September 2018

Ups ! ...

@evoskuil or anybody else... do you know any good grammar parser systems in C++? I'm currently using flex + bison

@narodnik you could look at Boost Spirit

I'm wondering, would it be possible to replace the logging solution in libbitcoin? What I mean by that is, could I supply my own logger that is then used also by the library? My app has different logging requirements and I would be interested in structured (json) logging

@vdeurzen Of course it's possible, you have the code 😉

hehe 😛

If you want structured json, I don't see how else you could get it without changing where it's used

The log system has configured sinks. However it is intentional that log line format is unstructured.

Hmm, I could possibly change the sinks indeed, that might be exactly the right approach @evoskuil and @neillm that would be a good structure to start with 🙂

@pmienk implemented the current log system.

@pmienk did you consider using fmt rather than boost::format?

Hello, thanks for inviting me.

Hi @Daniel Krawisz how can we help?

Oh I just want to be able to talk to you if I need anything.

Cool, yes we are already working on incorporating BCH, LTC and FTC into node and server config. @Fil is leading that work. There should be no changes required in the wallet APIs.

Oh well I'm really glad to hear that then

We do have a design principle in place for SPV, but it’s very loose.

Do you have the spv design principle written down anywhere?

No, but @Mlmikael can probably lay out the principles for you. He suggested it a while back.

Ok thanks

This pertains to a light node design, not the wallet store. For current work on a wallet store/API you might contact @narodnik .

Sure, I don't know anything about that.

We are also working to integrate Electrum and Libbitcoin Server via the ZeroMQ interface.

Oh cool well I know Justus Ranvier loves zeromq

@neillm is working on the websockets (and JSON RPC compat) server interface, which sits on the ZeroMQ interface.

Thanks!

@Daniel Krawisz are you working for Stash Crypto / Chris Odom ?

Yes

Still with Open Transactions ?

Bitprim has written it on a fork.

Oh ok cool

It is libbitcoin, with some forward features.

Oh ok great.

They are front-running some features that will take us longer to incorporate because of our additional requirements. But happy to have them doing it.

Cool well I'm working on some strange ideas but I think that I'll be able to put the wallet together that I have to make and it'll work.

@evoskuil I've been re-thinking your proposed solution for safety regarding the derivable values in bc::settings. I came to the conclusion that if we use standard C++ encapsulation mechanisms we can get it all: safety and performance. So we have non-derivable values upon which other (derivable) values depend. I would make both of them private members and only make them accessible through getter methods. For the non-derivable values we'd then implement setter methods that will also update the dependent values. Consequently, the derivation of the values would be an one-off computation rather than recurring for each use. This implementation would still be compatible with boost::program_options because the bc::settings setter methods can be passed into the boost::program_options's notifer method which then causes the value to be set according to the configuration. The only drawback would be that bc::settings would not be perfectly consistent with the other settings classes. But with the current solution it's not either.

@evoskuil when will you get around to do some review? Would be nice to get this one off my plate https://github.com/libbitcoin/libbitcoin/pull/1022

Sorry, been on the road.

No worries. And thanks for the quick review 👍

Back home for the next 12 days, then Honeybadger, then Scaling.

@evoskuil top of the file in src/math/hash.cpp:

#include "../math/external/crypto_scrypt.h"
#include "../math/external/hmac_sha256.h"
#include "../math/external/hmac_sha512.h"
#include "../math/external/pkcs5_pbkdf2.h"
#include "../math/external/ripemd160.h"
#include "../math/external/sha1.h"
#include "../math/external/sha256.h"
#include "../math/external/sha512.h"

Is this a mistake? Why not simply "external/..."?

No, why does it seem like a mistake?

@narodnik It looks like a relative path to what's included in the CPPFLAGS: src_libbitcoin_la_CPPFLAGS = -I${srcdir}/include from the src dir, where ${srcdir} is ".". So backing up one from src/include gives src for src/math/external/*.h

@evoskuil well surely #include "external/..." is sufficient?

root is sufficient 🙂

(hash.cpp and external/ share the same directory)

It's a hierarchy.

ok

These are the external files used by math classes/fns

@neillm generally for -I includes we use < ... > but it's a style choice

These headers are not supposed to be exposed though, so I thought that was the distinction. Not sure

"" is used for local includes, as opposed to published.

Where should I look for a method that gets me the latest block of the longest chain? I see that block_chain has is_stale() but I would like to perform an action already when I'm N blocks from the top ...

This is not possible, in any implementation.

The network doesn't have a chain. A chain exists only in the databases of individual nodes. There is no way to know if a block obtained from any node is on any chain at all unless you validate its full history. There is also no way to know that a block on a validated chain is on the longest chain unless you validate each candidate chain. And given that you may not see every chain, you can't actually know even if you validate every chain you see. The best you can do is validate each chain you see and from that obtain the strongest block. In libbitcoin this is the top block of the confirmed chain.

ok, makes sense, thank you @evoskuil

Are you working in v3?

bitprim at the moment, should indeed check the v3 libbitcoin (I'm currently 18.7% memory usage (24GB total), 600K blocks received on testnet)

👻

master is staging for next release, and is still early. V3 memory usage is very flat, increasing slightly with overall chain/store size. The store is memory-mapped files, and the operating system is fee to use any available memory for caching pages.

I was expecting something like that indeed, it seems to have started levelling off a bit indeed

From my limited testing it’s about half the memory usage of bitcoind during initial block download. And since we don’t have a memory pool it should do even better after.

it's pretty quick so far

v3 initial block download is actually very slow. that's a major piece of work in v4.

looking forward to seeing that work 🙂

@Daniel Krawisz, we'll need to wait for V4 to get stable, it will have header sync.

How does the Bitcoin address index work in LibBitcoin? I see the "fetch-history" aka "fetch_history3" call from Bitcoin Explorer and Client to the Server, which leads to this logic in libbitcoin-server's src/interface/blockchain.cpp: node.chain().fetch_history(address_hash, limit, from_height, std::bind(&blockchain::history_fetched, _1, _2, request, handler));

It looks to me that there's a configuration setting [server.]index_addresses which enables/disables the address index, and, it's enabled by default. I don't find this setting included in the documentation at https://github.com/libbitcoin/libbitcoin-server/wiki/Database-Settings , shouldn't it be there?

(@evoskuil that setting is about modulating the address index' behavior, not enabling/disabling it)

same thing

how could you use that setting to disable the address index, by setting it to #7fffffff...?

@evoskuil question, what exactly does address_database index over? The code in the block_chain::fetch_history that invokes the address_database and then translates address_database's response into the data format that the history_fetch_handler receives, I don't understand it so it's here https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/interface/block_chain.cpp#L1538 :

yup

@evoskuil, does address_database resolve **UTXO:s** (rather than Bitcoin transactions), and specifically, both "unspent" and spent UTXO:s?

it indexes payments to standard addresses.

What does history_fetch_handler actually get?

payments to/from a standard address

@evoskuil, does the address database allow lookups by height interval (in a particularly performant way, for incremental syncing of transaction/utxo history)?

@MlmikaelHow far away is version 4?

Planned end of year release.

Oh wow that's awesome.

The major work is the continuous parallel block download, which is largely complete and merged, though still working out issues. The websockets work is also largely complete, as is the tx metadata pool (which is merged but not integrated yet).

Yeah I remember you talking about that at a conference.

Any video of those talks ?

Yeah, youtube, Satoshi’s Vision conf has a good talk.

👻

cool, thx!

We implement all of the network types, but purposely avoid implementing protocols for bloom filters.

Why not?

Privacy, DOS, client-server protocol in the P2P network...

ok

It's so problematic it became the primary justification for introducing identity into the p2p protocol.

Where are the network types implemented?

yeah, proactor can be hard to follow.

Oh there we go. Thanks!

👍

@evoskuil if the database get's corrupted, is there anyway to fix it without having to resync the entire chain?

No, there is no database repair. If it’s corrupt there is no way to know how it is corrupt.

pity, thx though 🙂

This is the case with Core as well, they just don’t deal with it.

wouldn't that be a nice feature to have though?

No, the store is a cache of public data. It would be a design flaw to implement any feature that traded performance for general recoverability.

Sure, but having recoverability doesn't have to decrease performance imho

Yes it does.

why?

First, you cannot detect arbitrary corruption. The hardware can fail, and you use hardware to detect failure...

imho, I think that cost might actually be low enough compared to the time needed for a sync, but I haven't looked into it enough to say more about it

There's no clear ideal ratio for cost tradeoff. How much performance (profit) should a miner sacrifice so that they don't have to restore the chain if a server is hard stopped? How much time is actually required to restore the chain from the network?

that would help indeed

That’s easy, just don’t kill the process. You cannot back up the libbitcoin store while it is open for write. Also, some more info here: https://github.com/libbitcoin/libbitcoin-server/wiki/Frequently-Asked-Questions

That is a very informative faq

Will the adoptions of bip150 and 151 on bitcoin core (as a soft fork I imagine) affect non conforming nodes directly or only in a discriminatory manner?

Those are not forks, they are additions to the P2P protocol.

Oh

If widely adopted they will lead to privatization of the P2P network.

But not as an immediate effect. That's all I meant by soft fork I suppose

That's not a soft fork.

👍

@evoskuil @vdeurzen it should be possible to implement a self-consistency check on the data in LibBitcoin's blockchain files, without doing writing. the major expense involved in a LibBitcoin deployment is SSD wear, right, you want the SSD to work for several years even if there were. the self-consistency check could self-heal either by repairing, or by truncating/cutting files at the block height where inconsistencies were found

You would be reproducing code integrated into the devices themselves.

Even if it was possible to detect a failing drive with a "consistency check" it is not possible to stop in the midst of what needs to be an atomic commit (across multiple files) and expect the resulting store to be consistent. But again, this should not be possible since the drive should itself detect and disable failing sectors. All you should notice is your drive getting smaller.

usually the correct way is there to exist a lockfile when the file is opened but not closed properly. This would be implemented in addition to the boost lockfile currently used that prevents concurrent write access to the database.

This is already implemented.

The novel application of BLS aggregate threshold signatures could be a major game changer to the internationalization of existing centralized equity issuance processes.

FYI, I'm on vacation till the end of this month. Please contact @lclc if you need something from the FDF during this time. See you at scaling @evoskuil!

@evoskuil what's WebSockets used for in LibBitcoin?

@evoskuil Bitcoin Core does have some logic to rewind the last N blocks after an unexpected shutdown. i also did not see any logic to recalculate the whole Bitcoin blockchain as a self-validation measure in Bitcoin Core.

<higher prio topic coming below :)) >

@evoskuil no you're missing my point - it is the responsibility of a storage device (e.g. SSD) to keep its data store in integrity. Indeed it may not keep integrity. A filesystem (e.g. ZFS) may add a checksumming layer to storage device access, so that reads will fail if its underlying disk data was corrupted. In case of unexpected shutdowns in particular, self-validation to identify anything broken and rewind all affected, would make all sense - rather than for instance terminate and require administrator intervention which is LibBitcoin's current behavior. I agree that it not is reasonable for software behavior to be coherent if persistent storage breaks, just like it not is reasonable for software behavior to be coherent if RAM or CPU break. However, unexpected shutdowns happen, and also a user may like to run self-checking logics - in particular given the fiat cost of SSD wear for rewriting the whole blockchain, and the time it takes to resync the blockchain from scratch, some self-validation would make sense. I don't follow LibBitcoin's database models very well yet so I don't understand yet what degree of complexity they are. </higher prio topic coming below :)) >

> > This is the case with Core as well, they just don’t deal with it.

that’s not dealing with it, that’s denial.

The bitcoin blockchain is a cache, not an authoritative record. as such it’s requirements are different than general purpose databases. If there is a hard shutdown libbitcoin always knows whether the store is corrupt. Scanning the store is a complete waste of time. Validating a corrupt store does not make sense. It has nothing to do with the database structures.

@evoskuil @narodnik @neillm (as of v4:) can you please list to me the persistent stores and indexes LibBitcoin maintains now, as in, key(s) and value(s), and logical store order, and data format? Is it documented anywhere? Specifically: * Which are the databases (in the logical sense of database) actually now, what are their names? * How many copies of a transaction and of a block header does LibBitcoin store today, in what files&db:s? * Gossiped tx:es with sufficient txfee are pushed directly into the txdb? * Are transactions spread randomly accross a logical txid->tx raw content persistent hashtable, or is it stored as part of the blockchain blocks storage (tree structure)? * What is the actual block storage model, and what is the disk ordering, including for main chain vs orphan blocks?

@Mlmikael no, not documented.

@evoskuil i am looking at the code but also some things lack. would you mind writing a couple of lines here, briefly respond to each above? also - what are the conditions for storage of orphaned blocks and their transactions, are they stored indefinitely?

@Mlmikael The v4 libbitcoin-database has been heavily re-organized and is incomplete (subject to change before release). As mentioned, the code is the best place if you'd like to track it (generally true anyway).

@neillm yes reviewing the v4 code, also yes understand some things will change. i understand there's no point in writing documents when the code is in flux. do you have a minute to describe the approx databases and their formats now?

No, I haven't reviewed the latest changes

@neillm formats are changing because the header-first sync requires it?

Not sure, I haven't contributed to libbitcoin-database (yet)

ok

No, I'd have to look at the code

Does LibBitcoin have any "defragmentation" logic where it's moving around data to put the data relating to the (now-) main chain in a sequence?

Without looking at the code, I can't say. But I don't think so, I think it re-indexes existing data, rather than moving everything around (inefficient and unnecessary)

libbitcoin::database seems unaware of running in testnet / regtest, which means it will store addresses in mainnet format rather than testnet format.

I just talked to people about which full node Bitcoin nodes are out there and indeed it's only Bitcoin Core and LibBitcoin (!). BTCD apparently is a line-by-line transcription of an old Bitcoin Core version to GoLang, mostly used to drive Lightning development.

I seem to also have an interesting versioning issue with transactions within a mined block, the address cache seems to have been parsed with a non-testnet version configured 😕

s/parsed/populated/

@evoskuil is there any way to identify / traverse / access stale/orphan blocks?

Also, what meaning does the word "candidate" have (it's regarding blocks) in https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/interface/block_chain.cpp , is it when comparing total work done between two chains?

Continuous parallel block download requires it.

Previous block hash is contained in each block’s header.

No, libbitcoin never moves or removes a record that it has stored. Blocks form a tree. A “chain” is just an index into that tree.

Right, it would be a very significant deoptimization.

The database has no need of consensus rules. Addresses are not stored, address hashes are indexed. The format is unrelated to the coin rules.

The version value is not part of the node or server, it’s a wallet thing.

In v4 a weak block can be queried by its hash, and of course its txs and ancestors can be as well.

The candidate chain is a pow chain with more proven work than the confirmed chain, and is being downloaded and validated. Once it is validated it is reorged as the confirmed chain.

It’s my opinion that no fork of a large codebase will survive, especially as a language translation, unless the original codebase dies off.

The “primitives” templates implement the entire database. Nothing is read or written if not through them. These implement just a small number of data structures. These rest on an abstract memory model, implemented in the “memory” types. In test cases we implement memory as a data chunk, in production we implement it using operating system file mapping. The “database” classes organize the templates and memory model into specific files each representing a complex data structure.

The ADTs are: array, hash table, and linked list. These maintain data elements as record (fixed size) or slab (variable size). There are three arrays of records (candidate and confirmed indexes of heights and the tx index of tx offsets for a block), one record hash table (headers) one slab hash table (txs) and one record hash table of linked lists (addresses).

Hash tables also use the linked list for maintaining the set of hash conflicts.

how do you enumerate a block's children? you have one main chain child, and then >0 stale (or orphan) child blocks right

this has for instance this consequence right: transactions are stored on disk, in order of arrival, yes? (and not in order represented in blocks, or in order of hash(txid) or the like.)

What effect does this phenomenon have on LibBitcoin? Anyhow Bitcoin is different from traditional software compatibility problems, in that the introduction of implementational incompatibilities is greaetly restrained from the fact that Bitcoin's ability to add new features which could cause such incompatibilities, is very low due to protocol conservacy.

meh

@evoskuil the "primitives templates" and "memory types" are classes, or are logical memory regions with their produce/retrieve logics implemented as classes (or both! ?), and the block_chain file is made up of a sequence of those primitive/memory structures only? (By "in test cases we implement memory as a data chunk" you mean your logics are operating on the memory addresses malloc:ed for the data chunk object, instead of on mmap():ed memory addresses.)

Cool! ADT here means Abstract Data Type yes? So block_chain is.. mostly append only? Just to expand what you said above: block_chain contains these six structures?: * the index of height of candidate blocks (type: array of record) * the index of height of confirmed blocks (type: array of record) * the tx index of tx offsets for a block * block headers hash table, i guess key = block hash and value = block header data * transactions slab hash table, i guess key = tx hash and value = raw tx data * address index: hash table of key = bitcoin address hash, value = linked list

Ah yes, now I see what you mean 😃 : Yes BTCD forked Bitcoin Core and then translated it to another language. Indeed that is difficult to keep alive yes. LibBitcoin is a separate Bitcoin node implementation and has incentives for it per above, plus Bitcoin Core's obvious limitations, and more.

How do I enumerate which weak block children a particular block has 😄

What is the name/URL of the blockchain explorer that uses LibBitcoin again?

is it blockdozer?

You either read all headers and then monitor for new ones or you don’t do it. There is no scenario in Bitcoin that requires it.

An independent implementation would not exist if it hadn’t built up the necessary community to keep it alive.

aha you mean, the way to monitor weak block children is simply, listen for blocks as they come in and register their block hashes, and then track which of the previously registered block hashes are not in the main chain anymore - those are the weak blocks. That's the sole way to track them?

👍

Sure, but that’s just the creation of an index. It’s is not at all clear to me what scenario requires this blocktree children (vs blockchain child) index.

wait what do you mean?

What scenario requires this information?

There are an unlimited number of things that could be indexed. The choice to do so implies an important scenario.

wait, by "blockchain child" you mean "blocks within the main chain" and by "blocktree children" you mean "blocks within the main chain or stale chains", right?

The nature of the blockchain is a tree structure where one branch normally grows longer than all the others, and this is the general form in reality, a tree structure is what it is. The question is not what particular utility accessing the tree structure has, but instead, the tree structure is what it is and data structures should reflect that.

The cost of making a stale block is approx 80,000 USD. If someone spent this money to make a block, I like to inspect it. 🙂 Stale blocks are not a DOS vector.

Ok, thank you for your clarification that in the v4 API, a previously recorded block hash (that was the main chain then) can still be queried even when it's stale now! 👍

Blocks form a tree

Actually no, that’s not the question one asks in such a design exercise. You are describing the normal form of the data, not it’s access patterns based on actual use cases. It is never the case that implementation “should” reflect anything but usage requirements.

Then query for it by its hash. But again, what you personally would like is not what drives our design.

Yes, because it is not only useful but is facilitated by the fact that we never delete data (so low cost). This is not the same as navigating children.

Libbitcoin does not exhibit the recent bitcoind (Core, et. al.) consensus bug or DoS bug pertaining to internal double spend checks. As these checks do not require chain state they are performed in a context-free manner.

👍👍👍

Reserve Currency Fallacy (Baltic Honeybadger 2018): https://www.youtube.com/watch?v=66ZoGUAnY9s&amp;t=2147

Where is the equivalent of static const int COINBASE_MATURITY = 100; https://github.com/bitcoin/bitcoin/blob/master/src/consensus/consensus.h#L19 within libbitcoin?

@skaht https://github.com/libbitcoin/libbitcoin/blob/master/include/bitcoin/bitcoin/constants.hpp#L82

Thanks

Is it fair to say libbitcoin did not suffer from CVE-2018-17144?

@skaht Yes

Doing a little horn tooting about the need for diversity to Bitcoin Core.

Looks like all those other language transcriptions of Bitcoin Core versions 0.14.0 up to 0.16.2 in different languages will need to be updated.

Hello guys, I have a simple question. can anyone explain how to generate scriptPubKey from given address ?

@shota See: https://github.com/libbitcoin/libbitcoin/wiki/Building-Transactions

I read all of that but in those examples they always assume hat previous output is always pay to key hash. there is no checking according to address structure what kind of scriptPubKey is required ....

The part pointed out generates a script for a destination address

yes, but only for pay to key hash pattern ...

If you look at the second link i sent, the next method is for other patterns

let's say you are given an address. how can you check if address is paying to witness hash script (and scriptPubKey will be (00) + (32 bytes of hash(witness script) ) ) or not

I'm not the best person to answer here, but if given an address only, I don't know if you can make that determination. You need the script (or full tx that contains it) to determine that

so when you are the one who should make that transaction it's difficult 😄

I recommend reading this: https://github.com/libbitcoin/libbitcoin-explorer/wiki/How-to-Spend-Bitcoin

that's what I'm looking for. creating transaction without access to blockchain.

My hesitation on some witness addresses is that I think they are indistiguishable from P2SH or multisig, just by looking at them (could be wrong there). In any case, I don't think that's true of all witness addresses, particularly those that are obvious (bech32 encoded)

that's the address format for segwit

As for creating transactions without access to the blockchain ... I don't understand the purpose but ... I think the only way to do that is to ensure that you are the only one sending/receiving between yourself onchain, and even then, you can't know if any are ever confirmed. Maybe you have a better scenario in mind?

i am rebuilding libbitcoin with the install script on ubuntu 16.04. now when i try to build my project i get missing symbols for the libsecp256. adding -lgmp on the command line fixes it. don't see much changes in the install scripts, besides new boost and new zmq...

Hrm, that shouldn't be necessary. Check if there's a shared library conflict, or try to build statically

all the libraries are static (built it with --disable-shared) and still the same issue. not sure how to check for a shared library conflict...

If you built bs or bx, just use ldd

@pmienk may be able to chime in. Requiring a user application to additionally add -lgmp instead of just using the pkg-config --libs is a small extra step, but I guess it's an extra step. Perhaps it's not possible to detect which bignum library the secp256k1 build chose and therefore it isn't assumed?

both bx and my app (built with -lgmp) show all fine. can't check my other linux box right now. but i didn't recall having that issue

Anything built statically (bx/bs) should be shielded from this. The case I'm talking about is building an external application that uses libbitcoin, which uses libsecp256k1, which may or may not be using libgmp

i have external application that uses libbitcoin (hence libsec256k1), it seems to be linked statically (both my app and the libbitcoin libraries)

Right, but is the libsecp256k1 linkage to gmp also static? It appears to not be, if you have to specify it

correct!

That might be my mistake. I don't believe on static linking we're properly forcing pkg-config to invoke with --static, which may fix the problem.

@pmienk Cool, didn't know about that option, but adding --static to the pkg-config line does appear to include -lgmp on my system

i ran into this a while back thanks to cmake work, but wasn't thinking to add it to autotools. will augment

@zoranc In the meantime, give that a go

before i go digging through secp's config/make/etc, anyone know why they're getting a dependency on gmp?

It’s an optional perf optimization I believe.

autodetected it seems

adding --static to each pkg-config line fixes the problem too, looks better than adding -lgmp