March 2021

Who runs bitcoin.it?

Someone going by ccedk

Looking for admin contact info. My password expired and their reset doesn't work.

Ahah

I think I know him, just can't remember who it is.

Hm, saw that page but missed the IRC, thx.

Not super useful, but a channel is specified

Yup irc

looks good, thx

I noticed the deps lagging a bit

This dependency work is a rabbit hole

Maybe we can add openssl back in someday ...

I removed like 10, and you added three.

Yeah, that's probably the single biggest pain point

We've got several hacks to make boost with ICU compile, since they don't use package config to discover it (we do it for them), and boost itself is a completely non-standard build that doesn't export package config.

Unfortunately ICU is required for normalization, except in win32, where I wrote it in win32 natively (as normalization is native to the OS and boost doesn't link ICU statically in Win32). Boost also fails to link statically on Linux/macOS, but we work around that with a hack. They explicitly don't like static linkage for ICU, which is stupid. https://svn.boost.org/trac10/ticket/9685

I didn't write the BIP!

Dragging ICU into a BIP had to be the one of the dumbest things I've seen.

If there are other normalization options, I think we should be able to use them (like win32).

We do, but as far as I know nothing on linux/osx. Please feel free to find something!

I'll add it to the list for when I have time 😅 Been fried on weekends lately. Hoping it gets better later this year

👍

looks like belcher has bitcoin.it

Hello! Guys I see write_reverse() defined in some places in the sourcecode, but it is actually never used. what is it meant for?

I believe it writes bytes to a stream in reverse order.

To the best of my C++ skills it looks like write_reverse() is not used in any LB repo. Is there oe call to write_reverse() potentially being made by boost or by libstdc++, I presume not, or ??? At least there is no explicit call to write_reverse() anywhere in LibBitcoin

🤷🏻‍♂️

Is libbitcoin-system’s bitcoin/..../serializer intended to ever be used from outside of LibBitcoin? to my best awareness there is no legitimate usecase when a user of LibBitcoin would access the blockchain file directly, so therefore write_reverse() should never be used, because write_reverse() in LibBitcoin is always a write to the blockchain file?

The use is for writing bytes in reverse order. It's a common scenario when dealing with endianess issues. For example, bitcoin deals with hashes in both orders. And of course libbitcoin reads and writes blockchain data directly, we implement a full node.

hi, I switched from version 3 to 4 recently, and I replaced, say, bx fetch-balance msWHhBL1vLycmZtQ5M1j7xWuUYvienydfq with bx address-to-key msWHhBL1vLycmZtQ5M1j7xWuUYvienydfq | bx fetch-balance , also changed servers in bx.cfg to version 4 servers. Issue is balance shown is now different than what Electrum is showing. Before, in version 3, it was always agreeing with Electrum, tried 3 different addresses, they all show different balances than Electrum. Could you advise what I am doing wrong? Also, are there any mainnet libbitcoin version 4 servers?

No public ones other than an unofficial testnet server (as of yet). I believe a v4 mainnet server will be coming online at some point in the near future though. The balance indexing is different in v3/v4, with v4 having the more complete history

Thank you for your answer. Does it mean that Electrum shows correct balance for the entire wallet, but incorrect balances for particular addresses?

Unfortunately, I don't know much about electrum indexing at the moment, but heard it was supposed to be changing to the same kind we use in v4

Given v4 server isn’t complete (or hosted) it’s not surprising. Did you build and sync your own?

I see, makes sense. I was thinking about running my own soon, yet in version 3 didn't feel urgent need. Now I think I will do it.

Note that v3 cannot index for Electrum

Got a testnet v4 up for testing: http://testnet2.libbitcoin.net:29071/ (endpoint ports are 29091/29092/29093/29094)

If there are issues in v4 testnet it’s worth looking into, since it does sync. Tho there is no indexation of the tx pool, so only confirmed txs.

My addresses were confirmed few days ago, I could give some examples with possibly wrong balances

There could be an error in the indexed data (database) or the client (where the results are processed/accumulated). See if you can find where the problem originates

ok, looks like other addresses have similar issues, strange though that in version 3 I did not have any discrepancies for same addresses

^ Also to avoid another potential problem, best to try it on your own sync'd server

yes, I will do that, I see

V4 is good for resting the new interface, but blockchain/store is unfinished.

reason I switched to V4 was that V3 does not have electrum.cpp in wallet, only mnemonic.cpp, so I could not find working code for electrum mnemonic seed, only BIP39, in V4 both electrum and BIP39 work fine, so I am kind of reluctant to go back to V3

That won’t affect the change. You could run v3 bx for Electrum and v4 for client server work. If you need them integrated you could backport the Electrum command to v3 bx.

I see, I need it integrated, so I think I will backport

If you make the necessary PR to v3 libbitcoin-system I will do the code generation for libbitcoin-explorer and drop in the src and test changes.

ok, thanks, I'll try to do that

Note that v3 or v2 had bx Electrum wallet support. It was prematurely dropped in favor of BIP39 and then restored in v4. You might be able to rebase the old code.

yes, that's what I thought, that it was there before, I'll try to dig it up, also will look if and how Electrum updated their format since then

If I recall correctly, the old electrum code is unrelated to the new electrum code

Better to backport then.

I see, ok

Hi, I created pull request, https://github.com/libbitcoin/libbitcoin-system/pull/1199

I've completed the removal of qrencode, png and zlib external dependencies. Implemented our own two-color uncompressed tiff encoder from spec, with support for all possible size and shape images. Incorporated qrencode into our sources and redesigned the encoding conversion using our bit writer/reader classes. Added a complete set of tests, generated all manner of images, scanned them for verification and parsed some using online readers. So... the failing test has been repaired 🙂. And we now have full cross platform support for all features, and are back to just boost, secp256k1 and zeromq (for client-server libs).

Can png files still be output?

Nope, but you can trivially convert tiff to png.

Thanks, I’ll give it a good look in the next couple of days

Also, I removed the raw output format. The uncompressed TIFF output is sufficient to obtain the logical pixel data in a reliable manner, just using margin 0 and module pixels 1. I placed the image at the end of the file at a fixed offset, with the offset published from the tiff class header. Of course one can also use the API directly as well.

I didn’t check, but I believe the qrencode test was failing after upgrading qrencode because its output is a private format. It includes metadata that can change without affecting the module (visual) presentation. This is a good reason to not publish it from bx. The TIFF format (or any uncompressed image format), without margin or scaling, is a strict binary representation of the matrix of modules.

Thanks for looking into this.

👍 — master subject to interface changes.

Upgrade to boost 1.72, current secp256k1, zeromq and consensus sources, as well as removal of qrencode, png and zlib dependencies has been merged. NuGet packages for secp256k1 have been uploaded, though I'm not intending to update NuGet for zeromq. Instead I'm planning to move to vcpkg, which is c++ oriented and includes more current boost packages, and includes well-maintained zeromq packages. secp256k1 packaging continues to be a problem as the library is not versioned, so I'll end up building (only) and maintaining a package for it. Once this is complete I will create vs2019 project files and remove generation and testing of vs2013 (vc120/c++11 CTP). There are Travis breaks on all macOS builds due to a Travis change affecting boost installation. I'm going to resolve this next, and add builds for the Travis big-endian and Windows platforms. Also on the build roadmap is generation for mingw builds and VS code projects for all three platforms.

Nice work, will kick off new local builds this weekend to give it a go!

Why is the scriptsig field when spending p2sh-pwpkh input prefixed twice with a data push?

link?

segwit scripts can be embedded in p2sh scripts.

Dunno, I don't look to much at the bytes.

Ok, thanks - just asking out of curiosity

In segwit the signature script is in the witness.

Hello guys! I’d like to understand how can I add listener hooks and how to poll for the block hash and the block height in respect of the block header syncing process and block body syncing process, respectively. I’ll use these two datums (/data streams) to produce a 0-100% progress bar of the syncing process (a la the one in Bitcoin Core’s UI).

You can register callbacks for blocks and transactions (and headers I think too?) You can see this in the node implementation code (libbitcoin-node). At least that is what I did to get a stream of blocks / transactions to process 🙂

blocks are headers

I was talking about the C++ API and indeed I meant reorganization and transaction

Server (zeromq) provides broadcast block and transaction subscription endpoints (which can also be accessed directly via the C++ API): https://github.com/libbitcoin/libbitcoin-server/wiki/Transaction-Service https://github.com/libbitcoin/libbitcoin-server/wiki/Block-Service

The block objects are already in memory during the reorg, so there's only a performance hit in recreating the blocks as headers. You just read the header out of the block pointer.

Well but in the header-first sync part of the sync, a header is not a block yet right - so there are two phases of the sync?

There’s one continuous phase. Headers are always being requested and downloaded and blocks are being filled in behind them in parallel. We notify only when we’ve organized a validated block.

I see. Actually I’m using v4. I wish to get reports of block header sync aspect too. This will help me produce a better 0-100% progress bar to illustrate the sync status to the user in an UI

ah - for clarity: currently what do you use, what is the recommended way, to estimate the % sync status of of the blockchain?

Confirmed height relative to some target. The reorg message included fork height.

Valid headers, and txs valid relative to the confirmed chain (neither have to be in a confirmed block), are stored once in the header table and tx table, never moved or purged. We download unpopulated candidate blocks in parallel when the candidate chain is stronger. Txs are stored in parallel when blocks arrive, then validated and confirmed in block order.

What do you mean by “We download unpopulated candidate blocks .. when the candidate chain is stronger”>

An unpopulated block is one where we have its header only and are missing txs.

Is the log output configurable? My node instance prints me a lot of ‘Reject duplicate’ messages. I wonder, does this show that something is broken under the hood in my LibBitcoin setup, OR is this only LibBitcoin telling me that a block I already received from one peer LibBitcoin now received from another peer too, and that is supposed to happen all the time always right so it is not an error? I guess the latter applies, please confirm however as I did some deep plumbing in my LibBitcoin and wish to confirm to myself that my LibBitcoin works, I did not break it.

Yes, see config settings.

yes

v3 produces a lot of block duplicate messages, IIRC that’s not common in v4.

So you say the header-sync-first is implemented within LibBitocoin’s design of internally tracking a candidate chain and a confirmed chain. When you say “We download unpopulated candidate blocks in parallell when the candidate chain is stronger” - are you saying that the headers-first sync ONLY downloads block headers into the candidate chain, or that it downloads then into the candidate or confirmed chain depending on situation; Does “confirm chain” have the notion to it that all transactions have been downloaded already? To get the supposed total height of the Bitcoin blockchain during sync, should I use as data source, listening/polling for the height of the candidate chain?

Bitcoin is a tree index of headers, and blocks are indexes of txs per header. The data all has two levels of validation. Static validation is structural, and independent of any external state. Contextual validation is relations to other elements. The tree can be built with static and limited contextual validation (blocks to each other and txs to blocks). This relies on work to give storage a cost.

As an optimization we maintain an index of the confirmed branch and the next strongest branch, as more than one competitive candidate is extremely unlikely. A 51% attack is two competing chains. Furthermore, maintaining weak branches incurs almost no storage cost, so a decision to preclude weak branches below some arbitrary depth becomes important, though arbitrary.

So the full tree is a pruned (based on configured max reorg depth) memory structure of headers. Similarly, the tx DAG is limited to sufficient fee against currently confirmed ancestor tx outputs (or currently confirmed).

I am in v4 (master chain). I do indeed get a lot of [2021-03-24 15:24:15.729142] [0x000007cdc65cda40] [info] Rejected duplicate header [0000000064fc01456747cf00e6f9dd25cbdd3d444823c54e94cbed3681d9909a] from [24.222.43.8:8333] Does this hint at any problem? Is it just a coincidence, due to my Internet connection and proximity to peers such, that I get extra copies of same block from different peer such?

Oh, headers... yeah. As I said, v4 competes on headers, v3 on blocks. These are expected and necessary.

When you build the tree you ask each peer for its headers using an exponentially backed-off index of your own. The peer will send its headers starting at the highest known common block. This is standard p2p behavior. We do not want to allow a subset of peers determine the chain, so these requests are promiscuous. And there is no prioritization but response time (unlike block download) apart from the configured peer timeouts, since we also don’t want to let fast peers determine the chain. All peers compete to built the tree, and the set of headers that a peer has is not knowable until they are received. So if all peers are in sync there will be up to N times duplicate headers. This doesn’t always happen in practice since the fastest responder causes subsequent requests to have higher common points.

The longest header chain available in a Node, how do I poll for it, given LB’s persistence format?

You use the zmq interface.

wait wait, i meant block header sync only i guess LB Node’s normal metric of block height is blocks that have been downloaded and verified in full so that should be that the ZMQ RPC reflects - fully downloaded and verified blocks, isn’t it - so aside from that how do I get the highest header-only chain that the Node knows of?

Height is an index of confirmation. While headers in weak branches all have height, we would have to identify the branch to make the information meaningful. So we don’t have a height query for unconfirmed blocks. But if you know the hash you can query. Though we may not currently support emission of unpopulated or unconfirmed blocks. That’s a fairly trivial distinction tho, can easily be updated. Same for txs. If you have the hash you can request it.

So you are saying that LibBitcoin is not really indexing the block headers and not counting the height of the highest ‘block headers chain’ found. What about instead then that from my program I construct that counter function myself, by starting from the genesis block and then asking the LB Blockchain persistence, which child block headers it has - and then ask those block headers which children they have - etc - and this way i traverse the whole tree, would that make sense?

What do you mean by emission, you mean queryability in the LB blockchain query api?

The only real reason to maintain a height index is so that people may query by it. There is one optimization in validation (coinbase depth), but we store height with the block header, so the index is not actually necessary.

We should step back and start with your objectives. As I understand it you want to provide UI feedback regarding chain state, including currency.

As for unconfirmed header state, and unconfirmed block download state. These are not currently broadcast but could be. There is logically a pool of peers with header list requests outstanding, a pool of unique blocks requested (by header) but not received (unpopulated blocks of unknown size in the strong chain), knowledge of the top confirmed and top candidate block (therefore times and heights), a config (time) setting that determines when the stronger candidate chain is current enough to start downloading.

Builds are all back up to snuff on master, with updated/dropped dependencies and Travis config. Version3 is also updated, but I still have to work through warnings. I'm also in the process of completing the Electrum backport for version3 and will be backporting qrencode and tiff integration after that. The current version3 now has a common file system configuration (in -system) and the -build support for version3 and master has been consolidated to just a different generate.xml for each branch. Both branches now have the same set of dependencies and Travis builds, with the exception of mbedtls on master.

> The only real reason to maintain a height index is so people can query by it Querying by height is a disaster recipe, you want to query by block hash always, though sure keeping hammering the function “give me the block hash of the highest block you have” is core agrede.

Hi @evoskuil i have watched today this video that you mention about decentralization of mining pool, this something of my personal interest i would like to learn more on possible paths in order to make it happen.

> I don’t recommend node for anything else than a replacement for server. Yeah that’s kind of what we do, we’re running our own Node instance inside our own C++ program which then does blockchain explorer operations etc. on the Node. So kind of “embedded Bitcoin Core” usecase kinda you see what i mean

Not sure I follow.

Cool, @jungly is heading up this effort right now.

Ok, as long as you aren’t trying to scale that out, it’s optimal for a single host. Likewise Core isn’t much of a scale server.

But yes, a top query is useful and doesn’t require a height query.

The confirmed height index is mostly used to quickly identify the path through the strong branch for internal purposes, such as for generating a block requestor message.

@evoskuil Thank you for all this information and all your effort to contribute to my learning!

👍 thanks for being a libbitcoiner

> it’s optimal for scaling out. also Core doesn’t scale Yeah. I’ll implement some Bitcoin indexing a la colored coins, query service, RPC that kind of thing on top of Node

I think that was rule #2 when Amir first built it

@evoskuil just to confirm, LB uses bind() for callbacks rather than C++11 lambdas, because code conciseness is higher with bind() than with lambdas, right?

Yes

Hello guys! So I was checking bn’s code and I noticed that `close()` needs to be called from the main thread. Inside full_node’s code it leads me to check “join”. But I don’t understand what is “join” a reference to. Could you help me understand that?

// Close must be called from main thread.
if (node_->close())

// This must be called from the thread that constructed this class (see join).
bool full_node::close()

Join refers to the thread join. The process cannot terminate until threads are all coalesced, which join waits on.