Bitcoin full node and query server

February 2020

Archive of the libbitcoin Slack #general channel.

Sunday, February 2, 2020

lclc13:43
No, it’s false (like in the default config)

Saturday, February 8, 2020

N
narodnik08:35
Did you try contacting Eric? Possibly to give him server access to debug? @evoskuil
evoskuil08:45
I would not set up manual connections to Core seed nodes, they behave differently than Bitcoin nodes. Consider them ways to get addresses only, and not for Libbitcoin. Libbitcoin expects standard nodes as seeds, and we provide those by default. Core doesn’t want nodes syncing from its seeds, but the manual connections hold onto the seed nodes, preventing others if you are already at your total.
lclc12:51
I only added the manual ones later, after we didn’t get nearly any blocks
evoskuil15:26
Do you have a log without them?

Sunday, February 9, 2020

neillm01:35
@lclc Unlikely to help, but I kicked off a new v3 sync from scratch lastnight w/no checkpoints, 0 inbound, 4 outbound, and no local peers (so just random internet peers). I seeded from mainnet1/mainnet2. It's not complete, but currently at ~478950. So long as I don't run out of space, I'll let you know if any issues are spotted around 510K.
👍 1
skaht01:38
FYSA - almost have SLIP10 functioning for ed25519. Lessons learned could feed libbitcoin hd-new and hd-private code to accomodate SLIP 10.
(edited)
👍 3
4 replies · skaht · last Feb 18, 2020
skaht21:18
The bottom of https://monero.stackexchange.com/questions/9815/support-for-a-ledger-nano-s-recovery-plan-b/10922#10922 shows the piped scripting for how SLIP 10 can be implemented to support ed25519 to mimic Trezor’s Model T XMR key and address behaviors. The indices for m/44'/128'/0' were implicitly assumed to be hardened. The C++ code used to implement hd_hmac had a libsodium dependency.
Monero Stack Exchange
For a Ledger Nano S recovery contingency plan, how can one compute an associated vector of 25 Electrum wallet recovery words for each primary address using BIP 44 technology that is seeded from BIP...
skaht21:22
hd_hmac.cpp C++ · 14.6 KB
#include <iostream>
#include <string>
#include <regex>

#include <stdio.h>
#include <ctype.h>
#include <stdlib.h>
#include <string.h>
#include <sodium.h>

using namespace std;

static const int MAX = 512;                                // MAX input data-message size of 512 bits to hashed using hmac-sha512
static void      sprintkey( unsigned char*, const char* ); // Converts hexadecimal char* to a MPI represented as an unsigned char sequence

enum class Curve { undefined, ed25519, secp256k1, nistp256 };

/** 
    From libgcrypt-1.8.5/cipher/ecc-curves.c:

    --ed25519   = apply Bernstein's ed25519 curve   ( order = 0x1000000000000000000000000000000014DEF9DEA2F79CD65812631A5CF5D3ED }
    --secp256k1 = apply the Bitcoin secp256k1 curve { order = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141 }
    --nistp256  = apply the NIST nistp256 curve     { oder  = 0xffffffff00000000ffffffffffffffffbce6faada7179e84f3b9cac2fc632551 }
**/

int main( const int   argc, 
          const char *argv[] )
{
  const char    *root_chaincode_ed25519   = "656432353531392073656564",     // "ed25519 seed"
                *root_chaincode_secp256k1 = "426974636f696e2073656564",     // "Bitcoin seed";
                *root_chaincode_nistp256  = "4e69737432353670312073656564"; // "Nist256p1 seed";

  string         line;
  smatch         matches;
  regex          pattern_help1{     R"(-h)"          },         // help
                 pattern_help2{     R"(--help)"      },         // help
                 pattern_index1{    R"(-i)"          },         // index
                 pattern_index2{    R"(--index)"     },         // index
                 pattern_index_val{ R"([0-9]{1,10})" },         // extracts integer index to be treated as hardened
                 pattern_ed25519{   R"(--ed25519)"   },         // sets SLIP 10 defined          chaincode of root for the ed25519 curve
full file
skaht21:24
Most of the overhead was to parse the input arguments.
skaht02:50
Forgot to mention that only SLIP 10 ed25519 behaviors are functional for hardened index numbers.

Monday, February 10, 2020

lclc07:25
Thanks @neillm! The sync-speed problems already started earlier though.
1 reply · neillm · last Feb 10, 2020
neillm12:52
Ok, the sync completed.
evoskuil07:27
Your log showed lots of churning on manual peers.
lclc07:27
I start to think it might have been a problem with the server hardware, but anyway: Bad news: The project got killed by the client, so we don’t need that libbitcoin-server anymore.
The manual peers I added after we had the syncing speed problem
evoskuil07:28
Can only see what’s in the log.
lclc07:28
I’ll see if I can get more logs; someone in the management though it’s a good idea to immediately shutdown the server to save on costs
evoskuil07:28
Did you have any hard shut down?
lclc07:29
Maybe now, but not before.
But we never had to delete the lock file
evoskuil07:29
It’s generally pretty easy to see the cause of a stall in the log.
lclc07:30
I’ll restart the server
evoskuil07:31
Reducing the number of peers can help in v3, say 3.
It’s not an efficient sync and there can be lots of redundancy. But it shouldn’t stop.
Now that you have time come to Hanoi.
lclc07:51
ha, good point. Restarted now with 0 incoming, 3 outgoing, no manual peers. But just had a seg fault. Unfortunately someone did a hardshutdown so the blockchain database is probably corrupted now (tried it anyway by removing the lock file).
restarted again
I can give you access to the server if you give me your SSH key if you are interested in checking the problem, but we are going to kill it anyway later today otherwise.
syncing is already much better now with less peers it seems, but still slow
evoskuil08:22
Unfortunately I won’t have time for troubleshooting until after the conf. V3 sync is slow.
It’s optimized for query but not IBD.
lclc08:23
Ok. Next time we’ll use v4 🙂
vdeurzen13:05
With the v4 sync, after header sync and downloading + validating all blocks on test net I get the following warning: 12:57:18.371577 ERROR [node] Internal failure generating block locator for [167.71.203.100:18333] object does not exist Seems to be part of the header_in protocol, anyone seen this before? I'll do some debugging to see if I can find it
neillm13:06
^I've also seen that before. Not sure of the cause
vdeurzen13:08
I'll retry the sync with flush = true, maybe it's not written to disk and having trouble?
vdeurzen13:17
Seems to imply that the candidate database cannot find the block at the top height

Wednesday, February 12, 2020

evoskuil02:37
Flushing wouldn’t make a difference. The database is consistent independent of whether it is flushed to disk.
Seems like a bug.
V4 is designed for optimal sync, but optimizations not fully implemented. There are not known bugs except what’s documented, so pls submit issue.
vdeurzen09:57
I'll debug a little bit more to get you the required details, from what I can gather so far:
  1. It starts after validation of all blocks (which I'm not a 100% sure finished fully)
  2. In the logs I see a new peer connecting, a transaction handler being called, and then immediately a message stating the failure to generate a block locator (I'm thinking it might be a not-yet-propagated block, validation being stuck, or an invalid request)
  3. I don't see any new blocks coming in at this point.
  4. Transactions are coming in just fine
vdeurzen13:22
I get the feeling there might be some kind of deadlock issue in the threadpool implementation in libbitcoin-system. I've had a number of debug sessions showing p2p::stop() ... thread::join_noexcept() waiting on a whole bunch of thread_pool::spawn_once() ... scheduler::run(), with nothing happening 😕

Thursday, February 13, 2020

vdeurzen17:10
I think I've figured out the issue, I'm not sure it needs to be a github issue, but let me know if you want me to write it up:
My steps were, start node with only a peer, no inbound or outbound connections.
  • sync headers
  • stop node when it starts downloading blocks
  • enable inbound and outbound connections
  • get parallel block downloads
  • When blocks are done, we see the error
I stepped through in a debugger and it looked like at least one thread was dealing with a chain reorganization. Any protocol_header_in instances in other threads, upon startup, requested a particular header, which is then not found and the error is printed.
So, this is not, I understand, a great way to sync a bitcoin node, and I expect that given time it would calibrate and find the correct chain. I've re-synced with mostly default settings and did not encounter the issue.
I followed the previous steps because I read something in this channel that indicated that might be the quickest way to get a node synced up. However, upon reflection that is probably not necessarily the most correct way?
The deadlock issue remains open, oftentimes when I try to stop the node it will eventually hang and never stop, even after leaving it for several hours
evoskuil17:24
Are you restarting the sync after a deadlock?
Not sure by what you mean by “not a great way to sync a node”. A bug is never the intended way. One thread handles the reorg for the block that caused it. Reorganize are sequential operations.
vdeurzen17:33
What I mean, I'm syncing from 1 source, so I'm always going to get a somewhat one-sided 'truth' of the chain
I have had to kill the sync each time after a deadlock, using kill -9, which, when done too quickly, causes a broken database. Plenty of times I've been able to remove the lock files and restart the sync without issue
The deadlock seemed to be between the threadpool stopping and a resubscriber, or between the threadpool stopping and a spawn-once, at least from what I remember from the gdb session 🙂
evoskuil18:01
V3 uses the same concurrency model and doesn’t deadlock, but there’s a lot of new code in V4 and I haven’t had time to track down the deadlock. It’s almost always the case that a deadlock manifests at shutdown, which is extra fun to troubleshoot given the unrelated flushing delay at shutdown. But it’s pretty easy to see the distinction by looking at perf counters.
The block locator error message implies that the code believes the store is in an inconsistent state. This probably isn’t the case. I doubt it’s the cause of the shutdown deadlock. While concurrency is broad in Libbitcoin, there are very few places where the code executing under a critical section can not be easily visually verified to terminate. It’s the other couple of cases that can cause deadlock given reentry into the critical section.
Locator generation is pretty simple. That aspect of your issues should be easy to resolve. It’s probably block state being used inconsistently.
(edited)
vdeurzen22:59
I had the same deadlock issue in v3, discussed it here about a year ago, it's entirely possible I'm using the library incorrectly, but this is something I've had for a while
When syncing with a 'normal' set of peers the locator issue does not reproduce (so far)
let me know if I can be of help 🙂
evoskuil23:09
No, you haven’t been able to remove the lock file without issue. It’s a broken database. anything that happens after that must be discarded in troubleshooting. The database is corrupt, whether you see it immediately or years from now.
Any result you have reported after removing a lock file is not usable information.
If you did this in v3, same issue. We’ve never seen deadlock in years of running four v3 community servers.
(edited)

Friday, February 14, 2020

vdeurzen19:13
Like I said, entirely possible I'm doing something wrong, but I ran into it many times, may also be a linux configuration issue of course, I don't know. There were, at the time, too many variables to check exactly what was going wrong, and a backup was a good enough solution.

Saturday, February 15, 2020

evoskuil03:54
@vdeurzen I’m only saying that if you delete the lock file and continue, you are definitely doing something wrong. Otherwise I don’t see any issue with the approach.
I added the hard shutdown detection in v3 because we were getting reports of all sorts of unexplainable problems, and people couldn’t be relied upon to know if they had corrupted the store. After adding it those problems all went away, unless people were deleting the lock file.

Monday, February 17, 2020

vdeurzen10:32
I didn't really expect the store to work after that, but a few times it didn't crash outright 😜

Thursday, February 27, 2020

vdeurzen21:30
What might be wrong if my mainnet sync, using mostly default parameters is taking almost 36 hours by now and seems to be crawling forward? I'm using v4 and have an SSD for storage, 15G of ram and 15 cores (most of which are not being used)
evoskuil23:53
What does the log say it’s doing?

Friday, February 28, 2020

vdeurzen10:21
It's validating on one instance, seems to be happening sequentially and rather slowly, the other node is still downloading blocks, but very slowly, log output is minimal. I'm seeing lots of outbound connections being made, some transactions coming in, but that is about it
evoskuil10:28
Yeah, that’s the unfinished part
vdeurzen10:43
Ok, so this is expected then?
In that case, how long does it take for you (roughly) to sync the full mainnet chain? Then I know roughly what to expect 🙂
Is the plan to do the validation fully in parallel?
evoskuil11:02
It downloads in parallel and stores with pow and other static validation against the header chain, but full validation must still be competed sequentially. It currently reads each block in order to do so.
So the lack of sufficient caching makes the validation stage pretty slow.
Neill might know better than me. But if it’s still validating it’s working correctly. It can also get delayed if it’s missing a single block in order, but should complete downloads quickly. I’ve done full download in about 15 mins.
vdeurzen12:03
That sounds similar to my experience, not 15 minutes but certainly much faster than previously
evoskuil12:20
Yeah, my test platform is a beast.
1 reply · fpelliccioni · last Mar 02, 2020
fpelliccioni08:06
Hi Eric, what are the specs of this beast?
vdeurzen12:29
😄
I'm wondering, for my particular use case I am mainly interested in kind of 'seeing whats happening on the network'. I want to be sure I'm on the longest chain, but I don't really care about any block except for the last 100 or so and any new one coming in. Could I create a custom node that 1) downloads the header chain and then 2) starts following blocks starting at current - N? Would I then also need to change the blockchain library to allow it to ignore missing blocks 🤔 ?
evoskuil12:32
You don’t care if the blocks are valid?
vdeurzen12:33
I think I do, but I don't ever query them after validation. My main concern is getting my service to start quickly and then start creating events for transactions and blocks
evoskuil12:34
Well you generally need all blocks to validate.
vdeurzen12:35
Right
Hmm, unfortunate
Alternatively, I could customize it such that I change blockchain + database so it can store a state that I can trust has valid blocks, but only stores what I need. That database can then be backed up and then I can start syncing again?
I.e. I would have a bootstrap process that populates the database and validates all blocks, but then it only stores the data I need, allowing the fast startup and the correctness 🤔
Does that make sense?
evoskuil12:38
Not if you want to be able to validate new blocks.
vdeurzen12:39
What would I need for that, the full chain?
evoskuil12:46
Yes, all blocks with outputs that a new block may be spending.
vdeurzen12:47
ah, right, that does make sense
And if I ignore that then I am susceptible to someone pushing double-spending blocks to me and I will happily accept them?
1 reply · evoskuil · last Feb 28, 2020
evoskuil13:40
Yes
vdeurzen12:58
I could build in a delay and verify with a number of peers that I'm getting the same blocks 🤔
1 reply · evoskuil · last Feb 28, 2020
evoskuil13:41
🤣
evoskuil13:40
replied to a thread: view
Yes
evoskuil13:41
replied to a thread: view
🤣
vdeurzen13:41
Yeah, I'm trying to actively reduce the safety of my solution 😛