March 2020

In an effort to get something running while my node syncs I'm connecting with the ZMQ client (following the examples). I have everything compiling, however, interestingly I'm getting the same transaction all the time when connected to the transaction_publishing port mentioned here: https://github.com/libbitcoin/libbitcoin-server/wiki/Community-Servers

nvm, I'm an idiot

Hi Eric, what are the specs of this beast?

64 core, 256 GB I think, not sure. It’s about three years old now.

I buy it 😜

That is a beast indeed #envious

The block subscription on testnet seems to stall from time to time, I've not received a new block even though there are 2 new since the last one I've received. Anything I can do about that?

Are the stored database formats between v3 and v4 compatible? I'm wondering if I can use an old back-up to get my node up-to-date a bit quicker

No

thx

You should be able to query while syncing.

I'm looking to push the latest blocks and transactions into another service, don't really have a need for older blocks or transactions

Does getting that working require recent blocks?

it's just a passive view on the latest events in the blockchain, I only care that I'm not sending invalid blocks or following a completely wrong chain

Confirmed blocks are valid, and the syncing chain in v4 is based on a current strongest header chain.

My code is essentially start node subscribe_blocks(my_call_back) subscribe_transactoins(my_tx_callback) in my_call_back: if node->chain()->is_blocks_stale() { return true } same for tx's when it's no longer stale I start sending data to the rest of my services

No reason for wait for recent blocks, the old ones are valid on the current strong chain.

I'm not sure I follow your point? The goal of this service is to show the latest events that happened so that I can have another service track deposits. I.e I'm not interested what is on the strong chain unless I'm at the head of that chain.

But you are coding right, all you need is test data. Just increase the staleness criteria in the config.

This would be for a production service, most of the coding is done, just need to get synced so I can start pushing events. So, it's not really test data that I need, I would like to get up and running on mainnet as soon as I can, but syncing in a container on a VM is slow (I accept that, this is the env I have to run on for now). So, I'm looking to figure out how I can either 1) sync faster (or smarter) or 2) get the latest block and transaction events from a different source (eg. ZMQ and connecting with the public nodes) while I wait for my slow sync to finish after which I will switch to my wrapped node solution

For the block_service (https://github.com/libbitcoin/libbitcoin-server/wiki/Block-Service) I find that after ~2 blocks the service just doesn't return anything anymore. @neillm you suggested I need to resubscribe, but the example only shows a 'subscriber' socket in zmq and an infinite loop, should I be doing something else ?

Just try again, I restarted the mainnet node, don't have time to look at at the moment, but can later

It worked for a bit, but stopped after 2 blocks last time 😅

The server times out the client in 10 minutes (due to my server config). The client would have to resubscribe to keep getting more after that

Ok, so an implementation using zmq should reconnect after ~5-10 minutes in order to keep receiving blocks? This did not seem required for transactions btw, they've been broadcast continuously since yesterday without any reconnect

Sorry, I was incorrectly thinking of the client subscribe calls, which are totally separate from the broadcast subscriptions. Should be good as-is.

Ok, I'll try again then 🙂

So, for a bit of robustness, could I use a zmq poller and then after some time out (say 11 minutes) reconnect? Or does that not make any sense?

No need to resubscribe for these

I had the python example running since 620041, it's still waiting for block 620042 using mainnet1.libbitcoin.net:9093

Use mainnet, which goes to mainnet2

Ok 👍

Odd, will have to look at later

thank you

5 blocks in the last 5 minutes on mainnet. Interesting (python client has been running for a bit now)

that seems quick 😅

Regarding a v4 full node, I'm still quite consistently running into some kind of loop / lock / deadlock when shutting down, which is making it very difficult to save a snapshot of the database. Anyone else seeing this?

From my gdb and printf debugging I'm seeing that p2p::close() is hanging, specifically it's stuck joining all threads in the threadpool, somewhere in there something is causing some kind of deadlock. I've seen this on my production nodes, my local dev nodes on my laptop and my desktop machine, it's quite reproducible, but hard to track down

Experimenting with weird ideas I changed the full_node code to close the chain first then the p2p connections, this seems to at least reduce the probability of it hanging. My guess is, without my changes, that one of the p2p connections is waiting for the chain to do something, the chain is 'stopped' and not doing much and as a result everything just hangs

@neillm Even connected with all 4 public servers and using a poller to check if a block has been published, I'm seeing 2 blocks and then silence 😕

Something sounds wrong. I started the python block subscription client earlier and it's still running without issue. Is there a way to test locally against your own server (generally always recommended if possible)

I had the block subscription running through the night, multiplexing all 4 servers but it stopped receiving blocks after 2AM 😕 I honest don't know what's breaking it, maybe it's a networking issue that's causing my socket to be disconnected? At least the ZMQ interface from libbitcoin-protocol doesn't seem to tell me if that is the case

testnet would be nice for testing, but I'm trying get something, anything up and running for production while mainnet is still syncing

Well if it’s deadlocking on shutdown it may simply be the deadlock. It doesn’t generally occur on shutdown, but the failure to close the locked thread prevents the shutdown.

Any ideas on how to track which thread that might be? I.e. any logging or tooling already in-place that I could enable ?

Blocks get three orders of magnitude bigger around that point.

ah, ouch

Actually a little earlier, around 300k

Yeah, I'm guessing that wouldnt help indeed

I debug into the deadlock upon close in Visual Studio, observing the call stack of each thread. At least one will stand out.

Which one? I have a pretty good reproduction path so I can try and track it, see if I can contribute back a bit 👍

That would be awesome. It’s in a libbitcoin-system class, used in subscriptions, where handlers are injected into the subscriber instance. Handlers are arbitrary code and execute under the subscriber lock. They cannot be allowed to renter the subscriber.

Ok, will dig into that while I switch to v3 and try syncing mainnet again 🙂

Thx!!

ah, because that would try to recursively lock the unique_lock?

There are several subscriber instances, in network, node and server.

I'll try to find out if any subscriber gets stuck here.

Maybe run node only (no server). Maybe you can isolate it to server or not.

I've only had it happen in node so far

Ok, well then it will happen in server eventually.

Happy to help. Thank you for the help explaining things 🙂

No prob.

Close is idempotent, it gets called more than once.

Read something in the boost docs that suggested it might be an issue, but it's quite likely that was in fact talking about contexts or services

I don’t believe you will find any actual issue in the thread pool, but I can’t comment on specifics from my phone.

If a thread is spawned it implies there is a close outstanding.

I made a python script that launchs and stops libbitcoin several times. I was able to probe that deadlock exist. I think I shared to Eric and Neil, about 2 years ago,...

Yes, that helped isolate at least one of the deadlocks in v3.

I don't remember where that script is. I can write it again

If a thread is being initialized it has a close that is registered after the initialization.

It is managed by requiring a close handler to be registered upon completion of any thread spawn call.

I just ran into something on my environment and I'm wondering if there is anyway of dealing with it configuration wise: Apparently (and shockinly) our environment (google cloud) doesn't allow us to connect to ipv6 addresses, but my host pool contains a whole bunch of them and this stalls the node as it's essentially going from peer to peer getting and failing to resolve them. Anything I can do here? Can I configure something to force only ipv4 addresses to be added to the host pool?

You should be able to configure ipv4 only on your system, outside of libbitcoin

For example: http://terokarvinen.com/2016/prefer-ipv4-on-ubuntu-16-04-lts-xenial-etcgai-conf-precedence-ffff0096-100 (summary is to prefer ipv4 to ipv6, rather than the other way around: uncomment line "precedence ::ffff:0:0/96  100" in /etc/gai.conf, if applicable to your system). I've done that on the community servers and haven't seen any issues btw.

@neillm Thx, I'll give that a try

hello

Odd. Try to also link -ldl

my bad I showed the one without that @neillm, same result including that

version3 or master?

I followed this so version3 https://github.com/libbitcoin/libbitcoin-system#debianubuntu

Add -ldl as the first listed lib, before boost

Something seems odd though, perhaps some environmental mismatch. -ldl shouldn't be needed on the static build actually

So libbitcoin and tests build, just not the example code?

it's that example code

Add -static to the pkg-config line

The default config file from libbitcoin server needs to be updated (minor change): /subsidy_interval_blocks/subsidy_interval/

it worked with static! @neillm

@narodnik PR?

interesting --static added -lpthread -ldl -lm while -lpthread -ldl were already there it gives the same error if I remove that second -lpthread -ldl part but still keep -lm

It's just telling pkg-config to use the static options

output difference is those -lpthread -ldl -lm

@pmienk may have more detail

and removing those two breaks it again (keeping only -lm)

@evoskuil https://github.com/libbitcoin/libbitcoin-server/pull/537

Thx!