Bitcoin full node and query server
June 2018
Archive of the libbitcoin Slack #general channel.
Saturday, June 2, 2018
Fil03:17
It doesn't 🙂 I'm just testing my patches (which I did on master) and I've noticed that the blocks are not syncing. Now, I know that it's currently expected so it's difficult to asses if all verifications and checkpoints are implemented correctly. I came to the conclusion that I need to test my patches on the
version3 branch. Can you confirm that?Eric Voskuil
Hi @Fil, why does the database require modification?
Fil03:24
@evoskuil we have discussed in Zug that some constants from
constants.h in libbitcoin should be made configurable à la settings.cpp in other libbitcoin libraries. Did you envision this change to go into V3 or V4?Fil03:33
I can get started on that if it doesn't conflict with the work of anybody else. Should all constants be moved or only the ones relevant for us?
evoskuil20:22
@Fil You are welcome to do so, but parameterizing those values into all uses will be fairly pervasive. Expect a lot of comments. All constants that could vary by coin should be parameterized. There should be a settings class analogous to that of other repos (we want no more than one per repo). There are trade-offs to passing a full settings reference (vs. individual values) into a class, method or function. I'd be looking for consistency here and generally we opt for passing individual values if there is just one or two and no expectation that the set would expand over time, and the settings object if the use is pervasive within another class. If values are used in more than one place in a class we prefer passage via the constructor, and otherwise also to simplify later method calls. The genesis block must also become a setting (base16 encoded canonical/wire serialization). Thanks!
(edited)@Fil Once this is complete I would pull it into master and then look to update bx, bn and bs configuration. The changes should be fairly straightforward at that point based on the existing patterns.
Sunday, June 3, 2018
@evoskuil what do you mean by named parameters require c++14? where does my code use them?]
@narodnik I’ve got an update pending.
A
Alex23:24
@evoskuil Thanks for the pointers to bx message sign/verify. I will make a note in our specs. It is good to keep the sign/verify code within the existing libbitcoin ecosystem.
👍 1
Monday, June 4, 2018
Fil04:34
@evoskuil Thank you for your detailed response. Given the huge variety of coins it's quite difficult to assess all required parametrizations. Parametrizing the ones needed for Feathercoin and also the obvious ones should be a fairly good start though.
Because I'm a newcomer to the codebase my plan is to start by parametrizing only a few parameters and then ask for early feedback. That way we can limit the amount of comments needed to get the development conventions and code layout across.
Regarding the pervasiveness of this task my idea was to remove constants, that have been parametrized, from
constants.h only after they are not in use in other repos any more (if any). That way the whole migration process can be split into multiple PRs which makes it more managable for everyone involved.Eric Voskuil
@Fil You are welcome to do so, but parameterizing those values into all uses will be fairly pervasive. Expect a lot of comments. All constants that could vary by coin should be parameterized. There should be a settings class analogous to that of other repos (we want no more than one per repo). There are trade-offs to passing a full settings reference (vs. individual values) into a class, method or function. I'd be looking for consistency here and generally we opt for passing individual values if there is just one or two and no expectation that the set would expand over time, and the se…
Fil04:44
@narodnik I'm with the Feathercoin Development Foundation which is based in Zug. We think that libbitcoin is a well designed and well maintained piece of software which we would like to become a core part of our ecosystem. The first use case is using libbitcoin-server as a electrumX replacement. This is related to the work @haarts is currently doing on the electrum side.
Amir Taaki
@Fil what are you working on in zug?
👍 2
Tuesday, June 5, 2018
evoskuil02:54
@Fil I still don’t have a response on my query regarding why feathercoin requires database changes. This was not anticipated and I need to better understand the issue.
@Fil it’s okay if you want to start with only your required paraneterizations, but these will need to get pushed through all repos before we commit to them.
Fil03:26
@evoskuil you did get my response on your query regarding why feathercoin requires database changes in my reply below. I assumed that when I reply to your message using the share button you'd get notified. Will use @-mentions from now on.
Fil Gop
It doesn't 🙂 I'm just testing my patches (which I did on master) and I've noticed that the blocks are not syncing. Now, I know that it's currently expected so it's difficult to asses if all verifications and checkpoints are implemented correctly. I came to the conclusion that I need to test my patches on the
version3 branch. Can you confirm that?
https://libbitcoin.slack.com/archives/C6LH74ACQ/p1527678724000139Fil03:29
@evoskuil I understand that the project needs to be in a consistent state. I wouldn't approach it differently. I will make sure to open PRs in all affected repos.
@evoskuil no worries 🙂
evoskuil03:30
Version 3 is production and you should work on that if you require a chain. I should have master syncing in the next few weeks.
Sunday, June 10, 2018
3 replies · evoskuil, narodnik · last Jun 10, 2018
GitHub
These classes simplify EC arithmetic operations and should be two-way compatible with ec_secret and ec_compressed: ec_compressed Q = ...; ec_secret z = ...; ec_secret e = ...; const auto Z = Q * z ...
Wednesday, June 13, 2018
The wiki is in need of updating
Ah, ok the server heartbeat wiki page is blank/TODO
neillm12:40
We also have bx subscribe block/tx commands, but not heartbeat (e.g. https://github.com/libbitcoin/libbitcoin-explorer/blob/master/src/commands/subscribe-tx.cpp)
neillm12:42
If there's a need for a bx heartbeat subscriber, that could be added. Just wasn't sure if it would be useful.
I'm trying to establish if a connection has been sucessfully established in my Python client library.
I think fetching 'last_height' once should do.
(edited)Friday, June 15, 2018
neillm18:09
Yes, it does. Posted this today: https://github.com/libbitcoin/libbitcoin-server/wiki/Python-Subscription-Examples
Sunday, June 17, 2018
skaht02:11
BX can easily create P2WPKH Addresses. Look at the test vector from https://github.com/bitcoin/bips/blob/master/bip-0049.mediawiki#test-vectors and the bx commands below.
% echo cULrpoZGXiuC19Uhvykx7NugygA3k86b3hmdCeyvHYQZSxojGyXJ | bx base58-decode | cut -c 3-66 | bx ec-to-public | bx bitcoin160
38971f73930f6c141d977ac4fd4a727c854935b3% echo "0 [38971f73930f6c141d977ac4fd4a727c854935b3]" | bx script-to-address -v 196
2Mww8dCYPUpKHofjgcXcBCEGmniw9CoaiD2% echo 0000111122223333444455556666777788889999aaaabbbbccccddddeeeeffff | cut -c 1-32 | bx mnemonic-new
abandon above dust dust case often baby melt fever creek oven urban
The following mimics newer Trezor Bitcoin behaviors,
P2WPKH Address for M/49'/0'/0'/0/0:
% echo "abandon above dust dust case often baby melt fever creek oven urban" | bx mnemonic-to-seed | bx hd-new | bx hd-private -d -i 49 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-private -i 0 | bx hd-private -i 0 | bx hd-to-ec | bx ec-to-public | bx bitcoin160
9b4e7ab01095dc96f74676fd09bb44ec9857fb7e% echo "0 [9b4e7ab01095dc96f74676fd09bb44ec9857fb7e]" | bx script-to-address -v 5
3KN5WUE5KJNyn3usHkm4spWG33sUf2jWQCskaht03:01
Correction, the example above is for
P2WPKH nested in BIP16 P2SH Trezor Address for M/49'/0'/0'/0/0:
Saturday, June 23, 2018
Sunday, June 24, 2018
N
narodnik20:02
@evoskuil sorry for delaying on the pull request. will do that in these days, was very busy and travelling lots last few days
@neillm I built python bindings before for libbitcoin but they were difficult to maintain
I want to update them though again, and need to find a good way to manage them
GitHub
python-libbitcoin - Libbitcoin Python Interface [contributed library, not maintained].
The problem is maintaining the library against git is difficult, maybe against the releases is better, but still there needs to be a way to track API changes between releases so the Python bindings can be updated. It works by dynamically loading the libbitcoin library
GitHub
python-libbitcoin - Libbitcoin Python Interface [contributed library, not maintained].
which is based off this library https://github.com/libbitcoin/libbitcoin-c
GitHub
libbitcoin-c - Libbitcoin C Interface [contributed library, not maintained].
when is the next libbitcoin release coming out?
Release v4 is no later than the end of the year.
That will keep it in sync.
Monday, June 25, 2018
haarts10:47
Is the Block Service endpoint ready to use? I mean should I consider it stable? There's not much in the way of documentation thereof that's why I ask. (the python example is sufficient for me atm)
N
narodnik19:43
@evoskuil if I have an operator bool() defined in a class, does it need a corresponding function call?
or is the operator bool() alone sufficient
evoskuil19:47
The operator is sufficient. We have two patterns for object validity, is_valid() and bool(). Yours should match the wallet classes so that it can be their base.
N
narodnik19:55
ok but the operator bool for scalar is different to is_valid(), since the operator bool() also tests if the value is zero
and there is an internal flag called valid_ for the scalar. I can rename is_valid() and valid_ if that is not sufficient
and your other points also https://github.com/libbitcoin/libbitcoin/pull/969
GitHub
These classes simplify EC arithmetic operations and should be two-way compatible with ec_secret and ec_compressed: ec_compressed Q = ...; ec_secret z = ...; ec_secret e = ...; const auto Z = Q * z ...
bool ec_scalar::is_valid() const
{
return valid_;
}
ec_scalar::operator bool() const
{
return is_valid() && scalar_ != null_hash;
}
(edited)how do I show c++ snippets?
or is it without colour?
evoskuil20:22
If zero is always invalid, then we would just use it as the invalidity sentinel. If not then we would just have an is_zero() test independent of validity.
But it’s not obvious that an is_zero() test would be useful, as in that case zero would just be another valid value.
Thursday, June 28, 2018
Z
zoranc15:03
i am trying to use the libbitcoin for POC project in my organization, with a intention to make it open source eventually. however i am struggling to run the bs bitcoin server reliably. tried many different configurations with no luck so far. is there recommended set of tags for each library?
what typically happens is that bs does the initialization fine and then on the next run hangs forever. is there a way to debug it?
i am on ubuntu 16.06. tried those versions of libs: master, version3, tags: v3.2, v3.3 mostly with the latest boost build from source, but also with the system one
(ubuntu 16.04 i meant)
@zoranc For libbitcoin-server, try:
./install.sh --build-boost --build-zmq --prefix=/home/me/myprefix
as your user
Z
zoranc15:25
yes. prefix dir in my home was a must. even rolled my own install script. thanks a lot for recommendations, will try them now!
neillm15:26
Please don't roll your own install script unless absolutely necessary. The existing one should suffice in most cases
Let me know if you run into issues checking out version3 and running the install as shown above
Z
zoranc16:39
build went fine but the result is the same. initialization goes fine and then hangs when started next time. both on my local and cloud machines
the prebuilt binary runs fine
Which pre-built binary?
Z
zoranc16:41
linux-vdso.so.1 => (0x00007ffc1dfbd000)
libbitcoin-server.so.0 => /home/work/root/lib/libbitcoin-server.so.0 (0x00007f6870a08000)
libbitcoin-database.so.0 => /home/work/root/lib/libbitcoin-database.so.0 (0x00007f6870781000)
libbitcoin.so.0 => /home/work/root/lib/libbitcoin.so.0 (0x00007f68702ac000)
libboost_filesystem.so.1.57.0 => /home/work/root/lib/libboost_filesystem.so.1.57.0 (0x00007f6870093000)
libboost_log.so.1.57.0 => /home/work/root/lib/libboost_log.so.1.57.0 (0x00007f686fdb5000)
libboost_system.so.1.57.0 => /home/work/root/lib/libboost_system.so.1.57.0 (0x00007f686fbb1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f686f994000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f686f612000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f686f3fc000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f686f032000)
libbitcoin-protocol.so.0 => /home/work/root/lib/libbitcoin-protocol.so.0 (0x00007f686edf9000)
libbitcoin-node.so.0 => /home/work/root/lib/libbitcoin-node.so.0 (0x00007f686ead1000)
libbitcoin-network.so.0 => /home/work/root/lib/libbitcoin-network.so.0 (0x00007f686e760000)
libboost_program_options.so.1.57.0 => /home/work/root/lib/libboost_program_options.so.1.57.0 (0x00007f686e4e1000)
libboost_thread.so.1.57.0 => /home/work/root/lib/libboost_thread.so.1.57.0 (0x00007f686e2ba000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f686dfb1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6870cba000)
libboost_regex.so.1.57.0 => /home/work/root/lib/libboost_regex.so.1.57.0 (0x00007f686dca2000)
libsecp256k1.so.0 => /home/work/root/lib/libsecp256k1.so.0 (0x00007f686da7d000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f686d875000)
libzmq.so.5 => /home/work/root/lib/libzmq.so.5 (0x00007f686d5f1000)
libbitcoin-blockchain.so.0 => /home/work/root/lib/libbitcoin-blockchain.so.0 (0x00007f686d35a000)
libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007f686cfc6000)
libicui18n.so.55 => /usr/lib/x86_64-linux-gnu/libicui18n.so.55 (0x00007f686cb64000)
libbitcoin-consensus.so.0 => /home/work/root/lib/libbitcoin-consensus.so.0 (0x00007f686c929000)
libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007f686ae72000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f686ac6e000)
Also, is there any information in the logs?
Z
zoranc16:44
16:38:35.985346 INFO [server] ================= startup Thu Jun 28 18:38:35 2018 ==================
16:38:35.985365 WARNING [server] ================= startup Thu Jun 28 18:38:35 2018 ==================
16:38:35.985400 ERROR [server] ================= startup Thu Jun 28 18:38:35 2018 ==================
16:38:35.985418 FATAL [server] ================= startup Thu Jun 28 18:38:35 2018 ==================
16:38:35.985436 INFO [server] Using config file: "/home/work/root/etc/libbitcoin/bs.cfg"
16:38:35.985444 INFO [server] Please wait while the server is starting...
16:38:35.985991 INFO [network] Starting manual session.
16:38:35.986130 INFO [network] Contacting seed [mainnet1.libbitcoin.net:8333]
16:38:35.986247 INFO [network] Contacting seed [mainnet2.libbitcoin.net:8333]
16:38:35.986283 INFO [network] Contacting seed [mainnet3.libbitcoin.net:8333]
16:38:35.986303 INFO [network] Contacting seed [mainnet4.libbitcoin.net:8333]
nope. can't kill it with Ctrl-C
have to use kill =9
(i.e. ncat -vv mainnet1.libbitcoin.net 8333, or telnet mainnet1.libbitcoin.net 8333)
evoskuil16:49
When you say “wait for a while” what are you waiting on? What is the node doing? Does it complete seeding? How long do you wait?
the printout i pasted above comes. and that's it. somewhat different if i remove the bs.cfg file...
the bs -i finishes just fine
evoskuil16:52
The log above shows it seeding, but it would not be doing that if you initialized with a hosts file.
Z
zoranc16:52
@neillm i do connect to those even use tcp://mainnet.libbitcoin.net:9091 in my code with no problem
neillm16:52
Right. Also, i recommend commenting out mainnet4.libbitcoin.net, since it appears not reachable
👍 1
So, edit the bs.cfg, remove the hosts.cache, and try to re-run and paste output if not moving
init went good
started it...
Z
zoranc16:57
16:54:48.491469 DEBUG [server] ================= startup Thu Jun 28 18:54:48 2018 ==================
16:54:48.491545 INFO [server] ================= startup Thu Jun 28 18:54:48 2018 ==================
16:54:48.491566 WARNING [server] ================= startup Thu Jun 28 18:54:48 2018 ==================
16:54:48.491593 ERROR [server] ================= startup Thu Jun 28 18:54:48 2018 ==================
16:54:48.491608 FATAL [server] ================= startup Thu Jun 28 18:54:48 2018 ==================
16:54:48.491640 INFO [server] Using config file: "/home/work/root/etc/libbitcoin/bs.cfg"
16:54:48.491665 INFO [server] Please wait while the server is starting...
16:54:48.491774 DEBUG [database] Buckets: block [650000], transaction [110000000], spend [250000000], history [107000000]
16:54:48.492023 DEBUG [database] Mapping: "blockchain/block_table" [5200177] (4096)
16:54:48.492037 DEBUG [database] Mapping: "blockchain/block_index" [12] (4096)
16:54:48.492057 DEBUG [database] Mapping: "blockchain/transaction_table" [880000267] (4096)
16:54:48.492071 DEBUG [database] Mapping: "blockchain/spend_table" [1000000008] (4096)
16:54:48.492084 DEBUG [database] Mapping: "blockchain/history_table" [428000036] (4096)
16:54:48.492090 DEBUG [database] Mapping: "blockchain/history_rows" [55] (4096)
16:54:48.492101 DEBUG [database] Mapping: "blockchain/stealth_rows" [4] (4096)
16:54:48.492242 INFO [network] Starting manual session.
16:54:48.492369 INFO [network] Contacting seed [mainnet1.libbitcoin.net:8333]
16:54:48.492448 INFO [network] Contacting seed [mainnet2.libbitcoin.net:8333]
16:54:48.492486 INFO [network] Contacting seed [mainnet3.libbitcoin.net:8333]
So seeding is failing, and not timing out (30 secs by default)?
Z
zoranc17:00
17:00:04.759306 DEBUG [server] ================= startup Thu Jun 28 19:00:04 2018 ==================
17:00:04.759380 INFO [server] ================= startup Thu Jun 28 19:00:04 2018 ==================
17:00:04.759403 WARNING [server] ================= startup Thu Jun 28 19:00:04 2018 ==================
17:00:04.759438 ERROR [server] ================= startup Thu Jun 28 19:00:04 2018 ==================
17:00:04.759457 FATAL [server] ================= startup Thu Jun 28 19:00:04 2018 ==================
17:00:04.759474 INFO [server] Using config file: "/home/work/root/etc/libbitcoin/bs.cfg"
17:00:04.759483 INFO [server] Please wait while the server is starting...
17:00:04.759544 DEBUG [database] Buckets: block [650000], transaction [110000000], spend [250000000], history [107000000]
17:00:04.759767 DEBUG [database] Mapping: "blockchain/block_table" [5200177] (4096)
17:00:04.759780 DEBUG [database] Mapping: "blockchain/block_index" [12] (4096)
17:00:04.759801 DEBUG [database] Mapping: "blockchain/transaction_table" [880000267] (4096)
17:00:04.759814 DEBUG [database] Mapping: "blockchain/spend_table" [1000000008] (4096)
17:00:04.759827 DEBUG [database] Mapping: "blockchain/history_table" [428000036] (4096)
17:00:04.759833 DEBUG [database] Mapping: "blockchain/history_rows" [55] (4096)
17:00:04.759847 DEBUG [database] Mapping: "blockchain/stealth_rows" [4] (4096)
17:00:04.759985 INFO [network] Starting manual session.
17:00:04.761608 DEBUG [network] Seeding is not required because there are 1000 cached addresses.
17:00:04.761622 INFO [server] Seeding is complete.
17:00:04.761644 INFO [node] Node start height is (0).
17:00:04.761690 INFO [network] Not configured for accepting incoming connections.
17:00:04.761714 INFO [network] Starting outbound session.
17:00:04.761783 DEBUG [network] Connecting to [77.187.150.158:8333]
17:00:04.761852 DEBUG [network] Connecting to [115.171.61.103:8333]
17:00:04.761868 DEBUG [network] Connecting to [208.98.207.210:8333]
17:00:04.761880 DEBUG [network] Connecting to [45.55.197.77:8333]
17:00:04.761891 DEBUG [network] Connecting to [87.81.145.91:8333]
17:00:04.761902 DEBUG [network] Connecting to [27.17.73.66:8333]
17:00:04.761913 DEBUG [network] Connecting to [5.35.87.70:8333]
17:00:04.761924 DEBUG [network] Connecting to [74.195.176.159:8333]
17:00:04.761934 DEBUG [network] Connecting to [104.249.230.177:8333]
17:00:04.761944 DEBUG [network] Connecting to [24.134.16.117:8333]
17:00:04.762269 DEBUG [server] Applied address authentication to socket [query]
17:00:04.762331 INFO [server] Bound public query service to tcp://*:9091
17:00:04.762470 DEBUG [server] Connected public query worker to inproc://public_query
17:00:04.762689 DEBUG [server] Applied address authentication to socket [heartbeat]
17:00:04.762721 INFO [server] Bound public heartbeat service to tcp://*:9092
17:00:04.762820 DEBUG [server] Applied address authentication to socket [block]
17:00:04.762860 INFO [server] Bound public block service to tcp://*:9093
17:00:04.762952 DEBUG [server] Applied address authentication to socket [transaction]
17:00:04.762993 INFO [server] Bound public transaction service to tcp://*:9094
17:00:04.763033 INFO [server] Server is started.
(debug.log with hosts.cache copied before 2. start of bs)
Outgoing connections not going anywhere. Please post cfg file as well.
Z
zoranc17:05
@zoranc uploaded a file: bs.cfg
bs.cfg
# Libbitcoin Server configuration file [log] # The debug log file path, defaults to 'debug.log'. debug_file = debug.log # The error log file path, defaults to 'error.log'. error_file = error.log # The log archive directory, defaults to 'archive'. archive_directory = archive # The size at which a log is archived, defaults to 10000000 (0 disables). rotation_size = 10000000 # The minimum free space required in the archive directory, defaults to 0. minimum_free_space = 0 # The maximum combined size of archived logs, defaults to 0 (maximum). maximum_archive_size = 0 # The maximum number of logs to archive, defaults to 0 (maximum). maximum_archive_files = 0 # The address of the statistics collection server, defaults to none. statistics_server = 0.0.0.0:0 # Enable verbose logging, defaults to false. verbose = false [network] # The minimum number of threads in the network threadpool, defaults to 0 (physical cores). threads = 0 # The maximum network protocol version, defaults to 70013. protocol_maximum = 70013 # The minimum network protocol version, defaults to 31402. protocol_minimum = 31402 # The services exposed by network connections, defaults to 9 (full node, witness). services = 9 # The advertised services that cause a peer to be dropped, defaults to 176. invalid_services = 176 # The magic number for message headers, defaults to 3652501241 (use 118034699 for testnet). identifier = 3652501241 # Validate the checksum of network messages, defaults to false. validate_checksum = false # The port for incoming connections, defaults to 8333 (use 18333 for testnet). inbound_port = 8333 # The target number of incoming network connections, defaults to 0.full file
I suspect the build, specifically boost linkage. I would use the build that @neillm recommended, but use static linkage only. This will ensure that the runtime links only the boost version that you build into the prefix dir.
👍 1
Otherwise I would look into whether the operating system is somehow blocking process communication.
It's not an issue we've seen, but bad builds due to boost mismatching are common and produce undefined behavior.
@neillm Could you run an ubuntu build with latest v3 just to verify?
thanks a bunch for your fast reaction and great help!
👍 1
evoskuil18:11
Unfortunately we don't seem to be able to gain control over linkage in dynamic builds, though we do direct the build to the prefix dir for dynamic builds.
Or I guess that's another way of saying what you said
# Incorporate the custom libdir into each object, for runtime resolution.
export LD_RUN_PATH="$PREFIX/lib"
evoskuil18:27
So it makes the link and test execution work from the install process, but later you are hosed.
Z
zoranc18:28
yeah. will retry build on the same machine with the added path. also tomorrow with reversed path on the work machine
simple to test, LD_RUN_PATH="<path>" bs
and no rebuild is necessary to test that
Z
zoranc20:04
that didn't work. also with both set on command line, not in bashrc... i needed the rebuild because i've deleted the shared-lib executable
neillm21:15
@neillm uploaded a file: Untitled
Untitled
Interesting, a default install from libbitcoin-server on ubuntu 16.0.4 build without specifying static/shared and building boost/zmq, does in fact hang as described. I suspect it's a conflict with one of these files:
```
$ ldd bs | grep -v libbitcoin
linux-vdso.so.1 => (0x00007ffcbabe4000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f90bd4b6000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f90bd134000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f90bcf1e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f90bcb54000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f90bbad3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f90be7dc000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f90bb3be000)
```Friday, June 29, 2018
haarts11:57
Does the
subscription_expiration_minutes also affect subscriptions to the block publish endpoint?neillm12:06
Right, that one just loops forever until the program stops listening, or the service goes down: https://github.com/libbitcoin/libbitcoin-server/blob/master/data/subscribe-block.py
Z
zoranc12:31
i've ran the bs for a while. then stopped it with ctrl-c, then waited for it to exit. now when i restart it, it gives me: "... ERROR [node] Failure starting blockchain."
what might be problematic is that i built server, and THEN client and installed in the same PREFIX location...
neillm12:33
Hrm, if it shutdown properly, it should not have an error starting up. Were there other errors shown on shutdown?
Z
zoranc12:35
@zoranc uploaded a file: Untitled
Untitled
@zoran@zoran-lenovo:/mnt/disk2/bitcoin-bs/archive$ tail -30 debug-00006.log 12:11:05.395516 INFO [blockchain] Block [213048] 1 txs 1 ins 0 wms 0 vms 29 vµs 11 rµs 11 cµs 3 pµs 1 aµs 1 sµs 38 dµs 0.000000 12:11:06.311027 DEBUG [node] Connected block [000000000000006896ca760505253e9d5f2b5cbf488df6339ca74118adab5a4c] at height [213049] from [64.233.245.39:8333] (1073741894*, 1). 12:11:06.311070 INFO [blockchain] Block [213049] 651 txs 1135 ins 3 wms 6 vms 5 vµs 3 rµs 2 cµs 0 pµs 0 aµs 0 sµs 798 dµs 0.000000 12:11:06.347606 DEBUG [node] Connected block [000000000000024ed8ea1d19212755f9da14ab0844a3aac29ff482676b384d94] at height [213050] from [64.233.245.39:8333] (1073741894*, 1). 12:11:06.347649 INFO [blockchain] Block [213050] 574 txs 1036 ins 3 wms 5 vms 5 vµs 3 rµs 2 cµs 0 pµs 0 aµs 0 sµs 26 dµs 0.000000 12:11:06.392395 INFO [server] Stop signal detected (code: 2). 12:11:06.392452 INFO [server] Please wait while the server is stopping... 12:11:06.414436 DEBUG [node] Connected block [00000000000001f82c9114666bb075bd982c25a2821cea5db90460fa665b155a] at height [213051] from [64.233.245.39:8333] (1073741894*, 1). 12:11:06.414478 INFO [blockchain] Block [213051] 889 txs 1866 ins 5 wms 9 vms 5 vµs 4 rµs 1 cµs 0 pµs 0 aµs 0 sµs 28 dµs 0.000000 12:11:07.243047 DEBUG [network] Outbound channel stopped [64.233.245.39:8333] success 12:11:07.243090 DEBUG [network] Suspended outbound connection. 12:11:07.243095 DEBUG [network] Outbound channel stopped [104.220.88.225:8333] success 12:11:07.243120 DEBUG [network] Suspended outbound connection. 12:11:07.243105 DEBUG [network] Stopped block_in protocol for [64.233.245.39:8333]. 12:11:07.243135 DEBUG [network] Stopped block_in protocol for [104.220.88.225:8333]. 12:11:07.243140 DEBUG [network] Stopped block_out protocol for [64.233.245.39:8333]. 12:11:07.243155 DEBUG [network] Stopped block_out protocol for [104.220.88.225:8333]. 12:11:07.243170 DEBUG [network] Stopped transaction_in protocol for [104.220.88.225:8333]. 12:11:07.243161 DEBUG [network] Stopped transaction_in protocol for [64.233.245.39:8333]. 12:11:07.243180 DEBUG [network] Stopped transaction_out protocol for [104.220.88.225:8333]. 12:11:07.243183 DEBUG [network] Stopped transaction_out protocol for [64.233.245.39:8333]. 12:14:39.770556 INFO [server] Stop signal detected (code: 2). 12:14:39.812481 DEBUG [database] Unmapped: "blockchain/block_table" [358060221, 508018479] 12:14:39.845849 DEBUG [database] Unmapped: "blockchain/block_index" [1704428, 2332542] 12:14:40.029370 DEBUG [database] Unmapped: "blockchain/transaction_table" [5858239036, 8574300862] 12:14:40.079426 DEBUG [database] Unmapped: "blockchain/spend_table" [2477306592, 2784866076] 12:14:40.121216 DEBUG [database] Unmapped: "blockchain/history_table" [684040128, 902976168] 12:14:40.171303 DEBUG [database] Unmapped: "blockchain/history_rows" [3265374556, 4276004149] 12:14:40.196371 DEBUG [database] Unmapped: "blockchain/stealth_rows" [4, 4] 12:14:40.210474 INFO [node] Server stopped successfully.
cd ..
ops. sorry
block_index block_table flush_lock history_rows history_table spend_table stealth_rows transaction_table
i have database.flush_writes = false, maybe that's the problem...
neillm12:38
The presence of that
flush_lock
file is what's preventing the startup. Its presence indicates that there is corruption in the store. Unfortunately, you will need to erase this and re-init and start sync over. If you're tempted to remove that file and start back up, don't!
Well, during the initial sync, that is ok. Once caught up to the top height, definitely enable it
On spinning disk, it can be quite painfully slow
I recommend starting over with a new store, starting the sync, then interrupting it soon after to see if there is still corruption after proper shutdown. There was a zmq bug once that caused this behaviour
It doesn't look like it's happening here, but to sanity check
Z
zoranc12:42
i'll rebuild the full server again just to be on safe side. is there a way to build all libs at once?
All libs at once -- that's what install.sh does?
neillm12:43
Ah, no. What you can do is run install.sh from both libbitcoin-server and libbitcoin-explorer though. That's the closest thing I think
👍 1
I could see a benefit to a combined installer though. Make sure you use the same options to both installers (with regards to --enable-static --disable-shared, or whatever)
👍 1
The second time, you don't have to build boost/zmq, but you will need --with-boost and --with-zmq
skaht12:47
Generally speaking, don’t paths have to be supplied to the already compiled libraries when the --with-flags are set?
Boost has an option to, but the install.sh doesn't require that
skaht12:52
Interesting… I seem to remember a few years back providing explicit path on the Mac for already installed Homebrew packages. FYSA - Will be mucking around with building up from scratch a BS on testnet this evening and/or tomorrow on OSX.
skaht13:01
So assuming build and install directories remain the same the utility of the -with-flags for the appropriate bx, bn or bs*install.sh* is to minimize recompiling of libraries that have recently been compiled?
neillm13:03
The --with- flags tell the code what to conditionally compile in (e.g. --with-png will include the png related code). The --build- flags tell the installer what to build as a dependency
(edited)So you should always use the --with- flags you want, but you don't need to use --build- flags if you're installing to a target where you just built it already
evoskuil14:15
The only way the flush_lock can be left behind after process stop is if the process is terminated (killed). A controlled shutdown cannot result in a detected corruption.
This can’t result from hitting CTRL-C multiple times.
Z
zoranc14:33
weird then, either i made unconscious killall (unlikely) or there is a bug. can't reproduce it anymore - did 10 or so stops with Ctrl-C...
neillm14:34
I am aware of some unresolved ubuntu strangeness. I suppose there's a chance it crashed on exit
👍 1
evoskuil14:41
It’s not possible that the process terminated normally with the flush_lock in place. Either there was a hard shutdown/restart of the computer or the process aborted/died or was killed.
Enabling write flushing improves fault tolerance (at the significant expense of performance) in the case of uncontrolled shutdown (only) but does not improve reliability.
👍 1
Z
zoranc14:51
unfortunately i've deleted the logs so can't prove it now. but i do believe that there was a graceful shutdown in the log with "INFO [node] Server stopped successfully."
will keep the logs if it happens again...
haarts14:56
@neillm I'm starting to suspect the resubscription is necessary. I'm running the
subscribe-block script and I'm not getting any blocks after 529761 (there also seems to be an off by one error?)block 529762 is mined 16 minutes after 529761
evoskuil14:57
Resubscrition is not part of the broadcast endpoints, only address subscription, through the query endpoint.
Your client may need to keep itself alive.
evoskuil15:00
If the flush_lock file failed to delete there would be a logged shutdown error.
(edited)haarts15:01
but 'bx fetch-height' returns the heights as on a regular block explorer like https://blockchair.com/bitcoin/blocks
blockchair.com
Blockchair is a blockchain search and analytics engine featuring Bitcoin, Bitcoin Cash, and Ethereum explorers, as well as full-text search over blockchain data.
the blocks i'm getting with the subscription always lag this height by one.
perhaps it just starts at zero. prob that
The blocks I'm getting via the subscription are correct but lag one block
haarts15:07
idk. I'm deserializing the response into a CBlock from the python-bitcoinlib package: https://github.com/petertodd/python-bitcoinlib/blob/master/bitcoin/core/__init__.py#L577
GitHub
python-bitcoinlib - Python2/3 library providing an easy interface to the Bitcoin data structures and protocol.
the
(seq, height, block) tuple is like so (5214, 529760, CBlock(536870912, lx(0000000000000000000814ea61875b6e810c92ac7b0650705adbfb7c12efab61), lx(3f3a6b714764676f45f428de5b9889b36ea6f99c0abcee12093270a437de8bf5), 1530281897, 0x17376f56, 0x5b79bfc9))the first
lx in that representation should be the previous block hashit actually isn't ...
this might be logical mismatch
like I said, I haven't fully investigated.
@neillm you had to modify the script somewhat no?
that script I mean, yeah
neillm15:13
I'm not modifying anything. I'm waiting for the next block to see if the height matches bx fetch-height, as you indicated there was a difference
haarts15:14
odd. I had to make these changes to have it run at all: https://gist.github.com/haarts/a205fe70a26aa19d438b1492f47d19b6
I'm using Python 3.6.5 (as per magic line 1)
(edited)neillm15:18
Ah, interesting. There's no reason that's specifically python3, but I see what you mean. My default is python2, so it works. I'll add parens around the print statement to make it proper python3
I'm curious if your subscription also stops working after awhile
my debugging today was rather slower than normal.
The server doesn’t time out broadcast callers.
That’s the height provided by the subscription driving the broadcast.
I'll report back when I know more.
neillm15:45
@haarts Cool. I'm also testing a client with the keepalive and monitoring both the original and the new one
neillm16:34
@haarts So far the original client that you asked me to keep running is still running/working properly. Do you have a point of comparison on your side?
neillm16:48
True. The version with the keep alive doesn't seem to be hurting anything though, perhaps could add anyway. Not setting it uses the OS default, so maybe there's a difference there too
neillm17:50
@zoranc Static build of bs on ubuntu runs properly for some time and shuts down cleanly on ctrl-c (i.e. no hidden crash/flush_lock left). If you do reproduce again, please let me know
👍 1
haarts20:38
@neillm I don't have a good point yet. Can you tell me which range of heights the client worked thought? I'm curious.
Monday I have more time to dig into this. I'm on Ubuntu 18.04, from the docs of ZMQ it seems to matter which OS is ran (default is OS specific)
Using
socket.setsockopt(zmq.TCP_KEEPALIVE, 1) in my client now.I've set up a test on my server (ubuntu 16.04)
I'll check it tomorrow morning
one with and one without keep alive (I'm setting that option correct right?)
good night!
The option looks good, I updated the client earlier: https://github.com/libbitcoin/libbitcoin-server/blob/master/data/subscribe-block.py#L16
(edited)Also, fixed the 'off by one' discrepancy in the server, but that needs a server rebuild
oh, time between blocks
neillm20:56
Without. The one with is running still, but not logging to a file, so it's harder to check (not long scrollback)
But no, shouldn't matter
thanks for the help so far!
Saturday, June 30, 2018
N
narodnik11:15
how can I deploy the binary version of libbitcoin-server? There is no config file with it
I'll try on monday on my laptop
evoskuil13:52
@narodnik The binaries are a few minor versions back, so some potential issues. But you can just grab the sample config from the repo (/data). Also, there is the —settings command line option, which emits the settings metadata.
N
narodnik14:37
@evoskuil problem is that the settings file is not accepted by the binary because it's too new and --settings doesn't output the values.
btw check this out, boost has coroutines now which allows sequential linear programming with boost asio https://www.boost.org/doc/libs/1_61_0/libs/coroutine2/doc/html/coroutine2/coroutine/asymmetric.html
[x tmp]$ cat main.cpp
#include <iostream>
#include <boost/coroutine2/all.hpp>
typedef boost::coroutines2::coroutine<int> coro_t;
int main()
{
coro_t::pull_type source(
[&](coro_t::push_type& sink)
{
int first = 1, second = 1;
sink(first);
sink(second);
for(int i = 0; i < 8; ++i)
{
int third = first + second;
first = second;
second = third;
sink(third);
}
});
for (auto i: source)
std::cout << i << " ";
std::cout << std::endl;
return 0;
}
[x tmp]$ g++ main.cpp -lboost_system -lboost_context -std=c++11
[x tmp]$ ./a.out
1 1 2 3 5 8 13 21 34 55
evoskuil14:50
@evoskuil uploaded a file: bs 3.2.0 settings metadata
bs 3.2.0 settings metadata
$ bs --settings Configuration Settings: [blockchain] # A hash:height checkpoint, multiple entries allowed. checkpoint = <value> # The number of cores dedicated to block validation, defaults to 0 (physical cores). cores = <value> # Use high thread priority for block validation, defaults to true. priority = <value> # The maximum reorganization depth, defaults to 256 (0 for unlimited). reorganization_limit = <value> # Use libconsensus for script validation if integrated, defaults to false. use_libconsensus = <value> [database] # Block hash table size, defaults to 650000. block_table_buckets = <value> # The maximum number of entries in the unspent outputs cache, defaults to 10000. cache_capacity = <value> # The blockchain database directory, defaults to 'blockchain'. directory = <value> # Full database files increase by this percentage, defaults to 50. file_growth_rate = <value> # Flush each write to disk, defaults to false. flush_writes = <value> # History hash table size, defaults to 107000000. history_table_buckets = <value> # Spend hash table size, defaults to 250000000. spend_table_buckets = <value> # Transaction hash table size, defaults to 110000000. transaction_table_buckets = <value> [fork] # Add pay-to-script-hash processing, defaults to true (soft fork). bip16 = <value> # Disallow collision of unspent transaction hashes, defaults to true (hard fork). bip30 = <value> # Coinbase input must include block height, defaults to true (soft fork).full file
@narodnik See above. Also, what do you get when you type
$ bs --settings ?I don't want to fill them in by hand
You don't need to, those are defaults 🙂.
evoskuil14:52
No, <value> means, fill in a value. If you don't specify the setting, you get the default.
(edited)You could also just modify an existing or older default cfg file, they don't change much.
(edited)I need to create some new binaries, just busy.