Bitcoin full node and query server
May 2018
Archive of the libbitcoin Slack #general channel.
Tuesday, May 1, 2018
haarts10:40
When I'm watching an address with the explorer (like so
bx watch-address <some addres>) I get back notifications containing a transaction hash. That transaction hash I can look up again. That contains address_hashs. How can I relate a bitcoin address to the transaction?haarts11:35
OK, so from an EC private key you can derive a payment address (I believe that's the correct term) and a Hash160 version of the private key. So given the private key I can see how the
watch-address call works. But bx doesn't have that, it merely has a payment address. And going from one hash to the other is not possible. Or so I thought.You've all been excellent rubber ducks. 😉
👍 1😄 1
and
we got all the way up to linking with the "libbitcoin" repo 🙂 (so secp256k1 was successfullly installed).
libbitcoin configuration:
CC=/usr/local/bin/egcc \
CXX=/usr/local/bin/eg++ \
CFLAGS="-Og -g -fstack-protector-all" \
CXXFLAGS="-std=c++11 -Og -g -fstack-protector-all" \
PKG_CONFIG_PATH="$LIBBITCOIN_BUILD/lib/pkgconfig" \
./configure --prefix=$LIBBITCOIN_BUILD \
--exec-prefix=$LIBBITCOIN_BUILD \
\
--with-boost=$BOOST_DEPLOYMENT_DIR \
\
--with-icu \
--with-png \
--with-qrencode
build:
gmake -j3 V=1
failure output:
libtool: link: /usr/local/bin/eg++ -shared -fPIC -DPIC src/.libs/src_libbitcoin_la-error.o src/chain/.libs/src_libbitcoin_la-block.o src/chain/.libs/src_libbitcoin_la-chain_state.o src/chain/.libs/src_libbitcoin_la-compact.o src/chain/.libs/src_libbitcoin_la-header.o src/chain/.libs/src_libbitcoin_la-input.o src/chain/.libs/src_libbitcoin_la-output.o src/chain/.libs/src_libbitcoin_la-output_point.o src/chain/.libs/src_libbitcoin_la-point.o src/chain/.libs/src_libbitcoin_la-point_iterator.o src/chain/.libs/src_libbitcoin_la-point_value.o src/chain/.libs/src_libbitcoin_la-points_value.o src/chain/.libs/src_libbitcoin_la-script.o src/chain/.libs/src_libbitcoin_la-transaction.o src/chain/.libs/src_libbitcoin_la-witness.o src/config/.libs/src_libbitcoin_la-authority.o src/config/.libs/src_libbitcoin_la-base16.o src/config/.libs/src_libbitcoin_la-base2.o src/config/.libs/src_libbitcoin_la-base58.o src/config/.libs/src_libbitcoin_la-base64.o src/config/.libs/src_libbitcoin_la-checkpoint.o src/config/.libs/src_libbitcoin_la-directory.o src/config/.libs/src_libbitcoin_la-endpoint.o src/config/.libs/src_libbitcoin_la-hash160.o [sniiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiip] _libbitcoin_la-stealth_sender.o src/wallet/.libs/src_libbitcoin_la-uri.o src/wallet/parse_encrypted_keys/.libs/src_libbitcoin_la-parse_encrypted_private.o src/wallet/parse_encrypted_keys/.libs/src_libbitcoin_la-parse_encrypted_public.o src/wallet/parse_encrypted_keys/.libs/src_libbitcoin_la-parse_encrypted_token.o -L/home/phil/base/build/lib -lboost_chrono-mt -lboost_date_time-mt -lboost_filesystem -lboost_iostreams-mt -lboost_locale-mt -lboost_log-mt -lboost_program_options-mt -lboost_regex-mt -lboost_system -lboost_thread-mt -lpthread -L/usr/local/lib -licui18n -licuuc -licudata -lpng -lm -lz -lqrencode -L/home/phil/libbitcoin/build/lib -lsecp256k1 -lgmp -Og -pthread -pthread -o src/.libs/libbitcoin.so.0.0
mv -f examples/.deps/examples_libbitcoin_examples-main.Tpo examples/.deps/examples_libbitcoin_examples-main.Po
/usr/bin/ld: /home/phil/base/build/lib/libboost_filesystem.a(operations.o): relocation R_X86_64_PC32 against `_ZN5boost10filesystem6detail13dir_itr_closeERPvS3_' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
gmake: *** [Makefile:2154: src/libbitcoin.la] Error 1
(will send as "code snippet" under the + button next time.)
there is no "PIC" anywhere in https://github.com/libbitcoin/libbitcoin/blob/version3/install.sh
neillm12:00
There seems to be a linkage mismatch between boost and the other code. For example, install.sh --enable-static --disable-shared keeps all consistent
https://www.openbsd.org/faq/ports/specialtopics.html
"Tweaking ports builds to achieve the right names
Quite a few ports need tweaks to build shared libraries correctly anyways. Remember that building shared libraries should be done with
$ gcc -shared -fpic|-fPIC -o libfoo.so.4.5 obj1 obj2"
hmm
You'd have to look at the specific autotools files (configure/Makefile) per project for the flags that are used
install.sh does not control all project specific flags
But does tell them what they need to internally enable/disable
that's to produce a position independent executable, overall that's a good idea
i wonder why that argument not is implicit? 😃
only a complete idiot would roll binaries with static memory addresses lol
at least if the underlying build toolchain implements it well
anyhow maturation of the build toolchain may be very new
also Linux has been extremely ignorant in this area
neillm12:17
I don't agree, static linkage is fine for many uses and generally more secure than shared libraries (in that you can't load code that was replaced on you). Also performance is better without the load
it must have been a lot of work
I don't agree, static linkage is fine for many uses and generally more secure than shared libraries (in that you can't load code that was replaced on you). Also performance is better without the load
position independent code and static/shared linkage are orthogonal concepts
openbsd's kernel is also position independent now since ~6 mo or so
for OT remark, FreeBSD is a total nightmare with respect to low level security mitigations
i think they don't even do ASLR yet
ok very well, time to work around this one.
i would be curious about why i get this PIC error here and nowhere else, what the big picture is. anyhow now just make it work and get on with the next thing.
neillm12:24
I'm familiar with the ideas, but thought position independent implied dynamic loading. If static .o files can also be position independent, sure I don't know the advantage of fixed, other than if you're rolling hand crafted code made for stack smashing ;-)
M
Mlmikael12:24
other than if you're rolling hand crafted code made for stack smashing ;-)
that's a special case, you'd do "-fomit-pic" for that :))
but yeah
M
Mlmikael12:25
i think some quirks in Linux land are a function of stagnation in the community, coming from the fact that the ecosystem is so asynchronous - you have Linus sycing the kernel tree, and then you have a zillion distributions with low or zero incentive to cooperate or really go to the bottom with stuff
something liket hat
M
Mlmikael12:27
@neillm ok this is getting OT however my best impression is that Linux does have a lot of werid stuff going on
here is an example of a developer who found that to be the case: https://arcan-fe.com/2018/04/25/towards-secure-system-graphics-arcan-and-openbsd/
Arcan
Let me preface this by saying that this is a (very) long and medium-rare technical article about the security considerations and minutiae of porting (most of) the Arcan ecosystem to work under OpenBSD. The main point of this article is not so much flirting with the OpenBSD crowd or adding further noise to software engineering topics, but to go through the special considerations that had to be taken, as notes to anyone else that decides to go down this overgrown and lonesome trail, or are curious about some less than obvious differences between how these things “work” on Linux …
i haven't really studied Linux' sources myself so i can't comment
i do have the impression that systemd is a case of very irresponsible engineering
M
Mlmikael12:28
weird stuff should partially be a function of engineering to mitigate a communication gap, such as between distros or some other people
anyhow
now back to build focus. :)))
frequently "weird stuff" and security holes are correlated
weird stuff appears in technological systems when people did not coordinate, or did not care, or nobody designed and things were left arbitrary / to chance
so, just added "-fpic" to CFLAGS and CXXFLAGS.
oh interesting configure fail message for libbitcoin now, i wonder why it appeared now:
checking for pthread-config... no
configure: error: pthread library is required but was not found.
neillm12:38
I don't think you need to add -fPIC, configure will do that for you if you specify the proper shared/static option (passed down from install.sh)
hm
M
Mlmikael12:39
you mean some other mechanism in libbitcoin's build infrastructure will add "-fpic" if you specify "-shared" or somethingh
hm
ok let met try
M
Mlmikael12:41
interesting, the libbitcoin configure script outputs: checking for /usr/local/bin/egcc option to produce PIC... -fPIC -DPIC
and also "checking if /usr/local/bin/egcc PIC flag -fPIC -DPIC works... yes"
and also
checking for /usr/local/bin/eg++ option to produce PIC... -fPIC -DPIC
checking if /usr/local/bin/eg++ PIC flag -fPIC -DPIC works... yes
neat!
@neillm indeed I see the g++ invocations now when i gmake again (after gmake clean of course), the arguments "-fPIC -DPIC" are passed now. neat.
are these Autotools logics, to enable PIC only when you're build a library in shared-form only?
neillm12:47
I know the recommendation to use install.sh must get old, but there's really a lot going on that the script simplifies which can be done manually, but it's very difficult to get right/consistently
Yes, standard autotools
M
Mlmikael12:48
I know the recommendation to use install.sh must get old, but there's really a lot going on that the script simplifies which can be done manually, but it's very difficult to get right/consistently
right, i saw this in the past, that install.sh indeed is very finely tuned
i know it took Eric a month to put it together (the build system for libbitcoin)
neillm12:48
I understand, but the goal of porting to a new platform is to make sure that it does work 🙂
yes
and it's found both by the name /bin/sh and /bin/ksh
this means also when you want to run a script that has "#!/bin/bash" as its first line, you need to
yep. bash is GNU GPLv3 i think, so it is disqualified from being included in the OS distro for that reason, however it's provided as a package so you just do "pkg_add bash" and it installs as /usr/local/bin/bash .
if you want it.
you just need to throw in the "ksh " prefix when running install.sh, i think that's all
as in you run "ksh ./install.sh" instead of "./install.sh"
M
Mlmikael12:56
@neillm after doing for libbitcoin "./configure --enable-shared --disable-shared" as you suggested, i get the error again:
/usr/bin/ld: /home/myuser/base/build/lib/libboost_filesystem.a(operations.o): relocation R_X86_64_PC32 against `_ZN5boost10filesystem6detail13dir_itr_closeERPvS3_' can not be used when making a shared object; recompile with -fPIC
now.. this has to do with boost.. hmm
urr.. hmm.. hmm..
But also, boost was probably not built with the proper shared flags that you want
ah i meant
--enable-shared --disable-static
that's what i meant, sorry 😃
CC=/usr/local/bin/egcc \
CXX=/usr/local/bin/eg++ \
CFLAGS="-Og -g -fstack-protector-all" \
CXXFLAGS="-std=c++11 -Og -g -fstack-protector-all" \
PKG_CONFIG_PATH="$LIBBITCOIN_BUILD/lib/pkgconfig" \
./configure --prefix=$LIBBITCOIN_BUILD \
--exec-prefix=$LIBBITCOIN_BUILD \
\
--with-boost=$BOOST_DEPLOYMENT_DIR \
\
--with-icu \
--with-png \
--with-qrencode \
--enable-shared --disable-static
gmake -j3 V=1
this was what i tried now 😃 sry misquote
@Mlmikael uploaded a file: Untitled
Untitled
it ended liket his: libtool: link: /usr/local/bin/eg++ -shared -fPIC -DPIC src/.libs/src_libbitcoin_la-error.o src/chain/.libs/src_libbitcoin_la-block.o src/chain/.libs/src_libbitcoin_la-chain_state.o src/chain/.libs/src_libbitcoin_la-compact.o src/chain/.libs/src_libbitcoin_la-header.o src/chain/.libs/src_libbitcoin_la-input.o src/chain/.libs/src_libbitcoin_la-output.o src/chain/.libs/src_libbitcoin_la-output_point.o src/chain/.libs/src_libbitcoin_la-point.o src/chain/.libs/src_libbitcoin_la-point_iterator.o src/chain/.libs/src_libbitcoin_la-point_value.o src/chain/.libs/src_libbitcoin_la-points_value.o src/chain/.libs/src_libbitcoin_la-script.o src/chain/.libs/src_libbitcoin_la-transaction.o src/chain/.libs/src_libbitcoin_la-witness.o src/config/.libs/src_libbitcoin_la-authority.o src/config/.libs/src_libbitcoin_la-base16.o src/config/.libs/src_libbitcoin_la-base2.o src/config/.libs/src_libbitcoin_la-base58.o src/config/.libs/src_libbitcoin_la-base64.o src/config/.libs/src_libbitcoin_la-checkpoint.o src/config/.libs/src_libbitcoin_la-directory.o src/config/.libs/src_libbitcoin_la-endpoint.o src/config/.libs/src_libbitcoin_la-hash160.o src/config/.libs/src_libbitcoin_la-hash256.o src/config/.libs/src_libbitcoin_la-parameter.o src/config/.libs/src_libbitcoin_la-parser.o src/config/.libs/src_libbitcoin_la-printer.o src/config/.libs/src_libbitcoin_la-sodium.o src/formats/.libs/src_libbitcoin_la-base_10.o src/formats/.libs/src_libbitcoin_la-base_16.o src/formats/.libs/src_libbitcoin_la-base_58.o src/formats/.libs/src_libbitcoin_la-base_64.o src/formats/.libs/src_libbitcoin_la-base_85.o src/log/.libs/src_libbitcoin_la-file_collector.o src/log/.libs/src_libbitcoin_la-file_collector_repository.o src/log/.libs/src_libbitcoin_la-file_counter_formatter.o src/log/.libs/src_libbitcoin_la-sink.o src/log/.libs/src_libbitcoin_la-statsd_sink.o src/log/.libs/src_libbitcoin_la-udp_client_sink.o src/machine/.libs/src_libbitcoin_la-interpreter.o src/machine/.libs/src_libbitcoin_la-number.o src/machine/.libs/src_libbitcoin_la-opcode.o src/machine/.libs/src_libbitcoin_la-operation.o src/machine/.libs/src_libbitcoin_la-program.o src/math/.libs/src_libbitcoin_la-checksum.o src/math/.libs/src_libbitcoin_la-crypto.o src/math/.libs/src_libbitcoin_la-elliptic_curve.o src/math/.libs/src_libbitcoin_la-hash.o src/math/.libs/src_libbitcoin_la-secp256k1_initializer.o src/math/.libs/src_libbitcoin_la-stealth.o src/math/external/.libs/src_libbitcoin_la-aes256.o src/math/external/.libs/src_libbitcoin_la-crypto_scrypt.o src/math/external/.libs/src_libbitcoin_la-hmac_sha256.o src/math/external/.libs/src_libbitcoin_la-hmac_sha512.o src/math/external/.libs/src_libbitcoin_la-lax_der_parsing.o src/math/external/.libs/src_libbitcoin_la-pbkdf2_sha256.o src/math/external/.libs/src_libbitcoin_la-pkcs5_pbkdf2.o src/math/external/.libs/src_libbitcoin_la-ripemd160.o src/math/external/.libs/src_libbitcoin_la-sha1.o src/math/external/.libs/src_libbitcoin_la-sha256.o src/math/external/.libs/src_libbitcoin_la-sha512.o src/math/external/.libs/src_libbitcoin_la-zeroize.o src/message/.libs/src_libbitcoin_la-address.o src/message/.libs/src_libbitcoin_la-alert.o src/message/.libs/src_libbitcoin_la-alert_payload.o src/message/.libs/src_libbitcoin_la-block.o src/message/.libs/src_libbitcoin_la-block_transactions.o src/message/.libs/src_libbitcoin_la-compact_block.o src/message/.libs/src_libbitcoin_la-fee_filter.o src/message/.libs/src_libbitcoin_la-filter_add.o src/message/.libs/src_libbitcoin_la-filter_clear.o src/message/.libs/src_libbitcoin_la-filter_load.o src/message/.libs/src_libbitcoin_la-get_address.o src/message/.libs/src_libbitcoin_la-get_block_transactions.o src/message/.libs/src_libbitcoin_la-get_blocks.o src/message/.libs/src_libbitcoin_la-get_data.o src/message/.libs/src_libbitcoin_la-get_headers.o src/message/.libs/src_libbitcoin_la-header.o src/message/.libs/src_libbitcoin_la-headers.o src/message/.libs/src_libbitcoin_la-heading.o src/message/.libs/src_libbitcoin_la-inventory.o src/message/.libs/src_libbitcoin_la-inventory_vector.o src/message/.libs/src_libbitcoin_la-memory_pool.o src/message/.libs/src_libbitcoin_la-merkle_block.o src/message/.libs/src_libbitcoin_la-messages.o src/message/.libs/src_libbitcoin_la-network_address.o src/message/.libs/src_libbitcoin_la-not_found.o src/message/.libs/src_libbitcoin_la-ping.o src/message/.libs/src_libbitcoin_la-pong.o src/message/.libs/src_libbitcoin_la-prefilled_transaction.o src/message/.libs/src_libbitcoin_la-reject.o src/message/.libs/src_libbitcoin_la-send_compact.o src/message/.libs/src_libbitcoin_la-send_headers.o src/message/.libs/src_libbitcoin_la-transaction.o src/message/.libs/src_libbitcoin_la-verack.o src/message/.libs/src_libbitcoin_la-version.o src/unicode/.libs/src_libbitcoin_la-console_streambuf.o src/unicode/.libs/src_libbitcoin_la-ifstream.o src/unicode/.libs/src_libbitcoin_la-ofstream.o src/unicode/.libs/src_libbitcoin_la-unicode.o src/unicode/.libs/src_libbitcoin_la-unicode_istream.o src/unicode/.libs/src_libbitcoin_la-unicode_ostream.o src/unicode/.libs/src_libbitcoin_la-unicode_streambuf.o src/utility/.libs/src_libbitcoin_la-binary.o src/utility/.libs/src_libbitcoin_la-conditional_lock.o src/utility/.libs/src_libbitcoin_la-deadline.o src/utility/.libs/src_libbitcoin_la-dispatcher.o src/utility/.libs/src_libbitcoin_la-flush_lock.o src/utility/.libs/src_libbitcoin_la-interprocess_lock.o src/utility/.libs/src_libbitcoin_la-istream_reader.o src/utility/.libs/src_libbitcoin_la-monitor.o src/utility/.libs/src_libbitcoin_la-ostream_writer.o src/utility/.libs/src_libbitcoin_la-png.o src/utility/.libs/src_libbitcoin_la-prioritized_mutex.o src/utility/.libs/src_libbitcoin_la-random.o src/utility/.libs/src_libbitcoin_la-scope_lock.o src/utility/.libs/src_libbitcoin_la-sequencer.o src/utility/.libs/src_libbitcoin_la-sequential_lock.o src/utility/.libs/src_libbitcoin_la-socket.o src/utility/.libs/src_libbitcoin_la-string.o src/utility/.libs/src_libbitcoin_la-thread.o src/utility/.libs/src_libbitcoin_la-threadpool.o src/utility/.libs/src_libbitcoin_la-work.o src/wallet/.libs/src_libbitcoin_la-bitcoin_uri.o src/wallet/.libs/src_libbitcoin_la-dictionary.o src/wallet/.libs/src_libbitcoin_la-ec_private.o src/wallet/.libs/src_libbitcoin_la-ec_public.o src/wallet/.libs/src_libbitcoin_la-ek_private.o src/wallet/.libs/src_libbitcoin_la-ek_public.o src/wallet/.libs/src_libbitcoin_la-ek_token.o src/wallet/.libs/src_libbitcoin_la-encrypted_keys.o src/wallet/.libs/src_libbitcoin_la-hd_private.o src/wallet/.libs/src_libbitcoin_la-hd_public.o src/wallet/.libs/src_libbitcoin_la-message.o src/wallet/.libs/src_libbitcoin_la-mini_keys.o src/wallet/.libs/src_libbitcoin_la-mnemonic.o src/wallet/.libs/src_libbitcoin_la-payment_address.o src/wallet/.libs/src_libbitcoin_la-qrcode.o src/wallet/.libs/src_libbitcoin_la-select_outputs.o src/wallet/.libs/src_libbitcoin_la-stealth_address.o src/wallet/.libs/src_libbitcoin_la-stealth_receiver.o src/wallet/.libs/src_libbitcoin_la-stealth_sender.o src/wallet/.libs/src_libbitcoin_la-uri.o src/wallet/parse_encrypted_keys/.libs/src_libbitcoin_la-parse_encrypted_private.o src/wallet/parse_encrypted_keys/.libs/src_libbitcoin_la-parse_encrypted_public.o src/wallet/parse_encrypted_keys/.libs/src_libbitcoin_la-parse_encrypted_token.o -L/home/phil/base/build/lib -lboost_chrono-mt -lboost_date_time-mt -lboost_filesystem -lboost_iostreams-mt -lboost_locale-mt -lboost_log-mt -lboost_program_options-mt -lboost_regex-mt -lboost_system -lboost_thread-mt -lpthread -L/usr/local/lib -licui18n -licuuc -licudata -lpng -lm -lz -lqrencode -L/home/phil/libbitcoin/build/lib -lsecp256k1 -lgmp -Og -pthread -pthread -o src/.libs/libbitcoin.so.0.0 mv -f examples/.deps/examples_libbitcoin_examples-main.Tpo examples/.deps/examples_libbitcoin_examples-main.Po /usr/bin/ld: /home/phil/base/build/lib/libboost_filesystem.a(operations.o): relocation R_X86_64_PC32 against `_ZN5boost10filesystem6detail13dir_itr_closeERPvS3_' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status gmake: *** [Makefile:2154: src/libbitcoin.la] Error 1
so same, fourth last line shows the real error:
/usr/bin/ld: /home/phil/base/build/lib/libboost_filesystem.a(operations.o): relocation R_X86_64_PC32 against `_ZN5boost10filesystem6detail13dir_itr_closeERPvS3_' can not be used when making a shared object; recompile with -fPIC
it somehow spits out the Boost, hmm
neillm13:00
Right, boost was not build in the expected format. You'd have to build it just like the install.sh builds it
M
Mlmikael13:01
@neillm I ask a central GNU GCC developer redi this question "the distinction shared versus static linking, is a concept ORTHOGONAL to whether code is position-independent, right?", and he answered "kinda".
he continues, "static vs shared linking is orthogonal to whether code is PI, except that some systems don't allow shared linking to non-PI code"
In the case that the code is not PI, my understanding is that the linker just re-maps everything needed at link time (more linker work, but mostly same idea)
M
Mlmikael13:04
if the compilation was not done PI, simulating PI afterwards may be impossible, i'm not updated abotu the details
This is OT again though 😉
It's been many years since I've read up on linkers. Let's address the issue that you're seeing. I recommend building boost the same way install.sh would do it given --enable-shared and seeing how that does
M
Mlmikael13:09
@neillm if I read install.sh right (https://github.com/libbitcoin/libbitcoin/blob/version3/install.sh) , its default behavior is to build Boost in both static and shared forms
In your case, that's what you want though
M
Mlmikael13:13
@neillm a question i get on this one is whether position-independent compilation can ever be "bad" as in break a build process. mangling position-independent to position-dependent should be straightforward.
neillm13:16
I can believe that ... but haven't written a linker or know that process, so I can't say for sure 😉
haarts13:16
Another question re the Query Service subscribe.address call (https://github.com/libbitcoin/libbitcoin-server/wiki/Query-Service#subscribeaddress); these notifications are put in the socket and it is for the connecting client to figure out for which address they are right? Eg if the client subscribes to more than one address is has to sort that out once receiving a notification. Is that correct?
haarts13:18
Then I've understood is correctly. It's a tad involved going from the tx hash to a tx to a list of address hashes to a list of payment hashes, but that's what this lib I'm writing is for!
neillm13:24
Ah, ok, without bindings to libbitcoin in that case, yeah, that sounds trickier. Unless you're using some other tools like pybitcoin or whatever?
haarts13:25
yeah, I'm using python-bitcoinlib for some of the deserializations. Didn't think to look if there is more I can use. Good to bring that to my attention again.
M
Mlmikael13:27
@neillm i think for us it's policy-time now!
https://en.wikipedia.org/wiki/Position-independent_code#PIE
see here - recent Mac OS X, iOS, Android, Windows, and most Linuxes all support PIE.
Wikipedia
In computing, position-independent code (PIC) or position-independent executable (PIE) is a body of machine code that, being placed somewhere in the primary memory, executes properly regardless of its absolute address. PIC is commonly used for shared libraries, so that the same library code can be loaded in a location in each program address space where it will not overlap any other uses of memory (for example, other shared libraries). PIC was also used on older computer systems lacking an MMU, so that the operating system could keep applications away from each other even within the single add…
this means any platform that ever runs any program i made, i can require it to support PIE
if you don't support PIE on your OS, you won't run my binary - so i can tailor all work i do to the position-independent usecase.
the fact that you even need to specify "-fpie" specifically to get position-independentness is so incredibly weird
neillm13:30
Depends on the architecture I guess. Sure, I think on most modern architectures it seems weird, although likely not on embedded systems on small chips. But that's OT again, and I don't really want to argue it right now 😉
architecture = AMD64, ARM64, SPARC64 etc.
all those do PIE.
M
Mlmikael13:39
@neillm ah you know what, Boost is not compiled by Autotools, so "--enable-shared --disable-static" would not deliver there in all cases
ok nice to have this one hopefully clarified.
@neillm thank you very much for your suggestions on this one 🙂
sorry if i appear grumpy at any moment, not intended. i have learnt a loot about how to build stuff recently. 🙂
and how to not build lol
M
Mlmikael13:46
@neillm i stumble into this question from people i talk to: Isn't it that LibBitcoin is trying to link STATICALLY to Boost?
they deduce that that is the reason why my build fails
they think the default option out there is that when you build a shared library, it's PIE
and when you build a static library it's not
and now i try to statically link a static library when making a shared library
hence the issue
neillm13:47
If you use install.sh --enable-static --disable-shared, which is a config I use often, then yes. But you can just as easily use the opposite
M
Mlmikael13:47
i guess i like the idea of static linking into LibBitcoin, as a way to omit various issues
if i do just "install.sh", then both static and shared is built
neillm13:48
From my quick glance, there may be bug that only builds the shared boost configuration in that case
Would have to look into more, not sure at the moment
M
Mlmikael13:48
https://0bin.net/paste/CJpOQvwZPKBfcn75#09AqOAyxkDXNm141TGhIwhQhfdIKuDRZFPmQjGapCch - this is the correct output
0bin.net
0bin is a client-side-encrypted pastebin featuring burn after reading, an history and a clipboard
of the error (built with "-j1" to ensure totally serial order)
aha
evoskuil14:10
If you dynamically link you accept whatever gets linked, AKA DLL hell in the Windows world.
You don’t need to enable-static or enable-shared, both are defaults (and either can be disabled).
It took a lot longer than 8 days to design the generated installer, which reliably builds libbitcoin in minutes :).
neillm14:15
The part that caught my eye is this: https://github.com/libbitcoin/libbitcoin/blob/version3/install.sh#L254
I mixed it up earlier, but it looks like if both are set, static linkage only is provided
Similar to the boost specific option (the last paste)
boost as shared in that case
If you disable one you get the other, otherwise both.
Ah, nevermind, I see
Was confused here: https://github.com/libbitcoin/libbitcoin/blob/version3/install.sh#L241
@haarts the client-server API is designed for performance, not caller convenience.
All work is pushed to the client where possible.
haarts14:44
@evoskuil Fair enough! But this does require me to make an additional call to the server to convert a tx hash to a tx. But that might just be my use case.
evoskuil14:47
Right, much better from a performance standpoint than the server pushing all affected txs to all subscribers. Client can handle each as it is able.
Sometimes by performance I’m referring to scalability, not just the fastest outcome in an ideal case.
If clients become backed up the server must either wait on each client (accumulating pending messages) or drop messages. It will do both as necessary, but it is much better for all involved to minimize waiting and thereby eliminate dropping.
(edited)This requires allowing the client to manage its processing rate and to push as much work as possible to the potentially millions of clients operating against the single server.
If the server drops a notification the client is informed on the next notification by the existence of a sequence gap. The client may then query the server to cover this gap.
If, with clients querying the server for txs, the server becomes overtaxed it simply causes the clients to wait, resulting in a distributed wait state as opposed to a message backlog and dropping.
(edited)I realize that many APIs behave differently, but they also exhibit poor performance and/or don’t scale. Core’s JSON-RPC for example.
And in the end the additional complexity is resolvable in client code.
evoskuil15:11
However... v3.4 BS does send the full tx. This was designed as a security best practice for stealth subscriptions. Stealth indexing is phased out in v4, which will allow this to revert.
M
Mlmikael15:40
wait, so i had this extremely comical issue - my Boost build built broken .so filenames, their postfix was ".so.", and for whatever reason Boost would not produce any .so symlinks to those
this led to that LibBitcoin's configure script only found the statically compiled Boost libraries, not the shared ones
and since LibBitcoin is compiled as a shared library, which is position-independently compiled by default, and Boost was static which is fixed-position by default, we got the clash where my build process failed and screamed for -fpic.
one thing that could be interesting here would be if LibBitcoin's configure would print out whether it will use a shared or static build of Boost. this is an extreme corner case though.
checking for exit in -lboost_locale-mt... yes
configure: boost_locale_LIBS : -lboost_locale-mt
checking whether the Boost::Log library is available... yes
checking for exit in -lboost_log-mt... yes
configure: boost_log_LIBS : -lboost_log-mt
checking whether the Boost::Program_Options library is available... yes
checking for exit in -lboost_program_options-mt... yes
our Boost build and deploy instructions document is now 20 A4 pages long lolol
evoskuil15:44
We don’t know what the linker will do, only how we configure it, but it should tell you.
@evoskuil it failed with this error: /usr/bin/ld: /home/myuser/build/lib/libboost_filesystem.a(operations.o): relocation R_X86_64_PC32 against `_ZN5boost10filesystem6detail13dir_itr_closeERPvS3_' can not be used when making a shared object; recompile with -fPIC
so indeed yes it did clarify how it failed
however if Boost's static build would have fit the critieria (= been position-independent) then LibBitcoin would have linked to it silently
as far as i'm aware at least
that could qualify as a silent problem
evoskuil15:48
Libbitcoin isn’t the linker, we don’t link to boost, the linker links us with boost into an executable. There are lots of ways that can fail that we cannot control.
We try to minimize user error in that process by providing a reliable turnkey script.
yep makes sense.
maybe we have the repo "libbitcoin" building here now finally :)))
first time in 2.5 years 😃
LibBitcoin is indeed a "clean" project with respect to dependencies, it's only the typical mmap, bsd sockets and posix OS API:s and then Boost, and that's it.
(and zmq, png, and one more)
@evoskuil in a while i should try LibBitcoin on an ARM64 computer :))) and try mmap() generally
maybe Linux has a 256GB cap on mmap() on ARM64, but maybe other unices don't, will see then.
libbitcoin built! :DDDD hooray! =D
@evoskuil oh a question now that i'm at it, do you ever add "--exec-prefix" to your ./configure in the libbitcoin repos?
if "--exec-prefix" is not provided then "--prefix" is the all-effective catch-all installation directory prefix configuration setting?
skaht16:01
@Mlmikael - On the Mac, I used to compile with --disable-shared and --prefix= configure flags via the appropriate
install.sh quite extensively. Based upon the chatter here, I’m guessing it is getting to be time for me to give libbitcoin a try again after recent updates.Wednesday, May 2, 2018
M
Mlmikael12:51
i have a practical question about libbitcoin version3 - what's the dependency build order among the repos, where can i read that?
I guess either way, the answer is in install.sh 🙂
hmm!
you may have a point lol
you're like a lutheran refering to the chatechism lol
blanket answer
wai twait neill
it just says "libbitcoin"
hmm
neillm12:53
I guess if I always answer all of your questions as install.sh, it's bound to be right someday 😉
thanks!
M
Mlmikael13:17
wait do you have the same release convention in libsecp256k1 as you have in the libbitcoin* repos?
there was lots of changes late 2017 in it however https://github.com/libbitcoin/secp256k1/tree/version5 's most recent github release is aug 10 2017
neillm13:19
The version used is more or less fixed for compatibility. We'd have to test the latest before tracking a later version from install.sh, if that's what you mena
neillm13:20
Oh, yes, I was incorrect, I thought we tracked a particular version, but I think I'm mixing that up with zmq, which we fixed somewhat recently
Fixed to a particular version I mean. So yes, it looks like we track version5 latest
to track version5 lates
are you talking about adding a "release" tag to your version5 repo to be synced with the version3 libbitcoin* repos i.e. presently "v3.5.0."?
or are you talking about pulling in code updates from upstream?
neillm13:24
That's an external project that has a version5 tag. In theory they will not break API compatibility. Our install.sh just grabs the latest version5 from their repo and uses that.
git clone https://github.com/libbitcoin/secp256k1
Looking at it, the syncing must be updated manually by @evoskuil
So no, until that manual step is done, we're not tracking upstream
I didn't realize that was the case for that repo. I'm still learning too 😉
M
Mlmikael14:00
So no, until that manual step is done, we're not tracking upstream
consensus logics are so deep that doing it manually makes sense.
I didn't realize that was the case for that repo. I'm still learning too 😉
M
Mlmikael14:27
so likely finally we have the build scripting for all the 11 LibBitcoin repos updated here, and it's building. 🙂
based on the work from mid 2015, when i had version2 running well. 🙂
Friday, May 4, 2018
M
Mlmikael08:28
@Mlmikael uploaded a file: Untitled
Untitled
libbitcoin
$ gmake V=1 check
gmake test/libbitcoin_test
gmake[1]: Entering directory '/home/phil/libbitcoin/src/libbitcoin'
/usr/local/bin/eg++ -DPACKAGE_NAME=\"libbitcoin\" -DPACKAGE_TARNAME=\"libbitcoin\" -DPACKAGE_VERSION=\"3.4.0\" -DPACKAGE_STRING=\"libbitcoin\ 3.4.0\" -DPACKAGE_BUGREPORT=\"eric@voskuil.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"libbitcoin\" -DVERSION=\"3.4.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -D__EXTENSIONS__=1 -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D_POSIX_PTHREAD_SEMANTICS=1 -D_TANDEM_SOURCE=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DBOOST_HAS_ICU=1 -DNDEBUG=1 -DBOOST_ALL_DYN_LINK=1 -DHAVE_BOOST=/\*\*/ -DHAVE_BOOST_CHRONO=/\*\*/ -DHAVE_BOOST_DATE_TIME=/\*\*/ -DHAVE_BOOST_FILESYSTEM=/\*\*/ -DHAVE_BOOST_IOSTREAMS=/\*\*/ -DHAVE_BOOST_LOCALE=/\*\*/ -DHAVE_BOOST_LOG=/\*\*/ -DHAVE_BOOST_PROGRAM_OPTIONS=/\*\*/ -DHAVE_BOOST_REGEX=/\*\*/ -DHAVE_BOOST_SYSTEM=/\*\*/ -DHAVE_BOOST_THREAD=/\*\*/ -DHAVE_BOOST_UNIT_TEST_FRAMEWORK=/\*\*/ -DHAVE_PTHREAD_PRIO_INHERIT=1 -I. -I./include -DWITH_ICU -DWITH_PNG -DWITH_QRENCODE -I/home/phil/base/build/include -I/usr/local/include -I/usr/local/include/libpng16 -I/usr/local/include -I/home/phil/libbitcoin/build/include -std=c++11 -Og -g -fstack-protector-all -std=c++11 -Wno-deprecated-declarations -Wno-missing-braces -fvisibility-inlines-hidden -fno-var-tracking-assignments -MT test/utility/test_libbitcoin_test-data.o -MD -MP -MF test/utility/.deps/test_libbitcoin_test-data.Tpo -c -o test/utility/test_libbitcoin_test-data.o `test -f 'test/utility/data.cpp' || echo './'`test/utility/data.cpp
test/utility/data.cpp: In member function 'void data_tests::data__splice_two__two_bytes_each__expected::test_method()':
test/utility/data.cpp:253:27: error: reference to 'splice' is ambiguous
const auto combined = splice(left, right);
^
In file included from /home/phil/base/build/include/boost/asio/detail/socket_types.hpp:77:0,
from /home/phil/base/build/include/boost/asio/detail/impl/pipe_select_interrupter.ipp:31,
from /home/phil/base/build/include/boost/asio/detail/pipe_select_interrupter.hpp:82,
from /home/phil/base/build/include/boost/asio/detail/select_interrupter.hpp:27,
from /home/phil/base/build/include/boost/asio/detail/kqueue_reactor.hpp:32,
from /home/phil/base/build/include/boost/asio/detail/reactor.hpp:23,
from /home/phil/base/build/include/boost/asio/detail/impl/scheduler.ipp:23,
from /home/phil/base/build/include/boost/asio/detail/scheduler.hpp:212,
from /home/phil/base/build/include/boost/asio/system_context.hpp:19,
from /home/phil/base/build/include/boost/asio/impl/system_executor.hpp:22,
from /home/phil/base/build/include/boost/asio/system_executor.hpp:129,
from /home/phil/base/build/include/boost/asio/associated_executor.hpp:21,
from /home/phil/base/build/include/boost/asio.hpp:21,
from ./include/bitcoin/bitcoin/utility/asio.hpp:24,
from ./include/bitcoin/bitcoin/chain/block.hpp:35,
from ./include/bitcoin/bitcoin.hpp:24,
from test/utility/data.cpp:24:
/usr/include/sys/socket.h:138:8: note: candidates are: struct splice
struct splice {
^
In file included from ./include/bitcoin/bitcoin/utility/data.hpp:156:0,
from ./include/bitcoin/bitcoin/math/hash.hpp:29,
from ./include/bitcoin/bitcoin/formats/base_16.hpp:24,
from ./include/bitcoin/bitcoin/config/checkpoint.hpp:27,
from ./include/bitcoin/bitcoin/constants.hpp:25,
from ./include/bitcoin/bitcoin.hpp:19,
from test/utility/data.cpp:24:
./include/bitcoin/bitcoin/impl/utility/data.ipp:100:35: note: template<long unsigned int Left, long unsigned int Middle, long unsigned int Right> libbitcoin::byte_array<((Left + Middle) + Right)> libbitcoin::splice(const std byte_array<Left + Middle + Right> splice(const std::array<uint8_t, Left>& left,
^full file"gmake V=1 test" on release 3.5.0 in the libbitcoin repo gives us this.
building test/utility/data.cpp fails like this:
test/utility/data.cpp:278:27: error: reference to 'splice' is ambiguous
const auto combined = splice(left, middle, right);
evoskuil15:20
Oh, I see - socket.h, just add an explicit bc:: namespace qualifier and a comment. This sort of contextual pollution of the global namespace is one reason you will need CI to keep your environment working.
Saturday, May 5, 2018
M
Mlmikael06:16
@evoskuil aha, can you clarify what i need to add to which line, to workaround it?
or did you commit it so i should just use the newest commit now?
(of the libbitcoin repo)
I'll get back to this previous issue in one second, now i have one interim question:
In our build process, somehow our GCC compilation - which is intended to use GNU libstdc++ of course - somehow our binaries got tainted with library dependencies to clang's lib abi
M
Mlmikael08:07
btw, interesting, OpenBSD switched to CLANG as default compiler for AMD64 and I386 a couple of months ago (https://www.openbsd.org/62.html)
M
Mlmikael08:27
interesting.
do you guys have any preference today for GCC over clang or the other way around, @neillm, @evoskuil?
M
Mlmikael10:43
soo, the news for the day is that we make the assessment that using the OS' default compiler is the overall most sensible choice, and, for OpenBSD since last fall, on all of AMD64, I386, ARM64, SPARC64 (and more?), the system compiler has changed to clang
.
and so, all the ports ("packages") are built using clang too.
tomorrow we'll proceed with compiling boost and libbitcoin for clang.
using clang is not required however we see that an OS (kernel + bundled programs and libs + packaged libs and programs) tends to be built using clang OR gcc
so we'll adapt to this, and in this exact usecase right now this means clang.
hopefully we
'll have libbitcoin and its CLI tools going at the end of the day tmrw!
neillm12:10
libbitcoin is regularly built (by CI) with both gcc and clang. personally, i prefer to use gcc, but that shouldn't matter either way
@Mlmikael As to your earlier question, try this: const auto combined = bc::splice(left, middle, right);
If that works, add a comment as to why it's changed, and issue a PR if you'd like
Sunday, May 6, 2018
M
Mlmikael13:33
aha, for reasons that are off-topic to libbitcoin, we will not use clang whatsoever, so on Unix we'll go with GCC all over.
Monday, May 7, 2018
James C.11:38
Hello 🙂 May I ask for help after troubles starting a v3.5 server installed via homebrew package manager. Running OSX behind a NAT, standard testnet config file. Failure:
Error seeding host addresses: unresponsive peer may be throttling. Unsuccessfully tried so far: increasing timeouts, outbound connections.@James C. uploaded a file: Untitled
Untitled
11:31:50.962798 INFO [server] ================= startup ================== 11:31:50.962855 WARNING [server] ================= startup ================== 11:31:50.963075 ERROR [server] ================= startup ================== 11:31:50.963202 FATAL [server] ================= startup ================== 11:31:50.963322 INFO [server] Using config file: "/usr/local/Cellar/libbitcoin-server/3.5.0_1/etc/libbitcoin/bs.cfg" 11:31:50.963374 INFO [server] Please wait while the server is starting... 11:31:50.964720 INFO [network] Starting manual session. 11:31:50.964880 INFO [network] Contacting seed [testnet-seed.bitcoin.jonasschnelli.ch:18333] 11:31:50.964999 INFO [network] Contacting seed [seed.tbtc.petertodd.org:18333] 11:31:50.965052 INFO [network] Contacting seed [testnet-seed.bluematt.me:18333] 11:31:50.965104 INFO [network] Contacting seed [testnet-seed.bitcoin.schildbach.de:18333] 11:31:50.965156 INFO [network] Contacting seed [testnet-seed.voskuil.org:18333] 11:31:50.996627 INFO [network] Connected seed [testnet-seed.bitcoin.jonasschnelli.ch:18333] as 5.9.121.164:18333 11:31:55.176124 INFO [network] Failure contacting seed [seed.tbtc.petertodd.org:18333] resolving hostname failed 11:31:55.399541 INFO [network] Connected seed [testnet-seed.bluematt.me:18333] as 198.251.83.19:18333 11:31:55.433578 INFO [network] Failure contacting seed [testnet-seed.voskuil.org:18333] operation failed 11:32:05.298594 INFO [network] Failure contacting seed [testnet-seed.bitcoin.schildbach.de:18333] connection timed out 11:32:05.298851 ERROR [network] Error seeding host addresses: unresponsive peer may be throttling 11:32:05.299366 ERROR [server] Server failed to start with error, unresponsive peer may be throttling. 11:32:05.299830 INFO [server] Please wait while the server is stopping... 11:33:10.714301 INFO [node] Server stopped successfully.
Many thanks in advance!
neillm11:49
@James C. Can you see how many lines are present in the hosts.cache file? It should be at least 100. If it's less than that, can you remove the file and restart and then check again?
The default config should have mainnet1/2/3/4 as seeds, so hopefully those were commented out and replaced with testnet1/2/3/4 listed below.
James C.11:54
thx for help @neillm - so i should delete host cache and run bs again? without rebuilding blockchain? (just left my comp, will be back in 20m).
ah ok config stil has jonas, todd, voskuil seeds
neillm11:55
Oh, that's the problem, we've phased those out. Hrm, the default config should not have those for v3.5
1 reply · James C. · last May 07, 2018
James C.12:18
I installed it from the v3.5 homebrew package. Perhaps that uses a the older default config?
As you have now seen, seeding with those nodes (jonas/todd/voskuil) doesn't work very well 😉
or do there servers just not respond well
*their
neillm11:58
bitcoind changed seed compatibility for some other scheme and no longer works well for us. we still exchange nodes via getaddr, but we seed more directly from libbitcoin-servers now
haarts12:04
What could be the reason for not getting Notifications from the
subscribe.address Query Service endpoint? The request call returns a 0 error code indicating everything is fine. But not Notifications appear on the socket (but they do appear on bx).haarts12:04
I've inspected the payload I put on the wire with Wireshark and compared it with the payload bx puts on the wire and I don't see an obvious difference
certainly, hold on
@neillm This is the originating point in the code: https://github.com/Feathercoin-Foundation/pylibbitcoin/blob/master/pylibbitcoin/client.py#L336-L346
GitHub
Contribute to pylibbitcoin development by creating an account on GitHub.
The file cli.py shows an example use: https://github.com/Feathercoin-Foundation/pylibbitcoin/blob/master/examples/cli.py
GitHub
Contribute to pylibbitcoin development by creating an account on GitHub.
I don't think I can mess up too badly with the request. And libbitcoin-server tells me when I do, it reports a 'bad_stream' if the payload is not of the exact length expected.
I've been looking at other clients using libbitcoin like
python-libbitcoin but I suspect they are using some old message format: https://github.com/libbitcoin/python-libbitcoin/blob/master/libbitcoin/server/client.py#L384-L388GitHub
python-libbitcoin - Libbitcoin Python Interface [contributed library, not maintained].
neillm12:20
I haven't tried just using zmq to communicate with the subscriptions, but ordinarily a callback handler is set and then called when data is received. I don't see anything that just streams the data back over a socket (i.e. response is retrieved and passed to the callback)
Ok, so reply sent. Are you saying you're not getting any response, or an unexpected one?
So the way I understand it is that I send a request to watch an address. I immediately get a response indicating success or failure. THEN, on that same socket, I should occasionally get a Notification message.
That message my software never gets.
The server DOES send it as evident by doing the same watch with the explorer.
I believe this to be the request send by the explorer: https://github.com/libbitcoin/libbitcoin-client/blob/master/src/proxy.cpp#L242-L251
haarts12:42
The link you posted on the notification_worker, is the first argument to that function the same socket as where all the requests from the client end up?
But I think for what you're asking, yes
And this is the part where the server receives and acks the subscription no? https://github.com/libbitcoin/libbitcoin-server/blob/ade2522f2fc4bba03f314b28df3385556148e1ae/src/interface/subscribe.cpp#L34-L53
Thanks
evoskuil13:08
Also, WRT seeding, libbitcoin should be able to seed off of a bitcoind node (or any node that implements the address protocol) just fine. It’s just the seed nodes that are problematic.
👍 1
haarts14:50
Ah! More Wireshark digging revealed a packet being send to my pylibbitcoin socket. It just didn't get picked up by it. I had that problem before with a ROUTER socket not prepending the originating socket.
I don't know why that is ATM... Investigating further.
Tuesday, May 8, 2018
haarts07:38
I believe my problem is due some identity management. The server (a ROUTER socket) send Notifications to my client (DEALER). But that message lacks an identity and my client then drops the message (this last bit is conjecture on my part).
@evoskuil How does this 'routing' stuff work? https://github.com/libbitcoin/libbitcoin-server/blob/master/src/messages/message.cpp#L133-L137
that seems related
M
Mlmikael10:01
@evoskuil Hi Eric, a quick Q, https://github.com/libbitcoin/libbitcoin/blob/version3/install.sh#L475 here you do "export LDFLAGS="-L$PREFIX/lib $LDFLAGS"" with the motivation " # Because ICU tools don't know how to locate internal dependencies." - do you remember what this was about?
M
Mlmikael14:13
btw about Boost not finding ICU when installed in a prefix, that is still the case in latest Boost. maaaybe with more advanced build arguments it could be fixed though, https://www.boost.org/doc/libs/1_66_0/libs/regex/doc/html/boost_regex/install.html discusses it at length, i'll look into it.
haarts15:37
I've successfully created a minimal subscription script and that works as expected. The problem must be in the way I handle the event loop in Python!
👍 1
evoskuil15:55
@haarts the server supports a REQ or DEALER client. REQ clients do not delimit the identifier but DELAERSs do, because there can be more than one. It’s also possible for a DEALER to explicitly not include the delimiter. So the server remembers whether the request was delimited and reconstructs the response in the same manner.
@Mlmikael if you can explicitly direct boost to the prefix dir without the installer hack that would be great, but we don’t want it to “search” in a manner that allows it to be found elsewhere.
Wednesday, May 9, 2018
haarts07:21
@evoskuil the delimiter being empty frame right? And you are alluding to this piece of code: https://github.com/libbitcoin/libbitcoin-server/blob/ade2522f2fc4bba03f314b28df3385556148e1ae/src/messages/message.cpp#L133-L137
At the moment the message looks like this from my end:
[---command----] [--identifier--] [---payload----]
All the envelope stuff I seem to be able to ignore.
(edited)From the server code I understand that ZMQ puts an address in from of this. And I've played with it in a spike and that seems how it works. The address is then some random bytes. But you're saying that the DEALER can set this address too? And that that is even preferred in the future?
evoskuil17:03
Yes. It could be that pyzmq is managing the envelope. Every routing step incorporates addressing so that responses can be routed. A REQ is a singular route for that end of the socket, so addressing is simpler.
The original libbitcoin client DEALER implementation did not follow the recommended practice of adding the optional delimiter. Therefore, for backward compatibility reasons, the newer server allows both REQ and DEALER connections but does not require the delimiter. For this reason it must manage whether the messages are delimited. If not for this issue we would simply have required the delimiter.
(edited)👍 2
Thursday, May 10, 2018
N
narodnik15:33
Sorry haarts, I just saw your message and added the COPYING file with the license, copied from libbitcoin
James C.15:44
Hello @evoskuil @neillm, I am working myself up the server api/protocol/client stack in terms of developer documentation. I have a brief question on the pure virtual "stream" class in the client library. Perhaps more c++ related, but I don't quite understand why a (pure virtual) stream reference is used in the read method: https://github.com/libbitcoin/libbitcoin-client/blob/master/include/bitcoin/client/stream.hpp#L39
(edited)Does that mean any object reference of a class derived from "stream" can be accepted in that parameter, such as proxy? Such as here: https://github.com/libbitcoin/libbitcoin-client/blob/master/examples/get_height/main.cpp#L76
(edited)neillm15:49
@James C. Yes, that looks correct. The reference just means you need an actual existing stream instance, where a copy/temporary isn't acceptable
(edited)James C.15:52
ok great, thx @neillm! Good to know ... so I will study what happens when proxy, dealer, socketstream objects are passed thru next ...
The pure virtual method just means that anything can be a stream so long as it implements the read (and that there's no fallback/base behaviour)
You can see that "stream" is defined as an interface for something else to implement (also in the comment). So dealer implements it. I think proxy does not, so it would use the dealer implementation
And not to put a damper on anything, but libbitcoin-client is due for a re-write/re-structure, which may happen before v4 is released (I believe the stream will go away). You'd have to check with @evoskuil about the timing though, not sure where it is just yet priority-wise.
James C.15:58
brilliant -> that makes sense, I can see the implementation in dealer now, as in dealer.write()
James C.15:59
thanks again neill 🙂 Ill take note of the stream rewrite, and comment here before when potentially I get to writing examples, but will try to review v3 for my understanding anyway.
N
narodnik17:45
Hello guys, I just finished my ring signatures implementation
https://github.com/libbitcoin/libbitcoin/pull/947
I'm going to take some time to review this code during the next week or two. I would appreciate others looking over it too. I can answer any questions about the algorithm.
I know this mathematical scheme pretty thoroughly now
There is a unit test which can be run with make check
GitHub
This adds Borromean ring signatures. The unit test is exactly the same as the experimental secp256k1-zkp branch. I will spend more time verifying and auditing this code, but it will also require so...
Friday, May 11, 2018
M
Mlmikael10:27
@evoskuil our experience with attempting to build LibBoost against ICU installed in a prefix, says that the experience you had&have with LibBoost 1.57.0 still applies:
LibBoost 1.66.0 also fails at detecting ICU in a prefix.
Incredibly weird that in all these years they haven't fixed it.
We're at like the 4:th week of building LibBitcoin now, almost there! lolol
only like 4 weeks full time
something must be wrong lolol
2 ppl
M
Mlmikael16:24
noo, the Boost Locale library was still poisoned with linking to the wrong ICU library file
Boost Regex not interestingly, and neither any other boost library.
Sunday, May 13, 2018
James C.09:17
Hello Libbitcoin Team! Quick question regarding setting certificate for zmq curve. I can't find any official documentation on the following comment in the protocol::socket object source:
// Use on client for both set_public_key and set_private_key from a cert.
// If CURVE is not required by server, call set_certificate({ null_hash })
// to generate an arbitrary client certificate for a secure socket.Is the null_hash passed to zmq with ZMQ_CURVE_SECRETKEY/ZMQ_CURVE_PUBLICKEY set to null, and zmq automatically generates a temp pair? If "curve is not required by server", doesn't the client have to also set ZMQ_CURVE_SERVER to true, since the (libbitcoin)server is not the curve server? Couldn't find answers to this case in the zmq docs. Any pointers much appreciated!
M
Mlmikael10:09
@evoskuil @neillm does libbitcoin-database depend on LibGMP and LibSodium? 😮 i didn't see that in the documentation anywhere
since i'm building in GCC on this otherwise-clang machine, libbitcoin-explorer crashes at start for this reason
because i have both a dep. to GNU's libstdc++, and a dep,. to clang's libc++ and libc++abi , and those don't work out of the box in tandem.
PKG_CONFIG_PATH does not suffice.
M
Mlmikael13:25
https://en.bitcoin.it/wiki/Libbitcoin_Common#Dependencies_.28Version3.29 - GMP and ZMQ should be added here
@evoskuil it works!!
FINALLY lol
and we became veeery good at the intricacies of the deps in the process lolol
M
Mlmikael13:48
Wait, does LibBitcoin have any dependency on LibSodium????
ZMQ can use LibSodium BUT it has its own internal logic implemented already so sodium is optional to it.
I read in I think the "libbitcoin" repo some "sodium.hpp" file, but then now i think we built all the libbitcoin repoes and "ldd" show NO dependency on sodium anywhere, @evoskuil are we sodium-free? 😃
M
Mlmikael14:47
then a contextual question - where/how is install.sh generated? it looks human-made to me but says "machine-generated". what's the point?
libbitcoin doesn't depend on libsodium
evoskuil16:04
libzmq includes tweetnacl. All build artifacts are generated, for reliability and ease of maintenance.
James C.16:12
Hi @evoskuil @neillm - reposting this Q in case it got pushed back to far 😉
James
Hello Libbitcoin Team! Quick question regarding setting certificate for zmq curve. I can't find any official documentation on the following comment in the protocol::socket object source:
```// Use on client for both set_public_key and set_private_key from a cert.
// If CURVE is not required by server, call set_certificate({ null_hash })
// to generate an arbitrary client certificate for a secure socket.```
Is the null_hash passed to zmq with ZMQ_CURVE_SECRETKEY/ZMQ_CURVE_PUBLICKEY set to null, and zmq automatically generates a temp pair? If "curve is not required by server", doesn't the clien…
@evoskuil libsecp256k1 depends on libGMP, and you do not build libGMP
neillm16:49
@Mlmikael Ok. You asked where install.sh was generated from. It's generated from libbitcoin-build.
M
Mlmikael16:49
yes i agree that custom-building all libbitcoin's dependencies yourself, except for C library, makes all sense in LibBitcoin where you need so high determinism
@neillm ah right, yes i know that but the content of the libbitcoin-build repo doesn't make sense to me 😃 what is it? =)))
M
Mlmikael16:55
@neillm ah, so you are not using third party software but it's your own homemade script that makes the install.sh:s - ok
ok i must go now, anyhow,
incredibly relieved to finally have "bn" syncing here. it was about time. :))
evoskuil17:00
GMP is an optional dependency of libsecp256k1 which we don’t require via the standard install.
You should not bother with libbitcoin-build unless you are a libbitcoin maintainer.
Monday, May 14, 2018
M
Mlmikael03:06
@evoskuil, libsecp256k1 did build against GMP by default on my machine. I got the impression it's an obligatory dependency.
M
Mlmikael03:23
https://github.com/libbitcoin/secp256k1/blob/master/configure.ac#L145 , ahh, there's a "--with-bignum=no" option whereby you can disable gmp
M
Mlmikael03:30
urr. why does it default to "auto" and so attempt to link to GMP by default?
the issue i got on my machine was that GMP, which has C++ components, was built by clang and so it caused my libbitcoin build to sigsegv at start.
evoskuil04:05
Probably because it’s more efficient with it linked and so if it’s installed it will link it. We don’t require it (if not installed it doesn’t link). We don’t however attempt to fix its “auto” which apparently cannot tell a clang compiled version from gcc.
(edited)M
Mlmikael07:17
@evoskuil interesting. the regarded secp256k1 functions are used in block validation right, so the speedup from GMP will lead to faster full sync speed right. i see some mention of a GMP vs not GMP benchmark here https://www.reddit.com/r/Bitcoin/comments/2weymr/experiment_bitcoin_core_0100_initial_sync_time/coqepsl/ , would be interesting to learn more, anyhow that is not prioritary on my end, now time to learn more about libbitcoin.
reddit
There was a build flag to do that (for testing purposes), but I think it stopped working sometime during the past couple months. I don't know if...
M
Mlmikael11:41
@evoskuil, syncing test net just died here:
11:38:17.107238 INFO [blockchain] Block [82700] 3 txs 3 ins 0 wms 0 vms 0 vµs 0 rµs 0 cµs 0 pµs 0 aµs 0 sµs 0 dµs 0.000000
and attempting to resume it gives "failure starting blockchain"
bn-windows-x64.exe -c libbitcoin-testnet.conf
11:39:38.739679 INFO [node] ================= startup 05/14/18 19:39:38 ==================
11:39:38.746661 WARNING [node] ================= startup 05/14/18 19:39:38 ==================
11:39:38.752644 ERROR [node] ================= startup 05/14/18 19:39:38 ==================
11:39:38.757631 FATAL [node] ================= startup 05/14/18 19:39:38 ==================
11:39:38.764613 INFO [node] Using config file: "libbitcoin-testnet.conf"
11:39:38.769600 INFO [node] Press CTRL-C to stop the node.
11:39:38.772593 INFO [node] Please wait while the node is starting...
11:39:38.778578 ERROR [node] Failure starting blockchain.
11:39:38.782565 ERROR [node] Node failed to start with error, operation failed.
11:39:38.787551 INFO [node] Please wait while the node is stopping...
11:39:38.791541 INFO [node] Node stopped successfully.
now on second try, it works, interesting.
in testnet sync. ctrl+c leads to a shutting down message but after 35 minutes it did not shut down still. to debug this further, some shortcut to get a printout of LibBitcoin's internal state could be useful, could it not?
haarts14:09
@evoskuil I'm currently rooting around in the source of Electrum. Quite a bit of effort has been expended to make it able to deal with loads of requests. I'm only aware of Electrum being used as a desktop wallet. Are there any other use cases?
evoskuil15:52
@Mlmikael if you terminate the process during an unflushed write the store is corrupt. There is zero point in inspection of the state. If it tells you it is corrupt, it is 99.999% chance it is true. If it doesn’t tell you it is corrupt it is 100% chance it is correct unless you have a bonafide hardware failure (which can’t be reliably detected).
@haarts I don’t know, are you talking about the client?
@Mlmikael You must have deleted the flush lock file to get it to restart. Don’t do that, your store is corrupt.
M
Mlmikael16:38
@evoskuil wait, the two situations were these:
first: i had just initialized the blockchain store ,and did the first sync. after 82000 blocks, "bn" just shut down all by itself. when i tried to start it again, it told me the blockchain database was broken.
so this was a non-sigsegv crash, which was weird. not sure how i'd debug it if it happens again!
second: i removed the old blockchian database and started syncing the new one, up to something like 130 000 blocks, then i pressed ctrl+c and it said Stop signal detected, Please wait while the node is stopping. then 30 minutes passed and it did not stop. pressing ctrl+c more times did not stop it also.
here maybe some debug shortcut so i could report back to you what was actually going on, would be useful, so we could figure what part of libbitcoin it was that erroneously not wanted to shut down. however, this is v3 and you're making v4 which i guess will have quite other logics anyhow, so maybe better debug this at first later.
(so these two instances of unexpected behavior were not related to trying to force LB to operate a blockchain store that was market broken.)
evoskuil16:57
In this case you have an unknown failure condition which nobody has reported to date in v3. If the process crashes you should obtain the dump. 30 minutes is long for shutdown, even during initial block download, unless maybe you have a very slow machine with a HDD, but I doubt it. In this case the debug log can be helpful. But given you produced two failures in two runs, where there have been none reported in about a year time frame, I suspect something is different in your build or that there is actual hardware trouble (though this tends to be rare).
Given you spent a couple of months getting a custom build to work on a yet-unsupported platform, I suspect the build, not the hardware.
The two failures may be related, but a shutdown hang is a different class of failure than a process abort. Hang implies the threadpool is not stopping due to an unreleased resource. This can sometimes be seen in the debug log. Crash typically implies out of memory, handles, or access of unowned memory.
The stack trace can be helpful in analyzing that, as can running with tools such as valgrind.
I’ll be on the road most of the next two weeks, so won’t be much help.
evoskuil18:37
Just saw you are using the last windows binary. That’s not a current build, prob getting close to a year old.
That’s version 3.2 released May 2, 2017. Need to review for issues closed since then.
evoskuil18:50
Could be this for example: https://github.com/libbitcoin/libbitcoin-database/issues/159
(edited)Tuesday, May 15, 2018
haarts07:30
@evoskuil yes, I'm talking about the client (https://github.com/spesmilo/electrum/). @lclc thought you'd know.
Wednesday, May 16, 2018
haarts14:57
Ah, seems that Electrum, then, is not making bloom requests.
I'm looking for an alternative to ElectrumXs
get_merkle query (http://docs.electrum.org/en/latest/protocol.html#blockchain-transaction-get-merkle)evoskuil15:05
Right, it does not use p2p. Libbitcoin has a similar call as a zmq query, see protocol spec on server wiki.
Thursday, May 17, 2018
M
Mlmikael09:27
Hi @evoskuil, i am seeing recent "bn" distribution EXE on Windows 10 fail for a second time by never terminating at ctrl+C now, should I make a github issue of it?
"8:38:31.320531 INFO [node] Please wait while the node is stopping..." like 20 minutes ago. last time i waited 35mins.
pressing ctrl+c more times each time yields "INFO [node] Stop signal detected (code: 2)." but does also not make bn shut down
M
Mlmikael09:34
Also one more question, adding "peer=127.0.0.1:18333" to [network] in my libbitcoin-testnet.conf , to connect to my local Bitcoin Core 0.16.0, gives an unending stream of this:
09:33:01.883035 INFO [network] Manual channel failed to start [127.0.0.1:18333] channel stopped
09:33:01.946866 INFO [network] Manual channel failed to start [127.0.0.1:18333] channel stopped
09:33:01.967810 INFO [network] Manual channel failed to start [127.0.0.1:18333] channel stopped
09:33:01.974790 INFO [network] Manual channel failed to start [127.0.0.1:18333] channel stopped
09:33:01.978781 INFO [network] Manual channel failed to start [127.0.0.1:18333] channel stopped
what's the sense here?
@haarts to your last question i think the answer is yes
(i now made sure that "bn" not tries to bind localhost:18333 too, by specifying "inbound_port=18334" under [network], however the channel stopped problem remains. testing with adding a connection to 18334 from within bitcoin core .. )
M
Mlmikael09:46
also "-addnode=127.0.0.1:18334" (or from console, "addnode "127.0.0.1:18334" "add"") in Bitcoin QT, fails at producing a link between Bitcoin Core and LibBitcoin, hm.
which were recently added
M
Mlmikael10:40
(ah, i should also say that there were occasions when "bn" has closed as it should on ctrl+c..)
M
Mlmikael14:03
@evoskuil any idea how I make LibBitcoin and Bitcoin Core connect to each other? obvioulsy they do on the open internet, so i'm asking myself why they fail in the case of a manual local connection. any idea?
M
Mlmikael16:36
the "bn" binary connects to the localhost bitcoin core and transmits about a megabyte, and then drops connection, and starts over. also connecting the other way does not work.
16:34:30.977975 INFO [network] Connected manual channel [127.0.0.1:18333] as [127.0.0.1:18333] (1)
16:34:31.200378 INFO [network] Connected manual channel [127.0.0.1:18333] as [127.0.0.1:18333] (1)
16:34:31.450673 INFO [network] Connected manual channel [127.0.0.1:18333] as [127.0.0.1:18333] (1)
16:34:31.667472 INFO [network] Connected manual channel [127.0.0.1:18333] as [127.0.0.1:18333] (1)
16:34:31.959779 INFO [network] Connected manual channel [127.0.0.1:18333] as [127.0.0.1:18333] (1)
16:34:32.184180 INFO [network] Connected manual channel [127.0.0.1:18333] as [127.0.0.1:18333] (1)
here is my conf:
@Mlmikael uploaded a file: Untitled
Untitled
[network] # The time limit for connection establishment, defaults to 5. connect_timeout_seconds = 30 peer=127.0.0.1:18333 identifier=118034699 inbound_port=18333 inbound_connections=0 outbound_connections = 8 # Testnet seed nodes. seed = testnet-seed.bitcoin.jonasschnelli.ch:18333 seed = seed.tbtc.petertodd.org:18333 seed = testnet-seed.bluematt.me:18333 seed = testnet-seed.bitcoin.schildbach.de:18333 seed = testnet-seed.voskuil.org:18333 [database] # The blockchain database directory, defaults to 'blockchain'. directory = blockchain-testnet [blockchain] # The number of cores dedicated to block validation, defaults to 0 (physical cores). cores = 1 # Use high thread priority for block validation, defaults to true. priority = false
this is the Win10 binary. it should be complete and work out of the box, should it not? any idea what i should look for?
anyhow i should try with the mainnet too.
the big defining difference between mainnet and testnet is the "identifier=118034699" right?
Yes, but you need at least the genesis block of whichever network you're running
M
Mlmikael16:41
template here, https://github.com/libbitcoin/libbitcoin-server/wiki/Blockchain-Settings , right
right. how does LibBitcoin choose which?
neillm16:43
For testnet: # Testnet checkpoints.
checkpoint = 000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943:0
In the default config, you need at least the first one uncommented (block 0) for either mainnet or testnet
See the bottom of the config section of the link you pasted
evoskuil16:46
If you don’t at least specify the genesis block it defaults to standard mainnet checkpoints.
hm
i thought a bitcoin node was supposed to bundle the genesis block by itself
but so you can just connect to another peer and ask it for the genesis block?
neillm16:48
It does. But that's different than specifying a starting point via config for validating blocks on top of, which is more or less what a checkpoint is used for
For example, if it wasn't a config option and we always expected the mainnet genesis block from peers, testnet would not work. Let alone alternative coins that people are playing with
wait, does Bitcoin Core and LibBitcoin bundle the 1MB genesis block image (so, two actually, one for testnet and one for mainnet)?
N
narodnik17:02
Eric, there's a bug in secp256k1-zkp. It's not generating proper random numbers. Should I just generate my own test vectors?
for example
k[0] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
k[1] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
k[2] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
k[3] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
sec[0] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
sec[1] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
sec[2] = 3a 26 d7 57 f ac bc 44 5 c9 9d 6 f9 bf 73 ba e3 eb 61 9f ac 5b ce f7 b1 34 37 2d 93 26 26 ef
sec[3] = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
s_(0, 0) = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
s_(0, 1) = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
s_(1, 0) = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
s_(1, 1) = ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fe ba ae dc e6 af 48 a0 3b bf d2 5e 8c d0 36 41 40
s_(2, 0) = c5 95 7a a5 dc 57 3c e d f5 e1 7d fd 2d 7c 8 d6 5d 9c 64 40 89 7a b1 75 44 ec b7 35 98 af 88
s_(2, 1) = 19 35 4f e4 b0 8a f2 1f 9f 4d 66 4b be a3 6b ec 9d 76 24 48 26 7c 68 66 da 8 9 55 97 3a 78 a0
s_(2, 2) = 65 f9 a3 c1 38 be a1 c 91 fb 93 bf a4 6b 7c 82 3d 1f 45 df d4 67 b5 d0 e5 6c 9c d1 48 86 af f5
s_(3, 0) = 78 8 43 86 51 79 53 71 26 44 ad 1e 0 44 96 9e f9 95 d6 33 dc 90 a2 69 e5 8e 4 1e de c3 d 80
s_(3, 1) = 0 32 c3 e9 44 aa 41 4b c c0 c9 4e 89 83 46 24 ed eb 32 51 c4 b3 77 f0 48 d6 24 f2 af 79 78 37
Another run
k[0] = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 k[1] = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 k[2] = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 sec[0] = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 sec[1] = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 sec[2] = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 s_(0, 0) = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 s_(0, 1) = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 s_(0, 2) = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 s_(0, 3) = 63 7b 93 e2 9a 73 fd 9e 26 d5 4b 8a 6e ab b2 3 ad b9 d 8d 62 49 36 cf 42 94 4 f1 5b cb 7a c7 s_(0, 4) = 82 e2 52 2a e8 d1 51 e1 6e ff 61 8f 98 19 88 fd a1 77 e8 97 a6 fa f8 55 3f 82 ca 2c 5e 69 ef c7 s_(0, 5) = b5 1d 16 23 b6 90 73 a8 62 28 bd b3 e4 a5 c4 36 b4 e1 86 3a 4c af 62 a9 93 11 e4 67 b7 6c 88 f9 s_(1, 0) = f0 93 dd 8d c0 b7 5c 43 10 a3 f6 0 4a e 61 b3 10 4e af 2f fd b9 40 d7 8a 39 d4 8c 62 b3 56 99 s_(1, 1) = c5 bc b9 a5 e7 25 2c ba 34 68 7d a3 9e c0 35 ba 32 a2 46 30 b2 5d 52 6d ce 7f 49 1f cc 48 6a bd s_(1, 2) = 6f e0 6d 1d f4 ef 69 b2 27 86 ea a9 6d 4 9 81 b6 a3 2f 7b c7 9e af 8a 95 30 fa c1 e2 2 85 f6 s_(1, 3) = 38 56 a2 8f 64 6b 24 ab 68 93 5f ca 10 d3 9d 7 20 36 b0 bd 40 93 5 94 96 c4 f6 9b d8 d9 6b d1 s_(1, 4) = 6d fa a8 9f 58 2c 6 a5 60 42 d0 53 20 16 9c ba 76 9a 6d 6e ef 88 42 ac 75 cb bd 95 27 8c 9d 35 s_(2, 0) = a5 4 cc 6e f4 f8 4e 3f 91 a a8 fd 87 98 12 99 c5 76 22 7c 32 b0 38 89 7d 15 6d 8e c0 a0 a4 36 s_(2, 1) = 5e 81 60 34 86 14 af 88 21 f2 24 3b 8f 12 fe 84 a9 e2 69 76 12 12 87 6 c2 11 b7 9f ef 60 c4 6e s_(2, 2) = b9 24 76 fc fe 75 22 9b 2 2e c0 10 2c 8c eb 5a 7 54 33 27 93 4a ab 1 73 3f ee 7 c4 9e e7 c
neillm17:04
@narodnik If you take those starting points, does your code produce the same result? If so, use those as the random vectors?
But it's broken. I think I will just generate my own random values.
N
narodnik17:06
those are the values used for generating the signature. But k should be completely random. The equation is s = k + ex
The verifier knows s and e. The equation works because k is meant to be random and x is your secret key
If k is not random, then you can calculate x
(or pre-computed)
and the same s values.
maybe they're actually trying to test corner cases, by generating one often
Friday, May 18, 2018
M
Mlmikael02:30
@narodnik if the objective is real randomness, than that code snipped you showed now, artificially lowers it (the entropy). is that a problem?
James C.16:17
Here are my first documentation chapters to the Server API, took longer than I planned, but hope they can be helpful. Thanks to @neillm @haarts for the help on this! Feedback/Corrections welcome!
https://github.com/libbitcoin/libbitcoin-server/wiki/Simple-Query-from-REQ-sockets
https://github.com/libbitcoin/libbitcoin-server/wiki/Subscriptions-from-DEALER-sockets
https://github.com/libbitcoin/libbitcoin-server/wiki/Round-Robin-Queries
(edited)👍 1
@evoskuil should I add an alias typedef for ec_secret -> ec_scalar in the code, since my use of ec_secret seems to confuse you.
typedef ec_secret ec_scalar;
evoskuil20:13
Can you give me a link to the use you are referring to? I’m traveling so it’s a little hard to follow. But generally if you are using the type ec_secret for something that is not an elliptic curve (secp256k1) secret then you should declare a new independent type.
they don't always represent secret keys, but usually that's how the API user will be using it for
the scalar (secret) is not always a secret value, sometimes it's just a number
(or types)
evoskuil22:05
I’m thinking then that we declare ec_secret as a type of ec_scalar and declare the corresponding parameters of each fn as ec_scalar.
Saturday, May 19, 2018
skaht21:11
To maximize the portability of BIP 39 across different spoken languages does it make sense to have the inverse of mnemonic-new? A quick look at the most recent posting at https://bitcoin.stackexchange.com/questions/68605/how-to-generate-a-valid-hash-for-a-bip39-seed-phrase shows this can be accomplished. Additionally, the Electrum Standard used by Monero is invertible.
Bitcoin Stack Exchange
I'm trying to prove to myself that I understand how bip39 seed phrases work... I'm doing most of this in a spreadsheet because I thought it would be easier but I'm having trouble with the checksum...
evoskuil21:22
Maybe a better question for the authors of bip39, but it is specially not intended by the proposal.
Sunday, May 20, 2018
I am negating and modifying values in the data to make sure the verify fails.
evoskuil14:06
I’m not certain I know what you mean, but as long as the test vectors don’t change it’s fine.
Monday, May 21, 2018
anyway I updated the unit test with test vectors exported from secp256k1-zkp
I would expect verify to fail?
https://github.com/libbitcoin/libbitcoin/pull/947 <- I've finished the last changes to this pull requests
if anyone wants to help review
neillm16:11
@narodnik Have you talked to @evoskuil about pulling the -zkp branch in? That seems a requirement, but I'm not sure of the state of that branch
and I don't think it's needed
this pull req doesn't depend on it.
evoskuil16:13
The libsec apis fail if values are invalid, but in some cases it may not be practical to find a failure vector.
aah just answered above. shouldve read
Revive19:41
@Revive uploaded a file: maybe it is needed?
maybe it is needed?
src/math/elliptic_curve.cpp:24:10: fatal error: secp256k1_recovery.h: No such file or directory #include <secp256k1_recovery.h>
neillm20:13
@Revive Try installing from the install.sh script, that shouldn't be an issue if all the dependencies are installed properly
evoskuil20:30
They aren’t, and they aren’t defined in -recovery either. I couldn’t see the cause of the missing definitions. Must be something too obvious.
Revive21:53
ah i misread my error message. so what, i needed to compile libbitcoin/secp256k1 with some more options? just curious what was missing from the autotools process
neillm23:44
@Revive Probably this: https://github.com/libbitcoin/libbitcoin/blob/master/install.sh#L404
Revive23:47
Ah yes, I didn't understand the readme. It appears to me to say that attribute is only if you use bitcoin cores secp256k1 version.
The versions are the same, we just branch them to maintain version control.
Since they don’t re-version the lib when they make breaking interface changes.
(edited)OK thanks
Tuesday, May 22, 2018
M
Mlmikael06:38
Question, when how do I retrieve all transactions within a block? i see that block_database gives you a block as a block_result structure, but i don't see (right now) how to retrieve the contained transactions, except for doing separate resolves based on transaction hash, to the transaction_database.
M
Mlmikael07:09
does the "block" class contain all transactions? i don't see any methods or properties that look like that, but it looks like its constructor can take transactions as argument
(edited)M
Mlmikael07:19
question, in the libbitcoin repo, the "message" directory what's special about the classes there? e.g. message::transaction is a subclass of chain::transaction, why the need for message:: variants of chain:: stuff like transaction, block, is this some kind of higher level structure that's better for passing around between threads for some reason?
aha the block_chain class is a subclass of safe_chain, great (Y) 🙂
how did the word "organize" get the meaning "submit"? e.g. there's a "transaction organizer", that you make "organize" a transaction for you which really means to submit a transaction - just for context 🙂
is it, for instance, an index where the indexing key is bitcoin address and the value is an ascending in the order of block height and transaction index position, list of block height + transaction index
or instead is the value not block height + transaction index, but transaction hash? is there some other ordering
btw also curious, how does LibBitcoin's UTXO database look, do you have an integer field for each UTXO saying "this one was spent at height N by transaction number M that has hash O", or what are you doing 🙂
N
narodnik14:15
@evoskuil the missing definitions are because libbitcoin secp256k1 needs to be updated
@Mlmikael sorry what is your question?
@Mlmikael you should just be dealing with the blockchain/interface/block_chain.hpp file
@Mlmikael the message classes are what the Bitcoin network passes around, the messages like version, verack .etc
but the actual blockchain types are in chain/ and are different
evoskuil14:30
The bc::chain types are the nominal definitions for the conceptual blockchain data structure. Some of the bc::message types inherit these, and make up the p2p protocol.
N
narodnik14:37
@Mlmikael the address is stored on disk as [address] -> [index] [index] -> linked_list(address rows) ... when you get the result with address_database, instead of it deserializing all the data at once, it returns an iterator (libbitcoin_database:include/.../results/address_iterator.hpp) . Dereferencing the iterator calls chain::payment_record::from_data() in libbitcoin:
https://github.com/libbitcoin/libbitcoin-database/blob/master/include/bitcoin/database/result/address_iterator.hpp
https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/payment_record.cpp#L182
GitHub
libbitcoin-database - Bitcoin High Performance Blockchain Database
the reason you have that pre-lookup from [address] -> [index] is because you don't want to store every new row with the full address key again of 20 bytes. That's inefficient.
M
Mlmikael14:58
@evoskuil here is a more involved matter, sorry for bugging you with a big topic :)) also @neillm 🙂
what is LibBitcoin's threading model?
so, LibBitcoin is allocating callback handles all over the place, right, e.g.
node_->start(
std::bind(&executor::handle_started,
this, _1));
in what threads are these run? what's the "scheduler"?
is there any tracker/enumeration facility that shows what's going on?
like, thread 1 is in the p2p module doing handle_started, thread 2 is having a queue of 5 tasks to be done namely .. , and so on.
next, what is the memory model in LB
do you do any refcounting?
i see the use of std::make_shared / std::shared_ptr and of std::ptr all over, what's the point here.
e.g.
session_seed::ptr p2p::attach_seed_session()
{
return attach<session_seed>();
}
(edited)any notable comments about what parts of or functions in LB are threadsafe and which are not?
what settings about how many OS threads to use, are there+
evoskuil15:07
This is too much to text... it took me a long time to get clarity on it. Proactor is unnatural :).
@evoskuil, can you comment each of these questions just a bit so i get some idea about what to look for
nothing exhaustive
like, working within LB, or bridging to it, is like wild guesswork when i'm just guessing the concurrency model for execution and data, lol
so just some pointers here for navigation would be great.
channel_subscriber_(std::make_shared<channel_subscriber>(threadpool_,
NAME "_sub"))
ah and last, locking, what's the big picture there
i see LB is using mutexes for instance, are those used just to serialize execution in some small parts only (e.g. when "organize":ing = submitting a transaction), or is it uses for resource management on a more complex level, or bot
Proactor is an execution pattern, it’s the inverse of what you expect.
M
Mlmikael15:16
@evoskuil https://en.wikipedia.org/wiki/Proactor_pattern , i think this is obvious
Wikipedia
Proactor is a software design pattern for event handling in which long running activities are running in an asynchronous part. A completion handler is called after the asynchronous part has terminated. The proactor pattern can be considered to be an asynchronous variant of the synchronous reactor pattern.
maybe it's just me.
@evoskuil what about operating threads, where is your operations scheduler?
also who / what / where is memory management?
And schedules threads.
but LB as a C++ program, asks the OS to start thread
so the threads are "manually managed".
LB is multithreaded.
where do you start threads? how many do you start?
maybe it was just an incorrect interpretation from my side, but, it seems to me that some of these handlers are intended to be used as work tasks for execution in *other' threads
so sure in a local thread's execution you run various callback handlers and what not, but internally in the ordinary flow of carrying out the peer 2 peer networking communication, validating transactions, operating the database and its indexes, aren'd you juggling / passing jobs between threads?
and when you do it you do it by locking up methods as std:bind instances too, then pass it to some job queue which eventually dispatches it in the suitable other thread - or?
each service has a threadpool, and distributes its work evenly across the threadpool through boost::asio
threadpool pool(4); // threadpool with 4 threads
blockchain chain(pool);
N
narodnik15:27
every time you or some code calls an operation on blockchain, it will immediately return, and put the work on the stack. The proactor loop (in boost::asio) will take the work off, call it then call the completion handler you passed when you first called the method.
and where in the code is that?
the threads setting under each subsection
M
Mlmikael15:36
is there any point where an LB node could get overloaded so it would stack up on an infinity of work tasks,
M
Mlmikael15:36
so that now that CPU is totally consumed, work tasks will keep accumulating and you run out of RAM?
M
Mlmikael15:36
right, the "threads" config setting means "# The minimum number of threads in the network threadpool, defaults to 0 (physical cores).".
is the "threads in the network threadpool" all threads that are ever created by LB?
for instance, "bs", are the RPC threads included in this count
or is only like, block validation work threads counted by the "threads" value
evoskuil15:38
The model is more complex than Amir remembers. Large tasks bypass the queue and are executed sequentially. This limits the peer’s control.
But small tasks can still fill the buffer.
so "0" means it'll autodetect how many CPU cores you have, and create e.g. 1 or 2 or 4
M
Mlmikael15:40
are these all the threads created by LB, which will do both RPC server serving, block syncing, network connection work, and so on?
are you using some asynchronous IO library to manage peer TCP connections, you could have some thousand inward TCP connections right
so you have a crossplatform-select() call somewhere
libzmq's scheduler (e.g. select() ) requires an exclusive OS thread to operate, doesn't it
M
Mlmikael15:48
do you have.. hmm... so like, when there's no more data from a Bitcoin p2p connection, i presume you... hand back the socket to Boost ASIO so that its select() will include that socket too, right.
when there's more data, i guess Boost ASIO trigs some "got more data" handler in the libbitcoin-network repo (a.k.a. p2p)?
is Boost ASIO doing work scheduling for you, so like you launch those OS threads and hand over its operation all to Boost,
and then Boost will dispatch all the std:bind - bound handlers, which work as your job units, on any OS thread presentyl is doing nothing?
ok, i should round up for today around now -
about memory management, do you use any refcounts, or like how do you track whether an std:bind handler should stay around on the heap or be delete:d (destroyed and its memory free:d)
Large tasks bypass the queue and are executed sequentially.
@evoskuil how do you define "large task" here?
But small tasks can still fill the buffer.
@evoskuil which buffer?
M
Mlmikael15:57
ah right, smart pointers = refcounting (https://en.wikipedia.org/wiki/Smart_pointer). noted.
Wikipedia
In computer science, a smart pointer is an abstract data type that simulates a pointer while providing added features, such as automatic memory management or bounds checking. Such features are intended to reduce bugs caused by the misuse of pointers, while retaining efficiency. Smart pointers typically keep track of the memory they point to, and may also be used to manage other resources, such as network connections and file handles. Smart pointers were first popularized in the programming language C++ during the first half of the 1990s as rebuttal to criticisms of C++'s lack of automatic garb…
@evoskuil how do you define large task here?
and which is the buffer / queue?
wait, how is work transfered between OS threads in LB?
i guess for instance you spread block validations across threads
M
Mlmikael16:01
you can't just "this_other_thread.DO_THIS_NOW( std:bind .. );" in C++, right, so you are having some facility in each OS thread to make it do stuff right
so the "threads" setting is the global cap LB applies for OS thread count?
oh wait, the definition is minimum number of threads - "The minimum number of threads in the network threadpool, defaults to 0 (physical cores)."
when would more OS threads be allocated?
evoskuil16:08
I suggest you follow the code, since I’m in Tokyo getting drunk and my advice is diminishing in quality.
M
Mlmikael16:10
evoskuil> The bc::chain types are the nominal definitions for the conceptual blockchain data structure. Some of the bc::message types inherit these, and make up the p2p protocol.
aha, so the bc::chain classes are more oriented toward the storage form, and the bc::message types more toward p2p protocol communication, hk.
M
Mlmikael16:16
the reason you have that pre-lookup from [address] -> [index] is because you don't want to store every new row with the full address key again of 20 bytes. That's inefficient.
@narodnik wait what does [index] denote here?
@narodnik the "payment_record" structure, is that the structure that the address index lookup contains a list of?
wait can you point out some real world example where that iterator is walked.
Maybe last question today, the safe_chain class, what's the point with calling it "safe" - is it in the sense that it promises to have a consistent view, in other words it's a view of the blockchain as snapshotted at the time of its creation??
each service has a threadpool, and distributes its work evenly across the threadpool through boost::asio threadpool pool(4); // threadpool with 4 threads blockchain chain(pool);
@narodnik ahh, so each service (of which there is a handful at least within an LB node) has its own separate threadpool - how many threads are we talking in total?
every time you or some code calls an operation on blockchain, it will immediately return, and put the work on the stack. The proactor loop (in boost::asio) will take the work off, call it then call the completion handler you passed when you first called the method.
@narodnik aha right here we go. now, can this be used as a way of load balancing work accross OS threads??
e.g. the user code is in OS thread A, and asks blockchain to do some work, meanwhiel Boost ASIO's proactor loop is running on threads B and C
the address is stored on disk as [address] -> [index] [index] -> linked_list(address rows) ...
@narodnik wait, linked list of "address rows" - what's the exact definition of "address row" here and in what order are they?
M
Mlmikael16:24
in LB's use of shared pointers, is there ever any cascading / propagation of refcounts, so you have any complex memory management situations? (e.g. as not to get circular loops, or to destroy things in a proper order, or some other ordering problem)
Was able to shutdown mainnets, so no resync should be needed. But power is still out, not sure when coming back 😓
Wednesday, May 23, 2018
neillm00:06
That's a bummer. Good luck. Power is back, hopefully for good. Will be a bit before I can get to the servers, but hopefully online within an hour or two.
haarts06:56
In order to create an SPV client I need a Merkle branch for a specific transaction. ElectrumX implements an API endpoint for that: https://electrumx.readthedocs.io/en/latest/protocol-methods.html?highlight=merkle#blockchain-transaction-get-merkle
I don't think libbitcoin has such functionality has it?
M
Mlmikael10:50
aha, the concurrency model is documented here https://www.boost.org/doc/libs/1_66_0/doc/html/boost_asio/overview/core/threads.html ,
"Thread pools
Multiple threads may call io_context::run() to set up a pool of threads from which completion handlers may be invoked. This approach may also be used with post() as a means to perform arbitrary computational tasks across a thread pool.
Note that all threads that have joined an io_context's pool are considered equivalent, and the io_context may distribute work across them in an arbitrary fashion. "
M
Mlmikael11:08
in libbitcoin-network's p2p, there's seed, manual, inbound, outbound .. attachment of … handlers or something, to drive those aspects of the p2p activity, where are their actual logics?
M
Mlmikael11:11
p2p.hpp exports this method:
/// Subscribe to connection creation events.
virtual void subscribe_connection(connect_handler handler);
can I use that one as a way to track and hijack connections? 🙂
(edited)when a special LB connects to an LB that advertises that, it can send a special message
M
Mlmikael11:12
hijacking in that sense, that my code via subscribe_connection can listen for when it hits such peers,
right.
so my code will track each connection LB initiates by doing p2p.subscribe_connection() right, and then somehow listen for handshake completion,
and, then, peek the service flags , right
correctly understood ?
evoskuil11:14
You need to implement a derived protocol to handle the additional communication semantics, attach the protocol via a derived session.
where is the service flag announcement?
(as in where LB is telling the peer its service flags value)
evoskuil11:16
Services are part of the version object of the peer, which is accessible from the connection object.
The handshake (version protocol) is the version negotiation.
(edited)M
Mlmikael11:17
aha yes. and, where does LB tell the peer its own service flag? "Dear peer, my service flag is 0x … , now you know." .flush();
M
Mlmikael11:18
mhm. a peer normally won't have problems that you tell it that you support all kinds of stuff, though, does it?
at least nonstandard stuff over which there have been no BIP or Bitcoin node version controversies in the past
Core is sloppy.
But if you start using service flags and also don’t conform the the protocol you may start getting dropped based on the service flags.
We do this for three service flags.
Core does as well for at least two.
(edited)if you wanted to advertise to people that you support a special subprotocol of the Bitcoin protocol, would you do it by devising a service flag bit for your subprotocol, or would you do it some other way?
this is a service flag that is central but still it's not primary Bitcoin stuff
M
Mlmikael11:25
publishing a BIP yes, but how would that subprotocol announce itself in a way that does not put it at risk of getting ditched by peers out there, for the reason of announcing
this service announcement and send and receive special messages, is the deepest special behaviour I need of LB.
I think all the other stuff I have an idea of how to do now.
yea maybe, it could work.
it's a bit severe maybe, i'd hope there was some less dramatic way. i should look into service flags.
What messages are you passing?
M
Mlmikael11:29
meaning there's a data overlay, over the normal Bitcoin channel data, i.e. a Bitcoin subprotocol implemented multiplexed with normal channel data
it's going to regard making index lookups on the other side
and gossipping stuff
It’s all data
neillm12:52
@narodnik @evoskuil The ringsignature branch build failure is simply because master's install.sh isn't updated to version5 of secp256k1 (it's still at version4; libbitcoin version3 install.sh is correctly updated to secp256k1 version5). @narodnik if you swap that out in the install.sh file and push it, travis-ci should just rebuild everything properly. Reference: https://github.com/libbitcoin/libbitcoin/blob/master/install.sh#L759
M
Mlmikael18:02
what's the motivation for the name "proxy" (libbitcoin-network's bitcoin/network/proxy.cpp class)?
a "proxy" instance owns a socket. channel - which is the class carrying a P2P connection, right - is a subclass of proxy.
so many classes have a start and a stop method. is there any particular ordering scheme through LB for how these cascade, for instance on LB node shutdown?
what's "enable_shared_from_base" about? (used as a template parent class to some classes like proxy)
in some places there's a sequence mutex_.lock_upgrade(); -> (operation here) -> mutex_.unlock_upgrade_and_lock(); -> (pool an async call here, std::bind all that) --> mutex_.unlock(); , what's the point with this pattern?
M
Mlmikael18:14
i see in the libbitcoin repo the message types to carry a message e.g. "ping", "get_data", etc.
pmienk18:16
@Mlmikael https://github.com/libbitcoin/libbitcoin/blob/master/include/bitcoin/bitcoin/utility/enable_shared_from_base.hpp - a derivable trampoline to std::enable_shared_from_this - looks like a common method to avoid code replication
ah yeah so i see the message types but where is their dispatch logic?
as in where is the logic that says "for a ping message: send a pong!"
"for a get_data: look up so and so, and send it over" etc
pmienk18:17
for the p2p types, I'd assume you'll find that within libbitcoin-node - the project that puts together the networking, database, etc to form a node
libbitcoin-node/src/protocols/protocol_block_out.cpp:bool protocol_block_out::handle_receive_get_data(const code& ec,
riight.
yeah, even better
anyhow proxy is inherited by channel, and you have send message and receive message in there
@pmienk thanks for clarifying 😃
btw do you think there are any particular "gotchas" in the LB codebase?
in particular about contextual stuff like memory management, concurrency
@pmienk ah, "safe_chain", the name "safe" here is just to denote "thread-safe" right, there is NO concept of any snapshotting so that safe_chain would give you a "consistent view", right?
pmienk18:24
not an expert on -network, but proxy looks to be common functionality isolation for message sending/subscription from channel management
by the comments, safe_chain is a threadsafe interface
what would "channel management" be, data transformations - i guess NOT -
but rather like, inactivity timer and such higher level stuff
anything higher than the message passing and its encoding
libbitcoin-network's bitcoin/network/protocols/protocol.hpp , defines the interesting macros SEND1/2/3, SUBSCRIBE2/3, SUBSCROBE_STOP1
protocol_ping_60001.cpp is using that to SEND2 and SUBSCRIBE3
(figuring how messages transmission and channel handling is done :)) )
mhm
@pmienk, what logics to drop an established channel is there? so i guess, socket write error would lead to termination somewhere, and .. inactivity, or not responding to ping?
do you know where the node's logic for randomly picking a host from its hosts list, to connect to it, is located?
pmienk18:33
i don't know, but if you search network and node for references to hosts.hpp from -network I assume you'll find it
M
Mlmikael18:34
i don't know, but if you search network and node for references to hosts.hpp from -network I assume you'll find it
ok
ah, all matches for hosts.hpp are:
libbitcoin-network/include/bitcoin/network/p2p.hpp:#include <bitcoin/network/hosts.hpp>
libbitcoin-network/include/bitcoin/network.hpp:#include <bitcoin/network/hosts.hpp>
libbitcoin-network/Makefile.am: include/bitcoin/network/hosts.hpp \
libbitcoin-network/src/hosts.cpp:#include <bitcoin/network/hosts.hpp>
libbitcoin-network/src/p2p.cpp:#include <bitcoin/network/hosts.hpp>
@pmienk, riight, libbitcoin-network's hosts.cpp's fetch method does: // Randomly select an address from the buffer.
const auto random = pseudo_random(0, buffer_.size() - 1);
so right, that's the one.
interesting that all peers have equal "attractiveness" to an LB node
neillm18:40
You have a lot of questions (nothing wrong with that). Maybe take some time to read through some parts to help narrow down the scope a bit?
M
Mlmikael18:41
@neillm, focus today was on understanding how messages go back and forth on channels and what logic they're hitting, that was the important thing and maybe i understand it
node is more context now
trying to briefly understand some things now, and will go into detail study next weeks. 🙂
neillm18:42
Ok, good. None of this stuff is trivial, and a lot of it requires deeper understanding of what it's built on. Trying to understand it quickly can be difficult is all.
sure, LB's node's mechanics are very involved indeed.
@neillm where do you choose which host to connect to?
You were correct about the fetch. I was thinking of the fetch that outputs the list (in version3)
M
Mlmikael18:58
aha, it uses libbitcoin-network's sessions/session.cpp fetch_address , which leads to p2p's fetch_address,
which leads to libbitcoin-network's network/hosts.cpp's fetch = same 👍
Thursday, May 24, 2018
haarts07:09
@evoskuil In order to create an SPV client I need a Merkle branch for a specific transaction. ElectrumX implements an API endpoint for that: https://electrumx.readthedocs.io/en/latest/protocol-methods.html?highlight=merkle#blockchain-transaction-get-merkle
I don't think libbitcoin has such functionality has it?
haarts08:45
The header call (blockchain.fetch_block_header) contains the merkle root. The transaction hashes call (blockchain.fetch_block_transaction_hashes) returns a long list of, well, transaction hashes. I'm not sure where to go from there.
I don't see how to reconstruct the merkle branch.
Or are you saying I don't need to? Eg. just find the transaction hash in the long list?
M
Mlmikael08:52
@haarts are you implementing an SPV wallet on LibBitcoin? that would be awesome.
of course SPV is an incomplete security model, however when you want to do lighter Bitcoin blockchain related operations for instance from a smartphone or embedded device, syncing the blockchain headers to follow the PoW chain, is extremely useful (whether you get that chain from the P2P network or some other source), and using LibBitcoin as a universal Bitcoin library that does that kind of stuff to, is relevant.
haarts08:53
@Mlmikael That is what I'm doing yeah. Sort of in any case. I'm taking Electrum and have it talk to libbitcoin instead of ElectrumX.
I'm aware of the shortcomings of SPV, but it has it's place.
M
Mlmikael08:55
@haarts cool! will this become a "libbitcoin-spv" repo, or do you roll it some other way?
@neillm / @evoskuil a question, in libbitcoin-network repo, sessions directory, i see that
* the "session_seed" class, is for the P2P peer connections that LB initiates to the seed nodes, which are specified in the config, right -
* the "session_manual" is the P2P peers that the user has hardcoded in the LB config, right - and
* the "session_outbound" is the normal P2P walking that LB full node does on its own initiative, - and
* the "session_inbound" is connections arising from other Bitcoin nodes connecting to the LB node's inbound connections port, right.
i guess "manual" and "outbound" and "seed", while all of them are outbound connections, the point with having different logics for them is that "manual" will retry forever and with highest prio, and then i guess there is some other logics difference between "outbound" and "seed", so i kind of appreciate the difference in function and motivation. so this far all is clear to me.
now what is the "session_batch" about - what is its function and motivation?
(oh, session_outbound is a subclass of session_batch, which is a subclass of session . session_manual , _seed and _inbound are direct subclasses of session.)
evoskuil09:00
All you need is the header and ordered set of hashes, that’s the Merkle block. You should retain that for each block for later verifying tx block commitments.
You can and should of course verify that the merkle block is valid.
haarts09:11
@evoskuil Is set of transaction hashes ordered from the API call? It's a list, and I'm looking for a tree afaict.
I'm still not fully versed in this material...
@Mlmikael First I've created a Python library to talk to libbitcoin: https://github.com/Feathercoin-Foundation/pylibbitcoin (now it is also evident who's spearheading this ;)).
GitHub
Contribute to pylibbitcoin development by creating an account on GitHub.
This library should go to the libbitcoin repo IMO once it survived first contact with a real world use case.
Second I've reworked the network module of Electrum. This PR is the last to be merged in that quest: https://github.com/spesmilo/electrum/pull/4376
GitHub
This is a continuation of #4371. The goal here was to make the code base easier to understand and more decoupled. All the ElectrumX protocol mentions are now in the network module. Except for two i...
I'm now in the position to determine what calls are made by Electrum, what calls libbitcoin supports and what the possible mapping between the two is: https://gist.github.com/haarts/b2ff66af364ac50f02cb77f7536ca70d
I'm currently working on closing that gap.
evoskuil09:18
The tree is derivable from the ordered list as otherwise the block itself would not be verifiable.
See an merkle hash commitment verification code.
That is great to hear.
My little Python library is set to grow quite a bit.
evoskuil10:16
We will be working on providing Electrum-style payment indexing/query/notification. This will improve libbitcoin and resolve the main incompatibility.
(edited)It does get a merkle branch from ElectrumX https://electrumx.readthedocs.io/en/latest/protocol-methods.html#blockchain-transaction-get-merkle
Isn't that just the subscribe address API call?
Does it verify the merkle tree?
it does
Friday, May 25, 2018
M
Mlmikael05:16
@evoskuil @neillm 1. to traverse each transaction in new/reorged blocks, first you full_node.subscribe_blockchain() right. ( blockchain::block_chain.subscribe_blockchain() is a NOOP . safe_chain is a block_chain.)
the reorganize_handler gets as incoming a block_const_ptr_list_const_ptr right, so that's a list of chain::block .
this one contains a vector of transaction. this means those transaction objects are fully loaded/populated alredy here, so traversal is easy.
did I get it right this far?
next, what is the memory management model here, "const_ptr" here just means pointer, not refcounting by shared pointers.
2. also the transactions strucutre.. is storing transactions by reference ie pointer and not by value, no?
3. if i want to keep access to the blocks and transactions after the handler returns, should i clone all those blocks, or what's the memory management model here?
I would be happy to just borrow LB's objects, however for this I need to understand the memory model as in how increase/decrease refcount or otherwise tell LB that i'm still using it / i'm done using it.
Saturday, May 26, 2018
M
Mlmikael07:59
block_const_ptr_list_const_ptr's definition is exactly what? (it's used by block_chain.hpp)
1. Yes
2. The list of txs is a list of instances, not references.
3. Keep the block pointer. Smart pointers, refcounted, destruct on refcount 0.
evoskuil08:50
Message objects are copied from the store so that holding the pointer has only a memory impact, it does not block the store.
Delayed-read result objects are passed from database to blockchain, but not exposed by it.
M
Mlmikael11:10
question, in chain::block and chain::transaction there's some mutex locking, what's the point with that, you're feeding blocks and transactions with data from more threads at the same time, so you use mutexes to not corrupt data, and to serialize processing finegrainedly where needed?
also, what's the point with using boost shared & upgradeable mutexes (lock_upgrade , unlock_upgrade, unlock_upgrade_and_lock , unlock)?
Sunday, May 27, 2018
M
Mlmikael10:33
interesting about the libbitcoin repo's include/bitcoin/utility/subscriber.hpp -
when you stop it, the only thing that stops, is the ability to add more subscriptions.
for any subscriptions that exist from before however, if there are .invoke():s or .relay():s, those will be cascaded indeed.
M
Mlmikael10:44
ah also interesting that "subscriber" is a one-shot dispatch facility where you need to re-subscribe explicitly in order to get back in the loop, and "resubscriber" will resubscribe you
M
Mlmikael11:40
wait just FYR, some feedback from some C++ coders on https://github.com/libbitcoin/libbitcoin/blob/version3/include/bitcoin/bitcoin/utility/atomic.hpp , they think it could be written in a better way, @evoskuil @narodnik
@Mlmikael uploaded a file: Untitled
Untitled
wait just FYR, some feedback from some C++ coders on https://github.com/libbitcoin/libbitcoin/blob/version3/include/bitcoin/bitcoin/utility/atomic.hpp , they think it could be written in a better way 1) "The use of std forward does the right thing but is not idiomatic. It should use std::move" 2) <SlashLife> atomic(Type&& instance): instance_(std::forward<Type>(instance)) // <-- Better question: What's the point of the std::forward? <SlashLife> imo: This (code) is BS. <SlashLife> The use of std::decay in it is, and a bunch of the other templating stuff as well. <Oxyd> That forward almost makes sense. Shame that Type isn't a forwarding reference. <Oxyd> It's like someone read a C++ tutorial and almost understood it. <SlashLife> Oxyd: It doesn't make sense the way it is used there. <SlashLife> Type is determined by the struct, not by the ctor. <SlashLife> Instead they should restrict the Type they take to non-refs (and potentially to non-cv as well) <SlashLife> Jmabsd: That last line was for you again. <SlashLife> Where they go wrong: They didn't think about what kind of types they'd want to allow, ending up with contradicting usage concepts, therefore incorrecly applying template programming concepts. <Oxyd> Well, one major thing is that it always uses a mutex. The point of std::atomic is to get a lock-free access (at least sometimes). <SlashLife> That std::forward should instead have been an std::move; almost all uses of Type within that struct should be replaced with decayed_type <SlashLife> Unless of course Type is a reference. <SlashLife> But then you have an atomic<int&>, which doesn't even store a reference to int, but is just the same as an int. <SlashLife> Except that it won't have move operations, but instead copy operations for both const and non-const refs. <SlashLife> So being able to pass T& to it makes you believe it could store a reference. But it doesn't. Instead it disables move semantics if T has them. <SlashLife> So if you pass T&, it is actively misleading in what it does. <SlashLife> Now you could say that maybe this is not intended to pass T&, and they just didn't add code to catch that - sure, that could be. But they did add a bunch of other code (including those std::forward) which specifically suggest graceful handling of reference types. <SlashLife> If your T is not a T&, then std::forward<T> means std::move
M
Mlmikael11:52
"It's a little bit weird that they don't seem to use the standard C++11 mutex and lock types"
M
Mlmikael13:01
@evoskuil what's the purpose of this atomic class? it looks to me like what it does is give an object slot, where the store and retrieval operations are threadsafe, hence protecting the object itself from corruption due to concurrent stores of two objects or concurrent store and load
what's the purpose of the libbitcoin repo's utility/synchronizer.hpp template class?
M
Mlmikael16:41
@evoskuil wait, in connector.cpp https://github.com/libbitcoin/libbitcoin-network/blob/version3/src/connector.cpp#L165 , isn't there an existing risk that both the timeout callback would trig, and that the connect success callback would?
in that case the connect_handler would be invoked twice, in no particular order, once with error::channel_timeout or another error code and null as channel argument, and once with error::success as error code and the newly created channel as channel argument.
https://github.com/libbitcoin/libbitcoin-network/blob/version3/src/p2p.cpp#L176-L177 what's the point here?
"There is no way to guarantee subscription before handler execution. So currently subscription for seed node connections is not supported."
N
narodnik21:26
Hello everyone, here is my rangeproof code using the new ringsigs feature in libbitcoin: https://gist.github.com/narodnik/9a53bb1560bfb677ab8e94e93acf5885
👍 2
evoskuil23:40
Critical sections in the bc::chain type system support caching of costly-to-compute computed properties (such as .hash()). This optimizes for CPU while not sacrificing code clarity. It is protected to allow safe concurrent reading of the object, since the properties are computed on read, and allows safe invalidation when underlying properties change.
Monday, May 28, 2018
A
Alex01:27
Hello everyone! The team at SBI BITS has mentioned to @evoskuil that we would like libbitcoin to support threshold signatures. If you are curious what this is, a good starting place is https://www.cs.princeton.edu/~stevenag/threshold_sigs.pdf. There are, however, several implementations of threshold signatures with different characteristics (threshold-optimal, computational speed) so from a high-level view, we feel flexible support from libbitcoin is best. We have considered "external" and "internal" support.
From bx, the external support adds a function
bx input-prepare [-ah] [--config VALUE] [--index VALUE] [--sign_type VALUE] CONTRACT [TRANSACTION]
whose arguments match that of bx input-sign. The argument EC_PRIVATE_KEY is omitted, however, as threshold signatures cannot provide the private key. The outputs of this call are the hex-encoded sighash as calculated at https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/script.cpp:867 and the hex-encoded sighash_type from line 876. The user then generates the EC signature using his implementation of threshold signatures and the sighash, appends the sighash_type, and hex-encodes this result to be used in a subsequent call to bx input-set. Pros: maximum flexibility in threshold signature algorithm implementation; not specific to threshold signatures -- allows future signature algorithms to be used as well. Cons: signature implementation not necessarily vetted by libbitcoin.
From bx, the internal support adds a function
bx input-sign-partial [-ah] [--config VALUE] [--index VALUE] [--sign_type VALUE] EC_PRIVATE_KEY_SHARE KEY_SHARE_TYPE INTERMEDIATES_PRIV INTERMEDIATES_PUB CONTRACT [TRANSACTION]
whose arguments mostly match that of bx input-sign. The argument EC_PRIVATE_KEY becomes EC_PRIVATE_KEY_SHARE as threshold signatures cannot provide the private key. To deal with the different implementations of threshold signatures, KEY_SHARE_TYPE is taken to select the cryptographic algorithm used. INTERMEDIATES_PRIV and INTERMEDIATES_PUB are hex-encoded serialized data structures that are specific to the cryptographic algorithm. On the first call to this function, INTERMEDIATES_PRIV and INTERMEDIATES_PUB are null so "" would have to be passed on the command line if they are positional arguments. Alternatively, they could be optional arguments. As existing threshold signature algorithms cannot be completed in one round (each user must provide his key share multiple times), the outputs of this call are the INTERMEDIATES_PRIV to be retained by the caller for his next call to this function and the INTERMEDIATES_PUB to be given to the next key share holder for his call to this function. Once enough key shares have been provided, then the output INTERMEDIATES_PRIV is null and the output INTERMEDIATES_PUB is the completed endorsement. The completed endorsement is used in a subsequent call to bx input-set. Pros: signature implementations vetted by libbitcoin. Cons: limited to the signature implementations provided by libbitcoin.
neillm01:33
@Alex Great to hear! I'm still catching up on threshold signatures, but are you saying you'd prefer the first option for flexibility over the second? Is it realistic that users would want to choose different implementations externally over having something internal?
A
Alex02:16
Hi @neillm! Yes, we are leaning towards the "external" option as SBI BITS has chosen an older algorithm that would likely not be the first choice for a libbitcoin-provided algorithm. The first option also does not preclude libbitcoin from providing vetted threshold signature implementations, perhaps via a function
bx sign-threshold [-h] EC_PRIVATE_KEY_SHARE KEY_SHARE_TYPE INTERMEDIATES_PRIV INTERMEDIATES_PUB [SIGHASH]
which after several calls produces the EC signature to be concatenated with the sighash_type.
Also, from my memory, no current threshold signature algorithm completes signing in less than 4 rounds so it is unlikely that anyone wants to use the command line for threshold signing. Users will likely prefer to wrap the library call instead, creating a communication network that fits their preferences and passing around the public intermediates until the signature is complete. So even if the second "internal" option is chosen, users are likely to externalize it anyway for ease-of-use and performance.
evoskuil05:09
I would consider the ability of libbitcoin to complete the process as essential for the library to adopt the functionally. The limitation on algorithms is not significant given the open source nature of the library.
On the other hand it’s not obvious to me that the scenarios are mutually exclusive.
KEY_SHARE_TYPE should be an option, since it can be defaulted.
@Mlmikael The batch session is an intermediate base for address pool based outbound session types. It reduces connection time by attempting concurrent connection to multiple peers.
@Mlmikael The race condition you describe is managed by the synchronizer object.
@Mlmikael The bc::atomic class serves the same purpose as std::atomic but is not limited to integral types. It is lock-free, the commenter misinterprets the term. Lock free does not mean mutex free. The purpose is the same as std::atomic, to provide atomicity. Implantation in both is opaque.
@Mlmikael Boost lock types are used for access to the necessary upgrade semantics.
@Mlmikael Subscribers contain a queue of handlers. These are closures over arbitrary state. An arbitrary closer must eventually be executed under all scenarios. Stop of the subscriber prevents new subscription so that a subsequent event can drain the handler queue with encountering a subscription race.
@narodnik Cool
M
Mlmikael11:23
The bc::atomic class serves the same purpose as std::atomic but is not limited to integral types. It is lock-free, the commenter misinterprets the term. Lock free does not mean mutex free. The purpose is the same as std::atomic, to provide atomicity. Implantation in both is opaque.
@evoskuil, right, i didn't understand the sense of those Boost mutex locking semantics also :))
it looks to me like that there are multiple lock modes in those mutexes, one that's called "upgrade", one that's called "shared", and one that's called "exclusive", and frequently in a code block you go something like -> upgrade mode -> exclusive mode -> unlock.
what's the point here?
M
Mlmikael12:15
@evoskuil what locking states and mechanisms is involved here over normal binary mutex lock mutex_lock(mutex); / mutex_unlock(mutex); ?
haarts12:55
I'm struggling with the History Correlation. For that I need to create a checksum of the hash and the index of a transaction.
I'm taking the tests as a guide: https://github.com/libbitcoin/libbitcoin/blob/master/test/chain/point.cpp#L279-L305
My code now produces this:
inital hash: ffffffffffffffffffffffff01234567aaaaaaaaffffffffffffffffffffffff last 20 bytes: 01234567aaaaaaaaffffffffffffffffffffffff flipped last 20: 0xffffffffffffffffffffffffaaaaaaaa67452301 masked last 20: 0xaaaaaaaa67450000 index: 0x89abcdef masked index: 0x4def(edited)
(debug output from my method)
My Python test fails: AssertionError: '0xaaaaaaaa67454def' != '0x01234567aaaacdef'
(actual != expected)
obviously the flipping is not working out, and I'm not sure that is what the C++ code does. The function
from_little_endian_unsafe is hard for me to understand.Can someone point me in the right direction?
Well, stating the problem clearly is half the solution.
Minor nag; if the tests would have used 'ffffffffffffffffffffffff01234567aaaaaaaa000000000000000000000000' in stead of 'ffffffffffffffffffffffff01234567aaaaaaaaffffffffffffffffffffffff' I would have found it earlier. 😉
A
Alex23:46
@evoskuil I understand that it doesn't make sense to support threshold signatures without providing at least one implementation. I think you are right that "internal" support can be a thin wrapper over "external" support plus one or more threshold signature implementations.
I made one large omission in the specification for
bx input-sign-partial and bx sign-threshold. When data is put into INTERMEDIATES_PUB to be passed to other key share holders, that data must be signed to prevent tampering. This means these two functions also need to know a private key for the caller and a public key for each of the other key share holders. I will think about how to get these parameters in and make a suggestion later.Tuesday, May 29, 2018
A
Alex07:50
@evoskuil visited us here in Tokyo and made a good case for making the signing of data in INTERMEDIATES_PUBLIC optional since tampering would result in an invalid signature anyway. As long as shared key holders do not blindly continue to reattempt signing, the attack fails. Signing of data in INTERMEDIATES_PUBLIC would occur in a separate
bx function. By corollary, verification of signed data in INTERMEDIATES_PUBLIC would occur in another bx function also. We are going to pursue this path, seeking to implement bx input-prepare and bx sign-threshold first. If there is no demand, we may simply leave the signing of data in INTERMEDIATES_PUBLIC as a feature request.Revive11:06
when i was building from @narodnik’s ringsignature branch, src/math/.deps/src_libbitcoin_la-elliptic_curve.Tpo was owned by root
not sure why
Fil11:32
Hi guys! I've noticed that currently the block_organizer does not subscribe: https://github.com/libbitcoin/libbitcoin-blockchain/blob/75b604613f87f71777c28fe4ac9d28f393199bff/src/interface/block_chain.cpp#L1314
Does this mean that libbitcoin::database::block_database is never populated?
Fil11:44
@evoskuil I'm currently porting Feathercoin to libbitcoin. To spare myself from a big rebasing pain I'm working on master. Is the DB stuff still WIP?
I just need to know if it's my patches that prevent
libbitcoin::database::block_database from being populated or if that's current behaviour on master.neillm11:51
No problem, as Eric mentioned version3 is probably where you want to start, but I understand the porting pain as well. Not sure what the best answer for you is
Fil11:54
Has the new DB design been documented somewhere? It would help to understand in what state it currently is. Atm, I can just guess 🙂
neillm11:56
In master, no, not that i'm aware of. There is also a set of changes still pending on it.
M
Mlmikael15:37
@neillm a quick question, how do you mitigate congestion out outbound messaging in a P2P channel in LB?
so i see here https://github.com/libbitcoin/libbitcoin-network/blob/version3/include/bitcoin/network/proxy.hpp#L62
"dispatch_.lock" here, that must be a way to serialize individual messages' submissions...
what does the dispatch lock do here actually, will it process that code (that is "&proxy::do_send") at first after unlock?
and also, say i'm the full synced node logics and i just got a valid block and now i like to broadcast it to all the peers that are connected to me, and there's a tooon of them of course, and some are slow or just tanked or random malfunction, right.
when the LB fully synced node wants to relay transactions and blocks, what are the logics it applies to discern if it should?
also say a P2P channel was jammed right now - the TCP connection was slow and a previously committed 1MB block was taking say 25 more seconds to go through - at the point it has gone through, is there any pushback feedback notification where the fully synced node would learn to know that a particular channel C was busy sending stuff when I wanted to forward to it before, but, now 25 secs later you can send?
where is that logics :))
so this means if you make many such send:s, then they pile up
so the only effect of the dispatch here is to serialize them (serialize in the sense, make a sequence of)
yes?
yeah sure that makes all sense.
and to last question, is there any logic in the fully synced node for block/tx relay where it checks if a channel is congested, as to decide whether to send or not?
neillm15:50
I'm not sure about that part at the moment. I recall the node dynamically disables relaying if it's too far behind on block sync, but not sure about individual channel relay considerations
overall I like the LB codebase.
i'm really happy engaged to delve into it vs. not-it lol.
👍 1
@neillm yeah but like, there should be a cap somewhere so not too much work is accumulated relating to outbound messages for a p2p channel
they could be slow but work - i guess in that case you have some timeout somewhere that kicks in
e.g. a block send must take no more than 10 minutes to push through the TCP networking stack.
it could be your network connection that's slow too, in that case if you have 1000 inbound connections, sending a block to all of them at the same time, may indeed take 10mins!
@neillm does LB have any limit on how many peers it tries to send a block to at the same time? ( i think i saw already that there is no limit on bandwidth incoming and outgoing. that could possibly be interesting to add some day. anyhow not prioritary.)
anyhow the question about handling slow/congetsed outbound messaging was what i was thinking about most.
neillm16:02
channels do have expiration and inactivity timers, but not specifically a slow send check
M
Mlmikael16:02
wait, when you send a block, the peer is supposed to acknowledge somehow when it arrived to him?
hm. i guess the outbound messages connection generally not should be a big deal. individual pending outbound messages take maybe 1-2KB each of RAM
and in the special cases where they would accumulate, you would not have so many peers anyhow, say 50,000 congested messages x 100 peers, = 5,000,000 messages = 2GB of RAM
LB instances running on various Single Board Computers (e.g. some ARM with 1,2,4GB) would die by that. :))
anyhow very rare. hm.
it would be great if LB had definable upper bounds of memory consumption
or less?
way less
a slim operating system can take something like 100MB
anyhow
running a fully synced node on, say, 100MB of smartphone RAM, is not relevant today..
the lowest-end hardware would be SPV use only (which LB can do someday I guess..)
and those will neither process transactions or blocks on the P2P network.
neillm16:08
I was thinking specifically of a raspberry pi on an SD card, since you mentioned ARM. That would be tremendously slow, if possible at all.
M
Mlmikael16:08
i think an ARM64 with 1GB of RAM and another 1GB of swap most likely runs a LB fully synced node excellently
ah sure, SD memory cards would be too slow, agreed.
the RK3399 ARM, www.pine64.org will start selling them for 70 USD soon (4GB RAM), have PCIe, so you can connect an M.2 NVMe PCIe SSD to it
other cheaper ARM:s, such as www.pine64.org or Raspberry Pi, right, would be SSD via USB
i've been told syncing libbitcoin on a magnet disk isn't that bad by someone who tried :)) he did it in not many days
neillm16:10
Interesting, that could do it. Was not aware of that as an option. As for SSD over USB, I don't think the raspberry pi usb is fast enough, or even full USB3, but I'm not sure. I still think it would be quite slow, but haven't tried.
M
Mlmikael16:11
all the mmap:ed files in LibBitcoin, if you hit the system RAM as a limit, then the operating system will just flush the cache to the disk and drop it, right.
so you get slower read performance, but LB still works perfectly.
neillm16:11
I've sync'd libbitcoin on a spinning disk too. It was brutal. What's not bad is if you copy an existing store over to spinning disk, running it from there on isn't bad at all
@evoskuil @neillm if there's any particular mechanism in LB that caps tx/block relay outbound messaging in case of channel congestion, let me know, would love to see how that aspect worked.
i wild-guess that right now each channel has some performance timer, like, "if sending block to peer takes more than 60 seconds then disconnect". which means only 60 seconds of P2P network gossip has time to build up before that channel, with all its pending outbound messages, is discarded
that in itself may work perfectly as long as total connection counts in LB are low.
if some LB node would have lots of nodes, and you knew its resources were limited, you could just connect to it and gossip it a very large number of transactions, quickly, to crash it. :))
Fil16:47
Who is working on this set of changes?
Neill Miller
In master, no, not that i'm aware of. There is also a set of changes still pending on it.
Fil16:56
I see. I'll take a look into https://github.com/evoskuil/libbitcoin-database/tree/v4_wip then 👍
GitHub
libbitcoin-database - Bitcoin High Performance Blockchain Database (version 3 only)
Wednesday, May 30, 2018
Revive03:55
hello, i'm learning to use libbitcoin. i had compiled with it successfully earlier and afaik i changed nothing, but now i keep getting this error. https://pastebin.com/fBiZimU1
Pastebin
maybe i should remove it all and start fresh
pmienk04:03
probably easiest to determine if there is something missing from the script during the autotools reinstall
Revive04:05
i mentioned this earlier but this is the libbtc version i've installed
Taylor
when i was building from @narodnik’s ringsignature branch, src/math/.deps/src_libbitcoin_la-elliptic_curve.Tpo was owned by root
pmienk04:11
I don't see a change to Makefile.am that would alter ownership. If you're installing to /usr/local you'd need to have sudo'd the installer which would do that.
evoskuil07:38
@Mlmikael There is no timer on an individual send, but on seeding, connection time, channel lifetime, inactivity and heartbeat (see config).
@Mlmikael The a locked dispatch is used to ensure ASIO does not interleave sends when a send exceeds the max chunk size (1024 bytes).
@Alex given that there are already bx message sign/verify commands we would expect those to provide the optional features in the bx workflow.
evoskuil11:37
@Mlmikael The important problem scenarios are (1) creation of a large backlog of work, essentially allowing a peer to create a memory exhaustion failure, and (2) thread starvation, where slow peers can monopolize the set of available logical threads.
Message backlogging is mitigated by not subscribing to read a new message until the previous is handled. Some messages are currently “handled” by buffering them. Others (tx, block, ping...) are not. This is configured in the message subscriber. So small messages could create an overflow. We will likely move all message handling to unbuffered in v4.
Peers that are slow to accept a message will delay ping processing, resulting in a dropped peer. The limit is configurable. However message send is asynchronous, so no network thread is held up by a slow peer.
(edited)evoskuil11:46
Broadcast messages are in effect limited by the control on inbound messages. The number of peers only multiplies the cost by a constant (number of peers), so is immaterial.
There is no facility for dropping inbound or outbound messages except at channel shutdown. Poor performing peers don’t slow down libbitcoin beyond the latency increase (which is strongly mitigated in v4 initial block download).
So there are no quotas, and statistically slower peers are dropped when we are catching up (to minimize latency).
@Mlmikael the tx protocol drops inbound txs when the chain is behind (so no notifications) and the block sync does not notify peers when the chain is behind. The definition of “behind” is configurable.
Thursday, May 31, 2018
Revive03:01
i still am held up with some error while trying to use libbitcoin. ive tried using a fresh prefix directory and tried production tag versions 3.5. it seemed to me to start occuring without any changes as far as i can remember
its strange because when i run gcc now it seems to start a process in the background by default
i was thinking that a system update moved gcc from v7 to v8 and broke something maybe?
i'm on manjaro
thanks
cant believe i didnt see the &
😬 1
haarts11:46
The list of transaction hashes from the blockchain.fetch_block_transaction_hashes API call are ordered. Ordered how? The transactions are ordered in a tree, so is it depth first ordered? Or breadth first? Something else?