Bitcoin full node and query server
October 2018
Archive of the libbitcoin Slack #general channel.
Monday, October 1, 2018
neillm11:40
No REST API in the works. Websockets and JSON-RPC interfaces still pending, but details of integration are being worked out
Did you find a solution to replace the library which had licencing issues (forgot the name) with something else ?
neillm11:42
Yes, a small http stack from scratch optionally using mbedtls (for SSL) is the workaround
Testing help needed (as always) 🙂
neillm11:44
GitHub
Bitcoin Full Node and Query Server. Contribute to thecodefactory/libbitcoin-server development by creating an account on GitHub.
Note that the build may have some issues that can be worked around pretty easily, but don't work out of the box (we're looking at the currently)
Should be fine without SSL support though
You wrote all the web part from scratch?
I spent several months on the initial integration working with mongoose, and tracking down bugs in it that manifested in weird ways and after wrestling with it a while and learning about the license issue on top of it all, it seemed straight-forward to just replace it
And since we now had the HTTP stack, adding JSON-RPC support was near trivial
I also spent a fair amount of time researching alternatives. There were not that many that were still active projects, and the one that looked most likely would have still needed SSL support (via mbedtls since OpenSSL isn't an option we ever want integrated again)
Feel free to review. The entire HTTP related code is about 2.5K lines
The performance reported via the wrk HTTP benchmark is also > 5x what we were getting from mongoose ...
It’s still planned to have it as part of v4?
(edited)I like (and have used in the past) seasocks: https://github.com/mattgodbolt/seasocks
GitHub
Simple, small, C++ embeddable webserver with WebSockets support - mattgodbolt/seasocks
👍 1
The biggest issues there are 1) requires C++14, which we don't, and 2) No SSL support. I considered adding it there, but ended up not.
Saturday, October 6, 2018
N
narodnik08:22
@evoskuil C++20 will add modules and coroutines:
https://medium.com/@wrongway4you/brief-article-on-c-modules-f58287a6c64
https://lewissbaker.github.io/2017/11/17/understanding-operator-co-await
Modules means no more forward declarations and include guards, vastly speeding up compilation and simplifying code.
Coroutines means fast simple flat async code.
Medium
C++ Modules is, certainly, one of the most anticipated C++ programming language features. All modern C++ developers (if you aren’t talking about people who writes all of their code in C and Assembly…
Asymmetric Transfer
Understanding how the co_await operator works can help to demystify the behaviour of coroutines and how they are suspended and resumed. In this post I will be explaining the mechanics of the co_await operator and introduce the related ‘Awaitable’ and ‘Awaiter’ type concepts.
@neillm C++14 is not that bad for server deployed software, it's more an issue for cross compilation across many desktop and mobile platforms.
👍 2
neillm12:06
@narodnik I'm a fan of C++14 (generic lambdas!) and beyond (IIRC C++17 has most of the filesystem stuff we're using in boost) -- was just pointing out that adding a requirement that required it, could force a shift of our entire code base, perhaps unnecessarily, or prematurely. Not that it was somehow bad 😉
Sunday, October 7, 2018
evoskuil07:57
I’m particularly dismayed by comments recently made to me by @risknerd. I’m publishing those comments in full here now:
“the bip141 issue is larger than I thought. if it is not larger than you knew already then we should probably have a talk before I take the next steps on this. I'd like to know if you were discouraging me from looking into this because you knew what I would find. be honest.”
“one might also conclude that you made a showing of particular dismay at my development effort around the config file mechanism because you knew what I would discover upon introducing my change to that existing mechanism. I will elaborate in detail if you don't already know what I'm talking about. if you knew these things in advance and your knowledge motivated your hostility toward me then it leads next to some very challenging questions, for which you may need a lawyer. I'm not trying to cause any problems for you, truly. if my efforts to help debug and finish v4 have inadvertently led me to uncover some problems that you have already created for yourself, legal problems in particular, then obviously I don't wish to add to those problems in any way. I have some ideas for how I would recommend that you handle this situation, depending upon what the facts are and how you choose to reply.”
I prefer all discussion be public and my specific critiques of his approach have been made publicly, on GitHub and Slack.
2 replies · lclc, evoskuil · last Oct 10, 2018
lclc21:44
I think I’m lacking context.
Why would you need a lawyer for anything and what is the problem ?
neillm12:36
I like the conspiracy aspect. This seems like it could be interpreted as: "Your unfinished code isn't perfect yet and while you told me that 10+ times, it wasn't what I wanted to hear ... so if you knew that there are bugs and didn't want me to find them, legal action may follow for what you said"
N
narodnik15:20
@neillm well I meant if you're making a server with websocks support, then if it's separate (not libbitcoin), the dependency will be less of a problem than if you were developing say a user app
(which then you'd want to deploy to mobile) - but yep, got your point
Monday, October 8, 2018
vdeurzen13:04
btw, modules for C++ aren't entirely certain yet afaik, next meeting (San Diego) they might be added, but there are still open questions I believe
vdeurzen13:38
was just wondering, we're currently testing our tools using regtest, but would be nice to start using testnet as it's an actual network, but would also like to have btc on demand 😜
and a faucet doesn't give that much, though that is probably what we'll end up using
neillm13:39
Makes sense. I may have some to spare (probably not as much as you'd like), how much you looking for?
vdeurzen13:40
we're thinking about automated testing, which would be very difficult if the rate at which we could get testnet BTC is unpredictable, so not really a specific amount, just something we can control somehow or build up a buffer
neillm13:41
I guess how large that buffer needs to be depends on what kind of testing you're doing and if the coins can be recycled during the tests, minus fees, of course
still, at some point with fees that will end-up being zero in our buffer
not sure about the exact amount of tests and how often they will run etc
vdeurzen13:43
That might be a great start indeed. Haven't actually set-up our end yet so no place to receive them yet. But thank you very much for your offer!
Tuesday, October 9, 2018
I'm currently looking into something and I'm not sure if it might be a bug: when fetching a transaction's
fees() I always get back 0, it seems that total_input_value() is always 0, my guess is that that is due to the previous output being missing for some reason. I'm testing this with the 100'000th block hex data (as found in libbitcoin's tests)i.e.:
auto const raw = bc::to_chunk(bc::base16_literal(
"010000007f110631052deeee06f0754a3629ad7663e56359fd5f3aa7b3e30a00"
"000000005f55996827d9712147a8eb6d7bae44175fe0bcfa967e424a25bfe9f4"
"dc118244d67fb74c9d8e2f1bea5ee82a03010000000100000000000000000000"
"00000000000000000000000000000000000000000000ffffffff07049d8e2f1b"
"0114ffffffff0100f2052a0100000043410437b36a7221bc977dce712728a954"
"e3b5d88643ed5aef46660ddcfeeec132724cd950c1fdd008ad4a2dfd354d6af0"
"ff155fc17c1ee9ef802062feb07ef1d065f0ac000000000100000001260fd102"
"fab456d6b169f6af4595965c03c2296ecf25bfd8790e7aa29b404eff01000000"
"8c493046022100c56ad717e07229eb93ecef2a32a42ad041832ffe66bd2e1485"
"dc6758073e40af022100e4ba0559a4cebbc7ccb5d14d1312634664bac46f36dd"
"d35761edaae20cefb16f01410417e418ba79380f462a60d8dd12dcef8ebfd7ab"
"1741c5c907525a69a8743465f063c1d9182eea27746aeb9f1f52583040b1bc34"
"1b31ca0388139f2f323fd59f8effffffff0200ffb2081d0000001976a914fc7b"
"44566256621affb1541cc9d59f08336d276b88ac80f0fa02000000001976a914"
"617f0609c9fabb545105f7898f36b84ec583350d88ac00000000010000000122"
"cd6da26eef232381b1a670aa08f4513e9f91a9fd129d912081a3dd138cb01301"
"0000008c4930460221009339c11b83f234b6c03ebbc4729c2633cbc8cbd0d157"
"74594bfedc45c4f99e2f022100ae0135094a7d651801539df110a028d65459d2"
"4bc752d7512bc8a9f78b4ab368014104a2e06c38dc72c4414564f190478e3b0d"
"01260f09b8520b196c2f6ec3d06239861e49507f09b7568189efe8d327c3384a"
"4e488f8c534484835f8020b3669e5aebffffffff0200ac23fc060000001976a9"
"14b9a2c9700ff9519516b21af338d28d53ddf5349388ac00743ba40b00000019"
"76a914eb675c349c474bec8dea2d79d12cff6f330ab48788ac00000000"));
bc::chain::block block100k;
block100k.from_data(raw);
i.e.
block100k.transactions().fees() is 0, which I would not expect?^ the last part is for each tx, not the list
vdeurzen11:35
ok, cool, but is that the correct behaviour? Or should I have the entire chain before I can get accurate fees?
neillm11:37
It seems correct to me. You can't provide a value to something like that without knowing the requirements (ins/outs)
vdeurzen11:37
Ok, so a test with only that hex blob will never be able to get the fees unless I include all previous blocks + transactions to ensure a proper chain of utxo?
vdeurzen12:19
master (and secretly also bitprim fork, but I've confirmed that part of the code is the same across both)evoskuil19:13
The call has a dependency on previous output metadata. If not populated the value is invalid.
vdeurzen20:40
and that data won't be available when decoding a single block from a hex blob, makes sense 👍 thx
Are there any particular projects / issues that could do with some help in the libbitcoin project? I've been talking to my colleagues and bosses about maybe contributing to OSS projects, this could be a good candidate imho 🙂
neillm20:42
@vdeurzen Of course! Take a look through the issues on any of the repos and see if anything catches your interest?
Of if you have a specific interest, let us know and I'm sure there's something to be done 😉
Wednesday, October 10, 2018
Monday, October 15, 2018
Fil02:53
@evoskuil continuing on my work the next task is to implement FTC's diff algo which itself consists of 3 hard forks. I'd like to discuss them here briefly so we can align ideas about implementation.
Let's call the 3 hard forks HF1, HF2, HF3.
HF1 is trivial. It just changes the value of a couple of settings parameters which yields to a shorter retargeting interval. Implementation-wise there are two possible approaches: either set the parameter value upon hard fork activation or add new dedicated parameters for HF1. The former feels a bit hacky but the latter clutters the settings a bit and adds conditional logic to the code.
HF2 and HF3 use an additional timestamp sample to compute the new required difficulty. Therefore, chain_state::data::timestamp::retarget is not sufficient anymore. Moreover, HF3 computes a new difficulty target for every block. Thus, it would make sense to increase the size of chain_state::data::timestamps::ordered such that the required values can be obtained from there.
3 replies · evoskuil, Fil · last Oct 16, 2018
evoskuil08:25
HF1, I would use the parameter change. This is the reason for exposing the parameters. I agree it feels a little hacky, but this pattern is already established with activations, checkpoints and network parameters. The flags are really for logic forks. Flags are recorded with validation, so in any case where a fork is conditionally activated it must be a flag.
(edited)evoskuil08:34
chain_state::data::timestamp::ordered is a dequeue with dynamically-computed size (timestamp_count(...)), so you just need to incorporate the different size computation.
(edited)vdeurzen16:35
Has anyone here ever used a ledger nano s to sign a transaction which is then broadcast using libbitcoin?
skaht17:39
@vdeurzen - Not smart enough to do so, but I did compile experimental Trezor firmware for Monero 2 years ago that functioned for a few months before a semiannual hard fork occurred. Likely to be game for testing using OSX.
Ignoring recent Ledger MITM firmware distribution attacks, I’m not exactly happy with the level of Nano S support for Monero, see https://www.reddit.com/r/ledgerwallet/comments/9nl6vx/eminent_monero_hardfork_support/ and https://monero.stackexchange.com/questions/9815/support-for-a-ledger-nano-s-recovery-plan-b.
Monero Stack Exchange
For a Ledger Nano S recovery contingency plan, how can one compute an associated vector of 25 Electrum wallet recovery words for each primary address using BIP 44 technology that is seeded from BIP...
vdeurzen18:24
@evoskuil why? We're working on our wallet infrastructure and were considering ledger usage in our first version (will upgrade to heavy duty HSMs at some point, still digging into that for now)
would love to know why one might want to avoid ledger 😄 0.o
though, I can attest to an odd situation when I first ordered a nano s where the shipment simply never arrived but was marked as 'delivered' by DPD 0.o'
Tuesday, October 16, 2018
skaht05:33
@vdeurzen - Nano S does not have a trusted display, plus is not delivered from source with tamper resistant seals. Have used bx to confirm if firmware has likely not been subjected to MITM distribution attacks.
@evoskuil @skaht other than nano, any hints / tips on how one might go about building a safe infrastructure using a HSM?
I've been googling around a bit but haven't found much yet 😕
(edited)are there better options?
What is the use case?
I need to implement some automated way to move around our bitcoins after we receive deposits, so that we can distribute it across different hot/cold wallets
I haven't worked directly on exchanges, but I can give you a few thoughts on security as it may pertain.
If you are concerned about state (and corporate) level threats you should realize that an HSM is a sweet target, just like a RNG.
that's a good point, but I am not concerned about that yet. I'm mostly thinking of having the private keys as secure as possible in our production system. Software tools only go so far in protecting the (private) keys. I want to ensure that even if an attacker gains entry into our systems that I can be reasonably sure that the time it would take to get to our keys is enough time for us to shut down the platform / take action
evoskuil08:49
It's also worth considering the spending process, because it is not necessary to obtain the keys to steal the coin.
change the transaction being signed to send it somewhere else?
evoskuil08:50
Not necessarily, it should just be an insider spending, or an outside attacker spending...
1 reply · evoskuil · last Oct 16, 2018
evoskuil08:51
IOW you will have a process for authorizing the spend and this will presumably be weaker than the HSM.
We might want to automate some of the spending of smaller UTXO
brb, have a daily right now 🙂
evoskuil08:52
Multisig adds strength only if the sigs are independent, which is hard to achieve.
(edited)An outside attacker can spend without the keys because all spending occurs without the keys (they are presumed to be retained in the HSM).
(edited)vdeurzen09:13
Ah, right, you mean an outside attacker can just send a transaction to the HSM if they have access to the platform, nothing we can do about it 😕
Ledger misapplied HSM technology by using a shared secret across devices for bootstrapping authorization, claiming "bank level security"...
as used in chip/pin. But banks don't do this with their cards, they embed a unique secret on every card.
To provide a comparable level of security the ledger would need to do the same. The HSM in these scenarios is designed as a "proof of something I have".
vdeurzen09:22
yeah, which is what I expected from ledger actually, rather disappointed they don't do that 0.o
What is the benefit of proving you have a ledger?
evoskuil09:24
And given the class secures a much higher reward, the financial disincentive to attack it is destroyed.
But let's assume you are willing to risk this because you assume that even if the ledger attestation key is compromised, the platform with which it communicates is not...
You are therefore trusting the platform (including the people with access to the HSM).
evoskuil09:30
But given an automated process, you are already in this boat, since the platform/operators must have the ability to direct the device to spend.
These devices, with trusted manufacture and proper design (identity vs. attestation) can reasonably secure against loss when the device is stolen or lost, assuming its operator is trusted (like when you operate your own).
vdeurzen09:34
the goal of the HSM is to slow down finding the keys, but with a successful attack you could send tx to the HSM for signing which would not compromise they but still compromise the funds, so the HSM is not going to help us much in that case, unless we have some way of ensuring only valid parties can send anything to the HSM
vdeurzen09:35
The isolation is something we are taking quite serious, so that will always be there, we have tried to design everything as isolated as possible
evoskuil09:36
Multisig is designed to facilitate isolation, but it remains very difficult to isolate authority if you also need to join that authority without too much manual process to spend.
(edited)vdeurzen09:37
that is a question of risk I guess, not easy to avoid unless you create rather convoluted secretive processes, which would slow down business
evoskuil09:38
Yes, but the point is that one should not assume that an HSM provides a security benefit in these scenarios. In fact it may increase the attack surface.
All things being equal, complexity is the enemy of security.
This is basically the design trade-off made by Trezor, which IMO is a very reasonable choice.
vdeurzen09:40
This is very helpful so far, will help immensely when discussing these topics with our security experts 🙂
vdeurzen09:40
What does Trezor have over ledger in this case? It's still an implementation of a HSM right?
It's just a small computer, not designed for tamper-evidence or resistance.
evoskuil09:46
In both models you must trust the manufacture of the device. The HSM (of done properly) simply raises the cost of getting the secrets out of the device (assuming one doesn't already have that authority).
i.e. those that would attack if it was cheap but not if the cost gets high enough?
evoskuil09:47
If the benefit is higher than the cost then the attack is profitable regardless of who carries it out.
evoskuil09:47
So it limits the attack to those who can raise the necessary capital to carry out the attack.
vdeurzen09:48
but you cant' really do anything against that realistically, the only option you have there is somehow distributing your funds such that any single attack will always cost more than its benefits
How might one logically "distribute" funds (physical distribution being largely irrelevant)?
(edited)There must be independent decision-makers (humans) in the loop.
(edited)vdeurzen09:51
you'd have to essentially split the funds between hot and cold wallets in such a way that any of the funds available in hot wallets is less than the cost of breaching those
but yeah, not an easy problem and will require a human decision indeed
Information provided by machines.
It's not an easy problem.
not suggesting we're going to solve that any time soon 🙂
just trying to build layers of security to increase the cost of an attack wherever possible
Complexity is risk, and if you have layers of complexity that can be circumvented at lower cost than going through them, then it's a net loss.
(edited)we should be able to reason about it easily
if we cannot achieve that we are shooting ourselves in our foot
evoskuil09:58
So if for example the software/person with access to the ledger can enumerate the wallet and get it to sign away all of its coin, there is no additional security provided by it.
evoskuil09:58
If that person is the individual who owns the wallet, it's a more reasonable option. This is why hot-hsm-wallet scenario doesn't make a lot of sense to me.
(edited)vdeurzen09:59
I expect we will have to think very hard about a good process for signing any high value transactions and how we can apply multisig, different devices and proper hot/cold distribution
evoskuil10:05
Yes, and the one thing to never lose sight of is that this is primarily a business process problem, not a technology problem.
Maybe review the process that banks use to move gold around.
Or the process that military organizations use to authorize a nuclear strike.
Bitcoin is more like these than account-based banking because its transactions are irreversible.
M
Mlmikael13:23
The P2WSH pubkey script, the hash it contains, is it a single- or dual-iteration SHA256 of the witness program (which is stored in the spending input's witness data)?
Could you give me some reference to where LibBitcoin validates or generates such a hash in the pubkey script?
7 replies · James C., Mlmikael, evoskuil · last Oct 17, 2018
James C.16:45
Hi Mikael! I think witness script validation is here: https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/witness.cpp#L454
The witness verify method is here:
https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/witness.cpp#L478
And the overall script evaluate method is here:
https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/script.cpp#L1381
Is that what you are looking for?
(edited)M
Mlmikael01:26
James, excellent thanks. Right the P2WSH pubkey hash is based on single-iteration SHA256. This was not so easy to read out of Bitcoin Core - there apparently this https://github.com/bitcoin/bitcoin/tree/master/src/script/standard.cpp#L24 is the main location, and there's some very secondary location such as this https://github.com/bitcoin/bitcoin/blob/master/src/script/interpreter.cpp#L1433 .
Actually now when talking about it, where does LibBitcoin do the HASH160 for the P2WPKH pubkey hash validation?
(edited)James C.07:56
Hi Mikael, glad that was what you were looking for. There is a P2PKH script “extraction” step from the witness object here:
https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/witness.cpp#L436
A couple lines above, you’ll find how the PKH from the witness is moved into a standard P2PKH script template for evaluation.
https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/witness.cpp#L363
(edited)James C.07:58
I wrote up a script machine documentation article for the wiki a while back, perhaps helpful?
https://github.com/libbitcoin/libbitcoin/wiki/Script-Machine
(edited)James C.08:03
The actual script run of the witness-extracted P2PKH script happens here:
https://github.com/libbitcoin/libbitcoin/blob/master/src/machine/interpreter.cpp#L29
(edited)James C.08:10
(the interpreter evaluates all script runs, not not P2W specific, and is called by program.evaluate() )
evoskuil18:31
I added comments to clarify the hash160 question: https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/witness.cpp#L436
(edited)skaht22:55
@vdeurzen - Both Trezor and Ledger devices native use cases are for warm storage, not cold storage.
(edited)Wednesday, October 17, 2018
M
Mlmikael01:26
replied to a thread: view
James, excellent thanks. Right the P2WSH pubkey hash is based on single-iteration SHA256. This was not so easy to read out of Bitcoin Core - there apparently this https://github.com/bitcoin/bitcoin/tree/master/src/script/standard.cpp#L24 is the main location, and there's some very secondary location such as this https://github.com/bitcoin/bitcoin/blob/master/src/script/interpreter.cpp#L1433 .
Actually now when talking about it, where does LibBitcoin do the HASH160 for the P2WPKH pubkey hash validation?
(edited)Thursday, October 18, 2018
M
Mlmikael09:21
on the topic of implementations, there is a javascript codebase out there that claims to be a bitcoin full node implementation, started Apr 2014, https://github.com/bcoin-org/bcoin/commit/8a3c9d2dcdadf17c904590343c13e9421211ad1c , called bcore.
i would remain questioning about its degree of actual existence as in how much it works, however someone has made many commits as least.
interesting i've never heard a comma about it.
GitHub
Javascript bitcoin library for node.js and browsers - bcoin-org/bcoin
evoskuil10:09
Libbitcoin does not store the transaction pool in RAM. Validated transactions are stored on "disk". Given the disk is file-mapped memory, txs may be memory-resident depending on operating system behavior and available resources. Version 4 will use a metadata graph for generation of block templates, but this is not a tx mempool.
(edited)vdeurzen12:41
Do I understand correctly that with recent versions we will be able to work with other chains like litecoin and bch ?
neillm12:43
@vdeurzen Work has been done to allow litecoin to be used, with only configuration differences. I don't know about the bch status, but it's the same idea (although I'm not sure all of the fork support for that is in)
how would one configure the use of litecoin?
(edited)neillm12:44
GitHub
Contribute to Feathercoin-Foundation/libbitcoin-config development by creating an account on GitHub.
vdeurzen12:48
On another note, I've been using bitprim specifically because it offers different coins (through recompilation) and because it supports conan. With the coin-agnostic update to libbitcoin, I'm only missing conan support. Is there any specific reason for the use of autotools and pkg-config? Would you be open to a PR with CMake and conan support?
@here Thank you for you awesome efforts!
😄 1
neillm12:50
I think conan came up here before, though I'm not familiar with it personally. If there's a way to integrate CMake/conan into the libbitcoin-build generation project cleanly, I don't see why it would be an issue.
neillm12:51
I don't think replacing autotools/pkg-config will happen though, it would probably have to work alongside
A PR to start discussion and see what we're looking at can't hurt though
lclc15:32
We’ve only added Litecoin so far, but if someone wants BCH too and can support us in this feel free to message me directly
Friday, October 19, 2018
vdeurzen13:12
Writing conan package recipes for the autotools build is a bit painful, would be very happy to know more about the CMake status 😄
Saturday, October 20, 2018
N
narodnik07:36
I saw someone here mention that libconsensus is not being updated / falling out of date? Was that you @neillm? What specifically is the problems with it?
Monday, October 22, 2018
vdeurzen14:34
When running master I get a segfault with the following backtrace:
#0 0x000055555584b400 in libbitcoin::chain::chain_state::median_time_past() const ()
#1 0x00005555556f9669 in libbitcoin::blockchain::populate_header::populate(std::shared_ptr<libbitcoin::blockchain::header_branch>, std::function<void (std::error_code const&)>&&) const ()
#2 0x00005555556e3c74 in libbitcoin::blockchain::validate_header::accept(std::shared_ptr<libbitcoin::blockchain::header_branch>, std::function<void (std::error_code const&)>) const ()
#3 0x00005555556d13e3 in libbitcoin::blockchain::header_organizer::organize(std::shared_ptr<libbitcoin::message::header const>, std::function<void (std::error_code const&)>) ()
#4 0x00005555556bf9ff in libbitcoin::blockchain::block_chain::organize(std::shared_ptr<libbitcoin::message::header const>, std::function<void (std::error_code const&)>) ()
#5 0x000055555567c106 in libbitcoin::node::protocol_header_in::store_header(unsigned long, std::shared_ptr<libbitcoin::message::headers const>) ()
#6 0x000055555567d338 in boost::asio::detail::completion_handler<std::_Bind<std::_Bind<std::_Bind<void (libbitcoin::node::protocol_header_in::*(std::shared_ptr<libbitcoin::node::protocol_header_in>, unsigned long, std::shared_ptr<libbitcoin::message::headers const>))(unsigned long, std::shared_ptr<libbitcoin::message::headers const>)> ()> ()> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) ()
#7 0x000055555561a142 in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#8 0x00005555558cff1a in boost::detail::thread_data<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::{lambda()#1}>::run() ()
#9 0x00005555559b3b55 in thread_proxy ()
#10 0x00007ffff7988a9d in start_thread () from /usr/lib/libpthread.so.0
#11 0x00007ffff758ab23 in clone () from /usr/lib/libc.so.6is master considered stable?
Or should I run one of the feature branches?
evoskuil14:36
No, master is staging for v4 and definitely not stable. Use version3 except for development.
(edited)vdeurzen14:37
ok, is there a list of features available in version3? I.e. does it have header sync? It was mentioned that for litecoin (altcoin) support master was the only option 🤔
I've successfully created conan packages for all libbitcoin elements btw, they would be a lot better with CMake, but they work with autotools / pkg-config at the moment 🙂
evoskuil14:39
Header first sync (and continuous parallel block download) is v4. Litecoin is v4 only. That work is done but there is other ongoing work.
vdeurzen14:40
I've manually written the conan recipe's that simply wrap the autotools and pkg-config based build
there are probably still some bugs, but so far it seems to work quite well
vdeurzen14:43
I have to double check with my company if I can simply publish this but I expect to be able to do so 🙂
evoskuil14:43
We will need to integrate this into libbitcoin-build, and will want to incorporate CI if possible.
libbitcoin-build generates all build artifacts for all repos.
vdeurzen14:44
I expect that won't be a problem, but this being the first (potential) open source publication from our side I'll have to check before I publish what we have 🙂
ah, version3 is more recent it seems
Tuesday, October 23, 2018
A
Alex07:52
When I download https://raw.githubusercontent.com/libbitcoin/libbitcoin-server/v3.5.0/install.sh and run it, the script clones the git repos for branch version3 and not tag 3.5.0. The resulting build is, therefore, not the tagged code. Is this the desired behavior?
jungly07:53
Hi @Alex, I think this is desired behaviour. You want to use the branch and not the tag to get v3 release
i am new to libbitcoin too, just that eric explained that yesterday, so it is kind of fresh in my head 🙂
A
Alex07:57
Hi @jungly, I see that Eric wants changes to v3 in branch version3. This makes sense to me as those changes should be tested and released in a future tag. But an "install" of 3.5.0 resulting in installation of version3 doesn't make sense to me.
jungly07:58
i think that is because the branch is considered to have the most recent "release" as opposed to the tag. I think that might be the reason.
James C.10:04
Hello, I have been working my way through the P2P network stack. Here’s a first stab at an introduction to the p2p library:
https://github.com/libbitcoin/libbitcoin-network/wiki/Introduction-to-the-Libbitcoin-Network-Library
I will follow up with more detailed code examples for the documentation wiki later this week. Suggestions/Corrections welcome as always!
(edited)👍 1
Some of the patterns described in the article are common to c++, so perhaps not 100% Libbitcoin specific, but still might help somebody following the code for the first time.
4 replies · James C., jungly · last Oct 23, 2018
James C.14:05
Hi @jungly, if there are unclear items on the wiki docs, please let me know, always looking to improve these.
(edited)jungly14:06
I am currently in the middle of trying to write tests for v4 transaction_database class. I haven't done much c++ for around a decade, so you can imagine, i was going around in circles figuring out how things work 🙂
jungly14:08
What would be awesome is even just the structure of mmapped files and how different sections map to various data structures like hash_table_header, hash_table, and maybe later even transaction structure (if that is not already documented)
James C.14:25
ok thanks for the pointer, I know there are some pretty neat memory-mapped design aspects there, I hope to contribute some docs in the coming weeks as I learn it myself! Glad to hear you are working on it as well … 🙂
vdeurzen13:03
I seem to be getting null / empty transactions when subscribed to transactions, I've enabled transaction relay and refresh in my node config. Is there anything else I should be configuring?
i.e.
!transaction is true with transaction being the object passed to my transaction handlertransaction should be a bc::transaction_const_ptrtestnet btw
neillm13:05
No particular command though, just in code? (was wondering if like watch-address or subscribe-tx)
maybe I was just impatient
The version3 branches are considered production code. Releases as tagged across the repos periodically to provide a consistent historical build with a knowable feature set. It is possible we might merge a patch into a release branch that could cause a break (though we are pretty careful about that). Building from a tag set would prevent that. This is only important when actually in production deployment.
The tx notification should only have a null tx ptr if there is an error code.
This is the case only during channel or service stop.
But channels stop frequently.
pmienk21:02
yep, its in, it works. it disables bindings. i also noticed when checking in that it only allows boolean for bash-completion-dir instead of processing a string. cmake declares data types for the parameters so it would require two settings
also, i encoded everything as cached variable settings instead of using the option construct for boolean flags
-D<with/enable>-<name>=<true/false/yes/no> for all but pkgconfigdir parameter, -DCMAKE_PREFIX_PATH=<prefix> -DCMAKE_INSTALL_PREFIX=<prefix> for prefix encoding
-DBUILD_SHARED_LIBS=<on/off/true/false> will toggle between shared/static
annoyed it took me all of friday and monday to track down that secp256k1's pc file was emitting libs backward. i was deep in the weeds looking at rpath parameterization because it doesn't look right either
pmienk21:24
when we'd talked before, you had said that builds take long enough and we'd really have to double the matrix to cover. i thought we'd said it wouldn't be covered by travis due to perf
i can double the matrix and see what it takes
Wednesday, October 24, 2018
47 replies · pmienk, vdeurzen · last Oct 25, 2018
pmienk17:12
An artifact of the xml as in autotools they are applied to both c and c++. Cmake doesn't carry that distinction as I could identify. It also results in a few warnings when noreorder isn't applicable to c.
vdeurzen09:14
Not entirely sure what you mean though, in CMake you only have to mention them once afaik. Also, they use the global
add_compiler_flag but the recommended way, I believe, is to use target-specific methods e.g target_add_compiler_flag. Is there a reason for using the other setup?pmienk09:32
The cmake artifact is generated from our libbitcoin-build project which generates from common information currently placed in a file generate.xml. I agree that duplicate declaration is not necessary, however it wasn't harmful and to avoid it would require detecting the redundancy in the generate.xml and excluding it. Given the intent for all such flags to apply to all targets, add_compiler_flag is the proper mode of declaration.
pmienk09:33
is the compiler_flag method what you're referring to when you say setup or are you pointing to something else?
vdeurzen09:33
yes that's what I meant. I should note though that using global flags can mess with situations where the library is used as a dependency, which can create complicated hard-to-debug builds
(edited)vdeurzen09:35
I'll try to integrate the new CMake build with Conan and see if I hit any issues, could be the builds are isolated well enough that it doesn't matter 🙂
pmienk09:35
I don't see how that could be the case. If you're linking the library as a dependency, you needn't rely on the CMakeList.txt. Cmake can invoke pkg-config - it's what I'm doing in FindLibbitcoin.cmake
vdeurzen09:36
someone could try to use libbitcoin with External_Project_Add from CMake, which would lead to the issue I mentioned
pmienk09:37
it also retains the pkg-config/pc file mechanism that is used in the autotools build. they generate identical .pc files and will link between build systems
vdeurzen09:37
not a recommended approach. The global flag approach excludes some uses of the library in certain build set-ups, which is why I mentioned it
(edited)vdeurzen09:38
It may also lead to issues for people building libbitcoin using
add_subdirectory and just copying the library to their local set-uppmienk09:38
certainly, however I thought was ill-advised and why Find<foo>.cmake mechanisms exist. While i don't install a copy of them with the installed library, I believed that was the desired mechanism for inclusion
vdeurzen09:40
certainly, that is the recommended way indeed, and I would say nowadays it's probably best to use Conan or another package manager. The main point being, you don't have control over or know exactly where your library will be used and how it will be built, using the globals means some techniques won't work, but that will be detected most likely through odd build failures
pmienk09:40
I'd assume that all non-target mechanisms would potentially cause problems, which would require a tradeoff between include directory inclusion at the target and the suppression of warnings due to SYSTEM declared directories.
vdeurzen09:41
I.e. unless you provide a very clear documentation stating exactly what your build set-up requires (e.g. examples of how to include libbitcoin in a CMake build) you may get some odd issues regarding build failures.
vdeurzen09:42
I'll get a conan example up and running, which is what I would recommend as library vendor to my users 🙂
pmienk09:42
i'll look forward to it. easy enough to point to any but the base library's mechanism for including other repository libraries.
vdeurzen09:43
my testing conanfile:
[requires] libbitcoin-node/0.1.0@blockport/stable fmt/5.1.0@bincrafters/stable spdlog/1.1.0@bincrafters/stable range-v3/0.3.7@blockport/stable dry_comparisons/0.1.0@blockport/stable tl_expected/0.2.1@blockport/stable [generators] cmake [imports] lib, *.so* -> ./lib lib, *.a -> ./lib
vdeurzen09:44
It took a moment to get all dependencies found correctly when using autotools and conan together, but the pkg-config being available allowed conan to reuse that 🙂
pmienk09:44
i'm not aware of conan, so i'm not clear what i'm looking at. i assume it's a package distribution system
vdeurzen09:45
package /dependency management for C++ (but generic enough to be used for other languages too)
vdeurzen09:45
It means a libbitcoin based project to build a custom node, would only need that line in a file called
conanfile.txt and then it would pull it in and build/link everything correctly 🙂vdeurzen09:46
Coming from a project where I had to manage all deps using CMake, switching to Conan has made my life much nicer 😄
pmienk09:47
well, if you can get it working, that'll be great. if it's generalizable then we might be able to pull together a gsl generator to maintain it.
vdeurzen09:48
and I need to find out from my employer if I can publish this as is, given I've made this at work
pmienk09:48
i'm not sure what the requires section contains. are those supposed to be version numbers?
vdeurzen09:49
Yeah, the syntax is not very clear there. Those are package version numbers. I picked 0.1.0 for the time being, but that should be the semver of libbitcoin when used officially
pmienk09:50
if you're using master, it's currently 4.x.x in anticipation. but it doesn't take much other than cherry picking to backport the generator to 3
vdeurzen09:51
I'm using version3 as I am actually building on top of libbitcoin myself and need a stable version 🙂
pmienk09:53
requires packages for all dependencies. but if you've got it working they mostly must already exist
pmienk09:55
anyway, if you happen to encounter any cmake failures due to environment or system changes, i'd be interested. don't have access to osx, don't know if it'll work. will be testing with travis in a bit, once i rework our installer script
at other times it's almost instant
neillm13:51
Hrm, I suppose it depends on some factors. I've noticed that GNU/Linux shutdowns are either near instant, or up to about 2.5-3 minutes. I don't know about regular shutdowns taking 20 minutes though
vdeurzen13:52
normally it's much faster indeed, but from time to time it takes very long though the process doesn't seem to be doing anything
a force kill get's the process to stop, but then the database is corrupt -.-'
yes, I do have it off
(edited)I mean, when the node is near the top/current, enable the write sync. At that point, negative perf impact is much less significant, and corruption is far less likely
(edited)I am curious if that might resolve the shutdown delay you're seeing
neillm13:59
Might be worth taking a 'known good' snapshot at some point to help speed up future syncs, particularly for v3 which still takes a while.
with the flushing on I haven't been able to reproduce the issue 🙂
stopping is nearly instant 😄
evoskuil15:09
It’s not possible to predict or control how long the os will take to flush writes, but if you enable write flushing after getting fully synced it will definitely not take long.
It does significantly slow sync however.
With write flushing off the time is loosely a function of write activity. More writes means potentially longer flush.
(edited)Thursday, October 25, 2018
vdeurzen11:49
When using libbitcoin-node, which function should I use to broadcast a newly created transaction? Is that something like
node.chain().organize(my_new_tx, [...](...){...} ) ?James C.12:23
Hm, I think you could do this
(edited)node.broadcast(message::transaction, channel_handler, result_handler), which sends the transaction message to all p2p peers. node::full_node inherits broadcast from network::p2p …thx
👍 1
vdeurzen12:57
@James C. just finished reading your network documentation, very well written and thorough !
2 replies · James C., vdeurzen · last Oct 25, 2018
James C.13:01
Thats great to hear somebody has read it already! 🙂 I’ve been learning the interface myself recently … like attaching protocols to channels or inheriting the p2p object, like full-node does, which the full_node does, I hope to finish them soon. I enjoyed working through the network stack!
vdeurzen13:02
It's great that documentation is getting such a great amount of attention, it's one of the main reasons why I think libbitcoin is such a great library 🙂
😄 1
vdeurzen13:11
btw @James C. I am assuming for now the 'normal' process is to broadcast the transaction immediately, rather than 'push' it to the blockchain?
or should I do something else entirely 🤔
evoskuil13:15
Actually the tx should be organized. This will cause it to be validated, and stored if valid. A notification is then picked up by the tx out protocol, resulting in broadcast to peers.
Just follow the logic in the tx in protocol for the receipt of a new tx.
I had previously suggested just broadcasting without validation…
or the protocol_transaction_in class, which seems to be more applicable
This seems like the right method:
(edited)bool protocol_transaction_in::handle_receive_transaction(const code& ec, transaction_const_ptr message) ?2 replies · James C. · last Oct 25, 2018
James C.13:50
Sorry. I’ve found what eric was referring to here. The protocol_out subscribes to the chain notifications here, which I assume would include your newly organised TX. https://github.com/libbitcoin/libbitcoin-node/blob/master/src/protocols/protocol_transaction_out.cpp#L79
(edited)James C.13:52
That TX is then announced to the peer here: https://github.com/libbitcoin/libbitcoin-node/blob/master/src/protocols/protocol_transaction_out.cpp#L283
(edited)vdeurzen13:42
to clarify, my intention is to scan a subtree of an HD wallet to find addresses that have unspent outputs and to create and sign transactions to move those outputs to selected addresses
I have a full node so I can effectively request the history for any of my addresses and see if I need to move coins around or not, if I find an address with unspent outputs I want to create and sign the transaction and then broadcast it to the network
vdeurzen13:56
should the
protocol_transaction_out be constructed by me or is there an instance I can request from somewhere or a utility method / function that will do the right thing ?You don’t use that. You just organize the tx.
thx
Look at the libbitcoin-server code that accepts a tx from a client.
You are basically doing that.
vdeurzen14:06
the handler I pass is essentially just to check the result, is that correct, i.e. check the
error_code and log / respond somehow?Friday, October 26, 2018
vdeurzen09:25
When calling
node->chain().fetch_history(...) I get a segmentation fault:0x00005555558ba132 in libbitcoin::database::record_multimap<std::array<unsigned char, 20ul> >::find (this=0x9b0, key=...) at ./include/bitcoin/database/impl/record_multimap.ipp:78 78 const auto begin_address = map_.find(key)
I call this from the block_handler callback using the regtest network and version3 builds of libbitcoin-node
Should I do something else to ensuse the database is ready?
the
this address is rather weird in any case 🤔hmm, might be settings issue then, index start height
vdeurzen12:17
So, I'm testing with the testnet and I'm completely caught up, however, using a public faucet I'm not seeing my transaction coming by anywhere 😕 does libbitcoin use a particular test network?
vdeurzen12:24
@neillm if you happen to have some testnet btc lying around, would you mind sending a small amount to
miJsK8hgg2i18Gs2CKNYshDhW1qGvAE8rS ? like 0.01 btc or so?Is there maybe a faucet available for this testnet?
vdeurzen12:40
I just now found my transaction from a faucet, thank you for sending it, that way I can double check it's being received correctly 😄
yup, I see it came through 😄
Sunday, October 28, 2018
L
ldz15:04
hello. I know the libbitcoin-build is for maintainers but I wanted to learn more about the libbitcion build process. I have looked at the gsl.copy_modules.sh and the function generate_copy_modules. I don't see where the "output(out_file)" function comes from. output isn't defined in the libbitcoin-build. the imatix documentation tells me about ".output filename". is output(filename) another syntax for the output file manipulation?
pmienk19:29
@ldz please see documentation: https://github.com/zeromq/gsl/blob/master/README.md#templates-and-scripts
(edited)James C.20:17
New p2p channel subscriptions: https://github.com/libbitcoin/libbitcoin-network/wiki/New-Channel-Subscriptions
Next p2p topics which I am working on: attaching custom channel protocols, simple broadcast.
(edited)It indicates gsl command vs template code in the appropriate mode.
(edited)Whatever documentation you're reading presumes a mode.
(edited)L
ldz23:32
I think I understand the difference between template and script-command but I didn't know it makes a difference for an output setting. Or maybe I don't quite follow you. In the documentation I can see that we are in a template mode a we set ".output "filename". "." means script mode, I know. in the function generate_copy_modules (script mode) we call "output(filename)". The question is about the parenthesis. Is it the same? "output filename" and "output(filename)"
L
ldz23:47
James C., I do not have much knowledge about bitcoin but you documentation promises to be very interesting. I will surely read all the topics 🙂.
1 reply · James C. · last Oct 29, 2018
James C.07:17
That’s great! Please let me know if things are unclear or you have suggestions! Also, I have found the source code to often be the best source of documentation on the Bitcoin protocol 😉
Monday, October 29, 2018
neillm set the topic: Discuss libbitcoin here or on IRC (freenode): #libbitcoin00:18
B
baiumbg08:41
Hey guys! We're having some trouble pairing a libbitcoin-server instance with a bitcoind one on regtest. Libbitcoin-server seems to be rejecting all block generated using the
generate
or
generatetoaddress
RPCs. The configuration is more or less identical to the one in the libbitcoin-server wiki. We're using version 3.4 for libbitcoin-server and 0.17.0 for bitcoind. Are there any settings that have not been described on the wiki? Any ideas would be greatly appreciated.
settings.conf
# Libbitcoin Server configuration file [log] # The debug log file path, defaults to 'debug.log'. debug_file = debug.log # The error log file path, defaults to 'error.log'. error_file = error.log # The log archive directory, defaults to 'archive'. archive_directory = archive # The size at which a log is archived, defaults to 10000000 (0 disables). rotation_size = 10000000 # The minimum free space required in the archive directory, defaults to 0. minimum_free_space = 0 # The maximum combined size of archived logs, defaults to 0 (maximum). maximum_archive_size = 0 # The maximum number of logs to archive, defaults to 0 (maximum). maximum_archive_files = 0 # The address of the statistics collection server, defaults to none. statistics_server = 0.0.0.0:0 # Enable verbose logging, defaults to false. verbose = true [network] # The minimum number of threads in the network threadpool, defaults to 0 (physical cores). threads = 0 # The maximum network protocol version, defaults to 70013. protocol_maximum = 70013 # The minimum network protocol version, defaults to 31402. protocol_minimum = 31402 # The services exposed by network connections, defaults to 9 (full node, witness). services = 9 # The advertised services that cause a peer to be dropped, defaults to 176. #invalid_services = 176 # The magic number for message headers, defaults to 3652501241 (use 118034699 for testnet). identifier = 3669344250 # Validate the checksum of network messages, defaults to false. validate_checksum = false # The port for incoming connections, defaults to 8333 (use 18333 for testnet). inbound_port = 18444 # The target number of incoming network connections, defaults to 0.full file
Snippet from the log
08:25:01.054454 DEBUG [server] ================= startup Mon Oct 29 10:25:01 2018 ================== 08:25:01.054630 INFO [server] ================= startup Mon Oct 29 10:25:01 2018 ================== 08:25:01.054669 WARNING [server] ================= startup Mon Oct 29 10:25:01 2018 ================== 08:25:01.054840 ERROR [server] ================= startup Mon Oct 29 10:25:01 2018 ================== 08:25:01.054872 FATAL [server] ================= startup Mon Oct 29 10:25:01 2018 ================== 08:25:01.054896 INFO [server] Using config file: "settings.conf" 08:25:01.054917 INFO [server] Please wait while the server is starting... 08:25:01.055230 DEBUG [database] Buckets: block [650000], transaction [110000000], spend [250000000], history [107000000] 08:25:01.056425 DEBUG [database] Mapping: "blockchain/block_table" [5200177] (4096) 08:25:01.056478 DEBUG [database] Mapping: "blockchain/block_index" [12] (4096) 08:25:01.057122 DEBUG [database] Mapping: "blockchain/transaction_table" [880000267] (4096) 08:25:01.057513 DEBUG [database] Mapping: "blockchain/spend_table" [1000000008] (4096) 08:25:01.057912 DEBUG [database] Mapping: "blockchain/history_table" [428000036] (4096) 08:25:01.057963 DEBUG [database] Mapping: "blockchain/history_rows" [55] (4096) 08:25:01.058570 DEBUG [database] Mapping: "blockchain/stealth_rows" [4] (4096) 08:25:01.059084 INFO [network] Starting manual session. 08:25:01.059180 INFO [network] Not configured to populate an address pool. 08:25:01.059234 INFO [server] Seeding is complete. 08:25:01.059268 INFO [node] Node start height is (0). 08:25:01.059572 INFO [network] Not configured for accepting incoming connections. 08:25:01.059622 INFO [network] Starting outbound session. 08:25:01.059665 WARNING [network] Failure fetching new address: object does not exist 08:25:01.059770 WARNING [network] Failure fetching new address: object does not exist 08:25:01.059806 WARNING [network] Failure fetching new address: object does not exist 08:25:01.059861 WARNING [network] Failure fetching new address: object does not exist 08:25:01.059902 WARNING [network] Failure fetching new address: object does not exist 08:25:01.059932 DEBUG [network] Failure connecting outbound: object does not exist 08:25:01.059964 WARNING [network] Failure fetching new address: object does not exist 08:25:01.059995 WARNING [network] Failure fetching new address: object does not exist 08:25:01.060021 WARNING [network] Failure fetching new address: object does not exist 08:25:01.060045 WARNING [network] Failure fetching new address: object does not exist 08:25:01.060077 WARNING [network] Failure fetching new address: object does not exist 08:25:01.060121 DEBUG [network] Failure connecting outbound: object does not exist 08:25:01.061447 DEBUG [server] Applied address authentication to socket [query] 08:25:01.061561 INFO [server] Bound public query service to tcp://*:19091 08:25:01.061694 DEBUG [server] Connected public query worker to inproc://public_query 08:25:01.061830 DEBUG [server] Applied address authentication to socket [heartbeat] 08:25:01.061906 INFO [server] Bound public heartbeat service to tcp://*:19092 08:25:01.062030 DEBUG [server] Applied address authentication to socket [block] 08:25:01.062121 INFO [server] Bound public block service to tcp://*:19093full file
B
baiumbg12:20
Okay, did some digging and found out that 0.15 works fine. Has this been looked at for 0.16+?
most likely something related to segwit changes made to regtest (genesis) blocks
👍 2
hasn't been fixed for use with 16+ yet
vdeurzen15:34
Every once in a while I get:
ERROR [network] Error seeding host addresses: unresponsive peer may be throttling which leads to the node being stopped, can this be controlled somehow? Out of 5 seed hosts I see ~3 connecting fine and 1 or 2 that have either some connection or DNS issue from time to timeall this seems to happen during startup, is there an ec I could check for and choose to ignore in certain cases? Or maybe retry automatically?
vdeurzen15:55
INFO [network] Failure contacting seed [testnet-seed.voskuil.org:18333] unable to reach remote host 😜I'm using the testnet config as specified by libbitcoin
interesting is that the failure to connect is not always leading to the node stopping 🤔 It just started again even though 2/5 seeds were unreachable
So there is a chance that it's another thing killing the node ... hmm, I'll start printing the error code
evoskuil16:14
My node is not up, that’s perfectly normal behavior. It will start if it gets enough addresses in total.
evoskuil16:16
It will try until timeout and stop if address count is insufficient at that time. If you want to guarantee start just provide hosts in the file and it will skip seeding.
(edited)vdeurzen16:17
right, good point, I currently don't save the hosts cache file, I'll ensure that is done 👍
evoskuil16:19
No prob. What’s this issue with regtest? It looks like it’s parsing segwit blocks without segwit enabled.
vdeurzen16:19
something changed in bitcoind v0.16+ when using regtest, I ran into this a while ago when trying to have a local bitcoin network on my machine, I haven't taken the time to debug it too much though
I tried going over their changelog but did not find anything that would explain the behaviour to me
so libbitcoin is happy to accept blocks generated on the regtest network with bitcoind 0.15, but nothing newer
evoskuil16:21
It doesn’t make sense to me that it would be their issue. Is that when they enable segwit?
vdeurzen16:21
I'm not 100% sure at the moment, their changelog did not indicate that (or I missed it)
It might be that they simply added segwit to all blocks, including the genesis block
but I haven't looked into it for a bit so I'm a little out of context at the moment
vdeurzen16:25
ok, the problem I think is introduced by bitcoind enabling the feature for all blocks 🤔
the problem is observed starting with bitcoind v0.16
B
baiumbg16:26
Segwit is now always active in regtest mode by default.
vdeurzen16:30
the observed behaviour is as follows:
1. start regtest bitcoind
2. start libbitcoin node, with a peer config pointing at the regtest bitcoind (v16+)
3. generate 1 block using the CLI tool for bitcoind
expected:
libbitcoin logs receival of a block and stores it
observed:
block is rejected and requested again, and rejected and requested again ad infinitum
👍 1
evoskuil16:36
That’s obvious from the log. The key entry is the zero inputs error. That signifies parsing of a segwit tx with segwit parsing disabled (assuming the block is valid).
That’s the result of parameterization of the parser, and that’s the result of self and peer config.
what can I do to help 🙂
evoskuil16:56
I’d probably get into the block deserialization in the block_in protocol and figure out what is the witness parameter, and work back from there.
vdeurzen22:44
some naive checking using the hex of the v0.17 regtest block and
bc::chain::block::from_data(...) the first thing that strikes me is that with the new regtest block is_valid_proof_of_work() returns false, the same goes for is_valid_coinbase and is_segregatedthis is when constructing the
block with witness set to false btwsame result with
witness set to true auto const raw = bc::to_chunk(bc::base16_literal(
"010000000000000000000000000000000000000000000000000000000000000000000"
"0003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4ada"
"e5494dffff7f200200000001010000000100000000000000000000000000000000000"
"00000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d"
"65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b2"
"06f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2"
"052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a6796"
"2e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b"
"6bf11d5fac00000000"));
bc::chain::block block_regtest;
block_regtest.from_data(raw); // or with witness = true
On a side note, I'm allowed to publish the conan recipes for libbitcoin, they'll need some work to be generalized but shouldn't be too much work. I'll also update them for CMake and provide them for both version3 and 3+
(edited)2 replies · evoskuil, vdeurzen · last Oct 29, 2018
evoskuil22:50
In order to maintain this in libbitcoin we'll want to incorporate the generation of these into libbitcoin-build.
vdeurzen22:54
regarding the deserialization of the block, I'm not quite sure where to find it? From the
(edited)::from_raw method it does seem to parse, given it doesn't give any error with the given HEX dataevoskuil22:56
and forwarded to chain::block
NOTE: is_valid_coinbase_claim(size_t height)
there we go, needed to set retarget to false indeed
evoskuil23:17
In addition to height coinbase claim requires chain state, it cannot validate if not populated.
I'll continue tomorrow, bed time now 😴
evoskuil23:17
is_valid_proof_of_work requires the retarget parameter, which is false if regtest.
(edited)ah
hehe 😜
I'll dig into this a bit more tomorrow and/or later this week
parsing at least does not fail immediately or obviously (using ::from_raw)
evoskuil23:19
The important consideration is that the block is not segregated, though it's not clear to me that it actually should be.
vdeurzen23:20
this is how bitcoind serializes the block (detailed view) to JSON:
{
"hash": "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
"confirmations": 2,
"strippedsize": 285,
"size": 285,
"weight": 1140,
"height": 0,
"version": 1,
"versionHex": "00000001",
"merkleroot": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
"tx": [
{
"txid": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
"hash": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
"version": 1,
"size": 204,
"vsize": 204,
"weight": 816,
"locktime": 0,
"vin": [
{
"coinbase": "04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73",
"sequence": 4294967295
}
],
"vout": [
{
"value": 50.00000000,
"n": 0,
"scriptPubKey": {
"asm": "04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f OP_CHECKSIG",
"hex": "4104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac",
"reqSigs": 1,
"type": "pubkey",
"addresses": [
"mpXwg4jMtRhuSpVq4xS3HFHmCmWp9NyGKt"
]
}
}
],
"hex": "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000"
}
],
"time": 1296688602,
"mediantime": 1296688602,
"nonce": 2,
"bits": "207fffff",
"difficulty": 4.656542373906925e-10,
"chainwork": "0000000000000000000000000000000000000000000000000000000000000002",
"nTx": 1,
"nextblockhash": "29a3efe9d8edf16f00e87515efb6acb08dc1156eab6c2da80b9f0d1451b5e08c"
}should you want to have a look
signing off for now though, thanks your help 🙂 👍 !
(edited)👍 1
Tuesday, October 30, 2018
vdeurzen12:24
on a completely different note, building version3 has a whole bunch of warnings like
extra ';', have these been address in master/version4?vdeurzen12:29
no, many of them inside libbitcoin I believe, though I don't know if that might still link through to being a boost error:
include/bitcoin/bitcoin/wallet/ec_public.hpp:64:27: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
...
include/bitcoin/bitcoin/impl/machine/operation.ipp:333:45: warning: comparison is always true due to limited range of data type [-Wtype-limits]
return value >= op_186 && value <= op_255;
....
include/bitcoin/bitcoin/impl/utility/pending.ipp:52:2: warning: extra ';' [-Wpedantic]
neillm12:30
Ah, interesting. Those are left to track down then. The reason they are there is because extra warning levels were enabled somewhat recently, both in v3 and master.
interesting
I can go through a few of those at some point, simple enough
👍 1
vdeurzen14:13
My node is hanging on shutdown, gdb shows that what it's stopped / waiting for is the threadpool.join() in p2p::close(), don't think it's still flushing ... seems to be hung on a condition variable
L
ldz19:25
is the "artifacts generated" list of libbitcoin-build complete? what about build_base.bat? why is build_base.bat an artifact and build_all.bat not?
evoskuil19:32
If it’s flushing it will be under a lock (which implies a condition variable), but this would also be the case if it deadlocked.
If sources never change they don’t really need to be generated.
vdeurzen19:58
@evoskuil given the time it's taking to shutdown I'm expecting some kind of deadlock here
It’s not very common. Need to trap it in a capable debugger in order to identify it.
There are very few locks in libbitcoin that are difficult to verify against deadlock. Really it’s just one, where a closure is executed under lock. The closure can contain arbitrary code, and if it attempts to reenter the critical section will deadlock.
Not until it’s known.
thread sanitizer is spitting out some hints 😄
it's picking up on a fair number of data races actually
some in boost log, some in destructors for boost mutex / condition variable some in libbitcoin from a quick read of the output. However, this is the first time I'm using tsan so I'm not sure how to exactly interpret the output yet
or
the second one has 4 spaces indent
L
ldz22:43
VS has a option "position each attribute on a separate line" for XAML. I am not sure about XML.
L
ldz22:51
I don't think so. They look cosistent. Only for readability. I have looked also on the libbitcoin style guide. Of course 80 characters is stupid for XML but with the "pretty print for attributes" you can read the file from top to down without horizontal scroll. sometimes you will need to scroll for values, but at least you can see all attributes without horizontal scrolling.
there is "align attributes each on a separate line" for VS XML.
evoskuil22:57
But it's not the default, so everyone has to change it or it will cause reformatting of files. I really don't see any reason to mess with these files. Developers should never even see most of them.
afk
Wednesday, October 31, 2018
On the right side, there's a menu under that section for each of the config sections
vdeurzen11:40
I've used the bn.cfg from libbitcoin-node for the moment, with the defaults mostly kept
thx!
vdeurzen15:06
I'm pretty consistently getting my process (node) hung when trying to 'organize' a (custom) tx
it seems anyway
gdb is having trouble getting passed that
at least in GDB I can't step over the lock and if I just run the node and try to organize a TX it never shows the output from my result handler function
evoskuil15:11
I’m pretty sure however that txs have been submitted to our public servers without deadlocking them. Are you using V3?
I'm building the transaction within the block handling callback and trying to organize it
evoskuil15:13
It’s seems likely that a closure in your custom subscriptions is calling back into the message handler.
evoskuil15:15
This is really the only deadlock risk in libbitcoin, since arbitrary code is invoked under a critical section.
evoskuil15:24
Most other critical sections have trivial-to-verify execution paths that cannot deadlock. Actually the organization critical sections also notify, and the notify is the also a set of subscription handlers.
This ensures completion of a message before accept of another, which is essential for DoS protection.
vdeurzen15:25
that makes sense then, it's indeed solved by moving the organizing of the TX out of the subscription callback
now I'm getting to why I'm not seeing a proper tx propagated
stack false, apparently some part of my TX script is not quite correct 😛Easy to debug in our interpreter.
1 reply · neillm · last Oct 31, 2018
neillm16:21
Perhaps this can help or be extended/improved: https://github.com/libbitcoin/libbitcoin-client/tree/master/examples/console
just a basic REPL
an interesting challenge 🙂
It’s set up as a machine with state that can be initialized with a simple set of parameters that can be read from the bx command line.
evoskuil16:44
You can populate all previous output metadata on inputs, create a chain_state object, and then call tx.check, tx.accept and tx.connect
Check is context free, requires no chainstate or metadata.
Connect evaluates scripts.
evoskuil16:46
Or you can use the script verify method to evaluate individual scripts (just do what’s done in .connect).
Verify works for block, tx, input, script.
(edited)evoskuil18:28
Note that the design is such that there is no necessary database interaction with validation. All validation can be performed in the base libbitcoin (system) library.
L
ldz21:37
I'm not deep in it, but why to use the new gsl.copy_modules.sh on the basis of generate.sh for cmake instead of the gsl.find.cmake on the basis of the current exists templates inside generate.xml? It seems to be a bit inconsistent but I'm not sure.
pmienk22:41
@ldz I made gsl.find.cmake to generate a the dependencies supporting pkg-config. Given that those files don't change after first generation and that there were other non-generated cmake modules, I did not feel the need to plug their generation in on every invocation of generate.sh and instead copy them along with the other custom Find<library>.cmake files.
L
ldz22:50
@pmienk if I have understood correctly, gsl.find.cmake should be something like a template to generate the find...cmake content?
pmienk22:51
gsl.find.cmake generates a subset of the content found in
cmake/ the directory containing Find<library>.cmake modules.it does not generate
FindDl.cmake, FindMbedtls.cmake or FindRt.cmake in that directory.