Bitcoin full node and query server
November 2018
Archive of the libbitcoin Slack #general channel.
Thursday, November 1, 2018
vdeurzen12:20
Where does one find the UTXO hash for spending? according to: http://aaronjaramillo.org/libbitcoin-building-a-raw-transaction, I need the UTXO hash, but the output returned by tx->outputs() don's have a hash method or anything that I see that would provide it ?
The Web Log of Aaron Jaramillo
The Testnet Building and signing a raw transaction in libbitcoin is relatively easy but can still be prone to costly errors. For this reason, I’ve adapted my previous HD_Wallet class to be fo…
though https://github.com/libbitcoin/libbitcoin/wiki/Building-Transactions this suggests the tx hash is what should be used. With the TX hash my script (when checked with tx->connect()) fails with the message 'previous output not found'
evoskuil12:31
There is no such thing as a uxto hash in a tx, the hash of the tx with the unspent output, and the ordinal number of that output within the tx, identifies the unspent output.
vdeurzen12:32
that's what I thought, and I can find the transaction and the output (at index 0), but using that information
connect() still returns the error mentioned aboveIt’s not talking to a blockchain service.
organizing the tx still leads to a
stack false result, so I'm still trying to find out whyevoskuil12:41
To call accept or connect methods you must populate metadata on the tx, specifically the previous outputs.
Or you can simply call verify for the specific script that is returning false (which requires the previous output as a parameter).
vdeurzen12:45
in what sense do I have to populate the tx, I've pushed the input I'm spending (identified by the tx hash and index)
evoskuil12:49
You are referring to the output with a pointer to it (the input), but to validate that input you need the actual output. The input has a metadata field for its output. Each not-provably-unspendable input requires an output.
(edited)evoskuil12:57
No, metadata is set directly on “metadata” fields. In this case the previous_output().metadata.cache
so if I have the original transaction (retrieved by fetch_transaction(tx_hash), is there an easy way to build a new transaction from there?
evoskuil13:01
You mean to add the metadata? Yes, just set the outputs on the fields of the tx you are validating.
(edited)vdeurzen13:21
it's rather confusing at this point tbh, I basically just want to broadcast a new transaction and it's costing quite some time just to get the right parameters together -.-'
I have a fair understanding, but the small details combined with the library api are not helping me at the moment
evoskuil13:26
If you aren’t familiar with the structure, such as the means of referring to a previous output using a tx hash and index, it will be a bit of a challenge.
And validating a tx or script in libbitcoin is trivial, but you have to have the tx or script and the previous outputs to do so.
In fact you have validated it via the server (which looks up the previous outputs for you) but you have a false script.
(edited)vdeurzen13:28
I have all of that and I am essentially familiar with the conceptual structures, however, combined with tutorials / wiki pages that apparently don't contain the full details needed to spend a transaction it becomes rather complex. On top of that I am not supposed to rely on the
validation member which I need to update with the previous output apparentlyWhich is the documented method on the wiki.
thx
I found the page on the wiki, I'll see if I can get through that
vdeurzen15:25
Is there an easy way to print the evaluation stack? I'm looking at raw bytes which is rather had to read
As far as I can see it should be a valid transaction, but after the prevout evaluation the verification returns with stack false, though from what I can see the signature is fine, and the input verification also returns successfully 😕
neillm15:31
@vdeurzen I've only loosely been following your discussion about this, but if you have a tx, can you use tx-decode in bx to look at it?
James C.16:08
I’m not sure this is self-explanatory, but this is a script to print out the stack after each operation: https://github.com/teachbitcoin/code-demos/blob/master/A_script_machine_tool/00_script_machine_testing_tool.ipynb
This is the print-out from a validation run.
=========== Input script evaluation =========== > Operation: 0 [00149a19a31c2fda7d0c30215ec954a20a542aa84ad3] > Stack after operation: 0 [00149a19a31c2fda7d0c30215ec954a20a542aa84ad3] =========== Output script evaluation ========== > Operation: 0 hash160 > Stack after operation: 0 [dcdc2f89b96c420751e3750da7d5073a81b16946] >> Operation: 1 [dcdc2f89b96c420751e3750da7d5073a81b16946] >> Stack after operation: 1 [dcdc2f89b96c420751e3750da7d5073a81b16946] [dcdc2f89b96c420751e3750da7d5073a81b16946] >>> Operation: 2 equal >>> Stack after operation: 2 [01] ----------- P2SH pattern detected ------------- ======= P2SH Embedded script evaluation ======= > Operation: 0 zero > Stack after operation: 0 [] >> Operation: 1full file
11 replies · James C., vdeurzen · last Nov 02, 2018
James C.16:17
let me know if you can get this to run … you can try these functions in https://github.com/teachbitcoin/code-demos/blob/master/A_script_machine_tool/supporting_scripts/script_machine.cpp
vdeurzen16:27
I'm doing exactly that at the moment, and I'm wondering what exactly I should use as entry point?
James C.17:12
sorry, github’s jupyter notebook preview is somehow not working right now, but here is a step by step demo. https://nbviewer.jupyter.org/github/teachbitcoin/code-demos/blob/master/A_script_machine_tool/00_script_machine_testing_tool.ipynb
(edited)James C.17:16
so in step 1, you fetch the previous utxo with BX, step 2 you can just paste your spending TX and spending input index, and step 3 run the libbitcoin interpreter.
vdeurzen09:44
ok, so jupyter, bash and cling is the best option, I'll follow your notebook. Impressive work!
James C.09:51
@vdeurzen tbh, you don’t need all that, you can follow the BX steps in your command line, then include the same
script_machine.cpp file in your cpp script.vdeurzen09:51
getting cling + jupyter installed anyway, it's actually quite nice for prototyping I guess 🙂
James C.09:54
yeah, I use it for live demos, although, with cling, you can’t run a jupyter cell more than once if you’re defining/constructing a new object, which is obvious, but makes it less interactive as with python. With python, a constructor/reassign is “interchangeable”, which can also be dangerous I guess… let me know what you think of cling!
vdeurzen09:55
I've tried it when it was just released, was a bit unstable at that point, but I'm really excited for it 🙂
👍 1
L
ldz18:08
@pmienk I have no idea about cmake but I noticed the following. We create the libbitocoin.import because libbitcoin is a dependency of libbitcoin. more precisely libbitcoin is a dependency of libbitcoin-test and libbitcoin-example. Similarly, for other repositories. But we don't create libbitcoin.cmake for libbitcoin (for libbitcoin-test and libbitcoin-example). Is that a problem?
pmienk18:23
@ldz Can you provide me some context for your question? Given the guards in cmake around producing anything for
msvc and the import files being utilized only in that environment to my knowledge I'm unclear as to whether or not your concerns are primarily for msvc or if you're aiming for a cross-platform understanding of how the dependency is satisfied.L
ldz18:40
@pmienk I am not saying the imports have something to do with cmake. That was an analogy. The question is only about cmake. The context: As a cmake-user, do I need to use a libbitcoin.cmake to build libbitcoin-test? If yes the libbitcoin.cmake is missing inside libbitcoin.
*inside the libbitcoin repository
... or not 🙂. I'm not sure.
pmienk18:44
@ldz libbitcoin, libbitcoin-examples and libbitcoin-test are all produced by the the same MakefileList.txt. They are linked by
target_link_libraries. Specifically, they are linked to the library project of the repository by target_link_libraries( <target> ${CANONICAL_LIB_NAME} ).The linkage between repositories is accomplished using the
Find<library>.cmake file emitted into the depending repository. That linkage utilizes the pkg-config .pc file emitted for that purpose the same as is used via Makefile.am.there is also libbitcoin.org
it's waiting for someone to perform the integration
maintainers don't control the .org domain
community effort 🙂
evoskuil21:35
I think D3#<d3@riseup.net> owns the domain and Jaromil<jaromil@dyne.org> maintains the content.
L
ldz21:39
@evoskuil are you going to make the github.io more up to date? some links are dead, the latest release is 2.0 etc...
Currently it's just dead code.
evoskuil21:45
Presumably we would move the content from current libbitcoin.org into github.io but there's no point in that until we have a build process that produces libbitcoin.org from github.io.
Friday, November 2, 2018
evoskuil01:43
Working my way though v4 parallel block download over the past few days. I resolved a couple of trivial but material issues in header sync. Getting closer.
👍 2
vdeurzen12:46
Related to building transactions, does it make sense to still create non-segwit transactions when I control both source and destination?
neillm12:47
That's a matter of preference, but yes, it makes sense. There's nothing wrong with p2kh/p2sh transactions 🙂
th
*thx -.-
Saturday, November 3, 2018
L
ldz20:30
@neillm I am debugging electrum due to the electrum test bug. Not sure but our (libbitcoin) test entropy is not supported by the electrum wallet. the numeric entropy need to be >=2^121. Our numeric entropy for the spanish words is 386744892 < 2^121.
L
ldz20:40
https://pastebin.com/4uxufRKV the result inside electrum. from (random) numeric entropy to seed https://pastebin.com/4uxufRKV
the boost cpp_int says "integer constant is too large" after replace the numeric_entropy inside visual studio
neillm21:57
@ldz Not sure what you mean. The entropy is not an unsigned long, and you specify what to use.
1. Electrum creates numeric entropy. If numeric entropy <2^121 then create again. We pass data_chunk entropy to the function. Then we convert the data_chunk entropy to numeric_entropy with the cpp_int type. Our numeric entropy seems to be 386744892<2^121. Electrum would say "to small create again". This seems to be the first difference.
2. I have replaced the cpp_int with uint256_t for tests and I have created the seed from the large entropy from the pastebin link (maybe it workd also with cpp_int good but I don't know). The first seed word from libbitcoin VS is different than the first word from electrum (linux). The same problem as our libbitcoin test on windows/linux. I am debbuging now to find out the difference.
electrum: increases i and pass to the function. libbitcoin: pass to function and increase. ok, it doesn't matter, is important only for my tests :D
neillm22:55
For #1, all entropy is passed in. If you pass something too small (or too big), it will run and not complain, but can't match electrum, as noted. No entropy is generated at all (unlike electrum). For 2, that doesn't make sense to me. I know what the issue is (regarding the normalization, as noted in the issue around it), but resolving it is tricky since I have to decode some python-isms. If it were the entropy, none of the other tests would pass either, so I don't think that's it.
To be clear, I'm not discouraging you from working on your findings though. If you think the entropy isn't matching up, I'm curious, since I've spent a lot of time verifying that against electrum on the initial version, and encoded some of it in the tests. My problem is that I only tested english setups, which is all that was supported in the version of electrum that I was using.
L
ldz23:15
1. OK, we can agree here :). 2. I am not talking it has something to do with entropy. If you say it has something to do with normalization I believe you. I have not tested so deep yet.
evoskuil23:50
Resolved an issue in node, so header sync is running more smoothly. Still need to apply some optimizations to improve its performance, but concurrent linear header sync with reorganization’s appears to be working well.
👍 2
Sunday, November 4, 2018
skaht00:11
@ldz - Not sure if the info below will help. Try the following with bx that assumes 386744892 is base 10, base16( 386744892) = 0x170D423C.
% echo 170D423C | bx mnemonic-new -l es
atún hallar millón
The entropy can be measured using http://www.fourmilab.ch/random/ the example below.
% echo 170D423C | bx base16-decode | ./ent -b
Entropy = 0.974489 bits per bit.
Optimum compression would reduce the size
of this 32 bit file by 2 percent.
Chi square distribution for 32 samples is 1.12, and randomly
would exceed this value 28.88 percent of the times.
Arithmetic mean value of data bits is 0.4062 (0.5 = random).
Monte Carlo value for Pi is nan (error nan percent).
Serial correlation coefficient is 0.093117 (totally uncorrelated = 0.0).
(edited)fourmilab.ch
ENT: A Pseudorandom Number Sequence Test Program
L
ldz00:13
@neillm I don't know how important it is but the normalize_text method from electrum removes accents. I don't see anythink like that inside libbitocin is_new_seed method. is it not important?
It doesn't affect English, which is why I didn't realize its broken
L
ldz00:22
@neillm I know but maybe you mean only the dependency like boost / py libraries. I assume it may have something to do with missing implemntation
The implementation is incomplete
skaht00:25
Should point out the following:
% echo 170D423C | bx mnemonic-new -l en
blade head monkey
blade maps to line 185 at
(edited)https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt#L185
head maps to line 849 at https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt#L849
monkey maps to 1146 at https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt#L1146
atún = https://github.com/bitcoin/bips/blob/master/bip-0039/spanish.txt#L185
hallar = https://github.com/bitcoin/bips/blob/master/bip-0039/spanish.txt#L849
millón = https://github.com/bitcoin/bips/blob/master/bip-0039/spanish.txt#L1146Looks like bx is performing as expected.
@ldz This is something we definitely need resolved before the v4 release, so it's actively being looked at. It's just that lots of things are 😉
neillm00:40
@skaht No worries. Not sure if you saw the new electrum stuff in v4 (only), paralleling the mnemonic stuff, mostly
PSA: Don't use it with non-english mnemonics right now 🙂
skaht01:16
FYSA - Electrum test vector synthesis?
For Electrum English words, look at https://xmr.llcoins.net/addresstests.html (Monero uses Electrum words), and place 170D423C into the Hexadecimal Seed field and click the the Gen1 button. Results are:
picked alerts evicted evicted
(The last Electrum word is always one of the previous word.)
I might be wrong (high probability), but the example at https://github.com/libbitcoin/libbitcoin-explorer/wiki/bx-electrum-new appears to have two extra hex digits as input, i.e., 05e669b4270f4e25bce6fc3736170d42 3c . Place 05e669b4270f4e25bce6fc3736170d42 into the Monero URL above.
Not sure if the approach below works for Electrum as it does for BIP 39… The Spanish words start 23 lines later.
picked
https://github.com/monero-project/monero/blob/master/src/mnemonics/english.h#L1121 = luz https://github.com/monero-project/monero/blob/master/src/mnemonics/spanish.h#L1144.
I will shut up now…neillm01:18
@skaht I think monero uses electrum v1 style mnemonics with a different word list. Could be wrong.
neillm01:21
Well, libbitcoin v4 will use electrum 2.x mnemonics. The word lists for electrum 2.x are the same as BIP39
skaht01:26
BIP 39 is essentially base 2048. Is the current Electrum 2.x list length going to be the same size? Older Electrum v1 lists are only 1626 words.
skaht01:31
Is there a test for end users to know they are dealing with a list of words used for BIP 39 or Electrum v2? Otherwise, confusion will emerge.
neillm01:35
Electrum v2 can't generate a word list that is electrum v1 compatible. It's part of the generation test (i.e. if it's a valid v1 mnemonic, skip it and use next)
That's the only reason you'll find the electrum v1 word list still inside libbitcoin at this point, since electrum v1 mnemonic support was dropped a while ago
skaht01:54
FYSA - Here is an interesting issue related to mapping BIP39 words to older Electrum v1 words needed for a Ledgerwallet Nano S to adequately support Monero.
https://monero.stackexchange.com/questions/9815/support-for-a-ledger-nano-s-recovery-plan-b
The post at https://www.reddit.com/r/ledgerwallet/comments/9nl6vx/eminent_monero_hardfork_support/ fell on deaf ears. The Ledgerwallet recovery tool to generate your Monero 25 word seed is essentially vaporware.
Monero Stack Exchange
For a Ledger Nano S recovery contingency plan, how can one compute an associated vector of 25 Electrum wallet recovery words for each primary address using BIP 44 technology that is seeded from BIP...
L
ldz02:01
@neillm Not sure but it could have something to do with the hmac_sha512_hash function inside is_new_seed. I have removed the whole stuff from normalize_text except the normalize('NKFD, seed) call inside electrum. The seed bit pattern are same before and after the function call as well in electrum as in VS. They differ after hmac call.
skaht02:13
It would be really cool if bx could directly support the hmac_sha512_hash function function.
neillm02:14
@ldz Interesting, send me some results and we can compare notes. Either way, the normalization is broken though.
% stretch -f sha512 -r 2048 “army van defense carry jealous true garbage claim echo media make crunch” mnemonic | sed ‘s/^b.//’ | sed ‘s/.$//’ | bx base64-decode | bx base16-encode
5b56c417303faa3fcba7e57400e120a0ca83ec5a4fc9ffba757fbe63fbd77a89a1a3be4c67196f57c39a88b76373733891bfaba16ed27a813ceed498804c0570
skaht02:15
% echo “army van defense carry jealous true garbage claim echo media make crunch” | bx mnemonic-to-seed -p “”
5b56c417303faa3fcba7e57400e120a0ca83ec5a4fc9ffba757fbe63fbd77a89a1a3be4c67196f57c39a88b76373733891bfaba16ed27a813ceed498804c0570
https://pypi.org/project/hmac_cli/ only works if the -key is ASCII, see https://github.com/msabramo/hmac_cli/issues/1.
L
ldz02:25
I am away. Have a nice day.
L
ldz11:32
I have looked again at the english seed. The hamac behaves same for english words. There is something strange with hmac and accents chars. https://pastebin.com/WATALDGa
apart from the hamac problem, electrum implementation doesn't work with accents but we do. Despite this, I need to look deeper at the hamac function another time.
L
ldz16:58
debugging through the original test case I see also normalization issue now. inside electrum: before nkfd call:
'647565c3b16f20636f646f20c3a16d62617220696e736563746f20766572626f206372c3a17465722063656c6f736f20656e74726172207461726a6574612073616c6120636f636f20667269746f'
'6475656ecc836f20636f646f2061cc816d62617220696e736563746f20766572626f20637261cc817465722063656c6f736f20656e74726172207461726a6574612073616c6120636f636f20667269746f'
after
inside the first loop
:/
VS is different :/
evoskuil17:10
HMAC is not going to vary by character (or platform), it consumes bytes. It’s the character conversion to bytes that is making the difference (which can be platform sensitive).
(edited)VS is not a different HMAC, it’s the conversion from non-ascii to bytes that is different. Windows uses UCS2 encoding whereas Linux uses UTF8, this requires a rather complex translation which is often poorly understood.
Libbitcoin fully implements http://utf8everywhere.org
If your test cases don’t deal with it properly they will exhibit these sort of differences.
evoskuil18:44
For more information see comments here: https://github.com/libbitcoin/libbitcoin/blob/master/include/bitcoin/bitcoin/unicode/unicode.hpp#L28
L
ldz19:23
@evoskuil I am not sure if I can agree with you. I mean the byte consumption (processing) seems to be different on (my) windows /VS libbitcoin environment. I don't compare words or chars but byte inputs and byte outputs. I am not working inside isolated test environment. All I do is inside libbitcoin solution.
evoskuil19:26
So you are suggesting that it treats some bytes differently than others? Specifically (or coincidentally) when one or more of those bytes are independently derived from non-ascii characters?
Seems not only unlikely, but inconsistent with results in the many other test cases and actual usage involving HMAC.
L
ldz19:35
“So you are suggesting that it treats some bytes differently than others?“ I don't know. I only say the byte input that was created form words with accents gives me the same output (bytes) after calling HMAC with electrum HMAC function and .NET HMAC function. The output is different with the libbitcoin HMAC function inside libbitcoin solution / Windows. Which tests do you mean? Do we have another tests with accents?
L
ldz20:09
@evoskuil ok, never mind. I am seeing now following á produces cc 81 (hex) inside electrum. visual studio gives me -61 -96 (byte). It looks totally wrong. I am calling create_mnomonic through the bc:main inside libbitcoin-examples.
How are you introducing the text?
String literal?
L
ldz20:16
@evoskuil: no, automatically. Through the mnemonic_encode method (mnemonic.push_back from es lexicon).
L
ldz20:36
sry, á is not cc 81. á is 61 cc 88. 61 = a and cc 88 = accent http://www.fileformat.info/info/unicode/char/0301/index.htm
I mean utf-8 and linux. I am not sure about windows.
but C3 A1 seems to be hex utf-8
L
ldz21:29
But there is one thing I don't understand (among other things :D). The char "á" has a certain bit sequence for utf-8. The sequence is platform independent. The "á" literal has following bit sequence inside libbitcoin solution -61 -96 (I assume it is utf-8 because of utf-everywhere, why not?). -61 = 11000011 and -95 = 10100001 and this i exactly the bit patttern of U+00E1 in utf-8. Why is it even possible two have two different utf-8 bit pattern for the same char? Or is my understanding about encoding totally wrong?
With two different utf-8 bit pattern I mean electrum and vs/libbitcoin.
L
ldz21:51
OK, I am seeing the difference now :). I thought at first electrum and libbitcoin language files are the same but they don't. After using the word from electrum VS shows me the same encoding. So, the nkfd normalization needs to be strange... Of course you are right :).
But generally, it’s “UTF8 Everywhere”
L
ldz22:01
So, the normalization for "á" looks like: electrum: 0x61 0xCC, 0x81 (nkfd)-> 0x61 0xCC 0x81, libbitcoin (windows): 0xC3, 0xA1 (nkfd)-> 0xC3, 0xA1. electrum != libbitcoin (windows)
1 reply · evoskuil · last Nov 04, 2018
.NET Framework: 0xC3, 0xA1 (nkfd)-> 0x61 0xCC 0x81, 0x61 0xCC 0x81 (nkfd)-> 0x61 0xCC 0x81
0x61 0xCC 0x81 is the nkfd form for á.
evoskuil23:05
I think we’ve established there is a normalization difference on Windows. But it’s not at all clear to me where you are sampling inputs and outputs for Windows. Specifically, I’m confused as to why you show a different input to the normalization call.
If you are sampling at UCS2 it could very well be the behaviors are identical.
What do you mean with different input?
I believe I am not sampling at UCS2 because I am working with the "original" libbitcoin code.
evoskuil23:20
You drew a picture of inputs and outputs to nkfd. You showed different inputs for Windows.
"libbitcoin (windows): 0xC3, 0xA1 (nkfd)-> 0xC3, 0xA1"
I didn't try with 0x61 input on libbitcoin
L
ldz23:24
@evoskuil I don't understand. 0xC3, 0xA1 is my input. This input returns some same output inside electrum / .NET. I would expect the same output inside libbitcoin (on Windows). Is my approach wrong?
"I think we’ve established there is a normalization difference on Windows.". I am not trying to convince you about a normalization difference. I am sure you know it better than me. I only looked for me, I wanted better understand how it works and I have noticed 0xC3, 0xA1 behaves different with nkfd inside electrum/.NET vs libbitcoin (Windows).
evoskuil23:30
“So, the normalization for "á" looks like: electrum: 0x61 0xCC, 0x81 (nkfd)-> 0x61 0xCC 0x81, libbitcoin (windows): 0xC3, 0xA1 (nkfd)-> 0xC3, 0xA1. electrum != libbitcoin (windows)”... you are providing different inputs on Linux/.Net vs Windows.
L
ldz23:34
@evoskuil I thought that is the goal of the normalization function. Moreover, I have tried with 0xC3, 0xA1 input on electrum.
evoskuil23:37
The goal of the normalization test was to show that the behavior was different given the same inputs. That was what you declared. But the inputs are different. If you want to make sure all three normalize identically, you must provide identical inputs to the normalization call.
L
ldz23:40
@evoskuil No, I have tried also the same input. I have written about 0xC3, 0xA1 input. This input normalizes to 0x61... inside electrum and .NET but not inside libbitcoin (windows).
electrum: 0x61 0xCC, 0x81 (nkfd)-> 0x61 0xCC 0x81
libbitcoin (windows): 0xC3, 0xA1 (nkfd)-> 0xC3, 0xA1
L
ldz23:42
" libbitcoin (windows): 0xC3, 0xA1 (nkfd)-> 0xC3, 0xA1", "electrum: 0xC3, 0xA1 (nkfd)-> 0x61 0xCC 0x81", ".NET Framework: 0xC3, 0xA1 (nkfd)-> 0x61 0xCC 0x81"
@evoskuil I understand you position, but your quotation has also a relevance.
I mean two letter looks same but have different encoding and the normalization functions creates different outputs is also meaningfulness for me
L
ldz23:53
At first I wanted to say the letter that are different but look same (like the spanish a) don't normalize. After that I tested another cases.
Sorry for the confusing you.
Monday, November 5, 2018
L
ldz00:05
@evoskuil inside the main.cpp inside libbitcoin-examples I have pasted the code from the spanish test (entropy, decode_base16, call create_mnemonic). that is all.
3 lines
I am watching the first iteration.
-61 -95 before, 195, 161 after
0x61CC81 is the NFKD form
The unit test is encoded with the NFKC form.
evoskuil00:19
And apparently in the Windows normalization it is not being converted to the NFKD form.
I mean it is not the root of the problem.
evoskuil00:21
Not itself, it should just get normalized. But it suggests that the dictionary is not normalized.
Assuming the tests are drawn from the dictionary.
BOOST_AUTO_TEST_CASE(unicode__to_normal_nfkd_form__61cc81__c3a1)
{
const std::string literal_c3a1 = "á";
// This is pasted from the the BIP39 encoding:
//// github.com/bitcoin/bips/blob/master/bip-0039/spanish.txt#L93
const std::string literal_61cc81 = "á";
const std::string string_c3a1 = "\xc3\xa1";
const std::string string_61cc81 = "\x61\xcc\x81";
BOOST_REQUIRE_EQUAL(string_c3a1, literal_c3a1);
BOOST_REQUIRE_EQUAL(string_61cc81, literal_61cc81);
const auto normalized_c3a1 = to_normal_nfkd_form(string_61cc81);
BOOST_REQUIRE_EQUAL(normalized_c3a1.c_str(), string_c3a1);
const auto unchanged_c3a1 = to_normal_nfkd_form(string_c3a1);
BOOST_REQUIRE_EQUAL(unchanged_c3a1.c_str(), string_c3a1);
}
(edited)IOW, according to the Windows Unicode implementation, the BIP39 reference encoding (61cc81) is not nfkd and is converted to nfkd (c3a1).
This assumes the paste of the character from the reference is unmolested. I don't have a reliable reference for what should be the nfkd character according to standard.
It also implies that the dictionary (and or test cases) in libbitcoin are not encoded consistently with the reference.
(edited)I verified that the normalization parameterization is correct. Harder to verify if the result is correct, need a reference.
This gets sort of complicated... http://www.unicode.org/reports/tr15/
(edited)evoskuil19:22
I fully synced and validated testnet with master last night. Header sync is slow, and validation also needs some tuning. It will really shine when we add milestones. This is just the scaffolding on which to build at this point. I’ll restore and verify transaction sync today, and update console logging for validation feedback. So it should at least be fully functional at that point, though certainly not optimal.
👍 5🎉 1
2 replies · jungly, evoskuil · last Nov 05, 2018
evoskuil22:18
It's actually been really close for a few months, just had to shake out the stupid stuff.
L
ldz20:12
@evoskuil I don't know if it is appropriate for libbitcoin but I have found something: "...On both machines boost uses ICU. I added fold and normalize functions to the listing above - also here I get different output on Windows and Linux....", "I fixed it - for anyone, who might step into the same trouble here is the solution: Although I didn't get any error message while compiling nor when the application was executed it was not sufficient to just include icuucd.lib for linking you have to include icuind.lib as well. Then you get the same result on Windows and Linux." Rainer https://stackoverflow.com/questions/22331487/string-conversion-with-boost-locale-different-behaviour-on-windows-and-linux#-windows-and-linux#
(edited)L
ldz21:23
There are two backends "winapi" and "std" on windows. maybe using "ICU" it feel back to winapi (becasue there is not icu backend on windows) and cannot handle utf-8 properly. I will try with "std"-backend.
the method's comment says backend is ignored if invalid (icu on windows) but I hope it is wrong.
unhandled exception using "std" as backend
*to use correctly, that means to get right output.
evoskuil22:15
boost has a silent failure mode if it fails to link ICU, so we have a one-time run-time test case that throws if a normalization test fails. So it is normalizing, something.
If we link to a non-integrated library on Windows we get another big build headache. Where do this libs come from, how are they versioned, are there NuGet packages, do we have to end up owning them...
Over a year since the last update.
L
ldz22:29
I think your test doesn't test the backend suifficient. It only test the normalization in some cases and makes implicitly assumptions. Not sure, but I believe the main problem of the normalization problem is the missing icu backend. I'm not convinced to be possible to use normalization correctly without icu. And I don't know why icu is missing on windows. maybe the stackoverflow solution will help (some missing libs). I don't know.
2 replies · ldz, evoskuil · last Nov 05, 2018
Tuesday, November 6, 2018
jungly10:50
Hey guys, I was trying to build the server stack for v4 on my machine, and was running make check on all repos as I was going. The libbitcoin-protocol make check hangs with the following in test.log
tail: test.log: file truncated Running 131 test cases... Platform: linux Compiler: GNU C++ version 7.3.0 STL : GNU libstdc++ version 20180415 Boost : 1.62.0 test/zmq/socket.cpp(57): fatal error: in "socket_tests/socket__push_pull__brickhouse__blocked": critical check !poller__.wait(1).contains(puller.id()) has failed test/zmq/socket.cpp(568): info: subscriber test/zmq/socket.cpp(703): info: check
In case it helps, I installed zmq, libpgm and libnorm via debian apt-get install.
neillm12:20
@jungly This is a known issue. If you're able to resolve it, or provide more input on it, it would be great. Most of the time it works on my debian setup, but does hang sometimes. It almost always hangs on travis-ci, so we've been looking at it off and on. https://github.com/libbitcoin/libbitcoin-protocol/issues/130
9 replies · evoskuil, Alex, neillm · last Nov 07, 2018
evoskuil16:39
Just to clarify, this is an issue with the test case setup, not production code. It doesn't reproduce in VC++ builds, which has made it harder to debug.
A
Alex23:15
When I built v3.5.0 with zmq v4.2.5, I could not stop bs without the blockchain directory becoming corrupt as zmq errorred out. I do not remember the exact error message, but there was an issue already logged in the zmq repo for it. Downgrading to zmq v4.2.3 avoids this problem.
(edited)neillm01:20
@Alex Hrm, strange, I recall hitting that myself on an older version of zmq and upgrading was the answer. I have not seen it since we moved to v4.2.5. I am curious if you have more details
A
Alex01:33
Looks like it has been fixed in master so building with install.sh should have no problem.
neillm01:36
Ah, no, I see, interesting. Ok, the server issue I posted is not a match for that, but moved us to 4.2.5. I have seen that zmq issue in a previous version, but don't recall which exactly. After upgrading, I did not see it again. I'll note that you see it with 4.2.5 though
6 replies · evoskuil, neillm · last Nov 06, 2018
neillm19:23
I think it was already pointing at mainnet3, from the last outage, which is good -- it's up also
evoskuil16:39
replied to a thread: view
Just to clarify, this is an issue with the test case setup, not production code. It doesn't reproduce in VC++ builds, which has made it harder to debug.
evoskuil16:41
Master has been fully synced in testnet for a while now, and properly maintaining the top. However I've reattached tx in/out protocols and it appears that most/all txs are failing validation. This could be the result of any number of things, either in the tx validation or in block.tx storage/confirmation. I don't expect this to be a big issue, but you will notice it once the node gets fully synced, if you have tx relay enabled.
yikes, it's getting late and I can't type anymore 😛
Wednesday, November 7, 2018
B
baiumbg11:04
Does libbitcoin-server have any way to catch transaction rejections? More specifically
(edited)txn-mempool-conflict. I’m 100% sure it’s a double-spend and I need some way to catch that. The error handler for transaction_pool_broadcast doesn’t seem to be called with that error.B
baiumbg15:29
@neillm
10:21:18.221662 DEBUG [network] Received tx reject (18) from [127.0.0.1:28333] 'txn-mempool-conflict' [5fc3adc6f024a5ac6d345ab0032ccd964896a5b6a59ca30594e3dc4ca0951ea4].This is the result of a double-spend, but is logged after the actual broadcast. The error handler of
transaction_pool_broadcast isn’t called at all.neillm15:36
@baiumbg So you mean the broadcast (made from client) was successful, but the network later rejected the tx, and you would expect that the client call learn that it's rejected and return an error for it?
Not sure which API I can use for that.
subscribe_transaction doesn’t exist for version3 😕You could also use the standalone script: https://github.com/libbitcoin/libbitcoin-server/blob/master/data/subscribe-tx.py The server does support tx subscriptions, but perhaps the client call wasn't backported to v3
B
baiumbg16:21
That’s what I’m already doing. My issue is that I’m broadcasting a transaction, which later gets rejected and I don’t see any way I can get notified of that happening.
neillm16:23
You either note that it never comes in via subscription, or you run a node to log the reject (assuming your node is configured to not accept that kind of tx). I'm not sure of other shortcuts to just knowing that information somehow.
Thursday, November 8, 2018
vdeurzen10:30
I'm trying to figure out if there is a way for me to call a function that checks the database and tells me whether it is valid or corrupt (for logging purposes), but I don't see anything that would clearly help, any hints where I could find something like that ?
vdeurzen10:36
would it make sense to just 'open' the database and if the return value is false log a message and exit?
will do that then
on a side note, all those methods (at least in v3) return bool, might make sense for them to return an EC or some kind of 'expected' (eg. tartanllama's expected now that std::expected is on hold)
the 'open' and 'start' methods that is
vdeurzen11:50
imho it makes sense because errors are propagated several levels up, meaning the final error is rather generic 'operation failed'
different question, I've been synching mainnet and after about 240k blocks it just stops, no error, just the following logs:
"Stop protocol_ping on [54.169.209.7:8555] connection timed out" "Stop protocol_reject on [54.169.209.7:8555] connection timed out" "Stop protocol_address on [54.169.209.7:8555] connection timed out" "Stop protocol_block_in on [54.169.209.7:8555] connection timed out" "Stopped block_in protocol for [54.169.209.7:8555]." "Stop protocol_block_out on [54.169.209.7:8555] connection timed out" "Stopped block_out protocol for [54.169.209.7:8555]." "Stop protocol_transaction_in on [54.169.209.7:8555] connection timed out" "Stopped transaction_in protocol for [54.169.209.7:8555]." "Stop protocol_transaction_out on [54.169.209.7:8555] connection timed out" "Stopped transaction_out protocol for [54.169.209.7:8555]." "Fired protocol_block_in timer on [37.59.38.74:8333] success" "Peer [37.59.38.74:8333] exceeded configured block latency."
I'm thinking maybe it's related to a big block that won't download in 60 seconds and too few peers being connected to cover the timeout?
trying to reproduce it now
Friday, November 9, 2018
M
Mlmikael07:50
Question, does LibBitcoin have a Bitcoin variable size integer serializer?
I'd like to push the number 32 to the stack.
I know I can do it as a pushdata, so that's just by: { std::vector<uint8_t>({32}) }
so is there instead some { LibBitcoinVariableSizeIntegerSerializer(32) } that I can run?
vdeurzen09:44
@Mlmikael this might have some pointers I think: https://github.com/libbitcoin/libbitcoin/wiki/Working-with-Serialised-Data
Currently synching mainnet, been going for about ~18 hours now and we're roughly at block 378k, and it's slowed down quite a bit, anything I can do to speed that up?
evoskuil09:59
More memory, more CPUs, faster network (ie sync from local node). V3 sync is not efficient.
To push a number just push the operation onto the stack. The encoding size is minimized by default.
M
Mlmikael13:59
@vdeurzen thanks for the page reference, it's about compressed pubkeys and such, nothing about the variable size integer encoding.
@evoskuil you mean you are looking forward to the Bech32 pull request yes?
i think we may pull this off, do you have any particular wishes for HOW the Bech32 API should look?
..sec..
here's an example impl
which is proofed and borrowable.
neillm15:01
@Mlmikael libbitcoin already has base32 encode/decode, so that should help (in src/format). I would think bech32 support would be an extension of src/wallet/payment_address. But in truth, I wouldn't worry about these details as much up front. If you have a working PR, any of us are happy to review and move things around if needed too
I also often get those things wrong btw (class/path/etc), but it's always easier to correct/re-factor after it's working 🙂
Sometimes it's not obvious where things belong anyway when they're new, so it may be better to decide later
evoskuil15:14
Agree. I’d probably call it wallet::witness_address based on existing naming conventions and pattern the implementation after payment_address and stealth_address.
jungly17:35
Hi @evoskuil - I am running into some mutex unlocking issues when trying to write some tests for block_database. I am trying to split the test into smaller tests, like so: https://gist.github.com/kulpreet/38e0a74a600a7e3b945c34b1ab6c4048
But I get an error like the following snippet when using gcc 7.3
Untitled
```(gdb) r --run_test=block_database_tests
Starting program: /home/kulpreet/projects/libbitcoin/v4/libbitcoin-database/test/.libs/libbitcoin-database-test --run_test=block_database_tests
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Running 5 test cases...
entering ..block_database__get__empty_database__false
leaving ..block_database__get__empty_database__false
entering ..block_database__store__only_genesis_no_index__height_not_available
leaving ..block_database__store__only_genesis_no_index__height_not_available
entering ..block_database__index___only_genesis_candidate__height_available
libbitcoin-database-test: tpp.c:84: __pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)' failed.
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff61a3801 in __GI_abort () at abort.c:79
#2 0x00007ffff619339a in __assert_fail_base (fmt=0x7ffff631a7d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7ffff6b09c60 "new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)",
file=file@entry=0x7ffff6b09c54 "tpp.c", line=line@entry=84, function=function@entry=0x7ffff6b09d10 <__PRETTY_FUNCTION__.7568> "__pthread_tpp_change_priority") at assert.c:92
#3 0x00007ffff6193412 in __GI___assert_fail (assertion=assertion@entry=0x7ffff6b09c60 "new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)", file=file@entry=0x7ffff6b09c54 "tpp.c", line=line@entry=84,
function=function@entry=0x7ffff6b09d10 <__PRETTY_FUNCTION__.7568> "__pthread_tpp_change_priority") at assert.c:101
#4 0x00007ffff6b0855f in __pthread_tpp_change_priority (previous_prio=previous_prio@entry=-1, new_prio=new_prio@entry=0) at tpp.c:82
#5 0x00007ffff6afea98 in __pthread_mutex_lock_full (mutex=0x7fffffffb018) at ../nptl/pthread_mutex_lock.c:541
#6 0x00007ffff6aff055 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7fffffffb018) at ../nptl/pthread_mutex_lock.c:71
#7 0x00007ffff7bb76c9 in boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock (m_=0x7fffffffb018, this=<synthetic pointer>) at /usr/include/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
#8 boost::condition_variable::notify_all (this=0x7fffffffb018) at /usr/include/boost/thread/pthread/condition_variable.hpp:136
#9 boost::shared_mutex::release_waiters (this=0x7fffffffafe8) at /usr/include/boost/thread/pthread/shared_mutex.hpp:168
#10 boost::shared_mutex::unlock_and_lock_upgrade (this=0x7fffffffafe8) at /usr/include/boost/thread/pthread/shared_mutex.hpp:519
#11 libbitcoin::database::file_storage::reserve (this=0x7fffffffaf90, size=4143, expansion=<optimised out>) at src/memory/file_storage.cpp:376
#12 0x00007ffff7bb7911 in libbitcoin::database::file_storage::reserve (this=<optimised out>, size=<optimised out>) at src/memory/file_storage.cpp:333
#13 0x00005555555fb579 in libbitcoin::database::record_manager<unsigned int>::allocate (this=0x7fffffffb268, count=1) at ./include/bitcoin/database/impl/record_manager.ipp:126
#14 0x00007ffff7bae261 in libbitcoin::database::list_element<libbitcoin::database::record_manager<unsigned int>, unsigned int, std::array<unsigned char, 32ul> >::create(std::array<unsigned char, 32ul> const&, std::function<void (libbitcoin::serializer<unsigned char*>&)>) (write=..., key=..., this=0x7fffffffad80) at ./include/bitcoin/database/impl/list_element.ipp:93
#15 libbitcoin::database::block_database::store (this=this@entry=0x7fffffffaf90, header=..., height=<optimised out>, height@entry=100, median_time_past=<optimised out>, median_time_past@entry=0, checksum=<optimised out>, checksum@entry=0, tx_start=<optimised out>,
tx_start@entry=0, tx_count=<optimised out>, state=<optimised out>) at src/databases/block_database.cpp:239
#16 0x00007ffff7bae373 in libbitcoin::database::block_database::store (this=this@entry=0x7fffffffaf90, header=..., height=height@entry=100, median_time_past=median_time_past@entry=0) at src/databases/block_database.cpp:251
#17 0x00005555555ac355 in block_database_tests::block_database__index___only_genesis_candidate__height_available::test_method (this=this@entry=0x7fffffffc08c) at test/databases/block_database.cpp:109
#18 0x00005555555b03a9 in block_database_tests::block_database__index___only_genesis_candidate__height_available_invoker () at test/databases/block_database.cpp:96
#19 0x00007ffff78e1fbe in boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) () from /usr/local/lib/libboost_unit_test_framework.so.1.62.0full fileAs is the classic story in multithreaded libs - if i comment out any of other tests in the gist, the failing tests pass. 🤷
I thought I'd dig around a bit, but now it is time to just ask 🙂
jungly17:38
yeah, i was starting to think along those lines, there is a comment in block_database code about it too
jungly17:40
what puzzled me was that if the block_database instance was destroyed then all locks should have been properly unlocked and deleted..
bad assumption on my part.. so whats the best way to write multiple tests for block_database?
So is the exclusive lock. And you need to be careful about allowing test concurrency.
jungly17:44
yes.. i was thinking of that too.. i tried to create file stores for the various indexes as separate ones. I was using the name of the test in the file path for the index files.
You should avoid using file stores where possible. Use the fake from the blockchain tests.
jungly17:46
ah.. that's a great tip.. thanks. Didn't know we had fakes available for it.
I'll give that a look, might solve this problem.
evoskuil17:49
The critical sections in the database cannot be left unlocked, that’s not possible. You are only dealing with necessarily persistent file system locks.
I created a data_chunk database for blockchain tests. It should be possible therefore to use it with database tests.
jungly17:54
I'll try and take a look.. the file system locks probably is the issue, but if I make each test use a different set of index files, should the file system locks problem not go away?
Just kind of curious, cause I had the tests use different fs paths for each test, but still had this lock error
jungly18:15
OK. I switched to a mac to see how things go there and it turns out I can replicate the the error with a single test case. The test is here: https://gist.github.com/kulpreet/1c09e864ee52a2c247dc88119f5c4aac
And the error is
Running 1 test case... unknown location:0: fatal error: in "block_database_tests/block_database__store__only_genesis_no_index__height_not_available": boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >: boost: mutex lock failed in pthread_mutex_lock: Invalid argument test/databases/block_database.cpp:71: last checkpoint
I am trying to use LLDB, but it complains that the architecture is not not set
error: './test/libbitcoin-database-test' doesn't contain the architecture x86_64
Anyone using a mac and has been able to use lldb?Saturday, November 10, 2018
L
ldz21:57
On Linux libbitcoin uses ICU 55.1 with boost 1.62. Do you think it is possible to build a dependency (like ICU) with old toolset (e.g. vs 2010) and the rest with toolset for VS 2017? Can it be done? I don't think so but I have not enought experience. ICU 55.1 doesn't want to build with VS 2017 toolset. The default toolset for ICU 55.1 seems to be v100 (vs 2010). The ICU 61.1 build works perfectly with vs 2017, all tests are passed. I can imagine ICU upgrade will be necessary because of vs builds.
neillm22:01
I think it can be done if needed. @pmienk would know for sure. The issue would have to be verified for windows, not sure I understand what isn't working.
I mean the visual studio 2017 libbitcoin version.
"The ICU 61.1 build works perfectly with vs 2017," Just to be clear. I mean here only the build process of icu 61.1. I have not crated boost locale with icu yet.
Sunday, November 11, 2018
neillm01:49
I need to look into this some more as well. I ran into a case today building everything from scratch where gcc 8.2 doesn't seem to want to compile icu 55.1 version either. Not sure if it's an environmental fluke because I thought I had done that before, but I tried gcc-7 as well and it errored also. gcc-5 works as expected. It may just be time that an upgrade of icu is in order.
L
ldz13:05
@neillm gcc 7.3.0 wants to compile icu 55.1 (--build-boost --build-icu --with-icu). my main problem is actually to build icu 55.1 with vs 2017 (v141 compiler) :).
L
ldz14:06
I am not convinced it can be a big challenge. I think at first it is important to know the right icu version we can use to build with all vc++ platforms (vs 2013, 2015, 2017). If there is no such version, we have to use different versions depending on platform. After that the boost locale can be build with icu for the appropriate platform. At the end nuget packets can be prepared ant the end user (someone who is building on windows) will not notice any difference :). And of course it would be nice to have something like optional build (like secp256k1) without nuget, for transparency. LocaleWithIcu.import.props or something like that. That is what my naive imagination looks like. On the other side, boost locale with icu maintained by Sergey Shandar would make it easier for us. Anyway, it would be best if @pmienk could evaluate it.
L
ldz16:15
What does it mean? What is the purpose of keeping old names? Does it have something to do with backward compatibility or another reason?
Monday, November 12, 2018
vdeurzen12:32
So, I've doubled the amount of memory and cpu for the import of mainnet, but it still takes ~5 seconds per block and is not anywhere near the end yet ... anything I could tweak with settings here?
I've mostly followed this example:
https://github.com/libbitcoin/libbitcoin-node/blob/version3/data/bn.cfg
no local node unfortunately
neillm12:36
Yeah, v3 sync is slow. It often takes ~5-6 days for a mainnet sync from scratch. I'm currently doing one that's been that long already, nearly complete (well ~504K height), and it looks like 10-15 seconds per block.
vdeurzen12:36
machine has 4 cores (virtual) and 10 GB memory, but from my monitoring it seems it's barely using the CPU
alright, then I'm still within the expected time frame I guess
If you watch the CPU, you should see spikes. Mine shows often ~30% but then jumps to ~300% at times
vdeurzen12:39
yeah, it has very regular spikes, but low ones, hasn't gone over 80%, but I'm not sure whether that's the monitoring telling me it's 80% of 4 cores or 80% of 1 core in this case
evoskuil12:40
It just means that other factors are more limiting... it’s not able to continuously validate.
network?
I/O could be slow indeed
didn't expect it to have that much effect though
With low RAM paging costs increase.
How much output cache is configured?
vdeurzen12:43
@evoskuil whatever the default is for the output cache, I don't have that in the config file it seems
[log] debug_file = /var/chaindata/mainnet.debug.log error_file = /var/chaindata/mainnet.error.log archive_directory = /var/chaindata/archive.mainnet rotation_size = 10000000 minimum_free_space = 0 maximum_archive_size = 0 maximum_archive_files = 20 verbose = true [network] protocol_maximum = 70013 protocol_minimum = 31402 services = 9 invalid_services = 176 # The magic number for message headers, defaults to 3652501241 (use 118034699 for testnet). identifier = 3652501241 validate_checksum = false inbound_port = 8333 inbound_connections = 5 outbound_connections = 10 manual_attempt_limit = 0 connect_batch_size = 5 connect_timeout_seconds = 5 channel_handshake_seconds = 30 channel_heartbeat_minutes = 5 channel_inactivity_minutes = 30 channel_expiration_minutes = 60 channel_germination_seconds = 30 host_pool_capacity = 1000 hosts_file = /var/chaindata/mainnet.hosts.cache self = 0.0.0.0:0 seed = mainnet1.libbitcoin.net:8333 seed = mainnet2.libbitcoin.net:8333 seed = mainnet3.libbitcoin.net:8333 seed = mainnet4.libbitcoin.net:8333 [database] directory = /var/chaindata/mainnet flush_writes = true [blockchain] priority = true # A hash:height checkpoint, multiple entries allowed, defaults shown. checkpoint = 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0 checkpoint = 0000000069e244f73d78e8fd29ba2fd2ed618bd6fa2ee92559f542fdb26e7c1d:11111 checkpoint = 000000002dd5588a74784eaa7ab0507a18ad16a236e7b1ce69f00d7ddfb5d0a6:33333 checkpoint = 00000000001e1b4903550a0b96e9a9405c8a95f387162e4944e8d9fbe501cd6a:68555 checkpoint = 00000000006a49b14bcf27462068f1264c961f11fa2e0eddd2be0791e1d4124a:70567 checkpoint = 0000000000573993a3c9e41ce34471c079dcf5f52a0e824a81e7f953b8661a20:74000 checkpoint = 00000000000291ce28027faea320c8d2b054b2e0fe44a773f3eefb151d6bdc97:105000 checkpoint = 000000000000774a7f8a7a12dc906ddb9e17e75d684f15e00f8767f9e8f36553:118000 checkpoint = 00000000000005b12ffd4cd315cd34ffd4a594f430ac814c91184a0d42d2b0fe:134444 checkpoint = 000000000000033b512028abb90e1626d8b346fd0ed598ac0a3c371138dce2bd:140700 checkpoint = 000000000000099e61ea72015e79632f216fe6cb33d7899acb35b75c8303b763:168000 checkpoint = 000000000000059f452a5f7340de6682a977387c17010ff6e6c3bd83ca8b1317:193000 checkpoint = 000000000000048b95347e83192f69cf0366076336c639f9b7228e9ba171342e:210000 checkpoint = 00000000000001b4f4b433e81ee46494af945cf96014816a4e2370f11b23df4e:216116 checkpoint = 00000000000001c108384350f74090433e7fcf79a606b8e797f065b130575932:225430 checkpoint = 000000000000003887df1f29024b06fc2200b55f8af8f35453d7be294df2d214:250000 checkpoint = 0000000000000001ae8c72a0b0c301f67e3afca10e819efa9041e458e9bd7e40:279000 checkpoint = 00000000000000004d9b4ef50f0f9d686fd69db2e03af35a100370c64632a983:295000 [fork] easy_blocks = false retarget = true bip16 = true bip30 = true bip34 = true bip66 = true bip65 = true bip90 = true bip68 = true bip112 = true bip113 = true bip141 = true bip143 = true bip147 = true [node] block_latency_seconds = 300 notify_limit_hours = 24 byte_fee_satoshis = 1 sigop_fee_satoshis = 100 minimum_output_satoshis = 500 relay_transactions = true refresh_transactions = true
my config ^
from within the container using top I see it's got 6.6gb resident, so the monitoring seems off
neillm12:46
I think default should be ok. If it was too large, or if other processes are putting pressure on the system's RAM, it would have a bigger negative effect during sync
You have 10 outbound plus inbound connections. That’s too high for v3 sync, lots of redundancy.
I forget the default output cache, but with low ram it should be set to around 10000.
so the 10 outbound should be ~5 or fewer?
If you abort the process it will be corrupt.
vdeurzen12:52
I've hit a deadlock issue a couple of times that required a sigkill to get the process to stop 😕
evoskuil12:52
You can restore write flushing after sync. It doesn’t impact much for one block at a time.
(edited)Then you are stuck with two bad options. I haven’t been able to repro a deadlock in a useful debugger, but it apparently reproduces.
the default is 0 inbound connections, is that just for nodes trying to initiate a session or should it actually be higher?
You don’t want to hold on to very slow peers.
Once you are synced to refresh will happen, but prob not important for you.
trying those tweaks, thx!
neillm12:58
Disabling write flushing is a huge performance difference, btw. You get shutdown hangs on initial sync?
neillm12:59
When you say hang, how long are we talking about? Could it be the virtualized storage system doing a flush for an extended amount of time?
vdeurzen13:00
in that situation I had flushing on too and the code simply deadlocked somewhere while destructing the threadpool and launching something elsle, at least that is what seemed to be happening
it was stuck for many minutes until I had to kill it by hand to continue working
haven't found the issue that causes it yet
vdeurzen13:03
hence why I wanted to have flushing on for my main deployment to ensure I can snapshot the database and expect the process to stop within not too long a period of time
vdeurzen14:09
Maybe an odd question, but is there maybe a pre-downloaded database available anywhere? I realize that the whole trustless nature makes that a bit unlikely, but it would speed up our testing significantly 🙂
evoskuil14:14
Must be from same version and hash buckets config, and portability assumes std::hash consistency across builds, which is not guaranteed but generally consistent by compiler.
alright
neillm14:18
If you want to give it a shot and have a place to store a ~350GB database, I've got a v3 snapshot at 537036
how would we do the download? could we just wget / curl it from somewhere?
neillm14:37
Hrm, haven't thought about it. I'd prefer to push it somewhere if possible (ftp/scp). It's not on an externally accessible machine
vdeurzen14:38
hmm, then I'd have to find a publicly accessible place, not sure I'll be able to get that given our security requirements 😕
neillm14:40
Not sure it would work on my end though. I'll think about, but pushing somewhere would definitely be easiest 😉
vdeurzen15:17
Wondering though, just to understand, other than larger / more complicated blocks, why does the sync slow down so much near the end?
Blocks are 3 orders of magnitude larger.
The size of past history has negligible impact as all queries are constant time.
when does that change? around which block?
or is it gradual
evoskuil15:27
With low ram historical size has a greater impact due to paging. True linearity requires no paging.
so that will help 🙂
(edited)evoskuil15:30
Yeah, at low ram you can churn. Libbitcoin is designed to make best use of maximal resources, not to trade that off to be more optimal with low resourcing.
It's an impressive and very readable library so far, I'm very happy that it exists!
I forget.
anything I could do to help get it done faster?
or sooner rather
Resolve the shutdown hang.
(edited)I'll keep investigating 🙂
In this case I really don't need the outputs to be queryable, this particular service is just to track the state of the chain and to push notifications about transactions and new blocks to an upstream service, would it help if I turn off the utxo cache?
But you could skip validation by using a high checkpoint (trust).
That speeds things up a lot.
vdeurzen15:36
I would want validation mainly for the last blocks and to be certain the chain is a valid one ofc
vdeurzen15:37
Ok, so, I could go to a blockchain explorer find a block near the top and add that to the list?
at least for our testing phase that should be a nice extra speed bump, for production I'd probably stick with more conservative config, but still, interesting 🙂
Tuesday, November 13, 2018
M
Mlmikael15:08
Guys, what is the normal way you convert a script object into its binary correspondent, so that you can calculate its hash?
E.g. for generating a P2WSH pubkeyscript hash (which is OP_0 OP_PUSHDATA <32B hash of the script, right>)?
{ opcode::size, opcode::equalverify }
how do I compile this to a binary chunk
2 replies · Mlmikael, evoskuil · last Nov 14, 2018
M
Mlmikael18:29
@evoskuil using the script::to_data() method, obviously, duh. Thanks.
Now, the method has one argument which is "bool prefix", what's this, should it be true or false in a normal script serialization usecase?
How critical would the english dictionary change be with respect to the old (v3) version? Can it influence any important logic?
I mean the @Ebcrypto's change.
I think the electrum seed generation process is not testable directly because of ecdsa dependency. I atually have a running debug version and can fake the dependency. I could generate seed from numeric tests unlimited for you in all languages if that is what you need.
I guess SomerNight's tests are useless in our context.
neillm20:06
@ldz That's not what I need. I specifically need electrum compatibility. And yes, @Ebcrypto's change can break BIP39 and electrum compatibility. I'm not sure it's needed either.
neillm20:17
@ldz I also have electrum tests running here, but generating my own that match up is just taking my own word that they match up. While not necessary, I'd prefer they were incorporated so that we can show compatibility. BIP-39 is a spec with provable test vectors. Unfortunately, electrum is an implementation, so the closest to provable test vectors is what they provide, not us.
L
ldz22:58
@neillm Sorry, I don't quite follow you. I didn't talk about electrum test which you can run. At first, what kind of test data do you need? I guess you need some input (like entropy) and output (like seed). Next you would pass the provided input to your libbitcoin implementation and compare your ouput with the provided seed, right? I said indirectly, currently you cannot get the mentioned input/output from electrum test envionment because electrum has a strong dependency. In other words, there isn't a function where you could pass entropy input and get output (seed). What can be done? a) To force the electrum devs to change their architecture to break away from the dependency and make the design more unit test friendly. b). Make the electrum code temporary more unit test friendly and prepare the input yourself. The option b) was the one I suggested. I only looked superficially at the electum's implementation, so I can be wrong. If yes, please correct me :).
neillm23:07
Perhaps to you, but not to me 😉 I've got all of them integrated into our test suite so far, with 1 that still needs work.
L
ldz23:15
@neillm ok, I have looked only on the make_seed method. No idea about mnemonic_to_seed. I to not exclude that it makes sense. I will wait for your merge and look more closely :D.
L
ldz23:58
@NickSt: At first I thought the English dictionary would be wrong. But it was a mistake made by ebCrypto, luckily :).
@NickSt sorry I mean @neillm
Wednesday, November 14, 2018
Is there something in particular you're asking about?
neillm13:18
The serializer types appear to be written clearly. Is there a question about something in particular there, or just generally what they are?
neillm13:29
This may generally help too: https://github.com/libbitcoin/libbitcoin/wiki/Developer-Documentation (The sub-topics under that on the right side)
neillm13:31
If you find that other topics need documentation and want to take a stab at it, that's always helpful. Or if you just want to know more about something in particular, feel free to ask. Perhaps docs or comments are needed.
But also, given that it's a large C++ code base implementing something non-trivial, expect to spend a lot of time reading code 😉
👩🏾💻 1👀 1🤷🏾 1
Revive13:37
its not a lack of documentation really, i'm a new recruit in both c++ and coding bitcoin so i didn't know what safe and unsafe meant, but after seeing the code i kind of see its about threading and irrelevant to what i wanted to do anyways
neillm13:40
The safe/unsafe refers to whether bounds checking is enabled or not, where applicable (for numeric types).
For example, given a stream of bytes that are being deserialized, it's ideal to make sure that when a length is specified, that the stream contains at least that many bytes, etc (otherwise it's bad input, an error condition)
(edited)M
Mlmikael18:29
replied to a thread: view
@evoskuil using the script::to_data() method, obviously, duh. Thanks.
Now, the method has one argument which is "bool prefix", what's this, should it be true or false in a normal script serialization usecase?
L
ldz21:33
where the cs, uk and ru wordlists come from? is there something like official reference? I see here all libbitcoin's languages except cs, uk and ru https://github.com/bitcoin/bips/blob/master/bip-0039/bip-0039-wordlists.md
L
ldz21:40
Apr 1, 2017: adds cz, russian and ukrainian (libbitcoin). Jan 8 2018, there are some bugfixes at least at the ukrainian.txt. I think it is a bad idea to provide support for these lists, even in main.
*in master
neillm22:05
@ldz That's valid. Alternatively, we need someone to monitor them more closely to make sure we're in sync 😉
Thursday, November 15, 2018
M
Mlmikael11:06
A general question, you know Bitcoin's Base58 encoding has a logic whereby somehow zero padding bytes are wiped (in the base58 encoding those are translated to the character "1").
I'm reading both LibBitcoin's and Bitcoin Core's Base58 code and trying to grasp the role and utility.
So, P2PKH's prefix data byte is zero, right.
Is the point that if I would make a P2PKH address where the hash of the public key starts with zero bytes, then somehow they would be wiped?
Buuut then, I see both in LibBitcoin and in Bitcoin Core logics for handling multiple leading 1 characters.
Is what's going on that each leading 0x00 BYTE is translated to ONE "1" base58 CHARACTER, hence offering a kind of compression for these as normally one byte will be ~1.3 base58 characters??
https://github.com/libbitcoin/libbitcoin/blob/master/src/formats/base_58.cpp#L109
https://github.com/bitcoin/bitcoin/blob/master/src/base58.cpp#L43
evoskuil11:22
I believe the idea is merely to preserve the number as well as the character representation of the number (ie to preserve leading zeroes which are otherwise ignored by numeric encoding), since it’s encoded numerically, but it’s actually a byte array (non-numeric).
(edited)M
Mlmikael12:01
I'm starting to understand how it works - Bitcoin's "base58" is actually a combination of base256 (applied for leading zero bytes only) and base58 (applied for all subsequent bytes).
The theory seems to be that base58 encoding views the byte array as one big-endian big-integer number and then splices off base58characters by doing modulo 58.
and then when there's only 0 left, you encode those zero input bytes as "1" prefix characters - this is the theory.
The encoding awakens some questions with me, such as "isn't there a situation where you could get a non-prefix-0x00 byte to encode within base58 to a leading "1" base58 character?" - when you decode it, that "1" character would be decoded to a full byte, whereas it was really intended to describe a 58/256:th part of a byte.
M
Mlmikael12:12
anyhow, i kind of understood the encoding format now. the fact that the base used is not a power of 2 makes it much more complex than if it would have been..
@evoskuil yes but, certain bytes after the leading zeroes, will produce a "1" base58 character, right.
why does that not trash the decoding step?
"1" is not only a prefix character but also one of the encoding's 58 body data characters
M
Mlmikael12:14
playing with https://incoherency.co.uk/base58/ i can't get it to produce any such "1" character, but, i can feed the decode step with non-prefix "1":s. still trying to get my mind around it.
@evoskuil yeah. anyhow, the big wisdom is that all base58 bitcoin addresses regard a 1 + 20 (160bit) + 4 byte chunk right
so you take a supposed Bitcoin address through base58 deode and if the decode result is not 25B, it's not a P2PKH or P2SH Bitcoin address. yes?
evoskuil12:19
Since “1” is also the zeroth position in the base58 chars array the result of the pack would have to be zero to select it. This is precluded as a leading character by stripping the leading zero bytes. It can occur later in the encoding however.
3 replies · Mlmikael · last Nov 15, 2018
M
Mlmikael12:45
@evoskuil I still don't see why the base58 encoding could not produce a leading "1" character.
M
Mlmikael13:45
@evoskuil
https://files.esotericnonsense.com/public/base58.txt
I think I see now, base58 is analogous to decimal string serialization, and the "1" character is used as zero.
As you go through the input - which is viewed as a number - you're serializing in an order from right to left, and,
the definition of reaching the leftmost point is that your DIV 58 and MOD 58 suddenly give ZERO!
..at which point you STOP SERIALIZING
So the reason you don't get any leading "1" char as significant base58 data is exactly that, that a leading "1" means "serialization is over".
M
Mlmikael13:46
So it's like doing decimal serialization of the value "000002356" - when you did the DIV 10, MOD 10 work for 2356, what you have left is ZERO, which will give DIV 10 = 0 and MOD 10 = 0 and these will kick you out of the serialization loop.
M
Mlmikael12:45
replied to a thread: view
@evoskuil I still don't see why the base58 encoding could not produce a leading "1" character.
M
Mlmikael13:45
replied to a thread: view
@evoskuil
https://files.esotericnonsense.com/public/base58.txt
I think I see now, base58 is analogous to decimal string serialization, and the "1" character is used as zero.
As you go through the input - which is viewed as a number - you're serializing in an order from right to left, and,
the definition of reaching the leftmost point is that your DIV 58 and MOD 58 suddenly give ZERO!
..at which point you STOP SERIALIZING
So the reason you don't get any leading "1" char as significant base58 data is exactly that, that a leading "1" means "serialization is over".
M
Mlmikael13:46
replied to a thread: view
So it's like doing decimal serialization of the value "000002356" - when you did the DIV 10, MOD 10 work for 2356, what you have left is ZERO, which will give DIV 10 = 0 and MOD 10 = 0 and these will kick you out of the serialization loop.
vdeurzen16:06
How random is the 'pseudo_random_fill' function in version3, it's marked as deprecated, but I'm not sure what is recommended as its alternative?
just the mt19337 from c++11?
on linux
is it good enough as a random source for an HD key is the real question I'm asking
Any recommendations for a replacement?
neillm16:14
@vdeurzen /dev/random provides more entropy than /dev/urandom, but it's not as secure as dice rolls 🙂
That’s why we call it PR, it’s a warning to developers.
👍 1
neillm16:17
In other words, ignore all the advice on the internet that states to use /dev/urandom or whatever for cryptographic keys
Either that, or you're trusting some hardware to be random (i.e. hardware wallets, etc)
vdeurzen16:18
ok, so let's say we do dice rolls, how would I convert that to a seed readable by libbitcoin 😛 ?
neillm16:19
Any encoding you'd like (1-3 = bit 0, 4-6 = bit 1). Or get a 16 sided die and use hex values (quicker)
right, so for 256 bit seed I'd make 256 dice rolls and write down the result ... and then convert each 8 bits to a byte / uint8_t and put that in a data_chunk?
I'll need to get some of those automatic dice rolling cups to avoid rolling bias 😛
Like identity and trutlessness
vdeurzen16:35
It's going to be an important process, one we're still carefully designing, for the time being, I'm happy we're still only just testing 🙂
TRNG is an environmental sensor.
Most likely hardware to be compromised.
evoskuil17:38
Better to use off the shelf sensors like keyboard, mouse, camera, microphone with manually-crafted input.
(edited)7 replies · neillm, ldz · last Nov 15, 2018
Force push change into your own repo.
What would be optimal or feature-complete here, anything notable from the v3 world that's not in v4 yet?
But anything is possible.
evoskuil18:14
Optimal is much faster. Features not complete are varied, but include successful tx sync, milestones, websockets, payment indexation, compact block protocols, and block template and mempool message population, and need to patch a v4 (only) consensus bug. Also need to resolve mnemonic and vc++ normalization bugs (wallet code) and extend bx for segwit.
But the full chain sync does work (albeit with a consensus bug) - the higher performance you talked about that V4 will have due to header first download and additional parallellization, is it there now?
evoskuil18:30
Yes, that’s what I mean wrt tx. You will not encounter the consensus bug. Higher perf and full parallelization is there, but it is not optimal yet.
(edited)afk
neillm18:41
@ldz Maybe per-repo, try to consolidate them all into one commit (or squash them, etc). Each one kicks off a new build on travis-ci otherwise
I will stop this.
M
Mlmikael19:05
A question, both payment_address and stealth_address have a "valid_" value. When you typecast a payment_address or stealth_address object to boolean, you get a copy of that valid_ property.
It seems to simply say if the class has been successfully initialized, some of the failure cases in constructors - e.g. failed string deserialization - lead to using the () constructor, and unlike all other constructors, that one sets valid_ to false.
Is having a valid_ property an overall LibBitcoin convention?
evoskuil20:34
That’s a private field, used to implement the object’s bool() cast representing validity. Cast to bool is one convention used to determine if a give object construction is valid. This is necessary for conditionally-successful constructions since constructors don’t return a value. The other convention is for the object to expose an .is_valid() property.
The cast convention is also used in Boost, such as by shared_ptr<>. But sometimes we prefer to be more explicit.
L
ldz22:41
@neillm about #1060. I'm not as familiar with the subject as you, but in my opinion the lists should remain immutable. I believe it is better to provide exactly the same lists as the original one. "To avoid confusion and assumptions".
Friday, November 16, 2018
B
baiumbg08:58
Hi again guys, I’m having more block rejection issues on regtest. libbitcoin-server stops publishing blocks/transactions after it gets a block that it rejects with
(edited)Rejected block [block hash] from [127.0.0.1:28333] transaction inputs or outputs empty
I had to fetch the block using bitcoin-cli, since v3 of libbitcoin-explorer doesn’t have the fetch-block command. After this block gets rejected by libbitcoin-server, it gets stuck in an Ask [127.0.0.1:28333] for headers after [block hash] loop. This happens every night, while my regtest is running (I have a script that generates transactions and auto-mines blocks using bitcoind RPCs). From what I’ve seen, this always happens at a different block height (always below 500). Is this another segwit issue? Does bitcoind switch some bip on after a certain block height or # of transactions? Has anyone encountered this before?Here’s the block:
Untitled
{
"hash": "0d97107287998ea7fbda63955d67994a7e4d0cf9d694ec0a0c3441eb448079c7",
"confirmations": 194,
"strippedsize": 37668,
"size": 37704,
"weight": 150708,
"height": 432,
"version": 536870912,
"versionHex": "20000000",
"merkleroot": "757b4586230223702d70d02dc4c4959db4aa7610e59aa7e1addfd28cce6072d4",
"tx": [
"573e51baed0c4fe3b6715d4026549a9b790b3684dbf74b35355ec7170d6e8aee",
"9affdf83dbdad6bf526f49a58a610b9cc7d695e49671f9e2f5ceea222369bece",
"992f4bfca5b0147758c017a21dc7a2b145243a8b0d4f3dd3ae7ebb699d543374",
"2fa63bc5467e1cfe5d2b4a8d0cafab60fbecab12a74d289db5728551b61ea70f",
"db7a51380e04396e1f751b52953b79d6f42eec85d6e45f51025b9608d0dc3d2c",
"ff2c4393cc6884852e4151f99f8018609bc98ce7d740cbcf5accc5f5579d903c",
"b2b798e963482464cce0b5b574aa6f568d064b8042d577b01483b7fb4208c44a",
"12d3eeb7537f01e48fa0b9ee438d9979de1e133875c6448e7c6dc548e39b2250",
"b83478b462e7dcccd65a7ac1d40332d754a7bd53ece38978cbb63fc2cae02e67",
"8a4238ddba561deaff5e56ab728f26ef41fc6a9f29d5a87309dae1a5a16aa68f",
"cf6c6cfc2fca3ac262fb8756a1f212e5934bc8dca48bb204c8003a7f378ae190",
"88a45674ca8f2b19795459c3a90ecbb0e539d2a15c353ad068b50f9398942a9a",
"0d160487d00f00b1a73915125a2fdda83dbd46edb2a544925d4e42eecffb4e9b",
"917c84aaa60a132207e7c5ebd5217aa10551c35d0fd26650aec6e8b3de7f6ca3",
"80ed5409068e7a6f504fce9677ac78994e97e08061c434b55309562b3d13b8b6",
"be5ca4e07773dd37c60645e38d68c5f88d7d232def9d39d884f8d22feba25bc3",
"78f79bd313e7f5cde9ee858b4db816363fdc1dd5bc46a43987e11c6aac057ed3",
"929bc8401d1843743d95427de16b8412c895f553ef76bfd3ad7660e9bef4dfd6",
"17b0dbb308057ca31a23c91f637297c41622eb49c38a48f628cd52e5c09920e5",
"163decbfd6bd333ea11dec933ceaca9439683ab7faf31387f4392219517d83e5",
"cbb583fcd439293f8f57a79877e507325bcbc89f24131544ee8cdebb217859ea",
"2c043df6da6786be5164ead5e7b2c9427c0673e7fa21349c9d0b403dfde584f2",
"84466e0a29ee31655c8c69def1fbea8dc22e4bb5062118c04eb04ee62a42d6f2",
"47ed3a84b47d77c8eb0e2de17fbe0e594e6f758926b10dc063ba589ffe1846f3",
"45bf91483e65c89aa1a9dfeb4565ed8df4abb1bb61f6fb77ccaccf269e3bd6f6",
"22d764b5f8dcf2508f8fa9caf03b6cdf0a07cc20e7d34fdbd1970e3482114f05",
"825ad2da2b10662ee74b5f034b27d5dd76fcfed5f6d03a4f15d069aef255c40d",
"8616dd279490a85fbf8d3f5a881b6c52689e3ac23f60b69976d8049c3680201e",
"56d2d13e2db128fb53be519278ef40fb30a59b6fd39a8803737aa89bbca83034", full fileB
baiumbg11:25
The one that I encountered with bitcoind 0.16.0
https://libbitcoin.slack.com/archives/C6LH74ACQ/p1540802490025000
This is the breaking change in 0.16.0: https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.16.0.md#testing-changes
Segwit is now always active in regtest mode by default.Veselin Dobrev
Hey guys! We're having some trouble pairing a libbitcoin-server instance with a bitcoind one on regtest. Libbitcoin-server seems to be rejecting all block generated using the ``
generate` or `generatetoaddress`` RPCs. The configuration is more or less identical to the one in the libbitcoin-server wiki. We're using version 3.4 for libbitcoin-server and 0.17.0 for bitcoind. Are there any settings that have not been described on the wiki? Any ideas would be greatly appreciated.Running a 0.15.2 bitcoind atm.
(edited)B
baiumbg11:29
I don’t think so. @vdeurzen was going to post one, I think. Dunno if he’s gotten to that.
evoskuil11:32
There are no segwit problems on either mainnet or testnet, so this is presumably a regtest configuration issue (related to segwit) but not actually a problem with segwit.
(edited)My libbitcoin-server regtest configuration is more or less the same as in the wiki.
B
baiumbg11:59
I just went through the regtest configuration page in the wiki again and noticed the last sentence on the page. 432 is the exact block height that my regtest setup breaks. My configuration matches exactly the one in the wiki (except for the port and peers). I’m not sure I understand the last sentence, though.
M
Mlmikael14:12
right, so for 256 bit seed I'd make 256 dice rolls and write down the result ... and then convert each 8 bits to a byte / uint8_t and put that in a data_chunk?
Wait, throwing a dice gives you an integer 1-6. You should need to throw the dice ~120 times then, anyhow way less than 256? @vdeurzen
neillm14:32
That depends on your encoding of the dice rolls. The proposed example scheme used each roll as a single bit.
M
Mlmikael14:48
What's the idea of declaring main/testnet_p2kh/p2sh consts in payment_address.hpp but then writing the actual const values in payment_adress.cpp?
.hpp:
static const uint8_t mainnet_p2kh;
static const uint8_t mainnet_p2sh;
static const uint8_t testnet_p2kh;
static const uint8_t testnet_p2sh;
.cpp:
const uint8_t payment_address::mainnet_p2kh = 0x00;
const uint8_t payment_address::mainnet_p2sh = 0x05;
const uint8_t payment_address::testnet_p2kh = 0x6f;
const uint8_t payment_address::testnet_p2sh = 0xc4;
is there an inheritance point here that if someone would make a "litecoin_address" or something, then the child could override in this code model?
evoskuil16:57
No, you just provide the parameter, no need for derivation. The impl is not BTC specific.
jungly22:36
Hi @evoskuil, I have a small question about how asserts are used and how that interacts with NDEBUG flag enabled by default in the makefiles.
Consider the update_confirmation_state method in bitcoin_database.
https://github.com/libbitcoin/libbitcoin-database/blob/a0f64d2f904ec1e53e1b5b93eb8ede4102a8198f/src/databases/block_database.cpp#L361
If asserts are disabled, say NDEBUG is enabled, which is default in the libbitcoin Makefiles, then asserts will not get triggered. Even if an invariant fails, a block’s state will get updated.
This could result in a not yet validated block being marked as confirmed, if data_base incorrectly calls the index method with such args. In fact, I was doing so in my first rewrites of the block database tests.
I might be missing a detail somewhere. But I wanted to ask if it will make sense to convert the asserts to a simple condition and a return in case of the invariants failing. Or, do we expect production builds to disable NDEBUG? I have now set up my environment to always use --disable-ndebug so I can catch any similar bad situations from my code.
4 replies · evoskuil, jungly · last Nov 19, 2018
evoskuil13:28
Keep in mind that assertions are not error handling, so it matters not whether the assertions are enabled of disabled by default or otherwise.
evoskuil13:30
The database methods are public but not safe. libbitcoin-database is intended for use by libbitcoin-blockchain, which should provide safety to its public methods. This is a performance tradeoff, as making the database methods safe generally would have a material cost.
(edited)evoskuil13:32
Since tests are verified in both debug and ndebug environments unsafe calls will be caught by our CI environment.
(edited)jungly13:35
OK. That makes sense. I didn't realise that my local setup had ndebug enabled. I was thinking I was doing something bad with the locks and all this while I had a bad call to unindex that was passing on my machine but failing on travis in one of the test configs 🙂
Saturday, November 17, 2018
pmienk02:07
@jungly the function you've noted is indicated to be a private function. Presumably, any necessary guards against parameters passed by an external caller are validated elsewhere. Thus the asserts are sanity checks against programmer error instead of verification of input and they are intended to be optimized out.
pmienk02:17
though in this case there may be an error as I don't see the only caller validating the deserialized value is guaranteed to pass the check. to properly verify, the population of hash_table_ has to be examined
jungly07:49
@pmienk, you make a good point. However, as you note, during my test development I am able to pass arguments to index, a public function, such that an block that is not yet validated is added to the confirmed index.
The test passes on Ubuntu and Mac because ndebug is enabled by default. It was only with Ubuntu, clang, dynamic config on Travis that the test failed because that config had ndebug disabled.
Now I am using --disable-ndebug for all my development work.
I understand that maybe it is the responsibilty of data_base to not pass such bad arguments.
Just wanted to raise this, because the out of box build scripts enable ndebug, as one expects. Also, in case there's some more context and history to this that will improve my understanding.
2 replies · evoskuil, jungly · last Nov 19, 2018
evoskuil13:35
As you suggest, the public database methods are the intended use exposed only through the data_base class. Sometimes we need to make methods more public than we would like to to the limitations of c++. For example in c# these methods would be limited to the module.
(edited)jungly13:37
Yup. I also realised that to check conditions in ndebug mode we'd have to do some more processing each time. It is a hard trade off, but since database is to be used by our code, the current approach makes sense.
pmienk07:51
If it can be invoked without access to private methods, it should be guarded (as in without asserts). Thanks for the heads up.
(edited)jungly09:33
@pmienk Since I am in the middle of it, I don’t mind making a small PR just to address this for the index method. We can then review it if it makes sense to make the change.
Monday, November 19, 2018
B
baiumbg10:09
@evoskuil @vdeurzen I managed to get a libbitcoin-server instance paired with a bitcoind (0.17.0.1) instance by delaying the segwit bip9 activation. Bitcoind added a
vbparams command line arg in 0.16, which allows this (I ran it with -vbparams=segwit:999999999999:999999999999, which basically disables segwit). I still have no idea why segwit blocks are rejected by libbitcoin-server, though - it obviously works on testnet, I see no reason why it wouldn’t work on regtest. I played around with libbitcoin-server’s configuration, but to no avail.evoskuil10:15
Possibly segwit isn’t getting activated. Need both forks and service bit set, and activation checkpoints must be correct.
B
baiumbg13:10
Nevermind, that seemed to fix the block rejections, but transactions are still rejected. But yeah, I suspect it’s a mismatch between the activation checkpoints.
Is the witness service bit set in config?
15 replies · evoskuil, baiumbg · last Nov 22, 2018
evoskuil13:40
It has something to do with config. But if bitcoind has not activated segwit I don’t think it should be relaying segwit txs.
B
baiumbg19:16
Indeed, something weird is going on there. Anyhow, I'm probably going to need segwit activated in the near future, so it's best for me to leave it on and just figure out what's going wrong between libbitcoin-server and bitcoind. Any idea where my configs could be broken? My libbitcoin-server configuration is basically 1:1 with the one on the wiki.
B
baiumbg21:15
Also tested with bitcoind v0.14.2 and v0.15.1 (user lastcanal reported that these versions worked correctly in https://github.com/libbitcoin/libbitcoin-server/issues/404) and the result was the same - libbitcoin-server starts rejecting all blocks after 431.
(edited)evoskuil13:10
Dunno, but the config issue could be internal (not exposed to the config file) in v3. See chain_state constants.
B
baiumbg13:55
Are you referring to the ones in
include/bitcoin/bitcoin/constants.hpp? The bip9 checkpoints? The block hash and height seem correctly set to the genesis block 🤔evoskuil12:45
No, thank you! You should create a github incident so I can link the patch back to you.
evoskuil13:28
replied to a thread: view
Keep in mind that assertions are not error handling, so it matters not whether the assertions are enabled of disabled by default or otherwise.
evoskuil13:30
replied to a thread: view
The database methods are public but not safe. libbitcoin-database is intended for use by libbitcoin-blockchain, which should provide safety to its public methods. This is a performance tradeoff, as making the database methods safe generally would have a material cost.
(edited)evoskuil13:32
replied to a thread: view
Since tests are verified in both debug and ndebug environments unsafe calls will be caught by our CI environment.
(edited)evoskuil13:35
replied to a thread: view
As you suggest, the public database methods are the intended use exposed only through the data_base class. Sometimes we need to make methods more public than we would like to to the limitations of c++. For example in c# these methods would be limited to the module.
(edited)Tuesday, November 20, 2018
Mr Hash14:53
hey there, i’m having a play with libbitcoin but i’m not clear on a few things. first of all libbitcoin does not require bitcoind correct? secondly, building libbitcoin/libbitcoin builds the whole library stack?
neillm15:47
@Mr Hash Correct that bitcoind isn't required. But building libbitcoin/libbitcoin just installs the core/system library. I recommend building either libbitcoin-explorer or libbitcoin-server, depending on what you want to play with. Explorer is probably the best starting spot
(edited)neillm15:49
Building libbitcoin-explorer for example will give you access to all of the commands along the right side here: https://github.com/libbitcoin/libbitcoin-explorer/wiki
neillm15:51
You can do that from the command line using libbitcoin-explorer. Otherwise, libbitcoin-client will allow you to send txs and libbitcoin-system allows you to generate wallet addresses
neillm16:14
@Mr Hash I should also recommend that you build from the version3 branches only, which is production/stable
looks like it does
neillm16:29
Cool, at some point that will change from master, so it's always safest to checkout the version3 branch, which includes the version3 installer
(edited)Thursday, November 22, 2018
Mr Hash18:56
hey i ran the
(edited)bx fetchbalance command. seems like this does some lookup to some other servers correct? is that query private? and if i wanted to query a local node then i shuold install and run libbitcoin-server?evoskuil19:07
It uses your configured servers. See bx wiki for config. It can be made privately (see config), but the server can of course still see it. So if you want full privacy you must control your own server (bs).
Mr Hash19:09
ok btw https://github.com/libbitcoin/libbitcoin-explorer/blob/version3/data/bx.cfg has
server.block_url and server.transaction_url which seem to be unnecessaryanother question, are there an RPC entrypoints into the bx command or elsewhere in the client stack?
(edited)There is no "rpc"
There is a client-server interface that is message-based.
The "entry point" is the URL.
Mr Hash21:36
is there somewhere i can find a test body for a message, for example to retrieve a block header?
just to make sure i can contact the test server at tcp://testnet1.libbitcoin.net:19091
Friday, November 23, 2018
Mr Hash00:55
ok i made some progress. I can send
(edited)blockchain.fetch_block_height but all i get back in the response is the command name, not other response as far as i can seeMr Hash01:18
actually i’m just seeing if i can construct a message in javascript so i’m not using bx for this test
How are you making the zeromq call in javascript?
1444747
i’m using zeromq.js and constructing it in a similar way to the python example
i’m getting a response but it’s just the command name returned so far so i guess my message is incomplete
this particular command requires no third frame i think
the response is a single message yes? not multiple i’m assuming
evoskuil01:23
0 1 2 3 4 5 6 7
+-------+-------+
| code |height |
+-------+-------+
Mr Hash01:24
this is the one i’m calling https://github.com/libbitcoin/libbitcoin-server/wiki/Query-Service#blockchainfetch_last_height
same response tho
does the server return any errors or just ignores bad requests?
vdeurzen13:29
I'm looking into communicating the keys I've generated (through dice-or-some-other-entropy-creation) to another service, i.e. I'd like to use encryption, but I'm having some trouble finding a good library for this, any ideas about this?
neillm13:32
Not sure, but it's encryption that might secure a private key, depending on how the communication goes. BIP38 might also be an option. Depends on what you need
Mr Hash13:45
hey i’ve been building a message based on this command https://github.com/libbitcoin/libbitcoin-server/wiki/Query-Service#blockchainfetch_last_height
i’ve managed to get what looks like the correct response but it looks like the height is 8bytes with 4 bytes of padding or something
code is
<Buffer d2 04 00 00> followed by <Buffer 00 00 00 00 fb 0b 16 00>fb0b1600 is current block height in testnet so that’s good
Mr Hash13:49
so the message being sent is correct since the response is correct. i’m just dumping what the server is returning. I’m assuming the response is 8bytes for height as opposed to 4 as given in the docs?
wonder why i have extra 4 bytes
neillm13:52
If you're not using client to send requests/read responses, maybe it helps to look there? Otherwise, if you're using raw zmq, the protocol docs are even more important
neillm13:56
When it's ready. Don't mean to be snarky, but we were targeting end of year, but there are still items outstanding, though it's close. I'd guess ~3 months?
keep up the good work
has anyone written a js client for this yet? seems surprising that i’d be the first
(edited)i got requests working and subs also
neillm20:01
I don't know if there's an official policy, but if you wanted to contribute examples back in the libbitcoin-server/data project (similar to the existing python based ones), feel free to issue a pull request when things are ready.
Mr Hash20:02
sure no problem. i’m working on a project and it might require fleshing this out into a library as well
👍 1
so i’m curious about that database structure. there’s no wiki libbitcoin-database - have you got any links to more docs on that?
neillm20:05
I don't have links. The sources are the way to go, but are still a moving target in v4 as far as I know
evoskuil20:14
The database for v4 is pretty stable, but work ongoing in blockchain. No documentation yet. The primitives (templates) define the entire structure. Those are implemented by the database classes, and those are organized by data_base.
Saturday, November 24, 2018
Mr Hash19:08
i’m trying to run
bs on testnet and it got about half way i think, but my machine rebooted. Now i can’t restart bs and continue syncing. How do i find out what’s up? Logs just show 19:05:30.696904 ERROR [node] Failure starting blockchain. 19:05:30.697270 ERROR [server] Server failed to start with error, operation failed
neillm19:08
If the server wasn't stopped properly, the store is corrupt and will detect it on startup and fail to start
Unfortunately, you'll have to
rm -rf the blockchain store and start the sync overi was running it in a vm and then my mac crashed. not sure why
Sunday, November 25, 2018
Revive03:44
i'm trying to follow how libbitcoin does things project-wise. there's a src directory, and the headers of those are in <libbitcoin-src-dir>/include/* and there seems to be a header file generated for each project there as well?
pmienk04:12
include -> public headers
src -> lib
test -> unit tests
console -> application(s)
(edited)include\bitcoin -> root of all headers
include\bitcoin\<project> -> project specific headers
include\bitcoin\<project>.hpp -> header to link in all headers of the project
(edited)pmienk04:25
as standard as I've found. build artifacts along with the include/bitcoin/<project>.hpp header are generated via gsl in the libbitcoin-build project.
👍 1
Monday, November 26, 2018
Mr Hash16:50
just out of curiosity, what version of core consensus rules does libbitcoin v1 use?
(edited)neillm16:52
Hrm, not sure. I'd guess whatever was current at the time, although I never actually ran libbitcoin v1. I think by the time I started looking at it, v2 was already the main one supported
that’s fine just curious
Tuesday, November 27, 2018
It's just a guess 😉
Wednesday, November 28, 2018
evoskuil01:14
I think that’s a fair estimate at this point. I had planned to be home for all of Nov/Dec to wrap things up, but travel continues.
It’s encouraging that continuous parallel block download is now operating in master, but there are some important optimizations still outstanding, and tx sync is currently encountering some issue.
👍 1
Why are my last 2 lines causing it to hang?
The ones that are commented
jungly10:16
@narodnik just tried it on master and it worked for me on gcc/ubuntu. Are you linking to version3 or master?
I'm linking to master
jungly13:46
I did, yes, and it worked. was going to try it on a mac with clang, in case it is a platform issue
records.commit();
Mr Hash15:30
ok thanks. so v4 will support the old zeromq interface and also add a native websockets interface?
neillm15:32
It's not old 😉 But yes, both will be supported. The websocket interface internally transports over the zmq local interface
Thursday, November 29, 2018
evoskuil07:39
By electrum integration I mean that we are providing ElectrumX semantics over the zeromq interface. We will then adapt Electrum to interact directly with BS.
vdeurzen14:09
When the sync is up-to-date with the network, how often should / could it happen that the block subscription triggers out of order or misses block entirely? I understand that in a p2p network it's hard to get/give any guarantees, but I'm just interested in learning a bit about your experience, I will need to handle it one way or the other 🙂
If over the zeromq interface you can miss a message (eg because you are too slow in handling it) but you would be able to tell this from the sequence id.
vdeurzen14:30
Then I guess my monitoring missed a message rather than it being missed one layer lower. This is using the callback on the node directly rather than the zeromq interface
evoskuil14:32
Because it operates out of process and over public networks the zeromq interface can drop messages to ensure availability.
The c++ interface is cooperative.
cool, thx 🙂
vdeurzen15:00
In an attempt to run a backup I've on our mainnet node, I've stopped the node, but hit the termination deadlock issue. I'm no longer receiving any notifications of transactions or blocks, lsof shows the process still holds on to the database, but it's been almost 30 minutes since 'stop()' was called. Do you recon I could safely rsync the data even though it's still opened by the node?
I'm afraid if I hard-kill the process I might cause more harm 😕
I’ve never actually seen a deadlock on shutdown.
though, not anymore I see 0o
I'm not sure where it's deadlocking, but it simply does not terminate the process at any time 😕
vdeurzen15:57
I guess 🤔 , it's very odd behaviour though ... flushing is on at this point so it shouldn't take >30 minutes to shutdown the process?
If the locks are not present (flush/exclusive), I think you're ok to kill it without corruption ... I may be wrong though and consider everything other than a clean shutdown to be corrupt
process has not stopped yet ... but I guess it's at least flushed everything
neillm16:00
That's good. See if you can attach gdb, although that just shows you the current thread state, not necessarily a proof of deadlock
Also, is this master or v3?
v3
vdeurzen16:02
It's unfortunate it's hard to reproduce, process had been running many hours before this became an issue, previous shutdowns were much faster
neillm16:11
@vdeurzen I'm sure I've asked before, but are you running Ubuntu and does the build disable shared linkage?
vdeurzen16:15
I believe only libicu is still linked as a shared lib, everything else is statically linked
actually, no in my production image that too is statically linked in
and indeed, ubuntu bionic (in docker)
neillm16:18
Ok, good deal. It might be worth trying something other than ubuntu, if you have a choice. There was a reported hang on ubuntu with shared linkage in boost that I was able to reproduce in a VM, though I was unable to determine why it was happening. Static linkage did not present the hang.
I realize that's not a great solution though 😉
That particular hang was right on/after startup, not shutdown, but also did not allow a proper shutdown
vdeurzen16:32
could just be a particular build of boost that fails there, or even the slightest timing different 😛
Friday, November 30, 2018
evoskuil09:54
There is a documented start-stop race (in node or server), so you may have hit that. I’ve never seen a stop deadlock on Windows.
James C.18:49
Hi @evoskuil, I am reading through the tx_database but have a couple questions:
How is it used during transaction validation?
—
(edited)tx_database::get_output fetches any stored output, and queries an in-memory unspent_output cache first, but it doesn’t return whether the output is spent or not, does it?
— Each tx output in the database can be marked with a spender_height, but I can’t find where that value is read anywhere.
So tx_database::get_output seems to be optimised for unspent outputs, but I can’t see how it tells the caller whether the output is indeed spent/unspent. What call should I use for validating whether prevout is valid?The
spend_database could provide that information, but it doesn’t have the unspent_outputs cache optimisation … so that’s confusing me a little.evoskuil20:10
The spend database used to be used for this, but it is now only for the spender client-server query. My plan is to remove it entirely, computing the spending input from the spender height alone.
The spender height is read by the output non-wire deserializer (in the output object).
This determines spentness during tx population.
The height is required so that spentness relative to the fork height is computable.
evoskuil20:15
get_output is used to populate the previous output of an input, and is optimized over input reading because prevout population is validation performance critical.
Yes.
skaht23:57
Is there a recommended bx command to explicitly set an output locking script, e.g., an OP_RETURN?
“return [486170707920373774682042697274686461792053696c76657220466f780a]”