Bitcoin full node and query server
July 2018
Archive of the libbitcoin Slack #general channel.
Sunday, July 1, 2018
N
narodnik08:49
[kk@kk ~]$ rm -fr usr/share/bs/blockchain/ [kk@kk ~]$ ls usr/share/bs/ [kk@kk ~]$ cat usr/etc/bs.cfg [database] directory = /home/kk/usr/share/bs/blockchain/ [log] archive_directory = /home/kk/usr/log/bs/archive/ debug_file = /home/kk/usr/log/bs/debug.log error_file = /home/kk/usr/log/bs/error.log [network] hosts_file = /home/kk/usr/share/bs/hosts.cache [kk@kk ~]$ bs --config usr/etc/bs.cfg --initchain 08:47:37.768798 INFO [server] ================= startup ================== 08:47:37.768834 WARNING [server] ================= startup ================== 08:47:37.768902 ERROR [server] ================= startup ================== 08:47:37.768926 FATAL [server] ================= startup ================== 08:47:37.768945 INFO [server] Using config file: "usr/etc/bs.cfg" 08:47:37.769024 ERROR [server] Failed because the directory "/home/kk/usr/share/bs/blockchain/" already exists.
what am I doing wrong here?
I had to create it, then manually move it to the path
neillm13:15
@narodnik Not sure, I don't see anything wrong there. So you created it somewhere else and then moved it there because it couldn't create it where you wanted it?
N
narodnik14:35
yeah exactly what it says, I had to just do --initchain without the config, then move the directory to the correct path
oh yes it does... well that's strange
Monday, July 2, 2018
haarts07:07
@neillm mine are still running on my server , I think it is safe to say this Just Works. I'm now running the scripts on my laptop
Let us see!
Z
zoranc09:09
running bitcoin-server for 4 days now and still at block 370'000. am i doing something wrong or just a little impatient. a week or so seems to be needed for a full sync...
neillm12:00
@haarts Not that I know of using zmq, but the p2p protocol has a version handshake, so that would probably be the best way.
It's not very urgent but it would be nice to report the version in the electrum UI
@evoskuil that feature request should be recorded on libbitcoin-server or libbitcoin?
neillm12:39
Sure, I suggested it's the best way to get the version number (for now). I'd prefer we had more rich info available. I don't see the issue though.
Tuesday, July 3, 2018
haarts07:12
@neillm I'm not sure what was up last Friday. Yesterday scripts both kept working on my laptop.
The only thing I can think of is that I was ignorant enough not to corrolate putting my laptop to sleep and the script stopping.
That's a case I do need to handle but that should not be that hard.
haarts09:36
how would one fetch many headers from a libbitcoin server? one by one with 'fetch_block_header'?
neillm11:52
@haarts Ok. My subscribers are still running since the other day. The longer running one being the original one (without the keepalive).
haarts13:31
What kind of address does the Query Service 'subscribe.address' (new in v3) call accept? I'm confused wrt the p2sh/scripthash/'regular' address.
haarts13:33
I think I don't know what I mean too... You have these options I believe: p2sh/p2pk/p2pkh
anything would work for the subscribe.address?
neillm13:35
The format of the address hashes looks different depending on the type, but they are all valid bitcoin addresses
haarts13:36
when having an address there one needs to know if it is a script hash or publickey hash
(edited)neillm13:38
If you're given the script and need to determine how to represent the payment address, you need to know how to interpret it. But given the address, it's not possible to tell what it was the hash of (other than basic differences like p2sh prefix, etc)
evoskuil13:42
The caller needs to know the address is either p2sh or p2pkh, and v3 doesn’t support other types (ie nonstandard and witness). V4 will support all payment indexing using the Electrum model. In that case the caller must have or create (and hash) the script.
@evoskuil thanks for clarifying
Wednesday, July 4, 2018
@evoskuil / @neillm , a Q, what's going on in https://github.com/libbitcoin/libbitcoin-explorer/wiki/How-to-Encrypt-a-Wallet-Seed ?
you generate a HD wallet, namely this one:
xprv9s21ZrQH143K4EdfsuwWHdxaGgcwGkTB1f71fbzJNfJma1Xga5XE3LqYxXKkwJJLevsp16iDRyk35MwvmKEEyyqLkHQVziTNs6VtPr1xGM8
then you password-encrypt it, then password-decrypt it, and out of the decryption you get this HD wallet:
xprv9s21ZrQH143K2uNeWu3crjqwic1ocvMMwRSypQyMDTk4yQedZv8zkBVUeq2gztk2HQCAvqNLhUfcHhbD1RGFQ1TTqDSWfTLW4qkxsPMdjNG
these are different, because as you see one goes "K4Ed.." and the othe rone "K2uNeW..". this means.. bug or what?
neillm15:12
@Mlmikael The example pointed out is generating the compressed form. To get the command to look correct, add the -u option to the ec-to-ek command
Actually, nevermind, the command appears correct as-is and the example looks to have a typo
Try it out and verify, and I'll update the doc
It works here
M
Mlmikael15:42
a question, how do you normally figure that a Bitcoin node is fully synced?
I guess what you can do is download the block header tree, and, you assume you got all the block headers, and then you can track the download of block data.
M
Mlmikael15:43
@evoskuil neat, right. normally miners' clocks should be fine. your local clock could be messed.
M
Mlmikael15:44
@evoskuil you could apply some rule like that, you wait for more block headers from the network until the time matches or a particular amount of time passed
M
Mlmikael15:45
(top block header's time is within interval no older than 60mins behind your local time)
oh what do you mean?
M
Mlmikael15:46
yes i see what you mean in the sense that the Bitcoin network is totally asynchronous; there can be network segmentation and any vile condition that would hold blocks away from your sight, right.
(URL to the consensus rules page you mentioned.)
skaht15:51
Notice the following works fine:
(edited)% bx seed -b 256
9acfd31c269f5314505faf6c2a3d1aa7d679a7db1fe135ae06ff0334fd432de8
% echo 9acfd31c269f5314505faf6c2a3d1aa7d679a7db1fe135ae06ff0334fd432de8 | bx ec-to-ek "my passphrase"
6PYT9cmibbGwWUgmype9KQpoBpvgZnnqzTfcoBXZ4ENu8tRqR5y4K9dqZz
% echo 6PYT9cmibbGwWUgmype9KQpoBpvgZnnqzTfcoBXZ4ENu8tRqR5y4K9dqZz | bx ek-to-ec "my passphrase"
9acfd31c269f5314505faf6c2a3d1aa7d679a7db1fe135ae06ff0334fd432de8
Now comes the issue:
% bx seed -b 256 | bx ec-to-ek "my passphrase"
6PYWUL4BgdFMz2AME2UhDqLpTZy4yKd1gDBaMXWxc85n1KK2ZNoNKz9b6B
% echo 6PYWUL4BgdFMz2AME2UhDqLpTZy4yKd1gDBaMXWxc85n1KK2ZNoNKz9b6B | bx ek-to-ec "my passphrase"
b570afec3135f807be67109b60857a540fc9d8215b3cbcef645080c1b173754aM
Mlmikael15:52
@evoskuil i mean, syncing the Bitcoin blockchain is about finding the longest chain of work, so if you ask your peers for headers, and they give them, and you have all blocks for the longest chain as reported in the header tree you got, then you're up to date, no?
That is what enforces reasonable time.
else if (!is_valid_timestamp())
return error::futuristic_timestamp;
skaht15:56
Suggest that the example uses an “echo” instead of a “seed” to make the example repeatable.
M
Mlmikael15:56
wait, so, syncing is configured to not accept blocks that are more than 2 hours into the future??
but that can easily be the case if your local timezone is off, there should be tons of users with broken tz settings
(timestamp_future_seconds)
also the Bitcoin protocol has a year 2038 problem??
/// BUGBUG: bitcoin 32bit unix time: en.wikipedia.org/wiki/Year_2038_problem
Those nodes are offline WRT Bitcoin.
M
Mlmikael16:31
@evoskuil the logic in bc::chain::header is to accept blocks that are not-newer than 2 hours into the future, right . this is a condition that a bad network connection could never produce. what's the point here?
It limits clock skew, necessary for difficulty targeting.
skaht23:42
@neillm Yes I can. Totally forgot
bx seed -b 256 returns a new random number when used.👍 1
Thursday, July 5, 2018
M
Mlmikael08:57
A question about BIP 38, implemented as ec<->ek in libbitcoin - the BIP (https://github.com/bitcoin/bips/blob/master/bip-0038.mediawiki) describes that it was made to help make Bitcoin paper wallets.
What BIP 38 is used for i practice now as i understand it, though, is general purpose password protection of individual Bitcoin private keys as well as of HD wallet private keys, right?
So the benefit of BIP 38 is that it gives you a checksummed string representation form.. are there any more benefits or caveats within BIP 38 for this general purpose use?
lukejr writes "BIP 38 is recommended NOT to be used".
M
Mlmikael10:21
https://github.com/libbitcoin/libbitcoin-explorer/wiki/bx-fetch-balance describes the "uncorrelated receipt" situation, which must always be handled. I don't really get the point from the description here. Could you please explain to me what's the point here?
(Also just for reference, Bitcoin Core provides a minimum confirmations setting here, https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list )
neillm11:45
@Mlmikael Re: BIP38 -- https://github.com/libbitcoin/libbitcoin/wiki/BIP38-Security-Considerations
@Mlmikael The balance page describes that if the server hasn't indexed a record required to compute the balance, it will be known (and checked for) because a sentinel value is used. The example of how that could happen is that the user tells the server to start indexing at a certain height, which is greater than the record being retrieved
M
Mlmikael12:24
https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki#Node_serialization writes "there's no point in using a special version magic described in section "Serialization format" of BIP32", what is meant here?? i can't find any mentioning of magic in that section (https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#Serialization_format). In all cases it's not the "Bitcoin seed" constant they mean, or ???
@neillm re the balance page - having LibBitcoin start indexing at a block above the genesis block is extremely uncommon isn't it, or?
@neillm re BIP 38 security, the page you referenced discusses an extremely specific application of BIP 38, relative to my usecase (which I think is also the common LibBitcoin usecase) which is simply that you want password phrase encryption of a Bitcoin private key.
for that usecase (password protect a private key), is BIP 38 solid?
ah, do you have any thoughts about what's a good entropy for HD wallet seeds?
someone who should know suggested that due to how the HD wallets and Bitcoin crypto are designed - Bitcoin's EC are internally 128bits security and HD wallets enforce a 256bit key bandwidth - then for this reason more than 256bit HD wallet entropy is always waste, he suggested.
Would you agree?
Do you think 256bit or 128bit or 512bit is more appropriate, any difference in usefulness?
M
Mlmikael16:00
@evoskuil @neillm any ideas?
also
within HD wallets, if you have a public derivation - hardened or non-hardened - then you CAN make public derivations of it, right??
M
Mlmikael16:05
@neillm https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#Child_key_derivation_CKD_functions says for "Public parent key → public child key" that "It is only defined for non-hardened child keys.". aha that is a quote. hm!
is the BIP telling us that we can't???
1 reply · bigphil · last Jul 10, 2018
bigphil07:31
It depends on what you want to do. If you make non-hardened public parent, you can create children w/o priv key. Making the key hardened prevents scenarios in which the extended private key can be discovered. There’s more here: https://bitcoin.org/en/developer-guide#hardened-keys
bitcoin.org
so you can only make a non-hardened public derivation, from a hardened parent public key.
that is not a problem.
a guy told me that Bitcoin Core's HD wallet deviates from the BIP 44 spec, in that apparently Bitcoin Core's change and address_index levels are hardened also (BIP 44 specifies them to be not hardened).
Saturday, July 7, 2018
neillm01:40
@pmienk @skaht has identified an OS X issue when building zmq on his system. It fails to build the documentation properly (configure auto detects if asciidoc or xmlto are installed and tries to build it if so). I noticed that none of our CI matrix builds appear to build zmq (though I could have sworn we had one, because I recall it not detecting asciidoc/xmlto and skipping the documentation build attempt when I first looked into this). We've found that using
--with-docs=no
to the zmq build works around the problem, but what I'm wondering is if that's acceptable, as opposed to determining why the documentation is having problems being built on his system (could be environmental, could be a broken zmq tarball, not sure). On my GNU/Linux system, zmq can be built properly (with docs), but regardless, libbitcoin doesn't require the zmq documentation, so I'm not sure if (sometimes) building it is desired when in fact it can cause failures on some systems. Thoughts?
skaht01:48
The approach above works, but
--with-docs=yes
, the latest issue is with
asciidoc.conf not being found on OSX. I suspect this doc is portable, can someone ship me the one associated with a successful build?
pmienk05:22
Sounds like a question to pose to zmq. I wouldn't advise a change to install.sh, as someone could intend to install docs. At best, faq/readme note. I'm not in the know regarding asciidoc, haven't touched it in awhile.
👍 1
Monday, July 9, 2018
Fil08:44
I've been in Lisbon for the building on bitcoin conference but now I'm back in Zug. @Mlmikael you're welcome to drop by our office the next time you're in town 😉
Mlmikael
@Fil I was in Zug just recently. :)) You there?
Ledger though it’s a good idea to implement their own C++ bitcoin library instead of using libbitcoin.
Strong NIH :S
evoskuil15:55
They probably don’t like the fact that I broke their security features in three successive versions.
evoskuil16:02
Long history, can be traced from here: https://twitter.com/evoskuil/status/976352775058542592?s=21
👍 3
Tuesday, July 10, 2018
skaht15:50
When can one start to experiment with libbitcoin’s Confidential Transaction (CT) tech on testnet and mainnet and what is the version that will support CT capabilities?
neillm15:51
@skaht Some of the pre-reqs are present in libbitcoin, but CT is not supported in Bitcoin yet (testnet or mainnet), and so also not in libbitcoin.
skaht15:54
Are we waiting on a Bitcoin hard or soft fork to support the implementation’s Ring CT tech?
neillm15:56
I haven't followed if there's a concrete plan/timeline to add this into Bitcoin just yet. Perhaps @narodnik knows?
evoskuil16:20
A CT was demonstrated on main net last year by Maxwell I believe. Look it up on bitcoin-dev.
With schnorr we can create PoS sidechains, then add zk transactions or whatever system we want to Bitcoin.
skaht20:22
@narodnik So will the Bitcoin-core team’s killer Schnorr Signatures also be complemented with Ring CT support in the same release?
skaht20:46
The
July 2018 bitcoin-dev Archive
references
(edited)https://github.com/sipa/bips/blob/bip-schnorr/bip-schnorr.mediawiki. I assume this is what is being talked about.Wednesday, July 11, 2018
the Liquid federated sidechain by Blockstream is the only instance i know of that uses CT
ct is cool, but still too radical for consensus in bitcoin. there's an inherit risk of inflation behind the scenes of homomorphic commitments by quantum computing. i dont know if these are valid assessments because my knowledge of crypto definitely ends before quantum math begins
https://web.stanford.edu/~buenz/pubs/bulletproofs.pdf (4.6) "An adversary that can break the binding property of the commitment scheme or the soundness of the proof system can generate coins out of thin air and thus create uncontrolled but undetectable inflation rendering the currency useless..."
evoskuil15:21
That context is a new coin. Use of CT in Bitcoin (as has already been done) cannot result in inflation (under the breaking assumption).
skaht17:14
@Revive - If CT stays on either mainnet or testnet, there should be absolutely no inflation other than that prescribed by Bitcoin consensus rules. Homomorphic encryption properties of Ring CT, see https://www.elementsproject.org/elements/confidential-transactions/investigation.html, also ensures no extra inflation is induced. From a quick examination of https://www.elementsproject.org/elements/confidential-transactions/, it appears a new address and transaction type are needed to on chain support of CT. I’m guessing https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki nVersion might have to be incremented to support a Ring CT capability.
(edited)evoskuil17:37
Any hard fork makes it a new coin and anything becomes possible. But using CT in Bitcoin creates no new risk, as the inflation potential would obviously already be present in Bitcoin.
Revive17:51
Aah, yes. I suppose I mean using CT to verify tx amounts, which would only be possible through a new coin. What's an example of using CT with bitcoin otherwise? A sidechain i suppose, like Liquid
“Computational Privacy and Unconditional Soundness”
skaht18:07
Will be gleaning through:
1. 2017-April.txt,
2. 2017-May.txt,
3. 2017-September.txt
4. 2017-November.txt
5. 2018-March.txt
(edited)skaht18:38
Range proofs associated with Pederson Commitments at time_continue=2820 ensure no cyclic addition occurs so there is no induced inflation.
A
Alex23:10
@evoskuil I know you're proud of libbitcoin's memory mapped database. A Rust implementation of this layer has come to my attention and claims high performance. Have you heard of it?
https://bitcrust.org/blog-spend-tree
bitcrust.org
BITCRUST, Second generation bitcoin software | Fast parallel block validation without UTXO-index
Thursday, July 12, 2018
evoskuil00:00
I’m familiar from a discussion we had on bitcoin-dev when it was announced. The parallel is validation, not storage. It’s a layer for bitcoind as I recall. The parallelism is silly as it only applies to concurrently-received sibling blocks that presumably result in equal difficulty chains.
evoskuil00:07
It achieves some efficiency over bitcoind by pre-validating more. We prevalidate everything possible.
A
Alex00:48
I don't doubt that libbitcoin is superior, but it would take me too long to do the comparison myself. The blog says it is a "novel storage engine," mentions "memory map pages," and that "transactions are stored immediately when they are script-verified, but still unconfirmed" so it sounds like they might be trying to recreate what you have already done.
A
Alex02:13
Thanks! I saw your huge post there. I will wait for bitcrust to post performance numbers before revisiting this.
fpelliccioni12:11
"... some people feel uncomfortable with conclusions that have not been confirmed in practice." - Edsger W. Dijkstra.
Fil15:11
Guys, I wanted to share some ideas on how to work with compiler warnings. In my opinion compiler warnings are very helpful most of the time. My impression is that you feel the same given the recent commits where you are fixing them.
The only way to prevent warnings from slipping through is by using the
(edited)-Werror switch. Else, they will keep coming back; especially if there are existing warnings that make it hard to spot new ones in the noise.
There is a workflow to softly transition to using -Werror effectively:
1. enable -Werror switch
2. disable errors for warnings that already exist using -Wno-error=$WARNING, e.g. -Wno-error=reorder (this doesn't disable the warning; it just makes it a non-error)
3. Then, over time, remove the -Wno-error=$WARNING switches one by one and fix the warnings alongside. This is a bit trickier in projects where the build flags are not maintained in the same repo as the project itself but still doable I think.
I'm aware that sometimes warnings can be false positives but it's possible to selectively disable them using #pragma statements alongside a comment that explains why a warning is being a false positive. Pragmas are supported by all relevant compilers.
There is one drawback when using -Werror: in refactorings where the code is changed heavily it can take some additional effort to make it compile (e.g. unused variables, etc.). I think that this annoyance is far outweighed by the benefit of getting helpful feedback from the complier and by preventing warnings from slipping back into the code base.
Last but not least, there's the possibility that some warnings are not emitted by all compilers and therefore someone might accidentally break the build. However, it's the whole point of CI to prevent that from happening. If one warning behaves very inconsistently across compilers then it can be made a no-error warning as described above.
What do you guys think? 🙂pmienk15:22
I think that as long as we allow for dependency warnings to surface, Werror is inappropriate.
Local runs with --enable-isystem and Werror parameterized by cflags is possible before a clean build is possible on travis
pmienk15:38
I agree that the the goal of ci running with Werror given the dependency issue suppression though.
Fil15:50
I remember working on a project where we managed to turn off dependency warnings. Or are the dependency warnings turned on intentionally?
pmienk16:59
I'm unaware of a universal approach in use. pkg-config provides a means to work around default -I include, which I've employees with the enable-isystem configuration option. Additionally those includes communicated by m4 discovery have been substituted via that option. However their are limits to the means for that substitution which is one of the reasons why it's surfaced as an option.
evoskuil18:47
@James C. is making a dent generally, and the performance numbers are definitely coming with the v4 release, but that's a bit premature presently. We also added a configurable statsd sink to the log system and will eventually emit performance events through that.
Working on getting @James C. on full time, and he'll be joining me in Beijing in a couple of weeks, giving his own presentation on libbitcoin.
👍 3
Friday, July 13, 2018
There needs to be a method or upgrade path for changes that can be migrated into the dependent APIs. You mentioned test suites and CI. Usually the biggest problem is huge compile errors.
Fil07:23
@pmienk thanks for the explanation. I wasn't aware that such an effort is needed to make this work with autotools. My previous experience is with CMake. There, it's as simple as using
include_directories(SYSTEM ) and it does the right thing.evoskuil07:34
@narodnik I'm not sure I understand your question. Are you referring to language bindings, such as SWIG?
We want to update those, and assign someone to keep them updated / be responsible for maintaining them
evoskuil08:01
@narodnik You can do that at any time as contrib libs, but I can't imagine supporting manually generated and maintained bindings. We have an ongoing effort to provide generation.
N
narodnik08:07
Yes but you said they should be generated. Is there a tool to do this, because last time I made them manually (by hand), but it was difficult to keep up with master changes. Currently we're thinking to base it against some fixed revision, then update it for v4 when finished.
evoskuil08:07
The reason I would only support generated bindings is because of that issue. Unless generated they will never be consistent.
The tool to do this is SWIG.
@Victor is working on this, and has experience with SWIG.
N
narodnik08:09
Oh really, where is his work? I used to work with SWIG several years ago making Python bindings for C++.
evoskuil08:09
Libbitcoin is a C++ library, so it will always be most straightforward to use in C++. SWIG generates bindings in dozens of languages.
You should contact @Victor for an update. I suspect it is in his repo.
N
narodnik08:11
ok thank you, my experience with swig is that it has important C++ support missing but maybe that's changed now
evoskuil08:11
I extended libbitcoin-build to generate and invoke SWIG code to generate and compile SWIG bindings. The issue is that some features of C++ that we use are not suported by SWIG. He has been working through those.
N
narodnik08:14
with the manual bindings, you make a c wrapper that is loaded and used automatically by Python dynamically. There is no generation step. It loads the symbols directly from the shared library.
To define operators or your own methods
C is much easier to work with in Python, than C++
Nothing to do with C
Python is much easier to work with in Python.
evoskuil08:24
These are the language bindings we get from SWIG generation: http://www.swig.org/compat.html
N
narodnik08:25
Well they need to be supported, so you would have a dedicated bindings maintainer which is a big work for a large project
evoskuil08:27
The bindings maintainer would be the code generator, just as it is now for all of the VS and Autotools project artifacts (soon CMake and XCode as well).
(edited)👍 2
That takes a certain amount of time to maintain, but that is trivial by comparison to manual and is perfectly reliable where manual is never reliable.
I think you underestimate how complicated SWIG can get in certain corner cases, where there is no solution... but maybe that was a few years ago and things have changed now.
Once the remaining issues are resolved it will simply be CI build failure that guards against breaks.
evoskuil08:48
If you find a better generator I’m all for it. Even a one-off like you just posted. But someone has to spend the time to get the builds working with it. Then we can integrate it into libbitcoin-build and CI.
Saturday, July 14, 2018
M
Mlmikael14:54
i'm coming by Helsinki, any /Lib/Bitcoiners there who are up for coffee/beer let me know :))
Sunday, July 15, 2018
neillm14:35
Mainnet/testnet 1 and 2 are down. Will ping when back up. Sorry for the unexpected interruption :-(
neillm16:53
Thanks for the link, was just looking it up. Servers are coming back up momentarily (mainnet2/testnet2 is up already)
Monday, July 16, 2018
M
Mlmikael21:59
wait, who is it again who is implementing an SPV wallet in LibBitcoin?
electr walet is doing it if i got it right
feathercoin is also doing the work to fully generalize libbitcoin config for alts.
so the electrum integration will be useful for bitcoin, cash, litecoin, and feathercoin - as well as all testnets
(edited)Tuesday, July 17, 2018
is anyone implementing an SPV module atop libbitcoin? i think someone said
by the way -
how does the system local time setting effect LibBitcoin's behavior?
(the local time affects block validation in a way that I didn't understand why - if a block is timestamped as >2h into the future, libbitcoin won't process it.
i'd trust the system time on server nodes only - there's lots of junk computers out there that can have system time reset to 1970 on every boot etc.)
yeah apparently Bitcoin Core does like this too (someone writes about core, "yes it should be correct to within 2 hours of network time (mean time of last 6 blocks i believe) or else core will not be able to validate blocks")
on any non-server deployment i do, i think i'd like that to be configurable.
there should be computers with incorrect timezone setting making them off by +- 14 hours too
Bitcoin Core is made to run on computers that have some kind of rational settings. I guess for random embedded or smartphone deployments, assuming the system time is good makes less sense.
the Bitcoin network cares about wall clock time because it's used for difficulty computations, https://en.bitcoin.it/wiki/Block_timestamp
arrrr, the page writes " "Network-adjusted time" is the median of the timestamps returned by all nodes connected to you. ", meaning you actually need to share your local time with ALL Bitcoin peers ..
what if you report say 14 hours off, i'm in Fiji and my system thinks local time it's UTC but actually it's in Fiji time
someone writes "it also says that network adjust time cannot bemore than 70 minutes off from local time so your messed up clock won't affect anyone else."
ok neat. yeah so for less-maintained shit-nodes in the wild, reporting time = 0 to peers, and ignoring the time validation rule for blocks, is fine. good.
evoskuil08:05
you don't report your time, you mine a time into blocks. if your block's time is more than two hours ahead, other nodes will reject it.
M
Mlmikael08:10
https://en.bitcoin.it/wiki/Protocol_documentation#Message_types
"version
When a node creates an outgoing connection, it will immediately advertise its version.
...
8 timestamp int64_t standard UNIX timestamp in seconds"
@evoskuil?
isn't this each Bitcoin network peer reporting its local time to every other peer at protocol handshake at each connect?
anyhow you can set it to zero without breaking anything
Ignoring the consensus rule is not a great idea though.
It’s not true that version advertisement is limited to outgoing connections.
libbitcoin ignores the version message's timestamp, ok.
@lclc @evoskuil what were your thoughts about Ledger spinning their own https://github.com/LedgerHQ/lib-ledger-core . also, while of course absence of confirmation button and display is a breeding ground for security issues, do Ledger have issues on their devices with display (and buttons)?
GitHub
Contribute to lib-ledger-core development by creating an account on GitHub.
M
Mlmikael08:51
interesting - and then they solicited a tooon of VC investment recently to fill the planet with their devices
skaht15:18
@evoskuil, @neillm, @Mlmikael - Concerning emerging libbitcoin support for alt coins. It might be worth the time to obtain funding from the Dash Treasury to have libbitcoin support Dash transactions. I’ll pony up the the 5 Dash required to submit a proposal.
skaht15:40
Assuming it is feasible, the projects/proposals above and below the Dash Treasury’s funding line are visible at
(edited)https://dashvotetracker.com/ . From the examination of the BitGo project, asking $50K to $75K in funding is not too unrealistic.
Dash superblocks, the Treasury payouts, occur once a month. The next superblock is scheduled for around August 1st. Maybe we could shoot for this October’s or November’s Superblock? The process for submitting a proposal is at https://dashpay.atlassian.net/wiki/spaces/DOC/pages/8585246/Governance+and+Budget+System .skaht15:50
Was it https://github.com/lastcanal that pulled off the alpha Litecoin port six months ago?
skaht16:05
Lobbying Evan Duffield and Moocowmoo should radically improve the odds for obtaining treasury funding for a libbitcoin Dash port. I believe Evan is in Hong Kong working on hardware accelerators for master nodes that are to support big blocks with monsterous number of transactions to be validated.
And what is the work involved to port existing full nodes to Dash ?
What are the major differences to Bitcoin ?
ah wups, just saw the other text above regarding the 50k
skaht16:27
With Dash governance, 45% of the block seignorage goes to miner, 45% goes to master nodes (~4500), and 10% goes to the Treasury (released to Dash addresses of masternode approved projects, per Dash governance).
Dash was originally a fork of Litecoin, and was afterwards merged with bitcoin-core. The latest Dash release (
v.0.12.3.2 - https://github.com/dashpay just a few weeks old) indicates it was forked from Bitcoin. I believe, the 1st objective should be to get libbitcoin to interoperate with the latest Dash Core full node, signed binaries are at https://www.dash.org/wallets/. Guessing @evoskuil, @neillm, and @narodnik will be the ones to make a technical risk assessment if this idea is worth pursuing.neillm16:32
@skaht If someone is willing to do the work and issue PRs, it's apparently worth pursuing 😉 As for trying to get funding for this: Maybe it makes sense to find someone willing to do it first by asking in their community, as there may be people very knowledgeable about what's needed. If there is funding, and someone to do the work, I don't see why it can't be researched/attempted.
Wednesday, July 18, 2018
N
narodnik07:19
Damn I should have been using weak_ptr all these years when writing callback code in libbitcoin:
void MyClass::startTimer()
{
std::weak_ptr<MyClass> weak = shared_from_this();
timer_.async_wait( [weak, this](const boost::system::error_code& ec)
{
auto self = weak.lock();
if (self)
{
self->handleTimeout();
}
else
{
std::cout << "Target object no longer exists!\n";
}
} );
} (edited)@skaht really interesting, thank you. BLS looks amazing because unlike EC, it also offers curve pairing and homomorphic multiplication
The math is complicated though. I wonder what library dash is using.
Monero ring sigs with key images are incredible too, there's so many applications of that
N
narodnik07:36
initial sync doesn't even check sigs anymore. It's too slow, and I don't see a huge problem with new incoming blocks which you have lots of time to validate.
In the example above a shared pointer is better. Losing the ability to handle the event isn’t generally what you want.
Well there were many cases where I wanted to allow the class to go out of scope (for example node channels), so I created complex design with a stop() and boolean, pointer within the class when I wanted the weak_ptr paradigm
Dash Blog
I’ve been working on some BLS (Boneh-Lynn-Sacham) related topics for quite a few months already and there is one thing that I keep…
evoskuil07:43
But you don’t want the class to go out of scope until existing closures are cleared.
(edited)You want to stop accepting new work, complete existing work, and allow the class to go out of scope when that work is complete.
Early termination of existing work is another issue, which the stop Boolean or the invocation ec can signal, but you can’t rely on taking the class out of scope to signal early termination because the class may be locked.
And there are cases where early termination still requires use of the class.
M
Mlmikael08:00
Does LibBitcoin today implement any data structures for SPV proofs and their validation?
evoskuil08:00
They are both valid patterns, and there was some of each in the original implementation.
M
Mlmikael08:01
So say there's a transaction X, that i hold in my LibBitcoin instance and that has been stored in block Y on the blockchain.
I now like to compile an SPV proof that contains the content of X (that is a verbatim copy of the transaction X), together with the SPV proof for X, meaning anyone with Y's merkle root will be able to deduce that X belongs in Y
i'd wildguess this data message is totally unstandardized today, or?
evoskuil08:03
Just search the merkle block hashes for the tx hash, and validate the header and its merkle hash.
(edited)Aside from validating the chain (parents), and maybe each tx, and finding the strongest chain, that’s it.
M
Mlmikael08:05
The point with such an SPV proof is that you could send it to an SPV wallet out there, and it would be able to deduce that X indeed is present in the blockchain, as the SPV wallet by itself tracks the latest block based on PoW validation only
M
Mlmikael08:06
so the data format of the SPV proof is something like, [transaction] [transaction's hash] and then a recursion of: the hash, and the neighbor's hash, and then the hsah of those two - all up to the root
mm indeed.
so only the neighbor hashes need to be passed (on the lowest level that's the hash of the neighbor tx, if any, and on all upper levels its the merkle hash)
right.
the point here would be that the party this SPV proof is sent to, only tracks merkle roots, not merkle trees.
evoskuil08:09
To validate the hash is in the block you need the header and to validate its merkle hash, which requires the merkle block (tree).
(edited)M
Mlmikael08:10
to validate that the hash is in the block, you need only that particular line from leaf to root within the merkle tree,
you don't need the whole merkle tree; right?
M
Mlmikael08:11
so the SPV proof says, transaction X's hash is 2345, its neighbor tx' hash is 3456, one level up the neighbor hash is 4567, one level up the neighbor hash is 6789
the SPV wallet on the other side is tracking block headers via the Bitcoin P2P network or a mirror proxy thereof
so the SPV wallet does something like, hash(hash(hash(hash(X) + 3456) + 4567) + 6789) and checks that its result equals the merkle root hash of the regarded block (Y)
(edited)all clear i guess - thanks for your feedback!
(less relevant, do you see any relevant synonyms for "SPV proof" - the idea is I give you cryptographic proof for you to check yourself that my datum is in your merkle-hashed data structure, right.)
M
Mlmikael08:23
wait, what is your thought with mentioning the bloom filters BIP on the topic of SPV proofs?
does LibBitcoin implement any data structure for this kind of proof (bip 37 calls it "partial Merkle branch format")?
implementing it oneself should be very straightforward of course.
N
narodnik10:02
@evoskuil class may need to be cleared up but usually (like in node channels), you just want the current operation to finish early and signal failure to complete (channel closed) to the callback.
@narodnik Yes, that is what I was referring to with “early termination”.
Thursday, July 19, 2018
M
Mlmikael05:09
@evoskuil, some other time i remember you mentioned that for a web wallet to listen to transactions, there's a better more privacy preserving method available than bloom filters. What was the method you mentioned again?
@skaht thinking further about BLS, they offer interesting properties- you can issue a new blinded key or preimage, sign it, then aggregate all the signatures and unblind your new key. We can generate new keys unlinked to the old ones with this method.
I can think of so many applications for this concept. Right now I'm trying to think if schnorr can do this too
There is also a C library too
N
narodnik09:04
ok this is the best BLS library I've found and am using now:
https://github.com/herumi/mcl/blob/master/sample/bls_sig.cpp
evoskuil09:41
I think the outstanding issue was the need to make the two new classes bases for the existing two wallet classes, preserving the interface of the latter.
ec_public and ec_private
btw monero offers something in their ring sigs called one time preimages, which disallow reuse of a public key, so maybe later we add an overload for this function which supports that. However I'm researching BLS and they seem to be more powerful. I think it could be possible to make the world's first decentralized anonymous cryptocurrency (with Bitcoin on a sidechain)
if bitcoin ever supports schnorr, that would be real good
ec_public and ec_private inherit ec_point and ec_scalar
skaht12:55
@narodnik - On a Dash Discord channel,
@codablock (Alexander Block) indicted that https://github.com/herumi/bls is being used by Dash . He also indicated I've created wrappers around the libraries which allow us to use the optimizations in a more convenient way, and code will be published in a few weeks.(edited)
Friday, July 20, 2018
since this channel is linked on the wiki, surely it must be opened for anyone to join
how can we generate the doxygen docs? or the API documentation
But I give out reusable links.
neillm11:53
@narodnik There is a short invite url pinned in this channel. I think that link can be posted anywhere for others to join.
on the right, only registered users can join currently
Saturday, July 21, 2018
M
Mlmikael20:08
Where in LibBitcoin's sourcecode do I find the code that checks that a new block's hash matches the PoW difficulty that applies to that block, and calculates the applicable PoW difficulty based on the block chain preceding the block?
pmienk20:11
I'd start looking around here: https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/chain_state.cpp#L370
..or not
it looks close to it
a block (header) on the Bitcoin P2P network does not copy the difficulty,
but instead YOU need to calculate it, right
and then YOU take the block (header) and calculate the block's hash,
given the nonce and extnonce and all as provided to you by the miner, and YOU count how many zeroes the block's hash starts with.
and then based on that YOU make up your mind about whether you deem that block to be valid or not
@pmienk also is there any method or type to quantify the total PoW behind a block and all its parents, so that I could compare two blocks total PoW to see which has more work?
pmienk20:18
@Mlmikael I don't remember offhand. I found the reference by searching the codebase for PoW. I'd advise you to start there next time. Given that chain_state exists to manage questions like you're raising, I'd examine it in greater detail if I were you.
(ah also small correction to what i said above, there is a bits value in the block header, so what i'm looking for is the logics to validate it, as well as validating a block [header])
pmienk20:22
are you trying to validate the consistency of a block and header or are you trying to answer the question of whether the work in a given block is sufficient to append to a chain?
actuallly when would the second option you mentioned be relevant?
for consistency of data, I'd look at the header::check method
additionally, the block::check method
M
Mlmikael20:26
ah there you go, header.cpp has bool header::is_valid_proof_of_work(bool retarget) const
in particular where does calibration happen
maybe this method only checks for data format validity
i think indeed not
it doesn't actually count the bits and compare it againts how many are required
that's done somewhere totally else
pmienk20:28
it doesn't involve chain_state, so it doesn't relate to the required level of work to append to the chain
pmienk20:28
chain_state has an internal variable work_required_ - the calculation thereof seems to be what you're talking about
because how many bits are required isn't a question of block/header consistency, but instead chain consistency (history)
Sunday, July 22, 2018
relinking here for everyone
:))
Monday, July 23, 2018
Tuesday, July 24, 2018
M
Mlmikael16:21
guys, LibBitcoin version 4 is in the "master" branches, right?
(there are no "version4" branches, alas 🙂 )
version4 branches will be created when it's stable/production-ready
building version4 (master branch) the same way as we build version3, gives this - "bn" starts and then just shuts down right away
M
Mlmikael17:32
@evoskuil tried with wiping the blockchain store directory and that did not resolve it
neillm17:36
@Phil Ok, but installs in non-standard directories may not have the same LD_LIBRARY_PATH as the 'normal' user when using sudo -- perhaps related to the link error. Bad links (particularly with boost) can cause odd errors. Would prefer to rule that out.
P
Phil18:42
@Phil uploaded a file: debug.log file
debug.log file
$ cat archive/debug-00002.log 14:41:40.741500 DEBUG [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.745623 INFO [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.746129 WARNING [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.746999 ERROR [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.747077 FATAL [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.747099 INFO [node] Using config file: "/home/user/libbitcoin4/build/etc/libbitcoin/bn.cfg" 14:41:40.747182 INFO [node] Press CTRL-C to stop the node. 14:41:40.747240 INFO [node] Please wait while the node is starting... 14:41:40.750573 DEBUG [database] Buckets: block [650000], transaction [110000000], address [107000000] 14:41:40.758881 DEBUG [database] Mapping: "blockchain/block_table" [2600143] (4096) 14:41:40.759204 DEBUG [database] Mapping: "blockchain/candidate_index" [4] (4096) 14:41:40.759236 DEBUG [database] Mapping: "blockchain/confirmed_index" [8] (4096) 14:41:40.759258 DEBUG [database] Mapping: "blockchain/transaction_index" [12] (4096) 14:41:40.760845 DEBUG [database] Mapping: "blockchain/transaction_table" [880000265] (4096) 14:41:40.763334 ERROR [node] Failure starting blockchain. 14:41:40.763949 ERROR [node] Node failed to start with error, operation failed. 14:41:40.765004 INFO [node] Please wait while the node is stopping... 14:41:40.769637 DEBUG [database] Unmapped: "blockchain/block_table" [2600143, 2600143] 14:41:40.769858 DEBUG [database] Unmapped: "blockchain/candidate_index" [4, 4] 14:41:40.770061 DEBUG [database] Unmapped: "blockchain/confirmed_index" [8, 8] 14:41:40.770265 DEBUG [database] Unmapped: "blockchain/transaction_index" [12, 12] 14:41:40.777032 DEBUG [database] Unmapped: "blockchain/transaction_table" [880000265, 880000265] 14:41:40.778612 INFO [node] Node stopped successfully.
Is there some useful information in the debug file to help us figure it out what's the problem?
P
Phil18:57
@Phil uploaded a file: Untitled and commented: I don't think so.
Untitled
$ cat archive/error-00002.log 14:41:40.746129 WARNING [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.746999 ERROR [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.747077 FATAL [node] ================= startup Tue Jul 24 15:41:40 2018 ================== 14:41:40.763334 ERROR [node] Failure starting blockchain. 14:41:40.763949 ERROR [node] Node failed to start with error, operation failed.
neillm19:43
@Phil I see this on a new build of node from master as well. Would you mind opening an issue for it, in order to track it?
Wednesday, July 25, 2018
M
Mlmikael06:15
@evoskuil, version 4 is in master right, is it supposed to sync the blockchain correctly?
It’s the staging area for v4 work, it compiles, not syncs.
M
Mlmikael06:20
aha. so as a LibBitcoin user i shouldn't touch version4 now but instead wait until September-October, yes?
and, the "download headers" vs "download blocks" distinction is a version 4 feature so i'll be able to have LibBitcoin do headers-only download at first then with v4 in Sep-Oct, correct?
M
Mlmikael06:40
👍 ok
about how much interface differences are there between v3 and v4 in the block and full node classes?
can i start beta-using the v4 node in Oct just to get on the new interfaces early?
anyhow thanks for saying you expected it to be really stable at first by EOY.
evoskuil06:46
There are smallish changes in libbitcoin and node, and bigger changes in database and blockchain.
@evoskuil anyhow so beta-testing version 4's node can start in.. Sept-Oct sometime? (and right now as a user better not touch it at all)
M
Mlmikael18:41
@evoskuil version4 should likely start syncing the whole chain and keep doing so in 2-3mo?
Tuesday, July 31, 2018
Fil17:00
Does
bx support converting compressed pubkeys to uncompressed ones? I didn't find a subcommand but maybe I've overlooked it.neillm17:10
I don't think there's a command just for that, although there are libbitcoin methods (compress/decompress)
It would probably look a lot like ec-to-public, without the last conversion to ec_public (i.e. just printing out the encoded compressed point)
Actually, that would be overkill
compared to the compress/decompress
skaht17:47
bx supports both compressed and uncompressed public key synthesis and associated addresses.skaht18:31
The works secp256k1 works is that you can’t go backwards from a public key to a private key. You do however have the option to move from a hex private key to either a base58check encoded compressed or uncompressed WIF keys, and associated compressed or uncompressed P2PKH addresses.
Applying a
sed 's/$/01/' operation converts an uncompressed private hex key a compressed private key.skaht18:42
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-private -i 0 | bx hd-private -i 0 | bx hd-to-ec | bx base58check-encode -v 128
5JtqqvK24Vpmf776RujgiXAxTvhrT1ux6vHUiEw56KBM7SsowWk
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-private -i 0 | bx hd-private -i 0 | bx hd-to-ec | sed 's/$/01/' | bx base58check-encode -v 128
L1ypavJXgSDr7BwtxwmQJGAwor3KFq2iD8BjK2Vf4TEfmHQU9CDk
skaht18:49
Uncompressed Address:
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-private -i 0 | bx hd-private -i 0 | bx hd-to-ec | bx ec-to-public -u | bx ec-to-address -v 0
1AvYwTGggjTezsGJYzRJhYMZNWfz38HeeB
Compressed Address:
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-private -i 0 | bx hd-private -i 0 | bx hd-to-ec | bx ec-to-public | bx ec-to-address -v 0
164AqjP54Mo7uvsaQF4HduCJaQCsTYbjoW
or
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-private -i 0 | bx hd-public -i 0 | bx hd-to-ec | bx ec-to-address -v 0
164AqjP54Mo7uvsaQF4HduCJaQCsTYbjoW
or
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-private -d -i 0 | bx hd-public -i 0 | bx hd-public -i 0 | bx hd-to-ec | bx ec-to-address -v 0
164AqjP54Mo7uvsaQF4HduCJaQCsTYbjoW
or
% echo shit | bx base16-encode | bx sha256 | cut -c 1-32 | bx mnemonic-new | bx mnemonic-to-seed -p "sucks" | bx hd-new | bx hd-private -d -i 44 | bx hd-private -d -i 0 | bx hd-public -d -i 0 | bx hd-public -i 0 | bx hd-public -i 0 | bx hd-to-ec | bx ec-to-address -v 0
164AqjP54Mo7uvsaQF4HduCJaQCsTYbjoW
Powerful Kungfu Eric’s team came up with after refactoring
sx !skaht19:08
PCI-DSS just got a hell of a lot easier with crypto commerce servers, no private keys required! Just need to ensure the integrity of extended public keys, associated public key synthesis code, and indexes being used.