Bitcoin full node and query server
March 2019
Archive of the libbitcoin Slack #general channel.
Tuesday, March 5, 2019
N
narodnik11:56
@vdeurzen my personal style is to use free functions to compose operations, and keep class interfaces to the base minimal. Free functions therefore avoid polluting the class interface.
But because of ADL they are still kind of like a member (for example overloading them in template specializations)
N
narodnik12:04
BTW I'm very excited for C++20 which will bring coroutines and modules, and many other strong improvements. With modules support, there is a general movement towards standardized package management (like python's pip install). Currently it seems CMake and Meson are the 2 competing build systems (with Meson the newer standard and which I prefer). For package management there is Conan and vcpkg, neither of which I like so far.
Short tutorial on Meson:
https://medium.com/@germandiagogomez/getting-started-with-meson-build-system-and-c-83270f444bee
Medium
Hello everyone. First of all, pardon my writing style and possible typos, it is my first public blog post ever and english is not my…
vdeurzen13:03
I've been using Conan at work a bit, it feels like the least worst option at the moment, made some of the new C++ projects I started a breeze to set-up with CI, certainly when compared to my previous cmake-and-bash-scripting
C++ 20 is going to be very interesting indeed!
Revive17:13
I'm not sure i'm excited about a movement towards package management in c++, i detest the newer hipster web world with its billion long dependency tree. i have liked so far being able to see and understand a clean dependency tree with c++. ive not noticed a mess in pip however
package management within code get out of control as i understand
(edited)
neillm17:57
I haven't been following it much, but am curious to see what standardized package management in C++ would look like, given all the architectures, etc. Or are the modules entirely source based (meaning standardizing breakage everywhere)? pip (and node's npm) don't really have that issue/distinction as far as I know
Revive18:16
is it a normal practice to distribute python binaries? ive never seen this in distribution, have seen compiled locally .pyc
what would go sweet with a c++ type package management would be something as they do with nix packages, hash tree
P
Phil19:46
I’m facing some trouble with a libbitcoin dylib file, I’m getting
Expected in: flat namespace in /path/to/lib/libbitcoin-system.0.dylibDo you know how to solve that? I found a guys with a similar problem. But it was a super specific solution.
https://stackoverflow.com/questions/35006614/what-does-symbol-not-found-expected-in-flat-namespace-actually-mean
(edited)Wednesday, March 6, 2019
N
narodnik09:51
Ruse style borrow checker in C++. Maybe I need to use Rust more to see why this is useful:
https://gist.github.com/foonathan/023ff0fe923c6b0312dfc15e17ebb595
@Phil The link on SO seems to imply it's because of linking against the wrong library or with the wrong compiler (compiled with clang, linked with gcc)
jungly17:33
@evoskuil @neillm - I was digging through how btcd does payments/address indexing and found something curious. Would love to hear your thoughts on it.
They index addresses, as v3 does. The interesting thing is that they index pubkeys from a multisig output as well. They convert those pubkeys into addresses, which seems kind of odd. But they index that transaction for all those addresses (obtained from the pubkeys) enabling queries for use of pubkeys in payments.
As I understand, with our approach to index hashes of output scripts, we won't be able to support querying the payments index for a single pubkey in the multisig transactions. Is that right? Also, is that a concern, given that LN might cause some growth in the share of multisig transactions?
In case you are curious, this is where they grab the pubkeys and convert them to addresses: https://github.com/btcsuite/btcd/blob/86fed781132ac890ee03e906e4ecd5d6fa180c64/txscript/standard.go#L596
neillm17:41
If I understand correctly, we could still index the hash of the output script for those -- or do you mean more like 'should we'?
evoskuil17:42
I considered it but decided against it since the recipient is presumed to be aware of the address.
i was thinking of shady third parties trying to figure out where pubkeys were used 😉 🙂
jungly17:45
The use of multisig with growing experimentation with LN is fascinating.. I don't know if they use the same pubkey to open multiple channels, but if they do, some way to query all my LN node's channels currently open could be a growing use case
but lets leave LN aside... the recipient knowing the multisig address is cool
Thursday, March 7, 2019
jungly15:03
Hi all, I am finally giving the cmake build a try, but am running into linker errors.
I do the following
cd builds/cmake && mkdir out && cd out && cmake .. && cmake --build .
and I get the following error at link time
`
[100%] Linking CXX executable libbitcoin-system-test
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
CMakeFiles/libbitcoin-system-test.dir/build.make:2876: recipe for target 'libbitcoin-system-test' failed
make[2]: *** [libbitcoin-system-test] Error 1
CMakeFiles/Makefile2:141: recipe for target 'CMakeFiles/libbitcoin-system-test.dir/all' failed
make[1]: *** [CMakeFiles/libbitcoin-system-test.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
`
Am I doing something wrong here?also, this line looks strange https://github.com/libbitcoin/libbitcoin-system/blob/master/builds/cmake/CMakeLists.txt#L611
I deleted that line, ran the whole
cmake . and cmake --build . again and still the same error as aboveJust a note, the autogen make process is working fine
oh dear.. it doesn't!! i assumed it did! 🙂
jungly16:17
but its a boost unit test thing.. they have their own way of describing main when you use the macros..
👍 1
vdeurzen16:18
might be an issue with the macro not being defined, maybe the autogen stuff defines it as a compiler flag?
e.g. -DBOOST_UNIT_BLAHBLAH
jungly16:19
yeah.. i'll check that.. i was just curious if anyone else had the same issue.. i don't have much cmake experience, so wanted to check if i was doing something really silly here
i think we are missing Boost_UNIT_TEST_FRAMEWORK_LIBRARY from target_link_libraries in the CMakeLists.txt
I'll try to put that in and see if it works
vdeurzen16:23
You may be able to specify that on the commandline too without needing to change the config, not a 100% certain on how, I believe just
-DMY_AWESOME_VAR=blahfor the
cmake . partjungly16:25
no crap.. i was wrong..the Boost_UNIT_TEST_FRAMEWORK_LIBRARY thing is in the target_link_libraries..
I'll see if i can pass a flag.. really don't know much about cmake.. live and learn!
vdeurzen16:27
it's not the most friendly tool to learn, but it's becoming a sort of default for many c++ projects
jungly16:31
unit_test_framework is definitely in the link command, checked the link.txt generated by cmake
i'll park this again, go back to lovely autotools for now 🙂
Friday, March 8, 2019
jungly15:45
I have a question about electrum server. This might be off topic, but just throwing the question here.
I looked into how they sync off a bitcoind node to build a database that the server uses. I don't get why they used the JSON-RPC api for the first sync. For an initial sync with bitcoind, would it not be faster to shutdown bitcoind, so electrum can open leveldb, and then read the data off the db directly? Just curious if @evoskuil or @narodnik knew more about this.
neillm15:46
I'd guess that's faster, but more involved/more coding/maintenance, etc. Since it's one time, it's more or less the cost of usage (like bitcoin initial sync).
jungly15:50
yeah.. they don't have to write two ways to read the blocks. Makes sense. But onboarding is going to get slower and slower, especially for the personal server
neillm15:51
Well, it's also the only current option (a stable API). Perhaps it wasn't a given that LevelDB would always be the de-facto storage unit, so maybe unsafe to write code for it (more coding argument). But if electrumx was built on top of a faster stable API, it could help them scale that ...
(edited)N
narodnik16:56
I think it's something that hasn't aged well, and they haven't been able to significant large improvements esp with lightning now they feel huge pressure
jungly17:05
I also worry about the routing of payments and the whole channel balancing thing that keeps coming up. There are some decent unresolved challenges there alright. I am not sure they are solvable without a long research roadmap.
It still is odd how man person hours are going into LN at the moment. Maybe with so much effort some of the challenged will get solved. 🤷
evoskuil17:15
ElectrumX could sync much faster off of the libbitcoin zeromq interface, but then again we already maintain the ElectrumX index ;).
(edited)Ln is a lot of fun from a geeks perspective. All that networking and routing of payments, not just packets. So exciting!! ;)
evoskuil17:35
You should start modifying the electrum client to sync off of libbitcoin. We need to at least get that proof of concept complete before we ship v4
Anything to stay away from writing tests for the blockchain code. ;) Which I really want to do too btw.
The outstanding issues will be fee estimation and server discovery.
evoskuil17:45
The first cut should be to get the wallet sync working against a single dedicated server using your new index, using only the pyzmq dependency.
The dependency is a deployment complexity for electrum, but any other issues aside I believe they would accept it.
Then we would look at possible revisions to libbitcoin-server to simplify the client.
(no custom eventloop)
evoskuil18:12
This has a dependency on pyzmq. Not sure the benefit of coroutines outweighs the downside of another dependency, especially if community is small. But worth looking at.
neillm18:13
While I like the python async stuff, it may be tricky to integrate into existing electrum. @jungly You'll have to review if that makes sense, given their existing architecture.
evoskuil18:15
I think electrum went async during the feathercoin team work on this integration. This is why it had to be done over.
(edited)jungly18:17
Yup, I saw asyncio and coroutines. But any context is great, will probably save me effort discovering things for myself
N
narodnik18:18
just don't use pyzmq asyncio, I had a very bad time. It wants you to install a custom eventloop which doesn't integrate well with other libraries like websockets or Qt
aiozmq doesn't use a custom eventloop... also dependencies in python aren't such a big problem because they can be bundled with the source code.
He’s how I got in touch with you in the beginning.
evoskuil18:22
Yeah, he also ran sx for a while. Saw him on the libbitcoin mailing list when I was doing my diligence.
Tried to get him onto libbitcoin for a long time. But he’s all over the place.
did you meet them yet?
evoskuil18:25
Yeah, we’re following up, but I don’t think expectations were well aligned. I can follow up with you privately.
Saturday, March 9, 2019
N
narodnik19:42
Ah something important I forgot to mention. There is fairly decent reference material for libbitcoin, but there needs to be a tutorial docs that guide people through building applications with the library. For example you might want to log network statistics or index OP_RETURN outputs in a special index in the database .etc This is the real power of the library, is its logical well defined layout that enables people to reconfigure or use the toolkit to build many different forms of applications.
About the Python bindings, I think there's no way around it, just somebody has to knuckle down and make them manually. The current standard system in C++ for that is pybind11. I wouldn't trust any automatic bindings generators as they're likely to get many things wrong especially with the new language features.
Having Python bindings + good tutorial documentation I feel is essential and will help to build a much larger community. You might consider it a target after the v4 release. It's a shame to do all this great work but it isn't visible to people. I think a tiny investment of energy on the forward facing parts (website, doc, lang bindings) would go a long way. For language bindings you only really need to support Python. Having C support would be a bonus but not required (like zmq which was written in C++ but had a C api). The main dominant languages today are C, C++ and Python.
👍 1
Here's the pybind11 Python binding library I mentioned: https://github.com/pybind/pybind11
We are planning to build an anonymous DW library with built in mixnet support. Not sure if Harry mentioned this to you, but this is a long project in the planning since 1+ year because we are advancing new crypto anonymity techniques (the ring sigs code in libbitcoin originally was part of that but we've moved on significantly since then). So this will be a strong application to have too, and we think it will help revive crypto from the current winter and give it back its radical focus on free market, anti-state, anonymous values.
*anonymous DW application
jungly19:49
Totally agree with the need for putting out more educational material. AFAIK, James is doing great work there. But a libbitcoin "branded" website and docs would be great.
👍 1
Classic
Sunday, March 10, 2019
Monday, March 11, 2019
Thursday, March 14, 2019
mind blown 😄
😂 1
Tuesday, March 19, 2019
B
baiumbg14:05
Hey guys. I have a question regarding libbitcoin's license. Am I supposed to open-source the entirety of my code if I link with the library (and use it for commercial purposes)? Also, if I need to customise something in the library, do I only need to make those changes publicly available or, again, the entirety of my code? I'm not very well versed in the language of software licensing 😅
neillm15:25
@baiumbg This might help: https://softwareengineering.stackexchange.com/questions/107883/agpl-what-you-can-do-and-what-you-cant
Software Engineering Stack Exchange
AGPL is a fairly new license that was meant to go GPL-over-networks. However, not being a lawyer, and actually not having read the whole license, I can't understand what exactly you can do freely and
B
baiumbg15:29
@neillm I've already read that, but I'm asking since this is in libbitcoin's README in github:
All files in this repository fall under the license specified in COPYING. The project is licensed as AGPL with a lesser clause. It may be used within a proprietary project, but the core library and any changes to it must be published online. Source code for this library must always remain free for everybody to access.(edited)
Linking does not make the linked code part of the core library.
if I need to customise something in the library, do I only need to make those changes publicly available...
Yes, preferrably via a pull request to the libbitcoin repo.
(edited)Thanks! That's what I thought as well, but I had to ask anyway, just in case. Looks like it's quite a bit more permissive than the regular GPL licenses.
Wednesday, March 20, 2019
B
baiumbg11:22
Are there any differences for a regtest setup between version3 and master (libbitcoin-server)?
(edited)Or is the regtest configuration in the wiki enough?
neillm11:25
There are many differences between v3 and master at the moment, not sure how regtest in particular is affected. I recommend using v3 regtest, as that's what the docs use.
B
baiumbg11:35
I'm aware that there are many differences, but I'm interested in trying it out. I'll check to see if there is anything else, that might need to be added to the docs for a v4 regtest.
👍 1
vdeurzen11:53
regtest with the latest bitcoind version was problematic a while ago, haven't had the time to look into it though, so I couldn't tell you if it's still an issue
B
baiumbg12:19
It's fine atm - the bip34 activation checkpoints were our of date, but that's been fixed.
That's also something that has to be changed in the config for v4 - just noticed they're no longer hardcoded 😄
Untitled
diff --git a/data/bs.cfg b/data/bs.cfg index e92feca..1463f74 100644 --- a/data/bs.cfg +++ b/data/bs.cfg @@ -30,33 +30,33 @@ timestamp_limit_seconds = 7200 # The difficulty retargeting period in seconds, defaults to 1209600. retargeting_interval_seconds = 1209600 # The proof of work limit, defaults to 486604799. -proof_of_work_limit = 486604799 +proof_of_work_limit = 545259519 # The initial block subsidy in bitcoin, defaults to 50. initial_block_subsidy_bitcoin = 50 # The subsidy halving period in number of blocks, defaults to 210000. subsidy_interval = 210000 # The genesis block, defaults to 0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000 (use 0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4adae5494dffff001d1aa4ae180101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000 for testnet). -genesis_block = 0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000 +genesis_block = 0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4adae5494dffff7f20020000000101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000 # The number of new version blocks required for bip34 style soft fork activation, defaults to 750 (use 51 for testnet). -activation_threshold = 750 +activation_threshold = 51 # The number of new version blocks required for bip34 style soft fork enforcement, defaults to 950 (use 75 for testnet). -enforcement_threshold = 950 +enforcement_threshold = 75 # The number of blocks considered for bip34 style soft fork activation, defaults to 1000 (use 100 for testnet). -activation_sample = 1000 +activation_sample = 100 # The block height to freeze the bip65 softfork as in bip90, defaults to 388381 (use 581885 for testnet). -bip65_freeze = 388381 +bip65_freeze = 1351 # The block height to freeze the bip66 softfork as in bip90, defaults to 363725 (use 330776 for testnet). -bip66_freeze = 363725 +bip66_freeze = 1251 # The block height to freeze the bip34 softfork as in bip90, defaults to 227931 (use 21111 for testnet). -bip34_freeze = 227931 +bip34_freeze = 0 # The activation time for bip16 in unix time, defaults to 1333238400 (use 1329264000 for testnet). -bip16_activation_time = 1333238400 +bip16_activation_time = 1329264000 # The hash:height checkpoint for bip34 activation, defaults to 000000000000024b89b42a942fe0d9fea3bb44ab7bd1b19115dd6a759c0808b8:227931 (use 0000000023b3a96d3484e5abb3755c413e7d41500f8e2a5c3f0dd01299cd8ef8:21111 for testnet). -bip34_active_checkpoint = 000000000000024b89b42a942fe0d9fea3bb44ab7bd1b19115dd6a759c0808b8:227931full file
👍 2
Wednesday, March 27, 2019
neillm16:06
mainnet1 will be down for some time. Sorry for unexpected down-time. Will notify when back up.
mainnet2 should be the primary (if possible to update DNS)
Sorry, nevermind, it restarted properly(!)
(edited)mainnet2 has been more stable anyway, so updating DNS anyway isn't a bad idea
Thursday, March 28, 2019
neillm13:27
@jungly What's the status of https://github.com/libbitcoin/libbitcoin-database/pull/271 ?
neillm21:54
Interesting mail on the mailing list about the private key discovery ... not sure if it's real though. Anyone else see it?