Bitcoin full node and query server

December 2019

Archive of the libbitcoin Slack #general channel.

Monday, December 2, 2019

A
Alex10:16
Can I expect master's libbitcoin-system, libbitcoin-client, and libbitcoin-protocol to query a v3 libbitcoin-server without issue? v3 doesn't have witness_address, which I'd like to use in the client application to display transaction information.
neillm13:45
No, master is pre-v4, and is not query compatible in some ways with v3
evoskuil22:56
Address query is the main distinction. V4 indexes payments, using a hashed search key derived from the associated script. This is consistent with current Electrum query, allowing recovery of any payment. V3 was limited to pre-segwit standard payments.
However there is no distinction in -protocol, and -client should ship with a compatibility shim, since the search key for an address can be generated from the associated standard script.
-server could also provide compatibility but I do not believe that is currently the case, and the namespace May currently conflict with v3 query.

Tuesday, December 3, 2019

N
narodnik01:49
programmer friend is confused by bcs::chain::script input_script({ { endorsement }, bcs::machine::operation({ public_key.point().begin(), public_key.point().end() }) }); ... might be worth adding that overload @evoskuil
image.png
evoskuil02:10
What kinda programmer? ;)
I’ll just add a templated length array overload and internally convert.
N
narodnik02:12
cool
evoskuil02:12
I like having the clean syntax too.
Just can’t use the slice because it’s too inefficient in the validation context.
(edited)
N
narodnik03:16
yep

Thursday, December 5, 2019

N
narodnik19:19
Is there an easy function to serialize bc::chain::history::list to just data and back again? I'm trying to make a proxy service for nym mixnet and libbitcoin server
evoskuil19:21
I don’t think so, since lists aren’t objects they don’t have aggregate serializations unless provided by a containing object.
but using a stream it’s just a couple lines of code.
N
narodnik19:23
That's also fine, where is the serialization for history located?
evoskuil19:26
In server.
Should be ez to find.
N
narodnik19:43
ok found it
void blockchain::history_fetched(const code& ec,
    const history_compact::list& history, const message& request,
    send_handler handler)
{
    static constexpr size_t row_size = sizeof(uint8_t) + point_size +
        sizeof(uint32_t) + sizeof(uint64_t);

    data_chunk result(code_size + row_size * history.size());
    auto serial = make_unsafe_serializer(result.begin());
    serial.write_error_code(ec);

    // TODO: add serialization to history_compact.
    for (const auto& row: history)
    {
        BITCOIN_ASSERT(row.height <= max_uint32);
        serial.write_byte(static_cast<uint8_t>(row.kind));
        serial.write_bytes(row.point.to_data());
        serial.write_4_bytes_little_endian(row.height);
        serial.write_8_bytes_little_endian(row.value);
    }

    handler(message(request, std::move(result)));
}
evoskuil19:49
Note that row.value is either a satoshi value or a hash reference. Changes were made to reduce probability of hash collision, but it can happen and should be handled. See client for algorithm.
Slackbot19:49
👻
N
narodnik21:07
ok thanks
evoskuil21:18
No!
Np! :))

Saturday, December 7, 2019

evoskuil23:14
Bitcoin Developer Network
In this tutorial, we will be taking a closer look at libbitcoin, a multipurpose bitcoin library targeted at high-end use. An ideal backend to build fast implementations on top: mobile apps, desktop clients and server API's.
👍 1

Monday, December 9, 2019

lclc11:54
How much disk space does the current synced libbitcoin-server use ?
neillm13:16
v3 mainnet looks like ~483GB at the moment
v4 will be less, but don't have those numbers at the moment
evoskuil13:18
Depends a lot on indexing. Payments (ie address) and Neutrino indexing are quite large.
Neutrino on v4.

Tuesday, December 10, 2019

Revive01:18
I thought you advocated against neutrino
evoskuil02:10
Nope
It is client-server, which is not ideal. It has privacy limits. But unlike bip37 it doesn’t require a peer (server) to maintain state for other peers (client).
(edited)
👍 1

Friday, December 13, 2019

L
Hi evoskuil, can you please explain me your comment? https://github.com/libbitcoin/libbitcoin-build/pull/217#discussion_r357791327 "bitcoin" instead of "libbitcoin" sounds for me a bit like a forgotten change after renaming the "bitcoin" to "libbitcoin". Do I undestand correctly that you deliberately left these names as "bitcoin"? If so, what are the advantages? For now, I can only see disadvantages like special cases (like this line you commented on).
evoskuil21:03
There was never any such rename. libbitcoin was renamed to libbitcoin-system.
“lib” is treated as a library prefix, not part of the name of the dependency.
The names are used in other contexts.
pmienk21:07
Additionally isn't there a difference by platform as to whether or not the prefix is applied. Windows generates bitcoin-<component>.lib vs linux/macos libbitcoin-<component>.a
(edited)
👍 1
L
"There was never any such rename." Oh, that's right. I mixed up something.
pmienk21:09
As that is a tooling issue, we condition cmake output to match visual studio instead of fighting that stack
L
I don't quite understand where the problem is right now. As a person who creates and manages the generate.xml template, I'm not interested in details and how certain tools work. From this I have gsl to do it for me. In this case "libbitcoin" is semantically correct, because this is the actually name of the dependency. If any of the tools do not work with the "lib" prefix, the gsl task should be to remove the prefix and use the "bitcoin" name needed for these tools. Am I wrong?
evoskuil21:33
Yes :)
“lib” is a tooling convention.
Libbitcoin (capitalized) is a reference to “bitcoin library”. The actual libraries generate both “libbitcoin” and “bitcoin.lib”.
and “bitcoin-explorer.exe”
etc.
The prefix on the sources and github repos is an anachronism.
Notice the the headers are “bitcoin/system/...”
L
@evoskuil OK, I see :). there is also <package library="bitcoins-system"... that makes sense :). I confused the name of the repository with the name of the library. I thought they were the same.
evoskuil21:41
Yeah, the repos should be “bitcoin-system”.
It’s been a process of rationalization over years.
It became very apparent with code gen.
L
Thanks for the clarification. I will change the PR soon. another question about the second topic. https://www.appveyor.com/pricing/ unlimited public projects and 5 self hosted jobs? what does it mean the case?
AppVeyor
Plans and Pricing
evoskuil21:55
@pmienk ?
pmienk21:57
Is that supposed to be a change? We have unlimited projects and 1 concurrent jobs. I don't know what self hosting is in this context.
Looks like if it's new they would be allowing us to hook our own build hardware or azure/Amazon hosting to perform builds.
So I guess if we pay for cloud compute we could open up to more than just the system project and maybe get things completed in a reasonable amount of time. Odd that all plans cap external addition to five jobs
L
@pmienk Am I correct that there is no limit to the number of repositories? So what speaks against Appveyor with other libbitcoin repos? In the worst case, it will take a long time if several PR come in, but better than not having these builds at all.
pmienk22:26
Each pr for each repo generates 4 builds. Due to the time and the 60 minute limit only the libbitcoin-system repo is currently enabled on appveyor
Due to build time, some repos like -explorer have little chance of completion and would consume the job. As libbitcoin-system takes north of 20 minutes and is built by all repo builds (excluding tests in dependencies), it was expressed that dependency repos should be disabled.
If we brought more jobs and were able to lift the 60 minute cap or cache and reuse the compiled dependencies, we would be positioned to reenable the other projects with a button press
L
However, libbitcoin-build has a very simple and small "build"
pmienk22:45
Libbitcoin-build doesn't compile. There is no value to it's execution on appveyor that we don't get on travis
If you'd like to go through the trouble of getting gsl working on appveyor, be my guest.
L
Ok, I'll see if I can do anything. I only noticed that I cunonsciously destroyed the Windows generation in my PR because I only focused on Linux (in the future I will call also the cmd locally). I have no idea if AppVeyor allows any installations like gsl. Maybe it will fail because of that :).
evoskuil23:04
I run GSL on windows. The primary objective would be to build all repos on appveyor, except -build, which is secondary.
L
@evoskuil but installing gsl on appveyor could be a problem
evoskuil23:10
Don’t need to, it’s only used in -build.
L
@evoskuil I am talking about build :). -build checks if generate.sh fails. it would be nice to check if generate.cmd fails also
"except -build, which is secondary." ok
evoskuil23:14
I use it frequently, good nuff.
L
:D
evoskuil23:15
That’s also the current basis of VS builds stability.
L
afk. have a nice day.
evoskuil23:34
👍

Tuesday, December 17, 2019

Hi guys, so we're using libbitcoin-server to query blocks and txs. The zmq interface uses the REQ/REP pattern. According to the ZMQ documentation "This socket type allows only an alternating sequence of zmq_send(request) and subsequent zmq_recv(reply) calls". Is there a standard way to scale this? First thing that came to mind is using multiple REQ sockets but it seems kind of wrong. As you can imagine, our process is wasting time just waiting to receive the reply. Thanks!
jungly18:32
I don't know much about the server interface, but the subscription service is not helpful?
Oh, that looks very promising indeed! Thanks bro 👍 I'll dig deeper tomorrow

Wednesday, December 18, 2019

I can confirm that using the DEALER socket fixed my issues.
👍 2

Thursday, December 19, 2019

vdeurzen16:14
Is there a released version of libbitoin that has the header-chain-first and continuous parallel block sync ?
9 replies · evoskuil, neillm, vdeurzen · last Dec 21, 2019
evoskuil19:41
No
neillm00:12
Not yet 🙂
vdeurzen10:17
Is master in a 'correct' state, minus unknown unknown issues?
evoskuil11:54
Yes, all issues are logged, it seems to work :).
vdeurzen11:55
Ok, so, if I want to have the newest sync updates, I could use master?
vdeurzen11:55
Also, awesome work!
evoskuil22:38
yes
evoskuil22:38
thx
evoskuil22:39
still grossly suboptimal in some ways... but the architecture is there to be explosively better.
H
Harry17:29
Hello! I’m having some issue compiling a program that uses libbitcoin. I used the install script that libbitcoin provides however when i try to compile this program i get No package 'libbitcoin-system' found
(edited)
1 reply · neillm · last Dec 20, 2019
neillm00:13
Are you using version3? If so, you need libbitcoin and not libbitcoin-system

Saturday, December 21, 2019

f, this link wend dead, still here: https://libbitcoin.dyne.org/libbitcoin-manifesto.pdf
(edited)
👍 1
should incorporate into new site.

Sunday, December 22, 2019

neillm13:52
^ @evoskuil Not sure if this is the site you meant where it should be added, but: https://github.com/LibbitcoinInstitute/libbitcoininstitute.github.io/pull/5
I saw some work was done to try to revive (or capture?) the content of libbitcoin.org. Does anyone know the status of bringing that back, or replacing it?
1 reply · jungly · last Dec 22, 2019
jungly14:57
We ressusicated libbitcoin dot info. as a github pages website.
👍 1
N
narodnik21:32
@neillm you guys need possession of libbitcoin.org domain name? My friend was formerly in control of that... did we transfer ownership already or not?
4 replies · evoskuil, neillm, narodnik · last Dec 22, 2019
evoskuil21:39
I asked several times, but didn’t get it, and the .org site has been dead for about two years. I’ll happily take it and point it to the same location as libbitcoin.info.
neillm22:19
@narodnik Yep, that works ☝️
N
narodnik22:20
ok the guy just had a baby so maybe is afk for this reason, but when I see him, I'll get this sorted
👍 3
neillm22:20
👍 Thanks, appreciated!

Thursday, December 26, 2019

N
narodnik06:41
@evoskuil they're implementing the rust borrow checker in C++... it's been made part of the core guidelines. You should enable this feature in MSVC https://devblogs.microsoft.com/cppblog/lifetime-profile-update-in-visual-studio-2019-preview-2/
C++ Team Blog
The C++ Core Guidelines’ Lifetime Profile, which is part of the C++ Core Guidelines, aims to detect lifetime problems, like dangling pointers and references, in C++ code. It uses the type information already present in the source along with some simple contracts between functions to detect defects at compile time with minimal annotation.