September 2019

@jungly I've been thinking that an easy compromise solution for the store's hard shutdown issue would be to implement an alternative database driver. The database itself is trivial, just a small set of templates. We even substitute it in text cases as you know, using just a std::vector. The memory map is optimal for server scenarios - specifically rapid query, therefore optimized for mining and validation. Providing a database driver would allow for compromise in the desktop/wallet scenario where these performance characteristics are not required.

Thanks @evoskuil. I really like the drivers idea. It gives us some freedom to experiment as well. I went down the rabbit hole of in memory databases, reading a bunch of papers from here - https://15721.courses.cs.cmu.edu/spring2019/schedule.html It's amazing how much work is being done in the area. It seems Microsoft's hekaton SQL server engine is doing a lot of excellent stuff. We can pick some good ideas from these projects, if and when needed. The drivers approach is really cool. Will give us an easy way to experiment. Btw, I also ran a small experiment of measuring the amount of data going to disk per second during IBD. Want to plot it on a chart before sharing.

My last work in blockchain was block caching during sync, so that a large memory machine could cache blocks received out of order up to the size of the entire chain, eliminating the block read required for validation purposes. The other remaining optimization to complete is minimizing the scope of confirmation updates.

So you will see much more reading and some more writing now than is ideal.

I didn't realise you were open to the idea of using an integrable store. I can check if any of the existing in memory stores are 1) stable enough, 2) do what we want and 3) have a license we can use. Though I also think, our use case is so application specific, we really don't need all the bloat a generic store will come with, as they try to be generic, by definition. It might be interesting to measure how using such a store performs or if they are usable at all.

Really all we would be after is the shutdown guarantee (to the extent that is possible).

That really is the first and the most important goal.

I believe it's the only advantage we require.

Get that thing to shutdown restart properly, without to much of a slowdown :)

Yup, I believe we will always outperform on other measures.

👍🏾

At least on an SSD.

Yeah. That's an assumption I am convinced is totally sane.

does this URL: https://libbitcoin.org reachable?

@RaGa, timeout for me

been getting that for quite a while....

What were you looking for?

for the website

What information were you looking for? I'm not sure what's on libbitcoin.org as I thought everything was hosted in github. I was hoping to be able to direct you to something useful for your purposes.

Mostly looking for what applications have been built using libbitcoin toolkit

that's way out of date, not the same

@pmienk TY, Watched the Airbitz video,I have started out with bitcoin during my spare time, so things are bit distorted to me now.....may be i should focus on understanding the satoshi's reference implementation first.

👻

@RaGa I guess it's all about what your goals are. In general I would argue for a logical understanding of the protocol independent of an implementation. What specifically are you trying to do?

If you want to understand Bitcoin generally, libbitcoin is much easier than the Satoshi prototype. The website doesn’t contain any information you would need, it’s all in github. Info about libbitcoin is also in the Bitcoin Wiki: https://en.bitcoin.it/wiki/Libbitcoin

👻

primarily I want to contribute as I see bitcoin is kinda revolutionary, so will try to make some contributions, plus, will help other people ( who are new to bitcoin or never heard of bitcoin) to understand what it is or does.

Well, I know we would welcome contributions. I agree with @evoskuil that I believe that our codebase is more readable as it was developed with the purpose of providing a developer api as opposed to the prototype

What do you mean by admittedly bias?

It also puts a much strong emphasis on readability and code self-documentation.

well, i've contributed to this one, so it's not surprising i think its easier to understand

Libbitcoin also avoids a lot of unnecessary implementation optimizations that have been implemented in the Satoshi prototype to overcome architectural issues.

👻

ok, thanks I will catch up.

@evoskuil did you see they are making new hard drives which are key-value stores (in hardware)?

Way cool

Hi, Hope thing are going good. I've been building libbitcoin-system repo, faced couple of issues, resolved some and finally stuck here CXX examples/examples_libbitcoin_system_examples-main.o CXXLD src/libbitcoin-system.la copying selected object files to avoid basename conflicts... ar: u' modifier ignored since D' is the default (see U')` CXXLD examples/libbitcoin-system-examples src/.libs/libbitcoin-system.so: undefined reference to secp256k1_ecdsa_recover'` src/.libs/libbitcoin-system.so: undefined reference to secp256k1_ecdsa_sign_recoverable'` src/.libs/libbitcoin-system.so: undefined reference to secp256k1_ecdsa_recoverable_signature_serialize_compact'` src/.libs/libbitcoin-system.so: undefined reference to secp256k1_ecdsa_recoverable_signature_parse_compact'` collect2: error: ld returned 1 exit status Makefile:2438: recipe for target 'examples/libbitcoin-system-examples' failed make: *** [examples/libbitcoin-system-examples] Error 1 I've secp256k1 library in /usr/local/include/ and headers are stashed at /usr/local/include/include/ Changed the Makefile like below secp256k1_BUILD_CPPFLAGS = -I/usr/local/include/include secp256k1_CFLAGS = -I/usr/local/include/include secp256k1_CPPFLAGS = -I/usr/local/include/include secp256k1_ISYS_CPPFLAGS = -isystem/usr/local/include secp256k1_LIBS = -L/usr/local/lib -lsecp256k1 Any help is much appreciated.

@RaGa https://github.com/libbitcoin/libbitcoin-system#autotools-advanced-users

Thanks, What do you think the problem I am facing? I really don't wanna install libbitcoin, yet wanna compile source, clearly I am getting linker error

Looks like you're missing the --enable-module-recovery configure option on secp256k1

I did build libbitcoin/secp256k1 with that switch too

Then it's not being detected in the proper location, or is finding a version without it built

Yup, that’s definitely the issue.

I did make clean, re configure on both...

This is a matter of where you installed to, and if it's being detected

Nevertheless, that’s definitely the issue :)

ok, I will check

If you add --prefix=/some/dir on both secp256k1 and then do make install and do the same for libbitcoin-system, it should work

Or just use install.sh :)

hold on, should I install secp256k1?

It's required by libbitcoin-system, so yes

that is the part I missed, because I thought it is optional

Check that link I sent, it's all stated clearly 🙂

"A minimal libbitcoin build requires boost and libsecp256k1. The libbitcoin/secp256k1 repository is forked from bitcoin-core/secp256k1 in order to control for changes and to incorporate the necessary Visual Studio build. The original repository can be used directly but recent changes to the public interface may cause build breaks. The --enable-module-recovery switch is required."

@neillm and @evoskuil Thank you, built successfully!!

Gotta read the readme.

OK

@Mlmikael check your PM

is mbetls a new dependency of libbitcoin-protocol?

I believe it’s a conditional dependency, for ssl/tls support.

dependency of which package? only libbitcoin-protocol? but it is also a part of libbitcoin-system's install.sh

I don't believe you're correct. The only reference to mbedtls in -system install.sh is a check against an undefined MBEDTLS_ARCHIVE variable.

@pmienk I didn't say mbetls was a dependency of libbitcoin-system :). I just wanted to know which libbitcoin packages need this dependency and that it is present (mentioned) in the install.sh script of libbitcoin-system

well, the question was 'dependency of which package?' But to resolve this, I believe mbedtls was included to facilitate an integrated web server. I believe if it's utilized that would be by -server

https://github.com/libbitcoin/libbitcoin-protocol/blob/master/install.sh Script options: # --with-mbedtls Compile with MbedTLS Support

@pmienk is correct that it's required for SSL support on the server. It's in protocol because that's where the code is used, but server is the user of that.

its possible that all functionality could be accomplished at the protocol layer

If you want to run the server without ssl support, it's not needed

@pmienk I think it’s all in -protocol, but the server install has protocol as a dependency on -protocol, so it is incorporated into the server install. Unless I misunderstand what you meant by “protocol level”.

@evoskuil I hadn't seen neil's message and was speculating that all changes were in libbitcoin-protocol

Yeah, should all be in there except inclusion of the full decency graph in the upstream installers.

At least we got rid of the web dependency, with Neill implementing it. I’m planning to rebase it on ASIO, which would greatly simplify it, if I can get my head around it again.

👍

I'm trying to understand the libbitcoin's dependency chain. I will talk about the client stack. Libbitcoin-explorer has libbitcoin-system as a (indirect) dependency, which in turn has another (optional) icu dependency. For this reason I can build libbitcoin-explorer with icu option, it makes sense. Let's say I just want to build only the libbitcoin-network. The libbitcoin-network build (same for protocol and client) doesn't provide the icu option. Do I understand it correctly, that it was meant that way? In other words the libbitcoin-network (as well as protocol and client) doesn't use any icu features of libbitcoin-system and that is the reason I shouldn't build libbitcoin-network with icu option, right? (ICU was just an example, you can replace it with another optional dependency like png).

@ldz the install script does not facilitate that option, correct.

Yes, that's correct. For ICU in particular, only system and explorer use features from it (optionally). So while there are optional method in -system, -explorer is the only other part of the stack that needs those (also optionally).

thanks

I don't expected network would ever use localization features. These are generally restricted to user interface, so bx and supporting functions in system would probably remain the only usage.

While network may not use localization, a developer utilizing network may wish to without depending upon explorer. The existing install.sh is not equipped for the development scenario.

Which installer is that, system? The dependency is optional.

Network, as it will overwrite any installed system library

There is no requirement to install into as a system library, and it's not recommended.

If I install to a prefix with the intention of developing a product which requires localization features of libbitcoin-system and libbitcoin-network no install.sh present provides me that functionality without also installing something bigger in the prefix ala libbitcoin-explorer

Are you saying that network should expose the system option to install ICU?

Given that our continual refrain is "use the install script", I am solely pointing out that it is insufficient without explaining the need to over-install or use a different procedure

You are correct that the installer is not intended to create a general purpose dev environment. That can be done by installing the repos independently using conventional methods.

Total agreement.

The installer serves four purposes. (1) reliable CI builds, (2) reliable end-user build, (3) reduces support load by demonstrating that the build succeeds, (4) memorializes the various workarounds necessary for developer to succeed.

The recent pull request by ldz has me again looking at the build project. I may try to expand what I've already done to expose the dependency issues in my own setup after I wrap on the database changes for 157

Hi perhaps I am misunderstanding something, looking at elliptic_curve.hpp there is a lot of good stuff about signing, however is it possible to public/private key encrypt and decrypt data with this library too?

I am planning to use this to transfer keys and small blobs of data, also what is the maximum size that the key can be?

All our elliptic curve work uses secp256k1 (https://github.com/bitcoin-core/secp256k1) which I don't believe supports arbitrary encryption of data.

Thanks I came to that conclusion to that too, I may have to look at the openssl implementation, however I was trying to avoid that, do you know if the two systems are easily compatible when sharing keys?

I don't recall anything regarding openssl other than it was a large dependency we wished to avoid.

isn't there AES symmetric encryption in libbitcoin?

void aes256_encrypt(const aes_secret& key, aes_block& block) void aes256_decrypt(const aes_secret& key, aes_block& block)

encrypted_keys uses these ala BIP38

@evoskuil and @neillm - If libbitcoin bx HD tech is ever enhanced take a quick look at https://github.com/satoshilabs/slips/blob/master/slip-0010.md. The idea is to make BIP 32 to not only implicitly support secp256k1 elliptic curve but to support other cryptographic curves such as ed25519 and p-256. This request was driven by Prusnak’s answer to https://github.com/monero-project/monero/issues/5744