July 2019

Not sure, but I think actually my server is slow and shows "connected outbound channel" most of the time instead of doing the job. my blockchain folder is actually about 80 GB. my log snippet https://pastebin.com/HamJGpiJ maybe everything is ok and I interpret the output wrong?

If you're syncing, you could try setting the outbound connections to something lower like 2. Inbound should be 0.

@neillm +3-4 GB after 2 hours. it sounds like a very poor result

Peer selection is somewhat random, so that result could vary widely.

@neillm and what about this one https://pastebin.com/2sENPL5C ? It looks like the validation takes a lot of time.

What's a long time? I see validation of 100 blocks in about a minute or so?

Oh! These are certaint chakpoints and not individual blocks? That is reassuring :)

Right, the block (download) and validation are displayed in batches of 100 during those heights. It'll show individual after you're sync'd

thanks

No problem. That said, performance is not yet optimal for v4.

Hey @evoskuil today I read the latest version of Mastering Bitcoin

Agreed, I liked it as well.

@narodnik got that started long ago.

amazing work

The book is well written. I was surprised too.

@narodnik - Chapters 4 through 6 of Mastering Bitcoin can be improved even more with the libbitcoin examples contained in slides 1 - 42. As you and Eric already know, instead of talking around things, bx provides most of the means to do things concisely and quickly.

You can go a PR into the book, as at least three of us have done in the past. But sometimes lower level examples are intended.

Good point, but suggested changes will need to be added incrementally and slowly. Will probably start such an endeavor this Fall.

@skaht That's a lot of great stuff!

Glad you enjoyed! Was built upon a framework laid by Titans… Still need to get slides #72 forward to be fully functional, and refactored to minimize details needed to be understood by technical readers.

Could I ask a favor?

The worst I can say is no.

🙂

@evoskuil - Is v4 to support the synthesis of P2WPKH addresses?

We've done the base work (base32) but I don't think the reset if complete yet. @neillm?

v3 bx commands can be applied to synthesize P2SH( P2WPKH ) addresses.

@evoskuil https://github.com/libbitcoin/libbitcoin-system/pull/1118 and https://github.com/libbitcoin/libbitcoin-explorer/pull/627 are proposals

:)

@neillm - Cool! What stable version of https://github.com/libbitcoin/libbitcoin-explorer/blob/master/install.sh do I need to execute to compile and utilize v4 bx ec-to-witness functionality?

@skaht There's no stable version, it's just a PR based on top of v4 master

If everything at https://travis-ci.org/libbitcoin is green, should that improve my chances of building off of the head?

Not sure what you're asking. v4 vanilla should definitely build (that's what I consider head)

v4 should be fully functional and of good quality. The main work to be done is in blockchain.

The master maps to v4, right?

Yes, if you are working in libbitcoin and bx it should be close to production.

Thanks…

Please post issue

See https://github.com/libbitcoin/libbitcoin-explorer/issues/630

@neillm after cloning https://github.com/thecodefactory/libbitcoin-explorer.git and checking out the witness-address branch, is there anything else I need to do to get ec-to-witness to compile? % ./install.sh --disable-shared --build-icu --build-zlib --build-png --build-qrencode --build-boost --build-zmq --build-dir=/Users/Scott/Projects/LibBitcoin/libbitcoin-explorer/build4 --prefix=/Users/Scott/Projects/LibBitcoin/libbitcoin-explorer/build4/core Builds but % ./build4/core/bin/bx help provides no details about ec-to-witness

You have to edit the install.sh to pull from thecodefactory and the witness-address branch for system/explorer, rather than libbitcoin and master. Just having them checked out locally won't do it (using install.sh anyway)

@neillm - got it working. Had to tweak 2 lines at the bottom of the master bx install shell to point to two of your repos. 1. build_from_github thecodefactory libbitcoin-system witness-payments $PARALLEL ${BITCOIN_SYSTEM_OPTIONS[@]} “$@” 2. build_from_travis thecodefactory libbitcoin-explorer witness-address $PARALLEL ${BITCOIN_EXPLORER_OPTIONS[@]} “$@” % echo 0279BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798 | ./build4/core/bin/bx ec-to-witness bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4 Matches the test vector from https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki#examples

Do we have those test cases in our unit test suites?

@evoskuil https://github.com/libbitcoin/libbitcoin-explorer/pull/627/files#diff-8d92b609d4ac40f23a209f185feefa8fR27

👋 everyone. thanks for all the hard work everyone’s put in on this project!

Greetings

We use libbitcoin in our app, and are currently under review to have it posted on bitcoin.org. It was brought to us that we aren’t using a CSPRNG, rather libbitcoin’s native PRNG. Looking into it further, I found a way to get secure data from Libsodium, but I wasn’t sure how to go about adding it via PR to the project (or if it would even be a solution that should be added)

If possible, a PR would be best. We could review it and add the libsodium dep if needed. Or do you mean, a PR isn't possible because you're not the one that integrated it? I would think a method similar to pseudo_random in the utility(.cpp) should suffice(?)

(particularly the response in https://github.com/libbitcoin/libbitcoin-system/issues/940 that triggered me going down the path to using sodium’s random number generator)

Sure. So as mentioned we have the pseudo_random code here: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/utility/pseudo_random.cpp ... adding a proposal for something like sodium_random.cpp or something is ok for review. Adding libsodium as a dep to the installer may be a bit trickier, so we could help with that if you write code that assumes it's already available

Ah, yes, good idea. I’d be happy to do that!

Whether or not libbitcoin is actually the best place for a 'secure random' is debatable, so that would have to be resolved in review/discussion as well. But in the meantime, PRs are always welcome to get that kicked off!

Also, small side question if I may: We are using libbitcoin at tag v3.5.0, as that was the latest release tag, and follows similarly to what Aaron Jaramillo used in his tutorials. It’s over a year old; will newer releases be tagged? or is it generally suggested to use master?

master is the current staging area for version4. It will be tagged as v4.0.0 when ready. I'd say to decide where you want to place the PR. Perhaps you want to add the code to the version3 branch if that's what you're using. We could forward port if needed

Awesome, thank you so much!

No problem

I would discourage the integration of any PRNG algorithm into the code for three reasons. (1) libbitcoin passes this call to the operating system (through boost I believe). This allows integration with hardware random number generation. (2) Software cannot produce random numbers, so we discourage the use of even our own single API for this in any security context relating to bitcoin. Even bx allows all seeds to be input as arguments. We do not want to imply otherwise. (3) It is more code to maintain, and non-trivial.

I generally agree, but I'm not sure PRNG has the option for hardware random number generation, since at least on systems like GNU/Linux, it sources from a pseudo-random pool (/dev/urandom). libsodium claims it has the hardware true random capability if it's available. Either way, I think point 3 is the most important in some sense, as if we provide something that people believe is true random when in fact, that can't be provided in a general configuration, it's a maintenance/support/security burden taken on for no concrete benefit.

The proper resolution to (1) would be to simply change our source if that is the case.

If I followed the code right, it looks like pseudo_random::fill eventually calls out to a Mersenne Twister algorithm, which is not cryptographically secure. https://en.m.wikipedia.org/wiki/Mersenne_Twister

That's correct. The method is pseudo_random for that reason (to be explicit)

Right.

One point @evoskuil was making is that there are no secure random software methods. They must rely on hardware modules, which may or may not be available on a consumer computing system.

Ah, gotchya.

That said, some libraries (such as libsodium and perhaps boost, though not sure about the latter) may provide a way to obtain true random from a hardware device on the machine if it's available. If there's a way to utilize that in boost, we could simply change our code to use that path instead. I don't know if that's possible or not at the moment though

Looks like random_device might be of interest. https://www.boost.org/doc/libs/1_46_0/doc/html/boost/random_device.html. I'm on mobile now but can dive deeper later.

That's also pseudo_random. I think it's used in the code path we have now too

Hmm, are you sure? https://stackoverflow.com/questions/23372245/how-do-i-use-boostrandom-device-to-generate-a-cryptographically-secure-64-bit

That says that it will use the OS's random, which is what we do. It's true that the random_device can specify /dev/random, instead of /dev/urandom, which helps, but I don't know that people generally consider that a cryptographically secure source of entropy (though it is widely used as one)

Yes, not possible for logic to be random.

This is quite interesting. Thanks for walking through this with me. @neillm do you think then that a PR utilizing sodium is even worth it, if it effectively does the same call to the OS's random?

I would think that if it utilized the hardware pass through (say for a particular hardware module) and was driving a concrete use-case (on some established installation base), it would probably be ok eventually, rather than requiring that vendor to maintain patches on top. If it's just being added so that others can use it and believe they're getting true random, as opposed to pseudo-random, when in fact they may be getting pseudo-random data, then no -- because that's somewhat dangerous.

Ok cool, good input. I'll discuss this with my team to see how we want to proceed.

Yes, non logic hardware.

Heh, I can already imagine someone selling a hardware random module that is basically GNU/Linux on a USB device that spits out the local /dev/urandom 😛

HRNG is a great place to back door.

A lot of people don't get this type of thing, that's why we removed all internal use of PRNG.

It is worth making the point that 25 years ago most all PRNGs were anything but random. Perform a modulo %128 operation on the stream of random integers generated by the PRNG and map the resulting modulo integer to an ASCII character and print to a screen. It won’t be long before a recognizable character sequence repeats, not comforting at all…

Circling back on this, I’m trying to dive into where Boost is called for the entropy, and I can’t find it (i’m on the v3.5.0 tag). It appears that pseudo_random_fill just gets a std::uniform_int_distribution<uint16_t> distribution(0, max_uint8); uniform distribution, then passed it to the twister, which just gets seed from the clock, now.time_since_epoch().count(). Is this what you were referring to about passing to the OS for obtaining entropy? And is this method considered secure enough? (just trying to learn as much about this, as I also have to give this info to my team, so by no means do I mean this in a negative or criticizing light)

I was guessing about boost. Really anything is secure enough for pseudorandom.

@BJ Miller The following links may help clarify: https://wiki.archlinux.org/index.php/Random_number_generation and https://www.boost.org/doc/libs/1_46_0/doc/html/boost/random_device.html

More related boost info: https://www.boost.org/doc/libs/1_59_0/doc/html/boost_random/reference.html

Thank you

That’s great information, but what I’m still struggling to find is where in libbitcoin a call to Boost is actually occurring for random_device or boost_random. The only thing I can find is getting the clock time (like you’d mentioned above) and sending that into the Mersenne Twister. Am I missing something?

You probably have to search in boost. The uniform_int_distribution uses the PRNG, which may be what you're looking for.

Scratch that, realized we're using std:: random stuff there. So search std::, not ::boost

Hi can anyone help, how to get wig key from child xprv key?

@Artem I don't understand your question. You may want to look at this as reference: https://github.com/libbitcoin/libbitcoin-system/blob/master/test/wallet/hd_private.cpp

@neillm how can i create wif format key, then i have xpriv key?

I still don't understand the question. If you have an xpriv, look here for deriving private keys: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/wallet/hd_private.cpp . If you want wif encoding, look at the private key class: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/wallet/ec_private.cpp#L160

^ The above question/issue was resolved in private messages ...

@ Artem - 1% echo xprv9s21ZrQH143K3t4UZrNgeA3w861fwjYLaGwmPtQyPMmzshV2owVpfBSd2Q7YsHZ9j6i6ddYjb5PLtUdMZn8LhvuCVhGcQntq5rn7JVMqnie | bx hd-to-ec b2a0d576b828b537688b561f2cfa8dac3602d54c62bde619ad5331e6c235ee26 or explicitly demarshalling 2% echo xprv9s21ZrQH143K3t4UZrNgeA3w861fwjYLaGwmPtQyPMmzshV2owVpfBSd2Q7YsHZ9j6i6ddYjb5PLtUdMZn8LhvuCVhGcQntq5rn7JVMqnie | bx base58-decode | cut -c 93-156 b2a0d576b828b537688b561f2cfa8dac3602d54c62bde619ad5331e6c235ee26

Hi can someone help me, i trying to add libbitcoin to my project build via cmake Any examples?

@pmienk ^ ?

@Artem subsequent projects which depend upon libbitcoin-system itself (ie libbitcoin-client/libbitcoin-server/etc) provide a model for including the dependency.

@pmienk ty

Quick update: mainnet1/2 and testnet1/2 are currently down. Might be a while before restored

👏🏾 👏🏾

mainnet1/2 are back online, but may take some time to catch up

Just realized that the version3 branch is a significant number of commits ahead of the previous release. For better or worse, I just tagged it where it is and bumped the release version.

I do this periodically. It's a bit of a process, so I had let it slide. But it's well past time. Normally I would create and sign new binaries for each release and uploaded them to github on the release pages, bx, bn and bs. I don't see any reason to continue this with bn, since it's really just a developer tool. There's also a page that lists releases.

Even just tagging the releases through github is a bit of a process, as just tagging repos is easily scriptable in comparison. Where do you generate the binaries? Or rather, if I publish the releases, can those be uploaded/copied in?

I generate them on my three platforms (windows, ubuntu, macos).

Ah. Do you do this all in one go (i.e. your process for publishing), in which case it's better I just back things out?

Generally I do, butnot strictly necessary, as long as the builds match the tags.

Considering I don't have ready access to all the platforms, would you be able to kick off some version3 builds to add the binaries?

I realize 🙂

I haven't done the binaries in a couple of releases. Those can wait. I've got IT tasks before I can crap out release binaries. As long as you come up with tags consistent with the old ones, and update the version histories, it's good with me.

If we ever get the win cross compile going, I'd be up for looking into travis or something like buildbot for automated binary generation

That would be awesome.

The server and explorer release descriptions are more extensive, but mostly pertain to the binaries. I wonder if it could just be tagged now for consistency and the description could be updated at the time of binary upload 😉

What descriptions are you referring to?

Those release descriptions start with something like "These binaries require no installation, simply download and run the single file. Mainnet and testnet are supported based on configuration differences. You can verify the downloads using the signed SHA256 hashes below."

Those are only associated with the binaries.

Oh, didn't realize, when I click releases, that's what I see. Yes, I see what you mean now though

I don't think you can add a description without the binary, so it's just the tag text.

I've been using github for years, but I've still not mastered all of it 😉

I had a whole process for this, with some partial automation.... glad to have to give it a shot :))

True, and that's a very good thing regarding maintenance/configuration!

huge

I borrowed that!

Yeah, we could probably do better with the descriptions, but up to you.

Ah, most repos appear to be tagged at Feb 2018

All should be the same date/version.

Yep

3.5 was < 2 months after 3.4

Ah, ok, fair enough. Summarizing from Feb 2018 to now is a bit difficult at the moment, could be done. I'd hope if we stay on top, it will be a bit more reasonable/fresh in mind though

This hasn't been updated: https://github.com/libbitcoin/libbitcoin-explorer/wiki/Version-History

Ah, I was thinking those were just binary updates 😛

Those should include the tag text and be updated. I don't think I stopped intentionally, maybe just forgot about them.

@channel Are there any volunteers that may have some time to update some version history documents (mostly wiki entries)? Seriously, @evoskuil or @neillm will help 🙂

Anything outstading for 3.6 should prob be advanced to 4.0

Oy, lots I hadn't thought about before noticing this gap ...

Release control is a job.

True, I've seen it be ... didn't know what was involved in this at the moment

I got spread too thin and some of this stuff fell thru the cracks. Thanks for picking it up.

Haha, I don't know that I volunteered for the full extent, but perhaps getting started anyway. Would be great if some others chime in though

Tags (with text), version pages pasted from tags, triage of issues, binary builds and signing, upload of binaries and release page text... that's the whole deal.

I plan to do a bx command update pass for version4, as several things have changed and there are new/deprecated commands. While that should also be minor release centered, I think of it more as major release

Adding new commands is no harm, and removed ones can remain, as they should be redirtected with comment text within BX (if we did it right). Changes are more of a hassle.

I don't know if this is 😂 or 🤣, but it's certainly 🤔 and 😲

Then building version 3 from cmake /Users/artem/buildlokotko/libbitcoin-system/src/math/elliptic_curve.cpp:24:10: fatal error: ‘secp256k1_recovery.h’ file not found #include <secp256k1_recovery.h> ^~~~~~~~~~~~~~~~~~~~~~ 1 error generated. make[2]: *** [CMakeFiles/bitcoin-system.dir/Users/artem/buildlokotko/libbitcoin-system/src/math/elliptic_curve.cpp.o] Error 1 make[1]: *** [CMakeFiles/bitcoin-system.dir/all] Error 2 make: *** [all] Error 2

@pmienk ^ Something about not picking up the --enable-recovery option, or something else?

i dont understand difference between ./install.sh and ./autogen.sh $ ./configure $ make $ sudo make install $ sudo ldconfig

install.sh installs all required dependencies and then all required libbitcoin repos, rather than the latter that you posted, which builds a single repo, perhaps with missing dependencies (not included on the configure line for example)

hi. my server doesn't start anymore. it has always worked very well. the server was syncing (validation at block ~400 000). I am starting it now and it doesn't start anymore. Nothing has been changed since the last successful start. I use the version 4 (it was created about 2 weak ago from master). https://pastebin.com/natkxbbQ

How was the last shutdown? It looks like it was not cleanly shut down and therefore corrupt (detected on startup).

@neillm I believe the shutdown was clean (as always ctrl+c). But I cannot insure it 100 %. Is there a possibility to clean it now?

Of course, but unfortunately the clean step is rm -rf blockchain; bs --init 😞

@neillm Is that also the case in version 3?

Yes

Was that somehow recorded (github issue)? the user experience is not the best. From the user's point of view, I would expect the error message is more meaningful, that the message tells me what I have to do... or much more, that the corrupt state can be restored. Is this somehow planned for the future?

Not that I'm aware of, no.

The proposal with the backup sound good. I will definitely do that next time.

Were you running a sync on v3 or master/v4?

master/v4

Ok. The v3 snapshots may be more worthwhile because master/v4 snapshots may change in time before it's finalized. Just a heads up.

I probably have to expect that future changes in master will destroy my blockchain.

Well, certainly not all changes, but anything database related. It's been a while since it's changed though, but I expect some changes before release.

@neillm: A little question. You have written "boost PRNG by default uses entropy from /dev/urandom but has the ability to use /dev/random, which is a better source of OS randomness". better than what?

/dev/random uses OS level entropy (sourced from hardware events). /dev/urandom is pseudo-random that is less random than /dev/random, but it provides quicker access to pseudo-random values. If you try to read data from /dev/urandom, you can read large quantities. If you try to read data from /dev/random, you will find that you are often blocking, waiting for the system to provide more entropy

@neillm I am not an expert I can only try to reproduce what I have read in other sources but I believe you are wrong. The only relevant difference between random and urandom is that the first one doesn't block and urandom blocks. Your article confirms it. Ok, maybe not quite, the is "Not all operating systems implement...". But I am talking about the linux kernel (>4). Electrum uses urandom because python crypt function uses urandom. In spesmilo/electrum there was an extremely long debate there with many myths and discrepancies but I think the conclusion was blocking/non blocking. the python's function used earlier random but there was also a debate there and the implementation changed from random to urandom. I think the main reason was a security aspect - urandom blocks unless not enought entropy, but the entropy source is the same with random, not different.

We can disagree, but if the entropy is unavailable and one waits for it and the other doesn't, it sounds like it's confirmed.

"it sounds like it's confirmed." - what do you mean with that?

You are confirming what's stated in the link that I posted. To continue the quote I pasted earlier ... "This means that the call will not block, but the output may contain less entropy than the corresponding read from /dev/random." If the goal is random, more entropy usually equates to more random.

@neillm I have twisted urandom with random and now I understand what you wanted to say :). random blocks and of course when not enough entropy the urandom is worse. sorry!

No problem. Regardless of which entropy source is a user's preferred method, libbitcoin's stance has generally been to never use system generated random values for anything related to storing coins, which is why the model chosen is to allow a user to provide their own entropy. If someone believes /dev/urandom is "good enough", they can pass in generated values on the command line. Others may believe that dice rolls are "good enough", so those can be passed in too. In other words, regardless of what I think about the GNU/Linux sources of randomness, libbitcoin doesn't necessarily prefer either where it matters.

@neillm I agree.

@neillm I have removed all files in my installation directory. the initialization was successfull but contacting seed doesn't work. should I try later? https://pastebin.com/aLvyai4a

@ldz All mainnet servers are down at the moment. Or rather, mainnet1/mainnet2 are up, but are still catching up and should be open to the public soon. I'll make mainnet2 public shortly, so you can try again. It's nearly caught up ...

@neillm works, thx

Quick update: mainnet2 is back online, caught up and running as normal. mainnet1 is still catching up.

thanks for the info

Somewhat surprisingly, I had a UPS fail during a brief power outage, so a complete restore of both machines was necessary. I have a replacement UPS now, but haven't set it up just yet. I plan to soon, perhaps tomorrow.

isn't the blockstream liquid sidechain prone to state censorship?

@narodnik I'm not as familiar with it as I'd like to be, but I believe that it is. From what little I know about it, the use case is high-volume businesses and large exchanges. Don't quote me, I should read more about it. Is there a claimed use-case for individuals (as an alternative to Bitcoin)?

From what I understand, the Liquid sidechain is a permissioned network where node operators essentially sign blocks. A block is considered final once 2/3rd of the nodes have signed a block. It is a pretty straight forward majority based consensus algorithm, with no byzantine fault tolerance because you trust the nodes. And since you trust nodes (which I think will largely be exchanges and financial institutions) they will be pretty obedient of the state, unless of course they chose to do whatever they do, fuck things up and then ask for a bail out. But I digress. I think Liquid and other sidechains (if any follow) will make bitcoin more accessible to financial institutions, so in all this is great stuff, if you want that. But once you peg your bitcoin off the bitcoin blockchain into a sidechain, you are completely in state/fintech land.

mainnet1/mainnet2/testnet1/testnet2 are restored, but I'm going to bring them down momentarily to swap out the failing UPS. I expect downtime of the services to be 30 minutes or less ... starting shortly

Ok, everything is back up now 👍