Bitcoin full node and query server

December 2024

Archive of the libbitcoin Slack #general channel.

Tuesday, December 3, 2024

evoskuil05:46

I had a typo in a constant that was preventing the shani rounds from functioning correctly, but that is all resolved. There's a significant efficiency loss somewhere, likely in the way I've fit the sha intrinsics into the existing template structure. I'll be working to track that down, but it is functional at this point.

(edited)

🎉 1

Wednesday, December 4, 2024

evoskuil02:39

I took a bit of a detour today on additional sha vectorizations. I added the vectorization of endianness conversions and or k-adding. I recently moved k-adding to its own function with additions enumerated across the buffer, but there was a bug in the vectorization. I tracked that down today and was able to enable this and the vectorized endianness conversions. These save approximately a total of 5% for smaller hashing runs, declining as the size of the buffer increases. They also save 1-3% of Merkle hashing runs, again with the primary benefit on the low end (this makes sense as they only affect the 1/2/3 lane merkle hashes (unvectorized)).

(edited)

02:42

These numbers are without avx512, which doubles the vectorization lanes, since I don't have a test platform for it yet.

evoskuil02:48

We are able to take advantage of several vectorizations that Core and typical hash libraries cannot, because we use buffered scheduling of the message, as opposed to variable rotation. This uses more memory, but allows significant net time savings through vectorization. The sha256 buffer is 4x the 64 byte block size, rotating. So that's not really significant. So we vectorize across the message buffer: sigma computations, endianness conversion, and k-addition but also vectorize across multiple message buffers (the other axis), and finally vectorize across independent hashes for Merkle (which Core does as well).

02:52

Even without avx512 (just avx2/sse4) this reduces non-Merkle hashing to 70-75% of the normal form (based on Core implementation) for larger amounts of data, and 90% for smaller data. And down to as low as 80% of normal for on small data due to precomputed padding.

02:55

This is all C++ (no C and no assembly) in a single template, which also includes the same optimizations for sha512 and sha160 where applicable.

02:56

I ran into an issue on clang (non-xcode) with the two new optimizations, so I've disabled those on that platform until these can be isolated. So now back to refactoring the shani implementation.

cguida16:19

Hey @evoskuil do you have that bn.exe binary you used to do the Windows benchmark? I got the same hardware and I want to replicate your result :)

evoskuil16:34

Sure, I'll build a current copy and run it to verify. What's your measured bandwidth?

🔥 1

36 replies · cguida, evoskuil · last Dec 11, 2024

cguida17:33

Bandwidth is 620Mbps according to fast.com

evoskuil17:35

Well that's only 27% of what I'm testing on, and since it's network bound on my network, it will be as well on the same machine on a slower network. So you should expect it to take around 3.7 hours.

🔥 1

cguida17:38

Excellent, that would still be a fantastic result!

evoskuil17:38

Lolz, feels so slow 🙂

cguida17:40

Haha it’s super fast for the country i’m in. Btw the machine cost US$530 here because of silly import duties as well. Again, 3.7 hours on a US$530 machine is still an amazing result, looking forward to confirming 🙂

evoskuil17:41

I'm waiting for it to complete. Once that's done I'll post you a zip with the executable. The command line is simply > bn

🙏 1

evoskuil17:41

You can reference a config file of your choosing by > bn -c path

cguida17:42

do you have a recommended config for this machine given the resources i have?

cguida17:42

haven’t used the windows cmd line in forever, might need to brush up haha

evoskuil17:48

It generates a rotating pair of log files named bn_end.log and bn_begin.log (the combined size can be configured). It starts with end and then rotat3es it over to begin and so-on. These are currently just an echo of whatever hits the console. The console and logs are buffering, so pausing the console doesn't really slow the node. The log will start with an entry that shows the name of the config file its using (or default) so you know if it has your config.

2024-11-24T19:39:57Z.0 ====================== startup =======================
2024-11-24T19:39:57Z.0 Using config file: "m:\\bn.cfg"

There's also some logging options dumped at the start of the log. Since log levels (and their message generation) can be entirely compiled out, this gives you some good info.

2024-11-24T19:39:57Z.0 Version Information...
libbitcoin-node:       4.0.0
libbitcoin-database:   4.0.0
libbitcoin-network:    4.0.0
libbitcoin-system:     4.0.0
2024-11-24T19:39:57Z.0 Press CTRL-C to stop the node.
2024-11-24T19:39:57Z.0 Log system configuration...
2024-11-24T19:39:57Z.0 [a]pplication.. compiled:1 enabled:1.
2024-11-24T19:39:57Z.0 [n]ews......... compiled:1 enabled:1.
2024-11-24T19:39:57Z.0 [s]ession...... compiled:1 enabled:1.
2024-11-24T19:39:57Z.0 [p]rotocol..... compiled:1 enabled:0.
2024-11-24T19:39:57Z.0 [x]proxy....... compiled:1 enabled:0.
2024-11-24T19:39:57Z.0 [r]emote....... compiled:1 enabled:1.
2024-11-24T19:39:57Z.0 [f]ault........ compiled:1 enabled:1.
2024-11-24T19:39:57Z.0 [q]uitting..... compiled:1 enabled:0.
2024-11-24T19:39:57Z.0 [o]bjects...... compiled:0 enabled:0.
2024-11-24T19:39:57Z.0 [v]erbose...... compiled:1 enabled:0.
2024-11-24T19:39:57Z.0 Please wait while network is starting...
2024-11-24T19:39:57Z.1 Candidate top [000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0].
2024-11-24T19:39:57Z.1 Fork point (0) unassociated (0).
2024-11-24T19:39:57Z.1 Loaded (10000) addresses.
2024-11-24T19:39:57Z.1 Bypassed seeding because of sufficient (10000 of 1000) address quantity.
2024-11-24T19:39:57Z.0 Node is started.
2024-11-24T19:39:57Z.1 Not configured for inbound connections.
2024-11-24T19:39:57Z.1 Create 200 connections 5 at a time.
2024-11-24T19:39:57Z.0 Node is running.

👍 1

evoskuil17:48

I use notepad++ to view logs, config and event files (text).

👍 1

evoskuil17:50

Windows Defender might flag the process. I don't think it's infected, but the process behavior is obviously a bit scary. If so you might need to give the file an exclusion or turn off Defender.

👍 1

evoskuil17:59

The recommended config for a milestone sync benchmark on this machine is the default config. There are a lot of (1) config settings, (2) command line options, and (3) runtime console options. Config settings are in an "ini" file format, and established using the --config or -c command line switch as described above. The --help or -h command dumps command line options:

>bn -c m:\bn.cfg -h

Usage: bn [-abdfhiklnrstvw] [--config value]

Info: Runs a full bitcoin node.

Options (named):

-a [--slabs]         Scan and display store slab measures.
-b [--backup]        Backup to a snapshot (can also do live).
-c [--config]        Specify path to a configuration settings file.
-d [--hardware]      Display hardware compatibility.
-f [--flags]         Scan and display all flag transitions.
-h [--help]          Display command line options.
-i [--information]   Scan and display store information.
-k [--buckets]       Scan and display all bucket densities.
-l [--collisions]    Scan and display hashmap collision stats (may exceed
                     RAM and result in SIGKILL).
-n [--newstore]      Create new store in configured directory.
-r [--restore]       Restore from most recent snapshot.
-s [--settings]      Display all configuration settings.
-t [--test]          Run built-in read test and display.
-v [--version]       Display version information.
-w [--write]         Run built-in write test and display.

(edited)

evoskuil18:00

Many of these options are for developer troubleshooting and store sizing. The --settings option dumps the entire set of config file settings and descriptions. It's a lot and again many of these are to aid tuning, but some are for general purpose use.

(edited)

evoskuil18:05

Okay, the sync completed to block 850,000 in 51 minutes.

🔥 1

evoskuil18:09

When the node is running, at the console you can enter various commands by hitting a letter key and then <enter>. The combination m <enter> brings up the console menu. These consist of toggles and options. Toggle varies the log levels that are compiled in (all except for objects in this build. I don't recommend leaving highly verbose levels turned on for a long time, it just wastes time.

2024-12-04T18:06:13Z.0 Toggle: [a]pplication
2024-12-04T18:06:13Z.0 Toggle: [n]ews
2024-12-04T18:06:13Z.0 Toggle: [s]ession
2024-12-04T18:06:13Z.0 Toggle: [p]rotocol
2024-12-04T18:06:13Z.0 Toggle: [x]proxy
2024-12-04T18:06:13Z.0 Toggle: [r]emote
2024-12-04T18:06:13Z.0 Toggle: [f]ault
2024-12-04T18:06:13Z.0 Toggle: [q]uitting
2024-12-04T18:06:13Z.0 Toggle: [o]bjects
2024-12-04T18:06:13Z.0 Toggle: [v]erbose
2024-12-04T18:06:13Z.0 Option: [b]ackup the store
2024-12-04T18:06:13Z.0 Option: [c]lose the node
2024-12-04T18:06:13Z.0 Option: [e]rrors in store
2024-12-04T18:06:13Z.0 Option: [g]o network communication
2024-12-04T18:06:13Z.0 Option: [h]old network communication
2024-12-04T18:06:13Z.0 Option: [i]nfo about store
2024-12-04T18:06:13Z.0 Option: [m]enu of options and toggles
2024-12-04T18:06:13Z.0 Option: [t]est built-in case
2024-12-04T18:06:13Z.0 Option: [w]ork distribution
2024-12-04T18:06:13Z.0 Option: [z]eroize disk full error

evoskuil18:10

You cannot really track progress from watching the log levels. There is a high speed event log that emits messages to a file sink (events.log) presently. You can use this to track the number of archivals (a block that's downloaded, checked, stored and indexed.

block_archived...... 551105 1217
block_archived...... 565941 1217
block_archived...... 563597 1217
block_archived...... 562631 1217
block_archived...... 571193 1217
block_archived...... 567873 1217
block_archived...... 589292 1217
block_archived...... 559177 1217
block_archived...... 572302 1217
block_archived...... 550289 1217
block_archived...... 587634 1217
block_archived...... 584910 1217
block_archived...... 564990 1217
block_archived...... 567301 1217

evoskuil18:12

Since these are out of order you should rely on the file line count to see the number of blocks downloaded. The default config is milestoned to 850k, and validate/confirm are disabled. Blocks get a tad bit slower above 850k, and also at the tail end of the chain just because there are fewer blocks remaining.

evoskuil18:14

The commands work at any time the node is running. <ctrl>-c stops the process normally, or you can also use c <enter>. If you type multiple or wrong characters at the command line, just hit <enter> to clear them, which you'll see at the console once it clears. Note that the node is highly asynchronous, so sometimes console actions are not immediate, but generally it's pretty quick.

evoskuil18:16

This is the i <enter> command at runtime (it provides a little more at the command line):

2024-12-04T18:12:49Z.0 Body sizes...
   header    :85577716
   txs       :4520884072
   tx        :69884209602
   point     :101072408544
   input     :390667618129
   output    :107407237477
   puts      :26885386434
   candidate :2619726
   confirmed :3
   spend     :67402563504
   strong_tx :12387474660
   valid_tx  :0
   valid_bk  :0
   address   :0
   neutrino  :0
2024-12-04T18:12:49Z.0 Table records...
   header    :873242
   tx        :1127164671
   point     :2807566904
   candidate :873242
   confirmed :1
   spend     :2808440146
   strong_tx :1032289555
   address   :0
2024-12-04T18:12:49Z.0 Head buckets...
   header    :524493
   txs       :524493
   tx        :551320125
   point     :546188501
   spend     :1459791875
   strong_tx :551320125
   valid_tx  :551320125
   valid_bk  :524493
   address   :1
   neutrino  :1
2024-12-04T18:12:49Z.0 Collision rates...
   header    :1.66493
   txs       :1.66493
   tx        :2.04448
   point     :5.14029
   spend     :1.92386
   strong_tx :1.8724
   valid_tx  :2.04448
   valid_bk  :1.66493
   address   :0
   neutrino  :0

(edited)

evoskuil18:18

Shutdown time from the full chain sync was 10 seconds:

2024-12-04T18:18:02Z.0 Database stopped successfully in 10 secs.
2024-12-04T18:18:02Z.0 Node stopped successfully.
2024-12-04T18:18:02Z.0 ====================== shutdown ======================
2024-12-04T18:18:02Z.0 Press <enter> to exit...

evoskuil18:21

You should always terminate the process normally, or it will require a full resync from the last snapshot. You can establish snapshots while running or once it's complete, or at the command line. Automatic snapshotting is currently off by default. A snapshot pauses the network and takes a couple minutes. There are only two in rotation. Restore is automatic, or can be invoked at the command line. If the disk fills it is detected, and will resume once space is available, or can be shut down normally.

(edited)

👍 1

evoskuil18:26

The store can also be hot backed-up from the file system, and will automatically recover to the last snapshot. So for hot backup simply snapshot first and then copy the directory once that completes, while it's running.

evoskuil18:35

You will probably have to configure seed nodes manually. The ones configured by default are probably not running. This can be done in the config file, any number of entries:

[network]
seed = ip:port
seed = host:port

👍 1

evoskuil18:37

By default the node maintains 10,000 host names in a text file named hosts.cache . IPv6 is supported but disabled by default. To skip seeding you can also just place this file in the network directory, as long as there are at least 500 entries (by default).

cguida19:36

Ok I'll try to get it started today!

👍🏻 1

cguida00:04

Ok I finished the sync and by my calculations it took just about 90 minutes

cguida00:04

I will run some more benchmarks to confirm, this result is insane haha

cguida00:15

Ok upon closer inspection the sync took 7605 seconds which is 126.75 minutes or 2.11 hours

cguida00:16

I am incredibly psyched about this result!! I am in awe of the work you guys are doing!

🤙🏻 1

cguida00:16

I'm wondering if the sync time would be massively impacted with only 16GB ram instead of 32...

evoskuil01:03

You only have 16GB RAM? I would have expected more of a hit than that. Nice.

cguida02:30

Oh sorry I didn’t see this reply. No, I used 32GB

cguida02:30

Will try with 16GB soon

cguida02:31

Man these threads are all over the place haha

evoskuil02:31

Threading on slack really sucks

👍 1

evoskuil17:25

I just added some minor sha256 optimizations (vectorization), seems to have given it a bit of a boost. Still no shani but that's getting close. This is what the run looks like half way through on 2.3Gbps Internet.

(edited)

cguida17:34

Also curious how the Linux benchmark is going, definitely want to get that up to speed with the Windows version if possible! All my nodes are Linux due to working better with Lightning

evoskuil18:23

Linux and macos are tested and run, but they have not been performance tuned and there is some issue on both that causes them to be very slow while using few resources. I haven't spent any time on it. Pretty busy lately and have prioritized getting shani integrated and validate/confirm fully tuned.

1 reply · cguida · last Dec 04, 2024

cguida18:28

ok cool, thanks for the update!

👍🏻 1

Thursday, December 5, 2024

evoskuil02:15

Initial implementation with shani accelerates sha256 to 18% of canonical algorithm and 19% of vectorized optimizations. IOW the current hashing code takes 5.5x more time than this new code on machines with shani. This will also accelerate the 1/2/3 lane hashing for merkle roots. This should dramatically lower CPU utilization on the mini.

(edited)

evoskuil04:48

It looks like a partial integration of shani has knocked 15% off the mini sync time. I inadvertently ran it with automated snapshots and a block upper limit. Running it again now, but I think it will hit around 45 minutes to block 850k.

evoskuil05:56

I ordered another one of these PALADN minicomputers, but with the AVX512 (and AVX2/SSE4/SHA like the ones I have) CPU, $421 delivered. This will allow me to run with all of our hardware accelerations.

evoskuil06:12

43 minutes to block 850,000

🙇🏾 1🚀 3😮 1

06:15

The theoretical limit on my 2.3Gbps network is 34 minutes. We are only 9 minutes above the time it would take to download the full set of blocks in a single file.

06:16

This is on a $350 Windows PC. With AVX512 and the completion of the SHA-NI integration, we'll shave a few minutes more off of the total time.

(edited)

evoskuil20:30

Weekly mtg in 30 mins.

2 replies · pmienk, jungly · last Dec 05, 2024

pmienk20:45

I wont be able to make it, still at the optometrist at the moment.

👍🏻 1👍🏾 1

jungly21:00

hmm.. I can't find the invite.. been a long time

evoskuil21:08

Ok, still figuring this thing out. Starting a new huddle in a sec.

Huddle with evoskuil, Marnix, jungly, parsevalbtc, pmienk · 97 min21:08

7 replies · evoskuil, cguida · last Dec 06, 2024

evoskuil22:27

https://help.apple.com/instruments/mac/current/#/dev2843b037

evoskuil22:27

https://github.com/wolfpld/tracy/releases

evoskuil22:43

https://github.com/libbitcoin/libbitcoin-network/blob/master/include/bitcoin/network/messages/enums/level.hpp

evoskuil22:45

https://github.com/libbitcoin/libbitcoin-system/blob/master/include/bitcoin/system/chain/enums/flags.hpp

cguida00:00

Ah darn, I missed it! When's the next one?

evoskuil01:00

Every Thursday at 4pm NYC time.

👍 1

cguida01:48

Ah looks like I might be on a plane during the next one

evoskuil23:09

https://github.com/utreexo/utreexo

Friday, December 6, 2024

evoskuil06:19

Took me all evening to track down a simple issue in the single block shani optimization (difference in endianness between the buffered and rotating sha algorithms). But we are now getting the same 5x benefit for single and padding blocks. Just a little more to do to get half and dual half blocks integrated, which target double hashing and Merkle hashing respectively.

evoskuil09:21

41 minutes to block 850,000

neillm13:37

All the work lately has been awesome! Just wondering, is there an isolated benchmark or test I could use to test out sha_ni perf on Linux (AMD Ryzen 9 7950X 16-Core Processor)?

13:38

Separately, I'd like to compare your test run times, but my bandwidth is limited to 300Mbps. Your bandwidth is impressive 😅

(edited)

evoskuil14:38

Just did another, 42 minutes.

14:41

There are hashing performance tests in the system test project. Because they are slow they have to be turned on by enabling HAVE_PERFORMANCE_TESTS:

/// Have slow test execution (scrypt is slow by design).
////#define HAVE_SLOW_TESTS
////#define HAVE_PERFORMANCE_TESTS

🙏 1

14:43

The test lib has a full canonical implementation of sha256 and ripemd, derived from Core, and a custom test runner and timer in performance.hpp. Test cases are in performance.cpp.

14:44

The AMD Ryzen 9 7950X has shani as well as sse4, avx2, and avx512. I haven't been able to perf test with avx512 yet but it's implemented.

18 replies · neillm, evoskuil · last Dec 06, 2024

neillm15:33

Ah, that would explain why I'm seeing a build error with avx512 enabled! Here for reference:

... snip ...
                 from ./include/bitcoin/system/settings.hpp:22,
                 from ./include/bitcoin/system.hpp:29,
                 from examples/main.cpp:21:
./include/bitcoin/system/intrinsics/xcpu/functional_512.hpp: In function 'libbitcoin::system::xint512_t libbitcoin::system::add(xint512_t, xint512_t)':
./include/bitcoin/system/intrinsics/xcpu/functional_512.hpp:163:31: error: cannot convert 'libbitcoin::system::xint512_t' {aka '__m512i'} to '__m256i'
  163 |         return mm256_add_epi8(a, b);
      |                               ^
      |                               |
      |                               libbitcoin::system::xint512_t {aka __m512i}
./include/bitcoin/system/intrinsics/xcpu/defines.hpp:224:61: note: in definition of macro 'mm256_add_epi8'
  224 |     #define mm256_add_epi8(a, b)            _mm256_add_epi8(a, b)
      |                                                             ^
In file included from /usr/lib/gcc/x86_64-linux-gnu/14/include/immintrin.h:53,
                 from /mnt/ssd1/libbitcoin/master/install/include/boost/multiprecision/cpp_int/intel_intrinsics.hpp:83,
                 from /mnt/ssd1/libbitcoin/master/install/include/boost/multiprecision/cpp_int/add_unsigned.hpp:10,
                 from /mnt/ssd1/libbitcoin/master/install/include/boost/multiprecision/cpp_int/add.hpp:12,
                 from /mnt/ssd1/libbitcoin/master/install/include/boost/multiprecision/cpp_int.hpp:2284,
                 from ./include/bitcoin/system/boost.hpp:54,
                 from ./include/bitcoin/system/exceptions.hpp:26,
                 from ./include/bitcoin/system/arena.hpp:22,
                 from ./include/bitcoin/system/allocator.hpp:28,
                 from ./include/bitcoin/system.hpp:17:
/usr/lib/gcc/x86_64-linux-gnu/14/include/avx2intrin.h:105:26: note:   initializing argument 1 of '__m256i _mm256_add_epi8(__m256i, __m256i)'
  105 | _mm256_add_epi8 (__m256i __A, __m256i __B)
      |                  ~~~~~~~~^~~
./include/bitcoin/system/intrinsics/xcpu/functional_512.hpp:165:32: error: cannot convert 'libbitcoin::system::xint512_t' {aka '__m512i'} to '__m256i'
  165 |         return mm256_add_epi16(a, b);
      |                                ^
      |                                |
      |                                libbitcoin::system::xint512_t {aka __m512i}
./include/bitcoin/system/intrinsics/xcpu/defines.hpp:225:62: note: in definition of macro 'mm256_add_epi16'
  225 |     #define mm256_add_epi16(a, b)           _mm256_add_epi16(a, b)
      |                                                              ^
/usr/lib/gcc/x86_64-linux-gnu/14/include/avx2intrin.h:112:27: note:   initializing argument 1 of '__m256i _mm256_add_epi16(__m256i, __m256i)'
  112 | _mm256_add_epi16 (__m256i __A, __m256i __B)
      |                   ~~~~~~~~^~~
./include/bitcoin/system/intrinsics/xcpu/functional_512.hpp:167:32: error: cannot convert 'libbitcoin::system::xint512_t' {aka '__m512i'} to '__m256i'
  167 |         return mm256_add_epi32(a, b);
      |                                ^
      |                                |
      |                                libbitcoin::system::xint512_t {aka __m512i}
./include/bitcoin/system/intrinsics/xcpu/defines.hpp:226:62: note: in definition of macro 'mm256_add_epi32'
  226 |     #define mm256_add_epi32(a, b)           _mm256_add_epi32(a, b)
      |                                                              ^
/usr/lib/gcc/x86_64-linux-gnu/14/include/avx2intrin.h:119:27: note:   initializing argument 1 of '__m256i _mm256_add_epi32(__m256i, __m256i)'
  119 | _mm256_add_epi32 (__m256i __A, __m256i __B)
      |                   ~~~~~~~~^~~
./include/bitcoin/system/intrinsics/xcpu/functional_512.hpp:169:32: error: cannot convert 'libbitcoin::system::xint512_t' {aka '__m512i'} to '__m256i'
  169 |         return mm256_add_epi64(a, b);
      |                                ^
      |                                |
      |                                libbitcoin::system::xint512_t {aka __m512i}
./include/bitcoin/system/intrinsics/xcpu/defines.hpp:227:62: note: in definition of macro 'mm256_add_epi64'
  227 |     #define mm256_add_epi64(a, b)           _mm256_add_epi64(a, b)
      |                                                              ^
/usr/lib/gcc/x86_64-linux-gnu/14/include/avx2intrin.h:126:27: note:   initializing argument 1 of '__m256i _mm256_add_epi64(__m256i, __m256i)'
  126 | _mm256_add_epi64 (__m256i __A, __m256i __B)
      |                   ~~~~~~~~^~~
make: *** [Makefile:4993: examples/libbitcoin_system_examples-main.o] Error 1

neillm15:35

For now I'm going to disable that, but can help test at some point

evoskuil15:36

AVX512 builds and runs, we execute it in our CI tests. How did you enable it?

(edited)

neillm15:38

Clang or Gcc? Everything builds and runs fine with that option disabled

neillm15:39

With it disabled (run just now):

./configure --with-tests --with-examples --with-pkgconfigdir=/mnt/ssd1/libbitcoin/master/install/lib/pkgconfig --prefix=/mnt/ssd1/libbitcoin/master/install --with-boost=/mnt/ssd1/libbitcoin/master/install --with-zmq --with-zlib --with-icu --with-qrencode --with-png --enable-static --disable-shared --enable-avx2 --enable-shani --enable-sse41

neillm15:39

I'm using gcc-14

neillm15:40

checking --with-pkgconfigdir option... /mnt/ssd1/libbitcoin/master/install/lib/pkgconfig
checking --with-tests option... yes
checking --with-examples option... yes
checking --with-icu option... yes
checking --enable-avx2 option... yes
checking --enable-avx512 option... no
checking --enable-sse41 option... yes
checking --enable-shani option... yes
checking --enable-ndebug option... yes
checking --enable-isystem option... no

evoskuil15:41

There’s no option for zlib, qrencode, png or zmq.

neillm15:41

Yeah, this was generated from a build script I use

neillm15:41

It passes more info than needed (and is also old)

evoskuil15:48

Above is a recent regression (yesterday). I guess we aren't running avx512 on CI unless there's an available platform. I know the tests used to pass that detected it. I'll patch now.

🙏 1

evoskuil15:50

Done

neillm15:51

Thanks, will try!

🕺🏻 1

neillm15:52

Yep, that did it 🔥

👍🏻 1

evoskuil15:53

https://github.com/libbitcoin/libbitcoin-system/blob/master/test/hash/performance/performance.cpp#L81C2-L118C2

evoskuil15:55

These tests are just Merkle with and without vectorization. They don't include shani. The interesting avx512 test is the ratio of the vector tests to non-vector here. it's about 21% with avx2. I'm expected about 10% with avx512.

evoskuil15:57

The disabled tests below are non-merkle. You can set up sha256a_comp (shani) and sha256a_both (shani and vectorization) to get the shani stuff. base_sha256a is the Core baseline.

(edited)

evoskuil15:58

You can do the same for the Merkle tests if you want. There's still several places where shani isn't fully incorporated though. It doesn't currently benefit Merkle.

evoskuil14:51

The total percentage of customers with 1Gbps or higher is 34%, which makes it the most popular internet tier among all Americans.

https://www.cnet.com/home/internet/best-multi-gig-internet-plans/

6 replies · neillm, evoskuil · last Dec 06, 2024

neillm16:09

The big ones in my area top out at just under 1Gbps and it's pricey. I'm on business class internet, but it's quite a jump from where I am. I didn't think to look at AT&T, but as it turns out AT&T Internet is not available at ... where I need it.

evoskuil16:19

Jeesh, I'm in a little NH farming/fishing village 😅

neillm16:22

With good internet, apparently!

evoskuil16:34

Site of the first transatlantic cable landing. I live a block from cable road. 😂

🙌 1

neillm16:36

I'll have to look at the internet options here again. When I login, I can see the upgrade plan pricing and it sucks. If I just search (not logged in), I'm seeing some better deals from the same provider 🙄

(edited)

evoskuil16:38

Yep, cancel and re-add service. That will get them to give you the current price.

(edited)

evoskuil14:52

10 Gbps is coming. That's a 10 minute sync.

14:54

To compile with intrinsics you need to define each of the symbols for the build (shani, sse41, avx2, avx512).

neillm16:24

Hrm:

$ ./test/libbitcoin-system-test --run_test=performance_tests
Running 2 test cases...

test____________: performance__sha256a_none__merkle
algorithm_______: N10libbitcoin6system3sha9algorithmINS1_4h256ILm256ELb1EEELb0ELb0ELb1ELb1EEE
test_rounds_____: 10240
bytes_per_round_: 1
native__________: 0
vectorized______: 0
cached__________: 1
chunked_________: 0
seconds_total___: 0.437522
mib_per_second__: 0.0223203
cycles_per_byte_: 128180
ms_per_round____: 0.0427268
ms_per_byte_____: 0.0427268
unknown location(0): fatal error: in "performance_tests/performance__sha256a_none__merkle": signal: SIGSEGV, si_code: 128 (memory access violation at address: 0x00000000)
test/hash/performance/performance.cpp(80): last checkpoint: "performance__sha256a_none__merkle" test entry

*** 1 failure is detected in the test module "libbitcoin_test"

16:26

Odd, something may be off in general with this build (or my setup/system):

$ ./test/libbitcoin-system-test
Running 3724 test cases...
unknown location(0): fatal error: in "settings_tests/settings__construct__regtest_context__expected": signal: SIGSEGV, si_code: 128 (memory access violation at address: 0x00000000)
test/settings.cpp(171): last checkpoint

*** 1 failure is detected in the test module "libbitcoin_test"

pmienk16:26

Did you enable an intrinsic your cpu doesn't support?

7 replies · neillm, pmienk, evoskuil · last Dec 06, 2024

neillm16:27

In theory, I think my CPU supports all 4 options, but I could try to enable one by one until I see an error

pmienk16:29

Why not start with lscpu

(edited)

neillm16:31

Right, I thought they were covered though, maybe not:

    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe
                          1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ss
                          se3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse
                           3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
                           perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx sm
                          ap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total c
                          qm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbya
                          sid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmul
                          qdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze

evoskuil16:31

These tests determine exactly what is present that we need, and can be run from a default build. https://github.com/libbitcoin/libbitcoin-system/blob/master/test/intrinsics/haves.cpp

evoskuil16:32

Specifically we need avx512bw

neillm16:32

That's in there

evoskuil16:33

Then try the same with the reset but not avx512, as I haven't tested with it.

👍 1

evoskuil16:29

Yeah, that looks like non-support. There is test code to determine what the CPU supports (from a default build).

16:29

The node will provide that info on the command line.

26 replies · neillm, evoskuil · last Dec 06, 2024

neillm16:39

That's a great feature btw (older bin)

$ ./master/install/bin/bn -d
Intrinsics...
arm..... platform:0.
intel... platform:1.
avx512.. platform:1 compiled:0.
avx2.... platform:1 compiled:1.
sse41... platform:1 compiled:1.
shani... platform:1 compiled:0.
neon.... platform:0 compiled:0.

evoskuil16:40

Ohh, no... it's there.

(edited)

evoskuil16:42

So run the tests without it and if ok that means it's prob just a bug in our avx512 feature.

(edited)

evoskuil16:42

I'll have an avx512 box soon to test on.

neillm16:58

Still no go, and this is with only shani and sse41

$ ./test/libbitcoin-system-test
Running 3716 test cases...
unknown location(0): fatal error: in "settings_tests/settings__construct__regtest_context__expected": signal: SIGSEGV, si_code: 128 (memory access violation at address: 0x00000000)
test/settings.cpp(171): last checkpoint

*** 1 failure is detected in the test module "libbitcoin_test"

neillm16:59

Let me sanity check and make sure it works w/o any

evoskuil17:15

Shani is new, and the CI tests may not have it.

neillm17:15

I am suspecting that, but can find out for sure. It's working btw w/o any flags enabled. Worth letting it run to get the baseline

evoskuil17:18

avx2 and sse4 are defaults for CI so those are well tested.

evoskuil17:19

Are you building with clang?

neillm17:19

gcc

evoskuil17:21

Ok, I had to disable a couple new vectorization optimizations in non-macos clang due to these type of failures. gcc, XCode and msvc have been fine tho.

evoskuil17:21

You could debug those too! :))

👍 1

neillm17:23

Hrm, so how do I log the test results to a file? I'm trying to redirect stdout/stderr and it's not showing up

evoskuil17:24

I suspect they have to do with memory alignment, but the techniques work in other similar areas on clang, and I’ve tried making sure that loadu and storeu are comprehensively used, but no luck.

👍 1

neillm17:24

hwloc-bind --cpubind core:5 -- ./test/libbitcoin-system-test --log_level=all --run_test=performance_tests > perf_tests_none.log 2>&1

evoskuil17:24

From what, tests?

neillm17:25

The perf tests in particular. A lot of stuff is logged using that, but not the parts I want, i.e.

test____________: performance__sha256a_none__merkle
algorithm_______: N10libbitcoin6system3sha9algorithmINS1_4h256ILm256ELb1EEELb0ELb0ELb1ELb1EEE
test_rounds_____: 10240
bytes_per_round_: 1
native__________: 0
vectorized______: 0
cached__________: 1
chunked_________: 0
seconds_total___: 0.452969
mib_per_second__: 0.0215592
cycles_per_byte_: 132706
ms_per_round____: 0.0442352
ms_per_byte_____: 0.0442352

evoskuil17:25

Does it dump to the console? The rest is up to the OS.

neillm17:26

Unless it's all buffered and written to the file later

neillm17:26

Ok, yeah, will wait for it and see (but yes, to the console)

evoskuil17:26

It writes to the console, which has its own flush logic.

👍 1

evoskuil17:27

I do notice it buffering when I run the perf tests. There’s a lot in there and it’s sequential and single thread.

evoskuil17:28

But typically under a minute for the two merkle tests.

neillm17:28

This is w/o optimizations, so it's taking longer than a minute for sure

neillm17:55

Still going. For example:

test____________: performance__sha256a_none__merkle
algorithm_______: N10libbitcoin6system3sha9algorithmINS1_4h256ILm256ELb1EEELb0ELb0ELb1ELb1EEE
test_rounds_____: 10240
bytes_per_round_: 256
native__________: 0
vectorized______: 0
cached__________: 1
chunked_________: 0
seconds_total___: 229.015
mib_per_second__: 0.0109163
cycles_per_byte_: 262086
ms_per_round____: 22.3647
ms_per_byte_____: 0.0873621

evoskuil18:13

I just synced with the full electrum address index on the mini (which used to be prohibitive). 79 minutes to 850k!

💥 3🙌 1

18:18

That's 121,533,492,738 additional bytes of storage, 2,893,654,589 additional hash table records, and the same number of additional sha256 hashes (hash of every tx.output).

evoskuil21:57

Here's a very complex set of query results and timings, run both from the running node console, and from a stopped node at the startup command line. These are run on the fully-synced mini with the full Electrum address index.

query.txt Plain Text · 19.4 MB

WARM START : 1.422 sec (from running node console)
==================================================================================================================================

2024-12-06T21:51:35Z.0 Getting first [100] output address hashes.
2024-12-06T21:51:35Z.0 Got first [100] unique addresses above tx [1000000000] in [0] ms.
2024-12-06T21:51:35Z.0 Fetched [1] unique payments to address [759310d9c152ed35c07c2c46b27550c6e3515b96b13f4ec8e4e7564e9794dd12].
2024-12-06T21:51:35Z.0 Fetched [1] unique payments to address [5ac3348a44a11bd18d0aa88a94c454f199a182222ddecc2224cca2a1af1ea213].
2024-12-06T21:51:35Z.0 Fetched [1] unique payments to address [3be7a9c5705aba0eba6594d5285c8cf6d72b9702411930fa86a36eff419b4f14].
2024-12-06T21:51:35Z.0 Fetched [5461] unique payments to address [89b9e9eb214625e6b07147397caaadfdcaac0fd2c5cbc12f87aa66b9bf13c514].
2024-12-06T21:51:35Z.0 Fetched [1] unique payments to address [57940fda486ad77f374cb591c0b9f642044f11c909aab302e83651b88b830f15].
2024-12-06T21:51:35Z.0 Fetched [1] unique payments to address [df25331de538e6cf5304c3cd556e8be9c86536fa7db5ea42d7432e4313e40716].
2024-12-06T21:51:35Z.0 Fetched [32] unique payments to address [837b95d18fd612c9fd2a119046e7ce9c5954f1de3ce6d65d9a4f5d3942dde61a].
2024-12-06T21:51:35Z.0 Fetched [5] unique payments to address [b4e9f5a1dc8efaa4c6502b5d2e1f47360467a796adca75dc640d393e9fb4671b].
2024-12-06T21:51:36Z.0 Fetched [307] unique payments to address [fb1c8c2404df894237f1f631bd5389e4e3a192d921a81fbbb4a2fb7dbf0e061f].
2024-12-06T21:51:36Z.0 Fetched [85] unique payments to address [96271fa2de1fa69c46429cb611a2ae74ee867cecb54ad10455e1ad45994ee71f].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [69735f2f55eddf67848d63462473f7a017a79aa52d383c95c7173920e51ee720].
2024-12-06T21:51:36Z.0 Fetched [234] unique payments to address [3b8926732fb3860883f64d5506292cd38c6096bd62471794e9053135fcff5e25].
2024-12-06T21:51:36Z.0 Fetched [155] unique payments to address [3e775a8fca43ba85557b98b7415d8865c59ee2fe98ee6262d4b0224f57365327].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [4b6c6c4e6cfb0f9fd1eb21cb3c15f12a41a6ca2d11003b06dd00ef4b47c28c27].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [6c6ed9788123330d0fb1e1fe71e8e8d83d0db495e620d1d94eabc7c4533f0528].
2024-12-06T21:51:36Z.0 Fetched [253] unique payments to address [fd0613ffabac395037c745379f05905660ecc48507439d0f3a10ac6d36fffd2c].
2024-12-06T21:51:36Z.0 Fetched [574] unique payments to address [052d975162e90600c8938c54f5e9e9803fcd0e76c69356b11c3ed37702effa2d].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [064bb9bab2a2f2f8f6b49c2087d9ba565945fc2a5b79acf5ab344aa18c223f35].
2024-12-06T21:51:36Z.0 Fetched [2] unique payments to address [ba94987ce80b299a798177745c26e3854b68ff1dcba74d346abd0f75b36f1138].
2024-12-06T21:51:36Z.0 Fetched [219] unique payments to address [f21502641273ae2a99543821d7f8fea41f5d1ca80d6f328befdb1982f9ab603c].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [8ec667e23a379adf6f9998ab8d067ba1428a78090cc0c46932a6b59ad0d63b42].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [e6b182fc64a4f75e2dab7794aedc78f6ac5ce795085823d32ed680c13a528144].
2024-12-06T21:51:36Z.0 Fetched [5] unique payments to address [ebbe53fd9fed91e2cc48ed093e532c928cf0efcdd71f34b5f1b2f34be4193d45].
2024-12-06T21:51:36Z.0 Fetched [56] unique payments to address [0a2823a9e4df2abeeb90f783c065a76da57d7b59da5225ec2f33d40abd9af745].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [b68f22396a131344efb5f0f153862ea647b2fdea1526ed4cdb68cf469938fe46].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [72be417db2a465f45f160d422da3ba5e979e69f2c3548c587a195a6209b5724d].
2024-12-06T21:51:36Z.0 Fetched [110] unique payments to address [2794bb272f5fb2fff9e7cc7f6e596dbdf8f472e34641003aba2461dc88b1bc51].
2024-12-06T21:51:36Z.0 Fetched [3] unique payments to address [22c59260b339e8998e3e48afeb79bcc7fdb65e94fc3f41b265ccb99d59e7b853].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [cddb9eb456bdb54690ac8ce18438b221a248e1b8cb409fa39995aa0746f47356].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [71df01d24de3b99818e9a3ef84fb81c9ef36d8f7dedf72d15407674876691e57].
2024-12-06T21:51:36Z.0 Fetched [493] unique payments to address [331a1a15370b3afa8e36896eeb5af574e68520cd42660c80642fae11521f295a].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [856f519fa608468cb0fa0655f86d74ccc641e7f87be618cf02acabf07c72be5b].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [7ea3f76e7339b8e10e83fd7adca93540a3adedddac1bc98d32475702890ce55c].
2024-12-06T21:51:36Z.0 Fetched [1] unique payments to address [e1870e91731b8c84463159af396bfd5f46992af39fe4a8c87c2a5f00df025b5e].
2024-12-06T21:51:36Z.0 Fetched [246] unique payments to address [1cc7573313280746c2dae3e05ec46bc819f5a75ba4e3dcc80a07a2f98c773060].

full file

evoskuil22:02

The ad-hoc query that produces this is here.

void executor::read_test(bool dump) const
{
    constexpr auto start_tx = 1'000'000'000_u32;
    constexpr auto target_count = 100_size;

    // Set ensures unique addresses.
    std::set<hash_digest> keys{};
    auto tx = start_tx;

    logger(format("Getting first [%1%] output address hashes.") % target_count);

    auto start = fine_clock::now();
    while (!cancel_ && keys.size() < target_count)
    {
        const auto outputs = query_.get_outputs(tx++);
        if (outputs->empty())
            return;

        for (const auto& put: *outputs)
        {
            keys.emplace(put->script().hash());
            if (cancel_ || keys.size() == target_count)
                break;
        }
    }

    auto span = duration_cast<milliseconds>(fine_clock::now() - start);
    logger(format("Got first [%1%] unique addresses above tx [%2%] in [%3%] ms.") %
        keys.size() % start_tx % span.count());

    struct out
    {
        hash_digest address;

        uint32_t bk_fk;
        uint32_t bk_height;
        hash_digest bk_hash;

        uint32_t tx_fk;
        size_t tx_position;
        hash_digest tx_hash;

        uint32_t sp_tx_fk;
        hash_digest sp_tx_hash;

        uint64_t input_fk;
        chain::input::cptr input{};

        uint64_t output_fk;
        chain::output::cptr output{};
    };

    std_vector<out> outs{};
    outs.reserve(target_count);
    using namespace database;

    start = fine_clock::now();
    for (auto& key: keys)
    {
        size_t found{};
        auto address_it = store_.address.it(key);
        if (cancel_ || address_it.self().is_terminal())
            return;

        do
        {
            table::address::record address{};
            if (cancel_ || !store_.address.get(address_it.self(), address))
                return;

            const auto out_fk = address.output_fk;
            table::output::get_parent output{};
            if (!store_.output.get(out_fk, output))
                return;

            const auto tx_fk = output.parent_fk;
            const auto block_fk = query_.to_block(tx_fk);

            table::header::get_height header{};
            if (!store_.header.get(block_fk, header))
                return;

            table::txs::get_position txs{ {}, tx_fk };
            if (!store_.txs.get(query_.to_txs(block_fk), txs))
                return;

            spend_link sp_fk{};
            input_link in_fk{};
            tx_link sp_tx_fk{};

            // Get first spender only (may or may not be confirmed).
            const auto spenders = query_.to_spenders(out_fk);
            if (!spenders.empty())
            {
                sp_fk = spenders.front();
                table::spend::record spend{};
                if (!store_.spend.get(sp_fk, spend))
                    return;

                in_fk = spend.input_fk;
                sp_tx_fk = spend.parent_fk;
            }

            ++found;
            outs.push_back(out
            {
                key,

                block_fk,
                header.height,
                query_.get_header_key(block_fk),

                tx_fk,
                txs.position,
                query_.get_tx_key(tx_fk),

                sp_tx_fk,
                query_.get_tx_key(sp_tx_fk),

                in_fk,
                query_.get_input(sp_fk),

                out_fk,
                query_.get_output(out_fk)
            });
        }
        while (address_it.advance());

        logger(format("Fetched [%1%] unique payments to address [%2%].") %
            found% encode_hash(key));
    }

    span = duration_cast<milliseconds>(fine_clock::now() - start);
    logger(format("Got all [%1%] payments to [%2%] addresses in [%3%] ms.") %
        outs.size() % keys.size() % span.count());

    if (!dump)
        return;

    // Write it all...
    logger(
        "output_script_hash, "
    
        "ouput_bk_fk, "
        "ouput_bk_height, "
        "ouput_bk_hash, "

        "ouput_tx_fk, "
        "ouput_tx_position, "
        "ouput_tx_hash, "

        "input_tx_fk, "
        "input_tx_hash, "

        "output_fk, "
        "output_script, "

        "input_fk, "
        "input_script"
    );

    for (const auto& row: outs)
    {
        if (cancel_) break;

        const auto output = !row.output ? "{error}" :
            row.output->script().to_string(chain::flags::all_rules);

        const auto input = !row.input ? "{unspent}" :
            row.input->script().to_string(chain::flags::all_rules);
    
        logger(format("%1%, %2%, %3%, %4%, %5%, %6%, %7%, %8%, %9%, %10%, %11%, %12%, %13%") %
            encode_hash(row.address) %

            row.bk_fk %
            row.bk_height %
            encode_hash(row.bk_hash) %

            row.tx_fk %
            row.tx_position %
            encode_hash(row.tx_hash) %

            row.sp_tx_fk %
            encode_hash(row.sp_tx_hash) %

            row.output_fk %
            output %

            row.input_fk %
            input);
    }
}

evoskuil22:10

On the mini this completes in 1.4 seconds from a warm start (dumps summary output to console while running), and 2.6 seconds from the command line (node starts, loads store, starts timer, runs queries, stops timer, dumps full output to console [piped to file], stops). There are 45,368 unique payments identified across 100 unique addresses: output, tx, block including confirmed spending input (if any) and its tx. The full input and output are deserialized, along with internal and public object identifiers, eg for one payment:

output_script_hash, ouput_bk_fk, ouput_bk_height, ouput_bk_hash, ouput_tx_fk, ouput_tx_position, ouput_tx_hash, input_tx_fk, input_tx_hash, output_fk, output_script, input_fk, input_script

b2823266b8d53f9cff4d230ca355f10e5bebc6a4309d0555c4e271028f198afd, 821210, 821210, 000000000000000000010e12839c836ee639cf589292e0e3d23942d38e82ce20, 22247971, 4195, 93753adc834dbf78269c0b568fe83d7ac88f45b789387c7332ac0493250619ce, 530454348, 909cd96517741d03259789eda9b542a4823bade421ab652384051173047ff9e4, 2139158293, dup hash160 [2f93163beabbd1316fd21ef32a2a3c18f75ec4b6] equalverify checksig, 186332222689, [30440220643b28ca21d18974c3f817a67205cb0234816d6acabba16763b0bf55295739a4022020f6bd52fc4fda90f6a365f26d4128dc372b952bf10f21a4a51b30b556cc193201] [035fbd15569de389d55211cf336e346f594e166844fbb44679dee2ac2958544efa]

22:11

With more RAM queries are quite a bit faster, as the entire store can become memory resident. Also, this was produced by a single sequential thread of execution, and can execute concurrently with any others and even with concurrent heavy store writing.

(edited)

Saturday, December 7, 2024

evoskuil05:08

Consider that each entry in the output above required 30.8 microseconds to generate (not including the format/log). That's 18 table queries per entry (to_spenders is internally complex), or about 1.7 microseconds per query. And several of these queries are hash table searches along deconfliction lists. Unspent entries are a little faster and spent entries require the full query set.

(edited)

evoskuil08:05