Bitcoin full node and query server

May 2025

Archive of the libbitcoin Slack #general channel.

Thursday, May 1, 2025

evoskuil16:30
I won’t make the mtg this week. Still on my way back from NY.
C
cguida17:30
how'd it go?
👍🏻 1

Friday, May 2, 2025

evoskuil00:14
Good crowd, good discussion.
🎉 1
evoskuil17:54
Just a heads up, there is a regression in the node, introduced in the past week. If you are testing with latest you will probably experience a halt before full validation completion. I'll update here when it's resolved.
(edited)
👍 1
evoskuil23:26
Found and resolved the issue. The newish arraymap primitive has an expanding head, but was never previously read past it's limit due to the nature of the confirmation prevout query that consumed it. I changed two more tables to arraymap (validated_bk and txs) and in these cases the tables are read for items that may not be found just due to lack of ordering. When the element searched is greater than the head allocation there is a read protection fault. When writing there is no problem because a write would just expand the head. The low level file does not guard against such overflows, because at such a low level that's a big performance expense for all tables, most of which wouldn't benefit from it. The arrayhead test cases didn't cover it, because they were derived from the hashmap tests, and hashhead doesn't resize. So test added, simple fix within the arrayhead. https://github.com/libbitcoin/libbitcoin-database/pull/629
(edited)
evoskuil23:36
These arraymaps are good for block-indexed tables, since the array head is small (<1M) vs. tx (>1B) and input/output (>2.6B). The array head can be thought of as a simple hashmap head with a fixed 1.0 load factor, and pass-thru hash function. Each bucket holds the singular element body position. This means that repeat entries for the same element get overwritten in the head and are essentially hidden in the body. This works well for state transitions where there is one state per header, and prevouts and txs association, where there is only one set per header/block. The arraymap heads are larger (e.g. than a 2.2 LF hashhead), but they are sequential, so more efficient paging, and the arraymap requires no keys or links within the body - so the tables are overall much smaller and also with less paging. The mapping allows elements to be written out of order and yet indexed with array efficiency (always O(1)).

Saturday, May 3, 2025

evoskuil00:26
So it appears that the arraymap conversions are done. I've got one more open issue before going back to Taproot. Should take no more than one day to complete.
👀 1
evoskuil17:59
Completed the last open issue. The entire stack is essentially lock-free, with synchronization primitives buried in C++ std lib (concurrent loops), ASIO, ZeroMQ, and our store primitives. But I used a simple std::mutex with std::unique_lock to synchronize between the candidate chain and confirmed chain. Basically we cannot allow popping of blocks from the candidate chain during the organization of blocks into the confirmed chain, since the candidate chain provides order for the confirmed chain. It would be possible to push this into a store atomic read of the full candidate branch, which hash the slight drawback of holding this set in a redundant temporary structure during organization. This would have to be captured before determining if the branch is strong. There would be a very small performance hit, but that would allow the implementation to not expose any synchronization primitives at the application layer.
evoskuil19:53
So I did that, the reorganization interlock hash been moved to the store. Additionally I moved three queries from the node confirmation class into the store. This simplifies testing and is better hiding, as these are really just store queries (computing work, relative work, and getting the current fork header identifiers).
evoskuil20:18
On to Taproot. Once that's done, it's on to header and block out protocols (which gives us the basic full node without tx relay) and then server.
evoskuil21:14
With this in place the bottlenecks have to move to things like network, block serialization, validation parallelism overheads… Development time would be better spent on optimizing those...
Core has some work to do.
I find skipping amount validation more concerning (potential inflation bugs)
It’s basically free so it should do that for sure. That said, there are so many lost/abandoned coins that if miners (with developer collusion) create a chain that unjustly rewards them, they’ll just steal coins rather than inflate.
And seems to be willing to accept broken validation to get there. https://delvingbitcoin.org/t/swiftsync-speeding-up-ibd-with-pre-generated-hints-poc/1562/15
😮 1

Tuesday, May 6, 2025

evoskuil23:33
Update on Taproot progress. I expect to finish up the ops soon and start on signature hashing tonight, finish that up tomorrow. Should be done this weekend.
x add Tagged Hash
x add libsecp2561 schnorr functions
x add Signature Parse
x add coverage::bip341_rule
x add coverage::bip342_rule
x upd script tests for coverage::all_rules
x add coverage::hash_default
x add transaction::is_sighash_valid(uint8_t)
x add operation::is_success(opcode)

x op_if (conditionally "minimal if")
x op_notif (conditionally "minimal if")
x op_checksigadd (conditionally invoke)
x op_checkmultisig (conditionally disable)
x op_checkmultisigverify (conditionally disable)
x op_checksigadd (conditionally enable)
- op_checksigadd (implement)
- op_checksig (conditionally modify)
- op_checksigverify (conditionally modify)
  op_checksigadd (conditionally serialize)

  Signature Hashing
  Signature Hashing Cache
  Signature Validation Invocation

  Taproot Witness Program Invocation
  Script Path Spending Parse and Validation
  Script Run Loop
  
  Tapscript Indication (unset BIP342 for non-tapscript program flags)
  Stacks element count limit (max_unified_stack_size also applied to initial stack)
  Sigops per-script budget (sigops in tapscripts do not count towards block limit)
  Script size limit (max_script_size does not apply)
  Non-push opcodes limit (max_counted_ops does not apply)
    
  Tagged Hash Midstate (optimization)
  Activation (simplified)
(edited)
🎉 1🚀 2

Wednesday, May 7, 2025

evoskuil02:58
Correction:
x add flags::bip341_rule
x add flags::bip342_rule
evoskuil03:55
Finished up the opcodes.

Thursday, May 8, 2025

Huddle with pmienk, evoskuil, Marnix · 72 min19:54
1 reply · pmienk · last May 08, 2025
pmienk20:11
Looks like now is a good time to reevaluate weekly meeting time. If there's a better time for anyone or if another time would facilitate attendance, throw out a day/time.
👍 1
neillm20:12
~2-2.5 hours earlier works for me 🤷‍♂️
👍🏻 1

Friday, May 9, 2025

evoskuil05:26
Good progress on taproot:
x add tagged hash
x add libsecp2561 schnorr functions
x add signature parse
x add flags::bip341_rule
x add flags::bip342_rule
x add coverage::hash_default
x upd coverage::all_rules script tests
x ren script_version::zero to segwit
x add script_version::taproot
x add operation::is_success(opcode)
x add transaction::is_sighash_valid(uint8_t)

x op_if (conditionally "minimal if")
x op_notif (conditionally "minimal if")
x op_checksigadd (conditionally invoke)
x op_checkmultisig (conditionally disable)
x op_checkmultisigverify (conditionally disable)
x op_checksigadd (conditionally enable)
x op_checksigadd (conditionally serialize)
x op_checksigadd (implement)
x op_checksig (conditionally modify)
x op_checksigverify (conditionally modify)

x Interpreter run loop changes
x Tapscript indication (unset BIP342 in program when not tapscript)
x Stacks element count limit (max_unified_stack_size also applied to initial stack)
x Script size limit (max_script_size does not apply)
x Non-push opcodes limit (max_counted_ops does not apply)
x Sigops per-script budget (sigops in tapscripts do not count towards block limit)
x Add success, failure (and roller) detection to script parser
x Cache success, failure (and roller) on script
x Modify script(operations) construction to set success, failure (and roller)
x Early success and fail on script run (from cache)
x Differentiate cached is_prefail and is_prevalid for tapscript execution.

- Signature Validation Invocation
  Signature Hashing Cache
  Signature Hashing

- Taproot Witness Program Invocation
  Taproot Witness Program Extraction
  Script Path Spending Parse and Validation
    
  Tagged Hash Midstate (optimization)
  Activation (simplified)
🔥 1
evoskuil05:34
On the call today I mentioned that we are now precomputing (1) success opcode, (2) invalid opcode, and (3) op_roll existence in all scripts during parse. This includes reading off the wire and from the store, though it's very efficient since the iteration is already necessary. Previously we were only testing for invalid during parse, and op_roll was computed on the fly prior to script execution. This saved one bit (bool) per script, which wasn't really worth it, since the two flags would just be packed into one byte by the compiler. And since we otherwise need another iteration for the success codes prior to tapscript execution, it was an easy choice to just put these all into the script parser, which basically looks like this (emplace invokes chain::op(reader) to perform an in-place construction on the operation stack).
    while (!source.is_exhausted())
    {
        ops_.emplace_back(source);
        const auto& op = ops_.back();
        easier_ |= op.is_success();
        failer_ |= op.is_invalid();
        roller_ |= op.is_roller();
    }
I also mentioned that because we can't really know if it will be executed as tapscript at the parse point, we record these for all scripts. But since the definition of invalid opcode has changed (reduced to just op_verif and op_vernotif in tapscript) and the unconditional success codes apply only in tapscript, it seemed that this would present some complexity and require additional parse results. But it turns out that above is all that is required. The last set of comments here explains how it works out, which was quite elegant.
TEMPLATE
INLINE error::script_error_t CLASS::
initialize() const NOEXCEPT
{
    const auto bip141 = is_enabled(flags::bip141_rule);
    const auto bip342 = is_enabled(flags::bip342_rule);

    // Apply stack element limit (520) to initial witness [bip141][bip342].
    if (bip141 && witness_ && !chain::witness::is_push_size(*witness_))
        return error::invalid_witness_stack;

    // Script size limit (10,000) [0.3.7+], removed [bip342].
    if (!bip342 && script_->is_oversized())
        return error::invalid_script_size;

    // Stacks element limit (1,000) applied to initial stack [bip342].
    if (bip342 && is_stack_overflow())
        return error::invalid_stack_size;

    // Succeed if any success code, one overrides all codes [bip342].
    if (bip342 && script_->is_prevalid())
        return error::prevalid_script;

    // Fail if last op underflow, lower priority than success codes [bip342].
    if (script_->is_underflow())
        return error::invalid_script;

    // Fail if any op invalid (invalid codes reduced in tapscript).
    // Should be after underflow check since underflow is also an invalid op.
    // Promoted success codes are not reachable here due to is_prevalid above.
    // So only op_verif/op_vernotif (unpromoted invalids) are caught here for
    // tapscript, otherwise is_prevalid bypassed and all invalids caught here.
    if (script_->is_prefail())
        return error::prefail_script;

    return error::script_success;
}
(edited)
The above code is invoked at the very beginning of the interpreter::run() method, which runs the script:
TEMPLATE
code CLASS::
run() NOEXCEPT
{
    // Enforce initial limits, determine early success or failure.
    if (const auto ec = state::initialize())
        return (ec == error::prevalid_script) ? error::script_success : ec;

    for (auto it = state::begin(); it != state::end(); ++it)
    {
        const auto& op = *it;

        if (op.is_oversized())
            return error::invalid_push_data_size;

        if (!state::ops_increment(op))
            return error::invalid_operation_count;

        if (state::if_(op))
        {
            if (const auto ec = run_op(it))
                return ec;

            if (state::is_stack_overflow())
                return error::invalid_stack_size;
        }
    }

    return state::is_balanced() ? error::script_success :
        error::invalid_stack_scope;
}
evoskuil18:15
I wrote this up in the comments, but basically I decided not to do anything regarding the supposedly new op_success codes. None are renamed, it's just another method to test whether a code is in the unconditional success category, and one invocation of that test in the above initialize() method. Otherwise nothing changes, and the codes retain their previous state (invalid or reserved) in non-tapscript executions. So just as there are no actual op_invalid codes, and no op_conditional codes, and some reserved codes that are not named op_reserved, unconditional success is just a new category - confused by the BIP presentation.
(edited)

Sunday, May 11, 2025

evoskuil04:41
Did a full review of signature operations counting, added back a much more efficient signature hash cache for op-multisig, integrated and rationalized tapscript op-codeseparator, did a bunch of refactoring. Gotta take a day off tomorrow, so may not be done for a couple more days.
C
cguida17:08
Did a talk at btc++ and libbitcoin featured prominently. Hope I didn't get too much stuff wrong haha
(edited)
🔥 1
evoskuil22:28
Seems like a bit of an identity crisis has been brewing. https://delvingbitcoin.org/t/antoine-poinsot-on-bitcoin-cores-priorities/1470
(edited)
Delving Bitcoin
@AntoineP wrote three interesting blog posts on Bitcoin Core: Antoine Poinsot - Bitcoin Core project direction “I think we are in this situation where everybody is working on their own thing and there is no broad agreement about what the scope of the project should even be. As a consequence, year after year we keep piling up more code, more features, more RPCs, etc… with at best a constant amount of competent reviewers’ time. This is not tenable, as a focus which gets further dispersed will i...

Monday, May 12, 2025

jungly01:38
It'll be very hard to agree on priorities and then to agree on how to address those. Even in centralized top down corporates this can be hard. In an open source community driven project, best of luck! Linus' strong leadership goes a long way in Linux. Do we have a Linus in Core?
TBH knots might have a better chance to evolve it's architecture. They have the social structure, but they might not have the desire to do any changes required
evoskuil01:38
Nope, it's a rudderless ship. Luke is taking the helm 😮
1 reply · cguida · last May 12, 2025
C
cguida01:55
pretty much haha
jungly01:39
Yeah. I see knots has the structure, and the capital. But do they want to take on scaling work?
evoskuil01:40
They will be lucky to keep up with dependency breaks, bugs and basic support.
💯 1
jungly01:40
TBH, the capital will also work against them, they need to focus on profits
evoskuil01:40
Much of the time we invested over the past decade has been in automation and maintainability. They have not made those investments. It's a rat's nest in there. I've been rooting around all week for Taproot. ugh
(edited)
That's true, money doesn't help until you're ready to scale.
(edited)
And of course I have to root around in taproot code because the BIP's are way underspecified.
1 reply · Enligths · last May 12, 2025
E
Enligths06:28
Is it so bad despite the fact you already wrote taproot in the past?
jungly01:54
Taproot, another symptom of the problems we have. I fundamentally think the problem is one implementation dominating the market. It can easily turn into an echo chamber
But then, what do I know. I am still trying to figure out Bitcoin 🙂‍↕️
😅 1
evoskuil02:09
Aren't we all.
💯 2
evoskuil02:43
There's just no way a single Bitcoin node to rule all others could hold up over time. They've become completely dysfunctional, and that has nothing to do with individual decisions such as op_return or zero-conf policies, taproot, segwit, or lack of action on covenants. The failure is built directly in to the organizational structure. It's led to extreme arrogance, and the backlash has created defensiveness and insularity. It was always bound to fail, just required a Schelling point, and maybe that's what this is. Because the reaction is far more significant than the technical issue. It's years of pent up frustration from all of the corners that they have pissed off, and Luke providing a trivial outlet.
😬 1👍 1
evoskuil02:55
I also think that we have made the difference here. Knots is not an answer, and everyone knows that. But we are coming, and they know that too. Core is no longer dismissing us, they are frantically attempting to figure out what to do about us. That's where the soul searching and renewed attempts at performance engineering are coming from, directly. Previously they would have laughed off the filter bois, but this time is different. They have been sitting on their collective ass for a decade without even knowing it - despite being repeatedly told. Now they have had to face reality. Their performance blows, their code is a nightmarishly brittle and tangled mess, a major rearchitecture is required, and both organizationally and technically they have no ability to fix it. It's a monumental undertaking, I know. And it will be nearly impossible given their organization tensions.
(edited)
💯 2👊🏻 1👊 1
10 replies · cguida, evoskuil · last May 13, 2025
C
cguida00:12
Previously they applied filters to fight spam, it is only recently that they've seemingly stopped trying that method
evoskuil00:27
They have all come to the inevitable realization that it doesn't work. Maxwell even had to recently walk back his much earlier comments in support of it. I have said it from the beginning. zero-conf, op-data, etc... all policy will fail. Which is why I put zero effort into it. If it's not enforced, it will fail.
evoskuil00:28
You could make the same argument that not validating is good enough. It's not, if people don't validate the system will fail.
C
cguida00:46
Seems to me that filters disincentivize spam if run by a supermajority of nodes
evoskuil02:03
Does one person validating disincentivize scamming other people who aren’t? Sure. Do you think that people not validating will get scammed constantly. Of course they will.
evoskuil02:06
This is why we don’t build systems that rely on zero conf. We explicitly want it to not be used. It’s the same reason we don’t rely on “discouragement” (though both were tried with policy).
💯 1
evoskuil02:12
Whatever space can be filled by consensus will be filled. The only question is how much will people have paid for it. No demand, free space.
evoskuil02:30
This is just how it works, and there is no way around it. Space is an inherent cost of the system, and fortunately that's a very, very small cost.
C
cguida13:35
Yes, I agree, in the context of full validation
evoskuil15:20
But we don’t relay invalid txs, so that limits the scamming.
evoskuil03:03
It's become so bad that there are now testimonials going up about how one company or anther will continue to run Core. It's like those old Buy American ads going up as Detroit went down.
✔️ 1😅 1
E
Enligths06:28
replied to a thread: view
Is it so bad despite the fact you already wrote taproot in the past?
evoskuil07:13
It was never completed in the past, and it's an extremely complex design - so not exactly keeping all of the consensus-critical details in my head for 5 years ;).
E
Enligths08:40
Oh yeah I see
evoskuil19:25
Last meeting I mentioned that I have a taproot task to expose a precomputed midstate constructor in the hash accumulator template. Well turns out I did that a few years ago. :))
    /// Sets initial state to Hash initialization vector.
    constexpr accumulator() NOEXCEPT;

    /// Accepts an initial state and count of blocks accumulated by it.
    constexpr accumulator(const state_t& state, size_t blocks) NOEXCEPT;
😄 1🎉 3
evoskuil19:55
Given that all of our hash functions (apart from accumulator) are constexpr, we can simply construct the midstate under consteval and initialize the accumulator with it for each streaming tagged hash.

Wednesday, May 14, 2025

pmienk15:45
If there's no objection, I'll move tomorrow's meeting two hours earlier to 1pm EST/10am PST.
👍🏻 1👍 1
C
cguida20:47
Eric’s interview in PubKey, good stuff 👍 https://open.spotify.com/episode/6OW3ZNetSmsokfQ29uiggI
🤓 1💥 1🎉 2

Thursday, May 15, 2025

Huddle with pmienk, evoskuil, parsevalbtc, eynhaender · 69 min16:50
neillm16:51
Gah, I wasn't planning on it today, but will try to for next week! Today has been pretty out of the ordinary 😅 This time is good for me though generally, so I hope it works for others.
1 reply · pmienk · last May 15, 2025
pmienk16:53
Not a problem, catch you next week.
🙏 1
evoskuil17:01
Be there in a minute
👍 1
eynhaender17:57
its the first time i try to talk on this computer
its the headset
Sounds great but its not the reason why Iḿ here
👍🏻 1
😉

Saturday, May 17, 2025

evoskuil05:17
Signature hashing is just about done (just need the tapscript bool and tapleaf hash values):
bool transaction::version1_sighash(hash_digest& out,
    const input_iterator& input, const script& script, uint64_t value,
    uint8_t sighash_flags) const NOEXCEPT
{
    constexpr uint8_t epoch{};

    // TODO: obtain tapscript indicator and access to tapleaf hash.
    constexpr bool tapscript{};
    constexpr hash_digest tapleaf{};

    const auto& in = **input;
    const auto& annex = in.witness().annex();

    // Mask anyone_can_pay, and set hash_all by default.
    // sighash_flags previously verified (see schnorr_split).
    const auto flag = mask_sighash(sighash_flags);
    const auto anyone = is_anyone_can_pay(sighash_flags);
    const auto single = (flag == coverage::hash_single);
    const auto all = (flag == coverage::hash_all);

    // Create tagged hash writer.
    stream::out::fast stream{ out };
    hash::sha256t::fast<"TapSighash"> sink{ stream };

    sink.write_byte(epoch);
    sink.write_byte(sighash_flags);
    sink.write_4_bytes_little_endian(version_);
    sink.write_4_bytes_little_endian(locktime_);

    if (!anyone)
    {
        sink.write_bytes(single_hash_points());
        sink.write_bytes(single_hash_amounts());
        sink.write_bytes(single_hash_scripts());
        sink.write_bytes(single_hash_sequences());
    }

    if (all)
    {
        sink.write_bytes(single_hash_outputs());
    }

    sink.write_byte(spend_type_v1(annex, tapscript));

    if (anyone)
    {
        in.point().to_data(sink);
        sink.write_8_bytes_little_endian(value);
        script.to_data(sink, true);
        sink.write_4_bytes_little_endian(in.sequence());
    }
    else
    {
        sink.write_4_bytes_little_endian(input_index(input));
    }

    if (annex)
    {
        sink.write_variable(annex.size());
        sink.write_bytes(annex.hash());
    }

    if (single)
    {
        const auto index = input_index(input);
        if (output_overflow(index))
            return false;

        sink.write_bytes(outputs_->at(index)->hash());
    }

    // Additional for tapscript [bip342].
    if (tapscript)
    {
        sink.write_bytes(tapleaf);
        sink.write_byte(to_value(key_version::tapscript));
        sink.write_4_bytes_little_endian(subscript_v1(script));
    }

    sink.flush();
    return true;
}
Ended up creating an annex object to provide the above clean abstraction. On order to make that fully general and safe it required a shared_ptr per input, and computationally it's almost neutral. Really cleans up the above code though. The tagged hash writer [hash::sha256t::fast<"TapSighash">] was also a nice simplification, with no performance compromises. Still have to complete the hash caching for this. Also some related refactoring allowed me to share code with the original and segwit signature hashing functions. I split up a few files also, which helped with organization.
(edited)
evoskuil05:46
During the meeting we discussed using fabricated scripts for both the key path spend execution and the fall-through success. These are now implemented, trivial, and very optimal.
static script checksig_script() NOEXCEPT
{
    static const script cached{ { { opcode::checksig } } };
    return cached;
}

static script::cptr success_script_ptr() NOEXCEPT
{
    constexpr auto op_success = opcode::reserved_80;
    static_assert(operation::is_success(op_success));
    static const auto cached = to_shared<script>({ { op_success } });
    return cached;
}

static script::cptr checksig_script_ptr() NOEXCEPT
{
    static const auto cached = to_shared<script>(checksig_script());
    return cached;
}        

...

        case script_version::taproot:
        {
            // witness stack : [annex]...
            if (program->size() == hash_size)
            {
                auto stack_size = out_stack->size();

                // If at least two elements, discard annex if present.
                if (drop_annex(*out_stack))
                    --stack_size;

                // p2ts (tapscript, script path spend)
                // witness stack : <control><script>[stack-elements]
                // input script  : (empty)
                // output script : <1> <32-byte-tweaked-public-key>
                if (stack_size > one)
                {
                    // The last stack element is the control block.
                    const auto& control = pop(*out_stack);

                    // Control length must be 33 + 32m, for integer m [0..128].
                    if (!is_valid_control_block(*control))
                        return error::invalid_witness;

                    // The second-to-last stack element is the script.
                    out_script = to_shared<script>(*pop(*out_stack), false);

                    // TODO: DO SOME NASTY SHIT WITH CONTROL AND SCRIPT.
                    // TODO: MUST OBTAIN tapleaf_hash for signature hash.
                    // q is referred to as `taproot output key`.
                    // p is referred to as `taproot internal key`.

                    // Execute script with remaining stack.
                    return error::script_success;
                }

                // p2tr (taproot, key path spend)
                // witness stack : <signature>
                // input script  : (empty)
                // output script : <1> <32-byte-tweaked-public-key>
                if (is_one(stack_size))
                {
                    // Stack element is a signature that must be valid for q.
                    // Program is q, a 32 byte bip340 public key, so push it.
                    out_stack->push_back(program);
                    out_script = checksig_script_ptr();

                    // out_stack  : <public-key><signature>
                    // out_script : <op_checksig>
                    return error::script_success;
                }

                // Fail if witness stack was empty.
                return error::invalid_witness;
            }

            // This op_success script succeeds immediately with bip342 active.
            out_script = success_script_ptr();
            out_stack->clear();

            // Version 1 other than 32 bytes remain unencumbered (success).
            return error::script_success;
        }
(edited)

Sunday, May 18, 2025

evoskuil02:27
Here's that nasty section we talked about at the meeting, and from above... now complete (comments stripped). All of the computation is pushed into verify_commitment(...). That can't be pushed into a script execution as with the other couple scenarios (assured success and simple checksig). But at least verify_commitment is static, so it can be moved out of this witness class parse method.
        case script_version::taproot:
        {
            if (program->size() == ec_xonly_size)
            {
                auto stack_size = out_stack->size();
                if (drop_annex(*out_stack))
                    --stack_size;

                if (stack_size > one)
                {
                    const auto control = pop(*out_stack);
                    if (!is_valid_control_block(*control))
                        return error::invalid_witness;

                    out_script = to_shared<script>(*pop(*out_stack), false);
                    if (const auto tap = taproot::parse(*control))
                    {
                        using namespace taproot;
                        out_leaf = to_shared(leaf_hash(tap.version, *out_script));
                        if (!verify_commitment(*control, *program, *out_leaf,
                            tap.parity)) return error::invalid_commitment;

                        return error::script_success;
                    }

                    out_script = success_script_ptr();
                    out_stack->clear();
                    return error::script_success;
                }

                if (is_one(stack_size))
                {
                    out_stack->push_back(program);
                    out_script = checksig_script_ptr();
                    return error::script_success;
                }

                return error::invalid_witness;
            }

            out_script = success_script_ptr();
            out_stack->clear();
            return error::script_success;
        }
That one function is all this:
namespace taproot {

static hash_digest leaf_hash(uint8_t version,
    const script& script) NOEXCEPT
{
    hash_digest out{};
    stream::out::fast stream{ out };
    hash::sha256t::fast<"TapLeaf"> sink{ stream };
    sink.write_byte(version);
    script.to_data(sink, true);
    sink.flush();
    return out;
}

static hash_digest tweak_hash(const ec_xonly& key,
    const hash_digest& merkle) NOEXCEPT
{
    hash_digest out{};
    stream::out::fast stream{ out };
    hash::sha256t::fast<"TapTweak"> sink{ stream };
    sink.write_bytes(key);
    sink.write_bytes(merkle);
    sink.flush();
    return out;
}

static hash_digest branch_hash(const hash_digest& left,
    const hash_digest& right) NOEXCEPT
{
    hash_digest out{};
    stream::out::fast stream{ out };
    hash::sha256t::fast<"TapBranch"> sink{ stream };

    if (std::lexicographical_compare(left.begin(), left.end(),
        right.begin(), right.end()))
    {
        sink.write_bytes(left);
        sink.write_bytes(right);
    }
    else
    {
        sink.write_bytes(right);
        sink.write_bytes(left);
    }

    sink.flush();
    return out;
}

static hash_digest merkle_root(const data_chunk& control,
    const hash_digest& tapleaf_hash) NOEXCEPT
{
    BC_ASSERT(is_valid_control_block(control));

    constexpr auto start = add1(ec_xonly_size);
    const auto bytes = floored_subtract(control.size(), start);
    const auto count = floored_divide(bytes, ec_xonly_size);
    const auto begin = std::next(control.data(), start);
    const auto nodes = unsafe_array_cast<ec_xonly, taproot_max_keys>(begin);

    hash_digest hash{ tapleaf_hash };
    for (size_t node{}; node < count; ++node)
        hash = branch_hash(hash, nodes.at(node));

    return hash;
}

static bool verify_commitment(const data_chunk& control,
    const data_chunk& program, const hash_digest& hash,
    bool parity) NOEXCEPT
{
    BC_ASSERT(is_valid_control_block(control));

    const auto out = program.data();
    const auto& out_key = unsafe_array_cast<uint8_t, ec_xonly_size>(out);
    const auto in = std::next(control.data());
    const auto& in_key = unsafe_array_cast<uint8_t, ec_xonly_size>(in);
    const auto root = merkle_root(control, hash);
    const auto tweak = tweak_hash(out_key, root);
    return schnorr::verify_commitment(in_key, tweak, out_key, parity);
}

struct tap
{
    operator bool() const NOEXCEPT { return version == tapscript_version; }
    uint8_t version;
    bool parity;
};

static tap parse(const data_chunk& control) NOEXCEPT
{
    BC_ASSERT(!control.empty());

    const auto byte = control.front();
    return
    {
        bit_and(byte, tapscript_mask),
        to_bool(bit_and(byte, bit_not(tapscript_mask)))
    };
}

} // namespace taproot
That's enough to ship into its own static class (vs. the namespace) and place into a new chain class/file. I prefer static classes over namespaces because they allow us to create private and protected utilities in the namespace, which are then reachable by derived test accessor.
So almost there:
x add tagged hash
x add libsecp2561 schnorr functions
x add signature parse
x add flags::bip341_rule
x add flags::bip342_rule
x add coverage::hash_default
x upd coverage::all_rules script tests
x ren script_version::zero to segwit
x add script_version::taproot
x add operation::is_success(opcode)
x add transaction::is_sighash_valid(uint8_t)
x add codeseparator index default constant

x op_if (conditionally "minimal if")
x op_notif (conditionally "minimal if")
x op_checksigadd (conditionally invoke)
x op_checkmultisig (conditionally disable)
x op_checkmultisigverify (conditionally disable)
x op_checksigadd (conditionally enable)
x op_checksigadd (conditionally serialize)
x op_checksigadd (implement)
x op_checksig (conditionally modify)
x op_checksigverify (conditionally modify)

x Interpreter run loop changes
x Tapscript indication (unset BIP342 in program when not tapscript)
x Stacks element count limit (max_unified_stack_size also applied to initial stack)
x Script size limit (max_script_size does not apply)
x Non-push opcodes limit (max_counted_ops does not apply)
x Sigops per-script budget (sigops in tapscripts do not count towards block limit)
x Add success, failure (and roller) detection to script parser
x Cache success, failure (and roller) on script
x Modify script(operations) construction to set success, failure (and roller)
x Early success and fail on script run (from cache)
x Differentiate cached is_prefail and is_prevalid for tapscript execution.

x Signature validation invocation
x Expose codeseparator index with default
x Signature operation legacy count review
x Multisig hash cache by sighash byte
x Signature hashing

x Taproot Witness Program Invocation
x Taproot Witness Program Extraction
x Script Path Spending Parse and Validation
x Key Path Spending Parse and Validation

x Tagged Hash Midstate (optimization)
x Tagged hash stream writers with midstate optimizations
  Incorporate script test vectors
  Activation (simplified)
  
evoskuil18:00
Just a heads up, there's a segwit regression in -system/master latest (for full validation).
(edited)

Wednesday, May 21, 2025

evoskuil22:51
Well, just a static keyword inexplicably out of place, only manifests under highly-concurrent signature hashing.
evoskuil23:53
Fix is merged.

Thursday, May 22, 2025

evoskuil01:50
Taproot initial implementation is complete, apart from new tests and activation.
🚀 4
evoskuil04:12
Recent changes for taproot, and some organizer optimizations, resulted in shaving ~5 mins off of the maxi milestone run, 52 mins.
evoskuil04:46
Full validation run now. Tracking close to the record for validate, but confirmation, which is loosely coupled, is significantly faster than previous runs. Like half the time. If that holds it’s def the result of the organizer/confirmer optimizations/simplifications I made. I was in a rabbit hole on that for three days because of the other big triggering candidate chain disorganization logic. I found some real waste in the confirmer as a side effect of improving them both.
That would show up more on the mini, as total chain confirmation cost on the maxi is only about 15 minutes out of ~4.5 hrs. But it could drop the mini down around 6.5 hrs.
Also this is with all the taproot code in, but without it active. Just getting a baseline before activation.
evoskuil05:15
At about 25% complete, validation has surpassed the record run by a couple minutes, and confirmation is 1/3 the time of the record run.
👀 1
evoskuil15:34
Maxi full validation in 4.11 hrs, a new record. Previous best was 4.42 hrs. Validation completed to 850,000 in 4.08 hrs and at that point confirmation was at 838,485. In the previous record run confirmation was at 680,286 when validation completed at 4.17 hrs. So validation got a little faster and confirmation got a lot faster. Or it could be that validation didn't change but the reduce cost of confirmation slowed it down less. Would have to run validation without confirmation to isolate that question.
evoskuil15:44
The major change to confirmation is the removal of work accumulation in the candidate branch for comparison to the confirmed branch in the case where no blocks in the confirmed branch are to be reorganized, since the comparison in that case is always against zero and all blocks have positive work. This removes database queries that are unnecessary in all typical confirmations (simplified).
void chaser_confirm::do_bumped() NOEXCEPT
{
    size_t fork_point{};
    const auto& query = archive();
    const auto top = query.get_top_confirmed();
    auto fork = query.get_validated_fork(fork_point, checkpoint());

    if (fork_point > top)
    {
        fault(error::confirm1);
        return;
    }

    // ADDED THIS CONDITION...
    if (fork_point < top)
    {
        uint256_t work{};
        if (!query.get_work(work, fork))
        {
            fault(error::confirm2);
            return;
        }

        bool strong{};
        if (!query.get_strong_fork(strong, work, fork_point))
        {
            fault(error::confirm3);
            return;
        }

        if (!strong)
            return;
    }

    reorganize(fork, top, fork_point);
}
(edited)
Huddle with pmienk, evoskuil, neillm, eynhaender, Marnix, Rowan Voskuil, parsevalbtc, Enligths · 219 min16:55
3 replies · neillm, parsevalbtc, Marnix · last May 22, 2025
neillm16:56
I'm on my way
P
parsevalbtc19:41
Sorry guys connection sucks
M
Marnix20:02
Gotta go, bye
eynhaender18:38
👀 1🔥 2
neillm18:49
As far as I got with getting server building 🤷‍♂️ https://github.com/libbitcoin/libbitcoin-server/pull/575
🔥 1
I'll switch to network 🙂
This clears up the questions I had on duplexing. Our p2p operates over TCP which is full duplex, but the semantics of the protocols are more like half duplex, where there is a request/response, even though either party can initiate and one party can simply broadcast to the other (like tx announcements). HTTP is the same, I don't see any reason the implementation would be affected. But interestingly, WebSockets can also be implemented on the same stack, since it's just a different semantic. IOW we can broadcast from a notification subscriber over the socket, and the client can also spend requests.
✅ 1🔥 1
evoskuil23:10
Awesome meeting this week, so much happening!
💯 4

Friday, May 23, 2025

eynhaender04:14
9uw3fb.jpg
🤣 1❤️ 4✔️ 2
evoskuil16:18
I just ran a full validation with Electrum address index (110GiB) and compact filters (10GiB) on the maxi. Last best run was 5.6 hrs (May 13). This completed in 4.6 hrs (4.592) a full hour faster! The previous run, without optional indexes, was 4.11 hrs, so just 30 minutes for the combined 120GiB of optional indexation on the maxi.
(edited)
🚀 4
S
Sartoshi23:29
Hey Eric what are the maxi specs?
3 replies · pmienk, evoskuil · last May 24, 2025
pmienk23:48
I think he's referring to a dual xeon E5-2683v4 with 256gb ram, nvme PCIe gen3 ssd with 2.5gb nic and 2.3gb connection. Not sure the model
evoskuil13:51
That's about right, it's a Dell Precision 7810.
evoskuil13:52
The mini is actually faster except for validation, despite having only 32GB RAM, since it's got faster I/O.
(edited)
👍 1

Saturday, May 24, 2025

evoskuil14:17
Does anyone have an ARM machine that they would like to use for testing sha vectorizations and native sha extensions?
5 replies · jungly, evoskuil · last May 26, 2025
jungly17:36
For my past sins I was given a Mac M1. I have Linux on it now.
(edited)
jungly17:37
Let me know what benchmarks/tests you want to run. I can try.
evoskuil17:43
Ok, I'll let you know. I had grok translate my intrinsics macros from intel into arm equivalents, and it thinks it will be drop-in equivalent. Fingers crossed.
🤞🏾 1👍 1
evoskuil03:07
I just learned that ARM SVE gives us up to 4 concurrent native SHA hashes via native sha in 512 bits (128 each), which should be 4x faster than SHANI.
💥 1
jungly03:21
wow.. sweet

Sunday, May 25, 2025

evoskuil17:21
I've been working through the process of getting us formally up on ARM, which means integrating the 128/256/512 bit vectorization and native sha CPU instructions, and basically doubling our platform support. One limitation that we presently have on the vc++ builds is that there are no ARM boost NuGet packages. Otherwise I would update the two that I maintain (secp256k1 and libzmq). Another is that I don't have an ARM box. And finally our CI ARM coverage is presently limited to macOS (and we apparently not longer have macOS on Intel CI). But this is all pretty manageable because I can at least cross-compile for now, and get the build working on vc++ ARM and macOS. And @jungly can debug on Linux builds if I can't get the optimizations working. After that they are at least in CI on macOS. Presently that runs but without any of the optimizations.
✔️ 2
I ran into a boost ARM bug in our latest version, where boost was invoking Intel intrinsics whenever that specific header is present, but the header is always present on ARM as well. No way to fix that except to upgrade, so I started at boost latest (1.87) and that has a problem in the NuGet packages (one unresolved external in the std lib) so I dropped down to 1.86 and it all looks good.
evoskuil17:48
In that process I spent some time cleaning up years of cruft in our boost dependencies. I was able to remove chrono, thread (except on non-win32 and with ICU), optional, utility, lexical-cast, algorithm, and will soon have regex removed. json has been moved to header only (required an include and a couple defines), along with its dependency container. And the remaining system was being NuGet included, along with others above, in vc++ but are header only. So those are now implicitly included (by the base "targets" directive).
🎉 2
evoskuil18:12
arm: 32/64, linux/macos/win, conditional sha, neon, sve (variable width) intrinsics intel: 32/64, linux/macos/win, conditional shani, sse41, avx2, avx512 intrinsics linux: make/cmake, gcc/clang, conditional ICU (unicode) macos: make/cmake, xcode, conditional ICU (unicode) win: visual studio all: release/debug builds, vscode support, installers, CI matrix
As you can see ARM and the additional 4 intrinsics scenarios are exploding the support matrix.

Thursday, May 29, 2025

evoskuil06:53
It looks like ARM vectorization is active and successful on CI via macOS. Really just came down to defining for ARM the same set of intrinsics wrappers that we have for Intel (below). The sha template just picks up this vs. the Intel version. And this file was entirely generated by Grok, with some minor fixes and cosmetics from me.
#ifndef LIBBITCOIN_SYSTEM_INTRINSICS_NEON_128_HPP
#define LIBBITCOIN_SYSTEM_INTRINSICS_NEON_128_HPP

#include <bitcoin/system/define.hpp>
#include <bitcoin/system/intrinsics/types.hpp>
#include <bitcoin/system/intrinsics/platforms/neon.hpp>

// All operations use NEON intrinsics.

namespace libbitcoin {
namespace system {

#if defined(HAVE_NEON)

namespace f {

/// bitwise primitives
/// ---------------------------------------------------------------------------
/// 32 bit functions are applied because xint128_t is typed as uint32x4_t.

INLINE xint128_t and_(xint128_t a, xint128_t b) NOEXCEPT
{
    return vandq_u32(a, b);
}

INLINE xint128_t or_(xint128_t a, xint128_t b) NOEXCEPT
{
    return vorrq_u32(a, b);
}

INLINE xint128_t xor_(xint128_t a, xint128_t b) NOEXCEPT
{
    return veorq_u32(a, b);
}

INLINE xint128_t not_(xint128_t a) NOEXCEPT
{
    return xor_(a, vdupq_n_u32(-1));
}

/// vector primitives
/// ---------------------------------------------------------------------------

template <auto B, auto S>
INLINE xint128_t shr(xint128_t a) NOEXCEPT
{
    if constexpr (S == bits<uint8_t>)
        return (xint128_t)vshrq_n_u8((uint8x16_t)a, B);
    if constexpr (S == bits<uint16_t>)
        return (xint128_t)vshrq_n_u16((uint16x8_t)a, B);
    if constexpr (S == bits<uint32_t>)
        return (xint128_t)vshrq_n_u32((uint32x4_t)a, B);
    if constexpr (S == bits<uint64_t>)
        return (xint128_t)vshrq_n_u64((uint64x2_t)a, B);
}

template <auto B, auto S>
INLINE xint128_t shl(xint128_t a) NOEXCEPT
{
    if constexpr (S == bits<uint8_t>)
        return (xint128_t)vshlq_n_u8((uint8x16_t)a, B);
    if constexpr (S == bits<uint16_t>)
        return (xint128_t)vshlq_n_u16((uint16x8_t)a, B);
    if constexpr (S == bits<uint32_t>)
        return (xint128_t)vshlq_n_u32((uint32x4_t)a, B);
    if constexpr (S == bits<uint64_t>)
        return (xint128_t)vshlq_n_u64((uint64x2_t)a, B);
}

template <auto B, auto S>
INLINE xint128_t ror(xint128_t a) NOEXCEPT
{
    return or_(shr<B, S>(a), shl<S - B, S>(a));
}

template <auto B, auto S>
INLINE xint128_t rol(xint128_t a) NOEXCEPT
{
    return or_(shl<B, S>(a), shr<S - B, S>(a));
}

template <auto S>
INLINE xint128_t add(xint128_t a, xint128_t b) NOEXCEPT
{
    if constexpr (S == bits<uint8_t>)
        return (xint128_t)vaddq_u8((uint8x16_t)a, (uint8x16_t)b);
    if constexpr (S == bits<uint16_t>)
        return (xint128_t)vaddq_u16((uint16x8_t)a, (uint16x8_t)b);
    if constexpr (S == bits<uint32_t>)
        return (xint128_t)vaddq_u32((uint32x4_t)a, (uint32x4_t)b);
    if constexpr (S == bits<uint64_t>)
        return (xint128_t)vaddq_u64((uint64x2_t)a, (uint64x2_t)b);
}

template <auto K, auto S>
INLINE xint128_t addc(xint128_t a) NOEXCEPT
{
    if constexpr (S == bits<uint8_t>)
        return add<S>(a, (xint128_t)vdupq_n_u8((uint8_t)K));
    if constexpr (S == bits<uint16_t>)
        return add<S>(a, (xint128_t)vdupq_n_u16((uint16_t)K));
    if constexpr (S == bits<uint32_t>)
        return add<S>(a, (xint128_t)vdupq_n_u32((uint32_t)K));
    if constexpr (S == bits<uint64_t>)
        return add<S>(a, (xint128_t)vdupq_n_u64((uint64_t)K));
}

} // namespace f

/// add/broadcast/get/set
/// ---------------------------------------------------------------------------

template <typename Word, if_integral_integer<Word> = true>
INLINE xint128_t add(xint128_t a, xint128_t b) NOEXCEPT
{
    if constexpr (is_same_type<Word, uint8_t>)
        return (xint128_t)vaddq_u8((uint8x16_t)a, (uint8x16_t)b);
    if constexpr (is_same_type<Word, uint16_t>)
        return (xint128_t)vaddq_u16((uint16x8_t)a, (uint16x8_t)b);
    if constexpr (is_same_type<Word, uint32_t>)
        return (xint128_t)vaddq_u32((uint32x4_t)a, (uint32x4_t)b);
    if constexpr (is_same_type<Word, uint64_t>)
        return (xint128_t)vaddq_u64((uint64x2_t)a, (uint64x2_t)b);
}

template <typename xWord, typename Word, if_integral_integer<Word> = true,
    if_same<xWord, xint128_t> = true>
INLINE xint128_t broadcast(Word a) NOEXCEPT
{
    if constexpr (is_same_type<Word, uint8_t>)
        return (xint128_t)vdupq_n_u8((uint8_t)a);
    if constexpr (is_same_type<Word, uint16_t>)
        return (xint128_t)vdupq_n_u16((uint16_t)a);
    if constexpr (is_same_type<Word, uint32_t>)
        return (xint128_t)vdupq_n_u32((uint32_t)a);
    if constexpr (is_same_type<Word, uint64_t>)
        return (xint128_t)vdupq_n_u64((uint64_t)a);
}

// Lane zero is lowest order word.
template <typename Word, auto Lane, if_integral_integer<Word> = true>
INLINE Word get(xint128_t a) NOEXCEPT
{
    if constexpr (is_same_type<Word, uint8_t>)
        return (Word)vgetq_lane_u8((uint8x16_t)a, Lane);
    if constexpr (is_same_type<Word, uint16_t>)
        return (Word)vgetq_lane_u16((uint16x8_t)a, Lane);
    if constexpr (is_same_type<Word, uint32_t>)
        return (Word)vgetq_lane_u32((uint32x4_t)a, Lane);
    if constexpr (is_same_type<Word, uint64_t>)
        return (Word)vgetq_lane_u64((uint64x2_t)a, Lane);
}

// Low order word to the left.
template <typename xWord, if_same<xWord, xint128_t> = true>
INLINE xint128_t set(uint64_t x01, uint64_t x02) NOEXCEPT
{
    return (xint128_t)vcombine_u64(vcreate_u64(x01), vcreate_u64(x02));
}

template <typename xWord, if_same<xWord, xint128_t> = true>
INLINE xint128_t set(
    uint32_t x01, uint32_t x02, uint32_t x03, uint32_t x04) NOEXCEPT
{
    const uint32_t a[]{ x01, x02, x03, x04 };
    return (xint128_t)vld1q_u32(a);
}

template <typename xWord, if_same<xWord, xint128_t> = true>
INLINE xint128_t set(
    uint16_t x01, uint16_t x02, uint16_t x03, uint16_t x04,
    uint16_t x05, uint16_t x06, uint16_t x07, uint16_t x08) NOEXCEPT
{
    auto low  = vcreate_u16(
        (uint64_t)x02 << 16 | (uint64_t)x01 << 0);
    low  = vset_lane_u16((uint16_t)x03, low, 2);
    low  = vset_lane_u16((uint16_t)x04, low, 3);
    auto high = vcreate_u16(
        (uint64_t)x06 << 16 | (uint64_t)x05 << 0);
    high = vset_lane_u16((uint16_t)x07, high, 2);
    high = vset_lane_u16((uint16_t)x08, high, 3);
    return (xint128_t)vcombine_u16(low, high);
}


template <typename xWord, if_same<xWord, xint128_t> = true>
INLINE xint128_t set(
    uint8_t x01, uint8_t x02, uint8_t x03, uint8_t x04,
    uint8_t x05, uint8_t x06, uint8_t x07, uint8_t x08,
    uint8_t x09, uint8_t x10, uint8_t x11, uint8_t x12,
    uint8_t x13, uint8_t x14, uint8_t x15, uint8_t x16) NOEXCEPT
{
    auto low = vcreate_u8(
        (uint64_t)x04 << 24 | (uint64_t)x03 << 16 |
        (uint64_t)x02 <<  8 | (uint64_t)x01 << 0);
    low = vset_lane_u8((uint8_t)x05, low, 4);
    low = vset_lane_u8((uint8_t)x06, low, 5);
    low = vset_lane_u8((uint8_t)x07, low, 6);
    low = vset_lane_u8((uint8_t)x08, low, 7);
    auto high = vcreate_u8(
        (uint64_t)x12 << 24 | (uint64_t)x11 << 16 |
        (uint64_t)x10 <<  8 | (uint64_t)x09 << 0);
    high = vset_lane_u8((uint8_t)x13, high, 4);
    high = vset_lane_u8((uint8_t)x14, high, 5);
    high = vset_lane_u8((uint8_t)x15, high, 6);
    high = vset_lane_u8((uint8_t)x16, high, 7);
    return (xint128_t)vcombine_u8(low, high);
}

/// endianness
/// ---------------------------------------------------------------------------

template <typename Word, if_same<Word, uint8_t> = true>
INLINE xint128_t byteswap(xint128_t a) NOEXCEPT
{
    return (xint128_t)a;
}

template <typename Word, if_same<Word, uint16_t> = true>
INLINE xint128_t byteswap(xint128_t a) NOEXCEPT
{
    return (xint128_t)vrev16q_u8((uint8x16_t)a);
}

template <typename Word, if_same<Word, uint32_t> = true>
INLINE xint128_t byteswap(xint128_t a) NOEXCEPT
{
    return (xint128_t)vrev32q_u8((uint8x16_t)a);
}

template <typename Word, if_same<Word, uint64_t> = true>
INLINE xint128_t byteswap(xint128_t a) NOEXCEPT
{
    return (xint128_t)vrev64q_u8((uint8x16_t)a);
}

/// load/store
/// ---------------------------------------------------------------------------

INLINE xint128_t load(const xint128_t& bytes) NOEXCEPT
{
    return vld1q_u32((uint32_t*)&bytes);
}

INLINE void store(xint128_t& bytes, xint128_t a) NOEXCEPT
{
    vst1q_u32((uint32_t*)&bytes, a);
}

INLINE xint128_t load_aligned(const xint128_t& bytes) NOEXCEPT
{
    return vld1q_u32((uint32_t*)&bytes);
}

INLINE void store_aligned(xint128_t& bytes, xint128_t a) NOEXCEPT
{
    vst1q_u32((uint32_t*)&bytes, a);
}

#endif // HAVE_NEON

} // namespace system
} // namespace libbitcoin

#endif
(edited)
💥 1
ARM Neon is the common intrinsics library (there's a newer one but not widely deployed). This usually includes SHA intrinsics, but is limited to 128 bit vectorization. On Intel we get 128, 256, and 512 with newer machines, as well as SHA (which often is not present). The 128 bit vectorization helps when SHA is not present, but it's usually present with Neon. And native SHA is much faster, so more important to get that incorporated. That's a similar process, though the semantics of the intrinsics differ between ARM and Intel, so it's not quite as straightforward to get a clean common implementation.
(edited)
evoskuil07:04
I also did a custom software implementation of intrinsics that Intel doesn't provide but are necessary to round out the vectorization library. Specifically that gives us the ability to vectorize sha512 on 32 bit builds. Obviously not the highest priority, but not a lot of work and cleans up some code. Unfortunately it's not quite done, as I have native sha512 test failures across the board on 32 bit Intel. It's probably good on 32 bit ARM with Neon, because those are all natively implemented, but I have no test platform for that.
The code splits at intrinsics so that we can maintain constexpr hash generations, and intrinsics can't be constexpr. And all of the constexpr tests for the same vectors are passing.
Huddle with pmienk, evoskuil, neillm, Marnix, cguida, Rowan Voskuil, eynhaender, Enligths · 111 min16:54
12 replies · cguida, Marnix, pmienk, neillm +2 · last May 29, 2025
C
cguida17:07
i can test on arm mac
M
Marnix17:10
Sry if I DoS attacked the meeting with joining. Slack kept telling me I was already in the huddle and kept crashing
pmienk17:11
no problem, didn't see anything
👍 1
C
cguida17:11
lol, saw you rejoining a bunch of times but it didn't disrupt anything
👍 1
neillm17:30
Gotta run for today 😞 Nice work, Eric!
👍🏻 1
Rowan Voskuil18:33
unfortunately I need to head out. Eric ill reach out about some questions later today. 👍
👍🏻 1
evoskuil19:34
Really doesn’t wanna come off.
IMG_9343
💪 1
evoskuil20:08
Trashed the first extractor…
IMG_9345IMG_9346
evoskuil20:10
Hardened steel. 🤨 Then I noticed I had a slightly larger one. Found a little better spot, and now after about 400 foot pounds, there’s a gap.
IMG_9348IMG_9349
evoskuil20:10
So now Kroil is doing its thing while I code. 🤓
IMG_9350IMG_9351
🎉 2
evoskuil23:52
Trashed the big extractors too, and hit the limit of my weight times the breaker bar. So got out the big guns. Few full hits with the sledge hammer on that big black hub remover and it’s out! I was a little worried.
IMG_9355IMG_9354
evoskuil18:47
These are all of the files that are generated from the automated build system (client stack excluded, but that adds two more repos). https://github.com/libbitcoin/libbitcoin-build/blob/master/MAINTAINED.md

Friday, May 30, 2025

evoskuil05:39
Turns out the test failures were me chasing a ghost for 24 hours. They only reproduce on one machine, and when I roll back any amount of time they are still there. And I had compiler failures in the same area. Something broke on that machine, no bugs in the code. updated intel with full vectorization coverage, and full neon vectorization coverage. intel sha256 has been good for a long time, next I'll finish up with neon sha256 and then start testing the taproot impl.
evoskuil06:53
Maxi completed 1 minute over its milestone record, and mini is running 1 minute ahead of its best time. So doesn't look like there were any unexpected side effects from the intrinsics/ARM work.
image.png
🔥 3
This run is the smoothest network utilization I've ever seen. Something is better.
evoskuil07:33
maxi 53 minutes, mini 50 minutes, 74 and 57 sec shutdown.
evoskuil16:36
We will be passing block 900k within a week, at which point I'll be bumping my benchmarks to that height.
✅ 1
evoskuil21:59
Presently milestone to block 899,101 is 65 mins vs. 53 mins to 850,000. The load factors are higher given the additional elements, and this can pull down the point table duplicate search. We can prob get this back under 60 mins with config.
Store size 813,924,158,906 bytes (758GiB), up from 715,734,115,162 bytes (667GiB), without adjustments to reduce load factor - though that won't register at the GiB level.
(edited)
✅ 1
evoskuil22:13
Total wire size of the chain at 899,101 is 662,821,642,114 bytes (617GiB). Total to 850,000 is 582,391,632,423 bytes (542GiB).
🎉 1

Saturday, May 31, 2025

evoskuil00:31
So I updated the point table load factor (more buckets) to the same 2.2 we've been running at 850k, and re-ran the mini milestone sync. Sure enough, the sync came in at 59 mins. It increases the total headers size by over 1GB, which is a tradeoff, but obviously worth it. Under 1 hours on the mini, at 900,000 blocks!
(edited)
🔥 2
evoskuil00:40
With an 8 second stop, so still under an hour including table create and shutdown.
And in other good news, I just completed the ARM Intrinsic SHA implementation, with all tests passing on the macOS CI platforms.
(edited)
🎉 1
evoskuil00:46
It's worth pointing out that, for the same hardware, the sync time went up just 6 minutes between block 850,000 and 900,000 (11 months of growth).
🚀 1
evoskuil00:52
Regarding the intrinsic SHA implementations, recall from the meeting that I was blocked on the apparent impedance mismatch between Intel and ARM native SHA intrinsics. Well with a little persistence I figured out that it doesn't really exist, it's just how all of the samples are written differently. There are differences, but they don't affect the logical flow.
evoskuil04:26
Just set a new milestone config record (on the mini) to 850k, of 46 minutes (previous record with necessary duplicate detection was 53 minutes). This was the result of me bumping the process (vs. thread) priority to high, which can result in better memory performance.
(edited)
evoskuil16:52
The full native sha implementation for arm and intel comes down to this.
image.pngimage.pngimage.pngimage.png
💥 1
Dropping in another is just a few lines of code, same for sha vectorization.
evoskuil20:10
It might not be obvious from the context, but the last two images are the Neon (ARM) and SHA-NI (Intel/AMD) implementations. The two on the left are the relevant part of the generic SHA template.
evoskuil22:09
If anyone happens to have a POWER8/9/10 box I'll drop in native SHA and vectorization for you to test.