Bitcoin full node and query server
April 2024
Archive of the libbitcoin Slack #general channel.
Wednesday, April 3, 2024
Adi Shankara14:30
replied to a thread: view
The bootcamp consists of series of readings and seminars over a month, to teach bitcoin technicals, mostly using Grokking Bitcoin as a textbook. It culminated last week. It also includes a hard programming assignment on bitcoin technicals that one needs to solve in order to pass the screening.
Adi Shankara14:32
All other orgs and projects have been published: Summer of Bitcoin 2024 Organizations. Take a peek to see the scope and variety. Would be nice to get in some libbitcoin projects in time for applicants to explore and prepare proposals by the Apr 30 deadline.
summerofbitcoin.org
Learn more about the organizations and projects participating in Summer of Bitcoin 2024.
Saturday, April 6, 2024
neillm02:34
Hey all -- for a while now, the weekly meetings have been at a certain day/time. I can probably start making them more regularly now if they could be moved ~2-3 hours earlier than usual. Any chance we can make that work?
👍🏻 1🎉 1👍🏾 1
Write a minimal algorithm to detect the presence of a merkle tree malleable set of tx hashes.
evoskuil17:47
No, just to detect the flaw in the most efficient manner. A search for hash duplicates is sufficient to detect the possibility, but produces false positives.
The type under section 2.
evoskuil20:00
So what we want to detect is a set of tx hashes that, by excluding one or more, results in the same Merkle hash (due to the Merkle algorithm addition of one at any level in order to create an even set for hashing at that level).
jungly20:05
You could generalise the problem and drive a probability that given n tx hashes you can end up with m combinations where the root hash will be the same.
Then knowing the size of unconfirmed transactions you know the probability of existence of malleable block hash.
The result could be to say we should strive to keep the number of unconfirmed txs below some N.
Maybe we have been accidentally lucky because of the mempool stupidity?
I have a naive question about the linked document. I don't get why hash(C) = hash(C || C). I didn't realise that was the case. But why?
1 || 1 = 1, 0 || 0 = 0
Sunday, April 7, 2024
pmienk04:07
May be the case, I didn't get a chance to look at the reference. Just assumed || was binary or not concat
pmienk05:27
you're right, the paper uses || as concatenation. He's generalizing the comment (F), (F, F) both equal to C
It's the statement in 2 Duplicate transactions - documentation from core source
hash(F) = hash(F, F) = C
evoskuil05:30
Given set (of tx hashes) with length s, where:
- p is non-negative
- s is even
- s - 2^p is odd and >1 [fill]
- s - 2^n is even for all n=0..p-1 where p>0 [no fill]
- T is hash tuple length 2^p in set
- If T is repeated once at end of set then set is mutable
These are the rules I’ve derived that determine whether the set is mutable. I believe they are correct. I’d like to turn that into a function.
“Fill” means that the condition represents the case where Merkle fills in an odd length row by repeating its last hash.
The mutability takes advantage of the first fill, which happens on the first odd-length set produced by the Merkle iteration. To duplicate the value that will be filled requires a tuple of hashes at the end of the original set, with a power of two length, where that power is the number of iterations before the first odd set is produced.
(edited)The reason F and FF come out the same is that Merkle requires pairs and repeats the last element in any odd length set in order to achieve that.
So a-b-c hashes as [[a||b]||[c||c]]
(edited)The singular hash isn’t duplicated, which is an exception to that rule.
The test that Core uses produces false positives and is quadratically inefficient.
This message was deleted.05:48
1 reply · evoskuil · last Apr 07, 2024
evoskuil05:51
The test they use is a loop within every row in the actual merkle root computation, to compare all sibling pairs for equality. Yet the only pairs that can produce mutability are at the end of the first odd length row.
(edited)evoskuil05:54
The test they use is a loop within every row in the actual merkle root computation, to compare all sibling pairs for equality. Yet the only pairs that can produce mutability are at
One way to do this is to simply enumerate all possible counts of txs in a block and look up the tuple lengths that, if the tuple is repeated at the end of the original set, implies mutability. I did a short map manually…
0:0 ()
1:0
2:0 (special case)
3:0
4:1 (3)
5:0
6:1
7:0
8:1,2 (7, 6/3)
9:0
10:1
11:0
12:1,2 (11, 10/5)
13:0
14:1
15:0
16:1,2,4 (15, 14/7, 12/6/3)
17:0
18:1
19:0
20:1,2 (19, 18/9)
21:0
22:1
23:0
24:1,2,4 (23, 22/11, 20/10/5)
25:0
26:1
27:0
28:1,2 (27, 26/13)
29:0
30:1
31:0
32:1,2,4 (31, 30/15, 28/14/7)
33:0
34:1
35:0
36:1,2 (35, 34/17)
37:0
38:1
39:0
40:1,2,4 (39, 38/19, 36/18/9)
41:0
42:1
43:0
44:1,2 (43, 42/21)
set-length:tuple-lengths (iteration lengths for each tuple)
So for example, with a set of 40 hashes we know that the last 1 or 2 or 4 of them could be a subset that repeats once at the end of the set, which would result in the same merkle root as the case where the duplicate subset is dropped. And if any of those three counts do repeat the set is mallied.
So we really just want to find the set of those subset lengths for a given set length.
where the subsets (tuples) are always a power of two length, so the power is sufficient. All even sets have at least a power zero mutable set, since the first iteration is odd with a subset of length 2^0 removed.
I can work up an iterative function to test odd/even -ness unless someone has a better idea.
evoskuil14:12
Thinking about this some more another approach altogether comes to mind. By simply eliminating the txs with duplicate hashes from any block while retaining order otherwise, we not only eliminate the possibility of seeing this type of malleation, we can also provably obtain the actual block from someone attempting DoS.
evoskuil14:24
Duplicated tx hash malleations always result in an invalid block, which always occurs before we store the block (association from header to txs) since it’s detected in block.check() as internal double spend. But there are two cases where we still need to know if it’s malleated. Under checkpoints, where we skip validation, we can’t trust the hash to ensure we have the block. And otherwise we want to be able to mark the header as invalid if the unmalleated block is invalid. By removing duplicates we know it’s either the right or wrong block (matches the header chain) and know whether it can be marked as invalid (because it’s the right block). Though we still would probably like to drop the peer.
evoskuil14:29
It may still be worth having an efficient detection (since de-duplication is sub-linear in tx count and in the checkpoint scenario we need to do it for all covered blocks). But resolution becomes simpler. With an efficient algorithm the detection is logarithmic.
evoskuil15:29
std::vector<size_t>
tuple_sizes_from_set_size(size_t set)
{
using namespace system;
if (set < 3) return {};
if (is_odd(set)) return {};
std::vector<size_t> out{};
for (auto tuple = one;
tuple <= to_half(set);
tuple *= two)
{
for (auto u = set - tuple,
auto t = tuple;
!is_zero(t);
t = to_half(t),
u = to_half(u))
{
if (is_odd(u))
{
if (t == one) out.push_back(tuple);
break;
}
}
}
return out;
} (edited)evoskuil15:38
So iteratively, O(2*log2(n)) to find the set of tuple sizes that could cause malleation. Though this is only dependent upon set size, so can be made constant for a given limited set size using a lookup table.
And using constant evaluation the table could be populated at compile time (as an array) from the above function.
(edited)1 reply · evoskuil · last Apr 07, 2024
evoskuil17:03
The 2x factor is essentially eliminated over the space due to the exclusion of odd set lengths (/2).
evoskuil16:29
But I can’t escape the feeling that you smart guys can make this algorithm non-iterative :)).
46 replies · evoskuil, pmienk · last Apr 07, 2024
evoskuil17:04
Yes, but further restricted such that at depth “i” the set has been collapsed to an odd length.
pmienk17:11
Not sure the sizing constraints to be applied. But a linear walk of the TX set can detect the duplicate match. And if there's a first duplicate without subsequent match its already invalid and we can bail.
pmienk17:15
Some invariant between size of set s, size of nonrepeating elements p, and the length of the repeated Txs r such that r itself would require duplication to merge relative to p. So s = p + 2r or s = 2^p + 2^(r-1) is valid and s = 2^p + 2^r is the mallied block
pmienk17:16
Note that the two forms define p and r differently, not sure which is more convenient for teasing out the relation
evoskuil17:30
Re your earlier comments. Detection of the possibility is trivial. A duplicated tx is necessary for the possibility and also invalidates the block. That’s not the problem however. We need to be able to know that the cause of the invalidation is malleation, since that alone precludes caching the invalidity.
pmienk17:42
Yes, the comment was that the first failure to match the repetition point after first duplicate is an ejection from detection of the subset. There is no, we'll maybe it has a smaller repeating duplicate later on if scanned from the front. If scanned from the tail I suppose there's no gain
pmienk17:44
But even that is wrong I suppose, as the repeating subset could be performed more than once
evoskuil17:46
The exponent of the repeated set length must match the depth of the first odd row in the binary reduction.
evoskuil17:47
Otherwise the merkle duplication either doesn’t occur or occurs too early or late to match the repetition (ie at the necessary depth).
(edited)evoskuil17:50
So if the repetition is 4 hashes (2^2) then the third (0..1..2) reduction must be an odd length set, and the first and second must not be.
evoskuil17:50
Yes, but not a given tx hash, since as you move up the tree you are duplicating a peer leaf node that is a combined hash.
evoskuil17:51
Merkle is a simple iteration until 1 hash remains, where pairs of leave are hashed into a new row. If the row is odd the last pair is a duplicate.
(edited)evoskuil17:57
So depths must be all even until the target depth, which must be odd. Where the target depth is determined by the dup set length base 2 exponent.
(edited)pmienk17:58
Sounds like you're telling me that the only mallied set can contain a single duplicate sequence of hashes, cool
evoskuil18:01
Core scans each entire row for duplicated peers. That would create false positives. In other words, failure to invalidate a hash that has no valid variant.
evoskuil18:01
The simplest specification I can think of is the iterative algorithm, which I don’t like :/.
evoskuil18:04
Also, there can be more than one mallied set for a malleable valid block. But only one dup set per mallied block is feasible.
(edited)Monday, April 8, 2024
evoskuil04:26
std::vector<size_t> malleable(size_t set) NOEXCEPT
{
std::vector<size_t> out{};
for (auto copy = one; copy <= to_half(set); copy *= two)
{
for (auto width = set - copy,
depth = one; depth <= copy; depth *= two,
width = to_half(width))
{
if (is_odd(width))
{
if ((depth == copy) && !is_one(width))
out.push_back(copy);
break;
}
}
}
return out;
}I compiled and ran this and it gives the same results that I generated manually (up to set of 44). I believe it accurately defines the set of duplicate sizes that can create malleation in any given hash set size (inclusive of the duplicated set). The previous code was missing the
(edited)!is_one(width) test, required because Merkle doesn't duplicate a single-valued row (it is instead the terminal condition).
Jose Robles17:27
hi everybody i found this in a link; i've been taken interest in libbitcoin lately and i would ask if you have something like "good first issues" to start to be familiar with the codebase and apport something? sorry if this is not for ask about that
👋 1👍🏻 1👍🏾 1
evoskuil17:30
libbitcoin-system is the place to start. Depends on where you are starting from. Maybe read through the /chain and /machine classes.
79 replies · Jose Robles, evoskuil · last Apr 11, 2024
Jose Robles09:43
ok i can't install... i'm on a Mac M2, i pass until make (i compiled bitcoin-core/secp256k1 first). the error it gives me is:
Jose Robles09:44
./include/bitcoin/system/intrinsics/haves.hpp:114:16: error: use of undeclared identifier 'get_cpu'
return get_cpu(eax, ebx, ecx, edx, cpu1_0::leaf, cpu1_0::subleaf)evoskuil19:49
Core doesn't version it. This works for them because they clone the sources into the bitcoin repo. But for anyone who wants to use it as a proper library, versioning is required. They introduce unversioned breaking changes that otherwise break our builds.
Jose Robles20:00
oh, i compiled and installed sec256k from libbitcoin and now it says this ./configure: checking for libsecp256k1 >= 0.1.0.20... no
configure: error: libsecp256k1 >= 0.1.0.20 is required but was not found.
evoskuil20:02
The link defaults to
(edited)/version3 (released) so if you want /master (👍🏻) get the install.sh from master.evoskuil20:05
We don't modify libsecp to version it, we just place it into different branches. That's why you got an error from master.
evoskuil20:12
It's a bug, and I can patch it right now. But you may have to work through more. That would be fine with me. We just haven't got that CI set up yet.
evoskuil20:12
evoskuil20:16
The particular bug is in the setup of compiler-specific optimizations. non-Intel/AMD had an issue where a call was execution excluded but not from compile.
Jose Robles20:41
In file included from ./boost/mpl/integral_c.hpp:32:
./boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
./boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
# define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)
^
2 errors generated.
...failed updating 3 targets...
evoskuil22:36
That's likely a boost compatibility issue and should be resolved by building it as shown above.
evoskuil22:37
boost and libsecp are the only required dependencies and build from the install.sh. Also ICU which is optional.
Jose Robles22:42
the error gets again with this:https://libbitcoin.slack.com/archives/C6LH74ACQ/p1712783523195809?thread_ts=1712597441.375849&cid=C6LH74ACQ
Eric Voskuil
$ ./install.sh --prefix=/home/me/myprefix --build-boost --disable-sharedevoskuil22:59
You are probably seeing it where we don't because you are on clang15 and we build with 14.
evoskuil23:02
You might try bumping the boost version, as they may have fixed it. Though that sometimes surfaces other issues.
evoskuil23:10
We build with clang14 on macos and clang15 on linux. The issue is a consequence of a llvm breaking change in clang16 that breaks boost and has no fix. You said you are on clang15 - which shouldn't be an issue. But I would try 14 if that's the case, since that's what we test with.
(edited)👍 1
Friday, April 12, 2024
evoskuil01:54
We didn't hold the weekly today, so here's a quick update on my end. I was in London for a few days, home for a day, and then in FL for a few days. Managed to get some work done though.
As you saw above, I was working out the algorithm to accurately detect malleability, which I haven't found anywhere. Core doesn't have it right and then interleave it with Merkle tree computation. Despite this complexity it's far less efficient. Their algorithm is:
uint256 ComputeMerkleRoot(std::vector<uint256> hashes, bool* mutated) {
bool mutation = false;
while (hashes.size() > 1) {
if (mutated) {
for (size_t pos = 0; pos + 1 < hashes.size(); pos += 2) {
if (hashes[pos] == hashes[pos + 1]) mutation = true;
}
}
if (hashes.size() & 1) {
hashes.push_back(hashes.back());
}
SHA256D64(hashes[0].begin(), hashes[0].begin(), hashes.size() / 2);
hashes.resize(hashes.size() / 2);
}
if (mutated) *mutated = mutation;
if (hashes.size() == 0) return uint256();
return hashes[0];
}
It's confusing but if
*mutated starts true then it attempts to set mutation and then returns that back in the *mutated in/out parameter. As you can see, for each row of Merkle computation it compares all hashes in the row as pairs - even though only the last pair can cause mutation. And that only happens on odd count rows - as a result of the push_back(hashes.back()) statement. And that can only result when the log2(count) of the duplicated hashes equals the depth of the first odd row. So clearly there is a lot of extra iteration and comparison happening.I've implemented, tested and merged several functions on
system::chain::block to use in node and database. The core algorithm is this:
static constexpr bool is_malleable32(size_t set, size_t width) NOEXCEPT
{
for (auto depth = one; depth <= width; depth *= two, set = to_half(set))
if (is_odd(set)) return depth == width && !is_one(set);
return false;
} (edited)evoskuil02:06
Given an integer count of tx hashes (set) and a proposed size of the duplicated hashes (width) the algorithm returns true if the set is mutable for the width. By enumerating all widths up to half the set size one determines all possible mutations. The sets of trailing txs that may make up a mutation can then be compared for duplication. On average that's an average of 1 comparison per block - at least for up to 44 txs (well below O(log(n))). By comparison Core's test is O(n) for any block.
This is for what I named malleable32 (since hashes are duplicated and each is 32 bytes). There is also a trivial method for malleable64 (since this applies to a block of all 64 byte txs).
bool block::is_malleable64(const transaction_cptrs& txs) NOEXCEPT
{
const auto two_leaves = [](const transaction::cptr& tx) NOEXCEPT
{
return tx->serialized_size(false) == two * hash_size;
};
return !txs.empty() && std::all_of(txs.begin(), txs.end(), two_leaves);
}malleable32 only applies to invalid blocks, so we discard the invalid block (though we could derive the valid block from it) before ever storing it. But we need to know for the case of checkpointed/milestoned blocks, since we only check the hash for these - and this implies the hash is correct by the block has redundant txs.
malleable64 does not imply the block is invalid, and we may not know that until we hit the validation or confirmation stages, which are after the block is stored. So we need a very efficient query to determine if a stored block is malleable64. So this is now implemented and tested in the store. I added a flag to the txs association, which is an independent table from blocks and txs. This allows a query to determine if the block requires validation and confirmation despite having been checkpointed, and also indicates that the block should not be marked as invalid if either of these stages fail (because a valid block with the same hash is presumed to exist). The query is extremely efficient as the data is already paged into RAM wherever it would be called. And it required no storage expansion as it was able to fit within an existing byte used for the block.txs.count field.
(edited)I'm now implementing three store queries that will use this new information. One will "disassociate" a block, which means to set the txs association to null such that it is treated as we never downloaded the block, just the header. The txs remain in the store to be validated at some point, as does the header. This gets set when a stored malleable64 block is found to be invalid.
Another query simply returns the malleable flag that is set on the txs association (and disassociation). Finally we require to know that, when receiving a mealleable block, whether we have already invalidated that variant of the block with the same hash - so we don't continue to store and invalidate the same block. So even though it's disassociated, we use the old association(s) to determine this condition. In this case the set of tx hashes will be distinct for a new variant, despite the block hash being the same. So we just enumerate the old associations (these are inherently reachable in the store hash table even though normally the most recently written is the only we retrieve) and compare the hash sets.
So almost done with malleability, and the result should not make a noticeable increase in sync time and no increase in store size.
Saturday, April 13, 2024
evoskuil17:02
Synced and confirmed 804,000 blocks in 1.5hrs, which is about 90% of my peak network bandwidth for the data download alone. This is with validation bypassed by milestone as that is not currently parallelized. However confirmation was formerly a very costly exercise in previous libbitcoin versions, since it required updates throughout the store. The new design simply writes a record for each confirmed tx and block so is very fast. This could also be accelerated under checkpoints/milestone because it can be done concurrently.
132 replies · giphy, evoskuil, jungly, Marnix · last Apr 14, 2024
evoskuil17:53
That's confirmation, but validation write costs are the same, just a different table. They are distinct because a block/txs can be valid but not confirmed and can become unconfirmed and reconfirmed - and txs can have validation context for different blocks.
evoskuil17:55
The remaining big costs are reading transactions from the store to validate scripts, reading prevouts for the same, and performing that validation. These are all fully parallelizable to the extent that prevouts exist (and in some cases without them). That's the outstanding work right now to complete sync = the parallelizations.
(edited)evoskuil17:59
I'll have to give Core a run on my machine. I've got 64 cores and 256 GB RAM, though it's an old machine.
(edited)evoskuil18:33
7 minutes in and I'm at block 28,181, and these blocks are like 1kb. Libbitcoin downloads, stores and indexes the first 300k blocks faster than that.
(edited)evoskuil18:40
It took 3x longer to sync headers. The pre-header sync thing it does to protect against DoS doubles the time and achieves nothing more than we do natively.
jungly18:42
iirc, there's a small config change you have to make to the default config. Afk, so can't check.
jungly18:47
Ah yes. The connections, they ask you to keep them below 8.l iirc. Maybe cause concurrent can't handle more.
jungly18:48
But I think you should definitely complete a sync with default config.
Lopp's yearly blog post might have some config suggestions
jungly18:50
This is what lopp recommends from 2022
connect=<local node IP address>
assumevalid=0
dbcache=24000
disablewallet=1
evoskuil18:57
It looks like the block overhead is so high that it crushes the download rate for smaller blocks. The rate is going up a bit now that blocks are bigger.
evoskuil18:58
CPU is at zero because there is no validation. This is just indexation and storage.
(edited)evoskuil19:17
The tests that JL has been putting out are super misleading because they don't take into account peer variability, which is the hard issue and real issue.
jungly19:18
An ideal repeatable test is to set up ten peers and connect the node under test to the same ten
evoskuil19:20
The node has to be able to manage the network. 10 peers is WAY too low, and doesn't allow for perf differences.
jungly19:26
I agree that being able to handle more connections is a huge plus.. and setting up all those nodes that you can handle will be a bit hard. Maybe then the test has to be on theb love network, but repeated a good few times. However, we aren't trying to get reproducible numbers, just a good enough ballpark
evoskuil19:27
The network is big and varied enough that you do get consistent numbers if you do it right.
(edited)evoskuil19:29
If he set up 5000 peers with varying speed from 0 to GB and ran each of us against them, we would max out the LAN bandwidth and Core would take whatever it gets.
evoskuil19:36
It's approaching the end of the checkpoint range (295k). We can process the first 300k in just a few minutes.
evoskuil19:41
It's hard to imagine that I started working on this when the top block was less than 300k. That's just a couple percent of the present chain.
(edited)evoskuil19:46
300 is an inflection point, blocks start getting bigger. Because they are sequential this increases the download rate, so now they are hitting maybe 75MB/s (vs. our consistent 980).
evoskuil19:51
Interesting, the number of outbound connections has increased to about 30 and still slow.
evoskuil19:54
Ok, that's 1.5 hours. They are at block 310k, which is about 5% of block 804k, where we are at 1.5 hrs.
evoskuil20:06
Interestingly, their shutdown time at this low store size is also long, like 30-45 seconds.
evoskuil20:07
Since they do not max out any aspect of the hardware, it implies no scalability. We are capped by WAN speed on my machine. If I had a 10Gb/sec network we could probably complete in 10 minutes.
evoskuil20:10
So you see why I wasn't sweating the store locking? it's so small compared to everything else I haven't found it's limit yet.
evoskuil20:13
Bitcoin Stack Exchange
Checkpoints prevented an attack where a node could mine many many low-difficulty blocks from an early point in the blockchain and serve those blocks to syncing nodes who would see their disks fill ...
evoskuil20:15
What huge steaming turd. We just load the headers into a hash table until they are valid and current with sufficient work or are under milestone. Then they organize into the candidate chain on disk. Simple and same result in a single pass.
evoskuil20:16
I've seen this complete in 10 secs, typically more like 30-45 since we don't manage peer perf for this, and we redundantly hit peers.
(edited)evoskuil20:19
The difference is that we could have the hash table blow up if we're getting sybil attacked, but in that case you just drop the table and peers and try again. Because it's a race between multiple peers, only one good peer has to beat the bad.
jungly21:47
I don't fully follow the "disk fill attack". Wouldn't all your peers have to send the same "spam" headers? Basically you need to be eclipsed before you can be attacked, right?
evoskuil21:56
No, a peer can give you weak headers forever and until you find a stronger and current branch (and even after) you have to take them somehow.
evoskuil21:56
Because they can eventually become stronger. The only limit is that they can't pass through a checkpoint. But there are infinite branches possible below and between checkpoints. And now above is feasible too.
(edited)evoskuil21:58
Below and between checkpoints can be resolved by simply enumerating all points up to the top checkpoint. Ugly but works and not a fork.
jungly21:59
Gotcha. So just in case they become stronger later on. But once at checkpoint you can clean them up.
Isn't the answer simpler if you throttle each peer when downloading weak headers? Connect to more peers instead to speed the up?
(edited)evoskuil22:00
You can reject anything below a checkpoint once you have the full set of hashes that reach the checkpoint (like the headers).
jungly22:02
So you just need to get to the first checkpoint in the headers chain and that requires a single honest peer to send you 9mb.
evoskuil22:03
My plan is to allow you to export the hash set up to any point, and then use it to bootstrap a node.
jungly22:04
I was thinking the same. Just the peers can see you 9mb in a single message as headers up to the first checkpoint
evoskuil22:06
And worst case they are blowing up your ram until you dump it (not your disk).
(edited)jungly22:07
You can do relative throttling tbh. Sounds complicated. But tells a spammer to slow down. Problem is slow nodes will slow down your sync
evoskuil22:08
The Core approach is reasonable, it just feels like some serious complexity and over-engineering.
evoskuil22:11
I don't get the 600 block gap though, since an infinite number of blocks fit between any gap of 1.
evoskuil23:47
Some of the poor performance will be coming from my machine not having sha-ni. What this means that that, because I haven't implemented that optimization in our sha template, it's a fair comparison.
evoskuil23:49
IOW when we that's implemented we will also have a 5x improvement of all sha hashing, which is a material reduction in overall time, which should be proportional for Core. If we both got a 20% boost that would knock our 1:30 time for 804k blocks down to 1:15.
💥 1
evoskuil23:51
Together with other optimizations that are still available I'm hopeful that I can get 800k blocks in an hour.
M
Marnix07:13
Can confirm from Core IBD experiences on "normal hardware" (few) peer connections always is an issue
👍🏻 1
evoskuil07:21
It's not that there is an insufficient number peer connections, it's that the peer connections are not perfomance-managed and that they are not fully utilized due to lack of concurrency.
(edited)👍 1
evoskuil07:23
Libbitcoin implements rapid re-connection, standard deviation rate management, and concurrent requests balanced against the peer set - as well as a fully write-concurrent store.
(edited)👍 1
Sunday, April 14, 2024
evoskuil05:52
Tested Core v26.0 with comparable configuration today. It achieves about 5% of the 804,000 blocks under its "assume valid". So about 20x slower than libbitcoin. But that's only because I'm up against my WAN bandwidth limit of 1GB/s. We could probably be 10x faster on a 10GB/s connection but Core would not be because it hits no local machine limits (not scalable).
7 replies · Jose Robles, evoskuil · last Apr 15, 2024
Jose Robles09:07
that's really fast with libbitcoin. you can run pruned with libbitcoin? i want to try to sync, but don't know if i need to buy a sdd extra first
evoskuil14:10
Pruning is part of the store design but it isn’t implemented yet. v4 work is ongoing and it’s still pretty early, though key aspects are working well. An SSD is important for performance. And the system can scale to use all available RAM (and will eventually do the same with CPU once validation is fanned out). My machine is 32x2 cores and 256 GB RAM, but like 7 years old.
(edited)Jose Robles15:56
when i get time to make it compile i will try to liberate or buy a ssd. a machine that can work with core will work with libbitcoin?
evoskuil05:56
And yes, that does imply that with sufficient bandwidth we could sync up in about 10 minutes. That's without validation and confirmation checks (under milestone). Working on maximizing concurrency for those next.
evoskuil06:02
While doing that I worked through a bug in the
chain_state where following a reorg chain state for a header's parent was not found because the parent wasn't in the candidate chain. As a result I refactored chain state so that it would not require the candidate index but instead just drive off of parent pointers. This actually simplified some code and should have the same performance, since chasing parent pointers is just as efficient as sequential height increments with header pointer lookup.I also noticed that after having downloaded only about 30GB of blocks (out of over 600) the Core shutdown time was 30-45 seconds. Actually comparable to what ours is at that level, if not longer. During that time a warning is presented to not shut down the computer - so there is an explicit acknowledgement that its store is not fault tolerant (which we know but they didn't formerly acknowledge).
(edited)And we have the ability to take hot snapshots that allow us to recover in the event of a hard shutdown, and we can always detect one - neither of which Core can do.
evoskuil06:25
Interestingly, even with limiting bn to just 10 outbound sync connections, we finish headers in 30 secs (vs. 2-3 minutes) and maintain a block download/index rate of over 300MB/s (vs. Core ramping from under 300KB/s to 50MB/s to 75MB/s over the height range of 0-330,000. And by the time they have reached that height their outbound connection count has increased from 10 to 30. So they hold 3x as many connections and get less than 1/4 the net speed. So the operation is 12x more wasteful of peer connections.
(edited)Tuesday, April 16, 2024
evoskuil20:57
Lost a day tracking a confirmation check failure.
In the node confirmation checks are implemented in store queries. The general purpose system::chain object checks (header/block/tx) are retained, but those expect fully populated objects and metadata fully populated from the store. These are used in header/tx/block check/accept/connect, but the last stage (confirm) is not implemented with the expectation of deserialized objects, and is distinct from validation due to the concurrency differences (validation is concurrent while confirmation is sequential - spentness implies ordering).
So to track down the failure I implemented a redundant confirmation test against the failing transaction - one with the chain objects and the other by query. The chain object test succeeded and comparison showed that the check
tx.version >= relative_locktime_min_version was missing.Wednesday, April 17, 2024
evoskuil01:08
Spent some time working out the query to handle the three forks that make up bip30 handling. This is the only one I hadn't worked out in the design, because it wasn't absolutely necessary. But for consistency we should have it as in some fork configurations it must be applied. It can be costly to implement, but I worked out a constant time solution.
The difficulty for us arises from parallel writes with minimal locking. It is possible to end up with duplicate instances of any block, header, tx... in the store. This is the result of races, but extremely rare and logically a non-issue. But in this one case, the number of instances of a duplicated coinbase tx matter, and the instances of associated confirmations of it, spends of it, and confirmations of the spenders. So it presented a bit of a challenge.
But I found that, because every spend association commits to is spending tx by surrogate key, and every tx confirmation commits to its confirming block, we can count up the total number of uniquely confirmed (by block key) duplicated coinbases, and the total number of unique spends (by spender key) uniquely confirmed. The difference gives us the desired result - 0 means all spent, 1 means one unspent, -n and n> means fault, n > 1 means fault. Since each query iterates only these unique instances and they are essentially always singletons, it's not too costly if/when it's enabled.
So I'll wrap that up and test it tomorrow, and a couple simple things left in the confirmation chaser. Then it's all functional apart from parallelizations in validation.
🎉 1
Rich Phillips17:22
Hey all, started to try and get this library functional in order to recover some BTC in a stealth wallet. Got a long way through getting it built but it stumbles at the last part and I think only on linking test files together which I can probably skip.
Main issue is I’m doing this on macOS on arm64, so a lot of what this expects to exist just doesn’t under ARM.
Current sticking point is the referencing of _main from <initial-undefines> is an undefined symbol for arm64 and I don’t know my way through all the source yet to know which parts to pull out or modify.
Any guidance?
4 replies · jungly, Rich Phillips, evoskuil · last Apr 17, 2024
jungly17:33
Difficult to know what's really happening on your system.
Just to be sure you should be using the version3 branch https://github.com/libbitcoin/libbitcoin-system/tree/version3 and not the master branch
Rich Phillips17:34
Just starting to wonder if that’s it as the explorer and blockchain projects had the version tagged at end of folder name
Thursday, April 18, 2024
evoskuil03:53
Tracked down and fixed another bug in confirmation today. There is a case in the tail end of block download where duplicate blocks can become stored. This is allowed, but I'll be cleaning that up at some point. However the condition resulted in an unexpected confirmation failure due to tx_->block reverse navigation link (based on confirmation) not being present on the duplicated blocks. The first instances got confirmed, resulting in the second instances being the first found on natural key search and therefore not showing confirmed. This search is part of spend verification, where spends are associated from inputs to outputs via the input point tx hash. I resolved this by enumerating all instances stored against the hash (almost always one), which ensures the confirmation will be found if set.
evoskuil04:00
Following that I ran a timed confirmation check against all blocks in order, from disk storage, which completed in 3.0 hours with no faults. Writing confirmation state adds somewhat to this, though that would be more like 30 additional minutes.
There are currently nearly 1 billion (989,316,053) tx records, and over 2.5 billion (2,549,137,812) spends.
(edited)evoskuil04:06
So we are processing 90,307 txs and 232,692 spends every second, including all disk reads, to determine whether each output being spent is confirmed and not spent by a confirmed tx input other than the evaluated spender.
There is no utxo store - all determinations of spentness are made using the append only relational object model, augmented by the
(edited)strong_tx table.evoskuil04:14
By associating a tx to a block, the tx becomes strong (confirmable but not confirmed until added to confirmed height index), and a subsequent negating entry (append only) can reorg each tx of a block out of confirmation.
evoskuil13:31
Writing confirmation state adds somewhat to this, though that would be more like 30 additional minutes.
Just did a full run with confirmation state written following evaluation and there was no material increase in cost, under 3 hours. So assuming that runs sequentially that would be a minimum full chain (no milestones or checkpoints) validation cost.
💯 1
6 replies · jungly, evoskuil · last Apr 18, 2024
evoskuil18:17
3 hours as a minimum for a full chain download, validation, and confirmation is excellent.
👏🏾 1
evoskuil18:18
The concurrent download can happen in half that time, and I'm hopeful that concurrent validation can beat 3 hours. But combining them will increase each unless maybe with full chain RAM.
evoskuil18:24
In any case, it's a very positive data point. The confirmation is extremely costly in theory. You must mark each output (or its tx) as confirmed, and must mark each spending input as confirmed (or its tx), ensure that when you spend you are only spending outputs that exist, and are confirmed in tx of a block height at/previous to the input tx, and must ensure that they are not previously spent by a confirmed block tx. Then there are coinbase maturity checks (spender must be > 100 blocks from coinfirmed cb height), and finally relative locktime checks (bip68), which require the height and mtp of spender and spent, as well as spender tx.sequence.
evoskuil18:25
Core does this by maintaining a redundant hash table of unspent outputs (the output object including script, indexed by tx hash). By adding and removing from this redundant data, these questions are more simply answered. But it introduces not only a large amount of redundant data, but a heavy ordering requirement.
evoskuil19:28
Our approach does not update any primary blockchain table data or maintain any redundant data. We just maintain a hash table of associations from tx to confirming block, one entry per tx. A negative entry in the same table reorgs out the confirmation. That's it. So getting those answers and updating the confirmation table that fast is pretty good.
Friday, April 19, 2024
evoskuil01:43
Just implemented the
(edited)bip30 check, which turned out a bit simpler than I described previously. More interestingly, it is almost free in the no-duplicates case, and a trivial cost in the duplicates case (which don't actually exist). This is considered a really expensive check by Core:
"The BIP30 check is expensive, as it requires going through all the outputs of all the transactions in the block, and checking if a UTxO with the same outpoint does not already exist."
Or just a single constant time hashmap lookup. 😂
https://bitcoin.stackexchange.com/a/120839/34482😄 1
Our custom hashmap allows iteration over duplicates. Any hash table must implement conflict lists for a given hash bucket. Our average list length is configured for ~1.5 conflicts. If a duplicate entry is created it naturally hides older entries (this is how we implement logical delete for reorg and progressive state changes during validation). However our primitives allow enumeration of the conflict list, so that any hash table can also be treated as a multimap. A simple iterator hides (steps over) non-matching conflicts, making the enumeration seamless.
(edited)evoskuil02:11
So this design is used to enumerate txs with the same hash but different content (how we manage mutability) and also multiple instances of the same tx - such as the two valid bip30 exceptions that may be written in a concurrent block race resulting in two duplicated entries. So we just enumerate these and check each for block association (implying confirmed). Then we just enumerate the outputs of each confirmed instance and ensure all are spent.
code query::unspent_duplicates(const tx_link& link,
const context& ctx) const noexcept
{
if (!ctx.is_enabled(system::chain::flags::bip30_rule))
return error::success;
// Self is not a spender in this case.
constexpr auto self = tx_link::terminal;
const auto coinbases = to_strongs(get_tx_key(link));
// self should be strong but was not found (fault).
if (coinbases.empty())
return error::integrity;
// assuming it is strong only self was found (optimization).
if (is_one(coinbases.size()))
return error::success;
// All that are found must be confirmed spent except self.
size_t strong_unspent_coinbase_count{};
for (const auto& coinbase: coinbases)
{
// All outputs must be spent or the coinbase is unspent.
for (size_t out = zero; out < output_count(coinbase.tx); ++out)
{
if (!spent_prevout(table::spend::compose(coinbase.tx, out), self))
{
// Could stop at two but very rare so condition is more costly.
++strong_unspent_coinbase_count;
continue;
}
}
}
switch (strong_unspent_coinbase_count)
{
// self should be unspent.
case zero: return error::integrity;
// only self is unspent.
case one: return error::success;
// at least one instance other than self is unspent.
default: return error::unspent_coinbase_collision;
}
} (edited)So in the 99.99999999% case there is nothing but a key lookup (direct memory access) and a hashmap search O(1) for the key, which will iterate the conflict list and return one record and return.
D
David Chocholatý13:24
Hi everyone👋, nice to be here. I'm David, and I'm joining via the Summer of Bitcoin. Just introduce myself briefly. I'm a student in the master's degree programme at FIT Brno University of Technology. I have experience contributing to open-source Tone Mapping Studio project for about three years and Mixxx organization through the Google Summer of Code in 2022.
I'm interested in the project SQLite Database for libbitcoin. I like the idea of the performance improvement for the less equipped computers (probably the same as mine 😄).
@SoB mentors: Please, can I ask you a small question? I have done the first two parts of the Competency Test in the project idea published on the Summer of Bitcoin website. I have a single-page document with the problems and ideas I got during the libbitcoin database installation and running a node. Should I post the document here or send it only to mentors? Thank you for your answer in advance.
👋 2🙌 1👋🏻 1
19 replies · neillm, jungly, David Chocholatý, evoskuil · last May 01, 2024
neillm14:21
Welcome, glad you made it! Posting a link to a github gist or something seems appropriate for notes to review. cc: @jungly for the SQLite DB areas.
jungly14:35
Hi @David Chocholatý - Thanks for taking the time to run the node. Feel free to share your experience here or in a gist as Neill recommended.
(edited)D
David Chocholatý18:28
Thank you both for welcoming me. Okay, I have converted the original LaTeX document into the GitHub gist, which is on the link:
GitHub gist
If there is something unclear or should add or change anything, I can update the gist of course (on the Summer of Bitcoin website is mentioned, that only problems that occurred should be written).
(edited)evoskuil18:42
FYI the recommended build is static, prefix directory (non-sudo), and build boost. This is because the linker tends to pick up shared dependencies despite configuration and boost installations vary widely and are often incompatible.
Also, you've got version3, which is the release build but very dated. The SQL project can only apply to version4 (/master). There is no current server build for master, we are working from libbitcoin-node (libbitcoin-server builds on that). There is a similar
(edited)install.sh. That build has system, network, and database dependencies.D
David Chocholatý18:58
Thank you very much for the clarification and explanation. I have chosen version 3 because it was explicitly required in the project idea (for both, database and node too) but of course, for the project part I would prefer the master branch version.
(edited)summerofbitcoin.org
A C++ Bitcoin toolkit library for asynchronous apps.
D
David Chocholatý19:29
Please, can I ask you a small question just for clarification? In the Competency Test part of the project idea, is written for the "stretch goal" that the repository with SQLite program in C++ can choose arbitrary table schema. Does it mean that an independent schema from the project or Bitcoin in general can be chosen too? For example the common schemas like online book store or something like that.
jungly19:31
That's great that you got everything running @David Chocholatý.
Yes, any table will do from any schema. It's just an exercise to have a SQLite table created with a couple of read and write queries executed on it
👍 1✅ 1
D
David Chocholatý19:37
Thank you. Great, I will take a look at the SQLite part and try to send you the repository ASAP.
D
David Chocholatý13:24
Hi, I have done the stretch goal SQLite task. It can be found in the public repository:
https://github.com/davidchocholaty/libbitcoin-sqlite-task
The table scheme is in the README file. I have tried to write some short descriptions into the README file too. Of course, I can explain other things here. I would like to ask for your feedback on this. Thank you in advance.
jungly13:32
Looks good. I was able to build and run the application. I liked the extensive comments you left. Nice work.
👍 1
D
David Chocholatý14:18
Great, thank you very much. It pleases me. Now I'm working on the proposal part. I will remain available on Slack.
I want to ask you too if some tasks have to be done and are good for the libbitcoin newbies. I ask because there are no "good first issues" and I would like to dive into the toolkit a little bit deeper.
jungly14:20
TBH, the sqlite database is something we are looking forward to a lot. Other than that, @evoskuil will most definitely have suggestions for other first issues.
D
David Chocholatý14:26
Ok, thank you. So I will rather focus on the proposal and SoB now up by 30 April to increase the chances of creating a SQLite database.
evoskuil14:33
The document that I posted is the only work done on it so far. I would start by building a schema that maps to the existing database and then writing the archive queries in SQL against that schema.
Note that tables are designed for efficient access during validation and confirmation stages, so any deviation should be considered carefully.
Also, the store is fully read/write concurrent and tables are all append-only. The two height indexes (candidate/confirmed) can however be popped (become smaller).
(edited)👍 1
evoskuil14:35
The final outcome should be a drop-in replacement for the store object and query class.
👍 1
D
David Chocholatý14:52
Thank you for tips and explanations. Sounds good to me. I will try to reflect this in the proposal and cover parts of the document.
👍🏻 1
D
David Chocholatý07:38
Hi all, I'm also writing in this discussion thread. I'm so sorry, but unfortunately, because of the time pressure during my studies, I could not complete the SoB assignment task on the minimal required number of points which is required for a successful application too. So I won't be able to work on the project via the Summer of Bitcoin programme. However, if @Tanay Raikhere has fulfilled this requirement, I wish him the best of luck in his successful selection to work on this project. Despite this failure of mine, I would like to stay in touch with this project.
(edited)D
David Chocholatý06:48
Thank you @evoskuil. I will. Just for your information, I will be short of time by about three weeks due to the university exam period. However, I should have time after that period and am open to contributing.
🔥 1
Saturday, April 20, 2024
BTW, those invalid
sendaddrv2 messages are not Core (because Core is non-compliant with the broken BIP) - they are some other impl that complies with the BIP.👍 1
evoskuil17:21
The bip30 check is is now a bit simplified and verified across the chain. No material cost to it even when implemented for all blocks.
error::error_t CLASS::unspent_duplicates(const tx_link& link,
const context& ctx) const NOEXCEPT
{
if (!ctx.is_enabled(system::chain::flags::bip30_rule))
return error::success;
const auto coinbases = to_strongs(get_tx_key(link));
if (coinbases.empty())
return error::integrity;
if (is_one(coinbases.size()))
return error::success;
// bip30: all (but self) must be confirmed spent or dup invalid (cb only).
size_t unspent{};
for (const auto& cb: coinbases)
for (index out{}; out < output_count(cb.tx); ++out)
if (!spent_prevout(cb.tx, out) && is_one(unspent++))
return error::unspent_coinbase_collision;
return is_zero(unspent) ? error::integrity : error::success;
} (edited)
>The bip30 check...
This is an example of Core pushing forks to try and mitigate performance issues that are a consequence of their implementation, not inherent to consensus. bip30 defined a soft fork, which was necessary. Then there was a hard fork which enforced it only in the case of a bip34 checkpoint, and then there was another soft fork which reenabled it because it was later determined that the previous hard fork would cause a chain split in the not-so-distant future. And of course there is an ongoing effort to resolve it again now that it becomes reenabled at that future point.
(edited)evoskuil17:39
The above code only runs once per block (coinbase) and unless there is a duplicate (which is extremely rare even in the above description), requires only two O(1) hashmap queries, and otherwise the cost is still not significant.
evoskuil18:08
Because Core uses a utxo set, spent outputs are not indexed. And when they prune the spends don't even exist in the archival data - though that's moot. We just index all spends, and could easily retain cb-spending txs during pruning.
Sunday, April 21, 2024
evoskuil18:49
A fresh full sync today took 1:30 to complete download (840,000) while simultaneously performing sequential validation and confirmation through the first milestone block (804,000), completing that in :15 additional minutes (total 1:45). IOW the node could be usable at current height with a high milestone in under 2 hrs.
🎉 1
On the same machine in that amount of time, also under milestone, Core is less than 10% synchronized (in terms of bytes/time).
evoskuil21:17
I think my summary might have been a little misleading. The validation and confirmations are under milestone, so while there is work there it's not full validation/confirmation checks. But this is the first time with all of the necessary checks (even bip30 implemented) and metadata writing (all txs/blocks marked confirmed).
(edited)Monday, April 22, 2024
evoskuil14:18
I’m trying to get everything solid and complete before moving into validation concurrency. I’ve got one thing left to knock down, which is to close out some block download redundancy at the top of the chain.
evoskuil14:24
Headers first, blocks first, concurrent blocks, validation, confirmation, org, reorg, disorg, block hash malleation, bip30… all working very well together in the event-driven model, which allows them to be well isolated and easy to follow while still being necessarily interdependent.
evoskuil14:30
Also snapshot/restore has been flawless and easy. Snaps should be automated at some point.
🔥 1
And we don’t yet have sha-ni, which will give us a 5x improvement in sha256. We do have avx512 but my machine doesn’t. That will almost 2x merkle root.
Tuesday, April 23, 2024
Rich Phillips14:17
@evoskuil can’t seem to download the Mac binary for v3.2 of explorer, get a 404 error. Can that be restored or is there someplace else I can grab it from?
157 replies · Rich Phillips, evoskuil, pmienk · last Apr 23, 2024
Rich Phillips14:20
😬 ok, was hoping that would be the simple button since the build is sucking right now. What’s the flag to build without boost library? Or do I have to hack it out of the install script?
evoskuil14:22
The default install.sh does not build boost, but commonly the local install of boost isn’t compatible.
Rich Phillips20:27
Ok either I’m being stupid here or I’m doing this wrong but running either auto gen /configure or just install results in a check for boostlib being present. I can’t force it to be ignored. I’ve even forced a false flag on the install scripts etc to try and skip it but as soon as the linker tries to build the system section it falls over here.
evoskuil20:33
Right, the question is only whether you build boost or rely on your preinstalled version.
evoskuil20:35
$ ./install.sh --with-icu --build-icu --build-boost --build-zmq --prefix=/home/me/myprefix
evoskuil20:35
evoskuil20:37
This can be reduced to the following since you probably don't need ICU.
$ ./install.sh --build-boost --build-zmq --prefix=/home/me/myprefix
Rich Phillips20:39
I’ve built it, installed boost a hundred times, added to path. And even with all that the linked won’t find the binaries/libraries. Its the last thing that seems to fall over
Rich Phillips20:55
Ok moved along now it won’t build or find the secp256 library. That worked before.
evoskuil21:01
BTW, you should also set
--disable-shared as we cannot fully control the linker and it tends to to try and dynamically link to installed libs like boost despite the build isolation and direction. Static linking prevents that.evoskuil21:04
FYI, this is the most recent bx static/osx/clang/build-boost/build-zmq CI build. https://github.com/libbitcoin/libbitcoin-explorer/actions/runs/8117844403/job/22190912983
(edited)Rich Phillips21:18
output
Libbitcoin installer configuration. -------------------------------------------------------------------- OS : Darwin PARALLEL : 10 CC : clang CXX : clang++ CPPFLAGS : CFLAGS : CXXFLAGS : LDFLAGS : LDLIBS : WITH_ICU : WITH_PNG : WITH_QRENCODE : BUILD_ICU : BUILD_ZLIB : BUILD_PNG : BUILD_QRENCODE : BUILD_BOOST : yes PREFIX : /Users/richphillips/bx/ BUILD_DIR : build-libbitcoin-explorer DISABLE_SHARED : yes DISABLE_STATIC : with_boost : --with-boost=/Users/richphillips/bx/ with_pkgconfigdir : --with-pkgconfigdir=/Users/richphillips/bx//lib/pkgconfig -------------------------------------------------------------------- Initialized empty Git repository in /Users/richphillips/libbitcoin-explorer-3.2.0/build-libbitcoin-explorer/.git/ ********************** Download boost_1_85_0.tar.bz2 ********************** Libbitcoin boost configuration. -------------------------------------------------------------------- variant : release threading : multi toolset : clang cxxflags : linkflags : link : static boost.locale.iconv : on boost.locale.posix : onfull file
evoskuil21:33
@pmienk Going back to /version4 there was no versioning in secp256k1, just [0.1] even with changes, which is why we retain branches for versioning.
evoskuil21:37
Master shows
0.4.2.0. As of our most recent /version8 they were still showing 0.1.0.0. But the install is checking for 0.1.0.19.evoskuil21:42
I'm not sure what's up, but the log output doesn't match our current /version3 CI build.
pmienk21:42
@evoskuil you are referring to constraints that preexist the build project merging versions 3 and 4. I do not recall those limits
pmienk21:45
Correct. The .19 ending has been generated for more than 3 years by my quick check of the file history
evoskuil21:45
Notice that in our latest /version3 CI the following text:
********************** Prepairing to aquire libbitcoin/secp256k1/version7 ********************** Cloning libbitcoin/secp256k1/version7...
evoskuil21:45
evoskuil21:50
And now see /version8 is actually at 0.1.0.20. It looks like we did transition to using the libsecp versioning, which I forgot.
evoskuil21:51
But the install script posted above doesn't match latest /version3, and is pulling /version4 branch libsecp (which is unversioned).
pmienk21:52
Agreed, had assumed no other alteration of a fresh install.sh. any url substitution should be performed on latest version3 install.sh
👍🏻 1
evoskuil21:54
So is that the master install.sh? Confused because of this:
********************** Download libbitcoin/secp256k1/version4 **********************
evoskuil22:11
Boost hits incompat with new compilers etc. It is not always easy to get a version that works. You may even have to downgrade your compiler.
(edited)evoskuil22:13
But this line emitted from a v3.2 installer doesn't make sense.
checking for libsecp256k1 >= 0.1.0.19... no
Rich Phillips22:14
I’m gonna try the 3.8 tag instead. Start there. I think as long as it’s v3 should be ok
evoskuil22:16
Major versions maintain compatibility. So v3 latest is where you should be. I am still curious though how a 7 year old build is trying to find a very recent version of libsecp256k1.
(edited)Rich Phillips22:16
That part I couldn’t tell you. I tried to trace where it was looking for v4 and couldn’t find it
evoskuil22:18
That part is correct. /version4 secp is not the same as /version4 libbitcoin. They aren't aligned.
evoskuil22:20
So that build would never work if looking for 0.1.0.19 - it didn't even exist until 5+ years later.
(edited)Rich Phillips22:20
As I said, no idea why it was looking for that version, didn’t edit those parts. Just the boost url
evoskuil22:27
This is from the /version3 head (v3.8) configure.ac:
# Require secp256k1 of at least version 0.1.0.19 and output ${secp256k1_CPPFLAGS/LIBS/PKG}.
#------------------------------------------------------------------------------
PKG_CHECK_MODULES([secp256k1], [libsecp256k1 >= 0.1.0.19],
[secp256k1_INCLUDEDIR="`$PKG_CONFIG --variable=includedir "libsecp256k1 >= 0.1.0.19" 2>/dev/null`"
secp256k1_OTHER_CFLAGS="`$PKG_CONFIG --cflags-only-other "libsecp256k1 >= 0.1.0.19" 2>/dev/null`"],
[AC_MSG_ERROR([libsecp256k1 >= 0.1.0.19 is required but was not found.])])
AC_SUBST([secp256k1_PKG], ['libsecp256k1 >= 0.1.0.19'])
AC_SUBST([secp256k1_CPPFLAGS], [${secp256k1_CFLAGS}])
AS_IF([test x${secp256k1_INCLUDEDIR} != "x"],
[AC_SUBST([secp256k1_ISYS_CPPFLAGS], ["-isystem${secp256k1_INCLUDEDIR} ${secp256k1_OTHER_CFLAGS}"])],
[AC_SUBST([secp256k1_ISYS_CPPFLAGS], [${secp256k1_OTHER_CFLAGS}])])
AC_MSG_NOTICE([secp256k1_CPPFLAGS : ${secp256k1_CPPFLAGS}])
AC_MSG_NOTICE([secp256k1_ISYS_CPPFLAGS : ${secp256k1_ISYS_CPPFLAGS}])
AC_MSG_NOTICE([secp256k1_OTHER_CFLAGS : ${secp256k1_OTHER_CFLAGS}])
AC_MSG_NOTICE([secp256k1_INCLUDEDIR : ${secp256k1_INCLUDEDIR}])
AC_MSG_NOTICE([secp256k1_LIBS : ${secp256k1_LIBS}])
AS_CASE([${enable_isystem}],[yes],
[AC_SUBST([secp256k1_BUILD_CPPFLAGS], [${secp256k1_ISYS_CPPFLAGS}])],
[AC_SUBST([secp256k1_BUILD_CPPFLAGS], [${secp256k1_CPPFLAGS}])])
AC_MSG_NOTICE([secp256k1_BUILD_CPPFLAGS : ${secp256k1_BUILD_CPPFLAGS}])evoskuil22:28
When you use the install.sh it does not build from tags, it builds latest from branches. So by using a v3.2 install.sh you were mixing updates to v3 that are 7 years apart.
Rich Phillips22:29
I literally trashed the folders and downloaded fresh from the tag version for each build
evoskuil22:30
The installer is designed to build the latest of the branch it resides in, which means the latest from all corresponding branches in the other libbitcoin repos (not secp256k1).
evoskuil22:32
So when you get the v3.2 installer it builds external dependencies like boost/zeromq/secp from its recorded versions/URLs. But all the libbitcoin repos are built from their /version3 branch tops. And /libbitcoin-system/version3/ (v3.8) requires a much more recent libsecp than the v3.2 install.sh is building.
(edited)Rich Phillips22:49
Ok moved on from that, building 3.8 explorer, now we get past most stuff , finds the boost 1.73 content ok but, can’t find version for the Chrono library. It’s definitely there in the boost directory, have rebuilt boost a few times to make sure.
Rich Phillips22:50
checking for boostlib >= 1.73.0 (107300) includes in "/Users/richphillips/boost_1_73_0//include"... no
checking for boostlib >= 1.73.0 (107300)... yes
configure: boost_CPPFLAGS :
configure: boost_ISYS_CPPFLAGS :
configure: boost_LDFLAGS :
configure: boost_BUILD_CPPFLAGS :
checking whether the Boost::Chrono library is available... yes
configure: error: Could not find a version of the Boost::Chrono library!
Rich Phillips22:55
If I have the installer build boost instead of using one already built it throws build errors
pmienk22:55
Reporting is spotty from those boost checks. The m4s commonly report the wrong library. Likely it's the one immediately preceeding
Rich Phillips22:57
If I force using —with-boost and give a path, it finds it but can’t version chrono.
Rich Phillips22:57
If I have install build it, I get out of range error for a static cast for an enumerator
(edited)pmienk23:00
The supplied m4s have this as a known issue. What other output are you relying on to say it's found with the provided parameterization? Did the boost build share static/dynamic parameterization as your libbitcoin build?
evoskuil23:00
As I said, reliance on local installs is problematic because parts are often missing or compiled differently.
evoskuil23:01
So I would ask, what errors are you seeing when you use the installer to build it into a prefix lib, are you building static and not into a system directory.
evoskuil23:02
And what is your compiler version. There are known issues with boost and recent compilers.
Rich Phillips23:04
And because I’m
Setting the boost url, it’s building from
The same source tarball as I use for a Static build
evoskuil23:07
There are different components and build parameters. This is why preinstalled packages aren't generally usable.
Rich Phillips23:09
In file included from libs/thread/src/pthread/thread.cpp:11:
In file included from ./boost/thread/thread_only.hpp:17:
In file included from ./boost/thread/pthread/thread_data.hpp:12:
In file included from ./boost/thread/lock_types.hpp:18:
In file included from ./boost/thread/thread_time.hpp:11:
In file included from ./boost/date_time/posix_time/posix_time_types.hpp:16:
In file included from ./boost/date_time/posix_time/posix_time_duration.hpp:15:
In file included from ./boost/numeric/conversion/cast.hpp:33:
In file included from ./boost/numeric/conversion/converter.hpp:13:
In file included from ./boost/numeric/conversion/conversion_traits.hpp:13:
In file included from ./boost/numeric/conversion/detail/conversion_traits.hpp:18:
In file included from ./boost/numeric/conversion/detail/int_float_mixture.hpp:19:
In file included from ./boost/mpl/integral_c.hpp:32:
./boost/mpl/aux_/integral_wrapper.hpp:73:31: error: integer value -1 is outside the valid range of values [0, 3] for this enumeration type [-Wenum-constexpr-conversion]
./boost/mpl/aux_/static_cast.hpp:24:47: note: expanded from macro 'BOOST_MPL_AUX_STATIC_CAST'
# define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)
Rich Phillips23:09
I get two of the same errors on this file. I can edit the cast to fix but next build those changes are overwritten again
evoskuil23:10
It's the compiler. The warning has been elevated to an error by clang and cannot be suppressed.
evoskuil23:11
Boost hasn't fixed it yet, so the only option is to regress the compiler, which is what we did in CI.
evoskuil23:19
I've got it down to libsecp and a much smaller boost subset, and zeromq for client/server and optional ICU.
Rich Phillips23:51
Well I got further again but now it throws 20 errors and fatal errors out. Gonna have to work through each and see if I can work around them.
Rich Phillips23:51
Looks like this is driven by compiler version but don’t have a choice (thanks Apple :))
evoskuil23:56
Yeah, it sucks that they just decided to break this, and that boost didn't get it resolved.
evoskuil23:58
We've been lucky to be able to remove a lot of boost as that code has worked it's way into c++20 etc. But we'll probably at least have ASIO for some time.
Thursday, April 25, 2024
evoskuil03:22
Resolved the issue of redundant downloads toward the top of the chain. Also implemented a simple moving download windows, so that concurrent downloads can be held to a certain height above the top validated block. This will allow the downloaded blocks to be cached for validation if the window is small enough.
Also tracked down and resolved a legit bug in the network stack. That was an unexpected first after a couple years of solid operation. But I hit a GPF twice when ending the process normally in the debugger. I was able to track it down to a null
const& session_ member, and I determined that was the result of not using a shared pointed for sessions references. That's resolved and has been working well.🙌 1
evoskuil03:34
I'm traveling for a few days, but this morning I also integrated population of the address index, which is just a few lines of code inside the store(tx) database function. There is almost nothing to that code, just serialize all outputs in a loop, sha256 each, and store each hash against the output foreign key. But the significant amount of hashing drives CPU up from < 5% to ~60%. SHANI should cut that back down to 10% or so. Neither CPU nor download or disk were maxed, but I think it was because I have network threads configured to 16, whereas I have 64 available cores. I think if I up that we'll hit max network or CPU. The run was showing almost 50% bandwidth, vs 100% without address indexing, so I anticipated a 3 hour total run. That's far better than Electrum X or Fulcrum, which measure indexation time against a local full node in terms of days!
(edited)🤯 1
evoskuil03:44
Finally, I think I have a resolution to the open design flaw of hard termination on disk full. It's a hard problem since each table is expended independently and generally in the midst of a logical transaction. There is no transactional rollback, so there must be sufficient space. But by ensuring a minimal amount of disk space after expansion we can ensure that a subsequent expansion will succeed at least to the extent of committing the current transaction. Otherwise the process can be terminated safely. The only failure scenarios then is that the disk becomes filled external to the process (which we could also try to manage). But with snapshots this is pretty reaonable.
The trick to the above is the ability to handle a remap failure and retry with a smaller map, successively until either suffcient to complete the transaction or hard failure.
Friday, April 26, 2024
Tanay Raikhere14:05
Hey everyone, I am glad to be here. All thanks to the Summer of Bitcoin Community. I am really enthusiastic about Bitcoin and the idea revolving around decentralised money. Moreover, I have been learning about databases and currently writing my own SQLite Database in C.
I am eager to contribute to this community and open to grab any issue.
Let's get Cooking! 👨🍳
👋 1🔥 1👋🏻 1
Tanay Raikhere14:12
P.S. - @evoskuil @jungly Under the project ideas of Summer of Bitcoin, there is only one idea posted - "SQLite Database for libbitcoin".
Eric on my email informed about the current occupancy of this task. He also informed about 'other task' that can picked up, but the task is not listed over there. Where can I get the information about the other project?
Saturday, April 27, 2024
Tanay Raikhere16:02
Sure! I just completed the competency tests. I would be glad if @jungly could clarify this, so I can proceed on to making my proposal.
Here is the repository link for the third task - https://github.com/DreadPirate07/Sqlite-test . If there are any changes that are needed, please let me know!
(edited)jungly16:27
Unfortunately, we didn't get the second task in. We have the proposal here if you want to take a look.
https://docs.google.com/document/d/1tgWwzTnRrGooLDRgEXGcamjzOsWKAlTXKSwH04c80cM/edit?usp=drivesdk
evoskuil19:51
Just got back from FL. Had very quickly set up a run to fully index all outputs (Electrum style indexing). This is nothing but a hash table of hashes of the output and its surrogate key - but there are like 85 Billion outputs.
evoskuil19:56
That's a huge amount of hashing, which takes up almost all of the time. The total storage is about 109GB. The run from scratch took 3.0 hours, so doubled the sync time. It writes the index while syncing. I didn't have a chance to properly size the header buckets, because I needed the count first. So should do a little better next time. By way of comparison, a local sync off of a fully-synced Core node by ElextrumX or Fulcrum takes days, similar size I think.
Since it's all sha256, it should be about 5x faster with SHANI.
🙌 1
Tanay Raikhere21:18
Hello @evoskuil @jungly, I've been exploring the Libbitcoin codebase and find the project very interesting. With your permission, I would like to discuss with Adi Shankara the possibility of allowing both myself and @David Chocholatý to collaborate and work on this project.
Thank you.
3 replies · David Chocholatý, Tanay Raikhere · last May 01, 2024
D
David Chocholatý20:27
Hi @Tanay Raikhere. It's nice to hear you are interested in the project. I'm sorry, but unfortunately, because of the time pressure during my studies, I could not complete the SoB assignment task on the minimal required number of points. So I won't be able to work on the project via the Summer of Bitcoin programme.
Tanay Raikhere19:56
Hey @David Chocholatý, I hope you're doing well. It's unfortunate that you won't be able to join the project this time, but I hope your studies are going well. Despite you not participating in this program, I'm hopeful that we'll still find an opportunity to collaborate on this project in the future. Take Care!
D
David Chocholatý06:40
Thank you so much. I will stay in contact with the project so I believe that we will find an opportunity to cooperate. Good luck. 😉
Sunday, April 28, 2024
evoskuil17:38
Hi @Tanay Raikhere - that's fine with me. Does that mean you want to collaborate on the SQL task?
evoskuil17:50
Spending some time looking at table configuration today. Hash tables are still between 1.5 and 2.0 average collisions per row, despite blockchain growth. I've added the sizing for the address table to
bn --measure outputs and others, as well as to my spreadsheet and config settings.I've had a task for a while to increase the link size in the txs hashmap from 4 to 5 bytes, since we are at 92% capacity with 4 bytes. So I've just done that, which adds 1 byte per bucket (~.5MB) to the header, and 1 byte per block (~.8MB) to the body. Due to good table design this was a single constant change in the schema header, an updated static assert, and updates to a few test cases.
I've also added a txs.size field so that we can quickly obtain the wire size of a block or the full chain. This data is also stored on each tx, but summarizing all txs is expensive. This adds about 2.5MB to the txs body.
I'll then scan this in
bn --measure and other summaries to show the current wire size of the stored strong chain.evoskuil19:54
With that done I've moved on to incorporating generation of the neutrino filters table. It looks like the initial design for this table isn't good. The compact headers need to be computed sequentially, and the filters need to be computed off of a deserialized block with populated prevouts (at least the
output.value). These are different contexts, with populated blocks only available during validation (which is skipped under checkpoint/milestone). While validation is ordered, blocks are not confirmed until the subsequent confirmation step, which doesn't have access to a populated block. And since the filters are blobs and need to be linked from a header there is also a late association problem. Solvable, but not in the existing table design.evoskuil21:26
Holy shit... by reducing the configured thread count the sync rate with address indexing turned on (and the new block size metadata computation and write and larger txs links) I and now getting a full 1Gb/sec block download rate. With 24 threads configured it is back to redlining bandwidth and has only increased CPU to 30-40% vs. 20-30% without address indexing.
I didn't believe it. I attached the debugger and stepped through the new code to make sure it was executing. This will remain constant for 1.5 hours and complete, since we are randomly downloading blocks across the chain and with 150 peers or so, the pattern remains almost perfectly constant. So basically, we are hashing and indexing 85 BILLION outputs (all of them) at a total cost of 10% additional CPU and the necessary storage cost of 109GB.
evoskuil21:41
The only way I can explain the difference from 64 to 32 to 24 threads, where CPU usage went down and network utilization went up, both about linearly, is context switching. Load was being evenly balanced across 64 threads, at about 1.5% each, and CPU was 100% with network at max 50%.
And now after dropping to 16 channel threads I'm able to hold more peers (200 vs 150) with normal download rates and CPU as dropped down to 25-35% - with network bandwidth still perfectly pegged.
CPU will definitely drop way down once we have shani (sha256 intrinsics). This is not only crazy fast, but is preserving CPU for concurrent validation.
evoskuil22:05
Went down to 8 threads and dropped off to 70% network, so interpolated and at am now hitting peak bandwidth at 12 network threads, at less than 30% CPU. Slightly higher than without address indexing, but almost not noticeable. This is not even with a full RAM load, just cold starts with periodic flush. If you can't tell I'm kinda flipping out. 😉.
Monday, April 29, 2024
evoskuil01:12
Sure enough, the performance held for the entire sync, completing from empty store to block 840,000 (halving) and with full electrum address indexing in 1:35. Confirmations up to that point completed in 1:45 total. So maybe 15 minutes longer. That's 85 billion output serializations, sha256 hashes, hash table header updates and hash table body serializations totaling 109GB in 15 minutes - while syncing and indexing the full blockchain.
😮 1
evoskuil01:19
So I finally have some precise sizing numbers for the wire serialization. The set of blocks up to 841,303 totals 568,590,567,136 bytes (530GB) and is archived and indexed by BN in 670,219,288,155 bytes on disk (624GB). This is 17.87% overhead. The estimate from my two year old sizing spreadsheet is 18.58% !!
evoskuil01:27
There is some room for compression in cases where flags and values are byte-aligned and can be merged in a few tables. The above also excludes validation metadata which is 100% prunable (and isn't very large). The address index and its header are ~115GB, which is reportedly smaller than Fulcrum and larger than ElectrumX, though I'm not certain about those references.
1 reply · evoskuil · last Apr 29, 2024
evoskuil14:24
Turns it that the ElectrumX ~25% size advantage is a result of partial key storage (11 vs. 32 bytes). This has the effect of increasing hash table conflicts 3 fold, but far worse, it causes the potential conflicts to have to be read, serialized and hashed for key comparison. This huge performance tradeoff for a modest space savings is way not worth it. In benchmarks it underperforms in query by orders of magnitude.
evoskuil01:50
Fulcrum is 144GB and takes 7 days to sync to a Pi4 8GB over fully-synced Core on same machine.
(edited)evoskuil02:26
This is the log from the last run. It spans 23:05:52 - 01:07:17, but the actual sync to 840K completes after ~1:45. I use the high speed event system to track progress while it's running. I vary the log output here somewhat (toggling [p]rotocol and [n]ews levels, and level 0 is pretty cryptic), and it's low verbosity most of the time. You can see the startup and shutdown chain and store sizings.
(edited)bn.log.txt
2024-04-28T23:05:52Z.0 ====================== startup ======================= 2024-04-28T23:05:52Z.0 Using config file: "f:\\bn.cfg" 2024-04-28T23:05:52Z.0 Press CTRL-C to stop the node. 2024-04-28T23:05:52Z.0 Please wait while the database is starting... 2024-04-28T23:05:52Z.0 open::open_file(header_head) 2024-04-28T23:05:52Z.0 open::open_file(header_body) 2024-04-28T23:05:52Z.0 open::open_file(point_head) 2024-04-28T23:05:52Z.0 open::open_file(point_body) 2024-04-28T23:05:52Z.0 open::open_file(input_head) 2024-04-28T23:05:52Z.0 open::open_file(input_body) 2024-04-28T23:05:52Z.0 open::open_file(output_head) 2024-04-28T23:05:52Z.0 open::open_file(output_body) 2024-04-28T23:05:52Z.0 open::open_file(puts_head) 2024-04-28T23:05:52Z.0 open::open_file(puts_body) 2024-04-28T23:05:52Z.0 open::open_file(tx_head) 2024-04-28T23:05:52Z.0 open::open_file(tx_body) 2024-04-28T23:05:52Z.0 open::open_file(txs_head) 2024-04-28T23:05:52Z.0 open::open_file(txs_body) 2024-04-28T23:05:52Z.0 open::open_file(address_head) 2024-04-28T23:05:52Z.0 open::open_file(address_body) 2024-04-28T23:05:52Z.0 open::open_file(candidate_head) 2024-04-28T23:05:52Z.0 open::open_file(candidate_body) 2024-04-28T23:05:52Z.0 open::open_file(confirmed_head) 2024-04-28T23:05:52Z.0 open::open_file(confirmed_body) 2024-04-28T23:05:52Z.0 open::open_file(spend_head) 2024-04-28T23:05:52Z.0 open::open_file(spend_body) 2024-04-28T23:05:52Z.0 open::open_file(strong_tx_head) 2024-04-28T23:05:52Z.0 open::open_file(strong_tx_body) 2024-04-28T23:05:52Z.0 open::open_file(bootstrap_head) 2024-04-28T23:05:52Z.0 open::open_file(bootstrap_body) 2024-04-28T23:05:52Z.0 open::open_file(buffer_head) 2024-04-28T23:05:52Z.0 open::open_file(buffer_body) 2024-04-28T23:05:52Z.0 open::open_file(neutrino_head) 2024-04-28T23:05:52Z.0 open::open_file(neutrino_body) 2024-04-28T23:05:52Z.0 open::open_file(validated_bk_head) 2024-04-28T23:05:52Z.0 open::open_file(validated_bk_body) 2024-04-28T23:05:52Z.0 open::open_file(validated_tx_head) 2024-04-28T23:05:52Z.0 open::open_file(validated_tx_body) 2024-04-28T23:05:52Z.0 open::load_file(header_head) 2024-04-28T23:05:52Z.0 open::load_file(header_body)full file
👀 1
evoskuil02:41
The milestone is set to 840k so full validation and confirmation kicks in at 840,001. This is not currently fanned out, so it plods along. You can’t see that in the log output but you can see that the confirmed chain has not reached the height of the candidate chain at shutdown.
(edited)💯 1
evoskuil03:34
I run on a high ram 7 yo machine with 32 dual cores and an SSD. I just dropped 3/4 of my RAM cards, which puts me at 64GB. This moved my sync from 100% network to about 70%. I then restarted with bn network thread configuration bumped from 16 to 20. That restores 100% network utilization without even a noticeable increase to CPU. In other words, super fast sync with the memory-mapped store does not require super hi RAM, it can compensate with cores.
3 replies · jungly, evoskuil · last Apr 29, 2024
evoskuil06:44
Yeah, it’s getting crazy. We’ve solved fault tolerance, will have solved disk full soon, and with periodic flushing have also managed memory thrashing. This will require some smart config/management, but is really amazing, because we are able to retain the incredible scale/perf benefits of the append only mmaps.
jungly06:45
I am so keen to try this out now... need to fix this broken old ubuntu I have knocking around!
The pc is misreporting the RAM config as 160, but I assure you it’s 64.
This is the first time I’ve tested RAM sensitivity, and this is great news.
evoskuil04:01
Also looks like Core is 25% overhead vs. our 18%, and they don’t even maintain the costly reverse indexing that we do. This has to be a consequence of redundant utxo storage.
evoskuil14:24
replied to a thread: view
Turns it that the ElectrumX ~25% size advantage is a result of partial key storage (11 vs. 32 bytes). This has the effect of increasing hash table conflicts 3 fold, but far worse, it causes the potential conflicts to have to be read, serialized and hashed for key comparison. This huge performance tradeoff for a modest space savings is way not worth it. In benchmarks it underperforms in query by orders of magnitude.
Tuesday, April 30, 2024
evoskuil02:59
Worked off a little technical debt today. Table sizing/bucketizing/config. Ran a couple full syncs, consistently 1.5 hours to block 840k. Discovered I used the output hash as opposed to the output script hash for the address table, so that will make the indexing slightly faster. Also wrote up a test case for a query against the Binance wallet address, with 1,380,169 payments. Once I resync with the proper address key I'll do some runs through that. I suspect it will collect all of the output links within a couple seconds. These are direct navigable (in memory pointers) to the output and its tx, which is then constant time searchable for the confirming block (if any), and the spending input, with direct link to its transaction (and search to block). We've got queries in place to find confirmed balance, etc.
🔥 2
Rich Phillips21:15
Hey! So, I have bx running now, good progress. I thought I had just under 6 btc in a stealth wallet, I have a previous transaction and some other keys etc, but so far the only wallet address I can find has 0 spent and received satoshis. How do I start to trace the other wallet addresses these could be stashed at?
evoskuil21:23
bx fetch-stealth -h
Usage: bx fetch-stealth [-h] [--config VALUE] [--format VALUE] [--height
VALUE] [PREFIX]
Info: Get metadata on potential payment transactions by stealth prefix.
Requires a Libbitcoin server connection.
Options (named):
-c [--config] The path to the configuration settings file.
-f [--format] The output format. Options are 'info', 'json' and
'xml', defaults to 'info'.
-h [--help] Get a description and instructions for this command.
-t [--height] The minimum block height of transactions to include.
Arguments (positional):
PREFIX The Base2 stealth prefix used to locate
transactions. Defaults to all stealth transactions.The prefix is part of the stealth address.
This search is linear, it might take a while, especially with a short prefix.
(edited)base2 means its going to be like:
10101110
I have public and private keys, transaction hashes, wallet addresses.
wallet address but not stealth address?
I have the output of ec-to-public
(edited)That green bubble top/center.
That's how payments are located by the recipient.
Rich Phillips21:34
Ok so I have stealth address, no of sigs and scan public key. Can I extract out the prefix from
That somehow?
(edited)Last payment going there of 5.8 btc
🔥 1
stealth_address
{
encoded 12TFFcDyvpZd4Zy1GAS7sp7Xz9sgRabovgf8xfD4EMGxenJw8ivsa3bBj8TzjR
filter 11111111110000000000111111111100
scan_public_key 031bab84e687e36514eeaf5a017c30d32c1f59dd4ea6629da7970ca374513dd006
signatures 1
spends
{
public_key 031bab84e687e36514eeaf5a017c30d32c1f59dd4ea6629da7970ca374513dd006
}
version 0
}filter is what you are looking for
Even you had all of the info, we didn't need to search for the payments.
Electrum never indexed stealth payments from the chain, and there would be no way to find them other than a search of all sufficient length op_return txs and testing then using your private key.
May have been dark wallet chrome plugin. Its Been a minute
evoskuil21:42
When BS stores a tx it checks for a pattern match for possible stealth and indexes it according to the possible prefix. Collisions with other txs are possible and intended.
That index is very expensive. BX will return to you every single tx with the matching filter, which is just the leading bits of the tx hash (I believe). I said from the op_return before, but I think it's the tx hash.
the op_return will carry the ephemeral key.
Yeah I have that key
Sorry, I mean shared secret. You can see the flow in the graph.
What you need to spend is the stealth secret. And you cannot produce that without the shared secret. And you cannot produce that without the ephemeral public key, which is in the op_return.
So you need the
(edited)op_return values (of txs with a matching filter, to get the ephemeral public key), and both your scan secret (to use with the ephemeral key to generate the shared secret: stealth-shared command) and spend secret (to generate the stealth secret using the shared secret: stealth-secret command).evoskuil21:52
You can simply try every single tx on the blockchain but this means signing to test the match, which is many orders of magnitude slower than indexing.
So I have a couple questions, the answer for which would help me help you.
First, the private keys that you have - did you extract them from DW or Electrum or something else?
(edited)evoskuil21:56
Ok, I never used it so forgive my ignorance on what that means. But I assume they are organized so that you know what you have. And it sounds like you have one set of keys - specifically the
Scan Secret and the Spend Secret from my diagram, right?Rich Phillips21:57
I have scan public and scan secret, also stealth shared and stealth secret from dw
And ephemeral public key from dw
evoskuil21:57
And you have
Stealth Address that parses with BX and produces a Scan Public Key and a single Spend Public Key?or just send me the address?
(edited)Rich Phillips21:59
vJmxvmsqJFDCnZDwLLY9Ck8P7vLXJqUS6tPhfstc9gca4c7GVs9uBuA8LjAVHURLPg5AQwRkoJAWMF5XCHNd4ByGTerYckHDyY1HgD
That’s stealth address
and no prefix 😕
Ok, back to the private key identification...
I have scan public and scan secret, also stealth shared and stealth secret from dw. And ephemeral public key from dw
These don't map exactly to the image, and if I can't see that I'm lost.
Left of purple circle
Scan public derived from scan secret
And ephemeral public which derives from ephemeral secret
Ephemeral Secret is created by spender, you can only obtain the Ephemeral Public from him (or the op_return).
stealth-shared is a commandSo I assume one is shared secret and other is scan secret
👎🏻 1
The arrow flow shows output.
Two input keys, one output
So yes I have shared secret
Ok, if you have that it means (1) you don't need to scan the chain for that particular payment, and (2) it's not something you should have been able to generate unless you were paying yourself.
To produce the
Stealth Secret you needed both the Scan Secret (from your wallet) and the EPK which is generated from the ES which is created by the payer. The EPK is stored in the tx.op_return. So I don't see how you have it if you haven't scanned for it.But maybe DW scanned it for you and then you just never spent the payment?
Rich Phillips22:13
It very possible. In effect I have an old log of a bunch of these keys, and final transaction
Trying to unravel them and access in theory some coin lurking
OK, well let's just assume that's the deal. You can use the
Shared Secret and the Spend Secret to generate the Stealth Secret using the stealth-secret command.evoskuil22:16
And can use the
(edited)Shared Secret and the Spend Public Key to generate the Stealth Public Key using the stealth-public command.There are multiple ways to get some things...
You can also get the
Stealth Public Key using the ec-to-public command with the Stealth Secret that you said you have the keys to generate.So you have that also.
evoskuil22:20
So to sum, you have generated the
Stealth Secret and from that derived the Stealth Public Key?evoskuil22:23
Using input-sign is a bit complicated, but basically you need to construct a tx with an input that signs the output that was spent to (I assume the same tx as the
(edited)op_return output with the EPK as was the convention). And I think p2pkh by convention.There is a detailed example here: https://github.com/libbitcoin/libbitcoin-explorer/wiki/How-to-Spend-Bitcoin
So do you have the tx.hash and index of the output with the payment to the
Stealth Public Key?If not we can scan the chain for it.
evoskuil22:30
$ bx ec-to-address 0230926948702e0b77d3b58eecc850726ec15ae426505c030638655cf7668c4d11 14vWwNqum1GoUEt7zWzk2JuXJiyBG6199M
evoskuil22:32
There are no standard payments to the address derived from that public key, and as I recall it that is what you need to find.
So what was this about Electrum, and the 5.8 btc?
Rich Phillips22:35
Sounds like it either didn’t complete, or never started. There were supposed to be btc in an old dark wallet from years back, but clearly not so much
Kinda sucks
Last info I have
evoskuil22:42
Was it a payment you were expecting or might it have just been a DW bug where it misidentified the stealth payment?
Easy come easy go.
Was it supposed to be a stealth payment? Did you give your stealth address only?
Happy to help again if you have any other thoughts/data.
👍 1
Notifications off all:)
I can try and clear it if you want.
It’s a heck of a maze
That diagram is really awesome
Tho
👍🏻 1
evoskuil22:55
It took a long time to get those commands organized to that level of symmetry and simplicity.
And the diagram was essential.
But hardly got any use AFAICT.
(edited)In other news, it looks like were are operating at one to two orders of magnitude faster query than the fastest existing Electrum/Fulcrum server options out there. Similar store size (smaller than the fast option - Fulcrum, and larger than the absurdly slow option - ElectrumX). Sync time under two hours, for the full node and address index, compared to a full Core sync plus a week or two for the address indexing.
(edited)🔥 1
This is the result of a 300ms query from new process start, to get details regarding 3000 unique payment addresses.
dump.csv
output_script_hash, output_fk , spend_fk , input_fk , ouput_tx_fk , ouput_tx_hash, ouput_tx_pos , ouput_bk_fk , ouput_bk_hash, ouput_bk_height , input_tx_fk , input_tx_hash, input_tx_pos , input_bk_fk , input_bk_hash, sinput_bk_height , output_script, input_script
0f73582dfee8a7beb0700b20fa4a92dcd0a7d144a72579937b766bec69be1600," 1,265,365 "," 4,294,967,295 "," 1,099,511,627,775 "," 15,656 ", 7e7cc7b60ba3f0248d79de984882a82491faab8b6b6407b297cdcbe4edf36b3d, - ," 25,875 ", 00000000bedacc2be841e9219cb9af2dd62caa98271bea78634b2f57daeb47a5," 25,875 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [04d04c3f703caedd6a2dddd4e56ddaa92936734b08a9987b339f96f00869e93f70dc9040897b49e10583bd71c86d5490b79115d2a71804c51d099b964f748a9760] checksig, {unspent}
90e5e7076c45e776a58991f75fc828ea4ff32948e41779e8c32c2e3843c92000," 1,271,683 "," 4,294,967,295 "," 1,099,511,627,775 "," 15,734 ", fdd9576e197d4ac3c7bd293638d7e5328785aa426520116722d32418280bd296, - ," 33,452 ", 00000000cdc6b2015e79a32290536b65f9fc230102974669cee3cb0b2fc76729," 33,452 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [0425c3bad2070a31a4ce77805cdbc58b20a9e4f0fc49e7ccb7bafbd4a36dc3335150b9182db54303889f85eb112e526578bbf04f6718c451c1c394618ad063b446] checksig, {unspent}
bcc5bf77671e2e86b90c1b040058138eb9062d8461e3872f9122a8c27b952600," 1,407,910 "," 23,698 "," 461,557 "," 17,423 ", 42073bb10a375271460be0a32ab1bd0acbd474aaabef6f62156a635dda03c4c8, - ," 40,410 ", 000000001ad53692dff24fd899eed7275242c807d18537fe2cf771c2bb6d65e7," 40,410 "," 20,477 ", 4391b11d991e7c9ad4f9a1a5a7ea9ed7f234643b0c883f49511e1394a5ab8ff5, 1 ," 41,741 ", 000000006352b2dcc349adc2dfd7196b383f36f4d6a953a66f24a6368239b493," 41,741 ", [0494851ba2da12058f4261575ed75e3820b23cacade86f3a944abaab087880c3da2b0cf57d127b32ce1d758a15748f9092abfcc5cb7d04e8c8c78efc3b3720cca9] checksig, [3045022012e89958e765350190e2d2a30a77100f9cd3e5e070b1ad0dfbd5211255d4456b022100c71675c30238749bf3f3791d22d7ffb6d4783c580479d6007e4aa58b9237895c01]
74b693a657b826c179091363dfd134cd3424abc0499bfd03a8e51ae1fc7a4c00," 1,273,303 "," 3,449,442 "," 453,801,197 "," 15,754 ", 9ddbff7eebb6eac92374b66eff4c025e32e0dbd6ceedde777091db8a733e11a1, - ," 25,946 ", 00000000086ea9e39dc2722fdd00383435315d9de7b50a5e9315d310ea6d1edb," 25,946 "," 1,859,628 ", 9a6ddcc68dff604985ee7a7fe8869cf53116657a82d794d5edb829ac42621e0d, 1 ," 152,951 ", 0000000000000961967e284fb33abde28900cdd24f9f24eb7fe6b3fc9283e2da," 152,951 ", [04e765ba745fe5a34d7d8fd9869c55994a13fd43754c025d47bebcfdef4b146dbf9f7bf5fc726acd84be42d7e9ebcd8e1ebe5c305cf3ddfe1ee2bcf0734cc793b3] checksig, [3045022100a649f0557e9d77fff6e4a45b16d6d7c34103b2caa25a4dd20dcecbc645166ad602201026adb84d6048b7cae13df294e580040787d28cab7bec8f521dd0773842cbf801]
54c2a5134e8e934514565c329fe886813956cbf576a6c790c977abcc92595600," 1,307,602 "," 1,584,000 "," 200,965,409 "," 16,179 ", 33706bd68a3faed5ffed523911bfe041d3aeb551d23fc930de3494c7745e42c5, - ," 20,992 ", 0000000016fcf2ebf167421e882826d0cee5d43a4f2e0186ace2f5fb703b6758," 20,992 "," 911,548 ", 4713d7f7c0baa56d83401ece58b98a011e12fb45cc8f27d15c2076c2d55acd46, 1 ," 150,799 ", 000000000000011ba2fa5a15a28c0b6d98d18597f1c120580385a363ef00c380," 150,799 ", [0401b94844752407495cb0df360b93ab93e17abd7a3baed0eb636b1fa631148b0ebbccd90feabb37ecc7bea6442be2c337bb71acb23b1267a02e1de42a72679fbd] checksig, [3046022100ff1eed41f660103776667760da47a8abf1dc95952a34debf3a8be028b709420b022100a0380ed95373bc009f5b1ba5df3b1526e8dfb2a8a664e1f4ce990747fd63374301]
d058f0ca45e5a50ff3f6a0220e74c3c9ded99634bfebcd3c5469c816f2a29200," 1,234,546 "," 4,294,967,295 "," 1,099,511,627,775 "," 15,275 ", 304dc57e7e8f7b2af0b8a8170df1e6519dcd577fce1dc7619ad301d9b2714dfb, - ," 33,331 ", 0000000027b224dc0acdd44e2b1ea5f05f3909847d6f6f1a4f98abfb22239b4c," 33,331 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [04a02f7f117a6b3d405c43ef2dac2c803d22c7d9b9be66a4b4320de7fce296007ff803c7ad9db5189d5d779a7e71dc5339a1ed2fa42b0fa6dca39e5e368e0c9fcf] checksig, {unspent}
8a0731c56eb4dac2d3e11e71034aab67d3c82c31b6d65da78bd75c80f5259400," 1,286,302 "," 202,859 "," 16,058,778 "," 15,915 ", 60e4d6dbc2c0b55facdc78562963013e0d7d47a0fb7dfec16ecca1b93368c094, - ," 33,492 ", 000000006094db8ec50a847083442e57be3bc90497365c841c17e740f02446ff," 33,492 "," 139,817 ", 50123bbb315f212c16fef2fd300cdfe301b43c6fb2cce3cbef56d8fdfc2d352b, 1 ," 101,191 ", 0000000000030337b7adcede9c7da122335fa04e64a925187281f495f5e3fb4c," 101,191 ", [046a360216d6028d929c8a982c1d6a2054fc99cd4c060219ae5858aa542e35c920ce3e0489a157a1d4a86691550181a79536d8e315409fbfd682aa9de8ef3a7b74] checksig, [3044022045b26ee62e1dca56edc68f34eaefc7bd1d39797b8d2b3e4281466d3ea5e654a302207f97c0daf1f9d2a320de550e7cea8c63a84354a414cf41fe80a45e0f50a1295801]
65e0fef80638407b61940e2a582cdc43f258b330dfedb9ff8945124be5c0b900," 1,359,559 "," 23,838 "," 465,787 "," 16,822 ", 518db647d22e7176d131195fa804df15a77ac17a24739c19a82f86cbb3856036, - ," 40,079 ", 000000008630914b2f3fa68fa948355272ed0d12c02fe72bf6c915a0da94f155," 40,079 "," 20,576 ", c2807194f82dc0fccab898df4a26c75a650ebb88f60c871e3d0e954776009bb9, 1 ," 41,794 ", 000000003b440de466fbd31a7889c87aabeadd342b418d2986127ead95bfb859," 41,794 ", [048ae63207992767af706092b7ee35e83bcd09c89cc5e0ad6f5788663b527ec70e9622906f737497668636ab2d7b7f99bcf463054418bbe4bc01b5cf72d3f8d317] checksig, [3045022100a145857e5c97140d8cd3380c342e865fea3477277ef189a1e48c3f118ab9323c02201e9bd75242554718ed8387fbaaa199a2dfa4d47e800c45600c16e25ec64a19b501]
bd2effd3f50e3660ae1db30de4a3b6b9a7ee3c349ab354f5dfcd89fab317d800," 1,260,262 "," 3,397,139 "," 446,869,344 "," 15,593 ", 2ee4b98f7fdf49b2f32558df05ef6eb50ead1c6ee661c2bbf7d9a045353d32d2, - ," 20,968 ", 00000000dd65c3aec7c1d2e6c6080935ae6f1a2c0e7142f5e9240a59423d5dcc," 20,968 "," 1,832,884 ", 2cbdb443bec99191a81c5503250c6d84e66e52d5bb953ff3abcd5f68f7ffd285, 1 ," 149,796 ", 00000000000001e4c4619a7b59d5122af3fda7428482732898f078947e122440," 149,796 ", [04926766c74f242e3115e2ead310cba5d9a42a1292902df72f8135e723e9af642c9b1cb3961b1ace55dbabda4865121f1928464a0736c7b10e725bc1fe5add130f] checksig, [30450220440fde93e55e6f694b46c4c125813a820ada72a4b02603f36f3de2f54368f28f022100b00172874aba76697f5e6540207c22e27ab786f46480687559855b7b9c120c6701]
f25075faaa86b4e5a9f83785955b1b284678164037092b43bedc0947d45fda00," 1,261,639 "," 18,467 "," 356,368 "," 15,610 ", 00362050d5006d30e4021c83baf62734a1b173073e3ad0d730aafdc8ca6a2d32, - ," 25,840 ", 000000007f39962611519c0d69085287051e99e88caca9185695a2a444eee916," 25,840 "," 15,987 ", 28c0959d3b71fda237462e0bacd9c77e079d9557d8ed4d2b7e92c435dd42730d, 1 ," 26,112 ", 00000000a5ec628ab7ebf616a3b5c2ce0ac82fc6a2b97e457a9c45534c817384," 26,112 ", [040ac264f92330ab8fe60cedf62d21b75775e94ae176c36d63112419e27bb40c2e35a38149000cfc39e266ba3ab839e79543107cc74d8d25bea49a497d0df15cb4] checksig, [3046022100a8ce1f8b0c1baeb84f3a5b0210622a7a3c788f5084c82d73c7f1efb952643e82022100fb8ee7f8d0c27d5a8e8ace9d6f2f0ebc6ff6a9dcaa5c0e544bc8d9e0d65b300001]
3a5ea6d5ed2c0b77dfc76fc0cef9d593903422aab97c71ca49dca514c146e200," 1,223,248 "," 811,799 "," 96,147,072 "," 15,135 ", 45bda08870ca861b0aa88e4748fa18c56a727dfa890f324c1ca2e67c177c5acf, - ," 39,799 ", 000000000898551e4c2e0c5d3b3b79d0f5b0e7cb1b4a5b8c226afd513b62f0e9," 39,799 "," 523,571 ", bd7904ba84e3dc85d03fbe074f49710b839225a030689ccf063347c36ade5a37, 2 ," 131,303 ", 000000000000083227fc89458209d62217f45d4bec0a9d8ec77f57701efc3d01," 131,303 ", [04841a2776574842640fec6a0c8103d7f68bcfd0607677d37f6556a509725f41305cc3cb9a5030e591d7d7650893f84be4823af62b5595eff686d13e18e8ffc13f] checksig, [304502206d4b9e97f0c1d9fb0fbac9e1ce96b7b340801a4d92fcec44909f4900c3050352022100e709a5551d25e8a777f0f18e613355350cf5fe8726a1ecd9cd7181301f433c5a01]
63fe0809ea3685b595b67cb8f647621af739d9dfab8766f859388a0761d90a01," 1,279,822 "," 72,011 "," 3,248,216 "," 15,835 ", 4031ff21955b0ae6f9fdf253d39ad19ca1674bb965246faed0d19f1db66c5879, - ," 26,003 ", 00000000fa648627ade8b1ee9a89c31b1ef6a81f25aa0d0d114ee67b66403a75," 26,003 "," 52,058 ", 6827872cccd912a2ed5f4b7f603a570b97f6e68a8c2a9048b4f88417691f8d33, 1 ," 70,692 ", 00000000002fcb227537689c83c8ef564478567d5cde080f870d2b64ef76244c," 70,692 ", [042b67a27bad8affb9576c22de147d8319b2ef5bc78988066c3d697c11b08864e7ea9a6e81012e409af13df4ac2c0e3c37d99ec646dc6eeb9cff6c74a25ee2d36a] checksig, [304402201ae555d7d8deeb66f6bab5229c4c27c85850d18f9cf2604903dfe513981eede60220397075b074e51d7b0565dbc4bdfae7cba4fa2170238a8ffc3fc8fcf49506008b01]
1e4611cac9a6bc32f332b694bb29a7bb6b9da478dfaf12f58dca101605bf1b01," 1,251,799 "," 238,761 "," 20,385,124 "," 15,488 ", ae4a5bfcdb12e699827e75d1a09f488cf82c089daad023d7ad5ecdc8674757a0, - ," 25,752 ", 00000000d8dc8ddc20772bd6e5c8367305ee7494ad24dda14c8705b0bde239e3," 25,752 "," 165,430 ", cff0225cfbf538b9f512b2ed69fd2d23306167792d503f6e0b2bef9120adf239, 1 ," 98,585 ", 0000000000040b923458721e36607697d1e742be04d12ff904772caf3ef76b81," 98,585 ", [04bf1d899cbde529dc49705e1b61bc6201c638acc1056957d8015d1944bf24c432b345c42d2cca589f4d8889d90b0b7025ae1562b23f3f95fa07e6d5d0a1ba84c9] checksig, [30450221008bb78c4f80c07f1d3cb411d7c69294a5c4c9da7051ff9e4529ce607069f3c2ba02201c4b1448f7baf2cf17df1e44c0621ad3a7c05235c13f16063d7294f1d9fafb6001]
27d4fee8c552d81e53ef9be7babcb3a4284568740f3349afee1ede357b031c01," 1,261,801 "," 4,294,967,295 "," 1,099,511,627,775 "," 15,612 ", 889ebda7d37e86b12626496eec0ee86d2edc9a99e554cb47bd929cac0dc93d50, - ," 25,842 ", 000000002b23e369dcb2ee4fe7261a28abc139d762ea2685aada593c41d01328," 25,842 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [04dff1512fc653d30f88e5570c8a69926330fc14071dde35ed17e6373d6921b363782989e47315c08994dd82534f1604bb31195abed0ccaf9522c98e731fa37b35] checksig, {unspent}
94d0dc8f676e9420e3936282b64a5e786f580c23cc17399f5f1d014b5b373201," 1,400,751 "," 171,812 "," 12,658,918 "," 17,335 ", a873fd38f32b8c2e72c5042bed33a17be73e3f512d13288ddff157595fb40178, - ," 33,895 ", 000000002a42e41bb7e681d39be6fbcb8fa4ede9a61afa39b95f23ccb447ae2b," 33,895 "," 116,543 ", 6e163ab42691e03cea2c24f14b951f37d7b254f5f7e920d160a4adc18449f97f, 1 ," 67,954 ", 00000000012f3c54167d721e010f98073cf66946aafffd031d190237832becbb," 67,954 ", [043cf81ef91a9f08f3394566e8eeb0285977838dc5f9e75bc20492092e9e0f1e23af98904a5592cba442332018124e6f5e325d571bf5add45017be84940f7e9916] checksig, [30440220444f6a3424abc80a69599860a87e671565b1d90fe1fb485e8240a2a8f678931d02206559a8d93ca01d8b9be073de6dff176a0a6c72f7e0522c1c4120033d572b1b0b01]
c0e2bb2b2815d037955382d845a42c33077abb81f97559d4ff8a457d6f1a3c01," 1,316,917 "," 4,294,967,295 "," 1,099,511,627,775 "," 16,294 ", 273401cc9b0ff5d5e1b75e9e380b957d81db1e096b0b1663291337483ea8c9ad, - ," 26,332 ", 00000000418f9aea44f0452c54eab8932b0427f0ba3172ff47e1e17ada6476b3," 26,332 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [04076b5a803811a45b288e2a6eb264d795c238773e9ec66e662bcbe1a55f93cc23b6b78b1a1a3763b027876dd7644ebaa900a87beb26450b69582e447b35953b86] checksig, {unspent}
dd188eb19c0ee7e3b2f6a20b25fce6b65e44bf53c3b27eef21b91ab961265301," 1,297,558 "," 4,294,967,295 "," 1,099,511,627,775 "," 16,055 ", 63149d1c958bb81d47611143ca82b58cee6817d4454681f30a72b1e3b2c010f3, - ," 26,160 ", 00000000731a19159614d13b2925d5783a3fc2a104ed99842f1f6fd7465c87ab," 26,160 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [046d577bbf35c1fbd88d0093b079af4cd264d081e986d130ca8ddbf28b95533d1338e0984cd6111997032a13345e44adfe8d78eeb844d646c64c0477199d2e8ded] checksig, {unspent}
cc63f7924227e2a438d2d29085a3058d4f70a1f5323aecb00b0fab461c1a5b01," 1,322,830 "," 13,463 "," 185,389 "," 16,367 ", af9606ff9dfcdb52659e140ef9a59051041554350cbfc34146c8583f83b9db06, - ," 33,600 ", 00000000922811d417808514e99a4ae78f610da31c53abd1db184fd430b0b7df," 33,600 "," 12,773 ", 45807808b04f84f4e8964b26a15e417af8f952492126e5b0f702514354810637, 1 ," 38,330 ", 000000003c935661944da00acd35ee9abfa6d6d0521526f8cb70ece985984c0e," 38,330 ", [04f43bea5634056a80b40b005e07a8cb60e883f1281315bc240e10c367898c12aea2cc52257f8a6c3041da10d49e6e3e1e18bc72817d7028cca83d3aa0d03555a8] checksig, [304502203d4bacde79589d12ced78903aa729d561707d61bb094319a41c3c973175e9b6b0221008e04e16e427ddcbb53a413a1d132cf53b3aa61fadc0deac35a35399176c4996801]
170da287ab1de4be9f513a3a9ff0579a28b10ecac44dfed68e7f262f832e7801," 1,286,707 "," 51,850 "," 1,749,183 "," 15,920 ", a85e3d27514c44aa99a1b7bd17296cbcded6c091ff230d5a97d7b7777f2e63d7, - ," 33,493 ", 00000000271d34e1a2056fe2aca129edec33487fe12f05f4fcc9b95ce293f189," 33,493 "," 39,313 ", b1ccc290ece8c6e6162f8dbc9c95600e2a0a4f1c6bf806cbd4560fbbbff74c2f, 1 ," 49,387 ", 000000001cf51af3d1ffc49659eae87189517b5e8040508c949498fbdb1190af," 49,387 ", [045ab2b3723a9677ad259f221027a99784289466091e11a4ec09397ccc8d9f7c8682cd1759494079f5331044beeccaf89c82f3ebedefc3a6f51472ebc024fdc6be] checksig, [3045022035a74139a952a062debaf30666f864a73e0433402c71f2ffe12a2de12535ad1b022100d65a3bd46198023619e4e51b0837517e9ff735e80f8e971e106b9bc61d77848e01]
e2038c754cceadd4a5b5f0c58be925e947e1197959fcccb7d0e28b37be6f9b01," 1,296,829 "," 18,921 "," 363,895 "," 16,046 ", 1dfc3d07a73b41d5518111d78f58ec83162dc5deca402cd4aad243a40740fda1, - ," 26,154 ", 0000000063c8a56727f702b2ce49c80f56eb41ba9fcabf36a2a4cf27993f8c90," 26,154 "," 16,394 ", 6029e51c4706c8b12bfdc9bab2311cc99d7fd5ccca3701bff4af2935c4a89215, 1 ," 26,402 ", 00000000c4ace881b3d82195dc6a4a5e1c03755cc025a8ab53244db62733422b," 26,402 ", [04e6a4e0fb8c3397c75c5cf3591f9ac4a1d870a35cce7cfd6e3c36aae48696611346610e5d87d9ace2ad676a3ac98422388b4a2641c2669a7f7203046ecc633cc5] checksig, [3045022100d3405b2f5ae8218251e4a5471f400b641958f66de229b8228cd819e8afabe7ee022057141fb475a9d541e9a4d8484675faf8a944c0ebef4e82029cff8ee41bcbe17c01]
e64d43c8c335b901439ef5eab8871bf1d8106ae762cffd31d8bcdb2b4ab0c101," 1,253,500 "," 18,473 "," 356,833 "," 15,509 ", 446747e9c1c8139b6f42f071c1216f8a7c5502bce45288daf4b73a2e2c0a396e, - ," 25,770 ", 0000000078843ac2caf45ba66da56381ef8d27aa35e8dac5aa9fa6536a84f618," 25,770 "," 15,987 ", 28c0959d3b71fda237462e0bacd9c77e079d9557d8ed4d2b7e92c435dd42730d, 1 ," 26,112 ", 00000000a5ec628ab7ebf616a3b5c2ce0ac82fc6a2b97e457a9c45534c817384," 26,112 ", [0413c3f7c3c94b09f6abb906e1feb60b1f245f99239832437fcf68eead6919c52bb3526babe471066b3c86dda96fa9cffb65edef97303850a56b9d23b142b3f7ac] checksig, [3045022005239bc9feb165b51ae1a99dedcb867f9328090aa170971fd7307ea9b54b16e2022100cc8715daf223cf7884d039997033f31400f1c04a73a17bd425cf7323706ed33401]
3348962e6abe95b44aedcbb29e4abad8a65e9981ced894443f47a57220f3da01," 1,314,487 "," 1,167,710 "," 144,052,069 "," 16,264 ", a9591bd6af9dd4a15eada1cc11e151115c6269114a0e27ef25adb86aeffe050c, - ," 26,312 ", 0000000098ddf30379e53011943e579c02d1dea53d643bd33124ffcbec3b27ab," 26,312 "," 703,485 ", 5f1a76d6151a1c9dcf486dcfbe53086a7ebce50d5c2e554c41ce33e5ae1d072f, 1 ," 156,150 ", 0000000000000885ce3b94937fd20842c838bd5bc015908032fcd4fd4f5b0c48," 156,150 ", [044a02f0a0f15b35540416644c9ea87ce5515d2dc21aecac785071584acfab14d0a0e4b90b362e46af2c0b85de7c90b06448be0e757a3ffef7e31380427bf2947e] checksig, [3044022063e82d307443ac07d4f074c7a58e52307013c9f83bd1fa03ce05270ed2d7717a02201a637671e9f9d210729c02a6d3b69e9bc63ff161c03db3328c7bd160489308ed01]
185f98098baab1389dd6be0aaffd2f55b18b357b4acb4fb8489d15f4db17e401," 1,323,883 "," 20,786 "," 401,060 "," 16,380 ", 7834fef33811d3bc041b41850683a5bbd3aae705d829aa25627a352f65c8c80e, - ," 26,393 ", 00000000a761d6f076139757c697aa6ea61f3251dbc5c04c38900e21662fe1fc," 26,393 "," 18,007 ", 6b85ad8bd569e29a4587a8f15ce22ef3e0372da2d1f9b160dabdaa50e9981c25, 1 ," 26,815 ", 000000005ec300d10e8dda3f80eae0059f61f3e1c6d67c6b4b2d9cd5f7e1a40f," 26,815 ", [049b55376c621a39cf8537d7df4cf6085f2090f36f387a8c43a5e3b41922d567c8949be8729f4b37612bbaf88a48ebd0937df30b11dcbed5ab4a623c9a27ab57d6] checksig, [3044022031d915929d44f46ac6eda485e34d4012acdb491789fbf142a70dd0f9720e3d5d02205a7370ada9f05d644e197645defb179d43576a0fb24673055d6080de54ade19201]
97d7e398944a53313c5cc500ef9fa5eff62d2705c6e1126877ff6ba81d47f801," 1,253,176 "," 70,226 "," 3,095,503 "," 15,505 ", 66d79fc1bd29051494ef671202d209de0e37ac4c8c849c071e98c03162edb82b, - ," 33,400 ", 000000005dac67ecccc6134c3218302a014e60b80aa2695a9874e35958c79f02," 33,400 "," 51,193 ", b9d69463c500b81dc2ff600d0b528b15fb816c4dab11e424151de0bcbea7f26f, 4 ," 70,527 ", 00000000010870a799f0abc04547f2d1483e6e760eac6c38e849f159bf2e93ff," 70,527 ", [04117d4580325df6014dc0b1aeddcfdddf3070bc26b73fb10601c91375cbe3899a24b641819cb93960ec2175f8df2356fa439a93b309901224056196c5572c2c80] checksig, [304402201b36906081550f92fd63943d5c8da2b63086fc0d561c774d0e4940b52bdf584b02201d46e36f9d92962da30762d72cdeabd90ae825ba94654d76f16e95d59cc8a5a201]
18180d70f1ffb83947ec6dc3699ddcfb6be8d95ae917c4d0de4150b0d3b32902," 1,283,386 "," 983,942 "," 119,323,066 "," 15,879 ", 68399e35c0e5c2b2719262e89dbbd6bdc6597e3f0829e5d63eab6af5df900d3d, - ," 26,035 ", 000000000ae70bc42a99858f329436738c0c8c0a962b15bea1dadffa0c90968a," 26,035 "," 611,264 ", 44d79db844f3f63293ca568655080cf6cf38b209480f64b97fb78369702d0768, 1 ," 155,532 ", 0000000000000a98679cd8ba557c11442963ac12829cfd20bd459973eb52056d," 155,532 ", [04af499d6fddfb795b801cd0ccfc815567847d40a3cef6bbdff5cc8499f31f3868b0d8587a76a241f0a107fe9cf14003fe2a9e4c85c76ebb0bf96d68fc7d889c1a] checksig, [3045022100ad55e7f69128bb55a3cf8a62716e80bc8c6645a61d9fca0cb04552055bfd651702204fa85d4be7568598c8c4680bd1e79267fef2075fc45a60c2c38ab4cdd988c3b201]
409e29b5cfd5ed2323d72a1521e8b2ad2eadf148f58db2bef59812f94bfe3302," 1,356,076 "," 112,875,237 "," 14,953,341,191 "," 16,779 ", af81c8a4f2f3a05cd0e938128943dabb8fff3d2400ed95caf85623c3f25ba065, - ," 40,053 ", 000000006eb68d4a89390b5699c77cbf1038d1a99f376dee3a7003b0d537615a," 40,053 "," 45,946,162 ", a21f2ee9a8ebad7bb3c45d80ede4aa4ca5ef9407534f71b265395c2e4ae0a958, 1 ," 126,409 ", 0000000000002eda614ac818a63fea56ef227463866dbc660a6a2ff41eb0f7b2," 126,409 ", [04c1a7fd380945f9c5aa33e637eef9868e69be2b0e6e574029e155a285c7fcec0ab215ea09d052337cef83ef41e55b8f69ef26518a2a7dcef0a06f4fa16a935a7a] checksig, [3044022031d29f5fa397d39fb78e96b58e7e0d06b0a0b3509d682a004d6b8ff78f5aab1402207c1f8e67aedc0cbf537477784437e20ae9cb8819dfac9050f7b70738ffd9c9b001]
698e5d18625d88b142426faf98f6bb6a6e5036ae22ed8b507f95ed6ef1f43a02," 1,213,419 "," 4,294,967,295 "," 1,099,511,627,775 "," 15,013 ", c54a74b3502ccb461e176733d9d6da18443446cadfb81ee69bf2e8e2378138a1, - ," 39,703 ", 0000000052dbcc004dc626f5c5c123ecb5833fdc284b90be3590f36398b4cb65," 39,703 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [040d7c42ca38b07cb6b0b8da4e6c9d647553529d31b7d9a5dc3b772cb60b4f9e8f33c61317da656a1dcdabe4a0c69d18061a3f291c7f7c4995f911299e3d2c0661] checksig, {unspent}
2b28723130b8c71b5b12caa17d7c4558d233964e4ccbec5bf10831bf04ea5302," 1,378,932 "," 4,294,967,295 "," 1,099,511,627,775 "," 17,062 ", 334ecbc3bd781bada1543c0c02587d3d4fae5379d58ed7526c4dfd6803fd1689, - ," 33,787 ", 00000000c8592e187f3177f759ab92afe0e315d8b10baa063868524d995af5ff," 33,787 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [0496d0d7f3eef0842c635ffd2716b642f040306495906e17b0bba4d17a84d05b5c014f28c6bcea197561fd95807cbe73054a9c8e1971f2bc9b0622fffada11b47a] checksig, {unspent}
421e8fc3ab3226f3bb951931c9d1fcf5975f1ecb5c39d32dc97beeebff5c5b02," 1,323,640 "," 4,294,967,295 "," 1,099,511,627,775 "," 16,377 ", e822f1f2fef9a36640bfe9b6b8d5329fe49d7977a063ada202a5579c0e01cff2, - ," 26,390 ", 00000000ae7603a45a6ec6f9f82cd8b1392b03056ca2962bfd1a71606385c021," 26,390 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [045d8e94db09f36f6218385a65bf5a315153ca45ecc03c5af7a4dc856cd8b121882bd4e1759f8d7bfb98ad472de05a2175fd8c5bd36d49df07a7a3e516bc330341] checksig, {unspent}
240434f67276e53d55781c9d3a14d81be19632b0a6d78fe92d501df3a2f59f02," 1,253,824 "," 4,298,628 "," 564,709,441 "," 15,513 ", 1608da84cc4bfdae78098c203129b986f90be3889e41f0dca01d208e7483998c, - ," 25,773 ", 0000000059db9fb3ccf39f490db20b65285ec352c46018a6beba32982bcd1a84," 25,773 "," 2,278,624 ", 873e0b5f48ea58a000811862e64b916783e20e5c0f32f6e9d44a85b4afc800d4, 1 ," 153,427 ", 0000000000000634c1d0ef38e7800b0f9deb40b257faa81a492a2dbf843bebe6," 153,427 ", [04960e278c4325e80fe93a2f3467c675877275fb4f555738bdc2c708d674629c0ecab3429012b5f6e03d99ff1325d45bee4c9f1f43612f4b87c54acac2677553d4] checksig, [3045022007a0045d1dd75754b4419afdeba7c05ab7afe049ba81f6392063c1bd3890b4b90221008d448e04e14c14e9ab29b7a7aec6e85d0087d2287b67244c188e147bd7ef992201]
a66d736f6b954f664f46fd2ca03040910b97fe4d20efba2c1840a71d28c9a202," 1,285,573 "," 110,509 "," 6,723,554 "," 15,906 ", 098ab4c3a6f0b074f2bc37d15252f18159ae9106dc266f8dbcbb4b21b627c747, - ," 26,057 ", 00000000c6a6988992020d1de38a98721f72e5607bb6eaedea50309a50c68cb7," 26,057 "," 76,686 ", aaaa2ef07c4aa47a15375ba6a920a0e93b82a4307d5d45ccaf2672d1e60d29e4, 3 ," 82,659 ", 00000000002a02ca13cf12115a8e1454831222ca0960e868ef5bbd34b10f7eff," 82,659 ", [04bd60ae32cd3a6c2749873662232e3a4f20f745a4e86f0cbf25a53127ca20fad096cdd18363e9288462e2ea6cde1f68faeac5a65d067b9fcb168cc3cadfe65ce3] checksig, [3044022070c76061463fc1412895f275f7b8db14d37e09778b94653230161e903273d3a902201aac3bad2d95b2773268b57c1ea9192dd288bbc4ef67267c4d5f3b0253abf61001]
e84c6c804bf24851c5bc377edd03839dd51265c34e77cdcc8fe4fcc278c3ac02," 1,298,287 "," 22,334 "," 428,701 "," 16,064 ", 2d4bd8f95f190a720b1f09611d928f0d8369f9f04f5075ac41ca2d49ca576eb9, - ," 26,168 ", 0000000043d0d59ae1a46b6aec71ea57f696bd243934b7eaedbf6340feba0713," 26,168 "," 19,401 ", b6af26bf360c141937de55a88ef8033852c372d188d38863a3e0ec1c71bd63eb, 1 ," 34,402 ", 00000000ad2f2440cf943ed113e6ce03f5edf23075d941a220fd1d0bcff5bea2," 34,402 ", [04744b067a7f79f99566e164e371a2762d8866ebb876d6fa0b393ebc495b5ce584fee567c8d87c22d143de8ed22bf173cc3c7640761e0417e79f17a44199cfd23a] checksig, [30460221008dcf8a616e4c82681a03c1e4bd18da0bad913cf3b3113ce33c27f21403387f950221008d0c39a5eb5a2c73c6f0e14a5faaae88028fd4bb9cb215b3aea0c633bae85f9801]
d6a2bd78093172d6d77bfbbd66679be9d3554fca7ada68a2ca196f43a639ad02," 1,330,726 "," 4,294,967,295 "," 1,099,511,627,775 "," 16,465 ", 01de3f1488561675e8342139da2d6d4236956d9f38b52e994719f0d89dd86f50, - ," 26,452 ", 00000000e873ef276067ffc982e7ff865c4e42b9aaa8bc1faf091ea4cfaee37b," 26,452 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [04bf523e04eb3e929f55c6c1b46d819df10273d08e678c18483750f053d0ea5128329345c826f4f9560dda04c362e38c1ee5203e54676ce177fc9498543bf20125] checksig, {unspent}
5dfc3f2fd1e2955a879b63e72d840e7df874572ac4508110296ac0029c4dd302," 1,316,350 "," 15,992 "," 266,116 "," 16,287 ", 9d5b1a1101619212a7664a1c83e78fe53a64873b076cd72c6f92a92534bc7c95, - ," 26,327 ", 00000000bda85634b719c1ce7e00960abfd04cd964da8d1ce45d041eb4016f7a," 26,327 "," 14,491 ", 0f5007ea2f309b3609c08a6caab8986e04e7f064db2e2b68446c27c6e7c52232, 1 ," 39,318 ", 000000003816cb4cb0e18de90a6d1bce7b600959c63f37f50009fcad80de36ca," 39,318 ", [0442e48d8a599b44fbbee7992af90056b9a9224d3d0274e40758c2a7791295a69ffcc82a814c7ac29e67420601be54a2ee9ba04ff4bced9c3d2bc9ef63c314fbe3] checksig, [3045022100b17b590f2c0f611ccb83471b1e36a2890b8e6a8819981ba96bf88d982d7ec05502201e3c83df69ff2e96e1ad6b19ec87f11594f0f07910a787a9de135892352076e001]
e449f7abbef6793cf07d229fc3889da43dd139a33a4fd6ce777bd2697872d402," 1,344,820 "," 15,930 "," 261,404 "," 16,639 ", 82195d47acd56ff914c9bcadbf5ca8ba3c93fb2bdbd802438691a615ceaac060, - ," 26,577 ", 00000000a6b0b7312aac3a85eceafc58fb36717d7ffea82bd64216ae1c4f27ea," 26,577 "," 14,491 ", 0f5007ea2f309b3609c08a6caab8986e04e7f064db2e2b68446c27c6e7c52232, 1 ," 39,318 ", 000000003816cb4cb0e18de90a6d1bce7b600959c63f37f50009fcad80de36ca," 39,318 ", [04da1af6b5d2de0f73396ccdd92fd2a9c1d49ef10a9daf0c3072cf8b30c0b84b8b4fe2c92d9f7d40f050ed59754399d6c6ed501eb2de9f1f181db2e202589d0873] checksig, [3046022100feba960044c0655101e70006855bc0406c273b43eb7578a60c62873380c13f44022100b07024e16230395cf338cecebe540735343b54ce1535039ba82e6ee8c2beefe201]
4504abf007ddb6ce41486125f856173a706d68b5298dd53ce0604af1ddb51f03," 1,452,343 "," 15,948 "," 262,777 "," 17,971 ", 13c987dbdd2047e608b0400dc0846e999f9d7df87748b7384ba1962e977aaa70, - ," 26,805 ", 0000000085811e25709d73805f723e230bc6d686a02b39f0e22ca11b2d204ebf," 26,805 "," 14,491 ", 0f5007ea2f309b3609c08a6caab8986e04e7f064db2e2b68446c27c6e7c52232, 1 ," 39,318 ", 000000003816cb4cb0e18de90a6d1bce7b600959c63f37f50009fcad80de36ca," 39,318 ", [041e3de5c48140f66c3df66842dbcf4285cda5620dea98518d8f4aab7ae172312225909041e600d6bcc33e8f13301f87da4134c5353da18b20336628f1d69971d6] checksig, [3044022037e19bf3646180f6150e46f5b6f476332a90127bf635d0146eaa04f56f7d14c102206fe1e8c26f268f271330691b2ccdd5cf46055e246d9ea3dba5e63bf7b233864401]
e229a094fc1295f7b2aadf4a06b5b5a2207ff020cec9a0a1269b8569628b5b03," 1,357,939 "," 4,294,967,295 "," 1,099,511,627,775 "," 16,802 ", 33e2551ff17fdf6464011ead1b62fa4a6a54c8254a8b98f254e7567115eaf98c, - ," 40,066 ", 000000003abff3c98bf4f54a10144adade2d016429659f6813560c2dfe4a0e4f," 40,066 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [04ccec093ee27e71ff806390384569734f578476c7b8f91a1b9b587fe6fefb05b7f1f332fb6a4d35934a2b75b8ca1b6c38c658f97aac0697d40aa0294616e2dd83] checksig, {unspent}
2ccb6e4e280ce77bdfd26737370f21e8def42481b7fd58aa1a8ac9b7627f9103," 1,222,195 "," 4,294,967,295 "," 1,099,511,627,775 "," 15,122 ", 072aee2b6ed9174d22fb932a4509d73617a5840dcd8fbaebe32a3fb6e409328b, - ," 39,788 ", 000000001d39fb012d340c2b0434eafa03098a5015a3c9b4d890c52b6b21c62c," 39,788 "," 4,294,967,295 ",0," 65,535 "," 16,777,215 ",0," 16,777,215 ", [043306a779de1e40b75c7f7a32062b4adadce113182da9f919f3542f39e618b2ca2b878e2c871544bda5dec4d6cd4bc4fde0e8a88f77095725145fc075c90b9ec3] checksig, {unspent}
c9a11f003d0ef4b86f393a82c4fcc5babc9d68eca4bdbd7a6e86c9f808d1a303," 1,454,083 "," 70,516 "," 3,127,026 "," 17,993 ", 101cc03a7687491c10cb9d8ff5822e5d5986098e7fb02f97f3f2624ff6cb5471, - ," 34,070 ", 00000000888fd249b2e3477905399cbea6403309f4910e378bcf3e3e210d4d15," 34,070 "," 51,193 ", b9d69463c500b81dc2ff600d0b528b15fb816c4dab11e424151de0bcbea7f26f, 4 ," 70,527 ", 00000000010870a799f0abc04547f2d1483e6e760eac6c38e849f159bf2e93ff," 70,527 ", [0464368f8d0f55be752656e21b72b53c7021954d2f30d15802090c5c2de6522c792e1c89551b5395b27563b75377e74d7de3b476ebd41836f0c88420dc4f70c99a] checksig, [3045022100c8a34b393d989ea2303ec2d3f42051ef0b156758f0a3d3e9718995d902edcf500220190e3ca583e374adf9fb2b510245ba8b8e658f23bb924952892d267453ed3e3b01]full fileThe minimal details (comparable to existing benchmarks) take less than 100ms. The above includes not only all payments to an address, but all spenders of the payments (of they exist) and all tx/block details for the output and input and their txs/positions and blocks/heights. Spenders is an expensive reverse index, similar to the payment address search itself.
(edited)