Bitcoin full node and query server
April 2021
Archive of the libbitcoin Slack #general channel.
Thursday, April 1, 2021
P
Phil12:58
Dear Eric,
After pondering for a while we now realize what the join() from thread remark in the code regards.
We suggest this slight clarification:
What about
“// This must be called from the thread that constructed this class (see join).”
->
“// This must be called from the thread that constructed this class (see join).
// Specifically, threadpool.join() will be invoked and for it to work, the threadpool.join()
// call must be made from a thread that is not itself a part of the LibBitcoin threadpool.
// An example of such a thread, is the OS process’ main thread, from which this
// class was created.”
https://github.com/libbitcoin/libbitcoin-node/blob/master/src/full_node.cpp#L357
https://github.com/libbitcoin/libbitcoin-node/blob/master/console/executor.cpp#L220
evoskuil14:25
Actually I believe that is not correct. Boost documentation states that it must be the same thread.
P
Phil17:31
Wait, I don’t understand what you are thinking about within Boost, can you pull out the documentation reference for the Boost docs quote you have on your mind?
2 replies · evoskuil, Phil · last Apr 08, 2021
evoskuil22:51
I remember writing that comment in reference to boost documentation. I'll look for it when I get some free time.
👍 1
P
Phil11:17
Please, let me know if you have any news on this. I checked the boost code for
thread::join() and there was no comments about that.Saturday, April 3, 2021
evoskuil22:49
The backport of electrum commands (and an update to [bip39] mnemonic commands) to version3 is complete.
👍 2
Tuesday, April 6, 2021
P
Phil13:32
Dear Eric,
From my testing, I can see that node.stop() is blocking. But, by reading node.stop()’s code i’m not quite clear what part of it actually blocks.
Just for being really clear, I wish to ask you to confirm: node.stop() is blocking, right?
Over on my side I am in GUI logic code that must not block, so I will then mitigate node.stop()’s blockingness, by spawning a separate OS thread wherein I will do node.stop(), and my GUI logics will nonblockingly wait/poll for that thread’s completion, thereby then knowing that node.stop() and node.close() completed.
In my main thread, I would be fine with a sub-1-millisecond block by node.stop() and node.close(), but not more, in which case I like to spawn the separate OS thread as I suggest here.
Any cue on what’s the appropriate action please let me know.
evoskuil13:49
Is the node hangs on shutdown, node stop is blocked. The block may very well come from server, not node. You cannot correctly mitigate it, it needs to be resolved to ensure store correctness. I will eventually get to it, just working off a lot of backlog right now.
Thursday, April 8, 2021
P
Phil11:17
replied to a thread: view
Please, let me know if you have any news on this. I checked the boost code for
thread::join() and there was no comments about that.evoskuil19:34
There's no news. As I said, it doesn't really matter, if the threads don't self-terminate then that is the issue that needs to be resolved. Focusing on other ways to close the threadpool is in my opinion not a good use of time.
P
Phil20:07
Eric:
In my application are two groups of threads:
(1) ‘My’ threads. This is the main thread created by the OS, and all the other threads I personally pthread_create():d in my logics. None of these threads are included in LB’s “threadpool_“.
(2) LibBitcoin worker threads. All of these are in LB’s “threadpool”_“.
Please confirm that in your view and to confirm what you are saying is, the treadpool::join call can be made from ANY of the group (1) OS threads, however from NONE of the (2) OS threads as if I’d do threadpool::join() in any of the group (2) threads of course I would get the termination problem that you mention, which would be that threadpool:join() would wait for all threadpool threads to terminate, and the caller would be one of those threads that are being waited for to terminate, alas a deadlock.
Please confirm to me that this is the whole picture and there is no more complexity in LibBitcoin or to your knowledge in Boost in respect of requirement for which thread to invoke full_node::close() from.
So this is also the full meaning of the comment
// This must be called from the thread that constructed this class (see join).
bool full_node::close()
https://github.com/libbitcoin/libbitcoin-node/blob/master/src/full_node.cpp#L358
Soo, did I get it right? :)
May I then suggest rewording that comment, just as it took me a week now to clarify that it actually meant what I thought it meant
Something like
// This must be called from the thread that constructed this class (see join).
// (Specifically, this must be called from a thread that is not one of LibBitcoin’s worker threads, that is any thread created by LibBitcoin. These are contained in the threadpool_ set.)
evoskuil20:25
No, because I wrote it for a reason and it's been a while since I've been in that code and I cannot say for sure your assumption about it is correct. Feel free to create an issue that described the question and I'll work it out when I get to that code.
evoskuil21:12
You can see in bn that I call close from a process thread monitoring for ctrl-c, I'm not aware of any problem with calling close from a thread independent of the threadpool.
Monday, April 12, 2021
P
Phil14:20
Dear Eric in safe_chain.hpp (line 166) what does the word “locator” in the method calls ... mean actually? Thx
virtual void fetch_locator_block_headers(
system::get_headers_const_ptr locator,
const system::hash_digest& threshold, size_t limit,
locator_block_headers_fetch_handler handler) const = 0;P
Phil14:30
btw, is this how you get the UTXO?
virtual void fetch_spend(const system::chain::output_point& outpoint,
spend_fetch_handler handler) const = 0;P
Phil18:52
Eric just for me to double check, using safe_chain how do I make this check: “Is block with header X in the chain genesisblock...Y?
This is instrumental for me to make queries that are scoped to one consistent view of the blockchain, defined as the blockchain given by genesisblock...Y.
Tuesday, April 13, 2021
evoskuil01:03
Request block zero and compare it to your genesis block if you must. But the chain will always have the same block zero after first run, since block zero is obtained from sources (or in the case of bn, config), not from the network, and there is only one chain per instance.
A server can always lie to you, so if you want to know for with reasonable assurance you must obtain the full chain of headers and (statically) validate them. If you want to know for sure you need to validate the entire chain. If it's your own server it's a fairly pointless exercise.
Spends are inputs and outputs relative to a certain address (output script). You know that they are spent if there is an input that spends the output, which is is provided in the query result.
The tx hash and index of an output that is not spent identifies a "utxo".
1 reply · Phil · last Apr 20, 2021
P
Phil12:36
Hi Eric, thank you very much for your extensive response on this one.
I am running the LB Node in my own OS process, so I am not suffering a risk that the RPC from embedded LB Client to LB RPC Server would be MITMed/hijacked.
In my own OS process/C++ program, I want to query the Bitcoin blockchain according to the convention of a consistent view. Just in case LibBitcoin does *not* implement this, I will smartly emulate it myself.
I want to describe my usecase to you however, to avoid traps and be clear.
so in a given moment, I will make a batch of queries to the blockchain.
Logically, at the **START** of my queries, the Node will be at one given head block that we can call H or HEAD.
The fact that the Node is at HEAD, means all blocks between and including GENESIS and HEAD are queryable.
I want to do the operation:
var myConsistentView = node.chain(); // or node.chain().giveMeConsistentView()
printf( “The node is at ” + myConsistentView.headBlock().blockHash “\n”);
myConsistentView.getBlock(“1928309182309812093182”) <- this one should succeed if said hash is within GENESIS ... HEAD, and fail otherwise
myConsistentView.getConfirmedTransaction(“18329837498239832094820934823") <- same here, this one should succeed if said tx hash is within a block GENESIS ... HEAD, and fail otherwise
finally regarding UTXO:s, so a UTXO within one chain (GENESIS ... HEAD) is introduced at one block , and then optionally spent at a future/higher block in the chain.
So to get to my higher level question:
Does LibBitcoin implement a consistent view, or do I need to emulate it myself in the way that ANY of the query operations I do, MAY be such that a reorg happened BEFORE it, meaning that the GENESIS .. HEAD chain is **NO LONGER queryable** , and so I need to rinse-repeat by picking up the now-latest HEAD, and then query against that one instead?
(For clarity, I understand that there are transactions that are not in any block, also I understand one transaction can be in more blocks on different forks - i understand all that and that LB does very sophisticated and efficient handling of the same. What I’m doing now is more like a very consistent comprehensive approach to ‘clean’, correct, best-practice blockchain queries so my questions related to that rather than to other LB design aspects / internals.)
Tuesday, April 20, 2021
P
Phil12:13
Eric, I am doing deep diving in the whole main code path of the lifecycle of a Node, including its blockchain explorer calls like get block etc.
I have dived for a full week now, and I estimate it’ll be at least 10 more days.
For my time to be used effectively here, any studies, overview comments, such, of this, that has been written by anyone else, can you please list me all such meaningful sources of information.
Things I want to learn are like,
* Will a callback that I provided to Node.chain().getBlock(), Node.start() etc., ALWAYS be invoked, or only SOMETIMES.
* Where is the guarantee that all of a Node’s resources are deallocated
* I want to follow failure code paths
The code is mostly uncommented, so I need to make sense of respective sections as I go along and this takes very much time. Possibly I can write some comments while doing this, and PR. Any references to material that speeds up my learning would be much appreciated.
Thanks & great to see all this solid code!
I'm not a fan of heavy commenting. I only use comments to describe things that are counterintuitive in the code, for people who are generally familiar with it. An explanation of how it actually works belongs somewhere else. The primary effort is to make the code itself readable. There is a good amount of work to do in blockchain to get to that point however.
Comments have a nasty way of becoming inaccurate as they are not verified by the compiler. I am planning to introduce tooling for managing comments on public interfaces, but this is more of a documentation exercise.
1 reply · Phil · last Apr 20, 2021
P
Phil12:37
Which up to date-enough documentation is there now?
So I’m following a Node’s whole code path from creation, through start, through operations get block/tx and then to node.close() and delete node;
P
Phil12:36
replied to a thread: view
Hi Eric, thank you very much for your extensive response on this one.
I am running the LB Node in my own OS process, so I am not suffering a risk that the RPC from embedded LB Client to LB RPC Server would be MITMed/hijacked.
In my own OS process/C++ program, I want to query the Bitcoin blockchain according to the convention of a consistent view. Just in case LibBitcoin does *not* implement this, I will smartly emulate it myself.
I want to describe my usecase to you however, to avoid traps and be clear.
so in a given moment, I will make a batch of queries to the blockchain.
Logically, at the **START** of my queries, the Node will be at one given head block that we can call H or HEAD.
The fact that the Node is at HEAD, means all blocks between and including GENESIS and HEAD are queryable.
I want to do the operation:
var myConsistentView = node.chain(); // or node.chain().giveMeConsistentView()
printf( “The node is at ” + myConsistentView.headBlock().blockHash “\n”);
myConsistentView.getBlock(“1928309182309812093182”) <- this one should succeed if said hash is within GENESIS ... HEAD, and fail otherwise
myConsistentView.getConfirmedTransaction(“18329837498239832094820934823") <- same here, this one should succeed if said tx hash is within a block GENESIS ... HEAD, and fail otherwise
finally regarding UTXO:s, so a UTXO within one chain (GENESIS ... HEAD) is introduced at one block , and then optionally spent at a future/higher block in the chain.
So to get to my higher level question:
Does LibBitcoin implement a consistent view, or do I need to emulate it myself in the way that ANY of the query operations I do, MAY be such that a reorg happened BEFORE it, meaning that the GENESIS .. HEAD chain is **NO LONGER queryable** , and so I need to rinse-repeat by picking up the now-latest HEAD, and then query against that one instead?
(For clarity, I understand that there are transactions that are not in any block, also I understand one transaction can be in more blocks on different forks - i understand all that and that LB does very sophisticated and efficient handling of the same. What I’m doing now is more like a very consistent comprehensive approach to ‘clean’, correct, best-practice blockchain queries so my questions related to that rather than to other LB design aspects / internals.)
P
Phil12:37
replied to a thread: view
Which up to date-enough documentation is there now?
So I’m following a Node’s whole code path from creation, through start, through operations get block/tx and then to node.close() and delete node;
P
Phil12:37
Oh by the way, for my clarity: after successful node.close(); as in node.close() gave me true, i can do “delete node;” right, and also all of LB’s worker threads have been shut down and there will be no callbacks to any of my callback handlers I gave to any aspect of Node/chain, right?
1 reply · Phil · last Apr 20, 2021
1 reply · Phil · last Apr 20, 2021
P
Phil13:07
Oh wait, I did not get this in full so, there is NOT a consistent view feature right - so I will emulate the consistent view feature myself in my logics that use Node.chain(). Did I get it right now?
P
Phil12:39
node.chain().getBlock() and node.chain().getTransaction() and some more blockchain query operations take a callback. will those callbacks *ALWAYS* be invoked, or only *SOMETIMES*?
E.g.
Node.chain().getBlock(“7314982498723984732”,function(block) { ...});
Node.stop();
evoskuil12:39
Whenever there is a reorg, the fork point and all incoming and outgoing blocks at included in the callback.
1 reply · Phil · last Apr 20, 2021
1 reply · Phil · last Apr 20, 2021
P
Phil12:47
Are you saying that all of the query calls under Node.chain(). , that take a result handler call back, that result handler callback will ALWAYS be invoked?
e.g. if you run node.stop() right after then all such callbacks are invoked before node.stop() finally shuts down the Node
Well, there's a good amount on the bitcoin.it wiki as well.
When a call is made that has a callback parameter, the callback is how the code continues. It's standard async behavior. If it doesn't get invoked the code would deadlock.
(edited)evoskuil12:43
If there are multiple callbacks, like error and success, they are always both invoked. This ensures resources are freed.
reorg implies fork. Every new block is a reorg, even if just adding to the top. Fork point is the top and one block in and zero out. This happens with every change to the chain.
If you subscribe to block announcements you always have the current chain.
P
Phil12:47
replied to a thread: view
Are you saying that all of the query calls under Node.chain(). , that take a result handler call back, that result handler callback will ALWAYS be invoked?
e.g. if you run node.stop() right after then all such callbacks are invoked before node.stop() finally shuts down the Node
After invocation a handler can resubscribe. There's a resubscriber that helps with this.
This ensures that no events are missed.
👍 1
Handlers hold resources in closures. They must be invoked to free these resources.
(edited)evoskuil12:59
As for query consistency, I'm not sure I follow your questions. Any tx or block can be queries by hash, confirmed or otherwise. Txs and blocks include a confirmation state that implies they are in the current strong chain, or not, and at what height. Height implies confirmation. Payments are indexed only against the strong chain. Upon reorg, all deconfirmed payments are deindexed.
Blocks queried by height are in the strong chain at the given height. If you subscribe to reorgs, your will obtain all the information required to track the chain to its top.
Since libbitcoin is append-only, deconfirmation of payments is essentially a reduction in the file length.
P
Phil13:07
replied to a thread: view
Oh wait, I did not get this in full so, there is NOT a consistent view feature right - so I will emulate the consistent view feature myself in my logics that use Node.chain(). Did I get it right now?
evoskuil13:08
Again, I do not know what this means. From my perspective libbitcoin is entirely consistent.
It would help if you would describe the problem you are trying to solve, not explaining how you would do it.
(edited)All data that is exposed by libbitcoin is valid and consistent. Txs and blocks are stored if they are valid, confirmed or otherwise. Txs and blocks are considered are confirmed if they are in the strong chain. Otherwise they are unconfirmed txs or weak blocks. Payments are an index of the strong chain. There isn't any inconsistency in the data.
P
Phil13:19
Eric wait, right so you are saying that any ‘old’ chain actually is still in LB’s persistence and is queryable, it’s just that it degraded from being “the strong chain” to being “a weak chain”.
Relative to a block HEAD, which is in some chain but not necessarily the strong chain, how do I get the block at block height N, which is under it in the chain?
evoskuil13:22
You cannot navigate from any block to its parent. You can only navigate from parent to child. Each child retains the hash of its parent. Heights can be navigated in any direction but only represent the strong chain. In fact heights are what defines the strong chain. Weak blocks of course have height, but they are not indexed by height.
(edited)P
Phil13:22
Thank you for explaining.
In regard of consistency, say that I get a reorg callback. My reorg callback will simply send a message over to my program’s main logics thread, so it’ll do a new round of blockchain queries:
function gotReorgCallback(newHEAD) { sendMessage(mainLogicsThread,{ gotANewBlock: newHEAD }); node.listenForReorgs(gotReorgCallback); }
node.chain().listenForReorgs(gotReorgCallback);
Now here applies, that because LibBitcoin is fully asynchronous, more reorgs can happen.
this means that the main UI thread, as it goes into doing a ton of node.chain(). query operations, *another* reorg may happen between and during any of those query operations right
So this just means that my node.chain(). querying code needs to be prepared that a reorg could happen anytime.
You are describing solutions to a problem that hasn't been stated.
And the solutions sound like a big race condition to me.
P
Phil13:25
So - except for payments (maybe) - there *IS* a consistent view in the way that any block that has been fully validated, will remain queryable even if it drops into a weak chain/becomes a weak block. Did I get it right?
What is a fast way to look up a block by height in a weak chain?
Juuuust in case I was querying on the strong chain which then behind the scenes while I was querying became a weak chain before i was done querying
What's your use case.
You cannot look up something that's not indexed.
P
Phil13:27
So just, “get me transaction”, “get me block by hash/height” *RELATIVE* to one given GENESIS .. HEAD chain
1 reply · Phil · last Apr 20, 2021
P
Phil13:30
so that among a group of participants/computers which each has HEAD, i KNOW that all of them will see the same thing
All blocks in the node are relative to the one and only genesis block.
P
Phil13:30
replied to a thread: view
so that among a group of participants/computers which each has HEAD, i KNOW that all of them will see the same thing
evoskuil13:31
If you want to freeze your view based on a given intermediate (or initial top) block, all you do is subscribe to reorgs, and when you get one you know if any past query has been invalidated. You can base this strictly on the fork height (depth) - or you can compare block hashes from the reorg.
(edited)You can never guarantee that what you have done in the past will remain "consistent" with the current chain.
All you can do is invalidate what is no longer consistent. This is how we manage the payments index.
If you want to ignore reorgs, and just continue to query against a chain that may no longer be strong, you cannot rely on payments or height-based query. These are relative to the strong chain only.
But you can still query txs and blocks by hash that were formerly in the strong chain, forever. Tx and block hashes are indexed.
I have been modifying the payments index to no longer remove unconfirmed payments, and to simply provide confirmation state. This is just simpler code.
Also, I misspoke previously. Each block retains the hash of its parent, so you can walk hashes in reverse from any block, to genesis. You just cannot do so by height.
(edited)We have all txs, so all inputs and outputs. They are all indexed.
spent or not spent.
Storing a database of utxo's is silly if you have all txs.
P
Phil13:49
ah right, there is no UTXO index, only TX index
so.. the “payments index” is the transactions that WON (became confirmed - relative to doublespend)
There are no double spends, just possible spends of the same outputs in different branches.
(edited)So for a given output you can have zero or one confirmed spend and any number of unconfirmed spends. But we don't currently index unconfirmed payments.
Actually in v4 we may already be indexing unconfirmed payments. I'd have to check if I implemented it when I last worked on it.
P
Phil13:55
By payments index... do you mean the DAG chain of transactions and their outputs being spent or not?
Txs don't form a useful graph. They are only relevant for confirmation state, as they point to blocks. Only inputs and outputs form the DAG.
Payment is the term we use for either an input or an output.
Tx's don't actually "point to" blocks, as that would be a reverse n-ary index that we don't retain. Blocks point to txs, and txs have height if confirmed. Outputs have height of the spending tx/block if they are confirmed spent.
(edited)evoskuil14:11
The DAG is navigable in either direction due to the presence of the payments index. Payments refer to points (tx and input or output index). These allows retrieval of in/out scripts, which can be used to query for payments. This allows forward traversal. Reverse traversal is simply following the points.
P
Phil14:20
Last question today i think -
In V4, how do I listen for the highest header downloaded in the headers-first sync process,
and using which node.chain(). query do I traverse that header chain.. just the same usual one, get_header()
If you want to traverse in reverse, get a header and extract its parent hash, then get that header, etc.
(edited)Hash queries are fast, constant time.
P
Phil14:32
oh, and this old question:
How do I disable the download and storage of block body and transactions?
I want to degrade my Node to an SPV wallet. Please tell me which configuration setting or which line is it
Nodes aren't wallets.
evoskuil14:46
You can go into node and disable protocols for downloading transactions and blocks. Then you'll just get headers. But nothing will confirm.
(edited)2 replies · Phil, evoskuil · last Apr 22, 2021
P
Phil15:51
How exactly can I disable protocols for downloading transactions and blocks? Which lines should I change?
evoskuil21:10
Look in the node class, it should be pretty obvious. You can derive a class and override the virtual methods.
1 reply · Phil · last May 04, 2021
P
Phil18:07
Thank you very much for explaining these details about how the blockchain is stored and is queryable
I perfectly understand that:
1 - Any block that is having the most known total work of any block given an amount of time, will hit persistence and do a reorg and then will remain queryable through the interface permanently, that is retrievable by hash using getBlock();
2 - Any transaction that was confirmed, as in if it was a member of such a block at any point in time also remains in persistence permanently and queryable, that is retrievable using getTransaction();
3 - You can look up transactions that are members of a block by their block object and index;
4 - You can also traverse any strong and weak chain by simply walking the parents of a block up to the Genesis block;
So now I’m not clear and please explain to me the following:
With respect to UTXO’s, UTXO’s is defined by the transaction hash and the index within that transaction of the output.
I was trying to figure out what do you mean by payment index specifically.
Then what I want to know within a particular chain GENESIS...HEAD is a particular UTXO is spent or not. And thinking about this problem, this requires a function where within the chain you can look up the membership of a tx id within the chain. And second, you need to have a tracking function whereby you track if another transaction within the same GENESIS...HEAD chain has spent that output. So is this information, whether there is an input that is spent or output, is that what you mean by payment index?
Is there any trace of information within LB for a weak chain about whether at a particular block hash, a particular UTXO, which exists within that chain, was spent within that chain at that block hash? Is there any way that I can pull this out of LB?
If I want to make a query relative to the strong chain of when the UTXO was spent, is there information about at what block height that a particular UTXO was spent, if it was, when I query on the strong chain
evoskuil18:22
2. A transaction does not have to ever be confirmed to be queryable. It only has to be confirmable at the time that it is downloaded. This is our tx pool, which is stored just as any other tx, apart from its confirmation state.
3. Not by block and index, just by block (hash or height) and list of tx hashes. "index" refers to the position of an input or or output in a tx (a "point" is txhash:index).
4. Yes, but we consider this walking "down" since it's a tree structure and the genesis block is called block zero.
UTXOs are unspent outputs. Outputs (and inputs) are defined by tx:index, spentness is relative to the strong chain.
The payment index records all inputs and outputs, making them retrievable by the output script as a search key.
All inputs refer to existing outputs, so one or more inputs and the output can both be located by their script.
(edited)All payment records refer to txs, which have associated height. So any spend can find its confirming block.
There is no association of payments to weak blocks. They are simply unconfirmed.
A single payment could exist in any number of weak blocks, but we do not index them. We index the spend relative to the tx.
(edited)If the tx is not confirmed, it has no associated block. You can consider it in the tx pool.
(edited)There is no guarantee whatsoever that you will see weak blocks. They are only retained as a performance optimization. If a node kept all possible weak blocks it would be flooded with data, like zero fee txs.
evoskuil18:45
Libbitcoin is probably the only node that retains weak blocks, but we limit this to those that have a potential to replace the strong chain.
P
Phil20:43
So, the payments index is an append-only functional hashtable, where key=txid+output index and value=one particular utxo(= subarray of a tx), is it?
Is all payment index information for the weak chain wiped?? Why is it. So I can’t query the payment index for any weak block, even while it was once in the past a strong block?
P
Phil23:25
Are the payment records structurally totally separate, segregated from the blocks and transactions?
The payment information for a utxo contains block height of the block in which it was included is it
Wednesday, April 21, 2021
1 reply · Phil · last Apr 21, 2021
P
Phil11:20
Please tell me using the node.chain(). interface, for a block ‘821398127398721398’ which is either a strong or a weak block, how do I tell if at that block (in its strong or weak chain), utxo 91309812098210938:3 is unspent/spent/nonexistant?
P
Phil11:16
Are the payment records structurally totally separate, segregated from the blocks and transactions?
The payment information for a utxo contains block height of the block in which it was included is it
The inputs and outputs are in txs, and not reproduced anywhere else, spent or otherwise.
The payments index is an index.
It allows for constant time retrieval of any input or output, given just knowledge of the output script.
(edited)P
Phil11:20
replied to a thread: view
Please tell me using the node.chain(). interface, for a block ‘821398127398721398’ which is either a strong or a weak block, how do I tell if at that block (in its strong or weak chain), utxo 91309812098210938:3 is unspent/spent/nonexistant?
evoskuil11:20
From that any related information is retrievable, including inputs/output, transaction, confirmation status, and spender height of any confirmed spent output.
Query the block, get it's height. If it has no height, it's not confirmed (weak).
Query the transaction, get its output script, hash it and query the spend.
Payments indicate the point (hash:index). You will see a record of an output and a spend of that output if it is spent. You will also receive other payments against that same script (address) if they exist.
Unspent outputs are not special, all outputs and inputs need to be queryable. LS allows query of any input or output by its id (point) by simply getting the tx. But it also allows retrieval of any point that touches an address.
Spends are inputs bound to the output's address.
We also have the ability to discover the spender of a specific point (output) because we store the spender's height with the output when it is spent. This allows us to get the block at that height and search it for the spend, which remains a constant time query. However we don't currently expose a query for this.
Spender height exists primarily for chain validation purposes, because when there are reorgs we need to know if any output has become unspent, which is quickly obtained from the output's spender height field.
(edited)P
Phil20:22
Wait Eric,
So, at a given block B with hash ‘19823983217492749827439832’
- this block may be a weak or strong block (in a weak or the strong chain) -
then i want to know if the UTXO txid:index ‘913283194879132834’:3, exists or not, and is spent or not.
How do I do this using node.chain(). ?
So if it's in a weak block, it's unspent. Same as if it is in an unconfirmed transaction, because weak blocks are unconfirmed.
You can query the tx hash for existence and query the payments of the tx output script to determine spentness.
Eventually I'll add a query to directly query the output point for spentness, but that's just an optimization.
👍 1
P
Phil21:13
if I do node.chain().fetch_spend(callback) , callback gets a input_point right, which is a txid:index
how do I learn at which height in the strong chain this input point is located?
Wait, thought you were asking about the client query. Hold on, I'll look.
"/// Fetch the inpoint which is spent by the specified output." This comment is reversed. The inpoint spends the output, and it's the inpoint (spender) that is returned. Looks like I already implemented this 🙂.
void blockchain::spend_fetched(const code& ec, const input_point& inpoint, const message& request, send_handler handler)
{
// [ code:4 ]
// [ hash:32 ]
// [ index:4 ]
auto result = build_chunk(
{
message::to_bytes(ec),
inpoint.to_data()
});
handler(message(request, std::move(result)));
} (edited)That's the response. The inpoint is the spender of the output.
evoskuil21:30
The height of the spender is not on this particular response, since if that is desired it's easily obtained by querying the tx of the returned inpoint.
(edited)I just noticed that in blockchain this query is not implemented, and will return error::not_implemented.
That's because, as a I described yesterday, this is based on updated store design that retains spender height (or an unspent sentinel) with all outputs. This allows for search of the confirmed block (at the given height) for the input that refers to the particular output, so that the input point (spender) can be returned.
(edited)This allows us to index all unspent output information with just four bytes per output. You can think of that spender height value as our utxo store/index.
Storing height allows for validation and reorg with no link traversal. The only scenario in which we look up the spender is if someone is interested in it for some other reason. This lookup is constant time in the size of the chain, and bound by the max block size, which is still very fast for non-validation scenarios.
Thursday, April 22, 2021
threadpools
P
Phil12:34
Dear Eric, at Node closure, how does LibBitcoin/Node coordinate so that all work is for sure drained out of the threadpool, is it a combination of setting the “stop” variable to TRUE which causes all work to stop and cancel soonenough, together with the threadpool.join() in node.close() which simply waits for them to reach their termination point, which marks that the AIO scheduler has ended?
At that point when the worker threads have ended, none of the Node’s resources are used anymore and thus “delete node;” will cause the cascade of object destroy+free that causes the final deallocation of the whole Node and all its work, right?
evoskuil12:36
This is boost asio behavior. when join is called no more work is created and once all threads coalesce the join unblocks. Nothing can force a thread to stop, unless you want state corruption and arbitrary behavior. For threads under the control of zeromq workers in server, a zmq signal is sent which causes the thread to terminate when not busy.
(edited)P
Phil15:51
replied to a thread: view
How exactly can I disable protocols for downloading transactions and blocks? Which lines should I change?
P
Phil18:16
Eric,
When I do node.close() the call blocks for approx 30 seconds.
I presume this is because Node tries to do some kind of clean closure of its peer network connections?
Doing such clean closure is not a Bitcoin protocol requirement really isn’t it - therefore - is there some call i can do to force instant closure, thereby node.close() would mostly be fclose() and thread termination, bringing down the 30 seconds to more like a millisecond?
Friday, April 23, 2021
evoskuil06:31
It is waiting on the operating system to flush pages from the memory-mapped database files to disk. It is most certainly a requirement.
However if you configure the write flushing setting it will flush after every write. This significantly slows down the node during sync, but is of little consequence after you are caught up. With the enabled the node closes quickly. Otherwise it's up to the operating system.
Essentially, with write flushing disabled (default) you are deferring writes until the OS needs some space for something else, or you shut down. Because libbitcoin is optimized for high end performance, there is little reason to made such a tradeoff for shutdown peformance.