Bitcoin full node and query server
January 2019
Archive of the libbitcoin Slack #general channel.
Tuesday, January 1, 2019
skaht03:21
Happy New Years.
If anyone can point me to Check Sequence Verify (CSV) examples with libbitcoin bx commands that will be helpful.
In particular, does anybody know if it is possible to use a CSV as an antecedent to P2PKH payment? Most of the CSV literature looks like it will only works with checksig and payments possibly with P2SH.
In particular on Testnet, I’m trying to redeem the 3rd payment of
(edited)bx fetch-tx e1b429277d53c6aec0850875ad37a93e4cf877826857d09c4deeb983c2770b71Was wondering if the redemption of “[0040000f] checksequenceverify drop dup hash160 [2a773c929b7932fa781b0d1154accaacc565935e] equalverify checksig” is a lost cause?
Wednesday, January 2, 2019
vdeurzen11:42
Is there any functionality in libbitcoin that tracks fees? I want to create a transaction and automatically ensure I have an acceptable fee
neillm13:14
By the time the tx creation is complete, the fee had to have been determined already. It's up to you to add logic on what is considered acceptable during that process.
The coin selection algorithms can help a little, but ensuring a reasonable fee is really something the application manages
vdeurzen14:29
Well, I'm considering taking the average or median fee of the last X transactions / blocks and then using that +Y%, but I was wondering if libbitcoin had something available to get the median or average fee over some period of time 🙂
skaht21:16
Should a fetch-balance and watch-address for v3.4 work for the older deprecated Pay-to-Public Key (P2PK) script where the address is a base58check-encode of the raw compressed or uncompressed public key?
(edited)skaht21:21
There may be people that want to use bx to query older P2PK payment balances. Can’t natively do so today.
evoskuil21:24
v3 does index p2pk payments, but you may need to validate the full chain to see them all.
v4 will index all payments.
skaht21:28
% echo 0000111122223333444455556666777788889999aaaabbbbccccddddeeeeffff | bx ec-to-public -u | bx base58check-encode
5sx9uBHxcA3ysfBVYQfQkYzKHxoT7hPyrKUS3Nk3WzxaJTVqqXunvZsJY1tv2sbySbRjkYJasPA7Bx91nwujF39MLwcQfcPF
Notice client-side behaviors. Appears to be before server query.
% bx fetch-balance 5sx9uBHxcA3ysfBVYQfQkYzKHxoT7hPyrKUS3Nk3WzxaJTVqqXunvZsJY1tv2sbySbRjkYJasPA7Bx91nwujF39MLwcQfcPF
Error: the argument (‘5sx9uBHxcA3ysfBVYQfQkYzKHxoT7hPyrKUS3Nk3WzxaJTVqqXunvZsJY1tv2sbySbRjkYJasPA7Bx91nwujF39MLwcQfcPF’) for option ‘--PAYMENT_ADDRESS’ is invalid
Commands above are on Testnet.
skaht21:51
A P2PK address uses the full length of the raw public key, no bitcoin160 hashing. While a P2PKH address performs a bitcoin160 of the public key that is implicit with
(edited)bx ec-to-address == bx bitcoin160 | bx base58check-encode.
The P2PK exposes the public key on the blockchain once the output payment is mined. While a P2PKH expose the bitcoin160 hash ( bx bitcoin160 = bx sha256 | bx ripemd160 ) of the public key.P2PK base58check-encoded address come in two sizes. Both sizes will be larger than P2PKH address sizes.
The client-server API cannot distinguish (nor is there good reason to).
skaht21:59
Ah, in summary, to query P2PK address one should query the corresponding bitcoin160 ec-to-address.
(edited)Good to know. Thanks for the assistance.
skaht23:40
@evoskuil- Noticed that the question posed yesterday pertaining sending fund to the following output script is non-standard payment : “[0040000f] checksequenceverify drop dup hash160 [2a773c929b7932fa781b0d1154accaacc565935e] equalverify checksig”.
Is it your opinion that to use a conjunctive CSV antecedent as part of a standard payment that one embedded the CSV inside a P2SH payment?
evoskuil23:41
If it's a non-standard script then standard addressing requires P2SH, but v3 will index all payments (including non-standard).
(edited)skaht23:47
Thanks for the guidance.
I noticed that besides use for authentication purposes, https://github.com/libbitcoin/libbitcoin-explorer/wiki/bx-message-validate can also be used for the Proof-of-Existence (PoE) of cold storage funds without the need to thaw the coins.
Also mimics a behavior of Trezor and Ledger hardware wallets.
(edited)Thursday, January 3, 2019
skaht05:58
Sweeeet, see the P2SH( CSV-P2PK ) “transaction currently locked” message during a validate-tx.
evoskuil13:18
I've triaged all open issues in all repos, revising the set of v4 open work items. This would be a good time to discuss any features that we may want to get in or think that we should still cut.
👍 2🤔 1
Parallel block download appears to be working well, and I've been working through necessary optimizations over the last few days. I just added a feature to preconfigure the target store file sizes, so that there is no need for expansion and people will be less likely to disk-fill after a long sync period.
skaht14:03
More good feedback for P2SH( CSV-P2PK ):
% bx tx-encode -v 2 -i 9adfeea2ceec089c52c450b3f33147b45eedcfd27ca48c638a1191bbe9466196:0:4194319 -o 2N9e4AjRBiQyvuu3QMhCahQEDijjavfmET3:10000 | bx input-set -i 0 "[30440220256996a816448204d1bc2ca905fe7f9bbfec2bcc60b0d66956534bbb7881b61502207bd0d7268c979836fc34b8872377f6b096fe0c20078ac858f827b8f0ee14318701] [14e3826cfda3e5233e946a8abc52ddae7d704815f5ac]" | bx validate-tx
insufficient transaction fee
skaht14:14
However for bx 3.4.0 pointed at tcp://testnet2.libbitcoin.net:19091, I upped the TX mining fee in the tx-encode, re input-sign, re input-set, and re input-validate
% bx tx-encode -v 2 -i 9adfeea2ceec089c52c450b3f33147b45eedcfd27ca48c638a1191bbe9466196:0:4194319 -o 2N9e4AjRBiQyvuu3QMhCahQEDijjavfmET3:5000 | bx input-set -i 0 "[3045022100859bf5813cc0b5ede7b96d6d4fe2354d6351357648e63ffb7b675da3dae122f502203a9f9cfb3fcce3cab537336bf27e671b75f4ec25084a6967702c377ab6e19a8b01] [02e18f47b78730f788b726c46459fdc29b57fd6d437b69587a1e59efa343c46a3b]" | bx validate-tx
stack false
Was also getting stack-false with a prior non-Standard CSV transaction.
vdeurzen15:52
I'm trying to calculate the proper fee for a transaction with a bunch of P2PKH inputs and a single P2PHK output. If I calculate the size of the transaction (after adding all inputs, outputs and endorsements) and then modify the output to subtract the fee, the transaction fails to organize. If I calculate the size before adding the endorsements I get a very small total size, which I expect is not correct. Am I going about this the right way? Is there a more straightforward way to do this (given I know the byte_fee_satoshis is correctly configured).
vdeurzen16:59
Stack Overflow posts hint that I could possibly use a formula
(148 * #inputs + 34 * #outputs + 10) * byte_fee_satoshis, if I try that, however, the organizer complains my fee is insufficient -.-evoskuil18:49
Stack false doesn’t relate to standardness, and bs doesn’t have policy apart from min fee.
Size is the total of all bytes in the wire serialization.
skaht19:00
Is libbitcoin BIP 68 nSequence and BIP112 CSV functionality considered production ready?
The reason I asked is because the process of spending a non-standard CSV-P2PKH payment and a standard P2SH( CSV-P2PK ) payment ultimately reach a stack false. I’m very comfortable composing P2PKH and P2SH transaction using bx.
1 reply · skaht · last Jan 04, 2019
skaht19:12
Forgot to put the CSV locktime in little endian format. I’m sure I was not the 1st to make this mistake and won’t be the last.
skaht19:07
I guess I need to check with Neill to see if https://github.com/libbitcoin/libbitcoin-server/wiki/Fork-Settings for BIP 68 and 112 are set to true for tcp://testnet2.libbitcoin.net:19091.
(edited)skaht20:57
Thanks for checking:-) Will be briefing a bunch of slides this evening at the ATL Bitcoin Embassy about how bx can be used to construct transactions.
👍 1
Friday, January 4, 2019
evoskuil10:24
You are welcome to try it out but both header and block sync are still missing important optimizations, so it’s still not nearly as fast as it will be.
any hints on how I could best get the fee done, short of building the transaction twice, once to calculate the fee once with a correct fee 🙂
vdeurzen10:39
I want to ensure the fee will (given a correct fee-per-byte) be mined in the next 10/20 minutes, it seems from my inspection that it makes most sense to work with the byte size?
I've used that now, but organizing still returns 'insufficient fee'
evoskuil11:05
The formula depends on the tx structure and rejection depends on configured fee policy.
(edited)vdeurzen11:49
The TX structure is a selection of both segwit and non-segwit P2PKH inputs probably mostly non-segwit, the output I can decide, but probably standard P2PKH. Where do I find the fee policy code?
6 replies · James C., vdeurzen · last Jan 04, 2019
James C.16:26
Hi @vdeurzen, maybe this notebook example might help: https://github.com/teachbitcoin/code-demos/blob/master/03_transactions_introduction/00_transaction_build_bx_example.ipynb
James C.16:31
There is both a byte (1mb) and sigops (20k) limit to blocks, so the fee should factor in both.
(edited)vdeurzen16:09
So, following the wiki on how to create a transaction, you construct an output_point from a tx_hash and the position of the output you want to spend, however, this leaves the output_point.validation.cache with a default constructed
output. Which will always present as 'missing' when calculating the total_input_value() for a transaction, which means the fees() of the transaction are always 0 and never enough? But I don't seen any other way of getting a non-missing output_point ... this is becoming rather frustrating 😕evoskuil17:19
You cannot spend an output without knowledge of that output. The output's position is sufficient to obtain the output's value from a server, simply query the server for the tx of the output.
Fee policy is configured in the bs config file.
There is no place in code to find the network's policy - each node can set whatever policy it wants.
vdeurzen19:39
With the correct fee (given from @James C.) the transaction now organizes. It's rather surprising, to a developer using the library, that calling fees() on a tx he's constructed always returns 0
vdeurzen19:47
after organizing the TX the
fees() call correctly returns a value, given my current search through the code I can understand the behaviour though. I wonder if an extra method here could be valuable, something on a TX to check the current different between inputs and outputs that have been set (given there does not seem to be any way to get a valid output_point in a TX from the outside)neillm19:59
@vdeurzen The fees method isn't returning any property of the tx like you're expecting. I'm not quite following what you're doing, but I don't see how you can create a tx without setting all inputs (which have values attached to them already, since they are previous outputs) and all outputs (which you set the value on). Any mismatch from your construction using those inputs and your outputs is the fee.
vdeurzen20:02
Well, I'm following along with the wiki, and for each input I need I set the previous_output by supplying a tx hash and the output index. However, constructing the previous output that way creates a default validation struct for the output_point. The default validation struct uses a default constructed output, which is invalid. This leads the unorganized, newly created tx to have
fees() return 0 as all inputs are technically 'missing' due to them referencing a default constructed previous_output.validation struct.I can still organize the TX if the fee is large enough, so my main problem is solved. However, checking the fee before doing so won't work. I can, however, check after the organization by running
chain.fetch_transaction(...) and calling fees() on the tx pointer passed to the callbackTLDR:
transaction tx
...
input in0;
in0.set_previous_output({old_tx_hash, 1});
...
tx.set_inputs(in0);
tx.set_outputs(....);
tx.fees() // this will return 0 (edited)neillm20:06
There's no way for it not to return 0 there, how would it know what the fee is given the inputs?
neillm20:07
Yes, but there are no values that are specified in your inputs. They have to be queried (online, or rather, on the blockchain) to retrieve spendable values
(edited)You'd think that would be done in order to properly select the inputs (i.e. ensure the output amount can be met by the input set), no?
neillm20:09
If you want to query for utxos, look into the fetch-utxo command, which contains some supported ways to do it (though there are many ways)
Select the inputs that meet the criteria for your tx, then use those inputs and specify the output amounts. At that point, you have all required values to know the fee
vdeurzen20:12
My confusion comes from following what happens when you do chain->organize(my_new_tx); which previously returned
insufficient fee when I ran it. When going through that code it calls if (tx->fees() < price(tx)) ... in transaction_organizer::handle_accept. This led me to believe I could check the fees on the tx by calling the fees() method.The reason the fee was insufficient is simple, I had not included the sigop cost
Regarding that, when I look online at the current byte_fee I can a clear number, where does the 100 satoshi per sigop come from and how do I know if that is enough at this point?
neillm20:14
There is nothing that can always tell you if a statically configured value is enough at any point
As @evoskuil mentioned, the network determines what's enough (i.e. not all participants accept the same things)
vdeurzen20:17
I understand. I was just wondering if there are similar graphs for satoshis/sigop as there are for satoshis/byte (https://bitcoinfees.info/)
we'll see how it goes on testnet and then test with smaller amount on mainnet 😄
thank you for pointing my sleep deprived self in the correct direction 😄
neillm20:28
No problem, good luck. Fee estimation at any given time isn't an exact science and takes some experimentation, unless you just always want to overshoot it.
evoskuil20:48
There is no objective minimum fee, and there is no way to know the value spent by a tx (including the fee value) without knowing the value of the previous output(s), which are not part of the tx.
evoskuil20:49
The fee() method computes over tx metadata (specifically prevouts), which must be populated, otherwise simply returns zero as a default.
(edited)evoskuil20:53
These are bitcoin behaviors not unique to libbitcoin. Documenting such things is well beyond the scope of our wiki. Even methods (such as fee()) are not documented. An objective of libbitcoin is self-documenting code. It’s relatively straightforward to trace the behavior of the fee method.
the code is very readable, very happy with that, but the callback structure hides the fact that the TX you organize is not the exact same as the one receive at
(edited)handle_accept, that one will have its metadata set 🙂evoskuil20:58
Organization is not where I would be looking for interpretation of tx.fee(). The txs are the same, but metadata population is part of organization. Blockchain code is complex and certainly not the place to learn about how fees are computed on a tx.
2 replies · vdeurzen · last Jan 05, 2019
vdeurzen13:25
I ended up here, because I was debugging in gdb. It was after calling
organize that I received a message stating my fee was insufficient (while I myself thought I had enough, based on the info I had).evoskuil22:10
Libbitcoin isolates nearly all core Bitcoin understanding to the primitives. The sole exception (which is found only in libbitcoin-blockchain) pertains to computation of cumulative work and therefore the strongest chain. IOW core concepts are can be learned with no need to understand a database or blockchain organizer.
P2P networking is isolated to libbitcoin-network (and libbitcoin-node for blocks and txs).
Saturday, January 5, 2019
vdeurzen13:26
I quite like the isolation and separation of concerns, creates a nicely composable system. I also like knowing about all the details and went a little but far down the rabbit hole and took a wrong turn which led to my confusion
👍 1
Sunday, January 6, 2019
T
Tralfaz17:48
Hi All, Is there a preferred place other than here for some tech support re: installing libbitcoin?
T
Tralfaz18:21
Ok, cool.....I've installed libbitcoin successfully on a linux system in the past, but for some reason I'm having issues on this ubuntu 18 setup. I run the ./install with the flags as described in the instructions, and with my correct path...all seems to be going fine until I get this series of statements: CC src/src_libzmq_la-tweetnacl.lo
CXXLD src/libzmq.la
ar:
u' modifier ignored since D' is the default (see `U')
CXX tools/curve_keygen.o
CXXLD tools/curve_keygen
/usr/bin/ld: src/.libs/libzmq.a(src_libzmq_la-tweetnacl.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
Makefile:3368: recipe for target 'tools/curve_keygen' failed
make[1]: *** [tools/curve_keygen] Error 1
Makefile:4881: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
real 15m50.804s
user 5m31.317s
sys 0m58.737sThere was some other thing about not having IPC, if I recall
I'm fluent in linux cli and python, but installing G++ and this package is presenting me with a lot of info that I don't recognize
neillm19:00
Ok, I suspected it might be. The only time I've run into issues like that is when you have both clang and g++ installed. What happens is that some dependencies get built using clang and some with g++ and it ends up being incompatible. I'm not entirely sure why that happens (particularly recently), but to eliminate that as an issue, re-run the installer, but prefix it with something like
(edited)CC=gcc CXX=g++ ./install.sh --prefix= ...T
Tralfaz19:02
ok, thanks for your time and thoughts....I'll give it a try and report back. I am able to use the bx-linux-x64-qrcode single-file, but I prefer the elegance of a proper install
T
Tralfaz19:24
@neillm no luck. Errors: /usr/bin/ld: src/.libs/libzmq.a(src_libzmq_la-tweetnacl.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
Makefile:3368: recipe for target 'tools/curve_keygen' failed
make[1]: *** [tools/curve_keygen] Error 1
Makefile:4881: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
vdeurzen19:31
Recent versions of GCC have changed some behaviour regarding PIE, might need to recompile all parts 😕
pmienk19:31
Can you provide the parameters provided to install.sh and any relevant environmental settings? Alternately, have you tried -fPIC as it suggests?
T
Tralfaz19:34
here's what I just ran: sudo CC=gcc CXX=g++ ./install.sh --prefix=/home/user/.local/bin --build-boost --build-zmq --disable-shared
sorry to say, I'm not sure how to try the -fPIC
neillm19:36
@Tralfaz Try the same line again, but remove sudo and use a different prefix (or sudo rm -rf ~/.local/bin)
pmienk19:37
@neillm is right, try isolating properly without superuser first. Probably some shared libs getting pulled in.
Monday, January 7, 2019
vdeurzen13:38
I'm somehow able to create two transactions, one with all inputs and a second one with only every second input, and then organize them both successfully, even though it is an explicit double spend? Both have a single output to the same address, but each with a different amount and fee
walking through the organizer flow, I don't see any check for double spends?
the broadcast transaction is, correctly, rejected by bitcoind though
jungly13:42
@vdeurzen I probably don't have an answer for you, but did you check out libbitcoin-system's chain::transaction::accept method?
i think that is the one used to validate incoming transactions
neillm13:45
@vdeurzen There's no such thing as a double spend unless you have a (confirmed) spend. A node can reject 'conflicting' txs on sight, but doesn't have to
so, it should fail if there is atleast 1 confirmation then?
I'm just exploring, btw, I intend to do the double spend check if it's not there yet
and of course it could simply be a transaction replacement action 🤔
vdeurzen13:54
effectively my context is:
1. we will receive btc on a selection of addresses
2. after x confirmations these outputs will be spent and transferred to a designated wallet
if somehow the process dies, is killed or some other issue occurs, I want to add a boot phase that simply replays the last Y blocks in the database and spends the received crypto.
So, the issue I'm trying to understand is whether or not it will be an issue if that boot-phase creates txs spending already spent outputs 🤔 I don't think, conceptually, it should be an issue, if the output is spent a duplicate transaction should be rejected by most other nodes and for me it doesn't matter because in the best case I've spent outputs that were unspent still and in the worst case I've created a few invalid txs ...
make sense?
step 2 is happening automatically, i.e. that's what I'm working on now
neillm13:56
The workflow doesn't make sense to me, but broadcasting invalid txs won't cause any harm since they'll be rejected as noted
neillm14:06
I'm not familiar with your database, and what a block replay after crash means. But you can do what you want with bitcoin, so I'm commenting on that.
vdeurzen14:14
well, maybe I should explain it a bit better. I have a list of addresses where I will receive btc, too many addresses to manage by hand, so what I want to do is automatically transfer any btc received to a single address. If the application is incorrectly configured, crashes or for any reason doesn't correctly track the received transactions, I want to have a fallback solution that goes through the libbitcoin block database and effectively re-transfers any received btc, but I cannot know for sure that I haven't spent those outputs yet (unless I start tracking that in yet another database).
neillm14:17
But you could also keep track of an HD index or something and in the 'worst' case, re-scan from scratch
HD index is an existing concept? How would that work exactly?
neillm14:19
I just mean you could have your addresses derived from a BIP32 derivation (such as BIP44) and with a single seed, you can check all address balances as needed. You could cache what index was last processed to avoid doing all the work again, but if that's lost, just re-scan/re-start
vdeurzen14:20
that is effectively what I'm doing actually, I'm currently looking at the re-scanning part, but I could have a look at getting address balances for the re-scanning
vdeurzen17:36
@neillm is your suggestion to utilize the history database? If so, when I get a
history_compact::list, how do I get the actual outputs? Am I overlooking some method?On a general note, I'm wondering why methods like
database::spends() and database::history() just dereference the pointer without checking if the database is initialized? When calling fetch_history on a block_chain object obtained from a full_node this will cause a segmentation fault 0.oneillm17:40
vdeurzen17:45
Ok, I'm using the node directly and using history requires some hacking, given that the node hardcodes turning off the history database 😕
So using the client API, you can query it and end up with the history list: https://github.com/libbitcoin/libbitcoin-client/blob/master/include/bitcoin/client/history.hpp
vdeurzen17:48
I'll have a look at that, but for the time being that's probably not the best option in my case 🤔 need to think about it
thx in any case 😄
The organizer populates the previous outputs and invokes accept().
2 replies · vdeurzen, evoskuil · last Jan 08, 2019
vdeurzen10:03
v3, but I realized it's not technically a double spend, it's just two unconfirmed transactions using the same outputs as input. Which is valid I believe. I was thinking, somehow, that maybe libbitcoin would track spending of an output as soon as it is added to a transaction, but that doesn't make too much sense in hindsight
evoskuil22:50
This is certainly useful information, but can be expensive to index. We will eventually track this from startup in order to support fee increments for RBF.
evoskuil23:36
Figured out why tx pool validation was failing. Formerly chain_state(chain_state) was the signature for constructing a promotion to pool state. Obviously that's also the signature for default copy construction. So when an additional system::settings parameter was added to that promotion signature, the existing callers became copy constructions, resulting in a failure to promote top block state to pool state... instead remaining one block lower.
👍 3
Wednesday, January 9, 2019
2 replies · jungly, evoskuil · last Jan 09, 2019
jungly09:17
Super awesome new comments on metadata fields 🙂 I feel like I understand a wee bit more now 🙂
vdeurzen14:53
I'm playing around with mnemonics and I am running into something I'm not quite sure about. I generate a seed, create a mnemonic, then decode the mnemonic into the seed again, but the decoded seed is different. Starting with a 32 byte seed, the decoded mnemonic gives me a long_hash, 64 bytes. Is that expected?
neillm14:54
The entropy used to generate the mnemonic is not the seed, nor is it a goal to recreate it
vdeurzen14:54
Right, the confusion comes from the naming in this blogpost I guess: http://aaronjaramillo.org/libbitcoin-hd-keys
The Web Log of Aaron Jaramillo
Modern Wallets use Hierarchical Deterministic(HD) Keys rather than simple keys in order to allow a user to use a different bitcoin address with every transaction. Today, We’re going to program an H…
neillm14:56
You can see in that example, the seedChunk is pseudo-random entropy. So yeah, it's named incorrectly.
An HD seed is not (pseudo)-random
thx!
neillm17:29
It should query both. This is related to the recent transaction_pool update @evoskuil mentioned in v4 though
(edited)skaht18:09
If the is no means to force a fetch-tx to only query only the blockchain, is there a means for using bx to ascertain if a locking particular UTXO’s script was successfully spent after having satisfied the the following: input-validate, validate-input, send-tx?
A watch-address won’t work for a script that does not have a standard address, and a watch-tx is not yet implemented.
(edited)skaht18:30
Parent TX (successfully mined):
% bx fetch-tx 4c52f6c486b7e3372d0dc900aa041fcb0c390c89539a4af1ee9b55e9b78fd759
Child TX (appears to be in mempool) that is trying to spend UTXO index 1 (CSV-P2PKH) from parent appears to hang:
% bx fetch-tx 7fba79ad9511a0f27863b712d186c297843cb182569db0aded5d34ff1ca28809
Locktime associated with CSV had to be little-endian encoded. The nSequence of the transmitted child was not little-endian encoded.Friday, January 11, 2019
jungly19:37
Hey guys. I didn't realise libbitcoin is showing up at advancing bitcoin in London!
Just came across some meetups where @evoskuil and @James C. are going.
Would be awesome to catch up in person.
👍 3
jungly23:51
@evoskuil - There is no issue on github about completing the implementation of the address_database#catalog method. Are we aiming to have that working for the v4 milestone?
I was kind of looking at having a go at it as I find some time next week. Will that be useful?
But the index in v4 will be different than v3. The key is a sha256 of the output script.
Otherwise it’s the same.
This means inputs are indexed strictly by their prevouts (eliminating ambiguities) and that we no longer use script->address extraction for indexing.
(edited)👍 1
Saturday, January 12, 2019
jungly00:03
I didn’t realise v3 was doing address indexing.. thanks for the tip, will go take a look at how that works first
skaht00:10
@evoskuil - Will payment indexing mitigate the need to implement https://github.com/libbitcoin/libbitcoin-explorer/wiki/bx-watch-tx?
Wednesday, January 16, 2019
I just merged the first iteration of read-ahead caching for block confirmation during initial/catch-up sync. There is a thread starvation bug that eventually deadlocks the process. However it is clear that block confirmation performance is significantly improved by pre-reading blocks in parallel.
👍 2
I'll be traveling for the next three weeks, so won't be able to complete the work, but it's very nice to know that the design is performing as expected.
👍 1
jungly17:46
Hi @evoskuil, I know you are travelling soon. But just a small question about payment_record and how we want to use it when cataloging payments. There's a couple of conflicting comments that are confusing me a bit.
Is the following the list of things we want to store as payment records for both inputs and outputs keyed by the relevant script hashes?
1. input or output flag
2. link (i.e. the offset into the transaction database for the tx that is the source of the input or the output)
3. Index into the inputs or outputs for the transaction
4. Point checksum
With this change, we don't store the point hash or the height where the transaction is found. Replacing both of them with the direct link to the transaction record in the transaction database.
Hope I am not miles off target with my understanding.
1 reply · jungly · last Jan 17, 2019
Monday, January 21, 2019
vdeurzen11:54
@here When I have a bitcoin address, specifically a script address (e.g. starting with 3) and I want to transfer money there I should use
to_pay_script_hash_pattern() ?neillm12:45
@vdeurzen That sounds right when creating an output from an amount and a script (that takes an operation::list, given the address hash)
See payment_address::hash()
So you can do something like
output destination1(amount, script{ to_pay_script_hash_pattern(ADDR.hash())});Wednesday, January 23, 2019
neillm14:07
@evoskuil @jungly Looks like we have a break here: https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/interface/block_chain.cpp#L1536
I've loosely been following how we're changing that index, but don't know where it stands at the moment
jungly14:11
Hi @neillm, thanks for spotting this. I didn't change the get method, but what is saved in the db is slightly changed, and thus the address_result is slightly different now. I'll run the tests for blockchain and see how we can streamline this.
aha.. i see.. the problem is fetch_history is querying the payments database (called address_database) with a short_hash, i.e. an address
jungly14:16
yup, so the address db really doesn't index by addresses anymore. It expects a sha256 hash of a script instead now
we could change the fetch_history to take in a chain::script? or a hash_digest that captures the hash of the chain::script instead?
I'll have to look at how and where fetch_history is called from, but still I might not be the best one to make a call on this. My gut says we could change fetch_history to accept a script and we do the hashing in fetch_history instead.
neillm14:20
Ok, we need to figure out how this affects fetch_history4 (already a break from v3, so I don't think we need a version bump on the API).
Well, it's called almost directly from the client, usually just given a payment address
Yes, in this case a payment_address
Is the intent to use the context free script extraction from payment addresses as the key?
jungly14:26
I think it was the other way around, we didn't want to extract address from inputs and outputs and instead simply index payments by the script itself
I could be completely off the mark here
neillm14:27
Right, I just mean is the intent to improve and use the script from payment_address::extract_input as the index? Or are we requiring a user provided script instead of address (will hurt usability I think)
I should have paid more attention to what @evoskuil said on this a while back 😉
Seems fragile, I'm missing something (likely simple) on the intent here
jungly14:31
yes. exactly, i think that might be one of the reasons eric might have wanted to get rid the addresses extraction bit
To resolve the immediate breakage, I could locally update fetch_history4 to read out an opaque hash_digest and use that to key in the db. I'd have to fudge the upstream for now, effectively disabling the usage.
jungly14:36
pass in address, we query all sorts of scripts it could have been part of, but that sounds completely nuts i know.
neillm14:38
Yeah, either I don't understand that, or it sounds nuts. We need the exact script, which could be anything given some patterns, if it's hashed as the key.
jungly14:41
Yes. I know. I was just trying to be too creative there. We do need the exact script.
So from bx, we want users to paste in the script to fetch history? Is that the envisioned ux?
jungly14:49
so the bx interface will need some work to use the script hash approach to index payments is the answer?
(edited)neillm14:50
If I understand correctly, hash of script would really limit what can be queried though. Parameters for standard patterns allows some more flexibility, but is still guesstimated/not exact. It should be consistent though.
jungly14:58
My understanding was the extracting the address from a script required some pattern matching attempts, but if we let the user say they want to query pattern X, with A, B, C params then we should get the exact script, no? Again, I could be missing something here.
evoskuil15:13
A hash of the script is “exact”. Pattern matching just results in bx generating the script and the hash from it. A full script would just be hashed. Backward compatibility is easily achieved by constructing the script and hashing it.
neillm15:28
Given the script, agreed, but that implies you know it. My concern is that if you want to query a balance that isn't yours, you provide a pattern guess to bx and hope the hash from the generated script is a match because there's no way to distinguish some types. So it might just appear that there are large holes in our db, or we do silly things like generate a few possible scripts and index them all (obviously not going to be done).
jungly15:31
Could it be the other way around? That is, generate a few possible scripts and query for them all instead of storing them all? So read time script generation from patterns, not write time.
neillm15:33
Yes, it's possible, but I'm not satisfied that we need this kind of guessing now, and I still think it's fragile for future patterns that may intentionally obscure tx pattern types.
I almost wonder if an addr hash to script hash table makes sense (at least for historical), but that still involves guesses going forward. I'm still missing something because it's not making sense as a better way to index. If this is electrum guided, I'll have to take a look there.
evoskuil15:43
You are missing something. There’s no guessing, just different commands to obtain the script hash.
neillm15:45
A hypothetical bx fetch-balance syntax would help a lot I think (preferably for something other than p2pkh)
where the key is the hash of the output script.
(edited)neillm15:47
Understood, but how is the key determined if you don't know the script and only know the address?
neillm15:55
Ok, derp, I didn't realize new addr formats contained the script hash the same way. I also haven't looked much at bech32 addrs, but assumed it would be impacted there as well
:)
neillm16:04
So the only real client change is that instead of using addr::hash() as the index, we use the hash of it?
So we could optionally take the user provided script and pass that, or just generate the hash of addr script and pass that (backwards compatible).
Since we have a build break now, I'll starting updating this
👍🏾 1
jungly16:12
Thanks @neillm. I should have realised this break earlier and pinged you about it. Sorry for the surprise.
I think the terms
script and hash are heavily overloaded in bitcoin btw. I still should have realized what we were talking about sooner though ,,,evoskuil16:48
We can add an additional command to generate a script from an address. We already have a command to serialize a script and to hash in accordance with keying requirements (double sha256).
counterpart
Thursday, January 24, 2019
Revive20:52
whats the best course of action for me if my system upgraded to boost 1.69 and am using libbitcoin 3.5.0
thanks
Friday, January 25, 2019
vdeurzen10:26
I have pretty good results with libbitcoin + boost 1.68, wouldn't expect 1.69 to be an issue?
4 replies · Revive, vdeurzen · last Jan 30, 2019
Revive20:36
it appears that Boost 1.69 is an issue for me when i try to compile libbitcoin-protocol with the error
Revive20:36
/usr/local/include/boost/multiprecision/cpp_int.hpp:181:4: error: right operand of shift expression ‘(1 << 63)’ is >= than the precision of the left operand [-fpermissive]
BOOST_STATIC_CONSTANT(limb_type, sign_bit_mask = 1u << (limb_bits - 1));
vdeurzen10:48
I ran into this I believe, might be a compile flag that triggers it, see https://github.com/boostorg/multiprecision/commit/f9c8f9ec091ad232c0a291904f7839d665d098e0 for a patch that supposedly fixes it
B
baiumbg12:36
Is there a reason
libbitcoin-server stops publishing transactions/blocks when there are a high number of active client connections?Anything I need to configure?
I’m currently load testing and when I’m initiating 300 query service connections. During the time that those connections are active, no blocks/transactions are published through the tx/block services.
When I do 200, everything seems fine.
B
baiumbg12:44
I’ve noticed that sometimes messages get sent in batches, but I don’t think that’s the issue I’m looking at.
There aren’t any “Published public transaction/block” messages in the logs at all.
B
baiumbg12:52
When I said 3.4, I didn’t mean the tagged version. That’s the output from
--version 😄 I rebuilt the executable from the v3 branch in november, so I have those changes for sure.neillm12:55
I'll have to re-test that then. I'm currently running 300 clients on master and subscriptions are fine, but I'll get a version3 test going soon
👍 1
neillm13:06
Also, how are you subscribing? Right now, I'm just using subscribe-tx via bx on mainnet.libbitcoin.net (so I can visually see) and launching a few hundred queries at it and don't see anything stopped
And last question there -- the server has a time limit on subscriptions before you need to resubscribe. While doing this kind of testing, make sure you're not failing to re-subscribe if needed.
B
baiumbg13:13
I’m using pylibbitcoin to send basic queries like
blockchain.fetch_last_height, blockchain.fetch_history3, transaction_pool.broadcast, etc. I’m not using subscriptions currently.On the other hand, I have a single process that subscribes to the tx/block publishing services.
neillm13:15
I'm not sure I'm following. So you are subscribing and you say it's working for some amount of time and your separate query processes are running and that subscription stops?
B
baiumbg13:17
Yep. Whenever I launch the 300 query processes, my tx/block listener stops receiving blocks/transactions. Looking at libbitcoin-server’s logs, it seems the server isn’t publishing at all (at least that’s what I gather from the lack of “Published public transaction/block” logs)
The logs are there before I start bombarding it with queries.
N
narodnik13:20
https://github.com/libbitcoin/pylibbitcoin/blob/master/pylibbitcoin/client.py#L550 <- I'm not sure that the code is sending the keep alive message for the subscription
B
baiumbg13:22
I only use pylibbitcoin for querying. The process that subscribes to the block/tx services is in C++ and I haven’t experienced any subscription drops before trying to load test.
(edited)N
narodnik13:24
oh right it doesn't need a keep alive. if enabled, then it should be publishing on that socket through zmq
👍 1
if you know zmq, just open the socket to the correct port and see if data is being published
evoskuil13:26
If the client doesn’t resubscribe it will be dropped at the timeout. There is no keepalive, just a subscription. Zeromq sockets now implement a keepalive of their own.
N
narodnik13:26
import zmq
context = zmq.Context()
socket = context.socket("tcp://ipaddress:port")
socket.setsockopt_string(zmq.SUBSCRIBE, '')
frame = socket.recv_multipart()
print(frame)
@evoskuil hi... I made the intro with you to binance guys on signal
👍 1
B
baiumbg13:28
Everything is fine on the client side - the server stops publishing whenever I start my load test.
neillm13:29
And if it's on regtest, you're sure that the txs and blocks are still being generated/processed?
B
baiumbg13:29
Does the server stop publishing if there are no clients connected to the block/tx service?
Yep.
B
baiumbg13:30
Like I said at the beginning - everything runs fine when I do 200 query connections. It’s when I up them to 300 that something breaks.
neillm13:30
It still processes them like they are to be published, but with no one listening, zmq does not send anything to anyone
neillm13:33
I somewhat regularly test hundreds and thousands of connections. It's been a while since I've tested v3 properly, but see no indication that it would stop after doing a quick hacked together test earlier (300 clients) while subscribing. I think I'd have to have a test setup more similar to yours to have a better idea of what's going on. There's pretty much nothing other than shutdown that would stop the publication on the subscribe tx/block services
neillm13:36
So to confirm, the regtest miner and tx generation still runs during this test? Are they being processed by the server during that time (i.e. discovered, validated, etc)?
They get forwarded to other nodes in the network at least.
12:30:01.804644 VERBOSE [server] Query transaction_pool.broadcast from [00e43c9a98] 12:30:01.805252 VERBOSE [network] Sent inv to [127.0.0.1:28333] (61 bytes) 12:30:01.836837 VERBOSE [network] Received getdata from [127.0.0.1:28333] (37 bytes) 12:30:01.836938 VERBOSE [network] Sent tx to [127.0.0.1:28333] (396 bytes)
Blocks are getting connected as well:
12:30:30.276162 VERBOSE [network] Received headers from [127.0.0.1:28333] (82 bytes) 12:30:30.276299 VERBOSE [network] Sent getdata to [127.0.0.1:28333] (61 bytes) 12:30:30.288756 DEBUG [node] Connected block [09d703528c7975e78f2c4c26e369b9c90180f176339f5075b76879a9b53a86f3] at height [6575] from [127.0.0.1:28333] (16374, 4).
Nothing published, though (except heartbeats 😄 ).
(edited)neillm13:51
@baiumbg Ok, I'll keep this in mind while testing. If you find out more details about what's going on, please let me know.
👍 1
evoskuil14:46
I was referring to address subscription only. Pub-sub sockets use only the zmq primitives.
neillm14:48
It wasn't clear what the setup was at first, but it seems it's zmq subscribe services only in this case, so standard zmq keepalive can be used and there's no need to resubscribe.
👍 1
Ty17:48
@baiumbg It sounds like you might have run out of file descriptors. Try setting
ulimit -n 4096. My v3 regtest server drops responses when there are more than ~200 concurrent requests, until I increase my fd limit from the default 256.B
baiumbg17:51
That crossed my mind, but the system I'm running the server on has a default value of 1024, so I thought that would be enough.
J
JChen18:28
Hello! I have a few of my altcoin nodes running on non-standard ports (7333 for P2P and 7332 for JSON-RPC), I'm trying to connect and extract the block height, but the connection keeps timing out 😕
(edited)
Any thoughts?
neillm18:31
Try the connect that just takes the endpoint instead of all the settings you're not using
Also, by altcoin nodes, do you mean altcoins based on bitcoind or litecoind? If so, client is not what you want to use to connect
J
JChen18:33
Oh god! The altcoin is based on litecoind - my understanding was that I could use the libbitcoin-client like an API to query the altcoin's blockchain
neillm18:35
libbitcoin-client is used to communicate with libbitcoin-server (and does not use the p2p protocol). It's designed to be high-performance, but you do need a server that has already indexed the blockchain. There was some work done to allow the server to work with litecoin and similar altcoins, so perhaps that's a route to try?
libbitcoin-node is the p2p full node, or you could write something using libbitcoin-network (there are docs on the wiki IIRC) if you want to go that route. There are no libbitcoin tools that communicate with bitcoind/litecoind via rpc, if that's what you're looking for though.
J
JChen18:46
Okay that's quite useful, I don't mind which way I achieve the communication to be honest. So I'm assuming libbitcoin-network has the capability to connect in a more customised way? From my reading I couldn't see other ways to connect to peer nodes, aside from libbitcoin-client, if you could link me to some documentation that would be great. I'm not really trying to achieve anything too complex either. This is just an experimental project. I'd like to mention that I have played around with the magic bytes etc of the original litecoin code! Though I don't think that will be an issue.
neillm18:47
I recommend starting here: https://github.com/libbitcoin/libbitcoin-network/wiki/Introduction-to-the-Libbitcoin-Network-Library
👍 1
1 reply · JChen · last Jan 28, 2019
J
JChen15:54
Hey! I'm trying to initialise a threadpool variable via:
threadpool myCoinThread(5, thread_priority::normal);
However, the compiler is complaining about the threadpool constructor being a deleted function:
error: use of deleted function ‘boost::shared_mutex::shared_mutex(const boost::shared_mutex&)’
class BC_API threadpool
error: use of deleted function ‘std::atomic<long unsigned int>::atomic(const std::atomic<long unsigned int>&)’
class BC_API threadpool
(edited)Ty21:17
@neillm I’ve been been playing around with the new web-socket interface and it’s been working really great!. I’ve built a block explorer using the new api, check it a demo here: http://regtest.bs-v4.utxo.to:9071/
👍 2
neillm21:21
Ah, nice. I've also built one for that branch. After I push that out, we can see about migrating over though, or maybe having an option of which or something 🙂
Ty21:23
Nice. I figured I'd let you know what I was up to. I've pushed to code up here if you want to check it out. https://github.com/lastcanal/libbitcoin-server/tree/webpage
neillm21:25
Nice work, and thanks for testing/using the interface. Obviously it needs some work, but it's refreshing to know that someone other than me tried it out! The more things built on it also can help highlight pain points or where the API needs work (post version4 release, I expect it'll be extended quite a bit).
might break the overlength-strings rule on some compilers.
neillm21:40
Shouldn't, I don't think. It might better to keep it as a char* too, only because I think at runtime the string(s) needs to be built correctly to honor the user settings/configuration.
Saturday, January 26, 2019
neillm02:45
Wolf McNally
Building on Libbitcoin Today I’m announcing the availability of the open source Bitcoin framework for iOS. I have been working with Blockchain Commons on wallet apps for iOS that will support airgapped transaction signing and BTCR Decentralized Identity (DID). As part of this effort we chose to build on the open source libbitcoin library. From
👍 1
Monday, January 28, 2019
neillm15:49
Should the address_database index be the bitcoin_hash of the serialized script, or the single sha256_hash? It looks like it's single sha256 at the moment. Also, I wonder if address_database needs a rename(?)
jungly15:51
I had the same question, should we use bitcoin_hash or a single sha256_hash? In the end, I think a double hash doesn't serve any purpose for this case. Unless I am missing something.
And yes, I'd like to rename address_database. Just holding back to get the fix the last PR and have it merged 🙂
neillm16:52
@jungly Based on local testing, payments are not indexed due to this check: https://github.com/libbitcoin/libbitcoin-database/blob/master/src/data_base.cpp#L292 Not suggesting that removing it is correct, but noting that I removed it for testing in order to get indexing going. Something is off there.
jungly16:56
Hey @neillm I ran a v4 server last week too and the payments weren't being indexed. I thought there was an issue with the catalog code, but didn't get a chance to look at why. In fact, I was thinking it was due to some other problem in the way I was using the multimap, and am in the middle of writing a test to confirm that hypothesis. I have a feeling I am going down the wrong track, but still continuing to see what I discover.
If we take away the line you linked to, will we have to provide another means to avoid indexing the same payment more than once?
neillm16:59
Ok, no problem. I haven't followed the implications of that check, just noted that without that check, it looks like indexing is working properly and I'm able to retrieve it as needed, etc. Decoding the format is not working yet (history decoding), but there's data there. That's as far as I've gotten on it
jungly16:59
if you see here: https://github.com/libbitcoin/libbitcoin-database/blob/master/src/databases/transaction_database.cpp#L310
There is clear use of the existed metadata to avoid duplicate indexing.
That's great to hear it worked for you without the flag. I think I could have also had a config issue or something. But I don't mind to try and spend time to figure more out .. in case there are some surprises lurking somewhere.
neillm17:01
That flag seems to check that it's written to the tx db, but not necessarily the address db
So maybe another flag is needed (i.e. exists in tx db, but not indexed (when indexing is enabled)). Haven't thought about enough, but seems straight-forward
jungly17:04
I think the reason for checking with transaction db, is because transactions can come back to be stored more than once. Say it once comes in through a candidate block A, then candidate block B, etc. That's what I thought was happening.
neillm17:06
Hrm, in that case it doesn't look like it would re-index the tx though (which is ok). This check is only for addr indexing, so it just looks like we're checking an incorrect state flag
If there's block metadata, we could skip reindex if it (the block) has already been stored though
Not sure, I haven't dug into database as much as I'd like. Would have to look into a bit more.
jungly17:17
i'll definitely check if the existed flag is being incorrectly used there, though it was already there in the data_base::catalog code, so i wonder if there is something in the blockchain code that needs to change instead.. tbh, don't know yet..
I'd say the payment indexing is still a WIP, given that i need to fix that PR and maybe discover new problems when running local tests 🙂
neillm17:48
@jungly The other thing I had to update is here: https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/interface/block_chain.cpp#L1538 (limit is 0, as called from server, so it needs to be changed to process the results retrieved)
(edited)