Bitcoin full node and query server
October 2019
Archive of the libbitcoin Slack #general channel.
Tuesday, October 1, 2019
skaht02:15
Saturday, October 5, 2019
neillm03:41
Just noticed mainnet1 is down. Not sure what happened, but I'll restore from snapshot and catch-up sync ... hopefully back up tomorrow. mainnet2 is still good.
Has it ever crashed without explanation?
neillm03:51
This was an oom kill, which is odd, since that machine doesn't usually show that (but I guess could under certain pressure). So it doesn't tell us anything about potential code errors (if any).
Unrelated, but I've been running a mainnet v4 server at 72.65.246.86 for some time, if testing is needed.
What’s oom?
1 reply · neillm · last Oct 05, 2019
neillm04:05
There are way too many acronyms in the computing space today 😉 I think this one stuck (for me) when there were some debates about the "oom killer" in Linux in particular (which is what apparently raised its head on this)
(edited)neillm03:58
No, I don't think so, given the long runs of many months at a time (usually only interrupted by power failures). Leaks tend to show up quicker.
The memory profile on that machine is often reached to a level and remains flat going forward
Wednesday, October 9, 2019
vdeurzen11:35
How stable is the current master branch at this time? I remember you were working on the parallel sync and more extensive configurability, are those still only on master or have they been ported to other tags / versions?
vdeurzen16:26
Are the CMake builds tested in CI? I believe I'm discovering some issues (could be changes in newer CMake versions, but the result is build failure)
vdeurzen16:28
Well, it's using some things that nowadays are considered anti-patterns (global include manipulation for example) and for libbitcoin-database the include for libbitcoin-system is using an empty variable, meaning it doesn't add any include path.
vdeurzen16:30
eg.
target_include_directories( ${CANONICAL_LIB_NAME} PRIVATE
"../../include"
${bitcoin_system_INCLUDE_DIRS} )
here
${bitcoin_system_INCLUDE_DIRS} is empty for me and that results in cmake finding the wrong library (the system installed version, not the one I want to use, i.e. version 4). Meaning I get compile errors because there is now a version mismatchThere is a chance something is incorrect with the globally installed version (i.e. it's not actually version 4 so cmake should reject it, but that isn't happening)
For context, I'm creating a conan package for all libraries
pmienk16:34
It's not in front of me at the moment, but isn't the dependency specific include for populated by it's pkgconfig retrieval?
vdeurzen16:36
It's using
find_package, and whether or not that creates that variable depends on what is happening internallypmienk16:37
all find_package mechanisms for libbitcoin are located in modules and are pkg-config wrappers
vdeurzen16:38
The custom find module that is included as a CMake module does not create the variables that are used
and as far as I know, it neither would pkgconfig within CMake
(edited)Current practice would be to add
Bitcoin-System::Bitcoin-System to target_link_libraries or target_include_directories, meaning CMake knows what to doif I do that, my build finds the right version
pmienk16:40
Which means it's relying on the pkg_check_modules to do it. As you are dealing with multiple installs have you properly restricted pkg-config so as to locate your preferred installation first?
vdeurzen16:40
I am dealing with multiple installs as I'm wrapping each library in a package (conan is a package manager)
It should also use the right env variables to point pkg_check_modules to the pkgconfig files that are relevant for my package (i.e. not the globally installed libraries)
pmienk16:41
(that doesn't tell me why the variable is empty instead of pointing to the correct location, but of it was correct then the problem wouldn't exist)
I wish pkg-config didn't use its own env variables for search as well. However the cmakefile.txt doesn't appear to know that. Like automake, it's an external concept.
vdeurzen16:46
I'm going to leave it be and just hack around the autotools version, I had that working but this is a bit of a time sink I'm afraid.
I'm guessing CI uses autotools for its build?
pmienk16:47
Looks like all you would need is to specify pkg_config_path to start with your prefix path.
vdeurzen16:48
There is a chance it's related to how Conan deals with pkg_config usage from CMake, so I've tried to get things working by hand, but results are odd, hence my questions here
pmienk16:49
If that's so, then pkg_check_modules behavior has changed or is broken in your configuration, which would be pretty significant
vdeurzen16:50
The confusing thing is that CMake tells me it has found Bitcoin-System 4.0.0 and happily continues, but it has not added any include path (inspecting the compile_commands file or verbose build output). Which leads to the compile looking in default locations (e.g. /usr/local/include) and finding the system installed version
pmienk16:55
Yeah, that would indicate pkg_check_modules handed us back an appropriate version without populating it's other variables. If there's been a version release with break in behavior and those variables no longer get set that could do it too.
vdeurzen17:25
It simply seems that everything combined, the library being named Bitcoin-System, the pc file being named libbitcoin-system and the custom find module are conflicting. Probably the CMake files are OK, but they are not 'package manager friendly'
Friday, October 11, 2019
vdeurzen21:19
My point yesterday was that for package manager, the way the Libbitcoin CMake files are designed makes package management complicated. So, from a build perspective, yes CMake will build successfully, from a packaging perspective it uses some constructs (such as global include and link path manipulation) that make life rather complicated. My issue specifically is that my package management solution is expecting a package named 'Bitcoin-System', but the CMake module is defining one named 'libbitcoin_system'. I can work around it to get CMake working, however, I have most of this working using autotools (which required its own set of hacks)
IOW would you be able to incorporate that generation into libbitcoin-build?
vdeurzen21:52
not really, it doesn't really generate an artifact for distribution. It's effectively a Python based recipe for running the libbitcoin build, and it understands how to manipulate that build to a certain extent (e.g. copying the resulting .so files to its own directories etc)
it's a recipe, not an artifact, it requires a fair bit of manual tweaking (at least that's the experience so far) 😕
but can't really start with generating, have to have something that can be generalized
vdeurzen22:11
well, at that point you could also just implement libbitcoin as a specification and generate any implementation you want 😜
I don't think generating this package description would be a smart idea, it already needs to work around shortcomings in either autotools or CMake (both the tool and libbitcoin's build files) and those you should not generate imho. Yes, a generated packaging description could work, but it's directly dependent on the other build systems, meaning any change needs both to change together, not doing so possibly leading to build failures and more questions 😜
pmienk22:16
that last part is actually a point for generation - the generator creates all artifacts for all repositories at the same time. manual modification is more likely to have changes made in one project but not the other
evoskuil22:17
Manual package generation is not sustainable. At least 4 times people have done this and not maintained it, and that's only for 1 or two of our repos.
I can't imagine packaging artifacts would be any more complex than the build artifacts we maintain for autotools, cmake and visual c++ (not to mention those for installation, continuous integration and test coverage).
vdeurzen22:36
I highly doubt it would be a successful venture, but we can always try, maybe you prove me wrong
On a side-note, generating the package recipe doesn't actually fix my problem at this time. We would need to change the CMake files for that (or stick with the autotools based package recipe)
vdeurzen22:44
I'll need to play around with them a bit more before I can give you a definitive answer for that. It's currently bedtime for me, so I'll have to get back to that next time I'm looking at it 👍
Tuesday, October 15, 2019
getting things like libbitcoin::system::chain::payment_record::unconfirmed': definition of dllimport static data member not allowed
keen to make cross platform build using only one build tool
pmienk20:29
cmake currently doesn't work on windows. the package location logic would need to be provided to make that work.
E
Edward Boggis-Rolfe21:19
CMake does work on windows, as I have several other packages compiling fine. Can you build it as a static library your way on windows rather than as a dll? If so I will have a go at fixing the CMake to build statically as a package. This is the clone of your branch that does package on linux using Hunter without icu. https://github.com/edwardbr/libbitcoin-system
pmienk21:21
I'm having a hard time parsing the claim. Are you saying that 'cmake can work on windows' or 'cmake does work on windows with our artifacts, but doesn't build statically'
I have no doubt packages can compile with cmake on windows.
pmienk21:22
I also know that there is parameterization available for cmake to restrict to static builds
E
Edward Boggis-Rolfe21:22
Are there options to compile your lib on windows as a static lib? Out of the box
(edited)not using cmake
I will need to find out how to switch off the dll export logic
pmienk21:25
the main issue with windows support for cmake that is untackled is logic for locating dependencies. note that all dependencies are located via the provided
Find<foo>.cmake modules. No effort has been placed to generate windows environment logic to fulfill this requirementE
Edward Boggis-Rolfe21:26
I have that sorted, I am using hunter to find my packages from the internet, download, build and link them
E
Edward Boggis-Rolfe21:27
Have a go at cloning my lib on a linux machine, it should bring down a copy of secp256k1 boost etc
pmienk21:27
if you'd like to feed back how that's accomplished, I'm sure we'd be happy to integrate it or generalize it for build generation
E
Edward Boggis-Rolfe21:28
the dependancies are declared here https://github.com/edwardbr/libbitcoin-system/blob/master/cmake/Hunter/config.cmake
The boost ICU option is not working, and I do not need it, however I had to make my own package of secp256k1 as there wasnt one on the hunter repo
check its inclusion in the main cmake: https://github.com/edwardbr/libbitcoin-system/blob/master/CMakeLists.txt
E
Edward Boggis-Rolfe21:33
yes you compile the source, it is slow to generate but you only need to do it once as you dont tend to muck around with 3rdparty code
your project sits lower down in the cmake file once the dependanices are downloaded and built
If you have many projects with the same toolchain you will find yourself not needing to rebuild that code
pmienk21:35
cool, looks like config.cmake in Hunter controls the build parameterization, i presume it passes on the appropriate installation location, etc. it depends on being able to invoke cmake in the root of the downloaded dependency?
and presumably that
Find<foo>.cmake is provided by the installed dependencyE
Edward Boggis-Rolfe21:37
It manages a dependancy tree of projects, to top level config.cmake overrides all the children, so you can have a later version of boost for etc than what you specified in your repo
All the find foo stuff is generated automatically
have a look at the bottom of the cmakelists.txt file you will see some install scripts that generate all that stuff
pmienk21:38
when you get it working and i get some bandwidth i'd be happy to incorporate it into our codegen
Checkout the other repos as I got down to libbitcoin-explorer
Wednesday, October 16, 2019
E
Edward Boggis-Rolfe16:19
OK I have made https://github.com/edwardbr/libbitcoin-system work
I had to change the cmake settings on my machine to be
Toolset: msvc_x64_x64
CMake generator: Visual Studio 16 2019 Win64 as ninga barfed on my machine
I like Punk IPA!
evoskuil21:49
Libbitcoin is not verified in VS builds beyond 2013 (with C++11 CTP), 2015 and 2017. What was the error produced by the cmake build?
Thursday, October 17, 2019
E
Edward Boggis-Rolfe07:56
Though I have not tested with earlier vs builds. I’m sure that they will work. You may have to tweak the windows api version option in the cmake file. The vs Ninja build errors looked nasty and did not investigate just switched to a different generator.
You may want to look at the secp256k1 library that I dragged in too. There was no package for that either so I made one. Both need to be in the hunter repo to be considered ‘official’
(edited)
BJ Miller12:56
small side tangent: it was great to hear @evoskuil on the “What Bitcoin Did” podcast. we use Libbitcoin in our app (DropBit) which is a sponsor of the show, so it was extra neat to hear. great show, Eric!
👍 2
1 reply · evoskuil · last Oct 21, 2019
Monday, October 21, 2019
skaht01:02
@evoskuil - concerning https://github.com/libbitcoin/libbitcoin-system/wiki/Hybrid-Mining-Fallacy
164 replies · skaht, evoskuil, neillm, Slackbot +1 · last Oct 21, 2019
skaht01:12
A consensus algorithm that uses a combination of tier 1 PoW mining like BTC when combined with staked tier 2 PoS full nodes (e.g. Dash Masternodes) with the application of chainlocks does add additional security.
It also vastly improves the security of zero confirmations over that of bitcoin-core. Dash rotates its chainlock quorum of 400 masternodes trustlessly from a pot of ~5K every 24 hours using BLS signatures. Dash rotates its InstantSend quorum of 30 masternodes trustlessly from a pot of ~5K every 24 hours using BLS signatures.
(edited)skaht01:19
https://github.com/tdickman/crypto51/issues/45 provides details for how to calculate the cost of a 51% attack for the Dash network’s Chainlock LLMQ, 60% of 400 masternodes.
evoskuil01:21
Random rotation doesn’t achieve anything if it’s a fair distribution (which it must be).
skaht01:23
Not with 2nd tier locking all network transactions associated with an InstantSend UTXO until the 1st tier completes the PoW confirmation for the given instantSend TX.
skaht01:24
BLS aggregate signatures are causal but results can’t be predicted in advanced and thus can’t be gamed.
evoskuil01:27
Just reading that one link there are obvious flaws. I’ve never looked at Dash, but the assumption of non-collusion is a fatal flaw, and the assumption that there isn’t sufficient available Dash to achieve a controlling stake cannot be true.
skaht01:29
A 1000 Dash stake is required by each masternode. That staking makes the Dash masternodes immune to Sybil attacks.
neillm01:31
Don't forget that Darkcoin/Dash is flawed for staking because when it was release, something like ~50% of the supply was 'instamined' in the first day or so. It was claimed to be a bug, but staking can't be trusted after that, as all masternodes (that require some darkcoin/dash) are almost certainly mostly the same person(s)
👍 1
skaht01:32
Is a Blockchain fullnode with extra consus rules and provides extra value added services, e.g. InstantSend, PrivateSend.
skaht01:34
Anybody that has 1000 Dash and is willing to field a masternode or outsource a masternode on a VPS.
neillm01:34
Yeah, Darkcoin has a very bad past. Dash was the rebrand and the community has done their best to distance itself from that origin. I remember when it all went down, and re-starting the chain was on the table, but of course didn't happen.
evoskuil01:35
So how is it assumed that these are distinct individuals. And how does one overpower this censoring group?
skaht01:35
BLS signatures are applied to seperate duties between masternode stakers, masternode operators, and masternode voters.
neillm01:38
So 1 person can in fact be 90% of the masternodes, which is the problem. Masternodes used to (perhaps still) do mixing as well, which is why it's can't be trusted. If you can have the ability to be most of them, mixing isn't actually mixing, it's just convoluted tracking.
skaht01:38
1 masternode maps to 1 vote for approving budget on the Dash masternode network. Can’t stake more than 1000 Dash per masternode, and staking must be at least 1000 Dash.
evoskuil01:40
Controlling 90% of mining doesn’t matter unless you censor. Once you do you can be evicted from control. In a PoS system you own it, forever.
skaht01:41
Is very censorship resistant in accordance with the Dash consensus rules. Can always run your own Dash full node and or masternode to connect your clients.
skaht01:45
There is no censorship resistance if you don’t run your own nodes. Not your nodes, not necessarily your routing rules. Dash also inherits its tier 1 consensus rules directly from Bitcoin-core.
skaht01:47
The Dash fast mine was an artifact of the same Litecoin fast mine that Charlie does not like to talk about.
neillm01:48
I haven't been following Dash that much, but as far as I know the issuance remained the same, which was based on the X11 mining algo, released as Darkcoin, which gave the instaminer's (presumably Evan the 'founder') a massive amount of funds to work with. Of course PoS and masternodes were his later invention.
evoskuil01:49
The actual events are irrelevant, the point is that it is not censorship resistant. A group can obtain sufficient state to forever control the future.
neillm01:49
I heard many excuses about the fast mine, but never that one. LTC didn't mine over half the supply in a day.
neillm01:50
But agreed, with @evoskuil here. It's a good chat to be had, and not trying to bash anyone or anything here unnecessarily.
evoskuil01:51
I’m just responding to the critique, which I don’t accept. PoS cannot be censorship resistant.
skaht01:52
Facts about the Dash fastmine are available at https://www.youtube.com/watch?time_continue=0&v=O39kWbOu11M.
neillm01:55
It was certainly not launched as xcoin, as the wallet and naming back then was called darkcoin 😉
skaht01:57
FYSA - Not aware of any coin control mechanisms that support Trezor, Ledger, and KeepKey other than that available at https://github.com/Bertrand256/dash-masternode-tool/releases that connect to the Dash-core wallet.
neillm01:57
I don't want to go down these routes though, been there years ago. It's not important as @evoskuil hinted. The model is still problematic.
neillm02:01
Interesting, so I have the original darkcoin git source and in fact, it was xcoin at one point. It was darkcoin by the time it was launched though, as that's the repo I initially retrieved at launch.
skaht02:06
We will agree to disagree for now. I don’t understand how the model is flawed.
Don’t confuse Bitcoin’s network effect lead with Dash being a flawed model. https://www.youtube.com/watch?time_continue=675&v=qrwgYDAoZV0 is a video from Andreas that speaks about Dash Chainlocks.
👍 1
evoskuil02:08
evoskuil02:11
This has no bearing whatsoever on censorship resistance or any other aspect of Bitcoin security.
skaht02:12
Censorship resistance has to do with not filtering transactions sent, transmitted by wallets. It is routing issue.
(edited)skaht02:14
There are other full nodes routing at the application level. Who is to say there are no full node intermediaries that are not censoring?
skaht02:19
Seem to remember that Slush gave priority to a Tone Vays Bitcoin transaction that by all rights did not pay adequate mining fees. Does a propaganda mine also qualify as censorship?
evoskuil02:20
The p2p network doesn’t matter. It’s not required for either validation or confirmation, and cannot be secured.
neillm02:20
Put simply, If I need a tx confirmed, I'll mine it myself -- no matter the cost.
(edited)💥 1
evoskuil02:21
A single miner can simply collect txs directly from merchants, via email... or pigeons.
neillm02:23
Put simply, if someone with more stake than me wanted my tx to not be confirmed, the chain may move without my tx.
evoskuil02:24
But this can be done with majority hash power too. The distinction is in the ability to resist the censor.
👍 1
skaht02:24
Miner and Full Nodes strike the balance. If mined blocks don’t conform to consensus rules, the full nodes won’t propagate the mined blocks. If signed transactions don’t conform to consensus rules they won’t be forwarded to miners.
skaht02:27
Censorship is a man-in-the-middle attack that prevents information from flowing to intended recipients.
evoskuil02:28
No, you have chose a definition that is not in common use. More importantly it is causing you to ignore an actual behavior regardless of what it’s called.
neillm02:28
@skaht What about the case of me setting up the biggest mining shop and mining empty blocks with NO peer connections? This is censorship.
skaht02:30
Mining blocks with zero transactions is denial of service which is a temporary form of censorship.
👍 1
skaht02:31
The Dash Network consensus rules can be tweaked to reject mined blocks not containing any TXs.
neillm02:32
That doesn't solve anything, a government/state actor could do that as well. It doesn't mean that YOUR tx will ever be confirmed though.
neillm02:34
@skaht I don't like to travel. Any chance you can make it to CryptoEcon 2020 (no idea where that is), but if you can make it, I think I'd like to make it as well.
skaht02:34
If a transaction completion must be timely, a delay in the confirmation has the effect of being censored.
skaht02:37
That is very much the case in the real business world. Products get sent to customers that effectively settle 1st.
neillm02:38
Yes, we're not talking about traditional everyday bidness txs (like coffee). We're talking about txs that need to happen at any cost.
neillm02:39
Like I said, I don't know anything about it, so @evoskuil is the one to hit up. BTW, it's exciting because I think this is the FIRST conference hosted/planned by @evoskuil(presumably with a team, but not sure).
neillm02:39
This space BADLY needs cryptoecon lessons. At least analyzing what it is we're dealing with honestly (and better yet, provably).
skaht02:41
Got it! Will be game if costs can be constrained, on my dime. Traveling to Anarchapulco during Feb. 2020.
neillm02:41
Cool yeah, no comment on the costs, but can't imagine it would be crazy weird. No idea though 🙂
skaht02:48
FYSA - There are around a half a dozen virtual Dash masternode services like https://masternode.me.
Hosting full nodes and/or masternodes is a form of centralization that State actors can effectively pull the plug. This issue is in common with all cryptos.
skaht02:51
The hosting of Bitcoin full nodes can accomplished on centralized VPS cloud service providers. That is what is in common.
(edited)neillm02:57
Agreed, but doesn't matter for the mined locally case ... in that I have a tx I want in, and I mine it. Doesn't matter if VPS nodes accept it (because it was valid from the start, just not approved by the censors).
skaht02:58
The more I think about, Dash masternodes are really staking based upon Proof of Service that also requires staking. Incentivizing masternodes (sharing 45% of the mined block rewards) ensures there are a substantial number of Dash network full nodes.
neillm02:59
If 90% of dash masternodes don't want a particular tx included, is there any way it could be not included in the chain?
skaht03:21
Lets see if the question you asked either teaches you or me something new.
If 60% of a transaction quorum don’t want a transaction to be mined only because the transaction is not consistent with the hard coded transaction consensus rules within the masternode code base maintained by the dash-core development team.
If miners don’t like new masternode consensus rules, they don’t have to mine newly formed transactions associated with a new Dash code base. Miners must approve all new Dash-core code releases.
Dash transaction quorums are rotated every 24 hours from a pool of ~5K masternodes deployed around the World. The mathematics behind BLS threshold signature technology is applied to randomly select the next transaction processing quorum of 30 masternodes every 24 hours.
It is important to realize the Dash-core wallet is a Dash full node without extended Dash masternode behaviors that provide value added services such as voting on Dash Treasury proposals (funded by 10% of Dash mining proceeds), InstantSend, PrivateSend, and eventually services such as IPFS…
neillm03:23
I'm talking about a 100% perfectly consensus valid tx. But it's a tx that that a state actors says can't happen because of XYZ.
skaht03:33
POSe scores penalize masternodes not performing their roles. If a State actor takes down a masternode performing a Long Lived Masternode Quorum (LLMQ) function properly. It is removed and the quorum’s masternodes are updated.
neillm03:36
I'm not out to fight this atm ... the masternode system is flawed, as stated earlier. Punish whoever you need, which I'm guessing isn't many.
skaht03:38
No masternode bonding is required because the master node selection process is trusteless and PoSe score non-compliance prevents non-compliant masternodes from collecting their round robin staking rewards.
skaht03:39
I’m not trying to fight. Trying to learn if there is something I don’t understand. Sorry if I’m coming across as being adamant.
skaht03:41
Owning 90% is a costly thing to accomplish. The mining of Dash is decentralized. Miners can also run masternodes. About 1/2 of Dash has yet to be mined.
skaht03:42
Acquiring 90% of the BTC will drive the market price of BTC up. The same is true for Dash.
skaht03:49
BTC havenings are approximately every 4 years. Dash block rewards are adjusted approximately every year and are reduced to 13/14 the rewards of the previous year (-7.2%), which currently puts Dash on approximately a 10 year halvening cycle.
neillm03:51
There's nothing more I want to argue here at the moment. None of this matters if you're 90% of the current masternodes.
skaht03:53
I agree that the Gini Coefficient is extramely important for cryptos. https://en.wikipedia.org/wiki/Gini_coefficient
Wikipedia
In economics, the Gini coefficient ( JEE-nee), sometimes called Gini index, or Gini ratio, is a measure of statistical dispersion intended to represent the income or wealth distribution of a nation's residents, and is the most commonly used measurement of inequality. It was developed by the Italian statistician and sociologist Corrado Gini and published in his 1912 paper Variability and Mutability (Italian: Variabilità e mutabilità).The Gini coefficient measures the inequality among values of a frequency distribution (for example, levels of income). A Gini coefficient of zero expresses perfect…
skaht03:57
With BTC being perceived as a store of value, that could ruin BTCs Gini coefficient. Being an easy to use medium of exchange could improve the Gini coefficient to counter the negative effects of staking.
skaht04:07
How many governments have survived the test of time with 90% thresholds in legislatures? Nothing gets done. A constitutional amendment requires 3/4 of the States to approve. Albeit, cryptos are about opt-in versus being forced-in.
A 90% consensus threshold by BTC could make it a technological dinosaur to cryptos that are slightly more agile with their governance.
neillm04:10
That's the point, no? Governance is basically making up new rules whenever/wherever, which is the problematic point.
skaht04:11
The network effect of BTC is not to be underestimated. If BTC starts to fail with its fungibility, XMR will become a relief valve. If BTC fails with scalability and processing speed, Dash is likely to become a relief valve with its innate point-of-sale capabilities.
neillm04:12
I'm not asking what you're holding, just pointing out that there are options for many people to make transactions.
skaht04:18
Hell is when the IMF SDR is viewed as the solution to Triffin’s Dilemma. Crypto is a also a Stateless solution to Triffin’s Dilemma.
Revive20:26
Xcoin/darkcoin/dash is a total marketing scam, I ran a masternode briefly when it was darkcoin
👍 1
Revive20:28
Not mentioned was during the early days when the "bug" that mined most of the coins happened, the coin supply was capped at litecoin's level of 81 mil I think, then sometime after the bug they moved it down to 21 mill or even lower I think
Revive20:31
Most if not all the work is just spun from someone else's research or ideas and marketed as innovation. Even the 1000 coin requirement was a ploy to up the price of the coin
6 replies · neillm, evoskuil, skaht · last Oct 21, 2019
skaht03:27
Hearing that https://tabconf.com/ will be moved to March. This is probably one of the best bang for the buck conferences, is not profit driven, is community driven. It had 3 parallel tracks last year over 2 days.
(edited)skaht13:12
@evoskuil - staying on topic this time for https://github.com/libbitcoin/libbitcoin-system/wiki/Hybrid-Mining-Fallacy
6 replies · skaht, evoskuil · last Oct 21, 2019
skaht13:14
Concerning - The cost of hybrid mining is the combined costs of work and staking, inclusive of capital cost. The return on mining investment necessarily equates to capital cost, as a consequence of competition. As mining is profitable, capital cost does not contribute to security. Achieving majority stake is no more costly than achieving majority hash power. The theory is therefore invalid.
skaht13:18
If staking requires providing network crypto infrastructure such as additional full nodes there is a security value for providing those staking services. However, most staking coins are not providing such a value added network security service.
skaht13:23
Increasing the number of full nodes protects the network’s ledger. A State actor can easily take down a few nodes, but taking down a 1000+ nodes is much harder. Staking that increases the number of full nodes is good for security of a crypto.
skaht13:27
The health of a crypto network can be measured by both its hashing power and the number of Internet network reachable persistent full nodes.
skaht13:35
Contrast the number of LTC full nodes documented at https://chainz.cryptoid.info/ltc/#!network to the ~5K routable full nodes at https://chainz.cryptoid.info/dash/#!network.
chainz.cryptoid.info
Litecoin CryptoID Block Explorer and Statistics. Access detailed blockchain information on Litecoin (ltc) transactions, blocks and addresses
chainz.cryptoid.info
Dash CryptoID Block Explorer and Statistics. Access detailed blockchain information on Dash (dash) transactions, blocks and addresses
Thursday, October 31, 2019
Fil06:59
Hi guys, I came across BlockSci. That's this tool from academia built to make very fast queries on the bitcoin blockchain. The advertise like this:
BlockSci’s design starts with the observation that blockchainsare append-only databases; further, the snapshots used for researchare static. Thus, the ACID properties of transactional databasesare unnecessary.
This reminds me a bit on libbitcoin's database design. Therefore, I wonder if it's possible to have such efficient queries in libbitcoin, too?
(edited)Here's some of their performance measurements.
The anomalous-fee query extracts all tx where the fee was > 0.1 BTC.
Fil07:12
The above quotation is missing a sentence, but slack is not allowing me to edit the message anymore. So here the fixed version:
BlockSci’s design starts with the observation that blockchainsare append-only databases; further, the snapshots used for researchare static. Thus, the ACID properties of transactional databasesare unnecessary. This makes an in-memory analytical database thenatural choice.
extracts == it returns an array of tx objects with fee > 0.1 BTC
Fil07:15
I'm not sure. They actually have a jupyter notebook interface for doing the queries so the low-level details are hidden.
evoskuil07:19
How much ram, how many cores, cpu speed, output format/accumulation... all material. Also one can cache the fee level of the tx to significantly optimize for such a query, but it’s an uncommon query. Finally one could index by fee level, which would be a huge optimization. It’s not clear if they are doing this, but scanning the entire store linearly even using multiple threads implies some sort of optimization if they have it down to 11sec.
But the basic structure they are describing is no different than libbitcoin. Append only is all we do, and memory-mapped.
jungly07:22
@Fil they are focused on the olap use case. Ingest data that won't be updated, and then run read queries on it. Which is somewhat different from us. We need to make sure tx are updated when spent, handle reorgs etc.
For olap use cases you can make a number of assumptions and then optimise.
evoskuil07:24
The txs don’t actually have to be moved or modified for reorg or spends, but that implies additional storage (eg utxo store).
This message was deleted.07:25
1 reply · evoskuil · last Oct 31, 2019
Fil07:26
That's what BlockSci does too. And they keep a delay to the chain tip to omit reorgs but if a reorg happens anyway they just fail 😕
evoskuil07:26
This is really only a question of read perf. The amount of metadata we store in the tax is very small.
Fil07:27
I'm currently evaluating tools to do cluster analysis. And I was wondering if it would be possible to extend libbitcoin to achieve that because it's robust and very maintainable. BlockSci can do the cluster analysis but it's rather meant for interactive use. They don't really care too much about robustness and maintainability.
then if an address A belongs to a cluster that is considered to be "bad" ppl can make decisions about address "A"
some kind of AML thing basically
jungly07:34
The challenge is to have enough ram so your transactions database is not paged out. You probably also want to read the block database?
Their readme says they had a 61gb ram the machine.
r4.2xlarge 8 61 EBS-Only Up to 10
So they are not all in memory either. Just noting.
👍 1
Fil07:36
@evoskuil That is not clear yet. The tech stack is chosen in parallel to the legal requirements being speced out. It's all new in the bitcoin banking world 😄
evoskuil07:36
Computing the fee requires obtaining the prevouts and differencing from the outputs. Very expensive, so no doubt cached and possibly indexed.
Fil07:37
yes, it's expensive that's why they chose that as a benchmark. But I don't think it's cached. They also have a Python version where the query takes 11h on a single core.
Fil07:38
https://arxiv.org/pdf/1709.02489.pdf FYI this is their paper. I'm currenlty setting up a machine to cross-verify their claims.
the indexes are stored in sqlite
Fee cached
I overlooked that one
And indexing it would reduce it to like a couple seconds.
jungly07:45
Oh they have an interesting trick. The transaction graph data (as they call it) is stored without transaction hashes or addresses, and then they have those details in an sqlite index from tx some-sort-of-id to that index. They claim most cluster analysis doesn't need the transaction hashes and address info. 🤷🏾♂️
Just create a partially ordered table of fee to tx hash, insertion sort on graph population. Read the results.
Validation already computes the fee, so just store it as txs are validated.
This can be done as the chain is synced, and can be maintained through reorgs (ie to count confirmed only txs) efficiently by using a bitmap with a hash table on tx index and a partial order on fee.
jungly07:49
They have a cute system man. The new data and updates go straight to disk. While for analysis you load the file once to start the analysis. Then new updates aren't visible to you.
"To load the le for analysis, it is mapped into memory.
The on-disk representation continues to grow (and be modied in
place), but the analysis library provides a static view (Section 2.4)."
Accademia man
Get published
Further your career
Add I was once told, "what results do you want?"
evoskuil07:51
I mean seriously, you can optimize for anything. This result can be computed while syncing the node at almost no cost.
Total time much less.
jungly07:52
But still. I'll read the paper with calm. Thanks @Fil I can only learn by seeing efforts of others. In memory databases have started to excite me a lot 😄
😉 1
like this one
evoskuil07:54
Ignore the cost of populating an intermediate database with the entire blockchain, then populating a static graph optimized for specific queries.
Fil07:55
That's why I like libbitcoin. The blockchain is written directly to a append-only, memory-mapped db.
👍 1
evoskuil07:55
Since the tx is already in memory as it arrives, and the fee already computed on the tx object as a consequence of validation, the subscriber can just ignore lower.m fee txs. Don’t even need to change any code.
Set up a zmq client and the results are in once sync is done.
Can even be done in-process.
And can bypass ZeroMQ by deriving directly from the node object.
Libbitcoin even caches the total input and output values on the tx object, so there's not even a recomputation to obtain the fee.
Yeah, the memory map ensures an OS optimized tradeoff for available RAM.
Fil08:07
Thanks! That sounds good. I'll keep digging into this. First I'll check what the blocksci performance on my machine is so I have a baseline to compare. But first the thing needs to finish parsing... 😂