February 2021

Hello guys! the mmap emulation code for Windows where in the codebase is it?

I'd start here: https://github.com/libbitcoin/libbitcoin-database/tree/master/src/mman-win32

Thank you!

@neillm For mmap function, at least on windows, what kind of data block sizes are used, in other words, the len argument is what? what about if there is a btc transaction that spans such memory block boundary?

Try here: https://github.com/libbitcoin/libbitcoin-database/blob/master/src/memory/file_storage.cpp#L411

oh ok, no problem! Thank you for the reference! It’s very helpful! Checking that..

The size of data doesn’t matter.

That book is amazing you guys!

@evoskuil so, in order to implement a custom IO/persistence backend for LibBitcoin,

As I’ve explained in DM, no.

@evoskuil but, uuh, the file_storage class is used as interface. And that one has a method .access()

Given your description of objectives you SHOULD implement your own virtual memory layer over your hardware memory API.

that gives you the memory pointer (i.e. a void*) to the virtual memory address range for that file,

And then tie that to the Libbitcoin interface.

Right, for an architecture with so much virtual memory, that works, sure

It works for any architecture.

Read about VM implementation. https://en.m.wikipedia.org/wiki/Virtual_memory

One would be that our custom file_storage implementation would malloc() the whole intended memory range

You must IMPLEMENT (or find an implementation) of VM over your hardware API.

And then propagate that memory range with said information before file_storage:open() returns

How you implement it is entirely up to you.

Anyhow I see your point, I'd need some abstraction to actually give me userland virtual memory

You can expect no changes in Libbitcoin to do this. Not because it’s hard, but because it would be wrong.

Wrong in what sense?

Architecturally, for both your implementation and for Libbitcoin.

Not for a high performance software usecase,

And performance would in no way be improved by accepting it.

In this situation the NVMe etc. hardware will load chunks into given memory addresses and store given memory addresses to persistence, respcetively

Under the VM

mmap() is an abstraction that segregates the physical IO operations + caching logic, from the place where the information is actually processed

Mmap isn’t relevant.

In a direct hardware access case, the userland process implements its own cache

You are implementing the interface. It does not have to be or use mmap.

Yeah though alternative to mmap is either malloc() based in which case RAM requirement for solution is hundreds of GB which is unpractical,

Doesn’t matter how it does it. That’s what abstraction layers are for.

or implement my own virtual memory backing thing which would be really custom, not aware how feasible

You optimize in the layer.

LibBitcoin having its own IO is not strange, OS IO is frequently archaic.

It would be a very poor architectural decision to write data access directly in the underlying API. Our interface is not mmap, it’s our abstraction over it.

@evoskuil wait we checked what is done in Win32, and it's a wrapping over CreateFileMappingW() and MapViewOfFile(), which are just a flavor of mmap()?

No, it is not a bridge to mmap. It is an implementation of data access that satisfies the interface above it using the Win32 APIs.

> It would be a very poor architectural decision to write data access directly in the underlying API. Our interface is not mmap, it’s our abstraction over it. I mean in practice what you do is expect one continuous memory address range of couple hundred GB, to represent the LB blockchain file?

Do you really understand what is meant by virtual memory?

Yes, a memory range that is not backed by physical memory

I REALLY suggest you study VM in the context of your questions.

mmap() is such that it guarantees the memory from a void* and the subsequent size bytes, to be reserved -

mmap is NOT relevant

I know what VM is

You may know what it is but your questions imply you don’t really understand it.

LibBitcoin is based on that LB's persistent memory is represented in whole by given logical memory address ranges

LOGICAL memory yes, PHYSICAL is NOT contiguous.

I suggest I understand it but there are limits to what virtual memory operations Unix offers one to do in userland

That’s the point of VM.

Yeah exactly. So now here I'm in userland, and have a direct hardware access based NVMe driver. It will give me load and store operations.

Logical memory will always be contiguous.

If I had something like "vmem_map(logical_address,physical address,len);" and "vmem_unmap(logical-address,len);"

It’s not a problem with or limitation of Libbitcoin.

You are missing my point - so, in userland when you access hardware directly, then you're operating at the physical medium's sector size e.g. 4KB, so you do NVMe load and store ops, at a very very high speed.

And you are missing my point. I have been exceedingly explicit about it. You need to IMPLEMENT the VM layer.

the latter would be a PhD thesis or two or three in itself

If you do not all you will end up doing is pushing that complexity into Libbitcoin, to no performance benefit.

it's barely clear how to implement custom page fault handlers in Linux

But that is what you WANT!?

> If you do not all you will end up doing is pushing that complexity into Libbitcoin, to no performance benefit. My impression of the performance of userland direct NVMe hardware access is that it surprisingly fast

Irrelevant

Ballpark 10-20x faster than when going through the host OS' IO subsystem

Still irrelevant

It's not about pushing it into LibBitcoin, but to push it into userland from OS,

And you have to compare apples to apples. It only matters to the extent you can USE it.

A friend told me that Google does all their persistent access through userland direct access

Great, so do it.

however hardware operates in terms of load/store operations, not a memory abstraction

And software does NOT

> I’m telling you how. How, without implementing vmem_map() and vmem_unmap() kind of virtual memory page allocate/deallocate virtual memory management support for my host OS?

Exactly, software requires the abstraction.

A competent friend suggests "I would argue that it's libbitcoin that lacks the hook you desire" (if he understood correctly)

The key phrase being “you desire” vs. “you require”.

points out "If you can map a page so that it's unreadable and unwritable and then trap on the sigsegv when someone tries to access it you could potentially remap they area"

The memory class implements reads and writes. You simply implement it as part of the interface.

@evoskuil that sounds great! How do I reach Kulpreet?

@jungly

Thanks! 😄 pinging In https://github.com/libbitcoin/libbitcoin-database/tree/master/include/bitcoin/database/memory I see you and Phillip

Point?

@evoskuil is Phillip Fil here on Slack?

Oh A clarifies one C++ism that applies here, neat!: "You can replace that memory_ptr with an abstract interface that mimcs the mechanics of a character array So you don't need to change the actual code, and the memory_ptr is replaceable yep you might even have to get into templates" yey, hm

So, he suggests that LibBitcoin's memory_ptr can be abstracted upon. LB's memory_ptr is defined as memory::ptr . Exploring.

right so memory is defined here https://github.com/libbitcoin/libbitcoin-database/blob/master/include/bitcoin/database/memory/memory.hpp#L33-L43 /// This interface defines remap safe unrestricted access to a memory map. class BCD_API memory { public: typedef std::shared_ptr<memory> ptr; /// Get the address indicated by the pointer. virtual uint8_t* buffer() = 0; /// Increment the pointer the specified number of bytes within the record. virtual void increment(size_t value) = 0; };

@evoskuil he suggests i can do operator overloading on the uint8_t, however he suggested a couple times I should talk to you for the possibility of a LibBitcoin redesign in some part, to operate on whole blobs instead of individual uint8_t:s - though if I can pull off the uint8_t operator overloading thing fast then that might be totally fine as a start

I did not suggest any change to Libbitcoin.

Very interesting, all places in LB where .buffer() is used are: .//libbitcoin-database/include/bitcoin/database/impl/slab_manager.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/slab_manager.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/record_manager.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/record_manager.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: return std::equal(key.begin(), key.end(), memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/list_element.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/hash_table_header.ipp: memset(memory->buffer(), (uint8_t)empty, file_size); .//libbitcoin-database/include/bitcoin/database/impl/hash_table_header.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/hash_table_header.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/hash_table_header.ipp: auto deserial = system::make_unsafe_deserializer(memory->buffer()); .//libbitcoin-database/include/bitcoin/database/impl/hash_table_header.ipp: auto serial = system::make_unsafe_serializer(memory->buffer()); .//libbitcoin-database/src/result/transaction_iterator.cpp: auto deserial = make_unsafe_deserializer(memory->buffer());

🤷🏻‍♂️

Wait what are the multiple OS threads in LibBitcoin doing again. So one thing you have parallellism for is to process input from different Bitcoin node peer connections in parallell.

Concurrent reads of the same segment are allowed.

@evoskuil is there any writeup anywhere of what different OS thread roles are involved in a full-fledged LibBitcoin node instance?

All threads are “OS threads”

I'm a program (= OS process) that runs my own LibBitcoin node, i just ->start():ed it and now i'm having my own N OS threads doing all kinds of blockchain traversal fun, and also broadcasting some txes

The write up is the code

= typical block explorer usecase really heh

Neat we're reading. @evoskuil/@neillm I will comprehensively answer the following question myself by deep reading the code, however if you have any comment would be much appreciated: When in my application OS thread I make a call into LibBitcoin to do blockchain queries like retrieve a block, retrieve a transaction, and thus traverse all kinds of aspects of the blockchain, do there exist any points in LibBitcoin codes during the course of those operations that could be BLOCKING, EXCEPT for the disk IO itself (i.e.: that is in vanilla LibBitcoin the reads to the mmapped memory)? E.g. some point where some LibBitcoin logic would go into a blocking wait for some LibBitcoin node-internal worker thread, such

There is no blocking in memory management except for allocation. Data access critical sections are in the store code, not at the lower level memory management. There is also a critical section in work queue/dequeue.

Great. So when you're an application thread doing blockchain explorer type work then the only thing that could block would be that the mmapped memory would need to read from disk. Great. Thanks for clarifying!

LibBitcoin does not use thread-local storage for anything does it?

It may in one case. Just search.

aha interesting, thanks for syaing

@evoskuil just to save me from derpy mistakes, in this one optional case where it may, through which API does LB use the thread local storage then?

Don’t recall, and it may just be a todo comment, but it would be through a boost threading class. Possibly in a base in -system. Should be an easy grep.

Why does it say “#1 new release” ?

🤷🏻‍♂️

Sweeeet!

Hi, I browsed the LB codebase for use of thread local storage, and i find one usage instance, which is that the Boost thread local storage abstraction is used to carry a PRNG, here: .//libbitcoin/src/utility/pseudo_random.cpp: static boost::thread_specific_ptr<std::mt19937> twister(deleter); Just curious why is this done, the OS for instance offers a random function?

I believe the random engine used is not thread-safe, so each thread uses its own to avoid some complex locking around it

Related info: https://stackoverflow.com/questions/40655814/is-mersenne-twister-thread-safe-for-cpp

Oh, I see. But why not just use OS random numbers facility?

Historically, not so random

Like /dev/urandom? If so, I believe those wrappers also take contributions from OS random, but not 100%

No, I mean the C API! 🙂

Because the code is C++? 🙂 And rand() has the same problem, and has different distribution properties than the one we're using

something like that https://linux.die.net/man/3/arc4random

Generally the idea is to stick to std C++, and then boost as needed.

why is this any relevant? the thing with your current solution is that it “”taints”" :-) the thread local storage. i’m doing some deep plumbing where i’ll use thread local storage for myself, so something needs to move 🙂 random() looks like a nice substitute in my case

I'm also not understanding the relevance here

Neil, i have an idea: to clear up the TLS for me, actually this would work fine: locate the PRNG to a static variable (= one global) and put a mutex around its accesses. This would be the mildest code rewrite, preserve LB as-is the most, while freeing up the TLS for me. :-) Could this be relevant for a LibBitcoin ./configure optional? 🙂 ./configure --don’t-touch-tls-please

I believe that's specifically what was trying to be avoided

Is that TLS only used in LB worker threads?

🤷🏻‍♂️ what is the issue?

I am making a C++ program which will contain an LB Node class instance i do all kinds of stuff in my program, and one of the exotic things i do is that i’m using boost thread local storage myself! how do I do this without tanking & corrupting LB + my program

What is the issue with "corruption"?

I'm considering dropping support for VS2013. Support for VC++11 in VS2013 is only available through a technology preview addition to VC++, and there are a couple things we have to do in the source to accommodate unsupported C++11 features. It's not a huge problem, but it would speed up development and testing, especially given the present need to add VS2019. Looking for anyone who has current hard dependencies on VS2013.