Bitcoin full node and query server

Oct 25, 2024

View in context in the libbitcoin Slack #general mirror.

evoskuil00:54

I'm seeing some warnings coming up in GCC 13.2 (default Ubuntu). We're running 11.4 in CI, which isn't much order (2 months) in terms of release dates. @pmienk do we have an upper bound on GCC support presently?

150 replies · pmienk, evoskuil, parsevalbtc · last Oct 29, 2024

pmienk01:01

I remember either clang or gcc has an issue after a certain version. I thought it was clang though

evoskuil01:02

Ok, prob better to not bump it up, but we should still delint this.

evoskuil01:29

Looks like all of the warnings are either in Boost (mostly std::unary_function<> deprecated) or in std::lib.

evoskuil01:30

Just completed the node build with current libsecp.

evoskuil01:31

The whole thing in just 5 fucking minutes, on my cheap ass box.

evoskuil01:57

How do I enable the avx2 and sse4 options?

pmienk02:04

--enable-isystem will hide errors in dependencies

evoskuil02:05

Everything is running, and seems blazing fast, but I'm getting frequent stalling on blocks.

pmienk02:06

--enable-avx2 --enable-sse41

🙌 1

evoskuil02:08

Thanks, are these just passively passed through the install.sh?

evoskuil02:09

Also, are both required or is it cumulative?

evoskuil02:10

IOW I believe that in the hardware avx512 implies avx2 implies sse4. It's not imposed by my code, they are independent flags. So I assume it's probably that way for yours.

pmienk02:18

yes, passively passed. they are independent flags from the perspective of configuration. while hardware-wise i'm unaware of any system which has avx512 but not avx2, there's no ripple checking done so if the code assumes avx512 means avx2, the flag for avx2 wouldn't have been set and the test wouldn't have been run

evoskuil02:19

ok, that's prob best

evoskuil02:19

This stalling is frustrating. Never happens in Windows builds, ever. Really have no tools to troubleshoot it.

evoskuil02:21

The other flag is --enable-avx512 ?

pmienk02:22

--enable-<avx512, avx2, sse41, shani>

pmienk02:24

translates to: avx512 - avx512f, avx512bw avx2 - avx, avx2 shani - msse4, msha sse4 - sse4.1

evoskuil02:24

That's for the compiler options, independent of our internal use right?

pmienk02:25

yes, those are the options passed to the compiler, prefixed with -m

evoskuil02:34

Cool, thanks.

evoskuil02:35

Are you using --enable-isystem in the CI builds?

pmienk02:36

Believe so

pmienk02:36

yep, it's emitted into the ci.yml

evoskuil02:37

AH, maybe that's the only difference. I'll try that now with the hardware flags rebuild.

👍 1

evoskuil02:43

Still one warning (twice) in system, but that's coming out of my use of std lib, so I'll track that down, otherwise clean.

(edited)

evoskuil02:45

configure: WARNING: unrecognized options: --enable-avx2, --enable-sse41

evoskuil02:46

This is coming up in all of the repos except for system. But each of the libs needs to compile with the options right?

pmienk02:47

yeah, that'll spit out for every repo that doesn't recognize them, which at min will be secp256k1, but probably some of our repos above system

pmienk02:48

the other repos will pick up the compiler flags and defines via pkgconfig

evoskuil02:48

Are you passing the WITH_ symbols to all repos?

pmienk02:48

defines go via pkgconfig - pc file

evoskuil02:48

Oh, missed that.

(edited)

evoskuil02:49

But does that mean these will be affected in each repo? --enable-avx2, --enable-sse41...

evoskuil02:50

So for example, avx512f, avx512bw gets passed?

pmienk02:50

no. it means if someone were to manually try to build a project and already had system with those options, then the system options would be used. no point in having two configuration flags passed that disagree

pmienk02:50

-mavx512f -mavx512bw, but yeah those end up as CPPFLAGS

evoskuil02:53

Ok, that's what I meant. The system options derived from our the args passed to system will cascade so that all libs get the same compiler options (and our preprocessor symbols).

pmienk02:53

along with WITH_AVX512

evoskuil02:53

Cool. Just another time where the unknown option thing gets confusing.

pmienk02:54

all libs get their antecedent flags plus their own

pmienk02:54

yeah, i never looked to suppress it, i think it's more useful to leave though

evoskuil02:56

checking --enable-avx2 option... yes
checking --enable-avx512 option... no
checking --enable-sse41 option... yes
checking --enable-shani option... no
checking --enable-ndebug option... yes
checking --enable-isystem option... yes

evoskuil02:56

This was spit out from the system build. But...

(edited)

evoskuil02:57

$ ./bn --hardware
Intrinsics...
arm..... platform:0.
intel... platform:1.
avx512.. platform:0 compiled:0.
avx2.... platform:1 compiled:0.
sse41... platform:1 compiled:0.
shani... platform:1 compiled:0.
neon.... platform:0 compiled:0.

evoskuil02:57

should be:

avx2.... platform:1 compiled:1.
sse41... platform:1 compiled:1.

pmienk02:58

what's in the pc file for system?

evoskuil02:58

That comes from the WITH_AVX2 and WITH_SSE41

evoskuil02:59

Sorry, don't follow

evoskuil02:59

evoskuil03:00

Cflags: -I${includedir}  -DWITH_AVX2   -DWITH_SSE41 -I/home/nixmini/Prefix/include

pmienk03:00

just wondering what the contents of libbitcoin-system.pc

pmienk03:00

so... they should be defined

evoskuil03:00

The others don't have it (in the pc file).

(edited)

pmienk03:02

well, that's annoying.

evoskuil03:02

Are they supposed to be copied into the pc's?

pmienk03:04

no, and because of the depth of one dependency, they get squished out. which would imply anything passed that way would require restatement in the pc file

pmienk03:05

basically, we'd loose any cflags for libsecp256k1 after libbitcoin-network/-database

evoskuil03:06

"because of the depth of one dependency" ?

pmienk03:07

configure.ac only references explicit dependencies from the project. node reaches to network and database, network reaches to system

pmienk03:07

if node were to pull in system explicitly, the problem would be solved.

pmienk03:08

its probably the proper solution

evoskuil03:08

You mean in the metadata?

pmienk03:08

no, in the actual configure.ac file

(edited)

evoskuil03:09

ok. what does this imply about passage of the specific hw compiler options?

(edited)

pmienk03:09

the xml is great as is. the configure.ac and Makefile.am just need to explicitly reference the full dependency tree

pmienk03:09

they won't get communicated past the system dependent project.

evoskuil03:10

are they supposed to be in system pc?

pmienk03:10

yes

evoskuil03:10

Looks like that's an issue then.

pmienk03:10

and they're there. the pc file just isn't coming in to the dependencies that aren't explicit.

pmienk03:11

yeah, thought it was fixed. thought it was verified. must have been for the database project and so this got missed

pmienk03:11

will look at it now

evoskuil03:13

K, I'll go back to building my performance lab. You can verify the PP symbols by running bn --hardware

evoskuil03:13

The compiler flags I dunno.

pmienk03:43

Yeah... I get a different result.

pmienk03:43

it looks like they're properly passed with Requires

pmienk03:44

i build with --enable-avx2 --enable-sse41 and

$ ./bn --hardware
Intrinsics...
arm..... platform:0.
intel... platform:1.
avx512.. platform:0 compiled:0.
avx2.... platform:1 compiled:1.
sse41... platform:1 compiled:1.
shani... platform:0 compiled:0.
neon.... platform:0 compiled:0.

pmienk03:46

I had forgotten about Requires

evoskuil03:53

ohh, cool

pmienk03:54

I'm retrying with install.sh to confirm that it's working, that was devsetup output

evoskuil03:54

So what would I need to do differently?

pmienk03:55

well, as long as there is nothing special between the two scripts, which there shouldn't be, all i did was --enable-avx2 --enable-sse41 --enable-isystem

evoskuil03:58

But that's what I did.

evoskuil03:58

What's "Requires"?

pmienk03:58

a parameter within the package config pc file which is how the system pc file is supposed to be brought in

evoskuil03:59

Did you just change that?

pmienk03:59

nope

pmienk04:01

same results with install.sh

pmienk04:01

so it's working

pmienk04:01

at least locally for me

evoskuil04:04

I'm rebuilding to get a clean log

evoskuil04:11

Well shit, that works here too

evoskuil04:11

So one last question on intrinsics...

pmienk04:12

glad it works, did not want to have to refactor that stuff

🎉 2

evoskuil04:13

Are you sure that the -m arguments are making it into each/any lib, given that they don't show in any of the pc files?

evoskuil04:14

I guess I could test by compiling for avx512 and letting it crash, but that wouldn't tell me for sure it got into more than just system.

evoskuil04:15

Does the log show it?

pmienk04:15

I wouldn't assume they're making it past -system when not in the pc file.

pmienk04:16

if they need to be added to cflags, that's possible

evoskuil04:17

The issue is that we would get the optimizations in system only, and can't really tell without inspecting the disassembly.

pmienk04:17

i guess the question is, are there headers/templates that would require it or is this a linking to the static/dynamic lib that was generated with the optimizations already?

evoskuil04:17

The latter

evoskuil04:18

The only header issue is for the PP symbols

evoskuil04:18

The rest is totally silent compiler stuff.

pmienk04:18

thought we had a discussion about this before, and because of that we concluded that the compiler flags were excluded but the declares were passed on

evoskuil04:20

The declares initially were not passed on, and I thought it was enough to just have them in system, but the prob is that the system header gets included in the other builds, and the symbols in the system header are defined independently for each lib. So I added the --hardware option to verify it in the node project.

evoskuil04:21

But the compiler flags are opaque to me, I have to assume each library must be compiled with the flags, and otherwise those optimizations will just not be present in the libs without the flags.

evoskuil04:22

Those allow the compiler to perform autovectorization.

evoskuil04:25

Unrelated: I can't find it in the log. Do we incorporate default compiler optimizations into the script?

pmienk04:27

default compiler optimizations?

evoskuil04:28

eg -O3

pmienk04:28

no, the install scripts don't. we'd have a collision detection problem if someone passed anything via CFLAGS/CXXFLAGS

evoskuil04:30

Right, and in v4 it's a critical setting. So gotta make sure we document. I think we decided -O3 right?

evoskuil04:30

max speed over size

pmienk04:31

sounds right, however the CI uses -Os for all but debug

pmienk04:31

compiler speed issue

👍🏻 1

evoskuil04:33

So now I'm up to:

--build-boost --disable-shared --enable-avx2 --enable-sse41 --enable-isystem CFLAGS=-O3 CXXFLAGS=-O3

pmienk04:34

well, CFLAGS=-O3 CXXFLAGS=-O3 ./install.sh --build-boost --disable-shared --enable-avx2 --enable-sse41 --enable-isystem

(edited)

pmienk04:35

I don't believe it would pick up the flags if they're declared after parameters. needs to prefix the script execution

evoskuil04:37

configure options:
--with-boost=/home/nixmini/Prefix
--with-pkgconfigdir=/home/nixmini/Prefix/lib/pkgconfig
--prefix=/home/nixmini/Prefix
--disable-shared
--enable-avx2
--enable-sse41
--enable-isystem
CFLAGS=-O3
CXXFLAGS=-O3
--enable-static

pmienk04:38

shouldn't need --with-pkgconfigdir, that'll be picked up with the prefix

evoskuil04:38

That's from our build log, and also libsecp:

onfigure: WARNING: unrecognized options: --enable-avx2, --enable-sse41, --enable-isystem

Build Options:
  with external callbacks = no
  with benchmarks         = yes
  with tests              = no
  with ctime tests        = no
  with coverage           = no
  with examples           = no
  module ecdh             = yes
  module recovery         = yes
  module extrakeys        = yes
  module schnorrsig       = yes
  module ellswift         = yes

  asm                     = x86_64
  ecmult window size      = 15
  ecmult gen table size   = 86 KiB

  valgrind                = no
  CC                      = gcc
  CPPFLAGS                =  -w
  SECP_CFLAGS             = -O2  -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wcast-align=strict -fvisibility=hidden 
  CFLAGS                  = -O3
  LDFLAGS                 =

pmienk04:38

oh, our output, okay

evoskuil04:39

So it's not complaining about the FLAGS as unknown, and reports them, and also libsecp is outputting the CFLAGS. If you do it prior to install it sets the process environment, but I think the way we are passing args it also would work.

(edited)

pmienk04:40

I'm surprised, as I don't know of that being documented. only the environment declearations

pmienk04:40

but hey, we pass it everywhere when it's not recognized, except for boost

pmienk04:41

so if you want boost to pick it up, you'll want to do the enviornment

evoskuil04:41

This is what libscp output on the run without me setting them:

Build Options:
  with external callbacks = no
  with benchmarks         = yes
  with tests              = no
  with ctime tests        = no
  with coverage           = no
  with examples           = no
  module ecdh             = yes
  module recovery         = yes
  module extrakeys        = yes
  module schnorrsig       = yes
  module ellswift         = yes

  asm                     = x86_64
  ecmult window size      = 15
  ecmult gen table size   = 86 KiB

  valgrind                = no
  CC                      = gcc
  CPPFLAGS                =  -w
  SECP_CFLAGS             = -O2  -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wcast-align=strict -fvisibility=hidden 
  CFLAGS                  = -g -O2
  LDFLAGS                 =

evoskuil04:41

So it got switched from -g -O2 to -O3

(edited)

pmienk04:41

yeah, any output from boost though?

evoskuil04:42

libsec was throwing in debugging :?

evoskuil04:45

Libbitcoin boost configuration.
--------------------------------------------------------------------
variant               : release
threading             : multi
toolset               : 
boost cxxflags        : cxxflags=-Wno-enum-constexpr-conversion
boost linkflags       : 
link                  : static
boost.locale.iconv    : on
boost.locale.posix    : on
-sNO_BZIP2            : 1
-sNO_ZSTD             : 1
-sICU_PATH            : 
-j                    : 16
-d0                   : [supress informational messages]
-q                    : [stop at the first error]
--reconfigure         : [ignore cached configuration]
--prefix              : /home/nixmini/Prefix
BOOST_OPTIONS         : --with-chrono --with-iostreams --with-json --with-locale --with-program_options --with-regex --with-system --with-thread --with-test
cxxflags (ignored)    : 
--------------------------------------------------------------------

pmienk04:45

likely your system environment default

evoskuil04:45

libbitcoin-node installer configuration.
--------------------------------------------------------------------
OS                    : Linux
PARALLEL              : 16
CC                    : 
CXX                   : 
CPPFLAGS              : 
CFLAGS                : 
CXXFLAGS              : 
LDFLAGS               : 
LDLIBS                : 
WITH_ICU              : 
BUILD_ICU             : 
BUILD_BOOST           : yes
BOOST_ROOT            : /home/nixmini/Prefix
BUILD_DIR             : build-libbitcoin-node
PREFIX                : /home/nixmini/Prefix
DISABLE_SHARED        : yes
DISABLE_STATIC        : 
with_boost            : --with-boost=/home/nixmini/Prefix
with_pkgconfigdir     : --with-pkgconfigdir=/home/nixmini/Prefix/lib/pkgconfig
--------------------------------------------------------------------

pmienk04:46

huh, well then maybe

evoskuil04:47

The flags don't show there, which is our output, prob from the env.

evoskuil04:47

I'll try it via env (prior to install)

(edited)

pmienk04:48

# If coverage is enabled, and the user has not overridden CFLAGS,
# override Autoconf's value "-g -O2" with "-g". Otherwise we'd end up
# with "-O0 --coverage -g -O2".

pmienk04:48

comment within their configure.ac

evoskuil04:48

Yeah, they get picked up in boost and libsec and in our output.

pmienk04:49

for them to have picked up in boost, it had to treat them as environment because the parameters wouldn't have made it to the relevant calls that way

evoskuil04:49

ok, we have it documented the other way on -system readme

pmienk04:50

you mean to prefix the calls? yeah

evoskuil04:51

It's documented like:

$ ./install.sh CFLAGS="-Og -g" --prefix=/home/me/myprefix

evoskuil04:53

There's no positive log feedback in the libbitcoin repos, but it's def in libsec and boost.

pmienk04:57

I don't know what switched the order on master vs version3, that wasn't how it was documented when the docs were scrubbed

evoskuil05:07

I blame Jungly 😐

pmienk05:10

ha, no idea. i never touched system/master README.md, so maybe it's a correction I made to version3

parsevalbtc08:16

I remember I had something similar back this summer. Eventually @pmienk solved the issue, at least for me. https://github.com/libbitcoin/libbitcoin-node/issues/654