Bug: neard binary built with clang-18 segfaulting after opening a RocksDB instance

[tschuyebuhl]

Contact Details

[email protected]

Which network are you using?

mainnet

What happened?

When building neard with the default, non-versioned clang on ubuntu 24.04, the neard binary segfaults shortly after running (see attached logs)

I've created a Dockerfile which should help with reproducing. Changing the source image tag to 22.04 results in a binary working just fine.

FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive \
    CARGO_TERM_COLOR=always

RUN apt-get update && apt-get install -y --no-install-recommends \
    clang \
    git \
    libssl-dev \
    make \
    pkg-config \
    curl \
    ca-certificates && \
    rm -rf /var/lib/apt/lists/*

RUN curl --proto '=https' --tlsv1.3 -sSf -o /tmp/rustup.sh https://sh.rustup.rs && \
    sh /tmp/rustup.sh -y --default-toolchain none && \
    rm /tmp/rustup.sh && \
    echo ". /root/.cargo/env" >> /etc/profile

ENV PATH="/root/.cargo/bin:$PATH"

RUN git clone --depth 1 --branch 2.4.0 https://github.com/near/nearcore.git /checkout

WORKDIR /checkout
RUN make release

RUN mkdir /out && cp target/release/neard /out

VOLUME ["/out"]

CMD ["ls", "-l", "/out"]

The above Dockerfile may be built with:

docker build -t neard-builder-defaultclang .
docker run --rm -v $(pwd)/out:/out neard-builder-defaultclang cp /checkout/target/release/neard /out/neard-defaultclang

Ubuntu 24.04 requires an additional workaround, to symlink clang-17:

FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive \
    CARGO_TERM_COLOR=always

RUN apt-get update && apt-get install -y --no-install-recommends \
    clang-17 \
    clang-tools-17 \
    git \
    libssl-dev \
    make \
    pkg-config \
    curl \
    ca-certificates && \
    rm -rf /var/lib/apt/lists/*

RUN curl --proto '=https' --tlsv1.3 -sSf -o /tmp/rustup.sh https://sh.rustup.rs && \
    sh /tmp/rustup.sh -y --default-toolchain none && \
    rm /tmp/rustup.sh && \
    echo ". /root/.cargo/env" >> /etc/profile

RUN update-alternatives --install /usr/bin/cc cc /usr/bin/clang-17 100
RUN update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++-17 100

ENV PATH="/root/.cargo/bin:$PATH"

RUN git clone --depth 1 --branch 2.4.0 https://github.com/near/nearcore.git /checkout

WORKDIR /checkout
RUN make release

RUN mkdir /out && cp target/release/neard /out

VOLUME ["/out"]

CMD ["ls", "-l", "/out"]

Similarly, the binary can be built like this:

docker build -t neard-builder-clang17 .
docker run --rm -v $(pwd)/out:/out neard-builder-clang17 cp /checkout/target/release/neard /out/neard-clang17

Binaries should be present in the out directory. Please take a look at this and do not hesitate to reach out to me if anything does not make sense or you need more information. Should we not run neard on Ubuntu 24.04, or is building with clang-17 the recommended path?
FYI: the error occured on versions 2.3.0, 2.3.1, 2.4.0, I have not tested other tags. For builds, we have used AMD Ryzen 9 5900X 12-Core Processor

Relevant log output

./neard run
2024-12-19T09:23:12.050482Z  INFO neard: version="2.4.0" build="2.4.0" latest_protocol=73
2024-12-19T09:23:12.050643Z  WARN neard: .near/config.json: encountered unrecognised fields: consensus.reduce_wait_for_missing_block, consensus.state_fetch_horizon
2024-12-19T09:23:12.050653Z  INFO config: Validating Config, extracted from config.json...
2024-12-19T09:23:12.052357Z  WARN genesis: Skipped genesis validation
2024-12-19T09:23:12.052367Z  INFO config: Validating Genesis config and records. This could take a few minutes...
2024-12-19T09:23:12.052530Z  INFO config: All validations have passed!
2024-12-19T09:23:12.054378Z  INFO neard: Reset the config "log_config.json" because the config file doesn't exist. err=Os { code: 2, kind: NotFound, message: "No such file or directory" }
2024-12-19T09:23:12.054430Z  WARN neard: config.json: encountered unrecognised fields: consensus.reduce_wait_for_missing_block, consensus.state_fetch_horizon
2024-12-19T09:23:12.054434Z  INFO config: Validating Config, extracted from config.json...
2024-12-19T09:23:12.054449Z  INFO neard: No validator key validator_key.json.
2024-12-19T09:23:12.054479Z  INFO near_o11y::reload: Updated the logging layer according to `log_config.json`
2024-12-19T09:23:12.054499Z  INFO db_opener: Opening NodeStorage path=".near/data" cold_path="none"
2024-12-19T09:23:12.054512Z  INFO db: Opened a new RocksDB instance. num_instances=1
2024-12-19T09:23:13.293482Z  INFO db: Closed a RocksDB instance. num_instances=0
2024-12-19T09:23:13.293501Z  INFO db_opener: The database exists. path=.near/data
2024-12-19T09:23:13.293515Z  INFO db: Opened a new RocksDB instance. num_instances=1
2024-12-19T09:23:14.919744Z  INFO db: Closed a RocksDB instance. num_instances=0
2024-12-19T09:23:14.919769Z  INFO db: Opened a new RocksDB instance. num_instances=1
2024-12-19T09:23:14.921869Z  INFO db: Closed a RocksDB instance. num_instances=0
2024-12-19T09:23:14.921880Z  INFO db: Opened a new RocksDB instance. num_instances=1
Segmentation fault