Command '['ninja', '-v']' returned non-zero exit status 1

[ccchobits]

Hi, we are facing the following issue while running Linux command line python script/run.py -c config/transductive/wn18rr.yaml --gpus [0]

Traceback (most recent call last):
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1667, in _run_ninja_build
    subprocess.run(
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "script/run.py", line 216, in <module>
    train_and_validate(cfg, model, train_data, valid_data, filtered_data=filtered_data)
  File "script/run.py", line 57, in train_and_validate
    pred = parallel_model(train_data, batch)
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/models.py", line 139, in forward
    output = self.bellmanford(data, h_index[:, 0], r_index[:, 0])  # (num_nodes, batch_size, feature_dim）
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/models.py", line 104, in bellmanford
    hidden = layer(layer_input, query, boundary, data.edge_index, data.edge_type, size, edge_weight)
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/layers.py", line 65, in forward
    output = self.propagate(input=input, relation=relation, boundary=boundary, edge_index=edge_index,
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/layers.py", line 89, in propagate
    out = self.message_and_aggregate(edge_index, **msg_aggr_kwargs)
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/layers.py", line 157, in message_and_aggregate
    from .rspmm import generalized_rspmm
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/rspmm/__init__.py", line 1, in <module>
    from .rspmm import generalized_rspmm
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/rspmm/rspmm.py", line 204, in <module>
    rspmm = load_extension("rspmm", [os.path.join(path, "rspmm.cpp"), os.path.join(path, "rspmm.cu")])
  File "/export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/rspmm/rspmm.py", line 199, in load_extension
    return cpp_extension.load(name, sources, extra_cflags, extra_cuda_cflags, **kwargs)
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1079, in load
    return _jit_compile(
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1292, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1404, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'rspmm': [1/2] /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc --generate-dependencies-with-compile --dependency-output rspmm.cuda.o.d -ccbin /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/bin/x86_64-conda_cos6-linux-gnu-cc -DTORCH_EXTENSION_NAME=rspmm -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include/TH -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.1/toolkit/11.1.0/include -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -std=c++14 -c /export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/rspmm/source/rspmm.cu -o rspmm.cuda.o
FAILED: rspmm.cuda.o
/cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc --generate-dependencies-with-compile --dependency-output rspmm.cuda.o.d -ccbin /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/bin/x86_64-conda_cos6-linux-gnu-cc -DTORCH_EXTENSION_NAME=rspmm -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include/TH -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.1/toolkit/11.1.0/include -isystem /export/home/s190009/myFile/tools/anaconda3/envs/py38pt180cuda111/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -std=c++14 -c /export/home/s190009/myFile/repository/python/NBFNet-PyG/nbfnet/rspmm/source/rspmm.cu -o rspmm.cuda.o
x86_64-conda_cos6-linux-gnu-cc: error trying to exec 'cc1plus': execvp: Permission denied
nvcc fatal   : Failed to preprocess host compiler properties.
ninja: build stopped: subcommand failed.

We created the required environment, which includes:

python                     3.8.11
torch                        1.8.0+cu111
torch-scatter            2.0.8
torch-sparse            0.6.12
torch-geometric      2.0.1
ninja                        1.10.2.3
easydict                   1.9
pyyaml                    5.4.1

And our gcc version is:

gcc version 7.3.0 (crosstool-NG 1.23.0.449-a04d0)

Could you please check out this issue, and provide some clues on how to resolve it? Thanks.

[sbonner0]

Seems you are missing the full compiler package and you are missing g++.

You could try installing gcc via conda if you don't have sudo rights to the underling machine.