libclang.so becomes difficult to get hold of (and completely missing for x86_64 Windows/Linux/Apple in the latest 18.1.2 clang distro) (LLVM-420)

[ivmarkov]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

I expect the <ESP-IDF-DIR>/tools/tools.json to have a tool ("clang" or "clang-libs" or similar) that does contain the libclang.so/.dll for all major operating systems, including Windows.

Actual behavior (suspected bug)

Good:
Versions of ESP-IDF <= 5.3 contain the "esp-clang" tool (for versions < 5.1 it is named differently - "xtensa-clang" but is still OK), which - inside - the TGZ tarball does contain both the clang binaries (which we don't need) as well as the libclang.so/dll library. So by downloading this tool, esp-idf-sys can make sure it does have a proper clang library for Rust "bindgen" ("bindgen" uses the library, not the binary)

Not good:
Versions of ESP IDF >= 5.4 still contain the "esp-clang" tool as per above, however with version updated to clang-esp-18.1.2_20240912. This clang version has multiple issues:

1. It does not contain the libclang.so lib, which makes it backwards incompatible, as found out here. Now, if there is a separate "libs" tarball introduced, the minimum I would expect from such a backwards incompatible change is that there would be a new "tool" in tools.json which is named esp-clang-libs or something, which I can instruct esp-idf-sys to download for newer ESP-IDF versions

2. However, what is worse is that there is no "libs" tarball for Windows/Linux/Mac x86_64 in the current 18.1.2_20240912 clang distro. How are we supposed to build esp-idf-sys + ESP IDF >= 5.4 then? Sure, I can use an external "clang" for the RiscV targets (though that's very suboptimal as we otherwise use all the ESP IDF tooling for everything else) but what about the xtensa chips?

See also:

• Build failed due to failure of loading libclang.dll when generating bindings esp-rs/espup#440 - when the problem started surfacing
• esp-17.0.1_20240419 full toolchain tarball doesn't contain libclang (LLVM-352) #99 - problem reported for the first time, but in a different context (espup)

Error logs or terminal output

No response

Steps to reproduce the behavior

1. Look at the tools.json of the cited ESP-IDF versions
2. Look at this download location for the Espressif clang toolchains: https://github.com/espressif/llvm-project/releases/

Project release version

ESP IDF 5.4 or higher

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

Windows/Linux/Mac x86_64

Operating system version

All

Shell

ZSH

Additional context

No response

[ivmarkov]

Updated the issue description, because it turns out libclang.so is missing not just for Windows, but for all of x86_64 - Linux, Mac, Windows...

[gerekon]

@ivmarkov libs are disributed in separate archives for all supported platforms. Please, look for libs-clang-esp* distros here.

I agree with you regarding item (1). After we moved to statically linked LLVM toolchain we need to add esp-clang-libs to tools.json. I will prepare MR for that.

[gerekon]

After thinking a bit on it...

@igrr @dobairoland Do you think it is acceptable to add esp-clang-libs distros to tools.json? I do not see big problem to keep an entry for those libs there as far as it should be installed on demand, but the problem is that we need to retrieve version for installed distro. For executables it is easy, just run it with --version option, but for libraries we can not do this. So we can not distinguish between multiple installed versions of esp-clang-libs and compose proper path to the distro under $HOME/.espressif/tools.

1. Possible solution is to automatically install esp-clang along with the libs upon request and run clang --version. So having the following version_cmd in tools.json for esp-clang-libs. Do we have a kind of deps tracking for tools.json?

      "version_cmd": [
        "clang",
        "--version"
      ],

2. Another solution is to add small tool to the libs distro to print version. So version_cmd in tools.json would look like below.

      "export_paths": [
        [
          "esp-clang-libs",
          "bin"
        ]
      ],
      "version_cmd": [
        "esp-clang-libs-version"
      ],

3. esp-idf-sys can do it itself.

• Download esp-clang (or install it via idf_tools.py).
• Run clang --version
• Download esp-clang-libs distro basing on clang version. URL for Clang can be retrieved from tools.json and clang distro filename should be prefixed with libs-. @dobairoland Is there any command for idf_tools.py to print URL for the tool distro?

Taking into account that none else needs those libraries in tools.json solution (3) seems preferable to me at this moment.

[ivmarkov]

@ivmarkov libs are disributed in separate archives for all supported platforms. Please, look for libs-clang-esp* distros here.

I'm an idiot, sorry :-) I've overlooked the "Show all 16 assets" link below the table.

[ivmarkov]

After thinking a bit on it...

@igrr @dobairoland Do you think it is acceptable to add esp-clang-libs distros to tools.json? I do not see big problem to keep an entry for those libs there as far as it should be installed on demand, but the problem is that we need to retrieve version for installed distro. For executables it is easy, just run it with --version option, but for libraries we can not do this. So we can not distinguish between multiple installed versions of esp-clang-libs and compose proper path to the distro under $HOME/.espressif/tools.

@gerekon Hmmm, is this really how tools are installed?

My understanding was that tools are installed in a sub-folder based on this version name field and the "version"/"version-regex" fields in the json (which by the way can be empty and are empty for non-elf tooling like e.g. "esp-rom-elfs") are only used for "testing"?

[gerekon]

I'm an idiot, sorry :-) I've overlooked the "Show all 16 assets" link below the table.

Sometimes I do the same :-)

My understanding was that tools are installed in a sub-folder based on this version name field

Yes, but I suppose that version_cmd is used to check the installed version when you do . ./export.sh and ask you to install the one which matches current IDF version. Regarding esp-rom-elfs. I suppose that ROMs are valid for all IDF versions because they are not changed from IDF to IDF, they are chip specific, i.e. ESP32-S3 ROM is the same for any IDF version.

In your case you want to get clang libs matching installed IDF version. This seems to be different case.

[dobairoland]

Do you think it is acceptable to add esp-clang-libs distros to tools.json?

Yes, but currently we are missing your requirements mentioned later.

As for the version command, I think you could put a simple Python one-liner which could read a file and print the library version. It could be something else than Python but it has to be cross-platform and already installed.

Do we have a kind of deps tracking for tools.json?

No. Every item is independent right now.

Is there any command for idf_tools.py to print URL for the tool distro?

Not really because the tools intention is not to let the downloading to other tools.

[gerekon]

As for the version command, I think you could put a simple Python one-liner which could read a file and print the library version. It could be something else than Python but it has to be cross-platform and already installed.

Ok. In the next release I will try to add simple Python script which will keep the version and print it when invoked.
After that I will add esp-clang-libs to tools,json