Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For GraalVM libaws-crt-jni is not extracted to the same folder in which the native image is built #834

Closed
klopfdreh opened this issue Sep 26, 2024 · 20 comments
Labels
guidance Question that needs advice or information. p3 This is a minor priority issue

Comments

@klopfdreh
Copy link

klopfdreh commented Sep 26, 2024

Describe the bug

As of the manual https://github.com/awslabs/aws-crt-java?tab=readme-ov-file#graalvm-support libaws-crt-jni.so should be extracted to the same folder the native image is generated to, but it isn't.

The native-image command is executed in /native-image-build/ set via WORKDIR /native-image-build/ in the dockerfile.

The native image is then moved from native-image-build to native-image

Docker build output

STEP 33/38: RUN mv /native-image-build/libaws-crt-jni.so /native-image/libaws-crt-jni.so
mv: cannot stat '/native-image-build/libaws-crt-jni.so': Datei oder Verzeichnis nicht gefunden
subprocess exited with status 1
subprocess exited with status 1
Error: building at STEP "RUN mv /native-image-build/libaws-crt-jni.so /native-image/libaws-crt-jni.so": exit status 1

Datei oder Verzeichnis nicht gefunden = File or Directory not found

Expected Behavior

The libaws-crt-jni.so, in case of linux should be present in the same folder of the native image.

Current Behavior

No other files than the native image is present in the native image build folder.

Reproduction Steps

N/A

Possible Solution

N/A

Additional Information/Context

I tried to copy the JNDI lib because of the initial error

Caused by: java.lang.ExceptionInInitializerError
at software.amazon.awssdk.crt.CrtResource.<clinit>(CrtResource.java:104)
at [email protected]/java.lang.Class.ensureInitialized(DynamicHub.java:599)
at software.amazon.awssdk.services.s3.internal.crt.S3NativeClientConfiguration.<init>(S3NativeClientConfiguration.java:72)
at software.amazon.awssdk.services.s3.internal.crt.S3NativeClientConfiguration$Builder.build(S3NativeClientConfiguration.java:272)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.initializeS3CrtAsyncHttpClient(DefaultS3CrtAsyncClient.java:181)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.initializeS3AsyncClient(DefaultS3CrtAsyncClient.java:142)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.<init>(DefaultS3CrtAsyncClient.java:83)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.<init>(DefaultS3CrtAsyncClient.java:77)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$DefaultS3CrtClientBuilder.build(DefaultS3CrtAsyncClient.java:313)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$DefaultS3CrtClientBuilder.build(DefaultS3CrtAsyncClient.java:184)
at io.awspring.cloud.autoconfigure.s3.S3CrtAsyncClientAutoConfiguration.s3AsyncClient(S3CrtAsyncClientAutoConfiguration.java:85)
at io.awspring.cloud.autoconfigure.s3.S3CrtAsyncClientAutoConfiguration__BeanDefinitions.lambda$getSAsyncClientInstanceSupplier$1(S3CrtAsyncClientAutoConfiguration__BeanDefinitions.java:42)
at org.springframework.util.function.ThrowingBiFunction.apply(ThrowingBiFunction.java:68)
at org.springframework.util.function.ThrowingBiFunction.apply(ThrowingBiFunction.java:54)
at org.springframework.beans.factory.aot.BeanInstanceSupplier.lambda$get$2(BeanInstanceSupplier.java:206)
at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:58)
at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:46)
at org.springframework.beans.factory.aot.BeanInstanceSupplier.invokeBeanSupplier(BeanInstanceSupplier.java:219)
at org.springframework.beans.factory.aot.BeanInstanceSupplier.get(BeanInstanceSupplier.java:206)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.obtainInstanceFromSupplier(DefaultListableBeanFactory.java:949)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.obtainFromSupplier(AbstractAutowireCapableBeanFactory.java:1237)
... 16 more
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Failed to load 'libaws-crt-jni.so'. Make sure this file is in the same directory as the GraalVM native image. UNKNOWN(-1)
at software.amazon.awssdk.crt.CRT.<clinit>(CRT.java:46)

aws-crt-java version used

0.31.1

Java version used

Java 17

Operating System and version

ubi 9

@klopfdreh klopfdreh added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 26, 2024
@klopfdreh
Copy link
Author

klopfdreh commented Sep 26, 2024

As a workaround I copied the libaws-crt-jni.so which can be used of our architecture from out of the aws-crt-0.31.1.jar.

@TingDaoK
Copy link
Contributor

Can you provide more detail of the build steps? Does the GraalVMNativeFeature load correct? Any failure during the native image build?

@klopfdreh
Copy link
Author

klopfdreh commented Sep 26, 2024

Hey @TingDaoK - our setup looks like this (this demo does not use AWS CRT but you could simply add it to the Spring Boot application: https://github.com/klopfdreh/native-cloud-config-test/blob/main/client/Dockerfile

There are no issues during the build and the native image is created without any hints that something went wrong.

We only received an exception during the runtime like mentioned in the ticket. After that I thought that the lib has to be in the same folder as the native image but it wasn‘t.

When we provided the lib and copied it beside the native image everything was working without any issues.

My assumption: There might be an issue while copying the file based on the architecture or during the detection of the architecture.

@jmklix jmklix added p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Sep 26, 2024
@TingDaoK
Copy link
Contributor

Can you share Docker build output for the native-image build process?

@klopfdreh
Copy link
Author

Sure - I have to trigger a build at work, tomorrow.

@klopfdreh
Copy link
Author

klopfdreh commented Sep 27, 2024

Here is the log of the native-image build

STEP 30/38: RUN native-image --no-fallback -march=x86-64 -Djavax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts --initialize-at-build-time=`cat ./InitializeAtBuildTime | tr '\n' ','` --initialize-at-run-time=`cat ./InitializeAtRunTime | tr '\n' ','` --enable-https --enable-url-protocols=https -H:ResourceConfigurationResources=META-INF/native-image/software.amazon.awssdk/sdk-core/resource-config.json -H:ReflectionConfigurationResources=META-INF/native-image/software.amazon.awssdk/apache-client/reflect-config.json -H:ResourceConfigurationResources=META-INF/native-image/software.amazon.awssdk/apache-client/resource-config.json -H:DynamicProxyConfigurationResources=META-INF/native-image/software.amazon.awssdk/apache-client/proxy-config.json -H:ResourceConfigurationResources=META-INF/native-image/s3-downloader-custom-definitions/resource-config.json -H:ReflectionConfigurationResources=META-INF/native-image/s3-downloader-custom-definitions/reflect-config.json -H:+UnlockExperimentalVMOptions --install-exit-handlers -cp .:BOOT-INF/classes:`find BOOT-INF/lib | tr '\n' ':'` ${MAIN_CLASS} -o ${BINARY_NAME}
Warning: Using a deprecated option --allow-incomplete-classpath from 'META-INF/native-image/software.amazon.awssdk/netty-nio-client/native-image.properties' in 'file:///native-image-build/BOOT-INF/lib/netty-nio-client-2.28.6.jar'. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
Warning: Using a deprecated option --allow-incomplete-classpath from 'META-INF/native-image/native-image.properties' in 'file:///native-image-build/BOOT-INF/lib/ojdbc11-23.5.0.24.07.jar'. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/software.amazon.awssdk/apache-client/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/software.amazon.awssdk/apache-client/resource-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:DynamicProxyConfigurationResources=META-INF/native-image/software.amazon.awssdk/apache-client/proxy-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/software.amazon.awssdk/sdk-core/resource-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/s3-downloader-custom-definitions/resource-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/s3-downloader-custom-definitions/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 's3-downloader' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                   (12,0s @ 0,37GB)
 Java version: 22.0.2+11, vendor version: Liberica-NIK-24.0.2-1
 Graal compiler: optimization level: 2, target machine: x86-64
 C compiler: gcc (redhat, x86_64, 11.4.1)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 4 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - oracle.jdbc.nativeimage.NativeImageFeature
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
 - org.springframework.aot.nativex.feature.PreComputeFieldFeature
------------------------------------------------------------------------------------------------------------------------
 4 experimental option(s) unlocked:
 - '-H:DynamicProxyConfigurationResources': Use a proxy-config.json in your META-INF/native-image/<groupID>/<artifactID> directory instead. (origin(s): command line)
 - '-H:ServiceLoaderFeatureExcludeServices' (origin(s): 'META-INF/native-image/org.springframework/spring-orm/native-image.properties' in 'file:///native-image-build/BOOT-INF/lib/spring-orm-6.1.13.jar')
 - '-H:ResourceConfigurationResources': Use a resource-config.json in your META-INF/native-image/<groupID>/<artifactID> directory instead. (origin(s): command line, command line, command line, 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/native-image.properties' in 'file:///native-image-build/BOOT-INF/lib/tomcat-embed-el-10.1.30.jar')
 - '-H:ReflectionConfigurationResources': Use a reflect-config.json in your META-INF/native-image/<groupID>/<artifactID> directory instead. (origin(s): command line, command line, 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/native-image.properties' in 'file:///native-image-build/BOOT-INF/lib/tomcat-embed-el-10.1.30.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 9,07GB of memory (75,6% of 12,00GB system memory, determined at start)
 - 5 thread(s) (100,0% of 5 available processor(s), determined at start)
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
Warning: RecomputeFieldValue.ArrayIndexScale automatic field value transformation failed. The automatic registration was attempted because a call to jdk.internal.misc.Unsafe.arrayIndexScale(Class) was detected in the class initializer of io.netty.util.internal.shaded.org.jctools.util.UnsafeLongArrayAccess. Detailed failure reason(s): Could not determine the field where the value produced by the call to jdk.internal.misc.Unsafe.arrayIndexScale(Class) for the array index scale computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 
[2/8] Performing analysis...  [******]                                                                 (123,5s @ 5,92GB)
   51.119 reachable types   (92,0% of   55.559 total)
   85.205 reachable fields  (59,4% of  143.387 total)
  254.993 reachable methods (62,5% of  408.081 total)
   15.931 types, 2.507 fields, and 18.087 methods registered for reflection
      166 types,   280 fields, and   133 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (21,1s @ 4,99GB)
[4/8] Parsing methods...      [***]                                                                      (9,7s @ 5,55GB)
[5/8] Inlining methods...     [***]                                                                      (6,3s @ 6,02GB)
[6/8] Compiling methods...    [*********]                                                               (82,5s @ 4,11GB)
[7/8] Laying out methods...   [****]                                                                    (15,7s @ 5,58GB)
[8/8] Creating image...       [****]                                                                    (14,0s @ 6,76GB)
 117,80MB (47,77%) for code area:   169.402 compilation units
 128,44MB (52,08%) for image heap:  962.476 objects and 5.624 resources
 381,15kB ( 0,15%) for other data
 246,62MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  17,23MB ojdbc11-23.5.0.24.07.jar                            36,61MB byte[] for code metadata
  16,56MB java.base                                           26,49MB byte[] for embedded resources
  15,50MB hibernate-core-6.5.3.Final.jar                      15,87MB byte[] for java.lang.String
   7,14MB regions-2.28.6.jar                                  13,37MB java.lang.Class
   4,96MB s3-2.28.6.jar                                        9,15MB java.lang.String
   4,66MB java.desktop                                         4,29MB com.oracle.svm.core.hub.DynamicHubCompanion
   4,08MB java.xml                                             3,48MB byte[] for reflection metadata
   2,90MB jdk.proxy4                                           2,28MB java.lang.reflect.Method
   2,30MB jackson-databind-2.17.2.jar                          1,81MB java.lang.String[]
   2,28MB svm.jar (Native Image)                               1,80MB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
  38,95MB for 323 more packages                               13,29MB for 10716 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
                      35,0s (12,0% of total time) in 1712 GCs | Peak RSS: 8,47GB | CPU load: 3,91
------------------------------------------------------------------------------------------------------------------------
Build artifacts:
 /native-image-build/libawt.so (jdk_library)
 /native-image-build/libawt_headless.so (jdk_library)
 /native-image-build/libawt_xawt.so (jdk_library)
 /native-image-build/libfontmanager.so (jdk_library)
 /native-image-build/libfreetype.so (jdk_library)
 /native-image-build/libjava.so (jdk_library_shim)
 /native-image-build/libjavajpeg.so (jdk_library)
 /native-image-build/libjvm.so (jdk_library_shim)
 /native-image-build/liblcms.so (jdk_library)
 /native-image-build/s3-downloader (executable)
========================================================================================================================
Finished generating 's3-downloader' in 4m 50s.
STEP 31/38: RUN mkdir -p /native-image/
STEP 32/38: RUN mv /native-image-build/${BINARY_NAME} /native-image/${BINARY_NAME}
STEP 33/38: COPY ./s3-downloader/target/classes/libaws-crt-jni.so /native-image/
STEP 34/38: WORKDIR /native-image/
STEP 35/38: RUN rm -Rf /native-image-build/

At Build artifacts: the libaws-crt-jni.so is missing, so I guess this might be the issue.

The maven dependency is used as it should be.
image

@TingDaoK
Copy link
Contributor

 4 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - oracle.jdbc.nativeimage.NativeImageFeature
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
 - org.springframework.aot.nativex.feature.PreComputeFieldFeature

Looks like the --features=software.amazon.awssdk.crt.internal.GraalVMNativeFeature was not picked up correctly from native-image

@TingDaoK
Copy link
Contributor

I am not sure why. Maybe the native-image.properties is not configured correctly? Looks like the feature got picked up is pretty similar, here.

But I think we can start with trying to add --features=software.amazon.awssdk.crt.internal.GraalVMNativeFeature to native-image build commend args?

In the mean time, I'll try to see if I can reproduce it. From the docker file, looks like you build the jar first, then use native-image to build the binary, right?

@klopfdreh
Copy link
Author

Yes we build the jar with spring-boot-maven-plugin and compile native image like explained here: https://docs.spring.io/spring-boot/reference/packaging/native-image/advanced-topics.html#packaging.native-image.advanced.converting-executable-jars.native-image

@klopfdreh
Copy link
Author

I am not sure why. Maybe the native-image.properties is not configured correctly? Looks like the feature got picked up is pretty similar, here.

But I think we can start with trying to add --features=software.amazon.awssdk.crt.internal.GraalVMNativeFeature to native-image build commend args?

In the mean time, I'll try to see if I can reproduce it. From the docker file, looks like you build the jar first, then use native-image to build the binary, right?

I am going to try out this argument during a build on monday. 👍

@TingDaoK
Copy link
Contributor

TingDaoK commented Sep 27, 2024

I tried to add CRT to the sample you linked above.
TingDaoK/native-cloud-config-test@1daf6c4#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R20

But I cannot reproduce the issue.

Looks like it successfully load the feature and extracted the shared library.

#20 [16/22] RUN jar -xvf native-cloud-config-test-client-1.0.0-SNAPSHOT-exe.jar
...
#20 0.234 extracted: BOOT-INF/lib/aws-crt-0.31.1.jar
...

#21 [17/22] RUN native-image     --initialize-at-build-time=`cat ./InitializeAtBuildTime | tr '\n' ','`     --initialize-at-run-time=`cat ./InitializeAtRunTime | tr '\n' ','`     --no-fallback     --enable-https     --install-exit-handlers     -H:Name=config-client-application     -cp .:BOOT-INF/classes:`find BOOT-INF/lib | tr '\n' ':'` nat.cloud.config.test.client.ConfigClientApplication
...
#21 5.127  3 user-specific feature(s)
#21 5.128  - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
#21 5.128  - org.springframework.aot.nativex.feature.PreComputeFieldFeature
#21 5.128  - software.amazon.awssdk.crt.internal.GraalVMNativeFeature
...

#22 [18/22] RUN ls
...
#22 0.111 config-client-application
#22 0.111 libaws-crt-jni.so
...

@klopfdreh
Copy link
Author

Ah maybe it is because I use Spring Cloud AWS and it is not initializing AWS CRT the right way.

I am going to add „new CRT()“ to a code which is initialized during build time.

I am going to report back if this is working for me.

@TingDaoK
Copy link
Contributor

I don't think new CRT() is related, I added it to verify that it can run successfully at runtime.
I tried to remove the code, it still extracted the shared lib.
Not sure why, but yeah, we can start from the sample and see what really cause the feature to not be loaded in your use case.

@klopfdreh
Copy link
Author

klopfdreh commented Sep 28, 2024

I may found the issue.

The S3CrtAsyncClientAutoConfiguration checks if S3AsyncClient is available: https://github.com/awspring/spring-cloud-aws/blob/main/spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/s3/S3CrtAsyncClientAutoConfiguration.java#L47

If so it uses crtBuilder() https://github.com/awspring/spring-cloud-aws/blob/main/spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/s3/S3CrtAsyncClientAutoConfiguration.java#L68

When you just use

<dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-aws-dependencies</artifactId>
    <version>${spring-cloud-aws-dependencies.version}</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

and

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
</dependency>

the following dependency is not automatically included and because of this the feature is not used.

<dependency>
    <groupId>software.amazon.awssdk.crt</groupId>
    <artifactId>aws-crt</artifactId>
</dependency>

For normal java environments this is not causing any issues, because you can just use s3 without aws-crt even if you use crtBuilder(). But if you use native images s3 requires you to manually include aws-crt when you use crtBuilder(), because the feature is not picked up otherwise.

Maybe it would be good to add a hint to the readme that if you use s3 with crtBuilder() you manually have to add the aws-crt dependency for native images. Otherwise I would suggest that the s3 should automatically ship aws-crt

So:
java: aws-crt + s3 + crtBuilder() ✅
java: s3 + crtBuilder() ✅
native: aws-crt + s3 + crtBuilder() ✅
native: s3 + crtBuilder() ❌

@TingDaoK
Copy link
Contributor

TingDaoK commented Sep 30, 2024

software.amazon.awssdk.s3 takes aws-crt as optional dependency. https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/pom.xml#L113-L119

So, from my understanding, if you need to use aws-crt related functionality and only takes dependencies on software.amazon.awssdk.s3, it will not include aws-crt, and SHOULD fail for both JVM and GraalVM?

So, IMHO, it's kind expected to have aws-crt declared as the dependency separately from software.amazon.awssdk.s3. eg: the README from s3-transfer-manager, which also takes aws-crt as optional dependency, indicates that you need to add aws-crt as dependency as well.

@klopfdreh
Copy link
Author

So, from my understanding, if you need to use aws-crt related functionality and only takes dependencies on software.amazon.awssdk.s3, it will not include aws-crt, and SHOULD fail for both JVM and GraalVM?

That’s what I meant with „java: s3 + crtBuilder()“ and it was working for me without the aws-crt

@klopfdreh
Copy link
Author

Anyway the original issue was solved by explicitly adding aws-crt 👍

@TingDaoK
Copy link
Contributor

That’s what I meant with „java: s3 + crtBuilder()“ and it was working for me without the aws-crt

Perhaps you have aws-crt installed to your local maven environment? In theory, it should not work for JVM as well.
Anyway, glad that we find the root cause.
I'll close the issue unless you think further discussion/action needed.

@TingDaoK TingDaoK added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Sep 30, 2024
@klopfdreh
Copy link
Author

Thanks so much for all the time and the clarification!

@TingDaoK TingDaoK added guidance Question that needs advice or information. and removed bug This issue is a bug. closing-soon This issue will automatically close in 4 days unless further comments are made. labels Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
guidance Question that needs advice or information. p3 This is a minor priority issue
Projects
None yet
Development

No branches or pull requests

3 participants