Skip to content

Commit

Permalink
Make it possible to run 'make benchmark-%' using jvm 17+ (apache#823)
Browse files Browse the repository at this point in the history
When using jvm 17+ spark needs extra jvm args to avoid getting errors
like
```
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.2.0:java (default-cli) on project comet-spark-spark3.4_2.12: An exception occurred while executing the Java class. class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x35b75242) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x35b75242 -> [Help 1]
```

These args are already present inside the main pom.xml. To avoid
duplicating the args we using maven to extract them. In order avoid
slowing down the Makefile the args are defined as a function and
therefore only evaluated when they are needed.
  • Loading branch information
eejbyfeldt authored Aug 13, 2024
1 parent 9d4afc1 commit a0734dc
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

.PHONY: all core jvm test clean release-linux release bench

define spark_jvm_17_extra_args
$(shell ./mvnw help:evaluate -Dexpression=extraJavaTestArgs | grep -v '\[')
endef

all: core jvm

core:
Expand Down Expand Up @@ -80,7 +84,7 @@ release:
release-nogit:
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo build --release
./mvnw install -Prelease -DskipTests $(PROFILES) -Dmaven.gitcommitid.skip=true
benchmark-%: clean release
cd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g' ../mvnw exec:java -Dexec.mainClass="$*" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="$(filter-out $@,$(MAKECMDGOALS))" $(PROFILES)
benchmark-%: release
cd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g ${call spark_jvm_17_extra_args}' ../mvnw exec:java -Dexec.mainClass="$*" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="$(filter-out $@,$(MAKECMDGOALS))" $(PROFILES)
.DEFAULT:
@: # ignore arguments provided to benchmarks e.g. "make benchmark-foo -- --bar", we do not want to treat "--bar" as target

0 comments on commit a0734dc

Please sign in to comment.