How to properly specify s3 credentials for mainApplicationFile in Spark operator?

[paulpaul1076]

I have this mainApplicationFile config:

mainApplicationFile: s3a://obs-zdp-warehouse-stage-mz/firestarter-1.0.0.jar

I specified an s3connection for it:

  s3connection:
    inline:
      host: url.....
      port: 443
      accessStyle: Path
      credentials:
        secretClass: s3-credentials-class

I also created 2 files:

1) s3-credentials.yml:
apiVersion: v1
kind: Secret
metadata:
 name: s3-credentials
 labels:
   secrets.stackable.tech/class: s3-credentials-class
stringData:
 accessKey: *****
 secretKey: *****

2)  s3-credentials-class.yml

apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
name: s3-credentials-class
spec:
backend:
  k8sSearch:
    searchNamespace:
      pod: {}

But with this config there's nothing happening, the pod that's supposed to start the spark application is in stuck "Pending" state.

Optional info:

I previously used (and it worked):

spec:
  env:
    -name:AWS_ACCESS_KEY_ID
    value:"***"
    -name:AWS_SECRET_ACCESS_KEY
    value:"***"

But now, for some reason it doesn't work properly when using multiple buckets in my Spark application. For some reason it applies these keys to all buckets that I use and I get an error, saying that my access and secret key is wrong.

I did specify

spark.hadoop.fs.s3a.bucket.bucket1.access.key
spark.hadoop.fs.s3a.bucket.bucket1.secret.key

spark.hadoop.fs.s3a.bucket.bucket2.access.key
spark.hadoop.fs.s3a.bucket.bucket2.secret.key|

in my spark config, but it still seems to use the keys from env for some weird reason.

I tried:

spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Any ideas on how to resolve this?

[adwk67]

What does the whole SparkApplication yaml look like (redacted where need be)? In a previous version we set the env variables, but don't use them in 23.11 (nor possibly earlier). And the credentials provider is set automatically depending on whether an S3 connection is detected: see https://github.com/stackabletech/spark-k8s-operator/blob/release-23.11/rust/crd/src/lib.rs#L443-L457.

[adwk67]

For each distinct bucket you'll need something like this:

spark.hadoop.fs.s3a.bucket.{bucket_name}.endpoint=...
spark.hadoop.fs.s3a.bucket.{bucket_name}.path.style.access=true
spark.hadoop.fs.s3a.bucket.{bucket_name}.access.key=...
spark.hadoop.fs.s3a.bucket.{bucket_name}.secret.key=...
spark.hadoop.fs.s3a.bucket.{bucket_name}.path.style.access=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

[paulpaul1076]

@adwk67 it works now, I was passing spark parameters incorrectly, sorry!

The whole file looks like this:

`apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
name: spark-action-event11

namespace: spark
spec:
version: "1.0"
mainApplicationFile: s3a://obs-zdp-warehouse-stage-mz/firestarter-1.0.0.jar
mainClass: com.zvuk.firestarter.spark.batch.job.impl.cs.v3.ClickstreamV3AppActionEventToIcebergBatchJob
args:
- "env=stage"
- "token=s.ZIUHmKckS0FXiKvVitlEg1wT"
- "event_date=2023-08-17"
sparkImage:
productVersion: 3.5.0
pullPolicy: IfNotPresent
mode: cluster
sparkConf:
spark.driver.extraClassPath: /dependencies/jars-apd/*
spark.driver.extraJavaOptions: -Divy.cache.dir=/home/stackable/.ivy2 -Divy.home=/home/stackable/.ivy2
spark.executor.extraClassPath: /dependencies/jars-apd/*
spark.executor.extraJavaOptions: -Divy.cache.dir=/home/stackable/.ivy2 -Divy.home=/home/stackable/.ivy2
spark.hadoop.fs.s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.access.key: ***
spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
spark.hadoop.fs.s3a.bucket.obs-de-cicd-mz.access.key: ***
spark.hadoop.fs.s3a.bucket.obs-de-cicd-mz.secret.key: ***
spark.hadoop.fs.s3a.bucket.obs-de-export-mz.access.key: ***
spark.hadoop.fs.s3a.bucket.obs-de-export-mz.secret.key: ***
spark.hadoop.fs.s3a.bucket.obs-de-temporary-mz.access.key: ***
spark.hadoop.fs.s3a.bucket.obs-de-temporary-mz.secret.key: ***
spark.hadoop.fs.s3a.bucket.obs-zdp-warehouse-stage-mz.access.key: ***
spark.hadoop.fs.s3a.bucket.obs-zdp-warehouse-stage-mz.secret.key: ***
spark.hadoop.fs.s3a.endpoint: ***
spark.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.secret.key: ***
spark.kubernetes.memoryOverheadFactor: '0.2'
spark.sql.ansi.enabled: 'true'
spark.sql.catalog.iceberg_catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.iceberg_catalog.s3.access.key: ***
spark.sql.catalog.iceberg_catalog.s3.delete-enabled: 'false'
spark.sql.catalog.iceberg_catalog.s3.endpoint: ***
spark.sql.catalog.iceberg_catalog.s3.path-style-access: 'true'
spark.sql.catalog.iceberg_catalog.s3.secret.key: ***
spark.sql.catalog.iceberg_catalog.s3a.path-style-access: 'true'
spark.sql.catalog.iceberg_catalog.type: hive
spark.sql.catalog.iceberg_catalog.uri: thrift://thrift-hive-stage-v2.zvq.me:9083
spark.sql.catalog.iceberg_catalog.warehouse: s3a://obs-zdp-warehouse-stage-mz/
spark.sql.catalog.iceberg_catalog: org.apache.iceberg.spark.SparkCatalog
spark.sql.catalogImplementation: hive
spark.sql.debug: 'false'
spark.sql.defaultCatalog: iceberg_catalog
spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.storeAssignmentPolicy: ANSI
spark.ui.proxyBase: /sparkui/spark-example
spark.ui.proxyRedirectUri: /

logFileDirectory:
s3:
prefix: eventlogs/
bucket:
inline:
bucketName: obs-zdp-warehouse-stage-mz
connection:
inline:
host: ***
tls:
verification:
server:
caCert:
webPki: {}
port: 443
accessStyle: Path
credentials:
secretClass: obs-s3-credentials-secret
volumes:
- name: job-deps
persistentVolumeClaim:
claimName: spark-pvc
env:
- name: AWS_ACCESS_KEY_ID
value: ""
- name: AWS_SECRET_ACCESS_KEY
value: ""

driver:
config:
resources:
cpu:
min: '250m'
max: "1"
memory:
limit: '8Gi'
volumeMounts:
- name: job-deps
mountPath: /dependencies/jars-apd
executor:
replicas: 3
config:
resources:
cpu:
min: '500m'
max: "3"
memory:
limit: '4Gi'
volumeMounts:
- name: job-deps
mountPath: /dependencies/jars-apd`

[adwk67]

Glad it's working!