Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG][Opensearch] It is not possible to deploy the chart with the default settings. #617

Closed
LemonDouble opened this issue Nov 8, 2024 · 4 comments · Fixed by #618
Closed

[BUG][Opensearch] It is not possible to deploy the chart with the default settings. #617

LemonDouble opened this issue Nov 8, 2024 · 4 comments · Fixed by #618
Labels
bug Something isn't working

Comments

@LemonDouble
Copy link
Contributor

Describe the bug
When deploying the OpenSearch chart with default settings, an error occurs in the Security plugin, causing the deployment to fail.

To Reproduce

When deploying the default chart with only the OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable set, the deployment fails.

Error Log 
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-124-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.18.0
Detected OpenSearch Security Version: 2.18.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Nov 08, 2024 11:43:26 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-11-08T23:43:26,441][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[2.18.0], pid[1], build[tar/99a9a81da366173b0c2b963b26ea92e15ef34547/2024-10-31T19:08:39.157471098Z], OS[Linux/5.15.0-124-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.5/21.0.5+11-LTS]
[2024-11-08T23:43:26,442][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-11-08T23:43:26,442][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-9408234510311292452, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-11-08T23:43:26,561][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [opensearch-cluster-master-0] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2024-11-08T23:43:27,108][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2024-11-08T23:43:27,108][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2024-11-08T23:43:27,195][WARN ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of {}
java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?]
at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:95) ~[netty-handler-4.1.114.Final.jar:4.1.114.Final]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin$4.run(OpenSearchSecuritySSLPlugin.java:218) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:213) [opensearch-security-2.18.0.0.jar:2.18.0.0]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) [opensearch-security-2.18.0.0.jar:2.18.0.0]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.18.0.jar:2.18.0]
[2024-11-08T23:43:27,288][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
uncaught exception in thread [main]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions
at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?]
at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions]
at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165)
at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85)
at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137)
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93)
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
at org.opensearch.node.Node.<init>(Node.java:523)
at org.opensearch.node.Node.<init>(Node.java:450)
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

Expected behavior

The chart should be deployed successfully with the default settings.

Chart Name
opensearch

Host/Environment (please complete the following information):

  • Helm Version: 3.13.1
  • Kubernetes Version: 1.30.4+k3s1
  • Opensearch chart version : 2.27.0

Additional context

"It seems that the Security Plugin's demo configuration is not working because the opensearch.yml file is declared in the values.yaml."

In the Security plugin, if the opensearch.yml file is missing, it generates demo certificates. However, since the values.yaml file creates an opensearch.yml file, the plugin skips generating demo certificates, leading to this issue. (See Link)

Previously, the Security Plugin could not detect nested YAML, resulting in the demo configuration of the Security Plugin always being executed. However, it appears that this bug has been fixed with this PR(from Security Plugin release 2.18.0 onwards).

Therefore, it appears that this issue started occurring from the Opensearch chart version 2.27.0.

Would it be okay if I submit a PR for this issue? It seems that commenting out lines 49 to 99 in values.yaml would resolve the problem.
@LemonDouble LemonDouble added bug Something isn't working untriaged Issues that have not yet been triaged labels Nov 8, 2024
@LemonDouble LemonDouble mentioned this issue Nov 9, 2024
Merged
3 tasks
@siennathesane
Copy link

I ran into this problem and fixed it with:

singleNode: true
rbac:
  create: true
  automountServiceAccountToken: true
ingress:
  enabled: true
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    kubernetes.io/ingress.class: traefik
  hosts:
    - opensearch.127.0.0.1.sslip.io
config:
  opensearch.yml: "" #disable the default due to https://github.com/opensearch-project/helm-charts/issues/617
extraEnvs:
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: <some-password>

for Rancher Desktop

@prudhvigodithi prudhvigodithi removed the untriaged Issues that have not yet been triaged label Nov 21, 2024
@StefanSchuhart
Copy link

StefanSchuhart commented Nov 22, 2024 •
edited
Loading

Can someone give me a working values.yaml configuration? I am facing the same problem with chart versions 2.26.1 and 2.27.0 . I tried with different combinations of:

config:
  opensearch.yml: "" 

# or the default one

and with or without

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"

(Also tried version 2.25.0 and got a different error: java.lang.IllegalArgumentException: Could not load codec 'Lucene95'. Did you forget to add lucene-backward-codecs.jar?)

@ms-semarchy
Copy link

It works for me emptying opensearch.yml:

config:
  opensearch.yml: ""
singleNode: true

@LemonDouble
Copy link
Contributor Author

@StefanSchuhart Could you share the error message that occurs when opensearch.yml is left as an empty string?

@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Engineering Effectiveness Board Nov 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Engineering Effectiveness Board
Status: ✅ Done
Milestone
No milestone
5 participants
@siennathesane @prudhvigodithi @LemonDouble @StefanSchuhart @ms-semarchy