Enable SSL in the Elastic Stack

[jsoriano]

Enable SSL in the Elastic Stack to ensure that everything works with recommended security settings.

Tasks:

• Create self-signed certificates
  • Use a common self-signed root CA?
  • Create a pair of key/cert per service
• Enable SSL in elasticsearch
• Enable SSL in package-registry
• Enable SSL in kibana
• Enable SSL in fleet-server
• Reduce use of insecure flags by configuring clients with proper CAs
• Ingestion to ES works (with hack)
• Check that elastic-package test works
  • Check with oldest supported stack version (7.14).
• Fix stack in 7.x
  • Templatize kibana config to use xpack.fleet.agents.elasticsearch.ca_sha256. This is not what I want to do, see comment.
• Configure Kubernetes agent
• Configure custom agent (introduced in Add config option to deploy custom elastic-agents as test services #786).
• Additional testing:
  • Check integrations CI with this branch Test integrations with SSL enabled in the stack integrations#3443.
  • Test with Windows.
  • Upgrade from version without SSL.
• Reorganize/reuse certpool builders
• Investigate problems with unexpectedly re-generated profile (and certificates). Seems solved at least since 64693ec.
• Integrate cert files better with the profile
• Keep support for non-SSL deployments? No.
• Print endpoints when elastic-package stack up completes. Print kibana and elasticsearch information on boot-up #861

Fixes #654

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-23T09:01:40.904+0000

• Duration: 36 min 5 sec

Test stats 🧪

Test	Results
Failed	0
Passed	747
Skipped	0
Total	747

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[v1v]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (31/31)	💚
Files	66.372% (75/113)	👍 0.3
Classes	61.392% (97/158)	👍 0.246
Methods	49.686% (316/636)	👍 0.56
Lines	33.345% (2847/8538)	👍 0.289
Conditionals	100.0% (0/0)	💚

[jsoriano]

Opening for review.

Additional testing on Windows would be nice, @ManojS-shetty has agreed to give it a try 🙂 Thanks!

[mtojek]

Minor comments

[mtojek]

Nice!

[ManojS-shetty]

Opening for review.

Additional testing on Windows would be nice, @ManojS-shetty has agreed to give it a try 🙂 Thanks!

@jsoriano Followed the below steps to test this on windows virtual machine:

• Generated the binary for elastic-package from the PR.
• Tried the command elastic-package stack up -d
• After few minutes i was able to hit the URL (https://localhost:5601)
• But i observed kibana container was going into unhealthy state.
• Below is the screenshot from the windows virtual machine.

[jsoriano]

But i observed kibana container was going into unhealthy state.

This looks like #856, let's confirm offline with logs.

[jsoriano]

After looking to Manoj's logs, I couldn't find a reason why it was failing, it didn't seem related to #856, more testing is going to be needed.

[jsoriano]

I did some manual testing on Windows and I found issues related to CRLF, see #869. After following the instructions added in this PR, elastic-package stack worked well for me with SSL.

[ManojS-shetty]

After following the steps in #869 . Able to run the elastic-package stack without any difficulty in windows development platform.

LGTM!

Thank you @jsoriano