Elastic Package signature

[jlind23]

We have the idea of signing packages in order to have a proof that a package is truly coming from Elastic developers.

The question here is - What if the keys used to sign packages have to be changed? (Updating for security purpose for example)

How should we communicate this change and how to handle package that we signed with the old keys?

[jlind23]

@akshay-saraswat from a user perspective, what is your opinion there?

[mtojek]

Side note:

Public keys in RPMS repositories are exposed inside. For example: https://evebox.org/files/rpm/development/RPM-GPG-KEY-evebox

Sample CRL file from Google: https://crls.pki.goog/gts1c3/QOvJ0N1sT2A.crl

[akshay-saraswat]

@akshay-saraswat from a user perspective, what is your opinion there?

My opinion as an Elastic user advocate.

Will we change the keys regularly?
It’s a good practice to keep rotating the encryption keys at regular intervals. But in this case, it may not be feasible to rotate the keys at regular intervals especially because we will have two different types of operational environments to support that will become difficult to keep in sync.

Where will the public and private keys reside and who will have access to them?

• The Private key should reside in a vault. Only the Elastic Infra team and a special signing pipeline chained after master build should have access to it.
• The Public key should be bundled with Kibana and should be available on the Downloads page so that it is easily available for updates from an authentic source.
• The Elastic Package Registry should have a public key checksum for validation of the public key itself in Kibana.

What if the keys used to sign packages have to be changed?
In rare occurrences, if we lose the private key or if it gets exposed, we will have to update the key pair and communicate the same to our customers shortly after. As soon as the risk is identified, we must generate a new pair and re-sign all the packages with the new key immediately.

• For Elastic Cloud deployments with internet access, this change must get reflected immediately because Integrations UI can check the sanity of the public key every time it receives a new list of integrations.
• For Elastic Cloud deployed in air-gapped environments, a platform admin should have to download the new public key from the official downloads page and update it in Integrations UI settings. This public key update must be accompanied by the update of the self-managed EPR hosting new signed packages.

How will we communicate the key update?
Key updates must be communicated to all the customers via an announcement email sent by their assigned account executives or SAs. The same announcement should also be published as a blog on our marketing website and in the Elastic community Slack. Also,

• SaaS Elastic Cloud users must see an error message if they navigate to the integrations UI page without manually downloading the public key from Elastic’s official downloads page and updating it into their Elastic cloud deployment.
• Elastic cloud users operating in the air-gapped environment should not see any error until they choose to upgrade either self-managed EPR or public key. And they should not see any error if they update both of them simultaneously.

[jlind23]

we must generate a new pair and re-sign all the packages with the new key immediately.

As signature will happens during build, it means that we must rebuild and push again all packages right? Should we immediately disabled all packages download or should we wait the end of the build pipeline?

[mtojek]

The question here is - What if the keys used to sign packages have to be changed? (Updating for security purpose for example)

It isn't so uncommon to happen. Check out the news:

https://threatpost.com/d-link-accidentally-leaks-private-code-signing-keys/114727/
https://www.bleepingcomputer.com/news/security/eu-investigating-leak-of-private-key-used-to-forge-covid-passes/
https://www.eweek.com/security/sony-left-passwords-code-signing-keys-virtually-unprotected/
https://www.theregister.com/2021/04/26/hashicorp_reveals_exposure_of_private/

How should we communicate this change and how to handle package that we signed with the old keys?

I will bring the quote from the link above:

Kaspersky Lab and other security experts have criticized Sony Pictures for not immediately revoking the stolen certificates. “Certificate revocation needs to be a top priority when responding to a major malware and breach incidents,” the company said.

If we decide to be in charge of managing the sub-keys for signing packages, we can prepare a procedure for revoking keys, rotating them, and resigning artifacts. If the responsibility stays on the Infra side, we need to coordinate the process with them or a security engineer.

I also looked deeper and it seems that's mandatory to resign all artifacts just for safety purposes. It's our responsibility to be able to do this in a reasonable time. I suppose it's a matter of preparing procedures.

It’s a good practice to keep rotating the encryption keys at regular intervals. But in this case, it may not be feasible to rotate the keys at regular intervals especially because we will have two different types of operational environments to support that will become difficult to keep in sync.

It isn't a good practice to rotate the signing keys. For example, Google Play suggests setting the expiry period to at least 25 years.

The Private key should reside in a vault. Only the Elastic Infra team and a special signing pipeline chained after master build should have access to it.

Closer to the machine access only, the better.

The Public key should be bundled with Kibana and should be available on the Downloads page so that it is easily available for updates from an authentic source.

@jsoriano mentioned investigating an option of running a Keyserver. See https://keyserver.ubuntu.com . AFAIK we're using this one: https://pgp.mit.edu . That's a job for you, @akshay-saraswat , to find the owner of this process :)

Should we also keep and expose it in the Package Storage?

The Elastic Package Registry should have a public key checksum for validation of the public key itself in Kibana.

So far we assumed that the Package Registry is dumb and can't validate any signatures. Kibana would be the ultimate source of trust. It will decide to reject of accepting the signature.

In rare occurrences, if we lose the private key or if it gets exposed, we will have to update the key pair and communicate the same to our customers shortly after. As soon as the risk is identified, we must generate a new pair and re-sign all the packages with the new key immediately.

Agree, technically we must be able to perform this.

How will we communicate the key update?

Again, please try to pair with somebody responsible for this process in Elastic, and ask about the option of launching own keyserver.