Notarization of Electron app fails due to hardened runtime disabled

[gyk]

I'm using rcodesign to sign and notarize an Electron app. The signing phase seems to succeed, and Apple's codesign tool can verify the result when issuing codesign --verify --deep --strict. However, rcodesign notary-submit consistently produce errors stating "The executable does not have the hardened runtime enabled", which is strange as I do pass the --code-signature-flags runtime flag. By inspecting the logs I find rcodesign sign prints "signing without an Apple signed certificate but signing settings contain a team name" for all object files, and according to this comment, it is unexpected.

System information

• apple-codesign 0.25.1
• macOS 10.15.7
• Electron 27.0.4
• electron-builder 24.6.4

Logs

$ rcodesign sign -v --code-signature-flags runtime --runtime-version "10.14.0" \
  --entitlements-xml-file ./resources/mac/entitlements.mac.plist --p12-file $P12_FILE --p12-password $P12_PASSWORD \
  --team-name $TEAM_NAME  dist/bundle/$APP.app/Contents/Frameworks/libavcodec.60.dylib

registering signing key
automatically registered Apple CA certificate: Developer ID Certification Authority
automatically registered Apple CA certificate: Apple Root CA
using time-stamp protocol server http://timestamp.apple.com/ts01
automatically setting team ID from signing certificate: $TEAM_NAME
adding code signature flag CodeSignatureFlags(RUNTIME) to main signing target
setting entitlements XML for main signing target from path ./resources/mac/entitlements.mac.plist
signing dist/bundle/$APP.app/Contents/Frameworks/libavcodec.60.dylib in place
signing dist/bundle/$APP.app/Contents/Frameworks/libavcodec.60.dylib as a Mach-O binary
inferring default signing settings from Mach-O binary
preserving existing binary identifier in Mach-O (libavcodec.60)
using team ID from settings
using code signature flags from settings
using runtime version from settings
using entitlements from settings
setting binary identifier to libavcodec.60
parsing Mach-O
signing Mach-O binary at index 0
deriving code requirements from signing certificate
deriving code requirements from signing certificate
binary targets macOS >= 10.14.0 with SDK 10.14.0
adding code signature flags from signing settings: CodeSignatureFlags(RUNTIME)
using hardened runtime version 10.14.0 from signing settings
signing without an Apple signed certificate but signing settings contain a team name; signature varies from Apple's tooling
code directory version: 132352
creating cryptographic signature with certificate Developer ID Application: Whatever Co., Ltd. ($TEAM_NAME)
Using time-stamp server http://timestamp.apple.com/ts01
total signature size: 80528 bytes
writing Mach-O to dist/bundle/$APP.app/Contents/Frameworks/libavcodec.60.dylib

$ rcodesign analyze-certificate --p12-file $P12_FILE
Please enter password for p12 file: [hidden]
# Certificate 0

Subject CN:                  Developer ID Application: Whatever Co., Ltd. ($TEAM_NAME)
Issuer CN:                   Developer ID Certification Authority
Subject is Issuer?:          false
Team ID:                     $TEAM_NAME
SHA-1 fingerprint:           ...
SHA-256 fingerprint:         ...
Key Algorithm:               RSA
Signature Algorithm:         SHA-256 with RSA encryption
Public Key Data:             ...
Signed by Apple?:            true
Apple Issuing Chain:
  - Developer ID Certification Authority
  - Apple Root CA
  - Apple Root Certificate Authority
Guessed Certificate Profile: DeveloperIdApplication
Is Apple Root CA?:           false
Is Apple Intermediate CA?:   false
Apple Extended Key Usage Purpose Extensions:
  - 1.3.6.1.5.5.7.3.3 (CodeSigning)
Apple Code Signing Extensions:
  - 1.2.840.113635.100.6.1.33 (DeveloperIdDate)
  - 1.2.840.113635.100.6.1.13 (DeveloperIdApplication)

// rcodesign notary-log --api-key-file ~/.appstoreconnect/key.json $SUBMIT_ID

{
  "archiveFilename": "$APP.app.zip",
  "issues": [
    {
      "architecture": "x86_64",
      "code": null,
      "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087724",
      "message": "The executable does not have the hardened runtime enabled.",
      "path": "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/MacOS/$APP",
      "severity": "error"
    },

    // Same messages for:
    // "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper.app/Contents/MacOS/$APP Helper",
    // "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Helpers/chrome_crashpad_handler",
    // "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt",
    // "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (Plugin).app/Contents/MacOS/$APP Helper (Plugin)",
    // "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (GPU).app/Contents/MacOS/$APP Helper (GPU)",
    // "$APP.app.zip/$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (Renderer).app/Contents/MacOS/$APP Helper (Renderer)",
  ],
  "jobId": "...",
  "logFormatVersion": 1,
  "sha256": "...",
  "status": "Invalid",
  "statusCode": 4000,
  "statusSummary": "Archive contains critical validation errors",
  "ticketContents": null,
  "uploadDate": "..."
}

[indygreg]

That team ID warning was fixed in 0.26.0, which was released a few hours ago.

That version also had a bug fix for --code-signature-flags not being respected. But I don't think you hit that bug.

Can you run rcodesign print-signature-info on the signed bundle and look for the flags: lines? You want to see something like flags: CodeSignatureFlags(RUNTIME). I suspect RUNTIME isn't present in a lot of nested frameworks because they are failing notarization.

You'll either need to:

1. Resign each framework separately and then sign the full bundle (the hardened runtime flags should be preserved automatically during resigning with rcodesign.
2. Add --code-signature-flags Contents/MacoOS/path/to/binary:runtime when signing the bundle so every binary gets a hardened runtime flag.

If this all previously worked before, this regression is likely fallout from an intended change in behavior in 0.25 (https://github.com/indygreg/apple-platform-rs/releases/tag/apple-codesign%2F0.25.1) where settings aren't recursively propagated to every nested binary any more.

[gyk]

Yes, the team ID warning has been fixed in 0.26.0, thank you!

Can you run rcodesign print-signature-info on the signed bundle and look for the flags: lines? You want to see something like flags: CodeSignatureFlags(RUNTIME).

As you expected, grep with 'CodeSignatureFlags\(RUNTIME\)' only returns a single match of the main app. Signed again with --code-signature-flags dist/bundle/$APP.app/Contents/MacOS/$APP.app/Contents/MacOS/$APP:runtime but unfortunately there was still a single "CodeSignatureFlags(RUNTIME)" and notarization returned the same "hardened runtime" error.

Resign each framework separately and then sign the full bundle

Yes this should work, I will give it a try later. Thanks!

[bigorn0]

@indygreg this issue wasn't present after the fix that made signing Electron applications working. If i test using commit 187aedd it is passing notarization after signing using rcodesign. (this commit was my first attempt for testing with signing each Mach-o file / Fwk before signing the whole app).

[indygreg]

As I mentioned, the behavior of --code-signature-flags was purposefully changed to not propagate in the 0.25 release. That's because if settings propagate by default, there's no way to turn off propagation.

I think the reason why OP is having issues is that scoped paths are relative to the path of the entity being signed - not relative to the directory being executed from. In the failing example, the path used was --code-signature-flags dist/bundle/$APP.app/Contents/MacOS/$APP.app/Contents/MacOS/$APP:runtime. I bet if that changes to --code-signature-flags $APP.app/Contents/MacOS/$APP.app/Contents/MacOS/$APP:runtime that things will just work.

Also, in this failing example, the flag is being specified for the bundle's main executable. --code-signature-flags Contents/MacOS/$APP.app:runtime` should also work in this scenario.

There may be room to restore a way to opt into recursive propagation. e.g. --code-signature-flags @all:runtime. But we'd need to consider how code signature bitflags interact when there are conflicting definitions. e.g. if you have --code-signature-flags @all:runtime and --code-signature-flags Contents/MacOS/other-bin:host, is Contents/MacOS/other-bin runtime | host or host?

I'll think about this additional scoping rule, as it could make things simpler for end-users.

[gyk]

Solved by this:

rcodesign sign \
    --code-signature-flags "Contents/MacOS/$APP.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Helpers/chrome_crashpad_handler":runtime \
    --code-signature-flags "Contents/MacOS/$APP.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt":runtime \
    --code-signature-flags "Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper.app":runtime \
    --code-signature-flags "Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (Plugin).app":runtime \
    --code-signature-flags "Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (GPU).app":runtime \
    --code-signature-flags "Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (Renderer).app":runtime \
    --code-signature-flags "Contents/MacOS/$APP.app":runtime \
    --code-signature-flags runtime \
    --runtime-version "10.15.0" --entitlements-xml-file … --p12-file … --p12-password …  --team-name … ./dist/bundle/$APP.app

Notes:

• Our Electron app has a non-standard structure where an $APP.app is nested inside another $APP.app.
• For frameworks, the scopes must be set to Mach-O binaries directly.

Now Apple is happy with the bundle. 🎉

---

Update: Although Apple accepts the submission, the app actually can no longer launch after signing.

./dist/bundle/$APP.app/Contents/MacOS/$APP.app/Contents/MacOS/$APP

<--- Last few GCs --->

[67421:0x7fa741f31000]      106 ms: Mark-Compact (reduce) 0.6 (3.2) -> 0.6 (1.7) MB, 2.42 / 0.00 ms  (average mu = 0.147, current mu = 0.019) last resort; GC in old space requested
[67421:0x7fa741f31000]      107 ms: Mark-Compact (reduce) 0.6 (1.7) -> 0.6 (1.7) MB, 1.19 / 0.00 ms  (average mu = 0.097, current mu = 0.011) last resort; GC in old space requested


<--- JS stacktrace --->


#
# Fatal JavaScript out of memory: CALL_AND_RETRY_LAST
#

Trace/BPT trap: 5

---

Update 2: Running rcodesign sign --code-signature-flags runtime on the following paths separately (the order matters) can produce a valid bundle:

"$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Helpers/chrome_crashpad_handler"
"$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt"
"$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper.app"
"$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (Plugin).app"
"$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (GPU).app"
"$APP.app/Contents/MacOS/$APP.app/Contents/Frameworks/$APP Helper (Renderer).app"
"$APP.app/Contents/MacOS/$APP.app"
"$APP.app"

But this is a bit slow and error-prone so we switch back to Apple's tools for now.

[jlskuz]

Hi! We recently started using rcodesign for KDE apps. It is really useful to have this tool to run on non-apple systems!

For security reasons we have separated machines building/packaging and signing the apps. We therefor use the recursive behavior of rcodesign when signing .apps However notarization of them fails with the "no hardened runtime error" even though we are using --code-signature-flags runtime. I found https://github.com/indygreg/apple-platform-rs/blob/main/apple-codesign/src/cli/mod.rs#L2095 and thought --code-signature-flags main:runtime might make it work, but it doesn't. So I think we are suffering from a similar problem and need what you suggested:

There may be room to restore a way to opt into recursive propagation. e.g. --code-signature-flags @all:runtime.

As long as this does not work we can not notarize our apps (only sign). Unfortunately I have zero rust knowledge so I don't feel qualified to open a PR. If I could help in a different way let me know!

But we'd need to consider how code signature bitflags interact when there are conflicting definitions. e.g. if you have --code-signature-flags @all:runtime and --code-signature-flags Contents/MacOS/other-bin:host, is Contents/MacOS/other-bin runtime | host or host?

While the first suggestion feels more intuitive, the second seems to be more flexible as there would be no other way to get only host than to not use @all:runtime and specify every path again

I'll think about this additional scoping rule, as it could make things simpler for end-users.

Looking forward to it!

[kobaltcore]

This change in behavior has broken our release pipeline as well, which was annoying to even figure out, so that was a fun two days of investigation.
As long as recursive propagation for these flags is no longer possible, we'll have to stay on versions below 0.25.0. Hopefully this ability will return eventually so we can benefit from all the other ongoing work in this project, which takes care of a very painful part of our release pipeline for us and which we're very grateful for.

[indygreg]

I'm sorry for the confusion, @kobaltcore. In my defense, the release notes for 0.25.1 did attempt to call out the breaking change in behavior. I do generally value backwards compatibility, as I've been on the receiving end of unwanted compatibility breaks too many times myself. In this case, the previous behavior resulted in unexpected behavior for many signing operations and we needed to correct that behavior before more damage was done.

I'm still thinking about how to best solve this UX problem. If you have suggestions, I'd love to hear them.

FWIW another idea I've had is to add a --for-notarization (or similar) argument that automatically engages common settings needed to support Apple notarization. IMO it is kind of annoying for end-users to have to know which signing settings are required for notarization: developers just want to publish software and signing + notarization is probably something they don't want to think about. This feature also conveniently means we can punt on harder-to-reason-about decisions around the signing settings UI design.

[kobaltcore]

Yeah, this is on the 0.x strain for a reason, so we can't really complain about sudden breaking changes, haha. Alas, this particular change not only broke something, it made a previous thing that this program did impossible, which is a whole 'nother level, I reckon.

In any case, I think as long as there can be the option of having some kind of all scope we'd be able to apply, everything's good.
A --for-notarization toggle might be useful as well for those who really don't want to tinker with the flags at all, but I fear that -should the requirements for notarization ever change from i.e. Apple's side- different versions of this tool will end up applying different flags, which would be a breaking change every time. You'd also put yourself in lock-step with Apple, at least moreso than right now. Might not be the most ideal situation to put yourself in as a maintainer, but maybe that's me overthinking it in this case.

Anyway, thanks for engaging on this and working with the community to come to a solution, much appreciated!

[indygreg]

I recognize that the change in behavior made more complex signing modes more burdensome than they needed to be.

In the past few days I've committed a few features to help reign back control.

• rcodesign sign now has a --shallow flag to prevent recursing into nested bundles. This can enable you to sign a single entity at a time, similar to the way Apple's tooling forces you to do it. This means that in the worst case you should be able to invoke rcodesign sign N times for more control over signing. This feature likely needs some more work to not sign nested Mach-O binaries within bundles. But it seems strictly better than having no feature at all.
• rcodesign sign now has a --for-notarization flag that automatically enables the hardened runtime flag as well as checks an appropriate signing certificate is being used. Again, probably not perfect. But feels better than nothing.

Both features are documented as having a non-stable set of semantics, with the intent that the semantics converge towards more user-friendly behavior over time.

I still want to add an easy way to recursively enable signing settings. I had a quick and dirty patch to attempt this and it broke tests in some unexpected ways. So I need to think about things some more. I'm leaving this issue open to track a longer term solution.

[jlskuz]

As an update on this the --for-notarization flag does satisfy our needs and we are able to run the whole pipeline from signing to notarization all with rcodesign now. Thanks!