Add prebuilds for ARM64 Macs #2354

jamesbvaughan · 2024-02-23T01:57:09Z

This is an updated attempt at #2245, making use of GitHub's new Apple Silicon runners.

With many Mac-based developers transitioning to Apple Silicon, this addition will have a significant impact on their cumulative yarn/npm install time.

jamesbvaughan · 2024-02-23T03:17:39Z

I've temporarily removed the Linux and Windows builds from the workflow in order to reduce the Actions usage on my personal account while testing.

The current status here is that I'm hitting an error about napi.h not being found and I'm not sure how to proceed. Here's where you can see the error: https://github.com/jamesbvaughan/node-canvas/actions/runs/8014121461/job/21892227263#step:4:279

adamcin · 2024-02-23T23:22:01Z

I've temporarily removed the Linux and Windows builds from the workflow in order to reduce the Actions usage on my personal account while testing.

The current status here is that I'm hitting an error about napi.h not being found and I'm not sure how to proceed. Here's where you can see the error: https://github.com/jamesbvaughan/node-canvas/actions/runs/8014121461/job/21892227263#step:4:279

Hi @jamesbvaughan , it looks like you are building canvas sources that have migrated from nan to napi by way of node-addon-api, with related changes to gyp scripts described in this commit: jamesbvaughan@ce29f69

EDIT: I submitted a PR for your branch that I think may help jamesbvaughan#1

jamesbvaughan · 2024-02-28T04:42:04Z

Alright, thanks to @adamcin's help, I think this is ready for a real review!

Let me know if you'd like me to clean up the git history at all.

zbjornson · 2024-02-29T18:10:31Z

Thanks for submitting this! Two notes:

We still haven't managed to ship prebuilds for any platform since the change to N-API affected where the built files reside. The latest on that is here: Port to node-addon-api (and N-API) and remove NAN, libuv, and v8 #2235 (comment)
Prebuilds are made from the prebuilds branch, so the changes here would need to be targeted to that branch: https://github.com/Automattic/node-canvas/tree/prebuilds. We should delete the prebuilds stuff from the master branch since it's confused a lot of people.

jamesbvaughan · 2024-02-29T18:20:29Z

Thanks @zbjornson. I'll hold off on further work here for now.

rafaelmaiolla · 2024-04-26T14:16:49Z

Is there any update in having arm64 pre-build binaries?

mn4367 · 2024-04-26T15:11:35Z

@rafaelmaiolla, in my opinion still boils down to bundling the generated native module with the native dependencies. You may have a look to issue #2036 which has a bit more info on why providing pre-built binaries is difficult.

Edit:

I just saw that executing/loading my bundled version let to a Killed: 9. The same error occurs, if one tries to execute a self-compiled executable on an Apple silicon chip if it isn't signed. Maybe I'll find the time to do this again and see what happens, if I sign the generated binary.

Follow up:

With regard to #2036 this is indeed a signing problem. Signing all generated/patched binaries (canvas.node, *.dylib) in build after running bundle.sh everything worked fine! This at least proves, that a prebuild self contained version can be built for Apple Silicon. I've done the signing with these tools. The interesting part here is that the tools work cross platform, so it's even possible to sign Apple Silicon binaries on a Windows or Linux machine!

net-tech · 2024-07-09T01:34:25Z

With regard to #2036 Signing all generated/patched binaries (canvas.node, *.dylib) in build after running bundle.sh everything worked fine! This at least proves, that a prebuild self contained version can be built for Apple Silicon. I've done the signing with these tools. The interesting part here is that the tools work cross platform, so it's even possible to sign Apple Silicon binaries on a Windows or Linux machine!

If singing is working then would the next step be for OP to add signing to their PR? I'm sorry if I sound impatient, but a prebuilt binary could remove around 50s on our GitHub actions which is why I'm eager for there to be a prebuilt binary.

jamesbvaughan · 2024-07-10T01:55:59Z

I've rebased these changes on the prebuilds branch and removed some unnecessary changes. If the code signing truly is all that's left, then I think we're close!

With regard to #2036 this is indeed a signing problem. Signing all generated/patched binaries (canvas.node, *.dylib) in build after running bundle.sh everything worked fine! This at least proves, that a prebuild self contained version can be built for Apple Silicon. I've done the signing with these tools. The interesting part here is that the tools work cross platform, so it's even possible to sign Apple Silicon binaries on a Windows or Linux machine!

@mn4367 Thanks for trying that out! Do you have any public code demonstrating the use of that code signing tool in a context like this? I haven't dealt with Apple's code signing before and if you have an example, it could save me a lot of time.

mn4367 · 2024-07-11T09:11:34Z

@jamesbvaughan

I haven't dealt with Apple's code signing before ...

Same here :-(.

Everything I did was to run prebuild/MacOS/bundle.sh and then run rcodesign sign <.dylib> with every single .dylib file in build/Release e.g. rcodesign sign ./build/Release/libcairo-gobject.2.dylib, rcodesign sign ./build/Release/libcairo.2.dylib etc.

After that I'm able to run all tests successfully but I've no idea if sign creates files that can be loaded on all users machines not only on my machine. I've uploaded a tarball here, maybe you can check if they work on your machine? Please ping me if you've downloaded the tarball so that I can delete it.

If it doesn't work directly after unpacking you could try calling xattr -dr com.apple.quarantine with every .dylib and .node and see if that works.

jamesbvaughan · 2024-07-11T16:29:32Z

@mn4367 Thanks for all that. I've downloaded and unpacked your tarball and npm test passes for me with those files.

To be honest, I'm wading in unfamiliar territory here and I'm curious for @zbjornson's input on code signing.

The tests pass in the Make prebuilds CI workflow for my branch as-is, which makes me wonder if this is already good to go: https://github.com/jamesbvaughan/node-canvas/actions/runs/9866841579/job/27246266340 (The overall workflow failed because I don't have the necessary secrets for uploading, which is expected.)

mn4367 · 2024-07-13T13:59:24Z

@jamesbvaughan, I did another quick test: I created a Node.js SEA, signed it with rcodesign (the same way like for canvas) and send it over via Dropbox to a friend to see, if it can be run on his machine. Upon first execution there the OS complained about it (unknown developer etc.) but after allowing it to execute (via the System Settings App) it runs without further problems. Then I analyzed this executable and the node executable used to create this application:

My application:

$ codesign --display --verbose=4 ./my-sample-app 
Executable=/Users/somepath/my-sample-app
Identifier=my-sample-app
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=672899 flags=0x2(adhoc) hashes=21023+2 location=embedded
VersionPlatform=1
VersionMin=720896
VersionSDK=786688
Hash type=sha256 size=32
CandidateCDHash sha256=de3c83d5c5ff6fd32cb899d01fc275fd0a9763b2
CandidateCDHashFull sha256=de3c83d5c5ff6fd32cb899d01fc275fd0a9763b21c6c1e68d924a5ac4f6678a6
Hash choices=sha256
CMSDigest=de3c83d5c5ff6fd32cb899d01fc275fd0a9763b21c6c1e68d924a5ac4f6678a6
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=66027520
Executable Segment flags=0x1
Page size=4096
CDHash=de3c83d5c5ff6fd32cb899d01fc275fd0a9763b2
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

Node.js executable:

$ codesign --display --verbose=4 ./node
Executable=/Users/somepath/node
Identifier=node
Format=Mach-O thin (arm64)
CodeDirectory v=20500 size=730000 flags=0x10000(runtime) hashes=22802+7 location=embedded
VersionPlatform=1
VersionMin=720896
VersionSDK=786688
Hash type=sha256 size=32
CandidateCDHash sha256=8f343dfa30ae01932ad6c41f57f8d83bd4a747ff
CandidateCDHashFull sha256=8f343dfa30ae01932ad6c41f57f8d83bd4a747ffb1382c42f56a85b548492900
Hash choices=sha256
CMSDigest=8f343dfa30ae01932ad6c41f57f8d83bd4a747ffb1382c42f56a85b548492900
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=66027520
Executable Segment flags=0x1
Page size=4096
CDHash=8f343dfa30ae01932ad6c41f57f8d83bd4a747ff
Signature size=8987
Authority=Developer ID Application: Node.js Foundation (HX7739G8FX)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=8. Jul 2024 at 15:26:11
Info.plist=not bound
TeamIdentifier=HX7739G8FX
Runtime Version=12.1.0
Sealed Resources=none
Internal requirements count=1 size=164

So the only difference is that with rcodesign sign <file> a simple adhoc signature is created which forces users to explicitly give permission to execute while 'proper' signing with a 'real' developer certificate doesn’t need any user permissions. But there is a caveat: the (Zip) file which was downloaded by my friend has been unpacked autoamtically by Safari (as usual) and the contained unpacked executable has been marked as 'Downloaded from the Internet' (or similar, he has a german OS).

So there are basically two questions:

Will macOS also complain for files downloaded by other means, e.g. by an npm script and not through Safari?
Is it even necessary to give permissions for dylibs (like in the case of Canvas)?

If the answer to both questions is no, then rcodesign sign <file> creating an adhoc signature should be just enough to create the prebuilds for darwin-arm64 and darwin-x64. Signing with a real certificate would of course be the best solution.

If I find the time, I'll create a test repository with a pre-built, standalone canvas module. Maybe some people here can then download it and test if it works out of the box.

Edit, PS:
Thanks for testing the tarball! 👍

mn4367 · 2024-07-13T14:49:04Z

I found something interesting in the rcodesign docs in the meantime here: the last paragraph in this section clearly states that the quarantine attribute (which makes macOS bark) is not added to files downloaded by non-browser tools like curl or wget. So using an adhoc signature should be sufficient to create prebuilds for macOS. In the end, it no longer seems to be a technical requirement, but rather a question of 'reputation' whether or not to use a real certificate for signing.

jamesbvaughan · 2024-07-16T21:34:54Z

I had a bit of time to work on this today and found that the creator of rcodesign now provides a GitHub Action for performing code signing: https://github.com/marketplace/actions/apple-code-signing

I've added that action to the workflow in this PR and it seems to be signing successfully.

@zbjornson Let me know if there's anything else I'm missing here.

kevinji · 2024-09-11T06:40:43Z

@zbjornson Is there anything blocking the release of this PR?

de-robat · 2024-09-24T10:38:59Z

Would like to know too. Just to add to the urgency, appflow will deprecated the old Mac builders and will switch to Silicon runners only at 1st of October 24. With e.g. jsdom relying on this module (which is a direct dependency of jest, isomorphic-dompurify, ...)) lots of builds will fail without this binding. What needs to be done to get this across the fnishline?

zbjornson

Sorry for the silence. This looks straightforward. I don't have a mac to test on, but I'm happy to merge this, make the prebuild for 3.0-rc2 and make sure it works for folks.

One tiny question...

zbjornson · 2024-09-26T01:34:43Z

.github/workflows/prebuild.yaml

@@ -169,14 +174,19 @@ jobs:
          npm install --ignore-scripts
          . prebuild/macOS/preinstall.sh
          cp prebuild/macOS/binding.gyp binding.gyp
-          node-gyp rebuild -j 2
+          node-gyp rebuild -j 2 --arch=${{ matrix.os.arch }}


node-gyp defaults to process.arch. Is this change needed?

Good question - I'm not experienced with node-gyp and I think I copied this from someplace else without testing it without that option. I'll try it out without it today.

Alright I've pushed a commit that removes that flag, and as far as I can tell, this step of the build process is completing as intended: https://github.com/jamesbvaughan/node-canvas/actions/runs/11115619922/job/30884399250#step:4:178

I should point out though that I'm wading in unfamiliar territory for me when it comes to node-gyp and macos binary signing, so I'd really appreciate if someone with more experience with those could help verify that everything looks correct here.

chearon

Thank you! I finally have Apple Silicon so I'll give this a whirl and make any changes needed.

changes to get macos prebuild to pass on arm64 local machine Add packages to the uninstall list Restore changes in prebuild.yaml Restore changes to tarball.sh Set a temporary canvas_tag Revert temporary canvas tag Update runners Remove unnecessary changes Add a code signing step Remove explicit `--arch` argument

chearon · 2024-12-07T15:43:16Z

Squashed because the squashed commit doesn't have conflicts

chearon · 2024-12-07T18:34:06Z

I haven't been able to get the ARM version to load without a SIGKILL. If I run rcodesign or the native codesign on all of the executables (canvas.node and dylib) then it works, ~~but only locally. If I run those on CI and distribute, I get the same SIGKILL.~~

@mn4367 that seems to be different from your findings, though. Do you have any other ideas why this is happening?

@napi-rs/canvas doesn't seem to do anything with code signing: CI.yaml.

chearon · 2024-12-07T19:30:18Z

Okay, the ARM builds are working. Anyone still following this can try

npm install canvas@next

I was wrong about codesign only working locally. I didn't realize that prebuild-install caches heavily, so I wasn't seeing my changes. Wasted a lot of time on that...
rcodesign doesn't seem to do anything codesign (the native tool) doesn't already do, and rcodesign has to either be installed with cargo which takes forever, or from a release URL which is a pain. Can't use the GitHub action since it doesn't support multiple files in an easy way.
Signing does seem to be required after our build steps, or else it SIGKILLs when you try to load it
But turning off that security feature in macOS doesn't fix that. I agree with @mn4367 that code signing should not matter because those restrictions are only for files downloaded from a browser. So something else must be messed up in the build that is making this necessary. 🤷‍♂️

mn4367 · 2024-12-10T12:56:04Z

Hi @chearon,

sorry for the late reply. If the native Apple codegen is available in the build pipeline for creating the prebuilds it is of course the better solution.

rcodesign could only have been a last resort if all else had failed. I was working in a team with mixed environments (Windows and Apple) and rcodesign enabled us to create self-contained packages of our app on Windows for Apple platforms (although in a different context, that is creating Node SEA apps). For this use case rcodesign is still the only solution I know of as today.

Anyway, glad to hear that you could solve the problem!

PS:

So something else must be messed up in the build that is making this necessary. 🤷‍♂️

I don't think that something is messed up. With Apple Silicon, the signature is now simply an absolute requirement.

Primajin · 2024-12-10T16:52:46Z

Thanks so much for the merge, I've been eagerly waiting for this ❤️

So I do have canvas set to next but I'm still getting:

npm error prebuild-install warn install No prebuilt binaries found (target=7 runtime=napi arch=arm64 libc= platform=darwin)

When trying to build in a github action.

Am I doing anything wrong?

Edit: I forgot to re-run and commit the lockfile with the latest changes 🤦🏻

chearon · 2024-12-10T16:57:09Z

Did you npm install canvas@next or canvas?

Primajin · 2024-12-10T17:00:00Z

Did you npm install canvas@next or canvas?

canvas@next 👍🏻

https://github.com/Primajin/pass-a-way/pull/204/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519

https://github.com/Primajin/pass-a-way/actions/runs/12260791189/job/34206257857?pr=204

chearon · 2024-12-10T17:04:18Z

It doesn't look like your package-lock.json changed. Try using 3.0.0-rc3 in your package.json instead of next.

Primajin · 2024-12-10T17:05:54Z

True, I completely forgot to rerun it recently. Thanks I'll let you know if that fixes it 👍🏻

Primajin · 2024-12-10T17:16:29Z

Looks like x-mas came early this year!

  🎄
🎁🎁🎁

Works like a charm now, thank you so much!

jamesbvaughan force-pushed the james/mac-arm64-prebuild branch 4 times, most recently from c255f18 to ac3db9a Compare February 23, 2024 02:44

jamesbvaughan force-pushed the james/mac-arm64-prebuild branch from 2fc9d9b to 886d26b Compare February 28, 2024 04:41

jamesbvaughan marked this pull request as ready for review February 28, 2024 04:42

zbjornson mentioned this pull request Mar 27, 2024

[cleanup] feat(prebuild): support of Alpine binaries #2366

Closed

1 task

styfle mentioned this pull request May 1, 2024

chore(deps): Bump amannn/action-semantic-pull-request from 5.3.0 to 5.5.2 vercel/ncc#1187

Closed

krassowski mentioned this pull request May 10, 2024

Error during build steps on macos jupyterlab/jupyterlab#16311

Closed

kevinji mentioned this pull request Jul 4, 2024

macOS: Fix deps in preinstall, reenable arm64 builds #2393

Closed

jamesbvaughan force-pushed the james/mac-arm64-prebuild branch from 886d26b to c41b736 Compare July 10, 2024 01:18

jamesbvaughan changed the base branch from master to prebuilds July 10, 2024 01:18

Primajin mentioned this pull request Jul 17, 2024

📦 update canvas package Primajin/pass-a-way#204

Merged

2 tasks

mn4367 mentioned this pull request Jul 19, 2024

Compile Canvas on Apple Silicon Mac using custom library paths and without installling using homebrew #2406

Open

1 task

k-yle mentioned this pull request Aug 7, 2024

Throws errors when launching Electron app and when reinstalling k-yle/pdf-to-img#214

Open

zbjornson reviewed Sep 26, 2024

View reviewed changes

btsimonh mentioned this pull request Nov 1, 2024

Add prebuild for macos arm64, enable prebuilds to be run individually. #2451

Open

1 task

chearon approved these changes Dec 7, 2024

View reviewed changes

chearon force-pushed the james/mac-arm64-prebuild branch from 7c36b66 to b0c9060 Compare December 7, 2024 15:42

chearon merged commit 76d1ded into Automattic:prebuilds Dec 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add prebuilds for ARM64 Macs #2354

Add prebuilds for ARM64 Macs #2354

jamesbvaughan commented Feb 23, 2024 •

edited

Loading

jamesbvaughan commented Feb 23, 2024

adamcin commented Feb 23, 2024 •

edited

Loading

jamesbvaughan commented Feb 28, 2024

zbjornson commented Feb 29, 2024

jamesbvaughan commented Feb 29, 2024

rafaelmaiolla commented Apr 26, 2024

mn4367 commented Apr 26, 2024 •

edited

Loading

net-tech commented Jul 9, 2024

jamesbvaughan commented Jul 10, 2024

mn4367 commented Jul 11, 2024

jamesbvaughan commented Jul 11, 2024

mn4367 commented Jul 13, 2024 •

edited

Loading

mn4367 commented Jul 13, 2024

jamesbvaughan commented Jul 16, 2024

kevinji commented Sep 11, 2024

de-robat commented Sep 24, 2024

zbjornson left a comment

zbjornson Sep 26, 2024

jamesbvaughan Sep 26, 2024

jamesbvaughan Sep 30, 2024

jamesbvaughan Sep 30, 2024

chearon left a comment

chearon commented Dec 7, 2024

chearon commented Dec 7, 2024 •

edited

Loading

chearon commented Dec 7, 2024

mn4367 commented Dec 10, 2024

Primajin commented Dec 10, 2024 •

edited

Loading

chearon commented Dec 10, 2024

Primajin commented Dec 10, 2024 •

edited

Loading

chearon commented Dec 10, 2024

Primajin commented Dec 10, 2024

Primajin commented Dec 10, 2024

Add prebuilds for ARM64 Macs #2354

Add prebuilds for ARM64 Macs #2354

Conversation

jamesbvaughan commented Feb 23, 2024 • edited Loading

jamesbvaughan commented Feb 23, 2024

adamcin commented Feb 23, 2024 • edited Loading

jamesbvaughan commented Feb 28, 2024

zbjornson commented Feb 29, 2024

jamesbvaughan commented Feb 29, 2024

rafaelmaiolla commented Apr 26, 2024

mn4367 commented Apr 26, 2024 • edited Loading

net-tech commented Jul 9, 2024

jamesbvaughan commented Jul 10, 2024

mn4367 commented Jul 11, 2024

jamesbvaughan commented Jul 11, 2024

mn4367 commented Jul 13, 2024 • edited Loading

mn4367 commented Jul 13, 2024

jamesbvaughan commented Jul 16, 2024

kevinji commented Sep 11, 2024

de-robat commented Sep 24, 2024

zbjornson left a comment

Choose a reason for hiding this comment

zbjornson Sep 26, 2024

Choose a reason for hiding this comment

jamesbvaughan Sep 26, 2024

Choose a reason for hiding this comment

jamesbvaughan Sep 30, 2024

Choose a reason for hiding this comment

jamesbvaughan Sep 30, 2024

Choose a reason for hiding this comment

chearon left a comment

Choose a reason for hiding this comment

chearon commented Dec 7, 2024

chearon commented Dec 7, 2024 • edited Loading

chearon commented Dec 7, 2024

mn4367 commented Dec 10, 2024

Primajin commented Dec 10, 2024 • edited Loading

chearon commented Dec 10, 2024

Primajin commented Dec 10, 2024 • edited Loading

chearon commented Dec 10, 2024

Primajin commented Dec 10, 2024

Primajin commented Dec 10, 2024

jamesbvaughan commented Feb 23, 2024 •

edited

Loading

adamcin commented Feb 23, 2024 •

edited

Loading

mn4367 commented Apr 26, 2024 •

edited

Loading

mn4367 commented Jul 13, 2024 •

edited

Loading

chearon commented Dec 7, 2024 •

edited

Loading

Primajin commented Dec 10, 2024 •

edited

Loading

Primajin commented Dec 10, 2024 •

edited

Loading