[heft] heft-jest-plugin crashes on ESM tests (even though jest alone works fine on them)

[ChristophP]

Summary

We are converting our repo to ESM. (There is no way around it, since some depencies won't support CommonJS anymore, ESM can import CJS and ESM, CJs can only require CJS).

Our tsconfig.json uses module: NodeNext and moduleResolution: NodeNext and our package.json contains type: module.
Everything including the build works well except for our jest tests which use heft-plugin-jest.

We see an error like this (which is common when Jest is run on ESM code without NODE_OPTIONS=--experimental-vm-modules set).

However, this must be something heft specific ,as the same tests DO WORK when run through jest alone but not when run through heft.

Repro steps

This repo contains a minimal example to reproduce the issue. It also highlights how the same tests run under jest alone but not with the heft-jest-plugin.

Expected result: Heft uses Jest to run the tests.

Actual result: Heft uses Jest and crashes saying it doesn't recognize import (which it should in an ESM context)

Details

Heft seems to do something unusal. Since for some reason it doesn't seem to respect the NODE_OPTIONS=--experimental-vm-modules flag. When run with jest alone the tests are run just fine.
Probably the best way to fix this is to find out the difference between the rushx jest-only and rushx heft-breaks command in the demo repo and make sure heft works the same way as it would if only jest were run in terms of treatment of ESM.

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
@rushstack/heft version?
Operating system?	Mac
Would you consider contributing a PR?	Yes (if the agreed upon fix will be something I can manage)
Node.js version (node -v)?	v18.16.0

[woss]

Hi @ChristophP,

I see here https://github.com/ChristophP/heft-jest-esm/blob/main/packages/lib/config/heft.json that you don't use typescript plugin, The idea is to compile it to the javascript ( cjs or esm) then run tests on that code, not on the typescript code.

If you want to run jest on the typescript files you should use ts-jest.

[ChristophP]

Hi @woss ,
thanks for your reply. I believe I am using the typescript plugin. I think I am on an older version of heft (0.50) which includes the typescript plugin implicitly, which the newer versions no longer do. Could that be it? https://rushstack.io/blog/2023/06/16/heft-migration-guide/#plugins-must-be-explicitly-loaded

[woss]

@ChristophP

i am not sure, i didn't move any of my code to pure ESM + heft-jest yet, although I would suggest 2 things:

1. use new heft ( and plugins )
2. post your question here https://rushstack.zulipchat.com/ for better collaboration

[ChristophP]

It seems to work like this with the latest version of heft and the heft-jest-plugin https://github.com/ChristophP/heft-jest-esm/compare/get-esm-to-work.

[ChristophP]

I the solution is we got it to work by upgrading Heft to 0.6+ and the heft jest plugin accordingly.

Note to Rush maintainers:
It would be great to add an example about how to truly use ESM with Rush/Heft in the rushstack repo.
There is one example that's called heft-node-everything-esm-module-test, but the name is misleading, because it only uses ESM in the typescript files but still compiles down to CJS.
It would be great to add an example that actually has { "type": "module" } in the package.json to signal to people trying to migrate to ESM (which will be everyone eventually) that Rush/Heft support this. (I believe it's important to communicate with emphasis, because from experience I know it's easy to think that "it's just not supported" when you try and run into issues that aren't super obvious to fix).

Feel free to close this

[dmichon-msft]

To date we haven't done any work on enabling Jest's native ESM support, since it still has a number of open issues and requires experimental flags.

[ChristophP]

I understand. Yeah, it's a pain. Jest's ESM support has been experimental for years. And that's not even through fault of their own, it's because they are using (and need to) use NodeJS's vm API for test isolation, which still isn't marked as stable in NodeJS.
The whole CJS/ESM topic is hard on everyone writing JS, so the more tools openly support it the easier it will be to get to the other side where ESM is the norm eventually.

[ChristophP]

Btw judging from this thread that situation seems to be quite frustrating for the Jest maintainer's as well

[iclanton]

@dmichon-msft - Do you know how much work this would be to add support for?

[ChristophP]

I don't wanna speak for @dmichon-msft , but from how I understand it, there is no extra work to be done by heft or the jest-pluginto support ESM in Jest. It already works if some preconditions are met

• type: module in the package.json or files renamed to .mjs
• Jest tests are run with NODE_OPTIONS=--experimental-vm-modules

So I believe it's mostly a matter of stating that it works in the heft docs, and linking to jest's docs about how to use it.

[Toxaris]

Jest tests are run with NODE_OPTIONS=--experimental-vm-modules

What is the best way to set this up in a cross-platform rush+heft monorepo?

The best I managed is to use cross-env in the package.json of every project:

  "scripts": {
    "_phase:build": "heft run --only build -- --clean",
    "_phase:test": "cross-env NODE_OPTIONS=--experimental-vm-modules NODE_NO_WARNINGS=1 heft run --only test --",
    "build": "heft run --to build -- --clean",
    "test": "cross-env NODE_OPTIONS=--experimental-vm-modules NODE_NO_WARNINGS=1 heft run --to test -- --clean"
  },

This seems to work, but I would prefer a solution where I don't have to repeat the cross-env noise in every package.json.

Ideas:

1. Could rush support setting environment variables for a command or phase (related to [rush] Env variables for custom commands #1839)
2. Could heft support node startup options (from the command line and/or from a configuration file)?
3. Could heft-jest-plugin run jest in a separate process and support node startup optons (from a configuration file)?