Assets missing + nondeterministic build (probable regression?)

[SmallhillCZ]

Is there an existing issue for this?

• I have searched the existing issues

Current behavior

This is a probable regression from already closed issues (see below). When machine is slow or slowed down, sometimes not all assets will get copied. This creates non deterministicly faulty outputs for NestJS builds.

This means when you are for example building Docker image using BuildKit, which has parallel execution, a CPU and HDD intensive task of another Docker build step will cause part of your assets not being copied.

As a direct consequence part of your application will not be working as usual. Also as this not being a code part, it will probably not fail during startup and you will learn the hard way, e.g. users not being able to register, because the HTML file with validation email cannot be loaded.

This was discussed previously in #1828 and #749 and it was presumably caused by the timeout in assets manager on L26. Also the note on L25 actually suggests this could happen for large files.

nest-cli/lib/compiler/assets-manager.ts

Lines 24 to 37 in 840b54d

	public closeWatchers() {
	// Consider adjusting this for larger files
	const timeoutMs = 500;
	const closeFn = () => {
	if (this.actionInProgress) {
	this.actionInProgress = false;
	setTimeout(closeFn, timeoutMs);
	} else {
	this.watchers.forEach((watcher) => watcher.close());
	}
	};
	setTimeout(closeFn, timeoutMs);
	}

Minimum reproduction code

The issue seems the same as before and the reproduction code will be hard to create, as it is a concurrency issue with other processes. So would like to discuss if it's needed before coding.

Steps to reproduce

1. Run NestJS build with either
   • lot of files
   • slow machine
   • machine slowed down with other parallel processes using HDD
2. Random part of your assets will not make it to the build

Expected behavior

All assets must be in the build no matter the build time.

Package version

9.3.0

NestJS version

9.4.0

Node.js version

18.14.2

In which operating systems have you tested?

• macOS
• Windows
• Linux

Other

At least the timeoutMs should be configurable in nest-cli.json or somewhere, but I think by default the copy process should never be killed before finishing.

[SmallhillCZ]

Looking at the code it looks weird, that the variable which is there to "avoid watches getting cutoff" here:

nest-cli/lib/compiler/assets-manager.ts

Lines 118 to 119 in 840b54d

	// Set action to true to avoid watches getting cutoff
	this.actionInProgress = true;

is set to false to cut the watchers off after timeout anyway here:

nest-cli/lib/compiler/assets-manager.ts

Lines 28 to 33 in 840b54d

	if (this.actionInProgress) {
	this.actionInProgress = false;
	setTimeout(closeFn, timeoutMs);
	} else {
	this.watchers.forEach((watcher) => watcher.close());
	}

[natoszme]

Hi there! Same thing happening here: sometimes in my deploys in aws beanstalk, the assets folder is missing in one of the instances. Never happened in my local environment, so could not reproduce it yet.
But it did happen several times since I started using nest :(

Nest version: 9.2.0
Node version: 14.18.0

[natoszme]

Hi everyone! I've been struggling with this bug several times since my last post, so I fixed it ensuring that every asset is copied to the production dist folder. To to it, I made a small change to my package.json deploy command:

Before: "deploy": "npm install && npm run build && npm run start:prod"
After: "deploy": "npm install && npm run build && npm run ensureBuild && npm run start:prod"

Where:

• "ensureBuild": "make copyAssets",
• and copyAssets is defined in a Makefile as:

assetsDirectory = notifications/templates //change it for yours
distDirectory = dist/${assetsDirectory}
srcDirectory = src/${assetsDirectory}

copyAssets:
	rm -rf $(distDirectory)
	mkdir $(distDirectory)
	cp -r $(srcDirectory)/. $(distDirectory)/

It's been a couple of weeks since I deployed it and had no problems since then. Hope it helps!

[simonbrunel]

We are experiencing the same, making deployment very unreliable. It happens on .hbs files (handlebar templates), which are dynamically loaded on demand, meaning that the server correctly starts and then randomly crashes when trying to access these assets. It's also very hard to reproduce (it only happens when built on CI), so I can't provide a reproduction repo.

@kamilmysliwiec is there anything that come to your mind that would help fixing / workaround this issue?

[artpumpkin]

Same here with .proto files.

[kamilmysliwiec]

Would you like to create a PR for this issue?

[earloc]

same here (for a handfull of hbs-files).

Occuring on Azure DevOps Ubuntu Build Agents.

We have a postbuild npm script, which ultimately triggers some actions expecting the hbs-files in dist to be present.

{
  "collection": "@nestjs/schematics",
  "sourceRoot": "src",
  "compilerOptions": {
    "assets": [
      {
        "include":"**/*.hbs",
        "outDir":"dist/src",
        "watch":true
      }
    ]
  }
}

[pintxxo]

@earloc can you point me out to some of those postbuild scripts ?

[earloc]

@pintxxo Basically just anything, which relies on the files being copied to the destination.

Really nothing special.

We are calling a utility script, which generates our open-API Definition by basically utilizing NestFactory. The way we included those *. hbs files requires them to be present at runtime, which they just aren't.

Somebody even prepended the call to this Script with a sleep 0.5, to give the copy-operation a bit more time to finish. But this is flaky as well 😅, just not as often anymore.