Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(eas-cli): swap node-fetch for undici to support Node 22 better #2414

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This is the log of notable changes to EAS CLI and related packages.
### 🐛 Bug fixes

- Correctly parse the EXPO_APPLE_PROVIER_ID environment variable. ([#2349](https://github.com/expo/eas-cli/pull/2349) by [@louix](https://github.com/louix))
- Swapped `node-fetch` for `undici` to support Node 22. ([#2414](https://github.com/expo/eas-cli/pull/2414) by [@byCedric](https://github.com/byCedric))

### 🧹 Chores

Expand Down
7 changes: 7 additions & 0 deletions packages/eas-cli/__mocks__/undici.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
const undici = require('undici');

// This workaround swaps `undici.fetch` for `global.fetch` to connect Nock with Undici.
// See: https://github.com/nock/nock/issues/2183
require('nock');

module.exports = { ...undici, fetch: global.fetch };
5 changes: 2 additions & 3 deletions packages/eas-cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@
"mime": "3.0.0",
"minimatch": "5.1.2",
"nanoid": "3.3.4",
"node-fetch": "2.6.7",
"node-forge": "1.3.1",
"nullthrows": "1.1.1",
"ora": "5.1.0",
Expand All @@ -81,6 +80,7 @@
"terminal-link": "2.1.1",
"tslib": "2.6.2",
"turndown": "7.1.2",
"undici": "6.18.2",
"untildify": "4.0.0",
"uuid": "9.0.1",
"wrap-ansi": "7.0.0"
Expand All @@ -98,7 +98,6 @@
"@types/getenv": "^1.0.0",
"@types/jsonwebtoken": "8.5.1",
"@types/mime": "3.0.1",
"@types/node-fetch": "2.6.2",
"@types/node-forge": "1.3.1",
"@types/pngjs": "6.0.4",
"@types/promise-retry": "1.1.3",
Expand All @@ -113,7 +112,7 @@
"express": "4.19.2",
"memfs": "3.4.13",
"mockdate": "3.0.5",
"nock": "13.4.0",
"nock": "14.0.0-beta.7",
"rimraf": "3.0.2",
"ts-deepmerge": "6.2.0",
"ts-mockito": "2.6.1",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,9 @@ import {
} from '@urql/core';
import { retryExchange } from '@urql/exchange-retry';
import { DocumentNode } from 'graphql';
import fetch from 'node-fetch';

import { getExpoApiBaseUrl } from '../../../api';
import { httpsProxyAgent } from '../../../fetch';
import fetch, { RequestError, RequestInfo, RequestInit } from '../../../fetch';

export interface ExpoGraphqlClient extends Client {
query<Data = any, Variables extends AnyVariables = AnyVariables>(
Expand Down Expand Up @@ -44,19 +43,17 @@ export function createGraphqlClient(authInfo: {
}),
fetchExchange,
],
// @ts-expect-error Type 'typeof fetch' is not assignable to type '(input: RequestInfo, init?: RequestInit | undefined) => Promise<Response>'.
fetch,
fetchOptions: (): RequestInit => {
// @ts-expect-error - Type '(url: RequestInfo, init?: RequestInit) => Promise<Response>' is not assignable to type '{ (input: RequestInfo | URL, init?: RequestInit | undefined): Promise<Response>; (input: string | Request | URL, init?: RequestInit | undefined): Promise<...>; }'.
fetch: (url: RequestInfo, init?: RequestInit) =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this fix the type error?

Suggested change
fetch: (url: RequestInfo, init?: RequestInit) =>
fetch: (url: RequestInfo | URL, init?: RequestInit) =>

Copy link
Member Author

@byCedric byCedric Jun 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately not, it seems that typescript interprets the @types/node differently.

TL;DR;

  • in Node, they use url: string | URL | globalThis.Request (see here)
  • in Undici, they use url: RequestInfo, where type RequestInfo = string | URL | Request (see here)

These should not conflict, but I have a feeling that it may be caused by this class extension in @types/node

Copy link
Member Author

@byCedric byCedric Jun 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we might be dropping support for Node 16 (see PR #2413), we could just avoid importing fetch from undici though. That should resolve this typing issue, and remove the need for the nock <> undici mock/workaround as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opening a stacked PR for this, we need to refactor more than just removing the import { fetch } from 'undici'. This stacked PR may or may not succeed, depending on how much work still is required.

#2419

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so there is an issue. The summary is:

  • FormData is used from the form-data package, which is old and uses util.isArray (last update was 3y ago)
  • form-data is different from undici's FormData / Node built-in FormData
  • form-data supports Node's fs.createReadStream instances (ReadStream types)
  • undici does NOT support Node's ReadStream type, and only works with File (added in 18.13.0) or Blob (added in 14.x.x)
  • Starting from Node 20.x.x, there is a fs.openAsBlob method to stream files as blobs, and pass it to FormData, but we support Node 18 still

There are rather "dirty" hacks to work around this limitation, e.g. this one. I also have concerns with undici's FormData not sending the Content-Length header.

Right now, undici also doesn't play nice with the form-data package. We'd need to add a Readable.toWeb(form) to make it work.

For now, I've reached my timebox. Happy to circle back to this later though.

Copy link
Member Author

@byCedric byCedric Jun 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is an example request made, when using the workaround from SO. It's still missing the Content-Length header when streaming. When using a blob (in-memory read), it adds the header.

Using built-in fetch, built-in FormData, and file stream image
Using built-in fetch, built-in FormData, and Blob (loading in-memory) image
Using built-in fetch, built-in FormData, and fs.openAsBlob (streaming, but Node 20+) image

While, originally, this is the request made:

Using node-fetch, form-data, and file stream image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Node 18's end of life will be in April 2025. So it's not soon, but eventually we'll be able to rely on fs.openAsBlob. Could you document these findings in a task assigned to yourself so that it's easier to follow up on this after we've dropped Node 18?

Copy link
Member Author

@byCedric byCedric Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One workaround for Node 18 that I've found to work well is this one:

// Create a FormData object, and populate data
const form = new FormData();
const file = await createBlobFromPath(testFile);
form.append('some-value', 'value');
form.append('file', file, path.basename(testFile));


// Use `fs.openAsBlob` if available, else polyfill on Node 18
async function createBlobFromPath(filePath) {
  if (typeof fs.openAsBlob === 'function') {
    console.log('Using fs.openAsBlob:', filePath);
    return await fs.openAsBlob(filePath);
  }

  console.log('Falling back to File polyfill:', filePath);
  const { size } = await fs.promises.stat(filePath);

  return {
    [Symbol.toStringTag]: 'File', // Trick Node into thinking this is a File reference
    get type() { return undefined }, // Always undefined, it's the mime type
    get size() { return size }, // Ensure the `content-length` header is set through form data
    stream() { return Readable.toWeb(fs.createReadStream(filePath)) }, // The magic part that makes it work

    // Not necessary as we override the filename through formdata
    // get name() { return name },

    // Could add the full API, but it's not necessary for form data
    // arrayBuffer(...args) { return this.stream().arrayBuffer(...args) },
    // text(...args) { return this.stream().text(...args) },
    // slice(...args) { return this.stream().slice(...args) }
  };
}

This seems to run fine on Node 18 for ubuntu, macos, and windows. It's still leveraging Node 20+ APIs, but with a weird fallback. It's not spec-compliant, but enough compliant for FormData itself, and includes the content-length header we need.

fetch(url, init).catch((error: RequestError) => error.response),
fetchOptions: () => {
const headers: Record<string, string> = {};
if (authInfo.accessToken) {
headers.authorization = `Bearer ${authInfo.accessToken}`;
} else if (authInfo.sessionSecret) {
headers['expo-session'] = authInfo.sessionSecret;
}
return {
...(httpsProxyAgent ? { agent: httpsProxyAgent } : {}),
headers,
};
return { headers };
},
}) as ExpoGraphqlClient;
}
41 changes: 23 additions & 18 deletions packages/eas-cli/src/fetch.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
import { Agent } from 'https';
import createHttpsProxyAgent from 'https-proxy-agent';
import fetch, { RequestInfo, RequestInit, Response } from 'node-fetch';
import { env } from 'node:process';
import {
ProxyAgent,
type RequestInfo,
type RequestInit,
type Response,
fetch,
getGlobalDispatcher,
setGlobalDispatcher,
} from 'undici';

export * from 'node-fetch';
export { Agent, Headers, type RequestInfo, type RequestInit, Response } from 'undici';

export class RequestError extends Error {
constructor(
Expand All @@ -13,23 +20,21 @@ export class RequestError extends Error {
}
}

function createHttpsAgent(): Agent | null {
const httpsProxyUrl = process.env.https_proxy;
if (!httpsProxyUrl) {
return null;
}
return createHttpsProxyAgent(httpsProxyUrl);
}

export const httpsProxyAgent: Agent | null = createHttpsAgent();

export default async function (url: RequestInfo, init?: RequestInit): Promise<Response> {
const response = await fetch(url, {
...init,
...(httpsProxyAgent ? { agent: httpsProxyAgent } : {}),
});
installProxyAgent();

const response = await fetch(url, init);
if (response.status >= 400) {
throw new RequestError(`Request failed: ${response.status} (${response.statusText})`, response);
}

return response;
}

function installProxyAgent(): void {
const httpsProxyUrl = env.https_proxy;

if (httpsProxyUrl && !(getGlobalDispatcher() instanceof ProxyAgent)) {
setGlobalDispatcher(new ProxyAgent(httpsProxyUrl));
}
}
3 changes: 1 addition & 2 deletions packages/eas-cli/src/uploads.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
import FormData from 'form-data';
import fs from 'fs-extra';
import { Response } from 'node-fetch';
import promiseRetry from 'promise-retry';

import { ExpoGraphqlClient } from './commandUtils/context/contextUtils/createGraphqlClient';
import fetch from './fetch';
import fetch, { type Response } from './fetch';
import { UploadSessionType } from './graphql/generated';
import { SignedUrl, UploadSessionMutation } from './graphql/mutations/UploadSessionMutation';
import { ProgressHandler } from './utils/progress';
Expand Down
104 changes: 104 additions & 0 deletions packages/eas-cli/src/utils/__tests__/download-test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
import nock from 'nock';

import { getExpoApiBaseUrl } from '../../api';
import { RequestError } from '../../fetch';
import { wrapFetchWithProgress } from '../download';

describe(wrapFetchWithProgress, () => {
const url = getExpoApiBaseUrl();

it('returns response with body', async () => {
nock(url)
.get('/test-body')
.reply(200, 'success', { 'Content-Length': String(Buffer.byteLength('success')) });

const response = await wrapFetchWithProgress()(`${url}/test-body`, {}, jest.fn());

expect(await response.text()).toBe('success');
});

it('calls progress handler when loading body', async () => {
const testSize = 1024 * 1024; // 1MB

nock(url)
.get('/test-progress')
.reply(200, Buffer.alloc(testSize), { 'Content-Length': String(testSize) });

const progressTracker = jest.fn();
const fetchWithProgress = wrapFetchWithProgress();
const response = await fetchWithProgress(`${url}/test-progress`, {}, progressTracker);

// Response should be successful
expect(response).toMatchObject({ ok: true });
// Load the the response body to trigger the progress events
expect(await response.blob()).not.toBeUndefined();
// Progress tracker should start at 0%
expect(progressTracker).toHaveBeenCalledWith({
isComplete: false,
progress: {
total: testSize,
percent: 0,
transferred: 0,
},
});
// Progress tracker should end at 100%
expect(progressTracker).toHaveBeenCalledWith({
isComplete: true,
progress: {
total: testSize,
percent: 1,
transferred: testSize,
},
});
});

it('skips progress events when request fails', async () => {
nock(url)
.get('/test-fail')
.reply(404, 'Not Found', { 'Content-Length': String(Buffer.byteLength('Not Found')) });

const progressTracker = jest.fn();
const response = await wrapFetchWithProgress()(`${url}/test-fail`, {}, progressTracker).catch(
(requestError: RequestError) => requestError.response
);

// Response should not be successful
expect(response).toMatchObject({ ok: false });
// Repsonse should contain our error message
expect(await response.text()).toBe('Not Found');
// No progression events should be called
expect(progressTracker).not.toHaveBeenCalled();
});

it('skips progress events when response is empty', async () => {
nock(url).get('/test-empty').reply(204, undefined, { 'Content-Length': '0' });

const progressTracker = jest.fn();
const response = await wrapFetchWithProgress()(`${url}/test-empty`, {}, progressTracker);

// Response should be successful
expect(response).toMatchObject({ ok: true });
// Body should be empty
expect(await response.text()).toBe('');
// No progression events should be called
expect(progressTracker).not.toHaveBeenCalled();
});

it('skips progress events when no content-length header is available', async () => {
nock(url).get('/test-missing-content-length').reply(200, 'success');

const progressTracker = jest.fn();
const response = await wrapFetchWithProgress()(
`${url}/test-missing-content-length`,
{},
progressTracker
);

// Response should be successful
expect(response).toMatchObject({ ok: true });
// Body should be empty
expect(await response.text()).toBe('success');
// No progression events should be called
expect(progressTracker).not.toHaveBeenCalled();
});
});
Loading
Loading