-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(eas-cli): swap node-fetch
for undici
to support Node 22 better
#2414
Draft
byCedric
wants to merge
3
commits into
main
Choose a base branch
from
@bycedric/eas-cli/swap-node-fetch-for-undici
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
const undici = require('undici'); | ||
|
||
// This workaround swaps `undici.fetch` for `global.fetch` to connect Nock with Undici. | ||
// See: https://github.com/nock/nock/issues/2183 | ||
require('nock'); | ||
|
||
module.exports = { ...undici, fetch: global.fetch }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,104 @@ | ||
import nock from 'nock'; | ||
|
||
import { getExpoApiBaseUrl } from '../../api'; | ||
import { RequestError } from '../../fetch'; | ||
import { wrapFetchWithProgress } from '../download'; | ||
|
||
describe(wrapFetchWithProgress, () => { | ||
const url = getExpoApiBaseUrl(); | ||
|
||
it('returns response with body', async () => { | ||
nock(url) | ||
.get('/test-body') | ||
.reply(200, 'success', { 'Content-Length': String(Buffer.byteLength('success')) }); | ||
|
||
const response = await wrapFetchWithProgress()(`${url}/test-body`, {}, jest.fn()); | ||
|
||
expect(await response.text()).toBe('success'); | ||
}); | ||
|
||
it('calls progress handler when loading body', async () => { | ||
const testSize = 1024 * 1024; // 1MB | ||
|
||
nock(url) | ||
.get('/test-progress') | ||
.reply(200, Buffer.alloc(testSize), { 'Content-Length': String(testSize) }); | ||
|
||
const progressTracker = jest.fn(); | ||
const fetchWithProgress = wrapFetchWithProgress(); | ||
const response = await fetchWithProgress(`${url}/test-progress`, {}, progressTracker); | ||
|
||
// Response should be successful | ||
expect(response).toMatchObject({ ok: true }); | ||
// Load the the response body to trigger the progress events | ||
expect(await response.blob()).not.toBeUndefined(); | ||
// Progress tracker should start at 0% | ||
expect(progressTracker).toHaveBeenCalledWith({ | ||
isComplete: false, | ||
progress: { | ||
total: testSize, | ||
percent: 0, | ||
transferred: 0, | ||
}, | ||
}); | ||
// Progress tracker should end at 100% | ||
expect(progressTracker).toHaveBeenCalledWith({ | ||
isComplete: true, | ||
progress: { | ||
total: testSize, | ||
percent: 1, | ||
transferred: testSize, | ||
}, | ||
}); | ||
}); | ||
|
||
it('skips progress events when request fails', async () => { | ||
nock(url) | ||
.get('/test-fail') | ||
.reply(404, 'Not Found', { 'Content-Length': String(Buffer.byteLength('Not Found')) }); | ||
|
||
const progressTracker = jest.fn(); | ||
const response = await wrapFetchWithProgress()(`${url}/test-fail`, {}, progressTracker).catch( | ||
(requestError: RequestError) => requestError.response | ||
); | ||
|
||
// Response should not be successful | ||
expect(response).toMatchObject({ ok: false }); | ||
// Repsonse should contain our error message | ||
expect(await response.text()).toBe('Not Found'); | ||
// No progression events should be called | ||
expect(progressTracker).not.toHaveBeenCalled(); | ||
}); | ||
|
||
it('skips progress events when response is empty', async () => { | ||
nock(url).get('/test-empty').reply(204, undefined, { 'Content-Length': '0' }); | ||
|
||
const progressTracker = jest.fn(); | ||
const response = await wrapFetchWithProgress()(`${url}/test-empty`, {}, progressTracker); | ||
|
||
// Response should be successful | ||
expect(response).toMatchObject({ ok: true }); | ||
// Body should be empty | ||
expect(await response.text()).toBe(''); | ||
// No progression events should be called | ||
expect(progressTracker).not.toHaveBeenCalled(); | ||
}); | ||
|
||
it('skips progress events when no content-length header is available', async () => { | ||
nock(url).get('/test-missing-content-length').reply(200, 'success'); | ||
|
||
const progressTracker = jest.fn(); | ||
const response = await wrapFetchWithProgress()( | ||
`${url}/test-missing-content-length`, | ||
{}, | ||
progressTracker | ||
); | ||
|
||
// Response should be successful | ||
expect(response).toMatchObject({ ok: true }); | ||
// Body should be empty | ||
expect(await response.text()).toBe('success'); | ||
// No progression events should be called | ||
expect(progressTracker).not.toHaveBeenCalled(); | ||
}); | ||
}); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this fix the type error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately not, it seems that typescript interprets the
@types/node
differently.TL;DR;
url: string | URL | globalThis.Request
(see here)url: RequestInfo
, wheretype RequestInfo = string | URL | Request
(see here)These should not conflict, but I have a feeling that it may be caused by this class extension in
@types/node
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we might be dropping support for Node 16 (see PR #2413), we could just avoid importing
fetch
from undici though. That should resolve this typing issue, and remove the need for thenock
<>undici
mock/workaround as well.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Opening a stacked PR for this, we need to refactor more than just removing the
import { fetch } from 'undici'
. This stacked PR may or may not succeed, depending on how much work still is required.#2419
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, so there is an issue. The summary is:
FormData
is used from theform-data
package, which is old and usesutil.isArray
(last update was 3y ago)form-data
is different fromundici
'sFormData
/ Node built-inFormData
form-data
supports Node'sfs.createReadStream
instances (ReadStream
types)undici
does NOT support Node'sReadStream
type, and only works withFile
(added in18.13.0
) or Blob (added in14.x.x
)20.x.x
, there is afs.openAsBlob
method to stream files as blobs, and pass it toFormData
, but we support Node 18 stillThere are rather "dirty" hacks to work around this limitation, e.g. this one. I also have concerns with
undici
'sFormData
not sending theContent-Length
header.Right now,
undici
also doesn't play nice with theform-data
package. We'd need to add aReadable.toWeb(form)
to make it work.For now, I've reached my timebox. Happy to circle back to this later though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is an example request made, when using the workaround from SO. It's still missing the
Content-Length
header when streaming. When using a blob (in-memory read), it adds the header.Using built-in fetch, built-in FormData, and file stream
Using built-in fetch, built-in FormData, and Blob (loading in-memory)
Using built-in fetch, built-in FormData, and fs.openAsBlob (streaming, but Node 20+)
While, originally, this is the request made:
Using node-fetch, form-data, and file stream
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Node 18's end of life will be in April 2025. So it's not soon, but eventually we'll be able to rely on
fs.openAsBlob
. Could you document these findings in a task assigned to yourself so that it's easier to follow up on this after we've dropped Node 18?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One workaround for Node 18 that I've found to work well is this one:
This seems to run fine on Node 18 for ubuntu, macos, and windows. It's still leveraging Node 20+ APIs, but with a weird fallback. It's not spec-compliant, but enough compliant for
FormData
itself, and includes thecontent-length
header we need.