Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

readAndParseFile add error handling #24

Open
MSIH opened this issue Mar 28, 2021 · 2 comments
Open

readAndParseFile add error handling #24

MSIH opened this issue Mar 28, 2021 · 2 comments

Comments

@MSIH
Copy link

MSIH commented Mar 28, 2021

I am getting error about Json not in correct format.

Please add error handle and include path to bad file.

@mnmkng
Copy link
Member

mnmkng commented Mar 29, 2021

Sorry, will need a bit more info before we can look into this. It would be great if you could provide it in the following format:

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Add an example of code that produces the problem you're reporting.
We should be able to copy your code, run it and get the same result.

Expected behavior
A clear and concise description of what you expected to happen.

System information:

  • OS: [e.g. MacOS]
  • Node.js version [e.g. 14.7]
  • library version [e.g. 0.1.4]

Additional context
Add any other context about the problem here.

@MSIH
Copy link
Author

MSIH commented Mar 30, 2021

I am on holiday so do not have access to the exact error. But sometimes when I run this code it will crash with an error about content not being json. this is odd because all of the files in the dataset folder were created using this library, so they should be json. The dataset has 22000 records. I am running on windows 10. This only happens when I set the offset and limit parameters. if they are set to 0 then works every time.

const openDataset = await Apify.openDataset(sourceDataset);
    const {
        items
    } = await openDataset.getData({
        offset: offset,
        limit: limit
    });

My request is you add try/catch to the _readAndParseFile function when converting to json and add to the exception message the exact file or full path to the file that failed.

Thank

private async _readAndParseFile(index: number): Promise<Record<string, unknown>> {

        const filePath = join(this.storeDir, this._getItemFileName(index));
const json = await readFile(filePath, 'utf8');
        return JSON.parse(json);
    }

looks like you already are doing it here:
https://github.com/apify/apify-cli/blob/6cd38ba1d9f3574147681f255ddc37da4e0feeba/src/commands/vis.js#L9

            try {
            const inputSchemaStr = fs.readFileSync(path).toString();
            inputSchemaObj = JSON.parse(inputSchemaStr);
        } catch (err) {
            throw new Error(`Input schema is not a valid JSON (${err})`);
        }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants