Stream scalability issues

[rubensworks]

This is a master issue that acts as an overview of the following issues:

• Ensure backpressure is maintained in streams #71
• Performance issues / memory core dumps with big files #65
• Stream goes into flowing mode immediately #66

[rubensworks]

For reference, this issue will require switching to a new internal JSON parser.
My tests indicate that stream-json is probably the way to go: https://github.com/rubensworks/test-performance-json-parse-stream

[thadguidry]

@ohler55 has some nice performant JSON parsers in his repo's https://github.com/ohler55

[rubensworks]

@wouterbeek @LaurensRietveld Do you still want to go forward with this bounty? @Tpt would be available to take up this issue.

[wouterbeek]

@rubensworks Definitely; thanks @Tpt for looking into this!

[Tpt]

As discussed with @rubensworks, I will work on this issue via the Comunica Association.

[Tpt]

Sadly stream-json does not work out of the box with web browser: uhop/stream-json#91

So, we might have to pick an other parser.

[rubensworks]

Sadly stream-json does not work out of the box with web browser: uhop/stream-json#91
So, we might have to pick an other parser.

Ah, that's too bad...

Any suggestions for alternatives?
Perhaps some web-compatible forks exist of that package? Or just some fully different implementations?

If no alternatives exist, we could propose a PR with making that lib web-compatible. I would guess the effort of that would be limited. (but then we'd first have to be certain that this lib really does everything we need)

[Tpt]

Ah, that's too bad...

Any suggestions for alternatives?
Perhaps some web-compatible forks exist of that package? Or just some fully different implementations?

Thank you! I have opened #100 to discuss it.

[thadguidry]

Streaming is a memory and IO bound operation. It's likely the best approach will be using WASM + Go and as a base, utilizing Peter Ohler's OjG http://www.ohler.com/ You can email him and he's very responsive and can answer any questions. The work effort will be around compiling the Go code into a .wasm file that you can launch with wasm_exec.js See https://binx.io/2022/04/22/golang-webassembly/ Essentially, compile Peter's code as needed into a .wasm file to be executed by javascript as needed for the heavy lifting of stream parsing JSON.

[rubensworks]

Thanks for the input @thadguidry.
While I think WASM is a very interesting direction, it would increase the scope of this issue quite significantly, and we don't have the bandwidth (and budget) to take this up unfortunately.

[thadguidry]

Ah, you want to stay in Javascript, no worries.

[rubensworks]

Unfortunately @Tpt was not able to continue working on this bounty anymore, so this bounty will be unlocked again for other developers.

Current status on this bounty is that a new library has been created to parse JSON in a streaming manner (memory-safe): https://github.com/comunica/json-event-parser.js

The next step in this bounty is to plug it into this parser, and handle parsing in a spec-compliant manner.