Distributor Node Quality Checks - 1

[bwhm]

Workflow

Dev points: 15
JSG Reviewer: @DzhideX

Notes

PR to be made to this repo
Feel free to ask questions, but ask them publicly in the issue, and ping me on discord bwhm#6514 so I'll see it :)

Scope

Measuring the quality of service by the Distributors without proper tools is a very manual (and error prone) job.

First, we create a tool that fetches thumbnails from the front page. The tool has to be somewhat customizable, as in reality, users don't need to get every single asset at the same time.

Scope 1

Use puppeteer, or a similar free and open source tool, to create a script that mimics a user opening play.joystream.org and other sub-pages without caching locally.

Each time the script runs, it logs:

• when the test was run
• which files were requested, and for each of these:
  • from which distributor (URL) it was fetched
  • how long it took to get a response from the server
  • how big the file is, and the time/speed it took to render

Then, get the latency to each distributor, and get some data from the query-node to determine:

• Average response time for each distributor after latency has been removed
• Average response time for the system as a whole
• Average speed for each distributor
• Average speed for the system as a whole

To avoid unfair rulings:

• do not include the time it takes to get a response from orion.joystream.org.
• ideally, verify that the file was accepted

[singulart]

As per @traumschule this is covered by #679. Moving to "In Review"

[traumschule]

Covered would mean fully implemented. The code can be reused but does not have the purpose to simulate atlas user experience with puppeteer. It performs axios head (no download speed test) requests for each asset and sends results to a (jsstats) backend for collection (simply accepting unauthorized post requests and very easy to re-implement). All code for comparison between nodes or averages stored by a QN would need to be added there.
Also the jsstats site is no longer under active development, happy to share code if relevant though. i do not plan to spend time on that right now, feel free to use and complement according to needs.

[singulart]

@freakstatic any updated on the internal review?

[freakstatic]

@freakstatic any updated on the internal review?

sorry for the late reply, I was able to run it and got this file as output.
this tool can be handy to check if assets are available in the SP and DP but doesn't seem to be good for checking the quality of the QN as this issue requests

[freakstatic]

so I suggest that we develop something with puppeteer if the objective is to check how good are the distributors when using atlas.
if we gonna check how good are distributors in general I suggest adapting another tool which was done for SP's

[singulart]

Ok, clear. Moving back to To Do, then

[traumschule]

Jsgenesis Review was never the right status anyway.
JSG performed these tests. Latest THUMBNAIL_SCORE was 1, SERVICE_SCORE 0.997 and PLAYABLITIY_SCORE will be replaced:

Note that this will be changed in the future, as it turns out to be a flawed metric.

Requirements

[adovrodion]

Bounty link - https://dao.joystream.org/#/bounty/preview/68
UPD: Bounty was not fulfilled

[singulart]

Bounty submission failed the internal review, sent back to the worker and got stuck. Moving back to ToDo. Please consider this vacant and ready to be picked up.

[m00n620]

excited to join!

[singulart]

Assigned to @m00n620

[singulart]

Worker asked to un-assign this piece of work.