Unable to correct devdoc.io recipe

[kelson42]

If I want to change https://farm.openzim.org/recipes/devdocs.io/edit to put „WaitUntil“ to value „networkidle“ then I get an error:
https://farm.openzim.org/pipeline/cb6a074a-e33a-4552-9c45-932486a1dde9/debug

BTW the documentation of this field is insufficient as it does not list acceptable values and their respective behaviour.

We should have a selectbox here with possible values and input should be validated, see #783

[rgaudin]

Oh it's not select box ?!

[benoit74]

Problem was there were an old newContext flag which is not used (anymore probably) in zimit. It has nothing to do with the waitUntil flag you wanted to modified. I removed the newContext flag (it is the only recipe which had such a flag) and propose to continue the discussion on #783 about choice fields / invalid old flags / invalid old flag values. I've set the waitUntil to networkidle, I let you confirm the recipe is configured as intended and continue your work.

[kelson42]

Problem was there were an old newContext flag which is not used (anymore probably) in zimit.

Can we secure that when a scraper argument set is changed/updated, then the dataset is checked/fixed as well?

[kelson42]

I reopen, I have listed a bunch of dysfunctional things and we need proper handling at least a list of tickets.

[benoit74]

Can we secure that when a scraper argument set is changed/updated, then the dataset is checked/fixed as well?

This is now tracked in #911

I reopen, I have listed a bunch of dysfunctional things and we need proper handling at least a list of tickets.

Does it has anything to do with devdoc.io recipe? (it is the title of this issue, and I prefer to have one topic per issue)

[kelson42]

I reopen, I have listed a bunch of dysfunctional things and we need proper handling at least a list of tickets.

Does it has anything to do with devdoc.io recipe? (it is the title of this issue, and I prefer to have one topic per issue)

Tickets are and should be open from the user perspective AFAP. What comes next is the under the lead of the dev team. We can not just close ticket by knowing we have bugs/problems and hoping someone else will open the dedicated tickets.

So here is the list of things and I'm not convinced each of them need a dedicated ticket:

• Upgrade of scraper requires to check/update recipe data-set Regularly check that recipe flags are still valid #911
• Dataset should be checked for outdated newContext
• Document option waitUntil properly in Zimfarm (I understand we have tickets around this, be as a quickwin we could at least list the acceptable values and put a link to documentation).
• WaitUntil should be a selectbox

@benoit74 Maybe you can fix the dataset and I open a dedicated ticket for the field WaitUntil of the Zimit recipes?

[benoit74]

Dataset should be checked for outdated newContext

Done, as reported in #910 (comment)

WaitUntil should be a selectbox

I thought you already transferred this topic to #783

Document option waitUntil properly in Zimfarm (I understand we have tickets around this, be as a quickwin we could at least list the acceptable values and put a link to documentation).

Once the acceptable values are correctly controlled via a selectbox, it will be useless to list acceptable values in the usage. I wouldn't change the Zimfarm usage which is correct, the details you want to add are way too complex / long for it ; these details have to be explained properly, and this should be done in the scraper documentation from my PoV. I wouldn't stack as much of this documentation as possible in the usage. And yes scraper documentation needs to be done, but we are not making much progress unfortunately.

[kelson42]

@benoit74 OK, will look to close this ticket if everything OK.