mwoffliner:dev cannot download articles with forward slash (/) in the title when using --forceRender="RestApi"

[SageSystems]

It is not possible to download articles from the RestApi with forward slashes

example below:

podman run --volume=/home/sagesys/Downloads/out -ti ghcr.io/openzim/mwoffliner:dev mwoffliner --mwUrl https://bulbapedia.bulbagarden.net --adminEmail r.com --speed 1 --requestTimeout 1200 --forceRender="RestApi"
starting redis-server in the background…
[error] [2024-11-30T00:46:04.383Z] Error downloading article Steelix_(Pokémon)/Generation_III_learnset
[error] [2024-11-30T00:46:05.861Z] Error downloading article Tornadus_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:46:06.036Z] Error downloading article Donphan_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:46:11.388Z] Error downloading article List_of_Battle_Subway_Trainers/Biker
[error] [2024-11-30T00:46:13.089Z] Error downloading article Blaziken_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:13.921Z] Error downloading article Ilima/Quotes
[error] [2024-11-30T00:46:16.476Z] Error downloading article Arceus_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:17.881Z] Error downloading article Gastly_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:18.310Z] Error downloading article Rhydon_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:46:20.009Z] Error downloading article Rosa_(game)/Pokéstar_Studios
[error] [2024-11-30T00:46:21.646Z] Error downloading article Garbodor_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:25.546Z] Error downloading article Baltoy_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:46:30.219Z] Error downloading article List_of_Battle_Tower_Trainers_in_Pokémon_Brilliant_Diamond_and_Shining_Pearl/Jogger
[error] [2024-11-30T00:46:32.950Z] Error downloading article Wild_Area_News/2021
[error] [2024-11-30T00:46:34.355Z] Error downloading article Bellsprout_(Pokémon)/Generation_V_learnset
[error] [2024-11-30T00:46:34.876Z] Error downloading article List_of_Battle_Subway_Trainers/Policeman
[error] [2024-11-30T00:46:37.655Z] Error downloading article Chatot_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:46:38.717Z] Error downloading article Castform_(Pokémon)/Generation_V_learnset
[error] [2024-11-30T00:46:43.129Z] Error downloading article Omastar_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:45.414Z] Error downloading article Poliwrath_(Pokémon)/Generation_IV_learnset
[error] [2024-11-30T00:46:46.212Z] Error downloading article Tower_Colosseum/Battle_Mode_Double
[error] [2024-11-30T00:46:46.582Z] Error downloading article Golduck_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:46:48.065Z] Error downloading article Carvanha_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:51.948Z] Error downloading article Medicham_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:46:53.346Z] Error downloading article Pokémon_League_(Alola)/Title_Defense
[error] [2024-11-30T00:46:58.560Z] Error downloading article Cufant_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:46:59.228Z] Error downloading article Hoppip_(Pokémon)/Generation_IV_learnset
[error] [2024-11-30T00:46:59.964Z] Error downloading article Timburr_(Pokémon)/Generation_V_learnset
[error] [2024-11-30T00:47:00.823Z] Error downloading article Metang_(Pokémon)/Generation_III_learnset
[error] [2024-11-30T00:47:04.930Z] Error downloading article Machop_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:47:09.130Z] Error downloading article Corviknight_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:47:14.010Z] Error downloading article Magikarp_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:47:18.506Z] Error downloading article Cutiefly_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:47:24.121Z] Error downloading article Ekans_(Pokémon)/Generation_V_learnset
[error] [2024-11-30T00:47:24.812Z] Error downloading article Cubone_(Pokémon)/Generation_V_learnset
[error] [2024-11-30T00:47:27.945Z] Error downloading article Weedle_(Pokémon)/Generation_II_learnset
[error] [2024-11-30T00:47:29.061Z] Error downloading article Popplio_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:47:30.360Z] Error downloading article Togetic_(Pokémon)/Generation_V_learnset
[error] [2024-11-30T00:47:32.158Z] Error downloading article Arcanine_(Pokémon)/Generation_I_learnset
[error] [2024-11-30T00:47:33.209Z] Error downloading article Naganadel_(Pokémon)/Generation_VII_learnset
[error] [2024-11-30T00:47:35.479Z] Error downloading article Taillow_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:47:35.732Z] Error downloading article Mina/Quotes
[error] [2024-11-30T00:47:38.174Z] Error downloading article Nidorino_(Pokémon)/Generation_I_learnset
[error] [2024-11-30T00:47:39.717Z] Error downloading article Jirachi_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:47:40.276Z] Error downloading article Dragapult_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:47:41.190Z] Error downloading article List_of_Battle_Tower_Trainers_in_Pokémon_Ruby_and_Sapphire/Ninja_Boy
[error] [2024-11-30T00:47:42.931Z] Error downloading article Hawlucha_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:47:44.137Z] Error downloading article Koga/Quotes
[error] [2024-11-30T00:47:44.433Z] Error downloading article List_of_Battle_Tower_Trainers_in_Pokémon_Ruby_and_Sapphire/Rich_Boy
[error] [2024-11-30T00:47:46.727Z] Error downloading article Charmeleon_(Pokémon)/Generation_II_learnset
[error] [2024-11-30T00:47:47.435Z] Error downloading article Starmie_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:47:56.217Z] Error downloading article Dwebble_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:47:56.456Z] Error downloading article Quilava_(Pokémon)/Generation_II_learnset
[error] [2024-11-30T00:47:58.204Z] Error downloading article Rayquaza_(Pokémon)/Generation_VI_learnset
[error] [2024-11-30T00:48:01.070Z] Error downloading article Mudbray_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:48:02.007Z] Error downloading article Scolipede_(Pokémon)/Generation_VIII_learnset
[error] [2024-11-30T00:48:04.116Z] Error downloading article Stantler_(Pokémon)/Generation_II_learnset
[error] [2024-11-30T00:48:07.136Z] Error downloading article V_/_Santa_Claus_at_850_Yen_Per_Hour

[audiodude]

Thanks for the report!

There might be something wrong with the server configuration, because even when I percent encode the name, I can't find the page:

https://bulbapedia.bulbagarden.net/w/rest.php/v1/page/Steelix_%28Pok%C3%A9mon%29%2FGeneration_III_learnset

Also the error code is coming from Apache, not MediaWiki, so it might be something in the way Apache is processing paths under that directory.

[kelson42]

Yes, this things are sanitized/normalised at start AFAIK. Of course, we could always have a bug there, but this has been specifically adressed.

[audiodude]

That's right @kelson42. The code here escapes the article name when building the URL:

https://github.com/openzim/mwoffliner/blob/main/src/util/builders/url/rest-api.director.ts#L16

[kelson42]

I use to test this with "AC/DC" on Wikipedia, but we need to be careful and test as well with what MediaWiki call subpages (lots of them on Wikibooks), which also have slashes but are handled slightly differently.

[audiodude]

Agreed. I believe the pages listed in the original post are all subpages in that sense, yes.