Itch.io Integration

[Aeonitis]

Is your feature request related to a problem? Please describe.
Integration of Itch.io games into the Junkstore plugin. First of all, I love all your work here!

I would love to do this but I am unfortunately busy on my own non-steam app :'( Just adding this here so you can correct me, and make it easier for others to jump in and smash it into reality. No rush in responding, take your time!

Draft Plan

### Operation Itch
- [ ] API - User Browser Third-Party Login
- [ ] API - Fetch my_games list
- [ ] API - Get game details
- [ ] API - Download images
- [ ] API - Install Game
- [ ] DB  - Persist to db GameSet
- [ ] UI  - Update UX for download progress
- [ ] UI  - Update UI (e.g. Itch Tab)

Next Steps

• CLI Auth Login via Butler |Site & Github| may be needed, if the CLI Legendary tool can't work for Itch, then an Itch.io specific launcher (just for py or CLI-based) would be nice. I haven't confirmed if this GOG plugin for Legendary was used here, just discovered there is a dev branch, info might be there.
• Confirm process the web browser currently opened for user log in to capture user token within game mode.
• More questions in Questions section below.

Current Observations [Epic Integration]

Login

How the third-party in-game-mode browser user login is handled for other stores (Epic)...

• Login setup initiated via Plugin in main.py
  

  junkstore/main.py

  Line 240 in e115250

  if os.path.exists(os.path.join(decky_plugin.DECKY_PLUGIN_RUNTIME_DIR, "init.json")):

  
  static.json button

  junkstore/defaults/scripts/Extensions/Epic/static.json

  Line 88 in e115250

  "Id": "Login",

• static.json->junk-store.sh constructs function name based on the platform Epic and the action login
• Now valid function exists Epic_login store.sh
  

  junkstore/defaults/scripts/Extensions/Epic/store.sh

  Line 308 in e115250

  function Epic_login(){

  
  where the steam env variables ensure the environment specific to Steam context is correctly set
  

  junkstore/defaults/scripts/shared.sh

  Line 200 in e115250

  function get_steam_env() {

  
  and it moves on to
  

  junkstore/defaults/scripts/Extensions/Epic/store.sh

  Line 312 in e115250

  function loginlaunchoptions () {

{
   "Type": "RunExe",
   "Content": {
       "Exe": "/path/to/DECKY_PLUGIN_DIR/scripts/Extensions/Epic/login.sh",
       "Options": "",
       "WorkingDir": "/path/to/DECKY_PLUGIN_LOG_DIR",
       "Name": "Epic Games Login",
       "Compatibility": "",
       "CompatToolName": ""
   }
}

to construct a JSON for login...

Note: UNSURE why a JSON needs to be constructed, see in Questions section below.

Uses tool to authenticate via CLI on Epic specific launcher Legendary

junkstore/defaults/scripts/Extensions/Epic/login.sh

Line 13 in e115250

$LEGENDARY auth -v &>> "${DECKY_PLUGIN_LOG_DIR}/epiclogin.log"

• Legendary CLI Auth which I assume works like other auth-cli
  • sets up port internally
  • opens site to specific url, probably with with params too.
  • Mints a new key or token.
  • If auth is not cancelled/failed, Key/Token received by CLI as success page is shown.
  • Moving forward, that Key/Token is used in Header to interact with API

Arg info from Legendary documentation -v, --debug means Set loglevel to debug

Current Auth ID data in game logs, stored as appId.log e.g. cd1b8a6********************cf7b7f536.log which is 'Night in the Woods'.

-AUTH_LOGIN=unused: Login method not being used, userId is optional.
-AUTH_PASSWORD=8fe9*****************************422e: Hashed user password used for authentication.
-AUTH_TYPE=exchangecode: Specifies type of authentication being used, exchange code.
-epicapp=cd1b8a6********************cf7b7f536: Unique id for this Epic Games app.
-epicenv=Prod: Set to production.
-epicusername=Aeonitis: Your Epic Games account username.
-epicuserid=8d7********************************6c: Unique user ID for the Epic Games account.
-epiclocale=en: locale/language as English.
-epicsandboxid=abd***********************************3: Identifier for the sandbox environment being used.
Running: “/home/deck/.local/share/Steam/ubuntu12_32/reaper”: Path to executable being run.
“SteamLaunch” “AppId=4**********7” “–”: Steam launcher being used with specified app ID.
“/home/deck/.local/share/Steam/ubuntu12_32/steam-launch-wrapper”: Path to the Steam launch wrapper being used.

Client caching on deck is handled by sqlite 3:

junkstore/defaults/scripts/shared/GamesDb.py

Line 2 in befe4d0

import sqlite3

&

junkstore/defaults/scripts/shared/GameSet.py

Line 10 in befe4d0

import sqlite3

Questions

Fundamental questions here to help make the process easier for whichever dev takes on some of this work, based on what I learned from skimming through the repo and especially this commit

• Why is a launchoptions JSON constructed instead of just running sh scripts with args?
• The get_game_info function in GamesDb.py seems to be GOG specific, where is the Epic store's equivalent function?
• It seems that you are not using the GOG python sdk, but the web api via shell scripts, is that the same for Epic?
• Are we supposed to use API keys for each store or our login for Epic & GOG, I didn't understand how logins work for each store yet. I'm assuming logins or API key is needed for downloads to work. I guess itch is free for most game downloads anyway, linking Itch.io API unsure if needed
• I have sample shell and python script below, which would you recommend for consistency with your current code (unsure why there was an epic-config.py but no gog-config.py) but would understand if it was the quickest way to get things done :)
• Is the UmuId related to Umu Launcher or Proton id of games stored in the GameSet.py, would anything additional be needed for Itch games (I guess I am asking if there is an UmuId or Proton ID designated for itch games?) I guess I am unsure of what further requirements are needed on that end.
• Seems like installs are done through @shared.sh, and displayed progress here @store.sh. The install_deps seems like a global junkstore deps installer, not app-specific...

Describe the solution we'd like

1. API integration

   • Sample Shell Script:

     #!/bin/bash
     
     # Replace 'OUR_API_KEY' with our actual itch.io API key
     API_KEY="OUR_API_KEY"
     
     # Function to get user info
     login() {
       curl -H "Authorization: Bearer $API_KEY" "https://itch.io/api/1/key/me"
     }
     
     # Function to get list of our games
     get_my_games() {
       curl -H "Authorization: Bearer $API_KEY" "https://itch.io/api/1/key/my-games"
     }
     
     # Function to download a game by ID
     download_game() {
       game_id=$1
       curl -H "Authorization: Bearer $API_KEY" "https://itch.io/api/1/key/game/$game_id/download"
     }
     
     # Example usage
     echo "Logging in..."
     login
     
     echo "Fetching list of games..."
     get_my_games
     
     # Replace 'GAME_ID' with ID of game we want to download
     GAME_ID="GAME_ID"
     echo "Downloading game with ID: $GAME_ID"
     download_game $GAME_ID

   • Sample Python Code:

     import requests
     
     # Replace 'OUR_API_KEY' with our actual itch.io API key
     api_key = 'OUR_API_KEY'
     headers = {'Authorization': f'Bearer {api_key}'}
     
     # Function to log in and get our user info
     def login():
         response = requests.get('https://itch.io/api/1/key/me', headers=headers)
         response.raise_for_status()  # Will raise an exception for HTTP errors
         return response.json()
     
     # Function to get a list of our games
     def get_my_games():
         response = requests.get('https://itch.io/api/1/key/my-games', headers=headers)
         response.raise_for_status()
         return response.json()['games']
     
     # Function to download a game by ID
     def download_game(game_id):
         download_url = f'https://itch.io/api/1/key/game/{game_id}/download'
         response = requests.get(download_url, headers=headers)
         response.raise_for_status()
         return response.json()
     
     # Example usage
     user_info = login()
     print(f"Logged in as: {user_info['user']['username']}")
     
     my_games = get_my_games()
     print("My games:")
     for game in my_games:
         print(f"{game['id']}: {game['title']}")
     
     # Replace 'GAME_ID' with ID of game we want to download
     game_id = 'GAME_ID'
     game_download_info = download_game(game_id)
     print(f"Download URL for {game_id}: {game_download_info['url']}")

2. Scripting and Configuration Adjustments

   • Add defaults/scripts/Extensions/Itch/store.sh:

     • Handle itch.io specific command-line operations, such as game list refreshes, downloads, and status checks.
     • Implement filtering and caching mechanisms appropriate for itch.io's data structure and update frequency.

   • Add defaults/scripts/itch-config.py:

     • Config handling script to incorporate itch.io as selectable store.
     • Add necessary args and environment variables to manage interactions for itch.io.

   • Backend Python Modifications

     • Add defaults/scripts/itch.py:
       • Develop or extend a Python class specifically for interacting with itch.io, handling API calls, data processing, and exception management.
       • Implement methods for fetching game lists, game details, downloading images, and other media associated with games.
     • Revise defaults/scripts/shared/GameSet.py and GamesDb.py:
       • Update shared base classes to include methods and attributes that support itch.io's integration.
       • Ensure that database schema accommodates any new itch.io-specific fields (e.g., DRM status (May not need), direct download links).

   • Database Schema Enhancement

     • Modify Database Tables (GameSet.py?):
       • Update existing tables or create new ones specific to itch.io if necessary, ensuring all game metadata and image links are correctly stored and indexed.

3. Frontend Integration

   • Adjust src/Components/GameDetailsItem.tsx:
     • Update TypeScript component to handle display and interaction of itch.io game details within this plugin's UI.
     • Ensure that any new data elements specific to itch.io (like DRM-free indicators) are appropriately rendered.

Again, my two cents just to push us a little forward, Keep up the awesome work :D

[snowkat]

As I understand it, the itch.io server-side and OAuth APIs are for sellers to provide integration for games, and there isn't a way to use it for getting games you bought, generate download URLs, etc. The official itch app (https://github.com/itchio/itch) uses butler, but the APIs it uses require running it as a JSON-RPC daemon.

The butlerd docs are available at https://docs.itch.ovh/butlerd/master/, but as a general reference, getting authenticated works like such:

1. Spawn a new butler process. For example:

   butler --json                        \
       --user-agent "test app (Linux)"  \
       daemon                           \
       --dbpath ".../path/to/butler.db"
   

2. Connect to the TCP server by listening for the message on stdout, as described in https://docs.itch.ovh/butlerd/master/#/?id=making-requests. Notably, every TCP connection needs to call Meta.Authenticate with the secret provided over stdout.
3. (Optional) Try getting a saved profile by calling Profile.List and Profile.UseSavedLogin. If you get a profile this way, you're done! Skip to step 7.
4. Show the user a form with a username and password, and use them as parameters when calling Profile.LoginWithPassword.¹
5. Before Profile.LoginWithPassword returns, you may receive a couple server->client calls, Profile.RequestCaptcha and/or Profile.RequestTOTP.
   • Profile.RequestTOTP is simple: ask the user for the TOTP token, and respond.
   • Profile.RequestCaptcha is, bluntly, a mess. More on that in a moment.
6. Once any requested challenges have been completed, Profile.LoginWithPassword returns with the logged in profile.
7. Save the returned profile, or at least its profileId field, for use in later requests (e.g. Fetch.GameRecords).

For Profile.RequestCaptcha, you'll be provided a recaptchaUrl, most likely https://itch.io/captcha. This page needs to be embedded as a BrowserView or new window (not an iframe²) where you can inject JS to catch the CAPTCHA response. For example:

// function called by itch's reCAPTCHA on success.
// otherwise you could watch document.getElementById("g-recaptcha-response")
function onRecaptcha(response) {
    // "namespace" our message so we don't try to consume a stray Steam message
    window.opener.postMessage("__ITCH_CAPTCHA_" + response, "*");
}

Once solved, you can return the response string in the RPC call.

The other API calls are pretty straightforward, but handling two-way JSON-RPC in shell scripts would be a challenge. Logs are also sent through JSON-RPC calls, so you'd likely want at least one long-lived TCP connection to output them.

Footnotes

1. I was able to pull this off with a modal dialog. It doesn't feel great, but it works... ↩

2. https://itch.io/captcha sets X-Frame-Options: SAMEORIGIN, so we can't embed it in a frame. ↩

[Aeonitis]

Awesome! Thanks for your research!

Sorry for my delay, I had been distracted by many life events...

It seems from my little knowledge of frontend dev that you have successfully gone through the process of authenticating via a web form login through butler, including handling CAPTCHA (which sucks) and TOTP challenges, but I am not totally confident you have received the valid token, it seems you're saying the end product is a login within the browser, is that correct?

I may need to look at butler docs for token output info I guess...

Not sure if I am being a bother, but if you're ever free and willing, is there a way you can prepare a simple shell script to test auth, and maybe use jq to handle json tasks if necessary?

If not, just write down the simplest steps so I can try in future (WHEN I EVER HAVE TIME >_<)

The ideal way to know we have succeeded on "authorized-for-download" status for this is if the user token is fetched/printed/written I think.

Feel free to say hi or ask any questions on my discord @Aeonitis if interested

[snowkat]

It seems from my little knowledge of frontend dev that you have successfully gone through the process of authenticating via a web form login through butler, including handling CAPTCHA (which sucks) and TOTP challenges, but I am not totally confident you have received the valid token, it seems you're saying the end product is a login within the browser, is that correct?

The flow I followed is a bit different, but the end result is the same! itch.io only provides non-creator APIs (i.e., listing owned games, downloading, etc) for what they call cookie type keys.

That said, since writing that message, I found the login "API" is better described in their Go library, in endpoints_login.go. It'd be entirely possible to sidestep butler, but I've been hesitant about that for a couple reasons:

• butler uses an itch.io library called dash, which I've found to be really good at picking out the right file to run. It's especially useful in an environment like itch.io, where there's no standard for how games are installed or run. I've made a small binary wrapper called traipse for my testing.
• Many itch.io games use something called "wharfs," which I believe are similar to your average CI/CD artifacts. butlerd seems to handle most of whatever's happening there for you.
• butler handles installation and updates via what they call "caves"-- basically, dedicated game storage locations (a la the steamapps folder).
• While the butlerd docs say the API is unstable, it hasn't changed in four years. Granted, neither has the itch.io API... but considering the itch.io app relies on butlerd directly, I'd expect butlerd to stay close to the same.

Not sure if I am being a bother, but if you're ever free and willing, is there a way you can prepare a simple shell script to test auth, and maybe use jq to handle json tasks if necessary?

Not a bother, but it's unfortunately not easy to handle in a shell script. I was looking into making a custom backend, but ran out of steam building it for that reason. The biggest issues are the CAPTCHA (which requires a web browser you can inject JS into¹) and two-way TCP communication being a nuisance through bash/netcat.

That said: I was building a demonstration Decky plugin, mostly to convince myself any of this was possible.I think the Python backend could be useful for better explaining the JSON-RPC API, although a flow diagram might help more. I haven't had much time to keep working on it, but I'll upload it to GitHub (and possibly make said flow diagram?) when I next have a chance.

Footnotes

1. The Decky library does provide that through the SteamClient APIs! I just don't think there's a way we can do it from a custom Junk Store backend, and I could understand hesitation to allowing that. ↩

[Aeonitis]

@snowkat that's amazing stuff.

FYI months ago I made a python app, my own itch client on a free weekend that does login successfully but without any webpage, just username and password authentication. Cookie type key is what it is for me too actually 😂

Unfortunately the project requires web based login, I was told.

My app also managed downloads and images successfully, and persistence with shell, python and sqlite, which I chose to be consistent with this project and other technical reasons too...

Pretty much many steps have already ticked out but I haven't made more time for that project.

See what works for you, but again, I'm here https://discord.gg/s2BZeQgQ