Database Support

.gitignore

@@ -160,4 +160,5 @@ cogs/cache/
 .vscode/
 *.code-workspace
 pycord/
-docker-compose-dev.yml
+docker-compose-dev.yml
+scripts

CHANGELOG.md

@@ -1,103 +1,9 @@
-# Changelog v2.1.0 - Iridium
+# Changelog v2.2.0 - Caesium
 
-## Added
-- Algalon has finally left the house and touched the sky. Algalon can now be found on Bluesky as [@algalon.bsky.social](https://bsky.app/profile/algalon.bsky.social/).
-- Algalon is now able to notify your via DM about certain metadata field updates for specific game branches.
-   - As an example, you can receive a DM every time the `keyring` field is updated for `wow_beta`. Pretty fancy, huh? Just me? Okay.
-- Now observing `fenrishf`, `fenrisvendor4` and `fenrisvendor5`
+- Algalon is moving up in the world. With increased demand, speed has dropped and latency has increased. Algalon has now been (mostly) transitioned over to using an actual database instead of JSON files.
+   - Ideally this would come with zero user-facing changes apart from increased speed and snapiness. Ideally.
+   - If you encounter and issues or oddities, please let me know ASAP.
 
-## Fixed
-- Cleared up a few annoying bugs that would cause confusing error messages or failed commands
-
-# Changelog v2.0.0 - Fermium
-
-## Added
-- Algalon now supports user-installation, allowing you to setup DM notifications without being in a server with Algalon.
-   - If you have any suggestions for other features that might be nice to include with user installation, let me know!
-- A graphical UI for editing your watchlist, either for a server or for yourself, that can be used to add or remove branches from your watchlist
-   - Due to Discord limitations, these are capped at 25 elements. This means I've had to split them up by game. Sorry :c
-
-## Fixed
-- Improved version check time from ~13 seconds to ~200 milliseconds. Blame synchronous HTTP requests.
-   - This should make commands feel **much** more response and they should no longer randomly timeout.
-
-# Changelog v1.9.2 - Gallium
-
-## Added
-- DM notifications have been enabled frfr on all server Algalon is a part of.
-
-## Changed
-- Commands have been restructured.
-   - Watchlist commands have been moved to `/watchlist {add|remove|view}`
-   - Channel commands have been moved to `/channel {set|get}`
-   - DM commands have been moved to `/dm {subscribe|unsubscribe|view}`
-
-# Changelog v1.9 - Yttrium
-
-## Added
-
-- Algalon is now able to DM users when a build update appears. These DMs are purposefully slimmed down to allow them to be readable on mobile push notifications.
-- Added a live config file that can be used to update certain configuration values without needing to rebuild and redeploy the bot. Namely, product names can now be updated live.
-- Added support for Battlenet `catalogs`.
-
-## Removed
-
-- The `cdn` prefix on all of the slash commands has been deleted.
-
-## Fixed
-
-- Update checking is now significantly faster.
-- Global config objects are now handled more efficiently.
-
-# Changelog v1.8 - Niobium
-
-## Added
-
- - `wowlivetest2`, `fenristest`, `gryphon`, `gryphonb`, `gryphondev` products can now be observed.
-
-## Fixed
-
- - Various backend fixes and optimizations to improve uptime and performance.
-
-# Changelog v1.7 - Beryllium
-
-## Added
-
- - Algalon can now be added to any server by clicking the 'Add to Server' button on his profile on Discord! Use at your own risk. ;)
- - As such, I've added a new welcome message when Algalon first joins a server. This message will be sent to the system channel if possible, otherwise it'll fall back to the 'community updates' channel. Both of these channels are set by server admins in the server settings, and if neither are available he just remains silent.
- - Algalon will now publish messages to a channel in the Official™️ GhostBots Discord server, so if you don't want to add Algalon to your server, let me know and I'll invite to the GhostBots server where you can follow the announcement channel, and have all the messages propagated to your server! (you lose the ability to customize branches you wanna watch though)
- - Some minor debugging/administration commands.
-
-## Fixed
-
- - Servers subscribed to Diablo 4 updates but that haven't set their Diablo 4 channel will now automatically have the Diablo 4 channel set to whatever their Warcraft channel is.
- - Minor optimizations in the guild configuration area.
-
-
-
-# Changelog v1.6 - Diamond
-
-## Added
-
- - `/cdnbranches` allows you to view a list of all branches that Algalon can watch for your server.
- - Support for Diablo IV returns.
- - You can now split up notifications for Diablo IV and Warcraft into seperate channels if you so desire.
-
-## Changed
-
- - Update embeds are now a little prettier.
- - Most commands have been restricted to only being available in servers only and are now unable to be called from DMs.
- - The paginator command has been changed to `/cdndata`.
-    - The paginator returned by the above command is no longer missing a newline on the second-to-last line.
- - `/cdnsetchannel` now allows you to specify a game when setting your notification channel. Defaults to Warcraft.
-    - Existing servers will have their Diablo IV notification channel automatically set to the channel that is currently selected for notifications.
-    - `/cdngetchannel` has also been updated to reflect this change.
- - `/cdnaddtowatchlist` now supports comma-delimited lists of branches to add to the watchlist.
- - A few slash commands now have proper docstrings.
- - The `/cdnaddtowatchlist` and `/cdnremovefromwatchlist` commands now have a clickable command button to view all valid branches.
-
-## Removed
-
- - The commented out code for the watchlist editor UI has been removed  (rip). This will return in the future when I have the time (and patience) to work on it.
- - The commands for locale and region configuration have been removed. They didn't actually do anything in the first place and it's safer to remove them for now until I can get them working correctly.
- - The Blizzard API integration (`/cdntokenprice`) has been disabled. This wasn't really useful and doesn't fit with the rest of the bot's functionality, so it's getting yeeted.
+### Removed
+- The commands to add/remove single branches to/from your guild watchlist and user watchlist have been removed. These have been replaced by the graphical editors found using `/watchlist edit` or `/dm edit`.
+   - I know the UI is not amazing. My hands are tied by Discord, however, I do plan on building a website to handle configuration so we can finally save ourselves from this UX nightmare of Discord's making.

README.md

@@ -1,13 +1,13 @@
 # better-algalon
 ![GitHub License](https://img.shields.io/github/license/ghostopheles/better-algalon?style=for-the-badge) ![GitHub commit activity](https://img.shields.io/github/commit-activity/m/ghostopheles/better-algalon?style=for-the-badge) ![GitHub last commit](https://img.shields.io/github/last-commit/ghostopheles/better-algalon?display_timestamp=author&style=for-the-badge)
 
-v2.1.0 - Iridium
+v2.2.0 - Caesium
 
 A bot that watches Blizzard's CDN and automatically posts new build updates to specified Discord channels.
 
-Inspired by, and vaguely based on the original [Algalon bot by Ellypse](https://github.com/Ellypse/Algalon).
+Inspired by the original [Algalon bot by Ellypse](https://github.com/Ellypse/Algalon).
 
-Includes some social integrations to post updates to different social media sites alongside Discord. This bot can be found on Twitter as [@algalon_ghost](https://algalon.ghst.tools/) and on Bluesky as [@algalon.bsky.social](https://bsky.app/profile/algalon.bsky.social/).
+Includes some social integrations to post updates to different social media sites alongside Discord. This bot can be found on Twitter as [@algalon_ghost](https://algalon.ghst.tools/) and on Bluesky as [@algalon.ghst.tools](https://bsky.app/profile/algalon.ghst.tools/).
 
 Check out the [changelog](CHANGELOG.md) to view the most recent changes.
 
@@ -73,24 +73,16 @@ A lock indicates that the given branch is encrypted and not accessible to the pu
 
 ## Commands
 
-Algalon provides a number of commands to control your guild's (server) watchlist.
-
 \*admin privileges required.
 
 ### CDN Watching
 
-`/cdndata`: Returns a paginator containing the currently cached CDN data.
-
 `/lastupdate`: Returns a timestamp displaying when Algalon last checked for CDN updates.
 
 `/branches`: Returns a formatted list of all observable branches.
 
 #### Watchlist Controls
 
-`/watchlist add`*: Adds a specific branch to your guild's watchlist. Specify multiple branches at once by separating them with a comma.
-
-`/watchlist remove`*: Removes a specific branch to your guild's watchlist.
-
 `/watchlist edit`*: Returns a graphical editor for editing your guild's watchlist.
 
 `/watchlist view`: Returns your guild's current watchlist.
@@ -103,10 +95,6 @@ Algalon provides a number of commands to control your guild's (server) watchlist
 
 #### User DM Updates
 
-`/dm subscribe`: Subscribes the user to DM updates for the given branches. Supports comma-delimited input.
-
-`/dm unsubscribe`: Unsubscribes the user from DM updates for the given branches. Supports comma-delimited input.
-
 `/dm edit`*: Returns a graphical editor for editing your personal watchlist.
 
 `/dm view`: Returns all the branches you're currently subscribed to.

cogs/__init__.py

@@ -14,4 +14,5 @@
     "utils",
     "watcher",
     "monitoring",
+    "db",
 ]

cogs/admin.py

@@ -2,10 +2,12 @@
 import logging
 import discord
 
+from enum import Enum
 from discord.ext import commands
 
 from cogs.bot import Algalon
 from cogs.config import LiveConfig as cfg
+from cogs.db import AlgalonDB as DB
 
 logger = logging.getLogger("discord.admin")
 
@@ -50,7 +52,7 @@ async def reload_cog(self, ctx: discord.ApplicationContext, cog_name: str):
         logger.debug(f"{cog_name_internal} reloaded successfully.")
 
         await ctx.interaction.response.send_message(
-            f"`{cog_name}` reloaded successfully."
+            f"`{cog_name}` reloaded successfully.", ephemeral=True, delete_after=300
         )
 
     @commands.is_owner()
@@ -68,7 +70,7 @@ async def get_all_guilds(self, ctx: discord.ApplicationContext):
 
         message_bytes = io.BytesIO(message.encode())
         file = discord.File(message_bytes, filename="guilds.txt")
-        await ctx.respond(file=file)
+        await ctx.respond(file=file, ephemeral=True, delete_after=120)
         message_bytes.close()
 
     @commands.is_owner()

cogs/cdn_cache.py

@@ -8,10 +8,10 @@
 
 from typing import Any
 
-from cogs.user_config import Monitorable
-from .api.blizzard_tact import BlizzardTACTExplorer
-from .config import LiveConfig, CacheConfig
-from .ribbit_async import RibbitClient
+from cogs.api.blizzard_tact import BlizzardTACTExplorer
+from cogs.config import LiveConfig, CacheConfig
+from cogs.ribbit_async import RibbitClient
+from cogs.db import AlgalonDB as DB
 
 logger = logging.getLogger("discord.cdn.cache")
 
@@ -33,33 +33,8 @@ def __init__(self):
             os.mkdir(self.cache_path)
             self.init_cdn()
 
-        self.patch_cdn_keys()
-
         self.monitor = None
 
-    def patch_cdn_keys(self):
-        with open(self.cdn_path, "r+") as file:
-            logger.debug("Patching CDN file...")
-            file_json = json.load(file)
-            build_data = file_json["buildInfo"]
-            try:
-                for branch in build_data:
-                    for key, value in self.CONFIG.REQUIRED_KEYS_DEFAULTS.items():
-                        if key not in build_data[branch]:
-                            logger.debug(
-                                f"Adding {key} to {branch} with value {value}..."
-                            )
-                            build_data[branch][key] = value
-
-            except KeyError:
-                logger.error("KeyError while patching CDN file", exc_info=True)
-
-            file_json["buildInfo"] = build_data
-
-            file.seek(0)
-            json.dump(file_json, file, indent=4)
-            file.truncate()
-
     def init_cdn(self):
         """Populates the `cdn.json` file with default values if it does not exist."""
         with open(self.cdn_path, "w") as file:
@@ -73,13 +48,15 @@ def init_cdn(self):
     def register_monitor_cog(self, cog):
         self.monitor = cog
 
-    def notify_watched_field_updated(self, branch: str, field: str, new_data: Any):
+    async def notify_watched_field_updated(
+        self, branch: str, field: str, new_data: Any
+    ):
         if not self.monitor:
             return
 
-        self.monitor.on_field_update(branch, field, new_data)
+        await self.monitor.on_field_update(branch, field, new_data)
 
-    def compare_builds(self, branch: str, newBuild: dict) -> bool:
+    async def compare_builds(self, branch: str, newBuild: dict) -> bool:
         """
         Compares two builds.
 
@@ -104,9 +81,10 @@ def compare_builds(self, branch: str, newBuild: dict) -> bool:
                 logger.warning(f"Lower sequence number found for {branch}")
                 return False
 
+            metadata_fields = await DB.get_all_metadata_fields()
             build_info = file_json["buildInfo"]
             for area in newBuild:
-                if area not in Monitorable._value2member_map_:
+                if area not in metadata_fields:
                     continue
 
                 if branch not in build_info:
@@ -116,7 +94,9 @@ def compare_builds(self, branch: str, newBuild: dict) -> bool:
                     build_info[branch][area] = None
 
                 if build_info[branch][area] != newBuild[area]:
-                    self.notify_watched_field_updated(branch, area, newBuild[area])
+                    await self.notify_watched_field_updated(
+                        branch, area, newBuild[area]
+                    )
 
             for area in self.CONFIG.AREAS_TO_CHECK_FOR_UPDATES:
                 if branch in build_info:
@@ -218,7 +198,7 @@ async def fetch_branch_ribbit(self, branch: str):
         data = _data.__dict__()
 
         logger.debug(f"Comparing build data for {branch}")
-        is_new = self.compare_builds(branch, data)
+        is_new = await self.compare_builds(branch, data)
 
         if is_new:
             output_data = data.copy()