Skip to content

Commit

Permalink
New year new Algalon (#11)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Ghostopheles
Ghostopheles authored Mar 25, 2024
2 parents 0bf3acd + 3e8e9b0 commit 0a9b1f1
Show file tree
Hide file tree
Showing 8 changed files with 321 additions and 116 deletions.
10 changes: 10 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
# Changelog v1.8 - Niobium

## Added

- `wowlivetest2`, `fenristest`, `gryphon`, `gryphonb`, `gryphondev` products can now be observed.

## Fixed

- Various backend fixes and optimizations to improve uptime and performance.

# Changelog v1.7 - Beryllium

## Added
Expand Down
15 changes: 12 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# better-algalon
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) ![LastUpdate](https://img.shields.io/github/last-commit/Ghostamoose/better-algalon?style=flat-square) [![Docker](https://github.com/Ghostamoose/better-algalon/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/Ghostamoose/better-algalon/actions/workflows/docker-publish.yml)

v1.7 - Beryllium
v1.8 - Niobium

A bot that watches Blizzard's CDN and automatically posts new build updates to specified Discord channels.

Inspired by, and vaguely based on the original [Algalon bot by Ellypse](https://github.com/Ellypse/Algalon).

Includes a Twitter integration to post updates to Twitter alongside Discord. This will be replaced in the future with Algalon 3.0 (lol). This bot can be found on Twitter as [@algalon_ghost](https://twitter.kivatech.io).
Includes a Twitter integration to post updates to Twitter alongside Discord. This will be replaced in the future with Algalon 3.0 (lol). This bot can be found on Twitter as [@algalon_ghost](https://algalon.ghst.tools/).

Check out the [changelog](CHANGELOG.md) to view the most recent changes.

Expand All @@ -27,7 +27,8 @@ A lock indicates that the given branch is encrypted and not accessible to the pu
| wow_classic_era_beta | Classic Era Beta |
| wow_classic_era_ptr | Classic Era PTR |
| wowz | Submission |
| wowlivetest | Live Test|
| wowlivetest | Live Test |
| wowlivetest2 :lock: | Live Test Internal |
| wowdev :lock: | Internal |
| wowdev2 :lock: | Internal 2 |
| wowdev3 :lock: | Internal 3 |
Expand All @@ -45,13 +46,21 @@ A lock indicates that the given branch is encrypted and not accessible to the pu
| ----------- | ----------- |
| fenris | Diablo IV |
| fenrisb | Diablo IV Beta |
| fenristest | Diablo IV PTR |
| fenrisdev :lock: | Diablo IV Internal |
| fenrisdev2 :lock: | Diablo IV Internal 2 |
| fenrise :lock: | Diablo IV Event |
| fenrisvendor1 :lock: | Diablo IV Vendor |
| fenrisvendor2 :lock: | Diablo IV Vendor 2 |
| fenrisvendor3 :lock: | Diablo IV Vendor 3 |

### Warcraft Rumble
| Branch Name | Readable Name |
| ----------- | ----------- |
| gryphon | Warcraft Rumble Live |
| gryphonb | Warcraft Rumble Beta |
| gryphondev :lock: | Warcraft Rumble Internal |

## Commands

Algalon provides a number of commands to control your guild's (server) watchlist.
Expand Down
9 changes: 5 additions & 4 deletions cogs/api/blizzard_tact.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,12 @@ def construct_url(self, host: str, path: str, hash: str):
return f"http://{host}/{path}/{hash[:2]}/{hash[2:4]}/{hash}"

async def is_encrypted(self, branch: str, product_config_hash: str):
async with httpx.AsyncClient() as client:
async with httpx.AsyncClient(timeout=2) as client:
url = f"{self.__API_URL}{branch}{self.__API_ENDPOINT}"
try:
response = await client.get(url)
except httpx.ConnectTimeout as exc:
self.logger.error("TACT CDN info request timed out.")
self.logger.error(exc)
return None

if response.status_code != 200:
Expand All @@ -80,7 +79,9 @@ async def is_encrypted(self, branch: str, product_config_hash: str):
cdn_config_url = self.construct_url(host, path, product_config_hash)

try:
self.logger.info("Attempting to fetch product config...")
self.logger.info(
f"Attempting to fetch product config for {branch}..."
)
cdn_response = await client.get(cdn_config_url)

if cdn_response.status_code != 200:
Expand All @@ -90,7 +91,7 @@ async def is_encrypted(self, branch: str, product_config_hash: str):
continue
else:
self.logger.info(
"Product config found, returning encryption status..."
f"{branch} product config found, returning encryption status..."
)
product_config = cdn_response.json()

Expand Down
148 changes: 53 additions & 95 deletions cogs/cdn_cache.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
import httpx
import shutil
import logging
import asyncio

from .api.blizzard_tact import BlizzardTACTExplorer
from .config import CacheConfig, FETCH_INTERVAL
from .ribbit_async import RibbitClient

logger = logging.getLogger("discord.cdn.cache")

Expand All @@ -32,7 +34,7 @@ def patch_cdn_keys(self):
with open(self.cdn_path, "r+") as file:
logger.info("Patching CDN file...")
file_json = json.load(file)
build_data = file_json[self.CONFIG.indices.BUILDINFO]
build_data = file_json["buildInfo"]
try:
for branch in build_data:
for key, value in self.CONFIG.REQUIRED_KEYS_DEFAULTS.items():
Expand All @@ -45,7 +47,7 @@ def patch_cdn_keys(self):
except KeyError as exc:
logger.error("KeyError while patching CDN file", exc_info=exc)

file_json[self.CONFIG.indices.BUILDINFO] = build_data
file_json["buildInfo"] = build_data

file.seek(0)
json.dump(file_json, file, indent=4)
Expand All @@ -55,7 +57,7 @@ def init_cdn(self):
"""Populates the `cdn.json` file with default values if it does not exist."""
with open(self.cdn_path, "w") as file:
template = {
self.CONFIG.indices.BUILDINFO: {},
"buildInfo": {},
self.CONFIG.indices.LAST_UPDATED_BY: self.PLATFORM,
self.CONFIG.indices.LAST_UPDATED_AT: time.time(),
}
Expand All @@ -80,32 +82,28 @@ def compare_builds(self, branch: str, newBuild: dict) -> bool:
file_json["encrypted"] = None

if (
file_json[self.CONFIG.indices.BUILDINFO][branch]["encrypted"] == True
file_json["buildInfo"][branch]["encrypted"] == True
and newBuild["encrypted"] == None
):
newBuild["encrypted"] = True

# ignore builds with lower seqn numbers because it's probably just a caching issue
if (newBuild["seqn"] > 0) and newBuild["seqn"] < file_json[
self.CONFIG.indices.BUILDINFO
][branch]["seqn"]:
new_seqn, old_seqn = int(newBuild["seqn"]), int(
file_json["buildInfo"][branch]["seqn"]
)
if (new_seqn > 0) and new_seqn < old_seqn:
logger.warning(f"Lower sequence number found for {branch}")
return False

for area in self.CONFIG.AREAS_TO_CHECK_FOR_UPDATES:
if branch in file_json[self.CONFIG.indices.BUILDINFO]:
if (
file_json[self.CONFIG.indices.BUILDINFO][branch][area]
!= newBuild[area]
):
if branch in file_json["buildInfo"]:
if file_json["buildInfo"][branch][area] != newBuild[area]:
logger.debug(f"Updated info found for {branch} @ {area}")
return True
else:
return False
else:
file_json[self.CONFIG.indices.BUILDINFO][branch][area] = newBuild[
area
]
file_json["buildInfo"][branch][area] = newBuild[area]
return True
return False

Expand All @@ -115,7 +113,7 @@ def set_default_entry(self, name: str):
def get_all_config_entries(self):
with open(self.cdn_path, "r") as file:
file_json = json.load(file)
return file_json[self.CONFIG.indices.BUILDINFO].keys()
return file_json["buildInfo"].keys()

def create_cache_backup(self):
logger.info("Backing up CDN cache file...")
Expand All @@ -142,7 +140,7 @@ def save_build_data(self, branch: str, data: dict):
"""Saves new build data to the `cdn.json` file."""
with open(self.cdn_path, "r+") as file:
file_json = json.load(file)
file_json[self.CONFIG.indices.BUILDINFO][branch] = data
file_json["buildInfo"][branch] = data

file.seek(0)
json.dump(file_json, file, indent=4)
Expand All @@ -152,97 +150,57 @@ def load_build_data(self, branch: str):
"""Loads existing build data from the `cdn.json` file."""
with open(self.cdn_path, "r") as file:
file_json = json.load(file)
if branch in file_json[self.CONFIG.indices.BUILDINFO]:
return file_json[self.CONFIG.indices.BUILDINFO][branch]
if branch in file_json["buildInfo"]:
return file_json["buildInfo"][branch]
else:
file_json[self.CONFIG.indices.BUILDINFO][branch] = {
self.CONFIG.settings.REGION["name"]: self.CONFIG.defaults.REGION,
self.CONFIG.indices.BUILD: self.CONFIG.defaults.BUILD,
self.CONFIG.indices.BUILDTEXT: self.CONFIG.defaults.BUILDTEXT,
file_json["buildInfo"][branch] = {
"region": self.CONFIG.defaults.REGION,
"build": self.CONFIG.defaults.BUILD,
"build_text": self.CONFIG.defaults.BUILDTEXT,
}
return False

async def fetch_cdn(self):
"""This is a disaster."""
"""This is sort of a disaster."""
logger.info(self.CONFIG.strings.LOG_FETCH_DATA)
self.create_cache_backup()
async with httpx.AsyncClient() as client:
new_data = []
for branch in self.CONFIG.PRODUCTS:
branch = branch.name
try:
logger.info(f"Grabbing version for {branch}")
url = self.CONFIG.CDN_URL + branch + "/versions"
coros = [
self.fetch_branch_ribbit(branch.name) for branch in self.CONFIG.PRODUCTS
]
new_data = await asyncio.gather(*coros)
new_data = [i for i in new_data if i is not None]

res = await client.get(url, timeout=20)
logger.info(self.CONFIG.strings.LOG_PARSE_DATA)
data = await self.parse_response(branch, res.text)
return new_data

if data and res.status_code == 200:
logger.debug(f"Version check payload: {data}")
logger.info(f"Comparing build data for {branch}")
is_new = self.compare_builds(branch, data)
async def fetch_branch_ribbit(self, branch: str):
logger.info(f"Grabbing versions for {branch}")
_data, seqn = await RibbitClient().fetch_versions_for_product(product=branch)

if is_new:
output_data = data.copy()
if not _data:
logger.error(f"No response for {branch}.")
return

old_data = self.load_build_data(branch)
_data = _data["us"]
data = _data.__dict__()

if old_data:
output_data["old"] = old_data
logger.info(f"Comparing build data for {branch}")
is_new = self.compare_builds(branch, data)

output_data["branch"] = branch
new_data.append(output_data)
logger.debug(f"Updated build payload: {output_data}")
if is_new:
output_data = data.copy()

logger.info(f"Saving new build data for {branch}")
self.save_build_data(branch, data)
else:
logger.warning(
f"Invalid response {res.status_code} for branch {branch}"
)
except httpx.ConnectError as exc:
logger.error(
f"Connection error during CDN check for {branch} using url {url or None}" # type: ignore
)
continue
except Exception as exc:
logger.error(f"Error during CDN check for {branch}", exc_info=exc)
continue

return new_data

async def parse_response(self, branch: str, response: str):
"""Parses the API response and attempts to return the new data."""
try:
data = response.split("\n")
if len(data) < 3:
return False
seqn = data[1].replace("## seqn = ", "")
data = data[2].split("|")
region = data[0]
build_config = data[1]
cdn_config = data[2]
build_number = data[4]
build_text = data[5].replace(build_number, "")[:-1]
product_config = data[6]

output = {
self.CONFIG.settings.REGION["name"]: region,
"build_config": build_config,
"cdn_config": cdn_config,
"build": build_number,
"build_text": build_text,
"product_config": product_config,
"encrypted": await self.TACT.is_encrypted(branch, product_config),
"seqn": int(seqn),
}
old_data = self.load_build_data(branch)

return output
except KeyError or IndexError as exc:
logger.warning(
f"Encountered an error parsing API response for branch: {branch}.",
exc_info=exc if isinstance(exc, IndexError) else None,
)
if old_data:
output_data["old"] = old_data

return False
output_data["branch"] = branch
logger.debug(f"Updated build payload: {output_data}")

logger.info(f"Saving new build data for {branch}")
self.save_build_data(branch, data)

return output_data
else:
logger.debug(f"No new data found for {branch}")
return
Loading

0 comments on commit 0a9b1f1

Please sign in to comment.