From 444d75c8037d59eb515e7e316f3a1c12ee3567d3 Mon Sep 17 00:00:00 2001 From: FoxxMD Date: Fri, 12 Jul 2024 14:31:11 -0400 Subject: [PATCH 1/2] docs(feat): Add architecture overview, common dev structures, and dev guidance for Sources --- docsite/docs/development/_category_.json | 8 + docsite/docs/development/dev-client.md | 8 + docsite/docs/development/dev-common.md | 168 ++++++ docsite/docs/development/dev-source.md | 552 ++++++++++++++++++ docsite/docs/installation/installation.md | 1 - package.json | 2 +- src/backend/common/infrastructure/Atomic.ts | 55 +- .../infrastructure/config/source/sources.ts | 36 +- 8 files changed, 819 insertions(+), 11 deletions(-) create mode 100644 docsite/docs/development/_category_.json create mode 100644 docsite/docs/development/dev-client.md create mode 100644 docsite/docs/development/dev-common.md create mode 100644 docsite/docs/development/dev-source.md diff --git a/docsite/docs/development/_category_.json b/docsite/docs/development/_category_.json new file mode 100644 index 00000000..6942c16c --- /dev/null +++ b/docsite/docs/development/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Development", + "position": 3, + "link": { + "type": "generated-index", + "description": "Developing for Multi-Scrobbler and tutorials" + } +} diff --git a/docsite/docs/development/dev-client.md b/docsite/docs/development/dev-client.md new file mode 100644 index 00000000..214d9089 --- /dev/null +++ b/docsite/docs/development/dev-client.md @@ -0,0 +1,8 @@ +--- +toc_min_heading_level: 2 +toc_max_heading_level: 5 +sidebar_position: 3 +title: Client Development/Tutorial +--- + +To do... diff --git a/docsite/docs/development/dev-common.md b/docsite/docs/development/dev-common.md new file mode 100644 index 00000000..728899fa --- /dev/null +++ b/docsite/docs/development/dev-common.md @@ -0,0 +1,168 @@ +--- +toc_min_heading_level: 2 +toc_max_heading_level: 5 +sidebar_position: 1 +title: Common Development +description: Start here for MS development +--- + +# Development + +## Architecture + +Multi-scrobbler is written entirely in [Typescript](https://www.typescriptlang.org/). It consists of a backend and frontend. The backend handles all Source/Client logic, mounts web server endpoints that listen for Auth callbacks and Source ingress using [expressjs](https://expressjs.com/), and serves the frontend. The frontend is a standalone [Vitejs](https://vitejs.dev/) app that communicates via API to the backend in order to render the dashboard. + +## Project Setup + +Development requires [Node v18.19.1](https://nodejs.org/en) or higher is installed on your system. It is recommended to use [nvm](https://github.com/nvm-sh/nvm) to manage the installed node version. + + +Clone this repository somewhere and then install from the working directory + +```shell +git clone https://github.com/FoxxMD/multi-scrobbler.git . +cd multi-scrobbler +git checkout --track origin/develop +nvm use # optional, to set correct Node version +npm install +npm run start +``` + +Use the [`develop`](https://github.com/FoxxMD/multi-scrobbler/tree/develop) branch as the target for any Pull Requests. The `master` branch is reserved for releases and minor documentation updates only. + +## Common Development + +:::info + +In this document, when referring to aspects of Sources and Clients that are shared between both, the Source/Client will be referred to as a **Component.** + +::: + +A Component is composed of two parts: + +* Typescript interfaces describing structure of configuration for that Component +* A concrete class inheriting from a common "startup" abstract class that enforces how the Component is built and operates + +In both parts Source/Clients share some common properties/behavior before diverging in how they operate. + +### Config + +The configuration for a Component should always have this minimum shape, enforced respectively by the interfaces [CommonSourceConfig](https://github.com/FoxxMD/multi-scrobbler/blob/develop/src/backend/common/infrastructure/config/source/index.ts#L105) and [CommonClientConfig](https://github.com/FoxxMD/multi-scrobbler/blob/ce1c70a4e1e87fb5bea7cca960eaafbd15881a1f/src/backend/common/infrastructure/config/client/index.ts#L68): + +```ts +interface MyConfig { + name: string + data?: object + options?: object +} +``` + +* `data` contains data that is required for a Component to operate such as credentials, callback urls, api keys, endpoints, etc... +* `options` are **optional** settings that can be used to fine-tune the usage of the Component but are not required or do not majorly affect behavior. EX additional logging toggles + +### Concrete Class + +Components inherit from an abstract base class, [`AbstractComponent`](https://github.com/FoxxMD/multi-scrobbler/blob/develop/src/backend/common/AbstractComponent.ts), that defines different "stages" of how a Component is built and initialized when MS first starts as well as when restarting the Component in the event it stops due to an error/network failure/etc... + +#### Stages + +Stages below are invoked in the order listed. All stages are asynchronous to allow fetching network requests or reading files. + +The stage function (described in each stage below) should return a value or throw: + +* return `null` if the stage is not required +* return `true` if the stage succeeded +* return a `string` if the stage succeeded and you wish to append a result to the log output for this stage +* throw an `Exception` if the stage failed for any reason and the Component should not continue to run/start up + +##### Stage: Build Data + +This stage should be used to validate user configuration, parse any additional data from async sources (file, network), and finalize the shape of any configuration/data needed for the Component to operate. + +:::info + +Implement [`doBuildInitData`](https://github.com/FoxxMD/multi-scrobbler/blob/develop/src/backend/common/AbstractComponent.ts#L71) in your child class to invoke this stage. + +:::: + +
+ +Examples + +* Parse a full URL like `http://SOME_IP:7000/subfolder/api` from user config containing a base url like `data.baseUrl: 'SOME_IP'` and then store this in the class config +* Validate that config `data` contains required properties `user` `password` `salt` +* Read stored credentials from `${this.configDir}/currentCreds-MySource-${name}.json`; + +
+ +##### Stage: Check Connection + +This stage is used to validate that MS can communicate with the service the Component is interacting with. This stage is invoked on MS startup as well as any time the Component tries to restart after a failure. + +If the Component depends on **ingress** (like Jellyfin/Plex webhook) this stage is not necessary. + +:::info + +Implement [`doCheckConnection`](https://github.com/FoxxMD/multi-scrobbler/blob/develop/src/backend/common/AbstractComponent.ts#L103) in your child class to invoke this stage. + +:::: + +
+ +Examples + +* Make a [`request`](https://nodejs.org/docs/latest-v18.x/api/http.html#httprequesturl-options-callback) to the service's server to ensure it is accessible +* Open a websocket connection and check for a ping-pong + +
+ +##### Stage: Test Auth + +MS determines if Auth is required for a Component based on two class properties. You should set these properties during `constructor` initialization for your Component class: + +* `requiresAuth` - (default `false`) Set to `true` if MS should check/test Auth for this Component +* `requiresAuthInteraction` - (default `false`) Set to `true` if user interaction is required to complete auth IE user needs to visit a callback URL + +If the Component requires authentication in order to communicate with a service then any required data should be built in this stage and a request made to the service to ensure the authentication data is valid. + +This stage should return: + +* `true` if auth succeeded +* `false` if auth failed without unexpected errors + * IE the authentication data is not valid and requires user interaction to resolve the failure +* throw an exception if network failure or unexpected error occurred + +You _should_ attempt to re-authenticate, if possible. Only throw an exception or return `false` if there is no way to recover from an authentication failure. + +:::info + +Implement [`doAuthentication`](https://github.com/FoxxMD/multi-scrobbler/blob/develop/src/backend/common/AbstractComponent.ts#L111) in your child class to invoke this stage. + +:::: + +
+ +Examples + +* Generate a Bearer Token for Basic Auth from user/password given in config and store in class properties +* Make a request to a known endpoint with Authorization token from read credentials file to see if succeeds or returns 403 +* Catch a 403 and attempt to reauthenticate at an auth endpoint with user/password given in config + +
+ +### Play Object + +The **PlayObject** is the standard data structure MS uses to store listen (track) information and data required for scrobbling. It consists of: + +* Track Data -- a standard format for storing track, artists, album, track duration, the date the track was played at, etc... +* Listen Metadata -- Optional but useful data related to the specific play or specifics about the Source/Client context for this play such as + * Platform specific ID, web URL to track, device/user ID that played this track, etc... + +Both Sources and Clients use the **PlayObject** interface. When a Component receives track info from its corresponding service it must transform this data into a PlayObject before it can be interacted with. + +For more refer to the TS documentation for `PlayObject` or [`AmbPlayObject`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/core/Atomic.ts#L141) in your project + +## Creating Clients and Sources + +* [Source Development and Tutorial](dev-source.md) +* [Client Development and Tutorial](dev-client.md) diff --git a/docsite/docs/development/dev-source.md b/docsite/docs/development/dev-source.md new file mode 100644 index 00000000..13b111b3 --- /dev/null +++ b/docsite/docs/development/dev-source.md @@ -0,0 +1,552 @@ +--- +toc_min_heading_level: 2 +toc_max_heading_level: 5 +sidebar_position: 2 +title: Source Development/Tutorial +--- + +This document will provide a step-by-step guide for creating a (trivial) new Source in MS alongside describing what aspects of the Source need to be implemented based on the service you use. Before using this document you should review [Common Development](dev-common.md#common-development). + +## Scenario + +You are the developer of a fancy, new self-hosted web-based media player called **Cool Player.** Cool Player has a slick interface and many bells and whistles, but most importantly it has an API. The API: + +* Has an unauthenticated health endpoint at `/api/health` that returns `200` if the service is running properly +* Has authenticated endpoints that require a user-generated token in the header `Authorization MY_TOKEN` + * Has a `/api/recent` endpoint that lists recently played tracks with a timestamp + * Has a `/api/now-playing` endpoint that returns information about the state of the player like current track, player position in the track, etc... +* Cool Player is by default accessed on port `6969` +* Your personal instance of Cool Player is hosted at `http://192.168.0.100:6969` and the api is accessed at `http://192.168.0.100:6969/api` + +Because there is an API that MS can actively read this will be a **polling** Source where MS sends requests to Cool Player to get scrobble information -- as opposed to an **ingress** Source like Jellyfin/Plex that uses webhooks from the service to send data to MS. + +## Minimal Implementation + +### Define and Implement Config + +We will create a new config interface for Cool Player using the [Common Config](dev-common.md#config) and tell MS it is a valid config that can be used. + +Create a new file for your config: + +```ts title="/src/backend/common/infrastructure/config/source/coolplayer.ts" +import { PollingOptions } from "../common.js"; +import { CommonSourceConfig, CommonSourceData } from "./index.js"; + +// all of the required data for the Build Data and Test Auth stages (from Common Development docs) +// should go here +export interface CoolPlayerSourceData extends CommonSourceData, PollingOptions { +// remember to annotation your properties! + + /** + * The user-generated token for Cool Player auth created in Cool Player -> Settings -> User -> Tokens + * + * @example f243331e-cf5b-49d7-846b-0845bdc965b4 + * */ + token: string + /** + * The host and port where Cool Player is hosted + * + * @example http://192.168.0.100:6969 + * */ + baseUrl: string +} + +export interface CoolPlayerSourceConfig extends CommonSourceConfig { + data: CoolPlayerSourceData +} + +export interface CoolPlayerSourceAIOConfig extends CoolPlayerSourceConfig { + // when using the all-in-one 'config.json' this is how users will identify this source + type: 'coolplayer' +} +``` + +Add the new interface to the list of valid interfaces for sources: + +```ts title="src/backend/common/infrastructure/config/source/sources.ts" +import { ChromecastSourceAIOConfig, ChromecastSourceConfig } from "./chromecast.js"; +// ... +// highlight-next-line +import { CoolPlayerSourceAIOConfig, CoolPlayerSourceConfig } from "./coolplayer.js"; + +export type SourceConfig = + SpotifySourceConfig + // ... + // highlight-next-line + | CoolPlayerSourceConfig; + +export type SourceAIOConfig = + SpotifySourceAIOConfig + // ... + // highlight-next-line + | CoolPlayerSourceAIOConfig; +``` + +Finally, add the source type identifier to the list of valid identifiers + +```ts title="src/backend/common/infrastructure/Atomic.ts" +export type SourceType = + 'spotify' + // ... + // highlight-next-line + | 'coolplayer'; + +export const sourceTypes: SourceType[] = [ + 'spotify', + // ... + // highlight-next-line + 'coolplayer' +]; +``` + +Now we will create a new Source inheriting from [`AbstractComponent`](dev-common.md#concrete-class) that: + +* accepts our config interface +* implements a function to transform CoolPlayer's track data into a [**PlayObject**](dev-common.md#play-object) +* implements required [stages](dev-common.md#stages) +* implements required methods to current player state and/or now playing track + +### Create CoolPlayer Source + +First we create a new Source called `CoolPlayerSource` and setup our constructor to accept the config and [specify Auth behavior.](dev-common.md#stage-test-auth) + +```ts title="src/backend/sources/SpotifySource.ts" +import { CoolPlayerSourceConfig } from "../common/infrastructure/config/source/coolplayer.js"; +import MemorySource from "./MemorySource.js"; +import { + InternalConfig, +} from "../common/infrastructure/Atomic.js"; + +// MemorySource is the base class used for polling-type Sources +export default class CoolPlayerSource extends MemorySource { + + // type hints for TS to know what the base class config looks like + declare config: CoolPlayerSourceConfig; + + constructor(name: any, config: CoolPlayerSourceConfig, internal: InternalConfig, emitter: EventEmitter) { + super('coolplayer', name, config, internal, emitter); + + // Cool Player required authentication + this.requiresAuth = true; + // but does not require user interaction for auth to work + this.requiresAuthInteraction = false; + // tells MS this is a Source that can be activity monitored through API + this.canPoll = true; + } +} +``` + +### Initialize Source from Config + +When MS starts it reads all configs and determines which Source to build based on the configs found. We need to tell it to build a `CoolPlayerSource` when a `coolplayer` config type is found. + +We modify `ScrobbleSources.ts` to add `CoolPlayerSource` as an option: + +```ts title="src/backend/sources/ScrobbleSources.ts" +// ... +import { CoolPlayerSource, CoolPlayerData } from "./CoolPlayerSource.js"; + +export default class ScrobbleSources { + // ... + buildSourcesFromConfig = async (additionalConfigs: ParsedConfig[] = []) => { + // ... + + // if CoolPlayerSource should be able to be built from ENVs only + // then add it as a case statement here + for (const sourceType of sourceTypes) { + switch (sourceType) { + // ... + case 'musikcube': + // ... + break; + // highlight-start + case 'coolplayer': + const cp = { + baseUrl: process.env.COOL_URL, + token: process.env.COOL_TOKEN + } + if (!Object.values(cp).every(x => x === undefined)) { + configs.push({ + type: 'coolplayer', + name: 'unnamed', + source: 'ENV', + mode: 'single', + configureAs: defaultConfigureAs, + data: cp as CoolPlayerData + }); + } + break; + // highlight-end + default: + break; + } + } + } + + // ... + + // (required) create new CoolPlayerSource if source config type is 'coolplayer' + addSource = async (clientConfig: ParsedConfig, defaults: SourceDefaults = {}) => { + // ... + let newSource: AbstractSource; + switch (type) { + // ... + case 'musikcube': + // ... + break; + // highlight-start + case 'coolplayer': + newSource = await new CoolPlayerSource(name, compositeConfig as CoolPlayerSourceConfig, internal, this.emitter); + break; + // highlight-end + default: + break; + } + } +} +``` + +### Implement Play Object Transform + +Now we will create a static function that is used to take the track data returned from Cool Player's API and return a standard [`PlayObject`.](dev-common.md#play-object) + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import dayjs from "dayjs"; +import { + FormatPlayObjectOptions, +} from "../common/infrastructure/Atomic.js"; +import { PlayObject } from "../../core/Atomic.js"; + +export default class CoolPlayerSource extends MemorySource { + // ... + + // 'obj' should ideally be a real TS interface + // if CoolPlayer has a ts/js client we would use that otherwise + // we should build our own interfaces to represent track data from Cool Player's API + static formatPlayObj(obj: any, options: FormatPlayObjectOptions = {}): PlayObject { + const { + trackName, + artistName, + albumName, + duration, + playedAt, + } = obj; + + return { + data: { + artists: [artistName], + album: albumName, + track: trackName, + // assuming seconds + duration, + // assuming playedAt is an ISO8601 timestamp + playDate: dayjs(playedAt) + }, + meta: { + source: 'CoolPlayer' + } + } + } +} +``` + +### Implement Stages + +Next we will implement the [Stages](dev-common.md#stages) required to get CoolPlayerSource running. + +#### Build Data + +First we implement the [Build Data Stage](dev-common.md#stage-build-data). We will check that the `token` and `baseUrl` properties have been provided by the user. Additionally, we will parse the baseUrl and add default ports/prefix. + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import { URL } from "url"; +// ... + +export default class CoolPlayerSource extends MemorySource { + + baseUrl!: URL; + + // ... + + static parseConnectionUrl(val: string) { + const normal = normalizeUrl(val, {removeTrailingSlash: false, normalizeProtocol: true}) + const url = new URL(normal); + + if (url.port === null || url.port === '') { + url.port = '6969'; + } + if (url.pathname === '/') { + url.pathname = '/api/'; + } + return url; + } + + protected async doBuildInitData(): Promise { + const { + token, + baseUrl + } = this.config; + if (token === null || token === undefined || (typeof token === 'string' && token.trim() === '')) { + throw new Error(`'token' must be defined`); + } + + if (baseUrl === null || baseUrl === undefined || (typeof baseUrl === 'string' && baseUrl.trim() === '')) { + throw new Error(`'baseUrl' must be defined`); + } + try { + this.baseUrl = CoolPlayerSource.parseConnectionUrl(baseUrl); + } catch (e) { + throw new Error(`Could not parse baseUrl: ${baseUrl}`, {cause: e}); + } + + this.logger.verbose(`Config URL: ${baseUrl} => Normalized: '${this.url.toString()}'`); + return true; + } +} +``` + +#### Check Connection + +Second we will implement the [Check Connection Stage](dev-common.md#stage-check-connection): + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import request from 'superagent'; +import { UpstreamError } from "../common/errors/UpstreamError.js"; +// ... +export default class CoolPlayerSource extends MemorySource { + + // ... + + protected async doCheckConnection(): Promise { + try { + const resp = await request.get(`${this.baseUrl}/health`); + return true; + // if /health returned version info we could instead read response and return a string with version info for visibility to the user + // return `Cool Player Version: ${resp.body.version}`; + } catch (e) { + throw e; + } + } +} +``` + +#### Test Auth + +Finally, we will implement [Auth Test Stage](dev-common.md#stage-test-auth): + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import request from 'superagent'; +import { UpstreamError } from "../common/errors/UpstreamError.js"; +// ... +export default class CoolPlayerSource extends MemorySource { + + // ... + + doAuthentication = async () => { + try { + const resp = await request + .get(`${this.baseUrl}/recent`) + .set('Authorization', `Token ${this.config.token}`); + return true; + } catch (e) { + // if Cool Player returned an error as json we could parse it from error body and throw new Error with the message + throw e; + } + } +} +``` + +### Implement Polling + +The majority of Sources MS monitors primarily operate as a source of truth for a **music player** rather than a **played music history.** Only Listenbrainz and Last.fm operate as a source of music history. + +To this end, MS implements a [state machine](https://www.freecodecamp.org/news/state-machines-basics-of-computer-science-d42855debc66/) that emulates the behavior of a music player in order to keep track of when a song you are listening to should be scrobbled. It does this by monitoring the "currently playing" track reported by a Source's service, with varying degrees of accuracy depending on what information is returned from the service. The state machine is implemented in `MemorySource` which our `CoolPlayerSource` inherits from. + +For a polling Source to work properly we need to implement a function, [`getRecentlyPlayed`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/sources/AbstractSource.ts#L92), that returns PlayObjects that are "newly" played. These are then checked against previously "discovered" plays and their timestamp to determine if they should be surfaced to Clients to scrobble. + +To take advantage of the `MemorySource` state machine we will additionally use [`processRecentPlays`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/sources/MemorySource.ts#L113) from `MemorySource` inside `getRecentlyPlayed`. We pass track and/or player state returned from the Source service to `processRecentPlayers`. It then takes care of deriving Source player state based on how this data changes over time. The advantage to using `processRecentPlays` is that our Source service does not necessarily need to pass any player information -- as long as the track info has a **duration** we can more-or-less determine if it has been played long enough to scrobble. + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import request from 'superagent'; +import { + SourceData, + PlayerStateData, + SINGLE_USER_PLATFORM_ID +} from "../common/infrastructure/Atomic.js"; +// ... +export default class CoolPlayerSource extends MemorySource { + + // ... + + protected async getRecentlyPlayed(options: RecentlyPlayedOptions = {}): Promise { + const plays: SourceData[] = []; + try { + // currently playing tracks/player state data + const resp = await request + .get(`${this.baseUrl}/now-playing`) + .set('Authorization', `Token ${this.config.token}`); + const { + body: { + playerState, // 'playing' or 'stopped' or 'paused'... + position, // number of seconds into the track IE at position 48 -> ( 0:48/3:56 in player UI ) + play: { /* track data */} + } + } = resp; + + // transform into standard player state data + const playerData: PlayerStateData = { + platformId: SINGLE_USER_PLATFORM_ID, + play: CoolPlayerSource.formatPlayObj(play), + position + }; + + // if Cool Player does return player state we can also push a regular PlayObject to this list + plays.push(playerData); + } catch (e) { + throw e; + } + + // process player state through state machine + // if the track changes or player state changes + // and currently played track has been listened to long enough to be scrobbled it will return in newPlays + const newPlays = this.processRecentPlays(plays); + + // finally, we return new plays and MS checks to see if they have been previously seen + // before signalling to Clients that they can be scrobbled + return newPlays; + } +} +``` + +Congratulations! Your `CoolPlayerSource` has been minimally implemented and can now be used in multi-scrobbler. + +## Further Implementation + +### Backlog + +To have your Source try to scrobble "missed" tracks when MS starts up the Source's service must be able to provide: + +* track information +* timestamp of when the track was played + +In your Source implement [`getBackloggedPlays`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/sources/AbstractSource.ts#L235) and set setting in constructor indicating it has backlogging capabilities: + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import request from 'superagent'; +import { + PlayObject, +} from "../common/infrastructure/Atomic.js"; +// ... +export default class CoolPlayerSource extends MemorySource { + + constructor(/* ... */) { + super(/* ... */); + // ... + + // tell MS it should try to get backlogged tracks on startup + this.canBacklog = true; + } + + // ... + + protected getBackloggedPlays = async (options: RecentlyPlayedOptions): Promise => { + try { + const resp = await request + .get(`${this.baseUrl}/recent`) + .set('Authorization', `Token ${this.config.token}`); + + // assuming list from body looks like track info returned in + // "Implement Play Object Transform" section + const { + body = [] + } = resp; + + return body.map(x => CoolPlayerSource.formatPlayObj(x)); + } catch (e) { + throw new Error('Error occurred while getting recently played', {cause: e}); + } + + } +} +``` + +### Other Source Types + +There are some scenarios where polling and/or state machine is not the right tool to handle determining if incoming data should be scrobbled: + +* The Source service handles scrobble threshold internally, the data being received should always be scrobbled (WebScrobbler, Plex, Tautulli, Listenbrainz, Last.fm) +* You prefer to handle the scrobble determination yourself + +#### Music History Source + +If the Source is still polling but the track returned should always be scrobbled if not already seen IE the Source service is a **music history source** (Listenbrainz, Last.fm), rather than a music player, then simply indicate to MS the source of truth type by setting it in the constructor. The state machine will always return a track if it is new and not seen, regardless of how recently it was seen: + + +```ts title="src/backend/sources/CoolPlayerSource.ts" +import { SOURCE_SOT } from "../../core/Atomic.js"; +// ... +export default class CoolPlayerSource extends MemorySource { + + constructor(/* ... */) { + super(/* ... */); + // ... + + // tell MS it should immediately scrobble any new, unseen tracks from the upstream service + this.playerSourceOfTruth = SOURCE_SOT.HISTORY; + } +} +``` + +#### Non-Polling Source + +**Ingress** Sources (like Plex, Tautulli, Webscrobbler, Jellyfin) do not having a polling mechanism because the upstream service contacts MS when there is an event, rather than MS contacting the upstream service. + +For these Sources you will need to implement endpoints in `src/service/api.ts` and corresponding files. See the existing Sources in the project as references for how to do this. + +You may still wish to use the state machine `MemorySource` (like Jellyfin) if the events received are not "scrobble" events but instead of implementing `getRecentlyPlayed` you will implement your own function in your Source class, like `handle()`, that receives data and then uses `processRecentPlays`. + +After new plays have been determined see the next section for how to scrobble... + +#### Basic Source + +At the core of a Source that implements `AbstractSource`'s functionality is the ability to **Discover** and **Scrobble** plays. + +These functions are not seen in the MVP `CoolPlayerSource` because they are automatically done by the polling functionality after being returned from `getRecentlyPlayed`. + +##### Discovery + +A Source keeps track of all the "plays" that are determined to be valid for scrobbling. When a play is valid it is checked to see if it has already been "discovered" by comparing the track info and timestamp of the play against already discovered plays. This prevents duplicate scrobbling by using the Source's own data and simplifies scrobbling for Sources by allowing your implementation to "always" ingest track data without having to worry about whether its new or not -- `AbstractSource` and `discover()` will take care of that for you. + +```ts title="src/backend/sources/MyBasicSource.ts" +export default class MyBasicSource extends AbstractSource { + handle(somePlay: PlayObject) { + // if the track is "new" and not seen before it is returned in the discovered list + // we then know it is OK to be sent to Clients for scrobbling + const discovered: PlayObject[] = this.discover([somePlay]); + } +} +``` + +This additionally will be surfaced to the user in the Dashboard in the "Tracks Discovered" page. + +##### Scrobbling + +After a play is verified to be discovered we can then scrobble it. This will emit the plays to the ScrobbleClients service which then disseminates the play to all Clients that were configured to listen in the Source's config. + +```ts title="src/backend/sources/MyBasicSource.ts" +export default class MyBasicSourceSource extends AbstractSource { + handle(somePlay: PlayObject) { + const discovered: PlayObject[] = this.discover([somePlay]); + // emit plays that can be scrobbled by clients + this.scrobble(discovered); + } +} +``` + +If your service only emits an event when a play is scrobbled you can _technically_ skip using `discover()` but it is good practice to use it unless you have a very good reason not to. + +:::note + +Using `scrobble()` does not guarantee a track is actually scrobbled! The Scrobble Clients also check the play against their own "recently scrobbled" list to prevent duplicates. + +::: diff --git a/docsite/docs/installation/installation.md b/docsite/docs/installation/installation.md index a56a5e4a..41e6ba59 100644 --- a/docsite/docs/installation/installation.md +++ b/docsite/docs/installation/installation.md @@ -16,7 +16,6 @@ git clone https://github.com/FoxxMD/multi-scrobbler.git . cd multi-scrobbler nvm use # optional, to set correct Node version npm install -npm run build npm run start ``` diff --git a/package.json b/package.json index 5b4c3694..b2a16b30 100644 --- a/package.json +++ b/package.json @@ -29,7 +29,7 @@ } }, "engines": { - "node": ">=18.0.0", + "node": ">=18.19.1", "npm": ">=9.1.0" }, "repository": { diff --git a/src/backend/common/infrastructure/Atomic.ts b/src/backend/common/infrastructure/Atomic.ts index 402d72c5..6cdf80cc 100644 --- a/src/backend/common/infrastructure/Atomic.ts +++ b/src/backend/common/infrastructure/Atomic.ts @@ -6,13 +6,54 @@ import { FixedSizeList } from 'fixed-size-list'; import { PlayMeta, PlayObject } from "../../../core/Atomic.js"; import TupleMap from "../TupleMap.js"; -export type SourceType = 'spotify' | 'plex' | 'tautulli' | 'subsonic' | 'jellyfin' | 'lastfm' | 'deezer' | 'ytmusic' | 'mpris' | 'mopidy' | 'listenbrainz' | 'jriver' | 'kodi' | 'webscrobbler' | 'chromecast' | 'musikcube'; -export const sourceTypes: SourceType[] = ['spotify', 'plex', 'tautulli', 'subsonic', 'jellyfin', 'lastfm', 'deezer', 'ytmusic', 'mpris', 'mopidy', 'listenbrainz', 'jriver', 'kodi', 'webscrobbler', 'chromecast', 'musikcube']; - -export const lowGranularitySources: SourceType[] = ['subsonic','ytmusic']; - -export type ClientType = 'maloja' | 'lastfm' | 'listenbrainz'; -export const clientTypes: ClientType[] = ['maloja', 'lastfm', 'listenbrainz']; +export type SourceType = + 'spotify' + | 'plex' + | 'tautulli' + | 'subsonic' + | 'jellyfin' + | 'lastfm' + | 'deezer' + | 'ytmusic' + | 'mpris' + | 'mopidy' + | 'listenbrainz' + | 'jriver' + | 'kodi' + | 'webscrobbler' + | 'chromecast' + | 'musikcube'; + +export const sourceTypes: SourceType[] = [ + 'spotify', + 'plex', + 'tautulli', + 'subsonic', + 'jellyfin', + 'lastfm', + 'deezer', + 'ytmusic', + 'mpris', + 'mopidy', + 'listenbrainz', + 'jriver', + 'kodi', + 'webscrobbler', + 'chromecast', + 'musikcube' +]; + +export const lowGranularitySources: SourceType[] = ['subsonic', 'ytmusic']; + +export type ClientType = + 'maloja' + | 'lastfm' + | 'listenbrainz'; +export const clientTypes: ClientType[] = [ + 'maloja', + 'lastfm', + 'listenbrainz' +]; export type InitState = 0 | 1 | 2; export const NOT_INITIALIZED: InitState = 0; diff --git a/src/backend/common/infrastructure/config/source/sources.ts b/src/backend/common/infrastructure/config/source/sources.ts index 3fb92f89..6a42ed80 100644 --- a/src/backend/common/infrastructure/config/source/sources.ts +++ b/src/backend/common/infrastructure/config/source/sources.ts @@ -16,6 +16,38 @@ import { WebScrobblerSourceAIOConfig, WebScrobblerSourceConfig } from "./webscro import { YTMusicSourceAIOConfig, YTMusicSourceConfig } from "./ytmusic.js"; -export type SourceConfig = SpotifySourceConfig | PlexSourceConfig | TautulliSourceConfig | DeezerSourceConfig | SubSonicSourceConfig | JellySourceConfig | LastfmSourceConfig | YTMusicSourceConfig | MPRISSourceConfig | MopidySourceConfig | ListenBrainzSourceConfig | JRiverSourceConfig | KodiSourceConfig | WebScrobblerSourceConfig | ChromecastSourceConfig | MusikcubeSourceConfig; +export type SourceConfig = + SpotifySourceConfig + | PlexSourceConfig + | TautulliSourceConfig + | DeezerSourceConfig + | SubSonicSourceConfig + | JellySourceConfig + | LastfmSourceConfig + | YTMusicSourceConfig + | MPRISSourceConfig + | MopidySourceConfig + | ListenBrainzSourceConfig + | JRiverSourceConfig + | KodiSourceConfig + | WebScrobblerSourceConfig + | ChromecastSourceConfig + | MusikcubeSourceConfig; -export type SourceAIOConfig = SpotifySourceAIOConfig | PlexSourceAIOConfig | TautulliSourceAIOConfig | DeezerSourceAIOConfig | SubsonicSourceAIOConfig | JellySourceAIOConfig | LastFmSouceAIOConfig | YTMusicSourceAIOConfig | MPRISSourceAIOConfig | MopidySourceAIOConfig | ListenBrainzSourceAIOConfig | JRiverSourceAIOConfig | KodiSourceAIOConfig | WebScrobblerSourceAIOConfig | ChromecastSourceAIOConfig | MusikcubeSourceAIOConfig; +export type SourceAIOConfig = + SpotifySourceAIOConfig + | PlexSourceAIOConfig + | TautulliSourceAIOConfig + | DeezerSourceAIOConfig + | SubsonicSourceAIOConfig + | JellySourceAIOConfig + | LastFmSouceAIOConfig + | YTMusicSourceAIOConfig + | MPRISSourceAIOConfig + | MopidySourceAIOConfig + | ListenBrainzSourceAIOConfig + | JRiverSourceAIOConfig + | KodiSourceAIOConfig + | WebScrobblerSourceAIOConfig + | ChromecastSourceAIOConfig + | MusikcubeSourceAIOConfig; From ca5cc3bb303a58007c431a8bbc3e3a16448765c5 Mon Sep 17 00:00:00 2001 From: FoxxMD Date: Fri, 12 Jul 2024 14:37:52 -0400 Subject: [PATCH 2/2] docs(feat): Add TOC to dev docs and links in readme --- README.md | 4 ++++ docsite/docs/development/dev-common.md | 21 +++++++++++++++++++ docsite/docs/development/dev-source.md | 28 ++++++++++++++++++++++++++ docsite/src/pages/index.mdx | 4 ++++ 4 files changed, 57 insertions(+) diff --git a/README.md b/README.md index 1b98c657..9b85d1a7 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,10 @@ On first startup you may need to authorize Spotify and/or Last.fm by visiting th Having issues with connections or configuration? Check the [FAQ](/docsite/docs/FAQ.md) before creating an issue! +## Development + +[Detailed architecture and development guides for Sources/Clients](/docsite/docs/development/dev-common.md) + ## License MIT diff --git a/docsite/docs/development/dev-common.md b/docsite/docs/development/dev-common.md index 728899fa..23e13eb2 100644 --- a/docsite/docs/development/dev-common.md +++ b/docsite/docs/development/dev-common.md @@ -8,6 +8,27 @@ description: Start here for MS development # Development +
+ +Table of Contents + + +* [Development](#development) + * [Architecture](#architecture) + * [Project Setup](#project-setup) + * [Common Development](#common-development) + * [Config](#config) + * [Concrete Class](#concrete-class) + * [Stages](#stages) + * [Stage: Build Data](#stage-build-data) + * [Stage: Check Connection](#stage-check-connection) + * [Stage: Test Auth](#stage-test-auth) + * [Play Object](#play-object) + * [Creating Clients and Sources](#creating-clients-and-sources) + + +
+ ## Architecture Multi-scrobbler is written entirely in [Typescript](https://www.typescriptlang.org/). It consists of a backend and frontend. The backend handles all Source/Client logic, mounts web server endpoints that listen for Auth callbacks and Source ingress using [expressjs](https://expressjs.com/), and serves the frontend. The frontend is a standalone [Vitejs](https://vitejs.dev/) app that communicates via API to the backend in order to render the dashboard. diff --git a/docsite/docs/development/dev-source.md b/docsite/docs/development/dev-source.md index 13b111b3..a02fe791 100644 --- a/docsite/docs/development/dev-source.md +++ b/docsite/docs/development/dev-source.md @@ -5,6 +5,34 @@ sidebar_position: 2 title: Source Development/Tutorial --- +
+ +Table of Contents + + + * [Scenario](#scenario) + * [Minimal Implementation](#minimal-implementation) + * [Define and Implement Config](#define-and-implement-config) + * [Create CoolPlayer Source](#create-coolplayer-source) + * [Initialize Source from Config](#initialize-source-from-config) + * [Implement Play Object Transform](#implement-play-object-transform) + * [Implement Stages](#implement-stages) + * [Build Data](#build-data) + * [Check Connection](#check-connection) + * [Test Auth](#test-auth) + * [Implement Polling](#implement-polling) + * [Further Implementation](#further-implementation) + * [Backlog](#backlog) + * [Other Source Types](#other-source-types) + * [Music History Source](#music-history-source) + * [Non-Polling Source](#non-polling-source) + * [Basic Source](#basic-source) + * [Discovery](#discovery) + * [Scrobbling](#scrobbling) + + +
+ This document will provide a step-by-step guide for creating a (trivial) new Source in MS alongside describing what aspects of the Source need to be implemented based on the service you use. Before using this document you should review [Common Development](dev-common.md#common-development). ## Scenario diff --git a/docsite/src/pages/index.mdx b/docsite/src/pages/index.mdx index 7dfae6bc..697b7264 100644 --- a/docsite/src/pages/index.mdx +++ b/docsite/src/pages/index.mdx @@ -99,6 +99,10 @@ On first startup you may need to authorize Spotify and/or Last.fm by visiting th Having issues with connections or configuration? Check the [FAQ](docs/FAQ) before creating an issue! +## Development + +[Detailed architecture and development guides for Sources/Clients](docs/development/dev-common) + ## License MIT