[woolfg]

Summary

First step of a platform-independent schema and API v2, validated with one metric: Spotify plays. Plays are Spotify streams, meaning 60 seconds or more.

Schema (migration 19 + schema.sql)

• Lookup tables:
  • platforms: spotify, apple, anchor, podigee.
  • metrics: plays, with an additive flag. Unique metrics, such as listeners, will later be stored with additive=0 and never summed.
• episodes: own INT UNSIGNED AUTO_INCREMENT IDs, unique across the database. The podcast ID stays podcasts.account_id.
• ID mappings:
  • podcastPlatformIds and episodePlatformIds map our IDs to each platform's IDs.
  • Episodes are mapped per podcast, so the same Spotify show under two accounts does not collide.
• Metric values: podcastMetrics and episodeMetrics store a value per (id, metric, platform, start_date, end_date).
  • A daily value has start = end.
  • A lifetime value runs from the release date to the date the data was fetched.
• Backfill: the migration copies the existing spotify{Podcast,Episode}DetailedStreams history and assigns episode IDs to all known Spotify episodes.

Why auto-increment instead of UUIDs:

• At about 100M rows, BINARY(16) adds about 12 bytes per row and per secondary index entry, roughly 1.2 GB or more.
• Random UUIDs fragment the B-tree.
• The usual UUID benefits, generating IDs offline and merging databases, aren't needed here.

Write path

SpotifyConnector keeps its v1 writes and also writes to v2:

• detailedStreams: daily plays, at podcast or episode level.
• metadata and episodeMetadata: lifetime plays and platform ID mappings.

Writes are single bulk upserts, so refetching days to fill gaps is idempotent.

Read API

The new queries in db_schema/queries/v2/ go through the existing /analytics/:version/:podcastId/:query route:

• podcastPlays, podcastPlaysTotal and podcastPlaysLifetime. The total includes days, the number of days with data.
• The episode equivalents: episodePlays, episodePlaysTotal and episodePlaysLifetime.
• episodes: our episode IDs mapped to platform IDs and names.
• podcastPlaysMissingDays: days with no data, for gap filling.

Each row is {podcastId|episodeId, metric, platform, startDate, endDate, value}.

The API docs now cover v1 and v2. make dev-sql and make send-analytics-% take an optional VERSION.

Fixes along the way

• Access check for every version: the podcast check in /analytics only ran for v1, so v2 would have been open to any podcast.
• 401 responses: the three 401 branches didn't return, so the handler kept running after sending the error.
• Migrations: DBInitializer didn't await each migration. Note that a migration file name must not contain digits after its number, otherwise the file is not found.

Review notes

• On large databases, the episode streams backfill is a single INSERT ... SELECT. Expect it to take a while during the first deploy.
• Lifetime rows are not backfilled. They appear with the next metadata fetch.
• Cross-platform episode merging is out of scope for now.

Test plan

• make test: 64 unit tests pass, including the new MetricsRepository and SpotifyConnector tests.
• Lint and build pass.
• Migration from 18 to 19 on a MySQL 8 database seeded with v1 data: v1 and v2 row counts and stream sums match. The same Spotify episode under two podcasts gets two IDs, and AUTO_INCREMENT continues after the backfilled IDs.
• Fresh install from schema.sql: the migrations maximum is 19 and the seed rows are present.
• tests/api_e2e/analytics_v2.test.js: 9/9 pass. This was run against a separate DB and port (API_BASE_URL override), not through make e2e-tests.
• Run make e2e-tests in the standard setup.

🤖 Generated with Claude Code

[mre]

As discussed, there are a couple of downsides:

• values can be other than BIGINT
• no schema changes required for new metrics but danger of storing incorrect values under the wrong metric id
• potentially large metrics table
• can't easily optimize the metrics table
• queries can get quite complex

We decided to go with separate tables (and not all metrics in one).