Overture docs update by MitchellShiell · Pull Request #911 · overture-stack/SONG

MitchellShiell · 2026-07-30T13:36:32Z

Documentation for the Overture docs site.

The docs still described the pre-5.3.0 data model: donor/specimen/sample entities in the base schema, ID management delegated to an external service. They now describe the consolidated 5.3.0 schema and its migration path. Separately, docs/02-Usage/ → docs/02-Reference/ to match the other components, and the setup guide is rewritten around this repo's own Compose stack instead of the retired Prelude quickstart branch.

Source changes are limited to Makefile and docker-compose.yml: dev tooling, no application logic, no Java.

Song 5.3.0 data model

04-data-model-management.md: base schema corrected to the three fields it actually requires (studyId, analysisType, files); the old text listed donor/specimen/sample IDs and "essential cancer sample characteristics". Adds a Schema Options section for options.fileTypes and options.externalValidations, including that omitting an option preserves its previous value while an empty list clears it.
06-id-management.md: rewritten, replacing a 184-line page. Song generates both IDs it tracks, with no external ID service: analysis IDs are random UUIDs, file IDs deterministic UUID5 hashes of the analysis ID and file name. IDs are immutable, so changing a contributing value means UNPUBLISH + SUPPRESS + resubmit.
11-database-migration.md (new): pre-5.3.0 data spanned multiple tables, 5.3.0 writes to one, so existing databases must be migrated. Documents the TypeScript migration script, opening with a back-up warning.

Each carries a version note recording the old behaviour, including the retired "federated" ID mode.

Setup guide

The old guide opened with git clone -b quickstart .../prelude.git. That branch is retired, so it now uses the docker-compose.yml and Makefile already in this repo.

make start-deps → Keycloak 9082, keycloak-db 9444, Score 8087, Minio 8085. make start-song-server adds song-db 8432 and song-server 8080 (debug 5006).
Every port in the old table was the Quickstart's (8180, 9000, 5433), not ours. Corrected, the Conductor row is gone, and songDev.svg, which drew the Quickstart topology, is dropped.
CLIENT_ACCESS_TOKEN=68fb42b4-… and CLIENT_STUDY_ID=demo are now placeholders. Both were Quickstart values that worked nowhere else, and this stack starts with an empty database, so there is no demo study to query.
New walkthrough for getting an API key: set the realm admin password in Keycloak, get a token via the system client's direct access grant, exchange it at apikey/api_key for a song.WRITE/score.WRITE key.
Also noted: Keycloak imports myrealm and downloads keycloak-apikeys at start-up; the targets build with ./mvnw, so a JDK is needed even for the supporting services alone. make clean and make log-song-server documented.

Source changes

Makefile. The guide does not work without these:

Compose is detected, not assumed: probes docker compose version, falls back to the standalone binary. V1 no longer ships with Docker Desktop, so every Compose target failed outright on a current install.
MVN_EXE prefers the bundled ./mvnw over a system mvn.
ROOT_DIR moves above the detection block that now depends on it.

docker-compose.yml. Without this the stack could not pull at all. bitnami/keycloak:22 and bitnami/postgresql:11 → bitnamilegacy/*; Bitnami archived its versioned tags in 2025. Commented in place.

Expect a dirty tree after your first build. fmt-maven-plugin binds format to install, so the documented ./mvnw clean install reformats nineteen committed files under song-server/. Deliberately kept out of this PR. Worth a separate formatting pass on develop.

`docs/02-Usage/` → `docs/02-Reference/`

Every page moves; 02-Usage.mdx becomes 02-Reference.mdx (retitled "Reference", glossary retained). docs/custom-schemas.md is deleted, because its Options content is now the Schema Options section of 04-data-model-management.md, where anyone configuring an analysis type already is.

One real fix came along in the move: 01-retrieving-metadata.md and 02-updating-metadata.md showed a multi-analysis update querying analysis/search/id?submitterDonorId=<sample>, an endpoint that does not serve submitter sample lookups. Song has no server-side search for submitter sample or donor IDs, so the example now pages through the study's analyses, filters client-side, and patches each match by analysis ID.

Overview and README fixes

00-overview.md: the repo tree said /song-servers (actual: /song-server); the song-java-sdk entry described a package that no longer exists; song-python-sdk was described as being for "java applications".
01-setup.md: the services table credited Keycloak with OAuth2 "for Score" on the Song page, and the Song-server application.yml link pointed into the score repo.
README.md: stale docs/core-software/Song/* links → /develop/Song/*, contributing → /develop/contributing, getting started → /deploy.

The docs site's under-development section has been removed; Lyric (the tabular data submission service) is now the canonical destination for the 'data submission system' reference, so link to its core-software overview instead of the retired /docs/under-development/ page.

The site renamed /docs/core-software/* to /build/* to /develop/*, moved /docs/contribution to /develop/contributing, /guides/getting-started to /deploy, and /guides/administration-guides to /use. Each old path only resolved through a redirect hop. Verified every new target against a local site build.

Bare site paths (/community/support, /use/cli-submissions, /develop/Lyric/overview, /develop/Maestro/overview) only resolve once these files are assembled into the docs site. Read directly on GitHub, in a clone, or in an IDE they 404. Links pointing at content outside this repository now use the full URL form, matching the README. 18 links across 13 files.

Ten links to Song's own pages were written as absolute site paths (/develop/Song/Reference/..., /develop/Song/setup#song-client-setup). Relative paths resolve both standalone and through the aggregated site; the absolute form only works in the latter. 01-setup.md points into 02-Reference/, 00-submitting-metadata.md and 10-client-reference.md point at siblings, and 03-publication-controls.md points back up at 01-setup.md. This matches the form the docs already use elsewhere in 02-Reference/, for example ./11-database-migration.md. Both anchors verified against their target headings: '## Song-Client Setup' in 01-setup.md and '### Song Terminology' in 02-Reference.mdx.

…arget Three links used the retired /guides/* structure. All three resolve only through a plugin-client-redirects hop today. Two were worse than stale. 04-data-model-management.md: /guides/administration-guides/updating-the-data-model now points at /use/administration/building-song-schemas. The page was renamed, not just moved, so the link text changes from "updating data models" to match. 09-api-reference.mdx carried two links whose targets do not hold the content they promise: - "accessing an API key from a deployed Overture platform" pointed at /guides/user-guides/cli-submissions#generate-an-api-key. That anchor does not exist, and the current page (/use/cli-submissions) has no API key content at all. Retargeted at the Keycloak deployment guide, which is where the Overture API Key Provider is actually installed on a deployment. - "administration guide on identity & access management" pointed at /guides/administration-guides, which redirects to the generic /use overview. No such administration guide exists. The content it describes, generating and managing API keys through Keycloak, is this repo's own Configuring Keycloak page, so it now points there relatively. Both link texts updated to name what the reader will find.

leoraba · 2026-08-05T19:32:32Z

+Clone the migration repository and install its dependencies:
+
+```bash
+git clone https://github.com/overture-stack/song_5_3_migration.git


this repo should not be used anymore, the migration script was added in the same SONG repo.

so users can do:

# clone SONG repo git clone https://github.com/overture-stack/SONG.git # Go to folder cd SONG/data-migration # Install pnpm install

Fixed, now clones SONG and uses SONG/data-migration. Also fixed data-migration/README.md, which had the same stale clone.

leoraba · 2026-08-05T19:46:23Z

-DOCKER_COMPOSE_EXE := $(shell which docker-compose)
+# Prefer the Compose V2 plugin ("docker compose"), falling back to the standalone
+# V1 binary. V1 is no longer shipped with current Docker Desktop releases.
+DOCKER_COMPOSE_EXE := $(shell docker compose version >/dev/null 2>&1 && echo "docker compose" || which docker-compose)


leoraba · 2026-08-05T20:04:58Z

 ```python
 import requests

 url = "https://song.virusseq-dataportal.ca"


found in this file and other files an used to real URL (now iMs) but Song shouldn't point to any of these URLs

Swapped all these with https://. Also replaced the real VirusSeq study and analysis IDs (LSPQ-QC, MCPL-MB, 4861730f-...) with the ABC123 placeholders.

leoraba · 2026-08-05T20:27:22Z

+- **[Song Overview](https://docs.overture.bio/develop/Song/overview)**
+- [**Setting up the Development Environment**](https://docs.overture.bio/develop/Song/setup)
+- [**Reference Docs**](https://docs.overture.bio/develop/Song/Reference)


I noticed these URLs atm are not valid and have changed with path /develop/. I guess this is expected to change docs URL based on branches?

Expected, yes. Those paths go live when overture-stack/docs#48 merges

leoraba

great work keeping the docs up to date. just left minor changes.

leoraba

LGTM

MitchellShiell mentioned this pull request Jul 30, 2026

Overture docs V2 update overture-stack/docs#48

Open

5 tasks

MitchellShiell force-pushed the overtureDocsUpdate branch from 8b67b0a to 69da766 Compare July 30, 2026 16:04

MitchellShiell added 8 commits July 30, 2026 13:30

url updates

b5ca45b

usage > reference

f795333

docs update

0d90a83

schema migration update

8c88c02

setup page update

04ae328

updated dev quickstart, retiring prelude on all components

bb13738

MitchellShiell force-pushed the overtureDocsUpdate branch from 69da766 to bb13738 Compare July 30, 2026 17:33

MitchellShiell requested a review from leoraba July 30, 2026 17:37

MitchellShiell added 6 commits August 4, 2026 09:49

fix duplicated docs link, typo, and an unclosed italic

a9f1342

removed quickstart references from readme

c2bab2a

contributing link updated to in repo ref

2312aa5

leoraba reviewed Aug 5, 2026

View reviewed changes

docs: address review feedback, drop real deployment URLs

c562e8e

MitchellShiell requested a review from leoraba August 5, 2026 20:46

leoraba approved these changes Aug 5, 2026

View reviewed changes

Navigation Menu

Uh oh!

Overture docs update - #911

Overture docs update#911
MitchellShiell wants to merge 15 commits into
developfrom
overtureDocsUpdate

MitchellShiell commented Jul 30, 2026 •

edited

Loading

Uh oh!

leoraba Aug 5, 2026

Uh oh!

MitchellShiell Aug 5, 2026

Uh oh!

leoraba Aug 5, 2026

Uh oh!

leoraba Aug 5, 2026

Uh oh!

MitchellShiell Aug 5, 2026

Uh oh!

leoraba Aug 5, 2026

Uh oh!

MitchellShiell Aug 5, 2026

Uh oh!

leoraba left a comment

Uh oh!

leoraba left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

MitchellShiell commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Song 5.3.0 data model

Setup guide

Source changes

docs/02-Usage/ → docs/02-Reference/

Overview and README fixes

Uh oh!

leoraba Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

MitchellShiell Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

leoraba Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

leoraba Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

MitchellShiell Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

leoraba Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

MitchellShiell Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

leoraba left a comment

Choose a reason for hiding this comment

Uh oh!

leoraba left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MitchellShiell commented Jul 30, 2026 •

edited

Loading

`docs/02-Usage/` → `docs/02-Reference/`