Skip to content

Refactor schema development with generated artifacts, tests, and automation - #224

Open
anitamnd wants to merge 20 commits into
mainfrom
refactor
Open

anitamnd wants to merge 20 commits into
mainfrom
refactor

Conversation

@anitamnd

Copy link
Copy Markdown
Contributor

This pull request refactors biotoolsSchema into a reproducible, testable schema-development workflow.

It introduces JSON vocabulary files as the source of truth and adds tooling to generate the XSD, JSON Schema, and Python vocabulary module for bio.tools. It also adds automated validation, SPDX license synchronization, and CI/release workflows.

What changed

  • Added source vocabulary files under src/enums
  • Added scripts for:
    • Generating the XSD
    • Generating the JSON Schema
    • Generating the Python vocabulary module
    • Fetching and updating SPDX licenses
  • Added pyproject.toml entry points:
    • build-schema
    • fetch-spdx
  • Added tests covering:
    • Consistency between source vocabularies and generated artifacts
    • XSD format and structural validity
    • JSON Schema validity
    • Validation of XML examples
    • Rejection of invalid enum values
    • SPDX license vocabulary shape and expected license coverage
  • Added GitHub Actions workflows for:
    • Building and testing the schema
    • Creating draft releases
    • Synchronizing SPDX licenses monthly and opening an update pull request
  • Added pre-commit and repository configuration

New schema update workflow

Manual schema updates are made in two ways:

  1. Update descriptions and structural definitions in the relevant schema templates under src/.
  2. Add new vocabulary values to the corresponding file under src/enums.

The build process then uses these sources to generate the schema files.

Compatibility behavior

Deprecated vocabulary values are excluded from the generated XSD and JSON Schema so that new records use the active vocabulary. They remain available in the generated Python vocabulary module so existing bio.tools records containing deprecated values can still be represented. TBD?

Actual changes to the schema

The changes can be reviewed in biotools_dev.xsd. The large diff is primarily caused by a formatting change: indentation now uses spaces instead of tabs. Content-wise, relatively little has changed:

  • Added Preprint as publication type
  • Updated license list
  • Updated programming languages list

Review

The biotools_dev.xsd and biotoolsj.json files in the repository root are the resulting built schemas generated by this update.

If this pull request is approved, the existing stable/ and jsonschema/ directories will be removed because their contents will be replaced by the reproducible build process and the generated artifacts in each release.

Co-authored-by: Copilot copilot@github.com

@anitamnd
anitamnd marked this pull request as ready for review September 12, 2026 16:06

@veitveit veitveit left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks super-useful and I could not find anything off.
It would be good to have Herve taking a look as well.

@marielacour

marielacour commented Sep 14, 2026 via email

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants