[#24] SEO: declare the sitemap in robots.txt, add description and OpenGraph tags

[vharseko]

Fixes #24

Changes

• antora-playbook.yml: site.robots is now the text of robots.txt instead of allow, so it also declares the sitemap index:

  User-agent: *
  Allow: /
  Sitemap: https://doc.openidentityplatform.org/sitemap.xml
  

  Antora writes this value verbatim, so a comment next to it asks to keep the Sitemap: host in sync with site.url. The file also gets the CDDL license header (Copyright 2024-2026 3A Systems, LLC.: the playbook has been written by 3A Systems since 2024).

• supplemental-ui/partials/head-meta.hbs:

  • <meta name="description"> for pages without :description:. The default UI emits it only when the page sets :description:, and no page does today;
  • OpenGraph: og:type, og:site_name, og:title, og:description, og:url (the canonical URL; absent on the 404 page), og:image;
  • twitter:card = summary.

  og:image is the GitHub organization avatar, the same image www.openidentityplatform.org uses, so no image is added to the repository.

• supplemental-ui/partials/meta-description.hbs (new): the generated description:

  • product pages: <page title> - <component title> documentation, Open Identity Platform, e.g. "Installing OpenAM Core Services - OpenAM: Access Management documentation, Open Identity Platform";
  • start page and 404: a description of the platform naming the four products.

Verification

Local Antora build of this branch:

• robots.txt has the Sitemap: line;
• all 285 pages have every tag exactly once; og:url is missing only on 404.html;
• titles with special characters (Release Levels & Interface Stability, Standards, RFCs, & Internet-Drafts) are escaped correctly; no tags or unresolved entities are left in the attributes;
• a page given a temporary :description: with quotes, & and <b> gets a single description tag (emitted by the UI) and an identical og:description, escaped the same way;
• no new Antora errors.

[maximthomas]

praise: The new tags are placed so that they never collide with what the default UI already emits.

• The generated description is wrapped in {{#unless page.description}}, so a page that sets :description: keeps the single tag the UI writes (supplemental-ui/partials/head-meta.hbs:22-24).
• og:url sits inside {{#with page.canonicalUrl}}, so the 404 page gets no og:url tag rather than an empty one (supplemental-ui/partials/head-meta.hbs:29-31).
• Titles pass through {{{detag … attribute=true}}} before they go into attributes (supplemental-ui/partials/head-meta.hbs:27, supplemental-ui/partials/meta-description.hbs:18).

---

suggestion (non-blocking): No automated check covers the new head tags or robots.txt.

.github/workflows/publish.yml:3

publish.yml is the only workflow. It runs on push to master and on workflow_dispatch, never on pull_request, and package.json has no test script. The checks in the description (285 pages, each tag exactly once, og:url missing only on 404) were done once in a local build. A later template edit that drops or duplicates a tag would still build green and deploy, and so would a HEAD UI-bundle change (snapshot: true) that stops detag … attribute=true from escaping.

# .github/workflows/check.yml
name: Check site build
on: pull_request
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: '24'
    - run: npm i antora
    - run: npm run build
    - name: Check robots.txt and head tags
      run: |
        grep -qx 'Sitemap: https://doc.openidentityplatform.org/sitemap.xml' build/site/robots.txt
        fail=0
        while IFS= read -r f; do
          grep -q 'http-equiv="refresh"' "$f" && continue
          for tag in '<meta name="description"' '<meta property="og:title"' '<meta property="og:description"' '<meta property="og:image"' '<meta name="twitter:card"'; do
            n=$(grep -cF "$tag" "$f" || true)
            [ "$n" -eq 1 ] || { echo "$f: $n x $tag"; fail=1; }
          done
        done < <(find build/site -name '*.html')
        exit $fail

Pin: deleting any of the new tags from head-meta.hbs, or the Sitemap: line from the playbook, turns this job red.

---

suggestion (non-blocking): The Sitemap: line in robots.txt repeats the host instead of following site.url.

antora-playbook.yml:22, :18

Antora writes a custom site.robots value verbatim (only trimEnd()), while sitemap.xml is built from the effective site URL. So npx antora --url https://staging.example antora-playbook.yml (or URL set in the environment, or a later edit of line 18 alone) produces a staging sitemap.xml with a robots.txt that still points to the production one. The output is correct today. Antora cannot generate the line itself, so a comment is the cheapest guard.

  url: https://doc.openidentityplatform.org
  # Antora writes robots verbatim: keep the Sitemap host in sync with site.url above.
  robots: |
    User-agent: *
    Allow: /
    Sitemap: https://doc.openidentityplatform.org/sitemap.xml

---

nitpick (non-blocking): The new header in antora-playbook.yml says "Portions Copyright"; it should say "Copyright".

antora-playbook.yml:13

At base this file had no license header and no copyright line, so 3A Systems is the sole copyright holder. "Portions copyright" is for a contribution to a file that someone else holds, as in head-meta.hbs (ForgeRock 2017). The new meta-description.hbs already gets this right.

# Copyright 2026 3A Systems, LLC.

---

nitpick (non-blocking): The new CDDL headers point to legal/CDDLv1.0.txt, which this repository does not have.

antora-playbook.yml:5, :9, supplemental-ui/partials/meta-description.hbs:6, :10

The head tree has no legal/ directory and no license file. The only legal/ paths are the JDK notices under openam/apidocs/legal/. The same boilerplate was already in head-meta.hbs at base, so this copies a precedent.

---

note (non-blocking): A change that the description does not mention.

• antora-playbook.yml:1-13 — a new CDDL license header.

[vharseko]

Re: no automated check of the head tags and robots.txt — #28 adds the pull request check this asks for: .github/workflows/build.yml runs on pull_request to master, with npm ci and Node 24 action versions. It also replaces the UI bundle from GitLab HEAD (snapshot: true) with ui/ui-bundle.zip kept in the repository, so a UI change can no longer silently stop detag … attribute=true from escaping. What #28 does not do is the tag check itself; I will add it as a step of that build.yml once #28 and this PR are merged, rather than as a second workflow here: tracked in #37.

Re: Sitemap: host repeated — added the comment above robots:.

Re: "Portions Copyright" — changed to Copyright 2024-2026 3A Systems, LLC.: the playbook has been written by 3A Systems since 2024, so the range starts there. #28 adds the same header to this file (and to publish.yml), so I changed it there in the same way; the two PRs still merge without a conflict.

Re: legal/CDDLv1.0.txt — no change here: the same header is in hundreds of files on master, the documentation pages included. Adding the license file belongs to the whole repository, not to this PR.

Re: the header not mentioned in the description — added to the description.

[maximthomas]

praise: Both round-1 points are fixed as asked, and every page still gets each head tag once.

• antora-playbook.yml:13 now reads Copyright 2024-2026 3A Systems, LLC.. The file had no earlier holder and dates from 2024.
• antora-playbook.yml:19 ties the hard-coded Sitemap: host to site.url.
• The description fallback is gated on {{#unless page.description}} (supplemental-ui/partials/head-meta.hbs:22-24), which exactly complements the default UI's own tag. og:url sits inside {{#with page.canonicalUrl}} (:29-31).