[#25] CI: build pull requests and make the site build reproducible

[vharseko]

Fixes #25, together with #33 — this PR does items 1–4 of it: a reproducible build and a pull request check that fails on new Antora errors and broken links. Item 5, the DocSearch migration, is #33.

Changes

• publish.yml: npm i antora → npm ci, so the versions in package-lock.json are used.

• GitHub Actions on Node 24: checkout v7, setup-node v7, configure-pages v6, upload-pages-artifact v5, deploy-pages v5 — the Node 20 versions are deprecated and were forced onto Node 24 with a warning. The only dotfile of the site, .nojekyll, which upload-pages-artifact v4+ leaves out, is not needed by an Actions-based Pages deployment.

• antora-playbook.yml: the UI bundle was taken from GitLab HEAD with snapshot: true, so any upstream UI change reached production unreviewed. It is now ui/ui-bundle.zip, kept in the repository; the playbook records its source (antora-ui-default job 15385706035, bundle-stable) and sha256. A link to a specific GitLab job is not used because job artifacts expire.

• publish.yml, antora-playbook.yml: the CDDL license header, Copyright 2024-2026 3A Systems, LLC. (both files have been written by 3A Systems since 2024).

• build.yml (new): on every pull request to master (and manually) builds the site with the API docs and writes to the job summary:

  • Antora messages: errors and warnings with file and line (errors also as annotations), and missing attribute counts per file; also when Antora itself fails, whose fatal error goes only to the log file;
  • broken links from lychee --offline over the generated HTML, API docs excluded. The anchor of a link is not checked yet: CI: check the anchors of links in the pull request build #36.

• .github/build-baseline (new): the job fails on an Antora error or a broken link that is not in the known lists — antora-errors.txt (2 known errors) and broken-links.txt (10 known links), all tracked in the product repositories (below). A known one that is gone is reported as a notice so that it is removed from the list. A second occurrence of a known error counts as new. check.sh does the comparison and fails on a missing or malformed build output; check.sh --update rewrites the lists from a build. A self-test step checks that check.sh fails on a new Antora error and on a new broken link, and passes once they are in the lists.

  publish.yml is not gated: one broken docs upload of a product must not stop the publication of the docs of all products; there Antora's messages appear only in the job log, and links are not checked.

Known problems (the baseline)

Problem	Tracked in
OpenIG: level-0 section, duplicate id, unterminated open block	OpenIdentityPlatform/OpenIG#182
OpenIDM: incomplete table row in appendix-scripting.adoc	OpenIdentityPlatform/OpenIDM#232
OpenIDM: 6 broken links to old OpenDJ 3.5 / OpenAM 13, 13.5 docs (404 on the site)	OpenIdentityPlatform/OpenIDM#232
OpenIDM: 4 links with one .. too many — they load on the site, because the browser stops at the host root, but break under any path prefix	OpenIdentityPlatform/OpenIDM#232

Fixed in the product repositories and gone from master since this PR was opened: the .:chap-jee-agent-config.adoc xrefs (OpenIdentityPlatform/commons#315), the legacy ForgeRock links in OpenAM (OpenIdentityPlatform/OpenAM#1154), OpenIdentityPlatform/OpenAM#1155, the opendj/javadoc links (OpenIdentityPlatform/OpenDJ#1129) and the 3,523 missing attribute warnings of the OpenDJ reference (OpenIdentityPlatform/OpenDJ#1131).

Verification

Run locally with the same commands as the workflow:

• the site built with the vendored bundle is byte-identical to the one built from the GitLab snapshot, except lastmod in the sitemaps;
• npm ci installs Antora 3.1.15 from the lock file;
• report on the current master: 2 errors, 2 other warnings, 31 missing attribute warnings; lychee: 43,108 links checked, 10 broken. Without --fallback-extensions html lychee reports ~24,000 false positives, because the site uses extensionless URLs (html_extension_style: drop);
• the baseline check passes on master; with a broken xref, a broken link and a stale baseline entry added on purpose it fails with two "New …" errors and one "no longer found" notice;
• after the review: check.sh exits non-zero without build/ or with a non-JSON log line, and --update then leaves the lists untouched; a repeated known error fails it; % and CR in an annotation are sent as %25 and %0D; a multi-line Antora message stays in one row of the summary table; the self-test step passes against check.sh and fails against it without failed=true, with the comm operands swapped, and with .error_map misspelled.

On GitHub, the Build check of this pull request passes with the baseline check enabled, with no "no longer found" notice, so the lists match the build on the runner.

[maximthomas]

praise: The gate is reproducible and its baselines provably match the runner build.

• ui/ui-bundle.zip hashes to the sha256 recorded in antora-playbook.yml:44 (e0ac81aa…e70d), so the vendored bundle can be checked against its source.
• .github/workflows/build.yml:86-89 tells lychee's exit 2 (broken links) apart from a failure of the check itself.
• The Build run on 8646d34 annotates the four known errors with workspace-relative paths and emits no "No longer found" notice, so both lists match the build on the runner.

---

issue (non-blocking): check.sh passes when its input is missing or malformed, and --update then empties both lists.

.github/build-baseline/check.sh:83-84

"$(antora_errors)" and "$(broken_links)" are arguments to compare, so the non-zero status of jq | sort is discarded and set -e never fires. Without build/ (or with a non-JSON line in antora.log) every known entry becomes a "no longer found" notice and the script exits 0; with --update both files are rewritten to 0 entries, exit 0. In CI the jq calls in build.yml:51 and :93 fail first, so the gate itself holds; the local --update road does not. A plain assignment does stop set -euo pipefail:

errors=$(antora_errors)
links=$(broken_links)
$update || echo "## Compared with the known problems" >> "${GITHUB_STEP_SUMMARY:-/dev/null}"
compare "Antora errors" antora-errors.txt "$errors" "file <TAB> message"
compare "broken links" broken-links.txt "$links" "page <TAB> link"

---

issue (non-blocking): build/antora.log is appended to across local builds.

.github/workflows/build.yml:44-45

Antora's log file destination appends by default (runtime.log.destination.append: true), and output.clean removes only build/site. Re-running the documented commands locally keeps errors from earlier builds in the log: check.sh still counts an error that is already fixed, and check.sh --update writes it back into antora-errors.txt. CI starts from a fresh checkout and is not affected.

      run: |
        mkdir -p build
        rm -f build/antora.log
        npx antora antora-playbook.yml --log-format=json --log-file=build/antora.log
        npm run copyApiDocs

---

question (non-blocking): Should a second occurrence of a known Antora error in the same file fail the build?

.github/build-baseline/check.sh:36, :59

The key is file + message without the line number, and both lists go through sort -u. A second target of xref not found: .:chap-jee-agent-config.adoc#configure-j2ee-policy-agent in chap-jetty.adoc collapses into the known key: 0 new, exit 0, while the header says the check "fails on any new one". If collapsing is intended (a partial included from several pages repeats its errors), this needs nothing. If not, comm counts repeated lines, so plain sort keeps the key free of line numbers and still catches the repeat; run check.sh --update afterwards:

    | "\(.file.path // "" | ltrimstr($root))\t\(.msg)"' "$root/build/antora.log" | LC_ALL=C sort

  known=$({ grep -v '^#' "$file" || true; } | LC_ALL=C sort)

---

suggestion (non-blocking): Nothing in CI exercises the failure path of check.sh.

.github/build-baseline/check.sh:72, .github/workflows/build.yml:101

The Build run only reaches the "0 new, 0 no longer found" path; the failing path was checked once by hand. A mutant that deletes failed=true (line 72), or swaps the comm operands so that new is always empty, keeps every Build run green. The step below passes against the head's check.sh and fails against the mutant without failed=true:

    - name: Self-test the baseline check
      run: |
        t=$(mktemp -d)
        mkdir -p "$t/.github" "$t/build"
        cp -r .github/build-baseline "$t/.github/"
        echo '{"level":"error","msg":"self-test","file":{"path":"'"$t"'/x.adoc"}}' > "$t/build/antora.log"
        echo '{"error_map":{}}' > "$t/build/lychee.json"
        if GITHUB_STEP_SUMMARY=/dev/null "$t/.github/build-baseline/check.sh" > /dev/null; then
          echo "check.sh passed a new Antora error" >&2
          exit 1
        fi

Pin: placed before "Compare with the known problems", it turns red as soon as a new Antora error stops failing check.sh.

---

suggestion (non-blocking): When Antora itself fails, the job shows no cause.

.github/workflows/build.yml:47

With --log-file, Antora's CLI logs a fatal error to the configured logger — the file, not the console — and "Report Antora messages" has no if:, so it is skipped once "Generate Site" fails. A pull request that breaks the build gets a red check with nothing in the console, the summary or the annotations. The report step already annotates fatal:

    - name: Report Antora messages
      if: ${{ !cancelled() && hashFiles('build/antora.log') != '' }}

---

suggestion (non-blocking): The link check does not cover anchors.

.github/workflows/build.yml:80-85

lychee 0.24.2 checks fragments only with --include-fragments, and Antora resolves only the page part of an xref. Renaming a section id so that xref:page.adoc#old-id points nowhere gives neither an Antora error nor a broken link, and the check passes. Enabling it may add existing broken anchors to broken-links.txt (not measured):

          --offline --no-progress --format json --include-fragments \

---

issue (non-blocking): Annotation text is not escaped for workflow commands.

.github/build-baseline/check.sh:70, :76, .github/workflows/build.yml:52

The runner decodes %25, %0D and %0A in command data, so a broken link c%25d.html is annotated as c%d.html, and %0A splits the line. The comparison uses the raw strings and no current entry contains %, so only the annotation text is affected.

    msg=${line//%/%25}
    echo "::error title=New $title::${msg//$'\t'/ — }"

In build.yml:52: \(.msg | gsub("%"; "%25") | gsub("\r"; "%0D") | gsub("\n"; "%0A")).

[vharseko]

Done in 6264be5 (the branch is rebased onto the current master):

• check.sh fails on a missing or malformed input (assignment first, as suggested); --update no longer empties the lists.
• build/antora.log is removed before the build.
• Repeated Antora errors are kept (sort instead of sort -u); the baseline is unchanged, the build has no repeats.
• The self-test step runs before "Compare with the known problems"; it fails against check.sh without failed=true.
• "Report Antora messages" runs when Antora fails.
• Annotation text is escaped.

Anchors: measured with --include-fragments on a local build: 379 more broken links, of which 342 point to the id of the target page itself (Antora renders the page title without an id, so they are harmless) and 37 are real, in OpenAM, OpenDJ and OpenIDM; the check takes 3.5 min instead of 1. This needs a filter for the first kind and issues in the product repositories, so it is #36, not this PR.

[maximthomas]

praise: The round-1 fixes land where the bugs were, and the gate fails closed.

• .github/build-baseline/check.sh:84-87 assigns both inputs before compare under set -euo pipefail. A missing or malformed antora.log or lychee.json now stops the script instead of reading as an empty build, and --update cannot empty the lists.
• check.sh:36 and :59 sort without -u and compare with comm, so a second occurrence of a known Antora error counts as new.
• .github/workflows/build.yml:91 separates lychee's exit 2 (broken links found) from a failure of the check itself.

---

issue (non-blocking): A multi-line Antora message breaks the "Errors and warnings" table in the job summary.

.github/workflows/build.yml:72

The row escapes only | in .msg; line 56 also escapes CR and LF for the annotation of the same message. With an invalid component antora.yml, Antora exits with a fatal error whose message is "antora.yml has invalid syntax; …" followed by js-yaml's blank line and source snippet (@antora/content-aggregator 3.1.15:718, js-yaml 4.3.2 exception.js). The row splits, and the blank line ends the table. This is the fatal error the step now runs to show (:51). The gate is not affected.

            | "| \(.level) | \(.msg | gsub("\\|"; "\\\\|") | gsub("\r?\n|\r"; "<br>")) | \(.file.path // "" | ltrimstr($ws)) | \(.file.line // "") |"' "$log"

---

suggestion (non-blocking): The self-test does not catch swapped comm operands, and it never runs the broken-links path.

.github/workflows/build.yml:104-115

The step copies the real baseline, feeds one new Antora error and no links, and passes if check.sh exits non-zero. Measured with mutants of check.sh:

• Swapping comm -23 and comm -13 (:60-61) keeps the self-test green, because the 4 known errors that are missing from the fixture read as "new". Against the real build, that mutant exits 0 and lets a new error or link through with only a notice.
• .errormap for .error_map (:41) also keeps it green, and it turns the 42 known links into "no longer found" notices with exit 0.

The step below passes on the head and fails on both mutants and on check.sh without failed=true:

    - name: Self-test the baseline check
      run: |
        # check.sh must fail on a new Antora error and a new broken link, and pass once they are known
        t=$(mktemp -d)
        mkdir -p "$t/.github/build-baseline" "$t/build"
        cp .github/build-baseline/check.sh "$t/.github/build-baseline/"
        : > "$t/.github/build-baseline/antora-errors.txt"
        : > "$t/.github/build-baseline/broken-links.txt"
        echo '{"level":"error","msg":"self-test","file":{"path":"'"$t"'/x.adoc"}}' > "$t/build/antora.log"
        echo '{"error_map":{"/site/p.html":[{"url":"file:///site/q"}]}}' > "$t/build/lychee.json"
        out=$(GITHUB_STEP_SUMMARY=/dev/null "$t/.github/build-baseline/check.sh" 2> /dev/null) && rc=0 || rc=$?
        if [ "$rc" -ne 1 ] \
          || ! grep -qxF '::error title=New Antora errors::x.adoc — self-test' <<< "$out" \
          || ! grep -qxF '::error title=New broken links::p.html — q' <<< "$out"; then
          echo "check.sh did not report the new Antora error and the new broken link (exit $rc): $out" >&2
          exit 1
        fi
        printf 'x.adoc\tself-test\n' > "$t/.github/build-baseline/antora-errors.txt"
        printf 'p.html\tq\n' > "$t/.github/build-baseline/broken-links.txt"
        GITHUB_STEP_SUMMARY=/dev/null "$t/.github/build-baseline/check.sh" > /dev/null

Pin: with empty known lists, the comm swap leaves new empty and the exit code 0, so the step fails. Asserting on the output catches the .errormap mutant.

---

suggestion (non-blocking): Six of the known broken links load on the live site. They have one .. too many, so they are not legacy paths.

.github/workflows/build.yml:84, .github/build-baseline/broken-links.txt

lychee checks the site mounted at /site. A page-relative link with one .. more than the page is deep leaves /site and resolves to file:///<component>/…. A browser stops the extra .. at the host root. These targets return HTTP 200 on doc.openidentityplatform.org:

• openam/jee-users-guide/chap-apache-tomcat.html and chap-jetty.html → openam/admin-guide/chap-cdsso
• openidm/getting-started/chap-where-to-go.html and openidm/samples-guide/chap-ldap-samples.html → opendj/install-guide
• openidm/samples-guide/chap-fullstack-sample.html → openam/install-guide#configure-openam-custom
• openidm/samples-guide/chap-workflow-samples.html → opendj/install-guide#gui-install

In the last two, the anchor is defined on another page. The gate is right to fail such a link, since it breaks under any path prefix. Only the label in the description and in OpenIdentityPlatform/OpenAM#1154 and OpenIdentityPlatform/OpenIDM#232 is wrong, and it sends maintainers after pages that load. The fix in the source is one .. less:

* Any LDAPv3-compliant directory, including link:../../opendj/install-guide[OpenDJ, window=\_blank] and Active Directory (see xref:connectors-guide:chap-ldap.adoc#chap-ldap["Generic LDAP Connector"] in the __Connectors Guide__).

That is openidm/modules/getting-started/pages/chap-where-to-go.adoc:93. The openam/13, openam/13.5, docs/openam/13 and openam-*-policy-agents targets return 404 and are really broken.

---

nitpick (non-blocking): check.sh escapes % but not CR in annotation text. build.yml:56 escapes both, and 6264be5 says both files do.

.github/build-baseline/check.sh:70, :77

A lone CR inside an Antora error message reaches ::error and ::notice raw, and the runner cuts the annotation there. An LF cannot get there, because read splits on it first. No current entry has a CR, and the comparison is not affected.

    msg=${line//%/%25}
    msg=${msg//$'\r'/%0D}

---

nitpick (non-blocking): The description says that publish.yml reports the problems, but it reports none.

.github/workflows/publish.yml:50

publish.yml runs npm run buildAll, with no --log-file, no job summary and no lychee. On publish, Antora's messages appear only as console lines in the job log, and links are not checked at all. Suggested wording:

`publish.yml` is not gated: one broken docs upload of a product must not stop the publication of the docs of all products; there Antora's messages appear only in the job log, and links are not checked.

[vharseko]

Done in c7ef1cd (the branch is rebased onto the current master):

• The "Errors and warnings" row turns CR/LF in a message into <br>, as suggested; checked with a multi-line fatal message.
• The self-test step is the suggested one: empty lists, a new Antora error and a new broken link, then both known. It passes on check.sh and fails against the three mutants (no failed=true, swapped comm operands, .errormap).
• check.sh escapes CR in the annotation text as well.
• The description: the publish.yml wording is yours.

The six links with one .. too many: the two OpenAM ones are gone from master (OpenIdentityPlatform/OpenAM#1158 turned them into xrefs). The four OpenIDM ones are listed apart in the description; the fix proposed in OpenIdentityPlatform/OpenIDM#232, a cross-component xref, holds for them too.

The rebase onto master also changed the baseline: the OpenAM and OpenDJ docs uploaded since then fixed 2 of the 4 known errors and 32 of the 42 known links, so a2c1a17 removes them (check.sh --update on a local build; 0 new). The table of known problems in the description is updated accordingly.

[maximthomas]

praise: The self-test closes the gap from the previous round, and it fails for the right reason.

• .github/workflows/build.yml:114-117 captures check.sh's stdout and checks the exact ::error lines with grep -qxF, not just the exit code, so the step fails on all three mutants named last round.
• .github/build-baseline/check.sh:70-71 escapes % before CR, which is the order the runner unescapes them in. build.yml:72 keeps a multi-line fatal Antora message in one summary row.
• The Build run at a2c1a17 passes with the shrunk baseline and shows no "No longer found" notice.

---

suggestion (non-blocking): The self-test never makes check.sh print a "No longer found" notice.

.github/workflows/build.yml:104-123, .github/build-baseline/check.sh:61, :80

Pass 1 runs with empty known lists and pass 2 with known == build, so fixed is empty both times. Pass 2 also discards stdout. Three mutants keep the step green: comm -13 → comm -12 at check.sh:61, fixed=, and ::notice → ::debug at :80. Under the first one, every known entry that is still present gets announced as "no longer found", and an entry that really is stale does not. The gate stays green either way, because notices never set failed=true. This step is the pin suggested in the previous round, so the gap is the reviewer's.

printf 'x.adoc\tself-test\ny.adoc\tgone\n' > "$t/.github/build-baseline/antora-errors.txt"
out=$(GITHUB_STEP_SUMMARY=/dev/null "$t/.github/build-baseline/check.sh" 2> /dev/null) && rc=0 || rc=$?
if [ "$rc" -ne 0 ] \
  || [ "$out" != '::notice title=No longer found; remove from .github/build-baseline/antora-errors.txt::y.adoc — gone' ]; then
  echo "check.sh did not report only the stale known error (exit $rc)" >&2
  exit 1
fi

Pin: append this after build.yml:123. A stale known entry must give exactly one notice and exit 0. It passes on head and fails on all three mutants.

---

suggestion (non-blocking): No test feeds a CR, so the new %0D escape can be deleted without any test failing.

.github/build-baseline/check.sh:71, :79, .github/workflows/build.yml:72

The self-test's message is self-test, which has no CR or %. Deleting msg=${msg//$'\r'/%0D} at :71 and :79 keeps the step green. The code at head is correct: 50%\r becomes 50%25%0D. The pin below catches the deletion at :71. The notice copy at :79 and the <br> replacement at build.yml:72 remain untested.

echo '{"level":"error","msg":"50%\r","file":{"path":"'"$t"'/c.adoc"}}' > "$t/build/antora.log"
out=$(GITHUB_STEP_SUMMARY=/dev/null "$t/.github/build-baseline/check.sh" 2> /dev/null) && rc=0 || rc=$?
if [ "$rc" -ne 1 ] || ! grep -qxF '::error title=New Antora errors::c.adoc — 50%25%0D' <<< "$out"; then
  echo "check.sh did not escape % and CR in the annotation (exit $rc)" >&2
  exit 1
fi

Pin: append this as the last block of the step, since it replaces the Antora log. It passes on head and fails once the CR escape is deleted.

---

issue (non-blocking): When the self-test fails, echoing check.sh's output turns its ::error lines into PR annotations.

.github/workflows/build.yml:118

$out holds check.sh's workflow commands, and line 118 sends it to stderr. The runner parses stderr for commands too: ScriptHandler.cs:336 hands stderr to an OutputManager, and OutputManager.cs:87 calls TryProcessCommand (actions/runner). The first line of $out follows the message prefix on the same line, so it stays plain text. Every line after it becomes an annotation, for example ::error title=New broken links::p.html — q, which names a page and a link that do not exist. This only happens when the self-test is already red, so the job still fails, but the annotation points at the wrong thing. Leading spaces do not prevent it, because the runner trims them before looking for ::.

echo "check.sh did not report the new Antora error and the new broken link (exit $rc); its output:" >&2
sed 's/^/> /' <<< "$out" >&2

---

note (non-blocking): The description no longer matches the diff.

• antora-playbook.yml: "Changes" still says this PR adds the CDDL header here. BASE already has Copyright 2024-2026 3A Systems, LLC. at line 13 (872dec2), so only .github/workflows/publish.yml gets the header in this PR.

[vharseko]

Closing: everything in this pull request is already on master. #38 was stacked on its head a2c1a17 and was squash-merged as 52f0bf9, so antora-playbook.yml, ui/, publish.yml and .github/ on master match this branch, plus the additions of #38.

The third review round is done in #40 (issue #39), from master, since the code it touches now lives there, including the parts #38 added:

• The self-test checks the "No longer found" path: a stale known entry must give exactly one notice and exit 0.
• The self-test feeds 50% + CR, to a new entry and to a stale one, and expects 50%25%0D in both the ::error and the ::notice line.
• On failure, the output of check.sh and head-meta.sh goes to stderr quoted with > , so that its ::error lines do not become annotations. This applies to all three places: the two in the baseline self-test and the one in the head meta self-test that [#36] [#37] CI: check link anchors, head meta tags and the robots.txt Sitemap line #38 added.

The note on the description is correct: antora-playbook.yml already had Copyright 2024-2026 3A Systems, LLC. on the base (872dec2), so only publish.yml got the header here.

[vharseko]

@maximthomas, on the third round (review 5389582272): done in #40, since this PR is already on master through #38. Please review there.

• "No longer found" path: added your block. It fails on comm -12, fixed= and ::debug.
• CR escape: added your block, extended to a stale entry. It expects 50%25%0D in both the ::error and the ::notice line, and fails when the escape is deleted from either one.
• Quoted output on failure: sed 's/^/> /' in all three places, including the two that [#36] [#37] CI: check link anchors, head meta tags and the robots.txt Sitemap line #38 added; no stderr line of a failing self-test starts with ::.
• The description note: correct, the playbook header was already on the base (872dec2).