Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/chain-file.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ chain_file() {
}

if [ "${1-}" = --check ]; then
[ "$(chain_file 'katoptra/dropbox/.github/workflows/sync.yml@refs/heads/main')" = sync.yml ]
[ "$(chain_file 'katoptra/ctan/.github/workflows/sync.yml@refs/heads/main')" = sync.yml ]
[ "$(chain_file 'katoptra/ctan/.github/workflows/hourly.yml@refs/heads/josh/topic')" = hourly.yml ]
[ "$(chain_file 'o/r/.github/workflows/sync.yml@81fb6ee34abc7703087cfe4a303263febc912c6f')" = sync.yml ]
echo "chain-file: PASS"
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,18 @@ jobs:
timeout-minutes: 30
strategy:
matrix:
variant: [rsync, proton]
variant: [rsync]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: ./.github/actions/toolbox
# The test cases of each guard show the inputs that the guard rejects. The build shows
# an input that the guard accepts.
- name: The security guards refuse what they should
if: matrix.variant == 'rsync'
run: |
sh .github/validate-vars.sh --check
sh .github/chain-file.sh --check
sh docker/gpg-gate-check.sh
- name: The op verb tries again only when the command did not start
if: matrix.variant == 'rsync'
run: sh .github/op-retry-check.sh
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
# Actions keeps the layers in its cache, with one scope for each variant. release
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# On a semver tag, this workflow does these steps:
# 1. It makes each image for amd64 and arm64.
# 1. It makes the image for amd64 and arm64.
# 2. It pushes the two tags.
# 3. It moves the floating v<major> tag.
# 4. It publishes the GitHub release.
Expand All @@ -19,7 +19,7 @@ jobs:
timeout-minutes: 60
strategy:
matrix:
variant: [rsync, proton]
variant: [rsync]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
Expand Down
40 changes: 13 additions & 27 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
This repository is the toolbox that all katoptra mirrors include from a URL. `README.md` is
the manual. It has sections about these parts of the system:

- The layers, the verbs and the engines
- The layers, the verbs and the engine
- The secrets and the storage
- The images and the workflows.
- The image and the workflows.

It also tells how a mirror uses these parts and how a mirror changes them. This file gives
the items that a change must not break. Each item is a verified fact about go-task or a
Expand Down Expand Up @@ -45,14 +45,14 @@ Read that section before you write.
- Use only the keys `trusted-hosts` and `cache-expiry` in `.taskrc.yml`. Task ignores a key
that it does not know, and it shows no warning. In that condition, it downloads the remote
Taskfiles again each time that it runs.
- Each tool in an image is from `toolchain.lock.toml`. `docker/lock.py` reads that file
for the Dockerfiles and for the toolbox action, and the build examines each tool before
- Each tool in the image is from `toolchain.lock.toml`. `docker/lock.py` reads that file
for the Dockerfile and for the toolbox action, and the build examines each tool before
it uses it. Most tools have a recorded sha256. The AWS CLI zip has no sha256, because
upstream publishes only a detached PGP signature for it. Thus, the fetch stage verifies
that signature with `docker/aws-cli.pub`, and it must find the fingerprint that
`toolchain.lock.toml` pins. It uses the same GOODSIG and VALIDSIG awk check as the engine
uses for TeX Live.
- The images set `TASK_REMOTE_OFFLINE=1`. In a run, task gets the included Taskfiles from
- The image sets `TASK_REMOTE_OFFLINE=1`. In a run, task gets the included Taskfiles from
the `.task/remote` cache of the mirror. This cache goes into the container with the
repository (a bind mount). Task does not get the Taskfiles from the network.
- `run` and `render` put the git top level of the repository at `/work` (a bind mount).
Expand All @@ -70,19 +70,13 @@ Read that section before you write.
3.53.1), but bash and dash print `on`. For a guard that must operate in all these shells,
make each step exit manually.
- The built-in shell of task has no `umask`. To make a file with the mode 0600, use
`install -m 600 /dev/null "$f"`. Then write the file. The `age` verb of the proton engine
does this.
- `pd` runs each command of the Proton CLI. After each command, with all exit codes, `pd`
examines the session, and it writes the session back to the bucket if the refresh token
changed. The token changes each time that the CLI uses the session. If a run keeps a
changed token and does not write it back, Proton rejects the token of the next run. A new
login is then necessary. For the same cause, two mirrors do not use one session.
`install -m 600 /dev/null "$f"`. Then write the file.
- Pin each action to a full SHA, with the version in a comment at the end of the line. A
mirror pins the two reusable workflows in this method. Each reusable workflow does a
checkout of this repository at the commit of the workflow (`github.job_workflow_sha`) for
the toolbox action and `toolchain.lock.toml`. Thus, the pin of the workflow is the only
pin. The URLs in `includes:` use the git tag `v2`, and the image uses the tag
`<variant>-v2`. The two tags move. When they move, all mirrors get the release.
`rsync-v2`. The two tags move. When they move, all mirrors get the release.
- Give `-r` to each `xargs` if a filter can make its input empty. GNU `xargs` runs its
command one time on empty input. A guard on the file that a pipe reads is not a guard on
the input of `xargs`. For example, the `SLASH` awk of `pages` removes the root, which has
Expand All @@ -109,12 +103,12 @@ Read that section before you write.
container branch and the decision-batch branch use `.run/tl` only if the batch has a
`TL` path.
- Do not make a task internal if a host verb starts it with its name in the image. go-task
does not start an internal task from the command line. For example, `empty-trash` starts
`empty-trash-pipeline` with `task op`. Thus, `empty-trash-pipeline` is not internal.
does not start an internal task from the command line. For example, `plan` starts
`plan-pipeline` with `task op`. Thus, `plan-pipeline` is not internal.
- go-task reads the global vars of the Taskfiles that one Taskfile includes in a random
sequence at each run (verified, 3.53.1 and 3.54.0). Thus, an engine global var that reads
a toolbox var (`RUN`) can get that var with no value in the tasks of a mirror. For this
cause, `SESSION` and `UPLOAD` in `proton.yml` give `ROOT_DIR/.run` in full. Also, if a
cause, in an engine global var, write `ROOT_DIR/.run` in full, not `RUN`. Also, if a
mirror sets `IMAGE` in its root vars, a run can use the default of the engine. If the
toolbox entry of `includes:` sets `IMAGE`, the run always uses that value.
- Do not put an empty field first in a line that `read` gets with a TAB in `IFS`. A TAB is
Expand Down Expand Up @@ -146,14 +140,13 @@ Read that section before you write.
## Verifying a change

```sh
cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline
cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline
cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline
sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh
sh .github/op-retry-check.sh
```

After a change to a verb, run `task render-update` to update the `render.txt` files.
`offline` is the check of each engine.
After a change to a verb, run `task render-update` to update `examples/rsync/render.txt`.
`offline` is the check of the engine.

The rsync check runs on `examples/rsync/fixtures/`:

Expand All @@ -172,11 +165,6 @@ The rsync check runs on `examples/rsync/fixtures/`:
Make the tree again, with a new key, only to change its shape. There is no copy of the
private half of the test key.

The proton check runs `confirm` on `examples/proton/fixtures/`: an upload summary that it
accepts and one that it rejects. Then it does a dry run of `empty-trash-pipeline` from the
command line. Last, it encrypts and decrypts a file with the `age` verb and a test
identity.

`task check` does not examine a change to `README.md`. For that change, the review is the check.
The other repositories of the org link to headings of `README.md`. Do not change the text of
these headings. If you change it, the links from those repositories cannot find them:
Expand All @@ -187,8 +175,6 @@ these headings. If you change it, the links from those repositories cannot find
- `#the-rsync-engine`
- `#when-a-run-fails`
- `#content-types`
- `#the-proton-engine`
- `#the-session`
- `#r2-specifics`
- `#monitoring`
- `#rules-a-mirror-keeps`.
23 changes: 10 additions & 13 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,38 +7,35 @@ are applicable to this repository. This repository adds three rules:
sets.** If a `vars:` block has a default, the run uses that default, not the root value
of the mirror. Put each default inline in the command: `{{.X | default N}}`. If two
commands of a verb read the var, put the default in a task var with a different name
(`BATCH_SIZE: '{{.BATCH_GB | default 4}}'` in `split`). `IMAGE` is different: each
(`BATCH_SIZE: '{{.BATCH_GB | default 4}}'` in `split`). `IMAGE` is different: the
engine sets `IMAGE: '{{.IMAGE | default "..."}}'` in `vars:`. Thus, if the toolbox entry
of `includes:` gives `IMAGE`, the run uses that value.
- **Do not change the name of a verb in `toolbox.yml` or in an engine.** A mirror includes
these files with `flatten: true`. Then the host verbs and the verbs in the container use
one namespace.
A new name for one of these verbs is a new major version.
- **A change to a verb that a pipeline runs is a change to a `render.txt`.** `render` makes
a dry run of the `pipeline` of each example. A change to a verb changes these files:
- A toolbox verb: the `render.txt` of each example that runs it
- A verb of the rsync engine: `examples/rsync/render.txt`
- A verb of the proton engine: `examples/proton/render.txt`.

A host verb, for example `run` or `session-seal`, is not in a render. Run
`task render-update` in each example. Commit the result with the change. The diff of the
pull request is the review. A verb with `silent: true` renders no command. Thus, for
a dry run of the `pipeline` of the example. A change to a toolbox verb or to a verb of
the rsync engine changes `examples/rsync/render.txt`.

A host verb, for example `run` or `image-build`, is not in a render. Run
`task render-update` in `examples/rsync`. Commit the result with the change. The diff of
the pull request is the review. A verb with `silent: true` renders no command. Thus, for
`report` and `report-engine`, the review of their rows is in the diff of the file that
has the verb.

## Checking a change

```sh
cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline
cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline
cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline
sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh
```

CI does these steps for each image, on each pull request and on each push to `main`:
CI does these steps for the image, on each pull request and on each push to `main`:

1. It builds the image.
2. It runs `task run -- task tools`.
3. It runs `task check`.
4. It runs `task run -- task offline`.

It also runs the three guards one time, in the job of the `rsync` image.
It also runs the three guards before it builds the image.
Loading
Loading