Skip to content

Add a reusable workflow for manual secret writes #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lowie27 wants to merge 1 commit into
base: v1
Choose a base branch
from
Open

Add a reusable workflow for manual secret writes #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
261 changes: 261 additions & 0 deletions .github/actions/infra/secrets/preflight/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,261 @@
---
name: Secret Preflight
description: Prove that a manual secret write can succeed, before anyone is asked to approve it
inputs:
vendor:
description: The vendor to communicate with (scaleway)
required: true
project:
description: Application name, the prefix of the vendor project
required: true
environment:
description: Environment name, the suffix of the vendor project
required: true
key:
description: Variable name to add or overwrite
required: true
value:
description: Secret value, checked for length and for collisions with strings in the log
required: true
# Vendor: Scaleway
scaleway-organization-id:
description: Scaleway organization ID
required: false
scaleway-project-id:
description: Scaleway project ID
required: false
scaleway-region:
description: Scaleway deployment region (such as nl-ams)
required: false
scaleway-access-key:
description: Scaleway API key ID
required: false
scaleway-secret-key:
description: Scaleway API key secret
required: false

outputs:
project-name:
description: Resolved vendor project name
value: ${{ steps.target.outputs.project-name }}
secret-name:
description: Resolved secret name
value: ${{ steps.target.outputs.secret-name }}
versions:
description: Number of enabled versions the secret has now
value: ${{ steps.secret.outputs.versions }}
warning:
description: Warning to surface to the approver, empty when there is nothing to warn about
value: ${{ steps.secret.outputs.warning }}
available:
description: Environments that do exist, for the error when the target does not
value: ${{ steps.projects.outputs.available }}

runs:
using: composite
steps:
- name: Validate inputs
shell: bash
env:
VENDOR: ${{ inputs.vendor }}
run: |
echo "Validating inputs for vendor: $VENDOR"
case "$VENDOR" in
scaleway)
;;
digitalocean)
echo "DigitalOcean has no secret manager, so there is nothing to write to"
exit 1
;;
azure)
echo "Azure Key Vault is not implemented, see Docs/infra-secret-write.md"
exit 1
;;
*)
echo "Unsupported vendor '$VENDOR'"
exit 1
;;
esac

# Vendor-neutral: the key becomes an env var name whatever stores it, and the masking rules are
# GitHub's. Before the CLI setup deliberately, so a bad input fails without any network work.
- name: Validate the key and the value
shell: bash
env:
KEY: ${{ inputs.key }}
SECRET_VALUE: ${{ inputs.value }}
run: |
set -euo pipefail

# [[ =~ ]] rather than grep: grep's ^ and $ are line anchors, so a key containing a
# newline would pass as long as one of its lines matched. The key becomes an env var name
# in the pod through envFrom, and envFrom silently skips one that is not a valid name.
# Not echoed back, because a key holding a newline would break the annotation.
# A vendor whose own naming is stricter adds that check in its own block below.
if [[ ! "$KEY" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
echo "::error::invalid key: letters, digits and underscores only, not starting with a" \
"digit. It becomes an environment variable name, so no hyphens"
exit 1
fi

# Masking is literal replacement, so a value occurring inside a string the reader knows is
# recoverable by subtraction: value "test" logged new-env-*** against new-env-test.
if [ "${#SECRET_VALUE}" -lt 8 ]; then
echo "::error::value is ${#SECRET_VALUE} characters, minimum 8," \
"shorter values are recoverable from their own redaction"
exit 1
fi

# Quoted inside the pattern, so this is a substring test and not a glob.
case "$KEY" in
*"$SECRET_VALUE"*)
echo "::error::value occurs inside the key, so its redaction would reveal it." \
"Pick a value that does not collide"
exit 1
;;
esac

# Scaleway names every project <app>-<env> and every manual secret <project>-secrets-manual,
# from sysops-tf-modules/monorepo/scw/environment. The collision check needs that name, so it
# cannot run above.
- name: Resolve the Scaleway target
id: target
if: ${{ inputs.vendor == 'scaleway' }}
shell: bash
env:
PROJECT: ${{ inputs.project }}
ENVIRONMENT: ${{ inputs.environment }}
SECRET_VALUE: ${{ inputs.value }}
run: |
set -euo pipefail

project_name="${PROJECT}-${ENVIRONMENT}"
secret_name="${project_name}-secrets-manual"

case "$secret_name" in
*"$SECRET_VALUE"*)
echo "::error::value occurs inside the environment or the secret name, so its" \
"redaction would reveal it. Pick a value that does not collide"
exit 1
;;
esac

echo "project-name=$project_name" >> "$GITHUB_OUTPUT"
echo "secret-name=$secret_name" >> "$GITHUB_OUTPUT"

- uses: wisemen-digital/devops-github-actions/.github/actions/infra/common/setup@v1
if: ${{ inputs.vendor == 'scaleway' }}
with:
vendor: ${{ inputs.vendor }}
scaleway-access-key: ${{ inputs.scaleway-access-key }}
scaleway-secret-key: ${{ inputs.scaleway-secret-key }}
scaleway-organization-id: ${{ inputs.scaleway-organization-id }}
scaleway-project-id: ${{ inputs.scaleway-project-id }}

# Without this a stale dropdown option surfaces further down as "secret not found", which
# reads like a permissions or region problem. Says what does exist, so a wrong guess gets one.
- name: Verify the target project exists
id: projects
if: ${{ inputs.vendor == 'scaleway' }}
shell: bash
env:
PROJECT_NAME: ${{ steps.target.outputs.project-name }}
PROJECT: ${{ inputs.project }}
run: |
set -euo pipefail

projects="$(scw account project list -o json)"

# <app>-infra-shared shares the prefix but holds the cluster and the registry, never a
# manual secret, so it is not an environment and is not offered as one.
available="$(printf '%s' "$projects" \
| jq -r --arg p "$PROJECT-" '.[].name
| select(startswith($p)) | select(endswith("-infra-shared") | not)' \
| sort | paste -sd, - | sed 's/,/, /g')"

# A wrong app matches no prefix, and "none" would be the whole error. Widen to every
# project, in full, so the reader still gets something to compare against. "default" is
# dropped only here, since no app name matches it above.
if [ -z "$available" ]; then
available="$(printf '%s' "$projects" \
| jq -r '.[].name | select(endswith("-infra-shared") | not) | select(. != "default")' \
| sort | paste -sd, - | sed 's/,/, /g')"
fi

# Exported before the failure below, so the summary can show it either way.
echo "available=${available:-none}" >> "$GITHUB_OUTPUT"

project_id="$(printf '%s' "$projects" \
| jq -r --arg n "$PROJECT_NAME" '.[] | select(.name == $n) | .id')"

if [ -n "$project_id" ]; then
echo "Target project $PROJECT_NAME exists ($project_id)."
echo "project-id=$project_id" >> "$GITHUB_OUTPUT"
exit 0
fi

echo "::error::No Scaleway project named $PROJECT_NAME." \
"Projects that do exist: ${available:-none}"
exit 1

# Fails rather than warns: write has `needs: preflight`, so nobody is asked to approve a write
# that could not have succeeded.
- name: Verify the target secret exists
id: secret
if: ${{ inputs.vendor == 'scaleway' }}
shell: bash
env:
SECRET_NAME: ${{ steps.target.outputs.secret-name }}
PROJECT_ID: ${{ steps.projects.outputs.project-id }}
REGION: ${{ inputs.scaleway-region }}
run: |
set -euo pipefail

# The API's name filter is not guaranteed to be exact, so filter again here and assert
# exactly one match, the same way manual_secrets.sh does.
secrets="$(scw secret secret list name="$SECRET_NAME" project-id="$PROJECT_ID" \
region="$REGION" -o json)"
match="$(printf '%s' "$secrets" | jq --arg n "$SECRET_NAME" '[.[] | select(.name == $n)]')"
count="$(printf '%s' "$match" | jq 'length')"

if [ "$count" -eq 0 ]; then
# Terraform creates this secret with prevent_destroy, so its absence means the
# environment was never applied rather than that someone deleted it.
echo "::error::No secret named $SECRET_NAME in project $PROJECT_ID, region $REGION." \
"Terraform creates it, so the environment has probably never been applied"
exit 1
fi
if [ "$count" -ne 1 ]; then
echo "::error::$count secrets named $SECRET_NAME; refusing to guess"
exit 1
fi

secret_id="$(printf '%s' "$match" | jq -r '.[0].id')"
secret_type="$(printf '%s' "$match" | jq -r '.[0].type')"

if [ "$secret_type" != "key_value" ]; then
echo "::error::secret $SECRET_NAME is type $secret_type, not key_value"
exit 1
fi

# Enabled versions only, not version_count, which also counts versions scheduled for
# deletion. Same reasoning as resolve_secret in manual_secrets.sh.
versions="$(scw secret version list "$secret_id" region="$REGION" -o json)"
enabled="$(printf '%s' "$versions" | jq '[.[] | select(.status == "enabled")] | length')"
total="$(printf '%s' "$versions" | jq 'length')"

# Nothing readable, so the write starts from {} and the keys already stored stop being
# synced. Warned rather than refused: the old versions survive and can be re-enabled, and
# refusing would need the Scaleway access this workflow exists to avoid needing.
warning=""
if [ "$enabled" -eq 0 ] && [ "$total" -gt 0 ]; then
warning="**Warning:** all $total version(s) of this secret are disabled, so the write"
warning="$warning starts from empty and every other key in it stops being synced."
fi

{
echo "secret-id=$secret_id"
echo "versions=$enabled"
echo "warning=$warning"
} >> "$GITHUB_OUTPUT"
echo "Secret $SECRET_NAME exists ($secret_id), $enabled of $total version(s) enabled."
113 changes: 113 additions & 0 deletions .github/actions/infra/secrets/write/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
name: Secret Write
description: Add or overwrite one key in a manual secret, keeping every other key it holds
inputs:
vendor:
description: The vendor to communicate with (scaleway)
required: true
secret-name:
description: Name of the secret to write, as resolved by the preflight action
required: true
project-name:
description: Name of the vendor project holding the secret
required: true
key:
description: Variable name to add or overwrite
required: true
value:
description: Secret value
required: true
description:
description: Description recorded on the new version
required: false
default: ''
# Vendor: Scaleway
scaleway-organization-id:
description: Scaleway organization ID
required: false
scaleway-project-id:
description: Scaleway project ID
required: false
scaleway-region:
description: Scaleway deployment region (such as nl-ams)
required: false
scaleway-access-key:
description: Scaleway API key ID
required: false
scaleway-secret-key:
description: Scaleway API key secret
required: false

outputs:
log:
description: Path to the run log, for a caller that wants to summarise it
value: ${{ steps.write.outputs.log }}

runs:
using: composite
steps:
- name: Validate inputs
shell: bash
env:
VENDOR: ${{ inputs.vendor }}
run: |
echo "Validating inputs for vendor: $VENDOR"
case "$VENDOR" in
scaleway)
;;
digitalocean)
echo "DigitalOcean has no secret manager, so there is nothing to write to"
exit 1
;;
azure)
echo "Azure Key Vault is not implemented, see Docs/infra-secret-write.md"
exit 1
;;
*)
echo "Unsupported vendor '$VENDOR'"
exit 1
;;
esac

- uses: wisemen-digital/devops-github-actions/.github/actions/infra/common/setup@v1
if: ${{ inputs.vendor == 'scaleway' }}
with:
vendor: ${{ inputs.vendor }}
scaleway-access-key: ${{ inputs.scaleway-access-key }}
scaleway-secret-key: ${{ inputs.scaleway-secret-key }}
scaleway-organization-id: ${{ inputs.scaleway-organization-id }}
scaleway-project-id: ${{ inputs.scaleway-project-id }}

# Project by name: the name is derivable from <app>-<env>, the id is not. Region, project and
# organization are all explicit, because a wrong one returns an empty list rather than an error.
- name: Write the new secret version
id: write
if: ${{ inputs.vendor == 'scaleway' }}
shell: bash
env:
SECRET_NAME: ${{ inputs.secret-name }}
SCW_SECRETS_PROJECT_NAME: ${{ inputs.project-name }}
SCW_SECRETS_ORGANIZATION_ID: ${{ inputs.scaleway-organization-id }}
SCW_SECRETS_REGION: ${{ inputs.scaleway-region }}
KEY: ${{ inputs.key }}
DESCRIPTION: ${{ inputs.description }}
SECRET_VALUE: ${{ inputs.value }}
run: |
set -euo pipefail
umask 077
export TMPDIR="$RUNNER_TEMP"

args=(set "$SECRET_NAME" - --yes)
if [ -n "$DESCRIPTION" ]; then
args+=(--description "$DESCRIPTION")
fi

echo "log=$RUNNER_TEMP/manual-secret.log" >> "$GITHUB_OUTPUT"

# $GITHUB_ACTION_PATH rather than github.action_path, because of a bug in GA
# Refs: https://github.com/actions/runner/issues/2185
#
# jq reads $ENV and the patch goes over stdin: no shell string, no argv, no file in this step.
jq -n '{($ENV.KEY): $ENV.SECRET_VALUE}' \
| "$GITHUB_ACTION_PATH/scripts/manual_secrets.sh" "${args[@]}" 2>&1 \
| tee "$RUNNER_TEMP/manual-secret.log"
Loading