Skip to content

Wait for the mission architecture deploy slot - #21

Merged
jamesray merged 2 commits into
mainfrom
jamesray/wait-for-architecture-slot
Oct 7, 2026
Merged

jamesray merged 2 commits into
mainfrom
jamesray/wait-for-architecture-slot

Conversation

@jamesray

@jamesray jamesray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Why

After a Lambda image is pushed, this build calls StartBuild on the mission architecture project (build_<mission>_sdc_aws_pipeline_architecture) to deploy it. Since the deployment-root apply on 2026-09-28/29, that project allows one build at a time, and CodeBuild rejects a StartBuild over the limit instead of queueing it.

A mission base-image rebuild starts all of that mission's Lambda builds at once (PADRE: sorting, processing, artifacts, concating). They finish within a minute or so of each other, so today only the first one would deploy. The others would push their image to ECR and then fail at the hand-off, leaving the Lambda on its old image.

Running the architecture project in parallel instead isn't a good fix. Each deploy plans, then applies that saved plan, and Terraform refuses a saved plan once another deploy has changed the state in between ("Saved plan is stale"). So the hand-off waits for the slot instead. This is the same pattern sdc_aws_architecture #41 used for the dependency triggers.

What

  • Retry StartBuild every 30 seconds while the architecture project is busy.
  • Retry only throttling and limit errors (AccountLimitExceededException, ThrottlingException, …). Anything else, such as a denied call or a missing project, fails immediately.
  • Stop retrying 55 minutes after this build started (CODEBUILD_START_TIME), inside the project's 60-minute build timeout set in sdc_aws_architecture. One deploy takes about 3–4 minutes, so a full mission fan-out, development and production together, fits comfortably.
  • The architecture project's deploy logic and guard are unchanged.

Verification

  • The buildspec parses as YAML, and every command block passes bash -n.
  • I ran the new hand-off under set -euo pipefail against a stubbed aws CLI:
Case Result
Busy twice, then free Retried twice, started the deploy, exit 0
AccessDeniedException One call, no retry, exit 1
Still busy 55 minutes after the build started Gave up with a clear message, exit 1

The same change is going into sorting, processing, artifacts and concating.

🤖 Generated with Claude Code

The architecture project runs one build at a time and CodeBuild rejects a
StartBuild over that limit instead of queueing it. A base-image rebuild starts
every Lambda build for a mission at once, so all but the first hand-off were
dropped and those Lambdas kept their old images. Retry throttled StartBuild
calls every 30 seconds, for up to 55 minutes from the build's start, and fail
immediately on any other error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 08:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The loop can issue another deployment request after the 55-minute deadline.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds resilient deployment hand-off when the mission architecture project is busy.

Changes:

  • Retries retryable CodeBuild limit/throttling failures every 30 seconds.
  • Fails immediately for non-retryable errors.
  • Adds a build-relative 55-minute retry deadline.
File Description
buildspec.yml Adds retry and timeout handling for architecture deployment startup.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread buildspec.yml Resolved
The deadline was checked only before the fixed 30-second sleep, so a busy
response just before the 55-minute cutoff still led to one more StartBuild
after it. Cap each wait at the time remaining and check the deadline again
before every retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jamesray
jamesray merged commit 92f783b into main Oct 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants