Repository navigation
Wait for the mission architecture deploy slot - #21
Merged
Merged
Conversation
The architecture project runs one build at a time and CodeBuild rejects a StartBuild over that limit instead of queueing it. A base-image rebuild starts every Lambda build for a mission at once, so all but the first hand-off were dropped and those Lambdas kept their old images. Retry throttled StartBuild calls every 30 seconds, for up to 55 minutes from the build's start, and fail immediately on any other error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The loop can issue another deployment request after the 55-minute deadline.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Adds resilient deployment hand-off when the mission architecture project is busy.
Changes:
- Retries retryable CodeBuild limit/throttling failures every 30 seconds.
- Fails immediately for non-retryable errors.
- Adds a build-relative 55-minute retry deadline.
| File | Description |
|---|---|
buildspec.yml |
Adds retry and timeout handling for architecture deployment startup. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The deadline was checked only before the fixed 30-second sleep, so a busy response just before the 55-minute cutoff still led to one more StartBuild after it. Cap each wait at the time remaining and check the deadline again before every retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
After a Lambda image is pushed, this build calls
StartBuildon the mission architecture project (build_<mission>_sdc_aws_pipeline_architecture) to deploy it. Since the deployment-root apply on 2026-09-28/29, that project allows one build at a time, and CodeBuild rejects aStartBuildover the limit instead of queueing it.A mission base-image rebuild starts all of that mission's Lambda builds at once (PADRE: sorting, processing, artifacts, concating). They finish within a minute or so of each other, so today only the first one would deploy. The others would push their image to ECR and then fail at the hand-off, leaving the Lambda on its old image.
Running the architecture project in parallel instead isn't a good fix. Each deploy plans, then applies that saved plan, and Terraform refuses a saved plan once another deploy has changed the state in between ("Saved plan is stale"). So the hand-off waits for the slot instead. This is the same pattern
sdc_aws_architecture#41 used for the dependency triggers.What
StartBuildevery 30 seconds while the architecture project is busy.AccountLimitExceededException,ThrottlingException, …). Anything else, such as a denied call or a missing project, fails immediately.CODEBUILD_START_TIME), inside the project's 60-minute build timeout set insdc_aws_architecture. One deploy takes about 3–4 minutes, so a full mission fan-out, development and production together, fits comfortably.Verification
bash -n.set -euo pipefailagainst a stubbedawsCLI:AccessDeniedExceptionThe same change is going into sorting, processing, artifacts and concating.
🤖 Generated with Claude Code