Skip to content

feat: restore nested cross-resource references after Create and Update - #267

Merged
ack-prow[bot] merged 1 commit into
aws-controllers-k8s:mainfrom
gustavodiaz7722:feat/ensure-references
Sep 16, 2026
Merged

ack-prow[bot] merged 1 commit into
aws-controllers-k8s:mainfrom
gustavodiaz7722:feat/ensure-references

Conversation

@gustavodiaz7722

@gustavodiaz7722 gustavodiaz7722 commented Aug 27, 2026 •

Copy link
Copy Markdown
Member

Summary

A cross-resource reference (*Ref) is generated as a sibling of the concrete field it resolves into — spec.vpcConfig.subnetRefs next to spec.vpcConfig.subnetIDs. A resource manager builds its return value from an AWS API response, which has no concept of a reference, so rebuilding the containing struct drops every *Ref inside it.

That disables ClearResolvedReferences, which suppresses a resolved value only while it can still see the sibling:

if ko.Spec.VPCConfig != nil {
	if len(ko.Spec.VPCConfig.SubnetRefs) > 0 {   // false once the *Ref is gone
		ko.Spec.VPCConfig.SubnetIDs = nil
	}
}

So the spec patch deletes the declared *Ref and stores the resolved value in its place — what aws-controllers-k8s/community#2431 reports: a declared securityGroupRefs replaced by securityGroupIDs.

Reconciliation continues until the manifest is applied again, from Helm, Argo, Flux or kubectl apply. That apply restores the *Ref beside the now-stored value, and validateReferenceFields rejects the pair:

message: Reference resolution failed
reason: 'both resource reference wrapper and ID cannot be used together:
         VPCConfig.SubnetIDs,VPCConfig.SubnetRefs'

This PR adds an optional ReferenceEnsurer interface and invokes it on the object a resource manager hands back from Create and from Update, sourcing the references from the declared resource.

Fixes aws-controllers-k8s/community#2431, and addresses the struct-nested half of aws-controllers-k8s/community#2361. Pairs with aws-controllers-k8s/code-generator#738, which generates the method.

Backwards compatible

ReferenceEnsurer is deliberately separate from ReferenceManager and reached through a type assertion, so every controller generated before the method existed still satisfies AWSResourceManager and compiles unchanged. Those controllers take the existing path untouched and opt in by regenerating. TestReconcilerUpdate_WithoutEnsurerIsUnaffected pins that.

Where the restoration runs

On what a resource manager returns from Create and from Update — the two paths where the object about to be patched back was rebuilt from an API response while the patch base is still the resource the user declared. It runs before the error is inspected, since a manager may hand back a resource alongside a requeue error while an asynchronous operation is in flight and that object reaches the caller either way.

Not on AdoptionPolicy_Adopt. Under that policy the spec is populated from the observed AWS resource, so a declared spec is expected to be replaced rather than preserved, and a declared *Ref is replaced along with every other declared field. Restoring it would make the reference the one exception. TestReconcilerAdopt_DoesNotEnsureReferences asserts that.

AdoptionPolicy_AdoptOrCreate does keep a declared reference, for a structural reason rather than because the restoration runs: that branch marks the resource managed and adopted and requeues, and that patch's base is a DeepCopy of its own target, so nothing ever patches the spec with the declared resource as the base. TestReconcilerAdoptOrCreate_PreservesReferences asserts that too — both halves, because moving the restoration into patchResourceMetadataAndSpec would pick up the adopt branch automatically and silently change it.

Also not on the late-initialization patch, whose base is the AWS-observed object and carries no references — TestReconcilerUpdate_LateInitializeIsNotAffectedByEnsureReferences keeps that boundary asserted.

Why desired and not reconcileDesired

reconcileDesired is what gets handed to rm.Update, and a resource manager may mutate the object it is given: apigateway's ApiKey sdkUpdate assigns desired.ko.Spec.StageKeys straight from the UpdateApiKey response (sdk.go:331), and applyIgnoredFields merges observed values into it for a resource carrying the ignore-field-drift annotation. Only desired is a clean record of what the user declared, and it is also the base of the spec patch that follows, so the two agree by construction.

Create now gets a copy of desired

createResource passed desired itself to rm.Create. Generated sdkCreate only
deep-copies the resource it is given partway through -- a custom_implementation
returns before that point, and a sdk_create_pre_build_request hook runs before it
-- so either could mutate the object the user declared. desired is both the patch
base and the reference source, so it has to stay a clean record, and Create gets
desired.DeepCopy(). This mirrors updateResource, which already hands Update a
copy for the same reason.

The copy is taken after setResourceManaged and EnsureTags, so it carries the
finalizer and the controller tags.

No hook template in the fleet writes to desired.ko, and neither custom Create
implementation (elasticache's CustomCreateSnapshot, apigatewayv2's
customCreateApi) mutates its input -- the former deep-copies first, the latter
only reads -- so this is hardening rather than a fix.
TestReconcilerCreate_PassesCopyToCreate pins it.

What the generated method does

Detailed in aws-controllers-k8s/code-generator#738. Summarised here because it bounds this PR's blast radius:

  • top-level *Ref — nothing emitted; it cannot be lost, since every write path starts from a DeepCopy of the object it was handed.
  • reached through structs — only the reference field is assigned, so every concrete value the service reported stands. This is the whole of what is emitted, and it codifies a pattern eks/cluster, lambda/function and opensearchservice/domain already hand-maintain in sdk_*_post_set_output hooks.
  • reached through a list — nothing emitted; no fixed address to assign to, and no sound way to pair an observed element with a declared one. These behave exactly as they do today.

So 117 struct-nested references across 37 resources in 25 controllers change behaviour; the 315 top-level and 38 list-nested ones are untouched.

Testing

Eleven tests in pkg/runtime/reconciler_test.go:

Test Asserts
TestReconcilerCreate_EnsuresReferencesAfterCreate runs after Create, sourced from desired
TestReconcilerUpdate_EnsuresReferencesAfterUpdate the same after Update, and the returned object is what gets patched
TestReconcilerCreate_EnsuresReferencesOnCreateError still runs when Create returns an error
TestReconcilerUpdate_EnsuresReferencesOnUpdateError the same on the update error path
TestReconcilerAdopt_DoesNotEnsureReferences AdoptionPolicy_Adopt deliberately does not restore
TestReconcilerAdoptOrCreate_PreservesReferences AdoptionPolicy_AdoptOrCreate keeps a declared reference
TestReconcilerUpdate_LateInitializeIsNotAffectedByEnsureReferences never runs on the late-init patch
TestReconcilerUpdate_EnsureReferencesToleratesNilLatest a manager returning (nil, err) passes through instead of panicking
TestReconcilerUpdate_WithoutEnsurerIsUnaffected a manager not implementing the interface is untouched
TestReconcilerCreate_PassesCopyToCreate Create gets a copy, so desired stays a clean record
TestReconcilerUpdate_PassesCopyOfDesiredToUpdate the pre-existing counterpart on the update path

Full runtime suite passes.

Verified on a cluster, A/B against the unmodified controller. An ecs Service declaring networkConfiguration.awsVPCConfiguration.subnetRefs and .securityGroupRefs keeps both through create, update, re-apply and repeated resyncs, with no resolved IDs written to the spec. The same manifest under the unmodified controller loses both and stores the resolved subnets/securityGroups in their place; re-applying it then halts the resource on both resource reference wrapper and ID cannot be used together, which is the failure this prevents. Deletion completes cleanly. adopt-or-create against a pre-existing AWS resource keeps the references; strict adopt replaces them, as intended. Separately, an ecr Repository whose EncryptionConfiguration.KMSKey is both a reference and late_initialize reaches ACK.LateInitialized=True with the reference kept and no churn.

Not addressed

References reached through a list. No fixed address to assign to, and no sound way to pair an observed element with a declared one: an AWS response need not preserve request order, and 75 of the roughly 470 configured references resolve to a Spec.* path with no uniqueness guarantee. These behave exactly as they do today. This is the remaining half of #2361.

AdoptionPolicy_Adopt replaces a declared reference along with the rest of the declared spec. That is intentional, not a gap — see above.

The existing read-path hooks. lambda/function, eks/cluster and opensearchservice/domain restore these references by hand in sdk_read_one_post_set_output. Their create-path halves are subsumed. The read-path halves must stay — the AdoptionPolicy_Adopt and delete-path spec writes are not covered here.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@gustavodiaz7722

Copy link
Copy Markdown
Member Author

/retest

Comment thread pkg/types/reference_manager.go Outdated
Comment thread pkg/types/reference_manager.go Outdated
@knottnt

knottnt commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

/retest

ack-prow Bot pushed a commit that referenced this pull request Sep 8, 2026
Description of changes:

`unit-test` is currently failing on every PR, including unmodified `main`. It
fails in the `mocks` target, which `make test` depends on, so no test runs:

```
building mocks for pkg/types ...
internal error: package "k8s.io/apimachinery/pkg/apis/meta/v1" without types was imported from "github.com/aws-controllers-k8s/runtime/pkg/types"
make: *** [Makefile:24: mocks] Error 1
```

`scripts/install-mockery.sh` builds mockery from source with whatever Go the CI
image provides, and aws-controllers-k8s/test-infra#1084 moved `go_version` from
1.26.5 to 1.27.1 on 2026-09-02. mockery v2.53.3 pins
`golang.org/x/tools v0.30.0`, whose `go/packages` predates Go 1.27 and cannot
type-check its standard library, which is what produces the `without types`
loader error.

The timeline matches: the last `unit-test` run before the image bump passed
(PR #267, 2026-09-02T19:04Z), and runs after it fail.

v2.53.7 pins `golang.org/x/tools v0.49.0`, which handles Go 1.27.

The mocks are regenerated so the committed output matches the new version. The
only content changes are the generated-by header and import grouping — no mock
behaviour changes.

This also drops `mocks/pkg/types/resolved_reference_manager.go`. Its
`ResolvedReferenceManager` interface no longer exists in `pkg/types`, so mockery
does not generate it and nothing references it. It survived because the `mocks`
target overwrites files rather than starting from a clean directory, so a
`make clean-mocks && make mocks` cycle would otherwise always leave the tree
dirty.

### Testing

Reproduced and verified locally against both Go versions, building mockery from
source exactly as the job does:

| Go | mockery | `make mocks` |
| --- | --- | --- |
| 1.26.0 | v2.53.3 | passes |
| 1.27.1 | v2.53.3 | fails with the error above |
| 1.27.1 | v2.53.7 | passes |

Under Go 1.27.1 with this change, all six mock sets generate and the full
`go test ./...` passes. I also confirmed v2.53.3 fails on unmodified `main` under
Go 1.27.1, so this is not specific to any open PR.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@ack-prow ack-prow Bot added the approved label Sep 11, 2026
@knottnt

knottnt commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

/retest

1 similar comment
@gustavodiaz7722

Copy link
Copy Markdown
Member Author

/retest

@gustavodiaz7722 gustavodiaz7722 added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 14, 2026
gustavodiaz7722 added a commit to gustavodiaz7722/ack-ws-lambda-controller that referenced this pull request Sep 14, 2026
Demonstration only, not for merge. Shows what
aws-controllers-k8s/code-generator#738 emits in the context of a full
service controller.

Regenerated with no other change, so the diff is exactly the generated
EnsureReferences methods. go.mod is untouched: the method compiles
against the current runtime and stays inert until
aws-controllers-k8s/runtime#267 lands, which is what invokes it.

lambda covers all three reference shapes, so the per-shape behaviour is
visible in one controller:

  function       struct-nested   Code.S3BucketRef, VPCConfig.SecurityGroupRefs,
                                 VPCConfig.SubnetRefs -> emitted
  layer_version  struct-nested   emitted
  event_source_mapping           list-nested -> nothing emitted
  alias, code_signing_config, function_url_config, version
                                 top-level only -> nothing emitted

function is the shape reported in
aws-controllers-k8s/community#2431.
@gustavodiaz7722 gustavodiaz7722 removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 14, 2026
@gustavodiaz7722
gustavodiaz7722 force-pushed the feat/ensure-references branch 3 times, most recently from 109a805 to b0377f7 Compare September 14, 2026 23:04
gustavodiaz7722 added a commit to gustavodiaz7722/ack-ws-lambda-controller that referenced this pull request Sep 14, 2026
Demonstration only, not for merge. Shows what
aws-controllers-k8s/code-generator#738 emits in the context of a full
service controller.

Regenerated with no other change, so the diff is exactly the generated
EnsureReferences methods. go.mod is untouched: the method compiles
against the current runtime and stays inert until
aws-controllers-k8s/runtime#267 lands, which is what invokes it.

lambda covers all three reference shapes, so the per-shape behaviour is
visible in one controller:

  function       struct-nested   Code.S3BucketRef, VPCConfig.SecurityGroupRefs,
                                 VPCConfig.SubnetRefs -> emitted
  layer_version  struct-nested   emitted
  event_source_mapping           list-nested -> nothing emitted
  alias, code_signing_config, function_url_config, version
                                 top-level only -> nothing emitted

function is the shape reported in
aws-controllers-k8s/community#2431.
A cross-resource reference (*Ref) is generated as a sibling of the
concrete field it resolves into. A resource manager builds its return
value from an AWS API response, which has no concept of a reference, so
rebuilding the containing struct drops every *Ref inside it.

That disables ClearResolvedReferences, which suppresses a resolved value
only while the sibling *Ref is visible, so the spec patch deletes the
declared *Ref and stores the resolved value in its place. The next apply
of the manifest puts the *Ref back beside that value, a pair
validateReferenceFields rejects, stopping reconciliation.

Add an optional ReferenceEnsurer interface and invoke it on what a
resource manager returns from Create and from Update -- the two paths
where the object about to be patched back was rebuilt from an API
response while the patch base is still the resource the user declared.
The source is `desired`, not the copy handed to the manager, because a
manager may mutate what it is given: apigateway's ApiKey sdkUpdate
assigns desired.ko.Spec.StageKeys straight from the response.

The restoration runs before the error from Create or Update is inspected.
A resource manager may hand back a non-nil resource alongside a requeue
error while an asynchronous operation is in flight, and many do; that
object reaches the caller either way.

Deliberately not applied to AdoptionPolicy_Adopt. Under that policy the
spec is populated from the observed AWS resource, so a declared spec is
expected to be replaced rather than preserved, and a declared *Ref is
replaced along with every other declared field. Restoring it would make
the reference the one exception.
AdoptionPolicy_AdoptOrCreate does keep a declared reference, for a
structural reason rather than because the restoration runs: that branch
marks the resource managed and adopted and requeues, and that patch's
base is a DeepCopy of its own target, so nothing ever patches the spec
with the declared resource as the base. Both halves are asserted, because
moving the restoration into patchResourceMetadataAndSpec would pick up the
adopt branch automatically and silently change it.

Also not applied to the late-initialization patch, whose base is the
AWS-observed object and carries no references, nor in deleteResource,
where the CR is removed immediately afterwards and the write is never
observed.

Independently of the above, Create is now handed a copy of `desired`
rather than `desired` itself. Generated sdkCreate only deep-copies the
resource it is given partway through -- a custom_implementation returns
before that point and a sdk_create_pre_build_request hook runs before it
-- so either could mutate what the user declared, and `desired` is both
the patch base and the reference source. Update already took a copy for
the same reason. The copy is taken after setResourceManaged and
EnsureTags so it carries the finalizer and the controller tags.

Which shapes are covered is a property of the generated method rather
than of this interface.

Pairs with aws-controllers-k8s/code-generator#738, which generates the
method.

Issue aws-controllers-k8s/community#2431
Issue aws-controllers-k8s/community#2361
gustavodiaz7722 added a commit to gustavodiaz7722/ack-ws-lambda-controller that referenced this pull request Sep 14, 2026
Demonstration only, not for merge. Shows what
aws-controllers-k8s/code-generator#738 emits in the context of a full
service controller.

Regenerated with no other change, so the diff is exactly the generated
EnsureReferences methods. go.mod is untouched: the method compiles
against the current runtime and stays inert until
aws-controllers-k8s/runtime#267 lands, which is what invokes it.

lambda covers all three reference shapes, so the per-shape behaviour is
visible in one controller:

  function       struct-nested   Code.S3BucketRef, VPCConfig.SecurityGroupRefs,
                                 VPCConfig.SubnetRefs -> emitted
  layer_version  struct-nested   emitted
  event_source_mapping           list-nested -> nothing emitted
  alias, code_signing_config, function_url_config, version
                                 top-level only -> nothing emitted

function is the shape reported in
aws-controllers-k8s/community#2431.

@knottnt knottnt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gustavodiaz7722 Changes to the primary create/update paths look good. However, there may be a remaining edge case in the pre-delete-sync(prd) path where the prd update succeeds and the reference is overwritten, but for whatever reason the update fails or is delayed. I believe the most common path that edge case could be hit is a resource with an async delete that use a requeue error to inform the controller that it needs to wait for the resource to disappear or hit a terminal DELETED state.

@gustavodiaz7722
gustavodiaz7722 force-pushed the feat/ensure-references branch 4 times, most recently from ce2c1eb to ab76824 Compare September 15, 2026 20:53
@ack-prow

ack-prow Bot commented Sep 15, 2026

Copy link
Copy Markdown

@gustavodiaz7722: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
s3-controller-test 7ec6f14 link true /test s3-controller-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@knottnt knottnt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ack-prow ack-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 16, 2026
@ack-prow

ack-prow Bot commented Sep 16, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gustavodiaz7722, knottnt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ack-prow
ack-prow Bot merged commit 26e42fd into aws-controllers-k8s:main Sep 16, 2026
9 checks passed
gustavodiaz7722 added a commit to gustavodiaz7722/ack-ws-code-generator that referenced this pull request Sep 16, 2026
Picks up aws-controllers-k8s/runtime#267, which restores struct-nested
cross-resource references after Create and Update and adds the optional
ReferenceEnsurer interface this PR generates an implementation of.

Bundled here rather than as a follow-up because the two halves are only
useful together. auto-generate-controllers resolves the runtime version from
this module's go.mod (cd/auto-generate/auto-generate-controllers.sh:91)
rather than from the latest runtime release, so merging the generator change
while this pin still said v0.63.0 would open fleet-wide PRs adding a
generated EnsureReferences method against a runtime that never calls it.

The bump is go.mod and go.sum only, with no generated-output changes.
`go build ./...` and `go test ./pkg/generate/...` pass.
knottnt pushed a commit to aws-controllers-k8s/code-generator that referenced this pull request Sep 17, 2026
…erences (#738)

* feat: generate EnsureReferences for nested refs

A cross-resource reference (*Ref) is generated as a sibling of the
concrete field it resolves into. A resource manager builds its return
value from an AWS API response, which has no concept of a reference, so
rebuilding the containing struct drops every *Ref inside it.

That disables ClearResolvedReferences, which suppresses a resolved value
only while the sibling *Ref is visible, so the spec patch deletes the
declared *Ref and stores the resolved value in its place. The next apply
of the manifest puts the *Ref back beside that value, a pair
validateReferenceFields rejects, stopping reconciliation.

Generate an EnsureReferences method that restores such a reference from
the declared resource. Only a reference reached through structs is
emitted, at its one fixed address, so every value the service reported
stands. A top-level *Ref is skipped because it cannot be lost. One
reached through a list is also skipped and behaves as it does today:
neither position nor resolved value is a sound key for pairing an
observed element with a declared one.

This codifies a pattern eks/cluster, lambda/function and
opensearchservice/domain already hand-maintain in set-output hooks, with
stricter guards.

Requires the runtime's optional ReferenceEnsurer interface, which invokes
the method after Create and after Update. Controllers generated before
the method existed are unaffected and opt in by regenerating.

Issue aws-controllers-k8s/community#2431
Issue aws-controllers-k8s/community#2361

* chore: update ACK runtime to v0.64.0

Picks up aws-controllers-k8s/runtime#267, which restores struct-nested
cross-resource references after Create and Update and adds the optional
ReferenceEnsurer interface this PR generates an implementation of.

Bundled here rather than as a follow-up because the two halves are only
useful together. auto-generate-controllers resolves the runtime version from
this module's go.mod (cd/auto-generate/auto-generate-controllers.sh:91)
rather than from the latest runtime release, so merging the generator change
while this pin still said v0.63.0 would open fleet-wide PRs adding a
generated EnsureReferences method against a runtime that never calls it.

The bump is go.mod and go.sum only, with no generated-output changes.
`go build ./...` and `go test ./pkg/generate/...` pass.

* refactor: share a container's guard across the references inside it

EnsureReferences emitted the full guard chain and materialisation once per
reference, so two references in one container restated every enclosing
guard. Order the references by enclosing container and emit each container's
guard once, nesting a deeper container's guard inside its parent's.

ecs/CapacityProvider, the deepest case in the fleet, goes from 87 to 75
lines and from four separate guard chains to one tree; ecs/Service 42 to 38
and lambda/Function 46 to 40. The generated behaviour is unchanged.

The materialisation stays per-reference. It has to sit inside the guard
establishing that the source holds a reference to put in the container --
otherwise an empty container reaches the spec patch -- and sharing it would
need the source-side guards OR'd together, which is the concatenation the
nesting exists to avoid. It is a nil-check either way, so repeating it costs
nothing.

Also narrow the line-length assertion in
Test_EnsureReferences_GuardsAreNestedNotConcatenated to the guard lines,
which is what nesting controls. It read as a property of the whole output,
but an assignment line is as long as its two field paths and gofmt wraps
neither form: ecs/CapacityProvider's deepest reference produces a
210-character one, as the hand-written hooks this replaces already do.

Add an eks Cluster fixture carrying both shapes -- two references in one
container, and a container nested inside one that holds a reference of its
own -- and a test pinning that each guard is emitted once.

Regenerated ecs-, lambda- and eks-controller: all build, all gofmt-clean,
and the diffs are the same restorations under fewer guards.

* docs: correct referenceAncestors' account of the map rejection

The comment claimed the rejection was "kept in one place", but
iterReferenceValues makes the same check while walking the path itself, so
there are two. Say what is actually true: this keeps it in one place for
callers that do not walk the path.

Comment only. Sharing the check would mean extracting the lookup-and-reject
core into a third helper, since referenceAncestors stops at the first list
while iterReferenceValues has to emit a range loop and keep descending -- so
the former cannot serve as the latter's walker. That touches the function
behind ReferenceFieldsValidation, ResolveReferencesForField and
ClearResolvedReferencesForField, and belongs in its own change rather than
here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The ACK Lambda Controller modifies the object spec

2 participants