feat: restore nested cross-resource references after Create and Update - #267
Conversation
c279560 to
c2f3d04
Compare
|
/retest |
c2f3d04 to
e4cb381
Compare
90115f5 to
3e40e33
Compare
|
/retest |
Description of changes: `unit-test` is currently failing on every PR, including unmodified `main`. It fails in the `mocks` target, which `make test` depends on, so no test runs: ``` building mocks for pkg/types ... internal error: package "k8s.io/apimachinery/pkg/apis/meta/v1" without types was imported from "github.com/aws-controllers-k8s/runtime/pkg/types" make: *** [Makefile:24: mocks] Error 1 ``` `scripts/install-mockery.sh` builds mockery from source with whatever Go the CI image provides, and aws-controllers-k8s/test-infra#1084 moved `go_version` from 1.26.5 to 1.27.1 on 2026-09-02. mockery v2.53.3 pins `golang.org/x/tools v0.30.0`, whose `go/packages` predates Go 1.27 and cannot type-check its standard library, which is what produces the `without types` loader error. The timeline matches: the last `unit-test` run before the image bump passed (PR #267, 2026-09-02T19:04Z), and runs after it fail. v2.53.7 pins `golang.org/x/tools v0.49.0`, which handles Go 1.27. The mocks are regenerated so the committed output matches the new version. The only content changes are the generated-by header and import grouping — no mock behaviour changes. This also drops `mocks/pkg/types/resolved_reference_manager.go`. Its `ResolvedReferenceManager` interface no longer exists in `pkg/types`, so mockery does not generate it and nothing references it. It survived because the `mocks` target overwrites files rather than starting from a clean directory, so a `make clean-mocks && make mocks` cycle would otherwise always leave the tree dirty. ### Testing Reproduced and verified locally against both Go versions, building mockery from source exactly as the job does: | Go | mockery | `make mocks` | | --- | --- | --- | | 1.26.0 | v2.53.3 | passes | | 1.27.1 | v2.53.3 | fails with the error above | | 1.27.1 | v2.53.7 | passes | Under Go 1.27.1 with this change, all six mock sets generate and the full `go test ./...` passes. I also confirmed v2.53.3 fails on unmodified `main` under Go 1.27.1, so this is not specific to any open PR. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
/retest |
1 similar comment
|
/retest |
3e40e33 to
94da9ce
Compare
Demonstration only, not for merge. Shows what aws-controllers-k8s/code-generator#738 emits in the context of a full service controller. Regenerated with no other change, so the diff is exactly the generated EnsureReferences methods. go.mod is untouched: the method compiles against the current runtime and stays inert until aws-controllers-k8s/runtime#267 lands, which is what invokes it. lambda covers all three reference shapes, so the per-shape behaviour is visible in one controller: function struct-nested Code.S3BucketRef, VPCConfig.SecurityGroupRefs, VPCConfig.SubnetRefs -> emitted layer_version struct-nested emitted event_source_mapping list-nested -> nothing emitted alias, code_signing_config, function_url_config, version top-level only -> nothing emitted function is the shape reported in aws-controllers-k8s/community#2431.
109a805 to
b0377f7
Compare
Demonstration only, not for merge. Shows what aws-controllers-k8s/code-generator#738 emits in the context of a full service controller. Regenerated with no other change, so the diff is exactly the generated EnsureReferences methods. go.mod is untouched: the method compiles against the current runtime and stays inert until aws-controllers-k8s/runtime#267 lands, which is what invokes it. lambda covers all three reference shapes, so the per-shape behaviour is visible in one controller: function struct-nested Code.S3BucketRef, VPCConfig.SecurityGroupRefs, VPCConfig.SubnetRefs -> emitted layer_version struct-nested emitted event_source_mapping list-nested -> nothing emitted alias, code_signing_config, function_url_config, version top-level only -> nothing emitted function is the shape reported in aws-controllers-k8s/community#2431.
b0377f7 to
21de5c2
Compare
A cross-resource reference (*Ref) is generated as a sibling of the concrete field it resolves into. A resource manager builds its return value from an AWS API response, which has no concept of a reference, so rebuilding the containing struct drops every *Ref inside it. That disables ClearResolvedReferences, which suppresses a resolved value only while the sibling *Ref is visible, so the spec patch deletes the declared *Ref and stores the resolved value in its place. The next apply of the manifest puts the *Ref back beside that value, a pair validateReferenceFields rejects, stopping reconciliation. Add an optional ReferenceEnsurer interface and invoke it on what a resource manager returns from Create and from Update -- the two paths where the object about to be patched back was rebuilt from an API response while the patch base is still the resource the user declared. The source is `desired`, not the copy handed to the manager, because a manager may mutate what it is given: apigateway's ApiKey sdkUpdate assigns desired.ko.Spec.StageKeys straight from the response. The restoration runs before the error from Create or Update is inspected. A resource manager may hand back a non-nil resource alongside a requeue error while an asynchronous operation is in flight, and many do; that object reaches the caller either way. Deliberately not applied to AdoptionPolicy_Adopt. Under that policy the spec is populated from the observed AWS resource, so a declared spec is expected to be replaced rather than preserved, and a declared *Ref is replaced along with every other declared field. Restoring it would make the reference the one exception. AdoptionPolicy_AdoptOrCreate does keep a declared reference, for a structural reason rather than because the restoration runs: that branch marks the resource managed and adopted and requeues, and that patch's base is a DeepCopy of its own target, so nothing ever patches the spec with the declared resource as the base. Both halves are asserted, because moving the restoration into patchResourceMetadataAndSpec would pick up the adopt branch automatically and silently change it. Also not applied to the late-initialization patch, whose base is the AWS-observed object and carries no references, nor in deleteResource, where the CR is removed immediately afterwards and the write is never observed. Independently of the above, Create is now handed a copy of `desired` rather than `desired` itself. Generated sdkCreate only deep-copies the resource it is given partway through -- a custom_implementation returns before that point and a sdk_create_pre_build_request hook runs before it -- so either could mutate what the user declared, and `desired` is both the patch base and the reference source. Update already took a copy for the same reason. The copy is taken after setResourceManaged and EnsureTags so it carries the finalizer and the controller tags. Which shapes are covered is a property of the generated method rather than of this interface. Pairs with aws-controllers-k8s/code-generator#738, which generates the method. Issue aws-controllers-k8s/community#2431 Issue aws-controllers-k8s/community#2361
21de5c2 to
ab76824
Compare
Demonstration only, not for merge. Shows what aws-controllers-k8s/code-generator#738 emits in the context of a full service controller. Regenerated with no other change, so the diff is exactly the generated EnsureReferences methods. go.mod is untouched: the method compiles against the current runtime and stays inert until aws-controllers-k8s/runtime#267 lands, which is what invokes it. lambda covers all three reference shapes, so the per-shape behaviour is visible in one controller: function struct-nested Code.S3BucketRef, VPCConfig.SecurityGroupRefs, VPCConfig.SubnetRefs -> emitted layer_version struct-nested emitted event_source_mapping list-nested -> nothing emitted alias, code_signing_config, function_url_config, version top-level only -> nothing emitted function is the shape reported in aws-controllers-k8s/community#2431.
knottnt
left a comment
There was a problem hiding this comment.
@gustavodiaz7722 Changes to the primary create/update paths look good. However, there may be a remaining edge case in the pre-delete-sync(prd) path where the prd update succeeds and the reference is overwritten, but for whatever reason the update fails or is delayed. I believe the most common path that edge case could be hit is a resource with an async delete that use a requeue error to inform the controller that it needs to wait for the resource to disappear or hit a terminal DELETED state.
ce2c1eb to
ab76824
Compare
|
@gustavodiaz7722: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
7ec6f14 to
ab76824
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gustavodiaz7722, knottnt The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Picks up aws-controllers-k8s/runtime#267, which restores struct-nested cross-resource references after Create and Update and adds the optional ReferenceEnsurer interface this PR generates an implementation of. Bundled here rather than as a follow-up because the two halves are only useful together. auto-generate-controllers resolves the runtime version from this module's go.mod (cd/auto-generate/auto-generate-controllers.sh:91) rather than from the latest runtime release, so merging the generator change while this pin still said v0.63.0 would open fleet-wide PRs adding a generated EnsureReferences method against a runtime that never calls it. The bump is go.mod and go.sum only, with no generated-output changes. `go build ./...` and `go test ./pkg/generate/...` pass.
…erences (#738) * feat: generate EnsureReferences for nested refs A cross-resource reference (*Ref) is generated as a sibling of the concrete field it resolves into. A resource manager builds its return value from an AWS API response, which has no concept of a reference, so rebuilding the containing struct drops every *Ref inside it. That disables ClearResolvedReferences, which suppresses a resolved value only while the sibling *Ref is visible, so the spec patch deletes the declared *Ref and stores the resolved value in its place. The next apply of the manifest puts the *Ref back beside that value, a pair validateReferenceFields rejects, stopping reconciliation. Generate an EnsureReferences method that restores such a reference from the declared resource. Only a reference reached through structs is emitted, at its one fixed address, so every value the service reported stands. A top-level *Ref is skipped because it cannot be lost. One reached through a list is also skipped and behaves as it does today: neither position nor resolved value is a sound key for pairing an observed element with a declared one. This codifies a pattern eks/cluster, lambda/function and opensearchservice/domain already hand-maintain in set-output hooks, with stricter guards. Requires the runtime's optional ReferenceEnsurer interface, which invokes the method after Create and after Update. Controllers generated before the method existed are unaffected and opt in by regenerating. Issue aws-controllers-k8s/community#2431 Issue aws-controllers-k8s/community#2361 * chore: update ACK runtime to v0.64.0 Picks up aws-controllers-k8s/runtime#267, which restores struct-nested cross-resource references after Create and Update and adds the optional ReferenceEnsurer interface this PR generates an implementation of. Bundled here rather than as a follow-up because the two halves are only useful together. auto-generate-controllers resolves the runtime version from this module's go.mod (cd/auto-generate/auto-generate-controllers.sh:91) rather than from the latest runtime release, so merging the generator change while this pin still said v0.63.0 would open fleet-wide PRs adding a generated EnsureReferences method against a runtime that never calls it. The bump is go.mod and go.sum only, with no generated-output changes. `go build ./...` and `go test ./pkg/generate/...` pass. * refactor: share a container's guard across the references inside it EnsureReferences emitted the full guard chain and materialisation once per reference, so two references in one container restated every enclosing guard. Order the references by enclosing container and emit each container's guard once, nesting a deeper container's guard inside its parent's. ecs/CapacityProvider, the deepest case in the fleet, goes from 87 to 75 lines and from four separate guard chains to one tree; ecs/Service 42 to 38 and lambda/Function 46 to 40. The generated behaviour is unchanged. The materialisation stays per-reference. It has to sit inside the guard establishing that the source holds a reference to put in the container -- otherwise an empty container reaches the spec patch -- and sharing it would need the source-side guards OR'd together, which is the concatenation the nesting exists to avoid. It is a nil-check either way, so repeating it costs nothing. Also narrow the line-length assertion in Test_EnsureReferences_GuardsAreNestedNotConcatenated to the guard lines, which is what nesting controls. It read as a property of the whole output, but an assignment line is as long as its two field paths and gofmt wraps neither form: ecs/CapacityProvider's deepest reference produces a 210-character one, as the hand-written hooks this replaces already do. Add an eks Cluster fixture carrying both shapes -- two references in one container, and a container nested inside one that holds a reference of its own -- and a test pinning that each guard is emitted once. Regenerated ecs-, lambda- and eks-controller: all build, all gofmt-clean, and the diffs are the same restorations under fewer guards. * docs: correct referenceAncestors' account of the map rejection The comment claimed the rejection was "kept in one place", but iterReferenceValues makes the same check while walking the path itself, so there are two. Say what is actually true: this keeps it in one place for callers that do not walk the path. Comment only. Sharing the check would mean extracting the lookup-and-reject core into a third helper, since referenceAncestors stops at the first list while iterReferenceValues has to emit a range loop and keep descending -- so the former cannot serve as the latter's walker. That touches the function behind ReferenceFieldsValidation, ResolveReferencesForField and ClearResolvedReferencesForField, and belongs in its own change rather than here.
Summary
A cross-resource reference (
*Ref) is generated as a sibling of the concrete field it resolves into —spec.vpcConfig.subnetRefsnext tospec.vpcConfig.subnetIDs. A resource manager builds its return value from an AWS API response, which has no concept of a reference, so rebuilding the containing struct drops every*Refinside it.That disables
ClearResolvedReferences, which suppresses a resolved value only while it can still see the sibling:So the spec patch deletes the declared
*Refand stores the resolved value in its place — what aws-controllers-k8s/community#2431 reports: a declaredsecurityGroupRefsreplaced bysecurityGroupIDs.Reconciliation continues until the manifest is applied again, from Helm, Argo, Flux or
kubectl apply. That apply restores the*Refbeside the now-stored value, andvalidateReferenceFieldsrejects the pair:This PR adds an optional
ReferenceEnsurerinterface and invokes it on the object a resource manager hands back fromCreateand fromUpdate, sourcing the references from the declared resource.Fixes aws-controllers-k8s/community#2431, and addresses the struct-nested half of aws-controllers-k8s/community#2361. Pairs with aws-controllers-k8s/code-generator#738, which generates the method.
Backwards compatible
ReferenceEnsureris deliberately separate fromReferenceManagerand reached through a type assertion, so every controller generated before the method existed still satisfiesAWSResourceManagerand compiles unchanged. Those controllers take the existing path untouched and opt in by regenerating.TestReconcilerUpdate_WithoutEnsurerIsUnaffectedpins that.Where the restoration runs
On what a resource manager returns from
Createand fromUpdate— the two paths where the object about to be patched back was rebuilt from an API response while the patch base is still the resource the user declared. It runs before the error is inspected, since a manager may hand back a resource alongside a requeue error while an asynchronous operation is in flight and that object reaches the caller either way.Not on
AdoptionPolicy_Adopt. Under that policy the spec is populated from the observed AWS resource, so a declared spec is expected to be replaced rather than preserved, and a declared*Refis replaced along with every other declared field. Restoring it would make the reference the one exception.TestReconcilerAdopt_DoesNotEnsureReferencesasserts that.AdoptionPolicy_AdoptOrCreatedoes keep a declared reference, for a structural reason rather than because the restoration runs: that branch marks the resource managed and adopted and requeues, and that patch's base is aDeepCopyof its own target, so nothing ever patches the spec with the declared resource as the base.TestReconcilerAdoptOrCreate_PreservesReferencesasserts that too — both halves, because moving the restoration intopatchResourceMetadataAndSpecwould pick up the adopt branch automatically and silently change it.Also not on the late-initialization patch, whose base is the AWS-observed object and carries no references —
TestReconcilerUpdate_LateInitializeIsNotAffectedByEnsureReferenceskeeps that boundary asserted.Why
desiredand notreconcileDesiredreconcileDesiredis what gets handed torm.Update, and a resource manager may mutate the object it is given:apigateway'sApiKeysdkUpdateassignsdesired.ko.Spec.StageKeysstraight from theUpdateApiKeyresponse (sdk.go:331), andapplyIgnoredFieldsmerges observed values into it for a resource carrying the ignore-field-drift annotation. Onlydesiredis a clean record of what the user declared, and it is also the base of the spec patch that follows, so the two agree by construction.Create now gets a copy of
desiredcreateResourcepasseddesireditself torm.Create. GeneratedsdkCreateonlydeep-copies the resource it is given partway through -- a
custom_implementationreturns before that point, and a
sdk_create_pre_build_requesthook runs before it-- so either could mutate the object the user declared.
desiredis both the patchbase and the reference source, so it has to stay a clean record, and
Creategetsdesired.DeepCopy(). This mirrorsupdateResource, which already handsUpdateacopy for the same reason.
The copy is taken after
setResourceManagedandEnsureTags, so it carries thefinalizer and the controller tags.
No hook template in the fleet writes to
desired.ko, and neither custom Createimplementation (
elasticache'sCustomCreateSnapshot,apigatewayv2'scustomCreateApi) mutates its input -- the former deep-copies first, the latteronly reads -- so this is hardening rather than a fix.
TestReconcilerCreate_PassesCopyToCreatepins it.What the generated method does
Detailed in aws-controllers-k8s/code-generator#738. Summarised here because it bounds this PR's blast radius:
*Ref— nothing emitted; it cannot be lost, since every write path starts from aDeepCopyof the object it was handed.eks/cluster,lambda/functionandopensearchservice/domainalready hand-maintain insdk_*_post_set_outputhooks.So 117 struct-nested references across 37 resources in 25 controllers change behaviour; the 315 top-level and 38 list-nested ones are untouched.
Testing
Eleven tests in
pkg/runtime/reconciler_test.go:TestReconcilerCreate_EnsuresReferencesAfterCreateCreate, sourced fromdesiredTestReconcilerUpdate_EnsuresReferencesAfterUpdateUpdate, and the returned object is what gets patchedTestReconcilerCreate_EnsuresReferencesOnCreateErrorCreatereturns an errorTestReconcilerUpdate_EnsuresReferencesOnUpdateErrorTestReconcilerAdopt_DoesNotEnsureReferencesAdoptionPolicy_Adoptdeliberately does not restoreTestReconcilerAdoptOrCreate_PreservesReferencesAdoptionPolicy_AdoptOrCreatekeeps a declared referenceTestReconcilerUpdate_LateInitializeIsNotAffectedByEnsureReferencesTestReconcilerUpdate_EnsureReferencesToleratesNilLatest(nil, err)passes through instead of panickingTestReconcilerUpdate_WithoutEnsurerIsUnaffectedTestReconcilerCreate_PassesCopyToCreateCreategets a copy, sodesiredstays a clean recordTestReconcilerUpdate_PassesCopyOfDesiredToUpdateFull runtime suite passes.
Verified on a cluster, A/B against the unmodified controller. An
ecsServicedeclaringnetworkConfiguration.awsVPCConfiguration.subnetRefsand.securityGroupRefskeeps both through create, update, re-apply and repeated resyncs, with no resolved IDs written to the spec. The same manifest under the unmodified controller loses both and stores the resolvedsubnets/securityGroupsin their place; re-applying it then halts the resource onboth resource reference wrapper and ID cannot be used together, which is the failure this prevents. Deletion completes cleanly.adopt-or-createagainst a pre-existing AWS resource keeps the references; strictadoptreplaces them, as intended. Separately, anecrRepositorywhoseEncryptionConfiguration.KMSKeyis both a reference andlate_initializereachesACK.LateInitialized=Truewith the reference kept and no churn.Not addressed
References reached through a list. No fixed address to assign to, and no sound way to pair an observed element with a declared one: an AWS response need not preserve request order, and 75 of the roughly 470 configured references resolve to a
Spec.*path with no uniqueness guarantee. These behave exactly as they do today. This is the remaining half of #2361.AdoptionPolicy_Adoptreplaces a declared reference along with the rest of the declared spec. That is intentional, not a gap — see above.The existing read-path hooks.
lambda/function,eks/clusterandopensearchservice/domainrestore these references by hand insdk_read_one_post_set_output. Their create-path halves are subsumed. The read-path halves must stay — theAdoptionPolicy_Adoptand delete-path spec writes are not covered here.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.