Skip to content

ci: trial the agentic Open Source fix from prodsec-orb [AG-387] - #7054

Open
ShawkyZ wants to merge 1 commit into
mainfrom
feat/AG-387-trial-agentic-fix
Open

ci: trial the agentic Open Source fix from prodsec-orb [AG-387]#7054
ShawkyZ wants to merge 1 commit into
mainfrom
feat/AG-387-trial-agentic-fix

Conversation

@ShawkyZ

@ShawkyZ ShawkyZ commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What this does

Points the prodsec orb at the dev version built from snyk/prodsec-orb#166 so this repository can exercise the agentic Open Source fix before that PR is released, and switches it on for code-analysis.

Change Effect
prodsec: snyk/prodsec-orb@1@dev:ee7f5290 uses the branch build of #166
open-source-scan: highmedium the gate reports more
open-source-agentic-fix-enabled: true turns the feature on
prodsec-orb-runtime added to code-analysis contexts supplies the credentials the fix needs

When the Enhanced Gate blocks an Open Source scan the job still fails exactly as it does today. In addition, snyk fix --agentic runs at the same severity threshold the gate uses, commits to <branch>+remy_fix, and opens a pull request against the branch that failed. The fix step always exits 0, so it cannot turn a passing build red or mask a failure.

Please read before approving

1. medium is a behavioural change on its own, and it compounds here. The gate can now block on medium-severity findings that previously passed, so a build may go red for that reason alone — nothing to do with the fix. This repository also sets open-source-block-ignore-sla: true, so the gate already ignores remediation SLA. Together those widen what blocks considerably on a dependency tree this size.

The most likely outcome of the first blocked build is therefore not a tidy pull request: the fix has hard safety caps (50 files / 5000 lines / 1 MB patch) and will refuse to push anything that exceeds them, logging a skip instead. If you would rather the first trial be informative than broad, reverting this line to high is reasonable and I am happy to do it.

2. A github-cli/install step now runs on every build, not only blocked ones, because a CircleCI step cannot be made conditional on a runtime value. It is an ordinary step: if that install fails it fails the job even when the gate passed. Note this repository already declares gh: circleci/github-cli@2.1.0 at top level; the orb declares @3.0.0 internally. Those are separate orb instances rather than a conflict, but it is the first place to look if gh misbehaves.

3. snyk fix runs this repository's dependency lifecycle scripts in the same job that holds the context secrets. The GitHub token is read once, unset, and thereafter passed only as a single-command environment prefix, so it is not in the environment those scripts inherit. The Snyk token and the LLM key are exported job-wide by CircleCI and remain visible to them.

4. The orb reference is a dev version — mutable, and it expires after 90 days. This must move back to snyk/prodsec-orb@1 once #166 is released.

All four are documented on the orb's Agentic Fix page.

Verification

  • The config was validated by inlining the packed dev orb into a copy of it and running circleci config validate, which reproduces the compile step CircleCI performs. It passes.
  • prodsec-orb-runtime was confirmed to land on code-analysis only, not on any other job.
  • On the orb side, #166 is green: 82 checks, including four dedicated agentic_fix.sh test jobs.

Not verified: the feature has never completed end-to-end against a real LLM, a real gh install, or a real authenticated push. A parallel trial on snyk-intellij-plugin#880 has not yet produced a fix branch, and that failure is still undiagnosed. Treat the first blocked build here as the experiment, not as a working feature.

🤖 Generated with Claude Code

@ShawkyZ
ShawkyZ requested a review from a team as a code owner July 29, 2026 12:58
@snyk-io

snyk-io Bot commented Jul 29, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@snyk-pr-review-bot

This comment has been minimized.

@snyk-pr-review-bot

This comment has been minimized.

@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from 4d53dea to 7a1e0f9 Compare July 29, 2026 16:42
@snyk-pr-review-bot

This comment has been minimized.

@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from 7a1e0f9 to 896586c Compare July 29, 2026 18:35
@snyk-pr-review-bot

This comment has been minimized.

@snyk-pr-review-bot

This comment has been minimized.

@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from d025415 to a9b10eb Compare July 30, 2026 20:56
@snyk-pr-review-bot

This comment has been minimized.

@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from a9b10eb to f521597 Compare July 30, 2026 21:40
@snyk-pr-review-bot

This comment has been minimized.

@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from f521597 to 72d20b3 Compare July 31, 2026 10:19
@snyk-pr-review-bot

This comment has been minimized.

@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from 72d20b3 to 43ad0d7 Compare July 31, 2026 10:28
@snyk-pr-review-bot

This comment has been minimized.

Points the prodsec orb at the dev build of snyk/prodsec-orb#166 so this repository
can exercise the agentic Open Source fix before that PR is released, and switches
it on for the code-analysis job.

- prodsec orb: snyk/prodsec-orb@1 -> @dev:31a2fb2e92f1f80de62249143bd3a816b3836971
- open-source-scan: high -> medium
- open-source-agentic-fix-enabled: true
- prodsec-orb-runtime added to the code-analysis contexts, which is where
  LITELLM_API_KEY, LITELLM_BASE_URL, LITELLM_MODEL and REMY_GITHUB_TOKEN come
  from. Without it the feature skips, naming what is missing.

When the Enhanced Gate blocks an Open Source scan the job still fails exactly as
it does today, and in addition `snyk fix --agentic` runs at the same severity
threshold the gate uses and raises a pull request against the branch that failed.
The fix step always exits 0, so it cannot turn a passing build red or mask a
failure.

Worth knowing before this merges:

- Lowering to medium is a real behavioural change on its own: the gate can now
  block on medium-severity findings that previously passed. This repository also
  sets open-source-block-ignore-sla, so the gate already ignores remediation SLA -
  the two together widen what blocks considerably. Both this repository's AI
  reviewer and the orb author flagged that the most likely first outcome is a
  blocked main pipeline, and that the fix may exceed its own safety caps (50 files
  / 5000 lines / 1MB patch) and refuse to push anything. Reverting this one line to
  high is a reasonable call if the trial should be informative rather than broad.
- Enabling the feature adds a github-cli/install step that runs on every build of
  this repository, not only blocked ones, because a CircleCI step cannot be made
  conditional on a runtime value. If that install fails it fails the job even when
  the gate passed.
- `snyk fix` runs this repository's dependency lifecycle scripts in the same job
  that holds the context secrets. The LLM key, the Snyk token and - since the orb
  now exports GH_TOKEN for the GitHub CLI - a repository-write GitHub token are all
  readable by those scripts.

The dev orb reference is mutable and expires after 90 days, so this must move back
to snyk/prodsec-orb@1 once #166 is released - otherwise the orb stops resolving and
blocks every pull request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ShawkyZ
ShawkyZ force-pushed the feat/AG-387-trial-agentic-fix branch from 43ad0d7 to 65d3631 Compare July 31, 2026 11:32
@snyk-pr-review-bot

Copy link
Copy Markdown

PR Reviewer Guide 🔍

🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

High Blast Radius Change 🟠 [major]

Lowering open-source-scan to medium while open-source-block-ignore-sla is set to true will cause the CI pipeline to fail on any medium-severity vulnerability, regardless of whether it is within its remediation SLA. For a repository with a dependency tree as large as the Snyk CLI, this is highly likely to turn the build red immediately for existing issues, blocking all PR merges including critical fixes. The PR description acknowledges this risk but does not provide a plan for handling the expected influx of blocking findings.

open-source-scan: medium
open-source-block-ignore-sla: true
Brittle CI Dependency 🟡 [minor]

The PR switches to a dev version of the prodsec orb (snyk/prodsec-orb@dev:31a2fb2e92f1f80de62249143bd3a816b3836971). As noted in the code comments, dev versions expire after 90 days. This creates a 'time bomb' where the entire CI pipeline will fail when the orb version is garbage collected by CircleCI if not reverted or updated to a stable release in time.

prodsec: snyk/prodsec-orb@dev:31a2fb2e92f1f80de62249143bd3a816b3836971
📚 Repository Context Analyzed

This review considered 6 relevant code sections from 5 files (average relevance: 0.73)

🤖 Repository instructions applied (from AGENTS.md)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant