Add Trustabl Agent Scanner to CI - #68
Conversation
WalkthroughAdded a Trustabl Agent Scanner workflow for pushes to ChangesTrustabl scanning
Estimated code review effort: 2 (Simple) | ~10 minutes Mergeability Score: 🟡 Moderate · up to The CI workflow can currently pass while checkout or scanner operational failures are hidden, which may leave the repository without a reliable scan result. Merge should wait until failure handling is tightened or this behavior is explicitly accepted. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Warning |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/trustabl.yml:
- Around line 24-33: Move continue-on-error from the job level to the Trustabl
action step, assigning that step an id so its exit-code output can be
referenced. Add a final validation step that runs unless cancelled and fails
unless checkout succeeded, the Trustabl exit-code output is 0 or 1, and any
failed Trustabl step has exit code 1.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 67aaabea-5d49-43a4-b2fd-0093355cf7a8
📒 Files selected for processing (1)
.github/workflows/trustabl.yml
| # continue-on-error keeps this job advisory — findings are reported but | ||
| # CI does not go red so unrelated work is never blocked. | ||
| continue-on-error: true | ||
| steps: | ||
| - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: trustabl/trustabl-action@973f666d20b5fbb2e6a4511bd3846e965a08c28b # v0.4.1 | ||
| with: | ||
| version: v0.1.6 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- workflow ---'
cat -n .github/workflows/trustabl.yml
printf '%s\n' '--- repository references ---'
rg -n --hidden -S 'trustabl|exit-code|v0\.1\.6|continue-on-error' .github README.md 2>/dev/null || true
printf '%s\n' '--- tracked workflow files ---'
git ls-files '.github/workflows/*'
printf '%s\n' '--- action metadata from repository objects, if available ---'
git ls-tree -r --name-only HEAD | rg '(^|/)(action\.yml|action\.yaml)$|trustabl' || trueRepository: nicofretti/DataGenFlow
Length of output: 2014
🌐 Web query:
Trustabl v0.1.6 exit-code Trustabl action 0.4.1 scanner exit code 2
💡 Result:
The Trustabl scanner, including versions used in the trustabl-action, uses a specific exit code system to communicate scan results and operational status [1][2]. An exit code of 2 indicates a scanner or operational error [1][2]. According to official Trustabl documentation, an exit code of 2 is triggered by any of the following conditions [1][2]: - Scanner/IO error: This includes local installation faults (e.g., disk full, permission denied, or corrupt files) [3]. - No usable rules found: No rules could be located locally, and none were fetchable (run trustabl rules pull) [1][2][3]. - Verification failure: Use of a signed channel (via --rules-source) that failed verification. This includes bad signatures, untrusted/expired keys, expired or rolled-back statements, channel confusion, or digest mismatches [1][2][3]. For comparison, other exit codes include 0 (no findings of medium severity or higher) and 1 (at least one finding of medium severity or higher, or any finding if --strict is enabled) [1][2]. If you are encountering exit code 2 in a CI environment like GitHub Actions, it signifies that the scanner could not successfully complete its execution due to one of these operational or configuration issues rather than a reliability finding [1][2].
Citations:
- 1: https://github.com/trustabl/trustabl
- 2: https://github.com/trustabl/trustabl/blob/main/README.md
- 3: https://github.com/trustabl/trustabl/blob/main/ARCHITECTURE.md
🏁 Script executed:
#!/bin/bash
set -eu
cat -n .github/workflows/trustabl.yml
rg -n --hidden -S 'trustabl|exit-code|v0\.1\.6|continue-on-error' .github README.md 2>/dev/null || trueRepository: nicofretti/DataGenFlow
Length of output: 1780
🏁 Script executed:
#!/bin/bash
set -eu
action_sha=973f666d20b5fbb2e6a4511bd3846e965a08c28b
scanner_sha=bd7d707
printf '%s\n' '--- pinned action metadata ---'
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${action_sha}/action.yml"
printf '%s\n' '--- pinned action files ---'
curl -fsSL "https://api.github.com/repos/trustabl/trustabl-action/git/trees/${action_sha}?recursive=1" |
jq -r '.tree[]?.path' | head -80
printf '%s\n' '--- pinned action README references ---'
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${action_sha}/README.md" |
rg -n -C 3 'exit|output|version|failure' || true
printf '%s\n' '--- scanner README at referenced revision ---'
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl/${scanner_sha}/README.md" |
rg -n -C 5 'exit code|exit-code|Exit code|0.*1.*2|operational|finding' || true
printf '%s\n' '--- scanner source references ---'
curl -fsSL "https://api.github.com/repos/trustabl/trustabl/git/trees/${scanner_sha}?recursive=1" |
jq -r '.tree[]?.path' |
rg '(^|/)(README|ARCHITECTURE)|\.go$|\.rs$|\.py$' |
head -100Repository: nicofretti/DataGenFlow
Length of output: 30343
🌐 Web query:
site:github.com/trustabl/trustabl-action "exit-code"
💡 Result:
An exit code is a numeric value returned by a process to its parent process or operating system upon termination to indicate the result of its execution [1]. In standard computing conventions, an exit code of 0 typically indicates that the process completed successfully, while non-zero exit codes (such as 1 or 2) are used to signal that an error or unexpected condition occurred [1]. These codes allow parent processes, scripts, or automation tools to determine whether a task succeeded or failed and to respond accordingly [1].
Citations:
🏁 Script executed:
#!/bin/bash
set -eu
action_sha=973f666d20b5fbb2e6a4511bd3846e965a08c28b
scanner_sha=bd7d707
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${action_sha}/action.yml"
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${action_sha}/README.md" |
rg -n -C 3 'exit|output|version|failure' || true
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl/${scanner_sha}/README.md" |
rg -n -C 5 'exit code|exit-code|Exit code|operational|finding' || trueRepository: nicofretti/DataGenFlow
Length of output: 28926
🏁 Script executed:
#!/bin/bash
set -eu
sha=973f666d20b5fbb2e6a4511bd3846e965a08c28b
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${sha}/dist/index.js" > "$tmp"
printf '%s\n' '--- output assignment and exit handling ---'
rg -n -C 5 'exit-code|setOutput|setFailed|process\.exit|exec|spawn|native exit|result' "$tmp" | head -240
printf '%s\n' '--- action references to error handling ---'
rg -n -C 4 'catch|throw new Error|core\.setFailed|failed|error' "$tmp" | head -240
printf '%s\n' '--- bundle size ---'
wc -c "$tmp"Repository: nicofretti/DataGenFlow
Length of output: 27363
🌐 Web query:
GitHub Actions steps outcome continue-on-error step outputs documentation
💡 Result:
In GitHub Actions, when a step uses the continue-on-error: true setting, it fundamentally changes how subsequent steps evaluate the step's result [1][2]. To accurately check the status of such a step, you must distinguish between its outcome and its conclusion [1][3]. The two properties are defined as follows: 1. steps.<step_id>.outcome: Represents the result of the step before the continue-on-error logic is applied [1][2]. If the step fails, this value will be 'failure', regardless of the continue-on-error setting [1][4]. 2. steps.<step_id>.conclusion: Represents the final result of the step after the continue-on-error logic is applied [1][2]. If a step with continue-on-error: true fails, the conclusion is rewritten to 'success' [1][3]. Because the conclusion is forced to 'success' for steps with continue-on-error: true, conditional checks using failure or evaluating the step's conclusion in subsequent steps will fail to detect the actual failure [3][5]. To properly gate logic on the real result of a continue-on-error step, you should reference the outcome property in your conditional expressions [3][6]. Example: - id: my-step continue-on-error: true run: exit 1 - if: steps.my-step.outcome == 'failure' run: echo "The step actually failed, but the job continues." Note that steps.<step_id>.outputs continues to function normally for any outputs produced by the step, provided the step generated them before failing [1]. You can access these in the same way as any other step: ${{ steps.<step_id>.outputs.<output_name> }} [1].
Citations:
- 1: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts
- 2: https://github.com/actions/runner/blob/main/docs/adrs/0274-step-outcome-and-conclusion.md
- 3: https://latchkey.dev/learn/github-actions/github-actions-continue-on-error-outcome-not-propagated
- 4: https://robk.uk/posts/training/github/2025-github-actions/07-controlling-workflow/
- 5: Wrong behaviour when combining 'continue-on-error' and 'failure()' in subsequent steps actions/toolkit#1034
- 6: https://stackoverflow.com/questions/57850553/github-actions-check-steps-status
🏁 Script executed:
#!/bin/bash
set -eu
sha=973f666d20b5fbb2e6a4511bd3846e965a08c28b
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${sha}/dist/index.js" > "$tmp"
rg -n -C 5 'exit-code|setOutput|setFailed|process\.exit|exec|spawn|catch|throw new Error|core\.setFailed' "$tmp" | head -300Repository: nicofretti/DataGenFlow
Length of output: 16982
🏁 Script executed:
#!/bin/bash
set -eu
sha=973f666d20b5fbb2e6a4511bd3846e965a08c28b
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${sha}/dist/index.js" > "$tmp"
printf '%s\n' '--- exact Trustabl output markers ---'
rg -n -i -C 12 'exit.?code|readiness.?score|risk.?score|findings.?count|trustabl native|native exit|scan result|severity.?threshold' "$tmp" | tail -300
printf '%s\n' '--- final bundle section ---'
tail -260 "$tmp"Repository: nicofretti/DataGenFlow
Length of output: 50379
🌐 Web query:
site:github.com/trustabl/trustabl-action/blob v0.4.1 "setOutput" "exit-code"
💡 Result:
In v0.4.1:
setOutputdoes not appear inaction.yml. [1]exit-codeis declared as an action output with values:0= clean,1= findings,2= error. [2]
The implementation is likely in dist/index.js, referenced as the action’s main entry point. [2]
🏁 Script executed:
#!/bin/bash
set -eu
sha=973f666d20b5fbb2e6a4511bd3846e965a08c28b
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL "https://raw.githubusercontent.com/trustabl/trustabl-action/${sha}/dist/index.js" > "$tmp"
rg -n -i -C 12 'exit.?code|readiness.?score|risk.?score|findings.?count|trustabl native|native exit|scan result|severity.?threshold' "$tmp" | tail -300
tail -260 "$tmp"Repository: nicofretti/DataGenFlow
Length of output: 50379
Fail the workflow on Trustabl operational errors.
Job-level continue-on-error: true masks checkout and Trustabl failures. Move it to the Trustabl step. Add a final step that runs unless cancelled and fails unless checkout succeeds, steps.trustabl.outputs.exit-code is 0 or 1, and a failed Trustabl step has exit code 1. This keeps findings advisory while rejecting scanner, I/O, and missing-output failures.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/trustabl.yml around lines 24 - 33, Move continue-on-error
from the job level to the Trustabl action step, assigning that step an id so its
exit-code output can be referenced. Add a final validation step that runs unless
cancelled and fails unless checkout succeeded, the Trustabl exit-code output is
0 or 1, and any failed Trustabl step has exit code 1.
1 participant
We came across your repo and we like how you designed a minimal tool for generating and validating data, with a focus on user-friendliness and transparency. We scanned the repo, and noticed agent runtime reliability findings that might be worth reviewing.
[MEDIUM] Skill fetches untrusted external content
File: .claude/skills/configuring-models/SKILL.md
What it means: This skill's body references an external http(s) URL.
[MEDIUM] Skill fetches untrusted external content
File: .claude/skills/creating-pipeline-templates/SKILL.md
What it means: This skill's body references an external http(s) URL.
[MEDIUM] Skill fetches untrusted external content
File: .claude/skills/testing-pipeline-templates/SKILL.md
What it means: This skill's body references an external http(s) URL.
Recommendations are based on our understanding of agent runtime reliability, some findings may be intentional. Please let us know if this was intentional or if our findings are helpful so we can improve the accuracy of the scanner.
Best,
Trustabl.ai
Open-source AI agent reliability scanner (runs locally, GitHub Action)
Summary by CodeRabbit