diff --git a/plugins/codex-security/skills/publish-findings/SKILL.md b/plugins/codex-security/skills/publish-findings/SKILL.md index 4e8df0e7a..87d076588 100644 --- a/plugins/codex-security/skills/publish-findings/SKILL.md +++ b/plugins/codex-security/skills/publish-findings/SKILL.md @@ -1,6 +1,6 @@ --- name: publish-findings -description: Import selected Wiz package vulnerabilities, SAST, repository secrets, and IaC findings locally or to Codex Security Cloud after previewing the destination and evidence. Supports plain or gzip-compressed JSON and normalized JSONL; does not import workload secrets or cloud configuration, run a scan, or assess findings. +description: Import selected Wiz package vulnerabilities, SAST, repository secrets, and IaC findings locally or to Codex Security Cloud after previewing the destination and evidence. Supports JSON, normalized JSONL, repository vulnerability CSV, and gzip; does not import workload secrets or cloud configuration, run a scan, assess findings, or synchronize triage decisions. --- # Publish external findings @@ -27,15 +27,15 @@ Local records remain unassessed source evidence, separate from `findings list` a Prefer the user's configured Wiz plugin or authorized [Wiz MCP](https://www.wiz.io/blog/introducing-wiz-mcp) connection to read selected findings. Use an authorized Wiz API connection when the required records are not exposed by MCP. Save the actual selected records locally with their original occurrence IDs and evidence. Inspect the returned fields: a summary, Wiz Issue, or local Wiz CLI scan result is not automatically a supported finding record. Do not invent fields to make a response fit the mapper. -Finish the requested pagination; for an explicit subset, retain that selection and its named collection. The publisher reads plain or gzip-compressed JSON and normalized JSONL with `source_finding_id` and `evidence` as documented in the CLI guide. A supplied export can use the same path. The command does not fetch from Wiz or accept CSV, scan events, workload secrets, cloud configuration, or external network exports. +Finish the requested pagination; for an explicit subset, retain that selection and its named collection. The publisher reads JSON, normalized JSONL with `source_finding_id` and `evidence`, and the repository package-vulnerability CSV profile documented in the CLI guide. A supplied export can use the same path. The command does not fetch from Wiz or accept other CSV profiles, scan events, workload secrets, cloud configuration, or external network exports. Use a stable Wiz tenant/finding namespace as `--source-key`. A project filter is selection context, not a new identity. Confirm the selected assets or builds belong to the destination repository; do not infer a scanner branch or revision from the current checkout. Do not publish vendor credentials, unrelated records, or local assessments as original scanner evidence. -For package vulnerability UI exports, use **Vulnerability Findings → Save as → Report**, keeping the repository and finding filters. Select **Repository Branch** (remove the default Virtual Machine selection), **JSON**, and **Detailed** columns. Download the completed report; default gzip compression is supported, even with a `.json` filename. **Raw Event** on Code & Build Scans contains scan metadata and cannot supply findings. Do not create a scheduled report or enable notifications or external storage unless requested. +For a package vulnerability report, use **Vulnerability Findings → Save as → Report**, keeping the repository and finding filters. Select **Repository Branch** (remove the default Virtual Machine selection), **JSON** or comma-separated **CSV**, and **Detailed** columns. Pass supported CSV reports directly to the command; their GitHub `CloudProviderURL` must identify the destination repository branch. The importer retains every original cell in `source_data.row` and uses the reported `ID` as occurrence identity. Download the completed report; gzip compression is supported regardless of the filename extension. **Raw Event** on Code & Build Scans contains scan metadata and cannot supply findings. Do not create a scheduled report or enable notifications or external storage unless requested. For SAST, repository secrets, and IaC, retain the explicit `data.sastFindings.nodes`, `data.secretInstances.nodes`, or `data.iacFindings.nodes` collection. Bare arrays cannot identify these finding families. Include a verified URL on the source repository metadata or supply `data.versionControlResources.nodes` with matching `repository.id` and `repository.url`. Retain the documented inventory `platform` field; explicit `GITHUB` establishes case-insensitive matching on custom GitHub Enterprise hosts. Resolve conflicting platform or URL metadata for a repository ID before publishing. For secrets, the repository is `resource.typedProperties.repository` and resource type must be `REPOSITORY_BRANCH`. Do not map names, inventory node IDs, `providerID`, or Wiz UUIDs directly to Cloud repository IDs. The importer verifies source URLs against the selected destination; resolve a missing or conflicting mapping before uploading. -Keep raw source records unchanged in `source_data`. New raw occurrence IDs are prefixed `sast:`, `secret:`, or `iac:`; existing package IDs are unchanged. For CSV report conversion, use a reviewed mapping of the actual header names into the documented normalized JSONL profile, set `evidence.details.kind` and the verified `details.repository.url`, and use the same prefixed occurrence ID. The command does not parse CSV directly. Select source paths and line numbers explicitly in secret reports; the standard metadata query lacks them. Preserve rule/CWE, range/snippet, detector validation, and IaC comparison fields in the typed details described in the CLI docs. Preserve empty comparison strings. Do not invent missing IDs, severity, revisions, or locations, retrieve credential values, test credentials, or apply suggested fixes. Vendor status, AI verdict, secret validation, introduced commits, and logical IaC resources are evidence, not Codex assessment or proof of runtime deployment. +Keep raw source records unchanged in `source_data`. New raw occurrence IDs are prefixed `sast:`, `secret:`, or `iac:`; existing package IDs are unchanged. SAST, repository-secret, and IaC CSV reports still require a reviewed mapping of the actual header names into the documented normalized JSONL profile: set `evidence.details.kind` and the verified `details.repository.url`, and use the same prefixed occurrence ID. Select source paths and line numbers explicitly in secret reports; the standard metadata query lacks them. Preserve rule/CWE, range/snippet, detector validation, and IaC comparison fields in the typed details described in the CLI docs. Preserve empty comparison strings. Do not invent missing IDs, severity, revisions, or locations, retrieve credential values, test credentials, or apply suggested fixes. Vendor status, AI verdict, secret validation, introduced commits, and logical IaC resources are evidence, not Codex assessment or proof of runtime deployment. When publishing to staging or another Cloud deployment, set `CODEX_SECURITY_CLOUD_BASE_URL` to its API root before both preview and upload. If the Cloud page or user supplies its app URL, also set `CODEX_SECURITY_CLOUD_WEB_URL` to that full URL so the command can return a repository link. The UI setting does not change API routing or authentication. Use that deployment's login and an isolated state directory for tests. Verify `cloudApiUrl` and any `cloudUrl` in the preview against the intended deployment; the former is the API endpoint and the latter is the UI link. Saved retries are separated by deployment. The native scan setting `CODEX_SECURITY_CLOUD_PUBLISH_URL` does not route vendor imports. Never fall back to production when staging is unavailable. diff --git a/sdk/typescript/docs/cli.md b/sdk/typescript/docs/cli.md index 4c60079dd..e6714242e 100644 --- a/sdk/typescript/docs/cli.md +++ b/sdk/typescript/docs/cli.md @@ -1281,8 +1281,9 @@ Before the first import: Saved package-vulnerability reports are also supported. In Wiz **Vulnerability Findings**, retain the repository and finding filters, choose **Save as → Report**, -select **Repository Branch**, **JSON**, and **Detailed** columns, then download -the completed report. Gzip is detected regardless of the filename extension. +select **Repository Branch**, **CSV** or **JSON**, and **Detailed** columns, then +download the completed report. CSV must use the columns described below and a +comma delimiter. Gzip is detected regardless of the filename extension. **Raw Event** on Code & Build Scans contains scan metadata rather than findings. For an alternate Cloud deployment, set `CODEX_SECURITY_CLOUD_BASE_URL` to its @@ -1308,7 +1309,7 @@ the manual navigation guidance rather than guessing a UI host or port. Preview the selection and destination: ```bash -codex-security publish findings selected-wiz.json \ +codex-security publish findings selected-wiz.csv \ --to cloud --repository https://github.com/example/project --provider wiz \ --source-key TENANT_ID/vulnerability-finding --dry-run --format json ``` @@ -1330,12 +1331,32 @@ exclusions. `--dry-run --format json` retains all normalized findings and comple evidence for inspection. A prompt explicitly includes any records that will be skipped. -Package input can be a downloaded Wiz JSON vulnerability report, a vulnerability +Package input can be a downloaded Wiz Code vulnerability CSV, a Wiz JSON vulnerability report, a vulnerability finding, an array, a complete `data.vulnerabilityFindings.nodes` response, or JSONL with one package record per line. Plain and gzip-compressed files are accepted. A response that advertises another page is rejected. Complete the requested pages or save only the explicitly selected records; retain the named collection for -SAST, repository secrets, and IaC. Normalized JSONL is also accepted: +SAST, repository secrets, and IaC. + +The CSV mapper requires `ID`, `Name`, `DetailedName`, `AssetID`, `WizURL`, +`CloudPlatform`, and `CloudProviderURL`. Check the actual report headers against +this profile; other Wiz report types use different fields. +For repository imports, `CloudPlatform` must identify GitHub and +`CloudProviderURL` must identify a repository branch. The repository URL must +match the selected Cloud repository or, for local import, one of the checkout's +Git remotes when configured. This also applies when the parsed findings are saved and +read again as normalized JSONL. Other resource types are excluded. + +CSV imports use `ID` as the occurrence identity and retain every parsed field, +including unknown columns and quoted multiline values, in `source_data.row`. +Severity comes from `Severity`, then `VendorSeverity`, then `CVSSSeverity`. +Vendor resolution status stays in the original evidence. The importer does not +derive a branch or revision from a tree URL, or treat last-detected time as a +record-update timestamp. Duplicate headers, damaged CSV rows, invalid UTF-8, +and duplicate occurrence IDs stop the import. Review any excluded records in +the preview before publishing. + +Normalized JSONL is also accepted: ```json { @@ -1348,7 +1369,7 @@ SAST, repository secrets, and IaC. Normalized JSONL is also accepted: } ``` -Use Wiz's finding `id`, not a CVE, as identity. Keep the source key stable across +Use Wiz's finding `id` (`ID` in CSV), not a CVE, as identity. Keep the source key stable across exports; a selected project filter does not change the vendor namespace. Confirm that the selected assets or builds map to this repository. The raw Wiz adapter retains each record in `source_data`, uses reported severity, and leaves unknown @@ -1436,7 +1457,7 @@ does not establish that it is deployed. Precise original timestamps remain in timestamps. Native SAST and secret report downloads can be CSV, including gzip-compressed -CSV. CSV is not parsed directly. Convert a selected report with a reviewed mapping +CSV. Those CSV profiles are not parsed directly. Convert a selected report with a reviewed mapping for its actual column headers into normalized JSONL, preserving each original row in `source_data`. Set `details.kind` and `details.repository.url` explicitly after verifying the source repository. Use the same prefixed occurrence ID shown above diff --git a/sdk/typescript/scripts/package-dist-files.mjs b/sdk/typescript/scripts/package-dist-files.mjs index 957509fdb..b2f95dc48 100644 --- a/sdk/typescript/scripts/package-dist-files.mjs +++ b/sdk/typescript/scripts/package-dist-files.mjs @@ -31,6 +31,7 @@ const distModules = [ "external-findings-publish", "external-import-contract", "external-import-models", + "wiz-csv", "wiz-findings", "codex-prompt", "codex-sdk-environment", diff --git a/sdk/typescript/src/cli.ts b/sdk/typescript/src/cli.ts index ab0a9ae85..02d77d543 100644 --- a/sdk/typescript/src/cli.ts +++ b/sdk/typescript/src/cli.ts @@ -3503,14 +3503,14 @@ export async function main( publication.command("findings", { description: "Preview and import selected Wiz repository findings locally or to Cloud.", - hint: "Supply a package vulnerability JSON report, named SAST/repository-secret/IaC JSON collections with repository metadata, or normalized JSONL. Gzip is supported; CSV, scan events, workload secrets, and cloud configuration exports are not. --to local stores unassessed evidence for a local repository path without Cloud login. --full-output requires --format json or --format jsonl.\nExample: codex-security publish findings selected-wiz.json --to local --repository ./project --provider wiz --source-key TENANT_ID/vulnerability-finding --dry-run --format json", + hint: "Supply a Wiz Code vulnerability CSV for GitHub repositories, a package vulnerability JSON report, named SAST/repository-secret/IaC JSON collections with repository metadata, or normalized JSONL. Gzip is supported; other CSV profiles, scan events, workload secrets, and cloud configuration exports are not. --to local stores unassessed evidence for a local repository path without Cloud login. --full-output requires --format json or --format jsonl.\nExample: codex-security publish findings selected-wiz.json --to local --repository ./project --provider wiz --source-key TENANT_ID/vulnerability-finding --dry-run --format json", destructive: true, mcp: false, args: z.object({ file: z .string() .describe( - "Saved Wiz finding collections, package vulnerability JSON, or normalized JSONL.", + "Saved Wiz finding collections, package vulnerability JSON/CSV, or normalized JSONL.", ), }), options: z.object({ diff --git a/sdk/typescript/src/external-findings-publish.ts b/sdk/typescript/src/external-findings-publish.ts index a7ad767ab..3f9a2d9a6 100644 --- a/sdk/typescript/src/external-findings-publish.ts +++ b/sdk/typescript/src/external-findings-publish.ts @@ -28,6 +28,7 @@ import { readExternalFindings, type ExternalFindingsReaderOptions, } from "./external-findings-read.js"; +import { wizCsvRepositoryUrl } from "./wiz-csv.js"; import type { ExternalFindings, ExternalPublicationInput, @@ -704,15 +705,21 @@ async function preparePublicationInput( let requests = saved?.requests; if (!requests) { for (const finding of parsed.findings) { - const sourceRepository = finding.evidence.details?.repository; - if ( - sourceRepository && - cloudRepositoryUrlKey(sourceRepository.url) !== - cloudRepositoryUrlKey(destination.url) - ) - throw new CodexSecurityError( - `Finding ${JSON.stringify(finding.source_finding_id)} belongs to ${JSON.stringify(sourceRepository.url)}, which does not match the selected Cloud repository ${JSON.stringify(destination.url)}. Verify the source repository mapping before publishing.`, - ); + const sourceRepositories = [ + finding.evidence.details?.repository?.url, + ...(options.provider === "wiz" + ? [wizCsvRepositoryUrl(finding.evidence)] + : []), + ]; + for (const sourceRepository of sourceRepositories) + if ( + sourceRepository && + cloudRepositoryUrlKey(sourceRepository) !== + cloudRepositoryUrlKey(destination.url) + ) + throw new CodexSecurityError( + `Finding ${JSON.stringify(finding.source_finding_id)} belongs to ${JSON.stringify(sourceRepository)}, which does not match the selected Cloud repository ${JSON.stringify(destination.url)}. Verify the source repository mapping before publishing.`, + ); } let preparedCount = 0; progress({ diff --git a/sdk/typescript/src/external-findings-read.ts b/sdk/typescript/src/external-findings-read.ts index e92defd32..706f03a63 100644 --- a/sdk/typescript/src/external-findings-read.ts +++ b/sdk/typescript/src/external-findings-read.ts @@ -3,6 +3,7 @@ import type { ExternalPublicationInput, ExternalSourceRepository, } from "./external-findings.js"; +import { wizCsvRepositoryUrl } from "./wiz-csv.js"; import { isGitHubCloudHost } from "./git-remote.js"; import { isRecord } from "./record.js"; import { readVendorFindingsForPublication } from "./wiz-findings.js"; @@ -29,6 +30,8 @@ export async function readExternalFindings( // Raw package exports keep repository-branch URLs in retained source data. const sourceRepositories = publication.current.findings.flatMap( ({ evidence }) => { + const csvUrl = wizCsvRepositoryUrl(evidence); + if (csvUrl) return [{ url: csvUrl, platform: "github" as const }]; const asset = evidence.source_data?.["vulnerableAsset"]; if (!isRecord(asset) || typeof asset["cloudProviderURL"] !== "string") return []; diff --git a/sdk/typescript/src/wiz-csv.ts b/sdk/typescript/src/wiz-csv.ts new file mode 100644 index 000000000..5dc1dd7f2 --- /dev/null +++ b/sdk/typescript/src/wiz-csv.ts @@ -0,0 +1,136 @@ +import Papa from "papaparse"; +import { validateExternalEvidence } from "./external-import-contract.js"; +import type { ExternalFindingEvidence } from "./external-import-models.js"; +import type { VendorFinding } from "./wiz-findings.js"; + +const profile = "wiz-code-vulnerability-csv"; +const requiredHeaders = [ + "ID", + "Name", + "DetailedName", + "AssetID", + "WizURL", + "CloudPlatform", + "CloudProviderURL", +]; + +function text(value: unknown): string | null { + return typeof value === "string" && value.trim() ? value : null; +} + +/** Recognize the supported Wiz vulnerability CSV columns, retaining every cell. */ +export function readWizCsv( + contents: string, +): Record[] | undefined { + // JSONL can retain CSV column names in its evidence. Recognize the first + // JSON value before interpreting all of its quoted records as CSV. + try { + JSON.parse(contents.trimStart().split(/\r?\n/u, 1)[0]!); + return undefined; + } catch { + // A CSV header is not a JSON value. + } + // Read arrays so duplicate headers cannot be silently renamed, and unknown + // columns (including object-prototype names) remain ordinary source data. + const parsed = Papa.parse(contents, { + delimiter: ",", + skipEmptyLines: true, + }); + const [headers, ...rows] = parsed.data; + if (!headers || !requiredHeaders.every((name) => headers.includes(name))) + return undefined; + if (parsed.errors.length) + throw new Error(`Invalid Wiz CSV: ${parsed.errors[0]!.message}`); + if (new Set(headers).size !== headers.length) + throw new Error("Wiz CSV contains duplicate column names."); + return rows.map((cells, index) => { + if (cells.length !== headers.length) + throw new Error(`Wiz CSV row ${index + 2} has the wrong column count.`); + return Object.fromEntries(headers.map((name, i) => [name, cells[i]!])); + }); +} + +function repositoryUrl(row: Record): string { + if (text(row["CloudPlatform"])?.trim().toLowerCase() !== "github") + throw new Error( + "Wiz CSV imports require GitHub repository-branch findings. Workload and other provider CSV reports need an explicit normalized JSONL mapping.", + ); + const value = text(row["CloudProviderURL"]); + const url = value ? new URL(value) : undefined; + const origin = value?.match(/^(https?:\/\/[^/?#]+)/iu)?.[1]; + const path = url?.pathname.match(/^\/([^/]+)\/([^/]+)\/tree\/(.+)$/u); + if ( + !url || + !origin || + !["http:", "https:"].includes(url.protocol) || + url.username || + url.password || + !path + ) + throw new Error("Wiz CSV finding requires a GitHub repository-branch URL."); + // URL.origin removes explicit default ports, which the Cloud contract keeps. + return `${origin}/${path[1]}/${path[2]}`; +} + +/** Keep repository binding available after a normalized JSONL round trip. */ +export function wizCsvRepositoryUrl( + evidence: ExternalFindingEvidence, +): string | undefined { + const source = evidence.source_data; + if (source?.["format"] !== profile) return undefined; + const row = source["row"]; + if (!row || typeof row !== "object" || Array.isArray(row)) + throw new Error("Wiz CSV source evidence is missing its original row."); + return repositoryUrl(row as Record); +} + +export function wizCsvFinding( + row: Record, + sourceFindingId: string, +): VendorFinding { + repositoryUrl(row); + const name = text(row["Name"]); + const packageName = text(row["DetailedName"]); + if (!name || !packageName || !text(row["AssetID"])) + throw new Error( + "Wiz CSV finding requires Name, DetailedName, and AssetID.", + ); + const path = text(text(row["LocationPath"])?.replace(/^\/+/, "")); + const fixedVersion = text(row["FixedVersion"]); + const severity = ( + text(row["Severity"]) ?? + text(row["VendorSeverity"]) ?? + text(row["CVSSSeverity"]) + ) + ?.trim() + .toLowerCase(); + return { + source_finding_id: sourceFindingId, + evidence: validateExternalEvidence({ + title: [packageName, text(row["Version"]), name] + .filter(Boolean) + .join(" "), + description: text(row["Description"]), + severity, + url: text(row["WizURL"]), + advisory_ids: [name], + packages: [ + { + name: packageName, + ecosystem: null, + installed_version: text(row["Version"]), + manifest_path: path, + fixed_versions: fixedVersion ? [fixedVersion] : [], + }, + ], + locations: path ? [{ path }] : [], + // A tree URL does not distinguish a slash-containing ref from a directory. + branch: null, + code_revision: null, + source_scan_id: null, + source_updated_at: null, + image_digests: [], + source_data: { format: profile, row }, + }), + }; +} diff --git a/sdk/typescript/src/wiz-findings.ts b/sdk/typescript/src/wiz-findings.ts index 177c81ec7..126dafc38 100644 --- a/sdk/typescript/src/wiz-findings.ts +++ b/sdk/typescript/src/wiz-findings.ts @@ -4,6 +4,7 @@ import { validateExternalEvidence, } from "./external-import-contract.js"; import { parseJsonOrJsonl, readScannerText } from "./scanner-import.js"; +import { readWizCsv, wizCsvFinding } from "./wiz-csv.js"; import type { ExternalFinding as VendorFinding, ExternalFindings as VendorFindings, @@ -113,7 +114,7 @@ type RepositoryFindingKind = "sast" | "secret" | "iac"; type InputRecord = { value: unknown; - kind?: "sca" | RepositoryFindingKind | "cloud_configuration"; + kind?: "sca" | "sca_csv" | RepositoryFindingKind | "cloud_configuration"; }; type RepositoryMetadata = { @@ -436,7 +437,7 @@ function records(payload: unknown): FindingInput { legacyRepositories, }; throw new Error( - "Unsupported Wiz export. Supply vulnerability finding objects, named sastFindings/secretInstances/iacFindings collections, or normalized JSONL. CSV reports must be converted to the documented normalized JSONL profile.", + "Unsupported Wiz export. Supply repository vulnerability CSV, vulnerability finding objects, named sastFindings/secretInstances/iacFindings collections, or normalized JSONL. Other CSV reports need an explicit normalized JSONL mapping.", ); } @@ -505,10 +506,23 @@ export async function readVendorFindingsForPublication(path: string): Promise<{ async function readFindingInput(path: string): Promise { try { - return records(parseJsonOrJsonl(await readScannerText(path))); + const contents = await readScannerText(path); + let payload: unknown; + try { + payload = JSON.parse(contents); + } catch { + const csv = readWizCsv(contents); + if (csv) + return { + records: csv.map((value) => ({ value, kind: "sca_csv" })), + repositories: new Map(), + }; + payload = parseJsonOrJsonl(contents); + } + return records(payload); } catch (cause) { throw new CodexSecurityError( - `Could not read vendor findings as UTF-8 JSON or JSONL: ${cause instanceof Error ? cause.message : String(cause)}`, + `Could not read vendor findings as UTF-8 JSON, JSONL, or Wiz CSV: ${cause instanceof Error ? cause.message : String(cause)}`, { cause }, ); } @@ -530,7 +544,9 @@ function normalizeFindings( try { if (!record) throw new Error("Finding must be an object."); let finding: VendorFinding; - if ("source_finding_id" in record) { + if (item.kind === "sca_csv") { + finding = wizCsvFinding(record, sourceFindingId(record["ID"])); + } else if ("source_finding_id" in record) { if ( Object.keys(record).some( (key) => !["source_finding_id", "evidence"].includes(key), @@ -574,13 +590,16 @@ function normalizeFindings( } identities.add(finding.source_finding_id); result.findings.push(finding); - if ("source_finding_id" in record) normalized?.add(finding); + if (item.kind !== "sca_csv" && "source_finding_id" in record) + normalized?.add(finding); } catch (error) { if (error instanceof CodexSecurityError) throw error; result.excluded.push({ position: index + 1, source_finding_id: - text(record?.["source_finding_id"]) ?? text(record?.["id"]), + item.kind === "sca_csv" + ? text(record?.["ID"]) + : (text(record?.["source_finding_id"]) ?? text(record?.["id"])), reason: error instanceof Error ? error.message : String(error), }); } diff --git a/sdk/typescript/tests-ts/external-findings-publish.test.ts b/sdk/typescript/tests-ts/external-findings-publish.test.ts index 743ee60be..01ba8d029 100644 --- a/sdk/typescript/tests-ts/external-findings-publish.test.ts +++ b/sdk/typescript/tests-ts/external-findings-publish.test.ts @@ -1703,7 +1703,7 @@ test.each(["JSON", "JSONL"])( }); expect(stdout).not.toContain("\u001b"); expect(stderr).toContain( - "Could not read vendor findings as UTF-8 JSON or JSONL:", + "Could not read vendor findings as UTF-8 JSON, JSONL, or Wiz CSV:", ); expect(stderr).toContain("malformed"); expect(stderr).not.toContain("\u001b"); diff --git a/sdk/typescript/tests-ts/wiz-findings-local.test.ts b/sdk/typescript/tests-ts/wiz-findings-local.test.ts new file mode 100644 index 000000000..ec60f0999 --- /dev/null +++ b/sdk/typescript/tests-ts/wiz-findings-local.test.ts @@ -0,0 +1,248 @@ +import { access, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { gzipSync } from "node:zlib"; +import { expect, test } from "bun:test"; +import { prepareLocalExternalImport } from "../src/external-findings-import.js"; +import { listLocalExternalFindings } from "../src/external-findings-local.js"; +import { readVendorFindings } from "../src/wiz-findings.js"; +import { localFindingsFixture } from "./external-findings-local-helpers.js"; + +const repositoryUrl = "https://github.com/example/project"; +const row: Record = { + ID: "csv-occurrence-1", + WizURL: "https://vendor.example.test/findings/csv-occurrence-1", + Name: "CVE-2026-0001", + DetailedName: "example-package", + AssetID: "synthetic-repository-branch", + CloudPlatform: "GitHub", + CloudProviderURL: `${repositoryUrl}/tree/main`, + Severity: "HIGH", + Description: 'Synthetic "description", with a comma.\r\nAnother line.', + LocationPath: "/package-lock.json", + Version: "1.0.0", + FixedVersion: "2.0.0", +}; + +function csvReport(source: Record): string { + const quote = (value: string) => '"' + value.replaceAll('"', '""') + '"'; + return [Object.keys(source), Object.values(source)] + .map((cells) => cells.map(quote).join(",")) + .join("\r\n"); +} + +test.each(["CSV", "CSV gzip", "normalized JSONL", "normalized JSONL gzip"])( + "Wiz %s matches GitHub Enterprise repository names without changing evidence", + async (format) => { + const fixture = await localFindingsFixture({ + remote: "git@git.example.test:example/project.git", + }); + try { + const file = join(fixture.root, "wiz-report"); + const options = { + repository: fixture.repository, + provider: "wiz" as const, + sourceKey: "example-project:main", + }; + const dependencies = { environment: fixture.environment }; + const database = join(fixture.stateDirectory, "workbench.sqlite3"); + const writeReport = async (source: Record) => { + let contents = csvReport(source); + if (format.startsWith("normalized")) { + await writeFile(file, contents); + contents = (await readVendorFindings(file)).findings + .map((finding) => JSON.stringify(finding)) + .join("\n"); + } + await writeFile( + file, + format.endsWith("gzip") ? gzipSync(contents) : contents, + ); + }; + + for (const url of [ + "https://Git.Example.test/Example/Other/tree/main", + "https://other.example.test/Example/Project/tree/main", + ]) { + await writeReport({ ...row, CloudProviderURL: url }); + await expect( + prepareLocalExternalImport(file, options, dependencies), + ).rejects.toThrow("does not match"); + await expect(access(database)).rejects.toThrow(); + } + + const source = { + ...row, + CloudProviderURL: + "https://Git.Example.test/Example/Project/tree/feature/imports", + }; + await writeReport(source); + const parsed = await readVendorFindings(file); + const prepared = await prepareLocalExternalImport( + file, + options, + dependencies, + ); + expect(prepared.preview.findings).toEqual(parsed.findings); + expect((await prepared.import()).counts).toEqual({ + created: 1, + updated: 0, + unchanged: 0, + }); + const stored = await listLocalExternalFindings(options, dependencies); + expect(stored).toHaveLength(1); + expect(stored[0]!.evidence).toEqual(parsed.findings[0]!.evidence); + expect(stored[0]!.evidence.source_data).toEqual({ + format: "wiz-code-vulnerability-csv", + row: source, + }); + + const before = await readFile(database); + await writeReport({ + ...source, + CloudProviderURL: "https://Git.Example.test/Example/Other/tree/main", + }); + await expect( + prepareLocalExternalImport(file, options, dependencies), + ).rejects.toThrow("does not match"); + expect(await readFile(database)).toEqual(before); + expect(await listLocalExternalFindings(options, dependencies)).toEqual( + stored, + ); + } finally { + await fixture.dispose(); + } + }, +); + +test("Wiz CSV imports into SQLite and updates the same source finding", async () => { + const fixture = await localFindingsFixture({ remote: repositoryUrl }); + try { + const file = join(fixture.root, "wiz.csv"); + const options = { + repository: fixture.repository, + provider: "wiz" as const, + sourceKey: "example-project:main", + }; + const dependencies = { environment: fixture.environment }; + await writeFile(file, csvReport(row)); + const prepared = await prepareLocalExternalImport( + file, + options, + dependencies, + ); + expect(prepared.preview.excluded).toEqual([]); + expect((await prepared.import()).counts).toEqual({ + created: 1, + updated: 0, + unchanged: 0, + }); + const [stored] = await listLocalExternalFindings(options, dependencies); + expect(stored).toMatchObject({ + provider: "wiz", + sourceKey: options.sourceKey, + sourceFindingId: row["ID"], + evidence: prepared.preview.findings[0]!.evidence, + assessment: { state: "not_assessed" }, + version: 1, + }); + expect(stored!.evidence).toMatchObject({ + locations: [{ path: "package-lock.json", line: null }], + source_data: { format: "wiz-code-vulnerability-csv", row }, + branch: null, + code_revision: null, + }); + expect(stored!.evidence).not.toHaveProperty("confidence"); + + const repeated = await prepareLocalExternalImport( + file, + options, + dependencies, + ); + expect((await repeated.import()).counts).toEqual({ + created: 0, + updated: 0, + unchanged: 1, + }); + expect(await listLocalExternalFindings(options, dependencies)).toEqual([ + stored!, + ]); + + const updatedRow = { ...row, Description: "Updated scanner evidence" }; + await writeFile(file, csvReport(updatedRow)); + const changed = await prepareLocalExternalImport( + file, + options, + dependencies, + ); + expect((await changed.import()).counts).toEqual({ + created: 0, + updated: 1, + unchanged: 0, + }); + const records = await listLocalExternalFindings(options, dependencies); + expect(records).toHaveLength(1); + expect(records[0]).toMatchObject({ + id: stored!.id, + sourceFindingId: stored!.sourceFindingId, + evidence: changed.preview.findings[0]!.evidence, + assessment: { state: "not_assessed" }, + version: 2, + createdAt: stored!.createdAt, + }); + expect(records[0]!.evidence.source_data).toEqual({ + format: "wiz-code-vulnerability-csv", + row: updatedRow, + }); + } finally { + await fixture.dispose(); + } +}); + +test("Wiz CSV retains a package advisory without fabricated locations, fixes or confidence", async () => { + const fixture = await localFindingsFixture({ remote: repositoryUrl }); + try { + const file = join(fixture.root, "wiz.csv"); + const sparse = { + ...row, + Severity: "UNKNOWN", + Description: "", + LocationPath: "", + Version: "", + FixedVersion: "", + }; + await writeFile(file, csvReport(sparse)); + const options = { + repository: fixture.repository, + provider: "wiz" as const, + sourceKey: "example-project:main", + }; + const dependencies = { environment: fixture.environment }; + const prepared = await prepareLocalExternalImport( + file, + options, + dependencies, + ); + expect((await prepared.import()).counts.created).toBe(1); + const [stored] = await listLocalExternalFindings(options, dependencies); + expect(stored!.evidence).toEqual(prepared.preview.findings[0]!.evidence); + expect(stored!.evidence).toMatchObject({ + description: null, + severity: "unknown", + locations: [], + packages: [ + { + name: "example-package", + installed_version: null, + manifest_path: null, + fixed_versions: [], + }, + ], + source_data: { format: "wiz-code-vulnerability-csv", row: sparse }, + }); + expect(stored!.evidence).not.toHaveProperty("confidence"); + expect(stored!.evidence).not.toHaveProperty("remediation"); + expect(stored!.assessment).toEqual({ state: "not_assessed" }); + } finally { + await fixture.dispose(); + } +}); diff --git a/sdk/typescript/tests-ts/wiz-findings.test.ts b/sdk/typescript/tests-ts/wiz-findings.test.ts index e35ba7e1d..9239016ae 100644 --- a/sdk/typescript/tests-ts/wiz-findings.test.ts +++ b/sdk/typescript/tests-ts/wiz-findings.test.ts @@ -12,6 +12,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { Writable } from "node:stream"; import { gzipSync } from "node:zlib"; +import { isDeepStrictEqual } from "node:util"; import { afterEach, expect, test } from "bun:test"; import { main } from "../src/cli.js"; import { @@ -580,7 +581,7 @@ async function cloudFixture(payload: unknown) { const results = request.items.map((item) => { const previous = reports.get(item.source_finding_id); const outcome = previous - ? JSON.stringify(previous.evidence) === JSON.stringify(item.evidence) + ? isDeepStrictEqual(previous.evidence, item.evidence) ? "unchanged" : "updated" : "created"; @@ -755,6 +756,342 @@ test("a wrong source repository stops raw and normalized imports before source l ).toBeUndefined(); }); +const csvRow: Record = { + ID: "csv-occurrence-1", + WizURL: "https://app.wiz.io/vulnerability-findings/csv-occurrence-1", + Name: "CVE-2026-0001", + DetailedName: "example-package", + AssetID: "synthetic-repository-branch", + AssetName: "example/project/feature/parser", + CloudPlatform: "GitHub", + CloudProviderURL: "https://github.com/example/project/tree/feature/parser", + Severity: "HIGH", + VendorSeverity: "LOW", + CVSSSeverity: "CRITICAL", + Description: 'A quoted "description", with a comma.\r\nAnd another line.', + LocationPath: "/package-lock.json", + Version: "1.0.0", + FixedVersion: "2.0.0", + DetectionMethod: "Library", + FindingStatus: "Resolved", + LastDetected: "2026-01-02T03:04:05Z", +}; + +function csvReport( + rows: Record[], + headers = Object.keys(rows[0] ?? csvRow), +): string { + const quote = (value: string) => '"' + value.replaceAll('"', '""') + '"'; + return [headers, ...rows.map((row) => headers.map((key) => row[key] ?? ""))] + .map((cells) => cells.map(quote).join(",")) + .join("\r\n"); +} + +test.each(["", "\n \r\n"])( + "JSONL retains CSV-like column names as evidence with prefix %j", + async (prefix) => { + const f = await fixture({}); + const records = ["first", "second"].map((id) => ({ + source_finding_id: id, + evidence: { + title: "Synthetic normalized finding", + severity: "high", + source_data: { columns: ["extra", ...Object.keys(csvRow), "other"] }, + }, + })); + await writeFile( + f.file, + prefix + records.map((record) => JSON.stringify(record)).join("\r\n"), + ); + const parsed = await readVendorFindings(f.file); + expect(parsed.excluded).toEqual([]); + expect(parsed.findings.map((finding) => finding.source_finding_id)).toEqual( + ["first", "second"], + ); + expect( + parsed.findings.map((finding) => finding.evidence.source_data), + ).toEqual(records.map((record) => record.evidence.source_data)); + }, +); + +test.each(["plain", "gzip", "bom"])( + "Wiz Code CSV maps source claims and retains every original cell (%s)", + async (encoding) => { + const f = await fixture({}); + const csv = csvReport([csvRow]); + await writeFile( + f.file, + encoding === "gzip" + ? gzipSync(csv) + : (encoding === "bom" ? "\ufeff" : "") + csv, + ); + const parsed = await readVendorFindings(f.file); + expect(parsed.read).toBe(1); + expect(parsed.excluded).toEqual([]); + expect(parsed.findings[0]).toMatchObject({ + source_finding_id: "csv-occurrence-1", + evidence: { + title: "example-package 1.0.0 CVE-2026-0001", + description: csvRow["Description"], + severity: "high", + advisory_ids: ["CVE-2026-0001"], + packages: [ + { + name: "example-package", + ecosystem: null, + installed_version: "1.0.0", + manifest_path: "package-lock.json", + fixed_versions: ["2.0.0"], + }, + ], + locations: [{ path: "package-lock.json", line: null }], + branch: null, + code_revision: null, + source_scan_id: null, + source_updated_at: null, + source_data: { format: "wiz-code-vulnerability-csv", row: csvRow }, + }, + }); + expect(parsed.findings[0]!.evidence.details).toBeUndefined(); + }, +); + +test("Wiz CSV keeps opaque columns and formula-like text without interpreting them", async () => { + const row = Object.fromEntries([ + ...Object.entries(csvRow), + ["__proto__", '{"admin":true}'], + ["constructor", "opaque constructor"], + ["source_finding_id", "opaque source id"], + ["evidence", "opaque evidence"], + ["Extra", "'=SUM(1,2)"], + ]); + const f = await fixture({}); + await writeFile(f.file, csvReport([row])); + const parsed = await readVendorFindings(f.file); + expect(parsed.excluded).toEqual([]); + expect(parsed.findings[0]!.source_finding_id).toBe(csvRow["ID"]!); + expect(parsed.findings[0]!.evidence.source_data!["row"]).toEqual(row); + expect( + Object.getPrototypeOf(parsed.findings[0]!.evidence.source_data!["row"]), + ).toBe(Object.prototype); +}); + +test.each([ + { + Severity: "", + VendorSeverity: "MEDIUM", + CVSSSeverity: "HIGH", + expected: "medium", + }, + { Severity: "", VendorSeverity: "", CVSSSeverity: "LOW", expected: "low" }, + { + Severity: "Informational", + VendorSeverity: "", + CVSSSeverity: "", + expected: "informational", + }, +])( + "Wiz CSV uses the reported severity fallback %j", + async ({ expected, ...severity }) => { + const f = await fixture({}); + await writeFile(f.file, csvReport([{ ...csvRow, ...severity }])); + const parsed = await readVendorFindings(f.file); + expect(parsed.findings[0]!.evidence.severity).toBe(expected); + }, +); + +test.each([ + { ID: "" }, + { AssetID: "" }, + { DetailedName: "" }, + { Severity: "not-a-severity" }, + { CloudPlatform: "AWS" }, + { CloudProviderURL: "https://github.com/example/project" }, + { CloudProviderURL: "not a URL" }, + { LocationPath: "/../outside/package.json" }, + { LocationPath: "C:\\outside\\package.json" }, +] as Record[])( + "Wiz CSV excludes a malformed or unsupported occurrence without losing other rows: %j", + async (change) => { + const f = await fixture({}); + await writeFile( + f.file, + csvReport([csvRow, { ...csvRow, ID: "bad-row", ...change }]), + ); + const parsed = await readVendorFindings(f.file); + expect(parsed.read).toBe(2); + expect(parsed.findings.map((finding) => finding.source_finding_id)).toEqual( + [csvRow["ID"]!], + ); + expect(parsed.excluded).toHaveLength(1); + expect(parsed.excluded[0]!.position).toBe(2); + }, +); + +test("Wiz CSV accepts an empty report and never uses an advisory as occurrence identity", async () => { + const f = await fixture({}); + await writeFile(f.file, csvReport([])); + expect(await readVendorFindings(f.file)).toEqual({ + read: 0, + findings: [], + excluded: [], + }); + await writeFile( + f.file, + csvReport([csvRow, { ...csvRow, ID: "csv-occurrence-2" }]), + ); + expect( + (await readVendorFindings(f.file)).findings.map( + (finding) => finding.source_finding_id, + ), + ).toEqual(["csv-occurrence-1", "csv-occurrence-2"]); + await writeFile(f.file, csvReport([csvRow, csvRow])); + await expect(readVendorFindings(f.file)).rejects.toThrow( + "Duplicate source finding id", + ); +}); + +test.each([ + "duplicate header", + "short row", + "long row", + "unterminated quote", + "invalid UTF-8", +])( + "Wiz CSV rejects a damaged report before publication (%s)", + async (damage) => { + const f = await cloudFixture({}); + let contents = csvReport([csvRow]); + if (damage === "duplicate header") + contents = csvReport([csvRow], [...Object.keys(csvRow), "ID"]); + if (damage === "short row") contents = csvReport([]) + '\r\n"short"'; + if (damage === "long row") contents += ',"extra"'; + if (damage === "unterminated quote") contents = contents.slice(0, -1); + await writeFile( + f.file, + damage === "invalid UTF-8" + ? Buffer.concat([Buffer.from(contents), Buffer.from([0xff])]) + : contents, + ); + await expect( + prepareExternalPublication(f.file, f.options, f.deps), + ).rejects.toThrow(); + expect(f.posts).toEqual([]); + }, +); + +test.each([false, true])( + "Wiz CSV binds the source repository before uploads, including normalized input (%p)", + async (normalized) => { + const f = await cloudFixture({}); + await writeFile( + f.file, + csvReport([ + { + ...csvRow, + CloudProviderURL: "https://github.com/other/project/tree/main", + }, + ]), + ); + if (normalized) + await writeFile( + f.file, + JSON.stringify((await readVendorFindings(f.file)).findings), + ); + await expect( + prepareExternalPublication(f.file, f.options, f.deps), + ).rejects.toThrow("does not match"); + expect(f.posts).toEqual([]); + expect(f.calls.filter((call) => call.includes("/source_reports"))).toEqual( + [], + ); + }, +); + +for (const normalized of [false, true]) { + test.each([ + [ + "https://github.example.test:443", + "https://github.example.test:443", + true, + ], + ["http://github.example.test:80", "http://github.example.test:80", true], + [ + "https://github.example.test:8443", + "https://github.example.test:8443", + true, + ], + ["https://github.example.test:443", "https://github.example.test", false], + ["http://github.example.test:80", "http://github.example.test", false], + ["https://github.example.test", "https://github.example.test:443", false], + ["http://github.example.test", "http://github.example.test:80", false], + ])( + `Wiz CSV preserves repository ports in ${normalized ? "normalized JSONL" : "native CSV"}: %s → %s`, + async (source, destination, matches) => { + const f = await cloudFixture({}); + f.destination.url = `${destination}/example/project`; + const options = { ...f.options, repository: f.destination.id }; + const rows = ["csv-occurrence-1", "csv-occurrence-2"].map((ID) => ({ + ...csvRow, + ID, + CloudProviderURL: `${source}/example/project/tree/main`, + })); + await writeFile(f.file, csvReport(rows)); + if (normalized) + await writeFile( + f.file, + (await readVendorFindings(f.file)).findings + .map((finding) => JSON.stringify(finding)) + .join("\n"), + ); + if (matches) { + const prepared = await prepareExternalPublication( + f.file, + options, + f.deps, + ); + expect(prepared.preview.findings).toHaveLength(2); + expect(prepared.preview.excluded).toEqual([]); + } else { + await expect( + prepareExternalPublication(f.file, options, f.deps), + ).rejects.toThrow("does not match"); + expect( + f.calls.filter((call) => call.includes("/source_reports")), + ).toEqual([]); + } + expect(f.posts).toEqual([]); + }, + ); +} + +test("Wiz CSV retries retain acknowledged requests across column reordering and repeated imports stay unchanged", async () => { + const f = await cloudFixture({}); + const rows = [csvRow, { ...csvRow, ID: "csv-occurrence-2" }]; + await writeFile(f.file, csvReport(rows)); + const original = await prepareExternalPublication(f.file, f.options, f.deps); + expect(f.posts).toEqual([]); + f.state.failReadback = true; + await expect(original.publish()).rejects.toBeInstanceOf( + ExternalPublicationError, + ); + await writeFile(f.file, csvReport(rows, Object.keys(csvRow).reverse())); + const resumed = await prepareExternalPublication(f.file, f.options, f.deps); + expect(resumed.preview.resumed).toBe(true); + expect(resumed.preview.requests).toEqual(original.preview.requests); + f.state.failReadback = false; + expect((await resumed.publish()).verified).toBe(2); + expect(f.posts).toHaveLength(1); + const repeated = await prepareExternalPublication(f.file, f.options, f.deps); + const result = await repeated.publish(); + expect(result.verified).toBe(2); + expect(result.receipts[0]!.counts.unchanged).toBe(2); + expect([...f.reports.values()].map((report) => report.version)).toEqual([ + 1, 1, + ]); + expect(f.posts[0]!.items[0]!.evidence.source_data!["row"]).toEqual(csvRow); +}); + test("saved typed requests resume by repository ID after a repository rename", async () => { const f = await cloudFixture(envelope("sastFindings", [sast])); const options = { ...f.options, repository: f.destination.id }; @@ -1028,58 +1365,89 @@ test("unidentified source hosts still require exact supplied and inventory URL p expect(parsed.excluded[0]!.reason).toContain("conflicting repository URLs"); }); -test("Node reads a large complete named collection without a function argument limit", async () => { - const count = 150_000; - const records = Array.from({ length: count }, (_, index) => ({ - id: `vendor-${index}`, - name: "CVE-2099-0001", - detailedName: "example-package", - severity: "HIGH", - vulnerableAsset: { id: "synthetic-asset" }, - })); - const f = await fixture(envelope("vulnerabilityFindings", records, false)); - // Run the actual parser under Node; Bun does not have the same argument limit. - const bundle = await mkdtemp(join(import.meta.dir, "..", ".wiz-parser-")); - directories.push(bundle); - const runner = join(bundle, "large-input.mts"); - await writeFile( - runner, - ` - import { readVendorFindings } from "../src/wiz-findings.js"; - const parsed = await readVendorFindings(process.argv[2]!); +test.each(["named collection", "normalized JSONL"])( + "Node reads a large %s without parsing limits", + async (format) => { + const count = 150_000; + const records = Array.from({ length: count }, (_, index) => ({ + id: `vendor-${index}`, + name: "CVE-2099-0001", + detailedName: "example-package", + severity: "HIGH", + vulnerableAsset: { id: "synthetic-asset" }, + })); + const f = await fixture(envelope("vulnerabilityFindings", records, false)); + if (format === "normalized JSONL") + await writeFile( + f.file, + records + .map((record) => + JSON.stringify({ + source_finding_id: record.id, + evidence: { + title: "Synthetic package vulnerability", + severity: "high", + source_data: { + ...record, + vendorSeverity: "HIGH", + description: "Synthetic package vulnerability evidence.", + }, + }, + }), + ) + .join("\n"), + ); + // Run the actual parser under Node; Bun does not have the same argument limit. + const bundle = await mkdtemp(join(import.meta.dir, "..", ".wiz-parser-")); + directories.push(bundle); + const runner = join(bundle, "large-input.mts"); + await writeFile( + runner, + ` + import { readVendorFindings, readVendorFindingsForPublication } from "../src/wiz-findings.js"; + const parsed = process.argv[3] === "normalized JSONL" + ? (await readVendorFindingsForPublication(process.argv[2]!)).current + : await readVendorFindings(process.argv[2]!); console.log(JSON.stringify({ read: parsed.read, ready: parsed.findings.length, excluded: parsed.excluded.length, first: parsed.findings[0]?.source_finding_id, last: parsed.findings.at(-1)?.source_finding_id, })); `, - ); - const built = await Bun.build({ - entrypoints: [runner], - outdir: bundle, - target: "node", - format: "esm", - packages: "external", - }); - expect(built.success).toBe(true); - const child = Bun.spawn( - [Bun.which("node")!, built.outputs[0]!.path, f.file], - { stdout: "pipe", stderr: "pipe" }, - ); - const [exitCode, stdout, stderr] = await Promise.all([ - child.exited, - new Response(child.stdout).text(), - new Response(child.stderr).text(), - ]); - expect(exitCode, stderr).toBe(0); - expect(JSON.parse(stdout)).toEqual({ - read: count, - ready: count, - excluded: 0, - first: "vendor-0", - last: `vendor-${count - 1}`, - }); -}); + ); + const built = await Bun.build({ + entrypoints: [runner], + outdir: bundle, + target: "node", + format: "esm", + packages: "external", + }); + expect(built.success).toBe(true); + const child = Bun.spawn( + [ + Bun.which("node")!, + ...(format === "normalized JSONL" ? ["--max-old-space-size=256"] : []), + built.outputs[0]!.path, + f.file, + format, + ], + { stdout: "pipe", stderr: "pipe" }, + ); + const [exitCode, stdout, stderr] = await Promise.all([ + child.exited, + new Response(child.stdout).text(), + new Response(child.stderr).text(), + ]); + expect(exitCode, stderr).toBe(0); + expect(JSON.parse(stdout)).toEqual({ + read: count, + ready: count, + excluded: 0, + first: "vendor-0", + last: `vendor-${count - 1}`, + }); + }, +); test.each(["migrate", "distinct", "duplicate"])( "Unicode readback resumes historical checkpoints (%s)",