Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,7 @@
/requirements-grpc-smg.txt @NVIDIA/trt-llm-oss-compliance
/requirements-openengine.txt @NVIDIA/trt-llm-oss-compliance
/requirements.txt @NVIDIA/trt-llm-oss-compliance
/scripts/vendor_sources.py @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance
/scripts/vendor/ @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance
/setup.py @NVIDIA/trt-llm-oss-compliance
/tensorrt_llm/_torch/attention/backends/prims_ts/** @NVIDIA/trt-llm-torch-attention-devs @NVIDIA/trt-llm-oss-compliance
/tests/unittest/api_stability/ @NVIDIA/trt-llm-noncommitted-api-review-committee
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1597,7 +1597,7 @@ repos:
pass_filenames: false
- id: vendor-sources-check
name: verify vendored sources are in sync
entry: python scripts/vendor_sources.py check
entry: python scripts/vendor/manage.py check
language: python
additional_dependencies: [PyYAML]
always_run: true
Expand Down
68 changes: 47 additions & 21 deletions 3rdparty/vendor-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ the selected files, any persistent compatibility patch and its content digest,
and a digest of the materialized destination. A short branch or tag may explain
where the commit came from, but the full commit is authoritative.

Use `scripts/vendor_sources.py` for every lock or vendor-state change. Do not
Use `scripts/vendor/manage.py` for every lock or vendor-state change. Do not
edit the YAML, generated patches, or digests by hand. All examples below use the
default lock. For an isolated test or another consumer repository, place
`--lock PATH` before the subcommand.
Expand Down Expand Up @@ -91,10 +91,10 @@ existing compatibility patch, then use `pin`.
List entries, or run the offline integrity status for all or one vendor:

```bash
python scripts/vendor_sources.py list
python scripts/vendor_sources.py status
python scripts/vendor_sources.py status VENDOR
python scripts/vendor_sources.py check VENDOR
python scripts/vendor/manage.py list
python scripts/vendor/manage.py status
python scripts/vendor/manage.py status VENDOR
python scripts/vendor/manage.py check VENDOR
```

`status` and the default `check` exit unsuccessfully if the destination has a
Expand All @@ -107,7 +107,7 @@ When neither the lock entry nor destination exists, create both from an
immutable commit and a local upstream checkout:

```bash
python scripts/vendor_sources.py create VENDOR \
python scripts/vendor/manage.py create VENDOR \
--url https://example.com/organization/repository.git \
--branch main \
--commit FULL_COMMIT \
Expand All @@ -124,7 +124,7 @@ If the destination already exists but has no lock entry, adopt it. Use `exact`
to require an exact upstream match:

```bash
python scripts/vendor_sources.py create VENDOR \
python scripts/vendor/manage.py create VENDOR \
--url https://example.com/organization/repository.git \
--commit FULL_COMMIT \
--source path/in/upstream \
Expand All @@ -143,7 +143,7 @@ Discard destination edits and reproduce the currently locked upstream commit
plus its persistent patch:

```bash
python scripts/vendor_sources.py sync VENDOR --repo /path/to/upstream
python scripts/vendor/manage.py sync VENDOR --repo /path/to/upstream
```

This overwrites the selected destination files. It does not update the lock,
Expand All @@ -155,21 +155,21 @@ After editing an exact destination for a change that must remain downstream,
create its persistent patch:

```bash
python scripts/vendor_sources.py patch VENDOR create --repo /path/to/upstream
python scripts/vendor/manage.py patch VENDOR create --repo /path/to/upstream
```

After intentionally changing an already patched destination, regenerate the
patch:

```bash
python scripts/vendor_sources.py patch VENDOR refresh --repo /path/to/upstream
python scripts/vendor/manage.py patch VENDOR refresh --repo /path/to/upstream
```

Drop a no-longer-needed patch only after the destination exactly matches the
currently locked upstream selection:

```bash
python scripts/vendor_sources.py patch VENDOR drop --repo /path/to/upstream
python scripts/vendor/manage.py patch VENDOR drop --repo /path/to/upstream
```

Generated patches live under `3rdparty/vendor_patches/`. Review them, but update
Expand All @@ -184,7 +184,7 @@ branch at the currently locked commit **before** exporting:

```bash
git -C /path/to/upstream switch -c trtllm-vendor-fix LOCKED_FULL_COMMIT
python scripts/vendor_sources.py export VENDOR --repo /path/to/upstream
python scripts/vendor/manage.py export VENDOR --repo /path/to/upstream
```

The upstream checkout's selected source must be clean before export and its
Expand All @@ -205,7 +205,7 @@ git -C /path/to/upstream push -u origin trtllm-vendor-fix
Finally, pin the committed revision from that checkout:

```bash
python scripts/vendor_sources.py pin VENDOR \
python scripts/vendor/manage.py pin VENDOR \
--url https://example.com/my-fork/repository.git \
--branch trtllm-vendor-fix \
--commit NEW_FULL_COMMIT \
Expand Down Expand Up @@ -236,7 +236,7 @@ Remove a lock entry and its generated compatibility patch while preserving the
destination:

```bash
python scripts/vendor_sources.py remove VENDOR
python scripts/vendor/manage.py remove VENDOR
```

The preserved destination is no longer protected by the lock. Delete or move
Expand All @@ -247,8 +247,8 @@ it separately as part of the reviewed migration that removes the vendor.
The default check is deliberately offline:

```bash
python scripts/vendor_sources.py check
python scripts/vendor_sources.py check --offline
python scripts/vendor/manage.py check
python scripts/vendor/manage.py check --offline
```

It validates the lock schema and path safety, patch metadata, and the checked-in
Expand All @@ -260,22 +260,22 @@ When network access is available, attempt verification against every recorded
upstream:

```bash
python scripts/vendor_sources.py check --upstream
python scripts/vendor/manage.py check --upstream
```

An inaccessible repository is reported as unavailable rather than failing. If
a commit can be obtained, a source, patch, or destination mismatch is an error.
Trusted maintainer CI can require access to every source:

```bash
python scripts/vendor_sources.py check --upstream --require-access
python scripts/vendor/manage.py check --upstream --require-access
```

To verify one vendor against an existing checkout without contacting the
recorded URL, provide it explicitly:

```bash
python scripts/vendor_sources.py check VENDOR --repo /path/to/upstream
python scripts/vendor/manage.py check VENDOR --repo /path/to/upstream
```

The checkout's configured remote may differ from the lock URL; it only needs to
Expand All @@ -289,6 +289,32 @@ local repository, and URL or commit changes require vendor CODEOWNER review.
Never put credentials in a lock URL. Run checks that use internal credentials
only in a trusted environment, not with pull-request-controlled scripts.

## Promotion and local monitoring

The generic toolkit under `scripts/vendor/` supports every vendor key in this
lock. `promote.py` prepares a verified lock-only follow-up after a source-update
PR merges; `bot.py` can monitor PRs locally, request missing contributor metadata,
match upstream commits, and publish promotions with squash auto-merge.

See the [toolkit guide](../scripts/vendor/README.md) for the workflow diagram,
contributor description format, author resolution of ambiguous matches,
authentication, configurable private workdirs, daemon launch/stop, and recovery.
No GitHub App or systemd setup is required. All remote writes are opt-in with
`--publish`; operator settings and credentials are not committed to this repo.

Each monitor handles one vendor and consumer base branch. Main and release
branches can use different canonical source branches, but require separate
workdirs and explicit launches. A periodic upstream refresh remains a separate
code-changing PR with normal CI: stop its monitor or configure an audited
`--refresh-branch` target before opening it, then migrate the canonical branch
in the existing workdir after merge. See the toolkit guide's
[base-branch setup](../scripts/vendor/README.md#different-consumer-base-branches)
and [refresh cutover](../scripts/vendor/README.md#switching-canonical-branches-after-a-refresh).

Use the entry points under `scripts/vendor/` directly. The old top-level scripts
and PrimTS-specific CLI aliases are not retained; callers must migrate to the
generic commands and supply their vendor and repository configuration explicitly.

## License and attribution

The vendor lock is a reproducibility record, not a license manifest. Before
Expand All @@ -310,6 +336,6 @@ files remain exact upstream copies.
Use the normal commands with `flashinfer-prims-ts`, for example:

```bash
python scripts/vendor_sources.py status flashinfer-prims-ts
python scripts/vendor_sources.py check flashinfer-prims-ts
python scripts/vendor/manage.py status flashinfer-prims-ts
python scripts/vendor/manage.py check flashinfer-prims-ts
```
7 changes: 7 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,13 @@ The bot-to-bot semantic review workflow is disabled. See
[.github/semantic-review.md](.github/semantic-review.md) for its current status and
historical implementation details.

### Vendor source tools and promotion

Use the tools under `scripts/vendor/` to manage locked vendor sources, prepare
lock-only promotions, and optionally monitor source-update PRs. Read
[Vendored Sources](3rdparty/vendor-sources.md) for commands, contributor metadata,
and operator guidance before changing vendor state or publishing promotions.

### Triggering CI

CI is triggered by posting comments on the PR. Basic commands:
Expand Down
Loading
Loading