diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 49c023bc..34557b66 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -58,6 +58,7 @@ jobs: sparse-checkout: | docs/design docs/perf + docs/stability.md src/tileops scripts/nightly_report.py sparse-checkout-cone-mode: false diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index f1cb7077..784ef792 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -26,6 +26,7 @@ jobs: sparse-checkout: | docs/design docs/perf + docs/stability.md src/tileops scripts/nightly_report.py sparse-checkout-cone-mode: false diff --git a/.github/workflows/render-benchmarks.yml b/.github/workflows/render-benchmarks.yml index 4223d955..a4224312 100644 --- a/.github/workflows/render-benchmarks.yml +++ b/.github/workflows/render-benchmarks.yml @@ -40,6 +40,7 @@ jobs: sparse-checkout: | docs/design docs/perf + docs/stability.md src/tileops scripts/nightly_report.py sparse-checkout-cone-mode: false diff --git a/CLAUDE.md b/CLAUDE.md index 5e61cc72..f4a7b491 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,7 +88,7 @@ Everything else goes the other way. |------|--------| | Coverage | Whichever pages have a `.zh.md` — `ls docs/**/*.zh.md` | | Never translate | `api/` and `benchmarks/`, both generated; `design/`, mirrored English | -| Mirrored, translated | `performance-guides/trace-timeline.md` mirrors TileOPs `docs/perf/trace-timeline.md`; its `.zh.md` is a full translation. When upstream changes that file, update the translation in step | +| Mirrored, translated | `performance-guides/trace-timeline.md` mirrors TileOPs `docs/perf/trace-timeline.md`, and `stability.md` mirrors TileOPs `docs/stability.md`; each `.zh.md` is a full translation. When upstream changes either file, update the translation in step | | Missing translation | Falls back to English at the same URL, and `hooks.py` prepends a "本页暂无中文版" notice. The fallback runs zh → en only: a page that exists only as `.zh.md` leaves its `nav` entry on a missing file and the English sidebar renders a dead link | | Figures | A figure with text needs one SVG per language: `img/.svg` for English and `img/.zh.svg` beside it, which the `zh` build picks up for the same reference. Translate the `` nodes and the `aria-label`, keep the geometry. English runs longer than Chinese — grow the `viewBox` rather than let text overflow. The user-guide figures are drawn from sources under `figures/user-guide/` (`.zh.puml`, `.en.puml`, `manifest/overview.py`): edit the source, then run `figures/user-guide/render.sh` | | Nav labels | `nav_translations` in the `i18n` plugin block; keep an entry for every `nav` title | diff --git a/docs/api/attention.md b/docs/api/attention.md index f731fb70..755ed0df 100644 --- a/docs/api/attention.md +++ b/docs/api/attention.md @@ -27,6 +27,12 @@ what runs when you call `op(...)`. heading_level: 3 members: ["__init__", "forward"] +::: tileops.attention.GQADenseWithLSEFwdOp + options: + show_root_heading: true + heading_level: 3 + members: ["__init__", "forward"] + ::: tileops.attention.GQAPagedFwdOp options: show_root_heading: true diff --git a/docs/api/index.md b/docs/api/index.md index 374bd8e0..68568795 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -49,6 +49,8 @@ Two things this reference does not carry: - **How fast it is.** Device time against the fastest alternative on each workload is on the [Benchmarks](../benchmarks/index.md) pages. +Which names are public, and what a release may change, is on [API Stability](../stability.md). + These pages are generated from the docstrings in TileOPs, so an op whose docstring is thin reads thin here. The fix belongs upstream, in [`src/tileops/_ops/`](https://github.com/tile-ai/TileOPs/tree/main/src/tileops/_ops). diff --git a/docs/api/moe.md b/docs/api/moe.md index 766a3d49..db52f80b 100644 --- a/docs/api/moe.md +++ b/docs/api/moe.md @@ -91,3 +91,25 @@ the routes are few enough to read the weights once per route rather than once pe show_root_heading: true heading_level: 3 members: ["__init__", "forward"] + +## Layout and epilogue specs + +`MoEPrePermuteFwdOp`, `MoEGroupedGEMMFwdOp`, `MoEExpertMLPFwdOp` and +`MoEPostPermuteFwdOp` take a `layout`, a `ContiguousLayoutSpec` or a `MaskedLayoutSpec`; +`MoEGroupedGEMMFP8FwdOp` takes a `MaskedLayoutSpec`. `MoEPostPermuteFwdOp` also takes an +`epilogue`, a `RoutingEpilogueSpec`. + +::: tileops.moe.ContiguousLayoutSpec + options: + show_root_heading: true + heading_level: 3 + +::: tileops.moe.MaskedLayoutSpec + options: + show_root_heading: true + heading_level: 3 + +::: tileops.moe.RoutingEpilogueSpec + options: + show_root_heading: true + heading_level: 3 diff --git a/docs/backends.md b/docs/backends.md index 3f41ab59..0f2303a2 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -63,8 +63,8 @@ target's devices. A composite op, which only runs sub-ops, needs no builder. ## The protocol: four functions -`tileops.backend` defines only the outward-facing interface, expressed with Python -structural typing (`typing.Protocol`). A backend subclasses no base class and implements +`tileops.backend` defines only the outward-facing interface: function signatures and a few +data types. A backend subclasses no base class and implements no abstract method; it writes plain functions with matching signatures and registers them. The op layer checks a backend's return value structurally as well, with `callable()`. @@ -171,14 +171,16 @@ Called by the backend, once per op it takes over. It registers the kernel builde def register_device_facts(device_type: str, read: Callable[[torch.device], DeviceFacts]) -> None: ... ``` -Called by the backend, once per device type. Kernel selection reads four facts about the -call's device: `arch`, `calibration`, `sm_count` and `smem_budget`. +Called by the backend, once per device type. Kernel selection reads three facts about the +call's device, the fields of `DeviceFacts`: `arch`, `sm_count` and `smem_budget`. A CUDA +device's `arch` is `major * 10 + minor`; for another device type, the backend that +registers it states how its `arch` is encoded. | Device type | Where its facts come from | | --- | --- | | `cuda` | TileOPs reads them itself | | Another registered type | The registered `read`, which takes the call's `torch.device` and returns `DeviceFacts` | -| An unregistered type, such as `cpu` | Nowhere: it reports `-1 / None / 0 / 0` | +| An unregistered type, such as `cpu` | Nowhere: it reports `arch=-1, sm_count=0, smem_budget=0` | - Facts are read by implementations that are selected by architecture, SM count or shared memory. An implementation that declares @@ -194,6 +196,8 @@ call's device: `arch`, `calibration`, `sm_count` and `smem_budget`. - `read` runs the first time a call spec's facts are read, and the result stays on that call spec. It also runs when a kernel that declares `supported_archs` is constructed. If reading is expensive, the backend caches inside `read`. +- `DeviceFacts` is constructed by keyword only. A field added in a later release comes with a + default, so code that constructs it by keyword keeps working. - If `read` raises or returns something invalid (not a `DeviceFacts`, a field of the wrong type, a negative `sm_count` or `smem_budget`), the op layer raises a `BackendError` that names the `read`. @@ -205,7 +209,7 @@ import acme_runtime # a vendor runtime, ima def read_acme(device: torch.device) -> DeviceFacts: props = acme_runtime.device_properties(device.index) - return DeviceFacts(arch=props.arch, calibration=None, + return DeviceFacts(arch=props.arch, # encoded as this backend states sm_count=props.compute_units, smem_budget=props.shared_memory_per_block) @@ -215,14 +219,16 @@ register_device_facts("acme", read_acme) # "acme" is torch's dev ### `TensorSpec` ```python -class TensorSpec(NamedTuple): +@dataclasses.dataclass(frozen=True, kw_only=True) +class TensorSpec: device: torch.device dtype: torch.dtype shape: tuple[int, ...] ``` Defined by the protocol, built by the op layer and passed into `build_kernel`. It describes -a tensor's properties and does not contain the tensor. +a tensor's properties and does not contain the tensor. It is constructed by keyword only; +`TensorSpec.of(tensor)` describes a tensor. ```python # what a build_kernel argument looks like @@ -285,8 +291,8 @@ d = op(a, b) # every input on one device: a.device == b.devi # a later call at the same device, dtypes and shapes hits and jumps to the last step # ── backend: the op layer calls build_gemm, with TensorSpecs, not tensors ── -# build_gemm(TensorSpec("acme:0", float16, (4096, 4096)), -# TensorSpec("acme:0", float16, (4096, 4096)), +# build_gemm(TensorSpec(device=torch.device("acme:0"), dtype=float16, shape=(4096, 4096)), +# TensorSpec(device=torch.device("acme:0"), dtype=float16, shape=(4096, 4096)), # trans_a=False, trans_b=True) # params by their manifest names # → returns something callable diff --git a/docs/backends.zh.md b/docs/backends.zh.md index 8b470610..f2018627 100644 --- a/docs/backends.zh.md +++ b/docs/backends.zh.md @@ -48,7 +48,7 @@ TileLang 是支持多种 backend 的 DSL,每种硬件各有一套独立的 ker ## 协议中的四个函数 -`tileops.backend` 只定义对外接口,用 Python 的结构化类型(`typing.Protocol`)表达:backend 不继承基类,也不实现抽象方法,只需写出签名相符的普通函数并注册。op 层检查返回值时同样只看结构,即 `callable()`。 +`tileops.backend` 只定义对外接口,即函数的签名与几个数据类型:backend 不继承基类,也不实现抽象方法,只需写出签名相符的普通函数并注册。op 层检查返回值时同样只看结构,即 `callable()`。 backend 实现 `detect` 与 `build_kernel`,再调用 `register_detector` 与 `register_kernel_builder` 登记它们。此外还有协议定义的 `TensorSpec`,以及 `pyproject.toml` 中的一条 entry point: @@ -138,13 +138,13 @@ def register_kernel_builder(op: str, target: str, build_kernel: BuildKernel) -> def register_device_facts(device_type: str, read: Callable[[torch.device], DeviceFacts]) -> None: ... ``` -由 backend 调用,每种设备类型一次。kernel 选择读取调用设备的四项设备事实:`arch`、`calibration`、`sm_count`、`smem_budget`。 +由 backend 调用,每种设备类型一次。kernel 选择读取调用设备的三项设备事实,即 `DeviceFacts` 的三个字段:`arch`、`sm_count`、`smem_budget`。CUDA 设备的 `arch` 是 `major * 10 + minor`;其他设备类型的 `arch` 如何编码,由注册它的 backend 写明。 | 设备类型 | 设备事实的来源 | | --- | --- | | `cuda` | TileOPs 自己读取 | | 已注册的其他类型 | 注册的 `read`:收到调用的 `torch.device`,返回 `DeviceFacts` | -| 未注册的类型,如 `cpu` | 没有设备事实:`-1 / None / 0 / 0` | +| 未注册的类型,如 `cpu` | 没有设备事实:`arch=-1, sm_count=0, smem_budget=0` | - 读取设备事实的是按架构、SM 数或共享内存选择的实现。声明了 `supported_archs` 的实现在未注册的设备类型上永远不可用。 - 只服务整个 op 的 target 不需要注册:target 的 kernel 自己决定怎么选。 @@ -152,6 +152,7 @@ def register_device_facts(device_type: str, read: Callable[[torch.device], Devic - `arch` 只在同一种设备类型内可比,所以声明 `supported_archs` 的实现,其 `devices` 只能有一种设备类型。 - `read` 收到的设备:设备是进程 accelerator 的类型时,op 层先补上设备序号;`cpu` 这类不编号的设备原样传入。 - `read` 的调用时机:每个 call spec 第一次读设备事实时调用一次,结果只存在这个 call spec 上;声明了 `supported_archs` 的 kernel 构造时也调用一次。开销大时由 backend 在 `read` 内缓存。 +- `DeviceFacts` 只能按关键字构造。之后的版本新增字段时会带默认值,按关键字构造的代码不受影响。 - `read` 抛异常或返回值不合规(不是 `DeviceFacts`、字段类型不对、`sm_count` 或 `smem_budget` 为负)时,op 层报 `BackendError`,指出是哪个 `read`。 ```python @@ -161,7 +162,7 @@ import acme_runtime # 厂商 runtime,这 def read_acme(device: torch.device) -> DeviceFacts: props = acme_runtime.device_properties(device.index) - return DeviceFacts(arch=props.arch, calibration=None, + return DeviceFacts(arch=props.arch, # 编码方式由本 backend 写明 sm_count=props.compute_units, smem_budget=props.shared_memory_per_block) @@ -171,13 +172,14 @@ register_device_facts("acme", read_acme) # "acme" 是 torch 的 ### `TensorSpec` ```python -class TensorSpec(NamedTuple): +@dataclasses.dataclass(frozen=True, kw_only=True) +class TensorSpec: device: torch.device dtype: torch.dtype shape: tuple[int, ...] ``` -协议定义的类型,由 op 层构造后传入 `build_kernel`。它描述一个张量的属性,不包含张量本身。 +协议定义的类型,由 op 层构造后传入 `build_kernel`。它描述一个张量的属性,不包含张量本身。它只能按关键字构造;`TensorSpec.of(tensor)` 由一个张量得到它的描述。 ```python # build_kernel 收到的实参 @@ -234,8 +236,8 @@ d = op(a, b) # 所有输入必须在同一设备上:a.devi # 设备、dtype 与形状都相同的下一次调用命中,直接进入最后一步 # ── backend:op 层调用 build_gemm,张量已转为 TensorSpec ─────────────── -# build_gemm(TensorSpec("acme:0", float16, (4096, 4096)), -# TensorSpec("acme:0", float16, (4096, 4096)), +# build_gemm(TensorSpec(device=torch.device("acme:0"), dtype=float16, shape=(4096, 4096)), +# TensorSpec(device=torch.device("acme:0"), dtype=float16, shape=(4096, 4096)), # trans_a=False, trans_b=True) # params 按 manifest 中的名字传入 # → 返回一个可调用对象 diff --git a/docs/performance-guides/trace-timeline.zh.md b/docs/performance-guides/trace-timeline.zh.md index 9429e28a..0147dded 100644 --- a/docs/performance-guides/trace-timeline.zh.md +++ b/docs/performance-guides/trace-timeline.zh.md @@ -69,6 +69,8 @@ def build_gemm(m, n, k, dtype="float16", traced=False): 1. [`trace.dag(src, dst)`](../api/trace.md#tileops.trace._api._Trace.dag) 声明从一个命名区间指向另一个区间的依赖箭头(`arrive` → `wait`),每出现一次画一条。 1. [`trace.finalize(func, traced, max_events)`](../api/trace.md#tileops.trace._api._Trace.finalize) 在 `traced` 时 lower 标记并加上 `slots` 输出,否则把标记 strip 成零开销。`traced` **必须是构建函数缓存键的一部分**,这样同一形状下带追踪和不带追踪的两次构建才不会冲突。 +标记把名字记入按线程保存的注册表,`trace.finalize` 读取并清空它。多个线程可以同时构建带追踪的 kernel。一个线程同一时间只构建一个带追踪的 kernel:在一个线程上 finalize 第一个之前就定义第二个,不受支持。 + ## 运行带追踪的 kernel { #running-a-traced-kernel } 调用构建函数时传入 `traced=trace.enabled`,再把编译好的 kernel 交给 `trace.run`。同一个 `forward` 在两种模式下都能使用:追踪关闭时原样返回输出;追踪开启时写出时间线,并且只返回真正的输出。因此调用方不需要自己按开关分支。 diff --git a/docs/stability.md b/docs/stability.md new file mode 100644 index 00000000..d02b094d --- /dev/null +++ b/docs/stability.md @@ -0,0 +1,3 @@ +{% + include-markdown "../TileOPs/docs/stability.md" +%} diff --git a/docs/stability.zh.md b/docs/stability.zh.md new file mode 100644 index 00000000..066b9f1d --- /dev/null +++ b/docs/stability.zh.md @@ -0,0 +1,50 @@ +# API 稳定性 + +本页规定 TileOPs 支持哪些公开 API,以及一次发布可以改动什么。 + +## 公开模块 + +| 模块 | 包含的内容 | +| --- | --- | +| `tileops` | 各 family 模块、`Op`,以及它重新导出的 backend 名字。 | +| `tileops.` | 一个 family 的 op,以及这些 op 的构造函数接收的值类型(例如 `tileops.moe.MaskedLayoutSpec`)。 | +| `tileops.backend` | target 选择,以及 in-tree 之外的 kernel 的登记。 | +| `tileops.authoring` | `Op`、`Kernel`、`KernelInterface`、`CallSpec`、`Entry`、`SignatureCall`:TileOPs 之外的代码编写新 op 所用的类。 | +| `tileops.manifest` | 它的 `__all__` 中的名字、`tileops.manifest.primitives` 模块,以及 spec 格式。 | +| `tileops.trace` | kernel 内的时间线追踪工具。 | + +一个名字列在某个公开模块的 `__all__` 中,它就是公开的。 +名字以下划线开头的模块、函数、类、属性或方法都是内部的,包括 `tileops._ops`、`tileops._kernels`、`tileops._utils` 与 `tileops._perf`。 +内部名字在任何一次发布中都可能改动,不另行通知。 + +`tileops.elementwise.HardtanhFwdOp` 这类具体 op 公开的是构造与调用,不支持继承。 + +## 兼容性 + +在 0.x 版本内,minor 与 patch 发布对公开 API 只做增加: + +- 公开的名字、签名、参数默认值、返回结构与文档写明的行为保持不变。 +- 一次发布可以增加模块、名字、可选参数,或新接受的输入。 +- 其他改动等到 major 发布。 + +`scripts/ci/check_api_compat.py` 在每个 PR 上对照最新发布做检查,一个调用若按最新发布写成、在 PR 上会被破坏,检查就失败。 + +## 支持范围 + +满足以下条件的调用受支持: + +- 一个 op 实例同一时间只被一个线程、一个 CUDA stream 使用。 +- 一个实例的 CUDA graph 回放不与该实例的 eager 调用重叠。 +- 编译出的可调用对象(`torch.compile` 的输出)的生命周期不超过它调用的 op 实例。 +- 调用不写入的输入是连续的,op 另有说明的除外。 +- `Bwd` op 直接调用,不接入 `torch.autograd`。 + +## 在 TileOPs 之外编写 op + +TileOPs 之外的代码继承 `tileops.authoring.Op` 来编写新 op: + +- 类以类属性 `spec` 给出它的 spec,格式与 manifest 条目相同([manifest.md](design/manifest.md))。 +- `__init__` 先赋值 spec 中的参数,再调用 `super().__init__(target=target)`。 +- 类在 `kernel_types` 中声明它的 `Kernel` 子类,在 `interfaces` 中声明它的 `KernelInterface` 调用位置,并编写 `forward`,用 `kernel_for(interface, call)` 取得 kernel。 + +之后像内置 op 一样以 `op(...)` 调用。 diff --git a/docs/user-guide/development.md b/docs/user-guide/development.md index b83962b3..2c87cc1b 100644 --- a/docs/user-guide/development.md +++ b/docs/user-guide/development.md @@ -122,16 +122,19 @@ pre-commit run --all-files # lint, the same as CI's p ## Running benchmarks { #bench } ```bash -PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev,bench]' -c constraints.txt +PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev]' --group bench -c constraints.txt python -m pytest -q ``` - A PR that changes a kernel or an op includes benchmark results, compared against an implementation outside TileOPs. -- The baseline libraries install through the `bench` extra. The dev image installs - them at build time too; a baseline that fails to install there does not stop the - build, and `sgl-kernel` is not in the image. A baseline missing from the - container is installed at the version the repository declares. +- The baseline libraries install through the `bench` dependency group; `--group` needs + pip 25.1 or later. FlashAttention-3 (`flash_attn_interface`) is not on PyPI and is not + in the group: build it from the FlashAttention repository's `hopper/` directory, or use + the dev image. +- The dev image installs the baseline libraries at build time too. A failed install of + some of them does not stop the build, and `sgl-kernel` is not in the image. A baseline + missing from the container is installed at the version the repository declares. - With `--tileops-verify`, a benchmark only checks correctness and times nothing, which is a quick way to confirm the numerics after changing a kernel. - How to write a benchmark file is in [Writing benchmarks](benchmark/writing.md); how the diff --git a/docs/user-guide/development.zh.md b/docs/user-guide/development.zh.md index 1fea002a..d9ff0c25 100644 --- a/docs/user-guide/development.zh.md +++ b/docs/user-guide/development.zh.md @@ -102,12 +102,13 @@ pre-commit run --all-files # 代码检查,与 CI ## 运行 benchmark { #bench } ```bash -PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev,bench]' -c constraints.txt +PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev]' --group bench -c constraints.txt python -m pytest -q ``` - 改动 kernel 或 op 的 PR 需附上 benchmark 结果,对比对象是 TileOPs 以外的实现。 -- benchmark 的对比库通过 `bench` extra 安装。dev 镜像在构建时也会安装这些库,其中个别库安装失败不会中止构建,`sgl-kernel` 则不在镜像中;容器内缺少的库按仓库声明的版本另行安装。 +- benchmark 的对比库通过 `bench` 依赖组安装,`--group` 需要 pip 25.1 及以上。FlashAttention-3(`flash_attn_interface`)不在 PyPI 上,也不在这个依赖组里:从 FlashAttention 仓库的 `hopper/` 目录编译安装,或使用 dev 镜像。 +- dev 镜像在构建时也会安装这些库,其中个别库安装失败不会中止构建,`sgl-kernel` 则不在镜像中;容器内缺少的库按仓库声明的版本另行安装。 - 加上 `--tileops-verify` 时,benchmark 只做正确性校验,不计时,适合在改动 kernel 后先确认数值。 - benchmark 文件的写法见[编写 benchmark](benchmark/writing.md),计时方式见 [benchmark 的计时方法](../timing.md)。 diff --git a/mkdocs.yml b/mkdocs.yml index 45242744..658bfa5e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -75,6 +75,7 @@ nav: # unrelated algorithms. Trace comes last — a tool rather than an op. - API Reference: - api/index.md + - API Stability: stability.md - Elementwise: api/elementwise.md - RoPE: api/rope.md - Reduction: api/reduction.md @@ -212,6 +213,7 @@ plugins: Layer Boundaries: 层间边界 Elementwise: Elementwise API Reference: API 参考 + API Stability: API 稳定性 Reduction: Reduction Normalization: Normalization GEMM: GEMM @@ -224,6 +226,7 @@ plugins: Linear Attention: Linear Attention Mamba: Mamba FFT: FFT + Transform: Transform mHC: mHC MoE: MoE Sampling: Sampling diff --git a/scripts/dev.sh b/scripts/dev.sh index 2bcfe467..e2c8225b 100755 --- a/scripts/dev.sh +++ b/scripts/dev.sh @@ -41,6 +41,7 @@ done # workflows check out. An existing ./TileOPs — a symlink to a clone elsewhere, # say — is left alone. It stays at the revision it was cloned at while CI reads # current upstream, so an op renamed there fails CI a local build had passed. +sparse_paths=(docs/design docs/perf docs/stability.md src/tileops scripts/nightly_report.py) if [ -L TileOPs ] && [ ! -e TileOPs ]; then # `-e` follows the link, so a symlink whose target is gone reads as missing, # and the clone below would fail on a path that already exists. @@ -51,10 +52,14 @@ elif [ ! -e TileOPs ]; then echo "==> cloning tile-ai/TileOPs into ./TileOPs" git clone --depth 1 --filter=blob:none --sparse \ https://github.com/tile-ai/TileOPs.git TileOPs - git -C TileOPs sparse-checkout set --no-cone \ - docs/design docs/perf src/tileops scripts/nightly_report.py + git -C TileOPs sparse-checkout set --no-cone "${sparse_paths[@]}" elif [ "$update" = 1 ]; then echo "==> updating ./TileOPs" + # A sparse clone made before a path joined the list does not have it yet. `add` keeps + # the paths already selected; a full checkout is left as it is. + if [ "$(git -C TileOPs config --get core.sparseCheckout)" = true ]; then + git -C TileOPs sparse-checkout add "${sparse_paths[@]}" + fi git -C TileOPs pull --ff-only fi