Repository navigation
feat: add auto_tune, which picks the fastest cut depth - #56
Draft
lslusarczyk wants to merge 8 commits into
Draft
lslusarczyk wants to merge 8 commits into
lslusarczyk wants to merge 8 commits into
Conversation
`tt.default_strategy(device)` is `HighestThroughputStrategy` over every backend that runs on the device and keeps the results of the model. On XPU these are Torch Inductor, XPU graph, OpenVINO, and eager. On CPU these are Torch Inductor, OpenVINO, and eager. TorchAO is left out because quantization changes the results. On an Arc Pro B70 it was the fastest or close to it on five of six models. OpenVINO wins on whole ViT and DistilBERT models and loses on small modules, so it is measured, not excluded. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
…device) A module without a strategy used one shared FirstWinsStrategy. FirstWins keeps the first backend that builds, not the fastest one. The shared instance also carried state from one module to the next. Each graph now gets its own default_strategy for the tuning device. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`auto_tune` tunes the model at every depth from `min_depth` to `max_depth` and keeps the fastest cut. Modules shared between depths are tuned once. The shallowest depth within the tie tolerance wins. The model is left untuned when no depth beats the untuned model. The default strategy does not search the maximum batch size. Modules that fail to tune keep their original forward, and a depth where every module fails is skipped. Weights that a backend moved to CPU go back to their device before the next depth. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Every parameter after `dataset` is keyword-only. `print(result)` shows the latency of every depth and the selected one. The docs say that `obj` stays untuned when the whole model wins. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`auto_tune` starts no new depth after `time_budget` seconds. The depth in progress finishes. Depths not started are reported with the error "time budget used up". Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`auto_tune` uses `tt.default_strategy(device)` when no strategy is given, without the search for the maximum batch size. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The first measurement can run before the device reaches its full clock. On an Arc Pro B70 the same untuned T5 encoder measured 2.8 ms to 8.3 ms across runs, which favored tuning. The untuned model is measured again after the last depth, and the lower latency of every sample counts. The second measurement is skipped when no depth was measured. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The example tunes a four-layer transformer encoder with `auto_tune`. A dry run shows how many modules every depth wraps. `print(result)` shows the latency of every depth against the untuned model. The winner is saved, loaded, and checked against the untuned output. CI runs it next to the minimal example. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tt.auto_tunetunes a model at every cut depth and keeps the fastest one.Depends on #55.
The first two commits belong to #55. Review from
feat: add auto_tuneon.It inspects the model once and wraps each distinct cut from
min_depthtomax_depth.Every cut is tuned and measured end to end on the dataset.
A module tuned at one depth is reused at deeper depths.
The shallowest depth within
tie_toleranceof the fastest wins.The model stays untuned when no depth beats it.
datasetare keyword-only.print(result)shows the table.time_budgetstops the search between depths once the budget is used up.tt.default_strategy(device), without the search for the maximum batch size.examples/AutoTunetunes a four-layer transformer encoder.On an Arc Pro B70, depth 0 is 2.4x faster than eager and depth 2 (32 modules) is 0.64x.