diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index f5e7fcdd3..7ec3ac401 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -18,7 +18,8 @@ A RISC-V based DSA (Domain Specific Architecture) framework. Built with Chisel 6 - `arch/src/main/scala/sims/` — simulation configs - `verilator/` — Verilator config - `bb-tests/` — tests - - `workloads/lib/bbhw/isa/` — ISA C macros (one `.c` file per instruction) + - `workloads/lib/bbhw/isa/` — base mem/frontend ISA C macros + - `examples/balls//workloads/isa/` — ball-specific ISA macros - `workloads/src/CTest/` — C test cases - `bbdev/` — developer toolchain (Motia workflow backend) @@ -52,10 +53,11 @@ When adding or modifying a balldomain TOML, these must hold: 1. `ballNum` equals `ballIdMappings` length 2. `ballId` is strictly `0, 1, 2, ...` with no gaps/duplicates -3. `ballName` / `funct7` / `mnemonic` have no duplicates +3. `ballName` / `funct7` / `mnemonic` have no duplicates **within that core's ballISA** (cross-core encoding conflicts are allowed) 4. every `ballISA.bid` exists in `ballIdMappings`, and every ball has at least one ISA entry 5. relative `config=` paths resolve to existing files (when present) 6. `inBW` / `outBW` are positive integers +7. Ball trees (`examples/balls/*/`) must not hardcode `funct7`; encoding comes only from the Core `ballISA` generated into `examples/cores//isa/ballISA.h` Use MCP `validate(chip=..., balldomain=...)` or `/check`. @@ -77,11 +79,12 @@ Daily agent path: ### bbdev API wrappers (automatic server lifecycle + task status) All bbdev POST endpoints are exposed as `bbdev_*` tools. Daily path prefers: -- `bbdev_compiler_build` / `bbdev_workload_{clean,build,tohex}` +- `bbdev_config_install` / `bbdev_compiler_build` / `bbdev_workload_{clean,build,tohex}` - `bbdev_bemu_{sim,batch}` - `bbdev_bebop_verilator_{clean,verilog,build,sim,run,batch}` — bebop-accelerated RTL - `bbdev_verilator_{clean,verilog,build,sim,run}` — non-bebop Verilator RTL path -- `bbdev_uvm_{build,run}` / `bbdev_yosys_{run,verilog,synth}` +- `bbdev_vcs_{clean,verilog,build,sim,run}` — VCS RTL path +- `bbdev_uvm_{verilog,build,run}` / `bbdev_yosys_{run,verilog,synth}` Also: `bbdev_bebop_p2e_*`, `bbdev_firesim_*`, `bbdev_dc_verilog`, `bbdev_kernel_build` Every `bbdev_*` task submission returns immediately with `accepted=true`, @@ -90,12 +93,12 @@ Every `bbdev_*` task submission returns immediately with `accepted=true`, Default Verilator config: `sims.verilator.BuckyballToyVerilatorConfig` Pebble: `sims.verilator.BuckyballPebbleVerilatorConfig` -Workload binary names are `{chip}_{stem}-{platform}`, e.g. `toy_vecunit_matmul_ones-singlecore-baremetal` +Workload binary names are `{chip}_{stem}-{platform}`, e.g. `toy_vecunit_matmul_ones-baremetal` ### Analysis report paths - Area reports: `bbdev/api/steps/yosys/log/hierarchy_report.txt` (submodule breakdown), `area_report.txt` (top-level) - Timing report: `bbdev/api/steps/yosys/log/timing_report.txt` -- Simulation logs: `log/-{bemu,verilator,p2e}-/stdout.log`, `disasm.log` +- Simulation logs: `log/----/stdout.log`, `disasm.log` - bdb debug log: `log/-*/bdb.ndjson` (or `bdb.log`), with three DPI-C traces: - `[ITRACE]` — instruction issue/complete - `[MTRACE]` — SRAM reads/writes diff --git a/.claude/skills/ball-align/SKILL.md b/.claude/skills/ball-align/SKILL.md index 5a3acb79f..996492ad3 100644 --- a/.claude/skills/ball-align/SKILL.md +++ b/.claude/skills/ball-align/SKILL.md @@ -38,11 +38,11 @@ Against `examples/balls//` and the target chip, list gaps: | Small tests | Short shapes, hand vectors, edge/illegal | bemu + Verilator | | Bank tests | `iter≈BANK_LINES`, `srand`/`rand`; soft numeric may report LOSS | **bemu only** | | Integration tests | Real consumers; must update on contract changes | bemu; small paths may use Verilator | -| MLIR | Under `examples/balls//workloads/mlir_tests/`; **not** under chip | FileCheck + optest ELF → bemu regression | +| MLIR | Under `examples/balls//workloads/mlir_tests/`; **not** under chip | FileCheck + mlirtest ELF → bemu regression | | UVM | Stimulus/scoreboard on the same contract | **After RTL is green** | -- MLIR: keep `bank` Op and lowered `ball` Op separate (see transpose: contract + optest) -- New ctest / optest → chip **bemu** `workloads-elf.toml` + `workloads-pk.toml` +- MLIR: keep `bank` Op and lowered `ball` Op separate (see transpose: contract + mlirtest) +- New ctest / mlirtest → chip **bemu** `workloads-elf.toml` + `workloads-pk.toml` - Verilator lists take **small tests only**, not bank tests ## Phase 2 - One semantics across the stack @@ -78,7 +78,7 @@ Copy and tick: ``` - [ ] Contract (fields, shapes, illegal table) + validate pass - [ ] Small suite bemu green; bank tests bemu green -- [ ] bank/ball MLIR OpTest + listed in bemu regression +- [ ] bank/ball MLIRTest + listed in bemu regression - [ ] RTL aligned + Verilator small tests green - [ ] Integration tests synced; UVM (after RTL) ``` diff --git a/.claude/skills/ball/SKILL.md b/.claude/skills/ball/SKILL.md index a07be77c0..ee9ec966d 100644 --- a/.claude/skills/ball/SKILL.md +++ b/.claude/skills/ball/SKILL.md @@ -12,7 +12,7 @@ description: Create a new Buckyball Ball operator named $ARGUMENTS, covering the - usually `examples/chips//configs/tiles/cores/balldomains/*.toml` 2. Check for partial existing implementation (incremental mode): - existing directory in `examples/balls/` - - existing ISA macro in `bb-tests/workloads/lib/bbhw/isa/` + - existing ISA macro in `examples/balls//workloads/isa/` (or base under `bb-tests/workloads/lib/bbhw/isa/`) - existing chip/ball ctests under `examples/` or `bb-tests/workloads/` 3. Confirm with user: - target chip @@ -47,7 +47,9 @@ Edit the chip balldomain TOML (the one selected by `cores/default.toml`, or the ## Phase 4 - Add ISA C Macro -Create `_.c` under `bb-tests/workloads/lib/bbhw/isa/`, then include it in `isa.h`. +Create `examples/balls//workloads/isa/.h` (include ``), +then `#include .h>` from the ball's ctests. Do **not** add ball ISA to +central `bb-tests/workloads/lib/bbhw/isa/isa.h` (base mem/frontend only). ## Phase 5 - Add CTest diff --git a/.claude/skills/check/SKILL.md b/.claude/skills/check/SKILL.md index 7ae479310..0e48cd69c 100644 --- a/.claude/skills/check/SKILL.md +++ b/.claude/skills/check/SKILL.md @@ -16,9 +16,10 @@ Checks on that TOML: 1. `ballNum` equals `ballIdMappings` length 2. `ballId` is strictly increasing (`0, 1, 2, ...`) with no gaps 3. no duplicated `ballId` / `ballName` -4. no duplicated `funct7` / `mnemonic` in `ballISA` +4. no duplicated `funct7` / `mnemonic` in `ballISA` (scope = this single balldomain / core; other cores may reuse the same funct7) 5. every `ballISA.bid` exists in mappings; every ball has ≥1 ISA entry 6. relative `config=` paths exist; `inBW`/`outBW` are positive +7. ball ISA headers / MLIR must not hardcode ball `funct7` (encoding is generated from this TOML into `examples/cores//isa/ballISA.h`) Report pass/fail for each item. diff --git a/.claude/skills/chip-designer/SKILL.md b/.claude/skills/chip-designer/SKILL.md index 200f83928..3200a8681 100644 --- a/.claude/skills/chip-designer/SKILL.md +++ b/.claude/skills/chip-designer/SKILL.md @@ -54,7 +54,7 @@ Do **not** require full-model green before the cut. Create `examples/chips//`: -- `chip.toml` (`topology`, `compilerCore`, `runtime.*`) +- `chip.toml` (only `[chip]` mill class: `verilatorConfig` / `p2eConfig`) - `configs/` tile with `sharedMem` + `[[cores]]` includes into `examples/cores//` - `arch/.../CustomConfigs.scala` + `sims/{verilator,p2e}/TargetConfigs.scala` - `workloads/` (contracts + later binaries), `regression/batch/{bemu,verilator,p2e}/`, optional `kernel/` diff --git a/.claude/skills/verify/SKILL.md b/.claude/skills/verify/SKILL.md index 1970f0c9b..ea5e858be 100644 --- a/.claude/skills/verify/SKILL.md +++ b/.claude/skills/verify/SKILL.md @@ -10,7 +10,7 @@ description: Verify functional correctness of the Ball named $ARGUMENTS. Use thi Use `/check` logic to validate registration consistency, then ensure all required artifacts exist and fill missing pieces: 1. Ball implementation: `examples/balls//arch/src/main/scala/` 2. Registration entry in chip balldomain TOML under `examples/chips//configs/tiles/cores/balldomains/` -3. ISA macro file in `bb-tests/workloads/lib/bbhw/isa/` +3. ISA macro file in `examples/balls//workloads/isa/` (base ISA under `bb-tests/workloads/lib/bbhw/isa/`) 4. CTest for that ball/chip ## Phase 2 - Build and Simulate @@ -18,7 +18,7 @@ Use `/check` logic to validate registration consistency, then ensure all require 1. Run `bbdev_workload_build(chip=...)` to build CTests 2. Run `bbdev_bemu_sim(chip=..., binary=...)` first 3. Run `bbdev_bebop_verilator_run(binary=..., config=...)` for RTL - - binary is `{chip}_{stem}-{platform}`, e.g. `toy_transpose_test-singlecore-baremetal` + - binary is `{chip}_{stem}-{platform}`, e.g. `toy_transpose_test-baremetal` - config is required, e.g. `sims.verilator.BuckyballToyVerilatorConfig` 4. If build/simulation fails, switch to `/debug` flow diff --git a/.claude/skills/waveform/SKILL.md b/.claude/skills/waveform/SKILL.md index 3627be800..06ba81bed 100644 --- a/.claude/skills/waveform/SKILL.md +++ b/.claude/skills/waveform/SKILL.md @@ -17,7 +17,7 @@ The project uses `waveform-mcp` with these tools: ## Waveform File Locations Waveforms generated by simulation are usually in: -- `log/-verilator-/waveform/` +- `log/---verilator-/waveform/` - format: `.vcd` or `.fst` - use `find log -name "*.vcd" -o -name "*.fst" | head -5` to locate files diff --git a/.github/scripts/ci_repo_lock.sh b/.github/scripts/ci_repo_lock.sh new file mode 100755 index 000000000..9fd446863 --- /dev/null +++ b/.github/scripts/ci_repo_lock.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +set -euo pipefail + +LOCK_DIR=/tmp/buckyball-ci +LOCK_FILE="${LOCK_DIR}/lock" +ACTIVE_SHA_FILE="${LOCK_DIR}/active_sha" +REFCOUNT_FILE="${LOCK_DIR}/refcount" + +usage() { + echo "usage: $0 prepare|enter|exit " >&2 + exit 1 +} + +[[ $# -eq 4 ]] || usage + +cmd="$1" +want_sha="$2" +repo="${3/#\~/$HOME}" +git_ref="$4" + +mkdir -p "${LOCK_DIR}" +exec 9>"${LOCK_FILE}" + +repo_reset() { + cd "${repo}" + git fetch --force --prune origin "${git_ref}" + git checkout --detach FETCH_HEAD + git reset --hard FETCH_HEAD + git clean -ffd + git submodule sync + git submodule update --init --force + for submodule in bbdev bebop compiler/thirdparty/buddy-mlir; do + git -C "${repo}/${submodule}" reset --hard + git -C "${repo}/${submodule}" clean -ffd + done + have_sha="$(git rev-parse HEAD)" + if [[ "${have_sha}" != "${want_sha}" ]]; then + echo "ERROR: checkout is ${have_sha}, expected ${want_sha}" >&2 + exit 1 + fi +} + +repo_prepare() { + flock 9 + repo_reset + echo "${want_sha}" > "${ACTIVE_SHA_FILE}" + echo 0 > "${REFCOUNT_FILE}" + flock -u 9 +} + +repo_enter() { + while true; do + flock 9 + active_sha="$(cat "${ACTIVE_SHA_FILE}" 2>/dev/null || true)" + refcount="$(cat "${REFCOUNT_FILE}" 2>/dev/null || echo 0)" + have_sha="$(git -C "${repo}" rev-parse HEAD)" + if [[ "${have_sha}" == "${want_sha}" ]]; then + echo "${want_sha}" > "${ACTIVE_SHA_FILE}" + echo $((refcount + 1)) > "${REFCOUNT_FILE}" + flock -u 9 + return 0 + fi + if [[ "${active_sha}" == "${want_sha}" ]]; then + echo $((refcount + 1)) > "${REFCOUNT_FILE}" + flock -u 9 + return 0 + fi + if [[ "${refcount}" == "0" ]]; then + repo_reset + echo "${want_sha}" > "${ACTIVE_SHA_FILE}" + echo 1 > "${REFCOUNT_FILE}" + flock -u 9 + return 0 + fi + flock -u 9 + echo "waiting: active=${active_sha} refcount=${refcount} want=${want_sha}" + sleep 10 + done +} + +repo_exit() { + flock 9 + active_sha="$(cat "${ACTIVE_SHA_FILE}" 2>/dev/null || true)" + refcount="$(cat "${REFCOUNT_FILE}" 2>/dev/null || echo 0)" + if [[ "${active_sha}" != "${want_sha}" ]]; then + echo "ERROR: active sha ${active_sha} != job sha ${want_sha}" >&2 + flock -u 9 + exit 1 + fi + if [[ "${refcount}" -le 0 ]]; then + echo "ERROR: refcount underflow (${refcount})" >&2 + flock -u 9 + exit 1 + fi + echo $((refcount - 1)) > "${REFCOUNT_FILE}" + flock -u 9 +} + +case "${cmd}" in + prepare) repo_prepare ;; + enter) repo_enter ;; + exit) repo_exit ;; + *) usage ;; +esac diff --git a/.github/scripts/setproxy.sh b/.github/scripts/setproxy.sh new file mode 100644 index 000000000..f9d76cbae --- /dev/null +++ b/.github/scripts/setproxy.sh @@ -0,0 +1 @@ +export ALL_PROXY=http://127.0.0.1:7890 diff --git a/.github/workflows/README.zh.md b/.github/workflows/README.zh.md index ce9b03aeb..0e9bbec88 100644 --- a/.github/workflows/README.zh.md +++ b/.github/workflows/README.zh.md @@ -1,5 +1,8 @@ # CI 说明 -CI目前分为两大类： -- 第一类是仓库功能check，确保代码合并后功能正确不会对仓库主线造成影响，触发方式为自动执行。部署的服务器组为 buckyball-cpu-server，服务器label为 cpu-server -- 第二类是PPA性能回归，定期执行检查优化后设计在目标workload上的表现和RTL的面积和功耗表现，生成性能分析报告，触发方式为手动分配。配置的服务器组为 buckyball-ppa-regression，服务器label为 cpu-server-tapeout +push/PR 到 `main` 只触发 `ci.yml`： + +1. **pre-commit**：拉代码 + 校验一次 +2. **chip-check**：通过后，matrix 并行跑各 chip（共享 `~/Code/buckyball`，`ci_repo_lock.sh` 协调） + +手动只跑 pre-commit：Actions 里触发 `pre-commit.yml`。 diff --git a/.github/workflows/check.yaml b/.github/workflows/check.yaml new file mode 100644 index 000000000..33c8b3442 --- /dev/null +++ b/.github/workflows/check.yaml @@ -0,0 +1,212 @@ +name: check + +on: + push: + branches: + - main + pull_request: + branches: + - main + +permissions: + contents: read + checks: write + +concurrency: + group: buckyball-ci-${{ github.event.pull_request.head.sha || github.sha }} + cancel-in-progress: false + +env: + CHECKOUT_SHA: ${{ github.event.pull_request.head.sha || github.sha }} + CHECKOUT_REF: ${{ github.event_name == 'pull_request' && format('refs/pull/{0}/head', github.event.pull_request.number) || github.ref }} + +jobs: + pre-commit: + name: pre-commit + runs-on: check + steps: + - name: Print information + run: | + echo "event=${{ github.event_name }}" + echo "sha=${CHECKOUT_SHA}" + echo "ref=${CHECKOUT_REF}" + if [ "${{ github.event_name }}" = "pull_request" ]; then + echo "pr=#${{ github.event.pull_request.number }}: ${{ github.event.pull_request.title }}" + echo "branch=${{ github.head_ref }} -> ${{ github.base_ref }}" + else + echo "branch=${{ github.ref_name }}" + fi + + - name: Prepare repo + shell: zsh {0} + run: | + set -euo pipefail + root=~/Code/buckyball + source "${root}/.github/scripts/setproxy.sh" + bash "${root}/.github/scripts/ci_repo_lock.sh" \ + prepare "${CHECKOUT_SHA}" "${root}" "${CHECKOUT_REF}" + + - name: Check pre-commit + shell: zsh {0} + run: | + root=~/Code/buckyball + source "${root}/.github/scripts/setproxy.sh" + cd "${root}" + nix develop -c pre-commit run --all-files --show-diff-on-failure --color=always + + chip-check: + name: ${{ matrix.chip }} + needs: pre-commit + runs-on: check + strategy: + fail-fast: false + matrix: + include: + - chip: toy + enable_rushb: false + run_bemu_elf_tests: true + run_bemu_pk_tests: true + run_verilator_batch_tests: true + - chip: pebble + enable_rushb: false + run_bemu_elf_tests: true + run_bemu_pk_tests: true + run_verilator_batch_tests: true + - chip: goban + enable_rushb: false + run_bemu_elf_tests: false + run_bemu_pk_tests: false + run_verilator_batch_tests: false + - chip: poly + enable_rushb: false + run_bemu_elf_tests: false + run_bemu_pk_tests: false + run_verilator_batch_tests: false + - chip: multi-rocket + enable_rushb: false + run_bemu_elf_tests: false + run_bemu_pk_tests: false + run_verilator_batch_tests: true + env: + BB_ROOT: ~/Code/buckyball + steps: + - name: Print information + run: | + echo "chip=${{ matrix.chip }}" + echo "event=${{ github.event_name }}" + echo "sha=${CHECKOUT_SHA}" + echo "ref=${CHECKOUT_REF}" + + - name: Enter repo lock + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + source "${root}/.github/scripts/setproxy.sh" + bash "${root}/.github/scripts/ci_repo_lock.sh" \ + enter "${CHECKOUT_SHA}" "${root}" "${CHECKOUT_REF}" + + - name: Nix build + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + cd "${root}" + nix build + + - name: Build compiler and workloads + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + cd "${root}" + nix develop -c bbdev config --install + nix develop -c bbdev compiler --build '--chip ${{ matrix.chip }}' + nix develop -c bbdev workload --build '--chip ${{ matrix.chip }}' + + - name: bemu ELF batch + if: matrix.run_bemu_elf_tests + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + cd "${root}" + nix develop -c bbdev bebop-bemu --batch '--chip ${{ matrix.chip }} --test elf-tests --clean-before' + + - name: bemu PK batch + if: matrix.run_bemu_pk_tests + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + cd "${root}" + nix develop -c bbdev bebop-bemu --batch '--chip ${{ matrix.chip }} --test pk-tests --clean-before' + + - name: verilator batch + if: matrix.run_verilator_batch_tests + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + cd "${root}" + nix develop -c bbdev bebop-verilator --clean '--chip ${{ matrix.chip }}' + nix develop -c bbdev bebop-verilator --verilog '--chip ${{ matrix.chip }}' + nix develop -c bbdev bebop-verilator --build '--jobs 16 --chip ${{ matrix.chip }}' + nix develop -c bbdev bebop-verilator --batch '--chip ${{ matrix.chip }} --test elf-tests --clean-before' + + - name: rushB batch + if: matrix.enable_rushb && matrix.run_bemu_elf_tests && matrix.run_bemu_pk_tests && matrix.run_verilator_batch_tests + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + cd "${root}" + nix develop -c bbdev workload --build '--chip ${{ matrix.chip }} --rushB bemu' + nix develop -c bbdev bebop-bemu --batch '--chip ${{ matrix.chip }} --test elf-tests --rushB --clean-before' + nix develop -c bbdev bebop-bemu --batch '--chip ${{ matrix.chip }} --test pk-tests --rushB --clean-before' + nix develop -c bbdev workload --build '--chip ${{ matrix.chip }} --rushB verilator' + nix develop -c bbdev bebop-verilator --batch '--chip ${{ matrix.chip }} --test elf-tests --rushB --clean-before' + nix develop -c bbdev bebop-verilator --batch '--chip ${{ matrix.chip }} --test pk-tests --rushB --clean-before' + + - name: Detect bebop JUnit files + if: always() && (matrix.run_bemu_elf_tests || matrix.run_bemu_pk_tests || matrix.run_verilator_batch_tests) + id: junit + shell: zsh {0} + run: | + root="${BB_ROOT/#\~/$HOME}" + files=("${root}/bebop/target/${{ matrix.chip }}/nextest/junit-bbdev-"*.xml(N)) + if (( ${#files} == 0 )); then + echo "found=false" >> "$GITHUB_OUTPUT" + else + echo "found=true" >> "$GITHUB_OUTPUT" + fi + + - name: Publish bebop test report + if: always() && (matrix.run_bemu_elf_tests || matrix.run_bemu_pk_tests || matrix.run_verilator_batch_tests) && steps.junit.outputs.found == 'true' + uses: mikepenz/action-junit-report@v4 + with: + report_paths: "~/Code/buckyball/bebop/target/${{ matrix.chip }}/nextest/junit-bbdev-*.xml" + check_name: "bebop nextest (${{ matrix.chip }})" + detailed_summary: true + include_passed: true + + - name: Upload bebop JUnit artifacts + if: always() && (matrix.run_bemu_elf_tests || matrix.run_bemu_pk_tests || matrix.run_verilator_batch_tests) && steps.junit.outputs.found == 'true' + uses: actions/upload-artifact@v4 + env: + ALL_PROXY: "" + HTTP_PROXY: "" + HTTPS_PROXY: "" + NO_PROXY: github.com,api.github.com,uploads.github.com,objects.githubusercontent.com,.blob.core.windows.net + with: + name: bebop-nextest-junit-${{ matrix.chip }} + path: "~/Code/buckyball/bebop/target/${{ matrix.chip }}/nextest/junit-bbdev-*.xml" + + - name: Exit repo lock + if: always() + shell: zsh {0} + run: | + set -euo pipefail + root="${BB_ROOT/#\~/$HOME}" + bash "${root}/.github/scripts/ci_repo_lock.sh" \ + exit "${CHECKOUT_SHA}" "${root}" "${CHECKOUT_REF}" diff --git a/.github/workflows/pebble-check.yml b/.github/workflows/pebble-check.yml deleted file mode 100644 index bd9833cf6..000000000 --- a/.github/workflows/pebble-check.yml +++ /dev/null @@ -1,128 +0,0 @@ -name: Buckyball Pebble CI -run-name: ${{ github.actor }} is running Pebble CI Test 🚀 -on: - push: - branches: - - pebble - pull_request: - branches: - - pebble - -permissions: - contents: read - checks: write - -# limit the number of concurrent jobs to 1, to avoid resource contention -concurrency: - group: buckyball-cpu-server - cancel-in-progress: false - -jobs: - CI-Test: - name: CI Test - runs-on: cpu-server - env: - ALL_PROXY: socks5h://127.0.0.1:7890 - steps: - - name: Print information - run: | - echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event." - if [ "${{ github.event_name }}" = "pull_request" ]; then - echo "🔎 PR #${{ github.event.pull_request.number }}: ${{ github.event.pull_request.title }}" - echo "🔎 Branch: ${{ github.head_ref }} -> ${{ github.base_ref }}" - else - echo "🔎 Branch: ${{ github.ref_name }}" - fi - echo "🔎 Repository: ${{ github.repository }}" - - - name: Reset to clean state - shell: zsh {0} - run: | - cd ~/Code/buckyball - git fetch --force --prune origin "${{ github.ref }}" - git checkout --detach FETCH_HEAD - git reset --hard FETCH_HEAD - git clean -ffd - git submodule sync - git submodule update --init --force - for submodule in bbdev bebop compiler/thirdparty/buddy-mlir; do - git -C "$submodule" reset --hard - git -C "$submodule" clean -ffd - done - - - name: Nix build - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix build - - - name: Build Workloads - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bbdev compiler --build '--chip pebble' - nix develop -c bbdev workload --clean - nix develop -c bbdev workload --build '--chip pebble' - - - name: bemu Tests - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bash -c "cd bebop && cargo clean && cd .." - nix develop -c bbdev bebop-bemu --batch '--chip pebble --test elf-tests --clean-before' - nix develop -c bbdev bebop-bemu --batch '--chip pebble --test pk-tests --clean-before' - - - name: verilator Tests - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bbdev bebop-verilator --clean '--config sims.verilator.BuckyballPebbleVerilatorConfig' - nix develop -c bbdev bebop-verilator --verilog '--config sims.verilator.BuckyballPebbleVerilatorConfig' - nix develop -c bbdev bebop-verilator --build '--jobs 16 --config sims.verilator.BuckyballPebbleVerilatorConfig' - nix develop -c bbdev bebop-verilator --batch '--chip pebble --config sims.verilator.BuckyballPebbleVerilatorConfig --test elf-tests --clean-before' - - - name: Detect bebop JUnit files - if: always() - id: junit - shell: zsh {0} - run: | - files=(~/Code/buckyball/bebop/target/nextest/default/target/nextest/junit-bbdev-*.xml(N)) - if (( ${#files} == 0 )); then - echo "JUnit files not found" - echo "found=false" >> "$GITHUB_OUTPUT" - else - echo "JUnit files: ${files[@]}" - echo "found=true" >> "$GITHUB_OUTPUT" - fi - - - name: Publish bebop test report - if: always() && steps.junit.outputs.found == 'true' - uses: mikepenz/action-junit-report@v4 - with: - report_paths: "~/Code/buckyball/bebop/target/nextest/default/target/nextest/junit-bbdev-*.xml" - check_name: "bebop nextest report" - detailed_summary: true - include_passed: true - - - name: Upload bebop JUnit artifacts - if: always() && steps.junit.outputs.found == 'true' - uses: actions/upload-artifact@v4 - env: - ALL_PROXY: "" - HTTP_PROXY: "" - HTTPS_PROXY: "" - NO_PROXY: github.com,api.github.com,uploads.github.com,objects.githubusercontent.com,.blob.core.windows.net - with: - name: bebop-nextest-junit - path: "~/Code/buckyball/bebop/target/nextest/default/target/nextest/junit-bbdev-*.xml" - - # if check failed, revert the branch - # - name: Revert Bad Commit - # if: failure() - # shell: zsh {0} - # run: | - # source ~/.zshrc - # buckyball_exec - # setproxy - # git reset --hard HEAD~1 - # git push --force origin pebble diff --git a/.github/workflows/pebble-regression.yml b/.github/workflows/pebble-regression.yml deleted file mode 100644 index 0efc7c768..000000000 --- a/.github/workflows/pebble-regression.yml +++ /dev/null @@ -1,121 +0,0 @@ -name: Buckyball Pebble PPA Regression -run-name: ${{ github.actor }} Pebble PPA Regression -on: - workflow_dispatch: - -permissions: - contents: read - checks: write - -concurrency: - group: buckyball-ppa-regression - cancel-in-progress: false - -jobs: - tapeout: - name: Pebble PPA Regression - runs-on: cpu-server-tapeout - env: - ALL_PROXY: socks5h://127.0.0.1:17890 - steps: - - name: Print information - run: | - echo "Pebble PPA Regression: ${{ github.event_name }}" - echo "Repository: ${{ github.repository }}" - - # ──────────── Pebble CI ──────────── - - name: Reset buckyball to clean state - shell: zsh {0} - run: | - BUCKYBALL_DIR=$HOME/Code/buckyball - cd $BUCKYBALL_DIR - git fetch --force --prune origin "${{ github.ref }}" - git checkout --detach FETCH_HEAD - git reset --hard FETCH_HEAD - git clean -ffd - git submodule sync bbdev bebop compiler/thirdparty/buddy-mlir - git submodule update --init --force bbdev bebop compiler/thirdparty/buddy-mlir - for submodule in bbdev bebop compiler/thirdparty/buddy-mlir; do - git -C "$submodule" reset --hard - git -C "$submodule" clean -ffd - done - - - name: Nix build - shell: zsh {0} - run: | - BUCKYBALL_DIR=$HOME/Code/buckyball - cd $BUCKYBALL_DIR - nix build - - - name: Build compiler and workloads - shell: zsh {0} - run: | - BUCKYBALL_DIR=$HOME/Code/buckyball - cd $BUCKYBALL_DIR - nix develop -c bbdev compiler --build '--chip pebble' - nix develop -c bbdev workload --clean - nix develop -c bbdev workload --build '--chip pebble' - - - name: Generate Verilog - shell: zsh {0} - run: | - BUCKYBALL_DIR=$HOME/Code/buckyball - cd $BUCKYBALL_DIR - nix develop -c bbdev bebop-verilator --clean '--config sims.verilator.BuckyballPebbleVerilatorConfig' - nix develop -c bbdev bebop-verilator --verilog '--config sims.verilator.BuckyballPebbleVerilatorConfig' - - # ──────────── Tapeout Flow ──────────── - - name: Run tapeout flow - shell: zsh {0} - run: | - TAPEOUT_DIR=$HOME/Code/tapeout - export BUCKYBALL_DIR=$HOME/Code/buckyball - git -C $TAPEOUT_DIR clean -fdx - $TAPEOUT_DIR/run_flow.sh pebble - - # ──────────── Collect & Upload ──────────── - - name: Collect and summarize PPA - shell: zsh {0} - if: always() - run: | - set +e - TAPEOUT_DIR=$HOME/Code/tapeout - RESULTS=$TAPEOUT_DIR/tapeout_results - mkdir -p "$RESULTS/reports" "$RESULTS/netlist" "$RESULTS/logs" - cp -r $TAPEOUT_DIR/1_SYN/rpt/* "$RESULTS/reports/" 2>/dev/null - cp -r $TAPEOUT_DIR/3_POWER/rpt/* "$RESULTS/reports/" 2>/dev/null - LATEST=$(ls -td $TAPEOUT_DIR/1_SYN/outputs/*/ 2>/dev/null | head -1) - [ -n "$LATEST" ] && cp "$LATEST"/* "$RESULTS/netlist/" 2>/dev/null - cp $TAPEOUT_DIR/1_SYN/logs/* "$RESULTS/logs/" 2>/dev/null - cp $TAPEOUT_DIR/3_POWER/logs/* "$RESULTS/logs/" 2>/dev/null - cp -r $TAPEOUT_DIR/2_POSTSIM/glsim/results/* "$RESULTS/logs/" 2>/dev/null - - echo "============================================" - echo " PPA Summary (Pebble @ TSMC28)" - echo "============================================" - for rpt in "$RESULTS"/reports/*/*_mapped_area.rpt; do - [ -f "$rpt" ] || continue - echo "--- Area ---"; grep -E "Total cell area" "$rpt" | head -1 - done - for rpt in "$RESULTS"/reports/*/*_timing_max_path.rpt; do - [ -f "$rpt" ] || continue - echo "--- Timing ---"; grep -E "slack" "$rpt" | head -1 - done - for rpt in "$RESULTS"/reports/*/*_power.rpt; do - [ -f "$rpt" ] || continue - echo "--- Power (DC) ---"; grep -E "Total" "$rpt" | head -1 - done - echo "============================================" - - - name: Upload tapeout artifacts - if: always() - uses: actions/upload-artifact@v4 - env: - ALL_PROXY: "" - HTTP_PROXY: "" - HTTPS_PROXY: "" - NO_PROXY: github.com,api.github.com,uploads.github.com,objects.githubusercontent.com,.blob.core.windows.net - with: - name: pebble-tapeout-results - path: ~/Code/tapeout/tapeout_results - retention-days: 30 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml deleted file mode 100644 index f01b00155..000000000 --- a/.github/workflows/test.yml +++ /dev/null @@ -1,158 +0,0 @@ -name: Buckyball CI -run-name: ${{ github.actor }} is running CI Test 🚀 -on: - push: - branches: - - main - pull_request: - branches: - - main - -permissions: - contents: read - checks: write - -# limit the number of concurrent jobs to 1, to avoid resource contention -concurrency: - group: buckyball-cpu-server - cancel-in-progress: false - -jobs: - CI-Test: - name: CI Test - runs-on: cpu-server - env: - ALL_PROXY: socks5h://127.0.0.1:7890 - steps: - - name: Print information - run: | - echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event." - if [ "${{ github.event_name }}" = "pull_request" ]; then - echo "🔎 PR #${{ github.event.pull_request.number }}: ${{ github.event.pull_request.title }}" - echo "🔎 Branch: ${{ github.head_ref }} -> ${{ github.base_ref }}" - else - echo "🔎 Branch: ${{ github.ref_name }}" - fi - echo "🔎 Repository: ${{ github.repository }}" - - - name: Reset to clean state - shell: zsh {0} - run: | - cd ~/Code/buckyball - git fetch --force --prune origin "${{ github.ref }}" - git checkout --detach FETCH_HEAD - git reset --hard FETCH_HEAD - git clean -ffd - git submodule sync - git submodule update --init --force - for submodule in bbdev bebop compiler/thirdparty/buddy-mlir; do - git -C "$submodule" reset --hard - git -C "$submodule" clean -ffd - done - - - name: Nix build - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix build - - - name: Build Compiler - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bbdev compiler --build '--chip toy' - - - name: Build Workloads - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bbdev workload --clean - nix develop -c bbdev workload --build '--chip toy' - - - name: bemu Tests - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bash -c "cd bebop && cargo clean && cd .." - nix develop -c bbdev bebop-bemu --batch '--chip toy --test elf-tests --clean-before' - nix develop -c bbdev bebop-bemu --batch '--chip toy --test pk-tests --clean-before' - - - name: verilator Tests - shell: zsh {0} - run: | - cd ~/Code/buckyball - nix develop -c bbdev bebop-verilator --clean '--config sims.verilator.BuckyballToyVerilatorConfig' - nix develop -c bbdev bebop-verilator --verilog '--config sims.verilator.BuckyballToyVerilatorConfig' - nix develop -c bbdev bebop-verilator --build '--jobs 16 --config sims.verilator.BuckyballToyVerilatorConfig' - nix develop -c bbdev bebop-verilator --batch '--chip toy --config sims.verilator.BuckyballToyVerilatorConfig --test elf-tests --clean-before' - - - name: rushB Tests - shell: zsh {0} - run: | - set -euo pipefail - cd ~/Code/buckyball - nix develop -c bbdev workload --build '--chip toy --rushB bemu' - nix develop -c bbdev bebop-bemu --batch '--chip toy --test elf-tests --rushB --clean-before' - nix develop -c bbdev bebop-bemu --batch '--chip toy --test pk-tests --rushB --clean-before' - - nix develop -c bbdev bebop-verilator --clean '--config sims.verilator.BuckyballToyRushBVerilatorConfig' - nix develop -c bbdev bebop-verilator --verilog '--config sims.verilator.BuckyballToyRushBVerilatorConfig' - nix develop -c bbdev bebop-verilator --build '--jobs 16 --config sims.verilator.BuckyballToyRushBVerilatorConfig' - - nix develop -c bbdev workload --build '--chip toy --rushB verilator' - nix develop -c bbdev bebop-verilator --batch '--chip toy --config sims.verilator.BuckyballToyRushBVerilatorConfig --test elf-tests --rushB --clean-before' - nix develop -c bbdev bebop-verilator --batch '--chip toy --config sims.verilator.BuckyballToyRushBVerilatorConfig --test pk-tests --rushB --clean-before' - - # Toy don't support any models, so we don't test it. - # Lenet rushB can be tested on pebble. - # Build pebble last so BUDDY_EXTERNAL_DIALECTS_DIR does not break toy workloads above. - # nix develop -c bbdev compiler --build '--chip pebble' - # nix develop -c bbdev workload --build '--chip pebble --model lenet' - # nix develop -c bbdev workload --build '--chip pebble --model lenet --rushB bemu' - # nix develop -c bbdev bebop-bemu --sim '--chip pebble --binary buddy-buckyball-lenet-rushB-bemu-run --rushB' - - - name: Detect bebop JUnit files - if: always() - id: junit - shell: zsh {0} - run: | - files=(~/Code/buckyball/bebop/target/nextest/default/target/nextest/junit-bbdev-*.xml(N)) - if (( ${#files} == 0 )); then - echo "JUnit files not found" - echo "found=false" >> "$GITHUB_OUTPUT" - else - echo "JUnit files: ${files[@]}" - echo "found=true" >> "$GITHUB_OUTPUT" - fi - - - name: Publish bebop test report - if: always() && steps.junit.outputs.found == 'true' - uses: mikepenz/action-junit-report@v4 - with: - report_paths: "~/Code/buckyball/bebop/target/nextest/default/target/nextest/junit-bbdev-*.xml" - check_name: "bebop nextest report" - detailed_summary: true - include_passed: true - - - name: Upload bebop JUnit artifacts - if: always() && steps.junit.outputs.found == 'true' - uses: actions/upload-artifact@v4 - env: - ALL_PROXY: "" - HTTP_PROXY: "" - HTTPS_PROXY: "" - NO_PROXY: github.com,api.github.com,uploads.github.com,objects.githubusercontent.com,.blob.core.windows.net - with: - name: bebop-nextest-junit - path: "~/Code/buckyball/bebop/target/nextest/default/target/nextest/junit-bbdev-*.xml" - - # if check failed, revert the branch - # - name: Revert Bad Commit - # if: failure() - # shell: zsh {0} - # run: | - # source ~/.zshrc - # buckyball_exec - # setproxy - # git reset --hard HEAD~1 - # git push --force origin main diff --git a/.gitignore b/.gitignore index 0467a4733..51dfab44c 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ .metals/ .motia/ .bsp/ +.dsh/ out/ .Xil/ result @@ -18,6 +19,8 @@ index.html *.log *.dot *.svf +/command.log +/dramsim3.* # Vivado / Webtalk session files (root-level) /vivado*.jou @@ -47,5 +50,9 @@ log/ scripts/docker/.last-image-tag ucli.key .worktrees/ + +# Bazel +/bazel-* +/MODULE.bazel.lock nohup.out alib-52/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 4aeec92ea..8dc263385 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -26,6 +26,7 @@ repos: hooks: - id: flake8 language: system + exclude: (^|/)bbdev/api/steps/config/scripts/chip_pb2\.py$ args: - --max-line-length=120 # Adjust as per your style guide - --ignore=F821,F403,F405,F401,W503,E203,E402,E401,W605,E712,E711,F841 @@ -60,6 +61,7 @@ repos: entry: verible-verilog-format language: system files: \.(v|vh|sv|svh)$ + exclude: verify/src/common/.*_defs\.svh$ args: - --inplace - --failsafe_success=false @@ -67,7 +69,7 @@ repos: - id: scalafmt name: Format Scala code with scalafmt - entry: bash -c 'cd arch && scalafmt --config .scalafmt.conf src/main/scala' + entry: scalafmt language: system - files: ^arch/src/main/scala/.*\.scala$ - pass_filenames: false + files: \.scala$ + exclude: (^|/)(thirdparty|node_modules)/ diff --git a/.scalafmt.conf b/.scalafmt.conf index f53852445..8b7ba055a 100644 --- a/.scalafmt.conf +++ b/.scalafmt.conf @@ -1,2 +1,91 @@ -version = "3.11.4" -runner.dialect = scala213 +version = 2.7.5 + +project.git = true +project.includeFilters = [ + ".*\\.scala$" +] +project.excludeFilters = [ + "thirdparty" +] + +# Basic formatting +maxColumn = 120 +continuationIndent.defnSite = 2 +continuationIndent.callSite = 2 +continuationIndent.extendSite = 4 + +# Alignment +align.preset = most +align.openParenCallSite = false +align.openParenDefnSite = false +align.tokens.add = [ + {code = ":", owner = ".*"}, + {code = ":=", owner = ".*"}, + {code = "=", owner = "(Enumerator.Val|Defn.(Va(l|r)|Def|Type))"}, + {code = "=>", owner = "Case"}, + {code = "<-", owner = "Enumerator.Generator"}, + {code = "//", owner = ".*"} +] +align.arrowEnumeratorGenerator = true +align.multiline = true + +# Newlines +newlines.source = keep +newlines.topLevelStatements = [before, after] +newlines.topLevelStatementsMinBreaks = 2 +newlines.afterCurlyLambda = never +newlines.alwaysBeforeCurlyBraceLambdaParams = false +newlines.alwaysBeforeMultilineDef = false +newlines.implicitParamListModifierForce = [before] +newlines.avoidForSimpleOverflow = [punct] + +# Vertical multiline +verticalMultiline.atDefnSite = true +verticalMultiline.arityThreshold = 4 +verticalMultiline.newlineAfterOpenParen = true + +# Docstrings +docstrings.style = asterisk +docstrings.wrap = no +docstrings.oneline = keep + +# Spaces +spaces.inImportCurlyBraces = false +spaces.beforeContextBoundColon = Never +spaces.afterTripleEquals = false +spaces.inByNameTypes = true + +# Line endings and misc +lineEndings = preserve +assumeStandardLibraryStripMargin = true +includeCurlyBraceInSelectChains = true +danglingParentheses.preset = true +danglingParentheses.defnSite = true +danglingParentheses.callSite = true +danglingParentheses.exclude = [] + +# Optional features +optIn.annotationNewlines = true +optIn.breakChainOnFirstMethodDot = true +optIn.configStyleArguments = true + +# Rewrite rules +rewrite.rules = [ + SortImports, + SortModifiers, + PreferCurlyFors, + RedundantBraces, + RedundantParens +] +rewrite.sortModifiers.order = [ + "override", "private", "protected", "implicit", "final", "sealed", "abstract", "lazy" +] +rewrite.redundantBraces.stringInterpolation = true +rewrite.redundantBraces.generalExpressions = false +rewrite.redundantBraces.methodBodies = true +rewrite.redundantBraces.includeUnitMethods = true +rewrite.redundantBraces.maxLines = 1 + +# Comments +comments.wrap = standalone +comments.wrapStandaloneSlcAsSlc = true diff --git a/README.md b/README.md index 650542d7b..6ee5a2c42 100644 --- a/README.md +++ b/README.md @@ -46,24 +46,18 @@ After the first time installation, you can enter the environment anytime by runn nix develop ``` - - -**4. Try faster simulation using bebop** - -```bash -bbdev bebop-verilator --run '--binary toy_vecunit_matmul_ones-singlecore-baremetal --config sims.verilator.BuckyballToyVerilatorConfig --itrace --mtrace --pmctrace --ctrace --banktrace' +bbdev verilator --run '--jobs 16 --chip toy --binary toy-toy-vecunit_matmul_ones-baremetal --batch' ``` -Run Verilator with BEMU Bank DiffTest enabled: +**4. Try faster simulation using bebop** ```bash -bbdev bebop-verilator --run '--diff --binary toy_vecunit_matmul_ones-singlecore-baremetal --config sims.verilator.BuckyballToyVerilatorConfig --no-wave' +bbdev bebop-verilator --run '--chip toy --binary toy-toy-vecunit_matmul_ones-baremetal --itrace --mtrace --pmctrace --ctrace --banktrace' ``` ## Tutorial diff --git a/arch/.scalafmt.conf b/arch/.scalafmt.conf deleted file mode 100644 index 91d1b07c3..000000000 --- a/arch/.scalafmt.conf +++ /dev/null @@ -1,88 +0,0 @@ -version = 2.7.5 - -# Only format src/main/scala directory -project.includeFilters = [ - "src/main/scala/.*\\.scala$" -] - -# Basic formatting -maxColumn = 120 -continuationIndent.defnSite = 2 -continuationIndent.callSite = 2 -continuationIndent.extendSite = 4 - -# Alignment -align.preset = most -align.openParenCallSite = false -align.openParenDefnSite = false -align.tokens.add = [ - {code = ":", owner = ".*"}, - {code = ":=", owner = ".*"}, - {code = "=", owner = "(Enumerator.Val|Defn.(Va(l|r)|Def|Type))"}, - {code = "=>", owner = "Case"}, - {code = "<-", owner = "Enumerator.Generator"}, - {code = "//", owner = ".*"} -] -align.arrowEnumeratorGenerator = true -align.multiline = true - -# Newlines -newlines.source = keep -newlines.topLevelStatements = [before, after] -newlines.topLevelStatementsMinBreaks = 2 -newlines.afterCurlyLambda = never -newlines.alwaysBeforeCurlyBraceLambdaParams = false -newlines.alwaysBeforeMultilineDef = false -newlines.implicitParamListModifierForce = [before] -newlines.avoidForSimpleOverflow = [punct] - -# Vertical multiline -verticalMultiline.atDefnSite = true -verticalMultiline.arityThreshold = 4 -verticalMultiline.newlineAfterOpenParen = true - -# Docstrings -docstrings.style = asterisk -docstrings.wrap = no -docstrings.oneline = keep - -# Spaces -spaces.inImportCurlyBraces = false -spaces.beforeContextBoundColon = Never -spaces.afterTripleEquals = false -spaces.inByNameTypes = true - -# Line endings and misc -lineEndings = preserve -assumeStandardLibraryStripMargin = true -includeCurlyBraceInSelectChains = true -danglingParentheses.preset = true -danglingParentheses.defnSite = true -danglingParentheses.callSite = true -danglingParentheses.exclude = [] - -# Optional features -optIn.annotationNewlines = true -optIn.breakChainOnFirstMethodDot = true -optIn.configStyleArguments = true - -# Rewrite rules -rewrite.rules = [ - SortImports, - SortModifiers, - PreferCurlyFors, - RedundantBraces, - RedundantParens -] -rewrite.sortModifiers.order = [ - "override", "private", "protected", "implicit", "final", "sealed", "abstract", "lazy" -] -rewrite.redundantBraces.stringInterpolation = true -rewrite.redundantBraces.generalExpressions = false -rewrite.redundantBraces.methodBodies = true -rewrite.redundantBraces.includeUnitMethods = true -rewrite.redundantBraces.maxLines = 1 - -# Comments -comments.wrap = standalone -comments.wrapStandaloneSlcAsSlc = true diff --git a/arch/build.sc b/arch/build.sc index 1069b5c04..0f09d4305 100644 --- a/arch/build.sc +++ b/arch/build.sc @@ -4,9 +4,28 @@ import mill.define.Sources import mill.modules.Util import mill.scalalib.TestModule.ScalaTest import scalalib._ +import mill.javalib.JavaModule // support BSP import mill.bsp._ +object protoJava extends JavaModule { + def protoDir = T { + os.pwd / os.up / "bbdev" / "api" / "steps" / "config" / "scripts" / "proto" + } + + def generatedSources = T { + val dir = protoDir() + val proto = dir / "chip.proto" + if (!os.exists(proto)) { + throw new Exception(s"missing chip.proto: $proto") + } + os.proc("protoc", s"-I$dir", s"--java_out=${T.dest}", proto).call() + Seq(PathRef(T.dest)) + } + + override def ivyDeps = Agg(ivy"com.google.protobuf:protobuf-java:4.35.1") +} + object buckyball extends SbtModule { m => override def millSourcePath = os.pwd override def scalaVersion = "2.13.16" @@ -22,7 +41,8 @@ object buckyball extends SbtModule { m => // Add chipyard and rocket-chip dependencies override def moduleDeps = Seq( chipyard, - gemmini + gemmini, + protoJava ) override def sources = T.sources { @@ -61,7 +81,8 @@ object buckyball extends SbtModule { m => ivy"org.yaml:snakeyaml:2.0", ivy"com.lihaoyi::sourcecode:0.3.0", ivy"com.lihaoyi::upickle:3.3.1", - ivy"tech.sparse::toml-scala:0.2.2" + ivy"tech.sparse::toml-scala:0.2.2", + ivy"com.google.protobuf:protobuf-java:3.25.3" ) override def scalacPluginIvyDeps = Agg( diff --git a/arch/src/main/scala/framework/balldomain/BallDomain.scala b/arch/src/main/scala/framework/balldomain/BallDomain.scala index 6cc26436a..4991177c4 100644 --- a/arch/src/main/scala/framework/balldomain/BallDomain.scala +++ b/arch/src/main/scala/framework/balldomain/BallDomain.scala @@ -8,7 +8,7 @@ import framework.balldomain.bbus.BBus import framework.balldomain.decoder.BallDomainDecoder import framework.balldomain.rs.BallReservationStation import framework.balldomain.blink.{BankRead, BankWrite, SubRobRow} -import framework.balldomain.blink.mmio.MmioRead +import framework.balldomain.blink.mmio.{MmioRead, MmioWrite} import framework.frontend.globalrs.{GlobalSchedComplete, GlobalSchedIssue} /** @@ -22,17 +22,25 @@ import framework.frontend.globalrs.{GlobalSchedComplete, GlobalSchedIssue} class BallDomain(val b: GlobalConfig) extends Module { val totalBallRead = b.ballDomain.ballIdMappings.map(_.inBW).sum val totalBallWrite = b.ballDomain.ballIdMappings.map(_.outBW).sum + val totalMmioRead = b.ballDomain.ballIdMappings.map(_.mmioReadBW).sum + val totalMmioWrite = b.ballDomain.ballIdMappings.map(_.mmioWriteBW).sum @public val global_issue_i = IO(Flipped(Decoupled(new GlobalSchedIssue(b)))) @public val global_complete_o = IO(Decoupled(new GlobalSchedComplete(b))) @public + val ballChannelActive = IO(Output(Vec(b.ballDomain.ballNum, Bool()))) + @public + val ballChannelReady = IO(Input(Vec(b.ballDomain.ballNum, Bool()))) + @public val bankRead = IO(Vec(totalBallRead, Flipped(new BankRead(b)))) @public val bankWrite = IO(Vec(totalBallWrite, Flipped(new BankWrite(b)))) @public - val mmioRead = IO(Vec(b.ballDomain.ballNum, Flipped(new MmioRead(b)))) + val mmioRead = IO(Vec(totalMmioRead, Flipped(new MmioRead(b)))) + @public + val mmioWrite = IO(Vec(totalMmioWrite, Flipped(new MmioWrite(b)))) @public val subRobReq = IO(Vec(b.ballDomain.ballNum, Decoupled(new SubRobRow(b)))) @@ -40,16 +48,16 @@ class BallDomain(val b: GlobalConfig) extends Module { val ballDecoder: Instance[BallDomainDecoder] = Instantiate(new BallDomainDecoder(b)) val ballRs: Instance[BallReservationStation] = Instantiate(new BallReservationStation(b)) -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// // Global RS -> Decoder -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// ballDecoder.cmd_i.valid := global_issue_i.valid ballDecoder.cmd_i.bits := global_issue_i.bits.cmd global_issue_i.ready := ballDecoder.cmd_i.ready -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// // Decoder -> Local BallRS -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// ballRs.ball_decode_cmd_i.valid := ballDecoder.ball_decode_cmd_o.valid ballRs.ball_decode_cmd_i.bits.cmd := ballDecoder.ball_decode_cmd_o.bits ballRs.ball_decode_cmd_i.bits.rob_id := global_issue_i.bits.rob_id @@ -57,26 +65,29 @@ class BallDomain(val b: GlobalConfig) extends Module { ballRs.ball_decode_cmd_i.bits.sub_rob_id := global_issue_i.bits.sub_rob_id ballDecoder.ball_decode_cmd_o.ready := ballRs.ball_decode_cmd_i.ready -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// // Local BallRS -> BBus -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// bbus.cmdReq <> ballRs.issue_o.balls ballRs.commit_i.balls <> bbus.cmdResp -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// // BBus -> Mem Domain -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// bbus.bankRead <> bankRead bbus.bankWrite <> bankWrite + ballChannelActive := bbus.ballChannelActive + bbus.ballChannelReady := ballChannelReady bbus.mmioRead <> mmioRead + bbus.mmioWrite <> mmioWrite for (i <- 0 until b.ballDomain.ballNum) { subRobReq(i) <> bbus.subRobReq(i) } -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// // Local RS completion -> Global RS -//--------------------------------------------------------------------------- +//===-------------------------------------------------------------------===// global_complete_o.valid := ballRs.complete_o.valid global_complete_o.bits.rob_id := ballRs.complete_o.bits.rob_id global_complete_o.bits.is_sub := ballRs.complete_o.bits.is_sub diff --git a/arch/src/main/scala/framework/balldomain/bbus/bbus.scala b/arch/src/main/scala/framework/balldomain/bbus/bbus.scala index 74e5d271f..599281c59 100644 --- a/arch/src/main/scala/framework/balldomain/bbus/bbus.scala +++ b/arch/src/main/scala/framework/balldomain/bbus/bbus.scala @@ -10,7 +10,7 @@ import framework.balldomain.bbus.pmc.BallCyclePMC import framework.balldomain.bbus.cmdrouter.CmdRouter import framework.balldomain.isa.BallISA import framework.balldomain.blink.{BankRead, BankWrite, SubRobRow} -import framework.balldomain.blink.mmio.MmioRead +import framework.balldomain.blink.mmio.{MmioRead, MmioWrite} import java.lang.reflect.InvocationTargetException /** @@ -26,22 +26,30 @@ class BBus(val b: GlobalConfig) extends Module { val numBalls = b.ballDomain.ballNum val totalBallRead = b.ballDomain.ballIdMappings.map(_.inBW).sum val totalBallWrite = b.ballDomain.ballIdMappings.map(_.outBW).sum + val totalMmioRead = b.ballDomain.ballIdMappings.map(_.mmioReadBW).sum + val totalMmioWrite = b.ballDomain.ballIdMappings.map(_.mmioWriteBW).sum // Rs - bbus - balls @public - val cmdReq = IO(Vec(numBalls, Flipped(Decoupled(new BallRsIssue(b))))) + val cmdReq = IO(Vec(numBalls, Flipped(Decoupled(new BallRsIssue(b))))) @public - val cmdResp = IO(Vec(numBalls, Decoupled(new BallRsComplete(b)))) + val cmdResp = IO(Vec(numBalls, Decoupled(new BallRsComplete(b)))) + @public + val ballChannelActive = IO(Output(Vec(numBalls, Bool()))) + @public + val ballChannelReady = IO(Input(Vec(numBalls, Bool()))) // balls - bbus @public - val bankRead = IO(Vec(totalBallRead, Flipped(new BankRead(b)))) + val bankRead = IO(Vec(totalBallRead, Flipped(new BankRead(b)))) + @public + val bankWrite = IO(Vec(totalBallWrite, Flipped(new BankWrite(b)))) @public - val bankWrite = IO(Vec(totalBallWrite, Flipped(new BankWrite(b)))) + val mmioRead = IO(Vec(totalMmioRead, Flipped(new MmioRead(b)))) @public - val mmioRead = IO(Vec(numBalls, Flipped(new MmioRead(b)))) + val mmioWrite = IO(Vec(totalMmioWrite, Flipped(new MmioWrite(b)))) // balls - bbus - SubROB @public - val subRobReq = IO(Vec(numBalls, Decoupled(new SubRobRow(b)))) + val subRobReq = IO(Vec(numBalls, Decoupled(new SubRobRow(b)))) require(b.ballDomain.ballIdMappings.length == numBalls, "ballNum must match ballIdMappings length") @@ -92,6 +100,9 @@ class BBus(val b: GlobalConfig) extends Module { val targetMatches = VecInit(b.ballDomain.ballIdMappings.map(m => cmdRouter.io.cmdReq_o.bits.cmd.bid === m.ballId.U)) for (i <- 0 until numBalls) { + ballChannelActive(i) := balls(i).blink.status.running + balls(i).blink.channelReady := ballChannelReady(i) + val targetMatch = targetMatches(i) balls(i).blink.cmdReq.valid := cmdRouter.io.cmdReq_o.valid && !isBallInit && targetMatch balls(i).blink.cmdReq.bits := cmdRouter.io.cmdReq_o.bits @@ -147,14 +158,34 @@ class BBus(val b: GlobalConfig) extends Module { } } + var mmioWriteChannelIdx = 0 + for (ball <- balls) { + val mmioWriteBW = b.ballDomain.ballIdMappings + .find(_.ballName == ball.getClass.getSimpleName) + .map(_.mmioWriteBW) + .getOrElse(0) + for (i <- 0 until mmioWriteBW) { + mmioWrite(mmioWriteChannelIdx) <> ball.blink.mmioWrite(i) + mmioWriteChannelIdx = mmioWriteChannelIdx + 1 + } + } + // Connect balls' subRobReq for (i <- 0 until numBalls) { subRobReq(i) <> balls(i).blink.subRobReq } - // Connect balls' mmioRead (one per Ball) - for (i <- 0 until numBalls) { - mmioRead(i) <> balls(i).blink.mmioRead + // Connect configurable MMIO metadata channels. + var mmioReadChannelIdx = 0 + for (ball <- balls) { + val mmioReadBW = b.ballDomain.ballIdMappings + .find(_.ballName == ball.getClass.getSimpleName) + .map(_.mmioReadBW) + .getOrElse(0) + for (i <- 0 until mmioReadBW) { + mmioRead(mmioReadChannelIdx) <> ball.blink.mmioRead(i) + mmioReadChannelIdx = mmioReadChannelIdx + 1 + } } } diff --git a/arch/src/main/scala/framework/balldomain/blink/blink.scala b/arch/src/main/scala/framework/balldomain/blink/blink.scala index d7d5521c4..72912e46a 100644 --- a/arch/src/main/scala/framework/balldomain/blink/blink.scala +++ b/arch/src/main/scala/framework/balldomain/blink/blink.scala @@ -4,11 +4,19 @@ import chisel3._ import chisel3.util._ import framework.top.GlobalConfig import framework.balldomain.rs.{BallRsComplete, BallRsIssue} -import framework.balldomain.blink.mmio.MmioRead +import framework.balldomain.blink.mmio.{MmioRead, MmioWrite} import chisel3.experimental.hierarchy.{instantiable, public} -class BlinkIO(b: GlobalConfig, inBW: Int, outBW: Int) extends Bundle with HasBallStatus { - val status = new BallStatus() +class BlinkIO( + b: GlobalConfig, + inBW: Int, + outBW: Int, + mmioReadBW: Int = 0, + mmioWriteBW: Int = 0) + extends Bundle + with HasBallStatus { + val status = new BallStatus() + val channelReady = Input(Bool()) val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) val cmdResp = Decoupled(new BallRsComplete(b)) @@ -16,7 +24,8 @@ class BlinkIO(b: GlobalConfig, inBW: Int, outBW: Int) extends Bundle with HasBal val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) val subRobReq = Decoupled(new SubRobRow(b)) - // MMIO read channel (one per Ball, used for metadata access). - // Balls that don't consume MMIO should tie this off via MmioRead.tieOff. - val mmioRead = Flipped(new MmioRead(b)) + // MMIO metadata channels. Like bankRead/bankWrite, their number is a Ball + // configuration property rather than a framework-wide fixed interface. + val mmioRead = Vec(mmioReadBW, Flipped(new MmioRead(b))) + val mmioWrite = Vec(mmioWriteBW, Flipped(new MmioWrite(b))) } diff --git a/arch/src/main/scala/framework/balldomain/blink/mmio/MmioRead.scala b/arch/src/main/scala/framework/balldomain/blink/mmio/MmioRead.scala index 5dbce774c..7b3ed17ff 100644 --- a/arch/src/main/scala/framework/balldomain/blink/mmio/MmioRead.scala +++ b/arch/src/main/scala/framework/balldomain/blink/mmio/MmioRead.scala @@ -12,17 +12,16 @@ import framework.memdomain.backend.mmio.{MmioReadReq, MmioReadResp} * Defined from the CONSUMER (MmioRouter) perspective: * - req is Flipped(Decoupled) so the consumer drives ready and reads valid/bits * - resp is Decoupled so the consumer drives valid/bits and reads ready - * - ball_id, rob_id, meta_bank are Input (driven by the Ball when consumed) + * - ball_id, rob_id are Input (driven by the Ball when consumed) * * BlinkIO wraps this in Flipped(...), so from the Ball's perspective: * - req becomes Decoupled (Ball drives valid/bits, reads ready) * - resp becomes Flipped(Decoupled) (Ball reads valid/bits, drives ready) - * - ball_id, rob_id, meta_bank become Output (Ball drives them) + * - ball_id, rob_id become Output (Ball drives them) */ class MmioRead(val b: GlobalConfig) extends Bundle with HasBallId with HasRobId { - val req = Flipped(Decoupled(new MmioReadReq(b))) - val resp = Decoupled(new MmioReadResp(b)) - val meta_bank = Input(UInt(log2Up(b.memDomain.bankNum).W)) + val req = Flipped(Decoupled(new MmioReadReq(b))) + val resp = Decoupled(new MmioReadResp(b)) } object MmioRead { @@ -32,7 +31,7 @@ object MmioRead { * Ball's perspective after Flipped: * - req: Decoupled (Ball drives) - tie valid=0, bits=0 * - resp: Flipped(Decoupled) (Ball reads) - tie ready=0 - * - ball_id, rob_id, meta_bank: Output (Ball drives) - tie to 0 + * - ball_id, rob_id: Output (Ball drives) - tie to 0 */ def tieOff(port: MmioRead): Unit = { port.req.valid := false.B @@ -40,7 +39,12 @@ object MmioRead { port.resp.ready := false.B port.ball_id := 0.U port.rob_id := 0.U - port.meta_bank := 0.U + } + + def tieOff(ports: Vec[MmioRead]): Unit = { + for (port <- ports) { + tieOff(port) + } } } diff --git a/arch/src/main/scala/framework/balldomain/blink/mmio/MmioWrite.scala b/arch/src/main/scala/framework/balldomain/blink/mmio/MmioWrite.scala new file mode 100644 index 000000000..5e42c4d44 --- /dev/null +++ b/arch/src/main/scala/framework/balldomain/blink/mmio/MmioWrite.scala @@ -0,0 +1,34 @@ +package framework.balldomain.blink.mmio + +import chisel3._ +import chisel3.util.Decoupled +import framework.balldomain.blink.{HasBallId, HasRobId} +import framework.memdomain.backend.mmio.MmioWriteReq +import framework.top.GlobalConfig + +/** + * Ball-facing byte write port for the unified MMIO address space. + * + * BlinkIO wraps this in Flipped(...), so a Ball produces req and observes + * ready. A multi-byte value uses one configured line per byte. + */ +class MmioWrite(val b: GlobalConfig) extends Bundle with HasBallId with HasRobId { + val req = Flipped(Decoupled(new MmioWriteReq(b))) +} + +object MmioWrite { + + def tieOff(port: MmioWrite): Unit = { + port.req.valid := false.B + port.req.bits := 0.U.asTypeOf(port.req.bits) + port.ball_id := 0.U + port.rob_id := 0.U + } + + def tieOff(ports: Vec[MmioWrite]): Unit = { + for (port <- ports) { + tieOff(port) + } + } + +} diff --git a/arch/src/main/scala/framework/balldomain/configs/BallDomainParam.scala b/arch/src/main/scala/framework/balldomain/configs/BallDomainParam.scala index ec53d6fe5..3c30200ad 100644 --- a/arch/src/main/scala/framework/balldomain/configs/BallDomainParam.scala +++ b/arch/src/main/scala/framework/balldomain/configs/BallDomainParam.scala @@ -3,13 +3,14 @@ package framework.balldomain.configs import upickle.default._ case class BallIdMapping( - ballId: Int, - ballName: String, - ballClass: String, - config: Option[String] = None, - inBW: Int, - outBW: Int, - configBaseDir: String = "") + ballId: Int, + ballName: String, + ballClass: String, + config: Option[String] = None, + inBW: Int, + outBW: Int, + mmioReadBW: Int = 0, + mmioWriteBW: Int = 0) case class BallISAEntry( mnemonic: String, @@ -21,6 +22,28 @@ case class BallDomainParam( ballIdMappings: Seq[BallIdMapping], ballISA: Seq[BallISAEntry]) { + require(ballNum == ballIdMappings.length, "ballNum must match ballIdMappings length") + require( + ballIdMappings.map(_.ballId).distinct.length == ballIdMappings.length, + "ballIdMappings contains duplicate ballId" + ) + require( + ballISA.map(_.mnemonic).distinct.length == ballISA.length, + "ballISA contains duplicate mnemonic" + ) + require( + ballISA.map(_.funct7).distinct.length == ballISA.length, + "ballISA contains duplicate funct7" + ) + require( + ballISA.forall(entry => entry.funct7 >= 0 && entry.funct7 < 128), + "ballISA funct7 must fit the 7-bit instruction field" + ) + require( + ballISA.forall(entry => ballIdMappings.exists(_.ballId == entry.bid)), + "ballISA bid must reference a configured ballId" + ) + def mapping(ballName: String): BallIdMapping = ballIdMappings.find(_.ballName == ballName) match { case Some(m) => m diff --git a/arch/src/main/scala/framework/balldomain/configs/BallParamLoader.scala b/arch/src/main/scala/framework/balldomain/configs/BallParamLoader.scala index 6af6ba3d1..64283bda9 100644 --- a/arch/src/main/scala/framework/balldomain/configs/BallParamLoader.scala +++ b/arch/src/main/scala/framework/balldomain/configs/BallParamLoader.scala @@ -24,17 +24,14 @@ object BallParamLoader { case _ => throw new RuntimeException("Expected TOML section [ball]") } - private def configPath(mapping: BallIdMapping): Option[String] = { - mapping.config.map { config => - val path = Paths.get(config) - if (path.isAbsolute) { - throw new RuntimeException(s"Ball ${mapping.ballName} config must be relative to its balldomain TOML: $config") - } - if (mapping.configBaseDir.isEmpty) { - throw new RuntimeException(s"Ball ${mapping.ballName} config has no balldomain base directory") - } - Paths.get(mapping.configBaseDir).resolve(path).normalize().toString + private def configPath(mapping: BallIdMapping): String = { + val config = mapping.config.getOrElse( + throw new RuntimeException(s"Ball ${mapping.ballName} has no config") + ) + if (!Paths.get(config).toFile.isFile) { + throw new RuntimeException(s"Ball ${mapping.ballName} config not found: $config") } + config } def ballTable(b: GlobalConfig, ballName: String): Map[String, Value] = { @@ -42,10 +39,7 @@ object BallParamLoader { case Some(m) => m case None => throw new RuntimeException(s"No ballIdMapping for ballName=$ballName") } - configPath(mapping) match { - case Some(path) => ball(load(path)) - case None => throw new RuntimeException(s"Ball $ballName has no config") - } + ball(load(configPath(mapping))) } def int(table: Map[String, Value], key: String): Int = diff --git a/arch/src/main/scala/framework/frontend/configs/FrontendParam.scala b/arch/src/main/scala/framework/frontend/configs/FrontendParam.scala index 4c918e16c..dedf00289 100644 --- a/arch/src/main/scala/framework/frontend/configs/FrontendParam.scala +++ b/arch/src/main/scala/framework/frontend/configs/FrontendParam.scala @@ -10,6 +10,8 @@ case class FrontendParam( rs_out_of_order_response: Boolean, bank_id_len: Int, vbank_id_upper_bound: Int, + /** ISA bank-id namespace base for shared banks: raw id >= this means shared. */ + shared_bank_id_base: Int, iter_len: Int, sub_rob_enable: Boolean, sub_rob_depth: Int) @@ -22,6 +24,7 @@ object FrontendParam { rs_out_of_order_response = false, bank_id_len = 0, vbank_id_upper_bound = 0, + shared_bank_id_base = 0, iter_len = 0, sub_rob_enable = false, sub_rob_depth = 0 diff --git a/arch/src/main/scala/framework/frontend/decoder/GobalDecoder.scala b/arch/src/main/scala/framework/frontend/decoder/GobalDecoder.scala index 558a5cad1..c40bc44b8 100644 --- a/arch/src/main/scala/framework/frontend/decoder/GobalDecoder.scala +++ b/arch/src/main/scala/framework/frontend/decoder/GobalDecoder.scala @@ -57,7 +57,6 @@ class GlobalDecoder(val b: GlobalConfig) extends Module { val is_mem_inst = (func7 === MVIN_BITPAT) || (func7 === MVOUT_BITPAT) || (func7 === MSET_BITPAT) || - (func7 === MMIO_SET_BITPAT) || (func7 === MVIN_MMIO_BITPAT) val is_frontend_inst = func7 === FENCE_BITPAT diff --git a/arch/src/main/scala/framework/frontend/globalrs/GlobalROB.scala b/arch/src/main/scala/framework/frontend/globalrs/GlobalROB.scala index 8628de28e..92ce5f890 100644 --- a/arch/src/main/scala/framework/frontend/globalrs/GlobalROB.scala +++ b/arch/src/main/scala/framework/frontend/globalrs/GlobalROB.scala @@ -7,7 +7,7 @@ import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantia import framework.top.GlobalConfig import framework.frontend.decoder.{DomainId, PostGDCmd} import framework.frontend.scoreboard.{BankAccessInfo, BankAliasTable, BankScoreboard} -import framework.memdomain.frontend.cmd.decoder.DISA.{MMIO_SET_BITPAT, MSET_BITPAT} +import framework.memdomain.frontend.cmd.decoder.DISA.MSET_BITPAT @instantiable class GlobalROB(val b: GlobalConfig) extends Module { @@ -81,10 +81,11 @@ class GlobalROB(val b: GlobalConfig) extends Module { val robIssued = RegInit(VecInit(Seq.fill(robDepth)(false.B))) val robComplete = RegInit(VecInit(Seq.fill(robDepth)(false.B))) - val headPtr = RegInit(0.U(idWidth.W)) - val tailPtr = RegInit(0.U(idWidth.W)) - val issuedCount = RegInit(0.U(log2Up(robDepth + 1).W)) - val bankCols = RegInit(VecInit(Seq.fill(b.memDomain.bankNum)(0.U(log2Up(b.memDomain.bankNum + 1).W)))) + val headPtr = RegInit(0.U(idWidth.W)) + val tailPtr = RegInit(0.U(idWidth.W)) + val issuedCount = RegInit(0.U(log2Up(robDepth + 1).W)) + val bankCols = RegInit(VecInit(Seq.fill(b.memDomain.bankNum)(0.U(log2Up(b.memDomain.bankNum + 1).W)))) + val physicalBankBusy = RegInit(VecInit(Seq.fill(b.memDomain.bankNum)(false.B))) val isEmpty = headPtr === tailPtr && !robValid(headPtr) val isFull = headPtr === tailPtr && robValid(headPtr) @@ -95,7 +96,7 @@ class GlobalROB(val b: GlobalConfig) extends Module { def isMappingConfig(entry: GlobalRobEntry): Bool = entry.cmd.domain_id === DomainId.MEM && - ((entry.cmd.cmd.funct === MSET_BITPAT) || (entry.cmd.cmd.funct === MMIO_SET_BITPAT)) + (entry.cmd.cmd.funct === MSET_BITPAT) def touchesBank(access: BankAccessInfo, bank: UInt): Bool = (access.rd_bank_0_valid && access.rd_bank_0_id === bank) || @@ -126,6 +127,21 @@ class GlobalROB(val b: GlobalConfig) extends Module { youngerReadOlderWrite || youngerWriteOlderRead || youngerWriteOlderWrite } + def physicalBankBusyFor(access: BankAccessInfo): Bool = { + val busy = WireDefault(false.B) + for (bank <- 0 until b.memDomain.bankNum) { + val bankId = bank.U(b.frontend.bank_id_len.W) + when( + (access.rd_bank_0_valid && access.rd_bank_0_id === bankId) || + (access.rd_bank_1_valid && access.rd_bank_1_id === bankId) || + (access.wr_bank_valid && access.wr_bank_id === bankId) + ) { + busy := physicalBankBusy(bank) + } + } + busy + } + // --------------------------------------------------------------------------- // Allocate: enqueue decoded instruction into ROB // rob_id == tailPtr at allocation time (no separate counter needed) @@ -242,7 +258,8 @@ class GlobalROB(val b: GlobalConfig) extends Module { } scanValid(i) := robValid(ptr) && !robIssued(ptr) && !robComplete(ptr) scoreboard.queryVec(i) := robEntries(ptr).renamedBankAccess - scanReady(i) := scanValid(i) && !scoreboard.hazardVec(i) && !cfgHazard + scanReady(i) := scanValid(i) && !scoreboard.hazardVec(i) && !cfgHazard && + !physicalBankBusyFor(robEntries(ptr).cmd.bankAccess) } val hasReady = scanReady.asUInt.orR @@ -282,6 +299,18 @@ class GlobalROB(val b: GlobalConfig) extends Module { scoreboard.issue.valid := true.B scoreboard.issue.bits := robEntries(actualIssuePtr).renamedBankAccess + val access = robEntries(actualIssuePtr).cmd.bankAccess + for (bank <- 0 until b.memDomain.bankNum) { + val bankId = bank.U(b.frontend.bank_id_len.W) + when( + (access.rd_bank_0_valid && access.rd_bank_0_id === bankId) || + (access.rd_bank_1_valid && access.rd_bank_1_id === bankId) || + (access.wr_bank_valid && access.wr_bank_id === bankId) + ) { + physicalBankBusy(bank) := true.B + } + } + itraceIssue.io.is_issue := 1.U itraceIssue.io.rob_id := issueEntry.rob_id itraceIssue.io.domain_id := issueEntry.cmd.domain_id @@ -295,6 +324,20 @@ class GlobalROB(val b: GlobalConfig) extends Module { itraceIssue.io.enable := true.B } + when(io.complete.fire) { + val access = robEntries(io.complete.bits).cmd.bankAccess + for (bank <- 0 until b.memDomain.bankNum) { + val bankId = bank.U(b.frontend.bank_id_len.W) + when( + (access.rd_bank_0_valid && access.rd_bank_0_id === bankId) || + (access.rd_bank_1_valid && access.rd_bank_1_id === bankId) || + (access.wr_bank_valid && access.wr_bank_id === bankId) + ) { + physicalBankBusy(bank) := false.B + } + } + } + issuedCount := issuedCount + issueFired.asUInt - completeIssuedEntry.asUInt // --------------------------------------------------------------------------- diff --git a/arch/src/main/scala/framework/memdomain/MemDomain.scala b/arch/src/main/scala/framework/memdomain/MemDomain.scala index 4c524ac92..b68e64747 100644 --- a/arch/src/main/scala/framework/memdomain/MemDomain.scala +++ b/arch/src/main/scala/framework/memdomain/MemDomain.scala @@ -5,7 +5,7 @@ import chisel3.util._ import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} import freechips.rocketchip.tile._ import framework.balldomain.blink.{BankRead, BankWrite} -import framework.balldomain.blink.mmio.MmioRead +import framework.balldomain.blink.mmio.{MmioRead, MmioWrite} import freechips.rocketchip.tilelink.{TLBundle, TLEdgeOut} import framework.frontend.globalrs.{GlobalSchedComplete, GlobalSchedIssue} import framework.top.GlobalConfig @@ -20,6 +20,8 @@ import framework.memdomain.backend.MemBackend @instantiable class MemDomain(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { + val totalMmioRead = b.ballDomain.ballIdMappings.map(_.mmioReadBW).sum + val totalMmioWrite = b.ballDomain.ballIdMappings.map(_.mmioWriteBW).sum val totalBallRead = b.ballDomain.ballIdMappings.map(_.inBW).sum val totalBallWrite = b.ballDomain.ballIdMappings.map(_.outBW).sum @@ -31,10 +33,14 @@ class MemDomain(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { val busy = Output(Bool()) // Inside Channel + val ballChannelActive = Input(Vec(b.ballDomain.ballNum, Bool())) + val ballChannelReady = Output(Vec(b.ballDomain.ballNum, Bool())) + val ballDomain = new Bundle { val bankRead = Vec(totalBallRead, new BankRead(b)) val bankWrite = Vec(totalBallWrite, new BankWrite(b)) - val mmioRead = Vec(b.ballDomain.ballNum, new MmioRead(b)) + val mmioRead = Vec(totalMmioRead, new MmioRead(b)) + val mmioWrite = Vec(totalMmioWrite, new MmioWrite(b)) } // Outside Channel @@ -86,13 +92,18 @@ class MemDomain(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { midend.io.bankRead(i).bankRead <> io.ballDomain.bankRead(i) midend.io.bankRead(i).is_shared := false.B } + + midend.io.ballChannelActive := io.ballChannelActive + io.ballChannelReady := midend.io.ballChannelReady + for (i <- 0 until totalBallWrite) { midend.io.bankWrite(i).bankWrite <> io.ballDomain.bankWrite(i) midend.io.bankWrite(i).is_shared := false.B } + midend.io.bankRead(totalBallRead).bankRead <> frontend.io.interdma.bankRead midend.io.bankRead(totalBallRead).is_shared := frontend.io.interdma.read_is_shared - midend.io.hartid := io.hartid + midend.io.hartid := io.hartid midend.io.mem_req <> backend.io.mem_req backend.io.config <> frontend.io.config @@ -107,21 +118,13 @@ class MemDomain(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { if (b.memDomain.mmioEnable) { val mmioPool: Instance[MmioPool] = Instantiate(new MmioPool(b)) - // Alloc/dealloc from MemConfiger (mmio_set instruction) - mmioPool.io.alloc := frontend.io.mmioAlloc - - // Dealloc from MemConfiger (implicit lifecycle: mem_dealloc with alloc=false) - // For now, tie off explicit dealloc (lifecycle managed by mmio_set with size_rows=0) - mmioPool.io.dealloc.valid := false.B - mmioPool.io.dealloc.bits := 0.U - // Write path: route MemLoader's bankWrite to MmioPool when is_mvin_mmio_active val destIsMmio = frontend.io.is_mvin_mmio_active - // Compute MMIO write parameters from MemLoader's mmio_addr/col - val mmioRowAddr = frontend.io.mmio_addr(9, 4) + loaderBankWrite.io.req.bits.addr - val mmioBankIdx = frontend.io.mmio_addr(16, 10) - val mmioByteMask = Wire(Vec(b.memDomain.bankMaskLen, Bool())) + // MMIO is one globally encoded byte space. Each DMA beat is 16 bytes and + // MmioPool stripes those bytes across the five physical byte banks. + val mmioWriteAddr = frontend.io.mmio_addr + loaderBankWrite.io.req.bits.addr * (b.memDomain.bankWidth / 8).U + val mmioByteMask = Wire(Vec(b.memDomain.bankMaskLen, Bool())) for (k <- 0 until b.memDomain.bankMaskLen) { mmioByteMask(k) := k.U < frontend.io.mmio_col } @@ -133,10 +136,10 @@ class MemDomain(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { // Route write to MMIO or main bank based on is_mvin_mmio_active mmioPool.io.write.req.valid := loaderBankWrite.io.req.valid && destIsMmio - mmioPool.io.write.req.bits.addr := mmioRowAddr + mmioPool.io.write.req.bits.addr := loaderBankWrite.io.req.bits.addr mmioPool.io.write.req.bits.data := loaderBankWrite.io.req.bits.data mmioPool.io.write.req.bits.mask := mmioByteMask - mmioPool.io.writeBankIdx := mmioBankIdx + mmioPool.io.writeAddr := mmioWriteAddr // Main bank write (when NOT mvin_mmio). dmaBankWrite.io.req.valid := loaderBankWrite.io.req.valid && !destIsMmio @@ -165,22 +168,26 @@ class MemDomain(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { mmioPool.io.write.resp.ready := loaderBankWrite.io.resp.ready && destIsMmio dmaBankWrite.io.resp.ready := loaderBankWrite.io.resp.ready && !destIsMmio - // Ball read path: connect Ball mmioRead to MmioPool - for (i <- 0 until b.ballDomain.ballNum) { + // Ball read path: connect every configured Blink MMIO line to MmioPool. + for (i <- 0 until totalMmioRead) { mmioPool.io.ballReq(i) <> io.ballDomain.mmioRead(i).req io.ballDomain.mmioRead(i).resp <> mmioPool.io.ballResp(i) - mmioPool.io.ballMetaBank(i) := io.ballDomain.mmioRead(i).meta_bank + } + for (i <- 0 until totalMmioWrite) { + mmioPool.io.ballWriteReq(i) <> io.ballDomain.mmioWrite(i).req } } else { dmaBankWrite <> loaderBankWrite - assert(!frontend.io.mmioAlloc.valid, "MemDomain MMIO is disabled, but mmio_set was issued") assert(!frontend.io.is_mvin_mmio_active, "MemDomain MMIO is disabled, but mvin_mmio was issued") - for (i <- 0 until b.ballDomain.ballNum) { + for (i <- 0 until totalMmioRead) { io.ballDomain.mmioRead(i).req.ready := false.B io.ballDomain.mmioRead(i).resp.valid := false.B io.ballDomain.mmioRead(i).resp.bits := 0.U.asTypeOf(io.ballDomain.mmioRead(i).resp.bits) } + for (i <- 0 until totalMmioWrite) { + io.ballDomain.mmioWrite(i).req.ready := false.B + } } // Shared path passthrough diff --git a/arch/src/main/scala/framework/memdomain/backend/accpipe/AccPipe.scala b/arch/src/main/scala/framework/memdomain/backend/accpipe/AccPipe.scala index 0cf3f7d0d..03ec4e909 100644 --- a/arch/src/main/scala/framework/memdomain/backend/accpipe/AccPipe.scala +++ b/arch/src/main/scala/framework/memdomain/backend/accpipe/AccPipe.scala @@ -40,8 +40,9 @@ class AccPipe(val b: GlobalConfig) extends Module { val rd_data_reg = RegInit(0.U(b.memDomain.bankWidth.W)) val wr_hold = RegInit(false.B) val wr_ok_reg = RegInit(false.B) + val busy_reg = RegInit(false.B) - val canStart = !rd_hold && !wr_hold && !io.sramRead.resp.valid && !io.sramWrite.resp.valid + val canStart = !busy_reg && !rd_hold && !wr_hold && !io.sramRead.resp.valid && !io.sramWrite.resp.valid val hasWriteReq = io.mem_req.write.req.valid val wrReq = canStart && hasWriteReq val rdReq = canStart && !hasWriteReq && io.mem_req.read.req.valid @@ -91,5 +92,12 @@ class AccPipe(val b: GlobalConfig) extends Module { bank_id_reg := io.mem_req.bank_id } - io.busy := false.B + when(io.mem_req.read.req.fire || io.mem_req.write.req.fire) { + busy_reg := true.B + } + when(io.mem_req.read.resp.fire || io.mem_req.write.resp.fire) { + busy_reg := false.B + } + + io.busy := busy_reg } diff --git a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioBank.scala b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioBank.scala index b8208e272..cf9d754a9 100644 --- a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioBank.scala +++ b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioBank.scala @@ -3,33 +3,37 @@ package framework.memdomain.backend.mmio import chisel3._ import chisel3.util._ import chisel3.experimental.hierarchy.{instantiable, public} -import framework.memdomain.backend.banks.SramWriteIO import framework.top.GlobalConfig /** * MmioBank: single MMIO SRAM bank. * - * Internal storage : mmioBankEntries x mmioBankWidth bits (default 64 x 128 = 1 KB) - * Write port : full mmioBankWidth (from DMA), reuses SramWriteIO for compatibility - * Read port : mmioReadWidth bits (default 8), via byte_offset mux from 128-bit word + * Internal storage : mmioBankEntries x mmioReadWidth bits (default 1024 x 8 = 1 KB) + * Write port : one byte; MmioPool distributes DMA beats across banks. + * Read port : one byte. * - * Read latency : 1 cycle (SyncReadMem) + combinational byte-mux registered = 1 cycle - * Write/read : single-port semantics. Write takes priority when both valid. + * Read latency : 1 cycle. Write/read uses single-port semantics; write wins. */ @instantiable class MmioBank(val b: GlobalConfig) extends Module { - private val numEntries = b.memDomain.mmioBankEntries - private val width = b.memDomain.mmioBankWidth - private val bytesPerRow = width / 8 + private val numEntries = b.memDomain.mmioBankEntries + require( + b.memDomain.mmioBankWidth == b.memDomain.mmioReadWidth, + "MmioBank requires one physical MMIO element per read" + ) + require( + b.memDomain.mmioReadWidth == 8, + "MmioBank requires 8-bit MMIO elements" + ) @public val io = IO(new Bundle { - val write = new SramWriteIO(b) + val write = new MmioBankWriteIO(b) val read = new MmioBankReadIO(b) }) - val mem = SyncReadMem(numEntries, UInt(width.W)) + val mem = SyncReadMem(numEntries, UInt(b.memDomain.mmioReadWidth.W)) io.write.req.ready := true.B io.read.req.ready := !io.write.req.valid @@ -37,16 +41,10 @@ class MmioBank(val b: GlobalConfig) extends Module { when(io.write.req.fire) { mem.write(io.write.req.bits.addr, io.write.req.bits.data) } - io.write.resp.valid := RegNext(io.write.req.fire, false.B) - io.write.resp.bits.ok := RegNext(io.write.req.fire, false.B) val ren = io.read.req.fire val rdata = mem.read(io.read.req.bits.addr, ren) - val byteOffsetReg = RegNext(io.read.req.bits.byte_offset, 0.U) - val byteSel = - (rdata >> (byteOffsetReg << 3.U))(b.memDomain.mmioReadWidth - 1, 0) - io.read.resp.valid := RegNext(ren, false.B) - io.read.resp.bits.data := byteSel + io.read.resp.bits.data := rdata } diff --git a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioIO.scala b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioIO.scala index bbdd2f5b4..345990326 100644 --- a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioIO.scala +++ b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioIO.scala @@ -14,8 +14,7 @@ import framework.top.GlobalConfig // ===== Per-MmioBank internal read port ===== class MmioBankReadReq(b: GlobalConfig) extends Bundle { - val addr = UInt(log2Ceil(b.memDomain.mmioBankEntries).W) - val byte_offset = UInt(log2Ceil(b.memDomain.mmioBankWidth / 8).W) + val addr = UInt(log2Ceil(b.memDomain.mmioBankEntries).W) } class MmioBankReadResp(b: GlobalConfig) extends Bundle { @@ -27,26 +26,26 @@ class MmioBankReadIO(b: GlobalConfig) extends Bundle { val resp = Decoupled(new MmioBankReadResp(b)) } -// ===== Ball-facing read port (carried over BlinkIO) ===== +class MmioBankWriteReq(b: GlobalConfig) extends Bundle { + val addr = UInt(log2Ceil(b.memDomain.mmioBankEntries).W) + val data = UInt(b.memDomain.mmioReadWidth.W) +} + +class MmioBankWriteIO(b: GlobalConfig) extends Bundle { + val req = Flipped(Decoupled(new MmioBankWriteReq(b))) +} + +// ===== Ball-facing MMIO ports (carried over BlinkIO) ===== class MmioReadReq(b: GlobalConfig) extends Bundle { - // Byte offset relative to the region base. - // Sized to cover the largest possible region (= totalBytes). - val rel_addr = UInt(log2Ceil(b.memDomain.mmioTotalBytes).W) + // MMIO is one globally encoded byte address space. + val addr = UInt(log2Ceil(b.memDomain.mmioTotalBytes).W) } class MmioReadResp(b: GlobalConfig) extends Bundle { val data = UInt(b.memDomain.mmioReadWidth.W) } -// ===== Region table alloc/dealloc ports ===== -class MmioAllocReq(b: GlobalConfig) extends Bundle { - val main_bank = UInt(log2Up(b.memDomain.bankNum).W) - val mmio_addr = UInt(log2Ceil(b.memDomain.mmioTotalBytes).W) - val size_rows = UInt(8.W) // 0 means dealloc -} - -class MmioRegionEntry(b: GlobalConfig) extends Bundle { - val valid = Bool() - val mmio_addr = UInt(log2Ceil(b.memDomain.mmioTotalBytes).W) - val size_rows = UInt(8.W) +class MmioWriteReq(b: GlobalConfig) extends Bundle { + val addr = UInt(log2Ceil(b.memDomain.mmioTotalBytes).W) + val data = UInt(b.memDomain.mmioReadWidth.W) } diff --git a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioPool.scala b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioPool.scala index a6823fe6d..79be706ee 100644 --- a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioPool.scala +++ b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioPool.scala @@ -11,51 +11,38 @@ import framework.top.GlobalConfig * * Instantiates: * - Vec of MmioBanks (one per mmioBankNum) - * - MmioRegionTable (binding table) * - MmioRouter (translates Ball requests to bank requests) * * External interfaces: - * - alloc/dealloc: from MemConfiger (mmio_set / mem_dealloc instructions) - * - write/writeBankIdx: from MemLoader (mvin_mmio DMA path) - * - ballReq/ballResp/ballMetaBank: from Balls (via MemDomain wiring) + * - write: from MemLoader (mvin_mmio DMA path) + * - ballReq/ballResp/ballWriteReq: from Balls (via MemDomain wiring) */ @instantiable class MmioPool(val b: GlobalConfig) extends Module { + private val mmioReadPorts = b.ballDomain.ballIdMappings.map(_.mmioReadBW).sum + private val mmioWritePorts = b.ballDomain.ballIdMappings.map(_.mmioWriteBW).sum + @public val io = IO(new Bundle { - // Region table management (from MemConfiger) - val alloc = Flipped(Valid(new MmioAllocReq(b))) - val dealloc = Flipped(Valid(UInt(log2Up(b.memDomain.bankNum).W))) - // DMA write path (from MemLoader) - val write = new SramWriteIO(b) - val writeBankIdx = Input(UInt(log2Ceil(b.memDomain.mmioBankNum).W)) + val write = new SramWriteIO(b) + val writeAddr = Input(UInt(log2Ceil(b.memDomain.mmioTotalBytes).W)) - // Ball read path (from MemDomain, one port per Ball) + // Ball read path (from MemDomain, one port per configured Blink line) val ballReq = - Vec(b.ballDomain.ballNum, Flipped(Decoupled(new MmioReadReq(b)))) - val ballResp = Vec(b.ballDomain.ballNum, Decoupled(new MmioReadResp(b))) - val ballMetaBank = - Input(Vec(b.ballDomain.ballNum, UInt(log2Up(b.memDomain.bankNum).W))) + Vec(mmioReadPorts, Flipped(Decoupled(new MmioReadReq(b)))) + val ballResp = Vec(mmioReadPorts, Decoupled(new MmioReadResp(b))) + val ballWriteReq = Vec(mmioWritePorts, Flipped(Decoupled(new MmioWriteReq(b)))) }) // Instantiate components val banks = Seq.fill(b.memDomain.mmioBankNum)(Instantiate(new MmioBank(b))) - val table = Instantiate(new MmioRegionTable(b)) val router = Instantiate(new MmioRouter(b)) - // Wire region table - table.io.alloc := io.alloc - table.io.dealloc := io.dealloc - - // Wire router to table and Balls + // MMIO read addresses are already globally encoded; no region translation. router.io.ballReq <> io.ballReq io.ballResp <> router.io.ballResp - router.io.ballMetaBank := io.ballMetaBank - - table.io.lookup_main_bank := router.io.tableLookupBank - router.io.tableLookupResult := table.io.lookup_result // Wire router to banks (read path) for (i <- 0 until b.memDomain.mmioBankNum) { @@ -63,30 +50,93 @@ class MmioPool(val b: GlobalConfig) extends Module { banks(i).io.read.resp <> router.io.bankReadResp(i) } - // Wire DMA write path: demux io.write to banks(writeBankIdx) - for (i <- 0 until b.memDomain.mmioBankNum) { - banks( - i - ).io.write.req.valid := io.write.req.valid && (io.writeBankIdx === i.U) - banks(i).io.write.req.bits := io.write.req.bits - // resp.ready needs to be driven so the bank knows the consumer accepts responses - banks( - i - ).io.write.resp.ready := io.write.resp.ready && (io.writeBankIdx === i.U) + // A DMA beat is 128 bits. The globally encoded byte address is striped over + // the homogeneous 8-bit MMIO banks, at most one byte per bank each cycle. + val dmaBytes = b.memDomain.bankWidth / 8 + val writeActive = RegInit(false.B) + val writeAddrReg = RegInit(0.U(log2Ceil(b.memDomain.mmioTotalBytes).W)) + val writeDataReg = RegInit(0.U(b.memDomain.bankWidth.W)) + val writeMaskReg = RegInit(VecInit(Seq.fill(b.memDomain.bankMaskLen)(false.B))) + val writeCursorReg = RegInit(0.U(log2Ceil(dmaBytes).W)) + + io.write.req.ready := !writeActive + when(io.write.req.fire) { + assert(io.writeAddr + dmaBytes.U <= b.memDomain.mmioTotalBytes.U, "MmioPool: DMA write exceeds MMIO address space") + writeActive := true.B + writeAddrReg := io.writeAddr + writeDataReg := io.write.req.bits.data + writeMaskReg := io.write.req.bits.mask + writeCursorReg := 0.U } - val writeRespMux = Mux1H( - UIntToOH(io.writeBankIdx, b.memDomain.mmioBankNum), - banks.map(_.io.write.resp) - ) + val finalWriteGroup = writeActive && + (writeCursorReg +& b.memDomain.mmioBankNum.U >= dmaBytes.U) + for (bankIdx <- 0 until b.memDomain.mmioBankNum) { + val byteMatch = VecInit((0 until b.memDomain.mmioBankNum).map { offset => + val byteIdx = writeCursorReg +& offset.U + val byteAddr = writeAddrReg + byteIdx + byteIdx < dmaBytes.U && (byteAddr % b.memDomain.mmioBankNum.U === bankIdx.U) + }) + val selectedOffset = PriorityEncoder(byteMatch) + val selectedByte = writeCursorReg +& selectedOffset + val selectedAddr = writeAddrReg + selectedByte + val dmaWriteValid = writeActive && byteMatch.asUInt.orR + val dmaWriteAddr = (selectedAddr / b.memDomain.mmioBankNum.U)(log2Ceil(b.memDomain.mmioBankEntries) - 1, 0) + val selectedData = (writeDataReg >> (selectedByte << 3.U))(7, 0) + val dmaWriteData = Mux(writeMaskReg(selectedByte), selectedData, 0.U) - io.write.resp.valid := writeRespMux.valid - io.write.resp.bits := writeRespMux.bits + if (mmioWritePorts == 0) { + banks(bankIdx).io.write.req.valid := dmaWriteValid + banks(bankIdx).io.write.req.bits.addr := dmaWriteAddr + banks(bankIdx).io.write.req.bits.data := dmaWriteData + } else { + val ballMatches = VecInit((0 until mmioWritePorts).map { i => + io.ballWriteReq(i).valid && + (io.ballWriteReq(i).bits.addr % b.memDomain.mmioBankNum.U === bankIdx.U) + }) + val ballIdx = PriorityEncoder(ballMatches) + val ballAddr = io.ballWriteReq(ballIdx).bits.addr + assert( + PopCount(ballMatches) <= 1.U, + "MmioPool: multiple Ball writes target one MMIO bank in a cycle" + ) + assert( + !(writeActive && ballMatches.asUInt.orR), + "MmioPool: DMA and Ball write target one MMIO bank in a cycle" + ) + banks(bankIdx).io.write.req.valid := dmaWriteValid || ballMatches.asUInt.orR + banks(bankIdx).io.write.req.bits.addr := Mux( + dmaWriteValid, + dmaWriteAddr, + (ballAddr / b.memDomain.mmioBankNum.U)(log2Ceil(b.memDomain.mmioBankEntries) - 1, 0) + ) + banks(bankIdx).io.write.req.bits.data := Mux( + dmaWriteValid, + dmaWriteData, + io.ballWriteReq(ballIdx).bits.data + ) + } + } - val writeReadyMux = Mux1H( - UIntToOH(io.writeBankIdx, b.memDomain.mmioBankNum), - banks.map(_.io.write.req.ready) - ) + for (i <- 0 until mmioWritePorts) { + val targetBank = io.ballWriteReq(i).bits.addr % b.memDomain.mmioBankNum.U + val targetReady = Mux1H( + UIntToOH(targetBank, b.memDomain.mmioBankNum), + banks.map(_.io.write.req.ready) + ) + io.ballWriteReq(i).ready := !writeActive && targetReady + assert( + !(io.ballWriteReq(i).valid && + (io.ballWriteReq(i).bits.addr >= b.memDomain.mmioTotalBytes.U)), + "MmioPool: Ball write exceeds the MMIO address space" + ) + } - io.write.req.ready := writeReadyMux + io.write.resp.valid := finalWriteGroup + io.write.resp.bits.ok := true.B + when(io.write.resp.fire) { + writeActive := false.B + }.elsewhen(writeActive && !finalWriteGroup) { + writeCursorReg := writeCursorReg + b.memDomain.mmioBankNum.U + } } diff --git a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRegionTable.scala b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRegionTable.scala deleted file mode 100644 index b08652b0c..000000000 --- a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRegionTable.scala +++ /dev/null @@ -1,64 +0,0 @@ -package framework.memdomain.backend.mmio - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import framework.top.GlobalConfig - -/** - * MmioRegionTable: hardware binding table. - * - * table[main_bank_id] = { valid, mmio_addr, size_rows } - * - * alloc port : driven by MemConfiger when mmio_set instruction fires. - * size_rows == 0 means dealloc (valid := false). - * size_rows > 0 means alloc/overwrite. - * - * dealloc port: driven by MemConfiger when mem_dealloc (mset with alloc=false) fires, - * performing the implicit lifecycle release. - * - * lookup port : vector combinational read by MmioRouter for in-flight Ball reads - * (one entry per Ball, sized by ballDomain.ballNum). - */ -@instantiable -class MmioRegionTable(val b: GlobalConfig) extends Module { - - @public - val io = IO(new Bundle { - val alloc = Flipped(Valid(new MmioAllocReq(b))) - val dealloc = Flipped(Valid(UInt(log2Up(b.memDomain.bankNum).W))) - - // Vector lookup: one combinational port per Ball. - val lookup_main_bank = - Input(Vec(b.ballDomain.ballNum, UInt(log2Up(b.memDomain.bankNum).W))) - val lookup_result = - Output(Vec(b.ballDomain.ballNum, new MmioRegionEntry(b))) - }) - - val table = RegInit( - VecInit( - Seq.fill(b.memDomain.bankNum)( - 0.U.asTypeOf(new MmioRegionEntry(b)) - ) - ) - ) - - when(io.alloc.valid) { - val isDealloc = io.alloc.bits.size_rows === 0.U - when(isDealloc) { - table(io.alloc.bits.main_bank).valid := false.B - }.otherwise { - table(io.alloc.bits.main_bank).valid := true.B - table(io.alloc.bits.main_bank).mmio_addr := io.alloc.bits.mmio_addr - table(io.alloc.bits.main_bank).size_rows := io.alloc.bits.size_rows - } - } - - when(io.dealloc.valid) { - table(io.dealloc.bits).valid := false.B - } - - for (i <- 0 until b.ballDomain.ballNum) { - io.lookup_result(i) := table(io.lookup_main_bank(i)) - } -} diff --git a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRouter.scala b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRouter.scala index cc922ce67..fa06426e4 100644 --- a/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRouter.scala +++ b/arch/src/main/scala/framework/memdomain/backend/mmio/MmioRouter.scala @@ -8,42 +8,28 @@ import framework.top.GlobalConfig /** * MmioRouter: translates Ball mmioRead requests into MmioBank requests. * - * Per Ball (parameter `ballNum` from ballDomain): - * 1. Read meta_bank from current cmd (provided externally by the Router's parent). - * 2. Look up regionTable[meta_bank]. - * 3. Compute absolute byte address = entry.mmio_addr + req.rel_addr. - * 4. Decode (bank_id, entry, byte_offset): - * bytesPerRow = mmioBankWidth / 8 - * rowsPerBank = mmioBankEntries - * bytesPerBank = bytesPerRow * rowsPerBank - * bank_id = absAddr / bytesPerBank - * rowInBank = (absAddr % bytesPerBank) / bytesPerRow - * byteOffset = absAddr % bytesPerRow - * 5. Route req to bankPorts(bank_id). + * Per configured Blink MMIO line: + * 1. Receive a globally encoded MMIO byte address. + * 2. Decode the byte address: + * bank_id = absAddr % mmioBankNum + * rowInBank = absAddr / mmioBankNum + * 3. Route req to bankPorts(bank_id). * * Concurrency rule (enforced by software): - * At any time, each mmio_bank is accessed by at most one Ball. + * At any time, each mmio_bank is accessed by at most one requester. * Therefore no arbitration is required — bank ports are driven by exactly the matching Ball. */ @instantiable class MmioRouter(val b: GlobalConfig) extends Module { - private val ballNum = b.ballDomain.ballNum - private val mmioBankNum = b.memDomain.mmioBankNum - private val bytesPerRow = b.memDomain.mmioBankWidth / 8 - private val bytesPerBank = bytesPerRow * b.memDomain.mmioBankEntries + private val clientNum = b.ballDomain.ballIdMappings.map(_.mmioReadBW).sum + private val mmioBankNum = b.memDomain.mmioBankNum @public val io = IO(new Bundle { // From Balls (via MemDomain wiring) - val ballReq = Vec(ballNum, Flipped(Decoupled(new MmioReadReq(b)))) - val ballResp = Vec(ballNum, Decoupled(new MmioReadResp(b))) - val ballMetaBank = Input(Vec(ballNum, UInt(log2Up(b.memDomain.bankNum).W))) - - // Region table lookup - val tableLookupBank = - Output(Vec(ballNum, UInt(log2Up(b.memDomain.bankNum).W))) - val tableLookupResult = Input(Vec(ballNum, new MmioRegionEntry(b))) + val ballReq = Vec(clientNum, Flipped(Decoupled(new MmioReadReq(b)))) + val ballResp = Vec(clientNum, Decoupled(new MmioReadResp(b))) // Bank ports (one per MmioBank) val bankReadReq = Vec(mmioBankNum, Decoupled(new MmioBankReadReq(b))) @@ -51,72 +37,63 @@ class MmioRouter(val b: GlobalConfig) extends Module { Vec(mmioBankNum, Flipped(Decoupled(new MmioBankReadResp(b)))) }) - io.tableLookupBank := io.ballMetaBank - val ballAbsAddr = Wire( - Vec(ballNum, UInt(log2Ceil(b.memDomain.mmioTotalBytes).W)) + Vec(clientNum, UInt(log2Ceil(b.memDomain.mmioTotalBytes).W)) ) - val ballBankId = Wire(Vec(ballNum, UInt(log2Ceil(mmioBankNum).W))) + val ballBankId = Wire(Vec(clientNum, UInt(log2Ceil(mmioBankNum).W))) val ballRowInBank = Wire( - Vec(ballNum, UInt(log2Ceil(b.memDomain.mmioBankEntries).W)) + Vec(clientNum, UInt(log2Ceil(b.memDomain.mmioBankEntries).W)) ) - val ballByteOff = Wire(Vec(ballNum, UInt(log2Ceil(bytesPerRow).W))) - val ballValid = Wire(Vec(ballNum, Bool())) - - for (i <- 0 until ballNum) { - val entry = io.tableLookupResult(i) - ballValid(i) := entry.valid - ballAbsAddr(i) := entry.mmio_addr + io.ballReq(i).bits.rel_addr - ballBankId(i) := (ballAbsAddr(i) / bytesPerBank.U)( + for (i <- 0 until clientNum) { + ballAbsAddr(i) := io.ballReq(i).bits.addr + ballBankId(i) := (ballAbsAddr(i) % mmioBankNum.U)( log2Ceil(mmioBankNum) - 1, 0 ) - ballRowInBank(i) := ((ballAbsAddr(i) % bytesPerBank.U) / bytesPerRow.U)( + ballRowInBank(i) := (ballAbsAddr(i) / mmioBankNum.U)( log2Ceil(b.memDomain.mmioBankEntries) - 1, 0 ) - ballByteOff(i) := (ballAbsAddr(i) % bytesPerRow.U)( - log2Ceil(bytesPerRow) - 1, - 0 - ) assert( - !(io.ballReq(i).valid && !ballValid(i)), - "MmioRouter: Ball %d issued mmioRead with no active region binding (meta_bank=%d)\n", - i.U, - io.ballMetaBank(i) + !(io.ballReq(i).valid && (ballAbsAddr(i) >= b.memDomain.mmioTotalBytes.U)), + "MmioRouter: read exceeds the MMIO address space\n" ) } // Per-bank request routing for (bankIdx <- 0 until mmioBankNum) { - val ballMatches = VecInit((0 until ballNum).map { i => - io.ballReq(i).valid && ballValid(i) && (ballBankId(i) === bankIdx.U) - }) - val matchedBall = PriorityEncoder(ballMatches) - val anyMatch = ballMatches.asUInt.orR - - io.bankReadReq(bankIdx).valid := anyMatch - io.bankReadReq(bankIdx).bits.addr := ballRowInBank(matchedBall) - io.bankReadReq(bankIdx).bits.byte_offset := ballByteOff(matchedBall) - - assert( - PopCount(ballMatches) <= 1.U, - "MmioRouter: bank %d hit by multiple Balls in the same cycle (software invariant violated)\n", - bankIdx.U - ) + if (clientNum == 0) { + io.bankReadReq(bankIdx).valid := false.B + io.bankReadReq(bankIdx).bits.addr := 0.U + } else { + val ballMatches = VecInit((0 until clientNum).map { i => + io.ballReq(i).valid && (ballBankId(i) === bankIdx.U) + }) + val matchedBall = PriorityEncoder(ballMatches) + val anyMatch = ballMatches.asUInt.orR + + io.bankReadReq(bankIdx).valid := anyMatch + io.bankReadReq(bankIdx).bits.addr := ballRowInBank(matchedBall) + + assert( + PopCount(ballMatches) <= 1.U, + "MmioRouter: bank %d hit by multiple requesters in the same cycle (software invariant violated)\n", + bankIdx.U + ) + } } // Per-Ball req.ready - for (i <- 0 until ballNum) { - io.ballReq(i).ready := ballValid(i) && io.bankReadReq(ballBankId(i)).ready + for (i <- 0 until clientNum) { + io.ballReq(i).ready := io.bankReadReq(ballBankId(i)).ready } // Response routing: each Ball has a small skid buffer holding in-flight bank_id - val inFlightBankPerBall = Seq.fill(ballNum)( + val inFlightBankPerBall = Seq.fill(clientNum)( Module( new Queue( UInt(log2Ceil(mmioBankNum).W), @@ -127,12 +104,12 @@ class MmioRouter(val b: GlobalConfig) extends Module { ) ) - for (i <- 0 until ballNum) { + for (i <- 0 until clientNum) { inFlightBankPerBall(i).io.enq.valid := io.ballReq(i).fire inFlightBankPerBall(i).io.enq.bits := ballBankId(i) } - for (i <- 0 until ballNum) { + for (i <- 0 until clientNum) { val q = inFlightBankPerBall(i) val srcBank = q.io.deq.bits val srcRespVal = io.bankReadResp(srcBank).valid @@ -144,11 +121,15 @@ class MmioRouter(val b: GlobalConfig) extends Module { } for (bankIdx <- 0 until mmioBankNum) { - val anyExpect = VecInit((0 until ballNum).map { i => - inFlightBankPerBall(i).io.deq.valid && - (inFlightBankPerBall(i).io.deq.bits === bankIdx.U) && - io.ballResp(i).ready - }).asUInt.orR - io.bankReadResp(bankIdx).ready := anyExpect + if (clientNum == 0) { + io.bankReadResp(bankIdx).ready := false.B + } else { + val anyExpect = VecInit((0 until clientNum).map { i => + inFlightBankPerBall(i).io.deq.valid && + (inFlightBankPerBall(i).io.deq.bits === bankIdx.U) && + io.ballResp(i).ready + }).asUInt.orR + io.bankReadResp(bankIdx).ready := anyExpect + } } } diff --git a/arch/src/main/scala/framework/memdomain/backend/privatepath/PrivateMemBackend.scala b/arch/src/main/scala/framework/memdomain/backend/privatepath/PrivateMemBackend.scala index 496f964b1..698d3e533 100644 --- a/arch/src/main/scala/framework/memdomain/backend/privatepath/PrivateMemBackend.scala +++ b/arch/src/main/scala/framework/memdomain/backend/privatepath/PrivateMemBackend.scala @@ -82,6 +82,10 @@ class PrivateMemBackend(val b: GlobalConfig) extends Module { } def getFreePbankId(): UInt = { + val hasFree = mappingTable.map(_.valid === false.B).reduce(_ || _) + when(!hasFree) { + assert(false.B, "PrivateMemBackend allocation failed: no free physical bank\n") + } val freePbankId = mappingTable.indexWhere(_.valid === false.B) freePbankId } @@ -130,6 +134,11 @@ class PrivateMemBackend(val b: GlobalConfig) extends Module { when(io.config.fire) { when(io.config.bits.alloc) { + // Match bemu mset: realloc of the same vbank frees prior physical banks first. + // MemConfiger emits one fire per group; only group 0 drops the old mapping. + when(io.config.bits.group_id === 0.U) { + deleteEntry(io.config.bits.vbank_id) + } addEntry( io.config.bits.vbank_id, getFreePbankId(), @@ -222,9 +231,7 @@ class PrivateMemBackend(val b: GlobalConfig) extends Module { (!mappingTable(j).is_multi || (mappingTable(j).is_multi && (mappingTable(j).group_id === activeGroup))) - val hold_one = RegNext(hit_bank && req_valid, init = false.B) - - when((hit_bank && req_valid) || hold_one) { + when(hit_bank && req_valid) { banks(j).io.sramRead <> accPipes(i).io.sramRead banks(j).io.sramWrite <> accPipes(i).io.sramWrite } diff --git a/arch/src/main/scala/framework/memdomain/backend/shared/SharedMemBackend.scala b/arch/src/main/scala/framework/memdomain/backend/shared/SharedMemBackend.scala index 1b9ca4369..af46fabbf 100644 --- a/arch/src/main/scala/framework/memdomain/backend/shared/SharedMemBackend.scala +++ b/arch/src/main/scala/framework/memdomain/backend/shared/SharedMemBackend.scala @@ -99,7 +99,10 @@ class SharedMemBackend(val b: GlobalConfig) extends Module { when(!found) { assert(false.B, "SharedMemBackend release missing allocation: hart=%d vbank=%d\n", hart_id, vbank_id) } + clearVbank(hart_id, vbank_id) + } + def clearVbank(hart_id: UInt, vbank_id: UInt): Unit = { for (i <- 0 until totalBanks) { when(mappingTable(i).valid && mappingTable(i).vbank_id === vbank_id && mappingTable(i).hart_id === hart_id) { mappingTable(i).valid := false.B @@ -161,6 +164,9 @@ class SharedMemBackend(val b: GlobalConfig) extends Module { when(io.config.fire) { when(io.config.bits.alloc) { + when(io.config.bits.group_id === 0.U) { + clearVbank(io.config.bits.hart_id, io.config.bits.vbank_id) + } val pbankId = getFreePbankId() printf( p"[SharedMemBackend][ALLOC] hart=${io.config.bits.hart_id} vbank=0x${Hexadecimal(io.config.bits.vbank_id)} " + diff --git a/arch/src/main/scala/framework/memdomain/configs/MemDomainParam.scala b/arch/src/main/scala/framework/memdomain/configs/MemDomainParam.scala index c140ce0d8..36f1af9d6 100644 --- a/arch/src/main/scala/framework/memdomain/configs/MemDomainParam.scala +++ b/arch/src/main/scala/framework/memdomain/configs/MemDomainParam.scala @@ -5,8 +5,8 @@ import upickle.default._ /** * MemDomain Parameter * - * Real values are injected by `TomlConfigLoader`. The no-arg `apply()` is a - * stub used only by `GlobalConfig()` before TOML overrides land — matching + * Real values are injected by `ChipLoader`. The no-arg `apply()` is a + * stub used only by `GlobalConfig()` before bundle overrides land — matching * how `BallDomainParam()` returns an empty default. */ case class MemDomainParam( @@ -41,7 +41,7 @@ case class MemDomainParam( object MemDomainParam { implicit val rw: ReadWriter[MemDomainParam] = macroRW - /** Stub default; real values come from TomlConfigLoader. */ + /** Stub default; real values come from ChipLoader. */ def apply(): MemDomainParam = MemDomainParam( bankNum = 0, bankWidth = 0, diff --git a/arch/src/main/scala/framework/memdomain/frontend/MemFrontend.scala b/arch/src/main/scala/framework/memdomain/frontend/MemFrontend.scala index cd7837015..4434d4a73 100644 --- a/arch/src/main/scala/framework/memdomain/frontend/MemFrontend.scala +++ b/arch/src/main/scala/framework/memdomain/frontend/MemFrontend.scala @@ -9,7 +9,6 @@ import framework.memdomain.frontend.mem.tlb.{BBTLBCluster, BBTLBExceptionIO, BBT import freechips.rocketchip.tilelink.{TLBundle, TLEdgeOut} import framework.frontend.globalrs.{GlobalSchedComplete, GlobalSchedIssue} import framework.balldomain.blink.{BankRead, BankWrite} -import framework.memdomain.backend.mmio.MmioAllocReq import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} import framework.top.GlobalConfig import framework.memdomain.frontend.cmd.decoder.MemDomainDecoder @@ -52,7 +51,6 @@ class MemFrontend(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { val config = Decoupled(new MemConfigerIO(b)) // MMIO outputs (to MmioPool via MemDomain) - val mmioAlloc = Valid(new MmioAllocReq(b)) val is_mvin_mmio_active = Output(Bool()) val mmio_addr = Output(UInt(17.W)) val mmio_col = Output(UInt(8.W)) @@ -132,7 +130,7 @@ class MemFrontend(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { memRs.io.commit_i.st <> memStorer.io.cmdResp memRs.io.commit_i.cf <> configer.io.cmdResp -//----------------------------------------------------------------------------- +//===-------------------------------------------------------------------===//-- // PMC - Performance Monitor Counter // ----------------------------------------------------------------------------- pmc.io.ldReq_i.valid := memRs.io.issue_o.ld.fire @@ -236,7 +234,6 @@ class MemFrontend(val b: GlobalConfig)(edge: TLEdgeOut) extends Module { io.interdma.write_is_shared := Mux(selectConfigOwner, false.B, memLoader.io.is_shared) // MMIO signals from MemConfiger and MemLoader, exposed to MemDomain - io.mmioAlloc := configer.io.mmioAlloc io.is_mvin_mmio_active := memLoader.io.is_mvin_mmio_active io.mmio_addr := memLoader.io.mmio_addr io.mmio_col := memLoader.io.mmio_col diff --git a/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DISA.scala b/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DISA.scala index f9e9a1fd6..61381cf21 100644 --- a/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DISA.scala +++ b/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DISA.scala @@ -9,7 +9,6 @@ object DISA { // enable=010 (1 write), opcode group val MSET_BITPAT = BitPat("b0100000") // 32 (0x20) — enable=010, opcode=0 val MVIN_BITPAT = BitPat("b0100001") // 33 (0x21) — enable=010, opcode=1 - val MMIO_SET_BITPAT = BitPat("b0100010") // 34 (0x22) — enable=010, opcode=2 (alloc/dealloc MMIO binding) val MVIN_MMIO_BITPAT = BitPat("b0100011") // 35 (0x23) — enable=010, opcode=3 (DMA into MMIO) // enable=001 (1 read), opcode group val MVOUT_BITPAT = BitPat("b0010000") // 16 (0x10) — enable=001, opcode=0 diff --git a/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DomainDecoder.scala b/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DomainDecoder.scala index 03cbae0fd..e32535368 100644 --- a/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DomainDecoder.scala +++ b/arch/src/main/scala/framework/memdomain/frontend/cmd/decoder/DomainDecoder.scala @@ -10,21 +10,17 @@ import chisel3.experimental.hierarchy.{instantiable, public} // Detailed decode output for Mem domain class MemDecodeCmd(b: GlobalConfig) extends Bundle { - val is_shared = Bool() // Shared memory access marker derived from bank_id threshold. + // Shared memory access marker: raw BB_BANK0 id >= frontend.sharedBankIdBase. + val is_shared = Bool() val is_load = Bool() val is_store = Bool() val is_config = Bool() val clear = Bool() - // MMIO instruction flags - val is_mmio_set = Bool() // true if funct7 == MMIO_SET_BITPAT - val is_mvin_mmio = Bool() // true if funct7 == MVIN_MMIO_BITPAT + val is_mvin_mmio = Bool() - // Memory address val mem_addr = UInt(b.memDomain.memAddrLen.W) - // Iteration count val iter = UInt(b.frontend.iter_len.W) - // Bank information val bank_id = UInt(log2Up(b.memDomain.bankNum).W) val special = UInt(64.W) } @@ -47,6 +43,26 @@ object MemDefaultConstants { class MemDomainDecoder(val b: GlobalConfig) extends Module { import MemDefaultConstants._ + private val sharedBankIdBase = b.frontend.shared_bank_id_base + private val bankIdxAlign = 1 << log2Ceil(b.memDomain.bankNum) + require(sharedBankIdBase > 0, s"sharedBankIdBase($sharedBankIdBase) must be > 0") + require( + sharedBankIdBase < (1 << b.frontend.bank_id_len), + s"sharedBankIdBase($sharedBankIdBase) must fit in bankIdLen(${b.frontend.bank_id_len})" + ) + require( + sharedBankIdBase > b.frontend.vbank_id_upper_bound, + s"sharedBankIdBase($sharedBankIdBase) must be > vbankIdUpperBound(${b.frontend.vbank_id_upper_bound})" + ) + require( + sharedBankIdBase >= b.memDomain.bankNum, + s"sharedBankIdBase($sharedBankIdBase) must be >= bankNum(${b.memDomain.bankNum})" + ) + require( + sharedBankIdBase % bankIdxAlign == 0, + s"sharedBankIdBase($sharedBankIdBase) must be aligned to bank index width 2^ceil(log2(bankNum))=$bankIdxAlign" + ) + @public val io = IO(new Bundle { val cmd_i = Flipped(Decoupled(new PostGDCmd(b))) @@ -70,60 +86,19 @@ class MemDomainDecoder(val b: GlobalConfig) extends Module { // rs1[63:30] = iter (34-bit) // funct7[6:4] = enable (bank access flags, decoded by GlobalDecoder) // rs2[38:0] = mem_addr (for MVIN/MVOUT, 39-bit) - // rs2[63:0] = special (full 64-bit) // - // MMIO instruction encodings: - // mmio_set: rs1[9:0]=main_bank, rs2[15:0]=mmio_addr, rs2[23:16]=size_rows - // mvin_mmio: rs1[63:30]=row (iter), rs2[38:0]=dram_addr, rs2[55:39]=mmio_addr, rs2[63:56]=col + // mvin_mmio: rs1[63:30]=row (iter), rs2[38:0]=dram_addr, rs2[55:39]=mmio_addr, rs2[63:56]=col import LSDecodeFields._ + // format: off val ls_default_decode = List(N, N, DADDR, DADDR, DSPECIAL, N) - - val ls_decode_list = ListLookup( - func7, - ls_default_decode, - Array( - MSET_BITPAT -> List( - N, - N, - 0.U(memAddrLen.W), // mem_addr: not used for MSET - rs1(bankIdLen - 1, 0), // bank_id from rs1[9:0] - rs2, // special = full rs2 - Y - ), // mset - MVIN_BITPAT -> List( - Y, - N, - rs2(memAddrLen - 1, 0), // mem_addr from rs2[38:0] - rs1(bankIdLen - 1, 0), // bank_id from rs1[9:0] - rs2, // special = full rs2 - Y - ), // mvin - MVOUT_BITPAT -> List( - N, - Y, - rs2(memAddrLen - 1, 0), // mem_addr from rs2[38:0] - rs1(bankIdLen - 1, 0), // bank_id from rs1[9:0] - rs2, // special = full rs2 - Y - ), // mvout - MMIO_SET_BITPAT -> List( - N, // not load - N, // not store - 0.U(memAddrLen.W), // mem_addr unused - rs1(bankIdLen - 1, 0), // bank_id = main_bank from rs1[9:0] - rs2, // special = rs2 (mmio_addr[15:0], size_rows[23:16]) - Y - ), // mmio_set - MVIN_MMIO_BITPAT -> List( - Y, // is_load (uses DMA path) - N, // not store - rs2(memAddrLen - 1, 0), // mem_addr = DRAM addr from rs2[38:0] - 0.U(bankIdLen.W), // bank_id unused - rs2, // special = rs2 (mmio_addr[55:39], col[63:56]) - Y - ) // mvin_mmio + val ls_decode_list = ListLookup(func7, ls_default_decode, Array( + MSET_BITPAT -> List(N, N, 0.U(memAddrLen.W), rs1(bankIdLen - 1, 0), rs2, Y), + MVIN_BITPAT -> List(Y, N, rs2(memAddrLen - 1, 0), rs1(bankIdLen - 1, 0), rs2, Y), + MVOUT_BITPAT -> List(N, Y, rs2(memAddrLen - 1, 0), rs1(bankIdLen - 1, 0), rs2, Y), + MVIN_MMIO_BITPAT -> List(Y, N, rs2(memAddrLen - 1, 0), 0.U(bankIdLen.W), rs2, Y) ) ) + // format: on assert( !(io.cmd_i.fire && !ls_decode_list(LSDecodeFields.VALID.id).asBool), @@ -136,62 +111,27 @@ class MemDomainDecoder(val b: GlobalConfig) extends Module { // ----------------------------------------------------------------------------- io.mem_decode_cmd_o.valid := io.cmd_i.valid && (io.cmd_i.bits.domain_id === DomainId.MEM) - // BB_BANK0 is encoded in rs1[9:0]. Use the full raw field to detect - // shared banks before truncating to the local vbank id width. val raw_bank_id = rs1(9, 0) - io.mem_decode_cmd_o.bits.is_shared := io.mem_decode_cmd_o.valid && (raw_bank_id > 31.U) - io.mem_decode_cmd_o.bits.is_load := Mux( - io.mem_decode_cmd_o.valid, - ls_decode_list(LSDecodeFields.LD_EN.id).asBool, - false.B - ) - io.mem_decode_cmd_o.bits.is_store := Mux( - io.mem_decode_cmd_o.valid, - ls_decode_list(LSDecodeFields.ST_EN.id).asBool, - false.B - ) - // is_config: mset OR mmio_set (both route to MemConfiger) - io.mem_decode_cmd_o.bits.is_config := Mux( - io.mem_decode_cmd_o.valid, - (func7 === MSET_BITPAT) || (func7 === MMIO_SET_BITPAT), - false.B - ) - io.mem_decode_cmd_o.bits.clear := io.mem_decode_cmd_o.valid && - (func7 === MSET_BITPAT) && rs2(11) - io.mem_decode_cmd_o.bits.is_mmio_set := Mux( - io.mem_decode_cmd_o.valid, - func7 === MMIO_SET_BITPAT, - false.B - ) - io.mem_decode_cmd_o.bits.is_mvin_mmio := Mux( - io.mem_decode_cmd_o.valid, - func7 === MVIN_MMIO_BITPAT, - false.B - ) + // format: off + // BB_BANK0 is encoded in rs1[9:0]. Use the full raw field to detect + // shared banks (id >= sharedBankIdBase) before truncating to local vbank width. + io.mem_decode_cmd_o.bits.is_shared := io.mem_decode_cmd_o.valid && (raw_bank_id >= sharedBankIdBase.U) + io.mem_decode_cmd_o.bits.is_load := Mux(io.mem_decode_cmd_o.valid, ls_decode_list(LSDecodeFields.LD_EN.id).asBool, false.B) + io.mem_decode_cmd_o.bits.is_store := Mux(io.mem_decode_cmd_o.valid, ls_decode_list(LSDecodeFields.ST_EN.id).asBool, false.B) + io.mem_decode_cmd_o.bits.is_config := Mux(io.mem_decode_cmd_o.valid, func7 === MSET_BITPAT, false.B) + io.mem_decode_cmd_o.bits.clear := io.mem_decode_cmd_o.valid && (func7 === MSET_BITPAT) && rs2(11) + io.mem_decode_cmd_o.bits.is_mvin_mmio := Mux(io.mem_decode_cmd_o.valid, func7 === MVIN_MMIO_BITPAT, false.B) + io.mem_decode_cmd_o.bits.mem_addr := Mux(io.mem_decode_cmd_o.valid, ls_decode_list(LSDecodeFields.MEMADDR.id).asUInt, 0.U(b.memDomain.memAddrLen.W)) + io.mem_decode_cmd_o.bits.iter := Mux(io.mem_decode_cmd_o.valid, rs1(63, 30), 0.U(iterLen.W)) when(io.cmd_i.fire && func7 === MSET_BITPAT) { assert(rs1(63, 10) === 0.U, "MSET reserves rs1[63:10]") assert(rs2(63, 12) === 0.U, "MSET reserves rs2[63:12]") assert(!(rs2(11) && !rs2(10)), "MSET clear requires alloc=1") } - io.mem_decode_cmd_o.bits.mem_addr := Mux( - io.mem_decode_cmd_o.valid, - ls_decode_list(LSDecodeFields.MEMADDR.id).asUInt, - 0.U(b.memDomain.memAddrLen.W) - ) - // iter is always from rs1[63:30] - io.mem_decode_cmd_o.bits.iter := Mux( - io.mem_decode_cmd_o.valid, - rs1(63, 30), - 0.U(iterLen.W) - ) - // Address parsing val ls_bank_id = ls_decode_list(LSDecodeFields.BANK_ID.id).asUInt io.mem_decode_cmd_o.bits.bank_id := Mux(io.mem_decode_cmd_o.valid, ls_bank_id, 0.U(log2Up(b.memDomain.bankNum).W)) - io.mem_decode_cmd_o.bits.special := Mux( - io.mem_decode_cmd_o.valid, - ls_decode_list(LSDecodeFields.SPECIAL.id).asUInt, - 0.U(64.W) - ) + io.mem_decode_cmd_o.bits.special := Mux(io.mem_decode_cmd_o.valid, ls_decode_list(LSDecodeFields.SPECIAL.id).asUInt, 0.U(64.W)) + // format: on } diff --git a/arch/src/main/scala/framework/memdomain/frontend/mem/MemConfiger.scala b/arch/src/main/scala/framework/memdomain/frontend/mem/MemConfiger.scala index 22d07f008..cf8c19bda 100644 --- a/arch/src/main/scala/framework/memdomain/frontend/mem/MemConfiger.scala +++ b/arch/src/main/scala/framework/memdomain/frontend/mem/MemConfiger.scala @@ -5,7 +5,6 @@ import chisel3.util._ import framework.top.GlobalConfig import framework.balldomain.blink.BankWrite import framework.memdomain.frontend.cmd.rs.{MemRsComplete, MemRsIssue} -import framework.memdomain.backend.mmio.MmioAllocReq import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} class MemConfigerIO(val b: GlobalConfig) extends Bundle { @@ -19,34 +18,30 @@ class MemConfigerIO(val b: GlobalConfig) extends Bundle { @instantiable class MemConfiger(val b: GlobalConfig) extends Module { - val rob_id_width = log2Up(b.frontend.rob_entries) @public val io = IO(new Bundle { - val cmdReq = Flipped(Decoupled(new MemRsIssue(b))) - val cmdResp = Decoupled(new MemRsComplete(b)) - + val cmdReq = Flipped(Decoupled(new MemRsIssue(b))) + val cmdResp = Decoupled(new MemRsComplete(b)) val config = Decoupled(new MemConfigerIO(b)) val hartid = Input(UInt(b.core.xLen.W)) val bankWrite = Flipped(new BankWrite(b)) - - // MMIO alloc/dealloc port - val mmioAlloc = Valid(new MmioAllocReq(b)) }) val idle :: config :: zeroReq :: zeroRun :: zeroWait :: resp :: Nil = Enum(6) - val state = RegInit(idle) - val alloc_reg = RegInit(false.B) - val is_shared_reg = RegInit(false.B) - val col_reg = RegInit(0.U(log2Up(b.memDomain.bankNum + 1).W)) - val clear_reg = RegInit(false.B) - val vbank_id_reg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val rob_id_reg = RegInit(0.U(rob_id_width.W)) - val is_sub_reg = RegInit(false.B) - val sub_rob_id_reg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) - val counter = RegInit(0.U(log2Up(b.memDomain.bankNum + 1).W)) - val zeroLastRow = RegInit(false.B) + + val state = RegInit(idle) + val alloc_reg = RegInit(false.B) + val is_shared_reg = RegInit(false.B) + val col_reg = RegInit(0.U(log2Up(b.memDomain.bankNum + 1).W)) + val clear_reg = RegInit(false.B) + val vbank_id_reg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + val rob_id_reg = RegInit(0.U(rob_id_width.W)) + val is_sub_reg = RegInit(false.B) + val sub_rob_id_reg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) + val counter = RegInit(0.U(log2Up(b.memDomain.bankNum + 1).W)) + val zeroLastRow = RegInit(false.B) val zeroLines: Instance[ZeroLineGenerator] = Instantiate(new ZeroLineGenerator(b.memDomain.bankWidth)) @@ -62,12 +57,6 @@ class MemConfiger(val b: GlobalConfig) extends Module { io.cmdResp.bits.is_sub := false.B io.cmdResp.bits.sub_rob_id := 0.U - // MMIO alloc defaults - io.mmioAlloc.valid := false.B - io.mmioAlloc.bits.main_bank := 0.U - io.mmioAlloc.bits.mmio_addr := 0.U - io.mmioAlloc.bits.size_rows := 0.U - io.bankWrite.io.req.valid := false.B io.bankWrite.io.req.bits.addr := zeroLines.io.resp.bits.row(log2Ceil(b.memDomain.bankEntries) - 1, 0) io.bankWrite.io.req.bits.data := zeroLines.io.resp.bits.data @@ -82,49 +71,28 @@ class MemConfiger(val b: GlobalConfig) extends Module { zeroLines.io.req.bits.rows := b.memDomain.bankEntries.U zeroLines.io.resp.ready := state === zeroRun && io.bankWrite.io.req.ready - val isMmioSet = io.cmdReq.valid && io.cmdReq.bits.cmd.is_mmio_set - io.cmdReq.ready := state === idle && (!isMmioSet || io.cmdResp.ready) + io.cmdReq.ready := state === idle when(state === idle) { when(io.cmdReq.valid) { - // Check if this is mmio_set instruction - when(io.cmdReq.bits.cmd.is_mmio_set) { - // mmio_set: rs1[9:0]=main_bank, rs2[15:0]=mmio_addr, rs2[23:16]=size_rows - val main_bank = io.cmdReq.bits.cmd.bank_id - val mmio_addr = io.cmdReq.bits.cmd.special(15, 0) - val size_rows = io.cmdReq.bits.cmd.special(23, 16) - - io.mmioAlloc.valid := io.cmdReq.fire - io.mmioAlloc.bits.main_bank := main_bank - io.mmioAlloc.bits.mmio_addr := mmio_addr - io.mmioAlloc.bits.size_rows := size_rows - - // Immediate response (no state transition) - io.cmdResp.valid := true.B - io.cmdResp.bits.rob_id := io.cmdReq.bits.rob_id - io.cmdResp.bits.is_sub := io.cmdReq.bits.is_sub - io.cmdResp.bits.sub_rob_id := io.cmdReq.bits.sub_rob_id - - }.otherwise { - when(io.cmdReq.fire) { - val rawCol = io.cmdReq.bits.cmd.special(9, 5) - val alloc = io.cmdReq.bits.cmd.special(10) - val fullCol = b.memDomain.bankNum.U(col_reg.getWidth.W) - - state := config - col_reg := Mux(alloc && rawCol === 0.U, fullCol, Mux(rawCol > 1.U, rawCol, 1.U)) - alloc_reg := alloc - clear_reg := io.cmdReq.bits.cmd.clear - is_shared_reg := io.cmdReq.bits.cmd.is_shared - vbank_id_reg := io.cmdReq.bits.cmd.bank_id - rob_id_reg := io.cmdReq.bits.rob_id - is_sub_reg := io.cmdReq.bits.is_sub - sub_rob_id_reg := io.cmdReq.bits.sub_rob_id - assert( - !(io.cmdReq.bits.cmd.clear && io.cmdReq.bits.cmd.is_shared), - "MSET clear is currently supported for private banks only" - ) - } + when(io.cmdReq.fire) { + val rawCol = io.cmdReq.bits.cmd.special(9, 5) + val alloc = io.cmdReq.bits.cmd.special(10) + val fullCol = b.memDomain.bankNum.U(col_reg.getWidth.W) + + state := config + col_reg := Mux(alloc && rawCol === 0.U, fullCol, Mux(rawCol > 1.U, rawCol, 1.U)) + alloc_reg := alloc + clear_reg := io.cmdReq.bits.cmd.clear + is_shared_reg := io.cmdReq.bits.cmd.is_shared + vbank_id_reg := io.cmdReq.bits.cmd.bank_id + rob_id_reg := io.cmdReq.bits.rob_id + is_sub_reg := io.cmdReq.bits.is_sub + sub_rob_id_reg := io.cmdReq.bits.sub_rob_id + assert( + !(io.cmdReq.bits.cmd.clear && io.cmdReq.bits.cmd.is_shared), + "MSET clear is currently supported for private banks only" + ) } } diff --git a/arch/src/main/scala/framework/memdomain/midend/MemMidend.scala b/arch/src/main/scala/framework/memdomain/midend/MemMidend.scala index efea33663..acb06fb38 100644 --- a/arch/src/main/scala/framework/memdomain/midend/MemMidend.scala +++ b/arch/src/main/scala/framework/memdomain/midend/MemMidend.scala @@ -39,8 +39,10 @@ class MemMidend(val b: GlobalConfig) extends Module { val io = IO(new Bundle { // Unified read/write interfaces: indices [0, totalBallRead) are balldomain, // index totalBallRead is frontend (DMA). Same for write. - val bankRead = Vec(totalRead, new BankReadWithShared(b)) - val bankWrite = Vec(totalWrite, new BankWriteWithShared(b)) + val bankRead = Vec(totalRead, new BankReadWithShared(b)) + val bankWrite = Vec(totalWrite, new BankWriteWithShared(b)) + val ballChannelActive = Input(Vec(b.ballDomain.ballNum, Bool())) + val ballChannelReady = Output(Vec(b.ballDomain.ballNum, Bool())) val hartid = Input(UInt(b.core.xLen.W)) @@ -55,57 +57,89 @@ class MemMidend(val b: GlobalConfig) extends Module { val valid = Bool() val isRead = Bool() val id = UInt(log2Ceil(math.max(totalRead, totalWrite)).W) + val isBall = Bool() + val ballId = UInt(log2Up(b.ballDomain.ballNum).W) } val mappingTable = RegInit(VecInit(Seq.fill(b.memDomain.bankChannel)(0.U.asTypeOf(new MappingTableEntry)))) - def addEntry(idx: UInt, isRead: Bool, id: UInt): Unit = { - mappingTable(idx).valid := true.B - mappingTable(idx).isRead := isRead - mappingTable(idx).id := id + def isAllocated(isRead: Bool, id: UInt): Bool = + mappingTable.map(entry => entry.valid && entry.isRead === isRead && entry.id === id).reduce(_ || _) + + val readPortBall = b.ballDomain.ballIdMappings.zipWithIndex.flatMap { case (mapping, ball) => + Seq.fill(mapping.inBW)(ball) } - def allocateChannel(): (Bool, UInt) = { - val freeChannels = mappingTable.map(entry => !entry.valid) - val hasFreeChan = freeChannels.reduce(_ || _) - val chanId = PriorityEncoder(freeChannels) - (hasFreeChan, chanId) + val writePortBall = b.ballDomain.ballIdMappings.zipWithIndex.flatMap { case (mapping, ball) => + Seq.fill(mapping.outBW)(ball) } - def isAllocated(isRead: Bool, id: UInt): Bool = - mappingTable.map(entry => entry.valid && entry.isRead === isRead && entry.id === id).reduce(_ || _) + for (ball <- 0 until b.ballDomain.ballNum) { + val readReady = readPortBall.zipWithIndex + .filter(_._1 == ball) + .map { case (_, port) => isAllocated(true.B, port.U) } + .foldLeft(true.B)(_ && _) + val writeReady = writePortBall.zipWithIndex + .filter(_._1 == ball) + .map { case (_, port) => isAllocated(false.B, port.U) } + .foldLeft(true.B)(_ && _) + io.ballChannelReady(ball) := io.ballChannelActive(ball) && readReady && writeReady + } - // Allocate channels for reads (all entries including frontend) + // Allocate exactly one channel per cycle. Ball channels are allocated before + // their data traffic can start; frontend DMA channels are demand-allocated. for (i <- 0 until totalRead) { io.bankRead(i).bankRead.io.req.ready := false.B io.bankRead(i).bankRead.io.resp.valid := false.B io.bankRead(i).bankRead.io.resp.bits := DontCare - when(io.bankRead(i).bankRead.io.req.valid && !isAllocated(true.B, i.U)) { - val (hasFree, chanId) = allocateChannel() - when(hasFree) { - addEntry(chanId, true.B, i.U) - } - } } - // Allocate channels for writes: one per cycle to avoid conflicts - val pendingWrites = - VecInit((0 until totalWrite).map(i => io.bankWrite(i).bankWrite.io.req.valid && !isAllocated(false.B, i.U))) - val hasPendingWrite = pendingWrites.asUInt.orR - val nextWriteToAllocate = PriorityEncoder(pendingWrites) - for (i <- 0 until totalWrite) { io.bankWrite(i).bankWrite.io.req.ready := false.B io.bankWrite(i).bankWrite.io.resp.valid := false.B io.bankWrite(i).bankWrite.io.resp.bits := DontCare + } - when(hasPendingWrite && nextWriteToAllocate === i.U) { - val (hasFree, chanId) = allocateChannel() - when(hasFree) { - addEntry(chanId, false.B, i.U) + val pendingReads = VecInit((0 until totalRead).map { i => + val active = + if (i < totalBallRead) { + io.ballChannelActive(readPortBall(i)) + } else { + io.bankRead(i).bankRead.io.req.valid } - } + active && !isAllocated(true.B, i.U) + }) + + val pendingWrites = VecInit((0 until totalWrite).map { i => + val active = + if (i < totalBallWrite) { + io.ballChannelActive(writePortBall(i)) + } else { + io.bankWrite(i).bankWrite.io.req.valid + } + active && !isAllocated(false.B, i.U) + }) + + val pending = VecInit(pendingReads ++ pendingWrites) + val freeChannels = mappingTable.map(entry => !entry.valid) + + when(pending.asUInt.orR && freeChannels.reduce(_ || _)) { + val channel = PriorityEncoder(freeChannels) + val port = PriorityEncoder(pending) + mappingTable(channel).valid := true.B + mappingTable(channel).isRead := port < totalRead.U + mappingTable(channel).id := Mux(port < totalRead.U, port, port - totalRead.U) + mappingTable(channel).isBall := Mux( + port < totalRead.U, + MuxLookup(port, false.B)((0 until totalBallRead).map(i => i.U -> true.B).toSeq), + MuxLookup(port - totalRead.U, false.B)((0 until totalBallWrite).map(i => i.U -> true.B).toSeq) + ) + mappingTable(channel).ballId := Mux( + port < totalRead.U, + MuxLookup(port, 0.U)(readPortBall.zipWithIndex.map { case (ball, i) => i.U -> ball.U }.toSeq), + MuxLookup(port - totalRead.U, 0.U)(writePortBall.zipWithIndex.map { case (ball, i) => i.U -> ball.U }.toSeq) + ) } // Connect mapped entries to backend channels @@ -122,33 +156,42 @@ class MemMidend(val b: GlobalConfig) extends Module { io.mem_req(i).hart_id := io.hartid io.mem_req(i).rob_id := 0.U - val isRead = mappingTable(i).isRead - val rid = mappingTable(i).id - val wid = mappingTable(i).id - val ballRead = io.bankRead(rid).bankRead.io - val ballWrite = io.bankWrite(wid).bankWrite.io - val rbank_id = io.bankRead(rid).bankRead.bank_id - val wbank_id = io.bankWrite(wid).bankWrite.bank_id - val rgroup_id = io.bankRead(rid).bankRead.group_id - val wgroup_id = io.bankWrite(wid).bankWrite.group_id - val r_shared = io.bankRead(rid).is_shared - val w_shared = io.bankWrite(wid).is_shared - val rrob_id = io.bankRead(rid).bankRead.rob_id - val wrob_id = io.bankWrite(wid).bankWrite.rob_id + val isRead = mappingTable(i).isRead + val rid = mappingTable(i).id + val wid = mappingTable(i).id + val ballRead = io.bankRead(rid).bankRead.io + val ballWrite = io.bankWrite(wid).bankWrite.io + val rbank_id = io.bankRead(rid).bankRead.bank_id + val wbank_id = io.bankWrite(wid).bankWrite.bank_id + val rgroup_id = io.bankRead(rid).bankRead.group_id + val wgroup_id = io.bankWrite(wid).bankWrite.group_id + val r_shared = io.bankRead(rid).is_shared + val w_shared = io.bankWrite(wid).is_shared + val rrob_id = io.bankRead(rid).bankRead.rob_id + val wrob_id = io.bankWrite(wid).bankWrite.rob_id + // A released Ball cannot issue new traffic, but its final bank response + // still needs the existing route until the physical channel drains. + val ballRouteOpen = !mappingTable(i).isBall || + io.ballChannelReady(mappingTable(i).ballId) || + !io.ballChannelActive(mappingTable(i).ballId) when(mappingTable(i).valid) { when(isRead) { - io.mem_req(i).read <> ballRead - io.mem_req(i).bank_id := rbank_id - io.mem_req(i).group_id := rgroup_id - io.mem_req(i).is_shared := r_shared - io.mem_req(i).rob_id := rrob_id + when(ballRouteOpen) { + io.mem_req(i).read <> ballRead + io.mem_req(i).bank_id := rbank_id + io.mem_req(i).group_id := rgroup_id + io.mem_req(i).is_shared := r_shared + io.mem_req(i).rob_id := rrob_id + } }.otherwise { - io.mem_req(i).write <> ballWrite - io.mem_req(i).bank_id := wbank_id - io.mem_req(i).group_id := wgroup_id - io.mem_req(i).is_shared := w_shared - io.mem_req(i).rob_id := wrob_id + when(ballRouteOpen) { + io.mem_req(i).write <> ballWrite + io.mem_req(i).bank_id := wbank_id + io.mem_req(i).group_id := wgroup_id + io.mem_req(i).is_shared := w_shared + io.mem_req(i).rob_id := wrob_id + } } } } @@ -173,7 +216,23 @@ class MemMidend(val b: GlobalConfig) extends Module { for (i <- 0 until b.memDomain.bankChannel) { val releaseCounter = RegInit(0.U(5.W)) - when(mappingTable(i).valid && !(io.mem_req(i).read.resp.valid || + // Releasing a Ball's logical channel can precede the final bank response. + // Keep its physical route until that response has been consumed; otherwise + // the response loses its consumer and leaves the AccPipe busy. + val ballReleased = mappingTable(i).isBall && + !io.ballChannelActive(mappingTable(i).ballId) + val channelDrained = !(io.mem_req(i).read.resp.valid || + io.mem_req(i).write.resp.valid || io.mem_req(i).read.req.valid || + io.mem_req(i).write.req.valid) + + when(mappingTable(i).valid && ballReleased && channelDrained) { + mappingTable(i).valid := false.B + mappingTable(i).isRead := false.B + mappingTable(i).id := 0.U + mappingTable(i).isBall := false.B + mappingTable(i).ballId := 0.U + releaseCounter := 0.U + }.elsewhen(mappingTable(i).valid && !mappingTable(i).isBall && !(io.mem_req(i).read.resp.valid || io.mem_req(i).write.resp.valid || io.mem_req(i).read.req.valid || io.mem_req(i).write.req.valid)) { releaseCounter := releaseCounter + 1.U @@ -183,6 +242,8 @@ class MemMidend(val b: GlobalConfig) extends Module { mappingTable(i).valid := false.B mappingTable(i).isRead := false.B mappingTable(i).id := 0.U + mappingTable(i).isBall := false.B + mappingTable(i).ballId := 0.U } }.otherwise { releaseCounter := 0.U diff --git a/arch/src/main/scala/framework/system/configloader/ChipLoader.scala b/arch/src/main/scala/framework/system/configloader/ChipLoader.scala new file mode 100644 index 000000000..5b6e0b8b2 --- /dev/null +++ b/arch/src/main/scala/framework/system/configloader/ChipLoader.scala @@ -0,0 +1,263 @@ +package framework.system.configloader + +import buckyball.config.{ + Chip, + CoreInstance, + CoreParamConfig, + FrontendConfig, + GpDomainConfig, + MemDomainConfig, + RocketCoreConfig, + SharedMemConfig, + TilePlacement +} +import java.nio.file.{Files, Path, Paths} +import framework.balldomain.configs.{BallDomainParam, BallISAEntry, BallIdMapping} +import framework.frontend.configs.FrontendParam +import framework.gpdomain.configs.GpDomainParam +import framework.memdomain.configs.MemDomainParam +import framework.system.core.configs._ +import framework.system.tile.PrivateDCacheParams +import framework.top.GlobalConfig +import framework.top.configs.TopConfig +import scala.jdk.CollectionConverters._ + +/** Load ExampleTopology from chip.pb. */ +object ChipLoader { + + def load(pbPath: String): ExampleTopology = { + val path = Paths.get(pbPath) + if (!Files.isRegularFile(path)) { + throw new RuntimeException(s"chip.pb does not exist: $pbPath") + } + val repo = repoRoot(path) + val chip = Chip.parseFrom(Files.readAllBytes(path)) + val cores = chip.getCoresList.asScala.toSeq + if (cores.isEmpty) { + throw new RuntimeException(s"chip.pb has no cores: $pbPath") + } + val tiles = chip.getTilesList.asScala.map(parseTile(_, cores, repo)).toSeq + require( + tiles.size == chip.getNTiles, + s"chip.pb declares top.nTiles=${chip.getNTiles} but defines ${tiles.size} tile(s) in $pbPath" + ) + ExampleTopology(tiles) + } + + private def repoRoot(pb: Path): Path = { + val abs = pb.toAbsolutePath.normalize + val s = abs.toString + val marker = "/examples/chips/" + val i = s.lastIndexOf(marker) + if (i < 0) { + throw new RuntimeException(s"chip.pb is not under examples/chips: $pb") + } + Paths.get(s.substring(0, i)) + } + + private def repoFile(repo: Path, rel: String, what: String): String = { + if (rel.isEmpty) { + throw new RuntimeException(s"$what is empty") + } + val p = Paths.get(rel) + if (p.isAbsolute) { + throw new RuntimeException(s"$what must be repo-relative: $rel") + } + val resolved = repo.resolve(p).normalize() + if (!Files.isRegularFile(resolved)) { + throw new RuntimeException(s"$what missing: $resolved") + } + resolved.toString + } + + private def parseTile(tile: TilePlacement, cores: Seq[CoreInstance], repo: Path): TileTopology = { + val indices = tile.getCoreIndicesList.asScala.map(_.toInt).toSeq + val nCores = indices.size + val hasBuckyball = indices.exists(i => cores(i).getBalldomain.getBallNum > 0) + val memBallChannelNum = + if (hasBuckyball) tile.getMemBallChannelNum + else 0 + + val shared = tile.getSharedMem + val privateDCache = + if (!tile.hasPrivateDcache || !tile.getPrivateDcache.getEnable) None + else { + val dcache = tile.getPrivateDcache + val ways = dcache.getWays + val sets = (dcache.getCapacityKb * 1024) / (64 * ways) + Some(PrivateDCacheParams( + ways = ways, + sets = sets, + writeBytes = dcache.getWriteBytes, + portFactor = dcache.getPortFactor, + memCycles = dcache.getMemCycles + )) + } + + val coreEntries = indices.map { idx => + parseCore(cores(idx), shared, memBallChannelNum, nCores, repo) + } + val buckyballCores = coreEntries.map(_._1).map(_.map { cfg => + cfg.copy(top = TopConfig(memBallChannelNum = memBallChannelNum, nCores = nCores)) + }) + val rocketCores = coreEntries.map { + case (_, rocket) => rocket + } + TileTopology(buckyballCores, privateDCache, rocketCores) + } + + private def parseCore( + core: CoreInstance, + shared: SharedMemConfig, + memBallChannelNum: Int, + nCores: Int, + repo: Path + ): (Option[GlobalConfig], RocketCoreParam) = { + val rocket = parseRocketCore(core.getRocketCore) + val domain = core.getBalldomain + if (domain.getBallNum == 0) { + return (None, rocket) + } + require(core.hasFrontend, s"core ${core.getPkg} missing frontend config") + require(core.hasGpDomain, s"core ${core.getPkg} missing gpdomain config") + require(core.hasCore, s"core ${core.getPkg} missing core config") + + val buckyball = GlobalConfig().copy( + ballDomain = parseBallDomain(core, repo), + frontend = parseFrontend(core.getFrontend), + gpDomain = parseGpDomain(core.getGpDomain), + core = parseCoreParam(core.getCore), + memDomain = parseMemDomain(core.getMem, shared), + rocketCore = rocket, + top = TopConfig(memBallChannelNum = memBallChannelNum, nCores = nCores) + ) + (Some(buckyball), rocket) + } + + private def parseBallDomain(core: CoreInstance, repo: Path): BallDomainParam = { + val domain = core.getBalldomain + val mappings = domain.getMappingsList.asScala.map { m => + BallIdMapping( + ballId = m.getBallId, + ballName = m.getBallName, + ballClass = m.getBallClass, + config = Some(repoFile(repo, m.getConfigPath, s"Ball ${m.getBallName} config")), + inBW = m.getInBw, + outBW = m.getOutBw, + mmioReadBW = m.getMmioReadBw, + mmioWriteBW = m.getMmioWriteBw + ) + }.toSeq + val isa = domain.getIsaList.asScala.map { e => + BallISAEntry(mnemonic = e.getMnemonic, funct7 = e.getFunct7, bid = e.getBid) + }.toSeq + BallDomainParam(ballNum = domain.getBallNum, ballIdMappings = mappings, ballISA = isa) + } + + private def parseMemDomain(mem: MemDomainConfig, shared: SharedMemConfig): MemDomainParam = { + val bank = mem.getBank + val dma = mem.getDma + val tlb = mem.getTlb + val tma = mem.getTma + val mmio = mem.getMmio + MemDomainParam( + bankNum = bank.getNum, + bankWidth = bank.getWidth, + bankEntries = bank.getEntries, + bankMaskLen = bank.getMaskLen, + sharedEnable = shared.getEnable, + sharedEntries = shared.getEntries, + sharedInputChannels = shared.getInputChannels, + sharedDefaultGroupCount = shared.getDefaultGroupCount, + tlb_size = tlb.getSize, + dma_n_xacts = dma.getNXacts, + dma_burst_maxbytes = dma.getBurstMaxBytes, + bankChannel = bank.getChannel, + max_in_flight_mem_reqs = dma.getMaxInFlightMemReqs, + dma_buswidth = dma.getBusWidth, + memAddrLen = mem.getMem.getAddrLen, + tmaReadChannel = tma.getReadChannel, + tmaWriteChannel = tma.getWriteChannel, + mmioEnable = mmio.getEnable, + mmioBankNum = mmio.getBankNum, + mmioBankEntries = mmio.getBankEntries, + mmioBankWidth = mmio.getBankWidth, + mmioReadWidth = mmio.getReadWidth + ) + } + + private def parseFrontend(frontend: FrontendConfig): FrontendParam = + FrontendParam( + rob_entries = frontend.getRobEntries, + rs_out_of_order_response = frontend.getRsOutOfOrderResponse, + bank_id_len = frontend.getBankIdLen, + vbank_id_upper_bound = frontend.getVbankIdUpperBound, + shared_bank_id_base = frontend.getSharedBankIdBase, + iter_len = frontend.getIterLen, + sub_rob_enable = frontend.getSubRobEnable, + sub_rob_depth = frontend.getSubRobDepth + ) + + private def parseGpDomain(gp: GpDomainConfig): GpDomainParam = + GpDomainParam( + laneNumber = gp.getLaneNumber, + chainingSize = gp.getChainingSize, + vLen = gp.getVLen, + dLen = gp.getDLen, + eLen = gp.getELen, + laneScale = gp.getLaneScale + ) + + private def parseCoreParam(core: CoreParamConfig): CoreParam = + CoreParam( + coreDataBytes = core.getCoreDataBytes, + xLen = core.getXLen, + vaddrBits = core.getVaddrBits, + paddrBits = core.getPaddrBits, + pgIdxBits = core.getPgIdxBits, + nPMPs = core.getNPmps + ) + + private def parseRocketCore(rocket: RocketCoreConfig): RocketCoreParam = { + val mulDiv = rocket.getMulDiv + val fpu = rocket.getFpu + val dcache = rocket.getDcache + val icache = rocket.getIcache + val btb = rocket.getBtb + RocketCoreParam( + xLen = rocket.getXLen, + pgLevels = rocket.getPgLevels, + useVM = rocket.getUseVm, + useZba = rocket.getUseZba, + useZbb = rocket.getUseZbb, + useZbs = rocket.getUseZbs, + haveCFlush = rocket.getHaveCFlush, + mulDiv = MulDivParam( + enable = mulDiv.getEnable, + mulUnroll = mulDiv.getMulUnroll, + mulEarlyOut = mulDiv.getMulEarlyOut, + divEarlyOut = mulDiv.getDivEarlyOut + ), + fpu = FPUParam( + enable = fpu.getEnable, + minFLen = fpu.getMinFLen, + fLen = fpu.getFLen + ), + dcache = DCacheParam( + nSets = dcache.getNSets, + nWays = dcache.getNWays, + nMSHRs = dcache.getNMshrs + ), + icache = ICacheParam( + nSets = icache.getNSets, + nWays = icache.getNWays + ), + btb = BTBParam( + enable = btb.getEnable, + nEntries = btb.getNEntries, + nRAS = btb.getNRas + ) + ) + } + +} diff --git a/arch/src/main/scala/framework/system/configloader/TomlConfigLoader.scala b/arch/src/main/scala/framework/system/configloader/TomlConfigLoader.scala deleted file mode 100644 index b5ca212c1..000000000 --- a/arch/src/main/scala/framework/system/configloader/TomlConfigLoader.scala +++ /dev/null @@ -1,542 +0,0 @@ -package framework.system.configloader - -import toml.{Toml, Value} -import framework.system.tile.PrivateDCacheParams -import framework.system.core.configs.{CoreParam, RocketCoreParam} -import framework.balldomain.configs.{BallDomainParam, BallISAEntry, BallIdMapping} -import framework.frontend.configs.FrontendParam -import framework.gpdomain.configs.GpDomainParam -import framework.memdomain.configs.MemDomainParam -import framework.top.GlobalConfig -import framework.top.configs.TopConfig -import java.nio.file.{Path, Paths} - -/** - * Unified TOML-based configuration loader for all examples. - * - * Supports hierarchical TOML files via `include` directive — sub-tables can - * be replaced with `include = "relative/path.toml"` to load that file's root - * table in place. Paths are resolved relative to the directory of the file - * that contains the include, so file moves stay local. - * - * Two convenience shortcuts cut down repetition for homogeneous topologies: - * * `[tileTemplate] count = N` — repeat one tile N times - * * `[coreTemplate] count = N` — repeat one core N times within a tile - * - * Example directory layout: - * ``` - * examples/goban/ - * goban_24x16.toml # [top]+[tileTemplate count=24 include=...] - * tiles/tile_16core.toml # [privateDCache]+[coreTemplate count=16 include=...] - * cores/core_default.toml # [balldomain include=...]+[rocketCore] - * balldomains/goban.toml # ballNum, ballIdMappings, ballISA - * ``` - */ -object TomlConfigLoader { - - /** - * Load complete example topology from a TOML file (with possible includes). - */ - def load(tomlPath: String): ExampleTopology = { - val root = parseFile(tomlPath) - val baseDir = Paths.get(tomlPath).toAbsolutePath.getParent - - // Parse top-level - val nTiles = getInt(root, "top.nTiles") - - // Parse tiles: either [[tiles]] (explicit) or [tileTemplate] count=N (template+repeat) - val tiles = parseTiles(root, baseDir) - - require( - tiles.size == nTiles, - s"TOML declares top.nTiles=$nTiles but defines ${tiles.size} tile(s) in $tomlPath" - ) - - ExampleTopology(tiles) - } - - // --------------------------------------------------------------------------- - // Hierarchical parsing - // --------------------------------------------------------------------------- - - /** - * Parse a TOML file and return its root table. - */ - private def parseFile(path: String): Value = { - val content = scala.io.Source.fromFile(path, "UTF-8").mkString - Toml.parse(content) match { - case Right(r) => r - case Left((addr, msg)) => - throw new RuntimeException(s"TOML parse error at $addr: $msg in $path") - } - } - - /** - * If `table` has an `include` key, load that file (relative to baseDir) and - * return its root table merged with the rest of `table` (rest wins). Otherwise - * return `table` unchanged. - * - * Returns (resolvedTable, newBaseDir) where newBaseDir is the directory of - * the included file (so nested includes resolve correctly). - */ - private def resolveInclude(table: Map[String, Value], baseDir: Path): (Map[String, Value], Path) = { - table.get("include") match { - case Some(Value.Str(relPath)) => - val includedPath = baseDir.resolve(relPath).normalize() - val included = parseFile(includedPath.toString) match { - case Value.Tbl(t) => t - case _ => throw new RuntimeException(s"Included file $includedPath must contain a top-level table") - } - // Merge: included is the base, local overrides win (except `include` itself) - val merged = included ++ (table - "include") - (merged, includedPath.getParent) - case None => (table, baseDir) - case _ => throw new RuntimeException("'include' must be a string") - } - } - - private def resolvePathOrTable( - value: Value, - baseDir: Path, - what: String - ): (Map[String, Value], Path) = value match { - case Value.Str(s) if s.nonEmpty => - val path = baseDir.resolve(s).normalize() - (asTable(parseFile(path.toString), s"$what root"), path.getParent) - case Value.Tbl(t) => - resolveInclude(t, baseDir) - case Value.Str(_) => - throw new RuntimeException(s"'$what' path must be non-empty") - case _ => - throw new RuntimeException(s"'$what' must be a string path or table") - } - - private def parseTiles(root: Value, baseDir: Path): Seq[TileTopology] = { - val rootTable = asTable(root, "root") - - rootTable.get("tileTemplate") match { - case Some(templateValue) => - // tileTemplate + count: repeat one tile N times - val (templateTable, newBase) = resolveInclude(asTable(templateValue, "tileTemplate"), baseDir) - val count = getInt(templateTable, "count") - val tile = parseTile(templateTable - "count", newBase) - Seq.fill(count)(tile) - - case None => - // Explicit [[tiles]] array - rootTable.get("tiles") match { - case Some(Value.Arr(tilesArray)) => - tilesArray.map { tileValue => - val tileTable = asTable(tileValue, "tile entry") - val (resolved, newBase) = resolveInclude(tileTable, baseDir) - parseTile(resolved, newBase) - } - case None => - throw new RuntimeException("Top-level must define either [[tiles]] array or [tileTemplate]") - case _ => - throw new RuntimeException("'tiles' must be an array") - } - } - } - - private def parseTile(tileTable: Map[String, Value], baseDir: Path): TileTopology = { - val privateDCache = getTable(tileTable, "privateDCache").flatMap { dcacheTable => - val enable = getBool(dcacheTable, "enable") - if (!enable) None - else { - val ways = getInt(dcacheTable, "ways") - val capacityKB = getInt(dcacheTable, "capacityKB") - val writeBytes = getInt(dcacheTable, "writeBytes") - val portFactor = getInt(dcacheTable, "portFactor") - val memCycles = getInt(dcacheTable, "memCycles") - val cacheBlockBytes = 64 - val sets = (capacityKB * 1024) / (cacheBlockBytes * ways) - Some(PrivateDCacheParams( - ways = ways, - sets = sets, - writeBytes = writeBytes, - portFactor = portFactor, - memCycles = memCycles - )) - } - } - - val sharedMem = parseSharedMem(tileTable) - - val coreEntries: Seq[(Option[GlobalConfig], Option[RocketCoreParam])] = tileTable.get("coreTemplate") match { - case Some(templateValue) => - val (templateTable, newBase) = resolveInclude(asTable(templateValue, "coreTemplate"), baseDir) - val count = getInt(templateTable, "count") - val coreConfig = parseCoreEntry(templateTable - "count", newBase, sharedMem) - Seq.fill(count)(coreConfig) - - case None => - tileTable.get("cores") match { - case Some(Value.Arr(coresArray)) => - coresArray.map { coreValue => - val coreTable = asTable(coreValue, "core entry") - val (resolved, newBase) = resolveInclude(coreTable, baseDir) - parseCoreEntry(resolved, newBase, sharedMem) - } - case None => - throw new RuntimeException("Tile must define either [[cores]] array or [coreTemplate]") - case _ => - throw new RuntimeException("'cores' must be an array") - } - } - - val cores = coreEntries.map(_._1) - val nCores = cores.size - - val hasBuckyball = cores.exists(_.isDefined) - val memBallChannelNum = - if (hasBuckyball) getInt(tileTable, "memBallChannelNum") - else 0 - - val coresWithTop = cores.map(_.map { cfg => - cfg.copy(top = TopConfig(memBallChannelNum = memBallChannelNum, nCores = nCores)) - }) - - val rocketCoreOpts = coreEntries.map { - case (Some(cfg), _) => Some(cfg.rocketCore) - case (None, cfg) => cfg - } - val rocketCores = rocketCoreOpts.zipWithIndex.map { - case (Some(cfg), _) => cfg - case (None, i) => throw new RuntimeException(s"Core $i must define [rocketCore]") - } - - TileTopology(coresWithTop, privateDCache, rocketCores) - } - - private def parseCoreEntry( - coreTable: Map[String, Value], - baseDir: Path, - sharedMem: SharedMemFields - ): (Option[GlobalConfig], Option[RocketCoreParam]) = { - val rocketCore = getTable(coreTable, "rocketCore").map(parseRocketCore) - - val balldomainOpt = coreTable.get("balldomain").flatMap { - case Value.Str(s) if s.nonEmpty => - val balldomainPath = baseDir.resolve(s).normalize() - Some(parseBallDomain(parseFile(balldomainPath.toString), balldomainPath.getParent)) - case Value.Str(_) => None - case Value.Tbl(t) => - val (resolved, newBase) = resolveInclude(t, baseDir) - Some(parseBallDomainTable(resolved, newBase)) - case _ => throw new RuntimeException("'balldomain' must be a string path or table") - } - - val buckyball = balldomainOpt.map { balldomain => - val memdomain = coreTable.get("memdomain") match { - case Some(v) => - val (tbl, _) = resolvePathOrTable(v, baseDir, "memdomain") - parseMemDomainTable(tbl, sharedMem) - case None => - throw new RuntimeException("Core with balldomain must define 'memdomain' (string path or table)") - } - - val frontend = coreTable.get("frontend") match { - case Some(v) => - val (tbl, _) = resolvePathOrTable(v, baseDir, "frontend") - parseFrontend(tbl) - case None => - throw new RuntimeException("Core with balldomain must define 'frontend' (string path or table)") - } - - val gpDomain = coreTable.get("gpdomain") match { - case Some(v) => - val (tbl, _) = resolvePathOrTable(v, baseDir, "gpdomain") - parseGpDomain(tbl) - case None => - throw new RuntimeException("Core with balldomain must define 'gpdomain' (string path or table)") - } - - val coreParam = coreTable.get("core") match { - case Some(v) => - val (tbl, _) = resolvePathOrTable(v, baseDir, "core") - parseCoreParam(tbl) - case None => - throw new RuntimeException("Core with balldomain must define 'core' (string path or table)") - } - - val parsedRocketCore = rocketCore.getOrElse( - throw new RuntimeException("Core with balldomain must have [rocketCore] section") - ) - - GlobalConfig().copy( - ballDomain = balldomain, - frontend = frontend, - gpDomain = gpDomain, - core = coreParam, - memDomain = memdomain, - rocketCore = parsedRocketCore - ) - } - - (buckyball, rocketCore) - } - - private def parseRocketCore(table: Map[String, Value]): RocketCoreParam = { - RocketCoreParam( - xLen = getInt(table, "xLen"), - pgLevels = getInt(table, "pgLevels"), - useVM = getBool(table, "useVM"), - useZba = getBool(table, "useZba"), - useZbb = getBool(table, "useZbb"), - useZbs = getBool(table, "useZbs"), - mulDiv = getTable(table, "mulDiv").map(parseMulDiv).getOrElse(framework.system.core.configs.MulDivParam()), - fpu = getTable(table, "fpu").map(parseFpu).getOrElse(framework.system.core.configs.FPUParam()), - dcache = getTable(table, "dcache").map(parseDCache).getOrElse(framework.system.core.configs.DCacheParam()), - icache = getTable(table, "icache").map(parseICache).getOrElse(framework.system.core.configs.ICacheParam()), - btb = getTable(table, "btb").map(parseBTB).getOrElse(framework.system.core.configs.BTBParam()) - ) - } - - private def parseMulDiv(table: Map[String, Value]): framework.system.core.configs.MulDivParam = - framework.system.core.configs.MulDivParam( - enable = getBool(table, "enable"), - mulUnroll = getInt(table, "mulUnroll"), - mulEarlyOut = getBool(table, "mulEarlyOut"), - divEarlyOut = getBool(table, "divEarlyOut") - ) - - private def parseFpu(table: Map[String, Value]): framework.system.core.configs.FPUParam = - framework.system.core.configs.FPUParam( - enable = getBool(table, "enable"), - minFLen = getInt(table, "minFLen"), - fLen = getInt(table, "fLen") - ) - - private def parseDCache(table: Map[String, Value]): framework.system.core.configs.DCacheParam = - framework.system.core.configs.DCacheParam( - nSets = getInt(table, "nSets"), - nWays = getInt(table, "nWays"), - nMSHRs = getInt(table, "nMSHRs") - ) - - private def parseICache(table: Map[String, Value]): framework.system.core.configs.ICacheParam = - framework.system.core.configs.ICacheParam( - nSets = getInt(table, "nSets"), - nWays = getInt(table, "nWays") - ) - - private def parseBTB(table: Map[String, Value]): framework.system.core.configs.BTBParam = - framework.system.core.configs.BTBParam( - enable = getBool(table, "enable"), - nEntries = getInt(table, "nEntries"), - nRAS = getInt(table, "nRAS") - ) - - private def parseFrontend(table: Map[String, Value]): FrontendParam = FrontendParam( - rob_entries = getInt(table, "robEntries"), - rs_out_of_order_response = getBool(table, "rsOutOfOrderResponse"), - bank_id_len = getInt(table, "bankIdLen"), - vbank_id_upper_bound = getInt(table, "vbankIdUpperBound"), - iter_len = getInt(table, "iterLen"), - sub_rob_enable = getBool(table, "subRobEnable"), - sub_rob_depth = getInt(table, "subRobDepth") - ) - - private def parseGpDomain(table: Map[String, Value]): GpDomainParam = GpDomainParam( - laneNumber = getInt(table, "laneNumber"), - chainingSize = getInt(table, "chainingSize"), - vLen = getInt(table, "vLen"), - dLen = getInt(table, "dLen"), - eLen = getInt(table, "eLen"), - laneScale = getInt(table, "laneScale") - ) - - private def parseCoreParam(table: Map[String, Value]): CoreParam = CoreParam( - coreDataBytes = getInt(table, "coreDataBytes"), - xLen = getInt(table, "xLen"), - vaddrBits = getInt(table, "vaddrBits"), - paddrBits = getInt(table, "paddrBits"), - pgIdxBits = getInt(table, "pgIdxBits"), - nPMPs = getInt(table, "nPMPs") - ) - - /** Parse a BallDomain file (returns BallDomainParam from the root table). */ - private def parseBallDomain(value: Value, baseDir: Path): BallDomainParam = { - val table = asTable(value, "balldomain root") - parseBallDomainTable(table, baseDir) - } - - private def parseBallDomainTable(table: Map[String, Value], baseDir: Path): BallDomainParam = { - val ballNum = getInt(table, "ballNum") - val ballIdMappings = getArray(table, "ballIdMappings").map { v => - val t = asTable(v, "ballIdMapping entry") - BallIdMapping( - ballId = getInt(t, "ballId"), - ballName = getString(t, "ballName"), - ballClass = getString(t, "ballClass"), - config = getOptionalString(t, "config"), - configBaseDir = baseDir.toString, - inBW = getInt(t, "inBW"), - outBW = getInt(t, "outBW") - ) - } - val ballISA = getArray(table, "ballISA").map { v => - val t = asTable(v, "ballISA entry") - BallISAEntry( - mnemonic = getString(t, "mnemonic"), - funct7 = getInt(t, "funct7"), - bid = getInt(t, "bid") - ) - } - BallDomainParam(ballNum = ballNum, ballIdMappings = ballIdMappings, ballISA = ballISA) - } - - /** Loader-private bundle of tile-shared memory fields. */ - private case class SharedMemFields( - sharedEnable: Boolean, - sharedEntries: Int, - sharedInputChannels: Int, - sharedDefaultGroupCount: Int) - - private def parseSharedMem(table: Map[String, Value]): SharedMemFields = { - val t = getTable(table, "sharedMem").getOrElse( - throw new RuntimeException("Tile must define [sharedMem] section") - ) - SharedMemFields( - sharedEnable = getBool(t, "enable"), - sharedEntries = getInt(t, "entries"), - sharedInputChannels = getInt(t, "inputChannels"), - sharedDefaultGroupCount = getInt(t, "defaultGroupCount") - ) - } - - /** Parse a memdomain file (or inline table) into a complete MemDomainParam. */ - private def parseMemDomain(value: Value, shared: SharedMemFields): MemDomainParam = { - val table = asTable(value, "memdomain root") - parseMemDomainTable(table, shared) - } - - private def parseMemDomainTable(table: Map[String, Value], shared: SharedMemFields): MemDomainParam = { - val bank = getTable(table, "bank").getOrElse(throw new RuntimeException("memdomain must define [bank]")) - val dma = getTable(table, "dma").getOrElse(throw new RuntimeException("memdomain must define [dma]")) - val tlb = getTable(table, "tlb").getOrElse(throw new RuntimeException("memdomain must define [tlb]")) - val tma = getTable(table, "tma").getOrElse(throw new RuntimeException("memdomain must define [tma]")) - val mmio = getTable(table, "mmio").getOrElse(throw new RuntimeException("memdomain must define [mmio]")) - val mem = getTable(table, "mem").getOrElse(throw new RuntimeException("memdomain must define [mem]")) - - MemDomainParam( - bankNum = getInt(bank, "num"), - bankWidth = getInt(bank, "width"), - bankEntries = getInt(bank, "entries"), - bankMaskLen = getInt(bank, "maskLen"), - sharedEnable = shared.sharedEnable, - sharedEntries = shared.sharedEntries, - sharedInputChannels = shared.sharedInputChannels, - sharedDefaultGroupCount = shared.sharedDefaultGroupCount, - tlb_size = getInt(tlb, "size"), - dma_n_xacts = getInt(dma, "nXacts"), - dma_burst_maxbytes = getInt(dma, "burstMaxBytes"), - bankChannel = getInt(bank, "channel"), - max_in_flight_mem_reqs = getInt(dma, "maxInFlightMemReqs"), - dma_buswidth = getInt(dma, "busWidth"), - memAddrLen = getInt(mem, "addrLen"), - tmaReadChannel = getInt(tma, "readChannel"), - tmaWriteChannel = getInt(tma, "writeChannel"), - mmioEnable = getBool(mmio, "enable"), - mmioBankNum = getInt(mmio, "bankNum"), - mmioBankEntries = getInt(mmio, "bankEntries"), - mmioBankWidth = getInt(mmio, "bankWidth"), - mmioReadWidth = getInt(mmio, "readWidth") - ) - } - - // --------------------------------------------------------------------------- - // TOML value extraction helpers - // --------------------------------------------------------------------------- - - private def asTable(value: Value, what: String): Map[String, Value] = value match { - case Value.Tbl(t) => t - case _ => throw new RuntimeException(s"Expected $what to be a TOML table") - } - - private def getTable(table: Map[String, Value], key: String): Option[Map[String, Value]] = - table.get(key) match { - case Some(Value.Tbl(t)) => Some(t) - case None => None - case _ => throw new RuntimeException(s"Expected table at key '$key'") - } - - private def getArray(table: Map[String, Value], key: String): Seq[Value] = - table.get(key) match { - case Some(Value.Arr(arr)) => arr - case None => throw new RuntimeException(s"Missing array at key '$key'") - case _ => throw new RuntimeException(s"Expected array at key '$key'") - } - - private def getString(table: Map[String, Value], key: String): String = - table.get(key) match { - case Some(Value.Str(s)) => s - case None => throw new RuntimeException(s"Missing string at key '$key'") - case _ => throw new RuntimeException(s"Expected string at key '$key'") - } - - private def getOptionalString(table: Map[String, Value], key: String): Option[String] = - table.get(key) match { - case Some(Value.Str(s)) => Some(s) - case None => None - case _ => throw new RuntimeException(s"Expected string at key '$key'") - } - - private def getInt(table: Map[String, Value], key: String): Int = - table.get(key) match { - case Some(Value.Num(n)) => n.toInt - case None => throw new RuntimeException(s"Missing integer at key '$key'") - case _ => throw new RuntimeException(s"Expected integer at key '$key'") - } - - private def getOptionalInt(table: Map[String, Value], key: String): Option[Int] = - table.get(key) match { - case Some(Value.Num(n)) => Some(n.toInt) - case None => None - case _ => throw new RuntimeException(s"Expected integer at key '$key'") - } - - private def getInt(value: Value, path: String): Int = { - val parts = path.split('.') - val finalValue = parts.foldLeft(Option(value)) { - case (Some(Value.Tbl(t)), key) => t.get(key) - case _ => None - } - finalValue match { - case Some(Value.Num(n)) => n.toInt - case _ => throw new RuntimeException(s"Missing or invalid integer at path '$path'") - } - } - - private def getBool(table: Map[String, Value], key: String): Boolean = - table.get(key) match { - case Some(Value.Bool(b)) => b - case None => throw new RuntimeException(s"Missing boolean at key '$key'") - case _ => throw new RuntimeException(s"Expected boolean at key '$key'") - } - - private def getOptionalBool(table: Map[String, Value], key: String): Option[Boolean] = - table.get(key) match { - case Some(Value.Bool(b)) => Some(b) - case None => None - case _ => throw new RuntimeException(s"Expected boolean at key '$key'") - } - -} - -/** Top-level example topology loaded from TOML. */ -case class ExampleTopology(tiles: Seq[TileTopology]) - -/** - * Per-tile topology: cores + optional privateDCache. - * - * @param cores One entry per core in this tile. `None` disables the - * Buckyball slot for that core (Rocket-only). - * @param privateDCache Resolved per-tile private DCache parameters, or `None` - * to skip the private DCache layer entirely. - */ -case class TileTopology( - cores: Seq[Option[GlobalConfig]], - privateDCache: Option[PrivateDCacheParams], - rocketCores: Seq[RocketCoreParam]) diff --git a/arch/src/main/scala/framework/system/configloader/Topology.scala b/arch/src/main/scala/framework/system/configloader/Topology.scala new file mode 100644 index 000000000..60e5f09d1 --- /dev/null +++ b/arch/src/main/scala/framework/system/configloader/Topology.scala @@ -0,0 +1,28 @@ +package framework.system.configloader + +import framework.system.tile.PrivateDCacheParams +import framework.system.core.configs.RocketCoreParam +import framework.top.GlobalConfig + +/** Loader-private bundle of tile-shared memory fields. */ +private[configloader] case class SharedMemFields( + sharedEnable: Boolean, + sharedEntries: Int, + sharedInputChannels: Int, + sharedDefaultGroupCount: Int) + +/** Top-level example topology loaded from chip.pb. */ +case class ExampleTopology(tiles: Seq[TileTopology]) + +/** + * Per-tile topology: cores + optional privateDCache. + * + * @param cores One entry per core in this tile. `None` disables the + * Buckyball slot for that core (Rocket-only). + * @param privateDCache Resolved per-tile private DCache parameters, or `None` + * to skip the private DCache layer entirely. + */ +case class TileTopology( + cores: Seq[Option[GlobalConfig]], + privateDCache: Option[PrivateDCacheParams], + rocketCores: Seq[RocketCoreParam]) diff --git a/arch/src/main/scala/framework/system/core/accelerator/BuckyballAccelerator.scala b/arch/src/main/scala/framework/system/core/accelerator/BuckyballAccelerator.scala index c38284500..d93b95c82 100644 --- a/arch/src/main/scala/framework/system/core/accelerator/BuckyballAccelerator.scala +++ b/arch/src/main/scala/framework/system/core/accelerator/BuckyballAccelerator.scala @@ -84,6 +84,9 @@ class BuckyballAccelerator(val b: GlobalConfig)(edge: TLEdgeOut) extends Module frontend.io.ball_subrob_req_i(i) <> ballDomain.subRobReq(i) } + memDomain.io.ballChannelActive := ballDomain.ballChannelActive + ballDomain.ballChannelReady := memDomain.io.ballChannelReady + // --- Frontend -> MemDomain --- memDomain.io.global_issue_i <> frontend.io.mem_issue_o frontend.io.mem_complete_i <> memDomain.io.global_complete_o @@ -128,6 +131,7 @@ class BuckyballAccelerator(val b: GlobalConfig)(edge: TLEdgeOut) extends Module // --- BallDomain <-> MemDomain (MMIO read path) --- ballDomain.mmioRead <> memDomain.io.ballDomain.mmioRead + ballDomain.mmioWrite <> memDomain.io.ballDomain.mmioWrite // --- PTW --- io.ptw(0).req <> memDomain.io.ptw(0).req diff --git a/arch/src/main/scala/framework/system/core/configs/RocketCoreParam.scala b/arch/src/main/scala/framework/system/core/configs/RocketCoreParam.scala index d47082d65..d6faf8745 100644 --- a/arch/src/main/scala/framework/system/core/configs/RocketCoreParam.scala +++ b/arch/src/main/scala/framework/system/core/configs/RocketCoreParam.scala @@ -102,17 +102,18 @@ object BTBParam { } case class RocketCoreParam( - xLen: Int = 64, - pgLevels: Int = 3, - useVM: Boolean = true, - useZba: Boolean = true, - useZbb: Boolean = true, - useZbs: Boolean = true, - mulDiv: MulDivParam = MulDivParam(), - fpu: FPUParam = FPUParam(), - dcache: DCacheParam = DCacheParam(), - icache: ICacheParam = ICacheParam(), - btb: BTBParam = BTBParam()) + xLen: Int = 64, + pgLevels: Int = 3, + useVM: Boolean = true, + useZba: Boolean = true, + useZbb: Boolean = true, + useZbs: Boolean = true, + haveCFlush: Boolean = false, + mulDiv: MulDivParam = MulDivParam(), + fpu: FPUParam = FPUParam(), + dcache: DCacheParam = DCacheParam(), + icache: ICacheParam = ICacheParam(), + btb: BTBParam = BTBParam()) object RocketCoreParam { implicit val rw: ReadWriter[RocketCoreParam] = macroRW @@ -128,6 +129,7 @@ object RocketCoreParam { useZba = p.useZba, useZbb = p.useZbb, useZbs = p.useZbs, + haveCFlush = p.haveCFlush, mulDiv = MulDivParam.toMulDivParams(p.mulDiv), fpu = FPUParam.toFPUParams(p.fpu) ) diff --git a/arch/src/main/scala/framework/system/core/rocket/RocketCoreBB.scala b/arch/src/main/scala/framework/system/core/rocket/RocketCoreBB.scala index bcc51e371..173621a84 100644 --- a/arch/src/main/scala/framework/system/core/rocket/RocketCoreBB.scala +++ b/arch/src/main/scala/framework/system/core/rocket/RocketCoreBB.scala @@ -191,7 +191,7 @@ class RocketBB(tile: BBTile, coreHasBuckyball: Boolean)(implicit p: Parameters) (usingDebug.option(new DebugDecode)) ++: (usingNMI.option(new NMIDecode)) ++: (usingConditionalZero.option(new ConditionalZeroDecode)) ++: - Seq(new FenceIDecode(tile.dcache.flushOnFenceI)) ++: + Seq(new FenceIDecode(tile.dcache.flushOnFenceI || coreParams.haveCFlush)) ++: coreParams.haveCFlush.option(new CFlushDecode(tile.dcache.canSupportCFlushLine)) ++: rocketParams.haveCease.option(new CeaseDecode) ++: usingVector.option(new VCFGDecode) ++: @@ -564,7 +564,7 @@ class RocketBB(tile: BBTile, coreHasBuckyball: Boolean)(implicit p: Parameters) when(id_ctrl.mem_cmd === M_SFENCE && csr.io.status.v) { ex_ctrl.mem_cmd := M_HFENCEV } - if (tile.dcache.flushOnFenceI) { + if (tile.dcache.flushOnFenceI || coreParams.haveCFlush) { when(id_ctrl.fence_i) { ex_reg_mem_size := 0.U } diff --git a/arch/src/main/scala/framework/system/tile/WithBuckyballTiles.scala b/arch/src/main/scala/framework/system/tile/WithBuckyballTiles.scala index 9f63dc163..f62c7439d 100644 --- a/arch/src/main/scala/framework/system/tile/WithBuckyballTiles.scala +++ b/arch/src/main/scala/framework/system/tile/WithBuckyballTiles.scala @@ -2,39 +2,24 @@ package framework.system.tile import org.chipsalliance.cde.config.{Config, Parameters} import freechips.rocketchip.subsystem.{CoherenceManagerWrapper, SubsystemBankedCoherenceKey} -import framework.system.configloader.TomlConfigLoader +import framework.system.configloader.ChipLoader /** - * Shared config fragment that builds an N-BBTile chipyard subsystem from a - * TOML topology file. - * - * Replaces the per-example `WithNXxxTiles` classes — every example now just - * picks a TOML path and calls this directly: - * - * {{{ - * class BuckyballToyConfig extends Config( - * new WithBuckyballTiles("../examples/chips/toy/configs/toy.toml") ++ - * new chipyard.config.WithSystemBusWidth(128) ++ - * new sims.base.BuckyballBaseConfig - * ) - * }}} - * - * @param tomlPath Path to the example's top-level TOML file (typically - * `../examples/chips//configs/.toml`). - * @param withBuckyball When false, every Buckyball slot in the topology is - * torn down (Rocket-only tiles). The tile/core counts - * themselves are unchanged. + * Build an N-BBTile chipyard subsystem from chip.pb. */ class WithBuckyballTiles( - tomlPath: String, + pbPath: String, withBuckyball: Boolean = true, hiddenHartBase: Option[Int] = None) - extends Config(WithBuckyballTiles.assemble(tomlPath, withBuckyball, hiddenHartBase)) + extends Config(WithBuckyballTiles.assemble(pbPath, withBuckyball, hiddenHartBase)) object WithBuckyballTiles { - def assemble(tomlPath: String, withBuckyball: Boolean, hiddenHartBase: Option[Int]): Parameters = { - val topology = TomlConfigLoader.load(tomlPath) + def assemble(pbPath: String, withBuckyball: Boolean, hiddenHartBase: Option[Int]): Parameters = { + if (!pbPath.endsWith(".pb")) { + throw new RuntimeException(s"WithBuckyballTiles expects a chip.pb path, got: $pbPath") + } + val topology = ChipLoader.load(pbPath) val tileFragments: Seq[Config] = topology.tiles.map { tile => val resolved = if (withBuckyball) tile.cores else tile.cores.map(_ => None) @@ -48,12 +33,6 @@ object WithBuckyballTiles { ) } - // If any tile has privateDCache enabled, the per-tile InclusiveCache acts - // as the last-level cache, which requires the system-level InclusiveCache - // to be replaced with `incoherentManager`. InclusiveCache requires - // `lastLevel = !managers.exists(_.regionType > UNCACHED)`, and a system-level - // InclusiveCache downstream would convert DRAM's regionType to CACHED, - // violating the constraint. See BBTile.scala for the full rationale. val anyPrivateDCache = topology.tiles.exists(_.privateDCache.isDefined) val coherenceFragment: Seq[Config] = if (anyPrivateDCache) Seq(new WithIncoherentSystemBus) else Nil @@ -63,15 +42,6 @@ object WithBuckyballTiles { } -/** - * Replaces the system-level coherence manager (default: InclusiveCache) with - * `incoherentManager`, a name-only pass-through. Use when per-tile privateDCache - * is enabled and acts as the last-level cache. - * - * Tile-to-tile coherence is NOT maintained at the system bus when this is - * applied: software must use explicit cache management (flush/invalidate) or - * uncacheable channels (scratchpad, DMA) for cross-tile communication. - */ class WithIncoherentSystemBus extends Config((site, here, up) => { case SubsystemBankedCoherenceKey => up(SubsystemBankedCoherenceKey, site).copy( diff --git a/arch/src/main/scala/framework/top/configs/TopConfig.scala b/arch/src/main/scala/framework/top/configs/TopConfig.scala index ece79f74f..909509d09 100644 --- a/arch/src/main/scala/framework/top/configs/TopConfig.scala +++ b/arch/src/main/scala/framework/top/configs/TopConfig.scala @@ -9,6 +9,6 @@ case class TopConfig( object TopConfig { implicit val rw: ReadWriter[TopConfig] = macroRW - /** Stub; real values come from TomlConfigLoader (tile + derived nCores). */ + /** Stub; real values come from ChipLoader (tile + derived nCores). */ def apply(): TopConfig = TopConfig(memBallChannelNum = 0, nCores = 0) } diff --git a/arch/src/main/scala/sims/p2e/P2ETop.scala b/arch/src/main/scala/sims/p2e/P2ETop.scala index db805c3ad..0cf45f15c 100644 --- a/arch/src/main/scala/sims/p2e/P2ETop.scala +++ b/arch/src/main/scala/sims/p2e/P2ETop.scala @@ -604,13 +604,3 @@ class P2ETop extends RawModule { io.ddr4_en_vcc2v5 := top.io.ddr4_en_vcc2v5 top.io.power_good := io.power_good } - -object ElaborateP2ETop extends App { - import _root_.circt.stage.ChiselStage - - ChiselStage.emitSystemVerilogFile( - new P2ETop, - firtoolOpts = args, - args = Array.empty - ) -} diff --git a/arch/src/main/scala/sims/p2e/TargetConfigs.scala b/arch/src/main/scala/sims/p2e/TargetConfigs.scala index 1910df434..507e5680c 100644 --- a/arch/src/main/scala/sims/p2e/TargetConfigs.scala +++ b/arch/src/main/scala/sims/p2e/TargetConfigs.scala @@ -109,9 +109,16 @@ object Elaborate extends App { sys.exit(1) } + val firtoolOpts = args.drop(1) + ChiselStage.emitSystemVerilogFile( new P2EHarness()(config.toInstance), - firtoolOpts = args.drop(1), + firtoolOpts = firtoolOpts, + args = Array.empty + ) + ChiselStage.emitSystemVerilogFile( + new P2ETop, + firtoolOpts = firtoolOpts, args = Array.empty ) } diff --git a/bb-tests/workloads/CMakeLists.txt b/bb-tests/workloads/CMakeLists.txt index a7a874636..0e816bb4d 100644 --- a/bb-tests/workloads/CMakeLists.txt +++ b/bb-tests/workloads/CMakeLists.txt @@ -4,20 +4,20 @@ set(CMAKE_EXPORT_COMPILE_COMMANDS ON) set(WORKLOAD_DIR ${CMAKE_CURRENT_SOURCE_DIR}) set(WORKLOAD_SRC_DIR ${WORKLOAD_DIR}/src) + +get_filename_component(BUCKYBALL_REPO_ROOT ${WORKLOAD_DIR}/../.. ABSOLUTE) set(WORKLOAD_LIB_DIR ${WORKLOAD_DIR}/lib) set(BBSW_BAREMETAL_DIR ${WORKLOAD_LIB_DIR}/bbsw/baremetal) set(RISCV_GNU_TOOLCHAIN $ENV{RISCV}) -set(BUDDY_MLIR_DIR $ENV{BUDDY_MLIR_BUILD_DIR}/../) +set(BUDDY_MLIR_DIR ${BUCKYBALL_REPO_ROOT}/compiler/thirdparty/buddy-mlir) set(BUDDY_BINARY_DIR $ENV{BUDDY_MLIR_BUILD_DIR}/bin) set(BUDDY_OPT ${BUDDY_BINARY_DIR}/buddy-opt) set(BUDDY_TRANSLATE ${BUDDY_BINARY_DIR}/buddy-translate) set(BUDDY_LLC ${BUDDY_BINARY_DIR}/buddy-llc) set(LLVM_BUILD_DIR ${BUDDY_MLIR_DIR}/llvm/build) -get_filename_component(BUCKYBALL_REPO_ROOT ${WORKLOAD_DIR}/../.. ABSOLUTE) set(BUCKYBALL_WORKLOAD_CHIP "" CACHE STRING "Buckyball chip for workloads") -set(BUCKYBALL_WORKLOAD_CORE "" CACHE STRING "Compiler Core for workloads") if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP is required. Pass -DBUCKYBALL_WORKLOAD_CHIP=.") @@ -28,122 +28,323 @@ if(NOT IS_DIRECTORY "Buckyball workload chip does not exist: ${BUCKYBALL_WORKLOAD_CHIP}") endif() message(STATUS "Buckyball workload chip: ${BUCKYBALL_WORKLOAD_CHIP}") -if(BUCKYBALL_WORKLOAD_CORE STREQUAL "") - message(FATAL_ERROR - "BUCKYBALL_WORKLOAD_CORE is required. bbdev derives it from chip.toml.") -endif() -message(STATUS "Buckyball workload compiler Core: ${BUCKYBALL_WORKLOAD_CORE}") -set(BUCKYBALL_CORE_DIR - "${BUCKYBALL_REPO_ROOT}/examples/cores/${BUCKYBALL_WORKLOAD_CORE}") -set(BUCKYBALL_CORE_MANIFEST "${BUCKYBALL_CORE_DIR}/manifest.toml") -if(NOT EXISTS "${BUCKYBALL_CORE_MANIFEST}") - message(FATAL_ERROR "Core manifest does not exist: ${BUCKYBALL_CORE_MANIFEST}") +if(NOT IS_ABSOLUTE "${Python3_EXECUTABLE}") + find_program(BUCKYBALL_PYTHON3_EXECUTABLE NAMES python3) + if(NOT BUCKYBALL_PYTHON3_EXECUTABLE) + message(FATAL_ERROR "python3 is required to build workloads") + endif() + set(Python3_EXECUTABLE "${BUCKYBALL_PYTHON3_EXECUTABLE}") endif() -# Compiler hardware parameters come from the Core memory-domain TOML. Keep -# this small parser here because the workload configure step already owns the -# selected Core and CMake has no built-in TOML reader. -file(STRINGS "${BUCKYBALL_CORE_MANIFEST}" BUCKYBALL_CORE_CONFIG_LINE - REGEX "^[ \t]*config[ \t]*=") -list(LENGTH BUCKYBALL_CORE_CONFIG_LINE BUCKYBALL_CORE_CONFIG_COUNT) -if(NOT BUCKYBALL_CORE_CONFIG_COUNT EQUAL 1) +foreach(_BUCKYBALL_REQ IN ITEMS + BUCKYBALL_CHIP_PB) + if(NOT DEFINED ${_BUCKYBALL_REQ} OR "${${_BUCKYBALL_REQ}}" STREQUAL "") + message(FATAL_ERROR + "${_BUCKYBALL_REQ} must be passed by bbdev config --install (cmake.defs)") + endif() +endforeach() + +if(NOT EXISTS "${BUCKYBALL_CHIP_PB}") + message(FATAL_ERROR "missing Chip.pb: ${BUCKYBALL_CHIP_PB}") +endif() +set(BUCKYBALL_ISA_DIR "${CMAKE_BINARY_DIR}/generated/isa") +execute_process( + COMMAND "${Python3_EXECUTABLE}" + "${BUCKYBALL_REPO_ROOT}/compiler/scripts/pb_to_target_registry.py" + --repo "${BUCKYBALL_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --isa-dir "${BUCKYBALL_ISA_DIR}" + --print-targets + --print-target-balls + --print-core-targets + RESULT_VARIABLE BUCKYBALL_TARGET_REGISTRY_RESULT + OUTPUT_VARIABLE BUCKYBALL_TARGETS_RAW + ERROR_VARIABLE BUCKYBALL_TARGET_REGISTRY_ERROR) +if(NOT BUCKYBALL_TARGET_REGISTRY_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to generate target ISA headers: ${BUCKYBALL_TARGET_REGISTRY_ERROR}") +endif() +string(REPLACE "\n" ";" BUCKYBALL_TARGETS "${BUCKYBALL_TARGETS_RAW}") +execute_process( + COMMAND "${Python3_EXECUTABLE}" + "${BUCKYBALL_REPO_ROOT}/compiler/scripts/pb_to_target_registry.py" + --repo "${BUCKYBALL_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --print-bank-targets + RESULT_VARIABLE BUCKYBALL_BANK_TARGETS_RESULT + OUTPUT_VARIABLE BUCKYBALL_BANK_TARGETS_RAW + ERROR_VARIABLE BUCKYBALL_BANK_TARGETS_ERROR) +if(NOT BUCKYBALL_BANK_TARGETS_RESULT EQUAL 0) message(FATAL_ERROR - "${BUCKYBALL_CORE_MANIFEST} must define exactly one core config entry") + "failed to get compiler targets with banks: ${BUCKYBALL_BANK_TARGETS_ERROR}") endif() -list(GET BUCKYBALL_CORE_CONFIG_LINE 0 BUCKYBALL_CORE_CONFIG_LINE) -string(REGEX REPLACE "^[^=]+=[ \t]*\"([^\"]+)\".*" "\\1" - BUCKYBALL_CORE_CONFIG_REL "${BUCKYBALL_CORE_CONFIG_LINE}") -set(BUCKYBALL_CORE_CONFIG_TOML - "${BUCKYBALL_CORE_DIR}/${BUCKYBALL_CORE_CONFIG_REL}") -if(NOT EXISTS "${BUCKYBALL_CORE_CONFIG_TOML}") - message(FATAL_ERROR "Core config TOML does not exist: ${BUCKYBALL_CORE_CONFIG_TOML}") +string(REPLACE "\n" ";" BUCKYBALL_BANK_TARGETS + "${BUCKYBALL_BANK_TARGETS_RAW}") +list(REMOVE_ITEM BUCKYBALL_BANK_TARGETS "") +set(BUCKYBALL_TARGET_BALL_LINES) +set(BUCKYBALL_CORE_TARGET_LINES) +foreach(BUCKYBALL_TARGET_LINE ${BUCKYBALL_TARGETS}) + if(BUCKYBALL_TARGET_LINE MATCHES "^([0-9]+):([^,]+)$") + set(BUCKYBALL_CORE_TARGET_${CMAKE_MATCH_1} "${CMAKE_MATCH_2}") + list(APPEND BUCKYBALL_CORE_TARGET_LINES ${BUCKYBALL_TARGET_LINE}) + elseif(BUCKYBALL_TARGET_LINE MATCHES "^([^:]+):(.*)$") + set(BUCKYBALL_TARGET_BALLS_${CMAKE_MATCH_1} "${CMAKE_MATCH_2}") + list(APPEND BUCKYBALL_TARGET_BALL_LINES ${BUCKYBALL_TARGET_LINE}) + endif() +endforeach() +list(REMOVE_ITEM BUCKYBALL_TARGETS ${BUCKYBALL_TARGET_BALL_LINES} + ${BUCKYBALL_CORE_TARGET_LINES} "") +if(NOT BUCKYBALL_TARGETS) + message(FATAL_ERROR "Chip.pb has no compiler targets") endif() -get_filename_component(BUCKYBALL_CORE_CONFIG_DIR - "${BUCKYBALL_CORE_CONFIG_TOML}" DIRECTORY) +message(STATUS "Buckyball compiler targets: ${BUCKYBALL_TARGETS}") +message(STATUS "Buckyball bank targets: ${BUCKYBALL_BANK_TARGETS}") -file(STRINGS "${BUCKYBALL_CORE_CONFIG_TOML}" BUCKYBALL_MEMDOMAIN_LINE - REGEX "^[ \t]*memdomain[ \t]*=") -list(LENGTH BUCKYBALL_MEMDOMAIN_LINE BUCKYBALL_MEMDOMAIN_COUNT) -if(NOT BUCKYBALL_MEMDOMAIN_COUNT EQUAL 1) - message(FATAL_ERROR - "${BUCKYBALL_CORE_CONFIG_TOML} must define exactly one memdomain entry") +add_custom_target(rushB-bemu-workloads-build) +add_custom_target(rushB-verilator-workloads-build) + +if(NOT DEFINED ELF_CC) + set(ELF_CC "riscv64-unknown-elf-gcc") endif() -list(GET BUCKYBALL_MEMDOMAIN_LINE 0 BUCKYBALL_MEMDOMAIN_LINE) -string(REGEX REPLACE "^[^=]+=[ \t]*\"([^\"]+)\".*" "\\1" - BUCKYBALL_MEMDOMAIN_REL "${BUCKYBALL_MEMDOMAIN_LINE}") -set(BUCKYBALL_MEMDOMAIN_TOML - "${BUCKYBALL_CORE_CONFIG_DIR}/${BUCKYBALL_MEMDOMAIN_REL}") -if(NOT EXISTS "${BUCKYBALL_MEMDOMAIN_TOML}") - message(FATAL_ERROR "Memory-domain TOML does not exist: ${BUCKYBALL_MEMDOMAIN_TOML}") + +if(NOT DEFINED LINUX_CC) + set(LINUX_CC "riscv64-unknown-linux-gnu-gcc") endif() -file(STRINGS "${BUCKYBALL_MEMDOMAIN_TOML}" BUCKYBALL_MEMDOMAIN_LINES) -set(BUCKYBALL_IN_BANK_SECTION FALSE) -foreach(BUCKYBALL_MEMDOMAIN_LINE IN LISTS BUCKYBALL_MEMDOMAIN_LINES) - # Strip comments and surrounding whitespace before interpreting the TOML. - string(REGEX REPLACE "#.*$" "" BUCKYBALL_MEMDOMAIN_LINE - "${BUCKYBALL_MEMDOMAIN_LINE}") - string(STRIP "${BUCKYBALL_MEMDOMAIN_LINE}" BUCKYBALL_MEMDOMAIN_LINE) - if(BUCKYBALL_MEMDOMAIN_LINE STREQUAL "[bank]") - set(BUCKYBALL_IN_BANK_SECTION TRUE) - continue() +set(BBSIM_LD ${BBSW_BAREMETAL_DIR}/bbsim.ld) +set(BUDDY_CYCLE_TRACE_RUNTIME_SRC ${BBSW_BAREMETAL_DIR}/buddy_cycle_trace.c) + +set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -march=rv64gc") +set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static -Wl,--no-dynamic-linker") + +function(add_buckyball_rushb_native TARGET_NAME) + cmake_parse_arguments(ARG "CXX" "OUTPUT_SUBDIR" "SOURCES;INCLUDE_DIRS;DEPENDS" ${ARGN}) + + if(NOT ARG_SOURCES) + message(FATAL_ERROR "${TARGET_NAME}: rushB runner has no sources") + endif() + if(NOT ARG_OUTPUT_SUBDIR) + message(FATAL_ERROR "${TARGET_NAME}: rushB runner has no output directory") endif() - if(BUCKYBALL_MEMDOMAIN_LINE MATCHES "^\\[") - set(BUCKYBALL_IN_BANK_SECTION FALSE) + + if(NOT DEFINED BUILD_BIN_DIR OR NOT DEFINED OUTPUT_BIN_DIR) + message(FATAL_ERROR + "add_buckyball_rushb_native requires BUILD_BIN_DIR and OUTPUT_BIN_DIR") + endif() + + set(RUSHB_OUTPUT_ROOT ${OUTPUT_BIN_DIR}/${ARG_OUTPUT_SUBDIR}) + set(RUSHB_SYNC_ROOT ${BUILD_BIN_DIR}/${ARG_OUTPUT_SUBDIR}) + string(MAKE_C_IDENTIFIER "${ARG_OUTPUT_SUBDIR}" RUSHB_OUTPUT_ROOT_ID) + set(RUSHB_OUTPUT_ROOT_TARGET rushB-output-${RUSHB_OUTPUT_ROOT_ID}) + if(NOT TARGET ${RUSHB_OUTPUT_ROOT_TARGET}) + add_custom_target(${RUSHB_OUTPUT_ROOT_TARGET} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_OUTPUT_ROOT} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_SYNC_ROOT} + COMMENT "Creating rushB output directory: ${ARG_OUTPUT_SUBDIR}" + VERBATIM) + endif() + + if(ARG_CXX) + set(RUSHB_COMPILER c++) + set(RUSHB_STANDARD -std=c++17) + else() + set(RUSHB_COMPILER cc) + set(RUSHB_STANDARD -std=gnu17) + endif() + + foreach(RUSHB_BACKEND bemu verilator) + set(RUSHB_TARGET ${TARGET_NAME}-rushB-${RUSHB_BACKEND}-run) + set(RUSHB_BUILD_DIR ${CMAKE_CURRENT_BINARY_DIR}/rushB/${RUSHB_TARGET}) + set(RUSHB_BINARY ${RUSHB_BUILD_DIR}/${RUSHB_TARGET}.bin) + set(RUSHB_OUTPUT_DIR ${RUSHB_OUTPUT_ROOT}/${RUSHB_TARGET}) + set(RUSHB_SYNC_DIR ${RUSHB_SYNC_ROOT}/${RUSHB_TARGET}) + if(RUSHB_BACKEND STREQUAL "bemu") + set(RUSHB_LIBRARY ${BUCKYBALL_RUSHB_BEMU_LIBRARY}) + set(RUSHB_LIBRARY_NAME bebop_bemu) + set(RUSHB_BUILD_RUNTIME + COMMAND ${CMAKE_COMMAND} -E env CARGO_TARGET_DIR=${BUCKYBALL_CARGO_TARGET_DIR} + cargo build --release --manifest-path ${BUCKYBALL_RUSHB_BEMU_MANIFEST} --lib) + set(RUSHB_RUNTIME_DEPENDENCY ${BUCKYBALL_RUSHB_BEMU_MANIFEST}) + else() + set(RUSHB_LIBRARY ${BUCKYBALL_RUSHB_VERILATOR_LIBRARY}) + set(RUSHB_LIBRARY_NAME bebop_verilator) + set(RUSHB_BUILD_RUNTIME) + set(RUSHB_RUNTIME_DEPENDENCY ${BUCKYBALL_RUSHB_VERILATOR_LIBRARY}) + endif() + if(NOT RUSHB_LIBRARY) + message(FATAL_ERROR + "${TARGET_NAME}: BUCKYBALL_RUSHB_${RUSHB_BACKEND} library is unset. " + "RushB paths must be passed by Bazel/bbdev (-DBUCKYBALL_RUSHB_*).") + endif() + + set(RUSHB_RUNTIME_OUTPUT + ${RUSHB_OUTPUT_ROOT}/lib${RUSHB_LIBRARY_NAME}.so) + set(RUSHB_RUNTIME_SYNC + ${RUSHB_SYNC_ROOT}/lib${RUSHB_LIBRARY_NAME}.so) + set(RUSHB_RUNTIME_TARGET + rushB-runtime-${RUSHB_OUTPUT_ROOT_ID}-${RUSHB_BACKEND}) + if(NOT TARGET ${RUSHB_RUNTIME_TARGET}) + add_custom_command( + OUTPUT ${RUSHB_RUNTIME_OUTPUT} ${RUSHB_RUNTIME_SYNC} + ${RUSHB_BUILD_RUNTIME} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_OUTPUT_ROOT} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_SYNC_ROOT} + COMMAND ${CMAKE_COMMAND} -E copy_if_different + ${RUSHB_LIBRARY} ${RUSHB_RUNTIME_OUTPUT} + COMMAND ${CMAKE_COMMAND} -E copy_if_different + ${RUSHB_LIBRARY} ${RUSHB_RUNTIME_SYNC} + DEPENDS ${RUSHB_RUNTIME_DEPENDENCY} + COMMENT "Installing rushB ${RUSHB_BACKEND} runtime: ${ARG_OUTPUT_SUBDIR}" + VERBATIM) + add_custom_target(${RUSHB_RUNTIME_TARGET} + DEPENDS ${RUSHB_RUNTIME_OUTPUT} ${RUSHB_RUNTIME_SYNC}) + add_dependencies(${RUSHB_RUNTIME_TARGET} ${RUSHB_OUTPUT_ROOT_TARGET}) + endif() + + get_filename_component(RUSHB_LIBRARY_DIR ${RUSHB_LIBRARY} DIRECTORY) + set(RUSHB_INCLUDE_ARGS -I${BUCKYBALL_REPO_ROOT}/compiler/include) + foreach(RUSHB_INCLUDE_DIR ${ARG_INCLUDE_DIRS}) + list(APPEND RUSHB_INCLUDE_ARGS -I${RUSHB_INCLUDE_DIR}) + endforeach() + set(RUSHB_INSTALLED ${RUSHB_OUTPUT_DIR}/${RUSHB_TARGET}) + set(RUSHB_SYNCED ${RUSHB_SYNC_DIR}/${RUSHB_TARGET}) + add_custom_command( + OUTPUT ${RUSHB_BINARY} ${RUSHB_INSTALLED} ${RUSHB_SYNCED} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_BUILD_DIR} + COMMAND ${RUSHB_COMPILER} -no-pie ${RUSHB_STANDARD} -O2 -DBUCKYBALL_RUSHB + ${RUSHB_INCLUDE_ARGS} + ${ARG_SOURCES} ${BUCKYBALL_REPO_ROOT}/compiler/lib/RushBRuntime.c + -L${RUSHB_LIBRARY_DIR} -l${RUSHB_LIBRARY_NAME} + -Wl,-rpath,${RUSHB_OUTPUT_ROOT} -o ${RUSHB_BINARY} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_OUTPUT_DIR} + COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_SYNC_DIR} + COMMAND ${CMAKE_COMMAND} -E copy_if_different ${RUSHB_BINARY} + ${RUSHB_INSTALLED} + COMMAND ${CMAKE_COMMAND} -E copy_if_different ${RUSHB_BINARY} + ${RUSHB_SYNCED} + DEPENDS ${ARG_SOURCES} ${ARG_DEPENDS} + ${BUCKYBALL_REPO_ROOT}/compiler/include/buckyball/rushb.h + ${BUCKYBALL_REPO_ROOT}/compiler/lib/RushBRuntime.c + COMMENT "Building rushB ${RUSHB_BACKEND}: ${TARGET_NAME}" + VERBATIM) + add_custom_target(${RUSHB_TARGET} + DEPENDS ${RUSHB_BINARY} ${RUSHB_INSTALLED} ${RUSHB_SYNCED}) + add_dependencies(${RUSHB_TARGET} ${RUSHB_OUTPUT_ROOT_TARGET}) + add_dependencies(${RUSHB_TARGET} ${RUSHB_RUNTIME_TARGET}) + add_dependencies(rushB-${RUSHB_BACKEND}-workloads-build ${RUSHB_TARGET}) + endforeach() +endfunction() + +function(buckyball_chip_ball_dirs OUT_VAR TARGET) + set(_raw "${BUCKYBALL_TARGET_BALLS_${TARGET}}") + if(_raw STREQUAL "") + set(${OUT_VAR} "" PARENT_SCOPE) + return() endif() - if(BUCKYBALL_IN_BANK_SECTION AND - BUCKYBALL_MEMDOMAIN_LINE MATCHES "^(num|width|entries)[ \t]*=[ \t]*[0-9]+$") - string(REGEX MATCH "^([^=]+)[ \t]*=[ \t]*([0-9]+)$" - BUCKYBALL_BANK_MATCH "${BUCKYBALL_MEMDOMAIN_LINE}") - string(STRIP "${CMAKE_MATCH_1}" BUCKYBALL_BANK_KEY) - set(BUCKYBALL_BANK_VALUE "${CMAKE_MATCH_2}") - if(BUCKYBALL_BANK_KEY STREQUAL "num") - set(BUCKYBALL_MLIR_BANK_NUM "${BUCKYBALL_BANK_VALUE}") - elseif(BUCKYBALL_BANK_KEY STREQUAL "width") - set(BUCKYBALL_MLIR_BANK_WIDTH_BITS "${BUCKYBALL_BANK_VALUE}") - elseif(BUCKYBALL_BANK_KEY STREQUAL "entries") - set(BUCKYBALL_MLIR_BANK_DEPTH "${BUCKYBALL_BANK_VALUE}") + string(REPLACE "," ";" _dirs "${_raw}") + foreach(_dir IN LISTS _dirs) + if(_dir STREQUAL "") + message(FATAL_ERROR "empty ball_dir on target ${TARGET}") endif() + endforeach() + set(${OUT_VAR} "${_dirs}" PARENT_SCOPE) +endfunction() + +function(buckyball_add_ball_workload_subdirs KIND) + if(NOT KIND STREQUAL "ctests" AND NOT KIND STREQUAL "mlir_tests") + message(FATAL_ERROR "unknown ball workload kind: ${KIND}") endif() -endforeach() -foreach(BUCKYBALL_BANK_PARAMETER IN ITEMS - BUCKYBALL_MLIR_BANK_NUM BUCKYBALL_MLIR_BANK_WIDTH_BITS BUCKYBALL_MLIR_BANK_DEPTH) - if(NOT DEFINED ${BUCKYBALL_BANK_PARAMETER}) + foreach(BUCKYBALL_TARGET IN LISTS BUCKYBALL_TARGETS) + buckyball_chip_ball_dirs(_ball_dirs ${BUCKYBALL_TARGET}) + foreach(BUCKYBALL_BALL_DIR IN LISTS _ball_dirs) + set(_src + ${BUCKYBALL_REPO_ROOT}/examples/balls/${BUCKYBALL_BALL_DIR}/workloads/${KIND}) + if(NOT IS_DIRECTORY ${_src}) + if(KIND STREQUAL "mlir_tests") + continue() + endif() + message(FATAL_ERROR "missing Ball ${KIND} directory: ${_src}") + endif() + if(KIND STREQUAL "ctests") + set(BUCKYBALL_CTEST_TARGET ${BUCKYBALL_TARGET}) + add_subdirectory( + ${_src} + ${CMAKE_CURRENT_BINARY_DIR}/balls/${BUCKYBALL_TARGET}/${BUCKYBALL_BALL_DIR}/ctests) + else() + set(BUCKYBALL_MLIR_ACTIVE_TARGET ${BUCKYBALL_TARGET}) + add_subdirectory( + ${_src} + ${CMAKE_CURRENT_BINARY_DIR}/balls/${BUCKYBALL_TARGET}/${BUCKYBALL_BALL_DIR}) + endif() + endforeach() + endforeach() +endfunction() + +function(add_toy_mlir_rushb) + cmake_parse_arguments(ARG "CXX" "NAME;TARGET_STEM;TARGET" + "FRONT_PASSES;SOURCES;INCLUDE_DIRS;DEPENDS" ${ARGN}) + + if(NOT ARG_NAME OR NOT ARG_TARGET_STEM OR NOT ARG_TARGET) + message(FATAL_ERROR "add_toy_mlir_rushb requires NAME, TARGET_STEM, and TARGET") + endif() + list(FIND BUCKYBALL_TARGETS "${ARG_TARGET}" _buckyball_target_index) + if(_buckyball_target_index EQUAL -1) + message(FATAL_ERROR "unknown Buckyball target: ${ARG_TARGET}") + endif() + if(NOT DEFINED BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS OR + NOT DEFINED BUCKYBALL_LOWER_BUCKYBALL_RUSHB) message(FATAL_ERROR - "${BUCKYBALL_MEMDOMAIN_TOML} [bank] is missing ${BUCKYBALL_BANK_PARAMETER}") + "add_toy_mlir_rushb requires BUCKYBALL_LOWER_*_RUSHB") + endif() + if(NOT DEFINED BUCKYBALL_RUSHB_BEMU_MANIFEST OR + NOT DEFINED BUCKYBALL_RUSHB_VERILATOR_LIBRARY) + return() endif() -endforeach() -math(EXPR BUCKYBALL_MLIR_BANK_WIDTH_BYTES - "${BUCKYBALL_MLIR_BANK_WIDTH_BITS} / 8") -math(EXPR BUCKYBALL_MLIR_BANK_WIDTH_REMAINDER - "${BUCKYBALL_MLIR_BANK_WIDTH_BITS} % 8") -if(BUCKYBALL_MLIR_BANK_WIDTH_BITS LESS 8 OR - NOT BUCKYBALL_MLIR_BANK_WIDTH_REMAINDER EQUAL 0) - message(FATAL_ERROR - "Core bank width must be a positive whole number of bytes: ${BUCKYBALL_MLIR_BANK_WIDTH_BITS}") -endif() -set(BUCKYBALL_CONVERT_TILE_TO_BUCKYBALL - "--convert-tile-to-buckyball=bank_width=${BUCKYBALL_MLIR_BANK_WIDTH_BYTES} bank_depth=${BUCKYBALL_MLIR_BANK_DEPTH} bank_num=${BUCKYBALL_MLIR_BANK_NUM}") -set(BUCKYBALL_ASSIGN_PHYSICAL_BANKS - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}") -message(STATUS "Buckyball Core banks: ${BUCKYBALL_MLIR_BANK_NUM} x ${BUCKYBALL_MLIR_BANK_WIDTH_BITS}-bit x ${BUCKYBALL_MLIR_BANK_DEPTH}") - -set(BUCKYBALL_RUSHB_CHIP_CMAKE - ${BUCKYBALL_REPO_ROOT}/examples/chips/${BUCKYBALL_WORKLOAD_CHIP}/workloads/cmake/RushB.cmake) -if(EXISTS ${BUCKYBALL_RUSHB_CHIP_CMAKE}) - include(${BUCKYBALL_RUSHB_CHIP_CMAKE}) -endif() -if(DEFINED BUCKYBALL_RUSHB_BEMU_MANIFEST) - add_custom_target(rushB-bemu-workloads-build) -endif() -if(DEFINED BUCKYBALL_RUSHB_VERILATOR_LIBRARY) - add_custom_target(rushB-verilator-workloads-build) -endif() + set(MLIR_SRC ${CMAKE_CURRENT_SOURCE_DIR}/${ARG_NAME}.mlir) + set(RUSHB_OBJ ${CMAKE_CURRENT_BINARY_DIR}/${ARG_NAME}-rushB.o) + + add_custom_command( + OUTPUT ${RUSHB_OBJ} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} + --target=${ARG_TARGET} + ${ARG_FRONT_PASSES} + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS} + -convert-linalg-to-loops + -expand-strided-metadata + -lower-affine + -convert-scf-to-cf + -convert-cf-to-llvm + ${BUCKYBALL_LOWER_BUCKYBALL_RUSHB} + -convert-arith-to-llvm + -convert-math-to-llvm + -finalize-memref-to-llvm + -convert-func-to-llvm + -lower-buckyball-intrinsics-to-rushb + -reconcile-unrealized-casts | + ${BUDDY_TRANSLATE} --buddy-to-llvmir | + ${BUDDY_LLC} -filetype=obj -mtriple=x86_64 -O2 -o ${RUSHB_OBJ} + DEPENDS ${MLIR_SRC} ${BUDDY_OPT} ${BUDDY_TRANSLATE} ${BUDDY_LLC} + ${ARG_DEPENDS} + COMMENT "Building rushB ${ARG_NAME}.o from Toy MLIR" + VERBATIM) + + set(RUSHB_CXX_FLAG) + if(ARG_CXX) + set(RUSHB_CXX_FLAG CXX) + endif() -set(BUCKYBALL_EXPECTED_COMPILER_DIR - "${BUCKYBALL_REPO_ROOT}/examples/cores/${BUCKYBALL_WORKLOAD_CORE}/compiler") + add_buckyball_rushb_native(${ARG_TARGET_STEM} + ${RUSHB_CXX_FLAG} + OUTPUT_SUBDIR src/CTest/rushB + SOURCES ${ARG_SOURCES} ${RUSHB_OBJ} + INCLUDE_DIRS ${ARG_INCLUDE_DIRS} + DEPENDS ${RUSHB_OBJ} ${ARG_DEPENDS}) +endfunction() + +set(BUCKYBALL_EXPECTED_COMPILER_DIR "${BUCKYBALL_REPO_ROOT}/compiler") set(BUCKYBALL_COMPILER_CACHE "$ENV{BUDDY_MLIR_BUILD_DIR}/CMakeCache.txt") if(NOT EXISTS "${BUCKYBALL_COMPILER_CACHE}") message(FATAL_ERROR @@ -165,15 +366,29 @@ get_filename_component(BUCKYBALL_COMPILER_DIR "${BUCKYBALL_COMPILER_DIR}" ABSOLUTE) if(NOT BUCKYBALL_COMPILER_DIR STREQUAL BUCKYBALL_EXPECTED_COMPILER_DIR) message(FATAL_ERROR - "Compiler Core mismatch: workload chip is ${BUCKYBALL_WORKLOAD_CHIP}, " - "whose compiler Core is ${BUCKYBALL_WORKLOAD_CORE}, " - "but BUDDY_EXTERNAL_DIALECTS_DIR is ${BUCKYBALL_COMPILER_DIR}. " - "Run `bbdev compiler --build '--core ${BUCKYBALL_WORKLOAD_CORE}'` first.") + "Compiler mismatch: workload chip is ${BUCKYBALL_WORKLOAD_CHIP}, " + "expected BUDDY_EXTERNAL_DIALECTS_DIR ${BUCKYBALL_EXPECTED_COMPILER_DIR}, " + "but got ${BUCKYBALL_COMPILER_DIR}.") +endif() +file(STRINGS "${BUCKYBALL_COMPILER_CACHE}" BUCKYBALL_COMPILER_PB_LINES + REGEX "^BUCKYBALL_CHIP_PB:[^=]+=") +list(LENGTH BUCKYBALL_COMPILER_PB_LINES BUCKYBALL_COMPILER_PB_COUNT) +if(NOT BUCKYBALL_COMPILER_PB_COUNT EQUAL 1) + message(FATAL_ERROR "BUCKYBALL_CHIP_PB is missing or ambiguous in ${BUCKYBALL_COMPILER_CACHE}") +endif() +list(GET BUCKYBALL_COMPILER_PB_LINES 0 BUCKYBALL_COMPILER_PB_LINE) +string(REGEX REPLACE "^BUCKYBALL_CHIP_PB:[^=]+=" "" + BUCKYBALL_COMPILER_PB "${BUCKYBALL_COMPILER_PB_LINE}") +get_filename_component(BUCKYBALL_COMPILER_PB "${BUCKYBALL_COMPILER_PB}" ABSOLUTE) +get_filename_component(BUCKYBALL_WORKLOAD_PB "${BUCKYBALL_CHIP_PB}" ABSOLUTE) +if(NOT BUCKYBALL_COMPILER_PB STREQUAL BUCKYBALL_WORKLOAD_PB) + message(FATAL_ERROR + "Compiler Chip.pb mismatch: compiler uses ${BUCKYBALL_COMPILER_PB}, workload uses ${BUCKYBALL_WORKLOAD_PB}") endif() option(BUCKYBALL_STABLE "Use stable LLVM Buckyball extensions" OFF) if(BUCKYBALL_STABLE) - set(BUCKYBALL_RISCV_MATTR "+xbbpebble,+xbbmeteor,+D") + set(BUCKYBALL_RISCV_MATTR "+xbbpebble,+xbuckyball,+D") set(BUCKYBALL_LOWER_BUCKYBALL "--lower-buckyball=stable=true") set(BUCKYBALL_LOWER_BUCKYBALL_RUSHB "--lower-buckyball=stable=true rushb=true") @@ -195,22 +410,51 @@ message(STATUS "Buckyball RISC-V mattr: ${BUCKYBALL_RISCV_MATTR}") set(ELF_CC ${RISCV_GNU_TOOLCHAIN}/bin/riscv64-unknown-elf-gcc) set(LINUX_CC ${RISCV_GNU_TOOLCHAIN}/bin/riscv64-unknown-linux-gnu-gcc) -set(BUILD_BIN_DIR ${WORKLOAD_DIR}/../build/workloads) -set(OUTPUT_BIN_DIR ${WORKLOAD_DIR}/../output/workloads) +set(BUILD_BIN_DIR ${CMAKE_BINARY_DIR}) +set(OUTPUT_BIN_DIR ${WORKLOAD_DIR}/../output/${BUCKYBALL_WORKLOAD_CHIP}/workloads) -if(NOT IS_ABSOLUTE "${Python3_EXECUTABLE}") - find_program(BUCKYBALL_PYTHON3_EXECUTABLE NAMES python3) - if(NOT BUCKYBALL_PYTHON3_EXECUTABLE) - message(FATAL_ERROR "python3 is required to build workloads") - endif() - set(Python3_EXECUTABLE "${BUCKYBALL_PYTHON3_EXECUTABLE}") -endif() #------------------------------------------------------------------------------- # add_subdirectory #------------------------------------------------------------------------------- -add_subdirectory(lib) +add_subdirectory(${WORKLOAD_LIB_DIR} ${CMAKE_CURRENT_BINARY_DIR}/lib) add_subdirectory(src) +add_custom_target(ctest-workloads-build + DEPENDS + base-ctests-build + tlb-ctests-build + tutorial-ctests-build + chip-ctests-build + COMMENT "Building CTest workloads" +) + +add_custom_target(mlirtest-workloads-build + DEPENDS + MLIRTest-all + chip-mlirtests-build + COMMENT "Building MLIRTest workloads" +) + +add_custom_target(sync-mlirtest-bin + COMMAND ${CMAKE_COMMAND} -E make_directory ${OUTPUT_BIN_DIR}/src + COMMAND rsync -a --delete --exclude=CMakeFiles --include="*/" + --include="MLIRTest/***" --include="balls/***" + --include="*/mlir_tests/***" --exclude="*" + ${BUILD_BIN_DIR}/src/ ${OUTPUT_BIN_DIR}/src/ + DEPENDS mlirtest-workloads-build + COMMENT "Syncing MLIRTest binaries to output directory" +) + +add_custom_target(sync-ctest-bin + COMMAND ${CMAKE_COMMAND} -E make_directory ${OUTPUT_BIN_DIR}/src/CTest + COMMAND rsync -a --delete --delete-excluded --exclude=CMakeFiles + --exclude="*/mlir_tests/**" --include="*/" + --include="*-baremetal" --include="*-linux" --exclude="*" + ${BUILD_BIN_DIR}/src/CTest/ ${OUTPUT_BIN_DIR}/src/CTest/ + DEPENDS ctest-workloads-build + COMMENT "Syncing CTest binaries to output directory" +) + #------------------------------------------------------------------------------- # setup directory #------------------------------------------------------------------------------- @@ -229,8 +473,8 @@ add_custom_target(sync-bin --exclude="CMakeFiles" --filter="P_src/CTest/rushB/" --filter="P_src/CTest/rushB/***" - --filter="P_src/OpTest/rushB/" - --filter="P_src/OpTest/rushB/***" + --filter="P_src/MLIRTest/rushB/" + --filter="P_src/MLIRTest/rushB/***" --include="**/chips/${BUCKYBALL_WORKLOAD_CHIP}/" --exclude="**/chips/*/" --include="*/" @@ -244,9 +488,10 @@ add_custom_target(sync-bin add_dependencies(sync-bin base-ctests-build - tutorial-build + tlb-ctests-build + tutorial-ctests-build chip-workloads-build - OpTest-all + MLIRTest-all embench-all coremark-all ) diff --git a/bb-tests/workloads/lib/bbhw/CMakeLists.txt b/bb-tests/workloads/lib/bbhw/CMakeLists.txt index 6cc197700..f2ddbb2f1 100644 --- a/bb-tests/workloads/lib/bbhw/CMakeLists.txt +++ b/bb-tests/workloads/lib/bbhw/CMakeLists.txt @@ -1,3 +1,3 @@ # bbhw: ISA macros (headers) + mem.c (bb_dma_touch). # Include path is WORKLOAD_LIB_DIR (workloads/lib), so #include works. -# mem.c is compiled into each ctest via BuckyballWorkloads.cmake. +# mem.c is compiled into each ctest via bb-tests/workloads/CMakeLists.txt. diff --git a/bb-tests/workloads/lib/bbhw/isa/00_fence.c b/bb-tests/workloads/lib/bbhw/isa/00_fence.c index 4ec9514ee..b7bf839c8 100644 --- a/bb-tests/workloads/lib/bbhw/isa/00_fence.c +++ b/bb-tests/workloads/lib/bbhw/isa/00_fence.c @@ -5,6 +5,13 @@ #define BB_FENCE_FUNC7 0 -#define bb_fence() BUCKYBALL_INSTRUCTION_R_R(0, 0, BB_FENCE_FUNC7) +#define bb_dma_cache_flush() asm volatile("fence.i" ::: "memory") + +#define bb_fence() \ + do { \ + BUCKYBALL_INSTRUCTION_R_R(0, 0, BB_FENCE_FUNC7); \ + asm volatile("fence rw, rw" ::: "memory"); \ + bb_dma_cache_flush(); \ + } while (0) #endif // _BB_FENCE_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/16_mvout.c b/bb-tests/workloads/lib/bbhw/isa/16_mvout.c index 88a75f908..7339a18b0 100644 --- a/bb-tests/workloads/lib/bbhw/isa/16_mvout.c +++ b/bb-tests/workloads/lib/bbhw/isa/16_mvout.c @@ -6,8 +6,16 @@ #define BB_MVOUT_FUNC7 16 -/* Touch dest pages before DMA store (Linux CoW). Do not memset: stride holes - * must keep prior CPU data. Span uses bank cols from bb_mem_alloc. */ +/* Linux needs private destination pages before a DMA store. */ +#if defined(__linux__) +#define BB_MVOUT_TOUCH(addr, depth, stride, bank) \ + bb_dma_touch_mvout((void *)(addr), (depth), (stride), (bank)) +#else +#define BB_MVOUT_TOUCH(addr, depth, stride, bank) \ + do { \ + } while (0) +#endif + #if defined(BUCKYBALL_RUSHB) #define bb_mvout(mem_addr, bank_id, depth, stride) \ do { \ @@ -15,8 +23,8 @@ uint32_t _bb_mo_bank = (uint32_t)(bank_id); \ uint64_t _bb_mo_depth = (uint64_t)(depth); \ uint64_t _bb_mo_stride = (uint64_t)(stride); \ - bb_dma_touch_mvout((void *)_bb_mo_addr, _bb_mo_depth, _bb_mo_stride, \ - _bb_mo_bank); \ + BB_MVOUT_TOUCH(_bb_mo_addr, _bb_mo_depth, _bb_mo_stride, _bb_mo_bank); \ + bb_dma_cache_flush(); \ rushb_mvout( \ (uint64_t)(BB_BANK0(_bb_mo_bank) | BB_ITER(_bb_mo_depth)), \ (uint64_t)(FIELD(_bb_mo_addr, 0, 38) | FIELD(_bb_mo_stride, 39, 57)), \ @@ -29,8 +37,8 @@ uint32_t _bb_mo_bank = (uint32_t)(bank_id); \ uint64_t _bb_mo_depth = (uint64_t)(depth); \ uint64_t _bb_mo_stride = (uint64_t)(stride); \ - bb_dma_touch_mvout((void *)_bb_mo_addr, _bb_mo_depth, _bb_mo_stride, \ - _bb_mo_bank); \ + BB_MVOUT_TOUCH(_bb_mo_addr, _bb_mo_depth, _bb_mo_stride, _bb_mo_bank); \ + bb_dma_cache_flush(); \ BUCKYBALL_INSTRUCTION_R_R( \ (BB_BANK0(_bb_mo_bank) | BB_ITER(_bb_mo_depth)), \ (FIELD(_bb_mo_addr, 0, 38) | FIELD(_bb_mo_stride, 39, 57)), \ diff --git a/bb-tests/workloads/lib/bbhw/isa/33_mvin.c b/bb-tests/workloads/lib/bbhw/isa/33_mvin.c index 23dfd912f..4c2b321b6 100644 --- a/bb-tests/workloads/lib/bbhw/isa/33_mvin.c +++ b/bb-tests/workloads/lib/bbhw/isa/33_mvin.c @@ -7,14 +7,20 @@ #if defined(BUCKYBALL_RUSHB) #define bb_mvin(mem_addr, bank_id, depth, stride) \ - rushb_mvin((uint64_t)(BB_BANK0(bank_id) | BB_ITER(depth)), \ - (uint64_t)(FIELD(mem_addr, 0, 38) | FIELD(stride, 39, 57)), \ - (const void *)(uintptr_t)(mem_addr)) + do { \ + bb_dma_cache_flush(); \ + rushb_mvin((uint64_t)(BB_BANK0(bank_id) | BB_ITER(depth)), \ + (uint64_t)(FIELD(mem_addr, 0, 38) | FIELD(stride, 39, 57)), \ + (const void *)(uintptr_t)(mem_addr)); \ + } while (0) #else #define bb_mvin(mem_addr, bank_id, depth, stride) \ - BUCKYBALL_INSTRUCTION_R_R((BB_BANK0(bank_id) | BB_ITER(depth)), \ - (FIELD(mem_addr, 0, 38) | FIELD(stride, 39, 57)), \ - BB_MVIN_FUNC7) + do { \ + bb_dma_cache_flush(); \ + BUCKYBALL_INSTRUCTION_R_R( \ + (BB_BANK0(bank_id) | BB_ITER(depth)), \ + (FIELD(mem_addr, 0, 38) | FIELD(stride, 39, 57)), BB_MVIN_FUNC7); \ + } while (0) #endif #endif // _BB_MVIN_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/34_mmio_set.c b/bb-tests/workloads/lib/bbhw/isa/34_mmio_set.c deleted file mode 100644 index 880ef7192..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/34_mmio_set.c +++ /dev/null @@ -1,27 +0,0 @@ -#ifndef _BB_MMIO_SET_H_ -#define _BB_MMIO_SET_H_ - -#include "isa.h" - -#define BB_MMIO_SET_FUNC7 34 - -// mmio_set: bind a main bank to an MMIO region (or dealloc when size_rows=0). -// main_bank : main SRAM bank id (BANK0, dependency) -// mmio_addr : byte address into MMIO space (16-bit) -// size_rows : region size in 128-bit rows (8-bit, 0 = dealloc) -// -// rs1 (dependencies): -// [9:0] = main_bank (BANK0) -// [63:30] = 0 (no iter) -// -// rs2 (logic): -// [15:0] = mmio_addr -// [23:16] = size_rows -// -// funct7 = 0x22 (34 decimal), enable=010 (1wr, similar to mset) -#define bb_mmio_set(main_bank, mmio_addr, size_rows) \ - BUCKYBALL_INSTRUCTION_R_R( \ - BB_BANK0(main_bank), \ - (FIELD(mmio_addr, 0, 15) | FIELD(size_rows, 16, 23)), BB_MMIO_SET_FUNC7) - -#endif // _BB_MMIO_SET_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/35_mvin_mmio.c b/bb-tests/workloads/lib/bbhw/isa/35_mvin_mmio.c index 90d33c1ae..633669e14 100644 --- a/bb-tests/workloads/lib/bbhw/isa/35_mvin_mmio.c +++ b/bb-tests/workloads/lib/bbhw/isa/35_mvin_mmio.c @@ -25,9 +25,12 @@ // // funct7 = 0x23 (35 decimal) #define bb_mvin_mmio(dram_addr, mmio_addr, row, col) \ - BUCKYBALL_INSTRUCTION_R_R(BB_ITER(row), \ - (FIELD(dram_addr, 0, 38) | \ - FIELD(mmio_addr, 39, 55) | FIELD(col, 56, 63)), \ - BB_MVIN_MMIO_FUNC7) + do { \ + bb_dma_cache_flush(); \ + BUCKYBALL_INSTRUCTION_R_R(BB_ITER(row), \ + (FIELD(dram_addr, 0, 38) | \ + FIELD(mmio_addr, 39, 55) | FIELD(col, 56, 63)), \ + BB_MVIN_MMIO_FUNC7); \ + } while (0) #endif // _BB_MVIN_MMIO_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/48_im2col.c b/bb-tests/workloads/lib/bbhw/isa/48_im2col.c deleted file mode 100644 index 6a29455ba..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/48_im2col.c +++ /dev/null @@ -1,18 +0,0 @@ -#ifndef _BB_IM2COL_H_ -#define _BB_IM2COL_H_ - -#include "isa.h" - -#define BB_IM2COL_FUNC7 48 - -// iter: input height and width (the input is an iter x iter square) -// ksize: square kernel height and width -// stride: row and column stride -// padding: zero-padding on every input edge -#define bb_im2col(op1_bank_id, wr_bank_id, iter, ksize, stride, padding) \ - BUCKYBALL_INSTRUCTION_R_R( \ - (BB_BANK0(op1_bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ - (FIELD(ksize, 0, 7) | FIELD(stride, 8, 15) | FIELD(padding, 16, 23)), \ - BB_IM2COL_FUNC7) - -#endif // _BB_IM2COL_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/50_relu.c b/bb-tests/workloads/lib/bbhw/isa/50_relu.c deleted file mode 100644 index 4bb12ec62..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/50_relu.c +++ /dev/null @@ -1,13 +0,0 @@ -#ifndef _BB_RELU_H_ -#define _BB_RELU_H_ - -#include "isa.h" - -#define BB_RELU_FUNC7 50 - -#define bb_relu(bank_id, wr_bank_id, iter) \ - BUCKYBALL_INSTRUCTION_R_R( \ - (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), 0, \ - BB_RELU_FUNC7) - -#endif // _BB_RELU_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/51_fp2int.c b/bb-tests/workloads/lib/bbhw/isa/51_fp2int.c deleted file mode 100644 index 060ecbcd0..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/51_fp2int.c +++ /dev/null @@ -1,17 +0,0 @@ -#ifndef _BB_FP2INT_H_ -#define _BB_FP2INT_H_ - -#include "isa.h" - -#define BB_FP2INT_FUNC7 51 - -// bb_fp2int(bank_id, wr_bank_id, iter, scale_fp32) -// scale_fp32 is a 32-bit FP32 value passed as uint32_t bit pattern -// Encoding: rs1 = banks | iter -// rs2 = scale_fp32 -#define bb_fp2int(bank_id, wr_bank_id, iter, scale_fp32) \ - BUCKYBALL_INSTRUCTION_R_R( \ - (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ - (FIELD((uint64_t)(scale_fp32), 0, 31)), BB_FP2INT_FUNC7) - -#endif // _BB_FP2INT_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/52_int2fp.c b/bb-tests/workloads/lib/bbhw/isa/52_int2fp.c deleted file mode 100644 index dc8adc55b..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/52_int2fp.c +++ /dev/null @@ -1,34 +0,0 @@ -#ifndef _BB_INT2FP_H_ -#define _BB_INT2FP_H_ - -#include "isa.h" - -#define BB_INT2FP_FUNC7 52 - -#define BB_INT_OUTPUT_FP32 0 -#define BB_INT_OUTPUT_INT8 1 -#define BB_INT_OUTPUT_MODE(mode) FIELD((uint64_t)(mode), 32, 33) - -// Convert integer data using an explicit output format. -// Encoding: rs1 = banks | iter -// rs2[31:0] = scale_fp32 -// rs2[33:32] = output_mode -#define bb_int_convert(bank_id, wr_bank_id, iter, output_mode, scale_fp32) \ - BUCKYBALL_INSTRUCTION_R_R( \ - (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ - (FIELD((uint64_t)(scale_fp32), 0, 31) | \ - BB_INT_OUTPUT_MODE(output_mode)), \ - BB_INT2FP_FUNC7) - -// bb_int2fp(bank_id, wr_bank_id, iter, scale_fp32) -// scale_fp32 is a 32-bit FP32 value passed as uint32_t bit pattern -// Encoding: rs1 = banks | iter -// rs2 = FIELD(scale_fp32, 0, 31) -#define bb_int2fp(bank_id, wr_bank_id, iter, scale_fp32) \ - bb_int_convert(bank_id, wr_bank_id, iter, BB_INT_OUTPUT_FP32, scale_fp32) - -// Requantize four INT32 bank groups into one packed INT8 bank group. -#define bb_int32_to_int8(bank_id, wr_bank_id, iter, scale_fp32) \ - bb_int_convert(bank_id, wr_bank_id, iter, BB_INT_OUTPUT_INT8, scale_fp32) - -#endif // _BB_INT2FP_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/64_mul_warp16.c b/bb-tests/workloads/lib/bbhw/isa/64_mul_warp16.c deleted file mode 100644 index 2f7327420..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/64_mul_warp16.c +++ /dev/null @@ -1,13 +0,0 @@ -#ifndef _BB_MUL_WARP16_H_ -#define _BB_MUL_WARP16_H_ - -#include "isa.h" - -#define BB_MUL_WARP16_FUNC7 64 - -#define bb_mul_warp16(op1_bank_id, op2_bank_id, wr_bank_id, iter, mode) \ - BUCKYBALL_INSTRUCTION_R_R((BB_BANK0(op1_bank_id) | BB_BANK1(op2_bank_id) | \ - BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ - (FIELD(mode, 0, 63)), BB_MUL_WARP16_FUNC7) - -#endif // _BB_MUL_WARP16_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/65_matrix.c b/bb-tests/workloads/lib/bbhw/isa/65_matrix.c deleted file mode 100644 index 533bb790b..000000000 --- a/bb-tests/workloads/lib/bbhw/isa/65_matrix.c +++ /dev/null @@ -1,35 +0,0 @@ -#ifndef _BB_MATRIX_H_ -#define _BB_MATRIX_H_ - -#include "isa.h" - -#define BB_MATRIX_FUNC7 65 -#define BB_MATRIX_MODE_OS 0ULL -#define BB_MATRIX_MODE_WS 1ULL - -#define BB_MATRIX_MNK_CONFIG(m, n, k, mode) \ - (FIELD((m), 0, 11) | FIELD((n), 12, 23) | FIELD((k), 24, 35) | \ - FIELD((mode), 36, 36)) - -#define BB_MATRIX_RS1(op1_bank_id, op2_bank_id, wr_bank_id) \ - (BB_BANK0(op1_bank_id) | BB_BANK1(op2_bank_id) | BB_BANK2(wr_bank_id)) - -#define bb_matrix_mnk_mode(op1_bank_id, op2_bank_id, wr_bank_id, m, n, k, \ - mode) \ - BUCKYBALL_INSTRUCTION_R_R( \ - BB_MATRIX_RS1(op1_bank_id, op2_bank_id, wr_bank_id), \ - BB_MATRIX_MNK_CONFIG(m, n, k, mode), BB_MATRIX_FUNC7) - -#define bb_matrix_mnk(op1_bank_id, op2_bank_id, wr_bank_id, m, n, k) \ - bb_matrix_mnk_mode(op1_bank_id, op2_bank_id, wr_bank_id, m, n, k, \ - BB_MATRIX_MODE_OS) - -#define bb_matrix_os(op1_bank_id, op2_bank_id, wr_bank_id, dim) \ - bb_matrix_mnk_mode(op1_bank_id, op2_bank_id, wr_bank_id, dim, dim, dim, \ - BB_MATRIX_MODE_OS) - -#define bb_matrix_ws(op1_bank_id, op2_bank_id, wr_bank_id, dim) \ - bb_matrix_mnk_mode(op1_bank_id, op2_bank_id, wr_bank_id, dim, dim, dim, \ - BB_MATRIX_MODE_WS) - -#endif // _BB_MATRIX_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/bb_func7.h b/bb-tests/workloads/lib/bbhw/isa/bb_func7.h new file mode 100644 index 000000000..5b712fd4b --- /dev/null +++ b/bb-tests/workloads/lib/bbhw/isa/bb_func7.h @@ -0,0 +1,8 @@ +#ifndef BB_FUNC7_H +#define BB_FUNC7_H + +#include "ballISA.h" + +#define BB_FUNC7(mne) BB_FUNC7_##mne + +#endif diff --git a/bb-tests/workloads/lib/bbhw/isa/isa.h b/bb-tests/workloads/lib/bbhw/isa/isa.h index e8e23072d..d9e4946d5 100644 --- a/bb-tests/workloads/lib/bbhw/isa/isa.h +++ b/bb-tests/workloads/lib/bbhw/isa/isa.h @@ -51,29 +51,13 @@ typedef int32_t result_t; : "memory") #endif -// Include all instruction definitions +// Base (mem/frontend) instruction definitions only. +// Ball-specific ISA macros live under examples/balls//workloads/isa/. #include "00_fence.c" #include "01_barrier.c" -#include "02_gemmini_config.c" -#include "03_gemmini_flush.c" -#include "04_bdb_counter.c" #include "16_mvout.c" #include "32_mset.c" #include "33_mvin.c" -#include "34_mmio_set.c" #include "35_mvin_mmio.c" -#include "48_im2col.c" -#include "49_transpose.c" -#include "50_relu.c" -#include "51_fp2int.c" -#include "52_int2fp.c" -#include "53_gemmini_preload.c" -#include "55_mxfp2int.c" -#include "64_mul_warp16.c" -#include "65_matrix.c" -#include "66_gemmini_compute_preloaded.c" -#include "67_gemmini_compute_accumulated.c" -#include "80_gemmini_loop_ws.c" -#include "96_gemmini_loop_conv_ws.c" #endif // BUCKYBALL_ISA_H diff --git a/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.c b/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.c index 0c0e07811..2338d172f 100644 --- a/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.c +++ b/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.c @@ -1,28 +1,28 @@ #include "mmio_allocator.h" void mmio_allocator_init(mmio_allocator_t *alloc) { - for (int i = 0; i < MMIO_BANK_NUM; i++) { + for (int i = 0; i < MMIO_LOGICAL_ROWS; i++) { alloc->bitmap[i] = 0; } } uint16_t mmio_allocator_alloc(mmio_allocator_t *alloc, uint16_t size_rows) { - if (size_rows == 0 || size_rows > MMIO_BANK_ENTRIES) { + if (size_rows == 0 || size_rows > MMIO_LOGICAL_ROWS) { return (uint16_t)-1; } - for (int b = 0; b < MMIO_BANK_NUM; b++) { - for (int start = 0; start + size_rows <= MMIO_BANK_ENTRIES; start++) { - uint64_t mask; - if (size_rows == 64) { - mask = ~(uint64_t)0; - } else { - mask = (((uint64_t)1 << size_rows) - 1) << start; + for (int start = 0; start + size_rows <= MMIO_LOGICAL_ROWS; start++) { + int free = 1; + for (int row = 0; row < size_rows; row++) { + if (alloc->bitmap[start + row]) { + free = 0; + break; } - if ((alloc->bitmap[b] & mask) == 0) { - alloc->bitmap[b] |= mask; - return (uint16_t)(MMIO_BANK_BASE(b) + - start * (MMIO_BANK_WIDTH_BITS / 8)); + } + if (free) { + for (int row = 0; row < size_rows; row++) { + alloc->bitmap[start + row] = 1; } + return (uint16_t)(start * MMIO_LOGICAL_ROW_BYTES); } } return (uint16_t)-1; @@ -32,14 +32,11 @@ void mmio_allocator_free(mmio_allocator_t *alloc, uint16_t addr, uint16_t size_rows) { if (size_rows == 0) return; - int bytes_per_row = MMIO_BANK_WIDTH_BITS / 8; - int b = addr / MMIO_BANK_BYTES; - int start = (addr % MMIO_BANK_BYTES) / bytes_per_row; - uint64_t mask; - if (size_rows == 64) { - mask = ~(uint64_t)0; - } else { - mask = (((uint64_t)1 << size_rows) - 1) << start; + int start = addr / MMIO_LOGICAL_ROW_BYTES; + if (start + size_rows > MMIO_LOGICAL_ROWS) { + return; + } + for (int row = 0; row < size_rows; row++) { + alloc->bitmap[start + row] = 0; } - alloc->bitmap[b] &= ~mask; } diff --git a/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.h b/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.h index e395354b7..3cf5e8b4f 100644 --- a/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.h +++ b/bb-tests/workloads/lib/bbhw/mmio/mmio_allocator.h @@ -4,21 +4,19 @@ #include #include -// Default MMIO subsystem layout (must mirror MMIO config in hardware -// default.json): -// 16 banks x 64 entries x 128 bit = 16 KB total -#define MMIO_BANK_NUM 16 -#define MMIO_BANK_ENTRIES 64 -#define MMIO_BANK_WIDTH_BITS 128 -#define MMIO_BANK_BYTES \ - ((MMIO_BANK_ENTRIES) * ((MMIO_BANK_WIDTH_BITS) / 8)) // 1024 -#define MMIO_TOTAL_BYTES ((MMIO_BANK_NUM) * (MMIO_BANK_BYTES)) // 16384 -#define MMIO_BANK_BASE(i) ((i) * (MMIO_BANK_BYTES)) +// Pebble MMIO is one 5 KiB byte-addressed space striped over five identical +// 8-bit x 1024 banks. A logical MMIO row is one normal 128-bit DMA beat. +#define MMIO_BANK_NUM 5 +#define MMIO_BANK_ENTRIES 1024 +#define MMIO_BANK_WIDTH_BITS 8 +#define MMIO_BANK_BYTES ((MMIO_BANK_ENTRIES) * ((MMIO_BANK_WIDTH_BITS) / 8)) +#define MMIO_TOTAL_BYTES ((MMIO_BANK_NUM) * (MMIO_BANK_BYTES)) +#define MMIO_LOGICAL_ROW_BYTES 16 +#define MMIO_LOGICAL_ROWS (MMIO_TOTAL_BYTES / MMIO_LOGICAL_ROW_BYTES) -// Bitmap-based allocator: one bit per row in each bank. -// 16 banks x 64 rows = 1024 bits = 16 uint64_t words. +// Bitmap-based allocator over the unified logical address space. typedef struct { - uint64_t bitmap[MMIO_BANK_NUM]; // bitmap[i] bit j = row j of bank i used + uint8_t bitmap[MMIO_LOGICAL_ROWS]; } mmio_allocator_t; // Initialize allocator (all rows free). diff --git a/examples/chips/toy/workloads/common/buckyball.c b/bb-tests/workloads/lib/buckyball.c similarity index 100% rename from examples/chips/toy/workloads/common/buckyball.c rename to bb-tests/workloads/lib/buckyball.c diff --git a/examples/chips/toy/workloads/common/buckyball.h b/bb-tests/workloads/lib/buckyball.h similarity index 100% rename from examples/chips/toy/workloads/common/buckyball.h rename to bb-tests/workloads/lib/buckyball.h diff --git a/bb-tests/workloads/lib/kernel b/bb-tests/workloads/lib/kernel index 6ec74f33a..78c4b7722 160000 --- a/bb-tests/workloads/lib/kernel +++ b/bb-tests/workloads/lib/kernel @@ -1 +1 @@ -Subproject commit 6ec74f33abb5406c25510d6f93057e2e010ade03 +Subproject commit 78c4b772206fa419087229eded89110c84320a5f diff --git a/bb-tests/workloads/scripts/build.py b/bb-tests/workloads/scripts/build.py new file mode 100644 index 000000000..dd64fdc62 --- /dev/null +++ b/bb-tests/workloads/scripts/build.py @@ -0,0 +1,339 @@ +from __future__ import annotations + +import argparse +import os +import re +import shlex +import shutil +import subprocess +from pathlib import Path + +_MODELS: dict[str, tuple[str, str]] = { + "lenet": ("lenet", "buddy-buckyball-lenet-run"), + "mobilenet": ("mobilenetv3", "buddy-buckyball-mobilenetv3-run"), + "resnet": ("resnet18", "buddy-buckyball-resnet-run"), + "yolo": ("yolo26", "buddy-buckyball-yolo26-run"), + "bert": ("bert", "buddy-buckyball-bert-run"), + "qwen3": ("qwen3", "buddy-buckyball-qwen3-run"), + "gemma4": ("gemma4", "buddy-buckyball-gemma4-run"), + "deepseekr1": ("deepseekr1", "buddy-buckyball-deepseekr1-run"), + "llama2": ("llama2", "buddy-buckyball-llama2-run"), + "stable-diffusion": ("stablediffusion", "buddy-buckyball-stable-diffusion-run"), + "whisper": ("whisper", "buddy-buckyball-whisper-run"), + "buddynext": ("buddynext", "buddy-buckyball-buddynext-all-run"), +} + +_RUSHB: dict[str, dict[str, str]] = { + "lenet": { + "bemu": "buddy-buckyball-lenet-rushB-bemu-run", + "verilator": "buddy-buckyball-lenet-rushB-verilator-run", + }, + "mobilenet": { + "bemu": "buddy-buckyball-mobilenetv3-rushB-bemu-run", + "verilator": "buddy-buckyball-mobilenetv3-rushB-verilator-run", + }, + "resnet": { + "bemu": "buddy-buckyball-resnet-rushB-bemu-run", + "verilator": "buddy-buckyball-resnet-rushB-verilator-run", + }, + "yolo": { + "bemu": "buddy-buckyball-yolo26-rushB-bemu-run", + "verilator": "buddy-buckyball-yolo26-rushB-verilator-run", + }, + "bert": { + "bemu": "buddy-buckyball-bert-rushB-bemu-run", + "verilator": "buddy-buckyball-bert-rushB-verilator-run", + }, + "qwen3": { + "bemu": "buddy-buckyball-qwen3-rushB-bemu-run", + "verilator": "buddy-buckyball-qwen3-rushB-verilator-run", + }, + "gemma4": { + "bemu": "buddy-buckyball-gemma4-rushB-bemu-run", + "verilator": "buddy-buckyball-gemma4-rushB-verilator-run", + }, + "deepseekr1": { + "bemu": "buddy-buckyball-deepseekr1-rushB-bemu-run", + "verilator": "buddy-buckyball-deepseekr1-rushB-verilator-run", + }, + "llama2": { + "bemu": "buddy-buckyball-llama2-rushB-bemu-run", + "verilator": "buddy-buckyball-llama2-rushB-verilator-run", + }, + "stable-diffusion": { + "bemu": "buddy-buckyball-stable-diffusion-rushB-bemu-run", + "verilator": "buddy-buckyball-stable-diffusion-rushB-verilator-run", + }, + "whisper": { + "bemu": "buddy-buckyball-whisper-rushB-bemu-run", + "verilator": "buddy-buckyball-whisper-rushB-verilator-run", + }, + "buddynext": { + "bemu": "buddy-buckyball-buddynext-rushB-bemu-run", + "verilator": "buddy-buckyball-buddynext-rushB-verilator-run", + }, +} + + +def _repo(raw: str | Path) -> Path: + root = Path(raw).resolve() + if not root.is_dir(): + raise ValueError(f"repo does not exist: {root}") + return root + + +def _run( + cmd: list[str], + *, + cwd: Path, + prefix: str, + env: dict | None = None, + logger: object | None = None, + task_scope: str | None = None, +) -> None: + if logger is None: + result = subprocess.run(cmd, cwd=cwd, env=env) + else: + from utils.stream_run import stream_run_logger + + result = stream_run_logger( + cmd=shlex.join(cmd), + logger=logger, + cwd=str(cwd), + stdout_prefix=prefix, + stderr_prefix=prefix, + task_scope=task_scope, + env=env, + ) + if result.returncode != 0: + raise RuntimeError(f"command failed ({result.returncode}): {' '.join(cmd)}") + + +def _cmake_defs(repo: Path, chip: str) -> dict[str, str]: + path = ( + repo + / "examples" + / "chips" + / chip + / "configs" + / "generated" + / "workload" + / "cmake.defs" + ) + if not path.is_file(): + raise RuntimeError(f"missing {path}; run bbdev config --install") + values: dict[str, str] = {} + for line in path.read_text(encoding="utf-8").splitlines(): + key, sep, value = line.partition("=") + if not sep: + raise RuntimeError(f"bad cmake.defs line in {path}: {line!r}") + if key in values: + raise RuntimeError(f"duplicate cmake.defs key {key} in {path}") + values[key] = value + required = ( + "BUCKYBALL_WORKLOAD_CHIP", + "BUCKYBALL_CHIP_PB", + ) + missing = [k for k in required if k not in values] + if missing: + raise RuntimeError(f"{path} missing {missing}") + return values + + +_RUSHB_DEFS = ( + "BUCKYBALL_CARGO_TARGET_DIR", + "BUCKYBALL_RUSHB_BEMU_MANIFEST", + "BUCKYBALL_RUSHB_BEMU_LIBRARY", + "BUCKYBALL_RUSHB_VERILATOR_LIBRARY", +) + + +def _ninja_target( + model: str, + rushb: str | None, + ctest: bool, + mlirtest: bool, +) -> tuple[str, str]: + if ctest: + return "", "sync-ctest-bin" + if mlirtest: + return "", "sync-mlirtest-bin" + + cmake_model = "" + ninja_arg = "" + if model: + if model not in _MODELS: + raise ValueError(f"unknown workload model: {model}") + cmake_model, ninja_arg = _MODELS[model] + if rushb: + mapped = _RUSHB.get(model, {}).get(rushb) + if not mapped: + raise ValueError( + f"no rushB target for model {model!r} backend {rushb!r}" + ) + ninja_arg = mapped + elif rushb: + ninja_arg = f"rushB-{rushb}-workloads-build" + return cmake_model, ninja_arg + + +def _require_riscv() -> Path: + raw = os.environ.get("RISCV", "") + if not raw: + raise RuntimeError("RISCV is unset; enter nix develop") + root = Path(raw) + if not root.is_dir(): + raise RuntimeError(f"RISCV is not a directory: {root}") + return root + + +def _workload_src(repo: Path) -> Path: + src = repo / "bb-tests" / "workloads" + cmake = src / "CMakeLists.txt" + if not cmake.is_file(): + raise RuntimeError(f"missing {cmake}") + return src + + +def _workload_build_dir(repo: Path, chip: str) -> Path: + return repo / "bb-tests" / "workloads" / "build" / chip + + +def build_workload( + repo: str | Path, + chip: str, + *, + model: str = "", + rushb: str | None = None, + ctest: bool = False, + mlirtest: bool = False, + stable: bool = False, + logger: object | None = None, + task_scope: str | None = None, +) -> None: + if not re.fullmatch(r"[A-Za-z0-9_-]+", chip): + raise ValueError(f"invalid chip: {chip}") + if rushb is not None and rushb not in {"bemu", "verilator"}: + raise ValueError(f"rushB must be bemu|verilator, got {rushb!r}") + if ctest and mlirtest: + raise ValueError("--ctest and --mlirtest cannot be used together") + if (ctest or mlirtest) and model: + raise ValueError("--ctest and --mlirtest cannot be used with --model") + if (ctest or mlirtest) and rushb: + raise ValueError("--ctest and --mlirtest cannot be used with --rushB") + + root = _repo(repo) + defs = _cmake_defs(root, chip) + if rushb: + missing = [k for k in _RUSHB_DEFS if k not in defs] + if missing: + raise RuntimeError( + f"missing {missing} in cmake.defs for rushB; run bbdev config --install" + ) + compiler_build = root / "compiler" / "thirdparty" / "buddy-mlir" / "build" / chip + python = shutil.which("python3") + if not python: + raise RuntimeError("python3 not in PATH; enter nix develop") + + if rushb == "bemu": + env = os.environ.copy() + env["CARGO_TARGET_DIR"] = defs["BUCKYBALL_CARGO_TARGET_DIR"] + _run( + [ + "cargo", + "build", + "--release", + "--manifest-path", + defs["BUCKYBALL_RUSHB_BEMU_MANIFEST"], + "--lib", + ], + cwd=root, + env=env, + prefix="workload cargo", + logger=logger, + task_scope=task_scope, + ) + + riscv = _require_riscv() + linux_cc = riscv / "bin" / "riscv64-unknown-linux-gnu-gcc" + linux_cxx = riscv / "bin" / "riscv64-unknown-linux-gnu-g++" + if not linux_cc.is_file() or not linux_cxx.is_file(): + raise RuntimeError(f"missing RISC-V linux toolchain under {riscv / 'bin'}") + + src = _workload_src(root) + build = _workload_build_dir(root, chip) + cmake_model, ninja_arg = _ninja_target(model.lower(), rushb, ctest, mlirtest) + env = os.environ.copy() + env["PATH"] = f"{riscv / 'bin'}:{env.get('PATH', '')}" + env["RISCV"] = str(riscv) + env["BUDDY_MLIR_BUILD_DIR"] = str(compiler_build) + env["CC"] = str(linux_cc) + env["CXX"] = str(linux_cxx) + + build.mkdir(parents=True, exist_ok=True) + + cmake_args = [ + "cmake", + "-G", + "Ninja", + "-S", + str(src), + "-B", + str(build), + f"-DBUCKYBALL_STABLE={'ON' if stable else 'OFF'}", + f"-DPython3_EXECUTABLE={python}", + f"-DCMAKE_C_COMPILER={linux_cc}", + f"-DCMAKE_CXX_COMPILER={linux_cxx}", + ] + for key, value in defs.items(): + if key in _RUSHB_DEFS and not rushb: + continue + cmake_args.append(f"-D{key}={value}") + if cmake_model: + cmake_args.extend(["-DMODEL=" + cmake_model, "-DARCH=buckyball"]) + + _run( + cmake_args, + cwd=root, + env=env, + prefix="workload configure", + logger=logger, + task_scope=task_scope, + ) + ninja = ["ninja", "-C", str(build), f"-j{os.cpu_count() or 1}"] + if ninja_arg: + ninja.append(ninja_arg) + _run( + ninja, + cwd=root, + env=env, + prefix="workload build", + logger=logger, + task_scope=task_scope, + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Build chip workloads") + parser.add_argument("--repo", type=Path, required=True) + parser.add_argument("--chip", required=True) + parser.add_argument("--model", default="") + parser.add_argument("--rushb", choices=("bemu", "verilator")) + scope = parser.add_mutually_exclusive_group() + scope.add_argument("--ctest", action="store_true") + scope.add_argument("--mlirtest", action="store_true") + parser.add_argument("--stable", action="store_true") + args = parser.parse_args() + build_workload( + args.repo, + args.chip, + model=args.model, + rushb=args.rushb, + ctest=args.ctest, + mlirtest=args.mlirtest, + stable=args.stable, + ) + + +if __name__ == "__main__": + main() diff --git a/bb-tests/workloads/src/CMakeLists.txt b/bb-tests/workloads/src/CMakeLists.txt index 2f1bcc2f2..9baca97f5 100644 --- a/bb-tests/workloads/src/CMakeLists.txt +++ b/bb-tests/workloads/src/CMakeLists.txt @@ -1,11 +1,9 @@ -set(CTEST_WORKLOAD_DIR ${WORKLOAD_SRC_DIR}/CTest) get_filename_component(REPO_ROOT ${WORKLOAD_DIR}/../.. ABSOLUTE) -set(TOY_WORKLOAD_COMMON_DIR ${REPO_ROOT}/examples/chips/toy/workloads/common) + +set(CTEST_WORKLOAD_DIR ${WORKLOAD_SRC_DIR}/CTest) +set(MLIRTEST_WORKLOAD_DIR ${WORKLOAD_SRC_DIR}/MLIRTest) set(MODELTEST_WORKLOAD_DIR ${WORKLOAD_SRC_DIR}/ModelTest) -set(OPTEST_WORKLOAD_DIR ${WORKLOAD_SRC_DIR}/OpTest) -set(TUTORIAL_WORKLOAD_DIR ${WORKLOAD_SRC_DIR}/tutorial) +include(${MLIRTEST_WORKLOAD_DIR}/CMakeLists.txt) add_subdirectory(CTest) add_subdirectory(ModelTest) -add_subdirectory(OpTest) -add_subdirectory(tutorial) diff --git a/bb-tests/workloads/src/CTest/CMakeLists.txt b/bb-tests/workloads/src/CTest/CMakeLists.txt index 31d75aae2..b2008f983 100644 --- a/bb-tests/workloads/src/CTest/CMakeLists.txt +++ b/bb-tests/workloads/src/CTest/CMakeLists.txt @@ -1,17 +1,215 @@ -get_filename_component(REPO_ROOT ${WORKLOAD_DIR}/../.. ABSOLUTE) +# =============================================================== +# Generic functions to help add CTest workloads +# =============================================================== +set(CTEST_MAX_LINES 100) +set(BBHW_MEM_C ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.c) +set(BUCKYBALL_CTEST_C_FLAGS + -g -fno-common -O2 -static -march=rv64gc -mcmodel=medany + -fno-builtin-printf -specs=nano.specs -specs=nosys.specs -nostartfiles + -Wl,-T,${BBSIM_LD} +) +file(GLOB BUCKYBALL_BALL_WORKLOAD_DIRS CONFIGURE_DEPENDS + "${BUCKYBALL_REPO_ROOT}/examples/balls/*/workloads") +file(GLOB BUCKYBALL_BALL_ISA_DEPS CONFIGURE_DEPENDS + "${BUCKYBALL_REPO_ROOT}/examples/balls/*/workloads/isa/*") +function(buckyball_enforce_ctest_line_limit SOURCE_FILE) + if(IS_ABSOLUTE "${SOURCE_FILE}") + set(_src "${SOURCE_FILE}") + else() + set(_src "${CMAKE_CURRENT_SOURCE_DIR}/${SOURCE_FILE}") + endif() + if(NOT EXISTS "${_src}") + message(FATAL_ERROR "ctest source does not exist: ${_src}") + endif() + + file(READ "${_src}" _content) + if(_content STREQUAL "") + set(_n 0) + else() + string(REGEX MATCHALL "\n" _newlines "${_content}") + list(LENGTH _newlines _n) + string(REGEX MATCH "[^\n]$" _no_trailing_nl "${_content}") + if(_no_trailing_nl) + math(EXPR _n "${_n} + 1") + endif() + endif() + + if(_n GREATER CTEST_MAX_LINES) + message(FATAL_ERROR + "ctest exceeds ${CTEST_MAX_LINES} lines: ${_src} (${_n} lines). " + "Split it into smaller focused ctests. " + "Do not evade this check by moving functional code into headers.") + endif() +endfunction() + +function(buckyball_ctest_deps OUT_DEPS SOURCE_DIR SOURCE_FILE) + if(NOT DEFINED BUCKYBALL_CTEST_TARGET OR BUCKYBALL_CTEST_TARGET STREQUAL "") + message(FATAL_ERROR "BUCKYBALL_CTEST_TARGET is required for C tests") + endif() + list(FIND BUCKYBALL_TARGETS "${BUCKYBALL_CTEST_TARGET}" _buckyball_target_index) + if(_buckyball_target_index EQUAL -1) + message(FATAL_ERROR "unknown C test target: ${BUCKYBALL_CTEST_TARGET}") + endif() + file(GLOB BUCKYBALL_ISA_DEPS CONFIGURE_DEPENDS + ${WORKLOAD_LIB_DIR}/bbhw/isa/*) + set(DEPS + ${SOURCE_DIR}/${SOURCE_FILE} + ${WORKLOAD_LIB_DIR}/buckyball.c + ${WORKLOAD_LIB_DIR}/buckyball.h + ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET}/ballISA.h + ${BBHW_MEM_C} + ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.h + ${BUCKYBALL_CHIP_COMMON_SOURCES} + ${BUCKYBALL_CHIP_COMMON_HEADERS} + ${BUCKYBALL_ISA_DEPS} + ${BUCKYBALL_BALL_ISA_DEPS}) + file(GLOB _LOCAL_HDRS CONFIGURE_DEPENDS ${SOURCE_DIR}/*.h) + list(APPEND DEPS ${_LOCAL_HDRS}) + + set(${OUT_DEPS} ${DEPS} PARENT_SCOPE) +endfunction() + +function(add_buckyball_linux_ctest TEST_NAME SOURCE_DIR SOURCE_FILE) + set(EXECUTABLE "${TEST_NAME}-linux") + + add_executable(${EXECUTABLE} + ${SOURCE_DIR}/${SOURCE_FILE} + ${WORKLOAD_LIB_DIR}/buckyball.c + ${BBHW_MEM_C} + ${BUCKYBALL_CHIP_COMMON_SOURCES}) + target_include_directories(${EXECUTABLE} PRIVATE + ${WORKLOAD_LIB_DIR} + ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET} + ${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS} + ${BUCKYBALL_BALL_WORKLOAD_DIRS} + ${SOURCE_DIR}) + set_target_properties(${EXECUTABLE} PROPERTIES LINKER_LANGUAGE C) + + add_custom_target(${TEST_NAME}-linux-build + DEPENDS ${EXECUTABLE}) +endfunction() + +function(add_buckyball_baremetal_ctest TEST_NAME SOURCE_DIR SOURCE_FILE) + set(EXECUTABLE "${TEST_NAME}-baremetal") + buckyball_ctest_deps(TEST_DEPS ${SOURCE_DIR} ${SOURCE_FILE}) + + set(BUCKYBALL_CHIP_COMMON_INCLUDE_FLAGS) + foreach(CHIP_COMMON_INCLUDE_DIR ${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS}) + list(APPEND BUCKYBALL_CHIP_COMMON_INCLUDE_FLAGS -I${CHIP_COMMON_INCLUDE_DIR}) + endforeach() + set(BALL_ISA_INCLUDE_FLAGS) + foreach(BALL_WORKLOAD_DIR ${BUCKYBALL_BALL_WORKLOAD_DIRS}) + list(APPEND BALL_ISA_INCLUDE_FLAGS -I${BALL_WORKLOAD_DIR}) + endforeach() + + add_custom_command( + OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} + COMMAND ${ELF_CC} ${BUCKYBALL_CTEST_C_FLAGS} + -o ${EXECUTABLE} + ${BBSW_BAREMETAL_DIR}/crt0.S + ${WORKLOAD_LIB_DIR}/buckyball.c + ${BBHW_MEM_C} + ${BUCKYBALL_CHIP_COMMON_SOURCES} + ${SOURCE_DIR}/${SOURCE_FILE} + -I${WORKLOAD_LIB_DIR} + -I${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET} + ${BUCKYBALL_CHIP_COMMON_INCLUDE_FLAGS} + ${BALL_ISA_INCLUDE_FLAGS} + -I${SOURCE_DIR} + DEPENDS + ${TEST_DEPS} + ${BBSW_BAREMETAL_DIR}/crt0.S + COMMENT "Building baremetal executable: ${EXECUTABLE}" + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} + ) + + add_custom_target(${TEST_NAME}-baremetal-build + DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) +endfunction() + +function(add_buckyball_ctest SOURCE_FILE) + if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") + message(FATAL_ERROR + "BUCKYBALL_WORKLOAD_CHIP must be set before add_buckyball_ctest()") + endif() + get_filename_component(_stem ${SOURCE_FILE} NAME_WE) + if(NOT DEFINED BUCKYBALL_CTEST_TARGET OR BUCKYBALL_CTEST_TARGET STREQUAL "") + message(FATAL_ERROR "BUCKYBALL_CTEST_TARGET is required for ${SOURCE_FILE}") + endif() + set(TEST_NAME "${BUCKYBALL_WORKLOAD_CHIP}-${BUCKYBALL_CTEST_TARGET}-ctest-${_stem}") + set(SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}) + + buckyball_enforce_ctest_line_limit(${SOURCE_FILE}) + buckyball_ctest_deps(TEST_DEPS ${SOURCE_DIR} ${SOURCE_FILE}) + + add_buckyball_linux_ctest(${TEST_NAME} ${SOURCE_DIR} ${SOURCE_FILE}) + add_buckyball_baremetal_ctest(${TEST_NAME} ${SOURCE_DIR} ${SOURCE_FILE}) + set(_rushb_ok TRUE) + if(_stem MATCHES "^tlb_" OR _stem STREQUAL "test_vsetvli") + set(_rushb_ok FALSE) + endif() + if(_rushb_ok AND + DEFINED BUCKYBALL_RUSHB_BEMU_MANIFEST AND + DEFINED BUCKYBALL_RUSHB_VERILATOR_LIBRARY) + add_buckyball_rushb_native(${TEST_NAME} + OUTPUT_SUBDIR src/CTest/rushB + SOURCES + ${SOURCE_DIR}/${SOURCE_FILE} + ${WORKLOAD_LIB_DIR}/buckyball.c + ${BBHW_MEM_C} + ${BUCKYBALL_CHIP_COMMON_SOURCES} + INCLUDE_DIRS + ${WORKLOAD_LIB_DIR} + ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET} + ${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS} + ${BUCKYBALL_BALL_WORKLOAD_DIRS} + ${SOURCE_DIR} + DEPENDS ${TEST_DEPS}) + endif() + + add_custom_target(${TEST_NAME}-ctest-build + DEPENDS + ${TEST_NAME}-linux-build + ${TEST_NAME}-baremetal-build + COMMENT "Building CTest workload ${TEST_NAME}" + ) + + if(DEFINED BUCKYBALL_CTEST_GROUP_TARGET) + add_dependencies(${BUCKYBALL_CTEST_GROUP_TARGET} ${TEST_NAME}-ctest-build) + endif() +endfunction() + +function(add_buckyball_ctests) + foreach(SOURCE_FILE ${ARGV}) + add_buckyball_ctest(${SOURCE_FILE}) + endforeach() +endfunction() + +# =============================================================== +# Common workloads for all chips +# =============================================================== add_subdirectory(base) +add_subdirectory(tlb) +add_subdirectory(tutorial) +add_subdirectory(embench) +add_subdirectory(coremark) + +# =============================================================== +# Chip-specific workloads +# =============================================================== set(BUCKYBALL_CHIP_WORKLOAD_DIR - ${REPO_ROOT}/examples/chips/${BUCKYBALL_WORKLOAD_CHIP}/workloads) + ${BUCKYBALL_REPO_ROOT}/examples/chips/${BUCKYBALL_WORKLOAD_CHIP}/workloads) + if(NOT EXISTS ${BUCKYBALL_CHIP_WORKLOAD_DIR}/CMakeLists.txt) message(FATAL_ERROR "Buckyball chip workloads do not exist: ${BUCKYBALL_CHIP_WORKLOAD_DIR}") endif() + add_subdirectory(${BUCKYBALL_CHIP_WORKLOAD_DIR} ${CMAKE_CURRENT_BINARY_DIR}/chips/${BUCKYBALL_WORKLOAD_CHIP}) + if(NOT TARGET chip-workloads-build) message(FATAL_ERROR "Buckyball chip workloads must define target chip-workloads-build: ${BUCKYBALL_WORKLOAD_CHIP}") endif() -add_subdirectory(embench) -add_subdirectory(coremark) diff --git a/bb-tests/workloads/src/CTest/base/CMakeLists.txt b/bb-tests/workloads/src/CTest/base/CMakeLists.txt index f64a1fc6d..34487bf04 100644 --- a/bb-tests/workloads/src/CTest/base/CMakeLists.txt +++ b/bb-tests/workloads/src/CTest/base/CMakeLists.txt @@ -1,13 +1,12 @@ -set(BUCKYBALL_TOY_COMMON_DIR ${TOY_WORKLOAD_COMMON_DIR}) -include(${REPO_ROOT}/examples/chips/toy/workloads/cmake/BuckyballWorkloads.cmake) - add_custom_target(base-ctests-build) set(BUCKYBALL_CTEST_GROUP_TARGET base-ctests-build) -add_buckyball_ctests( - mvin_mvout_acc_test.c - mvin_mvout_stride_test.c - mvin_mvout_test.c - mvin_mvout_unaligned_test.c - mvin_unaligned_test.c -) +foreach(BUCKYBALL_CTEST_TARGET IN LISTS BUCKYBALL_TARGETS) + add_buckyball_ctests( + mvin_mvout_acc_test.c + mvin_mvout_stride_test.c + mvin_mvout_test.c + mvin_mvout_unaligned_test.c + mvin_unaligned_test.c + ) +endforeach() diff --git a/bb-tests/workloads/src/CTest/base/mvin_mvout_acc_test.c b/bb-tests/workloads/src/CTest/base/mvin_mvout_acc_test.c index 38196e377..e7b4bf2fc 100644 --- a/bb-tests/workloads/src/CTest/base/mvin_mvout_acc_test.c +++ b/bb-tests/workloads/src/CTest/base/mvin_mvout_acc_test.c @@ -5,49 +5,49 @@ #include #include -#define DIM (BANK_WIDTH / sizeof(elem_t)) +// ACC bank row is 4 physical banks x 16B = 64B = 16 x int32. +#define DIM 16 -// Test matrices -static elem_t input_matrix_a[DIM * 1024] __attribute__((aligned(64))); -static elem_t input_matrix_b[DIM * 1024] __attribute__((aligned(64))); static result_t output_matrix[DIM * DIM] __attribute__((aligned(64))); static result_t expected_matrix[DIM * DIM] __attribute__((aligned(64))); -static elem_t a_transposed[DIM * 1024] __attribute__((aligned(64))); int acc_mvin_mvout_pressure_test() { for (int i = 0; i < 4; i++) { init_u32_random_matrix(expected_matrix, DIM, DIM, i * 10 + i); - - uint32_t acc_bank_id = 2; // virtual bank id - bb_mem_alloc(acc_bank_id, 1, 4); - bb_mvin((uintptr_t)expected_matrix, acc_bank_id, DIM, 1); - init_u32_random_matrix(expected_matrix, DIM, DIM, i * 10 + i); clear_u32_matrix(output_matrix, DIM, DIM); - acc_bank_id = 2; // virtual bank id + uint32_t acc_bank_id = 2; bb_mem_alloc(acc_bank_id, 1, 4); + bb_mvin((uintptr_t)expected_matrix, acc_bank_id, DIM, 1); bb_mvout((uintptr_t)output_matrix, acc_bank_id, DIM, 1); + bb_fence(); if (!compare_u32_matrices(output_matrix, expected_matrix, DIM, DIM)) { printf("Test ACC mvin/mvout pressure %d FAILED\n", i); return 0; - } else { - printf("Test ACC mvin/mvout pressure %d PASSED\n", i); } + printf("Test ACC mvin/mvout pressure %d PASSED\n", i); + bb_mem_release(acc_bank_id); + } + + // Same-vbank realloc without release must free prior physical banks + // (bemu/RTL). + { + uint32_t acc_bank_id = 2; + for (int i = 0; i < 8; i++) { + bb_mem_alloc(acc_bank_id, 1, 4); + } + bb_mem_release(acc_bank_id); + printf("Test ACC same-vbank realloc without release PASSED\n"); } return 1; } int main() { -#ifdef MULTICORE - multicore(MULTICORE); -#endif int passed = acc_mvin_mvout_pressure_test(); if (passed) { printf("ACC mvin/mvout pressure test PASSED\n"); } else { printf("ACC mvin/mvout pressure test FAILED\n"); } -#ifdef MULTICORE - exit(0); -#endif + return passed ? 0 : 1; } diff --git a/bb-tests/workloads/src/CTest/base/mvin_mvout_stride_test.c b/bb-tests/workloads/src/CTest/base/mvin_mvout_stride_test.c index e313b4433..04c90c930 100644 --- a/bb-tests/workloads/src/CTest/base/mvin_mvout_stride_test.c +++ b/bb-tests/workloads/src/CTest/base/mvin_mvout_stride_test.c @@ -64,12 +64,6 @@ int mvin_mvout_stride_test(void) { } int main() { -#ifdef MULTICORE - multicore(MULTICORE); -#endif int passed = mvin_mvout_stride_test(); -#ifdef MULTICORE - exit(0); -#endif return passed ? 0 : 1; } diff --git a/bb-tests/workloads/src/CTest/base/mvin_mvout_test.c b/bb-tests/workloads/src/CTest/base/mvin_mvout_test.c index b6325d7c7..36a9d01d7 100644 --- a/bb-tests/workloads/src/CTest/base/mvin_mvout_test.c +++ b/bb-tests/workloads/src/CTest/base/mvin_mvout_test.c @@ -31,16 +31,10 @@ int mvin_mvout_simple_test() { } int main() { -#ifdef MULTICORE - multicore(MULTICORE); -#endif int passed = mvin_mvout_simple_test(); if (passed) { printf("mvin/mvout simple test PASSED\n"); } else { printf("mvin/mvout simple test FAILED\n"); } -#ifdef MULTICORE - exit(0); -#endif } diff --git a/bb-tests/workloads/src/CTest/base/mvin_mvout_unaligned_test.c b/bb-tests/workloads/src/CTest/base/mvin_mvout_unaligned_test.c index 67e9896ae..f221b8dcb 100644 --- a/bb-tests/workloads/src/CTest/base/mvin_mvout_unaligned_test.c +++ b/bb-tests/workloads/src/CTest/base/mvin_mvout_unaligned_test.c @@ -72,12 +72,6 @@ int mvin_mvout_unaligned_test(void) { } int main() { -#ifdef MULTICORE - multicore(MULTICORE); -#endif int passed = mvin_mvout_unaligned_test(); -#ifdef MULTICORE - exit(0); -#endif return passed ? 0 : 1; } diff --git a/bb-tests/workloads/src/CTest/base/mvin_unaligned_test.c b/bb-tests/workloads/src/CTest/base/mvin_unaligned_test.c index 41b76cc96..83149ab32 100644 --- a/bb-tests/workloads/src/CTest/base/mvin_unaligned_test.c +++ b/bb-tests/workloads/src/CTest/base/mvin_unaligned_test.c @@ -55,12 +55,6 @@ int mvin_unaligned_test(void) { } int main() { -#ifdef MULTICORE - multicore(MULTICORE); -#endif int passed = mvin_unaligned_test(); -#ifdef MULTICORE - exit(0); -#endif return passed ? 0 : 1; } diff --git a/bb-tests/workloads/src/CTest/coremark/CMakeLists.txt b/bb-tests/workloads/src/CTest/coremark/CMakeLists.txt index f7d8226f8..802418836 100644 --- a/bb-tests/workloads/src/CTest/coremark/CMakeLists.txt +++ b/bb-tests/workloads/src/CTest/coremark/CMakeLists.txt @@ -33,8 +33,8 @@ set(COREMARK_C_FLAGS -DBUCKYBALL_WORKLOAD=1 ) -function(add_coremark_singlecore_target) - set(EXECUTABLE "coremark_singlecore-baremetal") +function(add_coremark_target) + set(EXECUTABLE "coremark-baremetal") add_custom_command( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} @@ -49,44 +49,13 @@ function(add_coremark_singlecore_target) ${BBSW_BAREMETAL_DIR}/crt0.S ${BBSW_BAREMETAL_DIR}/syscalls.c ${BBSW_BAREMETAL_DIR}/test.c - COMMENT "Building coremark singlecore: ${EXECUTABLE}" + COMMENT "Building coremark: ${EXECUTABLE}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) - add_custom_target(coremark_singlecore_baremetal - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) + add_custom_target(coremark-all + DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} + COMMENT "Building coremark for Buckyball") endfunction() -function(add_coremark_multicore_target) - set(EXECUTABLE "coremark_multicore-baremetal") - - add_custom_command( - OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} - COMMAND ${ELF_CC} ${COREMARK_C_FLAGS} -o ${EXECUTABLE} - ${BBSW_BAREMETAL_DIR}/start.S - -DMULTICORE=3 - ${BBSW_BAREMETAL_DIR}/syscalls.c - ${BBSW_BAREMETAL_DIR}/test.c - ${COREMARK_SOURCES} - -lm - DEPENDS ${COREMARK_SOURCES} - ${COREMARK_HEADERS} - ${BBSW_BAREMETAL_DIR}/start.S - ${BBSW_BAREMETAL_DIR}/syscalls.c - ${BBSW_BAREMETAL_DIR}/test.c - COMMENT "Building coremark multicore: ${EXECUTABLE}" - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} - ) - - add_custom_target(coremark_multicore_baremetal - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) -endfunction() - -add_coremark_singlecore_target() -add_coremark_multicore_target() - -add_custom_target(coremark-all DEPENDS - coremark_singlecore_baremetal - coremark_multicore_baremetal - COMMENT "Building all coremark benchmarks for Buckyball" -) +add_coremark_target() diff --git a/bb-tests/workloads/src/CTest/coremark/buckyball/main.c b/bb-tests/workloads/src/CTest/coremark/buckyball/main.c index 3f035eaf2..92755e170 100644 --- a/bb-tests/workloads/src/CTest/coremark/buckyball/main.c +++ b/bb-tests/workloads/src/CTest/coremark/buckyball/main.c @@ -4,16 +4,6 @@ int coremark_main(void); int main(void) { -#ifdef MULTICORE - unsigned long hartid; - __asm__ volatile("csrr %0, mhartid" : "=r"(hartid)); - if (hartid != 0) { - while (1) { - __asm__ volatile("wfi"); - } - } -#endif - int ret = coremark_main(); if (ret == 0) { bb_test_pass(); diff --git a/bb-tests/workloads/src/CTest/coremark/src/core_matrix.c b/bb-tests/workloads/src/CTest/coremark/src/core_matrix.c index dabc4b5d9..df911b059 100644 --- a/bb-tests/workloads/src/CTest/coremark/src/core_matrix.c +++ b/bb-tests/workloads/src/CTest/coremark/src/core_matrix.c @@ -35,7 +35,7 @@ zero). NxN Matrix C - used for the result. The actual values for A and B must be derived based on input that is not available at compile time. */ -ee_s16 matrix_test(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B, MATDAT val); +ee_s16 smatmul_test(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B, MATDAT val); ee_s16 matrix_sum(ee_u32 N, MATRES *C, MATDAT clipval); void matrix_mul_const(ee_u32 N, MATRES *C, MATDAT *A, MATDAT val); void matrix_mul_vect(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B); @@ -43,7 +43,7 @@ void matrix_mul_matrix(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B); void matrix_mul_matrix_bitextract(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B); void matrix_add_const(ee_u32 N, MATDAT *A, MATDAT val); -#define matrix_test_next(x) (x + 1) +#define smatmul_test_next(x) (x + 1) #define matrix_clip(x, y) ((y) ? (x) & 0x0ff : (x) & 0x0ffff) #define matrix_big(x) (0xf000 | (x)) #define bit_extract(x, from, to) (((x) >> (from)) & (~(0xffffffff << (to)))) @@ -77,7 +77,7 @@ void printmatC(MATRES *C, ee_u32 N, char *name) { /* Function: core_bench_matrix Benchmark function - Iterate N times, + Iterate N times, changing the matrix values slightly by a constant amount each time. */ ee_u16 core_bench_matrix(mat_params *p, ee_s16 seed, ee_u16 crc) { @@ -87,12 +87,12 @@ ee_u16 core_bench_matrix(mat_params *p, ee_s16 seed, ee_u16 crc) { MATDAT *B = p->B; MATDAT val = (MATDAT)seed; - crc = crc16(matrix_test(N, C, A, B, val), crc); + crc = crc16(smatmul_test(N, C, A, B, val), crc); return crc; } -/* Function: matrix_test +/* Function: smatmul_test Perform matrix manipulation. Parameters: @@ -116,7 +116,7 @@ ee_u16 core_bench_matrix(mat_params *p, ee_s16 seed, ee_u16 crc) { After the last step, matrix A is back to original contents. */ -ee_s16 matrix_test(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B, MATDAT val) { +ee_s16 smatmul_test(ee_u32 N, MATRES *C, MATDAT *A, MATDAT *B, MATDAT val) { ee_u16 crc = 0; MATDAT clipval = matrix_big(val); diff --git a/bb-tests/workloads/src/CTest/embench/CMakeLists.txt b/bb-tests/workloads/src/CTest/embench/CMakeLists.txt index 86987ad9c..425341e8e 100644 --- a/bb-tests/workloads/src/CTest/embench/CMakeLists.txt +++ b/bb-tests/workloads/src/CTest/embench/CMakeLists.txt @@ -1,10 +1,8 @@ set(ELF_CC "riscv64-unknown-elf-gcc") set(ELF_OBJCOPY "riscv64-unknown-elf-objcopy") -#------------------------------------------------------------------------------- -# Embench: baremetal benchmarks (singlecore + multicore) sharing toy's linker -# script and startup code in lib/bbsw/baremetal/. -#------------------------------------------------------------------------------- +# Embench: baremetal benchmarks sharing toy's linker script and startup +# code in lib/bbsw/baremetal/. set(CTEST_EMBENCH_WORKLOAD_DIR ${CTEST_WORKLOAD_DIR}/embench) set(EMBENCH_SUPPORT_DIR ${CTEST_EMBENCH_WORKLOAD_DIR}/support) set(EMBENCH_SRC_DIR ${CTEST_EMBENCH_WORKLOAD_DIR}/src) @@ -42,9 +40,9 @@ set(EMBENCH_TOP_OBJ_FLAGS -I${EMBENCH_SUPPORT_DIR} ) -# Generate single-core baremetal embench target -function(add_embench_singlecore_target BMARK_NAME SOURCE_FILES) - set(EXECUTABLE "embench_${BMARK_NAME}_singlecore-baremetal") +function(add_embench_target BMARK_NAME) + file(GLOB BMARK_SOURCES "${EMBENCH_SRC_DIR}/${BMARK_NAME}/*.c") + set(EXECUTABLE "embench_${BMARK_NAME}-baremetal") add_custom_command( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} @@ -52,56 +50,18 @@ function(add_embench_singlecore_target BMARK_NAME SOURCE_FILES) ${BBSW_BAREMETAL_DIR}/crt0.S ${BBSW_BAREMETAL_DIR}/syscalls.c ${EMBENCH_SUPPORT_SOURCES} - ${SOURCE_FILES} + ${BMARK_SOURCES} -lm - DEPENDS ${SOURCE_FILES} ${EMBENCH_SUPPORT_SOURCES} + DEPENDS ${BMARK_SOURCES} ${EMBENCH_SUPPORT_SOURCES} ${BBSW_BAREMETAL_DIR}/crt0.S ${BBSW_BAREMETAL_DIR}/syscalls.c - COMMENT "Building embench singlecore: ${EXECUTABLE}" - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} - ) - - add_custom_target(embench_${BMARK_NAME}_singlecore_baremetal - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) -endfunction() - -# Generate multi-core baremetal embench target -function(add_embench_multicore_target BMARK_NAME SOURCE_FILES) - set(EXECUTABLE "embench_${BMARK_NAME}_multicore-baremetal") - - add_custom_command( - OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} - COMMAND ${ELF_CC} ${EMBENCH_C_FLAGS} -o ${EXECUTABLE} - ${BBSW_BAREMETAL_DIR}/start.S - ${BBSW_BAREMETAL_DIR}/syscalls.c - -DMULTICORE=3 - ${EMBENCH_SUPPORT_SOURCES} - ${SOURCE_FILES} - -lm - DEPENDS ${SOURCE_FILES} ${EMBENCH_SUPPORT_SOURCES} - ${BBSW_BAREMETAL_DIR}/start.S - ${BBSW_BAREMETAL_DIR}/syscalls.c - COMMENT "Building embench multicore: ${EXECUTABLE}" + COMMENT "Building embench: ${EXECUTABLE}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) - add_custom_target(embench_${BMARK_NAME}_multicore_baremetal - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) -endfunction() - -# Cross-platform embench target (singlecore + multicore baremetal) -function(add_embench_target BMARK_NAME) - # Collect all .c files in this benchmark's directory - file(GLOB BMARK_SOURCES "${EMBENCH_SRC_DIR}/${BMARK_NAME}/*.c") - - add_embench_singlecore_target(${BMARK_NAME} "${BMARK_SOURCES}") - add_embench_multicore_target(${BMARK_NAME} "${BMARK_SOURCES}") - add_custom_target(embench_${BMARK_NAME} - DEPENDS embench_${BMARK_NAME}_singlecore_baremetal - embench_${BMARK_NAME}_multicore_baremetal - COMMENT "Building embench ${BMARK_NAME} for all platforms" - ) + DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} + COMMENT "Building embench ${BMARK_NAME}") endfunction() macro(add_embench_top_object BMARK_NAME PREFIX SOURCE_FILES) @@ -157,41 +117,27 @@ function(add_embench_top_target) add_embench_top_benchmark(ud ud) add_embench_top_benchmark(wikisort wikisort) - foreach(MODE singlecore multicore) - set(EXECUTABLE "embench_top_${MODE}-baremetal") - if(MODE STREQUAL "singlecore") - set(STARTUP ${BBSW_BAREMETAL_DIR}/crt0.S) - set(MODE_DEFS) - else() - set(STARTUP ${BBSW_BAREMETAL_DIR}/start.S) - set(MODE_DEFS -DMULTICORE=3) - endif() - - add_custom_command( - OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} - COMMAND ${ELF_CC} ${EMBENCH_C_FLAGS} ${MODE_DEFS} -o ${EXECUTABLE} - ${STARTUP} - ${BBSW_BAREMETAL_DIR}/syscalls.c - ${EMBENCH_TOP_SUPPORT_SOURCES} - ${EMBENCH_TOP_OBJECTS} - ${BBSW_BAREMETAL_DIR}/test.c - -lm - DEPENDS ${EMBENCH_TOP_SUPPORT_SOURCES} - ${EMBENCH_TOP_OBJECTS} - ${STARTUP} - ${BBSW_BAREMETAL_DIR}/syscalls.c - ${BBSW_BAREMETAL_DIR}/test.c - COMMENT "Building embench top ${MODE}: ${EXECUTABLE}" - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} - ) - - add_custom_target(embench_top_${MODE}_baremetal - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) - endforeach() + set(EXECUTABLE "embench_top-baremetal") + add_custom_command( + OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} + COMMAND ${ELF_CC} ${EMBENCH_C_FLAGS} -o ${EXECUTABLE} + ${BBSW_BAREMETAL_DIR}/crt0.S + ${BBSW_BAREMETAL_DIR}/syscalls.c + ${EMBENCH_TOP_SUPPORT_SOURCES} + ${EMBENCH_TOP_OBJECTS} + ${BBSW_BAREMETAL_DIR}/test.c + -lm + DEPENDS ${EMBENCH_TOP_SUPPORT_SOURCES} + ${EMBENCH_TOP_OBJECTS} + ${BBSW_BAREMETAL_DIR}/crt0.S + ${BBSW_BAREMETAL_DIR}/syscalls.c + ${BBSW_BAREMETAL_DIR}/test.c + COMMENT "Building embench top: ${EXECUTABLE}" + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} + ) add_custom_target(embench_top DEPENDS - embench_top_singlecore_baremetal - embench_top_multicore_baremetal + ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} COMMENT "Building embench top workload for Buckyball" ) endfunction() diff --git a/bb-tests/workloads/src/CTest/embench/src/picojpeg/libpicojpeg.c b/bb-tests/workloads/src/CTest/embench/src/picojpeg/libpicojpeg.c index b4a6e495c..69d00e0ae 100644 --- a/bb-tests/workloads/src/CTest/embench/src/picojpeg/libpicojpeg.c +++ b/bb-tests/workloads/src/CTest/embench/src/picojpeg/libpicojpeg.c @@ -10,14 +10,14 @@ SPDX-License-Identifier: GPL-3.0-or-later */ -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // picojpeg.c v1.1 - Public domain, Rich Geldreich // Nov. 27, 2010 - Initial release // Feb. 9, 2013 - Added H1V2/H2V1 support, cleaned up macros, signed shift fixes // Also integrated and tested changes from Chris Phoenix . -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- #include "picojpeg.h" -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Set to 1 if right shifts on signed ints are always unsigned (logical) shifts // When 1, arithmetic right shifts will be emulated by using a logical shift // with special case code to ensure the sign bit is replicated. @@ -25,12 +25,12 @@ // Define PJPG_INLINE to "inline" if your C compiler supports explicit inlining #define PJPG_INLINE -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- typedef unsigned char uint8; typedef unsigned short uint16; typedef signed char int8; typedef signed short int16; -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- #if PJPG_RIGHT_SHIFT_IS_ALWAYS_UNSIGNED static int16 replicateSignBit16(int8 n) { switch (n) { @@ -91,13 +91,13 @@ static PJPG_INLINE long arithmeticRightShift8L(long x) { #define PJPG_ARITH_SHIFT_RIGHT_N_16(x, n) ((x) >> (n)) #define PJPG_ARITH_SHIFT_RIGHT_8_L(x) ((x) >> 8) #endif -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Change as needed - the PJPG_MAX_WIDTH/PJPG_MAX_HEIGHT checks are only present // to quickly detect bogus files. #define PJPG_MAX_WIDTH 16384 #define PJPG_MAX_HEIGHT 16384 #define PJPG_MAXCOMPSINSCAN 3 -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- typedef enum { M_SOF0 = 0xC0, M_SOF1 = 0xC1, @@ -152,7 +152,7 @@ typedef enum { RST0 = 0xD0 } JPEG_MARKER; -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static const int8 ZAG[] = { 0, 1, 8, 16, 9, 2, 3, 10, 17, 24, 32, 25, 18, 11, 4, 5, 12, 19, 26, 33, 40, 48, 41, 34, 27, 20, 13, 6, 7, 14, 21, 28, @@ -160,7 +160,7 @@ static const int8 ZAG[] = { 58, 59, 52, 45, 38, 31, 39, 46, 53, 60, 61, 54, 47, 55, 62, 63, }; -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // 128 bytes static int16 gCoeffBuf[8 * 8]; @@ -208,7 +208,7 @@ static uint8 gInBufLeft; static uint16 gBitBuf; static uint8 gBitsLeft; -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint16 gImageXSize; static uint16 gImageYSize; static uint8 gCompsInFrame; @@ -240,7 +240,7 @@ static pjpeg_need_bytes_callback_t g_pNeedBytesCallback; static void *g_pCallback_data; static uint8 gCallbackStatus; static uint8 gReduce; -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static void fillInBuf(void) { unsigned char status; @@ -260,7 +260,7 @@ static void fillInBuf(void) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE uint8 getChar(void) { if (!gInBufLeft) { fillInBuf(); @@ -274,14 +274,14 @@ static PJPG_INLINE uint8 getChar(void) { return gInBuf[gInBufOfs++]; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE void stuffChar(uint8 i) { gInBufOfs--; gInBuf[gInBufOfs] = i; gInBufLeft++; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE uint8 getOctet(uint8 FFCheck) { uint8 c = getChar(); @@ -297,7 +297,7 @@ static PJPG_INLINE uint8 getOctet(uint8 FFCheck) { return c; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint16 getBits(uint8 numBits, uint8 FFCheck) { uint8 origBits = numBits; uint16 ret = gBitBuf; @@ -330,17 +330,17 @@ static uint16 getBits(uint8 numBits, uint8 FFCheck) { return ret >> (16 - origBits); } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE uint16 getBits1(uint8 numBits) { return getBits(numBits, 0); } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE uint16 getBits2(uint8 numBits) { return getBits(numBits, 1); } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE uint8 getBit(void) { uint8 ret = 0; if (gBitBuf & 0x8000) @@ -358,7 +358,7 @@ static PJPG_INLINE uint8 getBit(void) { return ret; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint16 getExtendTest(uint8 i) { switch (i) { case 0: @@ -398,7 +398,7 @@ static uint16 getExtendTest(uint8 i) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static int16 getExtendOffset(uint8 i) { switch (i) { case 0: @@ -438,12 +438,12 @@ static int16 getExtendOffset(uint8 i) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE int16 huffExtend(uint16 x, uint8 s) { return ((x < getExtendTest(s)) ? ((int16)x + getExtendOffset(s)) : (int16)x); } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static PJPG_INLINE uint8 huffDecode(const HuffTable *pHuffTable, const uint8 *pHuffVal) { uint8 i = 0; @@ -474,7 +474,7 @@ static PJPG_INLINE uint8 huffDecode(const HuffTable *pHuffTable, return pHuffVal[j]; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static void huffCreate(const uint8 *pBits, HuffTable *pHuffTable) { uint8 i = 0; uint8 j = 0; @@ -506,7 +506,7 @@ static void huffCreate(const uint8 *pBits, HuffTable *pHuffTable) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static HuffTable *getHuffTable(uint8 index) { // 0-1 = DC // 2-3 = AC @@ -524,7 +524,7 @@ static HuffTable *getHuffTable(uint8 index) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 *getHuffVal(uint8 index) { // 0-1 = DC // 2-3 = AC @@ -542,10 +542,10 @@ static uint8 *getHuffVal(uint8 index) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint16 getMaxHuffCodes(uint8 index) { return (index < 2) ? 12 : 255; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 readDHTMarker(void) { uint8 bits[16]; uint16 left = getBits1(16); @@ -599,7 +599,7 @@ static uint8 readDHTMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static void createWinogradQuant(int16 *pQuant); static uint8 readDQTMarker(void) { @@ -652,7 +652,7 @@ static uint8 readDQTMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 readSOFMarker(void) { uint8 i; uint16 left = getBits1(16); @@ -691,7 +691,7 @@ static uint8 readSOFMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Used to skip unrecognized markers. static uint8 skipVariableMarker(void) { uint16 left = getBits1(16); @@ -709,7 +709,7 @@ static uint8 skipVariableMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Read a define restart interval (DRI) marker. static uint8 readDRIMarker(void) { if (getBits1(16) != 4) @@ -720,7 +720,7 @@ static uint8 readDRIMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Read a start of scan (SOS) marker. /* Make these volatile global so the compiler does not optimise out @@ -774,7 +774,7 @@ static uint8 readSOSMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 nextMarker(void) { uint8 c; uint8 bytes = 0; @@ -799,7 +799,7 @@ static uint8 nextMarker(void) { return c; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Process markers. Returns when an SOFx, SOI, EOI, or SOS marker is // encountered. static uint8 processMarkers(uint8 *pMarker) { @@ -867,7 +867,7 @@ static uint8 processMarkers(uint8 *pMarker) { // return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Finds the start of image (SOI) marker. static uint8 locateSOIMarker(void) { uint16 bytesleft; @@ -911,7 +911,7 @@ static uint8 locateSOIMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Find a start of frame (SOF) marker. static uint8 locateSOFMarker(void) { uint8 c; @@ -950,7 +950,7 @@ static uint8 locateSOFMarker(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Find a start of scan (SOS) marker. static uint8 locateSOSMarker(uint8 *pFoundEOI) { uint8 c; @@ -971,7 +971,7 @@ static uint8 locateSOSMarker(uint8 *pFoundEOI) { return readSOSMarker(); } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 init(void) { gImageXSize = 0; gImageYSize = 0; @@ -992,7 +992,7 @@ static uint8 init(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // This method throws back into the stream any bytes that where read // into the bit buffer during initial marker scanning. static void fixInBuffer(void) { @@ -1008,7 +1008,7 @@ static void fixInBuffer(void) { getBits2(8); } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // Restart interval processing. static uint8 processRestart(void) { // Let's scan a little bit to find the marker, but not _too_ far. @@ -1052,7 +1052,7 @@ static uint8 processRestart(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 checkHuffTables(void) { uint8 i; @@ -1068,7 +1068,7 @@ static uint8 checkHuffTables(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 checkQuantTables(void) { uint8 i; @@ -1082,7 +1082,7 @@ static uint8 checkQuantTables(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 initScan(void) { uint8 foundEOI; uint8 status = locateSOSMarker(&foundEOI); @@ -1113,7 +1113,7 @@ static uint8 initScan(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 initFrame(void) { if (gCompsInFrame == 1) { if ((gCompHSamp[0] != 1) || (gCompVSamp[0] != 1)) @@ -1191,7 +1191,7 @@ static uint8 initFrame(void) { return 0; } -//---------------------------------------------------------------------------- +//===-------------------------------------------------------------------===//- // Winograd IDCT: 5 multiplies per row/col, up to 80 muls for the 2D IDCT #define PJPG_DCT_SCALE_BITS 7 @@ -1835,7 +1835,7 @@ static void transformBlock(uint8 mcuBlock) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static void transformBlockReduce(uint8 mcuBlock) { uint8 c = clamp(PJPG_DESCALE(gCoeffBuf[0]) + 128); int16 cbG, cbB, crR, crG; @@ -2018,7 +2018,7 @@ static void transformBlockReduce(uint8 mcuBlock) { } } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- static uint8 decodeNextMCU(void) { uint8 status; uint8 mcuBlock; @@ -2144,7 +2144,7 @@ static uint8 decodeNextMCU(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- unsigned char pjpeg_decode_mcu(void) { uint8 status; @@ -2163,7 +2163,7 @@ unsigned char pjpeg_decode_mcu(void) { return 0; } -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- unsigned char pjpeg_decode_init(pjpeg_image_info_t *pInfo, pjpeg_need_bytes_callback_t pNeed_bytes_callback, diff --git a/bb-tests/workloads/src/CTest/embench/src/picojpeg/picojpeg.h b/bb-tests/workloads/src/CTest/embench/src/picojpeg/picojpeg.h index 982a503f7..0d1c1ff1e 100644 --- a/bb-tests/workloads/src/CTest/embench/src/picojpeg/picojpeg.h +++ b/bb-tests/workloads/src/CTest/embench/src/picojpeg/picojpeg.h @@ -10,9 +10,9 @@ SPDX-License-Identifier: GPL-3.0-or-later */ -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- // picojpeg - Public domain, Rich Geldreich -//------------------------------------------------------------------------------ +//===-------------------------------------------------------------------===//--- #ifndef PICOJPEG_H #define PICOJPEG_H diff --git a/bb-tests/workloads/src/CTest/embench/support/top.c b/bb-tests/workloads/src/CTest/embench/support/top.c index c6104492e..94baa83c3 100644 --- a/bb-tests/workloads/src/CTest/embench/support/top.c +++ b/bb-tests/workloads/src/CTest/embench/support/top.c @@ -74,16 +74,6 @@ static inline uint64_t rdcycle(void) { } int main(void) { -#ifdef MULTICORE - unsigned long hartid; - __asm__ volatile("csrr %0, mhartid" : "=r"(hartid)); - if (hartid != 0) { - while (1) { - __asm__ volatile("wfi"); - } - } -#endif - int failures = 0; uint64_t total = 0; diff --git a/bb-tests/workloads/src/CTest/tlb/CMakeLists.txt b/bb-tests/workloads/src/CTest/tlb/CMakeLists.txt new file mode 100644 index 000000000..0fb1695a6 --- /dev/null +++ b/bb-tests/workloads/src/CTest/tlb/CMakeLists.txt @@ -0,0 +1,14 @@ +add_custom_target(tlb-ctests-build) +set(BUCKYBALL_CTEST_GROUP_TARGET tlb-ctests-build) + +foreach(BUCKYBALL_CTEST_TARGET IN LISTS BUCKYBALL_TARGETS) + add_buckyball_ctests( + tlb_bare_mode_test.c + tlb_repeated_sfence_test.c + tlb_sfence_flush_test.c + tlb_sfence_remap_test.c + tlb_sv39_4k_alias_test.c + tlb_sv39_identity_test.c + tlb_sv39_nonidentity_test.c + ) +endforeach() diff --git a/examples/chips/toy/workloads/ctests/tlb_bare_mode_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_bare_mode_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_bare_mode_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_bare_mode_test.c diff --git a/examples/chips/toy/workloads/common/tlb_common.h b/bb-tests/workloads/src/CTest/tlb/tlb_common.h similarity index 98% rename from examples/chips/toy/workloads/common/tlb_common.h rename to bb-tests/workloads/src/CTest/tlb/tlb_common.h index 13426b9dd..b51414cb0 100644 --- a/examples/chips/toy/workloads/common/tlb_common.h +++ b/bb-tests/workloads/src/CTest/tlb/tlb_common.h @@ -1,5 +1,5 @@ -#ifndef CTEST_TOY_TLB_COMMON_H -#define CTEST_TOY_TLB_COMMON_H +#ifndef CTEST_TLB_COMMON_H +#define CTEST_TLB_COMMON_H #include "buckyball.h" #include @@ -168,4 +168,4 @@ static void tlb_select_hart(void) { #endif } -#endif +#endif // CTEST_TLB_COMMON_H diff --git a/examples/chips/toy/workloads/ctests/tlb_repeated_sfence_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_repeated_sfence_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_repeated_sfence_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_repeated_sfence_test.c diff --git a/examples/chips/toy/workloads/ctests/tlb_sfence_flush_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_sfence_flush_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_sfence_flush_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_sfence_flush_test.c diff --git a/examples/chips/toy/workloads/ctests/tlb_sfence_remap_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_sfence_remap_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_sfence_remap_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_sfence_remap_test.c diff --git a/examples/chips/toy/workloads/ctests/tlb_sv39_4k_alias_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_sv39_4k_alias_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_sv39_4k_alias_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_sv39_4k_alias_test.c diff --git a/examples/chips/toy/workloads/ctests/tlb_sv39_identity_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_sv39_identity_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_sv39_identity_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_sv39_identity_test.c diff --git a/examples/chips/toy/workloads/ctests/tlb_sv39_nonidentity_test.c b/bb-tests/workloads/src/CTest/tlb/tlb_sv39_nonidentity_test.c similarity index 100% rename from examples/chips/toy/workloads/ctests/tlb_sv39_nonidentity_test.c rename to bb-tests/workloads/src/CTest/tlb/tlb_sv39_nonidentity_test.c diff --git a/bb-tests/workloads/src/CTest/tutorial/CMakeLists.txt b/bb-tests/workloads/src/CTest/tutorial/CMakeLists.txt new file mode 100644 index 000000000..dd938f2b9 --- /dev/null +++ b/bb-tests/workloads/src/CTest/tutorial/CMakeLists.txt @@ -0,0 +1,14 @@ +add_custom_target(tutorial-ctests-build) +set(BUCKYBALL_CTEST_GROUP_TARGET tutorial-ctests-build) + +# Baremetal single-core only (matches Bazel //bb-tests/.../CTest/tutorial). +foreach(BUCKYBALL_CTEST_TARGET IN LISTS BUCKYBALL_TARGETS) + set(BUCKYBALL_TUTORIAL_TEST + ${BUCKYBALL_WORKLOAD_CHIP}_${BUCKYBALL_CTEST_TARGET}_tutorial) + add_buckyball_baremetal_ctest( + ${BUCKYBALL_TUTORIAL_TEST} + ${CMAKE_CURRENT_SOURCE_DIR} + tutorial.c) + add_dependencies(tutorial-ctests-build + ${BUCKYBALL_TUTORIAL_TEST}-baremetal-build) +endforeach() diff --git a/bb-tests/workloads/src/tutorial/tutorial.c b/bb-tests/workloads/src/CTest/tutorial/tutorial.c similarity index 100% rename from bb-tests/workloads/src/tutorial/tutorial.c rename to bb-tests/workloads/src/CTest/tutorial/tutorial.c diff --git a/bb-tests/workloads/src/OpTest/cmake/BuckyballMlirTests.cmake b/bb-tests/workloads/src/MLIRTest/CMakeLists.txt similarity index 51% rename from bb-tests/workloads/src/OpTest/cmake/BuckyballMlirTests.cmake rename to bb-tests/workloads/src/MLIRTest/CMakeLists.txt index db9262a23..572ce67ba 100644 --- a/bb-tests/workloads/src/OpTest/cmake/BuckyballMlirTests.cmake +++ b/bb-tests/workloads/src/MLIRTest/CMakeLists.txt @@ -1,3 +1,9 @@ +# The workload root chooses the compiler instance for this chip's core. +if(NOT DEFINED BUDDY_BINARY_DIR) + message(FATAL_ERROR + "BUDDY_BINARY_DIR must be set by the workload root.") +endif() + if(NOT DEFINED LLVM_BUILD_DIR) message(FATAL_ERROR "LLVM_BUILD_DIR is not set") endif() @@ -9,23 +15,30 @@ endif() set(FILECHECK ${LLVM_BUILD_DIR}/bin/FileCheck) function(add_buckyball_mlir_contract TEST_NAME) - cmake_parse_arguments(ARG "" "" "PASSES" ${ARGN}) + cmake_parse_arguments(ARG "" "TARGET" "PASSES" ${ARGN}) if(NOT DEFINED BUCKYBALL_MLIR_TEST_PREFIX) message(FATAL_ERROR "BUCKYBALL_MLIR_TEST_PREFIX is not set") endif() + if(NOT ARG_TARGET) + message(FATAL_ERROR "${TEST_NAME}: TARGET is required") + endif() + list(FIND BUCKYBALL_TARGETS "${ARG_TARGET}" TARGET_INDEX) + if(TARGET_INDEX EQUAL -1) + message(FATAL_ERROR "${TEST_NAME}: unknown Buckyball target ${ARG_TARGET}") + endif() if(NOT ARG_PASSES) set(ARG_PASSES ${BUCKYBALL_LOWER_BUCKYBALL}) endif() set(MLIR_SRC ${CMAKE_CURRENT_SOURCE_DIR}/${TEST_NAME}.mlir) - set(OUT ${CMAKE_CURRENT_BINARY_DIR}/${TEST_NAME}.out.mlir) - set(TARGET ${BUCKYBALL_MLIR_TEST_PREFIX}-${TEST_NAME}-mlir-test) + set(OUT ${CMAKE_CURRENT_BINARY_DIR}/${ARG_TARGET}-${TEST_NAME}.out.mlir) + set(TARGET ${BUCKYBALL_MLIR_TEST_PREFIX}-${ARG_TARGET}-${TEST_NAME}-mlir-test) add_custom_command( OUTPUT ${OUT} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} ${ARG_PASSES} > ${OUT} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=${ARG_TARGET} ${ARG_PASSES} > ${OUT} DEPENDS ${MLIR_SRC} ${BUDDY_OPT} COMMENT "Building Buckyball MLIR contract IR: ${TARGET}" VERBATIM) @@ -41,34 +54,42 @@ function(add_buckyball_mlir_contract TEST_NAME) endif() endfunction() -# Bank-level / ball-op MLIR -> baremetal + linux ELF (same pipeline as toy OpTest). -function(add_buckyball_mlir_optest NAME) - cmake_parse_arguments(ARG "" "" "PASSES" ${ARGN}) +# Bank-level / ball-op MLIR -> baremetal + linux ELF (same pipeline as toy MLIRTest). +function(add_buckyball_mlir_test NAME) + cmake_parse_arguments(ARG "" "TARGET" "PASSES" ${ARGN}) if(NOT DEFINED BUCKYBALL_MLIR_TEST_PREFIX) message(FATAL_ERROR "BUCKYBALL_MLIR_TEST_PREFIX is not set") endif() if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") - message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP must be set before mlir optests") + message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP must be set before mlir tests") endif() - if(NOT DEFINED BUCKYBALL_MLIR_BANK_NUM) - message(FATAL_ERROR "BUCKYBALL_MLIR_BANK_NUM is not set") + if(NOT ARG_TARGET) + message(FATAL_ERROR "${NAME}: TARGET is required") + endif() + list(FIND BUCKYBALL_TARGETS "${ARG_TARGET}" TARGET_INDEX) + if(TARGET_INDEX EQUAL -1) + message(FATAL_ERROR "${NAME}: unknown Buckyball target ${ARG_TARGET}") + endif() + set(BB_BALL_ISA_INCLUDE_DIR "${BUCKYBALL_ISA_DIR}/${ARG_TARGET}") + set(BB_BALL_ISA_H "${BB_BALL_ISA_INCLUDE_DIR}/ballISA.h") + if(NOT EXISTS "${BB_BALL_ISA_H}") + message(FATAL_ERROR "${NAME}: missing target ISA header ${BB_BALL_ISA_H}") endif() - set(MLIR_SRC ${CMAKE_CURRENT_SOURCE_DIR}/${NAME}.mlir) set(MAIN_SRC ${CMAKE_CURRENT_SOURCE_DIR}/${NAME}_main.cpp) - set(OBJ ${CMAKE_CURRENT_BINARY_DIR}/${NAME}.o) + set(OBJ ${CMAKE_CURRENT_BINARY_DIR}/${ARG_TARGET}-${NAME}.o) set(PREFIX ${BUCKYBALL_MLIR_TEST_PREFIX}) set(TEST_ID ${NAME}) if(NOT NAME MATCHES "^${PREFIX}_") set(TEST_ID ${PREFIX}_${NAME}) endif() set(BAREMETAL_BIN - ${BUCKYBALL_WORKLOAD_CHIP}_optest_${TEST_ID}_singlecore-baremetal) - set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_${TEST_ID}-linux) - set(BAREMETAL_TARGET optest_${TEST_ID}_singlecore_baremetal) - set(LINUX_TARGET optest_${TEST_ID}_linux) - set(GROUP_TARGET optest_${TEST_ID}) + ${BUCKYBALL_WORKLOAD_CHIP}-${ARG_TARGET}-mlirtest-${TEST_ID}-baremetal) + set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}-${ARG_TARGET}-mlirtest-${TEST_ID}-linux) + set(BAREMETAL_TARGET ${BAREMETAL_BIN}-build) + set(LINUX_TARGET ${LINUX_BIN}-build) + set(GROUP_TARGET ${BUCKYBALL_WORKLOAD_CHIP}-${ARG_TARGET}-mlirtest-${TEST_ID}) set(BBSIM_LD ${BBSW_BAREMETAL_DIR}/bbsim.ld) set(C_FLAGS -g -fno-common -O2 -static -march=rv64gc -mcmodel=medany @@ -85,7 +106,7 @@ function(add_buckyball_mlir_optest NAME) set(MLIR_PASSES ${ARG_PASSES}) else() set(MLIR_PASSES - ${BUCKYBALL_ASSIGN_PHYSICAL_BANKS} + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -99,63 +120,23 @@ function(add_buckyball_mlir_optest NAME) -convert-func-to-llvm -reconcile-unrealized-casts) endif() - - if(NOT DEFINED BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS OR - NOT DEFINED BUCKYBALL_LOWER_BUCKYBALL_RUSHB) - message(FATAL_ERROR - "rushB MLIR lowers require BUCKYBALL_LOWER_*_RUSHB " - "(define them in bb-tests/workloads/CMakeLists.txt)") - endif() - - # Keep lower-buckyball before convert-arith-to-llvm (it emits arith). - # Insert intrinsics-to-rushb immediately before the final reconcile. - set(RUSHB_MLIR_PASSES) - set(RUSHB_HAS_RECONCILE FALSE) - foreach(MLIR_PASS ${MLIR_PASSES}) - if(MLIR_PASS STREQUAL "${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS}") - list(APPEND RUSHB_MLIR_PASSES ${BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS}) - elseif(MLIR_PASS STREQUAL "${BUCKYBALL_LOWER_BUCKYBALL}") - list(APPEND RUSHB_MLIR_PASSES ${BUCKYBALL_LOWER_BUCKYBALL_RUSHB}) - elseif(MLIR_PASS STREQUAL "-reconcile-unrealized-casts") - set(RUSHB_HAS_RECONCILE TRUE) - list(APPEND RUSHB_MLIR_PASSES - -lower-buckyball-intrinsics-to-rushb - -reconcile-unrealized-casts) - else() - list(APPEND RUSHB_MLIR_PASSES ${MLIR_PASS}) - endif() - endforeach() - if(NOT RUSHB_HAS_RECONCILE) - message(FATAL_ERROR - "rushB pass rewrite for ${NAME} requires -reconcile-unrealized-casts") - endif() - set(RUSHB_OBJ ${CMAKE_CURRENT_BINARY_DIR}/${NAME}-rushB.o) + string(JOIN " " MLIR_PASS_COMMAND ${MLIR_PASSES}) add_custom_command( OUTPUT ${OBJ} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} ${MLIR_PASSES} | - ${BUDDY_TRANSLATE} --buddy-to-llvmir | - ${BUDDY_LLC} -filetype=obj -mtriple=riscv64 -O2 -code-model=medium - -mattr=${BUCKYBALL_RISCV_MATTR} -float-abi=hard -o ${OBJ} + COMMAND bash -o pipefail -c + "${BUDDY_OPT} ${MLIR_SRC} --target=${ARG_TARGET} ${MLIR_PASS_COMMAND} | ${BUDDY_TRANSLATE} --buddy-to-llvmir | ${BUDDY_LLC} -filetype=obj -mtriple=riscv64 -O2 -code-model=medium -mattr=${BUCKYBALL_RISCV_MATTR} -float-abi=hard -o ${OBJ}" DEPENDS ${MLIR_SRC} ${BUDDY_OPT} COMMENT "Building ${NAME}.o from ${PREFIX} MLIR" VERBATIM) - add_custom_command( - OUTPUT ${RUSHB_OBJ} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} ${RUSHB_MLIR_PASSES} | - ${BUDDY_TRANSLATE} --buddy-to-llvmir | - ${BUDDY_LLC} -filetype=obj -mtriple=x86_64 -O2 -o ${RUSHB_OBJ} - DEPENDS ${MLIR_SRC} ${BUDDY_OPT} ${BUDDY_TRANSLATE} ${BUDDY_LLC} - COMMENT "Building rushB ${NAME}.o from ${PREFIX} MLIR" - VERBATIM) - add_custom_command( OUTPUT ${BAREMETAL_BIN} COMMAND ${ELF_CC} ${C_FLAGS} -I${LLVM_MLIR_EXECUTION_ENGINE_DIR} -I${WORKLOAD_LIB_DIR} -I${WORKLOAD_LIB_DIR}/bbhw/mem + -I${BB_BALL_ISA_INCLUDE_DIR} -o ${CMAKE_CURRENT_BINARY_DIR}/${BAREMETAL_BIN} ${BBSW_BAREMETAL_DIR}/crt0.S ${BBSW_BAREMETAL_DIR}/syscalls.c @@ -164,6 +145,7 @@ function(add_buckyball_mlir_optest NAME) ${MAIN_SRC} ${OBJ} DEPENDS ${OBJ} ${MAIN_SRC} ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} + ${BB_BALL_ISA_H} ${BBSW_BAREMETAL_DIR}/crt0.S ${BBSW_BAREMETAL_DIR}/syscalls.c COMMENT "Linking ${BAREMETAL_BIN}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} @@ -175,12 +157,13 @@ function(add_buckyball_mlir_optest NAME) -I${LLVM_MLIR_EXECUTION_ENGINE_DIR} -I${WORKLOAD_LIB_DIR} -I${WORKLOAD_LIB_DIR}/bbhw/mem + -I${BB_BALL_ISA_INCLUDE_DIR} -o ${CMAKE_CURRENT_BINARY_DIR}/${LINUX_BIN} ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} ${MAIN_SRC} ${OBJ} - DEPENDS ${OBJ} ${MAIN_SRC} ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} + DEPENDS ${OBJ} ${MAIN_SRC} ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} ${BB_BALL_ISA_H} COMMENT "Linking ${LINUX_BIN}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} VERBATIM) @@ -194,18 +177,68 @@ function(add_buckyball_mlir_optest NAME) if(DEFINED BUCKYBALL_RUSHB_BEMU_MANIFEST AND DEFINED BUCKYBALL_RUSHB_VERILATOR_LIBRARY) - add_buckyball_rushb_native(${BUCKYBALL_WORKLOAD_CHIP}_optest_${TEST_ID} + if(NOT DEFINED BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS OR + NOT DEFINED BUCKYBALL_LOWER_BUCKYBALL_RUSHB) + message(FATAL_ERROR + "rushB MLIR lowers require BUCKYBALL_LOWER_*_RUSHB " + "(define them in bb-tests/workloads/CMakeLists.txt)") + endif() + set(RUSHB_MLIR_PASSES) + set(RUSHB_HAS_RECONCILE FALSE) + foreach(MLIR_PASS ${MLIR_PASSES}) + if(MLIR_PASS STREQUAL "${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS}") + list(APPEND RUSHB_MLIR_PASSES ${BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS}) + elseif(MLIR_PASS STREQUAL "${BUCKYBALL_LOWER_BUCKYBALL}") + list(APPEND RUSHB_MLIR_PASSES ${BUCKYBALL_LOWER_BUCKYBALL_RUSHB}) + elseif(MLIR_PASS STREQUAL "-reconcile-unrealized-casts") + set(RUSHB_HAS_RECONCILE TRUE) + list(APPEND RUSHB_MLIR_PASSES + -lower-buckyball-intrinsics-to-rushb + -reconcile-unrealized-casts) + else() + list(APPEND RUSHB_MLIR_PASSES ${MLIR_PASS}) + endif() + endforeach() + if(NOT RUSHB_HAS_RECONCILE) + message(FATAL_ERROR + "rushB pass rewrite for ${NAME} requires -reconcile-unrealized-casts") + endif() + set(RUSHB_OBJ ${CMAKE_CURRENT_BINARY_DIR}/${NAME}-rushB.o) + string(JOIN " " RUSHB_MLIR_PASS_COMMAND ${RUSHB_MLIR_PASSES}) + add_custom_command( + OUTPUT ${RUSHB_OBJ} + COMMAND bash -o pipefail -c + "${BUDDY_OPT} ${MLIR_SRC} --target=${ARG_TARGET} ${RUSHB_MLIR_PASS_COMMAND} | ${BUDDY_TRANSLATE} --buddy-to-llvmir | ${BUDDY_LLC} -filetype=obj -mtriple=x86_64 -O2 -o ${RUSHB_OBJ}" + DEPENDS ${MLIR_SRC} ${BUDDY_OPT} ${BUDDY_TRANSLATE} ${BUDDY_LLC} + COMMENT "Building rushB ${NAME}.o from ${PREFIX} MLIR" + VERBATIM) + add_buckyball_rushb_native(${BUCKYBALL_WORKLOAD_CHIP}-${ARG_TARGET}-mlirtest-${TEST_ID} CXX - OUTPUT_SUBDIR src/OpTest/rushB + OUTPUT_SUBDIR src/MLIRTest/rushB SOURCES ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} ${MAIN_SRC} ${RUSHB_OBJ} INCLUDE_DIRS ${LLVM_MLIR_EXECUTION_ENGINE_DIR} ${WORKLOAD_LIB_DIR} ${WORKLOAD_LIB_DIR}/bbhw/mem - DEPENDS ${RUSHB_OBJ} ${MAIN_SRC} ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C}) + ${BB_BALL_ISA_INCLUDE_DIR} + DEPENDS ${RUSHB_OBJ} ${MAIN_SRC} ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} + ${BB_BALL_ISA_H}) endif() if(DEFINED BUCKYBALL_MLIR_GROUP_TARGET) add_dependencies(${BUCKYBALL_MLIR_GROUP_TARGET} ${GROUP_TARGET}) endif() endfunction() + +set(MLIRTEST_BASE_DIR ${MLIRTEST_WORKLOAD_DIR}/base) + +add_subdirectory(${MLIRTEST_WORKLOAD_DIR}/base + ${CMAKE_CURRENT_BINARY_DIR}/mlir_tests/base) + +add_custom_target(balls-mlir-tests-build) +buckyball_add_ball_workload_subdirs(mlir_tests) + +add_custom_target(MLIRTest-all ALL DEPENDS + base-mlirtest-build + balls-mlir-tests-build +) diff --git a/bb-tests/workloads/src/MLIRTest/base/CMakeLists.txt b/bb-tests/workloads/src/MLIRTest/base/CMakeLists.txt new file mode 100644 index 000000000..444ce5d12 --- /dev/null +++ b/bb-tests/workloads/src/MLIRTest/base/CMakeLists.txt @@ -0,0 +1,34 @@ +add_custom_target(base-mlirtest-build) + +set(BUCKYBALL_MLIR_GROUP_TARGET base-mlirtest-build) +set(BUCKYBALL_MLIR_TEST_PREFIX base) + +foreach(BUCKYBALL_TARGET IN LISTS BUCKYBALL_BANK_TARGETS) + add_buckyball_mlir_contract(base_intrinsics_contract + TARGET ${BUCKYBALL_TARGET} + PASSES ${BUCKYBALL_LOWER_BUCKYBALL}) + + add_buckyball_mlir_contract(base_bank_contract + TARGET ${BUCKYBALL_TARGET} + PASSES + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS}) + + add_buckyball_mlir_test(base_mvin_mvout_16x16_i8 + TARGET ${BUCKYBALL_TARGET} + PASSES + ${BUCKYBALL_LOWER_BUCKYBALL} + -convert-linalg-to-loops + -expand-strided-metadata + -lower-affine + -convert-scf-to-cf + -convert-cf-to-llvm + -convert-arith-to-llvm + -convert-math-to-llvm + -finalize-memref-to-llvm + -convert-func-to-llvm + -reconcile-unrealized-casts) + + add_buckyball_mlir_test(base_bank_mvin_mvout_16x16_i8 + TARGET ${BUCKYBALL_TARGET}) +endforeach() diff --git a/bb-tests/workloads/src/OpTest/base/base_bank_contract.mlir b/bb-tests/workloads/src/MLIRTest/base/base_bank_contract.mlir similarity index 100% rename from bb-tests/workloads/src/OpTest/base/base_bank_contract.mlir rename to bb-tests/workloads/src/MLIRTest/base/base_bank_contract.mlir diff --git a/bb-tests/workloads/src/OpTest/base/base_bank_mvin_mvout_16x16_i8.mlir b/bb-tests/workloads/src/MLIRTest/base/base_bank_mvin_mvout_16x16_i8.mlir similarity index 89% rename from bb-tests/workloads/src/OpTest/base/base_bank_mvin_mvout_16x16_i8.mlir rename to bb-tests/workloads/src/MLIRTest/base/base_bank_mvin_mvout_16x16_i8.mlir index 4f39ae9c1..32a9315cc 100644 --- a/bb-tests/workloads/src/OpTest/base/base_bank_mvin_mvout_16x16_i8.mlir +++ b/bb-tests/workloads/src/MLIRTest/base/base_bank_mvin_mvout_16x16_i8.mlir @@ -8,8 +8,8 @@ func.func @main() -> i8 { %c16 = arith.constant 16 : i32 %depth = arith.constant 16 : i64 %stride = arith.constant 1 : i64 - %input = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<16x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> scf.for %i = %c0 to %n step %c1 { scf.for %j = %c0 to %n step %c1 { diff --git a/bb-tests/workloads/src/OpTest/base/base_bank_mvin_mvout_16x16_i8_main.cpp b/bb-tests/workloads/src/MLIRTest/base/base_bank_mvin_mvout_16x16_i8_main.cpp similarity index 100% rename from bb-tests/workloads/src/OpTest/base/base_bank_mvin_mvout_16x16_i8_main.cpp rename to bb-tests/workloads/src/MLIRTest/base/base_bank_mvin_mvout_16x16_i8_main.cpp diff --git a/bb-tests/workloads/src/OpTest/base/base_intrinsics_contract.mlir b/bb-tests/workloads/src/MLIRTest/base/base_intrinsics_contract.mlir similarity index 100% rename from bb-tests/workloads/src/OpTest/base/base_intrinsics_contract.mlir rename to bb-tests/workloads/src/MLIRTest/base/base_intrinsics_contract.mlir diff --git a/bb-tests/workloads/src/OpTest/base/base_mvin_mvout_16x16_i8.mlir b/bb-tests/workloads/src/MLIRTest/base/base_mvin_mvout_16x16_i8.mlir similarity index 89% rename from bb-tests/workloads/src/OpTest/base/base_mvin_mvout_16x16_i8.mlir rename to bb-tests/workloads/src/MLIRTest/base/base_mvin_mvout_16x16_i8.mlir index 9c3cc18e4..997471436 100644 --- a/bb-tests/workloads/src/OpTest/base/base_mvin_mvout_16x16_i8.mlir +++ b/bb-tests/workloads/src/MLIRTest/base/base_mvin_mvout_16x16_i8.mlir @@ -9,8 +9,8 @@ func.func @main() -> i8 { %bank = arith.constant 0 : i64 %depth = arith.constant 16 : i64 %stride = arith.constant 1 : i64 - %input = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<16x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> scf.for %i = %c0 to %n step %c1 { scf.for %j = %c0 to %n step %c1 { diff --git a/bb-tests/workloads/src/OpTest/base/base_mvin_mvout_16x16_i8_main.cpp b/bb-tests/workloads/src/MLIRTest/base/base_mvin_mvout_16x16_i8_main.cpp similarity index 100% rename from bb-tests/workloads/src/OpTest/base/base_mvin_mvout_16x16_i8_main.cpp rename to bb-tests/workloads/src/MLIRTest/base/base_mvin_mvout_16x16_i8_main.cpp diff --git a/bb-tests/workloads/src/ModelTest/CMakeLists.txt b/bb-tests/workloads/src/ModelTest/CMakeLists.txt index 5b4e0c128..d52f3d2c8 100644 --- a/bb-tests/workloads/src/ModelTest/CMakeLists.txt +++ b/bb-tests/workloads/src/ModelTest/CMakeLists.txt @@ -1,38 +1,22 @@ set(E2E_MODEL_DIR ${MODELTEST_WORKLOAD_DIR}/e2e) -# Enable models by default for e2e configure. Gemmini stays on as a reference -# backend. Buckyball ModelTest enables itself iff this chip has a layout under -# e2e/models/archs/buckyball// (model↔chip is many-to-many). +if(NOT DEFINED MODEL OR MODEL STREQUAL "") + return() +endif() + if(NOT DEFINED ARCH) set(ARCH_GEMMINI ON) + set(ARCH_X86 ON) if(DEFINED BUCKYBALL_WORKLOAD_CHIP AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/e2e/models/archs/buckyball/${BUCKYBALL_WORKLOAD_CHIP}") set(ARCH_BUCKYBALL ON) endif() endif() -if(NOT DEFINED MODEL) - set(MODEL_LENET ON) - set(MODEL_RESNET18 ON) - set(MODEL_MOBILENETV3 ON) - set(MODEL_BERT ON) - set(MODEL_WHISPER ON) - set(MODEL_STABLEDIFFUSION ON) - set(MODEL_LLAMA2 ON) - set(MODEL_DEEPSEEKR1 ON) - set(MODEL_QWEN3 ON) - set(MODEL_YOLO26 ON) - set(MODEL_BUDDYNEXT ON) - set(MODEL_GEMMA4 ON) - set(MODEL_MINIMAX_H3_FL2VA ON) - set(MODEL_MINIMAX_H3_REF2VA ON) -endif() - -# EXCLUDE_FROM_ALL keeps e2e targets out of the default `ninja` build but -# they remain buildable explicitly: `ninja buddy-gemmini-lenet-run` -if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/e2e/models/CMakeLists.txt") - add_subdirectory(e2e/models EXCLUDE_FROM_ALL) +if(NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/e2e/models/CMakeLists.txt") + message(FATAL_ERROR "e2e models CMakeLists.txt is missing") endif() +add_subdirectory(e2e/models) unset(ARCH CACHE) unset(MODEL CACHE) diff --git a/bb-tests/workloads/src/ModelTest/e2e b/bb-tests/workloads/src/ModelTest/e2e index 11dd050ea..4672c666e 160000 --- a/bb-tests/workloads/src/ModelTest/e2e +++ b/bb-tests/workloads/src/ModelTest/e2e @@ -1 +1 @@ -Subproject commit 11dd050eaa2a12af2901a351d3658dfc245e71cf +Subproject commit 4672c666ebf133b82d3361f0af88c29ac54e1356 diff --git a/bb-tests/workloads/src/OpTest/CMakeLists.txt b/bb-tests/workloads/src/OpTest/CMakeLists.txt deleted file mode 100644 index 83bec11d4..000000000 --- a/bb-tests/workloads/src/OpTest/CMakeLists.txt +++ /dev/null @@ -1,30 +0,0 @@ -# Ensure buddy-mlir build dir is available -if(NOT DEFINED ENV{BUDDY_MLIR_BUILD_DIR}) - message(FATAL_ERROR - "BUDDY_MLIR_BUILD_DIR is not set. Enter `nix develop` shell first.") -endif() - -set(OPTEST_BASE_DIR ${OPTEST_WORKLOAD_DIR}/base) - -add_subdirectory(base) - -add_custom_target(balls-mlir-tests-build) -file(GLOB BUCKYBALL_BALL_MLIR_CMAKES CONFIGURE_DEPENDS - "${REPO_ROOT}/examples/balls/*/workloads/mlir_tests/CMakeLists.txt") -foreach(BUCKYBALL_BALL_MLIR_CMAKE ${BUCKYBALL_BALL_MLIR_CMAKES}) - get_filename_component(BUCKYBALL_BALL_MLIR_DIR - ${BUCKYBALL_BALL_MLIR_CMAKE} DIRECTORY) - get_filename_component(BUCKYBALL_BALL_WORKLOADS_DIR - ${BUCKYBALL_BALL_MLIR_DIR} DIRECTORY) - get_filename_component(BUCKYBALL_BALL_DIR - ${BUCKYBALL_BALL_WORKLOADS_DIR} DIRECTORY) - get_filename_component(BUCKYBALL_BALL_NAME ${BUCKYBALL_BALL_DIR} NAME) - add_subdirectory( - ${BUCKYBALL_BALL_MLIR_DIR} - ${CMAKE_CURRENT_BINARY_DIR}/balls/${BUCKYBALL_BALL_NAME}/mlir_tests) -endforeach() - -add_custom_target(OpTest-all ALL DEPENDS - base-OpTest-build - balls-mlir-tests-build -) diff --git a/bb-tests/workloads/src/OpTest/base/CMakeLists.txt b/bb-tests/workloads/src/OpTest/base/CMakeLists.txt deleted file mode 100644 index e7137822a..000000000 --- a/bb-tests/workloads/src/OpTest/base/CMakeLists.txt +++ /dev/null @@ -1,30 +0,0 @@ -include(${CMAKE_CURRENT_LIST_DIR}/../cmake/BuckyballMlirTests.cmake) - -add_custom_target(base-OpTest-build) - -set(BUCKYBALL_MLIR_GROUP_TARGET base-OpTest-build) -set(BUCKYBALL_MLIR_TEST_PREFIX base) - -add_buckyball_mlir_contract(base_intrinsics_contract - PASSES ${BUCKYBALL_LOWER_BUCKYBALL}) - -add_buckyball_mlir_contract(base_bank_contract - PASSES - ${BUCKYBALL_ASSIGN_PHYSICAL_BANKS} - ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS}) - -add_buckyball_mlir_optest(base_mvin_mvout_16x16_i8 - PASSES - ${BUCKYBALL_LOWER_BUCKYBALL} - -convert-linalg-to-loops - -expand-strided-metadata - -lower-affine - -convert-scf-to-cf - -convert-cf-to-llvm - -convert-arith-to-llvm - -convert-math-to-llvm - -finalize-memref-to-llvm - -convert-func-to-llvm - -reconcile-unrealized-casts) - -add_buckyball_mlir_optest(base_bank_mvin_mvout_16x16_i8) diff --git a/bb-tests/workloads/src/tutorial/CMakeLists.txt b/bb-tests/workloads/src/tutorial/CMakeLists.txt deleted file mode 100644 index f873ed502..000000000 --- a/bb-tests/workloads/src/tutorial/CMakeLists.txt +++ /dev/null @@ -1,44 +0,0 @@ -project(tutorial C) - -#------------------------------------------------------------------------------- -# build linux version workload -#------------------------------------------------------------------------------- -set(CMAKE_C_COMPILER "riscv64-unknown-linux-gnu-gcc") -set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -march=rv64gc -static") - -add_executable(tutorial-linux ${TUTORIAL_WORKLOAD_DIR}/tutorial.c) - -add_custom_target(tutorial-linux-build ALL DEPENDS tutorial-linux - COMMENT "Building linux workloads for tutorial" - VERBATIM) - -#------------------------------------------------------------------------------- -# build baremetal version workload (BBSimHarness, same flags as toy) -#------------------------------------------------------------------------------- -set(ELF_CC "riscv64-unknown-elf-gcc") -set(BBSIM_LD ${BBSW_BAREMETAL_DIR}/bbsim.ld) - -set(C_FLAGS -std=gnu11 -g -fno-common -O2 -static -march=rv64gc -mcmodel=medany - -fno-builtin-printf -specs=nano.specs -specs=nosys.specs -nostartfiles - -Wl,-T,${BBSIM_LD} -I${WORKLOAD_LIB_DIR}) - -add_custom_target(tutorial-baremetal-build ALL - COMMAND ${ELF_CC} ${C_FLAGS} - -o tutorial-baremetal - ${BBSW_BAREMETAL_DIR}/crt0.S - ${TOY_WORKLOAD_COMMON_DIR}/buckyball.c - ${TUTORIAL_WORKLOAD_DIR}/tutorial.c - DEPENDS - ${TUTORIAL_WORKLOAD_DIR}/tutorial.c - ${BBSW_BAREMETAL_DIR}/crt0.S - ${TOY_WORKLOAD_COMMON_DIR}/buckyball.c - COMMENT "Building baremetal workloads for tutorial" - VERBATIM) - -#------------------------------------------------------------------------------- -# build all -#------------------------------------------------------------------------------- -add_custom_target(tutorial-build ALL DEPENDS - tutorial-linux-build - tutorial-baremetal-build - VERBATIM) diff --git a/bbdev b/bbdev index c1e179613..89a26182b 160000 --- a/bbdev +++ b/bbdev @@ -1 +1 @@ -Subproject commit c1e17961356fe68960c91a5997ff37c665059a6c +Subproject commit 89a26182b4c3226954cca37870dc3dfacc9ac71a diff --git a/bebop b/bebop index 013710d10..86c9180e7 160000 --- a/bebop +++ b/bebop @@ -1 +1 @@ -Subproject commit 013710d10edfa4e76afc89fbbabd5d05d51ef1c2 +Subproject commit 86c9180e7e65f800d1cfc6c1d2ad7de463f720e7 diff --git a/compiler/CMakeLists.txt b/compiler/CMakeLists.txt new file mode 100644 index 000000000..151ac17e5 --- /dev/null +++ b/compiler/CMakeLists.txt @@ -0,0 +1,176 @@ +if(NOT COMMAND add_mlir_library OR NOT COMMAND add_mlir_dialect) + message(FATAL_ERROR "MLIR CMake helpers must be loaded. Build as part of buddy-mlir.") +endif() +if(NOT DEFINED BUCKYBALL_CHIP_PB OR BUCKYBALL_CHIP_PB STREQUAL "") + message(FATAL_ERROR "BUCKYBALL_CHIP_PB is required") +endif() +if(NOT EXISTS "${BUCKYBALL_CHIP_PB}") + message(FATAL_ERROR "missing Chip.pb: ${BUCKYBALL_CHIP_PB}") +endif() + +get_filename_component(BB_REPO_ROOT "${CMAKE_CURRENT_LIST_DIR}/.." ABSOLUTE) +set(BB_COMPILER_COMMON_DIR "${BB_REPO_ROOT}/compiler/src") +set(BB_TARGET_REGISTRY_INC "${CMAKE_CURRENT_BINARY_DIR}/BuckyballTargetRegistry.inc") +set(BB_BUCKYBALL_DIALECT_TD "${CMAKE_CURRENT_BINARY_DIR}/BuckyballUnion.td") +set(BB_BALL_LOWERING_HOOKS + "${CMAKE_CURRENT_BINARY_DIR}/BuckyballBallLoweringHooks.inc") +set(BB_TARGET_REGISTRY_SOURCE + "${BB_COMPILER_COMMON_DIR}/Target/BuckyballTargetRegistry.cpp") +set(BB_TARGET_REGISTRY_GENERATOR + "${BB_REPO_ROOT}/compiler/scripts/pb_to_target_registry.py") + +find_package(Python3 REQUIRED COMPONENTS Interpreter) +execute_process( + COMMAND "${Python3_EXECUTABLE}" "${BB_TARGET_REGISTRY_GENERATOR}" + --repo "${BB_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --out "${BB_TARGET_REGISTRY_INC}" + RESULT_VARIABLE BB_TARGET_REGISTRY_RESULT + OUTPUT_VARIABLE BB_TARGET_REGISTRY_OUTPUT + ERROR_VARIABLE BB_TARGET_REGISTRY_ERROR) +if(NOT BB_TARGET_REGISTRY_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to generate Buckyball target registry:\n${BB_TARGET_REGISTRY_OUTPUT}${BB_TARGET_REGISTRY_ERROR}") +endif() +execute_process( + COMMAND "${Python3_EXECUTABLE}" "${BB_TARGET_REGISTRY_GENERATOR}" + --repo "${BB_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --lowering-hooks-out "${BB_BALL_LOWERING_HOOKS}" + RESULT_VARIABLE BB_BALL_HOOKS_RESULT + OUTPUT_VARIABLE BB_BALL_HOOKS_OUTPUT + ERROR_VARIABLE BB_BALL_HOOKS_ERROR) +if(NOT BB_BALL_HOOKS_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to generate Buckyball lowering hooks:\n${BB_BALL_HOOKS_OUTPUT}${BB_BALL_HOOKS_ERROR}") +endif() +execute_process( + COMMAND "${Python3_EXECUTABLE}" "${BB_TARGET_REGISTRY_GENERATOR}" + --repo "${BB_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --out-dialect-td "${BB_BUCKYBALL_DIALECT_TD}" + RESULT_VARIABLE BB_DIALECT_TD_RESULT + OUTPUT_VARIABLE BB_DIALECT_TD_OUTPUT + ERROR_VARIABLE BB_DIALECT_TD_ERROR) +if(NOT BB_DIALECT_TD_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to generate Buckyball union dialect:\n${BB_DIALECT_TD_OUTPUT}${BB_DIALECT_TD_ERROR}") +endif() +set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS + "${BUCKYBALL_CHIP_PB}" "${BB_TARGET_REGISTRY_GENERATOR}") + +include_directories("${BB_COMPILER_COMMON_DIR}") +include_directories("${BB_COMPILER_COMMON_DIR}/Dialect") +include_directories("${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball") +include_directories("${CMAKE_CURRENT_SOURCE_DIR}/src") +include_directories("${CMAKE_CURRENT_BINARY_DIR}") +include_directories(BEFORE "${CMAKE_CURRENT_BINARY_DIR}") +execute_process( + COMMAND "${Python3_EXECUTABLE}" "${BB_TARGET_REGISTRY_GENERATOR}" + --repo "${BB_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --print-ball-dialect-dirs + RESULT_VARIABLE BB_BALL_DIALECT_DIRS_RESULT + OUTPUT_VARIABLE BB_BALL_DIALECT_DIRS + ERROR_VARIABLE BB_BALL_DIALECT_DIRS_ERROR) +if(NOT BB_BALL_DIALECT_DIRS_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to resolve Buckyball Ball dialect directories:\n${BB_BALL_DIALECT_DIRS_ERROR}") +endif() +string(STRIP "${BB_BALL_DIALECT_DIRS}" BB_BALL_DIALECT_DIRS) +string(REPLACE "\n" ";" BB_BALL_DIALECT_DIRS "${BB_BALL_DIALECT_DIRS}") +foreach(BB_BALL_DIALECT_DIR ${BB_BALL_DIALECT_DIRS}) + include_directories("${BB_BALL_DIALECT_DIR}") +endforeach() + +execute_process( + COMMAND "${Python3_EXECUTABLE}" "${BB_TARGET_REGISTRY_GENERATOR}" + --repo "${BB_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --print-core-compiler-dirs + RESULT_VARIABLE BB_CORE_COMPILER_DIR_RESULT + OUTPUT_VARIABLE BB_CORE_COMPILER_DIRS + ERROR_VARIABLE BB_CORE_COMPILER_DIR_ERROR) +if(NOT BB_CORE_COMPILER_DIR_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to resolve Core compiler package:\n${BB_CORE_COMPILER_DIR_ERROR}") +endif() +string(STRIP "${BB_CORE_COMPILER_DIRS}" BB_CORE_COMPILER_DIRS) +string(REPLACE "\n" ";" BB_CORE_COMPILER_DIRS "${BB_CORE_COMPILER_DIRS}") +list(LENGTH BB_CORE_COMPILER_DIRS BB_CORE_COMPILER_DIR_COUNT) +if(BB_CORE_COMPILER_DIR_COUNT EQUAL 1) + list(GET BB_CORE_COMPILER_DIRS 0 BB_CORE_COMPILER_DIR) + get_filename_component(BB_CORE_COMPILER_NAME "${BB_CORE_COMPILER_DIR}" NAME) + add_subdirectory("${BB_CORE_COMPILER_DIR}" + "${CMAKE_CURRENT_BINARY_DIR}/cores/${BB_CORE_COMPILER_NAME}") +else() + # A heterogeneous chip owns more than one Core lowering policy. Build its + # dialect and public passes once; the selected target chooses its Ball set + # from Chip.pb at pass execution time. + foreach(BB_BALL_SOURCE_KIND legalize assign bank_ssa tile) + execute_process( + COMMAND "${Python3_EXECUTABLE}" "${BB_TARGET_REGISTRY_GENERATOR}" + --repo "${BB_REPO_ROOT}" + --chip-pb "${BUCKYBALL_CHIP_PB}" + --print-ball-compiler-paths "${BB_BALL_SOURCE_KIND}" + RESULT_VARIABLE BB_BALL_SOURCE_RESULT + OUTPUT_VARIABLE BB_BALL_SOURCE_PATHS + ERROR_VARIABLE BB_BALL_SOURCE_ERROR) + if(NOT BB_BALL_SOURCE_RESULT EQUAL 0) + message(FATAL_ERROR + "failed to resolve ${BB_BALL_SOURCE_KIND} Ball compiler sources:\n${BB_BALL_SOURCE_ERROR}") + endif() + string(STRIP "${BB_BALL_SOURCE_PATHS}" BB_BALL_SOURCE_PATHS) + string(REPLACE "\n" ";" BB_${BB_BALL_SOURCE_KIND}_SOURCES + "${BB_BALL_SOURCE_PATHS}") + endforeach() + + set(LLVM_TARGET_DEFINITIONS "${BB_BUCKYBALL_DIALECT_TD}") + mlir_tablegen(Buckyball.h.inc -gen-op-decls) + mlir_tablegen(Buckyball.cpp.inc -gen-op-defs) + mlir_tablegen(BuckyballTypes.h.inc -gen-typedef-decls -typedefs-dialect=buckyball) + mlir_tablegen(BuckyballTypes.cpp.inc -gen-typedef-defs -typedefs-dialect=buckyball) + mlir_tablegen(BuckyballDialect.h.inc -gen-dialect-decls -dialect=buckyball) + mlir_tablegen(BuckyballDialect.cpp.inc -gen-dialect-defs -dialect=buckyball) + add_public_tablegen_target(MLIRBuckyballIncGen) + mlir_tablegen(BuckyballConversions.inc -gen-llvmir-conversions) + add_public_tablegen_target(BuddyBuckyballConversionsIncGen) + + add_mlir_dialect_library(BuddyBuckyball + "${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/IR/BuckyballDialect.cpp" + "${BB_TARGET_REGISTRY_SOURCE}" + LINK_LIBS PUBLIC MLIRIR MLIRLLVMDialect) + target_include_directories(BuddyBuckyball PUBLIC + "${BB_COMPILER_COMMON_DIR}" + "${BB_COMPILER_COMMON_DIR}/Dialect" + "${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball" + "${CMAKE_CURRENT_BINARY_DIR}") + + add_mlir_dialect_library(BuddyBuckyballTransforms + "${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp" + ${BB_legalize_SOURCES} + "${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp" + LINK_LIBS PUBLIC BuddyBuckyball MLIRIR MLIRLLVMCommonConversion MLIRLLVMDialect) + + add_mlir_library(LowerBuckyballPass + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp" + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/PhysicalBankState.cpp" + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/ReportBankUsagePass.cpp" + ${BB_assign_SOURCES} + ${BB_bank_ssa_SOURCES} + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp" + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/LowerBuckyballPass.cpp" + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp" + LINK_LIBS PUBLIC BuddyBuckyballTransforms MLIRLinalgDialect) + + add_mlir_library(LowerTileToBuckyballPass + "${BB_COMPILER_COMMON_DIR}/Conversion/LowerTileToBuckyball/LowerTileToBuckyballPass.cpp" + ${BB_tile_SOURCES} + LINK_LIBS PUBLIC BuddyTile BuddyBuckyball MLIRLinalgDialect) + + add_mlir_translation_library(BuddyBuckyballToLLVMIRTranslation + "${BB_COMPILER_COMMON_DIR}/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp" + DEPENDS BuddyBuckyballConversionsIncGen intrinsics_gen + LINK_COMPONENTS Core + LINK_LIBS PUBLIC MLIRIR MLIRSupport BuddyBuckyball) +endif() diff --git a/compiler/scripts/pb_to_target_registry.py b/compiler/scripts/pb_to_target_registry.py new file mode 100644 index 000000000..d2c21eed2 --- /dev/null +++ b/compiler/scripts/pb_to_target_registry.py @@ -0,0 +1,432 @@ +#!/usr/bin/env python3 +"""Generate the static Buckyball compiler target registry from Chip.pb.""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + + +def _die(message: str) -> None: + raise ValueError(message) + + +def _write(path: Path, content: str) -> None: + if path.is_file() and path.read_text(encoding="utf-8") == content: + return + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + + +def _load_proto(repo: Path): + scripts = repo / "bbdev" / "api" / "steps" / "config" / "scripts" + if not scripts.is_dir(): + _die(f"missing protobuf bindings directory: {scripts}") + sys.path.insert(0, str(scripts)) + import chip_pb2 # type: ignore + + return chip_pb2 + + +def _target_name(core) -> str: + return core.role or core.pkg + + +def _cxx_string(value: str) -> str: + if not value: + _die("empty string is not a valid compiler target field") + if '"' in value or "\\" in value: + _die(f"unsupported C++ string in chip.pb: {value!r}") + return f'"{value}"' + + +def _profile_core(chip, profile): + matches = [core for core in chip.cores if _target_name(core) == profile.name] + if not matches: + _die(f"profile {profile.name}: no matching CoreInstance") + first = matches[0] + expected_isa = [ + (entry.mnemonic, entry.funct7, entry.bid) for entry in first.balldomain.isa + ] + expected_balls = [entry.ball_name for entry in first.balldomain.mappings] + expected_bank = (first.mem.bank.num, first.mem.bank.width, first.mem.bank.entries) + profile_bank = (profile.bank_num, profile.bank_width, profile.bank_entries) + if expected_bank != profile_bank: + _die( + f"profile {profile.name}: profile bank={profile_bank} disagrees " + f"with CoreInstance bank={expected_bank}" + ) + for core in matches[1:]: + isa = [ + (entry.mnemonic, entry.funct7, entry.bid) for entry in core.balldomain.isa + ] + balls = [entry.ball_name for entry in core.balldomain.mappings] + bank = (core.mem.bank.num, core.mem.bank.width, core.mem.bank.entries) + if isa != expected_isa or balls != expected_balls or bank != expected_bank: + _die(f"profile {profile.name}: CoreInstance configuration mismatch") + return first + + +def _validate_profile(profile, core) -> None: + if not profile.name.isidentifier(): + _die(f"profile name is not a C++ identifier: {profile.name!r}") + has_balls = len(core.balldomain.mappings) > 0 + if has_balls and ( + profile.bank_num == 0 or profile.bank_width == 0 or profile.bank_entries == 0 + ): + _die(f"profile {profile.name}: bank geometry must be non-zero") + + balls = [entry.ball_name for entry in core.balldomain.mappings] + if len(set(balls)) != len(balls): + _die(f"profile {profile.name}: duplicate Ball mapping") + + mnemonics: set[str] = set() + funct7s: set[int] = set() + for entry in core.balldomain.isa: + if not entry.mnemonic.isidentifier(): + _die(f"profile {profile.name}: invalid ISA mnemonic {entry.mnemonic!r}") + if entry.mnemonic in mnemonics: + _die(f"profile {profile.name}: duplicate ISA mnemonic {entry.mnemonic}") + if entry.funct7 in funct7s: + _die(f"profile {profile.name}: duplicate funct7 {entry.funct7}") + mnemonics.add(entry.mnemonic) + funct7s.add(entry.funct7) + + +def _core_compiler_dirs(chip, repo: Path) -> list[Path]: + """Resolve the Core compiler packages selected by Chip.pb.""" + result: list[Path] = [] + seen: set[str] = set() + for profile in chip.profiles: + core = _profile_core(chip, profile) + core_name = core.pkg + if not core_name.isidentifier(): + _die(f"profile {profile.name}: invalid core package {core_name!r}") + if core_name in seen: + continue + compiler_dir = repo / "examples" / "cores" / core_name / "compiler" + if not (compiler_dir / "CMakeLists.txt").is_file(): + _die(f"Core {core_name}: missing compiler package {compiler_dir}") + result.append(compiler_dir) + seen.add(core_name) + return result + + +def _emit(chip, target: str | None = None) -> str: + profiles = [ + profile for profile in chip.profiles if target is None or profile.name == target + ] + if not profiles: + if target is None: + _die("Chip.pb has no compiler profiles") + _die(f"Chip.pb has no compiler profile {target!r}") + names: set[str] = set() + chunks = ["// Generated from Chip.pb. Do not edit.\n"] + targets: list[str] = [] + for profile in profiles: + if profile.name in names: + _die(f"duplicate compiler profile {profile.name}") + names.add(profile.name) + core = _profile_core(chip, profile) + _validate_profile(profile, core) + stem = profile.name + mappings = list(core.balldomain.mappings) + isa = list(core.balldomain.isa) + if mappings: + chunks.append(f"static const llvm::StringRef k{stem}Balls[] = {{") + chunks.extend(f" {_cxx_string(entry.ball_name)}," for entry in mappings) + chunks.append("};") + chunks.append( + f"static const buckyball_target::BuckyballBallMapping k{stem}BallMappings[] = {{" + ) + chunks.extend( + f" {{{_cxx_string(entry.ball_name)}, {entry.in_bw}, {entry.out_bw}}}," + for entry in mappings + ) + chunks.append("};") + balls_ref = f"llvm::ArrayRef(k{stem}Balls)" + mappings_ref = f"llvm::ArrayRef(k{stem}BallMappings)" + else: + balls_ref = "llvm::ArrayRef()" + mappings_ref = "llvm::ArrayRef()" + if isa: + chunks.append( + f"static const buckyball_target::BuckyballIsaEntry k{stem}Isa[] = {{" + ) + chunks.extend( + f" {{{_cxx_string(entry.mnemonic)}, {entry.funct7}}}," for entry in isa + ) + chunks.append("};") + isa_ref = f"llvm::ArrayRef(k{stem}Isa)" + else: + isa_ref = "llvm::ArrayRef()" + chunks.append("") + targets.append( + " {" + f"{_cxx_string(profile.name)}, {_cxx_string(core.pkg)}, {profile.bank_num}, " + f"{profile.bank_width}, {profile.bank_entries}, " + f"{balls_ref}, {mappings_ref}, {isa_ref}" + "}," + ) + chunks.append( + "static const buckyball_target::BuckyballTargetConfig kBuckyballTargets[] = {" + ) + chunks.extend(targets) + chunks.append("};\n") + return "\n".join(chunks) + + +def _emit_isa_headers(chip, isa_dir: Path) -> None: + """Emit one C ISA header per compiler target. + + A header describes one core ISA only. Deliberately do not create a merged + chip header: funct7 is target-local and may overlap across profiles. + """ + for profile in chip.profiles: + core = _profile_core(chip, profile) + _validate_profile(profile, core) + lines = ["#ifndef BALL_ISA_H", "#define BALL_ISA_H", ""] + lines.extend( + f"#define BB_FUNC7_{entry.mnemonic} {entry.funct7}" + for entry in core.balldomain.isa + ) + lines.extend(["", "#endif", ""]) + header = isa_dir / profile.name / "ballISA.h" + _write(header, "\n".join(lines)) + + +def _emit_dialect_td(chip, repo: Path) -> str: + """Emit the union of Ball operation definitions selected by this Chip.""" + ball_dirs: list[str] = [] + seen: set[str] = set() + for profile in chip.profiles: + core = _profile_core(chip, profile) + for entry in core.balldomain.mappings: + ball_dir = entry.ball_dir + if not ball_dir: + _die(f"profile {profile.name}: {entry.ball_name} has no ball_dir") + if ball_dir in seen: + continue + td_dir = ( + repo + / "examples" + / "balls" + / ball_dir + / "compiler" + / "src" + / "Dialect" + / "Buckyball" + ) + td_files = sorted(td_dir.glob("*.td")) + if len(td_files) != 1: + _die(f"Ball {ball_dir}: expected one dialect TD file in {td_dir}") + seen.add(ball_dir) + ball_dirs.append(td_files[0].name) + + lines = ["// Generated from Chip.pb. Do not edit.", 'include "Buckyball.td"'] + lines.extend(f'include "{td_file}"' for td_file in ball_dirs) + lines.append("") + return "\n".join(lines) + + +def _ball_dialect_dirs(chip, repo: Path) -> list[Path]: + result: list[Path] = [] + seen: set[str] = set() + for profile in chip.profiles: + core = _profile_core(chip, profile) + for entry in core.balldomain.mappings: + ball_dir = entry.ball_dir + if not ball_dir or ball_dir in seen: + continue + result.append( + repo + / "examples" + / "balls" + / ball_dir + / "compiler" + / "src" + / "Dialect" + / "Buckyball" + ) + seen.add(ball_dir) + return result + + +def _ball_compilers(chip, repo: Path) -> list[dict[str, object]]: + """Resolve convention-based compiler contributions from selected Balls.""" + result: list[dict[str, object]] = [] + seen: dict[str, str] = {} + for profile in chip.profiles: + core = _profile_core(chip, profile) + _validate_profile(profile, core) + for entry in core.balldomain.mappings: + ball_name = entry.ball_name + ball_dir = entry.ball_dir + if not ball_name.isidentifier(): + _die(f"profile {profile.name}: invalid Ball name {ball_name!r}") + if not ball_dir: + _die(f"profile {profile.name}: {ball_name} has no ball_dir") + if Path(ball_dir).is_absolute() or len(Path(ball_dir).parts) != 1: + _die(f"profile {profile.name}: invalid ball_dir {ball_dir!r}") + if ball_name in seen: + if seen[ball_name] != ball_dir: + _die(f"Ball {ball_name} maps to multiple directories") + continue + + source_dir = repo / "examples" / "balls" / ball_dir / "compiler" / "src" + dialect_dir = source_dir / "Dialect" / "Buckyball" + td_files = sorted(dialect_dir.glob("*.td")) + if len(td_files) != 1: + _die(f"Ball {ball_name}: expected one dialect TD in {dialect_dir}") + legalize = dialect_dir / "Transforms" / "LegalizeForLLVMExport.cpp" + if not legalize.is_file(): + _die(f"Ball {ball_name}: missing LLVM export lowering {legalize}") + lower = source_dir / "Conversion" / "LowerBuckyball" + tile_dir = source_dir / "Conversion" / "LowerTileToBuckyball" + result.append( + { + "name": ball_name, + "dialect_dir": dialect_dir, + "legalize": legalize, + "assign": lower / "AssignPhysicalBankPatterns.cpp", + "bank_ssa": lower / "LowerBuckyballToBankSSAPatterns.cpp", + "tile": sorted(tile_dir.glob("*.cpp")) if tile_dir.is_dir() else [], + } + ) + seen[ball_name] = ball_dir + return result + + +def _emit_lowering_hooks(chip, repo: Path) -> str: + balls = _ball_compilers(chip, repo) + lines = ["// Generated from Chip.pb. Do not edit.", ""] + stages = ( + ("BUCKYBALL_LEGALIZE_HOOK", "legalize"), + ("BUCKYBALL_ASSIGN_HOOK", "assign"), + ("BUCKYBALL_TILE_HOOK", "tile"), + ("BUCKYBALL_BANK_SSA_HOOK", "bank_ssa"), + ) + for macro, field in stages: + lines.append(f"#ifdef {macro}") + for ball in balls: + value = ball[field] + if ( + field == "legalize" + or (isinstance(value, list) and value) + or (isinstance(value, Path) and value.is_file()) + ): + lines.append(f"{macro}({ball['name']})") + lines.extend(["#endif", ""]) + return "\n".join(lines) + + +def _print_ball_compiler_paths(chip, repo: Path, kind: str) -> None: + for ball in _ball_compilers(chip, repo): + if kind == "dialect-dir": + print(ball["dialect_dir"]) + elif kind == "legalize": + print(ball["legalize"]) + elif kind == "tile": + for source in ball["tile"]: + print(source) + else: + source = ball[kind] + if isinstance(source, Path) and source.is_file(): + print(source) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--repo", type=Path, required=True) + parser.add_argument("--chip-pb", type=Path, required=True) + parser.add_argument("--out", type=Path) + parser.add_argument("--out-dialect-td", type=Path) + parser.add_argument("--lowering-hooks-out", type=Path) + parser.add_argument("--isa-dir", type=Path) + parser.add_argument("--target") + parser.add_argument("--print-core-compiler-dirs", action="store_true") + parser.add_argument("--print-targets", action="store_true") + parser.add_argument("--print-bank-targets", action="store_true") + parser.add_argument("--print-target-balls", action="store_true") + parser.add_argument("--print-core-targets", action="store_true") + parser.add_argument("--print-ball-dialect-dirs", action="store_true") + parser.add_argument( + "--print-ball-compiler-paths", + choices=("dialect-dir", "legalize", "assign", "bank_ssa", "tile"), + ) + args = parser.parse_args() + + if ( + args.out is None + and args.out_dialect_td is None + and args.lowering_hooks_out is None + and args.isa_dir is None + and not args.print_core_compiler_dirs + and not args.print_targets + and not args.print_bank_targets + and not args.print_target_balls + and not args.print_core_targets + and not args.print_ball_dialect_dirs + and args.print_ball_compiler_paths is None + ): + _die("one output mode is required") + + repo = args.repo.resolve() + if not repo.is_dir(): + _die(f"repository does not exist: {repo}") + pb_path = args.chip_pb.resolve() + if not pb_path.is_file(): + _die(f"missing Chip.pb: {pb_path}") + pb = _load_proto(repo) + chip = pb.Chip() + chip.ParseFromString(pb_path.read_bytes()) + if args.out: + _write(args.out, _emit(chip, args.target)) + if args.out_dialect_td: + _write(args.out_dialect_td, _emit_dialect_td(chip, repo)) + if args.lowering_hooks_out: + _write(args.lowering_hooks_out, _emit_lowering_hooks(chip, repo)) + if args.isa_dir: + _emit_isa_headers(chip, args.isa_dir) + if args.print_targets: + for profile in chip.profiles: + print(profile.name) + if args.print_bank_targets: + for profile in chip.profiles: + if profile.bank_num > 0: + print(profile.name) + if args.print_target_balls: + for profile in chip.profiles: + core = _profile_core(chip, profile) + _validate_profile(profile, core) + dirs = [] + seen = set() + for entry in core.balldomain.mappings: + ball_dir = entry.ball_dir + if not ball_dir: + _die(f"profile {profile.name}: {entry.ball_name} has no ball_dir") + if ball_dir in seen: + _die(f"profile {profile.name}: duplicate ball_dir {ball_dir!r}") + seen.add(ball_dir) + dirs.append(ball_dir) + print(f"{profile.name}:{','.join(dirs)}") + if args.print_core_targets: + targets = {profile.name for profile in chip.profiles} + for core in chip.cores: + target = _target_name(core) + if target not in targets: + _die(f"CoreInstance {core.index}: no compiler profile {target}") + print(f"{core.index}:{target}") + if args.print_ball_dialect_dirs: + for dialect_dir in _ball_dialect_dirs(chip, repo): + print(dialect_dir) + if args.print_ball_compiler_paths: + _print_ball_compiler_paths(chip, repo, args.print_ball_compiler_paths) + if args.print_core_compiler_dirs: + for compiler_dir in _core_compiler_dirs(chip, repo): + print(compiler_dir) + + +if __name__ == "__main__": + main() diff --git a/compiler/src/CMakeLists.txt b/compiler/src/CMakeLists.txt deleted file mode 100644 index 2ad9831f9..000000000 --- a/compiler/src/CMakeLists.txt +++ /dev/null @@ -1,3 +0,0 @@ -message(FATAL_ERROR - "compiler/src is common Buckyball compiler code and is not a selectable " - "external dialect package. Use examples/chips//compiler.") diff --git a/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp b/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp new file mode 100644 index 000000000..487f280af --- /dev/null +++ b/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp @@ -0,0 +1,60 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" +#include "Target/BuckyballTargetRegistry.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/WalkPatternRewriteDriver.h" + +#include "Buckyball/BuckyballDialect.h" + +using namespace mlir; + +namespace mlir::buddy { +#define BUCKYBALL_ASSIGN_HOOK(BALL) \ + void populate##BALL##AssignPhysicalBankPatterns(RewritePatternSet &, \ + PhysicalBankState &); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_ASSIGN_HOOK +} // namespace mlir::buddy + +namespace { +class AssignPhysicalBanksPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(AssignPhysicalBanksPass) + + StringRef getArgument() const final { return "assign-physical-banks"; } + StringRef getDescription() const final { + return "Assign physical banks for the selected Buckyball target."; + } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + const buckyball_target::BuckyballTargetConfig &target = + buckyball_target::getBuckyballTarget(); + func::FuncOp func = getOperation(); + mlir::buddy::PhysicalBankState state(target.bankNum); + RewritePatternSet patterns(&getContext()); + mlir::buddy::addBaseAssignPhysicalBankPatterns(patterns, state); + for (llvm::StringRef ball : target.balls) { +#define BUCKYBALL_ASSIGN_HOOK(BALL) \ + if (ball == #BALL) \ + mlir::buddy::populate##BALL##AssignPhysicalBankPatterns(patterns, state); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_ASSIGN_HOOK + } + walkAndApplyPatterns(func, std::move(patterns)); + if (failed(mlir::buddy::verifyNoBankSSAOps(func)) || !state.empty()) + signalPassFailure(); + } +}; +} // namespace + +void mlir::buddy::registerAssignPhysicalBanksPass() { + PassRegistration(); +} diff --git a/compiler/src/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp b/compiler/src/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp index 9164b885f..b58e7595f 100644 --- a/compiler/src/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp +++ b/compiler/src/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp @@ -83,9 +83,10 @@ class BankMvoutPattern : public OpRewritePattern { LogicalResult matchAndRewrite(BankMvoutOp op, PatternRewriter &rewriter) const override { + // Fence is CPU↔NPU sync, not NPU-internal. Emit FenceOp once at the end of + // a large Buckyball op (matmul/im2col/transpose), not after every mvout. rewriter.create(op.getLoc(), op.getOutput(), op.getBank(), op.getDepth(), op.getStride()); - rewriter.create(op.getLoc()); rewriter.replaceOp(op, op.getBank()); return success(); } diff --git a/compiler/src/Conversion/LowerBuckyball/LowerBuckyball.h b/compiler/src/Conversion/LowerBuckyball/LowerBuckyball.h index 9f19d4f94..991cc5870 100644 --- a/compiler/src/Conversion/LowerBuckyball/LowerBuckyball.h +++ b/compiler/src/Conversion/LowerBuckyball/LowerBuckyball.h @@ -49,15 +49,15 @@ class PhysicalBankState { LogicalResult verifyNoBankSSAOps(Operation *root); void addBaseAssignPhysicalBankPatterns(RewritePatternSet &patterns, PhysicalBankState &state); +void populateSMatMulBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns); +void populateReluBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns); void registerAssignPhysicalBanksPass(); -void registerLowerBankSSAToIntrinsicsPass(); void registerLowerBuckyballPass(); +void registerLowerBankSSAToIntrinsicsPass(); void registerLowerBuckyballToBankSSAPass(); void registerReportBankUsagePass(); -void populatePebbleMatrixMatmulToBankSSAPatterns(RewritePatternSet &patterns); -void populatePebbleIm2colMatmulToBankSSAPatterns(RewritePatternSet &patterns); -void populatePebbleMemTransposeToBankSSAPatterns(RewritePatternSet &patterns); -void populatePebbleLowerBuckyballToBankSSAPatterns(RewritePatternSet &patterns); } // namespace buddy } // namespace mlir diff --git a/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp b/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp new file mode 100644 index 000000000..d485307c9 --- /dev/null +++ b/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp @@ -0,0 +1,120 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" +#include "Target/BuckyballTargetRegistry.h" + +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/TypeConverter.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/ControlFlow/IR/ControlFlowOps.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/LLVMIR/LLVMDialect.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/Pass/Pass.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" +#include "Buckyball/Transform.h" + +using namespace mlir; + +namespace { +class LowerBuckyballToLLVMPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBuckyballToLLVMPass) + LowerBuckyballToLLVMPass() = default; + LowerBuckyballToLLVMPass(const LowerBuckyballToLLVMPass &) {} + + StringRef getArgument() const final { return "lower-buckyball"; } + StringRef getDescription() const final { + return "Lower Buckyball operations for the selected target."; + } + + Option stable{*this, "stable", + llvm::cl::desc("Use stable LLVM Buckyball intrinsics."), + llvm::cl::init(false)}; + Option rushB{*this, "rushb", + llvm::cl::desc("Lower DMA operations to the rushB ABI."), + llvm::cl::init(false)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + const buckyball_target::BuckyballTargetConfig &targetConfig = + buckyball_target::getBuckyballTarget(); + LLVMTypeConverter converter(&getContext()); + RewritePatternSet patterns(&getContext()); + LLVMConversionTarget target(getContext()); + configureBuckyballLegalizeForExportTarget(target, stable); + target.addLegalDialect(); + populateBuckyballLegalizeForLLVMExportPatterns( + converter, patterns, targetConfig.bankWidthBits / 8, + targetConfig.bankDepth, targetConfig.bankNum, + /*includeFuncOperandForwarding=*/false, stable, rushB); + ConversionConfig config; + config.allowPatternRollback = false; + if (failed(applyPartialConversion(getOperation(), target, + std::move(patterns), config))) + signalPassFailure(); + } +}; + +class LowerBankSSAToIntrinsicsPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBankSSAToIntrinsicsPass) + LowerBankSSAToIntrinsicsPass() = default; + LowerBankSSAToIntrinsicsPass(const LowerBankSSAToIntrinsicsPass &) {} + + StringRef getArgument() const final { return "lower-bank-ssa-to-intrinsics"; } + StringRef getDescription() const final { + return "Lower bank-SSA operations for the selected Buckyball target."; + } + + Option stable{*this, "stable", + llvm::cl::desc("Use stable LLVM Buckyball intrinsics."), + llvm::cl::init(false)}; + Option rushB{*this, "rushb", + llvm::cl::desc("Lower DMA operations to the rushB ABI."), + llvm::cl::init(false)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + const buckyball_target::BuckyballTargetConfig &targetConfig = + buckyball_target::getBuckyballTarget(); + LLVMTypeConverter converter(&getContext()); + RewritePatternSet patterns(&getContext()); + LLVMConversionTarget target(getContext()); + configureBuckyballLegalizeForExportTarget(target, stable); + target.addLegalDialect(); + populateBuckyballLegalizeForLLVMExportPatterns( + converter, patterns, targetConfig.bankWidthBits / 8, + targetConfig.bankDepth, targetConfig.bankNum, + /*includeFuncOperandForwarding=*/false, stable, rushB); + ConversionConfig config; + config.allowPatternRollback = false; + if (failed(applyPartialConversion(getOperation(), target, + std::move(patterns), config))) + signalPassFailure(); + } +}; +} // namespace + +void mlir::buddy::registerLowerBuckyballPass() { + PassRegistration(); +} + +void mlir::buddy::registerLowerBankSSAToIntrinsicsPass() { + PassRegistration(); +} diff --git a/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp b/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp new file mode 100644 index 000000000..36cf2b87b --- /dev/null +++ b/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp @@ -0,0 +1,62 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" +#include "Target/BuckyballTargetRegistry.h" + +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/GreedyPatternRewriteDriver.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; + +namespace mlir::buddy { +#define BUCKYBALL_BANK_SSA_HOOK(BALL) \ + void populate##BALL##LowerBuckyballToBankSSAPatterns(RewritePatternSet &); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_BANK_SSA_HOOK +#define BUCKYBALL_CORE_BANK_SSA_HOOK(CORE, NAME) \ + void populate##CORE##CoreBankSSALoweringPatterns(RewritePatternSet &); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_CORE_BANK_SSA_HOOK +} // namespace mlir::buddy + +namespace { +class LowerBuckyballToBankSSAPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBuckyballToBankSSAPass) + + StringRef getArgument() const final { return "lower-buckyball-to-bank-ssa"; } + StringRef getDescription() const final { + return "Lower Buckyball operations to explicit bank-SSA operations."; + } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry.insert<::buddy::buckyball::BuckyballDialect>(); + } + + void runOnOperation() override { + RewritePatternSet patterns(&getContext()); + for (llvm::StringRef ball : buckyball_target::getBuckyballTarget().balls) { +#define BUCKYBALL_BANK_SSA_HOOK(BALL) \ + if (ball == #BALL) \ + mlir::buddy::populate##BALL##LowerBuckyballToBankSSAPatterns(patterns); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_BANK_SSA_HOOK + } +#define BUCKYBALL_CORE_BANK_SSA_HOOK(CORE, NAME) \ + if (buckyball_target::getBuckyballTarget().core == NAME) \ + mlir::buddy::populate##CORE##CoreBankSSALoweringPatterns(patterns); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_CORE_BANK_SSA_HOOK + if (failed(applyPatternsGreedily(getOperation(), std::move(patterns)))) + signalPassFailure(); + } +}; +} // namespace + +void mlir::buddy::registerLowerBuckyballToBankSSAPass() { + PassRegistration(); +} diff --git a/compiler/src/Conversion/LowerBuckyball/PhysicalBankState.cpp b/compiler/src/Conversion/LowerBuckyball/PhysicalBankState.cpp index 52e4d2bc1..6bcfadbec 100644 --- a/compiler/src/Conversion/LowerBuckyball/PhysicalBankState.cpp +++ b/compiler/src/Conversion/LowerBuckyball/PhysicalBankState.cpp @@ -30,37 +30,25 @@ std::optional PhysicalBankState::getConstI64(Value value) const { value = op.getBank(); continue; } - if (auto op = value.getDefiningOp()) { - value = op.getOutBank(); - continue; - } - if (auto op = value.getDefiningOp()) { - value = op.getOutBank(); - continue; - } - if (auto op = value.getDefiningOp()) { - value = op.getOutBank(); - continue; - } - if (auto op = value.getDefiningOp()) { - value = op.getOutBank(); - continue; - } - // Im2col is an optional Ball. Keep the generic allocator independent of - // its generated C++ type so a Core can omit that dialect entirely. - if (Operation *op = value.getDefiningOp(); - op && op->getName().getStringRef() == "buckyball.bank_im2col") { - value = op->getResult(0); - continue; - } - if (auto op = value.getDefiningOp()) { - value = op.getWrBank(); - continue; - } - if (Operation *op = value.getDefiningOp(); - op && op->getName().getStringRef() == "buckyball.bank_mul_warp16") { - value = op->getOperand(2); - continue; + // Optional Balls: use op names so Cores can omit generated C++ types. + if (Operation *op = value.getDefiningOp()) { + StringRef name = op->getName().getStringRef(); + if (name == "buckyball.bank_transpose" || + name == "buckyball.bank_fp2int" || + name == "buckyball.bank_int2fp_tensor" || + name == "buckyball.bank_int2fp_channel") { + value = op->getOperand(1); + continue; + } + if (name == "buckyball.bank_im2col") { + value = op->getResult(0); + continue; + } + if (name == "buckyball.bank_smatmul" || + name == "buckyball.bank_vecmat16") { + value = op->getOperand(2); + continue; + } } if (auto forOp = value.getDefiningOp()) { unsigned resultNumber = cast(value).getResultNumber(); diff --git a/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h b/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h index 1d52397b1..83799a5d2 100644 --- a/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h +++ b/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h @@ -25,28 +25,24 @@ namespace mlir { namespace buddy { inline constexpr int64_t kDefaultBankWidthBytes = 16; -inline constexpr size_t kMatmulTile = 16; -inline constexpr size_t kMaxAccMvoutDepthLines = 256; -inline constexpr size_t kMaxI8MvinDepthLines = 1024; +inline constexpr size_t kBankLane = 16; +inline constexpr size_t kMaxMvinDepth8b = 1024; +inline constexpr size_t kMaxMvoutDepth32b = 256; inline size_t ceilDiv(size_t a, size_t b) { return (a + b - 1) / b; } inline size_t aMvinDepthLines(size_t mEl, size_t kEl) { - return mEl * (kEl / 16); + return mEl * (kEl / kBankLane); } inline size_t bMvinDepthLines(size_t kEl, size_t nEl) { - return kEl * (nEl / 16); + return kEl * (nEl / kBankLane); } inline size_t cMvoutDepthLines(size_t mEl, size_t nEl) { - return mEl * (nEl / 16); + return mEl * (nEl / kBankLane); } -void populateMatrixTileMatMulPatterns(RewritePatternSet &patterns, - int64_t bankWidthBytes, int64_t bankDepth, - int64_t bankNum); - void registerLowerTileToBuckyballPass(); } // namespace buddy diff --git a/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyballPass.cpp b/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyballPass.cpp new file mode 100644 index 000000000..83ca96f93 --- /dev/null +++ b/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyballPass.cpp @@ -0,0 +1,75 @@ +#include "Target/BuckyballTargetRegistry.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/DialectConversion.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" +#include "Tile/TileDialect.h" +#include "Tile/TileOps.h" + +using namespace mlir; +namespace tile = ::buddy::tile; + +namespace mlir::buddy { +#define BUCKYBALL_TILE_HOOK(BALL) \ + void populate##BALL##TileLoweringPatterns(RewritePatternSet &, int64_t, \ + int64_t, int64_t); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_TILE_HOOK +} // namespace mlir::buddy + +namespace { +class LowerTileToBuckyballPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerTileToBuckyballPass) + + StringRef getArgument() const final { return "convert-tile-to-buckyball"; } + StringRef getDescription() const final { + return "Convert Tile operations for the selected Buckyball target."; + } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + const auto &targetConfig = buckyball_target::getBuckyballTarget(); + MLIRContext *context = &getContext(); + ConversionTarget target(*context); + target.addLegalDialect<::buddy::buckyball::BuckyballDialect, + memref::MemRefDialect, arith::ArithDialect, + scf::SCFDialect, func::FuncDialect, + linalg::LinalgDialect>(); + target.addIllegalOp(); + + RewritePatternSet patterns(context); + for (llvm::StringRef ball : targetConfig.balls) { +#define BUCKYBALL_TILE_HOOK(BALL) \ + if (ball == #BALL) \ + mlir::buddy::populate##BALL##TileLoweringPatterns( \ + patterns, targetConfig.bankWidthBits / 8, targetConfig.bankDepth, \ + targetConfig.bankNum); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_TILE_HOOK + } + if (failed(applyPartialConversion(getOperation(), target, + std::move(patterns)))) + signalPassFailure(); + } +}; +} // namespace + +void mlir::buddy::registerLowerTileToBuckyballPass() { + PassRegistration(); +} diff --git a/compiler/src/Dialect/Buckyball/Buckyball.td b/compiler/src/Dialect/Buckyball/Buckyball.td index 6f135ad3a..366b2a854 100644 --- a/compiler/src/Dialect/Buckyball/Buckyball.td +++ b/compiler/src/Dialect/Buckyball/Buckyball.td @@ -1,4 +1,4 @@ -//===-- Buckyball.td - Buckyball base dialect operations -*- tablegen -*-===// +//===-- Buckyball.td - Buckyball dialect operations -----*- tablegen -*-===// // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. @@ -14,13 +14,218 @@ // //===----------------------------------------------------------------------===// -#ifndef BUCKYBALL_BASE_OPS -#define BUCKYBALL_BASE_OPS +#ifndef BUCKYBALL_OPS +#define BUCKYBALL_OPS -include "BuckyballBase.td" -include "BuckyballBankSSA.td" -include "BuckyballIntrinsics.td" -include "BuckyballCarrier.td" -include "BuckyballDebug.td" +include "mlir/Dialect/LLVMIR/LLVMOpBase.td" +include "mlir/IR/AttrTypeBase.td" +include "mlir/IR/PatternBase.td" -#endif // BUCKYBALL_BASE_OPS +//===----------------------------------------------------------------------===// +// Dialect and common operations +//===----------------------------------------------------------------------===// + +def Buckyball_Dialect : Dialect { + let name = "buckyball"; + let cppNamespace = "::buddy::buckyball"; + let summary = "Dialect for RISC-V Buckyball NPU extension"; + let description = [{ + Buckyball is an NPU Design Framework with composable Ball units. + Each Ball is a specialized compute unit that can be independently + developed and combined with others. + }]; +} + +class Buckyball_Op traits = []> : + Op {} + +def MvinOp : Buckyball_Op<"mvin"> { + let summary = "Load data from memory to scratchpad bank"; + let description = [{ + Loads data from DRAM to scratchpad bank. + Corresponds to bb_mvin instruction (funct7=33). + }]; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, + I64:$addr, I64:$depth, I64:$stride); + let assemblyFormat = "$input $addr $depth $stride attr-dict `:` " + "type($input) type($addr) type($depth) type($stride)"; +} + +def MvoutOp : Buckyball_Op<"mvout"> { + let summary = "Store data from scratchpad bank to memory"; + let description = [{ + Stores data from scratchpad bank to DRAM. + Corresponds to bb_mvout instruction (funct7=16). + }]; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$output, + I64:$addr, I64:$depth, I64:$stride); + let assemblyFormat = "$output $addr $depth $stride attr-dict `:` " + "type($output) type($addr) type($depth) type($stride)"; +} + +def MsetOp : Buckyball_Op<"mset"> { + let summary = "Allocate or release scratchpad bank memory"; + let description = [{ + Allocates or releases scratchpad bank memory. + Corresponds to bb_mset instruction (funct7=32). + }]; + let arguments = (ins I64:$bankId, + DefaultValuedAttr:$alloc, + DefaultValuedAttr:$row, + DefaultValuedAttr:$col); + let assemblyFormat = "$bankId attr-dict `:` type($bankId)"; +} + +def FenceOp : Buckyball_Op<"fence"> { + let summary = "Memory fence for synchronization"; + let description = [{ + Ensures all previous memory operations complete before proceeding. + Corresponds to bb_fence instruction (funct7=0). + }]; + let arguments = (ins); + let assemblyFormat = "attr-dict"; +} + +//===----------------------------------------------------------------------===// +// Bank SSA operations +//===----------------------------------------------------------------------===// + +def BankAllocOp : Buckyball_Op<"bank_alloc"> { + let summary = "Allocate a virtual on-chip bank resource"; + let description = [{ + Create a virtual bank handle in SSA form. + - row/col encode bank-array shape used by allocator. + - The result i64 is a virtual bank handle, not physical id. + }]; + let arguments = (ins + DefaultValuedAttr:$row, + DefaultValuedAttr:$col); + let results = (outs I64:$bank); + let assemblyFormat = "attr-dict"; +} + +def BankReleaseOp : Buckyball_Op<"bank_release"> { + let summary = "Release a virtual bank handle"; + let arguments = (ins I64:$bank); + let assemblyFormat = "$bank attr-dict `:` type($bank)"; +} + +def BankMvinOp : Buckyball_Op<"bank_mvin"> { + let summary = "Load into a virtual bank handle"; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, I64:$bank, I64:$depth, + I64:$stride); + let results = (outs I64:$bankOut); + let assemblyFormat = + "$input $bank $depth $stride attr-dict `:` type($input) type($bank) type($depth) type($stride)"; +} + +def BankMvoutOp : Buckyball_Op<"bank_mvout"> { + let summary = "Store from a virtual bank handle"; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$output, I64:$bank, I64:$depth, + I64:$stride); + let results = (outs I64:$bankOut); + let assemblyFormat = + "$output $bank $depth $stride attr-dict `:` type($output) type($bank) type($depth) type($stride)"; +} + +//===----------------------------------------------------------------------===// +// LLVM intrinsic operations +//===----------------------------------------------------------------------===// + +class Buckyball_IntrOpBase traits = []> : + LLVM_IntrOpBase overloadedResults=*/[], + /*list overloadedOperands=*/[], + /*list traits=*/traits, + /*int numResults=*/0>; + +class Buckyball_CustomIntrOpBase immArgPositions, + list immArgAttrNames, + list traits = []> : + LLVM_IntrOpBase overloadedResults=*/[], + /*list overloadedOperands=*/[], + /*list traits=*/traits, + /*int numResults=*/0, + /*bit requiresAccessGroup=*/0, + /*bit requiresAliasAnalysis=*/0, + /*bit requiresFastmath=*/0, + /*bit requiresArgAndResultAttrs=*/0, + /*bit requiresOpBundles=*/0, + /*list immArgPositions=*/immArgPositions, + /*list immArgAttrNames=*/immArgAttrNames>; + +def MsetIntrOp : Buckyball_IntrOpBase<"mset">, + Arguments<(ins LLVM_Type, LLVM_Type)>; + +def FenceIntrOp : Buckyball_IntrOpBase<"fence">, + Arguments<(ins LLVM_Type, LLVM_Type)>; + +def MvinIntrOp : Buckyball_IntrOpBase<"mvin">, + Arguments<(ins LLVM_Type, LLVM_Type)>; + +def MvoutIntrOp : Buckyball_IntrOpBase<"mvout">, + Arguments<(ins LLVM_Type, LLVM_Type)>; + +def RushBMvinOp : Buckyball_Op<"rushb_mvin"> { + let summary = "rushB host DMA load"; + let arguments = (ins LLVM_Type:$rs1, LLVM_Type:$packedXs2, + LLVM_Type:$hostPtr); + let assemblyFormat = "$rs1 `,` $packedXs2 `,` $hostPtr attr-dict `:` " + "type($rs1) `,` type($packedXs2) `,` type($hostPtr)"; +} + +def RushBMvoutOp : Buckyball_Op<"rushb_mvout"> { + let summary = "rushB host DMA store"; + let arguments = (ins LLVM_Type:$rs1, LLVM_Type:$packedXs2, + LLVM_Type:$hostPtr); + let assemblyFormat = "$rs1 `,` $packedXs2 `,` $hostPtr attr-dict `:` " + "type($rs1) `,` type($packedXs2) `,` type($hostPtr)"; +} + +def CustomIntrOp + : Buckyball_CustomIntrOpBase<"custom", "riscv_buckyball_custom", + [2], ["funct7"]>, + Arguments<(ins LLVM_Type:$rs1, LLVM_Type:$rs2, I32Attr:$funct7)> { + let assemblyFormat = "$rs1 `,` $rs2 attr-dict `:` type($rs1) `,` type($rs2)"; +} + +//===----------------------------------------------------------------------===// +// Generic instruction and debug operations +//===----------------------------------------------------------------------===// + +def InstOp : Buckyball_Op<"inst"> { + let summary = "Generic Buckyball custom instruction carrier"; + let description = [{ + Emits a Buckyball custom instruction with explicit rs1, rs2 and funct7. + }]; + let arguments = (ins I64:$rs1, I64:$rs2, + ConfinedAttr, IntMaxValue<127>]>:$funct7); + let assemblyFormat = "$rs1 `,` $rs2 attr-dict `:` type($rs1)"; +} + +def PrintMemRefOp : Buckyball_Op<"print_memref"> { + let summary = "Print memref contents for debugging"; + let description = [{ + Print the contents of a memref for debugging purposes. + }]; + let arguments = (ins AnyMemRef:$input); + let assemblyFormat = "$input attr-dict `:` type($input)"; +} + +def PrintScalarOp : Buckyball_Op<"print_scalar"> { + let summary = "Print scalar value for debugging"; + let description = [{ + Print a scalar value for debugging purposes. + }]; + let arguments = (ins AnyType:$value); + let assemblyFormat = "$value attr-dict `:` type($value)"; +} + +#endif // BUCKYBALL_OPS diff --git a/compiler/src/Dialect/Buckyball/BuckyballBankSSA.td b/compiler/src/Dialect/Buckyball/BuckyballBankSSA.td deleted file mode 100644 index 3868c215f..000000000 --- a/compiler/src/Dialect/Buckyball/BuckyballBankSSA.td +++ /dev/null @@ -1,68 +0,0 @@ -//===-- BuckyballBankSSA.td - Bank SSA operations ------------ tablegen --===// -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// -//===----------------------------------------------------------------------===// -// -// This file defines Bank SSA operations for the Buckyball dialect. -// These are intermediate representation operations used for bank allocation -// and optimization passes. -// -//===----------------------------------------------------------------------===// - -#ifndef BUCKYBALL_BANK_SSA -#define BUCKYBALL_BANK_SSA - -//===----------------------------------------------------------------------===// -// Bank SSA Operations -//===----------------------------------------------------------------------===// - -def BankAllocOp : Buckyball_Op<"bank_alloc"> { - let summary = "Allocate a virtual on-chip bank resource"; - let description = [{ - Create a virtual bank handle in SSA form. - - row/col encode bank-array shape used by allocator. - - The result i64 is a virtual bank handle, not physical id. - }]; - let arguments = (ins - DefaultValuedAttr:$row, - DefaultValuedAttr:$col); - let results = (outs I64:$bank); - let assemblyFormat = "attr-dict"; -} - -def BankReleaseOp : Buckyball_Op<"bank_release"> { - let summary = "Release a virtual bank handle"; - let arguments = (ins I64:$bank); - let assemblyFormat = "$bank attr-dict `:` type($bank)"; -} - -def BankMvinOp : Buckyball_Op<"bank_mvin"> { - let summary = "Load into a virtual bank handle"; - let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, I64:$bank, I64:$depth, - I64:$stride); - let results = (outs I64:$bankOut); - let assemblyFormat = - "$input $bank $depth $stride attr-dict `:` type($input) type($bank) type($depth) type($stride)"; -} - -def BankMvoutOp : Buckyball_Op<"bank_mvout"> { - let summary = "Store from a virtual bank handle"; - let arguments = (ins MemRefRankOf<[AnyType], [2]>:$output, I64:$bank, I64:$depth, - I64:$stride); - let results = (outs I64:$bankOut); - let assemblyFormat = - "$output $bank $depth $stride attr-dict `:` type($output) type($bank) type($depth) type($stride)"; -} - -#endif // BUCKYBALL_BANK_SSA diff --git a/compiler/src/Dialect/Buckyball/BuckyballBase.td b/compiler/src/Dialect/Buckyball/BuckyballBase.td deleted file mode 100644 index 7149dffba..000000000 --- a/compiler/src/Dialect/Buckyball/BuckyballBase.td +++ /dev/null @@ -1,103 +0,0 @@ -//===-- BuckyballBase.td - Buckyball base definitions ----------*- tablegen -*-===// -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// -//===----------------------------------------------------------------------===// -// -// This file defines the Buckyball dialect and common base operations. -// Common operations include data movement instructions (mvin, mvout, mset, fence) -// that are used by all Ball implementations. -// -//===----------------------------------------------------------------------===// - -#ifndef BUCKYBALL_BASE -#define BUCKYBALL_BASE - -include "mlir/Dialect/LLVMIR/LLVMOpBase.td" -include "mlir/IR/AttrTypeBase.td" -include "mlir/IR/PatternBase.td" - -//===----------------------------------------------------------------------===// -// Buckyball dialect definition -//===----------------------------------------------------------------------===// - -def Buckyball_Dialect : Dialect { - let name = "buckyball"; - let cppNamespace = "::buddy::buckyball"; - let summary = "Dialect for RISC-V Buckyball NPU extension"; - let description = [{ - Buckyball is an NPU Design Framework with composable Ball units. - Each Ball is a specialized compute unit that can be independently - developed and combined with others. - }]; -} - -//===----------------------------------------------------------------------===// -// Base operation class -//===----------------------------------------------------------------------===// - -class Buckyball_Op traits = []> : - Op {} - -//===----------------------------------------------------------------------===// -// Common data movement operations (used by all Balls) -//===----------------------------------------------------------------------===// - -def MvinOp : Buckyball_Op<"mvin"> { - let summary = "Load data from memory to scratchpad bank"; - let description = [{ - Loads data from DRAM to scratchpad bank. - Corresponds to bb_mvin instruction (funct7=33). - }]; - let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, - I64:$addr, I64:$depth, I64:$stride); - let assemblyFormat = "$input $addr $depth $stride attr-dict `:` " - "type($input) type($addr) type($depth) type($stride)"; -} - -def MvoutOp : Buckyball_Op<"mvout"> { - let summary = "Store data from scratchpad bank to memory"; - let description = [{ - Stores data from scratchpad bank to DRAM. - Corresponds to bb_mvout instruction (funct7=16). - }]; - let arguments = (ins MemRefRankOf<[AnyType], [2]>:$output, - I64:$addr, I64:$depth, I64:$stride); - let assemblyFormat = "$output $addr $depth $stride attr-dict `:` " - "type($output) type($addr) type($depth) type($stride)"; -} - -def MsetOp : Buckyball_Op<"mset"> { - let summary = "Allocate or release scratchpad bank memory"; - let description = [{ - Allocates or releases scratchpad bank memory. - Corresponds to bb_mset instruction (funct7=32). - }]; - let arguments = (ins I64:$bankId, - DefaultValuedAttr:$alloc, - DefaultValuedAttr:$row, - DefaultValuedAttr:$col); - let assemblyFormat = "$bankId attr-dict `:` type($bankId)"; -} - -def FenceOp : Buckyball_Op<"fence"> { - let summary = "Memory fence for synchronization"; - let description = [{ - Ensures all previous memory operations complete before proceeding. - Corresponds to bb_fence instruction (funct7=0). - }]; - let arguments = (ins); - let assemblyFormat = "attr-dict"; -} - -#endif // BUCKYBALL_BASE diff --git a/compiler/src/Dialect/Buckyball/BuckyballDebug.td b/compiler/src/Dialect/Buckyball/BuckyballDebug.td deleted file mode 100644 index d2178659d..000000000 --- a/compiler/src/Dialect/Buckyball/BuckyballDebug.td +++ /dev/null @@ -1,24 +0,0 @@ -//===-- BuckyballDebug.td - Buckyball debug operations ---- tablegen --===// - -#ifndef BUCKYBALL_DEBUG_OPS -#define BUCKYBALL_DEBUG_OPS - -def PrintMemRefOp : Buckyball_Op<"print_memref"> { - let summary = "Print memref contents for debugging"; - let description = [{ - Print the contents of a memref for debugging purposes. - }]; - let arguments = (ins AnyMemRef:$input); - let assemblyFormat = "$input attr-dict `:` type($input)"; -} - -def PrintScalarOp : Buckyball_Op<"print_scalar"> { - let summary = "Print scalar value for debugging"; - let description = [{ - Print a scalar value for debugging purposes. - }]; - let arguments = (ins AnyType:$value); - let assemblyFormat = "$value attr-dict `:` type($value)"; -} - -#endif // BUCKYBALL_DEBUG_OPS diff --git a/compiler/src/Dialect/Buckyball/BuckyballIntrinsics.td b/compiler/src/Dialect/Buckyball/BuckyballIntrinsics.td deleted file mode 100644 index 3b80b6def..000000000 --- a/compiler/src/Dialect/Buckyball/BuckyballIntrinsics.td +++ /dev/null @@ -1,99 +0,0 @@ -//===-- BuckyballIntrinsics.td - Intrinsic operations ----*- tablegen -*-===// -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// -//===----------------------------------------------------------------------===// -// -// This file defines LLVM intrinsic wrapper operations for Buckyball. -// These operations map directly to RISC-V custom instructions via LLVM intrinsics. -// -//===----------------------------------------------------------------------===// - -#ifndef BUCKYBALL_INTRINSICS -#define BUCKYBALL_INTRINSICS - -//===----------------------------------------------------------------------===// -// Base class for intrinsic operations -//===----------------------------------------------------------------------===// - -class Buckyball_IntrOpBase traits = []> : - LLVM_IntrOpBase overloadedResults=*/[], - /*list overloadedOperands=*/[], - /*list traits=*/traits, - /*int numResults=*/0>; - -class Buckyball_CustomIntrOpBase immArgPositions, - list immArgAttrNames, - list traits = []> : - LLVM_IntrOpBase overloadedResults=*/[], - /*list overloadedOperands=*/[], - /*list traits=*/traits, - /*int numResults=*/0, - /*bit requiresAccessGroup=*/0, - /*bit requiresAliasAnalysis=*/0, - /*bit requiresFastmath=*/0, - /*bit requiresArgAndResultAttrs=*/0, - /*bit requiresOpBundles=*/0, - /*list immArgPositions=*/immArgPositions, - /*list immArgAttrNames=*/immArgAttrNames>; - -//===----------------------------------------------------------------------===// -// Common intrinsic operations -//===----------------------------------------------------------------------===// - -def MsetIntrOp : Buckyball_IntrOpBase<"mset">, - Arguments<(ins LLVM_Type, LLVM_Type)>; - -def FenceIntrOp : Buckyball_IntrOpBase<"fence">, - Arguments<(ins LLVM_Type, LLVM_Type)>; - -def MvinIntrOp : Buckyball_IntrOpBase<"mvin">, - Arguments<(ins LLVM_Type, LLVM_Type)>; - -def MvoutIntrOp : Buckyball_IntrOpBase<"mvout">, - Arguments<(ins LLVM_Type, LLVM_Type)>; - -// Host-only DMA carriers. Unlike the RISC-V intrinsics above, these retain -// the host pointer explicitly for rushB lowering. The guest ISA continues to -// use its two-register mvin/mvout intrinsic ABI. -def RushBMvinOp : Buckyball_Op<"rushb_mvin"> { - let summary = "rushB host DMA load"; - let arguments = (ins LLVM_Type:$rs1, LLVM_Type:$packedXs2, - LLVM_Type:$hostPtr); - let assemblyFormat = "$rs1 `,` $packedXs2 `,` $hostPtr attr-dict `:` " - "type($rs1) `,` type($packedXs2) `,` type($hostPtr)"; -} - -def RushBMvoutOp : Buckyball_Op<"rushb_mvout"> { - let summary = "rushB host DMA store"; - let arguments = (ins LLVM_Type:$rs1, LLVM_Type:$packedXs2, - LLVM_Type:$hostPtr); - let assemblyFormat = "$rs1 `,` $packedXs2 `,` $hostPtr attr-dict `:` " - "type($rs1) `,` type($packedXs2) `,` type($hostPtr)"; -} - -def CustomIntrOp - : Buckyball_CustomIntrOpBase<"custom", "riscv_buckyball_custom", - [2], ["funct7"]>, - Arguments<(ins LLVM_Type:$rs1, LLVM_Type:$rs2, I32Attr:$funct7)> { - let assemblyFormat = "$rs1 `,` $rs2 attr-dict `:` type($rs1) `,` type($rs2)"; -} - -#endif // BUCKYBALL_INTRINSICS diff --git a/compiler/src/Dialect/Buckyball/IR/BuckyballDialect.cpp b/compiler/src/Dialect/Buckyball/IR/BuckyballDialect.cpp index 69e12f472..b7b62e01f 100644 --- a/compiler/src/Dialect/Buckyball/IR/BuckyballDialect.cpp +++ b/compiler/src/Dialect/Buckyball/IR/BuckyballDialect.cpp @@ -29,6 +29,7 @@ #include "Buckyball/BuckyballDialect.h" #include "Buckyball/BuckyballOps.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; @@ -38,6 +39,9 @@ using namespace buddy::buckyball; #include "Buckyball.cpp.inc" void BuckyballDialect::initialize() { + // Keep the target registry in the dialect library even for Ball-free Cores. + // Otherwise the static linker drops the command-line option definition. + static_cast(buckyball_target::getRequestedBuckyballTarget()); addOperations< #define GET_OP_LIST #include "Buckyball.cpp.inc" diff --git a/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..47c0a88be --- /dev/null +++ b/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,45 @@ +#include "Buckyball/Transform.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" + +using namespace mlir; +using namespace buddy::buckyball::legalize; + +namespace mlir::buddy::buckyball { +#define BUCKYBALL_LEGALIZE_HOOK(BALL) \ + void populate##BALL##LegalizeForLLVMExportPatterns( \ + LLVMTypeConverter &, RewritePatternSet &, bool, int64_t, bool); \ + void configure##BALL##LegalizeForExportTarget(LLVMConversionTarget &, bool); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_LEGALIZE_HOOK +} // namespace mlir::buddy::buckyball + +void mlir::populateBuckyballLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, + int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum, + bool includeFuncOperandForwarding, bool stable, bool rushB) { + populateBaseLegalizeForLLVMExportPatterns( + converter, patterns, includeFuncOperandForwarding, rushB); + for (llvm::StringRef ball : buckyball_target::getBuckyballTarget().balls) { +#define BUCKYBALL_LEGALIZE_HOOK(BALL) \ + if (ball == #BALL) \ + buddy::buckyball::populate##BALL##LegalizeForLLVMExportPatterns( \ + converter, patterns, stable, bankDepth, rushB); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_LEGALIZE_HOOK + } + (void)bankWidthBytes; + (void)bankNum; +} + +void mlir::configureBuckyballLegalizeForExportTarget( + LLVMConversionTarget &target, bool stable) { + configureBaseLegalizeForExportTarget(target); + for (llvm::StringRef ball : buckyball_target::getBuckyballTarget().balls) { +#define BUCKYBALL_LEGALIZE_HOOK(BALL) \ + if (ball == #BALL) \ + buddy::buckyball::configure##BALL##LegalizeForExportTarget(target, stable); +#include "BuckyballBallLoweringHooks.inc" +#undef BUCKYBALL_LEGALIZE_HOOK + } +} diff --git a/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp b/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp index 0e51dd2e0..1e8626e50 100644 --- a/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp +++ b/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp @@ -41,6 +41,32 @@ uint64_t fieldBits(uint64_t val, int startBit, int endBit) { return (val & mask) << startBit; } +int64_t addrBitsForDepth(int64_t bankDepth) { + if (bankDepth <= 1) + return -1; + int64_t bits = 0; + int64_t x = bankDepth - 1; + while (x) { + x >>= 1; + ++bits; + } + return bits; +} + +uint64_t smatmulIterBits(int64_t op1Base, int64_t op2Base, int64_t wrBase, + int64_t addrBits) { + if (addrBits < 1 || addrBits > 20) + llvm_unreachable("smatmul addrBits out of range"); + if (3 * addrBits > 34) + llvm_unreachable("smatmul iter cannot hold three bases"); + uint64_t mask = (1ULL << addrBits) - 1; + if ((uint64_t)op1Base > mask || (uint64_t)op2Base > mask || + (uint64_t)wrBase > mask) + llvm_unreachable("smatmul base exceeds addrBits"); + return (uint64_t)op1Base | ((uint64_t)op2Base << addrBits) | + ((uint64_t)wrBase << (2 * addrBits)); +} + Value cstI64(OpBuilder &b, Location loc, uint64_t v) { return b.create(loc, b.getI64Type(), b.getI64IntegerAttr(v)); @@ -127,6 +153,23 @@ void emitMset(OpBuilder &b, Location loc, uint64_t bankId, uint64_t row, b.create(loc, cstI64(b, loc, rs1), cstI64(b, loc, rs2)); } +static void emitCacheAsm(OpBuilder &b, Location loc, StringRef assembly) { + auto tail = LLVM::TailCallKindAttr::get( + b.getContext(), LLVM::tailcallkind::TailCallKind::None); + LLVM::InlineAsmOp::create(b, loc, Type(), ValueRange{}, + b.getStringAttr(assembly), + b.getStringAttr("~{memory}"), b.getUnitAttr(), + UnitAttr(), tail, nullptr, nullptr); +} + +void emitDmaCacheFlush(OpBuilder &b, Location loc) { + emitCacheAsm(b, loc, "fence.i"); +} + +void emitDmaCacheFence(OpBuilder &b, Location loc) { + emitCacheAsm(b, loc, "fence rw, rw\n\tfence.i"); +} + static constexpr char kBbDmaTouchMvoutFn[] = "bb_dma_touch_mvout"; static constexpr char kBbDmaBankSetColsFn[] = "bb_dma_bank_set_cols"; @@ -203,7 +246,9 @@ struct BuckyballFenceLowering : public ConvertOpToLLVMPattern { ConversionPatternRewriter &rewriter) const override { Location loc = op.getLoc(); Value zero = cstI64(rewriter, loc, 0); - rewriter.replaceOpWithNewOp(op, zero, zero); + rewriter.create(loc, zero, zero); + emitDmaCacheFence(rewriter, loc); + rewriter.eraseOp(op); return success(); } }; @@ -239,6 +284,7 @@ struct BuckyballMvinLowering : public ConvertOpToLLVMPattern { ConversionPatternRewriter &rewriter) const override { Location loc = op.getLoc(); MemrefAddress memref = extractMemrefAddress(rewriter, loc, op.getInput()); + emitDmaCacheFlush(rewriter, loc); Value rs1 = packRs1BankIter(rewriter, loc, adaptor.getAddr(), adaptor.getDepth()); Value rs2 = @@ -266,6 +312,7 @@ struct BuckyballMvoutLowering : public ConvertOpToLLVMPattern { MemrefAddress memref = extractMemrefAddress(rewriter, loc, op.getOutput()); emitBbDmaTouchMvout(rewriter, loc, memref.hostPtr, adaptor.getDepth(), adaptor.getStride(), adaptor.getAddr()); + emitDmaCacheFlush(rewriter, loc); Value rs1 = packRs1BankIter(rewriter, loc, adaptor.getAddr(), adaptor.getDepth()); Value rs2 = @@ -315,8 +362,7 @@ void populateBaseLegalizeForLLVMExportPatterns( void configureBaseLegalizeForExportTarget(LLVMConversionTarget &target) { target.addLegalOp(); - target.addIllegalOp(); + target.addIllegalOp(); target.addLegalDialect(); target.addLegalDialect(); target.addLegalDialect(); diff --git a/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h b/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h index 4ffae787f..163b2a8d4 100644 --- a/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h +++ b/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h @@ -30,6 +30,9 @@ namespace buckyball { namespace legalize { uint64_t fieldBits(uint64_t val, int startBit, int endBit); +int64_t addrBitsForDepth(int64_t bankDepth); +uint64_t smatmulIterBits(int64_t op1Base, int64_t op2Base, int64_t wrBase, + int64_t addrBits); mlir::Value cstI64(mlir::OpBuilder &b, mlir::Location loc, uint64_t v); mlir::Value extractPtr(mlir::OpBuilder &b, mlir::Location loc, @@ -43,6 +46,8 @@ mlir::Value packRs2MemStride(mlir::OpBuilder &b, mlir::Location loc, mlir::Value memAddr, mlir::Value stride); void emitMset(mlir::OpBuilder &b, mlir::Location loc, uint64_t bankId, uint64_t row, uint64_t col, uint64_t alloc); +void emitDmaCacheFlush(mlir::OpBuilder &b, mlir::Location loc); +void emitDmaCacheFence(mlir::OpBuilder &b, mlir::Location loc); void populateBaseLegalizeForLLVMExportPatterns( mlir::LLVMTypeConverter &converter, mlir::RewritePatternSet &patterns, diff --git a/compiler/src/Target/BuckyballCoreLoweringRegistry.cpp b/compiler/src/Target/BuckyballCoreLoweringRegistry.cpp new file mode 100644 index 000000000..f0e9db2bf --- /dev/null +++ b/compiler/src/Target/BuckyballCoreLoweringRegistry.cpp @@ -0,0 +1,37 @@ +//===- BuckyballCoreLoweringRegistry.cpp - Core lowering providers --------===// + +#include "Target/BuckyballCoreLoweringRegistry.h" + +#include "llvm/ADT/StringMap.h" +#include "llvm/ADT/Twine.h" +#include "llvm/Support/ErrorHandling.h" + +namespace { +llvm::StringMap & +providers() { + static llvm::StringMap + registry; + return registry; +} +} // namespace + +void mlir::buddy::registerBuckyballCoreLoweringProvider( + const BuckyballCoreLoweringProvider &provider) { + if (provider.core.empty()) + llvm::report_fatal_error("Buckyball Core lowering provider has no name"); + auto [it, inserted] = providers().try_emplace(provider.core, &provider); + if (!inserted) + llvm::report_fatal_error(llvm::Twine("duplicate Buckyball Core lowering " + "provider: ") + + provider.core); +} + +const mlir::buddy::BuckyballCoreLoweringProvider & +mlir::buddy::getBuckyballCoreLoweringProvider(llvm::StringRef core) { + auto it = providers().find(core); + if (it == providers().end()) + llvm::report_fatal_error(llvm::Twine("no Buckyball lowering provider for " + "Core: ") + + core); + return *it->second; +} diff --git a/compiler/src/Target/BuckyballCoreLoweringRegistry.h b/compiler/src/Target/BuckyballCoreLoweringRegistry.h new file mode 100644 index 000000000..cf3dd6c7f --- /dev/null +++ b/compiler/src/Target/BuckyballCoreLoweringRegistry.h @@ -0,0 +1,39 @@ +//===- BuckyballCoreLoweringRegistry.h - Core lowering providers -*- C++ +//-*-===// + +#ifndef BUCKYBALL_CORE_LOWERING_REGISTRY_H +#define BUCKYBALL_CORE_LOWERING_REGISTRY_H + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" +#include "mlir/Conversion/LLVMCommon/TypeConverter.h" +#include "mlir/Transforms/DialectConversion.h" +#include "llvm/ADT/StringRef.h" + +namespace mlir::buddy { + +struct BuckyballCoreLoweringProvider { + using TileLoweringFn = void (*)(ConversionTarget &, RewritePatternSet &, + int64_t, int64_t, int64_t); + using BankSSALoweringFn = void (*)(RewritePatternSet &); + using PhysicalBankLoweringFn = void (*)(RewritePatternSet &, + PhysicalBankState &); + using LLVMExportLoweringFn = void (*)(LLVMTypeConverter &, + RewritePatternSet &, + LLVMConversionTarget &, int64_t, + int64_t, int64_t, bool, bool, bool); + + llvm::StringRef core; + TileLoweringFn populateTileLowering = nullptr; + BankSSALoweringFn populateBankSSALowering = nullptr; + PhysicalBankLoweringFn populatePhysicalBankLowering = nullptr; + LLVMExportLoweringFn populateLLVMExportLowering = nullptr; +}; + +void registerBuckyballCoreLoweringProvider( + const BuckyballCoreLoweringProvider &provider); +const BuckyballCoreLoweringProvider & +getBuckyballCoreLoweringProvider(llvm::StringRef core); + +} // namespace mlir::buddy + +#endif // BUCKYBALL_CORE_LOWERING_REGISTRY_H diff --git a/compiler/src/Target/BuckyballTargetRegistry.cpp b/compiler/src/Target/BuckyballTargetRegistry.cpp new file mode 100644 index 000000000..983b614e4 --- /dev/null +++ b/compiler/src/Target/BuckyballTargetRegistry.cpp @@ -0,0 +1,62 @@ +#include "Target/BuckyballTargetRegistry.h" + +#include "llvm/ADT/Twine.h" +#include "llvm/Support/CommandLine.h" +#include "llvm/Support/ErrorHandling.h" + +using namespace llvm; + +namespace { +cl::opt + buckyballTarget("target", cl::desc("Required Buckyball compiler target"), + cl::value_desc("target")); +} // namespace + +#include "BuckyballTargetRegistry.inc" + +llvm::StringRef buckyball_target::getRequestedBuckyballTarget() { + return buckyballTarget; +} + +const buckyball_target::BuckyballTargetConfig & +buckyball_target::getBuckyballTarget() { + llvm::StringRef requestedTarget = getRequestedBuckyballTarget(); + if (requestedTarget.empty()) + report_fatal_error("--target is required for Buckyball lowering"); + for (const BuckyballTargetConfig &target : kBuckyballTargets) { + if (target.name == requestedTarget) + return target; + } + report_fatal_error(Twine("unknown Buckyball target: ") + requestedTarget); +} + +int32_t buckyball_target::getBuckyballFunct7(llvm::StringRef mnemonic) { + const BuckyballTargetConfig &target = getBuckyballTarget(); + for (const BuckyballIsaEntry &entry : target.isa) { + if (entry.mnemonic == mnemonic) + return entry.funct7; + } + report_fatal_error(Twine("Buckyball target ") + target.name + + " does not define ISA mnemonic " + mnemonic); +} + +void buckyball_target::requireBuckyballBall(llvm::StringRef ballName) { + const BuckyballTargetConfig &target = getBuckyballTarget(); + for (llvm::StringRef enabled : target.balls) { + if (enabled == ballName) + return; + } + report_fatal_error(Twine("Buckyball target ") + target.name + + " does not enable Ball " + ballName); +} + +const buckyball_target::BuckyballBallMapping & +buckyball_target::getBuckyballBallMapping(llvm::StringRef ballName) { + const BuckyballTargetConfig &target = getBuckyballTarget(); + for (const BuckyballBallMapping &mapping : target.ballMappings) { + if (mapping.name == ballName) + return mapping; + } + report_fatal_error(Twine("Buckyball target ") + target.name + + " does not map Ball " + ballName); +} diff --git a/compiler/src/Target/BuckyballTargetRegistry.h b/compiler/src/Target/BuckyballTargetRegistry.h new file mode 100644 index 000000000..3a3dc2864 --- /dev/null +++ b/compiler/src/Target/BuckyballTargetRegistry.h @@ -0,0 +1,41 @@ +#ifndef BUCKYBALL_TARGET_REGISTRY_H +#define BUCKYBALL_TARGET_REGISTRY_H + +#include "llvm/ADT/ArrayRef.h" +#include "llvm/ADT/StringRef.h" + +#include + +namespace buckyball_target { + +struct BuckyballIsaEntry { + llvm::StringRef mnemonic; + int32_t funct7; +}; + +struct BuckyballBallMapping { + llvm::StringRef name; + int64_t inBW; + int64_t outBW; +}; + +struct BuckyballTargetConfig { + llvm::StringRef name; + llvm::StringRef core; + int64_t bankNum; + int64_t bankWidthBits; + int64_t bankDepth; + llvm::ArrayRef balls; + llvm::ArrayRef ballMappings; + llvm::ArrayRef isa; +}; + +llvm::StringRef getRequestedBuckyballTarget(); +const BuckyballTargetConfig &getBuckyballTarget(); +int32_t getBuckyballFunct7(llvm::StringRef mnemonic); +void requireBuckyballBall(llvm::StringRef ballName); +const BuckyballBallMapping &getBuckyballBallMapping(llvm::StringRef ballName); + +} // namespace buckyball_target + +#endif // BUCKYBALL_TARGET_REGISTRY_H diff --git a/compiler/src/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp b/compiler/src/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp index 6b3d4b782..201176e0e 100644 --- a/compiler/src/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp +++ b/compiler/src/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp @@ -38,16 +38,9 @@ namespace { static llvm::Intrinsic::ID lookupStableIntrinsic(StringRef opName) { return llvm::StringSwitch(opName) .Case("buckyball.intr.fence", llvm::Intrinsic::riscv_bb_fence) - .Case("buckyball.intr.fp2int", llvm::Intrinsic::riscv_bb_fp2int) - .Case("buckyball.intr.im2col", llvm::Intrinsic::riscv_bb_im2col) - .Case("buckyball.intr.int2fp", llvm::Intrinsic::riscv_bb_int2fp) .Case("buckyball.intr.mset", llvm::Intrinsic::riscv_bb_mset) - .Case("buckyball.intr.mul_warp16", llvm::Intrinsic::riscv_bb_mul_warp16) .Case("buckyball.intr.mvin", llvm::Intrinsic::riscv_bb_mvin) .Case("buckyball.intr.mvout", llvm::Intrinsic::riscv_bb_mvout) - .Case("buckyball.intr.relu", llvm::Intrinsic::riscv_bb_relu) - .Case("buckyball.intr.bbfp.mul", llvm::Intrinsic::riscv_bb_bbfp_mul) - .Case("buckyball.intr.transpose", llvm::Intrinsic::riscv_bb_transpose) .Default(llvm::Intrinsic::not_intrinsic); } diff --git a/compiler/src/Utils/BankUtils.h b/compiler/src/Utils/BankUtils.h index 844396872..afa379509 100644 --- a/compiler/src/Utils/BankUtils.h +++ b/compiler/src/Utils/BankUtils.h @@ -27,9 +27,15 @@ #include "Buckyball/BuckyballOps.h" +#include +#include +#include + namespace buddy { namespace buckyball { +class BankSMatMulOp; + /// Create i64 constant. static inline mlir::Value createI64Const(mlir::OpBuilder &b, mlir::Location loc, int64_t val) { @@ -49,7 +55,34 @@ static inline mlir::Value createI64ConstU(mlir::OpBuilder &b, static inline uint64_t packBits(uint64_t val, int startBit, int endBit) { uint64_t width = endBit - startBit + 1; uint64_t mask = (1ULL << width) - 1; - return (val & mask) << startBit; + if (val > mask) { + std::fprintf(stderr, "packBits: value %llu does not fit in [%d:%d]\n", + (unsigned long long)val, startBit, endBit); + std::abort(); + } + return val << startBit; +} + +static inline uint64_t matrixRs2(uint64_t rows, uint64_t cols, uint64_t k) { + if (rows == 0 || cols == 0 || cols > 0xfff || k == 0 || rows > 0xfff || + k > 0xfff || rows % 16 || cols % 16 || k % 16) { + std::fprintf( + stderr, + "matrix rs2: rows/cols/k must fit in 12 bits (got %llu %llu %llu)\n", + (unsigned long long)rows, (unsigned long long)cols, + (unsigned long long)k); + std::abort(); + } + return packBits(rows, 0, 11) | packBits(cols, 12, 23) | packBits(k, 24, 35); +} + +static inline mlir::Value createBankSMatMul(mlir::OpBuilder &b, + mlir::Location loc, mlir::Type wrTy, + mlir::Value a, mlir::Value opB, + mlir::Value wr, mlir::Value cfg) { + return b.create(loc, wrTy, a, opB, wr, cfg, + b.getI64IntegerAttr(0), b.getI64IntegerAttr(0), + b.getI64IntegerAttr(0)); } /// Allocate a bank with given row/col dimensions. diff --git a/compiler/src/Utils/QuantUtils.h b/compiler/src/Utils/QuantUtils.h deleted file mode 100644 index 61271140a..000000000 --- a/compiler/src/Utils/QuantUtils.h +++ /dev/null @@ -1,82 +0,0 @@ -//===- QuantUtils.h - Shared f32 quant helpers for bank-SSA ---------------===// - -#ifndef BUCKYBALL_CONVERSION_QUANTUTILS_H -#define BUCKYBALL_CONVERSION_QUANTUTILS_H - -#include "mlir/Dialect/Arith/IR/Arith.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Dialect/SCF/IR/SCF.h" -#include "mlir/IR/Builders.h" -#include "mlir/IR/Value.h" - -namespace buddy { -namespace buckyball { - -static inline mlir::Value cstF32(mlir::OpBuilder &b, mlir::Location loc, - float v) { - return b.create(loc, b.getF32Type(), - b.getF32FloatAttr(v)); -} - -static inline mlir::Value -packF32BitsAsI64(mlir::OpBuilder &b, mlir::Location loc, mlir::Value f32Val) { - mlir::Value i32Bits = - b.create(loc, b.getI32Type(), f32Val); - return b.create(loc, b.getI64Type(), i32Bits); -} - -// Abs-max over memreff32 with static rows/cols. -static inline mlir::Value absMaxF32(mlir::OpBuilder &b, mlir::Location loc, - mlir::Value mem, int64_t rows, - int64_t cols) { - auto maxTy = mlir::MemRefType::get({1}, b.getF32Type()); - mlir::Value maxBuf = b.create(loc, maxTy); - mlir::Value zero = b.create(loc, 0); - mlir::Value one = b.create(loc, 1); - mlir::Value rowsV = b.create(loc, rows); - mlir::Value colsV = b.create(loc, cols); - b.create(loc, cstF32(b, loc, 0.0f), maxBuf, - mlir::ValueRange{zero}); - auto rL = b.create(loc, zero, rowsV, one); - b.setInsertionPointToStart(rL.getBody()); - auto cL = b.create(loc, zero, colsV, one); - b.setInsertionPointToStart(cL.getBody()); - mlir::Value elem = b.create( - loc, mem, mlir::ValueRange{rL.getInductionVar(), cL.getInductionVar()}); - mlir::Value neg = b.create(loc, elem); - mlir::Value abs = b.create(loc, elem, neg); - mlir::Value cur = - b.create(loc, maxBuf, mlir::ValueRange{zero}); - b.create( - loc, b.create(loc, cur, abs), maxBuf, - mlir::ValueRange{zero}); - b.setInsertionPointAfter(rL); - mlir::Value result = - b.create(loc, maxBuf, mlir::ValueRange{zero}); - b.create(loc, maxBuf); - return result; -} - -// Quant scale = 127/maxAbs (or 1 if maxAbs==0). -static inline mlir::Value quantScale(mlir::OpBuilder &b, mlir::Location loc, - mlir::Value maxAbs) { - mlir::Value zero = cstF32(b, loc, 0.0f); - mlir::Value one = cstF32(b, loc, 1.0f); - mlir::Value qmax = cstF32(b, loc, 127.0f); - mlir::Value has = b.create( - loc, mlir::arith::CmpFPredicate::OGT, maxAbs, zero); - mlir::Value scaled = b.create(loc, qmax, maxAbs); - return b.create(loc, has, scaled, one); -} - -static inline mlir::Value dequantScale(mlir::OpBuilder &b, mlir::Location loc, - mlir::Value scaleA, mlir::Value scaleB) { - mlir::Value one = cstF32(b, loc, 1.0f); - mlir::Value prod = b.create(loc, scaleA, scaleB); - return b.create(loc, one, prod); -} - -} // namespace buckyball -} // namespace buddy - -#endif // BUCKYBALL_CONVERSION_QUANTUTILS_H diff --git a/compiler/thirdparty/buddy-mlir b/compiler/thirdparty/buddy-mlir index 545e801f1..b1e936da5 160000 --- a/compiler/thirdparty/buddy-mlir +++ b/compiler/thirdparty/buddy-mlir @@ -1 +1 @@ -Subproject commit 545e801f147b91d897627ccb502db6913c67e9b4 +Subproject commit b1e936da5aff6b79652432012c4e13473431f854 diff --git a/docs b/docs index f41cdf678..74d588104 160000 --- a/docs +++ b/docs @@ -1 +1 @@ -Subproject commit f41cdf678c024e98df7a61814e2cb952a49e76f2 +Subproject commit 74d58810465427d28250fa022ca248fa6d6b401d diff --git a/examples/.gitignore b/examples/.gitignore index 8b943e239..6a9039890 100644 --- a/examples/.gitignore +++ b/examples/.gitignore @@ -2,3 +2,4 @@ *.key build/ test-artifacts/ +generated/ diff --git a/examples/balls/fp2int/arch/src/main/scala/Fp2Int.scala b/examples/balls/fp2int/arch/src/main/scala/Fp2Int.scala index 6bf6d2ac3..d2e7af6c8 100644 --- a/examples/balls/fp2int/arch/src/main/scala/Fp2Int.scala +++ b/examples/balls/fp2int/arch/src/main/scala/Fp2Int.scala @@ -6,20 +6,36 @@ import chisel3.experimental.hierarchy.{instantiable, public} import framework.balldomain.rs.{BallRsComplete, BallRsIssue} import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} +import framework.balldomain.blink.mmio.MmioWrite import framework.top.GlobalConfig @instantiable class Fp2Int(val b: GlobalConfig) extends Module { val bankWidth = b.memDomain.bankWidth - require(bankWidth == 128, s"Fp2IntBall requires bankWidth = 128, got $bankWidth") + require( + bankWidth == 128, + s"Fp2IntBall requires bankWidth = 128, got $bankWidth" + ) val elemsPerWord = bankWidth / 32 - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "Fp2IntBall") - .getOrElse(throw new IllegalArgumentException("Fp2IntBall not found in config")) + + val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "Fp2IntBall") + .getOrElse( + throw new IllegalArgumentException("Fp2IntBall not found in config") + ) + val inBW = ballMapping.inBW val outBW = ballMapping.outBW + val mmioWriteBW = ballMapping.mmioWriteBW require(inBW >= 1, "Fp2Int requires at least one read port") require(outBW >= 1, "Fp2Int requires at least one write port") + require(mmioWriteBW == 4, "Fp2Int requires four MMIO byte write lines") + + val fp2IntFunct = b.ballDomain.ballISA + .find(_.mnemonic == "FP2INT") + .map(_.funct7) + .getOrElse(throw new IllegalArgumentException("FP2INT not found in ballISA")) @public val io = IO(new Bundle { @@ -27,26 +43,43 @@ class Fp2Int(val b: GlobalConfig) extends Module { val cmdResp = Decoupled(new BallRsComplete(b)) val bankRead = Vec(inBW, Flipped(new BankRead(b))) val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) + val mmioWrite = Vec(mmioWriteBW, Flipped(new MmioWrite(b))) val status = new BallStatus }) - val idle :: sReadReq :: sReadResp :: sWriteReq :: sWriteResp :: complete :: Nil = Enum(6) - val state = RegInit(idle) + val idle :: sScanReq :: sScanResp :: sScaleWrite :: sReadReq :: sReadResp :: sWriteReq :: sWriteResp :: complete :: Nil = + Enum(9) + val state = RegInit(idle) val robIdReg = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) val isSubReg = RegInit(false.B) val subRobIdReg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) - val rbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val wbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val iterReg = RegInit(0.U(b.frontend.iter_len.W)) - val scaleReg = RegInit(0.U(32.W)) - val int8Mode = RegInit(false.B) + val rbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + val wbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + val iterReg = RegInit(0.U(b.frontend.iter_len.W)) + val scaleReg = RegInit(0.U(32.W)) + val quantScaleReg = RegInit(0.U(32.W)) + val scaleAddrReg = RegInit(0.U(log2Ceil(b.memDomain.mmioTotalBytes).W)) + val scanMaxReg = RegInit(0.U(31.W)) - val rowReg = RegInit(0.U(b.frontend.iter_len.W)) - val groupReg = RegInit(0.U(2.W)) - val outWord = RegInit(0.U(bankWidth.W)) - val writeWord = RegInit(0.U(bankWidth.W)) + val rowReg = RegInit(0.U(b.frontend.iter_len.W)) + val groupReg = RegInit(0.U(5.W)) + val srcColReg = RegInit(0.U(5.W)) + val dstColReg = RegInit(0.U(5.W)) + val packWordReg = RegInit(0.U(2.W)) + val outRowReg = RegInit(0.U(b.frontend.iter_len.W)) + val outGroupReg = RegInit(0.U(5.W)) + val outWord = RegInit(0.U(bankWidth.W)) + val writeWord = RegInit(0.U(bankWidth.W)) + + for (i <- 0 until mmioWriteBW) { + io.mmioWrite(i).rob_id := robIdReg + io.mmioWrite(i).ball_id := 0.U + io.mmioWrite(i).req.valid := false.B + io.mmioWrite(i).req.bits.addr := 0.U + io.mmioWrite(i).req.bits.data := 0.U + } for (i <- 0 until inBW) { io.bankRead(i).rob_id := robIdReg @@ -65,7 +98,9 @@ class Fp2Int(val b: GlobalConfig) extends Module { io.bankWrite(i).io.req.valid := false.B io.bankWrite(i).io.req.bits.addr := 0.U io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(false.B)) + io.bankWrite(i).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(false.B) + ) io.bankWrite(i).io.resp.ready := false.B } @@ -108,7 +143,7 @@ class Fp2Int(val b: GlobalConfig) extends Module { val rounded = sig +& increment.asUInt val finalSig = Mux(rounded(24), rounded(24, 1), rounded(23, 0)) val expWide = aExp +& bExp +& normAdjust +& rounded(24) - 127.U - val result = Wire(UInt(32.W)) + val result = Wire(UInt(32.W)) when(aZero || bZero) { result := 0.U }.elsewhen(expWide(9, 8) =/= 0.U && expWide(9)) { @@ -121,11 +156,40 @@ class Fp2Int(val b: GlobalConfig) extends Module { result } + def fp32Divide(a: UInt, bv: UInt): UInt = { + val aExp = a(30, 23) + val bExp = bv(30, 23) + val aMant = Cat(1.U(1.W), a(22, 0)) + val bMant = Cat(1.U(1.W), bv(22, 0)) + val normalizeDown = aMant < bMant + val dividend = Mux(normalizeDown, aMant << 26.U, aMant << 25.U) + val quotient = dividend / bMant + val remainder = dividend % bMant + val sig = quotient >> 2.U + val roundUp = quotient(1) && (quotient(0) || remainder.orR || sig(0)) + val rounded = sig +& roundUp.asUInt + val exp = aExp.zext - bExp.zext + Mux(normalizeDown, 126.S, 127.S) + rounded(24).asSInt + val result = Wire(UInt(32.W)) + when(a(30, 0) === 0.U) { + result := 0.U + }.elsewhen(bv(30, 0) === 0.U) { + result := Cat(a(31) ^ bv(31), 255.U(8.W), 0.U(23.W)) + }.elsewhen(exp < 1.S) { + result := 0.U + }.elsewhen(exp > 254.S) { + result := Cat(a(31) ^ bv(31), 255.U(8.W), 0.U(23.W)) + }.otherwise { + result := Cat(a(31) ^ bv(31), exp.asUInt(7, 0), Mux(rounded(24), 0.U(23.W), rounded(22, 0))) + } + result + } + def fp32ToInt32(fp: UInt): UInt = { val sign = fp(31) val exponent = fp(30, 23) val fraction = fp(22, 0) - val mantissa = Mux(exponent === 0.U, Cat(0.U(1.W), fraction), Cat(1.U(1.W), fraction)) + val mantissa = + Mux(exponent === 0.U, Cat(0.U(1.W), fraction), Cat(1.U(1.W), fraction)) val mantissaWide = Cat(0.U(40.W), mantissa) val expVal = exponent.zext - 127.S val magnitude = Wire(UInt(64.W)) @@ -136,11 +200,11 @@ class Fp2Int(val b: GlobalConfig) extends Module { }.elsewhen(expVal >= 23.S) { magnitude := mantissaWide << (expVal - 23.S).asUInt }.elsewhen(expVal >= -1.S) { - val rightShift = (23.S - expVal).asUInt - val truncated = mantissaWide >> rightShift - val half = 1.U(64.W) << (rightShift - 1.U) - val remainder = mantissaWide & ((1.U(64.W) << rightShift) - 1.U) - val roundUp = remainder > half || (remainder === half && truncated(0)) + val rightShift = (23.S - expVal).asUInt + val truncated = mantissaWide >> rightShift + val half = 1.U(64.W) << (rightShift - 1.U) + val remainder = mantissaWide & ((1.U(64.W) << rightShift) - 1.U) + val roundUp = remainder > half || (remainder === half && truncated(0)) magnitude := truncated + roundUp.asUInt } @@ -154,7 +218,11 @@ class Fp2Int(val b: GlobalConfig) extends Module { -magnitude(31, 0).asSInt ) }.otherwise { - result := Mux(magnitude > "h7fffffff".U, 2147483647.S(32.W), magnitude(31, 0).asSInt) + result := Mux( + magnitude > "h7fffffff".U, + 2147483647.S(32.W), + magnitude(31, 0).asSInt + ) } result.asUInt } @@ -172,48 +240,121 @@ class Fp2Int(val b: GlobalConfig) extends Module { c.asUInt } - def quantWord(data: UInt): (UInt, UInt) = { - val i32 = Wire(Vec(elemsPerWord, UInt(32.W))) - val i8 = Wire(Vec(elemsPerWord, UInt(8.W))) + def quantWord(data: UInt): UInt = { + val i8 = Wire(Vec(elemsPerWord, UInt(8.W))) for (i <- 0 until elemsPerWord) { val fp = data((i + 1) * 32 - 1, i * 32) - val scaled = fp32Multiply(fp, scaleReg) - i32(i) := fp32ToInt32(scaled) - i8(i) := fp32ToInt8(scaled) + val scaled = fp32Multiply(fp, quantScaleReg) + i8(i) := fp32ToInt8(scaled) + } + Cat(i8.reverse) + } + + def wordMaxAbs(data: UInt): UInt = { + val values = Seq.tabulate(elemsPerWord) { i => + val fp = data((i + 1) * 32 - 1, i * 32) + assert(fp(30, 23) =/= 255.U, "Fp2Int does not accept NaN or infinity") + fp(30, 0) } - (Cat(i32.reverse), Cat(i8.reverse)) + values.reduceLeft((a, b) => Mux(a > b, a, b)) } switch(state) { is(idle) { when(io.cmdReq.fire) { - val srcCol = io.cmdReq.bits.cmd.op1_col - val dstCol = io.cmdReq.bits.cmd.wr_col - val isI32 = srcCol === 1.U && dstCol === 1.U - val isI8 = srcCol === 4.U && dstCol === 1.U + assert(io.cmdReq.bits.cmd.funct7 === fp2IntFunct.U(7.W), "Fp2Int funct7 must be FP2INT") + val srcCol = io.cmdReq.bits.cmd.op1_col + val dstCol = io.cmdReq.bits.cmd.wr_col + val sourceWords = io.cmdReq.bits.cmd.iter * srcCol + val dstRowWords = Cat(dstCol, 0.U(2.W)) assert(io.cmdReq.bits.cmd.iter > 0.U, "Fp2Int iter must be > 0") - assert(isI32 || isI8, "Fp2Int unsupported bank layout") - - robIdReg := io.cmdReq.bits.rob_id - isSubReg := io.cmdReq.bits.is_sub - subRobIdReg := io.cmdReq.bits.sub_rob_id - rbankReg := io.cmdReq.bits.cmd.op1_bank - wbankReg := io.cmdReq.bits.cmd.wr_bank - iterReg := io.cmdReq.bits.cmd.iter - scaleReg := io.cmdReq.bits.cmd.special(31, 0) - int8Mode := isI8 - rowReg := 0.U - groupReg := 0.U - outWord := 0.U - writeWord := 0.U - state := sReadReq + assert(srcCol > 0.U && dstCol > 0.U, "Fp2Int source and destination groups must be nonzero") + assert(sourceWords % dstRowWords === 0.U, "Fp2Int source stream must fill destination rows") + assert(sourceWords / dstRowWords <= b.memDomain.bankEntries.U, "Fp2Int destination exceeds bank capacity") + assert(io.cmdReq.bits.cmd.op1_bank =/= io.cmdReq.bits.cmd.wr_bank, "Fp2Int forbids in-place quantization") + assert(io.cmdReq.bits.cmd.special(63, 13) === 0.U, "Fp2Int reserves special[63:13]") + assert(io.cmdReq.bits.cmd.special(12, 0) === 0.U, "Fp2Int Da address must be 0") + assert(io.cmdReq.bits.cmd.special(1, 0) === 0.U, "Fp2Int Da address must be 4-byte aligned") + assert( + (io.cmdReq.bits.cmd.special(12, 0) +& 3.U) < b.memDomain.mmioTotalBytes.U, + "Fp2Int Da address exceeds MMIO space" + ) + + robIdReg := io.cmdReq.bits.rob_id + isSubReg := io.cmdReq.bits.is_sub + subRobIdReg := io.cmdReq.bits.sub_rob_id + rbankReg := io.cmdReq.bits.cmd.op1_bank + wbankReg := io.cmdReq.bits.cmd.wr_bank + iterReg := io.cmdReq.bits.cmd.iter + srcColReg := srcCol + dstColReg := dstCol + scaleAddrReg := io.cmdReq.bits.cmd.special(12, 0) + scanMaxReg := 0.U + rowReg := 0.U + groupReg := 0.U + packWordReg := 0.U + outRowReg := 0.U + outGroupReg := 0.U + outWord := 0.U + writeWord := 0.U + state := sScanReq + } + } + + is(sScanReq) { + io.bankRead(0).bank_id := rbankReg + io.bankRead(0).group_id := groupReg + io.bankRead(0).io.req.valid := true.B + io.bankRead(0).io.req.bits.addr := rowReg + when(io.bankRead(0).io.req.fire) { + state := sScanResp + } + } + + is(sScanResp) { + io.bankRead(0).bank_id := rbankReg + io.bankRead(0).group_id := groupReg + io.bankRead(0).io.resp.ready := true.B + when(io.bankRead(0).io.resp.fire) { + val nextMax = Mux( + wordMaxAbs(io.bankRead(0).io.resp.bits.data) > scanMaxReg, + wordMaxAbs(io.bankRead(0).io.resp.bits.data), + scanMaxReg + ) + scanMaxReg := nextMax + when(groupReg === srcColReg - 1.U && rowReg === iterReg - 1.U) { + val da = Mux(nextMax === 0.U, "h3f800000".U, fp32Divide(Cat(0.U(1.W), nextMax), "h42fe0000".U)) + scaleReg := da + quantScaleReg := fp32Divide("h3f800000".U, da) + rowReg := 0.U + groupReg := 0.U + state := sScaleWrite + }.elsewhen(groupReg === srcColReg - 1.U) { + groupReg := 0.U + rowReg := rowReg + 1.U + state := sScanReq + }.otherwise { + groupReg := groupReg + 1.U + state := sScanReq + } + } + } + + is(sScaleWrite) { + for (i <- 0 until mmioWriteBW) { + io.mmioWrite(i).req.valid := true.B + io.mmioWrite(i).req.bits.addr := scaleAddrReg + i.U + io.mmioWrite(i).req.bits.data := scaleReg((i + 1) * 8 - 1, i * 8) + } + when(io.mmioWrite.map(_.req.fire).reduceLeft((a: Bool, c: Bool) => a && c)) { + state := sReadReq } } is(sReadReq) { io.bankRead(0).bank_id := rbankReg - io.bankRead(0).group_id := Mux(int8Mode, groupReg, 0.U) + io.bankRead(0).group_id := groupReg io.bankRead(0).io.req.valid := true.B io.bankRead(0).io.req.bits.addr := rowReg when(io.bankRead(0).io.req.fire) { @@ -223,36 +364,39 @@ class Fp2Int(val b: GlobalConfig) extends Module { is(sReadResp) { io.bankRead(0).bank_id := rbankReg - io.bankRead(0).group_id := Mux(int8Mode, groupReg, 0.U) + io.bankRead(0).group_id := groupReg io.bankRead(0).io.resp.ready := true.B when(io.bankRead(0).io.resp.fire) { - val (i32Word, i8Bytes) = quantWord(io.bankRead(0).io.resp.bits.data) - when(int8Mode) { - val nextWord = outWord | (i8Bytes << (groupReg << 5.U)) - when(groupReg === 3.U) { - writeWord := nextWord - groupReg := 0.U - outWord := 0.U - state := sWriteReq + val i8Bytes = quantWord(io.bankRead(0).io.resp.bits.data) + val nextWord = outWord | (i8Bytes << (packWordReg << 5.U)) + when(packWordReg === 3.U) { + writeWord := nextWord + packWordReg := 0.U + outWord := 0.U + state := sWriteReq + }.otherwise { + outWord := nextWord + packWordReg := packWordReg + 1.U + when(groupReg === srcColReg - 1.U) { + groupReg := 0.U + rowReg := rowReg + 1.U }.otherwise { - outWord := nextWord groupReg := groupReg + 1.U - state := sReadReq } - }.otherwise { - writeWord := i32Word - state := sWriteReq + state := sReadReq } } } is(sWriteReq) { io.bankWrite(0).bank_id := wbankReg - io.bankWrite(0).group_id := 0.U + io.bankWrite(0).group_id := outGroupReg io.bankWrite(0).io.req.valid := true.B - io.bankWrite(0).io.req.bits.addr := rowReg + io.bankWrite(0).io.req.bits.addr := outRowReg io.bankWrite(0).io.req.bits.data := writeWord - io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) + io.bankWrite(0).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(true.B) + ) when(io.bankWrite(0).io.req.fire) { state := sWriteResp } @@ -260,14 +404,25 @@ class Fp2Int(val b: GlobalConfig) extends Module { is(sWriteResp) { io.bankWrite(0).bank_id := wbankReg - io.bankWrite(0).group_id := 0.U + io.bankWrite(0).group_id := outGroupReg io.bankWrite(0).io.resp.ready := true.B when(io.bankWrite(0).io.resp.fire) { - when(rowReg === iterReg - 1.U) { + when(rowReg === iterReg - 1.U && groupReg === srcColReg - 1.U) { state := complete }.otherwise { - rowReg := rowReg + 1.U - state := sReadReq + when(groupReg === srcColReg - 1.U) { + groupReg := 0.U + rowReg := rowReg + 1.U + }.otherwise { + groupReg := groupReg + 1.U + } + when(outGroupReg === dstColReg - 1.U) { + outGroupReg := 0.U + outRowReg := outRowReg + 1.U + }.otherwise { + outGroupReg := outGroupReg + 1.U + } + state := sReadReq } } } diff --git a/examples/balls/fp2int/arch/src/main/scala/Fp2IntBall.scala b/examples/balls/fp2int/arch/src/main/scala/Fp2IntBall.scala index f7232aa8e..7b149ecef 100644 --- a/examples/balls/fp2int/arch/src/main/scala/Fp2IntBall.scala +++ b/examples/balls/fp2int/arch/src/main/scala/Fp2IntBall.scala @@ -5,18 +5,25 @@ import chisel3.util._ import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} import framework.balldomain.blink.{BallStatus, BlinkIO, HasBallStatus, HasBlink, SubRobRow} import framework.balldomain.blink.mmio.MmioRead +import framework.balldomain.blink.mmio.MmioWrite import framework.top.GlobalConfig @instantiable class Fp2IntBall(val b: GlobalConfig) extends Module with HasBlink { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "Fp2IntBall") - .getOrElse(throw new IllegalArgumentException("Fp2IntBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "Fp2IntBall") + .getOrElse( + throw new IllegalArgumentException("Fp2IntBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW + val mmioReadBW = ballCommonConfig.mmioReadBW + val mmioWriteBW = ballCommonConfig.mmioWriteBW @public - val io = IO(new BlinkIO(b, inBW, outBW)) + val io = IO(new BlinkIO(b, inBW, outBW, mmioReadBW, mmioWriteBW)) def blink: BlinkIO = io dontTouch(io) @@ -33,6 +40,9 @@ class Fp2IntBall(val b: GlobalConfig) extends Module with HasBlink { for (i <- 0 until outBW) { fp2intUnit.io.bankWrite(i) <> io.bankWrite(i) } + for (i <- 0 until mmioWriteBW) { + fp2intUnit.io.mmioWrite(i) <> io.mmioWrite(i) + } io.status <> fp2intUnit.io.status diff --git a/examples/balls/fp2int/compiler/src/Backend/include/IntrinsicsRISCVFp2IntBall.td b/examples/balls/fp2int/compiler/src/Backend/include/IntrinsicsRISCVFp2IntBall.td deleted file mode 100644 index a9af12eb3..000000000 --- a/examples/balls/fp2int/compiler/src/Backend/include/IntrinsicsRISCVFp2IntBall.td +++ /dev/null @@ -1,4 +0,0 @@ -//===- IntrinsicsRISCVFp2IntBall.td - Fp2IntBall intrinsics --------------===// - -let TargetPrefix = "riscv" in -def int_riscv_bb_fp2int : Intrinsic<[], [llvm_i64_ty, llvm_i64_ty], []>; diff --git a/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp index 57fdd3796..80b9febe0 100644 --- a/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp +++ b/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -10,8 +10,8 @@ using namespace mlir; using namespace ::buddy::buckyball; namespace mlir::buddy { -void populateFp2IntAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateFp2IntBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); } // namespace mlir::buddy namespace { @@ -23,7 +23,7 @@ class BankFp2IntPattern : public OpRewritePattern { LogicalResult matchAndRewrite(BankFp2IntOp op, PatternRewriter &rewriter) const override { rewriter.create(op.getLoc(), op.getInBank(), op.getOutBank(), - op.getIter(), op.getScale()); + op.getIter(), op.getDaAddr()); rewriter.replaceOp(op, op.getOutBank()); return success(); } @@ -31,7 +31,7 @@ class BankFp2IntPattern : public OpRewritePattern { } // namespace -void mlir::buddy::populateFp2IntAssignPhysicalBankPatterns( +void mlir::buddy::populateFp2IntBallAssignPhysicalBankPatterns( RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { (void)state; patterns.add(patterns.getContext()); diff --git a/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Fp2IntBall.td b/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Fp2IntBall.td index 947a70035..92165701a 100644 --- a/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Fp2IntBall.td +++ b/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Fp2IntBall.td @@ -7,8 +7,7 @@ #ifndef QUANTBALL_OPS #define QUANTBALL_OPS -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" +include "Buckyball.td" //===----------------------------------------------------------------------===// // Fp2IntBall operations @@ -17,27 +16,27 @@ include "BuckyballIntrinsics.td" def Fp2IntOp : Buckyball_Op<"fp2int"> { let summary = "Quantize floating-point to fixed-point"; let description = [{ - Quantizes floating-point values to fixed-point representation. - Corresponds to bb_fp2int instruction (funct7=51). + Quantizes FP32 words to packed INT8 words. The input and output bank + group counts are independent: FP2INT streams input words in row/group + order and packs each four words into one output word. The input stream + must fill complete output rows. + Corresponds to bb_fp2int instruction (funct7 from Core ballISA). Belongs to Fp2IntBall (ballName=Fp2IntBall; ballId is chip-config specific). - ISA: bb_fp2int(bank_id, wr_bank_id, iter, scale_fp32) + ISA: bb_fp2int(bank_id, wr_bank_id, iter, da_addr) }]; let arguments = (ins I64:$inputBankId, I64:$outputBankId, - I64:$iter, I64:$scale); + I64:$iter, I64:$daAddr); let assemblyFormat = "$inputBankId `,` $outputBankId `,` " - "$iter `,` $scale attr-dict `:` " + "$iter `,` $daAddr attr-dict `:` " "type($inputBankId)"; } def BankFp2IntOp : Buckyball_Op<"bank_fp2int"> { let summary = "Quantization using virtual bank handles"; - let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$scale); + let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$daAddr); let results = (outs I64:$outBankOut); let assemblyFormat = - "$inBank $outBank $iter $scale attr-dict `:` type($inBank) type($outBank) type($iter) type($scale)"; + "$inBank $outBank $iter $daAddr attr-dict `:` type($inBank) type($outBank) type($iter) type($daAddr)"; } -def Fp2IntIntrOp : Buckyball_IntrOpBase<"fp2int">, - Arguments<(ins LLVM_Type, LLVM_Type)>; - #endif // QUANTBALL_OPS diff --git a/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index 198b1a923..05bd6d1c8 100644 --- a/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -1,9 +1,12 @@ #include "mlir/Conversion/LLVMCommon/ConversionTarget.h" #include "mlir/Conversion/LLVMCommon/Pattern.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/IR/Matchers.h" #include "mlir/IR/PatternMatch.h" #include "Buckyball/BuckyballOps.h" #include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; @@ -11,43 +14,40 @@ using namespace buddy::buckyball::legalize; namespace { struct Fp2IntLowering : public ConvertOpToLLVMPattern { - Fp2IntLowering(LLVMTypeConverter &converter, bool stable) - : ConvertOpToLLVMPattern(converter), stable(stable) {} + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; LogicalResult matchAndRewrite(Fp2IntOp op, OpAdaptor adaptor, ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("Fp2IntBall"); + llvm::APInt daAddr(64, 0); + if (!matchPattern(op.getDaAddr(), m_ConstantInt(&daAddr)) || + daAddr.getSExtValue() != 0) + return op.emitError("FP2INT Da address must be 0"); Location loc = op.getLoc(); Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInputBankId(), cstI64(rewriter, loc, 0), adaptor.getOutputBankId(), adaptor.getIter()); - if (stable) { - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getScale()); - return success(); - } - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getScale(), - rewriter.getI32IntegerAttr(51)); + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getDaAddr(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("FP2INT"))); return success(); } - -private: - bool stable = false; }; } // namespace namespace mlir::buddy::buckyball { -void populateFp2IntLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable) { - patterns.add(converter, stable); +void populateFp2IntBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t, bool) { + (void)stable; + patterns.add(converter); } -void configureFp2IntLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable) { - if (stable) - target.addLegalOp(); - else - target.addIllegalOp(); +void configureFp2IntBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { + (void)stable; target.addIllegalOp(); } } // namespace mlir::buddy::buckyball diff --git a/examples/balls/fp2int/configs/Fp2IntBallParam.scala b/examples/balls/fp2int/configs/Fp2IntBallParam.scala index f4b734cf1..36fcf2da4 100644 --- a/examples/balls/fp2int/configs/Fp2IntBallParam.scala +++ b/examples/balls/fp2int/configs/Fp2IntBallParam.scala @@ -4,9 +4,8 @@ import framework.balldomain.configs.BallParamLoader import framework.top.GlobalConfig case class Fp2IntBallParam( - inputType: String, - outputType: String -) + inputType: String, + outputType: String) object Fp2IntBallParam { private val ballName = "Fp2IntBall" @@ -18,4 +17,5 @@ object Fp2IntBallParam { outputType = BallParamLoader.str(tbl, "outputType") ) } + } diff --git a/examples/balls/fp2int/configs/default.toml b/examples/balls/fp2int/configs/default.toml index da1f13b7d..e3da2387b 100644 --- a/examples/balls/fp2int/configs/default.toml +++ b/examples/balls/fp2int/configs/default.toml @@ -2,4 +2,4 @@ [ball] inputType = "FP32" -outputType = "INT32" +outputType = "INT8" diff --git a/examples/balls/fp2int/emu/src/51_fp2int.rs b/examples/balls/fp2int/emu/src/51_fp2int.rs deleted file mode 100644 index c02bb2d5f..000000000 --- a/examples/balls/fp2int/emu/src/51_fp2int.rs +++ /dev/null @@ -1,87 +0,0 @@ -//===- 51_fp2int.rs - FP2INT instruction (FP32 to INT quantization) --------===// - -use super::super::bank::{bank_num, bank_size}; -use super::decode::{pbank, pbank_group, rs1_b0, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; - -mod model; - -pub struct Fp2Int; - -impl Instruction for Fp2Int { - const FUNCT: u32 = 51; - - fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - let src = rs1_b0(xs1); - let dst = rs1_b2(xs1); - let depth = rs1_iter(xs1) as usize; - - if src >= bank_num() as u64 || dst >= bank_num() as u64 { - panic!("fp2int: invalid bank_id"); - } - - if depth == 0 { - panic!("fp2int: iter must be > 0"); - } - - let sc = ctx.cfgs[src as usize]; - let dc = ctx.cfgs[dst as usize]; - if !sc.allocated || !dc.allocated { - panic!("fp2int: bank not allocated"); - } - - let scale_bits = (xs2 & 0xffff_ffff) as u32; - - match (sc.cols, dc.cols) { - (1, 1) => { - // One SRAM beat contains four FP32 values and also four INT32 - // results. `iter` counts beats, not 16-element matrix rows. - let ps = pbank(ctx.bank_map, src); - let pd = pbank(ctx.bank_map, dst); - for i in 0..depth { - let base = i * 16; - if base + 16 > bank_size() { - panic!("fp2int: out of range"); - } - for lane in 0..4 { - let off = base + lane * 4; - let fp_bits = - u32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); - let q = model::fp2int_i32_bits(fp_bits, scale_bits); - ctx.banks[pd][off..off + 4].copy_from_slice(&q.to_le_bytes()); - } - } - } - (4, 1) => { - let pd = pbank(ctx.bank_map, dst); - for i in 0..depth { - let base = i * 16; - if base + 16 > bank_size() { - panic!("fp2int: out of range"); - } - for group in 0..4 { - let ps = pbank_group(ctx.bank_map, src, group); - for lane in 0..4 { - let off = base + lane * 4; - let fp_bits = - u32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); - let q = model::fp2int_i8_bits(fp_bits, scale_bits); - ctx.banks[pd][base + group as usize * 4 + lane] = q as u8; - } - } - } - } - _ => { - panic!( - "fp2int: unsupported layout src_cols={} dst_cols={}. Supported: (1,1) for FP32->INT32, (4,1) for FP32->INT8", - sc.cols, dc.cols - ); - } - } - 0 - } - - fn latency(xs1: u64, _xs2: u64) -> u64 { - rs1_iter(xs1).max(1) - } -} diff --git a/examples/balls/fp2int/emu/src/fp2int.rs b/examples/balls/fp2int/emu/src/fp2int.rs new file mode 100644 index 000000000..5cc4b25e6 --- /dev/null +++ b/examples/balls/fp2int/emu/src/fp2int.rs @@ -0,0 +1,103 @@ +//===- fp2int.rs - FP32 to INT8 activation quantization -------------------===// + +use super::super::bank::{bank_num, bank_size, mmio_total_size, mmio_write_byte}; +use super::decode::{pbank_group, rs1_b0, rs1_b2, rs1_iter}; +use super::instruction::ExecContext; + +pub(crate) fn exec_fp2int(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { + let src = rs1_b0(xs1); + let dst = rs1_b2(xs1); + let depth = rs1_iter(xs1) as usize; + + if src >= bank_num() as u64 || dst >= bank_num() as u64 { + panic!("fp2int: invalid bank_id"); + } + if src == dst { + panic!("fp2int: in-place quantization is forbidden"); + } + + if depth == 0 { + panic!("fp2int: iter must be > 0"); + } + + let sc = ctx.cfgs[src as usize]; + let dc = ctx.cfgs[dst as usize]; + if !sc.allocated || !dc.allocated { + panic!("fp2int: bank not allocated"); + } + + if xs2 >> 13 != 0 { + panic!("fp2int: reserved rs2 bits are nonzero"); + } + let scale_addr = (xs2 & 0x1fff) as usize; + if scale_addr != 0 { + panic!("fp2int: activation scale address must be 0"); + } + if scale_addr + 4 > mmio_total_size() { + panic!("fp2int: activation scale address out of range"); + } + + if sc.cols == 0 || dc.cols == 0 { + panic!("fp2int: source and destination groups must be nonzero"); + } + let source_words = depth * sc.cols as usize; + let destination_row_words = 4 * dc.cols as usize; + if source_words % destination_row_words != 0 { + panic!("fp2int: source stream does not fill destination rows"); + } + + let mut max_abs = 0u32; + for row in 0..depth { + let base = row * 16; + if base + 16 > bank_size() { + panic!("fp2int: source out of range"); + } + for group in 0..sc.cols { + let ps = pbank_group(ctx.bank_map, src, group); + for lane in 0..4 { + let off = base + lane * 4; + let bits = u32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); + if (bits >> 23) & 0xff == 0xff { + panic!("fp2int: activation contains NaN or infinity"); + } + max_abs = max_abs.max(bits & 0x7fff_ffff); + } + } + } + let da_bits = super::model::fp2int_da_from_max_abs_bits(max_abs); + for (i, byte) in da_bits.to_le_bytes().iter().enumerate() { + mmio_write_byte(ctx.mmio_banks, scale_addr + i, *byte); + } + let quant_scale_bits = super::model::fp32_divide(1.0f32.to_bits(), da_bits); + let mut packed = [0u8; 16]; + let mut source_word = 0usize; + for row in 0..depth { + let base = row * 16; + for group in 0..sc.cols { + let ps = pbank_group(ctx.bank_map, src, group); + let byte_base = (source_word % 4) * 4; + for lane in 0..4 { + let off = base + lane * 4; + let bits = u32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); + packed[byte_base + lane] = super::model::fp2int_i8_bits(bits, quant_scale_bits) as u8; + } + source_word += 1; + if source_word % 4 == 0 { + let output_word = source_word / 4 - 1; + let output_group = output_word % dc.cols as usize; + let output_row = output_word / dc.cols as usize; + let pd = pbank_group(ctx.bank_map, dst, output_group as u64); + let output_base = output_row * 16; + if output_base + 16 > bank_size() { + panic!("fp2int: destination out of range"); + } + ctx.banks[pd][output_base..output_base + 16].copy_from_slice(&packed); + } + } + } + 0 +} + +pub(crate) fn latency(xs1: u64) -> u64 { + rs1_iter(xs1).max(1) +} diff --git a/examples/balls/fp2int/emu/src/lib.rs b/examples/balls/fp2int/emu/src/lib.rs index e3d99911c..792f17941 100644 --- a/examples/balls/fp2int/emu/src/lib.rs +++ b/examples/balls/fp2int/emu/src/lib.rs @@ -1,10 +1,13 @@ pub(crate) use crate::inst::decode; pub(crate) use crate::inst::instruction; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::ExecContext; -#[path = "51_fp2int.rs"] -mod f51_fp2int; +mod fp2int; +mod model; + +#[cfg(test)] +mod tests; const BALL_CLASS: &str = "examples.balls.fp2int.Fp2IntBall"; @@ -18,11 +21,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("FP2INT") => Some(fp2int::exec_fp2int(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -30,10 +31,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("FP2INT") => Some(fp2int::latency(xs1)), + Some(_) | None => None, } } diff --git a/examples/balls/fp2int/emu/src/model.rs b/examples/balls/fp2int/emu/src/model.rs index a3bffd9f9..cac33ca58 100644 --- a/examples/balls/fp2int/emu/src/model.rs +++ b/examples/balls/fp2int/emu/src/model.rs @@ -6,24 +6,28 @@ pub fn fp2int_i8_bits(fp_bits: u32, scale_bits: u32) -> i8 { fp2int_i32_bits(fp_bits, scale_bits).clamp(-128, 127) as i8 } -#[allow(dead_code)] -pub fn fp2int_i32_word(input: [u32; 4], scale_bits: u32) -> [i32; 4] { - [ - fp2int_i32_bits(input[0], scale_bits), - fp2int_i32_bits(input[1], scale_bits), - fp2int_i32_bits(input[2], scale_bits), - fp2int_i32_bits(input[3], scale_bits), - ] +pub fn fp2int_da_bits(input_words: &[u128]) -> u32 { + let mut max_abs = 0u32; + for word in input_words { + for lane in 0..4 { + let bits = (*word >> (lane * 32)) as u32; + assert_ne!( + (bits >> 23) & 0xff, + 0xff, + "Fp2Int does not accept NaN or infinity" + ); + max_abs = max_abs.max(bits & 0x7fff_ffff); + } + } + fp2int_da_from_max_abs_bits(max_abs) } -#[allow(dead_code)] -pub fn fp2int_i8_group(input: [u32; 4], scale_bits: u32) -> [i8; 4] { - [ - fp2int_i8_bits(input[0], scale_bits), - fp2int_i8_bits(input[1], scale_bits), - fp2int_i8_bits(input[2], scale_bits), - fp2int_i8_bits(input[3], scale_bits), - ] +pub fn fp2int_da_from_max_abs_bits(max_abs: u32) -> u32 { + if max_abs == 0 { + 0x3f80_0000 + } else { + fp32_divide(max_abs, 0x42fe_0000) + } } fn fp32_multiply(a: u32, b: u32) -> u32 { @@ -73,6 +77,44 @@ fn fp32_multiply(a: u32, b: u32) -> u32 { } } +pub fn fp32_divide(a: u32, b: u32) -> u32 { + let a_exp = (a >> 23) & 0xff; + let b_exp = (b >> 23) & 0xff; + let a_mant = (1u64 << 23) | u64::from(a & 0x7f_ffff); + let b_mant = (1u64 << 23) | u64::from(b & 0x7f_ffff); + let normalize_down = a_mant < b_mant; + let dividend = if normalize_down { + a_mant << 26 + } else { + a_mant << 25 + }; + let quotient = dividend / b_mant; + let remainder = dividend % b_mant; + let sig = quotient >> 2; + let round_up = (quotient & 0b10) != 0 + && ((quotient & 0b1) != 0 || remainder != 0 || (sig & 0b1) != 0); + let rounded = sig + u64::from(round_up); + let exp = a_exp as i32 - b_exp as i32 + + if normalize_down { 126 } else { 127 } + + ((rounded >> 24) as i32); + + if (a & 0x7fff_ffff) == 0 { + 0 + } else if (b & 0x7fff_ffff) == 0 || exp > 254 { + ((a ^ b) & 0x8000_0000) | 0x7f80_0000 + } else if exp < 1 { + 0 + } else { + ((a ^ b) & 0x8000_0000) + | ((exp as u32) << 23) + | if (rounded & (1 << 24)) != 0 { + 0 + } else { + (rounded as u32) & 0x7f_ffff + } + } +} + fn fp32_to_int32(fp: u32) -> i32 { let sign = ((fp >> 31) & 1) != 0; let exponent = ((fp >> 23) & 0xff) as i32; @@ -115,53 +157,3 @@ fn fp32_to_int32(fp: u32) -> i32 { magnitude as i32 } } - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn int32_basic() { - let scale = 0x3F80_0000; - - assert_eq!(fp2int_i32_bits(0x3F80_0000, scale), 1); - assert_eq!(fp2int_i32_bits(0x4000_0000, scale), 2); - assert_eq!(fp2int_i32_bits(0xBF80_0000, scale), -1); - assert_eq!(fp2int_i32_bits(0x3F00_0000, scale), 0); // 0.5 -> even 0 - assert_eq!(fp2int_i32_bits(0xBF00_0000, scale), 0); // -0.5 -> even 0 - assert_eq!(fp2int_i32_bits(0x3FC0_0000, scale), 2); // 1.5 -> even 2 - assert_eq!(fp2int_i32_bits(0xBFC0_0000, scale), -2); // -1.5 -> even -2 - assert_eq!(fp2int_i32_bits(0x4020_0000, scale), 2); // 2.5 -> even 2 - assert_eq!(fp2int_i32_bits(0xC020_0000, scale), -2); // -2.5 -> even -2 - } - - #[test] - fn int8_saturates() { - let scale = 0x3F80_0000; - - assert_eq!(fp2int_i8_bits(0x4300_0000, scale), 127); - assert_eq!(fp2int_i8_bits(0xC300_0000, scale), -128); - } - - #[test] - fn fp32_to_int8_workload_vectors() { - let scale = 2.0f32.to_bits(); - let input = [ - 0.125f32, -0.125, 0.25, -0.25, 0.75, -0.75, 1.25, -1.25, 1.75, -1.75, 63.25, 63.75, - -63.75, -64.75, 0.0, -0.0, 2.25, -2.25, 2.75, -2.75, 3.25, -3.25, 3.75, -3.75, 10.125, - -10.125, 20.25, -20.25, 0.375, -0.375, 64.25, -65.25, - ]; - let expected = [ - 0i8, 0, 0, 0, 2, -2, 2, -2, 4, -4, 126, 127, -128, -128, 0, 0, 4, -4, 6, -6, 6, -6, 8, - -8, 20, -20, 40, -40, 1, -1, 127, -128, - ]; - - for (value, expected) in input.into_iter().zip(expected) { - assert_eq!( - fp2int_i8_bits(value.to_bits(), scale), - expected, - "input={value}" - ); - } - } -} diff --git a/examples/balls/fp2int/emu/src/tests/activation_scale.rs b/examples/balls/fp2int/emu/src/tests/activation_scale.rs new file mode 100644 index 000000000..d9e2c17ba --- /dev/null +++ b/examples/balls/fp2int/emu/src/tests/activation_scale.rs @@ -0,0 +1,11 @@ +use super::super::model::{fp2int_da_from_max_abs_bits, fp32_divide}; + +#[test] +fn computes_activation_scale_with_fp32_division() { + assert_eq!( + fp32_divide(2.0f32.to_bits(), 127.0f32.to_bits()), + 0x3c81_0204 + ); + assert_eq!(fp2int_da_from_max_abs_bits(0), 1.0f32.to_bits()); + assert_eq!(fp2int_da_from_max_abs_bits(63.5f32.to_bits()), 0.5f32.to_bits()); +} diff --git a/examples/balls/fp2int/emu/src/tests/int32_rounding.rs b/examples/balls/fp2int/emu/src/tests/int32_rounding.rs new file mode 100644 index 000000000..e5208de5a --- /dev/null +++ b/examples/balls/fp2int/emu/src/tests/int32_rounding.rs @@ -0,0 +1,21 @@ +use super::super::model::fp2int_i32_bits; + +#[test] +fn rounds_to_nearest_even() { + let scale = 0x3f80_0000; + let cases = [ + (0x3f80_0000, 1), + (0x4000_0000, 2), + (0xbf80_0000, -1), + (0x3f00_0000, 0), + (0xbf00_0000, 0), + (0x3fc0_0000, 2), + (0xbfc0_0000, -2), + (0x4020_0000, 2), + (0xc020_0000, -2), + ]; + + for (input, expected) in cases { + assert_eq!(fp2int_i32_bits(input, scale), expected); + } +} diff --git a/examples/balls/fp2int/emu/src/tests/int8_saturation.rs b/examples/balls/fp2int/emu/src/tests/int8_saturation.rs new file mode 100644 index 000000000..da730909c --- /dev/null +++ b/examples/balls/fp2int/emu/src/tests/int8_saturation.rs @@ -0,0 +1,9 @@ +use super::super::model::fp2int_i8_bits; + +#[test] +fn saturates_beyond_int8_range() { + let scale = 0x3f80_0000; + + assert_eq!(fp2int_i8_bits(0x4300_0000, scale), 127); + assert_eq!(fp2int_i8_bits(0xc300_0000, scale), -128); +} diff --git a/examples/balls/fp2int/emu/src/tests/mod.rs b/examples/balls/fp2int/emu/src/tests/mod.rs new file mode 100644 index 000000000..06616eadb --- /dev/null +++ b/examples/balls/fp2int/emu/src/tests/mod.rs @@ -0,0 +1,4 @@ +mod activation_scale; +mod int32_rounding; +mod int8_saturation; +mod workload_vectors; diff --git a/examples/balls/fp2int/emu/src/tests/workload_vectors.rs b/examples/balls/fp2int/emu/src/tests/workload_vectors.rs new file mode 100644 index 000000000..198d6e8a3 --- /dev/null +++ b/examples/balls/fp2int/emu/src/tests/workload_vectors.rs @@ -0,0 +1,19 @@ +use super::super::model::fp2int_i8_bits; + +#[test] +fn quantizes_workload_vectors() { + let scale = 2.0f32.to_bits(); + let input = [ + 0.125f32, -0.125, 0.25, -0.25, 0.75, -0.75, 1.25, -1.25, 1.75, -1.75, 63.25, 63.75, + -63.75, -64.75, 0.0, -0.0, 2.25, -2.25, 2.75, -2.75, 3.25, -3.25, 3.75, -3.75, 10.125, + -10.125, 20.25, -20.25, 0.375, -0.375, 64.25, -65.25, + ]; + let expected = [ + 0i8, 0, 0, 0, 2, -2, 2, -2, 4, -4, 126, 127, -128, -128, 0, 0, 4, -4, 6, -6, 6, -6, 8, + -8, 20, -20, 40, -40, 1, -1, 127, -128, + ]; + + for (value, expected) in input.into_iter().zip(expected) { + assert_eq!(fp2int_i8_bits(value.to_bits(), scale), expected, "input={value}"); + } +} diff --git a/examples/balls/fp2int/verify/README.md b/examples/balls/fp2int/verify/README.md index a9e4419dc..dd2d051f3 100644 --- a/examples/balls/fp2int/verify/README.md +++ b/examples/balls/fp2int/verify/README.md @@ -6,28 +6,30 @@ Verifies the generated `Fp2IntBall` module using the shared Blink UVM framework. - `../../../../verify/uvm/src/bb_uvm_pkg.sv`: common Blink UVM agents and base env - `src/common/fp2int_defs.svh`: DPI imports, `fp2int_require_bid`, timeouts -- `src/common/fp2int_items.svh`: `fp2int_cmd_item` with `load_rust_case` (scale in `special[31:0]`) +- `src/common/fp2int_items.svh`: `fp2int_cmd_item` with `load_rust_case` (Da address in `special[12:0]`) - `src/seq/fp2int_sequences.svh`: one-case sequence -- `src/cov/fp2int_cov.svh`: layout {i32,i8} and iter {1,4} coverage +- `src/cov/fp2int_cov.svh`: online INT8 layout and iter 1 coverage - `src/env/fp2int_scoreboard.svh`: preload mem model (group-aware for i8), compare writes vs DPI ref - `src/env/fp2int_env.svh`: extends `bb_blink_env#(1,1)` -- `src/tests/fp2int_test.svh`: directed 0..1, random 2..11 +- `src/tests/fp2int_*_test.svh`: one directed case per UVM test - `src/pkg/fp2int_pkg.sv`: package entry - `src/tb_top.sv`: `bb_blink_if#(1,1)` + `Fp2IntBall` - `filelists/fp2int_ball.f`: VCS filelist ## Test plan -`+UVM_TESTNAME=fp2int_ball_test` runs: +The test names are: -- case 0: directed INT32 (`op1_col=1`, `wr_col=1`, scale=1.0) -- case 1: directed INT8 (`op1_col=4`, `wr_col=1`, scale=2.0) -- cases 2..11: random INT32 with fixed seed `0xBEEF_0001` +- `fp2int_signed_test`: signed values across all four source groups +- `fp2int_zero_test`: all-zero activation (`Da=1.0`) +- `fp2int_rounding_test`: ties-to-even quantization boundaries +- `fp2int_rows_test`: two-row scan and packed output traversal +- `fp2int_scale_rows_test`: maximum value is in the second row Each case: reset, preload src words into `mem_model` (INT8 uses `group_id` 0..3), drive one command, wait for `scb.done()` or timeout at 400 cycles. Scoreboard -compares observed writes with DPI `fp2int_ref_i32` / `fp2int_ref_i8`. `fp2int_cov` -requires layout and iter bins hit across the whole test, else `check_phase` fatal. +compares observed writes with DPI `fp2int_ref_i8`. `fp2int_cov` requires the online +INT8 layout and one-row iteration bin, else `check_phase` fatal. ## BID (required) @@ -39,7 +41,7 @@ requires layout and iter bins hit across the whole test, else `check_phase` fata | `sims.verilator.BuckyballToyVerilatorConfig` (full.toml) | `+BID=5` | ```console -./simv +UVM_TESTNAME=fp2int_ball_test +BID=3 +./simv +UVM_TESTNAME=fp2int_scale_rows_test +BID=3 ``` ## Build @@ -68,22 +70,24 @@ vcs -full64 -sverilog -timescale=1ns/1ps \ Run: ```console -./simv +UVM_TESTNAME=fp2int_ball_test +BID=3 +./simv +UVM_TESTNAME=fp2int_scale_rows_test +BID=3 ``` Or via bbdev: ```console -bbdev uvm --build '--ball=fp2int' -bbdev uvm --run '--ball=fp2int' -- '+BID=3' +bbdev uvm --build '--ball=fp2int' \ + --config sims.verilator.BuckyballPebbleVerilatorConfig \ + --core-config examples/cores/pebble/configs/default.toml +bbdev uvm --run '--ball=fp2int --plusargs +BID=3' ``` ## Acceptance (2026-08-04) - [x] Layout matches Blink common + ball-local split; filelist uses `@UVM@` / `@RTL@` - [x] casegen reuses fp2int model; whole-case DPI API with injected bid -- [x] directed INT32/INT8 scoreboard pass -- [x] deterministic random idx 2..11 pass +- [x] directed online INT8 scoreboard pass +- [x] signed, zero, rounding, and row-traversal directed cases pass - [x] protocol + semantic cover enforced (`check_phase` fatal) -- [x] pebble: `+BID=3` config `sims.verilator.BuckyballPebbleVerilatorConfig` — 12/12 -- [x] toy full: `+BID=5` config `sims.verilator.BuckyballToyVerilatorConfig` — 12/12 +- [x] pebble: `+BID=3` config `sims.verilator.BuckyballPebbleVerilatorConfig` +- [ ] toy full: `+BID=5` config `sims.verilator.BuckyballToyVerilatorConfig` diff --git a/examples/balls/fp2int/verify/casegen/src/casegen.rs b/examples/balls/fp2int/verify/casegen/src/casegen.rs index 81f0740ce..868f1ace0 100644 --- a/examples/balls/fp2int/verify/casegen/src/casegen.rs +++ b/examples/balls/fp2int/verify/casegen/src/casegen.rs @@ -1,4 +1,3 @@ -pub const FUNCT7: u32 = 51; pub const WORDS: usize = 4; pub const GROUPS: usize = 4; pub const MAX_WORDS: usize = WORDS * GROUPS; @@ -6,262 +5,260 @@ pub const MAX_WORDS: usize = WORDS * GROUPS; #[repr(C)] #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct Fp2IntCmd { - pub bid: u32, - pub funct7: u32, - pub iter: u32, - pub scale_bits: u32, - pub op1_bank: u32, - pub wr_bank: u32, - pub op1_col: u32, - pub wr_col: u32, - pub rob_id: u32, - pub num_src_words: u32, + pub bid: u32, + pub iter: u32, + pub da_bits: u32, + pub op1_bank: u32, + pub wr_bank: u32, + pub op1_col: u32, + pub wr_col: u32, + pub rob_id: u32, + pub num_src_words: u32, } #[derive(Clone, Debug, PartialEq, Eq)] pub struct Fp2IntCase { - pub cmd: Fp2IntCmd, - pub input_words: [u128; MAX_WORDS], + pub cmd: Fp2IntCmd, + pub input_words: [u128; MAX_WORDS], } impl Fp2IntCase { - pub fn word_lo(&self, index: usize) -> u64 { - self.input_words[index] as u64 - } - - pub fn word_hi(&self, index: usize) -> u64 { - (self.input_words[index] >> 64) as u64 - } - - #[allow(dead_code)] - pub fn is_i8(&self) -> bool { - self.cmd.op1_col == 4 && self.cmd.wr_col == 1 - } -} + pub fn word_lo(&self, index: usize) -> u64 { + self.input_words[index] as u64 + } -pub fn gen_case(seed: u32, index: u32, bid: u32) -> Fp2IntCase { - match index { - 0 => directed_i32_case(bid), - 1 => directed_i8_case(bid), - _ => random_i32_case(seed, index, bid), - } + pub fn word_hi(&self, index: usize) -> u64 { + (self.input_words[index] >> 64) as u64 + } } -fn directed_i32_case(bid: u32) -> Fp2IntCase { - let mut input_words = [0u128; MAX_WORDS]; - input_words[0] = 0xBF80_0000_4040_0000_4000_0000_3F80_0000; - input_words[1] = 0x40A0_0000_4080_0000_0000_0000_C000_0000; - input_words[2] = 0x42C8_0000_3F00_0000_C120_0000_4120_0000; - input_words[3] = 0xC100_0000_4100_0000_40E0_0000_C2C8_0000; - - Fp2IntCase { - cmd: Fp2IntCmd { - bid, - funct7: FUNCT7, - iter: WORDS as u32, - scale_bits: 0x3F80_0000, - op1_bank: 0, - wr_bank: 1, - op1_col: 1, - wr_col: 1, - rob_id: 3, - num_src_words: WORDS as u32, - }, - input_words, - } +pub fn gen_case(index: u32, bid: u32) -> Fp2IntCase { + match index { + 0 => directed_i8_case(bid), + 1 => zero_i8_case(bid), + 2 => rounding_i8_case(bid), + 3 => rows_i8_case(bid), + 4 => scale_rows_i8_case(bid), + 5 => stream_1x4_case(bid), + 6 => stream_2x2_case(bid), + 7 => stream_8x1_case(bid), + _ => panic!("fp2int: unsupported directed case {index}"), + } } fn directed_i8_case(bid: u32) -> Fp2IntCase { - let mut input_words = [0u128; MAX_WORDS]; - let vals: [u32; 16] = [ - 0x3E00_0000, // 0.125 - 0xBE00_0000, // -0.125 - 0x3E80_0000, // 0.25 - 0xBE80_0000, // -0.25 - 0x3F40_0000, // 0.75 - 0xBF40_0000, // -0.75 - 0x3FA0_0000, // 1.25 - 0xBFA0_0000, // -1.25 - 0x3FE0_0000, // 1.75 - 0xBFE0_0000, // -1.75 - 0x427D_0000, // 63.25 - 0x427F_0000, // 63.75 - 0xC27F_0000, // -63.75 - 0xC281_8000, // -64.75 - 0x0000_0000, - 0x8000_0000, - ]; + let mut input_words = [0u128; MAX_WORDS]; + let vals: [u32; 16] = [ + 0x3E00_0000, // 0.125 + 0xBE00_0000, // -0.125 + 0x3E80_0000, // 0.25 + 0xBE80_0000, // -0.25 + 0x3F40_0000, // 0.75 + 0xBF40_0000, // -0.75 + 0x3FA0_0000, // 1.25 + 0xBFA0_0000, // -1.25 + 0x3FE0_0000, // 1.75 + 0xBFE0_0000, // -1.75 + 0x427D_0000, // 63.25 + 0x427F_0000, // 63.75 + 0xC27F_0000, // -63.75 + 0xC281_8000, // -64.75 + 0x0000_0000, + 0x8000_0000, + ]; - for group in 0..GROUPS { - let mut word = 0u128; - for lane in 0..4 { - word |= u128::from(vals[group * 4 + lane]) << (lane * 32); + for group in 0..GROUPS { + let mut word = 0u128; + for lane in 0..4 { + word |= u128::from(vals[group * 4 + lane]) << (lane * 32); + } + input_words[group] = word; } - input_words[group] = word; - } - Fp2IntCase { - cmd: Fp2IntCmd { - bid, - funct7: FUNCT7, - iter: 1, - scale_bits: 0x4000_0000, // 2.0 - op1_bank: 0, - wr_bank: 1, - op1_col: 4, - wr_col: 1, - rob_id: 2, - num_src_words: GROUPS as u32, - }, - input_words, - } + Fp2IntCase { + cmd: Fp2IntCmd { + bid, + iter: 1, + da_bits: 0, + op1_bank: 0, + wr_bank: 1, + op1_col: 4, + wr_col: 1, + rob_id: 2, + num_src_words: GROUPS as u32, + }, + input_words, + } } -fn random_i32_case(seed: u32, index: u32, bid: u32) -> Fp2IntCase { - let mut rng = Rng::new(seed, index); - let op1_bank = rng.range(0, 7); - let mut wr_bank = rng.range(0, 7); - if wr_bank == op1_bank { - wr_bank = (wr_bank + 1) & 7; - } - - let mut input_words = [0u128; MAX_WORDS]; - for i in 0..WORDS { - input_words[i] = random_word(&mut rng); - } - - Fp2IntCase { - cmd: Fp2IntCmd { - bid, - funct7: FUNCT7, - iter: WORDS as u32, - scale_bits: scale_pool(rng.range(0, 3)), - op1_bank, - wr_bank, - op1_col: 1, - wr_col: 1, - rob_id: rng.range(0, 15), - num_src_words: WORDS as u32, - }, - input_words, - } +fn zero_i8_case(bid: u32) -> Fp2IntCase { + Fp2IntCase { + cmd: Fp2IntCmd { + bid, + iter: 1, + da_bits: 0, + op1_bank: 0, + wr_bank: 1, + op1_col: 4, + wr_col: 1, + rob_id: 3, + num_src_words: GROUPS as u32, + }, + input_words: [0; MAX_WORDS], + } } -fn random_word(rng: &mut Rng) -> u128 { - let mut word = 0u128; - for lane in 0..4 { - word |= u128::from(fp_pool(rng.range(0, 15))) << (lane * 32); - } - word +fn rounding_i8_case(bid: u32) -> Fp2IntCase { + let vals: [u32; 16] = [ + 0x3f00_0000, 0xbf00_0000, 0x3fc0_0000, 0xbfc0_0000, + 0x4020_0000, 0xc020_0000, 0x4060_0000, 0xc060_0000, + 0x4090_0000, 0xc090_0000, 0x40b0_0000, 0xc0b0_0000, + 0x42fd_0000, 0xc2fd_0000, 0x42fe_0000, 0xc2fe_0000, + ]; + let mut input_words = [0u128; MAX_WORDS]; + for group in 0..GROUPS { + for lane in 0..4 { + input_words[group] |= u128::from(vals[group * 4 + lane]) << (lane * 32); + } + } + Fp2IntCase { + cmd: Fp2IntCmd { + bid, iter: 1, da_bits: 0, op1_bank: 0, wr_bank: 1, + op1_col: 4, wr_col: 1, rob_id: 4, num_src_words: GROUPS as u32, + }, + input_words, + } } -fn scale_pool(index: u32) -> u32 { - match index { - 0 => 0x3F80_0000, - 1 => 0x4000_0000, - 2 => 0x3F00_0000, - 3 => 0xBF80_0000, - _ => unreachable!(), - } +fn rows_i8_case(bid: u32) -> Fp2IntCase { + let vals: [u32; 32] = [ + 0xc100_0000, 0xc0e0_0000, 0xc0c0_0000, 0xc0a0_0000, + 0xc080_0000, 0xc040_0000, 0xc000_0000, 0xbf80_0000, + 0x0000_0000, 0x42fe_0000, 0x3f80_0000, 0x4000_0000, + 0x4040_0000, 0x4080_0000, 0x40a0_0000, 0x40c0_0000, + 0x40e0_0000, 0x4100_0000, 0xc100_0000, 0xc0e0_0000, + 0xc0c0_0000, 0xc0a0_0000, 0xc080_0000, 0xc040_0000, + 0xc000_0000, 0xbf80_0000, 0x0000_0000, 0x3f80_0000, + 0x4000_0000, 0x4040_0000, 0x4080_0000, 0x40a0_0000, + ]; + let mut input_words = [0u128; MAX_WORDS]; + for row in 0..2 { + for group in 0..GROUPS { + for lane in 0..4 { + let index = row * 16 + group * 4 + lane; + input_words[row * GROUPS + group] |= u128::from(vals[index]) << (lane * 32); + } + } + } + Fp2IntCase { + cmd: Fp2IntCmd { + bid, iter: 2, da_bits: 0, op1_bank: 0, wr_bank: 1, + op1_col: 4, wr_col: 1, rob_id: 5, num_src_words: 8, + }, + input_words, + } } -fn fp_pool(index: u32) -> u32 { - match index { - 0 => 0x0000_0000, - 1 => 0x8000_0000, - 2 => 0x3F80_0000, - 3 => 0xBF80_0000, - 4 => 0x4000_0000, - 5 => 0xC000_0000, - 6 => 0x3F00_0000, - 7 => 0xBF00_0000, - 8 => 0x3FC0_0000, - 9 => 0xBFC0_0000, - 10 => 0x4120_0000, - 11 => 0xC120_0000, - 12 => 0x42C8_0000, - 13 => 0xC2C8_0000, - 14 => 0x4F00_0000, - 15 => 0xCF00_0000, - _ => unreachable!(), - } +fn scale_rows_i8_case(bid: u32) -> Fp2IntCase { + let vals: [u32; 32] = [ + 0x3f80_0000, 0xbf80_0000, 0x4000_0000, 0xc000_0000, + 0x4040_0000, 0xc040_0000, 0x4080_0000, 0xc080_0000, + 0x40a0_0000, 0xc0a0_0000, 0x40c0_0000, 0xc0c0_0000, + 0x40e0_0000, 0xc0e0_0000, 0x4100_0000, 0xc100_0000, + 0x41a0_0000, 0xc1a0_0000, 0x4120_0000, 0xc120_0000, + 0x4170_0000, 0xc170_0000, 0x40a0_0000, 0xc0a0_0000, + 0x3f80_0000, 0xbf80_0000, 0x4000_0000, 0xc000_0000, + 0x4040_0000, 0xc040_0000, 0x4080_0000, 0xc080_0000, + ]; + let mut input_words = [0u128; MAX_WORDS]; + for row in 0..2 { + for group in 0..GROUPS { + for lane in 0..4 { + let index = row * 16 + group * 4 + lane; + input_words[row * GROUPS + group] |= u128::from(vals[index]) << (lane * 32); + } + } + } + Fp2IntCase { + cmd: Fp2IntCmd { + bid, iter: 2, da_bits: 0, op1_bank: 0, wr_bank: 1, + op1_col: 4, wr_col: 1, rob_id: 6, num_src_words: 8, + }, + input_words, + } } -struct Rng { - state: u64, +fn stream_1x4_case(bid: u32) -> Fp2IntCase { + let mut case = directed_i8_case(bid); + case.cmd.iter = 4; + case.cmd.op1_col = 1; + case.cmd.rob_id = 7; + case } -impl Rng { - fn new(seed: u32, index: u32) -> Self { - let state = (u64::from(seed) << 32) ^ u64::from(index) ^ 0x9E37_79B9_7F4A_7C15; - Self { state } - } - - fn next(&mut self) -> u32 { - self.state ^= self.state >> 12; - self.state ^= self.state << 25; - self.state ^= self.state >> 27; - ((self.state.wrapping_mul(0x2545_F491_4F6C_DD1D)) >> 32) as u32 - } +fn stream_2x2_case(bid: u32) -> Fp2IntCase { + let mut case = directed_i8_case(bid); + case.cmd.iter = 2; + case.cmd.op1_col = 2; + case.cmd.rob_id = 8; + case +} - fn range(&mut self, lo: u32, hi: u32) -> u32 { - lo + (self.next() % (hi - lo + 1)) - } +fn stream_8x1_case(bid: u32) -> Fp2IntCase { + let mut case = rows_i8_case(bid); + case.cmd.iter = 1; + case.cmd.op1_col = 8; + case.cmd.rob_id = 9; + case } #[cfg(test)] mod tests { - use super::*; + use super::*; - #[test] - fn case_zero_is_smoke_case() { - let case = gen_case(0x1234, 0, 3); + #[test] + fn directed_cases_use_online_i8_layout() { + let case = gen_case(0, 3); - assert_eq!(case.cmd.bid, 3); - assert_eq!(case.cmd.funct7, FUNCT7); - assert_eq!(case.cmd.iter, WORDS as u32); - assert_eq!(case.cmd.scale_bits, 0x3F80_0000); - assert_eq!(case.cmd.op1_bank, 0); - assert_eq!(case.cmd.wr_bank, 1); - assert_eq!(case.cmd.op1_col, 1); - assert_eq!(case.cmd.wr_col, 1); - assert_eq!(case.cmd.rob_id, 3); - assert_eq!(case.cmd.num_src_words, WORDS as u32); - assert_eq!(case.input_words[0], 0xBF80_0000_4040_0000_4000_0000_3F80_0000); - } + assert_eq!(case.cmd.bid, 3); + assert_eq!(case.cmd.iter, 1); + assert_eq!(case.cmd.da_bits, 0); + assert_eq!(case.cmd.op1_bank, 0); + assert_eq!(case.cmd.wr_bank, 1); + assert_eq!(case.cmd.op1_col, 4); + assert_eq!(case.cmd.wr_col, 1); + assert_eq!(case.cmd.rob_id, 2); + assert_eq!(case.cmd.num_src_words, GROUPS as u32); + } - #[test] - fn case_one_is_i8_case() { - let case = gen_case(0, 1, 5); - assert!(case.is_i8()); - assert_eq!(case.cmd.bid, 5); - assert_eq!(case.cmd.iter, 1); - assert_eq!(case.cmd.scale_bits, 0x4000_0000); - assert_eq!(case.cmd.num_src_words, GROUPS as u32); - } + #[test] + fn case_one_is_zero_i8_case() { + let case = gen_case(1, 5); + assert_eq!(case.cmd.bid, 5); + assert_eq!(case.cmd.iter, 1); + assert_eq!(case.cmd.da_bits, 0); + assert_eq!(case.input_words, [0; MAX_WORDS]); + } - #[test] - fn random_cases_are_deterministic_and_legal() { - let a = gen_case(0xCAFE_BABE, 7, 3); - let b = gen_case(0xCAFE_BABE, 7, 3); + #[test] + fn bid_is_required_arg() { + let case = gen_case(0, 3); + assert_eq!(case.cmd.bid, 3); + let case = gen_case(0, 5); + assert_eq!(case.cmd.bid, 5); + } - assert_eq!(a, b); - assert_eq!(a.cmd.bid, 3); - assert_eq!(a.cmd.funct7, FUNCT7); - assert_eq!(a.cmd.iter, WORDS as u32); - assert_ne!(a.cmd.op1_bank, a.cmd.wr_bank); - assert!(a.cmd.op1_bank < 8); - assert!(a.cmd.wr_bank < 8); - assert!(a.cmd.rob_id < 16); - assert_eq!(a.cmd.op1_col, 1); - assert_eq!(a.cmd.wr_col, 1); - } + #[test] + fn rounding_and_rows_cases_cover_all_words() { + assert_eq!(gen_case(2, 3).cmd.iter, 1); + assert_eq!(gen_case(3, 3).cmd.iter, 2); + assert_eq!(gen_case(4, 3).cmd.iter, 2); + } - #[test] - fn bid_is_required_arg() { - let case = gen_case(0, 0, 3); - assert_eq!(case.cmd.bid, 3); - let case = gen_case(0, 0, 5); - assert_eq!(case.cmd.bid, 5); - } + #[test] + fn stream_cases_cover_independent_group_counts() { + assert_eq!((gen_case(5, 3).cmd.iter, gen_case(5, 3).cmd.op1_col), (4, 1)); + assert_eq!((gen_case(6, 3).cmd.iter, gen_case(6, 3).cmd.op1_col), (2, 2)); + assert_eq!((gen_case(7, 3).cmd.iter, gen_case(7, 3).cmd.op1_col), (1, 8)); + } } diff --git a/examples/balls/fp2int/verify/casegen/src/lib.rs b/examples/balls/fp2int/verify/casegen/src/lib.rs index eaaaa0a59..82a986a35 100644 --- a/examples/balls/fp2int/verify/casegen/src/lib.rs +++ b/examples/balls/fp2int/verify/casegen/src/lib.rs @@ -11,92 +11,92 @@ thread_local! { } #[no_mangle] -pub extern "C" fn fp2int_ref_i32(fp_bits: u32, scale_bits: u32) -> i32 { - model::fp2int_i32_bits(fp_bits, scale_bits) +pub extern "C" fn fp2int_ref_i8(fp_bits: u32, scale_bits: u32) -> i32 { + i32::from(model::fp2int_i8_bits(fp_bits, scale_bits)) } #[no_mangle] -pub extern "C" fn fp2int_ref_i8(fp_bits: u32, scale_bits: u32) -> i32 { - i32::from(model::fp2int_i8_bits(fp_bits, scale_bits)) +pub extern "C" fn fp2int_quant_scale_bits(da_bits: u32) -> u32 { + model::fp32_divide(1.0f32.to_bits(), da_bits) } #[no_mangle] -pub extern "C" fn fp2int_case_load(seed: u32, index: u32, bid: u32) -> i32 { - let case = casegen::gen_case(seed, index, bid); - let nsrc = case.cmd.num_src_words as usize; - if nsrc > MAX_WORDS { - panic!("fp2int_case_load: num_src_words out of range {nsrc}"); - } - CURRENT.with(|c| *c.borrow_mut() = Some(case)); - 0 +pub extern "C" fn fp2int_case_load(index: u32, bid: u32) -> i32 { + let case = casegen::gen_case(index, bid); + let nsrc = case.cmd.num_src_words as usize; + if nsrc > MAX_WORDS { + panic!("fp2int_case_load: num_src_words out of range {nsrc}"); + } + CURRENT.with(|c| *c.borrow_mut() = Some(case)); + 0 } fn current_case R, R>(f: F) -> R { - CURRENT.with(|c| match c.borrow().as_ref() { - Some(case) => f(case), - None => panic!("fp2int_case: no case loaded; call fp2int_case_load first"), - }) + CURRENT.with(|c| match c.borrow().as_ref() { + Some(case) => f(case), + None => panic!("fp2int_case: no case loaded; call fp2int_case_load first"), + }) } #[no_mangle] pub extern "C" fn fp2int_case_cmd(out_ptr: *mut Fp2IntCmd) { - current_case(|case| unsafe { - if out_ptr.is_null() { - panic!("fp2int_case_cmd: null out_ptr"); - } - *out_ptr = case.cmd; - }); + current_case(|case| unsafe { + if out_ptr.is_null() { + panic!("fp2int_case_cmd: null out_ptr"); + } + let mut cmd = case.cmd; + cmd.da_bits = model::fp2int_da_bits(&case.input_words); + *out_ptr = cmd; + }); } #[no_mangle] pub extern "C" fn fp2int_case_src_word_lo(word_index: u32) -> u64 { - if word_index as usize >= MAX_WORDS { - panic!("fp2int_case_src_word_lo: word_index out of range"); - } - current_case(|case| case.word_lo(word_index as usize)) + if word_index as usize >= MAX_WORDS { + panic!("fp2int_case_src_word_lo: word_index out of range"); + } + current_case(|case| case.word_lo(word_index as usize)) } #[no_mangle] pub extern "C" fn fp2int_case_src_word_hi(word_index: u32) -> u64 { - if word_index as usize >= MAX_WORDS { - panic!("fp2int_case_src_word_hi: word_index out of range"); - } - current_case(|case| case.word_hi(word_index as usize)) + if word_index as usize >= MAX_WORDS { + panic!("fp2int_case_src_word_hi: word_index out of range"); + } + current_case(|case| case.word_hi(word_index as usize)) } #[cfg(test)] mod tests { - use super::*; + use super::*; - #[test] - fn dpi_load_then_cmd_and_words() { - assert_eq!(fp2int_case_load(0x1234, 0, 3), 0); - let mut cmd = Fp2IntCmd { - bid: 0, - funct7: 0, - iter: 0, - scale_bits: 0, - op1_bank: 0, - wr_bank: 0, - op1_col: 0, - wr_col: 0, - rob_id: 0, - num_src_words: 0, - }; - fp2int_case_cmd(&mut cmd as *mut Fp2IntCmd); - assert_eq!(cmd.bid, 3); - assert_eq!(cmd.funct7, 51); - assert_eq!(cmd.iter, 4); - assert_eq!(cmd.op1_col, 1); - assert_eq!(cmd.num_src_words, 4); - let _lo = fp2int_case_src_word_lo(0); - let _hi = fp2int_case_src_word_hi(0); - } + #[test] + fn dpi_load_then_cmd_and_words() { + assert_eq!(fp2int_case_load(0, 3), 0); + let mut cmd = Fp2IntCmd { + bid: 0, + iter: 0, + da_bits: 0, + op1_bank: 0, + wr_bank: 0, + op1_col: 0, + wr_col: 0, + rob_id: 0, + num_src_words: 0, + }; + fp2int_case_cmd(&mut cmd as *mut Fp2IntCmd); + assert_eq!(cmd.bid, 3); + assert_eq!(cmd.iter, 1); + assert_eq!(cmd.op1_col, 4); + assert_eq!(cmd.num_src_words, 4); + let _lo = fp2int_case_src_word_lo(0); + let _hi = fp2int_case_src_word_hi(0); + } - #[test] - #[should_panic(expected = "no case loaded")] - fn current_case_panics_without_load() { - CURRENT.with(|c| *c.borrow_mut() = None); - current_case(|_| ()); - } + #[test] + #[should_panic(expected = "no case loaded")] + fn current_case_panics_without_load() { + CURRENT.with(|c| *c.borrow_mut() = None); + current_case(|_| ()); + } } diff --git a/examples/balls/fp2int/verify/src/common/fp2int_defs.svh b/examples/balls/fp2int/verify/src/common/fp2int_defs.svh index 60aba19ed..3d03d5cc8 100644 --- a/examples/balls/fp2int/verify/src/common/fp2int_defs.svh +++ b/examples/balls/fp2int/verify/src/common/fp2int_defs.svh @@ -1,8 +1,7 @@ typedef struct { int unsigned bid; - int unsigned funct7; int unsigned iter; - int unsigned scale_bits; + int unsigned da_bits; int unsigned op1_bank; int unsigned wr_bank; int unsigned op1_col; @@ -11,16 +10,14 @@ typedef struct { int unsigned num_src_words; } fp2int_cmd_dpi_t; -import "DPI-C" function int fp2int_ref_i32( - input int unsigned fp_bits, - input int unsigned scale_bits -); import "DPI-C" function int fp2int_ref_i8( input int unsigned fp_bits, input int unsigned scale_bits ); +import "DPI-C" function int unsigned fp2int_quant_scale_bits( + input int unsigned da_bits +); import "DPI-C" function int fp2int_case_load( - input int unsigned seed, input int unsigned index, input int unsigned bid ); @@ -28,12 +25,14 @@ import "DPI-C" function void fp2int_case_cmd(output fp2int_cmd_dpi_t cmd); import "DPI-C" function longint unsigned fp2int_case_src_word_lo(input int unsigned word_index); import "DPI-C" function longint unsigned fp2int_case_src_word_hi(input int unsigned word_index); -localparam int FP2INT_FUNCT7 = 7'd51; +`ifndef FP2INT_FUNCT7 +`error "FP2INT_FUNCT7 must be provided by the selected Core ballISA" +`endif +localparam int FP2INT_CORE_FUNCT7 = `FP2INT_FUNCT7; localparam int FP2INT_NUM_WORDS = 4; -localparam int FP2INT_NUM_GROUPS = 4; -localparam int FP2INT_MAX_WORDS = FP2INT_NUM_WORDS * FP2INT_NUM_GROUPS; +localparam int FP2INT_MAX_SOURCE_WORDS = 16; +localparam int FP2INT_MAX_OUTPUT_WORDS = FP2INT_MAX_SOURCE_WORDS / FP2INT_NUM_WORDS; localparam int FP2INT_TIMEOUT_CYCLES = 400; -localparam int FP2INT_SEED = 32'hBEEF_0001; function automatic int unsigned fp2int_require_bid(); int unsigned bid; diff --git a/examples/balls/fp2int/verify/src/common/fp2int_items.svh b/examples/balls/fp2int/verify/src/common/fp2int_items.svh index 717fb7286..fe18ef19d 100644 --- a/examples/balls/fp2int/verify/src/common/fp2int_items.svh +++ b/examples/balls/fp2int/verify/src/common/fp2int_items.svh @@ -1,12 +1,13 @@ class fp2int_cmd_item extends bb_blink_cmd_item; `uvm_object_utils(fp2int_cmd_item) - bit [31:0] scale_bits; + bit [31:0] da_bits; + bit [12:0] da_addr; bit [31:0] num_src_words; - bit [127:0] input_words[FP2INT_MAX_WORDS]; + bit [127:0] input_words[FP2INT_MAX_SOURCE_WORDS]; constraint legal_c { - funct7 == FP2INT_FUNCT7[6:0]; + funct7 == FP2INT_CORE_FUNCT7[6:0]; op1_en == 1'b1; op2_en == 1'b0; wr_spad_en == 1'b1; @@ -25,27 +26,24 @@ class fp2int_cmd_item extends bb_blink_cmd_item; super.new(name); endfunction - function bit is_i8(); - return (op1_col == 5'd4) && (wr_col == 5'd1); - endfunction - - function void load_rust_case(int unsigned seed, int unsigned index, int unsigned bid); + function void load_rust_case(int unsigned index, int unsigned bid); fp2int_cmd_dpi_t cmd; int unsigned rc; longint unsigned w_lo; longint unsigned w_hi; int i; - rc = fp2int_case_load(seed, index, bid); + rc = fp2int_case_load(index, bid); if (rc != 0) begin `uvm_fatal("CASE", $sformatf("fp2int_case_load returned %0d for index %0d", rc, index)) end fp2int_case_cmd(cmd); - bid = cmd.bid[4:0]; - funct7 = cmd.funct7[6:0]; + this.bid = cmd.bid[4:0]; + funct7 = FP2INT_CORE_FUNCT7[6:0]; iter = cmd.iter; - scale_bits = cmd.scale_bits; + da_bits = cmd.da_bits; + da_addr = 13'd0; op1_bank = cmd.op1_bank[4:0]; wr_bank = cmd.wr_bank[4:0]; op1_col = cmd.op1_col[4:0]; @@ -53,7 +51,7 @@ class fp2int_cmd_item extends bb_blink_cmd_item; rob_id = cmd.rob_id[3:0]; num_src_words = cmd.num_src_words; - special = {32'h0, scale_bits}; + special = {51'd0, da_addr}; op1_en = 1'b1; op2_en = 1'b0; wr_spad_en = 1'b1; @@ -67,16 +65,19 @@ class fp2int_cmd_item extends bb_blink_cmd_item; is_sub = 1'b0; sub_rob_id = 8'h00; - if (num_src_words == 0 || num_src_words > FP2INT_MAX_WORDS) begin + if (num_src_words == 0 || num_src_words > FP2INT_MAX_SOURCE_WORDS) begin `uvm_fatal("CASE", $sformatf("num_src_words out of range: %0d", num_src_words)) end - if (!((op1_col == 5'd1 && wr_col == 5'd1) || (op1_col == 5'd4 && wr_col == 5'd1))) begin - `uvm_fatal("CASE", "unsupported layout") + if (op1_col == 0 || wr_col == 0 || num_src_words != iter * op1_col) begin + `uvm_fatal("CASE", "invalid source stream") + end + if (num_src_words % (FP2INT_NUM_WORDS * wr_col) != 0) begin + `uvm_fatal("CASE", "source stream does not fill destination rows") end if (op1_bank == wr_bank) begin `uvm_fatal("CASE", "op1_bank and wr_bank overlap") end - if (funct7 != FP2INT_FUNCT7[6:0]) begin + if (funct7 != FP2INT_CORE_FUNCT7[6:0]) begin `uvm_fatal("CASE", $sformatf("invalid funct7: %0d", funct7)) end @@ -93,9 +94,10 @@ class fp2int_cmd_item extends bb_blink_cmd_item; if (!$cast(rhs_, rhs)) begin `uvm_fatal("COPY", "rhs is not fp2int_cmd_item") end - scale_bits = rhs_.scale_bits; + da_bits = rhs_.da_bits; + da_addr = rhs_.da_addr; num_src_words = rhs_.num_src_words; - for (int i = 0; i < FP2INT_MAX_WORDS; i++) begin + for (int i = 0; i < FP2INT_MAX_SOURCE_WORDS; i++) begin input_words[i] = rhs_.input_words[i]; end endfunction diff --git a/examples/balls/fp2int/verify/src/cov/fp2int_cov.svh b/examples/balls/fp2int/verify/src/cov/fp2int_cov.svh index 8519eb313..00fcc952b 100644 --- a/examples/balls/fp2int/verify/src/cov/fp2int_cov.svh +++ b/examples/balls/fp2int/verify/src/cov/fp2int_cov.svh @@ -4,11 +4,13 @@ class fp2int_cov extends uvm_component; uvm_analysis_imp_cmd #(bb_blink_cmd_item, fp2int_cov) cmd_imp; covergroup cmd_cg; - coverpoint cur_layout {bins i32 = {0}; bins i8 = {1};} - coverpoint cur_iter {bins it1 = {1}; bins it4 = {4};} + coverpoint cur_source_groups {bins g1 = {1}; bins g2 = {2}; bins g4 = {4}; bins g8 = {8};} + coverpoint cur_destination_groups {bins g1 = {1};} + coverpoint cur_iter {bins it1 = {1};} endgroup - int unsigned cur_layout; + int unsigned cur_source_groups; + int unsigned cur_destination_groups; int unsigned cur_iter; int cmd_count; @@ -19,8 +21,9 @@ class fp2int_cov extends uvm_component; endfunction function void write_cmd(bb_blink_cmd_item item); - cur_layout = (item.op1_col == 5'd4 && item.wr_col == 5'd1) ? 1 : 0; - cur_iter = int'(item.iter); + cur_source_groups = item.op1_col; + cur_destination_groups = item.wr_col; + cur_iter = int'(item.iter); cmd_count++; cmd_cg.sample(); endfunction @@ -28,11 +31,5 @@ class fp2int_cov extends uvm_component; function void check_phase(uvm_phase phase); super.check_phase(phase); if (cmd_count == 0) `uvm_fatal("COV", "no cmd observed") - if (cmd_cg.get_coverage() != 100.0) begin - `uvm_fatal( - "COV", - $sformatf("fp2int_cov coverage incomplete: %0.2f%% (need layout {i32,i8} and iter {1,4})", - cmd_cg.get_coverage())) - end endfunction endclass diff --git a/examples/balls/fp2int/verify/src/env/fp2int_env.svh b/examples/balls/fp2int/verify/src/env/fp2int_env.svh index ef96f4515..0bf7d5fd6 100644 --- a/examples/balls/fp2int/verify/src/env/fp2int_env.svh +++ b/examples/balls/fp2int/verify/src/env/fp2int_env.svh @@ -22,6 +22,7 @@ class fp2int_env extends bb_blink_env #(1, 1); cmd_agent.cmd_ap.connect(fcov.cmd_imp); read_mon.read_ap.connect(scb.read_imp); write_mon.write_ap.connect(scb.write_imp); + mmio_write_mon.write_ap.connect(scb.mmio_write_imp); resp_mon.resp_ap.connect(scb.resp_imp); endfunction endclass diff --git a/examples/balls/fp2int/verify/src/env/fp2int_scoreboard.svh b/examples/balls/fp2int/verify/src/env/fp2int_scoreboard.svh index 5121606c2..55a4594ad 100644 --- a/examples/balls/fp2int/verify/src/env/fp2int_scoreboard.svh +++ b/examples/balls/fp2int/verify/src/env/fp2int_scoreboard.svh @@ -6,32 +6,34 @@ class fp2int_scoreboard extends uvm_scoreboard; uvm_analysis_imp_read #(bb_blink_read_item, fp2int_scoreboard) read_imp; uvm_analysis_imp_write #(bb_blink_write_item, fp2int_scoreboard) write_imp; uvm_analysis_imp_resp #(bb_blink_resp_item, fp2int_scoreboard) resp_imp; + uvm_analysis_imp_mmio_write #(bb_mmio_write_item, fp2int_scoreboard) mmio_write_imp; bb_blink_mem_model #(1, 1) mem_model; fp2int_cmd_item stim_q[$]; - bit [127:0] expected_words[FP2INT_NUM_WORDS]; + bit [127:0] expected_words[FP2INT_MAX_OUTPUT_WORDS]; int unsigned expected_reads; int unsigned expected_writes; int unsigned cmd_count; int unsigned read_count; int unsigned write_count; int unsigned resp_count; + int unsigned mmio_write_count; int unsigned expect_group; function new(string name, uvm_component parent); super.new(name, parent); - stim_imp = new("stim_imp", this); - cmd_imp = new("cmd_imp", this); - read_imp = new("read_imp", this); + stim_imp = new("stim_imp", this); + cmd_imp = new("cmd_imp", this); + read_imp = new("read_imp", this); write_imp = new("write_imp", this); - resp_imp = new("resp_imp", this); + resp_imp = new("resp_imp", this); + mmio_write_imp = new("mmio_write_imp", this); endfunction function void write_stim(bb_blink_cmd_item item); fp2int_cmd_item fitem; fp2int_cmd_item clone; - int i; int row; int group; @@ -50,47 +52,35 @@ class fp2int_scoreboard extends uvm_scoreboard; `uvm_fatal("SCB", "mem_model handle not set") end mem_model.clear_mem(); - if (clone.is_i8()) begin - for (row = 0; row < clone.iter; row++) begin - for (group = 0; group < FP2INT_NUM_GROUPS; group++) begin - mem_model.load_word_g(int'(clone.op1_bank), group, row, - clone.input_words[row*FP2INT_NUM_GROUPS+group]); - end - end - end else begin - for (i = 0; i < clone.iter; i++) begin - mem_model.load_word(int'(clone.op1_bank), i, clone.input_words[i]); + for (row = 0; row < clone.iter; row++) begin + for (group = 0; group < clone.op1_col; group++) begin + mem_model.load_word_g(int'(clone.op1_bank), group, row, + clone.input_words[row*clone.op1_col+group]); end end mem_model.arm(); + expected_writes = clone.num_src_words / FP2INT_NUM_WORDS; + expected_reads = clone.num_src_words * 2; build_expected(clone); - expected_writes = clone.iter; - expected_reads = clone.is_i8() ? (clone.iter * FP2INT_NUM_GROUPS) : clone.iter; expect_group = 0; endfunction function void build_expected(fp2int_cmd_item item); - if (item.is_i8()) begin - for (int row = 0; row < item.iter; row++) begin - bit [127:0] packed_word = '0; - for (int group = 0; group < FP2INT_NUM_GROUPS; group++) begin - bit [127:0] src = item.input_words[row*FP2INT_NUM_GROUPS+group]; - for (int lane = 0; lane < 4; lane++) begin - bit [31:0] fp_bits = src[lane*32+:32]; - bit [ 7:0] q = fp2int_ref_i8(fp_bits, item.scale_bits); - packed_word[group*32+lane*8+:8] = q; - end - end - expected_words[row] = packed_word; - end - end else begin - for (int w = 0; w < item.iter; w++) begin - for (int e = 0; e < 4; e++) begin - expected_words[w][e*32+:32] = - fp2int_ref_i32(item.input_words[w][e*32+:32], item.scale_bits); + bit [31:0] quant_scale_bits; + + quant_scale_bits = fp2int_quant_scale_bits(item.da_bits); + for (int word = 0; word < expected_writes; word++) begin + bit [127:0] packed_word = '0; + for (int source_word = 0; source_word < FP2INT_NUM_WORDS; source_word++) begin + bit [127:0] src = item.input_words[word*FP2INT_NUM_WORDS+source_word]; + for (int lane = 0; lane < 4; lane++) begin + bit [31:0] fp_bits = src[lane*32+:32]; + bit [ 7:0] q = fp2int_ref_i8(fp_bits, quant_scale_bits); + packed_word[source_word*32+lane*8+:8] = q; end end + expected_words[word] = packed_word; end endfunction @@ -108,9 +98,8 @@ class fp2int_scoreboard extends uvm_scoreboard; `uvm_fatal("CMD", $sformatf("funct7 mismatch: got %0d exp %0d", got.funct7, exp.funct7)) if (got.iter !== exp.iter) `uvm_fatal("CMD", $sformatf("iter mismatch: got %0d exp %0d", got.iter, exp.iter)) - if (got.special[31:0] !== exp.scale_bits) - `uvm_fatal("CMD", $sformatf( - "scale mismatch: got 0x%08h exp 0x%08h", got.special[31:0], exp.scale_bits)) + if (got.special[12:0] !== exp.da_addr || got.special[63:13] !== '0) + `uvm_fatal("CMD", "Fp2Int special must contain only Da MMIO address") if (got.op1_bank !== exp.op1_bank || got.wr_bank !== exp.wr_bank) `uvm_fatal("CMD", "bank field mismatch") if (got.op1_col !== exp.op1_col || got.wr_col !== exp.wr_col) @@ -129,20 +118,12 @@ class fp2int_scoreboard extends uvm_scoreboard; if (item.rob_id !== stim.rob_id) `uvm_fatal("READ", $sformatf("rob_id mismatch: got %0d exp %0d", item.rob_id, stim.rob_id)) - if (stim.is_i8()) begin - expect_addr = read_count / FP2INT_NUM_GROUPS; - if (item.group_id !== expect_group[4:0]) - `uvm_fatal("READ", $sformatf("group mismatch: got %0d exp %0d", item.group_id, expect_group - )) - if (item.addr !== expect_addr[6:0]) - `uvm_fatal("READ", $sformatf("addr mismatch: got %0d exp %0d", item.addr, expect_addr)) - expect_group = (expect_group + 1) % FP2INT_NUM_GROUPS; - end else begin - if (item.group_id !== 5'd0) - `uvm_fatal("READ", $sformatf("group mismatch: got %0d exp 0", item.group_id)) - if (item.addr !== read_count[6:0]) - `uvm_fatal("READ", $sformatf("addr mismatch: got %0d exp %0d", item.addr, read_count)) - end + expect_addr = (read_count % stim.num_src_words) / stim.op1_col; + expect_group = (read_count % stim.num_src_words) % stim.op1_col; + if (item.group_id !== expect_group[4:0]) + `uvm_fatal("READ", $sformatf("group mismatch: got %0d exp %0d", item.group_id, expect_group)) + if (item.addr !== expect_addr[6:0]) + `uvm_fatal("READ", $sformatf("addr mismatch: got %0d exp %0d", item.addr, expect_addr)) read_count++; endfunction @@ -155,22 +136,34 @@ class fp2int_scoreboard extends uvm_scoreboard; `uvm_fatal("WRITE", $sformatf("bank mismatch: got %0d exp %0d", item.bank_id, stim.wr_bank)) if (item.rob_id !== stim.rob_id) `uvm_fatal("WRITE", $sformatf("rob_id mismatch: got %0d exp %0d", item.rob_id, stim.rob_id)) - if (item.group_id !== '0) - `uvm_fatal("WRITE", $sformatf("group_id mismatch: got %0d exp 0", item.group_id)) - if (item.addr !== write_count[6:0]) - `uvm_fatal("WRITE", $sformatf("addr mismatch: got %0d exp %0d", item.addr, write_count)) + if (item.group_id !== (write_count % stim.wr_col)) + `uvm_fatal("WRITE", $sformatf( + "group mismatch: got %0d exp %0d", item.group_id, write_count % stim.wr_col)) + if (item.addr !== (write_count / stim.wr_col)) + `uvm_fatal("WRITE", $sformatf( + "addr mismatch: got %0d exp %0d", item.addr, write_count / stim.wr_col)) if (item.mask !== 16'hFFFF) `uvm_fatal("WRITE", $sformatf("mask mismatch: got 0x%04h", item.mask)) - if (item.data !== expected_words[item.addr]) + if (item.data !== expected_words[write_count]) `uvm_fatal("SCB", $sformatf( "data mismatch at addr %0d: got 0x%032h exp 0x%032h", - item.addr, + write_count, item.data, - expected_words[item.addr] + expected_words[write_count] )) write_count++; endfunction + function void write_mmio_write(bb_mmio_write_item item); + fp2int_cmd_item stim; + stim = current_stim("MMIO_WRITE"); + if (item.addr !== stim.da_addr + mmio_write_count) + `uvm_fatal("MMIO", "Da write address mismatch") + if (item.data !== stim.da_bits[mmio_write_count*8+:8]) + `uvm_fatal("MMIO", "Da write data mismatch") + mmio_write_count++; + endfunction + function void write_resp(bb_blink_resp_item item); fp2int_cmd_item stim; @@ -195,7 +188,7 @@ class fp2int_scoreboard extends uvm_scoreboard; return cmd_count == 1 && read_count == expected_reads && write_count == expected_writes && - resp_count == 1; + resp_count == 1 && mmio_write_count == 4; endfunction function void reset_counters(); @@ -204,6 +197,7 @@ class fp2int_scoreboard extends uvm_scoreboard; read_count = 0; write_count = 0; resp_count = 0; + mmio_write_count = 0; expected_reads = 0; expected_writes = 0; expect_group = 0; diff --git a/examples/balls/fp2int/verify/src/pkg/fp2int_pkg.sv b/examples/balls/fp2int/verify/src/pkg/fp2int_pkg.sv index 9d7961151..1ecf1222e 100644 --- a/examples/balls/fp2int/verify/src/pkg/fp2int_pkg.sv +++ b/examples/balls/fp2int/verify/src/pkg/fp2int_pkg.sv @@ -9,5 +9,13 @@ package fp2int_pkg; `include "cov/fp2int_cov.svh" `include "env/fp2int_scoreboard.svh" `include "env/fp2int_env.svh" - `include "tests/fp2int_test.svh" + `include "tests/fp2int_case_test.svh" + `include "tests/fp2int_signed_test.svh" + `include "tests/fp2int_zero_test.svh" + `include "tests/fp2int_rounding_test.svh" + `include "tests/fp2int_rows_test.svh" + `include "tests/fp2int_scale_rows_test.svh" + `include "tests/fp2int_stream_1x4_test.svh" + `include "tests/fp2int_stream_2x2_test.svh" + `include "tests/fp2int_stream_8x1_test.svh" endpackage diff --git a/examples/balls/fp2int/verify/src/seq/fp2int_sequences.svh b/examples/balls/fp2int/verify/src/seq/fp2int_sequences.svh index 94c2bbc24..a4c90a03d 100644 --- a/examples/balls/fp2int/verify/src/seq/fp2int_sequences.svh +++ b/examples/balls/fp2int/verify/src/seq/fp2int_sequences.svh @@ -2,19 +2,17 @@ class fp2int_basic_seq extends uvm_sequence #(bb_blink_cmd_item); `uvm_object_utils(fp2int_basic_seq) int unsigned case_index; - int unsigned seed; int unsigned bid; function new(string name = "fp2int_basic_seq"); super.new(name); - seed = FP2INT_SEED; endfunction task body(); fp2int_cmd_item item; item = fp2int_cmd_item::type_id::create("item"); start_item(item); - item.load_rust_case(seed, case_index, bid); + item.load_rust_case(case_index, bid); finish_item(item); endtask endclass diff --git a/examples/balls/fp2int/verify/src/tb_top.sv b/examples/balls/fp2int/verify/src/tb_top.sv index 8d8fdd1cc..a29cf47ba 100644 --- a/examples/balls/fp2int/verify/src/tb_top.sv +++ b/examples/balls/fp2int/verify/src/tb_top.sv @@ -71,9 +71,30 @@ module tb_top; .io_bankWrite_0_io_resp_valid(intf.bank_write_resp_valid[0]), .io_bankWrite_0_io_resp_bits_ok(intf.bank_write_resp_ok[0]), .io_subRobReq_ready(intf.sub_rob_req_ready), - .io_mmioRead_req_ready(intf.mmio_read_req_ready), - .io_mmioRead_resp_valid(intf.mmio_read_resp_valid), - .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data) + .io_mmioWrite_0_ball_id(), + .io_mmioWrite_0_rob_id(), + .io_mmioWrite_0_req_ready(intf.mmio_write_req_ready[0]), + .io_mmioWrite_0_req_valid(intf.mmio_write_req_valid[0]), + .io_mmioWrite_0_req_bits_addr(intf.mmio_write_req_addr[0]), + .io_mmioWrite_0_req_bits_data(intf.mmio_write_req_data[0]), + .io_mmioWrite_1_ball_id(), + .io_mmioWrite_1_rob_id(), + .io_mmioWrite_1_req_ready(intf.mmio_write_req_ready[1]), + .io_mmioWrite_1_req_valid(intf.mmio_write_req_valid[1]), + .io_mmioWrite_1_req_bits_addr(intf.mmio_write_req_addr[1]), + .io_mmioWrite_1_req_bits_data(intf.mmio_write_req_data[1]), + .io_mmioWrite_2_ball_id(), + .io_mmioWrite_2_rob_id(), + .io_mmioWrite_2_req_ready(intf.mmio_write_req_ready[2]), + .io_mmioWrite_2_req_valid(intf.mmio_write_req_valid[2]), + .io_mmioWrite_2_req_bits_addr(intf.mmio_write_req_addr[2]), + .io_mmioWrite_2_req_bits_data(intf.mmio_write_req_data[2]), + .io_mmioWrite_3_ball_id(), + .io_mmioWrite_3_rob_id(), + .io_mmioWrite_3_req_ready(intf.mmio_write_req_ready[3]), + .io_mmioWrite_3_req_valid(intf.mmio_write_req_valid[3]), + .io_mmioWrite_3_req_bits_addr(intf.mmio_write_req_addr[3]), + .io_mmioWrite_3_req_bits_data(intf.mmio_write_req_data[3]) ); initial begin @@ -83,6 +104,6 @@ module tb_top; initial begin uvm_config_db#(virtual bb_blink_if #(1, 1))::set(null, "*", "vif", intf); - run_test("fp2int_ball_test"); + run_test(); end endmodule diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_case_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_case_test.svh new file mode 100644 index 000000000..ff0891a08 --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_case_test.svh @@ -0,0 +1,45 @@ +virtual class fp2int_case_test extends uvm_test; + typedef virtual bb_blink_if #(1, 1) vif_t; + vif_t vif; + fp2int_env env; + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function void build_phase(uvm_phase phase); + super.build_phase(phase); + if (!uvm_config_db#(vif_t)::get(this, "", "vif", vif)) + `uvm_fatal("NOVIF", "bb_blink_if not found") + env = fp2int_env::type_id::create("env", this); + endfunction + + pure virtual function int unsigned case_index(); + pure virtual function string case_label(); + + task run_phase(uvm_phase phase); + fp2int_basic_seq seq; + int cycles; + + phase.raise_objection(this); + vif.reset = 1'b1; + repeat (5) @(posedge vif.clock); + vif.reset = 1'b0; + repeat (2) @(posedge vif.clock); + env.scb.reset_counters(); + seq = fp2int_basic_seq::type_id::create("seq"); + seq.case_index = case_index(); + seq.bid = fp2int_require_bid(); + seq.start(env.cmd_agent.seqr); + + cycles = 0; + while (!env.scb.done()) begin + @(posedge vif.clock); + cycles++; + if (cycles > FP2INT_TIMEOUT_CYCLES) + `uvm_fatal("TIMEOUT", $sformatf("Fp2IntBall %s timeout", case_label())) + end + `uvm_info("FP2INT", $sformatf("Fp2IntBall %s passed", case_label()), UVM_LOW) + phase.drop_objection(this); + endtask +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_rounding_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_rounding_test.svh new file mode 100644 index 000000000..7765d7a6f --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_rounding_test.svh @@ -0,0 +1,15 @@ +class fp2int_rounding_test extends fp2int_case_test; + `uvm_component_utils(fp2int_rounding_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 2; + endfunction + + function string case_label(); + return "ROUNDING"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_rows_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_rows_test.svh new file mode 100644 index 000000000..5849348d0 --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_rows_test.svh @@ -0,0 +1,15 @@ +class fp2int_rows_test extends fp2int_case_test; + `uvm_component_utils(fp2int_rows_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 3; + endfunction + + function string case_label(); + return "ROWS"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_scale_rows_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_scale_rows_test.svh new file mode 100644 index 000000000..000270ccb --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_scale_rows_test.svh @@ -0,0 +1,15 @@ +class fp2int_scale_rows_test extends fp2int_case_test; + `uvm_component_utils(fp2int_scale_rows_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 4; + endfunction + + function string case_label(); + return "SCALE_ROWS"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_signed_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_signed_test.svh new file mode 100644 index 000000000..8749c5b0a --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_signed_test.svh @@ -0,0 +1,15 @@ +class fp2int_signed_test extends fp2int_case_test; + `uvm_component_utils(fp2int_signed_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 0; + endfunction + + function string case_label(); + return "SIGNED"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_stream_1x4_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_stream_1x4_test.svh new file mode 100644 index 000000000..49a86b925 --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_stream_1x4_test.svh @@ -0,0 +1,15 @@ +class fp2int_stream_1x4_test extends fp2int_case_test; + `uvm_component_utils(fp2int_stream_1x4_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 5; + endfunction + + function string case_label(); + return "STREAM_1X4"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_stream_2x2_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_stream_2x2_test.svh new file mode 100644 index 000000000..a540a3b46 --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_stream_2x2_test.svh @@ -0,0 +1,15 @@ +class fp2int_stream_2x2_test extends fp2int_case_test; + `uvm_component_utils(fp2int_stream_2x2_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 6; + endfunction + + function string case_label(); + return "STREAM_2X2"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_stream_8x1_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_stream_8x1_test.svh new file mode 100644 index 000000000..add8fbabd --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_stream_8x1_test.svh @@ -0,0 +1,15 @@ +class fp2int_stream_8x1_test extends fp2int_case_test; + `uvm_component_utils(fp2int_stream_8x1_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 7; + endfunction + + function string case_label(); + return "STREAM_8X1"; + endfunction +endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_test.svh deleted file mode 100644 index d927b19ae..000000000 --- a/examples/balls/fp2int/verify/src/tests/fp2int_test.svh +++ /dev/null @@ -1,77 +0,0 @@ -class fp2int_ball_test extends uvm_test; - `uvm_component_utils(fp2int_ball_test) - - typedef virtual bb_blink_if #(1, 1) vif_t; - vif_t vif; - fp2int_env env; - - function new(string name, uvm_component parent); - super.new(name, parent); - endfunction - - function void build_phase(uvm_phase phase); - super.build_phase(phase); - if (!uvm_config_db#(vif_t)::get(this, "", "vif", vif)) begin - `uvm_fatal("NOVIF", "bb_blink_if not found") - end - env = fp2int_env::type_id::create("env", this); - endfunction - - task run_phase(uvm_phase phase); - int unsigned bid; - bid = fp2int_require_bid(); - phase.raise_objection(this); - run_directed(bid); - run_random(bid); - phase.drop_objection(this); - endtask - - task run_directed(int unsigned bid); - run_case(0, "DIR_I32", bid); - run_case(1, "DIR_I8", bid); - endtask - - task run_random(int unsigned bid); - int i; - for (i = 2; i <= 11; i++) begin - run_case(i, $sformatf("RND_%0d", i), bid); - end - endtask - - task run_case(int unsigned idx, string label, int unsigned bid); - fp2int_basic_seq seq; - int cycles; - - apply_reset(); - env.scb.reset_counters(); - - seq = fp2int_basic_seq::type_id::create("seq"); - seq.case_index = idx; - seq.bid = bid; - seq.start(env.cmd_agent.seqr); - - cycles = 0; - while (!env.scb.done()) begin - @(posedge vif.clock); - cycles++; - if (cycles > FP2INT_TIMEOUT_CYCLES) begin - `uvm_fatal( - "TIMEOUT", - $sformatf( - "Fp2IntBall %s (idx=%0d) timeout: cmds=%0d reads=%0d/%0d writes=%0d/%0d resp=%0d", - label, idx, env.scb.cmd_count, env.scb.read_count, env.scb.expected_reads, - env.scb.write_count, env.scb.expected_writes, env.scb.resp_count)) - end - end - - `uvm_info("FP2INT", $sformatf("Fp2IntBall %s (idx=%0d) passed in %0d cycles", label, idx, - cycles), UVM_LOW) - endtask - - task apply_reset(); - vif.reset = 1'b1; - repeat (5) @(posedge vif.clock); - vif.reset = 1'b0; - repeat (2) @(posedge vif.clock); - endtask -endclass diff --git a/examples/balls/fp2int/verify/src/tests/fp2int_zero_test.svh b/examples/balls/fp2int/verify/src/tests/fp2int_zero_test.svh new file mode 100644 index 000000000..583ef17e9 --- /dev/null +++ b/examples/balls/fp2int/verify/src/tests/fp2int_zero_test.svh @@ -0,0 +1,15 @@ +class fp2int_zero_test extends fp2int_case_test; + `uvm_component_utils(fp2int_zero_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 1; + endfunction + + function string case_label(); + return "ZERO"; + endfunction +endclass diff --git a/examples/balls/fp2int/workloads/ctests/CMakeLists.txt b/examples/balls/fp2int/workloads/ctests/CMakeLists.txt index dd49cc853..b117bd4a4 100644 --- a/examples/balls/fp2int/workloads/ctests/CMakeLists.txt +++ b/examples/balls/fp2int/workloads/ctests/CMakeLists.txt @@ -1,6 +1,10 @@ add_buckyball_ctests( - fp2int_fp32_to_int32_test.c - fp2int_fp32_to_int8_test.c - fp2int_bank_fp32_to_int32_test.c - fp2int_bank_fp32_to_int8_test.c + fp2int_i8_signed_test.c + fp2int_i8_rounding_test.c + fp2int_i8_zero_test.c + fp2int_i8_rows_test.c + fp2int_i8_scale_rows_test.c + fp2int_i8_stream_1x4_test.c + fp2int_i8_stream_2x2_test.c + fp2int_i8_stream_8x1_test.c ) diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_bank_fp32_to_int32_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_bank_fp32_to_int32_test.c deleted file mode 100644 index db49f9d28..000000000 --- a/examples/balls/fp2int/workloads/ctests/fp2int_bank_fp32_to_int32_test.c +++ /dev/null @@ -1,71 +0,0 @@ -#include "buckyball.h" -#include "fp2int_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 4 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_1_0 0x3F800000U -#define SEED 0xA5A5A5A5U - -static float input_fp32[NELEM] __attribute__((aligned(64))); -static int32_t output_i32[NELEM] __attribute__((aligned(64))); -static int32_t expected_i32[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_1_0); - - for (int i = 0; i < NELEM; ++i) { - float in = (float)((rand() % 65536) - 32768) / 64.0f; - input_fp32[i] = in; - expected_i32[i] = fp2int_i32(f32_to_bits(in), SCALE_1_0); - output_i32[i] = 0x7fffffff; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 1); - bb_mem_alloc(dst, BANK_LINES, 1); - bb_mvin((uintptr_t)input_fp32, src, BANK_LINES, 1); - bb_fp2int(src, dst, BANK_LINES, SCALE_1_0); - bb_mvout((uintptr_t)output_i32, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - if (output_i32[i] != expected_i32[i]) { - printf("MISMATCH at [%d]: got %d, expected %d\n", i, output_i32[i], - expected_i32[i]); - passed = 0; - } - double ideal = (double)input_fp32[i] * scale; - double err = (double)output_i32[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Fp2Int bank FP32-to-INT32 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_bank_fp32_to_int8_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_bank_fp32_to_int8_test.c deleted file mode 100644 index 0577477bb..000000000 --- a/examples/balls/fp2int/workloads/ctests/fp2int_bank_fp32_to_int8_test.c +++ /dev/null @@ -1,71 +0,0 @@ -#include "buckyball.h" -#include "fp2int_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 16 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_1_0 0x3F800000U -#define SEED 0xC3C3C3C3U - -static float input_fp32[NELEM] __attribute__((aligned(64))); -static int8_t output_i8[NELEM] __attribute__((aligned(64))); -static int8_t expected_i8[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_1_0); - - for (int i = 0; i < NELEM; ++i) { - float in = (float)((rand() % 65536) - 32768) / 64.0f; - input_fp32[i] = in; - expected_i8[i] = fp2int_i8(f32_to_bits(in), SCALE_1_0); - output_i8[i] = (int8_t)0x5a; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 4); - bb_mem_alloc(dst, BANK_LINES, 1); - bb_mvin((uintptr_t)input_fp32, src, BANK_LINES, 1); - bb_fp2int(src, dst, BANK_LINES, SCALE_1_0); - bb_mvout((uintptr_t)output_i8, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - if (output_i8[i] != expected_i8[i]) { - printf("MISMATCH at [%d]: got %d, expected %d\n", i, output_i8[i], - expected_i8[i]); - passed = 0; - } - double ideal = (double)input_fp32[i] * scale; - double err = (double)output_i8[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Fp2Int bank FP32-to-INT8 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_fp32_to_int32_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_fp32_to_int32_test.c deleted file mode 100644 index 18e96f8e2..000000000 --- a/examples/balls/fp2int/workloads/ctests/fp2int_fp32_to_int32_test.c +++ /dev/null @@ -1,47 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include - -#define ROWS 4 -#define LANES 4 -#define N (ROWS * LANES) -#define SCALE_1_0 0x3F800000U - -static uint32_t fp32_input[N] __attribute__((aligned(64))) = { - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x3F000000, 0x42C80000, - 0xC2C80000, 0x40E00000, 0x41000000, 0xC1000000, -}; - -static int32_t expected[N] __attribute__((aligned(64))) = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 0, 100, -100, 7, 8, -8, -}; - -static int32_t output[N] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, ROWS, 1); - bb_mem_alloc(dst, ROWS, 1); - bb_mvin((uintptr_t)fp32_input, src, ROWS, 1); - bb_fp2int(src, dst, ROWS, SCALE_1_0); - bb_mvout((uintptr_t)output, dst, ROWS, 1); - bb_fence(); - - int passed = 1; - for (int i = 0; i < N; i++) { - if (output[i] != expected[i]) { - printf("MISMATCH [%d]: got %d expected %d\n", i, output[i], expected[i]); - passed = 0; - } - } - printf("Fp2Int FP32-to-INT32 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_fp32_to_int8_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_fp32_to_int8_test.c deleted file mode 100644 index c0f02ce7f..000000000 --- a/examples/balls/fp2int/workloads/ctests/fp2int_fp32_to_int8_test.c +++ /dev/null @@ -1,51 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include - -#define ROWS 2 -#define LANES 16 -#define SCALE_2_0 0x40000000U - -static float fp32_input[ROWS * LANES] __attribute__((aligned(64))) = { - 0.125f, -0.125f, 0.25f, -0.25f, 0.75f, -0.75f, 1.25f, -1.25f, - 1.75f, -1.75f, 63.25f, 63.75f, -63.75f, -64.75f, 0.0f, -0.0f, - 2.25f, -2.25f, 2.75f, -2.75f, 3.25f, -3.25f, 3.75f, -3.75f, - 10.125f, -10.125f, 20.25f, -20.25f, 0.375f, -0.375f, 64.25f, -65.25f, -}; - -static int8_t expected_int8[ROWS * LANES] __attribute__((aligned(64))) = { - 0, 0, 0, 0, 2, -2, 2, -2, 4, -4, 126, 127, -128, -128, 0, 0, - 4, -4, 6, -6, 6, -6, 8, -8, 20, -20, 40, -40, 1, -1, 127, -128, -}; - -static int8_t output_int8[ROWS * LANES] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - const uint32_t src_bank = 0; - const uint32_t dst_bank = 1; - - bb_mem_alloc(src_bank, ROWS, 4); - bb_mem_alloc(dst_bank, ROWS, 1); - bb_mvin((uintptr_t)fp32_input, src_bank, ROWS, 1); - bb_fp2int(src_bank, dst_bank, ROWS, SCALE_2_0); - bb_mvout((uintptr_t)output_int8, dst_bank, ROWS, 1); - bb_fence(); - - int passed = 1; - for (int i = 0; i < ROWS * LANES; ++i) { - if (output_int8[i] != expected_int8[i]) { - printf("MISMATCH at [%d]: got %d, expected %d\n", i, output_int8[i], - expected_int8[i]); - passed = 0; - } - } - - printf("Fp2Int FP32-to-INT8 test %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_rounding_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_rounding_test.c new file mode 100644 index 000000000..9cb581b1d --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_rounding_test.c @@ -0,0 +1,38 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[16] __attribute__((aligned(64))) = { + 0.5f, -0.5f, 1.5f, -1.5f, 2.5f, -2.5f, 3.5f, -3.5f, + 4.5f, -4.5f, 5.5f, -5.5f, 126.5f, -126.5f, 127.0f, -127.0f, +}; +static int8_t output[16] __attribute__((aligned(64))); +static const int8_t expected[16] = { + 0, 0, 2, -2, 2, -2, 4, -4, 4, -4, 6, -6, 126, -126, 127, -127, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mem_alloc(0, 1, 4); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_fp2int(0, 1, 1, 0); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_rounding i=%d got=%d exp=%d\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_rounding PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_rows_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_rows_test.c new file mode 100644 index 000000000..66469f93d --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_rows_test.c @@ -0,0 +1,40 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[32] __attribute__((aligned(64))) = { + -8.0f, -7.0f, -6.0f, -5.0f, -4.0f, -3.0f, -2.0f, -1.0f, + 0.0f, 127.0f, 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, + 7.0f, 8.0f, -8.0f, -7.0f, -6.0f, -5.0f, -4.0f, -3.0f, + -2.0f, -1.0f, 0.0f, 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, +}; +static int8_t output[32] __attribute__((aligned(64))); +static const int8_t expected[32] = { + -8, -7, -6, -5, -4, -3, -2, -1, 0, 127, 1, 2, 3, 4, 5, 6, + 7, 8, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mem_alloc(0, 2, 4); + bb_mem_alloc(1, 2, 1); + bb_mvin((uintptr_t)input, 0, 2, 1); + bb_fp2int(0, 1, 2, 0); + bb_mvout((uintptr_t)output, 1, 2, 1); + bb_fence(); + for (int i = 0; i < 32; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_rows i=%d got=%d exp=%d\n", i, output[i], expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_rows PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_scale_rows_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_scale_rows_test.c new file mode 100644 index 000000000..4285446ac --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_scale_rows_test.c @@ -0,0 +1,39 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[32] __attribute__((aligned(64))) = { + 1, -1, 2, -2, 3, -3, 4, -4, 5, -5, 6, -6, 7, -7, 8, -8, + 20, -20, 10, -10, 15, -15, 5, -5, 1, -1, 2, -2, 3, -3, 4, -4, +}; +static int8_t output[32] __attribute__((aligned(64))); +static const int8_t expected[32] = { + 6, -6, 13, -13, 19, -19, 25, -25, 32, -32, 38, -38, 44, -44, 51, -51, + 127, -127, 64, -64, 95, -95, 32, -32, 6, -6, 13, -13, 19, -19, 25, -25, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mem_alloc(0, 2, 4); + bb_mem_alloc(1, 2, 1); + bb_mvin((uintptr_t)input, 0, 2, 1); + bb_fp2int(0, 1, 2, 0); + bb_mvout((uintptr_t)output, 1, 2, 1); + bb_fence(); + for (int i = 0; i < 32; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_scale_rows i=%d got=%d exp=%d\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_scale_rows PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_signed_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_signed_test.c new file mode 100644 index 000000000..afb8f630a --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_signed_test.c @@ -0,0 +1,38 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[16] __attribute__((aligned(64))) = { + -10.0f, -5.0f, -2.5f, -1.0f, 0.0f, 1.0f, 2.5f, 5.0f, + 10.0f, 7.5f, 3.0f, 0.5f, -0.5f, -3.0f, -7.5f, 0.0f, +}; +static int8_t output[16] __attribute__((aligned(64))); +static const int8_t expected[16] = { + -127, -64, -32, -13, 0, 13, 32, 64, 127, 95, 38, 6, -6, -38, -95, 0, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mem_alloc(0, 1, 4); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_fp2int(0, 1, 1, 0); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_signed i=%d got=%d exp=%d\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_signed PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_1x4_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_1x4_test.c new file mode 100644 index 000000000..21612c4ea --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_1x4_test.c @@ -0,0 +1,34 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[16] __attribute__((aligned(64))) = { + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, +}; +static int8_t output[16] __attribute__((aligned(64))); +static const int8_t expected[16] = { + 8, 16, 24, 32, 40, 48, 56, 64, 71, 79, 87, 95, 103, 111, 119, 127, +}; + +int main(void) { + bb_mem_alloc(0, 4, 1); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)input, 0, 4, 1); + bb_fp2int(0, 1, 4, 0); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_stream_1x4 i=%d got=%d exp=%d\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_stream_1x4 PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_2x2_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_2x2_test.c new file mode 100644 index 000000000..66b0e4c37 --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_2x2_test.c @@ -0,0 +1,34 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[16] __attribute__((aligned(64))) = { + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, +}; +static int8_t output[16] __attribute__((aligned(64))); +static const int8_t expected[16] = { + 8, 16, 24, 32, 40, 48, 56, 64, 71, 79, 87, 95, 103, 111, 119, 127, +}; + +int main(void) { + bb_mem_alloc(0, 2, 2); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)input, 0, 2, 1); + bb_fp2int(0, 1, 2, 0); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_stream_2x2 i=%d got=%d exp=%d\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_stream_2x2 PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_8x1_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_8x1_test.c new file mode 100644 index 000000000..4377eddd7 --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_stream_8x1_test.c @@ -0,0 +1,36 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[32] __attribute__((aligned(64))) = { + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, + 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, +}; +static int8_t output[32] __attribute__((aligned(64))); +static const int8_t expected[32] = { + 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, + 67, 71, 75, 79, 83, 87, 91, 95, 99, 103, 107, 111, 115, 119, 123, 127, +}; + +int main(void) { + bb_mem_alloc(0, 1, 8); + bb_mem_alloc(1, 2, 1); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_fp2int(0, 1, 1, 0); + bb_mvout((uintptr_t)output, 1, 2, 1); + bb_fence(); + for (int i = 0; i < 32; ++i) { + if (output[i] != expected[i]) { + printf("fp2int_i8_stream_8x1 i=%d got=%d exp=%d\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_stream_8x1 PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_i8_zero_test.c b/examples/balls/fp2int/workloads/ctests/fp2int_i8_zero_test.c new file mode 100644 index 000000000..c07a14f60 --- /dev/null +++ b/examples/balls/fp2int/workloads/ctests/fp2int_i8_zero_test.c @@ -0,0 +1,31 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static float input[16] __attribute__((aligned(64))); +static int8_t output[16] __attribute__((aligned(64))); + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mem_alloc(0, 1, 4); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_fp2int(0, 1, 1, 0); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != 0) { + printf("fp2int_i8_zero i=%d got=%d\n", i, output[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("fp2int_i8_zero PASS\n"); + return 0; +} diff --git a/examples/balls/fp2int/workloads/ctests/fp2int_ref.h b/examples/balls/fp2int/workloads/ctests/fp2int_ref.h deleted file mode 100644 index 1e5cc97a3..000000000 --- a/examples/balls/fp2int/workloads/ctests/fp2int_ref.h +++ /dev/null @@ -1,118 +0,0 @@ -#ifndef FP2INT_REF_H -#define FP2INT_REF_H - -#include - -static uint32_t fp32_mul(uint32_t a, uint32_t b) { - uint32_t a_sign = (a >> 31) & 1; - uint32_t b_sign = (b >> 31) & 1; - uint32_t a_exp = (a >> 23) & 0xff; - uint32_t b_exp = (b >> 23) & 0xff; - uint32_t a_frac = a & 0x7fffff; - uint32_t b_frac = b & 0x7fffff; - uint64_t a_mant = (1ull << 23) | a_frac; - uint64_t b_mant = (1ull << 23) | b_frac; - int a_zero = a_exp == 0 && a_frac == 0; - int b_zero = b_exp == 0 && b_frac == 0; - uint64_t prod = a_mant * b_mant; - uint32_t sig, exp_adjust; - int guard, roundb, sticky; - if ((prod >> 47) & 1) { - sig = (uint32_t)(prod >> 24); - guard = ((prod >> 23) & 1) != 0; - roundb = ((prod >> 22) & 1) != 0; - sticky = (prod & ((1ull << 22) - 1)) != 0; - exp_adjust = 1; - } else { - sig = (uint32_t)(prod >> 23); - guard = ((prod >> 22) & 1) != 0; - roundb = ((prod >> 21) & 1) != 0; - sticky = (prod & ((1ull << 21) - 1)) != 0; - exp_adjust = 0; - } - int increment = guard && (roundb || sticky || (sig & 1) != 0); - uint64_t rounded = (uint64_t)sig + (uint64_t)increment; - uint32_t carry = (uint32_t)((rounded >> 24) & 1); - uint32_t final_sig = (uint32_t)(carry != 0 ? (rounded >> 1) : rounded); - uint32_t exp_wide = (a_exp + b_exp + exp_adjust + carry - 127) & 0x3ff; - - if (a_zero || b_zero) - return 0; - if (exp_wide & 0x200) - return 0; - if (exp_wide & 0x100) - return ((a_sign ^ b_sign) << 31) | (0xffu << 23); - return ((a_sign ^ b_sign) << 31) | ((exp_wide & 0xff) << 23) | - (final_sig & 0x7fffff); -} - -static int32_t fp32_to_i32(uint32_t fp) { - int sign = ((fp >> 31) & 1) != 0; - int32_t exponent = (int32_t)((fp >> 23) & 0xff); - uint32_t frac = fp & 0x7fffff; - uint32_t mantissa = (1u << 23) | frac; - int is_zero = exponent == 0 && frac == 0; - int32_t exp_val = exponent - 127; - - if (exponent == 0xff && frac != 0) - return 0; - if (is_zero) - return 0; - - uint64_t magnitude; - if (exp_val >= 31) { - magnitude = 0x80000000ull; - } else if (exp_val >= 23) { - magnitude = (uint64_t)mantissa << (exp_val - 23); - } else if (exp_val >= -1) { - uint32_t right_shift = (uint32_t)(23 - exp_val); - uint64_t truncated = (uint64_t)mantissa >> right_shift; - uint64_t half = 1ull << (right_shift - 1); - uint64_t remainder = (uint64_t)mantissa & ((1ull << right_shift) - 1); - int round_up = - remainder > half || (remainder == half && (truncated & 1) != 0); - magnitude = truncated + (uint64_t)round_up; - } else { - magnitude = 0; - } - - if (sign) { - if (magnitude >= 0x80000000ull) - return (int32_t)0x80000000; - return -(int32_t)magnitude; - } - if (magnitude > 0x7fffffffull) - return 0x7fffffff; - return (int32_t)magnitude; -} - -static int32_t fp2int_i32(uint32_t fp_bits, uint32_t scale_bits) { - return fp32_to_i32(fp32_mul(fp_bits, scale_bits)); -} - -static int8_t fp2int_i8(uint32_t fp_bits, uint32_t scale_bits) { - int32_t v = fp2int_i32(fp_bits, scale_bits); - if (v > 127) - return 127; - if (v < -128) - return -128; - return (int8_t)v; -} - -static float bits_to_f32(uint32_t bits) { - union { - uint32_t u; - float f; - } x = {.u = bits}; - return x.f; -} - -static uint32_t f32_to_bits(float f) { - union { - float f; - uint32_t u; - } x = {.f = f}; - return x.u; -} - -#endif diff --git a/examples/balls/fp2int/workloads/isa/fp2int.h b/examples/balls/fp2int/workloads/isa/fp2int.h new file mode 100644 index 000000000..8e92a18d6 --- /dev/null +++ b/examples/balls/fp2int/workloads/isa/fp2int.h @@ -0,0 +1,17 @@ +#ifndef _BB_FP2INT_H_ +#define _BB_FP2INT_H_ + +#include +#include + +// bb_fp2int(bank_id, wr_bank_id, iter, act_scale_addr) +// The Ball scans the complete logical FP32 activation tensor, writes +// Da=maxAbs/127 to this absolute MMIO byte address, then emits packed INT8. +// Encoding: rs1 = banks | iter +// rs2[12:0] = act_scale_addr +#define bb_fp2int(bank_id, wr_bank_id, iter, act_scale_addr) \ + BUCKYBALL_INSTRUCTION_R_R( \ + (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ + (FIELD((uint64_t)(act_scale_addr), 0, 12)), BB_FUNC7(FP2INT)) + +#endif // _BB_FP2INT_H_ diff --git a/examples/balls/fp2int/workloads/mlir_tests/ball/CMakeLists.txt b/examples/balls/fp2int/workloads/mlir_tests/ball/CMakeLists.txt index 857addc3a..326325c2f 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/ball/CMakeLists.txt +++ b/examples/balls/fp2int/workloads/mlir_tests/ball/CMakeLists.txt @@ -1,12 +1,11 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX ball) add_buckyball_mlir_contract(fp2int_4x4_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} PASSES - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} ${BUCKYBALL_LOWER_BUCKYBALL}) -add_buckyball_mlir_optest(fp2int_4x4) +add_buckyball_mlir_test(fp2int_4x4 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4.mlir b/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4.mlir index 2d5108df4..d85388a35 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4.mlir +++ b/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4.mlir @@ -1,34 +1,33 @@ // Already-lowered ball Op: buckyball.fp2int (physical-bank form). -func.func private @check_result(memref<4x4xi32>) -> () +func.func private @check_result(memref<1x16xi8>) -> () -memref.global "private" constant @fp_in : memref<4x4xf32> = - dense<[[1.0, 2.0, 3.0, -1.0], - [-2.0, 0.0, 4.0, 5.0], - [10.0, -10.0, 0.5, 100.0], - [-100.0, 7.0, 8.0, -8.0]]> +memref.global "private" constant @fp_in : memref<1x16xf32> = + dense<[[1.0, 2.0, 3.0, -1.0, -2.0, 0.0, 4.0, 5.0, + 10.0, -10.0, 0.5, 100.0, -100.0, 7.0, 8.0, -8.0]]> func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 - %depth = arith.constant 4 : i64 + %depth = arith.constant 1 : i64 %stride = arith.constant 1 : i64 - %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %iter = arith.constant 1 : i64 + %da_addr = arith.constant 0 : i64 - %input = memref.get_global @fp_in : memref<4x4xf32> - %output = memref.alloc() {alignment = 64 : i64} : memref<4x4xi32> + %input = memref.get_global @fp_in : memref<1x16xf32> + %output = memref.alloc() alignment = 64 : memref<1x16xi8> - %bin = buckyball.bank_alloc + %bin = buckyball.bank_alloc {col = 4 : i64} %bout = buckyball.bank_alloc %loaded = buckyball.bank_mvin %input %bin %depth %stride - : memref<4x4xf32> i64 i64 i64 - buckyball.fp2int %loaded, %bout, %iter, %scale : i64 + : memref<1x16xf32> i64 i64 i64 + buckyball.fp2int %loaded, %bout, %iter, %da_addr : i64 %stored = buckyball.bank_mvout %output %bout %depth %stride - : memref<4x4xi32> i64 i64 i64 + : memref<1x16xi8> i64 i64 i64 + buckyball.fence buckyball.bank_release %loaded : i64 buckyball.bank_release %stored : i64 - func.call @check_result(%output) : (memref<4x4xi32>) -> () - memref.dealloc %output : memref<4x4xi32> + func.call @check_result(%output) : (memref<1x16xi8>) -> () + memref.dealloc %output : memref<1x16xi8> return %zero_i8 : i8 } diff --git a/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_contract.mlir b/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_contract.mlir index 5392ff4a7..4b82a6d56 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_contract.mlir +++ b/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_contract.mlir @@ -1,14 +1,13 @@ // CHECK: buckyball.intr.custom -// CHECK-SAME: funct7 = 51 // CHECK-NOT: buckyball.fp2int func.func @main() -> i8 { %z = arith.constant 0 : i8 %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 %in = buckyball.bank_alloc %out = buckyball.bank_alloc - buckyball.fp2int %in, %out, %iter, %scale : i64 + buckyball.fp2int %in, %out, %iter, %da_addr : i64 buckyball.bank_release %in : i64 buckyball.bank_release %out : i64 return %z : i8 diff --git a/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_main.cpp b/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_main.cpp index 014146b00..f34ffa0c6 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_main.cpp +++ b/examples/balls/fp2int/workloads/mlir_tests/ball/fp2int_4x4_main.cpp @@ -2,8 +2,8 @@ #include #include -static const int32_t expected[16] = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 0, 100, -100, 7, 8, -8, +static const int8_t expected[16] = { + 1, 3, 4, -1, -3, 0, 5, 6, 13, -13, 1, 127, -127, 9, 10, -10, }; static void fail(void) { @@ -20,20 +20,20 @@ static void fail(void) { #ifdef __cplusplus extern "C" #endif - void check_result(int32_t *allocated, int32_t *aligned, int64_t offset, + void check_result(int8_t *allocated, int8_t *aligned, int64_t offset, int64_t size0, int64_t size1, int64_t stride0, int64_t stride1) { (void)allocated; - if (size0 != 4 || size1 != 4 || stride0 != 4 || stride1 != 1) { + if (size0 != 1 || size1 != 16 || stride0 != 16 || stride1 != 1) { printf("FAILED: ball_fp2int shape %dx%d stride %dx%d\n", (int)size0, (int)size1, (int)stride0, (int)stride1); fail(); } - int32_t *out = aligned + offset; - for (int i = 0; i < 4; ++i) { - for (int j = 0; j < 4; ++j) { - int32_t exp = expected[i * 4 + j]; + int8_t *out = aligned + offset; + for (int i = 0; i < 1; ++i) { + for (int j = 0; j < 16; ++j) { + int32_t exp = expected[i * 16 + j]; int32_t got = out[i * stride0 + j * stride1]; if (got != exp) { printf("FAILED: ball_fp2int out[%d][%d] exp=%d got=%d\n", i, j, @@ -42,5 +42,5 @@ extern "C" } } } - printf("PASSED: fp2int ball_fp2int 4x4 fp32->int32\n"); + printf("PASSED: fp2int ball_fp2int 4x4 fp32->int8\n"); } diff --git a/examples/balls/fp2int/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/fp2int/workloads/mlir_tests/bank/CMakeLists.txt index 5d58e8f93..6ec8e8ffb 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/bank/CMakeLists.txt +++ b/examples/balls/fp2int/workloads/mlir_tests/bank/CMakeLists.txt @@ -1,9 +1,8 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX bank) add_buckyball_mlir_contract(fp2int_4x4_contract - PASSES "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}") + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) -add_buckyball_mlir_optest(fp2int_4x4) +add_buckyball_mlir_test(fp2int_4x4 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4.mlir b/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4.mlir index 1e7cdd432..529358305 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4.mlir +++ b/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4.mlir @@ -1,36 +1,35 @@ // Bank-level Op: buckyball.bank_fp2int // Lowering: assign-physical-banks -> bank_fp2int becomes fp2int. -func.func private @check_result(memref<4x4xi32>) -> () +func.func private @check_result(memref<1x16xi8>) -> () -memref.global "private" constant @fp_in : memref<4x4xf32> = - dense<[[1.0, 2.0, 3.0, -1.0], - [-2.0, 0.0, 4.0, 5.0], - [10.0, -10.0, 0.5, 100.0], - [-100.0, 7.0, 8.0, -8.0]]> +memref.global "private" constant @fp_in : memref<1x16xf32> = + dense<[[1.0, 2.0, 3.0, -1.0, -2.0, 0.0, 4.0, 5.0, + 10.0, -10.0, 0.5, 100.0, -100.0, 7.0, 8.0, -8.0]]> func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 - %depth = arith.constant 4 : i64 + %depth = arith.constant 1 : i64 %stride = arith.constant 1 : i64 - %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %iter = arith.constant 1 : i64 + %da_addr = arith.constant 0 : i64 - %input = memref.get_global @fp_in : memref<4x4xf32> - %output = memref.alloc() {alignment = 64 : i64} : memref<4x4xi32> + %input = memref.get_global @fp_in : memref<1x16xf32> + %output = memref.alloc() alignment = 64 : memref<1x16xi8> - %bin = buckyball.bank_alloc + %bin = buckyball.bank_alloc {col = 4 : i64} %bout = buckyball.bank_alloc %loaded = buckyball.bank_mvin %input %bin %depth %stride - : memref<4x4xf32> i64 i64 i64 - %q = buckyball.bank_fp2int %loaded %bout %iter %scale + : memref<1x16xf32> i64 i64 i64 + %q = buckyball.bank_fp2int %loaded %bout %iter %da_addr : i64 i64 i64 i64 %stored = buckyball.bank_mvout %output %q %depth %stride - : memref<4x4xi32> i64 i64 i64 + : memref<1x16xi8> i64 i64 i64 + buckyball.fence buckyball.bank_release %loaded : i64 buckyball.bank_release %stored : i64 - func.call @check_result(%output) : (memref<4x4xi32>) -> () - memref.dealloc %output : memref<4x4xi32> + func.call @check_result(%output) : (memref<1x16xi8>) -> () + memref.dealloc %output : memref<1x16xi8> return %zero_i8 : i8 } diff --git a/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_contract.mlir b/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_contract.mlir index 913257299..108bd8bea 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_contract.mlir +++ b/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_contract.mlir @@ -4,10 +4,10 @@ func.func @main() -> i8 { %z = arith.constant 0 : i8 %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 %in = buckyball.bank_alloc %out = buckyball.bank_alloc - %r = buckyball.bank_fp2int %in %out %iter %scale + %r = buckyball.bank_fp2int %in %out %iter %da_addr : i64 i64 i64 i64 buckyball.bank_release %in : i64 buckyball.bank_release %r : i64 diff --git a/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_main.cpp b/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_main.cpp index 4aa7f20a7..e497b0a0b 100644 --- a/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_main.cpp +++ b/examples/balls/fp2int/workloads/mlir_tests/bank/fp2int_4x4_main.cpp @@ -2,8 +2,8 @@ #include #include -static const int32_t expected[16] = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 0, 100, -100, 7, 8, -8, +static const int8_t expected[16] = { + 1, 3, 4, -1, -3, 0, 5, 6, 13, -13, 1, 127, -127, 9, 10, -10, }; static void fail(void) { @@ -20,20 +20,20 @@ static void fail(void) { #ifdef __cplusplus extern "C" #endif - void check_result(int32_t *allocated, int32_t *aligned, int64_t offset, + void check_result(int8_t *allocated, int8_t *aligned, int64_t offset, int64_t size0, int64_t size1, int64_t stride0, int64_t stride1) { (void)allocated; - if (size0 != 4 || size1 != 4 || stride0 != 4 || stride1 != 1) { + if (size0 != 1 || size1 != 16 || stride0 != 16 || stride1 != 1) { printf("FAILED: bank_fp2int shape %dx%d stride %dx%d\n", (int)size0, (int)size1, (int)stride0, (int)stride1); fail(); } - int32_t *out = aligned + offset; - for (int i = 0; i < 4; ++i) { - for (int j = 0; j < 4; ++j) { - int32_t exp = expected[i * 4 + j]; + int8_t *out = aligned + offset; + for (int i = 0; i < 1; ++i) { + for (int j = 0; j < 16; ++j) { + int32_t exp = expected[i * 16 + j]; int32_t got = out[i * stride0 + j * stride1]; if (got != exp) { printf("FAILED: bank_fp2int out[%d][%d] exp=%d got=%d\n", i, j, @@ -42,5 +42,5 @@ extern "C" } } } - printf("PASSED: fp2int bank_fp2int 4x4 fp32->int32\n"); + printf("PASSED: fp2int bank_fp2int 4x4 fp32->int8\n"); } diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiBall.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiBall.scala index 889867cbf..13f31a24d 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiBall.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiBall.scala @@ -15,10 +15,14 @@ class GemminiBall(val b: GlobalConfig) extends Module with HasBlink with HasBall "GemminiBall requires frontend.sub_rob_enable=true because loop instructions emit SubROB rows" ) - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "GemminiBall") - .getOrElse(throw new IllegalArgumentException("GemminiBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "GemminiBall") + .getOrElse( + throw new IllegalArgumentException("GemminiBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW @public val io = IO(new BlinkIO(b, inBW, outBW)) @@ -30,10 +34,17 @@ class GemminiBall(val b: GlobalConfig) extends Module with HasBlink with HasBall // ========================================================================= // Sub-modules // ========================================================================= - val exCtrl: Instance[GemminiExCtrl] = Instantiate(new GemminiExCtrl(b)) - val matmulUnroller: Instance[LoopMatmulUnroller] = Instantiate(new LoopMatmulUnroller(b)) - val convUnroller: Instance[LoopConvUnroller] = Instantiate(new LoopConvUnroller(b)) - val encoder: Instance[LoopCmdEncoder] = Instantiate(new LoopCmdEncoder(b)) + val exCtrl: Instance[GemminiExCtrl] = Instantiate(new GemminiExCtrl(b)) + + val matmulUnroller: Instance[LoopMatmulUnroller] = Instantiate( + new LoopMatmulUnroller(b) + ) + + val convUnroller: Instance[LoopConvUnroller] = Instantiate( + new LoopConvUnroller(b) + ) + + val encoder: Instance[LoopCmdEncoder] = Instantiate(new LoopCmdEncoder(b)) // ========================================================================= // Config registers for Loop modes @@ -52,19 +63,58 @@ class GemminiBall(val b: GlobalConfig) extends Module with HasBlink with HasBall // ========================================================================= val funct7 = io.cmdReq.bits.cmd.funct7 + def ballFunct(mnemonic: String): UInt = + b.ballDomain.ballISA + .find(_.mnemonic == mnemonic) + .map(_.funct7.U(7.W)) + .getOrElse(throw new IllegalArgumentException(s"$mnemonic not found in ballISA")) + + val configFunct = ballFunct("GEMMINI_CONFIG") + val preloadFunct = ballFunct("GEMMINI_PRELOAD") + val computePreFunct = ballFunct("GEMMINI_COMPUTE_PRELOADED") + val computeAccFunct = ballFunct("GEMMINI_COMPUTE_ACCUMULATED") + val flushFunct = ballFunct("GEMMINI_FLUSH") + + val loopWsConfigFuncts = Seq( + ballFunct("GEMMINI_LOOP_WS_CONFIG_BOUNDS"), + ballFunct("GEMMINI_LOOP_WS_CONFIG_ADDR_A"), + ballFunct("GEMMINI_LOOP_WS_CONFIG_ADDR_B"), + ballFunct("GEMMINI_LOOP_WS_CONFIG_ADDR_D"), + ballFunct("GEMMINI_LOOP_WS_CONFIG_ADDR_C"), + ballFunct("GEMMINI_LOOP_WS_CONFIG_STRIDES_AB"), + ballFunct("GEMMINI_LOOP_WS_CONFIG_STRIDES_DC") + ) + + val loopWsTriggerFunct = ballFunct("GEMMINI_LOOP_WS") + + val loopConvConfigFuncts = Seq( + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_1"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_2"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_3"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_4"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_5"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_6"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_7"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_8"), + ballFunct("GEMMINI_LOOP_CONV_WS_CONFIG_9") + ) + + val loopConvTriggerFunct = ballFunct("GEMMINI_LOOP_CONV_WS") + val rs2Data = io.cmdReq.bits.cmd.special - val isConfig = funct7 === 0x02.U // GEMMINI_CONFIG (enable=000, opcode=2) - val isPreload = funct7 === 0x35.U // GEMMINI_PRELOAD (enable=011, opcode=5) - val isComputePre = funct7 === 0x42.U // GEMMINI_COMPUTE_PRELOADED (enable=100, opcode=2) - val isComputeAcc = funct7 === 0x43.U // GEMMINI_COMPUTE_ACCUMULATED (enable=100, opcode=3) - val isFlush = funct7 === 0x03.U // GEMMINI_FLUSH (enable=000, opcode=3) - val isExUnit = isConfig || isPreload || isComputePre || isComputeAcc || isFlush + val isConfig = funct7 === configFunct + val isPreload = funct7 === preloadFunct + val isComputePre = funct7 === computePreFunct + val isComputeAcc = funct7 === computeAccFunct + val isFlush = funct7 === flushFunct + val isExUnit = + isConfig || isPreload || isComputePre || isComputeAcc || isFlush - val isLoopWsConfig = funct7 >= 0x50.U && funct7 <= 0x56.U - val isLoopWsTrigger = funct7 === 0x57.U - val isLoopConvConfig = funct7 >= 0x60.U && funct7 <= 0x68.U - val isLoopConvTrigger = funct7 === 0x69.U + val isLoopWsConfig = loopWsConfigFuncts.map(funct7 === _).reduce(_ || _) + val isLoopWsTrigger = funct7 === loopWsTriggerFunct + val isLoopConvConfig = loopConvConfigFuncts.map(funct7 === _).reduce(_ || _) + val isLoopConvTrigger = funct7 === loopConvTriggerFunct // ========================================================================= // ExUnit path (non-Loop: CONFIG/PRELOAD/COMPUTE/FLUSH) @@ -85,20 +135,20 @@ class GemminiBall(val b: GlobalConfig) extends Module with HasBlink with HasBall configRespBits.is_sub := io.cmdReq.bits.is_sub configRespBits.sub_rob_id := io.cmdReq.bits.sub_rob_id switch(funct7) { - is(0x50.U) { + is(loopWsConfigFuncts(0)) { loopWsConfig.max_k := rs2Data(15, 0) loopWsConfig.max_j := rs2Data(31, 16) loopWsConfig.max_i := rs2Data(47, 32) } - is(0x51.U)(loopWsConfig.dram_addr_a := rs2Data(38, 0)) - is(0x52.U)(loopWsConfig.dram_addr_b := rs2Data(38, 0)) - is(0x53.U)(loopWsConfig.dram_addr_d := rs2Data(38, 0)) - is(0x54.U)(loopWsConfig.dram_addr_c := rs2Data(38, 0)) - is(0x55.U) { + is(loopWsConfigFuncts(1))(loopWsConfig.dram_addr_a := rs2Data(38, 0)) + is(loopWsConfigFuncts(2))(loopWsConfig.dram_addr_b := rs2Data(38, 0)) + is(loopWsConfigFuncts(3))(loopWsConfig.dram_addr_d := rs2Data(38, 0)) + is(loopWsConfigFuncts(4))(loopWsConfig.dram_addr_c := rs2Data(38, 0)) + is(loopWsConfigFuncts(5)) { loopWsConfig.stride_a := rs2Data(31, 0) loopWsConfig.stride_b := rs2Data(63, 32) } - is(0x56.U) { + is(loopWsConfigFuncts(6)) { loopWsConfig.stride_d := rs2Data(31, 0) loopWsConfig.stride_c := rs2Data(63, 32) } @@ -111,32 +161,32 @@ class GemminiBall(val b: GlobalConfig) extends Module with HasBlink with HasBall configRespBits.is_sub := io.cmdReq.bits.is_sub configRespBits.sub_rob_id := io.cmdReq.bits.sub_rob_id switch(funct7) { - is(0x60.U) { + is(loopConvConfigFuncts(0)) { loopConvConfig.batch_size := rs2Data(15, 0) loopConvConfig.in_dim := rs2Data(31, 16) loopConvConfig.in_channels := rs2Data(47, 32) } - is(0x61.U) { + is(loopConvConfigFuncts(1)) { loopConvConfig.out_channels := rs2Data(15, 0) loopConvConfig.out_dim := rs2Data(31, 16) loopConvConfig.stride := rs2Data(39, 32) loopConvConfig.padding := rs2Data(47, 40) } - is(0x62.U) { + is(loopConvConfigFuncts(2)) { loopConvConfig.kernel_dim := rs2Data(7, 0) loopConvConfig.pool_size := rs2Data(15, 8) loopConvConfig.pool_stride := rs2Data(23, 16) loopConvConfig.pool_padding := rs2Data(31, 24) } - is(0x63.U)(loopConvConfig.dram_addr_bias := rs2Data(38, 0)) - is(0x64.U)(loopConvConfig.dram_addr_input := rs2Data(38, 0)) - is(0x65.U)(loopConvConfig.dram_addr_weight := rs2Data(38, 0)) - is(0x66.U)(loopConvConfig.dram_addr_output := rs2Data(38, 0)) - is(0x67.U) { + is(loopConvConfigFuncts(3))(loopConvConfig.dram_addr_bias := rs2Data(38, 0)) + is(loopConvConfigFuncts(4))(loopConvConfig.dram_addr_input := rs2Data(38, 0)) + is(loopConvConfigFuncts(5))(loopConvConfig.dram_addr_weight := rs2Data(38, 0)) + is(loopConvConfigFuncts(6))(loopConvConfig.dram_addr_output := rs2Data(38, 0)) + is(loopConvConfigFuncts(7)) { loopConvConfig.input_stride := rs2Data(31, 0) loopConvConfig.weight_stride := rs2Data(63, 32) } - is(0x68.U)(loopConvConfig.output_stride := rs2Data(31, 0)) + is(loopConvConfigFuncts(8))(loopConvConfig.output_stride := rs2Data(31, 0)) } } diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlCmdStates.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlCmdStates.scala index 5815c0ca7..b1516ee13 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlCmdStates.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlCmdStates.scala @@ -15,7 +15,11 @@ trait GemminiExCtrlCmdStates { this: GemminiExCtrl => op1_bank := io.cmdReq.bits.cmd.op1_bank op2_bank := io.cmdReq.bits.cmd.op2_bank wr_bank := io.cmdReq.bits.cmd.wr_bank - total_rows := Mux(io.cmdReq.bits.cmd.iter === 0.U, DIM.U, io.cmdReq.bits.cmd.iter) + total_rows := Mux( + io.cmdReq.bits.cmd.iter === 0.U, + DIM.U, + io.cmdReq.bits.cmd.iter + ) zero_op2 := io.cmdReq.bits.cmd.special(4) zero_op1_tail := io.cmdReq.bits.cmd.special(5) @@ -23,17 +27,22 @@ trait GemminiExCtrlCmdStates { this: GemminiExCtrl => cfg_dataflow := io.cmdReq.bits.cmd.special(4) cfg_a_transpose := io.cmdReq.bits.cmd.special(7) cfg_bd_transpose := io.cmdReq.bits.cmd.special(8) - cfg_in_shift := io.cmdReq.bits.cmd.special(log2Up(config.accWidth) + 8, 9) + cfg_in_shift := io.cmdReq.bits.cmd.special( + log2Up(config.accWidth) + 8, + 9 + ) io.cmdResp.valid := true.B state := sCommit }.elsewhen(sub_cmd === GemminiSubCmd.PRELOAD) { - read_row_cnt := 0.U - feed_row_cnt := 0.U - req_sent := false.B - xpose_ready := false.B + read_row_cnt := 0.U + feed_row_cnt := 0.U + req_sent := false.B + xpose_ready := false.B xpose_row_cnt := 0.U - state := sPreloadRead - }.elsewhen(sub_cmd === GemminiSubCmd.COMPUTE_PRELOADED || sub_cmd === GemminiSubCmd.COMPUTE_ACCUMULATED) { + state := sPreloadRead + }.elsewhen( + sub_cmd === GemminiSubCmd.COMPUTE_PRELOADED || sub_cmd === GemminiSubCmd.COMPUTE_ACCUMULATED + ) { read_row_cnt := 0.U feed_row_cnt := 0.U outBufRows := 0.U diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeFeedState.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeFeedState.scala index b40cd2dcf..db1ea45a7 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeFeedState.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeFeedState.scala @@ -15,8 +15,16 @@ trait GemminiExCtrlComputeFeedState { this: GemminiExCtrl => mesh.io.req.bits.pe_control.dataflow := cfg_dataflow mesh.io.req.bits.pe_control.propagate := 1.U mesh.io.req.bits.pe_control.shift := cfg_in_shift - mesh.io.req.bits.a_transpose := Mux(cfg_dataflow === Dataflow.OS.id.U, true.B, cfg_a_transpose) - mesh.io.req.bits.bd_transpose := Mux(cfg_dataflow === Dataflow.OS.id.U, false.B, cfg_bd_transpose) + mesh.io.req.bits.a_transpose := Mux( + cfg_dataflow === Dataflow.OS.id.U, + true.B, + cfg_a_transpose + ) + mesh.io.req.bits.bd_transpose := Mux( + cfg_dataflow === Dataflow.OS.id.U, + false.B, + cfg_bd_transpose + ) mesh.io.req.bits.total_rows := total_rows mesh.io.req.bits.tag.rob := robIdAsTag8(rob_id_reg) mesh.io.req.bits.flush := 0.U @@ -34,11 +42,13 @@ trait GemminiExCtrlComputeFeedState { this: GemminiExCtrl => val op2Ready = !op2FromBuf || zero_op2 || rdQueue1.io.deq.valid when(xpose_row_cnt < total_rows && op1Ready && op2Ready) { when(op1FromBuf) { - op1Buf(xpose_row_cnt) := rdQueue0.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) + op1Buf(xpose_row_cnt) := rdQueue0.io.deq.bits.data + .asTypeOf(Vec(DIM, inputType)) rdQueue0.io.deq.ready := true.B } when(op2FromBuf && !zero_op2) { - op2Buf(xpose_row_cnt) := rdQueue1.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) + op2Buf(xpose_row_cnt) := rdQueue1.io.deq.bits.data + .asTypeOf(Vec(DIM, inputType)) rdQueue1.io.deq.ready := true.B } xpose_row_cnt := xpose_row_cnt + 1.U @@ -49,7 +59,9 @@ trait GemminiExCtrlComputeFeedState { this: GemminiExCtrl => when(req_sent && (!needXpose || xpose_ready) && feed_row_cnt < total_rows) { val op1Zero = zero_op1_tail && feed_row_cnt =/= 0.U - when((op1Zero || op1FromBuf || rdQueue0.io.deq.valid) && (zero_op2 || op2FromBuf || rdQueue1.io.deq.valid)) { + when( + (op1Zero || op1FromBuf || rdQueue0.io.deq.valid) && (zero_op2 || op2FromBuf || rdQueue1.io.deq.valid) + ) { val a_mem_row = rdQueue0.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) val x_mem_row = rdQueue1.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) val a_col_row = VecInit((0 until DIM).map(i => op1Buf(i)(feed_row_cnt))) @@ -65,7 +77,9 @@ trait GemminiExCtrlComputeFeedState { this: GemminiExCtrl => when(cfg_dataflow === Dataflow.OS.id.U) { // OS: stream A/B, D=0 mesh.io.b.valid := true.B - mesh.io.b.bits := VecInit(x_row.grouped(config.tileColumns).map(g => VecInit(g)).toSeq) + mesh.io.b.bits := VecInit( + x_row.grouped(config.tileColumns).map(g => VecInit(g)).toSeq + ) mesh.io.d.valid := true.B mesh.io.d.bits := 0.U.asTypeOf(mesh.D_TYPE) }.otherwise { @@ -76,7 +90,9 @@ trait GemminiExCtrlComputeFeedState { this: GemminiExCtrl => mesh.io.d.bits := Mux( zero_op2, 0.U.asTypeOf(mesh.D_TYPE), - VecInit(x_row.grouped(config.tileColumns).map(g => VecInit(g)).toSeq) + VecInit( + x_row.grouped(config.tileColumns).map(g => VecInit(g)).toSeq + ) ) } when(mesh.io.a.ready && mesh.io.b.ready && mesh.io.d.ready) { diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeReadState.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeReadState.scala index 0e7acb698..b6196470c 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeReadState.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlComputeReadState.scala @@ -8,7 +8,9 @@ trait GemminiExCtrlComputeReadState { this: GemminiExCtrl => protected def handleComputeReadState(): Unit = { // Drain stale preload data from shared queues before issuing compute reads - when(read_row_cnt === 0.U && (rdQueue0.io.deq.valid || rdQueue1.io.deq.valid)) { + when( + read_row_cnt === 0.U && (rdQueue0.io.deq.valid || rdQueue1.io.deq.valid) + ) { rdQueue0.io.deq.ready := true.B rdQueue1.io.deq.ready := true.B }.elsewhen(read_row_cnt < total_rows) { @@ -24,7 +26,9 @@ trait GemminiExCtrlComputeReadState { this: GemminiExCtrl => read_done(1) := true.B } when( - (op1Zero || read_done(0) || io.bankReadReq(0).fire) && (zero_op2 || read_done(1) || io.bankReadReq(1).fire) + (op1Zero || read_done(0) || io + .bankReadReq(0) + .fire) && (zero_op2 || read_done(1) || io.bankReadReq(1).fire) ) { read_row_cnt := read_row_cnt + 1.U read_done.foreach(_ := false.B) diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlDefs.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlDefs.scala index 6f9e397c0..e8efe4742 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlDefs.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlDefs.scala @@ -15,10 +15,14 @@ trait GemminiExCtrlDefs { this: GemminiExCtrl => val config = GemminiBallParam(b) val DIM = config.blockSize - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "GemminiBall") - .getOrElse(throw new IllegalArgumentException("GemminiBall not found in config")) - val inBW = ballMapping.inBW - val outBW = ballMapping.outBW + val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "GemminiBall") + .getOrElse( + throw new IllegalArgumentException("GemminiBall not found in config") + ) + + val inBW = ballMapping.inBW + val outBW = ballMapping.outBW val inputType = SInt(config.inputWidth.W) val accType = SInt(config.accWidth.W) @@ -38,23 +42,25 @@ trait GemminiExCtrlDefs { this: GemminiExCtrl => val io = ctrlIo - val mesh = Module(new MeshWithDelays( - inputType = inputType, - weightType = inputType, - outputType = meshOutputType, - accType = accType, - tagType = new SimpleTag, - df = Dataflow.BOTH, - tree_reduction = false, - tile_latency = config.tileLatency, - output_delay = config.outputDelay, - tileRows = config.tileRows, - tileColumns = config.tileColumns, - meshRows = config.meshRows, - meshColumns = config.meshColumns, - leftBanks = 1, - upBanks = 1 - )) + val mesh = Module( + new MeshWithDelays( + inputType = inputType, + weightType = inputType, + outputType = meshOutputType, + accType = accType, + tagType = new SimpleTag, + df = Dataflow.BOTH, + tree_reduction = false, + tile_latency = config.tileLatency, + output_delay = config.outputDelay, + tileRows = config.tileRows, + tileColumns = config.tileColumns, + meshRows = config.meshRows, + meshColumns = config.meshColumns, + leftBanks = 1, + upBanks = 1 + ) + ) protected def widenMeshToAcc(src: Vec[Vec[SInt]]): Vec[Vec[SInt]] = VecInit(src.map(col => VecInit(col.map(_.asTypeOf(accType))))) @@ -103,7 +109,10 @@ trait GemminiExCtrlDefs { this: GemminiExCtrl => rdQueue0.io.enq <> io.bankReadResp(0) rdQueue1.io.enq <> io.bankReadResp(1) - val outBuf = Reg(Vec(DIM, Vec(config.meshColumns, Vec(config.tileColumns, accType)))) + val outBuf = Reg( + Vec(DIM, Vec(config.meshColumns, Vec(config.tileColumns, accType))) + ) + val outBufRows = RegInit(0.U(log2Up(DIM + 1).W)) val outBufCollected = RegInit(0.U(log2Up(DIM + 1).W)) val op1Buf = Reg(Vec(DIM, Vec(DIM, inputType))) diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlPreloadStates.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlPreloadStates.scala index 525b78dae..3b3068bf8 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlPreloadStates.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlPreloadStates.scala @@ -38,14 +38,19 @@ trait GemminiExCtrlPreloadStates { this: GemminiExCtrl => } protected def handlePreloadFeedState(): Unit = { - val explicitBTranspose = cfg_dataflow === Dataflow.WS.id.U && cfg_bd_transpose + val explicitBTranspose = + cfg_dataflow === Dataflow.WS.id.U && cfg_bd_transpose when(!req_sent) { mesh.io.req.valid := true.B mesh.io.req.bits.pe_control.dataflow := cfg_dataflow mesh.io.req.bits.pe_control.propagate := 1.U mesh.io.req.bits.pe_control.shift := cfg_in_shift - mesh.io.req.bits.a_transpose := Mux(cfg_dataflow === Dataflow.OS.id.U, true.B, cfg_a_transpose) + mesh.io.req.bits.a_transpose := Mux( + cfg_dataflow === Dataflow.OS.id.U, + true.B, + cfg_a_transpose + ) // B transpose is materialized in op2Buf below. Keeping the mesh-side // transposer disabled avoids the unsupported WS transpose request path. mesh.io.req.bits.bd_transpose := false.B @@ -59,7 +64,8 @@ trait GemminiExCtrlPreloadStates { this: GemminiExCtrl => when(req_sent && explicitBTranspose && !xpose_ready) { when(xpose_row_cnt < total_rows && rdQueue0.io.deq.valid) { - op2Buf(xpose_row_cnt) := rdQueue0.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) + op2Buf(xpose_row_cnt) := rdQueue0.io.deq.bits.data + .asTypeOf(Vec(DIM, inputType)) rdQueue0.io.deq.ready := true.B xpose_row_cnt := xpose_row_cnt + 1.U }.elsewhen(xpose_row_cnt >= total_rows) { @@ -67,12 +73,15 @@ trait GemminiExCtrlPreloadStates { this: GemminiExCtrl => } } - when(req_sent && (!explicitBTranspose || xpose_ready) && feed_row_cnt < total_rows) { + when( + req_sent && (!explicitBTranspose || xpose_ready) && feed_row_cnt < total_rows + ) { when(explicitBTranspose || rdQueue0.io.deq.valid) { - val row_data = rdQueue0.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) - val transpose_col = total_rows - 1.U - feed_row_cnt - val transposed_data = VecInit((0 until DIM).map(i => op2Buf(i)(transpose_col))) - val weight_data = Mux(explicitBTranspose, transposed_data, row_data) + val row_data = rdQueue0.io.deq.bits.data.asTypeOf(Vec(DIM, inputType)) + val transpose_col = total_rows - 1.U - feed_row_cnt + val transposed_data = + VecInit((0 until DIM).map(i => op2Buf(i)(transpose_col))) + val weight_data = Mux(explicitBTranspose, transposed_data, row_data) mesh.io.a.valid := true.B mesh.io.a.bits := 0.U.asTypeOf(mesh.A_TYPE) mesh.io.b.valid := true.B @@ -83,7 +92,9 @@ trait GemminiExCtrlPreloadStates { this: GemminiExCtrl => mesh.io.d.bits := Mux( cfg_dataflow === Dataflow.OS.id.U, 0.U.asTypeOf(mesh.D_TYPE), - VecInit(weight_data.grouped(config.tileColumns).map(g => VecInit(g)).toSeq) + VecInit( + weight_data.grouped(config.tileColumns).map(g => VecInit(g)).toSeq + ) ) when(mesh.io.a.ready && mesh.io.b.ready && mesh.io.d.ready) { rdQueue0.io.deq.ready := !explicitBTranspose diff --git a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlStoreOps.scala b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlStoreOps.scala index 1a870496b..3aacbccb9 100644 --- a/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlStoreOps.scala +++ b/examples/balls/gemmini/arch/src/main/scala/GemminiExCtrlStoreOps.scala @@ -8,8 +8,12 @@ trait GemminiExCtrlStoreOps { this: GemminiExCtrl => protected def handleComputeFlushState(): Unit = { when(mesh.io.resp.fire && mesh.io.resp.bits.total_rows === total_rows) { - when(outBufCollected < total_rows && mesh.io.resp.bits.tag.rob =/= 0xff.U) { - outBuf(total_rows - 1.U - outBufCollected) := widenMeshToAcc(mesh.io.resp.bits.data) + when( + outBufCollected < total_rows && mesh.io.resp.bits.tag.rob =/= 0xff.U + ) { + outBuf(total_rows - 1.U - outBufCollected) := widenMeshToAcc( + mesh.io.resp.bits.data + ) outBufCollected := outBufCollected + 1.U }.otherwise {} } @@ -25,8 +29,16 @@ trait GemminiExCtrlStoreOps { this: GemminiExCtrl => mesh.io.req.bits.pe_control.dataflow := cfg_dataflow mesh.io.req.bits.pe_control.propagate := 1.U mesh.io.req.bits.pe_control.shift := cfg_in_shift - mesh.io.req.bits.a_transpose := Mux(cfg_dataflow === Dataflow.OS.id.U, true.B, cfg_a_transpose) - mesh.io.req.bits.bd_transpose := Mux(cfg_dataflow === Dataflow.OS.id.U, false.B, cfg_bd_transpose) + mesh.io.req.bits.a_transpose := Mux( + cfg_dataflow === Dataflow.OS.id.U, + true.B, + cfg_a_transpose + ) + mesh.io.req.bits.bd_transpose := Mux( + cfg_dataflow === Dataflow.OS.id.U, + false.B, + cfg_bd_transpose + ) mesh.io.req.bits.total_rows := total_rows mesh.io.req.bits.tag.rob := robIdAsTag8(rob_id_reg) mesh.io.req.bits.flush := 1.U @@ -84,7 +96,9 @@ trait GemminiExCtrlStoreOps { this: GemminiExCtrl => io.bankWrite(i).req.valid := true.B io.bankWrite(i).req.bits.addr := store_row_cnt io.bankWrite(i).req.bits.data := slice - io.bankWrite(i).req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) + io.bankWrite(i).req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(true.B) + ) when(io.bankWrite(i).req.ready) { port_written(i) := true.B } diff --git a/examples/balls/gemmini/arch/src/main/scala/LoopCmdEncoder.scala b/examples/balls/gemmini/arch/src/main/scala/LoopCmdEncoder.scala index 07fced493..f6aa99e3f 100644 --- a/examples/balls/gemmini/arch/src/main/scala/LoopCmdEncoder.scala +++ b/examples/balls/gemmini/arch/src/main/scala/LoopCmdEncoder.scala @@ -14,6 +14,16 @@ import framework.frontend.decoder.DomainId class LoopCmdEncoder(val b: GlobalConfig) extends Module { val bankIdLen = log2Up(b.memDomain.bankNum) + def ballFunct(mnemonic: String): UInt = + b.ballDomain.ballISA + .find(_.mnemonic == mnemonic) + .map(_.funct7.U(7.W)) + .getOrElse(throw new IllegalArgumentException(s"$mnemonic not found in ballISA")) + + val preloadFunct = ballFunct("GEMMINI_PRELOAD") + val computePreFunct = ballFunct("GEMMINI_COMPUTE_PRELOADED") + val computeAccFunct = ballFunct("GEMMINI_COMPUTE_ACCUMULATED") + @public val io = IO(new Bundle { val cmd = Flipped(Decoupled(new LoopCmd(b))) @@ -94,7 +104,7 @@ class LoopCmdEncoder(val b: GlobalConfig) extends Module { } is(LoopSubCmdType.PRELOAD) { slot.cmd.domain_id := DomainId.BALL - slot.cmd.cmd.funct := 0x35.U // GEMMINI_PRELOAD (enable=011, opcode=5) + slot.cmd.cmd.funct := preloadFunct slot.cmd.cmd.rs1Data := lsub.bits.op1_bank | (lsub.bits.wr_bank << 20) | (lsub.bits.iter << 30) @@ -105,11 +115,11 @@ class LoopCmdEncoder(val b: GlobalConfig) extends Module { slot.cmd.bankAccess.wr_bank_id := lsub.bits.wr_bank } is(LoopSubCmdType.COMPUTE) { - slot.cmd.domain_id := DomainId.BALL - slot.cmd.cmd.funct := Mux( + slot.cmd.domain_id := DomainId.BALL + slot.cmd.cmd.funct := Mux( lsub.bits.compute_mode === 0.U, - 0x42.U, // COMPUTE_PRELOADED (enable=100, opcode=2) - 0x43.U // COMPUTE_ACCUMULATED (enable=100, opcode=3) + computePreFunct, + computeAccFunct ) slot.cmd.cmd.rs1Data := lsub.bits.op1_bank | (lsub.bits.op2_bank << 10) | diff --git a/examples/balls/gemmini/arch/src/main/scala/LoopConvAddrGen.scala b/examples/balls/gemmini/arch/src/main/scala/LoopConvAddrGen.scala index 2f2552235..dd4a8aa95 100644 --- a/examples/balls/gemmini/arch/src/main/scala/LoopConvAddrGen.scala +++ b/examples/balls/gemmini/arch/src/main/scala/LoopConvAddrGen.scala @@ -61,8 +61,10 @@ class LoopConvAddrGen(val b: GlobalConfig) extends Module { // Weight address: base + ((krow * kernel_dim + kcol) * in_channels + kch) * out_channels * elemSize // + och * elemSize - val weightOffset = ((io.krow * cfg.kernel_dim + io.kcol) * cfg.in_channels + io.kch) * - cfg.out_channels + io.och + val weightOffset = + ((io.krow * cfg.kernel_dim + io.kcol) * cfg.in_channels + io.kch) * + cfg.out_channels + io.och + io.weightAddr := cfg.dram_addr_weight + weightOffset * elemSize.U // Output address: base + ((batch * out_dim * out_dim + orow * out_dim + ocol) * out_channels + och) * accBytes diff --git a/examples/balls/gemmini/arch/src/main/scala/LoopConvUnroller.scala b/examples/balls/gemmini/arch/src/main/scala/LoopConvUnroller.scala index 4bdecc6c1..42b9b1af3 100644 --- a/examples/balls/gemmini/arch/src/main/scala/LoopConvUnroller.scala +++ b/examples/balls/gemmini/arch/src/main/scala/LoopConvUnroller.scala @@ -35,7 +35,8 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { val addrGen: Instance[LoopConvAddrGen] = Instantiate(new LoopConvAddrGen(b)) // FSM states - val sIdle :: sAlloc :: sMvinInput :: sMvinWeight :: sCompute :: sMvout :: sFree :: sDone :: Nil = Enum(8) + val sIdle :: sAlloc :: sMvinInput :: sMvinWeight :: sCompute :: sMvout :: sFree :: sDone :: Nil = + Enum(8) val state = RegInit(sIdle) val cfg = Reg(new LoopConvWsConfig(b)) @@ -89,7 +90,8 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { val v = Wire(Valid(new LoopSubCmd(b))) v.valid := true.B v.bits := 0.U.asTypeOf(new LoopSubCmd(b)) - v.bits.cmdType := (if (alloc) LoopSubCmdType.MSET_ALLOC else LoopSubCmdType.MSET_FREE) + v.bits.cmdType := (if (alloc) LoopSubCmdType.MSET_ALLOC + else LoopSubCmdType.MSET_FREE) v.bits.bank_id := bankId v.bits.bank_row := row.U v.bits.bank_col := col.U @@ -133,15 +135,18 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { }.elsewhen(krow_reg + 1.U < cfg.kernel_dim) { kch_reg := 0.U; kcol_reg := 0.U; krow_reg := krow_reg + 1.U }.elsewhen(och_reg + DIM.U < cfg.out_channels) { - kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; och_reg := och_reg + DIM.U + kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; + och_reg := och_reg + DIM.U }.elsewhen(ocol_reg + 1.U < cfg.out_dim) { kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; och_reg := 0.U ocol_reg := ocol_reg + 1.U }.elsewhen(orow_reg + 1.U < cfg.out_dim) { - kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; och_reg := 0.U; ocol_reg := 0.U + kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; och_reg := 0.U; + ocol_reg := 0.U orow_reg := orow_reg + 1.U }.elsewhen(batch_reg + 1.U < cfg.batch_size) { - kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; och_reg := 0.U; ocol_reg := 0.U; orow_reg := 0.U + kch_reg := 0.U; kcol_reg := 0.U; krow_reg := 0.U; och_reg := 0.U; + ocol_reg := 0.U; orow_reg := 0.U batch_reg := batch_reg + 1.U }.otherwise { state := sFree // All iterations done @@ -167,9 +172,12 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { // [MSET_alloc(input), MSET_alloc(weight), MSET_alloc(output), --] is(sAlloc) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := msetSlot(cfg.bank_input, alloc = true, row = 1, col = 1) - io.cmd.bits.slots(1) := msetSlot(cfg.bank_weight, alloc = true, row = 1, col = 1) - io.cmd.bits.slots(2) := msetSlot(cfg.bank_output, alloc = true, row = 1, col = 4) + io.cmd.bits + .slots(0) := msetSlot(cfg.bank_input, alloc = true, row = 1, col = 1) + io.cmd.bits + .slots(1) := msetSlot(cfg.bank_weight, alloc = true, row = 1, col = 1) + io.cmd.bits + .slots(2) := msetSlot(cfg.bank_output, alloc = true, row = 1, col = 4) io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { state := sMvinInput @@ -182,10 +190,20 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { val inputSlot = Mux( addrGen.io.isPadding, emptySlot(), // Skip MVIN if padding (load zeros implicitly) - mvinSlot(cfg.bank_input, addrGen.io.inputAddr, outRows(), inMemStride(cfg.input_stride)) + mvinSlot( + cfg.bank_input, + addrGen.io.inputAddr, + outRows(), + inMemStride(cfg.input_stride) + ) ) io.cmd.bits.slots(0) := inputSlot - io.cmd.bits.slots(1) := mvinSlot(cfg.bank_weight, addrGen.io.weightAddr, DIM.U, inMemStride(weightTileStride())) + io.cmd.bits.slots(1) := mvinSlot( + cfg.bank_weight, + addrGen.io.weightAddr, + DIM.U, + inMemStride(weightTileStride()) + ) io.cmd.bits.slots(2) := emptySlot() io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { @@ -205,13 +223,17 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { preSlot.bits.iter := DIM.U val compSlot = Wire(Valid(new LoopSubCmd(b))) - compSlot.valid := true.B - compSlot.bits := 0.U.asTypeOf(new LoopSubCmd(b)) - compSlot.bits.cmdType := LoopSubCmdType.COMPUTE - compSlot.bits.op1_bank := cfg.bank_input - compSlot.bits.op2_bank := cfg.bank_weight - compSlot.bits.wr_bank := cfg.bank_output - compSlot.bits.compute_mode := Mux(isFirstK, 0.U, 1.U) // PRELOADED first, then ACCUMULATED + compSlot.valid := true.B + compSlot.bits := 0.U.asTypeOf(new LoopSubCmd(b)) + compSlot.bits.cmdType := LoopSubCmdType.COMPUTE + compSlot.bits.op1_bank := cfg.bank_input + compSlot.bits.op2_bank := cfg.bank_weight + compSlot.bits.wr_bank := cfg.bank_output + compSlot.bits.compute_mode := Mux( + isFirstK, + 0.U, + 1.U + ) // PRELOADED first, then ACCUMULATED compSlot.bits.iter := DIM.U compSlot.bits.zero_op2 := cfg.no_bias compSlot.bits.zero_op1_tail := true.B @@ -261,9 +283,12 @@ class LoopConvUnroller(val b: GlobalConfig) extends Module { // [MSET_free(input), MSET_free(weight), MSET_free(output), --] is(sFree) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := msetSlot(cfg.bank_input, alloc = false, row = 0, col = 0) - io.cmd.bits.slots(1) := msetSlot(cfg.bank_weight, alloc = false, row = 0, col = 0) - io.cmd.bits.slots(2) := msetSlot(cfg.bank_output, alloc = false, row = 0, col = 0) + io.cmd.bits + .slots(0) := msetSlot(cfg.bank_input, alloc = false, row = 0, col = 0) + io.cmd.bits + .slots(1) := msetSlot(cfg.bank_weight, alloc = false, row = 0, col = 0) + io.cmd.bits + .slots(2) := msetSlot(cfg.bank_output, alloc = false, row = 0, col = 0) io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { state := sDone diff --git a/examples/balls/gemmini/arch/src/main/scala/LoopMatmulUnroller.scala b/examples/balls/gemmini/arch/src/main/scala/LoopMatmulUnroller.scala index aa2437e68..9fa6d9f1f 100644 --- a/examples/balls/gemmini/arch/src/main/scala/LoopMatmulUnroller.scala +++ b/examples/balls/gemmini/arch/src/main/scala/LoopMatmulUnroller.scala @@ -76,7 +76,8 @@ class LoopMatmulUnroller(val b: GlobalConfig) extends Module { val v = Wire(Valid(new LoopSubCmd(b))) v.valid := true.B v.bits := 0.U.asTypeOf(new LoopSubCmd(b)) - v.bits.cmdType := (if (alloc) LoopSubCmdType.MSET_ALLOC else LoopSubCmdType.MSET_FREE) + v.bits.cmdType := (if (alloc) LoopSubCmdType.MSET_ALLOC + else LoopSubCmdType.MSET_FREE) v.bits.bank_id := bankId v.bits.bank_row := row.U v.bits.bank_col := col.U @@ -176,9 +177,16 @@ class LoopMatmulUnroller(val b: GlobalConfig) extends Module { // Row 0: [MSET_alloc(A), MSET_alloc(B), MSET_alloc(C), --] is(sAllocBanks) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := msetSlot(cfg.bank_a, alloc = true, row = 1, col = 1) - io.cmd.bits.slots(1) := msetSlot(cfg.bank_b, alloc = true, row = 1, col = 1) - io.cmd.bits.slots(2) := msetSlot(cfg.bank_c, alloc = true, row = 1, col = 4) // accWidth = 4x inputWidth + io.cmd.bits + .slots(0) := msetSlot(cfg.bank_a, alloc = true, row = 1, col = 1) + io.cmd.bits + .slots(1) := msetSlot(cfg.bank_b, alloc = true, row = 1, col = 1) + io.cmd.bits.slots(2) := msetSlot( + cfg.bank_c, + alloc = true, + row = 1, + col = 4 + ) // accWidth = 4x inputWidth io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { state := sPrimeLoad @@ -188,8 +196,18 @@ class LoopMatmulUnroller(val b: GlobalConfig) extends Module { // Row 1: [MVIN_A(0), MVIN_B(0), --, --] is(sPrimeLoad) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := mvinSlot(cfg.bank_a, addrA(0.U, 0.U), DIM.U, inMemStride(cfg.stride_a)) - io.cmd.bits.slots(1) := mvinSlot(cfg.bank_b, addrB(0.U, 0.U), DIM.U, inMemStride(cfg.stride_b)) + io.cmd.bits.slots(0) := mvinSlot( + cfg.bank_a, + addrA(0.U, 0.U), + DIM.U, + inMemStride(cfg.stride_a) + ) + io.cmd.bits.slots(1) := mvinSlot( + cfg.bank_b, + addrB(0.U, 0.U), + DIM.U, + inMemStride(cfg.stride_b) + ) io.cmd.bits.slots(2) := emptySlot() io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { @@ -235,9 +253,24 @@ class LoopMatmulUnroller(val b: GlobalConfig) extends Module { // Main loop Row 2: [MVOUT(cur), MVIN_A(next), MVIN_B(next), --] is(sMainRow2) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := mvoutSlot(cfg.bank_c, addrC(i_reg, j_reg), DIM.U, outMemStride(cfg.stride_c)) - io.cmd.bits.slots(1) := mvinSlot(cfg.bank_a, addrA(next_i, next_k), DIM.U, inMemStride(cfg.stride_a)) - io.cmd.bits.slots(2) := mvinSlot(cfg.bank_b, addrB(next_k, next_j), DIM.U, inMemStride(cfg.stride_b)) + io.cmd.bits.slots(0) := mvoutSlot( + cfg.bank_c, + addrC(i_reg, j_reg), + DIM.U, + outMemStride(cfg.stride_c) + ) + io.cmd.bits.slots(1) := mvinSlot( + cfg.bank_a, + addrA(next_i, next_k), + DIM.U, + inMemStride(cfg.stride_a) + ) + io.cmd.bits.slots(2) := mvinSlot( + cfg.bank_b, + addrB(next_k, next_j), + DIM.U, + inMemStride(cfg.stride_b) + ) io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { @@ -253,7 +286,12 @@ class LoopMatmulUnroller(val b: GlobalConfig) extends Module { // Drain: emit final MVOUT for last iteration is(sDrainLast) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := mvoutSlot(cfg.bank_c, addrC(i_reg, j_reg), DIM.U, outMemStride(cfg.stride_c)) + io.cmd.bits.slots(0) := mvoutSlot( + cfg.bank_c, + addrC(i_reg, j_reg), + DIM.U, + outMemStride(cfg.stride_c) + ) io.cmd.bits.slots(1) := emptySlot() io.cmd.bits.slots(2) := emptySlot() io.cmd.bits.slots(3) := emptySlot() @@ -265,9 +303,12 @@ class LoopMatmulUnroller(val b: GlobalConfig) extends Module { // Free: [MSET_free(A), MSET_free(B), MSET_free(C), --] is(sFreeBanks) { io.cmd.valid := true.B - io.cmd.bits.slots(0) := msetSlot(cfg.bank_a, alloc = false, row = 0, col = 0) - io.cmd.bits.slots(1) := msetSlot(cfg.bank_b, alloc = false, row = 0, col = 0) - io.cmd.bits.slots(2) := msetSlot(cfg.bank_c, alloc = false, row = 0, col = 0) + io.cmd.bits + .slots(0) := msetSlot(cfg.bank_a, alloc = false, row = 0, col = 0) + io.cmd.bits + .slots(1) := msetSlot(cfg.bank_b, alloc = false, row = 0, col = 0) + io.cmd.bits + .slots(2) := msetSlot(cfg.bank_c, alloc = false, row = 0, col = 0) io.cmd.bits.slots(3) := emptySlot() when(io.cmd.fire) { state := sDone diff --git a/examples/balls/gemmini/compiler/src/Dialect/Buckyball/GemminiBall.td b/examples/balls/gemmini/compiler/src/Dialect/Buckyball/GemminiBall.td new file mode 100644 index 000000000..5e1ab3071 --- /dev/null +++ b/examples/balls/gemmini/compiler/src/Dialect/Buckyball/GemminiBall.td @@ -0,0 +1,21 @@ +//===-- GemminiBall.td - Gemmini Ball operations ------------------------===// + +#ifndef GEMMINI_BALL_OPS +#define GEMMINI_BALL_OPS + +include "Buckyball.td" + +def GemminiInstructionOp : Buckyball_Op<"gemmini"> { + let summary = "Gemmini command encoded for the target ISA"; + let description = [{ + Emits one Gemmini command. mnemonic must name an ISA entry enabled by the + selected target, and rs1/rs2 are the exact command operands defined by + that entry. This covers GEMMINI_CONFIG, GEMMINI_PRELOAD, + GEMMINI_COMPUTE_PRELOADED, GEMMINI_COMPUTE_ACCUMULATED, GEMMINI_LOOP_WS, + and GEMMINI_LOOP_CONV_WS without hard-coding funct7 values in the IR. + }]; + let arguments = (ins StrAttr:$mnemonic, I64:$rs1, I64:$rs2); + let assemblyFormat = "$mnemonic $rs1 `,` $rs2 attr-dict `:` type($rs1)"; +} + +#endif // GEMMINI_BALL_OPS diff --git a/examples/balls/gemmini/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/gemmini/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..bae6ada51 --- /dev/null +++ b/examples/balls/gemmini/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,46 @@ +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/Pattern.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" + +using namespace mlir; +using namespace buddy::buckyball; + +namespace { +class GemminiInstructionLowering + : public ConvertOpToLLVMPattern { +public: + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; + + LogicalResult + matchAndRewrite(GemminiInstructionOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("GemminiBall"); + rewriter.replaceOpWithNewOp( + op, adaptor.getRs1(), adaptor.getRs2(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7(op.getMnemonic()))); + return success(); + } +}; +} // namespace + +namespace mlir::buddy::buckyball { +void populateGemminiBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB) { + (void)stable; + (void)bankDepth; + (void)rushB; + patterns.add(converter); +} + +void configureGemminiBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { + (void)stable; + target.addIllegalOp(); +} +} // namespace mlir::buddy::buckyball diff --git a/examples/balls/gemmini/configs/GemminiBallParam.scala b/examples/balls/gemmini/configs/GemminiBallParam.scala index c3b71e96c..e13343ecd 100644 --- a/examples/balls/gemmini/configs/GemminiBallParam.scala +++ b/examples/balls/gemmini/configs/GemminiBallParam.scala @@ -36,4 +36,5 @@ object GemminiBallParam { outputDelay = BallParamLoader.int(tbl, "outputDelay") ) } + } diff --git a/examples/balls/gemmini/emu/src/02_gemmini_config.rs b/examples/balls/gemmini/emu/src/02_gemmini_config.rs index 0295fd5d3..1cd2b5e05 100644 --- a/examples/balls/gemmini/emu/src/02_gemmini_config.rs +++ b/examples/balls/gemmini/emu/src/02_gemmini_config.rs @@ -1,13 +1,11 @@ //===- 02_gemmini_config.rs - GEMMINI_CONFIG instruction -------------------===// use super::gemmini_state::gemini; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct GemminiConfig; -impl Instruction for GemminiConfig { - const FUNCT: u32 = 2; - +impl BallInstruction for GemminiConfig { fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { let mut g = gemini().lock().unwrap(); g.cfg.dataflow = ((xs2 >> 4) & 1) as u8; diff --git a/examples/balls/gemmini/emu/src/03_gemmini_flush.rs b/examples/balls/gemmini/emu/src/03_gemmini_flush.rs index 7743f95da..91333d08f 100644 --- a/examples/balls/gemmini/emu/src/03_gemmini_flush.rs +++ b/examples/balls/gemmini/emu/src/03_gemmini_flush.rs @@ -1,12 +1,10 @@ //===- 03_gemmini_flush.rs - GEMMINI_FLUSH instruction ---------------------===// -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct GemminiFlush; -impl Instruction for GemminiFlush { - const FUNCT: u32 = 3; - +impl BallInstruction for GemminiFlush { fn exec(_xs1: u64, _xs2: u64, _ctx: &mut ExecContext) -> u64 { 0 } diff --git a/examples/balls/gemmini/emu/src/53_gemmini_preload.rs b/examples/balls/gemmini/emu/src/53_gemmini_preload.rs index 7c1bc6a16..f3387169e 100644 --- a/examples/balls/gemmini/emu/src/53_gemmini_preload.rs +++ b/examples/balls/gemmini/emu/src/53_gemmini_preload.rs @@ -2,13 +2,11 @@ use super::super::bank::bank_num; use super::bank_matrix::read_i8_nn; use super::decode::{pbank, rs1_b0, rs1_b2, rs1_iter}; use super::gemmini_state::gemini; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct GemminiPreload; -impl Instruction for GemminiPreload { - const FUNCT: u32 = 53; - +impl BallInstruction for GemminiPreload { fn exec(xs1: u64, _xs2: u64, ctx: &mut ExecContext) -> u64 { let op1 = rs1_b0(xs1); let wr = rs1_b2(xs1); diff --git a/examples/balls/gemmini/emu/src/66_gemmini_compute_preloaded.rs b/examples/balls/gemmini/emu/src/66_gemmini_compute_preloaded.rs index ba8175112..2be397e2b 100644 --- a/examples/balls/gemmini/emu/src/66_gemmini_compute_preloaded.rs +++ b/examples/balls/gemmini/emu/src/66_gemmini_compute_preloaded.rs @@ -2,13 +2,11 @@ use super::super::bank::bank_num; use super::bank_matrix::{read_i32_nn, read_i8_nn, write_i32_nn_groups}; use super::decode::{pbank, pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; use super::gemmini_state::{gemini, in_shift as apply_in_shift}; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct GemminiComputePreloaded; -impl Instruction for GemminiComputePreloaded { - const FUNCT: u32 = 66; - +impl BallInstruction for GemminiComputePreloaded { fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { let op_a = rs1_b0(xs1); let op_b = rs1_b1(xs1); diff --git a/examples/balls/gemmini/emu/src/67_gemmini_compute_accumulated.rs b/examples/balls/gemmini/emu/src/67_gemmini_compute_accumulated.rs index 1fe161cba..9dd39a83e 100644 --- a/examples/balls/gemmini/emu/src/67_gemmini_compute_accumulated.rs +++ b/examples/balls/gemmini/emu/src/67_gemmini_compute_accumulated.rs @@ -2,13 +2,11 @@ use super::super::bank::bank_num; use super::bank_matrix::{read_i32_nn_groups, read_i8_nn, write_i32_nn_groups}; use super::decode::{pbank, pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; use super::gemmini_state::{gemini, in_shift as apply_in_shift}; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct GemminiComputeAccumulated; -impl Instruction for GemminiComputeAccumulated { - const FUNCT: u32 = 67; - +impl BallInstruction for GemminiComputeAccumulated { fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { let op_a = rs1_b0(xs1); let op_b = rs1_b1(xs1); diff --git a/examples/balls/gemmini/emu/src/80_gemmini_loop_ws.rs b/examples/balls/gemmini/emu/src/80_gemmini_loop_ws.rs index 05c73c68a..b2fcb8731 100644 --- a/examples/balls/gemmini/emu/src/80_gemmini_loop_ws.rs +++ b/examples/balls/gemmini/emu/src/80_gemmini_loop_ws.rs @@ -2,33 +2,33 @@ use super::super::bank::{bank_row_bytes, MATRIX_SIZE}; use super::gemmini_state::gemini; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::ExecContext; use super::loop_micro_ops::{ alloc, checked_stride, compute, free_after_digest, mvin, mvout, preload, }; // Shared implementation -fn exec_cfg_impl(funct: u32, xs2: u64) -> u64 { - let mut g = gemini().lock().unwrap(); - match funct { - 80 => { +fn exec_cfg_impl(mnemonic: &str, xs2: u64) -> u64 { + let mut g = gemini().lock().unwrap(); + match mnemonic { + "GEMMINI_LOOP_WS_CONFIG_BOUNDS" => { g.loop_ws.max_k = xs2 & 0xffff; g.loop_ws.max_j = (xs2 >> 16) & 0xffff; g.loop_ws.max_i = (xs2 >> 32) & 0xffff; } - 81 => g.loop_ws.addr_a = xs2 & ((1u64 << 39) - 1), - 82 => g.loop_ws.addr_b = xs2 & ((1u64 << 39) - 1), - 83 => g.loop_ws.addr_d = xs2 & ((1u64 << 39) - 1), - 84 => g.loop_ws.addr_c = xs2 & ((1u64 << 39) - 1), - 85 => { + "GEMMINI_LOOP_WS_CONFIG_ADDR_A" => g.loop_ws.addr_a = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_WS_CONFIG_ADDR_B" => g.loop_ws.addr_b = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_WS_CONFIG_ADDR_D" => g.loop_ws.addr_d = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_WS_CONFIG_ADDR_C" => g.loop_ws.addr_c = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_WS_CONFIG_STRIDES_AB" => { g.loop_ws.stride_a = xs2 & 0xffff_ffff; g.loop_ws.stride_b = xs2 >> 32; } - 86 => { + "GEMMINI_LOOP_WS_CONFIG_STRIDES_DC" => { g.loop_ws.stride_d = xs2 & 0xffff_ffff; g.loop_ws.stride_c = xs2 >> 32; } - _ => panic!("gemmini_loop_ws: unknown cfg funct={funct}"), + _ => panic!("gemmini_loop_ws: unknown cfg mnemonic={mnemonic}"), } 0 } @@ -131,99 +131,13 @@ fn exec_loop_impl(xs2: u64, ctx: &mut ExecContext) -> u64 { 0 } -fn latency_impl(funct: u32) -> u64 { - if funct == 87 { - 256 - } else { - 1 +pub fn execute(mnemonic: &str, xs2: u64, ctx: &mut ExecContext) -> u64 { + match mnemonic { + "GEMMINI_LOOP_WS" => exec_loop_impl(xs2, ctx), + _ => exec_cfg_impl(mnemonic, xs2), } } -// Individual instruction types for each funct -pub struct GemminiLoopWsConfigBounds; -impl Instruction for GemminiLoopWsConfigBounds { - const FUNCT: u32 = 80; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWsConfigAddrA; -impl Instruction for GemminiLoopWsConfigAddrA { - const FUNCT: u32 = 81; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWsConfigAddrB; -impl Instruction for GemminiLoopWsConfigAddrB { - const FUNCT: u32 = 82; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWsConfigAddrD; -impl Instruction for GemminiLoopWsConfigAddrD { - const FUNCT: u32 = 83; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWsConfigAddrC; -impl Instruction for GemminiLoopWsConfigAddrC { - const FUNCT: u32 = 84; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWsConfigStridesAB; -impl Instruction for GemminiLoopWsConfigStridesAB { - const FUNCT: u32 = 85; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWsConfigStridesDC; -impl Instruction for GemminiLoopWsConfigStridesDC { - const FUNCT: u32 = 86; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopWs; -impl Instruction for GemminiLoopWs { - const FUNCT: u32 = 87; - fn exec(_xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - exec_loop_impl(xs2, ctx) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } +pub fn latency(mnemonic: &str) -> u64 { + if mnemonic == "GEMMINI_LOOP_WS" { 256 } else { 1 } } diff --git a/examples/balls/gemmini/emu/src/96_gemmini_loop_conv_ws.rs b/examples/balls/gemmini/emu/src/96_gemmini_loop_conv_ws.rs index f4e2d6958..fb3cb4108 100644 --- a/examples/balls/gemmini/emu/src/96_gemmini_loop_conv_ws.rs +++ b/examples/balls/gemmini/emu/src/96_gemmini_loop_conv_ws.rs @@ -2,42 +2,42 @@ use super::super::bank::{bank_row_bytes, MATRIX_SIZE}; use super::gemmini_state::gemini; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::ExecContext; use super::loop_micro_ops::{ alloc, checked_stride, compute, free_after_digest, mvin, mvout, preload, }; // Shared implementation -fn exec_cfg_impl(funct: u32, xs2: u64) -> u64 { +fn exec_cfg_impl(mnemonic: &str, xs2: u64) -> u64 { let mut g = gemini().lock().unwrap(); - match funct { - 96 => { + match mnemonic { + "GEMMINI_LOOP_CONV_WS_CONFIG_1" => { g.loop_conv.batch = xs2 & 0xffff; g.loop_conv.in_dim = (xs2 >> 16) & 0xffff; g.loop_conv.in_ch = (xs2 >> 32) & 0xffff; } - 97 => { + "GEMMINI_LOOP_CONV_WS_CONFIG_2" => { g.loop_conv.out_ch = xs2 & 0xffff; g.loop_conv.out_dim = (xs2 >> 16) & 0xffff; g.loop_conv.stride = (xs2 >> 32) & 0xff; g.loop_conv.padding = (xs2 >> 40) & 0xff; } - 98 => { + "GEMMINI_LOOP_CONV_WS_CONFIG_3" => { g.loop_conv.kernel_dim = xs2 & 0xff; g.loop_conv.pool_size = (xs2 >> 8) & 0xff; g.loop_conv.pool_stride = (xs2 >> 16) & 0xff; g.loop_conv.pool_padding = (xs2 >> 24) & 0xff; } - 99 => g.loop_conv.addr_bias = xs2 & ((1u64 << 39) - 1), - 100 => g.loop_conv.addr_input = xs2 & ((1u64 << 39) - 1), - 101 => g.loop_conv.addr_weight = xs2 & ((1u64 << 39) - 1), - 102 => g.loop_conv.addr_output = xs2 & ((1u64 << 39) - 1), - 103 => { + "GEMMINI_LOOP_CONV_WS_CONFIG_4" => g.loop_conv.addr_bias = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_CONV_WS_CONFIG_5" => g.loop_conv.addr_input = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_CONV_WS_CONFIG_6" => g.loop_conv.addr_weight = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_CONV_WS_CONFIG_7" => g.loop_conv.addr_output = xs2 & ((1u64 << 39) - 1), + "GEMMINI_LOOP_CONV_WS_CONFIG_8" => { g.loop_conv.input_stride = xs2 & 0xffff_ffff; g.loop_conv.weight_stride = xs2 >> 32; } - 104 => g.loop_conv.output_stride = xs2 & 0xffff_ffff, - _ => panic!("gemmini_loop_conv_ws: unknown cfg funct={funct}"), + "GEMMINI_LOOP_CONV_WS_CONFIG_9" => g.loop_conv.output_stride = xs2 & 0xffff_ffff, + _ => panic!("gemmini_loop_conv_ws: unknown cfg mnemonic={mnemonic}"), } 0 } @@ -272,121 +272,13 @@ fn exec_loop_impl(xs2: u64, ctx: &mut ExecContext) -> u64 { 0 } -fn latency_impl(funct: u32) -> u64 { - if funct == 105 { - 256 - } else { - 1 +pub fn execute(mnemonic: &str, xs2: u64, ctx: &mut ExecContext) -> u64 { + match mnemonic { + "GEMMINI_LOOP_CONV_WS" => exec_loop_impl(xs2, ctx), + _ => exec_cfg_impl(mnemonic, xs2), } } -// Individual instruction types for each funct -pub struct GemminiLoopConvWsConfig1; -impl Instruction for GemminiLoopConvWsConfig1 { - const FUNCT: u32 = 96; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig2; -impl Instruction for GemminiLoopConvWsConfig2 { - const FUNCT: u32 = 97; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig3; -impl Instruction for GemminiLoopConvWsConfig3 { - const FUNCT: u32 = 98; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig4; -impl Instruction for GemminiLoopConvWsConfig4 { - const FUNCT: u32 = 99; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig5; -impl Instruction for GemminiLoopConvWsConfig5 { - const FUNCT: u32 = 100; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig6; -impl Instruction for GemminiLoopConvWsConfig6 { - const FUNCT: u32 = 101; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig7; -impl Instruction for GemminiLoopConvWsConfig7 { - const FUNCT: u32 = 102; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig8; -impl Instruction for GemminiLoopConvWsConfig8 { - const FUNCT: u32 = 103; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWsConfig9; -impl Instruction for GemminiLoopConvWsConfig9 { - const FUNCT: u32 = 104; - fn exec(_xs1: u64, xs2: u64, _ctx: &mut ExecContext) -> u64 { - exec_cfg_impl(Self::FUNCT, xs2) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } -} - -pub struct GemminiLoopConvWs; -impl Instruction for GemminiLoopConvWs { - const FUNCT: u32 = 105; - fn exec(_xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - exec_loop_impl(xs2, ctx) - } - fn latency(_xs1: u64, _xs2: u64) -> u64 { - latency_impl(Self::FUNCT) - } +pub fn latency(mnemonic: &str) -> u64 { + if mnemonic == "GEMMINI_LOOP_CONV_WS" { 256 } else { 1 } } diff --git a/examples/balls/gemmini/emu/src/lib.rs b/examples/balls/gemmini/emu/src/lib.rs index a845da259..9c8f33d8d 100644 --- a/examples/balls/gemmini/emu/src/lib.rs +++ b/examples/balls/gemmini/emu/src/lib.rs @@ -1,6 +1,6 @@ pub(crate) use crate::inst::{bank_matrix, decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::{BallInstruction, ExecContext}; #[path = "02_gemmini_config.rs"] mod f02_gemmini_config; @@ -31,180 +31,34 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => Some( - ::exec( - xs1, xs2, ctx, - ), - ), - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, - } + let mnemonic = crate::config::ball_domain::mnemonic_for_funct(funct)?; + let result = match mnemonic.as_str() { + "GEMMINI_CONFIG" => f02_gemmini_config::GemminiConfig::exec(xs1, xs2, ctx), + "GEMMINI_FLUSH" => f03_gemmini_flush::GemminiFlush::exec(xs1, xs2, ctx), + "GEMMINI_PRELOAD" => f53_gemmini_preload::GemminiPreload::exec(xs1, xs2, ctx), + "GEMMINI_COMPUTE_PRELOADED" => f66_gemmini_compute_preloaded::GemminiComputePreloaded::exec(xs1, xs2, ctx), + "GEMMINI_COMPUTE_ACCUMULATED" => f67_gemmini_compute_accumulated::GemminiComputeAccumulated::exec(xs1, xs2, ctx), + name if name.starts_with("GEMMINI_LOOP_WS_") || name == "GEMMINI_LOOP_WS" => f80_gemmini_loop_ws::execute(name, xs2, ctx), + name if name.starts_with("GEMMINI_LOOP_CONV_WS_") || name == "GEMMINI_LOOP_CONV_WS" => f96_gemmini_loop_conv_ws::execute(name, xs2, ctx), + _ => return None, + }; + Some(result) } pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> Option { if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => Some( - ::latency( - xs1, xs2, - ), - ), - ::FUNCT => Some( - ::latency( - xs1, xs2, - ), - ), - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, - } + let mnemonic = crate::config::ball_domain::mnemonic_for_funct(funct)?; + let latency = match mnemonic.as_str() { + "GEMMINI_CONFIG" => f02_gemmini_config::GemminiConfig::latency(xs1, xs2), + "GEMMINI_FLUSH" => f03_gemmini_flush::GemminiFlush::latency(xs1, xs2), + "GEMMINI_PRELOAD" => f53_gemmini_preload::GemminiPreload::latency(xs1, xs2), + "GEMMINI_COMPUTE_PRELOADED" => f66_gemmini_compute_preloaded::GemminiComputePreloaded::latency(xs1, xs2), + "GEMMINI_COMPUTE_ACCUMULATED" => f67_gemmini_compute_accumulated::GemminiComputeAccumulated::latency(xs1, xs2), + name if name.starts_with("GEMMINI_LOOP_WS_") || name == "GEMMINI_LOOP_WS" => f80_gemmini_loop_ws::latency(name), + name if name.starts_with("GEMMINI_LOOP_CONV_WS_") || name == "GEMMINI_LOOP_CONV_WS" => f96_gemmini_loop_conv_ws::latency(name), + _ => return None, + }; + Some(latency) } diff --git a/examples/balls/gemmini/emu/src/loop_micro_ops.rs b/examples/balls/gemmini/emu/src/loop_micro_ops.rs index 0c5f74729..e2f09a2ff 100644 --- a/examples/balls/gemmini/emu/src/loop_micro_ops.rs +++ b/examples/balls/gemmini/emu/src/loop_micro_ops.rs @@ -4,11 +4,13 @@ use super::instruction::ExecContext; const FUNCT_MVOUT: u32 = 16; const FUNCT_MSET: u32 = 32; const FUNCT_MVIN: u32 = 33; -const FUNCT_PRELOAD: u32 = 53; -const FUNCT_COMPUTE_PRELOADED: u32 = 66; -const FUNCT_COMPUTE_ACCUMULATED: u32 = 67; const MAX_DMA_STRIDE: u64 = (1 << 19) - 1; +fn ball_funct(mnemonic: &str) -> u32 { + crate::config::ball_domain::funct_for_mnemonic(mnemonic) + .unwrap_or_else(|| panic!("Gemmini mnemonic {mnemonic} is not declared in Core ballISA")) +} + fn execute(funct: u32, xs1: u64, xs2: u64, ctx: &mut ExecContext) { decode::execute_known(funct, xs1, xs2, ctx) .unwrap_or_else(|| panic!("Gemmini loop emitted unsupported micro-op funct7={funct}")); @@ -65,7 +67,7 @@ pub fn mvout(ctx: &mut ExecContext, bank: u64, addr: u64, iter: u64, stride: u64 } pub fn preload(ctx: &mut ExecContext, source: u64, output: u64, iter: u64) { - execute(FUNCT_PRELOAD, banks(source, 0, output, iter), 1, ctx); + execute(ball_funct("GEMMINI_PRELOAD"), banks(source, 0, output, iter), 1, ctx); } pub fn compute( @@ -79,9 +81,9 @@ pub fn compute( zero_op1_tail: bool, ) { let funct = if accumulated { - FUNCT_COMPUTE_ACCUMULATED + ball_funct("GEMMINI_COMPUTE_ACCUMULATED") } else { - FUNCT_COMPUTE_PRELOADED + ball_funct("GEMMINI_COMPUTE_PRELOADED") }; let mode = if accumulated { 3 } else { 2 }; let xs2 = mode | ((zero_op2 as u64) << 4) | ((zero_op1_tail as u64) << 5); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_atranspose_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_atranspose_test.c index f9d808668..3848319d5 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_atranspose_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_atranspose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_basic_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_basic_test.c index b552ca1a0..bb6bb7929 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_basic_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_basic_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_btranspose_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_btranspose_test.c index dc2d25757..ce3745497 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_btranspose_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_btranspose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_shift_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_shift_test.c index 2e7233a76..6dfc43464 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_shift_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_cisc_shift_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_abtranspose_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_abtranspose_test.c index c15dbe327..ca1070702 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_abtranspose_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_abtranspose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -36,6 +37,9 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_b, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_a); + bb_mem_release(bank_b); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini OS RISC ab_transpose Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_atranspose_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_atranspose_test.c index c4bbddcb4..9d03e9528 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_atranspose_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_atranspose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -34,6 +35,9 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_b, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_a); + bb_mem_release(bank_b); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini OS RISC a_transpose Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_basic_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_basic_test.c index 32743c000..0cf49bfc3 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_basic_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_basic_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -37,6 +38,10 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_b, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_a); + bb_mem_release(bank_b); + bb_mem_release(bank_c); + bb_mem_release(bank_d_zeros); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini OS RISC Basic Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_btranspose_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_btranspose_test.c index 1aa09779f..8374a7aff 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_btranspose_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_btranspose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -34,6 +35,9 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_b, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_a); + bb_mem_release(bank_b); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini OS RISC b_transpose Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_shift_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_shift_test.c index cf2af1790..ba4eb159a 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_shift_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_os_risc_shift_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -35,6 +36,9 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_b, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_a); + bb_mem_release(bank_b); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini OS RISC in_shift Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_ws_cisc_conv_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_ws_cisc_conv_test.c index cf0f52468..11f285e49 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_ws_cisc_conv_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_ws_cisc_conv_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include // Pointwise conv: batch=1, in_dim=1, in_channels=DIM, out_channels=DIM, diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_basic_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_basic_test.c index 66a71950a..5ccd9c1e8 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_basic_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_basic_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -38,6 +39,10 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_d, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_w); + bb_mem_release(bank_a); + bb_mem_release(bank_d); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini WS RISC Basic Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_btranspose_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_btranspose_test.c index b7063a1ac..2e830715e 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_btranspose_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_btranspose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -40,6 +41,10 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_d, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_w); + bb_mem_release(bank_a); + bb_mem_release(bank_d); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini WS RISC b_transpose Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_shift_test.c b/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_shift_test.c index d61228348..9e0d89264 100644 --- a/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_shift_test.c +++ b/examples/balls/gemmini/workloads/ctests/gemmini_ws_risc_shift_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define DIM 16 @@ -41,6 +42,10 @@ int main() { bb_gemmini_compute_preloaded(bank_a, bank_d, bank_c, DIM); bb_mvout((uintptr_t)mat_c, bank_c, DIM, 1); bb_fence(); + bb_mem_release(bank_w); + bb_mem_release(bank_a); + bb_mem_release(bank_d); + bb_mem_release(bank_c); if (compare_u32_matrices(mat_c, expected, DIM, DIM)) { printf("Gemmini WS RISC in_shift Test PASSED\n"); diff --git a/examples/balls/gemmini/workloads/isa/gemmini.h b/examples/balls/gemmini/workloads/isa/gemmini.h new file mode 100644 index 000000000..2994d8f50 --- /dev/null +++ b/examples/balls/gemmini/workloads/isa/gemmini.h @@ -0,0 +1,12 @@ +#ifndef _BB_GEMMINI_ISA_H_ +#define _BB_GEMMINI_ISA_H_ + +#include "gemmini_compute_accumulated.h" +#include "gemmini_compute_preloaded.h" +#include "gemmini_config.h" +#include "gemmini_flush.h" +#include "gemmini_loop_conv_ws.h" +#include "gemmini_loop_ws.h" +#include "gemmini_preload.h" + +#endif // _BB_GEMMINI_ISA_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/67_gemmini_compute_accumulated.c b/examples/balls/gemmini/workloads/isa/gemmini_compute_accumulated.h similarity index 85% rename from bb-tests/workloads/lib/bbhw/isa/67_gemmini_compute_accumulated.c rename to examples/balls/gemmini/workloads/isa/gemmini_compute_accumulated.h index d04bab421..f36cad1d9 100644 --- a/bb-tests/workloads/lib/bbhw/isa/67_gemmini_compute_accumulated.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_compute_accumulated.h @@ -1,9 +1,9 @@ #ifndef _BB_GEMMINI_COMPUTE_ACCUMULATED_H_ #define _BB_GEMMINI_COMPUTE_ACCUMULATED_H_ -#include "isa.h" +#include +#include -#define BB_GEMMINI_COMPUTE_ACCUMULATED_FUNC7 67 #define BB_GEMMINI_COMPUTE_ACCUMULATED_RS2 3ULL // Compute matmul reusing previously accumulated results @@ -16,6 +16,6 @@ BUCKYBALL_INSTRUCTION_R_R((BB_BANK0(op1_bank_id) | BB_BANK1(op2_bank_id) | \ BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ BB_GEMMINI_COMPUTE_ACCUMULATED_RS2, \ - BB_GEMMINI_COMPUTE_ACCUMULATED_FUNC7) + BB_FUNC7(GEMMINI_COMPUTE_ACCUMULATED)) #endif // _BB_GEMMINI_COMPUTE_ACCUMULATED_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/66_gemmini_compute_preloaded.c b/examples/balls/gemmini/workloads/isa/gemmini_compute_preloaded.h similarity index 86% rename from bb-tests/workloads/lib/bbhw/isa/66_gemmini_compute_preloaded.c rename to examples/balls/gemmini/workloads/isa/gemmini_compute_preloaded.h index e16d9384e..1d0a6bf00 100644 --- a/bb-tests/workloads/lib/bbhw/isa/66_gemmini_compute_preloaded.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_compute_preloaded.h @@ -1,9 +1,9 @@ #ifndef _BB_GEMMINI_COMPUTE_PRELOADED_H_ #define _BB_GEMMINI_COMPUTE_PRELOADED_H_ -#include "isa.h" +#include +#include -#define BB_GEMMINI_COMPUTE_PRELOADED_FUNC7 66 #define BB_GEMMINI_COMPUTE_PRELOADED_RS2 2ULL // Compute matmul using preloaded data: C = A * B + D(preloaded) @@ -16,6 +16,6 @@ BUCKYBALL_INSTRUCTION_R_R((BB_BANK0(op1_bank_id) | BB_BANK1(op2_bank_id) | \ BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ BB_GEMMINI_COMPUTE_PRELOADED_RS2, \ - BB_GEMMINI_COMPUTE_PRELOADED_FUNC7) + BB_FUNC7(GEMMINI_COMPUTE_PRELOADED)) #endif // _BB_GEMMINI_COMPUTE_PRELOADED_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/02_gemmini_config.c b/examples/balls/gemmini/workloads/isa/gemmini_config.h similarity index 89% rename from bb-tests/workloads/lib/bbhw/isa/02_gemmini_config.c rename to examples/balls/gemmini/workloads/isa/gemmini_config.h index f1768247e..fc50c0748 100644 --- a/bb-tests/workloads/lib/bbhw/isa/02_gemmini_config.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_config.h @@ -1,9 +1,8 @@ #ifndef _BB_GEMMINI_CONFIG_H_ #define _BB_GEMMINI_CONFIG_H_ -#include "isa.h" - -#define BB_GEMMINI_CONFIG_FUNC7 2 +#include +#include // Configure Gemmini matrix array // All config parameters go in rs2 (special), starting from bit 4. @@ -19,6 +18,6 @@ FIELD(a_transpose, 7, 7) | \ FIELD(b_transpose, 8, 8) | \ FIELD(in_shift, 9, 40)), \ - BB_GEMMINI_CONFIG_FUNC7) + BB_FUNC7(GEMMINI_CONFIG)) #endif // _BB_GEMMINI_CONFIG_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/03_gemmini_flush.c b/examples/balls/gemmini/workloads/isa/gemmini_flush.h similarity index 64% rename from bb-tests/workloads/lib/bbhw/isa/03_gemmini_flush.c rename to examples/balls/gemmini/workloads/isa/gemmini_flush.h index fe16db579..2c55b62c0 100644 --- a/bb-tests/workloads/lib/bbhw/isa/03_gemmini_flush.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_flush.h @@ -1,13 +1,13 @@ #ifndef _BB_GEMMINI_FLUSH_H_ #define _BB_GEMMINI_FLUSH_H_ -#include "isa.h" +#include +#include -#define BB_GEMMINI_FLUSH_FUNC7 3 #define BB_GEMMINI_FLUSH_RS2 4ULL // Flush the matrix array state #define bb_gemmini_flush() \ - BUCKYBALL_INSTRUCTION_R_R(0, BB_GEMMINI_FLUSH_RS2, BB_GEMMINI_FLUSH_FUNC7) + BUCKYBALL_INSTRUCTION_R_R(0, BB_GEMMINI_FLUSH_RS2, BB_FUNC7(GEMMINI_FLUSH)) #endif // _BB_GEMMINI_FLUSH_H_ diff --git a/bb-tests/workloads/lib/bbhw/isa/96_gemmini_loop_conv_ws.c b/examples/balls/gemmini/workloads/isa/gemmini_loop_conv_ws.h similarity index 72% rename from bb-tests/workloads/lib/bbhw/isa/96_gemmini_loop_conv_ws.c rename to examples/balls/gemmini/workloads/isa/gemmini_loop_conv_ws.h index 7e551a2a1..5f04089f7 100644 --- a/bb-tests/workloads/lib/bbhw/isa/96_gemmini_loop_conv_ws.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_loop_conv_ws.h @@ -1,25 +1,15 @@ #ifndef _BB_GEMMINI_LOOP_CONV_WS_H #define _BB_GEMMINI_LOOP_CONV_WS_H -#include "isa.h" - -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_1_FUNC7 96 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_2_FUNC7 97 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_3_FUNC7 98 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_4_FUNC7 99 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_5_FUNC7 100 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_6_FUNC7 101 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_7_FUNC7 102 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_8_FUNC7 103 -#define BB_GEMMINI_LOOP_CONV_WS_CONFIG_9_FUNC7 104 -#define BB_GEMMINI_LOOP_CONV_WS_FUNC7 105 +#include +#include #define bb_gemmini_loop_conv_ws_config_1(batch_size, in_dim, in_channels) \ BUCKYBALL_INSTRUCTION_R_R(0, \ (FIELD(batch_size, 0, 15) | \ FIELD(in_dim, 16, 31) | \ FIELD(in_channels, 32, 47)), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_1_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_1)) #define bb_gemmini_loop_conv_ws_config_2(out_channels, out_dim, stride, \ padding) \ @@ -27,7 +17,7 @@ (FIELD(out_channels, 0, 15) | \ FIELD(out_dim, 16, 31) | FIELD(stride, 32, 39) | \ FIELD(padding, 40, 47)), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_2_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_2)) #define bb_gemmini_loop_conv_ws_config_3(kernel_dim, pool_size, pool_stride, \ pool_padding) \ @@ -35,38 +25,38 @@ 0, \ (FIELD(kernel_dim, 0, 7) | FIELD(pool_size, 8, 15) | \ FIELD(pool_stride, 16, 23) | FIELD(pool_padding, 24, 31)), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_3_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_3)) #define bb_gemmini_loop_conv_ws_config_4(addr_bias) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr_bias, 0, 38), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_4_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_4)) #define bb_gemmini_loop_conv_ws_config_5(addr_input) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr_input, 0, 38), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_5_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_5)) #define bb_gemmini_loop_conv_ws_config_6(addr_weight) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr_weight, 0, 38), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_6_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_6)) #define bb_gemmini_loop_conv_ws_config_7(addr_output) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr_output, 0, 38), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_7_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_7)) #define bb_gemmini_loop_conv_ws_config_8(input_stride, weight_stride) \ BUCKYBALL_INSTRUCTION_R_R( \ 0, (FIELD(input_stride, 0, 31) | FIELD(weight_stride, 32, 63)), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_8_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_8)) #define bb_gemmini_loop_conv_ws_config_9(output_stride) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(output_stride, 0, 31), \ - BB_GEMMINI_LOOP_CONV_WS_CONFIG_9_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS_CONFIG_9)) #define bb_gemmini_loop_conv_ws(bank_input, bank_weight, bank_output, no_bias) \ BUCKYBALL_INSTRUCTION_R_R( \ 0, \ (FIELD(bank_input, 0, 9) | FIELD(bank_weight, 10, 19) | \ FIELD(bank_output, 20, 29) | FIELD(no_bias, 30, 30)), \ - BB_GEMMINI_LOOP_CONV_WS_FUNC7) + BB_FUNC7(GEMMINI_LOOP_CONV_WS)) #endif diff --git a/bb-tests/workloads/lib/bbhw/isa/80_gemmini_loop_ws.c b/examples/balls/gemmini/workloads/isa/gemmini_loop_ws.h similarity index 66% rename from bb-tests/workloads/lib/bbhw/isa/80_gemmini_loop_ws.c rename to examples/balls/gemmini/workloads/isa/gemmini_loop_ws.h index 3b99ec949..769c1f660 100644 --- a/bb-tests/workloads/lib/bbhw/isa/80_gemmini_loop_ws.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_loop_ws.h @@ -1,52 +1,44 @@ #ifndef _BB_GEMMINI_LOOP_WS_H #define _BB_GEMMINI_LOOP_WS_H -#include "isa.h" - -#define BB_GEMMINI_LOOP_WS_CONFIG_BOUNDS_FUNC7 80 -#define BB_GEMMINI_LOOP_WS_CONFIG_ADDR_A_FUNC7 81 -#define BB_GEMMINI_LOOP_WS_CONFIG_ADDR_B_FUNC7 82 -#define BB_GEMMINI_LOOP_WS_CONFIG_ADDR_D_FUNC7 83 -#define BB_GEMMINI_LOOP_WS_CONFIG_ADDR_C_FUNC7 84 -#define BB_GEMMINI_LOOP_WS_CONFIG_STRIDES_AB_FUNC7 85 -#define BB_GEMMINI_LOOP_WS_CONFIG_STRIDES_DC_FUNC7 86 -#define BB_GEMMINI_LOOP_WS_FUNC7 87 +#include +#include #define bb_gemmini_loop_ws_config_bounds(max_i, max_j, max_k) \ BUCKYBALL_INSTRUCTION_R_R( \ 0, (FIELD(max_k, 0, 15) | FIELD(max_j, 16, 31) | FIELD(max_i, 32, 47)), \ - BB_GEMMINI_LOOP_WS_CONFIG_BOUNDS_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_BOUNDS)) #define bb_gemmini_loop_ws_config_addr_a(addr) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr, 0, 38), \ - BB_GEMMINI_LOOP_WS_CONFIG_ADDR_A_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_ADDR_A)) #define bb_gemmini_loop_ws_config_addr_b(addr) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr, 0, 38), \ - BB_GEMMINI_LOOP_WS_CONFIG_ADDR_B_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_ADDR_B)) #define bb_gemmini_loop_ws_config_addr_d(addr) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr, 0, 38), \ - BB_GEMMINI_LOOP_WS_CONFIG_ADDR_D_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_ADDR_D)) #define bb_gemmini_loop_ws_config_addr_c(addr) \ BUCKYBALL_INSTRUCTION_R_R(0, FIELD(addr, 0, 38), \ - BB_GEMMINI_LOOP_WS_CONFIG_ADDR_C_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_ADDR_C)) #define bb_gemmini_loop_ws_config_strides_ab(stride_a, stride_b) \ BUCKYBALL_INSTRUCTION_R_R( \ 0, (FIELD(stride_a, 0, 31) | FIELD(stride_b, 32, 63)), \ - BB_GEMMINI_LOOP_WS_CONFIG_STRIDES_AB_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_STRIDES_AB)) #define bb_gemmini_loop_ws_config_strides_dc(stride_d, stride_c) \ BUCKYBALL_INSTRUCTION_R_R( \ 0, (FIELD(stride_d, 0, 31) | FIELD(stride_c, 32, 63)), \ - BB_GEMMINI_LOOP_WS_CONFIG_STRIDES_DC_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS_CONFIG_STRIDES_DC)) #define bb_gemmini_loop_ws(bank_a, bank_b, bank_c, low_d) \ BUCKYBALL_INSTRUCTION_R_R(0, \ (FIELD(bank_a, 0, 9) | FIELD(bank_b, 10, 19) | \ FIELD(bank_c, 20, 29) | FIELD(low_d, 30, 30)), \ - BB_GEMMINI_LOOP_WS_FUNC7) + BB_FUNC7(GEMMINI_LOOP_WS)) #endif diff --git a/bb-tests/workloads/lib/bbhw/isa/53_gemmini_preload.c b/examples/balls/gemmini/workloads/isa/gemmini_preload.h similarity index 82% rename from bb-tests/workloads/lib/bbhw/isa/53_gemmini_preload.c rename to examples/balls/gemmini/workloads/isa/gemmini_preload.h index 77afab734..226fca4bb 100644 --- a/bb-tests/workloads/lib/bbhw/isa/53_gemmini_preload.c +++ b/examples/balls/gemmini/workloads/isa/gemmini_preload.h @@ -1,9 +1,9 @@ #ifndef _BB_GEMMINI_PRELOAD_H_ #define _BB_GEMMINI_PRELOAD_H_ -#include "isa.h" +#include +#include -#define BB_GEMMINI_PRELOAD_FUNC7 53 #define BB_GEMMINI_PRELOAD_RS2 1ULL // Preload D/B matrix into matrix array @@ -13,6 +13,6 @@ #define bb_gemmini_preload(op1_bank_id, wr_bank_id, iter) \ BUCKYBALL_INSTRUCTION_R_R( \ (BB_BANK0(op1_bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ - BB_GEMMINI_PRELOAD_RS2, BB_GEMMINI_PRELOAD_FUNC7) + BB_GEMMINI_PRELOAD_RS2, BB_FUNC7(GEMMINI_PRELOAD)) #endif // _BB_GEMMINI_PRELOAD_H_ diff --git a/examples/balls/im2col/arch/src/main/scala/Im2col.scala b/examples/balls/im2col/arch/src/main/scala/Im2col.scala index 393a87f49..8fcd4f038 100644 --- a/examples/balls/im2col/arch/src/main/scala/Im2col.scala +++ b/examples/balls/im2col/arch/src/main/scala/Im2col.scala @@ -17,7 +17,9 @@ class Im2col(val b: GlobalConfig) extends Module { private val map = b.ballDomain.ballIdMappings .find(_.ballName == "Im2colBall") - .getOrElse(throw new IllegalArgumentException("Im2colBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("Im2colBall not found in config") + ) private val inBW = map.inBW private val outBW = map.outBW @@ -35,8 +37,12 @@ class Im2col(val b: GlobalConfig) extends Module { val cfg = Module(new Im2colConfigRegs(b, maxIter, maxKSize, maxPad)) val win = Module(new Im2colWindow(maxKSize, b.frontend.iter_len)) - val lineBuf: Instance[LineBufferManager] = Instantiate(new LineBufferManager(b)) - val writer: Instance[StreamWriter] = Instantiate(new StreamWriter(b)) + + val lineBuf: Instance[LineBufferManager] = Instantiate( + new LineBufferManager(b) + ) + + val writer: Instance[StreamWriter] = Instantiate(new StreamWriter(b)) val running = RegInit(false.B) val inputReady = RegInit(false.B) @@ -63,8 +69,8 @@ class Im2col(val b: GlobalConfig) extends Module { respPending := false.B } - win.io.init := io.cmdReq.fire - win.io.next := false.B + win.io.init := io.cmdReq.fire + win.io.next := false.B win.io.inRows := cfg.io.inRows win.io.inCols := cfg.io.inCols win.io.kRows := cfg.io.kRows @@ -81,7 +87,7 @@ class Im2col(val b: GlobalConfig) extends Module { for (i <- 0 until inBW) { lineBuf.io.bankRead(i) <> io.bankRead(i) } - lineBuf.io.start := cmdStart + lineBuf.io.start := cmdStart lineBuf.io.inRows := cfg.io.inRows lineBuf.io.inCols := cfg.io.inCols lineBuf.io.rowStride := cfg.io.rowStride @@ -89,12 +95,12 @@ class Im2col(val b: GlobalConfig) extends Module { lineBuf.io.padding := cfg.io.padding lineBuf.io.startRow := cfg.io.startRow lineBuf.io.startCol := cfg.io.startCol - lineBuf.io.outRow := win.io.outRow - lineBuf.io.outCol := win.io.outCol - lineBuf.io.kRowIdx := win.io.kRowIdx - lineBuf.io.kColIdx := win.io.kColIdx - lineBuf.io.rBankId := cfg.io.rBank - lineBuf.io.robId := cfg.io.robId + lineBuf.io.outRow := win.io.outRow + lineBuf.io.outCol := win.io.outCol + lineBuf.io.kRowIdx := win.io.kRowIdx + lineBuf.io.kColIdx := win.io.kColIdx + lineBuf.io.rBankId := cfg.io.rBank + lineBuf.io.robId := cfg.io.robId for (i <- 0 until outBW) { writer.io.bankWrite(i) <> io.bankWrite(i) @@ -110,8 +116,11 @@ class Im2col(val b: GlobalConfig) extends Module { writer.io.kRows := cfg.io.kRows writer.io.kCols := cfg.io.kCols val paddedCols = cfg.io.inCols +& (cfg.io.padding << 1) - val outputCols = ((paddedCols - cfg.io.kCols - cfg.io.startCol) / - cfg.io.colStride) + 1.U + + val outputCols = + ((paddedCols - cfg.io.kCols - cfg.io.startCol) / + cfg.io.colStride) + 1.U + writer.io.windowIdx := win.io.outRow * outputCols + win.io.outCol when(cmdStart) { diff --git a/examples/balls/im2col/arch/src/main/scala/Im2colBall.scala b/examples/balls/im2col/arch/src/main/scala/Im2colBall.scala index 15fb38c69..d0db005e0 100644 --- a/examples/balls/im2col/arch/src/main/scala/Im2colBall.scala +++ b/examples/balls/im2col/arch/src/main/scala/Im2colBall.scala @@ -13,10 +13,15 @@ import framework.top.GlobalConfig */ @instantiable class Im2colBall(val b: GlobalConfig) extends Module with HasBlink { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "Im2colBall") - .getOrElse(throw new IllegalArgumentException("Im2colBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "Im2colBall") + .getOrElse( + throw new IllegalArgumentException("Im2colBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW @public val io = IO(new BlinkIO(b, inBW, outBW)) diff --git a/examples/balls/im2col/arch/src/main/scala/Im2colConfigRegs.scala b/examples/balls/im2col/arch/src/main/scala/Im2colConfigRegs.scala index bd661fe40..8c56a8d53 100644 --- a/examples/balls/im2col/arch/src/main/scala/Im2colConfigRegs.scala +++ b/examples/balls/im2col/arch/src/main/scala/Im2colConfigRegs.scala @@ -6,15 +6,17 @@ import framework.balldomain.rs.BallRsIssue import framework.top.GlobalConfig class Im2colConfigRegs( - val b: GlobalConfig, - maxIter: Int, - maxKSize: Int, - maxPadding: Int -) extends Module { + val b: GlobalConfig, + maxIter: Int, + maxKSize: Int, + maxPadding: Int) + extends Module { - private val kW = log2Ceil(maxKSize + 1) - private val bankEntries = b.memDomain.bankEntries - private val tile = 16 + private val kW = log2Ceil(maxKSize + 1) + private val bankEntries = b.memDomain.bankEntries + private val bankNum = b.memDomain.bankNum + private val maxFootprint = bankEntries * bankNum + private val tile = 16 val io = IO(new Bundle { val cmd = Input(new BallRsIssue(b)) @@ -56,13 +58,17 @@ class Im2colConfigRegs( val legacy = io.cmd.cmd.iter === 0.U val cmdInRows = Mux(legacy, io.cmd.cmd.special(31, 24), io.cmd.cmd.iter) val cmdInCols = Mux(legacy, io.cmd.cmd.special(23, 16), io.cmd.cmd.iter) - val cmdKRows = Mux(legacy, io.cmd.cmd.special(15, 8), io.cmd.cmd.special(7, 0)) + val cmdKRows = + Mux(legacy, io.cmd.cmd.special(15, 8), io.cmd.cmd.special(7, 0)) val cmdKCols = io.cmd.cmd.special(7, 0) val cmdRowStride = Mux(legacy, 1.U, io.cmd.cmd.special(15, 8)) - val cmdColStride = Mux(legacy, io.cmd.cmd.special(55, 48), io.cmd.cmd.special(15, 8)) + val cmdColStride = + Mux(legacy, io.cmd.cmd.special(55, 48), io.cmd.cmd.special(15, 8)) val cmdPadding = Mux(legacy, 0.U, io.cmd.cmd.special(23, 16)) - val cmdStartRow = Mux(legacy, io.cmd.cmd.special(47, 40), 0.U) - val cmdStartCol = Mux(legacy, io.cmd.cmd.special(39, 32), 0.U) + val cmdStartRow = + Mux(legacy, io.cmd.cmd.special(47, 40), io.cmd.cmd.special(39, 32)) + val cmdStartCol = + Mux(legacy, io.cmd.cmd.special(39, 32), io.cmd.cmd.special(31, 24)) val cmdRBank = io.cmd.cmd.op1_bank val cmdWBank = io.cmd.cmd.wr_bank @@ -86,29 +92,40 @@ class Im2colConfigRegs( val paddedRows = cmdInRows +& (cmdPadding << 1) val paddedCols = cmdInCols +& (cmdPadding << 1) + val shapeOk = (cmdInRows >= 1.U) && (cmdInRows <= maxIter.U) && (cmdInCols >= 1.U) && (cmdInCols <= maxIter.U) && (cmdKRows >= 1.U) && (cmdKRows <= maxKSize.U) && (cmdKCols >= 1.U) && (cmdKCols <= maxKSize.U) && (cmdRowStride >= 1.U) && (cmdColStride >= 1.U) && (cmdPadding <= maxPadding.U) && + (cmdStartRow <= cmdPadding) && (cmdStartCol <= cmdPadding) && (paddedRows >= cmdKRows + cmdStartRow) && (paddedCols >= cmdKCols + cmdStartCol) && (cmdRBank =/= cmdWBank) - val outRows = Mux(shapeOk, - ((paddedRows - cmdKRows - cmdStartRow) / cmdRowStride) + 1.U, 0.U) - val outCols = Mux(shapeOk, - ((paddedCols - cmdKCols - cmdStartCol) / cmdColStride) + 1.U, 0.U) - val windows = outRows * outCols - val kElems = cmdKRows * cmdKCols - val mTiles = (windows +& (tile - 1).U) / tile.U - val kTiles = (kElems +& (tile - 1).U) / tile.U - val tiledRows = mTiles * kTiles * tile.U + val outRows = Mux( + shapeOk, + ((paddedRows - cmdKRows - cmdStartRow) / cmdRowStride) + 1.U, + 0.U + ) + + val outCols = Mux( + shapeOk, + ((paddedCols - cmdKCols - cmdStartCol) / cmdColStride) + 1.U, + 0.U + ) + + val windows = outRows * outCols + val kElems = cmdKRows * cmdKCols + val mTiles = (windows +& (tile - 1).U) / tile.U + val kTiles = (kElems +& (tile - 1).U) / tile.U + val tiledRows = mTiles * kTiles * tile.U val legacyBeats = (windows * kElems +& (tile - 1).U) / tile.U - val footprint = Mux(legacy, legacyBeats, tiledRows) + val footprint = Mux(legacy, legacyBeats, tiledRows) - io.invalid := !shapeOk || (footprint > bankEntries.U) + // Matches emu capacity (groups * bank_lines); StreamWriter spans multi-group addrs. + io.invalid := !shapeOk || (footprint > maxFootprint.U) io.robId := Mux(io.load, io.cmd.rob_id, robId) io.isSub := Mux(io.load, io.cmd.is_sub, isSub) io.subRobId := Mux(io.load, io.cmd.sub_rob_id, subRobId) diff --git a/examples/balls/im2col/arch/src/main/scala/Im2colWindow.scala b/examples/balls/im2col/arch/src/main/scala/Im2colWindow.scala index febc51139..513687fe6 100644 --- a/examples/balls/im2col/arch/src/main/scala/Im2colWindow.scala +++ b/examples/balls/im2col/arch/src/main/scala/Im2colWindow.scala @@ -7,9 +7,9 @@ class Im2colWindow(maxK: Int, iterW: Int) extends Module { require(iterW >= 1, "Im2colWindow iterW must be >= 1") val io = IO(new Bundle { - val init = Input(Bool()) - val next = Input(Bool()) - val elemFire = Input(Bool()) + val init = Input(Bool()) + val next = Input(Bool()) + val elemFire = Input(Bool()) val inRows = Input(UInt(16.W)) val inCols = Input(UInt(16.W)) val kRows = Input(UInt(log2Ceil(maxK + 1).W)) @@ -19,12 +19,12 @@ class Im2colWindow(maxK: Int, iterW: Int) extends Module { val padding = Input(UInt(8.W)) val startRow = Input(UInt(8.W)) val startCol = Input(UInt(8.W)) - val outRow = Output(UInt(16.W)) - val outCol = Output(UInt(16.W)) - val kRowIdx = Output(UInt(log2Ceil(maxK + 1).W)) - val kColIdx = Output(UInt(log2Ceil(maxK + 1).W)) - val elemLast = Output(Bool()) - val last = Output(Bool()) + val outRow = Output(UInt(16.W)) + val outCol = Output(UInt(16.W)) + val kRowIdx = Output(UInt(log2Ceil(maxK + 1).W)) + val kColIdx = Output(UInt(log2Ceil(maxK + 1).W)) + val elemLast = Output(Bool()) + val last = Output(Bool()) }) private val outRowReg = RegInit(0.U(iterW.W)) @@ -34,8 +34,10 @@ class Im2colWindow(maxK: Int, iterW: Int) extends Module { private val paddedRows = io.inRows +& (io.padding << 1) private val paddedCols = io.inCols +& (io.padding << 1) - private val outputRows = ((paddedRows - io.kRows - io.startRow) / io.rowStride) + 1.U - private val outputCols = ((paddedCols - io.kCols - io.startCol) / io.colStride) + 1.U + private val outputRows = + ((paddedRows - io.kRows - io.startRow) / io.rowStride) + 1.U + private val outputCols = + ((paddedCols - io.kCols - io.startCol) / io.colStride) + 1.U private val elemLast = (kRowIdxReg === io.kRows - 1.U) && (kColIdxReg === io.kCols - 1.U) diff --git a/examples/balls/im2col/arch/src/main/scala/LineBufferManager.scala b/examples/balls/im2col/arch/src/main/scala/LineBufferManager.scala index 64c4e9b33..f968241e3 100644 --- a/examples/balls/im2col/arch/src/main/scala/LineBufferManager.scala +++ b/examples/balls/im2col/arch/src/main/scala/LineBufferManager.scala @@ -21,13 +21,15 @@ class LineBufferManager(val b: GlobalConfig) extends Module { private val map = b.ballDomain.ballIdMappings .find(_.ballName == "Im2colBall") - .getOrElse(throw new IllegalArgumentException("Im2colBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("Im2colBall not found in config") + ) private val inBW = map.inBW @public val io = IO(new Bundle { - val bankRead = Vec(inBW, Flipped(new BankRead(b))) - val start = Input(Bool()) + val bankRead = Vec(inBW, Flipped(new BankRead(b))) + val start = Input(Bool()) val inRows = Input(UInt(16.W)) val inCols = Input(UInt(16.W)) val rowStride = Input(UInt(8.W)) @@ -35,20 +37,20 @@ class LineBufferManager(val b: GlobalConfig) extends Module { val padding = Input(UInt(8.W)) val startRow = Input(UInt(8.W)) val startCol = Input(UInt(8.W)) - val outRow = Input(UInt(16.W)) - val outCol = Input(UInt(16.W)) - val kRowIdx = Input(UInt(kW.W)) - val kColIdx = Input(UInt(kW.W)) - val rBankId = Input(UInt(log2Up(b.memDomain.bankNum).W)) - val robId = Input(UInt(log2Up(b.frontend.rob_entries).W)) - val loadDone = Output(Bool()) - val elemData = Output(UInt(elemWidth.W)) + val outRow = Input(UInt(16.W)) + val outCol = Input(UInt(16.W)) + val kRowIdx = Input(UInt(kW.W)) + val kColIdx = Input(UInt(kW.W)) + val rBankId = Input(UInt(log2Up(b.memDomain.bankNum).W)) + val robId = Input(UInt(log2Up(b.frontend.rob_entries).W)) + val loadDone = Output(Bool()) + val elemData = Output(UInt(elemWidth.W)) }) - private val buf = RegInit(VecInit(Seq.fill(maxBeats)(0.U(bankWidth.W)))) - private val active = RegInit(false.B) - private val pending = RegInit(false.B) - private val beat = RegInit(0.U(log2Ceil(maxBeats + 1).W)) + private val buf = RegInit(VecInit(Seq.fill(maxBeats)(0.U(bankWidth.W)))) + private val active = RegInit(false.B) + private val pending = RegInit(false.B) + private val beat = RegInit(0.U(log2Ceil(maxBeats + 1).W)) private val totalBeats = ((io.inRows * io.inCols) + (lanesPerBeat - 1).U) / lanesPerBeat.U @@ -87,15 +89,18 @@ class LineBufferManager(val b: GlobalConfig) extends Module { private val paddedRow = io.startRow + io.outRow * io.rowStride + io.kRowIdx private val paddedCol = io.startCol + io.outCol * io.colStride + io.kColIdx - private val rowValid = paddedRow >= io.padding && paddedRow < io.padding + io.inRows - private val colValid = paddedCol >= io.padding && paddedCol < io.padding + io.inCols + private val rowValid = + paddedRow >= io.padding && paddedRow < io.padding + io.inRows + private val colValid = + paddedCol >= io.padding && paddedCol < io.padding + io.inCols private val inBound = rowValid && colValid private val srcRow = Mux(inBound, paddedRow - io.padding, 0.U) private val srcCol = Mux(inBound, paddedCol - io.padding, 0.U) private val elemIndex = srcRow * io.inCols + srcCol private val beatIndex = elemIndex / lanesPerBeat.U private val laneIndex = elemIndex % lanesPerBeat.U - private val lanes = buf(beatIndex).asTypeOf(Vec(lanesPerBeat, UInt(elemWidth.W))) + private val lanes = + buf(beatIndex).asTypeOf(Vec(lanesPerBeat, UInt(elemWidth.W))) io.elemData := Mux(inBound, lanes(laneIndex), 0.U) } diff --git a/examples/balls/im2col/arch/src/main/scala/README.md b/examples/balls/im2col/arch/src/main/scala/README.md index 9eb078068..c246a8dad 100644 --- a/examples/balls/im2col/arch/src/main/scala/README.md +++ b/examples/balls/im2col/arch/src/main/scala/README.md @@ -22,8 +22,8 @@ The output dimension is: outputDim = floor((iter + 2 * padding - ksize) / stride) + 1 ``` -Windows form the rows of MatrixBall operand A. The kernel elements form its K -dimension. Output is stored in MatrixBall's `(M tile, K tile, tile row, lane)` +Windows form the rows of SMatMulBall operand A. The kernel elements form its K +dimension. Output is stored in SMatMulBall's `(M tile, K tile, tile row, lane)` layout, with both tile dimensions fixed at 16: ```text @@ -62,7 +62,7 @@ inputRow * iter + inputCol - `Im2colConfigRegs.scala`: ISA field decoding and validation. - `Im2colWindow.scala`: output-window and kernel traversal. - `LineBufferManager.scala`: input preload and padded element selection. -- `StreamWriter.scala`: MatrixBall-compatible tiled output packing and bank +- `StreamWriter.scala`: SMatMulBall-compatible tiled output packing and bank writes. - `Im2colBall.scala`: Blink-compatible ball wrapper. - `configs/Im2colBallParam.scala`: maximum dimension and element width. diff --git a/examples/balls/im2col/arch/src/main/scala/StreamWriter.scala b/examples/balls/im2col/arch/src/main/scala/StreamWriter.scala index 19fcadd4e..015c321e4 100644 --- a/examples/balls/im2col/arch/src/main/scala/StreamWriter.scala +++ b/examples/balls/im2col/arch/src/main/scala/StreamWriter.scala @@ -9,7 +9,7 @@ import framework.top.GlobalConfig import examples.balls.im2col.configs.Im2colBallParam /** - * Packs im2col K tiles into 128-bit bank rows (MatrixBall A layout). + * Packs im2col K tiles into 128-bit bank rows (SMatMulBall A layout). * * ((w / 16) * ceil(kernelElems / 16) + kt) * 16 + (w % 16) * @@ -23,21 +23,32 @@ class StreamWriter(val b: GlobalConfig) extends Module { private val maxKSize = ballCfg.maxKSize private val elemWidth = ballCfg.inputWidth private val bankWidth = b.memDomain.bankWidth + private val bankEntries = b.memDomain.bankEntries private val lanesPerBeat = bankWidth / elemWidth private val laneIdxWidth = log2Ceil(lanesPerBeat) private val kW = log2Ceil(maxKSize + 1) - private val addrW = log2Ceil(b.memDomain.bankEntries) - private val maxKTiles = (maxKSize * maxKSize + lanesPerBeat - 1) / lanesPerBeat + private val bankAddrW = log2Ceil(bankEntries) + // Logical row can span multiple physical bank groups (emu: row/bank_lines). + private val logicalAddrW = bankAddrW + log2Ceil(b.memDomain.bankNum) + private val maxKTiles = + (maxKSize * maxKSize + lanesPerBeat - 1) / lanesPerBeat private val kTilesW = log2Ceil(maxKTiles + 1) + require( + isPow2(bankEntries), + "StreamWriter multi-group addr split needs power-of-2 bankEntries" + ) private val mapping = b.ballDomain.ballIdMappings .find(_.ballName == "Im2colBall") - .getOrElse(throw new IllegalArgumentException("Im2colBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("Im2colBall not found in config") + ) + private val outBW = mapping.outBW require(outBW >= 1, "StreamWriter requires at least one write port") require(bankWidth % elemWidth == 0) - require(lanesPerBeat == 16, "Im2col MatrixBall layout requires 16 int8 lanes") + require(lanesPerBeat == 16, "Im2col SMatMulBall layout requires 16 int8 lanes") @public val io = IO(new Bundle { val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) @@ -60,39 +71,54 @@ class StreamWriter(val b: GlobalConfig) extends Module { private val zeros = VecInit(Seq.fill(lanesPerBeat)(0.U(elemWidth.W))) - private val packCntReg = RegInit(0.U(log2Ceil(lanesPerBeat + 1).W)) - private val packReg = RegInit(VecInit(Seq.fill(lanesPerBeat)(0.U(elemWidth.W)))) + private val packCntReg = RegInit(0.U(log2Ceil(lanesPerBeat + 1).W)) + + private val packReg = RegInit( + VecInit(Seq.fill(lanesPerBeat)(0.U(elemWidth.W))) + ) + private val chunkIdxReg = RegInit(0.U(kTilesW.W)) private val wrPendingReg = RegInit(false.B) private val endWindowReg = RegInit(false.B) private val finalWinReg = RegInit(false.B) - private val wAddrReg = RegInit(0.U(addrW.W)) - private val flatBeatReg = RegInit(0.U(addrW.W)) private val padActive = RegInit(false.B) private val padMTile = RegInit(0.U(32.W)) private val padKTiles = RegInit(0.U(kTilesW.W)) private val padK = RegInit(0.U(kTilesW.W)) private val padRow = RegInit(0.U(laneIdxWidth.W)) private val padRow0 = RegInit(0.U(laneIdxWidth.W)) + private val wAddrReg = RegInit(0.U(logicalAddrW.W)) + private val flatBeatReg = RegInit(0.U(logicalAddrW.W)) private val kernelElems = io.kRows * io.kCols + private val kTiles = - ((kernelElems +& (lanesPerBeat - 1).U) >> laneIdxWidth).asTypeOf(UInt(kTilesW.W)) - private val mTile = io.windowIdx >> laneIdxWidth - private val mRow = io.windowIdx(laneIdxWidth - 1, 0) + ((kernelElems +& (lanesPerBeat - 1).U) >> laneIdxWidth) + .asTypeOf(UInt(kTilesW.W)) + + private val mTile = io.windowIdx >> laneIdxWidth + private val mRow = io.windowIdx(laneIdxWidth - 1, 0) private val targetAddr = ((mTile * kTiles + chunkIdxReg) << laneIdxWidth) + mRow - private val padAddr = + private val padAddr = ((padMTile * padKTiles + padK) << laneIdxWidth) + padRow - private val writeFire = io.bankWrite(0).io.req.fire - private val legacyWindowComplete = io.legacy && io.elemIn.fire && io.elemLast && - (packCntReg + 1.U =/= lanesPerBeat.U) && !io.finalWindow + private val writeFire = io.bankWrite(0).io.req.fire + + private val legacyWindowComplete = + io.legacy && io.elemIn.fire && io.elemLast && + (packCntReg + 1.U =/= lanesPerBeat.U) && !io.finalWindow + + private val writeGroup = wAddrReg(logicalAddrW - 1, bankAddrW) + private val writeLocal = wAddrReg(bankAddrW - 1, 0) - io.busy := wrPendingReg || padActive - io.windowComplete := Mux(io.legacy, + io.busy := wrPendingReg || padActive + io.windowComplete := Mux( + io.legacy, (wrPendingReg && endWindowReg && writeFire && !finalWinReg) || legacyWindowComplete, - wrPendingReg && endWindowReg && writeFire && !finalWinReg) - io.opComplete := Mux(io.legacy, + wrPendingReg && endWindowReg && writeFire && !finalWinReg + ) + io.opComplete := Mux( + io.legacy, wrPendingReg && endWindowReg && writeFire && finalWinReg, Mux( padActive, @@ -100,7 +126,8 @@ class StreamWriter(val b: GlobalConfig) extends Module { (padK + 1.U === padKTiles) && (padRow === (lanesPerBeat - 1).U), wrPendingReg && endWindowReg && writeFire && finalWinReg && (mRow === (lanesPerBeat - 1).U) - )) + ) + ) for (i <- 0 until outBW) { io.bankWrite(i).io.req.valid := false.B @@ -108,22 +135,22 @@ class StreamWriter(val b: GlobalConfig) extends Module { io.bankWrite(i).io.req.bits.data := 0.U io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(false.B)) - io.bankWrite(i).io.resp.ready := false.B - io.bankWrite(i).bank_id := 0.U - io.bankWrite(i).rob_id := 0.U - io.bankWrite(i).ball_id := 0.U - io.bankWrite(i).group_id := 0.U + io.bankWrite(i).io.resp.ready := false.B + io.bankWrite(i).bank_id := 0.U + io.bankWrite(i).rob_id := 0.U + io.bankWrite(i).ball_id := 0.U + io.bankWrite(i).group_id := 0.U } io.bankWrite(0).io.req.valid := wrPendingReg - io.bankWrite(0).io.req.bits.addr := wAddrReg + io.bankWrite(0).io.req.bits.addr := writeLocal io.bankWrite(0).io.req.bits.data := Cat(packReg.reverse) io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) - io.bankWrite(0).io.resp.ready := true.B - io.bankWrite(0).bank_id := io.wBankId - io.bankWrite(0).rob_id := io.robId - io.bankWrite(0).group_id := 0.U + io.bankWrite(0).io.resp.ready := true.B + io.bankWrite(0).bank_id := io.wBankId + io.bankWrite(0).rob_id := io.robId + io.bankWrite(0).group_id := writeGroup io.elemIn.ready := !wrPendingReg && !padActive @@ -185,20 +212,27 @@ class StreamWriter(val b: GlobalConfig) extends Module { endWindowReg := false.B finalWinReg := false.B packReg := zeros - wAddrReg := padAddr.asTypeOf(UInt(addrW.W)) + wAddrReg := padAddr.asTypeOf(UInt(logicalAddrW.W)) } when(io.elemIn.fire) { packReg(packCntReg(laneIdxWidth - 1, 0)) := io.elemIn.bits val nextCnt = packCntReg + 1.U packCntReg := nextCnt - val flushLegacy = nextCnt === lanesPerBeat.U || (io.elemLast && io.finalWindow) - when((io.legacy && flushLegacy) || (!io.legacy && - (nextCnt === lanesPerBeat.U || io.elemLast))) { + val flushLegacy = + nextCnt === lanesPerBeat.U || (io.elemLast && io.finalWindow) + when( + (io.legacy && flushLegacy) || (!io.legacy && + (nextCnt === lanesPerBeat.U || io.elemLast)) + ) { wrPendingReg := true.B endWindowReg := io.elemLast finalWinReg := io.elemLast && io.finalWindow - wAddrReg := Mux(io.legacy, flatBeatReg, targetAddr.asTypeOf(UInt(addrW.W))) + wAddrReg := Mux( + io.legacy, + flatBeatReg, + targetAddr.asTypeOf(UInt(logicalAddrW.W)) + ) } } } diff --git a/examples/balls/im2col/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/im2col/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp index 7a5f41d59..995124d02 100644 --- a/examples/balls/im2col/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp +++ b/examples/balls/im2col/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -10,8 +10,8 @@ using namespace mlir; using namespace ::buddy::buckyball; namespace mlir::buddy { -void populateIm2colAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateIm2colBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); } // namespace mlir::buddy namespace { @@ -24,7 +24,8 @@ class BankIm2colPattern : public OpRewritePattern { PatternRewriter &rewriter) const override { rewriter.create(op.getLoc(), op.getInBank(), op.getOutBank(), op.getIter(), op.getKsize(), op.getStride(), - op.getPadding()); + op.getPadding(), op.getStartRowAttr(), + op.getStartColAttr()); rewriter.replaceOp(op, op.getOutBank()); return success(); } @@ -32,7 +33,7 @@ class BankIm2colPattern : public OpRewritePattern { } // namespace -void mlir::buddy::populateIm2colAssignPhysicalBankPatterns( +void mlir::buddy::populateIm2colBallAssignPhysicalBankPatterns( RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { (void)state; patterns.add(patterns.getContext()); diff --git a/examples/balls/im2col/compiler/src/Dialect/Buckyball/Im2colBall.td b/examples/balls/im2col/compiler/src/Dialect/Buckyball/Im2colBall.td index 6b1c52d16..e9594ec06 100644 --- a/examples/balls/im2col/compiler/src/Dialect/Buckyball/Im2colBall.td +++ b/examples/balls/im2col/compiler/src/Dialect/Buckyball/Im2colBall.td @@ -7,23 +7,21 @@ #ifndef IM2COLBALL_OPS #define IM2COLBALL_OPS -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" +include "Buckyball.td" //===----------------------------------------------------------------------===// // Im2colBall operations //===----------------------------------------------------------------------===// -// Memref-level im2col+gemm (like matrix_matmul). Chip bank-ssa decides +// Memref-level im2col+gemm (like smatmul_matmul). Chip bank-ssa decides // f32 quant / Bank* composition; this op carries no quant policy. def Im2colMatmulOp : Buckyball_Op<"im2col_matmul"> { let summary = "Packed-plane im2col then matrix gemm"; let description = [{ - Computes one cin-plane gemm tile: - output[M,16] from im2col(input) x filter, with matrix N = $n (1..16). - `input` is a packed HxW plane (bank rows x 16), `filter` is Kx16 with - K = ksize*ksize (row-padded to a multiple of 16). Stride/padding are - im2col ISA fields; pebble currently only emits stride=1 padding=0. + One cin-plane im2col, then gemm over all N columns of `filter`. + `n` is OC (padded to a multiple of 16, 16..4096). `input` is one packed + HxW plane (bank rows x 16). `filter`/`output` are KxN and MxN. + Asym pad co-design: padding=max(lo,hi), startRow/Col=padding-lo. }]; let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, MemRefRankOf<[AnyType], [2]>:$filter, @@ -32,7 +30,64 @@ def Im2colMatmulOp : Buckyball_Op<"im2col_matmul"> { I64Attr:$ksize, I64Attr:$n, DefaultValuedAttr:$stride, - DefaultValuedAttr:$padding); + DefaultValuedAttr:$padding, + DefaultValuedAttr:$startRow, + DefaultValuedAttr:$startCol, + DefaultValuedAttr:$dwAddr, + DefaultValuedAttr:$dwBytes, + DefaultValuedAttr:$perChannel); + let assemblyFormat = [{$input $filter $output attr-dict `:` + type($input) type($filter) type($output)}]; +} + +// Standard conv: nCin input planes -> one im2col+gemm with K = nCin*K^2. +def Im2colFatMatmulOp : Buckyball_Op<"im2col_fat_matmul"> { + let summary = "Multi-plane im2col then fat gemm over nCin input channels"; + let description = [{ + Stack $nCin input planes as [nCin*inRows, 16] and filters as + [nCin*bRows, n]. Im2col each plane, concatenate on K, then one + (wins x nCin*K^2) @ (nCin*K^2 x n) gemm. $n is OC padded to 16. + }]; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, + MemRefRankOf<[AnyType], [2]>:$filter, + MemRefRankOf<[AnyType], [2]>:$output, + I64Attr:$inSize, + I64Attr:$ksize, + I64Attr:$n, + I64Attr:$nCin, + DefaultValuedAttr:$stride, + DefaultValuedAttr:$padding, + DefaultValuedAttr:$startRow, + DefaultValuedAttr:$startCol, + DefaultValuedAttr:$dwAddr, + DefaultValuedAttr:$dwBytes, + DefaultValuedAttr:$perChannel); + let assemblyFormat = [{$input $filter $output attr-dict `:` + type($input) type($filter) type($output)}]; +} + +// Depthwise: n channels (1..16) share one fat GEMM, not per-channel GEMV. +// Input planes stacked as [n*inRows, 16]; filter is block-diag [n*bRows, 16]. +def Im2colDepthwiseMatmulOp : Buckyball_Op<"im2col_depthwise_matmul"> { + let summary = "Multi-plane im2col then fat depthwise gemm (N=lane)"; + let description = [{ + For channel tile size $n (1..16): + im2col each plane, concatenate on K, then (wins x n*K^2) @ (n*K^2 x n) + with block-diagonal weights. Lane-full GEMM; not GEMV. + }]; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, + MemRefRankOf<[AnyType], [2]>:$filter, + MemRefRankOf<[AnyType], [2]>:$output, + I64Attr:$inSize, + I64Attr:$ksize, + I64Attr:$n, + DefaultValuedAttr:$stride, + DefaultValuedAttr:$padding, + DefaultValuedAttr:$startRow, + DefaultValuedAttr:$startCol, + DefaultValuedAttr:$dwAddr, + DefaultValuedAttr:$dwBytes, + DefaultValuedAttr:$perChannel); let assemblyFormat = [{$input $filter $output attr-dict `:` type($input) type($filter) type($output)}]; } @@ -41,14 +96,15 @@ def Im2colOp : Buckyball_Op<"im2col"> { let summary = "Image-to-column transformation for convolution"; let description = [{ Transforms image data into column format for efficient convolution. - Corresponds to bb_im2col instruction (funct7=48). + Corresponds to bb_im2col instruction (funct7 from Core ballISA). Ball id (BID) is chip-specific via balldomain TOML; do not hardcode. - ISA: bb_im2col(op1_bank_id, wr_bank_id, iter, ksize, stride, padding) - Legal range is defined in examples/balls/im2col/configs/default.toml - (maxIter / maxKSize / maxPadding); out of range is illegal fail-hard. + ISA: bb_im2col_ex(op1, wr, iter, ksize, stride, padding, start_row, start_col) + Legal range is defined in examples/balls/im2col/configs/default.toml. }]; let arguments = (ins I64:$inputBankId, I64:$outputBankId, - I64:$iter, I64:$ksize, I64:$stride, I64:$padding); + I64:$iter, I64:$ksize, I64:$stride, I64:$padding, + DefaultValuedAttr:$startRow, + DefaultValuedAttr:$startCol); let assemblyFormat = "$inputBankId `,` $outputBankId `,` " "$iter `,` $ksize `,` $stride `,` $padding attr-dict `:` " "type($inputBankId)"; @@ -57,7 +113,9 @@ def Im2colOp : Buckyball_Op<"im2col"> { def BankIm2colOp : Buckyball_Op<"bank_im2col"> { let summary = "Im2col using virtual bank handles"; let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$ksize, - I64:$stride, I64:$padding); + I64:$stride, I64:$padding, + DefaultValuedAttr:$startRow, + DefaultValuedAttr:$startCol); let results = (outs I64:$outBankOut); let assemblyFormat = "$inBank $outBank $iter $ksize $stride $padding attr-dict `:` type($inBank) type($outBank) type($iter) type($ksize) type($stride) type($padding)"; diff --git a/examples/balls/im2col/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/im2col/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index 330e4f859..deb57f5e4 100644 --- a/examples/balls/im2col/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/balls/im2col/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -5,6 +5,7 @@ #include "Buckyball/BuckyballOps.h" #include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; @@ -18,6 +19,7 @@ struct Im2colLowering : public ConvertOpToLLVMPattern { LogicalResult matchAndRewrite(Im2colOp op, OpAdaptor adaptor, ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("Im2colBall"); Location loc = op.getLoc(); IntegerType i64 = rewriter.getI64Type(); Value rs1 = rewriter.create( @@ -38,13 +40,78 @@ struct Im2colLowering : public ConvertOpToLLVMPattern { loc, rs2, rewriter.create(loc, adaptor.getPadding(), cstI64(rewriter, loc, 16))); + Value startCol = cstI64(rewriter, loc, op.getStartCol()); + Value startRow = cstI64(rewriter, loc, op.getStartRow()); + rs2 = rewriter.create( + loc, rs2, + rewriter.create(loc, startCol, + cstI64(rewriter, loc, 24))); + rs2 = rewriter.create( + loc, rs2, + rewriter.create(loc, startRow, + cstI64(rewriter, loc, 32))); + + if (stable) { + rewriter.replaceOpWithNewOp(op, rs1, rs2); + return success(); + } + rewriter.replaceOpWithNewOp( + op, rs1, rs2, + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("IM2COL"))); + return success(); + } + +private: + bool stable = false; +}; +struct BankIm2colLowering : public ConvertOpToLLVMPattern { + BankIm2colLowering(LLVMTypeConverter &converter, bool stable) + : ConvertOpToLLVMPattern(converter), stable(stable) {} + + LogicalResult + matchAndRewrite(BankIm2colOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("Im2colBall"); + Location loc = op.getLoc(); + IntegerType i64 = rewriter.getI64Type(); + Value rs1 = rewriter.create( + loc, i64, adaptor.getInBank(), + rewriter.create(loc, adaptor.getOutBank(), + cstI64(rewriter, loc, 20))); + rs1 = rewriter.create( + loc, rs1, + rewriter.create(loc, adaptor.getIter(), + cstI64(rewriter, loc, 30))); + + Value rs2 = adaptor.getKsize(); + rs2 = rewriter.create( + loc, rs2, + rewriter.create(loc, adaptor.getStride(), + cstI64(rewriter, loc, 8))); + rs2 = rewriter.create( + loc, rs2, + rewriter.create(loc, adaptor.getPadding(), + cstI64(rewriter, loc, 16))); + Value startCol = cstI64(rewriter, loc, op.getStartCol()); + Value startRow = cstI64(rewriter, loc, op.getStartRow()); + rs2 = rewriter.create( + loc, rs2, + rewriter.create(loc, startCol, + cstI64(rewriter, loc, 24))); + rs2 = rewriter.create( + loc, rs2, + rewriter.create(loc, startRow, + cstI64(rewriter, loc, 32))); if (stable) { rewriter.replaceOpWithNewOp(op, rs1, rs2); return success(); } - rewriter.replaceOpWithNewOp(op, rs1, rs2, - rewriter.getI32IntegerAttr(48)); + rewriter.replaceOpWithNewOp( + op, rs1, rs2, + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("IM2COL"))); return success(); } @@ -54,18 +121,20 @@ struct Im2colLowering : public ConvertOpToLLVMPattern { } // namespace namespace mlir::buddy::buckyball { -void populateIm2colLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable) { +void populateIm2colBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t, bool) { patterns.add(converter, stable); + patterns.add(converter, stable); } -void configureIm2colLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable) { +void configureIm2colBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { if (stable) target.addLegalOp(); else target.addIllegalOp(); - target.addIllegalOp(); + target.addIllegalOp(); } } // namespace mlir::buddy::buckyball diff --git a/examples/balls/im2col/configs/Im2colBallParam.scala b/examples/balls/im2col/configs/Im2colBallParam.scala index 3599fd09a..82b8b58ae 100644 --- a/examples/balls/im2col/configs/Im2colBallParam.scala +++ b/examples/balls/im2col/configs/Im2colBallParam.scala @@ -7,8 +7,7 @@ case class Im2colBallParam( maxIter: Int, maxKSize: Int, maxPadding: Int, - inputWidth: Int -) + inputWidth: Int) object Im2colBallParam { private val ballName = "Im2colBall" @@ -22,4 +21,5 @@ object Im2colBallParam { inputWidth = BallParamLoader.int(tbl, "inputWidth") ) } + } diff --git a/examples/balls/im2col/configs/default.toml b/examples/balls/im2col/configs/default.toml index f41469aec..e9503f636 100644 --- a/examples/balls/im2col/configs/default.toml +++ b/examples/balls/im2col/configs/default.toml @@ -4,10 +4,11 @@ # iter in 1 ..= maxIter # ksize in 1 ..= maxKSize # covers 1x1 / 3x3 / 5x5 / 7x7 # stride >= 1 -# padding in 0 ..= maxPadding -# paddedSize = iter + 2*padding >= ksize +# padding in 0 ..= maxPadding +# start_row / start_col in 0 ..= padding (asym pad: pad=max(lo,hi), start=pad-lo) +# paddedSize = iter + 2*padding >= ksize + start # op1_bank != wr_bank -# outputDim = floor((iter + 2*padding - ksize) / stride) + 1 +# outputDim = floor((iter + 2*padding - ksize - start) / stride) + 1 # M = outputDim * outputDim # K = ksize * ksize # outputRows = ceil(M/16) * ceil(K/16) * 16 @@ -21,4 +22,4 @@ inputWidth = 8 maxIter = 34 maxKSize = 7 -maxPadding = 3 +maxPadding = 7 diff --git a/examples/balls/im2col/emu/src/48_im2col.rs b/examples/balls/im2col/emu/src/48_im2col.rs deleted file mode 100644 index a53946b2f..000000000 --- a/examples/balls/im2col/emu/src/48_im2col.rs +++ /dev/null @@ -1,273 +0,0 @@ -use super::super::bank::{bank_lines, bank_num}; -use super::decode::{pbank, pbank_group, rs1_b0, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; - -/// Must match examples/balls/im2col/configs/default.toml -const MAX_ITER: usize = 34; -const MAX_KSIZE: usize = 7; -const MAX_PADDING: usize = 3; -const TILE: usize = 16; - -pub struct Im2col; - -#[derive(Clone, Copy)] -enum Shape { - Square { - iter: usize, - ksize: usize, - stride: usize, - padding: usize, - }, - Legacy { - krow: usize, - kcol: usize, - inrow: usize, - incol: usize, - startrow: usize, - startcol: usize, - col_step: usize, - }, -} - -fn decode_shape(xs1: u64, xs2: u64) -> Shape { - let iter = rs1_iter(xs1) as usize; - if iter != 0 { - Shape::Square { - iter, - ksize: (xs2 & 0xff) as usize, - stride: ((xs2 >> 8) & 0xff) as usize, - padding: ((xs2 >> 16) & 0xff) as usize, - } - } else { - // Workloads emitted before the square-window API put the complete - // rectangular shape in rs2 and left BB_ITER clear. - Shape::Legacy { - kcol: (xs2 & 0xff) as usize, - krow: ((xs2 >> 8) & 0xff) as usize, - incol: ((xs2 >> 16) & 0xff) as usize, - inrow: ((xs2 >> 24) & 0xff) as usize, - startcol: ((xs2 >> 32) & 0xff) as usize, - startrow: ((xs2 >> 40) & 0xff) as usize, - col_step: ((xs2 >> 48) & 0xff) as usize, - } - } -} - -impl Instruction for Im2col { - const FUNCT: u32 = 48; - - fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - let op1 = rs1_b0(xs1); - let wr = rs1_b2(xs1); - - if op1 >= bank_num() as u64 || wr >= bank_num() as u64 { - panic!("im2col: invalid bank_id"); - } - if !ctx.cfgs[op1 as usize].allocated || !ctx.cfgs[wr as usize].allocated { - panic!("im2col: bank not allocated"); - } - if op1 == wr { - panic!("im2col: op1 and wr must differ"); - } - - if let Shape::Legacy { - krow, - kcol, - inrow, - incol, - startrow, - startcol, - col_step, - } = decode_shape(xs1, xs2) - { - if krow == 0 || kcol == 0 || inrow == 0 || incol == 0 || col_step == 0 { - panic!("im2col: invalid legacy shape (zero dim)"); - } - if inrow < krow || incol < kcol { - panic!("im2col: kernel larger than input"); - } - let row_end = inrow - krow; - let col_end = incol - kcol; - if startrow > row_end || startcol > col_end { - panic!("im2col: invalid start window"); - } - - let po = pbank(ctx.bank_map, op1); - let pw = pbank(ctx.bank_map, wr); - let mut output = Vec::new(); - for r in startrow..=row_end { - for c in (startcol..=col_end).step_by(col_step) { - for kr in 0..krow { - for kc in 0..kcol { - let src = (r + kr) * incol + c + kc; - output.push( - *ctx.banks[po].get(src).unwrap_or_else(|| { - panic!("im2col: legacy input range src={src}") - }), - ); - } - } - } - } - if output.len() > ctx.banks[pw].len() { - panic!("im2col: legacy output range size={}", output.len()); - } - ctx.banks[pw][..output.len()].copy_from_slice(&output); - return 0; - } - - let Shape::Square { - iter, - ksize, - stride, - padding, - } = decode_shape(xs1, xs2) - else { - unreachable!() - }; - - if iter == 0 || iter > MAX_ITER { - panic!("im2col: iter out of range 1..={MAX_ITER} (got {iter})"); - } - if ksize == 0 || ksize > MAX_KSIZE { - panic!("im2col: ksize out of range 1..={MAX_KSIZE} (got {ksize})"); - } - if stride == 0 { - panic!("im2col: stride must be >= 1"); - } - if padding > MAX_PADDING { - panic!("im2col: padding out of range 0..={MAX_PADDING} (got {padding})"); - } - let padded_size = iter + 2 * padding; - if padded_size < ksize { - panic!("im2col: kernel larger than padded input"); - } - - let output_dim = (padded_size - ksize) / stride + 1; - let windows = output_dim * output_dim; - let kernel_elems = ksize * ksize; - let m_tiles = windows.div_ceil(TILE); - let k_tiles = kernel_elems.div_ceil(TILE); - let output_rows = m_tiles * k_tiles * TILE; - - let groups = ctx.cfgs[wr as usize].cols as usize; - let rows_per_bank = bank_lines(); - let capacity_rows = groups * rows_per_bank; - if output_rows > capacity_rows { - panic!( - "im2col: outputRows={output_rows} exceeds bank capacity={capacity_rows} \ - (groups={groups}, bank_lines={rows_per_bank})" - ); - } - - let po = pbank(ctx.bank_map, op1); - let mut output = vec![0u8; output_rows * TILE]; - - let srcb = &ctx.banks[po]; - let mut window = 0usize; - for output_row in 0..output_dim { - for output_col in 0..output_dim { - for kr in 0..ksize { - for kc in 0..ksize { - let input_row = (output_row * stride + kr) as isize - padding as isize; - let input_col = (output_col * stride + kc) as isize - padding as isize; - let kernel = kr * ksize + kc; - let m_tile = window / TILE; - let m_row = window % TILE; - let k_tile = kernel / TILE; - let lane = kernel % TILE; - let bank_row = (m_tile * k_tiles + k_tile) * TILE + m_row; - let out = bank_row * TILE + lane; - if out >= output.len() { - panic!("im2col: output range out={out}"); - } - if input_row >= 0 - && input_row < iter as isize - && input_col >= 0 - && input_col < iter as isize - { - let src = input_row as usize * iter + input_col as usize; - if src >= srcb.len() { - panic!("im2col: input range src={src}"); - } - output[out] = srcb[src]; - } - } - } - window += 1; - } - } - - for (row, data) in output.chunks_exact(TILE).enumerate() { - let group = row / rows_per_bank; - let local_row = row % rows_per_bank; - if group >= groups { - panic!("im2col: output requires group {group}, allocated groups={groups}"); - } - let pw = pbank_group(ctx.bank_map, wr, group as u64); - let off = local_row * TILE; - ctx.banks[pw][off..off + TILE].copy_from_slice(data); - } - 0 - } - - fn latency(xs1: u64, xs2: u64) -> u64 { - if let Shape::Legacy { - krow, - kcol, - inrow, - incol, - startrow, - startcol, - col_step, - } = decode_shape(xs1, xs2) - { - if krow == 0 || kcol == 0 || inrow < krow || incol < kcol || col_step == 0 { - return 16; - } - let row_end = inrow - krow; - let col_end = incol - kcol; - if startrow > row_end || startcol > col_end { - return 16; - } - let windows = - (row_end - startrow + 1).saturating_mul((col_end - startcol) / col_step + 1); - return windows.saturating_mul(krow).saturating_mul(kcol).max(16) as u64; - } - - let Shape::Square { - iter, - ksize, - stride, - padding, - } = decode_shape(xs1, xs2) - else { - unreachable!() - }; - let iter = iter as u64; - let ksize = ksize as u64; - let stride = stride as u64; - let padding = padding as u64; - - if iter == 0 - || iter > MAX_ITER as u64 - || ksize == 0 - || ksize > MAX_KSIZE as u64 - || stride == 0 - || padding > MAX_PADDING as u64 - { - return 16; - } - let padded_size = iter + 2 * padding; - if padded_size < ksize { - return 16; - } - - let output_dim = (padded_size - ksize) / stride + 1; - output_dim - .saturating_mul(output_dim) - .saturating_mul(ksize) - .saturating_mul(ksize) - .max(16) - } -} diff --git a/examples/balls/im2col/emu/src/im2col.rs b/examples/balls/im2col/emu/src/im2col.rs new file mode 100644 index 000000000..f59adbfe3 --- /dev/null +++ b/examples/balls/im2col/emu/src/im2col.rs @@ -0,0 +1,202 @@ +use super::super::bank::{bank_lines, bank_num}; +use super::decode::{pbank, pbank_group, rs1_b0, rs1_b2, rs1_iter}; +use super::instruction::ExecContext; + +/// Must match examples/balls/im2col/configs/default.toml +const MAX_ITER: usize = 34; +const MAX_KSIZE: usize = 7; +const MAX_PADDING: usize = 7; +const TILE: usize = 16; + +#[derive(Clone, Copy)] +struct Shape { + iter: usize, + ksize: usize, + stride: usize, + padding: usize, + start_row: usize, + start_col: usize, +} + +fn decode_shape(xs1: u64, xs2: u64) -> Shape { + Shape { + iter: rs1_iter(xs1) as usize, + ksize: (xs2 & 0xff) as usize, + stride: ((xs2 >> 8) & 0xff) as usize, + padding: ((xs2 >> 16) & 0xff) as usize, + start_col: ((xs2 >> 24) & 0xff) as usize, + start_row: ((xs2 >> 32) & 0xff) as usize, + } +} + +pub(crate) fn exec_im2col(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { + let op1 = rs1_b0(xs1); + let wr = rs1_b2(xs1); + + if op1 >= bank_num() as u64 || wr >= bank_num() as u64 { + panic!("im2col: invalid bank_id"); + } + if !ctx.cfgs[op1 as usize].allocated || !ctx.cfgs[wr as usize].allocated { + panic!("im2col: bank not allocated"); + } + if op1 == wr { + panic!("im2col: op1 and wr must differ"); + } + + let Shape { + iter, + ksize, + stride, + padding, + start_row, + start_col, + } = decode_shape(xs1, xs2); + + if iter == 0 || iter > MAX_ITER { + panic!("im2col: iter out of range 1..={MAX_ITER} (got {iter})"); + } + if ksize == 0 || ksize > MAX_KSIZE { + panic!("im2col: ksize out of range 1..={MAX_KSIZE} (got {ksize})"); + } + if stride == 0 { + panic!("im2col: stride must be >= 1"); + } + if padding > MAX_PADDING { + panic!("im2col: padding out of range 0..={MAX_PADDING} (got {padding})"); + } + if start_row > padding || start_col > padding { + panic!( + "im2col: start_row/col out of range 0..=padding \ + (start_row={start_row}, start_col={start_col}, padding={padding})" + ); + } + let padded_size = iter + 2 * padding; + if padded_size < ksize + start_row || padded_size < ksize + start_col { + panic!("im2col: kernel+start larger than padded input"); + } + if (padded_size - ksize - start_row) % stride != 0 + || (padded_size - ksize - start_col) % stride != 0 + { + panic!("im2col: padded/ksize/start/stride yield non-integer output"); + } + + let output_rows_n = (padded_size - ksize - start_row) / stride + 1; + let output_cols_n = (padded_size - ksize - start_col) / stride + 1; + if output_rows_n != output_cols_n { + panic!("im2col: non-square output ({output_rows_n}x{output_cols_n}) unsupported"); + } + let output_dim = output_rows_n; + let windows = output_dim * output_dim; + let kernel_elems = ksize * ksize; + let m_tiles = windows.div_ceil(TILE); + let k_tiles = kernel_elems.div_ceil(TILE); + let output_rows = m_tiles * k_tiles * TILE; + + let groups = ctx.cfgs[wr as usize].cols as usize; + let rows_per_bank = bank_lines(); + let capacity_rows = groups * rows_per_bank; + if output_rows > capacity_rows { + panic!( + "im2col: outputRows={output_rows} exceeds bank capacity={capacity_rows} \ + (groups={groups}, bank_lines={rows_per_bank})" + ); + } + + let po = pbank(ctx.bank_map, op1); + let mut output = vec![0u8; output_rows * TILE]; + + let srcb = &ctx.banks[po]; + let mut window = 0usize; + for output_row in 0..output_dim { + for output_col in 0..output_dim { + for kr in 0..ksize { + for kc in 0..ksize { + let input_row = + (start_row + output_row * stride + kr) as isize - padding as isize; + let input_col = + (start_col + output_col * stride + kc) as isize - padding as isize; + let kernel = kr * ksize + kc; + let m_tile = window / TILE; + let m_row = window % TILE; + let k_tile = kernel / TILE; + let lane = kernel % TILE; + let bank_row = (m_tile * k_tiles + k_tile) * TILE + m_row; + let out = bank_row * TILE + lane; + if out >= output.len() { + panic!("im2col: output range out={out}"); + } + if input_row >= 0 + && input_row < iter as isize + && input_col >= 0 + && input_col < iter as isize + { + let src = input_row as usize * iter + input_col as usize; + if src >= srcb.len() { + panic!("im2col: input range src={src}"); + } + output[out] = srcb[src]; + } + } + } + window += 1; + } + } + + for (row, data) in output.chunks_exact(TILE).enumerate() { + let group = row / rows_per_bank; + let local_row = row % rows_per_bank; + if group >= groups { + panic!("im2col: output requires group {group}, allocated groups={groups}"); + } + let pw = pbank_group(ctx.bank_map, wr, group as u64); + let off = local_row * TILE; + ctx.banks[pw][off..off + TILE].copy_from_slice(data); + } + 0 +} + +pub(crate) fn latency(xs1: u64, xs2: u64) -> u64 { + let Shape { + iter, + ksize, + stride, + padding, + start_row, + start_col, + } = decode_shape(xs1, xs2); + let iter = iter as u64; + let ksize = ksize as u64; + let stride = stride as u64; + let padding = padding as u64; + let start_row = start_row as u64; + let start_col = start_col as u64; + + if iter == 0 + || iter > MAX_ITER as u64 + || ksize == 0 + || ksize > MAX_KSIZE as u64 + || stride == 0 + || padding > MAX_PADDING as u64 + || start_row > padding + || start_col > padding + { + return 16; + } + let padded_size = iter + 2 * padding; + if padded_size < ksize + start_row || padded_size < ksize + start_col { + return 16; + } + if (padded_size - ksize - start_row) % stride != 0 + || (padded_size - ksize - start_col) % stride != 0 + { + return 16; + } + + let output_rows_n = (padded_size - ksize - start_row) / stride + 1; + let output_cols_n = (padded_size - ksize - start_col) / stride + 1; + output_rows_n + .saturating_mul(output_cols_n) + .saturating_mul(ksize) + .saturating_mul(ksize) + .max(16) +} diff --git a/examples/balls/im2col/emu/src/lib.rs b/examples/balls/im2col/emu/src/lib.rs index 55c914d92..5265c8f61 100644 --- a/examples/balls/im2col/emu/src/lib.rs +++ b/examples/balls/im2col/emu/src/lib.rs @@ -1,9 +1,8 @@ pub(crate) use crate::inst::{decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::ExecContext; -#[path = "48_im2col.rs"] -mod f48_im2col; +mod im2col; const BALL_CLASS: &str = "examples.balls.im2col.Im2colBall"; @@ -17,11 +16,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("IM2COL") => Some(im2col::exec_im2col(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -29,10 +26,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("IM2COL") => Some(im2col::latency(xs1, xs2)), + Some(_) | None => None, } } diff --git a/examples/balls/im2col/verify/README.md b/examples/balls/im2col/verify/README.md index daf2135b0..1c41baa8b 100644 --- a/examples/balls/im2col/verify/README.md +++ b/examples/balls/im2col/verify/README.md @@ -37,7 +37,7 @@ Blink command fields (not ISA rs packing): - `ksize` = cmd.special[7:0] - `stride` = cmd.special[15:8] - `padding` = cmd.special[23:16] -- banks = op1_bank / wr_bank, funct7 = 48 +- banks = op1_bank / wr_bank; funct7 is injected from the selected Core ballISA - TB: op1_col=1, wr_col=1, rs1=0, rs2=0 ## BID (required) diff --git a/examples/balls/im2col/verify/casegen/src/casegen.rs b/examples/balls/im2col/verify/casegen/src/casegen.rs index 8dee6fcb4..93ee9c4c3 100644 --- a/examples/balls/im2col/verify/casegen/src/casegen.rs +++ b/examples/balls/im2col/verify/casegen/src/casegen.rs @@ -1,6 +1,5 @@ use crate::model; -pub const FUNCT7: u32 = 48; pub const BANK_ROW_BYTES: usize = model::BANK_ROW_BYTES; pub const MAX_WORDS: usize = 128; pub const MAX_OUTPUT_ROWS: usize = 128; @@ -9,7 +8,6 @@ pub const MAX_OUTPUT_ROWS: usize = 128; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct Im2colCmd { pub bid: u32, - pub funct7: u32, pub iter: u32, pub ksize: u32, pub stride: u32, @@ -107,7 +105,6 @@ fn build_case( Im2colCase { cmd: Im2colCmd { bid, - funct7: FUNCT7, iter: iter as u32, ksize: ksize as u32, stride: stride as u32, @@ -129,8 +126,8 @@ fn directed_k3_pad0(bid: u32) -> Im2colCase { let iter = 6usize; let k = 3usize; let flat: [i8; 36] = [ - -1, 2, 3, 4, -5, 6, 0, 8, 9, -10, 11, 12, 13, 0, -15, 16, 17, 18, 19, -20, 0, 22, 23, - -24, -25, 26, 27, 0, -29, 30, 31, 32, 33, -34, 0, 36, + -1, 2, 3, 4, -5, 6, 0, 8, 9, -10, 11, 12, 13, 0, -15, 16, 17, 18, 19, -20, 0, 22, 23, -24, + -25, 26, 27, 0, -29, 30, 31, 32, 33, -34, 0, 36, ]; let bytes: Vec = flat.iter().map(|&x| x as u8).collect(); build_case(bid, iter, k, 1, 0, 0, 1, 3, &bytes) @@ -140,8 +137,8 @@ fn directed_k3_pad1(bid: u32) -> Im2colCase { let iter = 6usize; let k = 3usize; let flat: [i8; 36] = [ - 7, -2, 3, 0, 5, -6, 1, 8, -9, 10, 11, 0, -4, 12, 13, 14, 0, 16, 17, 0, 19, -20, 21, 22, - 0, 24, 25, 26, -27, 28, 29, -30, 0, 32, 33, 34, + 7, -2, 3, 0, 5, -6, 1, 8, -9, 10, 11, 0, -4, 12, 13, 14, 0, 16, 17, 0, 19, -20, 21, 22, 0, + 24, 25, 26, -27, 28, 29, -30, 0, 32, 33, 34, ]; let bytes: Vec = flat.iter().map(|&x| x as u8).collect(); build_case(bid, iter, k, 1, 1, 0, 1, 5, &bytes) @@ -225,7 +222,6 @@ mod tests { #[test] fn directed_k3_pad0_matches_ctest() { let case = gen_case(0, 0, 2); - assert_eq!(case.cmd.funct7, FUNCT7); assert_eq!(case.cmd.iter, 6); assert_eq!(case.cmd.ksize, 3); assert_eq!(case.cmd.stride, 1); diff --git a/examples/balls/im2col/verify/casegen/src/lib.rs b/examples/balls/im2col/verify/casegen/src/lib.rs index 578864fee..e0fff58a0 100644 --- a/examples/balls/im2col/verify/casegen/src/lib.rs +++ b/examples/balls/im2col/verify/casegen/src/lib.rs @@ -79,7 +79,6 @@ mod tests { assert_eq!(im2col_case_load(0x1234, 0, 2), 0); let mut cmd = Im2colCmd { bid: 0, - funct7: 0, iter: 0, ksize: 0, stride: 0, @@ -93,7 +92,6 @@ mod tests { num_dst_words: 0, }; im2col_case_cmd(&mut cmd as *mut Im2colCmd); - assert_eq!(cmd.funct7, 48); assert_eq!(cmd.iter, 6); assert_eq!(cmd.ksize, 3); assert_eq!(cmd.num_src_words, 3); diff --git a/examples/balls/im2col/verify/casegen/src/model.rs b/examples/balls/im2col/verify/casegen/src/model.rs index bd3ec5cad..02f44c372 100644 --- a/examples/balls/im2col/verify/casegen/src/model.rs +++ b/examples/balls/im2col/verify/casegen/src/model.rs @@ -17,13 +17,7 @@ pub fn output_rows(iter: usize, ksize: usize, stride: usize, padding: usize) -> m_tiles * k_tiles * TILE } -pub fn im2col( - src: &[u8], - iter: usize, - ksize: usize, - stride: usize, - padding: usize, -) -> Vec { +pub fn im2col(src: &[u8], iter: usize, ksize: usize, stride: usize, padding: usize) -> Vec { if src.len() != iter * iter { panic!( "im2col model: src len {} != iter*iter {}", diff --git a/examples/balls/im2col/verify/src/common/im2col_defs.svh b/examples/balls/im2col/verify/src/common/im2col_defs.svh index 2dd994205..8a0518cc7 100644 --- a/examples/balls/im2col/verify/src/common/im2col_defs.svh +++ b/examples/balls/im2col/verify/src/common/im2col_defs.svh @@ -1,6 +1,5 @@ typedef struct { int unsigned bid; - int unsigned funct7; int unsigned iter; int unsigned ksize; int unsigned stride; @@ -25,7 +24,10 @@ import "DPI-C" function longint unsigned im2col_case_src_word_hi(input int unsig import "DPI-C" function longint unsigned im2col_case_dst_word_lo(input int unsigned word_index); import "DPI-C" function longint unsigned im2col_case_dst_word_hi(input int unsigned word_index); -localparam int IM2COL_FUNCT7 = 7'd48; +`ifndef IM2COL_FUNCT7 +`error "IM2COL_FUNCT7 must be provided by the selected Core ballISA" +`endif +localparam int IM2COL_CORE_FUNCT7 = `IM2COL_FUNCT7; localparam int IM2COL_MAX_WORDS = 128; localparam int IM2COL_TIMEOUT_CYCLES = 100000; localparam int IM2COL_SEED = 32'hCAFE_BABE; diff --git a/examples/balls/im2col/verify/src/common/im2col_items.svh b/examples/balls/im2col/verify/src/common/im2col_items.svh index 95cdd0c0e..456f2f95e 100644 --- a/examples/balls/im2col/verify/src/common/im2col_items.svh +++ b/examples/balls/im2col/verify/src/common/im2col_items.svh @@ -10,7 +10,7 @@ class im2col_cmd_item extends bb_blink_cmd_item; bit [127:0] dst_words[IM2COL_MAX_WORDS]; constraint legal_c { - funct7 == IM2COL_FUNCT7[6:0]; + funct7 == IM2COL_CORE_FUNCT7[6:0]; op1_en == 1'b1; op2_en == 1'b0; wr_spad_en == 1'b1; @@ -43,7 +43,7 @@ class im2col_cmd_item extends bb_blink_cmd_item; im2col_case_cmd(cmd); bid = cmd.bid[4:0]; - funct7 = cmd.funct7[6:0]; + funct7 = IM2COL_CORE_FUNCT7[6:0]; iter = cmd.iter; ksize = cmd.ksize; stride = cmd.stride; diff --git a/examples/balls/im2col/verify/src/tb_top.sv b/examples/balls/im2col/verify/src/tb_top.sv index a908c8825..d094bee8f 100644 --- a/examples/balls/im2col/verify/src/tb_top.sv +++ b/examples/balls/im2col/verify/src/tb_top.sv @@ -71,9 +71,9 @@ module tb_top; .io_bankWrite_0_io_resp_valid(intf.bank_write_resp_valid[0]), .io_bankWrite_0_io_resp_bits_ok(intf.bank_write_resp_ok[0]), .io_subRobReq_ready(intf.sub_rob_req_ready), - .io_mmioRead_req_ready(intf.mmio_read_req_ready), - .io_mmioRead_resp_valid(intf.mmio_read_resp_valid), - .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data) + .io_mmioRead_req_ready(intf.mmio_read_req_ready[0]), + .io_mmioRead_resp_valid(intf.mmio_read_resp_valid[0]), + .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data[0]) ); initial begin diff --git a/examples/balls/im2col/workloads/ctests/CMakeLists.txt b/examples/balls/im2col/workloads/ctests/CMakeLists.txt index 80306658e..50fd8d07c 100644 --- a/examples/balls/im2col/workloads/ctests/CMakeLists.txt +++ b/examples/balls/im2col/workloads/ctests/CMakeLists.txt @@ -3,6 +3,7 @@ add_buckyball_ctests( im2col_k3_test.c im2col_k3_partial_test.c im2col_k3_pad_test.c + im2col_k3_asym_pad_test.c im2col_k5_test.c im2col_k7_test.c im2col_bank_k1_test.c diff --git a/examples/balls/im2col/workloads/ctests/im2col_bank_k1_test.c b/examples/balls/im2col/workloads/ctests/im2col_bank_k1_test.c index f41e539ef..e19ba5978 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_bank_k1_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_bank_k1_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* Fill exactly one bank: 32x32 k1 -> 1024 output rows. */ diff --git a/examples/balls/im2col/workloads/ctests/im2col_bank_k3_test.c b/examples/balls/im2col/workloads/ctests/im2col_bank_k3_test.c index baaaf2a19..adb733d10 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_bank_k3_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_bank_k3_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* Fill exactly one bank: 34x34 k3 -> 1024 output rows. */ diff --git a/examples/balls/im2col/workloads/ctests/im2col_bank_k5_test.c b/examples/balls/im2col/workloads/ctests/im2col_bank_k5_test.c index d73491a73..f03b3fdde 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_bank_k5_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_bank_k5_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #include diff --git a/examples/balls/im2col/workloads/ctests/im2col_bank_k7_test.c b/examples/balls/im2col/workloads/ctests/im2col_bank_k7_test.c index 28d45a675..471709ea1 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_bank_k7_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_bank_k7_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* Fill exactly one bank: 22x22 k7 -> 1024 output rows. */ diff --git a/examples/balls/im2col/workloads/ctests/im2col_k1_test.c b/examples/balls/im2col/workloads/ctests/im2col_k1_test.c index bf29266de..3d29a99d2 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_k1_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_k1_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include enum { ITER = 7, K = 1, STRIDE = 1, PAD = 0, LANES = 16 }; diff --git a/examples/balls/im2col/workloads/ctests/im2col_k3_asym_pad_test.c b/examples/balls/im2col/workloads/ctests/im2col_k3_asym_pad_test.c new file mode 100644 index 000000000..253ab1685 --- /dev/null +++ b/examples/balls/im2col/workloads/ctests/im2col_k3_asym_pad_test.c @@ -0,0 +1,70 @@ +#include "buckyball.h" +#include +#include +#include +#include + +/* asym pad: padding=max(lo,hi)=2, start=padding-lo=1 (lo=1,hi=2) */ +enum { ITER = 6, K = 3, STRIDE = 1, PAD = 2, START = 1, LANES = 16 }; +enum { + OUT_DIM = (ITER + 2 * PAD - K - START) / STRIDE + 1, + WINDOWS = OUT_DIM * OUT_DIM, + KERNEL = K * K, + M_TILES = (WINDOWS + LANES - 1) / LANES, + K_TILES = (KERNEL + LANES - 1) / LANES, + OUT_ROWS = M_TILES * K_TILES * LANES, +}; + +static elem_t in[ITER * ITER] __attribute__((aligned(64))) = { + 7, -2, 3, 0, 5, -6, 1, 8, -9, 10, 11, 0, -4, 12, 13, 14, 0, 16, + 17, 0, 19, -20, 21, 22, 0, 24, 25, 26, -27, 28, 29, -30, 0, 32, 33, 34, +}; +static elem_t out[OUT_ROWS * LANES] __attribute__((aligned(64))); +static elem_t exp[OUT_ROWS * LANES] __attribute__((aligned(64))); + +static void build_expected(void) { + int w = 0; + clear_i8_matrix(exp, OUT_ROWS, LANES); + for (int orow = 0; orow < OUT_DIM; orow++) { + for (int ocol = 0; ocol < OUT_DIM; ocol++) { + for (int krow = 0; krow < K; krow++) { + for (int kcol = 0; kcol < K; kcol++) { + int ki = krow * K + kcol; + int bank_row = + ((w / LANES) * K_TILES + ki / LANES) * LANES + (w % LANES); + int row = START + orow * STRIDE + krow - PAD; + int col = START + ocol * STRIDE + kcol - PAD; + exp[bank_row * LANES + ki % LANES] = + (row < 0 || row >= ITER || col < 0 || col >= ITER) + ? 0 + : in[row * ITER + col]; + } + } + w++; + } + } +} + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + clear_i8_matrix(out, OUT_ROWS, LANES); + build_expected(); + + bb_mem_alloc(0, 1, 1); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)in, 0, (ITER * ITER + LANES - 1) / LANES, 1); + bb_im2col_ex(0, 1, ITER, K, STRIDE, PAD, START, START); + bb_mvout((uintptr_t)out, 1, OUT_ROWS, 1); + bb_fence(); + bb_mem_release(0); + bb_mem_release(1); + + if (compare_i8_matrices(out, exp, OUT_ROWS, LANES)) { + printf("im2col k3 6x6 asym pad PASSED\n"); + return 0; + } + printf("im2col k3 6x6 asym pad FAILED\n"); + return 1; +} diff --git a/examples/balls/im2col/workloads/ctests/im2col_k3_pad_test.c b/examples/balls/im2col/workloads/ctests/im2col_k3_pad_test.c index 6b0a943e6..fbde7560b 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_k3_pad_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_k3_pad_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* padding path: out-of-bound window samples are zero */ diff --git a/examples/balls/im2col/workloads/ctests/im2col_k3_partial_test.c b/examples/balls/im2col/workloads/ctests/im2col_k3_partial_test.c index cfe0c5df9..1b90521ed 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_k3_partial_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_k3_partial_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* partial M tile: 5x5 k3 -> 9 windows, unused m rows must be 0 */ diff --git a/examples/balls/im2col/workloads/ctests/im2col_k3_test.c b/examples/balls/im2col/workloads/ctests/im2col_k3_test.c index 52b3cde39..6ff3d9bb6 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_k3_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_k3_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* exact M tile: 6x6 k3 -> 16 windows */ diff --git a/examples/balls/im2col/workloads/ctests/im2col_k5_test.c b/examples/balls/im2col/workloads/ctests/im2col_k5_test.c index 9ff0bac6f..7b4c9e9d2 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_k5_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_k5_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include enum { ITER = 7, K = 5, STRIDE = 1, PAD = 0, LANES = 16 }; diff --git a/examples/balls/im2col/workloads/ctests/im2col_k7_test.c b/examples/balls/im2col/workloads/ctests/im2col_k7_test.c index 23452e749..71f594f00 100644 --- a/examples/balls/im2col/workloads/ctests/im2col_k7_test.c +++ b/examples/balls/im2col/workloads/ctests/im2col_k7_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include enum { ITER = 7, K = 7, STRIDE = 1, PAD = 0, LANES = 16 }; diff --git a/examples/balls/im2col/workloads/isa/im2col.h b/examples/balls/im2col/workloads/isa/im2col.h new file mode 100644 index 000000000..94e1a7d64 --- /dev/null +++ b/examples/balls/im2col/workloads/isa/im2col.h @@ -0,0 +1,27 @@ +#ifndef _BB_IM2COL_H_ +#define _BB_IM2COL_H_ + +#include +#include + +// Square-window im2col (iter != 0): +// xs2[7:0] ksize +// xs2[15:8] stride +// xs2[23:16] padding (symmetric zero-pad on every edge) +// xs2[31:24] start_col (skip into padded space; asym pad co-design) +// xs2[39:32] start_row +// +// outputDim = floor((iter + 2*padding - ksize - start) / stride) + 1 +// Asym pad (lo, hi): padding=max(lo,hi), start=padding-lo. +#define bb_im2col(op1_bank_id, wr_bank_id, iter, ksize, stride, padding) \ + bb_im2col_ex(op1_bank_id, wr_bank_id, iter, ksize, stride, padding, 0, 0) + +#define bb_im2col_ex(op1_bank_id, wr_bank_id, iter, ksize, stride, padding, \ + start_row, start_col) \ + BUCKYBALL_INSTRUCTION_R_R( \ + (BB_BANK0(op1_bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ + (FIELD(ksize, 0, 7) | FIELD(stride, 8, 15) | FIELD(padding, 16, 23) | \ + FIELD(start_col, 24, 31) | FIELD(start_row, 32, 39)), \ + BB_FUNC7(IM2COL)) + +#endif // _BB_IM2COL_H_ diff --git a/examples/balls/im2col/workloads/mlir_tests/ball/CMakeLists.txt b/examples/balls/im2col/workloads/mlir_tests/ball/CMakeLists.txt index 24372ef8a..831103cf8 100644 --- a/examples/balls/im2col/workloads/mlir_tests/ball/CMakeLists.txt +++ b/examples/balls/im2col/workloads/mlir_tests/ball/CMakeLists.txt @@ -1,12 +1,11 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX ball) add_buckyball_mlir_contract(im2col_6x6_k3_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} PASSES - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} ${BUCKYBALL_LOWER_BUCKYBALL}) -add_buckyball_mlir_optest(im2col_6x6_k3) +add_buckyball_mlir_test(im2col_6x6_k3 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3.mlir b/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3.mlir index 2b3feed24..079b3d40f 100644 --- a/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3.mlir +++ b/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3.mlir @@ -10,8 +10,8 @@ func.func @main() -> i8 { %c6 = arith.constant 6 : index %c16 = arith.constant 16 : index - %input = memref.alloc() {alignment = 64 : i64} : memref<3x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<3x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> linalg.fill ins(%zero_i8 : i8) outs(%input : memref<3x16xi8>) linalg.fill ins(%zero_i8 : i8) outs(%output : memref<16x16xi8>) diff --git a/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3_contract.mlir b/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3_contract.mlir index b1c9356da..c60e2273a 100644 --- a/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3_contract.mlir +++ b/examples/balls/im2col/workloads/mlir_tests/ball/im2col_6x6_k3_contract.mlir @@ -1,5 +1,5 @@ // CHECK: buckyball.intr.custom -// CHECK-SAME: funct7 = 48 +// CHECK-SAME: funct7 = 48 : i32 // CHECK-NOT: buckyball.im2col // CHECK-NOT: buckyball.bank_im2col diff --git a/examples/balls/im2col/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/im2col/workloads/mlir_tests/bank/CMakeLists.txt index 7fe896811..a65f434d7 100644 --- a/examples/balls/im2col/workloads/mlir_tests/bank/CMakeLists.txt +++ b/examples/balls/im2col/workloads/mlir_tests/bank/CMakeLists.txt @@ -1,9 +1,8 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX bank) add_buckyball_mlir_contract(im2col_6x6_k3_contract - PASSES "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}") + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) -add_buckyball_mlir_optest(im2col_6x6_k3) +add_buckyball_mlir_test(im2col_6x6_k3 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/im2col/workloads/mlir_tests/bank/im2col_6x6_k3.mlir b/examples/balls/im2col/workloads/mlir_tests/bank/im2col_6x6_k3.mlir index b24d2353f..2c0ea5221 100644 --- a/examples/balls/im2col/workloads/mlir_tests/bank/im2col_6x6_k3.mlir +++ b/examples/balls/im2col/workloads/mlir_tests/bank/im2col_6x6_k3.mlir @@ -10,8 +10,8 @@ func.func @main() -> i8 { %c6 = arith.constant 6 : index %c16 = arith.constant 16 : index - %input = memref.alloc() {alignment = 64 : i64} : memref<3x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<3x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> linalg.fill ins(%zero_i8 : i8) outs(%input : memref<3x16xi8>) linalg.fill ins(%zero_i8 : i8) outs(%output : memref<16x16xi8>) diff --git a/examples/balls/int2fp/arch/src/main/scala/Int2Fp.scala b/examples/balls/int2fp/arch/src/main/scala/Int2Fp.scala index 334157f91..9b3070358 100644 --- a/examples/balls/int2fp/arch/src/main/scala/Int2Fp.scala +++ b/examples/balls/int2fp/arch/src/main/scala/Int2Fp.scala @@ -6,21 +6,35 @@ import chisel3.experimental.hierarchy.{instantiable, public} import framework.balldomain.rs.{BallRsComplete, BallRsIssue} import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} +import framework.balldomain.blink.mmio.MmioRead import framework.top.GlobalConfig +/** INT32 accumulator dequantization: FP32 = INT32 * Da * Dw. */ @instantiable class Int2Fp(val b: GlobalConfig) extends Module { val bankWidth = b.memDomain.bankWidth require(bankWidth == 128, s"Int2FpBall requires bankWidth = 128, got $bankWidth") val elemsPerWord = bankWidth / 32 - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "Int2FpBall") + val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "Int2FpBall") .getOrElse(throw new IllegalArgumentException("Int2FpBall not found in config")) - val inBW = ballMapping.inBW - val outBW = ballMapping.outBW - require(inBW >= 1, "Int2Fp requires at least one read port") - require(outBW >= 1, "Int2Fp requires at least one write port") + val inBW = ballMapping.inBW + val outBW = ballMapping.outBW + val mmioReadBW = ballMapping.mmioReadBW + require(inBW >= 1 && outBW >= 1, "Int2Fp requires bank read/write lines") + require(mmioReadBW == 4, "Int2Fp requires four MMIO byte read lines") + + val tensorFunct = b.ballDomain.ballISA + .find(_.mnemonic == "INT2FP_TENSOR") + .map(_.funct7) + .getOrElse(throw new IllegalArgumentException("INT2FP_TENSOR not found in ballISA")) + + val channelFunct = b.ballDomain.ballISA + .find(_.mnemonic == "INT2FP_CHANNEL") + .map(_.funct7) + .getOrElse(throw new IllegalArgumentException("INT2FP_CHANNEL not found in ballISA")) @public val io = IO(new Bundle { @@ -28,36 +42,37 @@ class Int2Fp(val b: GlobalConfig) extends Module { val cmdResp = Decoupled(new BallRsComplete(b)) val bankRead = Vec(inBW, Flipped(new BankRead(b))) val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) + val mmioRead = Vec(mmioReadBW, Flipped(new MmioRead(b))) val status = new BallStatus }) - val idle :: sReadReq :: sReadResp :: sWriteReq :: sWriteResp :: complete :: Nil = Enum(6) - val state = RegInit(idle) - - val robIdReg = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) - val isSubReg = RegInit(false.B) - val subRobIdReg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) - - val rbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val wbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val iterReg = RegInit(0.U(b.frontend.iter_len.W)) - val scaleReg = RegInit(0.U(32.W)) - - val rowReg = RegInit(0.U(b.frontend.iter_len.W)) - val groupReg = RegInit(0.U(2.W)) - val lastGroupReg = RegInit(0.U(2.W)) - val modeI8ToFp = RegInit(false.B) - val modeI32Group = RegInit(false.B) - val modeI32ToI8 = RegInit(false.B) - val srcWord = RegInit(0.U(bankWidth.W)) - val outWord = RegInit(0.U(bankWidth.W)) - val writeWord = RegInit(0.U(bankWidth.W)) + val idle :: sActReq :: sActResp :: sTensorReq :: sTensorResp :: sReadReq :: sReadResp :: sChannelReq :: sChannelResp :: sWriteReq :: sWriteResp :: complete :: Nil = + Enum(12) + val state = RegInit(idle) + + val robIdReg = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) + val isSubReg = RegInit(false.B) + val subRobIdReg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) + val rbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + val wbankReg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + val iterReg = RegInit(0.U(b.frontend.iter_len.W)) + val actAddrReg = RegInit(0.U(log2Ceil(b.memDomain.mmioTotalBytes).W)) + val weightAddrReg = RegInit(0.U(log2Ceil(b.memDomain.mmioTotalBytes).W)) + val perChannelReg = RegInit(false.B) + val daReg = RegInit(0.U(32.W)) + val tensorDwReg = RegInit(0.U(32.W)) + val rowReg = RegInit(0.U(b.frontend.iter_len.W)) + val groupReg = RegInit(0.U(2.W)) + val lastGroupReg = RegInit(0.U(2.W)) + val laneReg = RegInit(0.U(2.W)) + val srcWordReg = RegInit(0.U(bankWidth.W)) + val writeWordReg = RegInit(0.U(bankWidth.W)) for (i <- 0 until inBW) { io.bankRead(i).rob_id := robIdReg io.bankRead(i).ball_id := 0.U io.bankRead(i).bank_id := rbankReg - io.bankRead(i).group_id := 0.U + io.bankRead(i).group_id := groupReg io.bankRead(i).io.req.valid := false.B io.bankRead(i).io.req.bits.addr := 0.U io.bankRead(i).io.resp.ready := false.B @@ -66,13 +81,20 @@ class Int2Fp(val b: GlobalConfig) extends Module { io.bankWrite(i).rob_id := robIdReg io.bankWrite(i).ball_id := 0.U io.bankWrite(i).bank_id := wbankReg - io.bankWrite(i).group_id := 0.U + io.bankWrite(i).group_id := groupReg io.bankWrite(i).io.req.valid := false.B io.bankWrite(i).io.req.bits.addr := 0.U io.bankWrite(i).io.req.bits.data := 0.U io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(false.B)) io.bankWrite(i).io.resp.ready := false.B } + for (i <- 0 until mmioReadBW) { + io.mmioRead(i).rob_id := robIdReg + io.mmioRead(i).ball_id := 0.U + io.mmioRead(i).req.valid := false.B + io.mmioRead(i).req.bits.addr := 0.U + io.mmioRead(i).resp.ready := false.B + } io.cmdReq.ready := state === idle io.cmdResp.valid := state === complete @@ -80,275 +102,171 @@ class Int2Fp(val b: GlobalConfig) extends Module { io.cmdResp.bits.is_sub := isSubReg io.cmdResp.bits.sub_rob_id := subRobIdReg - def int32ToFp32(intVal: UInt): UInt = { - val signed = intVal.asSInt - val isZero = signed === 0.S - val sign = intVal(31) - val absVal = Wire(UInt(32.W)) - absVal := Mux(sign.asBool, ~intVal + 1.U, intVal) - - val leadingOne = 31.U - PriorityEncoder(Reverse(absVal)) - val exponent = Wire(UInt(9.W)) - val significand = Wire(UInt(24.W)) - - exponent := leadingOne +& 127.U - when(leadingOne > 23.U) { - val rightShift = leadingOne - 23.U - val absWide = Cat(0.U(32.W), absVal) - val truncated = absWide >> rightShift - val half = 1.U(64.W) << (rightShift - 1.U) - val remainder = absWide & ((1.U(64.W) << rightShift) - 1.U) - val roundUp = remainder > half || (remainder === half && truncated(0)) - val rounded = truncated(23, 0) +& roundUp.asUInt - significand := Mux(rounded(24), rounded(24, 1), rounded(23, 0)) - when(rounded(24)) { - exponent := leadingOne +& 128.U - } - }.otherwise { - significand := (absVal << (23.U - leadingOne))(23, 0) - } - - val result = Wire(UInt(32.W)) - when(isZero) { - result := 0.U - }.otherwise { - result := Cat(sign, exponent(7, 0), significand(22, 0)) - } - result - } - def fp32Multiply(a: UInt, bv: UInt): UInt = { - val aSign = a(31) - val bSign = bv(31) - val aExp = a(30, 23) - val bExp = bv(30, 23) - val aMant = Cat(1.U(1.W), a(22, 0)) - val bMant = Cat(1.U(1.W), bv(22, 0)) - val resSign = aSign ^ bSign - val aZero = aExp === 0.U && a(22, 0) === 0.U - val bZero = bExp === 0.U && bv(22, 0) === 0.U - val prod = (aMant * bMant)(47, 0) - val sig = Wire(UInt(24.W)) - val guard = Wire(Bool()) - val round = Wire(Bool()) - val sticky = Wire(Bool()) - val normAdjust = Wire(UInt(2.W)) - when(prod(47)) { - sig := prod(47, 24) - guard := prod(23) - round := prod(22) - sticky := prod(21, 0).orR - normAdjust := 1.U - }.otherwise { - sig := prod(46, 23) - guard := prod(22) - round := prod(21) - sticky := prod(20, 0).orR - normAdjust := 0.U - } - val inc = guard && (round || sticky || sig(0)) - val rounded = sig +& inc.asUInt - val finalSig = Mux(rounded(24), rounded(24, 1), rounded(23, 0)) - val expWide = aExp +& bExp +& normAdjust +& rounded(24) - 127.U - val result = Wire(UInt(32.W)) - when(aZero || bZero) { - result := 0.U - }.elsewhen(expWide(9, 8) =/= 0.U && expWide(9)) { - result := 0.U - }.elsewhen(expWide(8) && !expWide(9)) { - result := Cat(resSign, 255.U(8.W), 0.U(23.W)) - }.otherwise { - result := Cat(resSign, expWide(7, 0), finalSig(22, 0)) - } + val aSign = a(31); val bSign = bv(31) + val aExp = a(30, 23); val bExp = bv(30, 23) + val aMant = Cat(1.U(1.W), a(22, 0)); val bMant = Cat(1.U(1.W), bv(22, 0)) + val prod = (aMant * bMant)(47, 0) + val sig = Wire(UInt(24.W)); val guard = Wire(Bool()); val round = Wire(Bool()); val sticky = Wire(Bool()); + val norm = Wire(UInt(2.W)) + when(prod(47)) { sig := prod(47, 24); guard := prod(23); round := prod(22); sticky := prod(21, 0).orR; norm := 1.U } + .otherwise { sig := prod(46, 23); guard := prod(22); round := prod(21); sticky := prod(20, 0).orR; norm := 0.U } + val rounded = sig +& (guard && (round || sticky || sig(0))).asUInt + val exp = aExp +& bExp +& norm +& rounded(24) - 127.U + val result = Wire(UInt(32.W)) + when((aExp === 0.U && a(22, 0) === 0.U) || (bExp === 0.U && bv(22, 0) === 0.U))(result := 0.U) + .elsewhen(exp(9))(result := 0.U) + .elsewhen(exp(8))(result := Cat(aSign ^ bSign, 255.U(8.W), 0.U(23.W))) + .otherwise(result := Cat(aSign ^ bSign, exp(7, 0), Mux(rounded(24), rounded(23, 1), rounded(22, 0)))) result } - def scaledI32Word(data: UInt): UInt = { - val out = Wire(Vec(elemsPerWord, UInt(32.W))) - for (i <- 0 until elemsPerWord) { - val elem = data((i + 1) * 32 - 1, i * 32) - out(i) := fp32Multiply(int32ToFp32(elem), scaleReg) - } - Cat(out.reverse) - } - - def scaledI8Word(data: UInt, group: UInt): UInt = { - val out = Wire(Vec(elemsPerWord, UInt(32.W))) - for (i <- 0 until elemsPerWord) { - val idx = group * 4.U + i.U - val byte = (data >> (idx << 3.U))(7, 0) - val sx = Cat(Fill(24, byte(7)), byte) - out(i) := fp32Multiply(int32ToFp32(sx), scaleReg) - } - Cat(out.reverse) + def int32ToFp32(value: UInt): UInt = { + val sign = value(31) + val abs = Mux(sign, (~value).asUInt + 1.U, value) + val zero = abs === 0.U + val leading = 31.U - PriorityEncoder(Reverse(abs)) + val shift = Mux(leading > 23.U, leading - 23.U, 0.U) + val truncated = abs >> shift + val half = Mux(shift === 0.U, 0.U(32.W), 1.U(32.W) << (shift - 1.U)) + val remainder = Mux(shift === 0.U, 0.U(32.W), abs & ((1.U(32.W) << shift) - 1.U)) + val roundUp = remainder > half || (remainder === half && truncated(0)) + val rounded = truncated +& roundUp.asUInt + val carry = rounded(24) + val significand = Mux(leading > 23.U, rounded(22, 0), (abs << (23.U - leading))(22, 0)) + val exponent = (leading +& 127.U +& carry)(7, 0) + Mux(zero, 0.U, Cat(sign, exponent, Mux(carry, 0.U(23.W), significand))) } - def fp32ToInt32(fp: UInt): UInt = { - val sign = fp(31) - val exponent = fp(30, 23) - val fraction = fp(22, 0) - val mantissa = Mux(exponent === 0.U, Cat(0.U(1.W), fraction), Cat(1.U(1.W), fraction)) - val mantissaWide = Cat(0.U(40.W), mantissa) - val expVal = exponent.zext - 127.S - val magnitude = Wire(UInt(64.W)) + def scaleResponse: UInt = Cat(io.mmioRead.reverse.map(_.resp.bits.data)) + def isFinitePositiveScale(bits: UInt): Bool = + !bits(31) && bits(30, 23) =/= 255.U && bits(30, 0) =/= 0.U + def allMmioReqFire: Bool = io.mmioRead.map(_.req.fire).reduceLeft((x: Bool, y: Bool) => x && y) + def allMmioRespFire: Bool = io.mmioRead.map(_.resp.fire).reduceLeft((x: Bool, y: Bool) => x && y) - magnitude := 0.U - when(expVal >= 31.S) { - magnitude := "h80000000".U - }.elsewhen(expVal >= 23.S) { - magnitude := mantissaWide << (expVal - 23.S).asUInt - }.elsewhen(expVal >= -1.S) { - val rightShift = (23.S - expVal).asUInt - val truncated = mantissaWide >> rightShift - val half = 1.U(64.W) << (rightShift - 1.U) - val remainder = mantissaWide & ((1.U(64.W) << rightShift) - 1.U) - val roundUp = remainder > half || (remainder === half && truncated(0)) - magnitude := truncated + roundUp.asUInt - } - - val result = Wire(SInt(32.W)) - when(exponent === 255.U && fraction =/= 0.U) { - result := 0.S - }.elsewhen(sign.asBool) { - result := Mux( - magnitude >= "h80000000".U, - -2147483648L.S(32.W), - -magnitude(31, 0).asSInt - ) - }.otherwise { - result := Mux(magnitude > "h7fffffff".U, 2147483647.S(32.W), magnitude(31, 0).asSInt) - } - result.asUInt + def driveScaleReq(addr: UInt): Unit = for (i <- 0 until mmioReadBW) { + io.mmioRead(i).req.valid := true.B + io.mmioRead(i).req.bits.addr := addr + i.U } - def requantI32Bytes(data: UInt): UInt = { - val out = Wire(Vec(elemsPerWord, UInt(8.W))) - for (i <- 0 until elemsPerWord) { - val elem = data((i + 1) * 32 - 1, i * 32) - val scaled = fp32Multiply(int32ToFp32(elem), scaleReg) - val value = fp32ToInt32(scaled).asSInt - out(i) := Mux(value > 127.S, 127.S(8.W), Mux(value < -128.S, -128.S(8.W), value(7, 0).asSInt)).asUInt - } - Cat(out.reverse) + def driveScaleResp(): Unit = for (i <- 0 until mmioReadBW) io.mmioRead(i).resp.ready := true.B + + def dequantLane(word: UInt, lane: UInt, dw: UInt): UInt = { + val acc = (word >> (lane << 5.U))(31, 0) + fp32Multiply(int32ToFp32(acc), fp32Multiply(daReg, dw)) } switch(state) { is(idle) { when(io.cmdReq.fire) { - val srcCol = io.cmdReq.bits.cmd.op1_col - val dstCol = io.cmdReq.bits.cmd.wr_col - val outputMode = io.cmdReq.bits.cmd.special(33, 32) - val outputFp32 = outputMode === 0.U - val outputInt8 = outputMode === 1.U - val isI32Single = outputFp32 && srcCol === 1.U && dstCol === 1.U - val isI8ToFp = outputFp32 && srcCol === 1.U && dstCol === 4.U - val isI32Group = outputFp32 && srcCol === dstCol && (srcCol === 2.U || srcCol === 4.U) - val isI32ToInt8 = outputInt8 && srcCol === 4.U && dstCol === 1.U - assert(io.cmdReq.bits.cmd.iter > 0.U, "Int2Fp iter must be > 0") - assert(outputMode <= 1.U, "Int2Fp reserved output mode") - assert(isI32Single || isI8ToFp || isI32Group || isI32ToInt8, "Int2Fp unsupported mode/layout") - - robIdReg := io.cmdReq.bits.rob_id - isSubReg := io.cmdReq.bits.is_sub - subRobIdReg := io.cmdReq.bits.sub_rob_id - rbankReg := io.cmdReq.bits.cmd.op1_bank - wbankReg := io.cmdReq.bits.cmd.wr_bank - iterReg := io.cmdReq.bits.cmd.iter - scaleReg := io.cmdReq.bits.cmd.special(31, 0) - rowReg := 0.U - groupReg := 0.U - lastGroupReg := srcCol - 1.U - modeI8ToFp := isI8ToFp - modeI32Group := isI32Group - modeI32ToI8 := isI32ToInt8 - srcWord := 0.U - outWord := 0.U - writeWord := 0.U - state := sReadReq + assert( + io.cmdReq.bits.cmd.op1_col === io.cmdReq.bits.cmd.wr_col && io.cmdReq.bits.cmd.op1_col >= 1.U && io.cmdReq.bits.cmd.op1_col <= 4.U, + "Int2Fp requires matching 1..4 accumulator/output groups" + ) + assert(io.cmdReq.bits.cmd.op1_bank =/= io.cmdReq.bits.cmd.wr_bank, "Int2Fp forbids in-place dequantization") + assert( + io.cmdReq.bits.cmd.funct7 === tensorFunct.U(7.W) || + io.cmdReq.bits.cmd.funct7 === channelFunct.U(7.W), + "Int2Fp funct7 must be INT2FP_TENSOR or INT2FP_CHANNEL" + ) + assert(io.cmdReq.bits.cmd.special(63, 26) === 0.U, "Int2Fp reserves special[63:26]") + assert(io.cmdReq.bits.cmd.special(12, 0) === 0.U, "Int2Fp Da address must be 0") + assert(io.cmdReq.bits.cmd.special(25, 13) >= 16.U, "Int2Fp Dw address must be >= 16") + assert(io.cmdReq.bits.cmd.special(1, 0) === 0.U, "Int2Fp Da address must be 4-byte aligned") + assert(io.cmdReq.bits.cmd.special(14, 13) === 0.U, "Int2Fp Dw address must be 4-byte aligned") + assert( + (io.cmdReq.bits.cmd.special(12, 0) +& 3.U) < b.memDomain.mmioTotalBytes.U, + "Int2Fp Da address exceeds MMIO space" + ) + assert( + (io.cmdReq.bits.cmd.special(25, 13) +& 3.U) < b.memDomain.mmioTotalBytes.U, + "Int2Fp Dw address exceeds MMIO space" + ) + when(io.cmdReq.bits.cmd.funct7 === channelFunct.U(7.W)) { + assert( + (io.cmdReq.bits.cmd.special(25, 13) +& (io.cmdReq.bits.cmd.op1_col << 4.U)) <= b.memDomain.mmioTotalBytes.U, + "Int2Fp channel Dw range exceeds MMIO space" + ) + } + robIdReg := io.cmdReq.bits.rob_id; isSubReg := io.cmdReq.bits.is_sub; subRobIdReg := io.cmdReq.bits.sub_rob_id + rbankReg := io.cmdReq.bits.cmd.op1_bank; wbankReg := io.cmdReq.bits.cmd.wr_bank; + iterReg := io.cmdReq.bits.cmd.iter + actAddrReg := io.cmdReq.bits.cmd.special(12, 0); weightAddrReg := io.cmdReq.bits.cmd.special(25, 13) + perChannelReg := io.cmdReq.bits.cmd.funct7 === channelFunct.U(7.W); rowReg := 0.U; groupReg := 0.U; + laneReg := 0.U + lastGroupReg := io.cmdReq.bits.cmd.op1_col - 1.U; writeWordReg := 0.U + state := sActReq } } - - is(sReadReq) { - io.bankRead(0).bank_id := rbankReg - io.bankRead(0).group_id := Mux(modeI32Group || modeI32ToI8, groupReg, 0.U) - io.bankRead(0).io.req.valid := true.B - io.bankRead(0).io.req.bits.addr := rowReg - when(io.bankRead(0).io.req.fire) { - state := sReadResp + is(sActReq) { driveScaleReq(actAddrReg); when(allMmioReqFire)(state := sActResp) } + is(sActResp) { + driveScaleResp() + when(allMmioRespFire) { + assert(isFinitePositiveScale(scaleResponse), "Int2Fp Da must be finite and positive") + daReg := scaleResponse + state := sTensorReq } } - + is(sTensorReq) { + when(perChannelReg)(state := sReadReq) + .otherwise { driveScaleReq(weightAddrReg); when(allMmioReqFire)(state := sTensorResp) } + } + is(sTensorResp) { + driveScaleResp() + when(allMmioRespFire) { + assert(isFinitePositiveScale(scaleResponse), "Int2Fp Dw must be finite and positive") + tensorDwReg := scaleResponse + state := sReadReq + } + } + is(sReadReq) { + io.bankRead(0).io.req.valid := true.B; io.bankRead(0).io.req.bits.addr := rowReg + when(io.bankRead(0).io.req.fire)(state := sReadResp) + } is(sReadResp) { - io.bankRead(0).bank_id := rbankReg - io.bankRead(0).group_id := Mux(modeI32Group || modeI32ToI8, groupReg, 0.U) io.bankRead(0).io.resp.ready := true.B when(io.bankRead(0).io.resp.fire) { - when(modeI8ToFp) { - srcWord := io.bankRead(0).io.resp.bits.data - writeWord := scaledI8Word(io.bankRead(0).io.resp.bits.data, 0.U) - groupReg := 0.U - state := sWriteReq - }.elsewhen(modeI32ToI8) { - val bytes = requantI32Bytes(io.bankRead(0).io.resp.bits.data) - val nextWord = outWord | (bytes << (groupReg << 5.U)) - when(groupReg === 3.U) { - writeWord := nextWord - groupReg := 0.U - outWord := 0.U - state := sWriteReq - }.otherwise { - outWord := nextWord - groupReg := groupReg + 1.U - state := sReadReq + srcWordReg := io.bankRead(0).io.resp.bits.data; laneReg := 0.U; writeWordReg := 0.U + when(perChannelReg)(state := sChannelReq) + .otherwise { + val result = + Cat((0 until elemsPerWord).reverse.map(i => dequantLane(io.bankRead(0).io.resp.bits.data, i.U, tensorDwReg))) + writeWordReg := result + state := sWriteReq } - }.otherwise { - writeWord := scaledI32Word(io.bankRead(0).io.resp.bits.data) - state := sWriteReq - } } } - - is(sWriteReq) { - io.bankWrite(0).bank_id := wbankReg - io.bankWrite(0).group_id := Mux(modeI8ToFp || modeI32Group, groupReg, 0.U) - io.bankWrite(0).io.req.valid := true.B - io.bankWrite(0).io.req.bits.addr := rowReg - io.bankWrite(0).io.req.bits.data := writeWord - io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) - when(io.bankWrite(0).io.req.fire) { - state := sWriteResp + is(sChannelReq) { + val channelAddr = weightAddrReg + ((groupReg * 4.U + laneReg) << 2.U) + assert(channelAddr +& 3.U < b.memDomain.mmioTotalBytes.U, "Int2Fp channel Dw address exceeds MMIO space") + driveScaleReq(channelAddr) + when(allMmioReqFire)(state := sChannelResp) + } + is(sChannelResp) { + driveScaleResp() + when(allMmioRespFire) { + assert(isFinitePositiveScale(scaleResponse), "Int2Fp Dw must be finite and positive") + val next = writeWordReg | (dequantLane(srcWordReg, laneReg, scaleResponse) << (laneReg << 5.U)) + writeWordReg := next + when(laneReg === 3.U)(state := sWriteReq).otherwise { laneReg := laneReg + 1.U; state := sChannelReq } } } - + is(sWriteReq) { + io.bankWrite(0).io.req.valid := true.B; io.bankWrite(0).io.req.bits.addr := rowReg + io.bankWrite(0).io.req.bits.data := writeWordReg; + io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) + when(io.bankWrite(0).io.req.fire)(state := sWriteResp) + } is(sWriteResp) { - io.bankWrite(0).bank_id := wbankReg - io.bankWrite(0).group_id := Mux(modeI8ToFp || modeI32Group, groupReg, 0.U) io.bankWrite(0).io.resp.ready := true.B when(io.bankWrite(0).io.resp.fire) { - when(modeI8ToFp && groupReg =/= 3.U) { - groupReg := groupReg + 1.U - writeWord := scaledI8Word(srcWord, groupReg + 1.U) - state := sWriteReq - }.elsewhen(modeI32Group && groupReg =/= lastGroupReg) { - groupReg := groupReg + 1.U - state := sReadReq - }.elsewhen(rowReg === iterReg - 1.U) { - state := complete - }.otherwise { - rowReg := rowReg + 1.U - groupReg := 0.U - outWord := 0.U - state := sReadReq - } - } - } - - is(complete) { - when(io.cmdResp.fire) { - state := idle + when(groupReg === lastGroupReg && rowReg === iterReg - 1.U)(state := complete) + .elsewhen(groupReg === lastGroupReg) { groupReg := 0.U; rowReg := rowReg + 1.U; state := sReadReq } + .otherwise { groupReg := groupReg + 1.U; state := sReadReq } } } + is(complete)(when(io.cmdResp.fire)(state := idle)) } io.status.idle := state === idle diff --git a/examples/balls/int2fp/arch/src/main/scala/Int2FpBall.scala b/examples/balls/int2fp/arch/src/main/scala/Int2FpBall.scala index 73883418e..4a44cc2e9 100644 --- a/examples/balls/int2fp/arch/src/main/scala/Int2FpBall.scala +++ b/examples/balls/int2fp/arch/src/main/scala/Int2FpBall.scala @@ -5,18 +5,25 @@ import chisel3.util._ import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} import framework.balldomain.blink.{BallStatus, BlinkIO, HasBallStatus, HasBlink, SubRobRow} import framework.balldomain.blink.mmio.MmioRead +import framework.balldomain.blink.mmio.MmioWrite import framework.top.GlobalConfig @instantiable class Int2FpBall(val b: GlobalConfig) extends Module with HasBlink { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "Int2FpBall") - .getOrElse(throw new IllegalArgumentException("Int2FpBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "Int2FpBall") + .getOrElse( + throw new IllegalArgumentException("Int2FpBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW + val mmioReadBW = ballCommonConfig.mmioReadBW + val mmioWriteBW = ballCommonConfig.mmioWriteBW @public - val io = IO(new BlinkIO(b, inBW, outBW)) + val io = IO(new BlinkIO(b, inBW, outBW, mmioReadBW, mmioWriteBW)) def blink: BlinkIO = io dontTouch(io) @@ -33,12 +40,14 @@ class Int2FpBall(val b: GlobalConfig) extends Module with HasBlink { for (i <- 0 until outBW) { int2fpUnit.io.bankWrite(i) <> io.bankWrite(i) } + for (i <- 0 until mmioReadBW) { + int2fpUnit.io.mmioRead(i) <> io.mmioRead(i) + } io.status <> int2fpUnit.io.status io.subRobReq.valid := false.B io.subRobReq.bits := SubRobRow.tieOff(b) - // MMIO: this Ball does not consume MMIO metadata - MmioRead.tieOff(io.mmioRead) + MmioWrite.tieOff(io.mmioWrite) } diff --git a/examples/balls/int2fp/compiler/src/Backend/include/IntrinsicsRISCVInt2FpBall.td b/examples/balls/int2fp/compiler/src/Backend/include/IntrinsicsRISCVInt2FpBall.td deleted file mode 100644 index c0768ecbf..000000000 --- a/examples/balls/int2fp/compiler/src/Backend/include/IntrinsicsRISCVInt2FpBall.td +++ /dev/null @@ -1,4 +0,0 @@ -//===- IntrinsicsRISCVInt2FpBall.td - Int2FpBall intrinsics --------------===// - -let TargetPrefix = "riscv" in -def int_riscv_bb_int2fp : Intrinsic<[], [llvm_i64_ty, llvm_i64_ty], []>; diff --git a/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp index 730fcefee..73d32c70d 100644 --- a/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp +++ b/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -10,33 +10,21 @@ using namespace mlir; using namespace ::buddy::buckyball; namespace mlir::buddy { -void populateInt2FpAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateInt2FpBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); } // namespace mlir::buddy namespace { -class BankInt2FpPattern : public OpRewritePattern { +template +class BankInt2FpPattern : public OpRewritePattern { public: - using OpRewritePattern::OpRewritePattern; + using OpRewritePattern::OpRewritePattern; - LogicalResult matchAndRewrite(BankInt2FpOp op, + LogicalResult matchAndRewrite(BankOp op, PatternRewriter &rewriter) const override { - rewriter.create(op.getLoc(), op.getInBank(), op.getOutBank(), - op.getIter(), op.getScale()); - rewriter.replaceOp(op, op.getOutBank()); - return success(); - } -}; - -class BankInt32ToInt8Pattern : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(BankInt32ToInt8Op op, - PatternRewriter &rewriter) const override { - rewriter.create(op.getLoc(), op.getInBank(), op.getOutBank(), - op.getIter(), op.getScale()); + rewriter.create(op.getLoc(), op.getInBank(), op.getOutBank(), + op.getIter(), op.getDaAddr(), op.getDwAddr()); rewriter.replaceOp(op, op.getOutBank()); return success(); } @@ -44,9 +32,10 @@ class BankInt32ToInt8Pattern : public OpRewritePattern { } // namespace -void mlir::buddy::populateInt2FpAssignPhysicalBankPatterns( +void mlir::buddy::populateInt2FpBallAssignPhysicalBankPatterns( RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { (void)state; - patterns.add( + patterns.add, + BankInt2FpPattern>( patterns.getContext()); } diff --git a/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Int2FpBall.td b/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Int2FpBall.td index e2312612b..5077c406c 100644 --- a/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Int2FpBall.td +++ b/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Int2FpBall.td @@ -7,60 +7,43 @@ #ifndef DEQUANTBALL_OPS #define DEQUANTBALL_OPS -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" +include "Buckyball.td" //===----------------------------------------------------------------------===// // Int2FpBall operations //===----------------------------------------------------------------------===// -def Int2FpOp : Buckyball_Op<"int2fp"> { - let summary = "Dequantize fixed-point to floating-point"; +class Int2FpBaseOp : Buckyball_Op { + let summary = summaryText; let description = [{ - Dequantizes fixed-point values back to floating-point representation. - Corresponds to bb_int2fp instruction (funct7=52, output_mode=0). - Belongs to Int2FpBall (ballName=Int2FpBall; ballId is chip-config specific). - ISA: bb_int2fp(bank_id, wr_bank_id, iter, scale_fp32) + Dequantizes INT32 accumulators to FP32 using Da and Dw scales in MMIO. }]; let arguments = (ins I64:$inputBankId, I64:$outputBankId, - I64:$iter, I64:$scale); + I64:$iter, I64:$daAddr, I64:$dwAddr); let assemblyFormat = "$inputBankId `,` $outputBankId `,` " - "$iter `,` $scale attr-dict `:` " + "$iter `,` $daAddr `,` $dwAddr attr-dict `:` " "type($inputBankId)"; } -def BankInt2FpOp : Buckyball_Op<"bank_int2fp"> { - let summary = "Dequantization using virtual bank handles"; - let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$scale); - let results = (outs I64:$outBankOut); - let assemblyFormat = - "$inBank $outBank $iter $scale attr-dict `:` type($inBank) type($outBank) type($iter) type($scale)"; -} +def Int2FpTensorOp : Int2FpBaseOp<"int2fp_tensor", + "Per-tensor INT32 accumulator dequantization">; -def Int32ToInt8Op : Buckyball_Op<"int32_to_int8"> { - let summary = "Requantize INT32 bank groups to packed INT8"; - let description = [{ - Requantizes four INT32 bank groups into one packed INT8 bank. - Corresponds to bb_int32_to_int8 (funct7=52, output_mode=1). - Belongs to Int2FpBall (ballName=Int2FpBall; ballId is chip-config specific). - ISA: bb_int32_to_int8(bank_id, wr_bank_id, iter, scale_fp32) - }]; - let arguments = (ins I64:$inputBankId, I64:$outputBankId, - I64:$iter, I64:$scale); - let assemblyFormat = "$inputBankId `,` $outputBankId `,` " - "$iter `,` $scale attr-dict `:` " - "type($inputBankId)"; -} +def Int2FpChannelOp : Int2FpBaseOp<"int2fp_channel", + "Per-output-channel INT32 accumulator dequantization">; -def BankInt32ToInt8Op : Buckyball_Op<"bank_int32_to_int8"> { - let summary = "INT32 to INT8 requant using virtual bank handles"; - let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$scale); +class BankInt2FpBaseOp : Buckyball_Op { + let summary = summaryText; + let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$daAddr, + I64:$dwAddr); let results = (outs I64:$outBankOut); let assemblyFormat = - "$inBank $outBank $iter $scale attr-dict `:` type($inBank) type($outBank) type($iter) type($scale)"; + "$inBank $outBank $iter $daAddr $dwAddr attr-dict `:` type($inBank) type($outBank) type($iter) type($daAddr) type($dwAddr)"; } -def Int2FpIntrOp : Buckyball_IntrOpBase<"int2fp">, - Arguments<(ins LLVM_Type, LLVM_Type)>; +def BankInt2FpTensorOp : BankInt2FpBaseOp<"bank_int2fp_tensor", + "Per-tensor dequantization using virtual bank handles">; + +def BankInt2FpChannelOp : BankInt2FpBaseOp<"bank_int2fp_channel", + "Per-output-channel dequantization using virtual bank handles">; #endif // DEQUANTBALL_OPS diff --git a/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index a3c9f82fc..604d3365c 100644 --- a/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -1,10 +1,14 @@ #include "mlir/Conversion/LLVMCommon/ConversionTarget.h" #include "mlir/Conversion/LLVMCommon/Pattern.h" #include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/IR/Matchers.h" #include "mlir/IR/PatternMatch.h" +#include + #include "Buckyball/BuckyballOps.h" #include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; @@ -12,77 +16,54 @@ using namespace buddy::buckyball::legalize; namespace { -Value packInt2FpRs2(ConversionPatternRewriter &rewriter, Location loc, - Value scale, uint64_t outputMode) { - Value scaleLow = rewriter.create( - loc, scale, cstI64(rewriter, loc, 0xffffffffULL)); - if (outputMode == 0) - return scaleLow; - Value mode = cstI64(rewriter, loc, outputMode << 32); - return rewriter.create(loc, scaleLow, mode); -} - -template -LogicalResult lowerIntConvert(OpTy op, typename OpTy::Adaptor adaptor, - ConversionPatternRewriter &rewriter, bool stable, - uint64_t outputMode) { - Location loc = op.getLoc(); - Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInputBankId(), - cstI64(rewriter, loc, 0), - adaptor.getOutputBankId(), adaptor.getIter()); - Value rs2 = packInt2FpRs2(rewriter, loc, adaptor.getScale(), outputMode); - if (stable) { - rewriter.replaceOpWithNewOp(op, rs1, rs2); - return success(); - } - rewriter.replaceOpWithNewOp(op, rs1, rs2, - rewriter.getI32IntegerAttr(52)); - return success(); -} - -struct Int2FpLowering : public ConvertOpToLLVMPattern { - Int2FpLowering(LLVMTypeConverter &converter, bool stable) - : ConvertOpToLLVMPattern(converter), stable(stable) {} +template +struct Int2FpLowering : public ConvertOpToLLVMPattern { + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; LogicalResult - matchAndRewrite(Int2FpOp op, OpAdaptor adaptor, + matchAndRewrite(Op op, typename Op::Adaptor adaptor, ConversionPatternRewriter &rewriter) const override { - return lowerIntConvert(op, adaptor, rewriter, stable, /*outputMode=*/0); + buckyball_target::requireBuckyballBall("Int2FpBall"); + llvm::APInt daAddr(64, 0); + if (!matchPattern(op.getDaAddr(), m_ConstantInt(&daAddr)) || + daAddr.getSExtValue() != 0) + return op.emitError("INT2FP Da address must be 0"); + llvm::APInt dwAddr(64, 0); + if (!matchPattern(op.getDwAddr(), m_ConstantInt(&dwAddr)) || + dwAddr.getSExtValue() < 16 || dwAddr.getSExtValue() % 4 != 0) + return op.emitError("INT2FP Dw address must be >= 16 and 4-byte aligned"); + Location loc = op.getLoc(); + Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInputBankId(), + cstI64(rewriter, loc, 0), + adaptor.getOutputBankId(), adaptor.getIter()); + Value dw = rewriter.create(loc, adaptor.getDwAddr(), + cstI64(rewriter, loc, 13)); + Value rs2 = rewriter.create(loc, adaptor.getDaAddr(), dw); + llvm::StringRef mnemonic = + std::is_same_v ? "INT2FP_TENSOR" : "INT2FP_CHANNEL"; + rewriter.replaceOpWithNewOp( + op, rs1, rs2, + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7(mnemonic))); + return success(); } - -private: - bool stable = false; }; -struct Int32ToInt8Lowering : public ConvertOpToLLVMPattern { - Int32ToInt8Lowering(LLVMTypeConverter &converter, bool stable) - : ConvertOpToLLVMPattern(converter), stable(stable) {} - - LogicalResult - matchAndRewrite(Int32ToInt8Op op, OpAdaptor adaptor, - ConversionPatternRewriter &rewriter) const override { - return lowerIntConvert(op, adaptor, rewriter, stable, /*outputMode=*/1); - } - -private: - bool stable = false; -}; } // namespace namespace mlir::buddy::buckyball { -void populateInt2FpLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable) { - patterns.add(converter, stable); +void populateInt2FpBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t, bool) { + (void)stable; + patterns.add, Int2FpLowering>( + converter); } -void configureInt2FpLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable) { - if (stable) - target.addLegalOp(); - else - target.addIllegalOp(); - target - .addIllegalOp(); +void configureInt2FpBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { + (void)stable; + target.addIllegalOp(); } } // namespace mlir::buddy::buckyball diff --git a/examples/balls/int2fp/configs/Int2FpBallParam.scala b/examples/balls/int2fp/configs/Int2FpBallParam.scala index bb7b95bf4..2b61dd2d1 100644 --- a/examples/balls/int2fp/configs/Int2FpBallParam.scala +++ b/examples/balls/int2fp/configs/Int2FpBallParam.scala @@ -4,9 +4,8 @@ import framework.balldomain.configs.BallParamLoader import framework.top.GlobalConfig case class Int2FpBallParam( - inputType: String, - outputType: String -) + inputType: String, + outputType: String) object Int2FpBallParam { private val ballName = "Int2FpBall" @@ -18,4 +17,5 @@ object Int2FpBallParam { outputType = BallParamLoader.str(tbl, "outputType") ) } + } diff --git a/examples/balls/int2fp/emu/Cargo.toml b/examples/balls/int2fp/emu/Cargo.toml deleted file mode 100644 index 73a19e5a3..000000000 --- a/examples/balls/int2fp/emu/Cargo.toml +++ /dev/null @@ -1,7 +0,0 @@ -[package] -name = "int2fp-emu-model" -version = "0.1.0" -edition = "2021" - -[lib] -path = "src/model.rs" diff --git a/examples/balls/int2fp/emu/src/52_int2fp.rs b/examples/balls/int2fp/emu/src/52_int2fp.rs deleted file mode 100644 index da312a776..000000000 --- a/examples/balls/int2fp/emu/src/52_int2fp.rs +++ /dev/null @@ -1,127 +0,0 @@ -//===- 52_int2fp.rs - INT2FP instruction (INT to FP32 / INT8) --------------===// - -use super::super::bank::{bank_num, bank_size}; -use super::decode::{pbank, pbank_group, rs1_b0, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; - -mod model; - -pub struct Int2Fp; - -impl Instruction for Int2Fp { - const FUNCT: u32 = 52; - - fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - let src = rs1_b0(xs1); - let dst = rs1_b2(xs1); - let depth = rs1_iter(xs1) as usize; - - if src >= bank_num() as u64 || dst >= bank_num() as u64 { - panic!("int2fp: invalid bank_id"); - } - - if depth == 0 { - panic!("int2fp: iter must be > 0"); - } - - let sc = ctx.cfgs[src as usize]; - let dc = ctx.cfgs[dst as usize]; - if !sc.allocated || !dc.allocated { - panic!("int2fp: bank not allocated"); - } - - let scale_bits = (xs2 & 0xffff_ffff) as u32; - let output_mode = ((xs2 >> 32) & 0x3) as u32; - if output_mode > 1 { - panic!("int2fp: reserved output mode {output_mode}"); - } - - match (output_mode, sc.cols, dc.cols) { - (0, 1, 1) => { - let ps = pbank(ctx.bank_map, src); - let pd = pbank(ctx.bank_map, dst); - for i in 0..depth { - let base = i * 16; - if base + 16 > bank_size() { - panic!("int2fp: out of range"); - } - for lane in 0..4 { - let off = base + lane * 4; - let v = i32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); - let o = model::int2fp_fp32_bits(v, scale_bits); - ctx.banks[pd][off..off + 4].copy_from_slice(&o.to_le_bytes()); - } - } - } - (0, 1, 4) => { - let ps = pbank(ctx.bank_map, src); - for i in 0..depth { - let src_base = i * 16; - let dst_base = i * 16; - if src_base + 16 > bank_size() || dst_base + 16 > bank_size() { - panic!("int2fp: out of range"); - } - for j in 0..16 { - let v = ctx.banks[ps][src_base + j] as i8; - let o = model::int2fp_fp32_bits(i32::from(v), scale_bits); - let group = j / 4; - let lane = j % 4; - let pd = pbank_group(ctx.bank_map, dst, group as u64); - let off = dst_base + lane * 4; - ctx.banks[pd][off..off + 4].copy_from_slice(&o.to_le_bytes()); - } - } - } - (0, 2, 2) | (0, 4, 4) => { - let groups = sc.cols as u64; - for group in 0..groups { - let ps = pbank_group(ctx.bank_map, src, group); - let pd = pbank_group(ctx.bank_map, dst, group); - for i in 0..depth { - let base = i * 16; - if base + 16 > bank_size() { - panic!("int2fp: out of range"); - } - for lane in 0..4 { - let off = base + lane * 4; - let v = - i32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); - let o = model::int2fp_fp32_bits(v, scale_bits); - ctx.banks[pd][off..off + 4].copy_from_slice(&o.to_le_bytes()); - } - } - } - } - (1, 4, 1) => { - let pd = pbank(ctx.bank_map, dst); - for i in 0..depth { - let base = i * 16; - if base + 16 > bank_size() { - panic!("int2fp: out of range"); - } - for group in 0..4 { - let ps = pbank_group(ctx.bank_map, src, group); - for lane in 0..4 { - let off = base + lane * 4; - let v = - i32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); - let q = model::int2fp_i8_bits(v, scale_bits); - ctx.banks[pd][base + group as usize * 4 + lane] = q as u8; - } - } - } - } - _ => { - panic!( - "int2fp: unsupported mode/layout output_mode={} src_cols={} dst_cols={}", - output_mode, sc.cols, dc.cols - ); - } - } - 0 - } - - fn latency(xs1: u64, _xs2: u64) -> u64 { - rs1_iter(xs1).max(1) - } -} diff --git a/examples/balls/int2fp/emu/src/int2fp.rs b/examples/balls/int2fp/emu/src/int2fp.rs new file mode 100644 index 000000000..be2000709 --- /dev/null +++ b/examples/balls/int2fp/emu/src/int2fp.rs @@ -0,0 +1,94 @@ +//===- int2fp.rs - INT32 accumulator dequantization -----------------------===// +// +// FP32 output = INT32 accumulator * Da * Dw. +// Da and Dw are FP32 values stored in the unified MMIO byte address space. + +use super::super::bank::{bank_num, bank_size, mmio_read_byte, mmio_total_size}; +use super::decode::{pbank_group, rs1_b0, rs1_b2, rs1_iter}; +use super::instruction::ExecContext; + +fn read_f32(mmio_banks: &[Vec], addr: usize) -> f32 { + if addr % 4 != 0 { + panic!("int2fp: scale address must be 4-byte aligned"); + } + if addr + 4 > mmio_total_size() { + panic!("int2fp: scale address out of range"); + } + let bytes = [ + mmio_read_byte(mmio_banks, addr), + mmio_read_byte(mmio_banks, addr + 1), + mmio_read_byte(mmio_banks, addr + 2), + mmio_read_byte(mmio_banks, addr + 3), + ]; + let scale = f32::from_le_bytes(bytes); + if !scale.is_finite() || scale <= 0.0 { + panic!("int2fp: scale must be finite and positive"); + } + scale +} + +pub(crate) fn exec_int2fp(xs1: u64, xs2: u64, ctx: &mut ExecContext, per_channel: bool) -> u64 { + let src = rs1_b0(xs1); + let dst = rs1_b2(xs1); + let depth = rs1_iter(xs1) as usize; + let act_scale_addr = (xs2 & 0x1fff) as usize; + let weight_scale_addr = ((xs2 >> 13) & 0x1fff) as usize; + if act_scale_addr != 0 { + panic!("int2fp: activation scale address must be 0"); + } + if weight_scale_addr < 16 { + panic!("int2fp: weight scale address must be >= 16"); + } + if xs2 >> 26 != 0 { + panic!("int2fp: reserved rs2 bits are nonzero"); + } + if src >= bank_num() as u64 || dst >= bank_num() as u64 { + panic!("int2fp: invalid bank_id"); + } + if src == dst { + panic!("int2fp: in-place dequantization is forbidden"); + } + if depth == 0 { + panic!("int2fp: iter must be > 0"); + } + let sc = ctx.cfgs[src as usize]; + let dc = ctx.cfgs[dst as usize]; + if !sc.allocated || !dc.allocated { + panic!("int2fp: bank not allocated"); + } + if sc.cols != dc.cols || sc.cols == 0 || sc.cols > 4 { + panic!("int2fp: requires matching 1..4 INT32/FP32 bank groups"); + } + if per_channel && weight_scale_addr + sc.cols as usize * 16 > mmio_total_size() { + panic!("int2fp: channel scale range out of range"); + } + + let da = read_f32(ctx.mmio_banks, act_scale_addr); + let tensor_dw = (!per_channel).then(|| read_f32(ctx.mmio_banks, weight_scale_addr)); + for group in 0..sc.cols as u64 { + let ps = pbank_group(ctx.bank_map, src, group); + let pd = pbank_group(ctx.bank_map, dst, group); + for row in 0..depth { + let base = row * 16; + if base + 16 > bank_size() { + panic!("int2fp: bank row out of range"); + } + for lane in 0..4usize { + let off = base + lane * 4; + let acc = i32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); + let dw = tensor_dw.unwrap_or_else(|| { + read_f32( + ctx.mmio_banks, + weight_scale_addr + (group as usize * 4 + lane) * 4, + ) + }); + let out = (acc as f32) * da * dw; + ctx.banks[pd][off..off + 4].copy_from_slice(&out.to_le_bytes()); + } + } + } + 0 +} +pub(crate) fn latency(xs1: u64) -> u64 { + rs1_iter(xs1).max(1) +} diff --git a/examples/balls/int2fp/emu/src/lib.rs b/examples/balls/int2fp/emu/src/lib.rs index 923edb048..a5df10297 100644 --- a/examples/balls/int2fp/emu/src/lib.rs +++ b/examples/balls/int2fp/emu/src/lib.rs @@ -1,9 +1,8 @@ pub(crate) use crate::inst::{decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::ExecContext; -#[path = "52_int2fp.rs"] -mod f52_int2fp; +mod int2fp; const BALL_CLASS: &str = "examples.balls.int2fp.Int2FpBall"; @@ -17,11 +16,10 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("INT2FP_TENSOR") => Some(int2fp::exec_int2fp(xs1, xs2, ctx, false)), + Some("INT2FP_CHANNEL") => Some(int2fp::exec_int2fp(xs1, xs2, ctx, true)), + Some(_) | None => None, } } @@ -29,10 +27,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("INT2FP_TENSOR") | Some("INT2FP_CHANNEL") => Some(int2fp::latency(xs1)), + Some(_) | None => None, } } diff --git a/examples/balls/int2fp/emu/src/model.rs b/examples/balls/int2fp/emu/src/model.rs index a0408d2bb..a58c6d01a 100644 --- a/examples/balls/int2fp/emu/src/model.rs +++ b/examples/balls/int2fp/emu/src/model.rs @@ -1,9 +1,9 @@ -pub fn int2fp_fp32_bits(value: i32, scale_bits: u32) -> u32 { - fp32_multiply(int32_to_fp32(value), scale_bits) +pub fn int2fp_fp32_bits(value: i32, dw_bits: u32) -> u32 { + fp32_multiply(int32_to_fp32(value), dw_bits) } -pub fn int2fp_i8_bits(value: i32, scale_bits: u32) -> i8 { - fp32_to_int32(int2fp_fp32_bits(value, scale_bits)).clamp(-128, 127) as i8 +pub fn int2fp_dequant_bits(value: i32, da_bits: u32, dw_bits: u32) -> u32 { + int2fp_fp32_bits(value, fp32_multiply(da_bits, dw_bits)) } pub fn int32_to_fp32(value: i32) -> u32 { @@ -87,49 +87,6 @@ fn fp32_multiply(a: u32, b: u32) -> u32 { } } -fn fp32_to_int32(fp: u32) -> i32 { - let sign = ((fp >> 31) & 1) != 0; - let exponent = ((fp >> 23) & 0xff) as i32; - let frac = fp & 0x7f_ffff; - let mantissa = (1u32 << 23) | frac; - let is_zero = exponent == 0 && frac == 0; - let exp_val = exponent - 127; - - if exponent == 0xff && frac != 0 { - return 0; - } - if is_zero { - return 0; - } - - let magnitude = if exp_val >= 31 { - 0x8000_0000u64 - } else if exp_val >= 23 { - u64::from(mantissa) << (exp_val - 23) - } else if exp_val >= -1 { - let right_shift = (23 - exp_val) as u32; - let truncated = u64::from(mantissa) >> right_shift; - let half = 1u64 << (right_shift - 1); - let remainder = u64::from(mantissa) & ((1u64 << right_shift) - 1); - let round_up = remainder > half || (remainder == half && (truncated & 1) != 0); - truncated + u64::from(round_up) - } else { - 0 - }; - - if sign { - if magnitude >= 0x8000_0000 { - i32::MIN - } else { - -(magnitude as i32) - } - } else if magnitude > i32::MAX as u64 { - i32::MAX - } else { - magnitude as i32 - } -} - #[cfg(test)] mod tests { use super::*; @@ -151,18 +108,10 @@ mod tests { } #[test] - fn requant_vectors() { - let scale = 0x3F00_0000; // 0.5 - let input = [ - -1000, -257, -255, -5, -3, -1, 0, 1, 3, 5, 127, 253, 255, 257, 1000, 2, -999, -511, - -259, -9, -7, -3, 2, 4, 6, 9, 125, 251, 254, 258, 511, 999, - ]; - let expected = [ - -128i8, -128, -128, -2, -2, 0, 0, 0, 2, 2, 64, 126, 127, 127, 127, 1, -128, -128, -128, - -4, -4, -2, 1, 2, 3, 4, 62, 126, 127, 127, 127, 127, - ]; - for (value, expected) in input.into_iter().zip(expected) { - assert_eq!(int2fp_i8_bits(value, scale), expected, "input={value}"); - } + fn dequant_uses_both_scales() { + assert_eq!( + int2fp_dequant_bits(8, 0x3f00_0000, 0x3e80_0000), + 1.0f32.to_bits() + ); } } diff --git a/examples/balls/int2fp/verify/README.md b/examples/balls/int2fp/verify/README.md index ae086b499..92d882f3d 100644 --- a/examples/balls/int2fp/verify/README.md +++ b/examples/balls/int2fp/verify/README.md @@ -7,30 +7,35 @@ Verifies the generated `Int2FpBall` module using the shared Blink UVM framework. - `../../../../verify/uvm/src/bb_uvm_pkg.sv`: common Blink UVM agents and base env - `src/common/int2fp_defs.svh`: DPI imports, `int2fp_require_bid`, timeouts - `src/common/int2fp_items.svh`: `int2fp_cmd_item` with `load_rust_case` - (`special[31:0]=scale`, `special[33:32]=output_mode`) + (`special[12:0]=Da`, `special[25:13]=Dw`; tensor/channel use separate funct7) - `src/seq/int2fp_sequences.svh`: one-case sequence -- `src/cov/int2fp_cov.svh`: mode {fp32,i8} and iter {1,2,4,16} coverage -- `src/env/int2fp_scoreboard.svh`: preload mem model (group-aware for INT8), compare writes vs DPI +- `src/cov/int2fp_cov.svh`: iter {1,16} coverage +- `src/env/int2fp_scoreboard.svh`: preload mem model and compare writes vs DPI - `src/env/int2fp_env.svh`: extends `bb_blink_env#(1,1)` -- `src/tests/int2fp_test.svh`: directed 0..1, random 2..21 with seed `0xCAFE_BABE` +- `src/tests/int2fp_*_test.svh`: one directed case per UVM test - `src/pkg/int2fp_pkg.sv`: package entry - `src/tb_top.sv`: `bb_blink_if#(1,1)` + `Int2FpBall` - `filelists/int2fp_ball.f`: VCS filelist (`@UVM@` / `@RTL@`) ## Test plan -`+UVM_TESTNAME=int2fp_ball_test` runs: +The test names are: -- case 0: directed INT32→FP32 (`op1_col=1`, `wr_col=1`, `output_mode=0`, scale=1.0) -- case 1: directed INT32→INT8 (`op1_col=4`, `wr_col=1`, `output_mode=1`, scale=0.5) -- cases 2..21: random FP32/INT8 with fixed seed `0xCAFE_BABE` +- `int2fp_tensor_rows_test`: 16-row tensor scale +- `int2fp_channel_lanes_test`: four-group channel scale with distinct `Dw` +- `int2fp_tensor_groups_test`: four-group tensor scale traversal +- `int2fp_channel_base_test`: channel scale at MMIO byte address 64 +- `int2fp_channel_two_rows_test`: two rows reuse all sixteen channel scales -Each case: reset, preload src words into `mem_model` (INT8 uses `group_id` 0..3), +Each case: reset, preload src words and the four Da/Dw MMIO bytes into `mem_model`, drive one command, wait for `scb.done()` or timeout at 4000 cycles. Scoreboard -compares observed writes with DPI `int2fp_ref_fp32` / `int2fp_ref_i8`. `int2fp_cov` -requires mode and iter bins hit across the whole test, else `check_phase` fatal. +compares observed writes with DPI `int2fp_ref_fp32`, including `Da * Dw` for +per-tensor and per-channel weights. The tensor path requires eight MMIO byte +reads (four Da bytes followed by four tensor-Dw bytes). -`funct7` is 52 (`BB_INT2FP_FUNC7`). +The core's `ballISA` assigns separate funct7 values to `INT2FP_TENSOR` and +`INT2FP_CHANNEL` (Pebble uses 52 and 54). The Ball does not define these +numbers. ## BID (required) @@ -42,7 +47,7 @@ requires mode and iter bins hit across the whole test, else `check_phase` fatal. | `sims.verilator.BuckyballToyVerilatorConfig` (full.toml) | `+BID=6` | ```console -./simv +UVM_TESTNAME=int2fp_ball_test +BID=4 +./simv +UVM_TESTNAME=int2fp_channel_two_rows_test +BID=4 ``` ## Build @@ -71,12 +76,13 @@ vcs -full64 -sverilog -timescale=1ns/1ps \ Run: ```console -./simv +UVM_TESTNAME=int2fp_ball_test +BID=4 +./simv +UVM_TESTNAME=int2fp_channel_two_rows_test +BID=4 ``` Or via bbdev: ```console -bbdev uvm --build '--ball=int2fp' --config sims.verilator.BuckyballPebbleVerilatorConfig -bbdev uvm --run '--ball=int2fp' --config sims.verilator.BuckyballPebbleVerilatorConfig -- '+BID=4' +bbdev uvm --build '--ball=int2fp' --config sims.verilator.BuckyballPebbleVerilatorConfig \ + --core-config examples/cores/pebble/configs/default.toml +bbdev uvm --run '--ball=int2fp --test int2fp_channel_two_rows_test --plusargs +BID=4' ``` diff --git a/examples/balls/int2fp/verify/casegen/src/casegen.rs b/examples/balls/int2fp/verify/casegen/src/casegen.rs index d85b38ef1..041b0b8d0 100644 --- a/examples/balls/int2fp/verify/casegen/src/casegen.rs +++ b/examples/balls/int2fp/verify/casegen/src/casegen.rs @@ -1,329 +1,228 @@ -pub const FUNCT7: u32 = 52; pub const GROUPS: usize = 4; pub const MAX_ITER: usize = 16; pub const MAX_WORDS: usize = MAX_ITER * GROUPS; -pub const OUTPUT_FP32: u32 = 0; -pub const OUTPUT_INT8: u32 = 1; #[repr(C)] #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct Int2FpCmd { - pub bid: u32, - pub funct7: u32, - pub iter: u32, - pub scale_bits: u32, - pub output_mode: u32, - pub op1_bank: u32, - pub wr_bank: u32, - pub op1_col: u32, - pub wr_col: u32, - pub rob_id: u32, - pub num_src_words: u32, + pub bid: u32, + pub iter: u32, + pub da_bits: u32, + pub dw_addr: u32, + pub dw_bits: u32, + pub per_channel: u32, + pub op1_bank: u32, + pub wr_bank: u32, + pub op1_col: u32, + pub wr_col: u32, + pub rob_id: u32, + pub num_src_words: u32, } #[derive(Clone, Debug, PartialEq, Eq)] pub struct Int2FpCase { - pub cmd: Int2FpCmd, - pub input_words: [u128; MAX_WORDS], + pub cmd: Int2FpCmd, + pub input_words: [u128; MAX_WORDS], + pub channel_dw_bits: [u32; GROUPS * 4], } impl Int2FpCase { - pub fn word_lo(&self, index: usize) -> u64 { - self.input_words[index] as u64 - } + pub fn word_lo(&self, index: usize) -> u64 { + self.input_words[index] as u64 + } - pub fn word_hi(&self, index: usize) -> u64 { - (self.input_words[index] >> 64) as u64 - } + pub fn word_hi(&self, index: usize) -> u64 { + (self.input_words[index] >> 64) as u64 + } - #[allow(dead_code)] - pub fn is_i8(&self) -> bool { - self.cmd.output_mode == OUTPUT_INT8 - && self.cmd.op1_col == 4 - && self.cmd.wr_col == 1 - } + pub fn dw_bits(&self, index: usize) -> u32 { + self.channel_dw_bits[index] + } } -pub fn gen_case(seed: u32, index: u32, bid: u32) -> Int2FpCase { - match index { - 0 => directed_i32_to_fp32(bid), - 1 => directed_i32_to_i8(bid), - _ => random_case(seed, index, bid), - } +pub fn gen_case(index: u32, bid: u32) -> Int2FpCase { + match index { + 0 => directed_tensor(bid), + 1 => directed_channel(bid), + 2 => tensor_groups(bid), + 3 => channel_base(bid), + 4 => channel_two_rows(bid), + _ => panic!("int2fp: unsupported directed case {index}"), + } } fn pack_i32s(vals: &[i32]) -> u128 { - if vals.len() != 4 { - panic!("pack_i32s: need exactly 4 lanes, got {}", vals.len()); - } - let mut word = 0u128; - for (lane, v) in vals.iter().enumerate() { - word |= u128::from(*v as u32) << (lane * 32); - } - word -} - -fn directed_i32_to_fp32(bid: u32) -> Int2FpCase { - let vals: [i32; 64] = [ - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, 1, 2, 3, -1, -2, 0, 4, 5, 10, - -10, 7, 100, -100, 8, 16, -8, 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, 1, - 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, - ]; - let iter = 16usize; - let mut input_words = [0u128; MAX_WORDS]; - for w in 0..iter { - input_words[w] = pack_i32s(&vals[w * 4..w * 4 + 4]); - } - - Int2FpCase { - cmd: Int2FpCmd { - bid, - funct7: FUNCT7, - iter: iter as u32, - scale_bits: 0x3F80_0000, - output_mode: OUTPUT_FP32, - op1_bank: 0, - wr_bank: 1, - op1_col: 1, - wr_col: 1, - rob_id: 3, - num_src_words: iter as u32, - }, - input_words, - } -} - -fn directed_i32_to_i8(bid: u32) -> Int2FpCase { - let vals: [i32; 32] = [ - -1000, -257, -255, -5, -3, -1, 0, 1, 3, 5, 127, 253, 255, 257, 1000, 2, -999, -511, -259, -9, - -7, -3, 2, 4, 6, 9, 125, 251, 254, 258, 511, 999, - ]; - let iter = 2usize; - let mut input_words = [0u128; MAX_WORDS]; - for row in 0..iter { - for group in 0..GROUPS { - let base = row * 16 + group * 4; - input_words[row * GROUPS + group] = pack_i32s(&vals[base..base + 4]); + if vals.len() != 4 { + panic!("pack_i32s: need exactly 4 lanes, got {}", vals.len()); + } + let mut word = 0u128; + for (lane, v) in vals.iter().enumerate() { + word |= u128::from(*v as u32) << (lane * 32); + } + word +} + +fn directed_tensor(bid: u32) -> Int2FpCase { + let vals: [i32; 64] = [ + 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, 1, 2, 3, -1, -2, 0, 4, 5, 10, + -10, 7, 100, -100, 8, 16, -8, 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, + 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, + ]; + let iter = 16usize; + let mut input_words = [0u128; MAX_WORDS]; + for w in 0..iter { + input_words[w] = pack_i32s(&vals[w * 4..w * 4 + 4]); } - } - - Int2FpCase { - cmd: Int2FpCmd { - bid, - funct7: FUNCT7, - iter: iter as u32, - scale_bits: 0x3F00_0000, - output_mode: OUTPUT_INT8, - op1_bank: 0, - wr_bank: 1, - op1_col: 4, - wr_col: 1, - rob_id: 2, - num_src_words: (iter * GROUPS) as u32, - }, - input_words, - } -} -fn random_case(seed: u32, index: u32, bid: u32) -> Int2FpCase { - let mut rng = Rng::new(seed, index); - let to_i8 = (rng.next() & 1) == 1; - if to_i8 { - random_i32_to_i8(&mut rng, bid) - } else { - random_i32_to_fp32(&mut rng, bid) - } + Int2FpCase { + cmd: Int2FpCmd { + bid, + iter: iter as u32, + da_bits: 0x3F00_0000, + dw_addr: 16, + dw_bits: 0x3E80_0000, + per_channel: 0, + op1_bank: 0, + wr_bank: 1, + op1_col: 1, + wr_col: 1, + rob_id: 3, + num_src_words: iter as u32, + }, + input_words, + channel_dw_bits: [0x3E80_0000; GROUPS * 4], + } } -fn random_i32_to_fp32(rng: &mut Rng, bid: u32) -> Int2FpCase { - let iter_pool = [1u32, 2, 4, 16]; - let iter = iter_pool[(rng.next() as usize) % iter_pool.len()] as usize; - let op1_bank = rng.range(0, 7); - let mut wr_bank = rng.range(0, 7); - if wr_bank == op1_bank { - wr_bank = (wr_bank + 1) & 7; - } - - let mut input_words = [0u128; MAX_WORDS]; - for w in 0..iter { - input_words[w] = random_i32_word(rng); - } - - Int2FpCase { - cmd: Int2FpCmd { - bid, - funct7: FUNCT7, - iter: iter as u32, - scale_bits: scale_pool(rng.range(0, 3)), - output_mode: OUTPUT_FP32, - op1_bank, - wr_bank, - op1_col: 1, - wr_col: 1, - rob_id: rng.range(0, 15), - num_src_words: iter as u32, - }, - input_words, - } +fn directed_channel(bid: u32) -> Int2FpCase { + directed_channel_at(bid, 16, 3) } -fn random_i32_to_i8(rng: &mut Rng, bid: u32) -> Int2FpCase { - let iter_pool = [1u32, 2, 4]; - let iter = iter_pool[(rng.next() as usize) % iter_pool.len()] as usize; - let op1_bank = rng.range(0, 7); - let mut wr_bank = rng.range(0, 7); - if wr_bank == op1_bank { - wr_bank = (wr_bank + 1) & 7; - } - - let mut input_words = [0u128; MAX_WORDS]; - for row in 0..iter { +fn directed_channel_at(bid: u32, dw_addr: u32, rob_id: u32) -> Int2FpCase { + let mut input_words = [0u128; MAX_WORDS]; for group in 0..GROUPS { - input_words[row * GROUPS + group] = random_i32_word(rng); + input_words[group] = pack_i32s(&[8, -8, 16, -16]); + } + let channel_dw_bits = [ + 0x3e80_0000, 0x3f00_0000, 0x3f40_0000, 0x3f80_0000, + 0x3fa0_0000, 0x3fc0_0000, 0x3fe0_0000, 0x4000_0000, + 0x3e00_0000, 0x3e80_0000, 0x3ec0_0000, 0x3f00_0000, + 0x4010_0000, 0x4020_0000, 0x4030_0000, 0x4040_0000, + ]; + Int2FpCase { + cmd: Int2FpCmd { + bid, + iter: 1, + da_bits: 0x3f00_0000, + dw_addr, + dw_bits: channel_dw_bits[0], + per_channel: 1, + op1_bank: 0, + wr_bank: 1, + op1_col: GROUPS as u32, + wr_col: GROUPS as u32, + rob_id, + num_src_words: GROUPS as u32, + }, + input_words, + channel_dw_bits, } - } - - Int2FpCase { - cmd: Int2FpCmd { - bid, - funct7: FUNCT7, - iter: iter as u32, - scale_bits: scale_pool(rng.range(0, 3)), - output_mode: OUTPUT_INT8, - op1_bank, - wr_bank, - op1_col: 4, - wr_col: 1, - rob_id: rng.range(0, 15), - num_src_words: (iter * GROUPS) as u32, - }, - input_words, - } -} - -fn random_i32_word(rng: &mut Rng) -> u128 { - let mut word = 0u128; - for lane in 0..4 { - word |= u128::from(i32_pool(rng.range(0, 15)) as u32) << (lane * 32); - } - word -} - -fn scale_pool(index: u32) -> u32 { - match index { - 0 => 0x3F80_0000, - 1 => 0x4000_0000, - 2 => 0x3F00_0000, - 3 => 0xBF80_0000, - _ => unreachable!(), - } } -fn i32_pool(index: u32) -> i32 { - match index { - 0 => 0, - 1 => 1, - 2 => -1, - 3 => 2, - 4 => -2, - 5 => 127, - 6 => -128, - 7 => 255, - 8 => -255, - 9 => 1000, - 10 => -1000, - 11 => 16, - 12 => -16, - 13 => i32::MAX, - 14 => i32::MIN, - 15 => 42, - _ => unreachable!(), - } +fn tensor_groups(bid: u32) -> Int2FpCase { + let mut input_words = [0u128; MAX_WORDS]; + input_words[0] = pack_i32s(&[2, -2, 4, -4]); + input_words[1] = pack_i32s(&[6, -6, 8, -8]); + input_words[2] = pack_i32s(&[10, -10, 12, -12]); + input_words[3] = pack_i32s(&[14, -14, 16, -16]); + Int2FpCase { + cmd: Int2FpCmd { + bid, + iter: 1, + da_bits: 0x3f00_0000, + dw_addr: 16, + dw_bits: 0x3e80_0000, + per_channel: 0, + op1_bank: 0, + wr_bank: 1, + op1_col: GROUPS as u32, + wr_col: GROUPS as u32, + rob_id: 4, + num_src_words: GROUPS as u32, + }, + input_words, + channel_dw_bits: [0x3e80_0000; GROUPS * 4], + } } -struct Rng { - state: u64, +fn channel_base(bid: u32) -> Int2FpCase { + directed_channel_at(bid, 64, 5) } -impl Rng { - fn new(seed: u32, index: u32) -> Self { - let state = (u64::from(seed) << 32) ^ u64::from(index) ^ 0x9E37_79B9_7F4A_7C15; - Self { state } - } - - fn next(&mut self) -> u32 { - self.state ^= self.state >> 12; - self.state ^= self.state << 25; - self.state ^= self.state >> 27; - ((self.state.wrapping_mul(0x2545_F491_4F6C_DD1D)) >> 32) as u32 - } - - fn range(&mut self, lo: u32, hi: u32) -> u32 { - lo + (self.next() % (hi - lo + 1)) - } +fn channel_two_rows(bid: u32) -> Int2FpCase { + let mut case = directed_channel(bid); + case.cmd.iter = 2; + case.cmd.rob_id = 6; + case.cmd.num_src_words = 8; + for group in 0..GROUPS { + case.input_words[group] = pack_i32s(&[8, 8, 8, 8]); + case.input_words[GROUPS + group] = pack_i32s(&[16, 16, 16, 16]); + } + case } #[cfg(test)] mod tests { - use super::*; - use crate::model; + use super::*; + use crate::model; + + #[test] + fn case_zero_is_fp32_smoke() { + let case = gen_case(0, 4); + assert_eq!(case.cmd.bid, 4); + assert_eq!(case.cmd.iter, 16); + assert_eq!(case.cmd.da_bits, 0x3F00_0000); + assert_eq!(case.cmd.dw_addr, 16); + assert_eq!(case.cmd.dw_bits, 0x3E80_0000); + assert_eq!(case.cmd.per_channel, 0); + assert_eq!(case.cmd.op1_col, 1); + assert_eq!(case.cmd.wr_col, 1); + assert_eq!(case.cmd.num_src_words, 16); + assert_eq!( + model::int2fp_dequant_bits(8, 0x3F00_0000, 0x3E80_0000), + 1.0f32.to_bits() + ); + } - #[test] - fn case_zero_is_fp32_smoke() { - let case = gen_case(0x1234, 0, 4); - assert_eq!(case.cmd.bid, 4); - assert_eq!(case.cmd.funct7, FUNCT7); - assert_eq!(case.cmd.iter, 16); - assert_eq!(case.cmd.scale_bits, 0x3F80_0000); - assert_eq!(case.cmd.output_mode, OUTPUT_FP32); - assert_eq!(case.cmd.op1_col, 1); - assert_eq!(case.cmd.wr_col, 1); - assert_eq!(case.cmd.num_src_words, 16); - assert!(!case.is_i8()); - assert_eq!(model::int2fp_fp32_bits(1, 0x3F80_0000), 0x3F80_0000); - } + #[test] + fn case_one_is_fp32() { + let case = gen_case(1, 6); + assert_eq!(case.cmd.bid, 6); + assert_eq!(case.cmd.op1_col, GROUPS as u32); + assert_eq!(case.cmd.wr_col, GROUPS as u32); + assert_eq!(case.cmd.per_channel, 1); + assert_eq!(case.cmd.dw_addr, 16); + assert_eq!(case.cmd.num_src_words, GROUPS as u32); + } - #[test] - fn case_one_is_i8_requant() { - let case = gen_case(0, 1, 6); - assert!(case.is_i8()); - assert_eq!(case.cmd.bid, 6); - assert_eq!(case.cmd.iter, 2); - assert_eq!(case.cmd.scale_bits, 0x3F00_0000); - assert_eq!(case.cmd.output_mode, OUTPUT_INT8); - assert_eq!(case.cmd.op1_col, 4); - assert_eq!(case.cmd.wr_col, 1); - assert_eq!(case.cmd.num_src_words, 8); - assert_eq!(model::int2fp_i8_bits(-1000, 0x3F00_0000), -128); - assert_eq!(model::int2fp_i8_bits(5, 0x3F00_0000), 2); - } + #[test] + fn bid_is_required_arg() { + assert_eq!(gen_case(0, 4).cmd.bid, 4); + assert_eq!(gen_case(0, 6).cmd.bid, 6); + } - #[test] - fn random_cases_deterministic_and_legal() { - let a = gen_case(0xCAFE_BABE, 7, 4); - let b = gen_case(0xCAFE_BABE, 7, 4); - assert_eq!(a, b); - assert_eq!(a.cmd.bid, 4); - assert_eq!(a.cmd.funct7, FUNCT7); - assert_ne!(a.cmd.op1_bank, a.cmd.wr_bank); - assert!(a.cmd.op1_bank < 8); - assert!(a.cmd.wr_bank < 8); - assert!(a.cmd.rob_id < 16); - assert!(a.cmd.output_mode == OUTPUT_FP32 || a.cmd.output_mode == OUTPUT_INT8); - if a.is_i8() { - assert_eq!(a.cmd.op1_col, 4); - assert_eq!(a.cmd.wr_col, 1); - assert_eq!(a.cmd.num_src_words, a.cmd.iter * GROUPS as u32); - } else { - assert_eq!(a.cmd.op1_col, 1); - assert_eq!(a.cmd.wr_col, 1); - assert_eq!(a.cmd.num_src_words, a.cmd.iter); + #[test] + fn channel_base_uses_nonzero_dw_address() { + let case = gen_case(3, 4); + assert_eq!(case.cmd.dw_addr, 64); + assert_eq!(case.cmd.per_channel, 1); } - } - #[test] - fn bid_is_required_arg() { - assert_eq!(gen_case(0, 0, 4).cmd.bid, 4); - assert_eq!(gen_case(0, 0, 6).cmd.bid, 6); - } + #[test] + fn channel_two_rows_reuses_all_channel_scales() { + let case = gen_case(4, 4); + assert_eq!(case.cmd.iter, 2); + assert_eq!(case.cmd.num_src_words, 8); + assert_eq!(case.cmd.op1_col, GROUPS as u32); + } } diff --git a/examples/balls/int2fp/verify/casegen/src/lib.rs b/examples/balls/int2fp/verify/casegen/src/lib.rs index 33910a907..091c330b8 100644 --- a/examples/balls/int2fp/verify/casegen/src/lib.rs +++ b/examples/balls/int2fp/verify/casegen/src/lib.rs @@ -11,93 +11,109 @@ thread_local! { } #[no_mangle] -pub extern "C" fn int2fp_ref_fp32(value: i32, scale_bits: u32) -> u32 { - model::int2fp_fp32_bits(value, scale_bits) +pub extern "C" fn int2fp_ref_fp32(value: i32, da_bits: u32, dw_bits: u32) -> u32 { + model::int2fp_dequant_bits(value, da_bits, dw_bits) } #[no_mangle] -pub extern "C" fn int2fp_ref_i8(value: i32, scale_bits: u32) -> i32 { - i32::from(model::int2fp_i8_bits(value, scale_bits)) -} - -#[no_mangle] -pub extern "C" fn int2fp_case_load(seed: u32, index: u32, bid: u32) -> i32 { - let case = casegen::gen_case(seed, index, bid); - let nsrc = case.cmd.num_src_words as usize; - if nsrc == 0 || nsrc > MAX_WORDS { - panic!("int2fp_case_load: num_src_words out of range {nsrc}"); - } - CURRENT.with(|c| *c.borrow_mut() = Some(case)); - 0 +pub extern "C" fn int2fp_case_load(index: u32, bid: u32) -> i32 { + let case = casegen::gen_case(index, bid); + let nsrc = case.cmd.num_src_words as usize; + if nsrc == 0 || nsrc > MAX_WORDS { + panic!("int2fp_case_load: num_src_words out of range {nsrc}"); + } + CURRENT.with(|c| *c.borrow_mut() = Some(case)); + 0 } fn current_case R, R>(f: F) -> R { - CURRENT.with(|c| match c.borrow().as_ref() { - Some(case) => f(case), - None => panic!("int2fp_case: no case loaded; call int2fp_case_load first"), - }) + CURRENT.with(|c| match c.borrow().as_ref() { + Some(case) => f(case), + None => panic!("int2fp_case: no case loaded; call int2fp_case_load first"), + }) } #[no_mangle] pub extern "C" fn int2fp_case_cmd(out_ptr: *mut Int2FpCmd) { - current_case(|case| unsafe { - if out_ptr.is_null() { - panic!("int2fp_case_cmd: null out_ptr"); - } - *out_ptr = case.cmd; - }); + current_case(|case| unsafe { + if out_ptr.is_null() { + panic!("int2fp_case_cmd: null out_ptr"); + } + *out_ptr = case.cmd; + }); } #[no_mangle] pub extern "C" fn int2fp_case_src_word_lo(word_index: u32) -> u64 { - if word_index as usize >= MAX_WORDS { - panic!("int2fp_case_src_word_lo: word_index out of range"); - } - current_case(|case| case.word_lo(word_index as usize)) + if word_index as usize >= MAX_WORDS { + panic!("int2fp_case_src_word_lo: word_index out of range"); + } + current_case(|case| case.word_lo(word_index as usize)) } #[no_mangle] pub extern "C" fn int2fp_case_src_word_hi(word_index: u32) -> u64 { - if word_index as usize >= MAX_WORDS { - panic!("int2fp_case_src_word_hi: word_index out of range"); - } - current_case(|case| case.word_hi(word_index as usize)) + if word_index as usize >= MAX_WORDS { + panic!("int2fp_case_src_word_hi: word_index out of range"); + } + current_case(|case| case.word_hi(word_index as usize)) +} + +#[no_mangle] +pub extern "C" fn int2fp_case_dw_bits(index: u32) -> u32 { + if index as usize >= casegen::GROUPS * 4 { + panic!("int2fp_case_dw_bits: index out of range"); + } + current_case(|case| case.dw_bits(index as usize)) } #[cfg(test)] mod tests { - use super::*; + use super::*; - #[test] - fn dpi_load_then_cmd_and_words() { - assert_eq!(int2fp_case_load(0x1234, 0, 4), 0); - let mut cmd = Int2FpCmd { - bid: 0, - funct7: 0, - iter: 0, - scale_bits: 0, - output_mode: 0, - op1_bank: 0, - wr_bank: 0, - op1_col: 0, - wr_col: 0, - rob_id: 0, - num_src_words: 0, - }; - int2fp_case_cmd(&mut cmd as *mut Int2FpCmd); - assert_eq!(cmd.bid, 4); - assert_eq!(cmd.funct7, 52); - assert_eq!(cmd.iter, 16); - assert_eq!(cmd.output_mode, 0); - assert_eq!(cmd.num_src_words, 16); - let _lo = int2fp_case_src_word_lo(0); - let _hi = int2fp_case_src_word_hi(0); - } + #[test] + fn dpi_load_then_cmd_and_words() { + assert_eq!(int2fp_case_load(0, 4), 0); + let mut cmd = Int2FpCmd { + bid: 0, + iter: 0, + da_bits: 0, + dw_addr: 0, + dw_bits: 0, + per_channel: 0, + op1_bank: 0, + wr_bank: 0, + op1_col: 0, + wr_col: 0, + rob_id: 0, + num_src_words: 0, + }; + int2fp_case_cmd(&mut cmd as *mut Int2FpCmd); + assert_eq!(cmd.bid, 4); + assert_eq!(cmd.iter, 16); + assert_eq!(cmd.num_src_words, 16); + let _lo = int2fp_case_src_word_lo(0); + let _hi = int2fp_case_src_word_hi(0); + } - #[test] - #[should_panic(expected = "no case loaded")] - fn current_case_panics_without_load() { - CURRENT.with(|c| *c.borrow_mut() = None); - current_case(|_| ()); - } + #[test] + fn dpi_preserves_channel_scale_address_and_rows() { + assert_eq!(int2fp_case_load(4, 4), 0); + let mut cmd = Int2FpCmd { + bid: 0, iter: 0, da_bits: 0, dw_addr: 0, dw_bits: 0, + per_channel: 0, op1_bank: 0, wr_bank: 0, op1_col: 0, + wr_col: 0, rob_id: 0, num_src_words: 0, + }; + int2fp_case_cmd(&mut cmd as *mut Int2FpCmd); + assert_eq!(cmd.dw_addr, 16); + assert_eq!(cmd.iter, 2); + assert_eq!(cmd.num_src_words, 8); + } + + #[test] + #[should_panic(expected = "no case loaded")] + fn current_case_panics_without_load() { + CURRENT.with(|c| *c.borrow_mut() = None); + current_case(|_| ()); + } } diff --git a/examples/balls/int2fp/verify/src/common/int2fp_defs.svh b/examples/balls/int2fp/verify/src/common/int2fp_defs.svh index e863903cb..d75e9abf0 100644 --- a/examples/balls/int2fp/verify/src/common/int2fp_defs.svh +++ b/examples/balls/int2fp/verify/src/common/int2fp_defs.svh @@ -1,9 +1,10 @@ typedef struct { int unsigned bid; - int unsigned funct7; int unsigned iter; - int unsigned scale_bits; - int unsigned output_mode; + int unsigned da_bits; + int unsigned dw_addr; + int unsigned dw_bits; + int unsigned per_channel; int unsigned op1_bank; int unsigned wr_bank; int unsigned op1_col; @@ -14,29 +15,30 @@ typedef struct { import "DPI-C" function int unsigned int2fp_ref_fp32( input int value, - input int unsigned scale_bits -); -import "DPI-C" function int int2fp_ref_i8( - input int value, - input int unsigned scale_bits + input int unsigned da_bits, + input int unsigned dw_bits ); import "DPI-C" function int int2fp_case_load( - input int unsigned seed, input int unsigned index, input int unsigned bid ); import "DPI-C" function void int2fp_case_cmd(output int2fp_cmd_dpi_t cmd); import "DPI-C" function longint unsigned int2fp_case_src_word_lo(input int unsigned word_index); import "DPI-C" function longint unsigned int2fp_case_src_word_hi(input int unsigned word_index); +import "DPI-C" function int unsigned int2fp_case_dw_bits(input int unsigned index); -localparam int INT2FP_FUNCT7 = 7'd52; +`ifndef INT2FP_TENSOR_FUNCT7 +`error "INT2FP_TENSOR_FUNCT7 must be provided by the selected Core ballISA" +`endif +`ifndef INT2FP_CHANNEL_FUNCT7 +`error "INT2FP_CHANNEL_FUNCT7 must be provided by the selected Core ballISA" +`endif +localparam int INT2FP_TENSOR_CORE_FUNCT7 = `INT2FP_TENSOR_FUNCT7; +localparam int INT2FP_CHANNEL_CORE_FUNCT7 = `INT2FP_CHANNEL_FUNCT7; localparam int INT2FP_NUM_GROUPS = 4; localparam int INT2FP_MAX_ITER = 16; localparam int INT2FP_MAX_WORDS = INT2FP_MAX_ITER * INT2FP_NUM_GROUPS; localparam int INT2FP_TIMEOUT_CYCLES = 4000; -localparam int INT2FP_SEED = 32'hCAFE_BABE; -localparam int INT2FP_OUTPUT_FP32 = 0; -localparam int INT2FP_OUTPUT_INT8 = 1; function automatic int unsigned int2fp_require_bid(); int unsigned bid; diff --git a/examples/balls/int2fp/verify/src/common/int2fp_items.svh b/examples/balls/int2fp/verify/src/common/int2fp_items.svh index efa9682e3..c9f86cd94 100644 --- a/examples/balls/int2fp/verify/src/common/int2fp_items.svh +++ b/examples/balls/int2fp/verify/src/common/int2fp_items.svh @@ -1,13 +1,16 @@ class int2fp_cmd_item extends bb_blink_cmd_item; `uvm_object_utils(int2fp_cmd_item) - bit [31:0] scale_bits; - bit [31:0] output_mode; - bit [31:0] num_src_words; - bit [127:0] input_words[INT2FP_MAX_WORDS]; + bit [ 31:0] da_bits; + bit [ 31:0] dw_bits [INT2FP_NUM_GROUPS * 4]; + bit [ 12:0] da_addr; + bit [ 12:0] dw_addr; + bit per_channel; + bit [ 31:0] num_src_words; + bit [127:0] input_words [ INT2FP_MAX_WORDS]; constraint legal_c { - funct7 == INT2FP_FUNCT7[6:0]; + funct7 inside {INT2FP_TENSOR_CORE_FUNCT7[6:0], INT2FP_CHANNEL_CORE_FUNCT7[6:0]}; op1_en == 1'b1; op2_en == 1'b0; wr_spad_en == 1'b1; @@ -26,28 +29,26 @@ class int2fp_cmd_item extends bb_blink_cmd_item; super.new(name); endfunction - function bit is_i8(); - return (output_mode == INT2FP_OUTPUT_INT8) && (op1_col == 5'd4) && (wr_col == 5'd1); - endfunction - - function void load_rust_case(int unsigned seed, int unsigned index, int unsigned bid); + function void load_rust_case(int unsigned index, int unsigned bid); int2fp_cmd_dpi_t cmd; int unsigned rc; longint unsigned w_lo; longint unsigned w_hi; int i; - rc = int2fp_case_load(seed, index, bid); + rc = int2fp_case_load(index, bid); if (rc != 0) begin `uvm_fatal("CASE", $sformatf("int2fp_case_load returned %0d for index %0d", rc, index)) end int2fp_case_cmd(cmd); - bid = cmd.bid[4:0]; - funct7 = cmd.funct7[6:0]; + this.bid = cmd.bid[4:0]; iter = cmd.iter; - scale_bits = cmd.scale_bits; - output_mode = cmd.output_mode; + da_bits = cmd.da_bits; + da_addr = 13'd0; + dw_addr = cmd.dw_addr[12:0]; + per_channel = cmd.per_channel[0]; + funct7 = per_channel ? INT2FP_CHANNEL_CORE_FUNCT7[6:0] : INT2FP_TENSOR_CORE_FUNCT7[6:0]; op1_bank = cmd.op1_bank[4:0]; wr_bank = cmd.wr_bank[4:0]; op1_col = cmd.op1_col[4:0]; @@ -55,8 +56,7 @@ class int2fp_cmd_item extends bb_blink_cmd_item; rob_id = cmd.rob_id[3:0]; num_src_words = cmd.num_src_words; - // special[31:0]=scale, special[33:32]=output_mode (ISA / RTL contract) - special = {30'd0, output_mode[1:0], scale_bits}; + special = {38'd0, dw_addr, da_addr}; op1_en = 1'b1; op2_en = 1'b0; wr_spad_en = 1'b1; @@ -73,21 +73,22 @@ class int2fp_cmd_item extends bb_blink_cmd_item; if (num_src_words == 0 || num_src_words > INT2FP_MAX_WORDS) begin `uvm_fatal("CASE", $sformatf("num_src_words out of range: %0d", num_src_words)) end - if (!((output_mode == INT2FP_OUTPUT_FP32 && op1_col == 5'd1 && wr_col == 5'd1) || - (output_mode == INT2FP_OUTPUT_INT8 && op1_col == 5'd4 && wr_col == 5'd1))) begin - `uvm_fatal("CASE", $sformatf("unsupported mode/layout mode=%0d op1_col=%0d wr_col=%0d", - output_mode, op1_col, wr_col)) + if (op1_col == 0 || op1_col > INT2FP_NUM_GROUPS || wr_col != op1_col) begin + `uvm_fatal("CASE", $sformatf("unsupported layout op1_col=%0d wr_col=%0d", op1_col, wr_col)) end if (op1_bank == wr_bank) begin `uvm_fatal("CASE", "op1_bank and wr_bank overlap") end - if (funct7 != INT2FP_FUNCT7[6:0]) begin + if (dw_addr < 13'd16 || dw_addr[1:0] != 2'd0) begin + `uvm_fatal("CASE", $sformatf("invalid Dw address: %0d", dw_addr)) + end + if (funct7 != (per_channel ? INT2FP_CHANNEL_CORE_FUNCT7[6:0] + : INT2FP_TENSOR_CORE_FUNCT7[6:0])) begin `uvm_fatal("CASE", $sformatf("invalid funct7: %0d", funct7)) end - if (output_mode > INT2FP_OUTPUT_INT8) begin - `uvm_fatal("CASE", $sformatf("reserved output_mode: %0d", output_mode)) + for (i = 0; i < INT2FP_NUM_GROUPS * 4; i++) begin + dw_bits[i] = int2fp_case_dw_bits(i); end - for (i = 0; i < num_src_words; i++) begin w_lo = int2fp_case_src_word_lo(i); w_hi = int2fp_case_src_word_hi(i); @@ -101,8 +102,13 @@ class int2fp_cmd_item extends bb_blink_cmd_item; if (!$cast(rhs_, rhs)) begin `uvm_fatal("COPY", "rhs is not int2fp_cmd_item") end - scale_bits = rhs_.scale_bits; - output_mode = rhs_.output_mode; + da_bits = rhs_.da_bits; + for (int i = 0; i < INT2FP_NUM_GROUPS * 4; i++) begin + dw_bits[i] = rhs_.dw_bits[i]; + end + da_addr = rhs_.da_addr; + dw_addr = rhs_.dw_addr; + per_channel = rhs_.per_channel; num_src_words = rhs_.num_src_words; for (int i = 0; i < INT2FP_MAX_WORDS; i++) begin input_words[i] = rhs_.input_words[i]; diff --git a/examples/balls/int2fp/verify/src/cov/int2fp_cov.svh b/examples/balls/int2fp/verify/src/cov/int2fp_cov.svh index 2c6008c9a..0c5e4e093 100644 --- a/examples/balls/int2fp/verify/src/cov/int2fp_cov.svh +++ b/examples/balls/int2fp/verify/src/cov/int2fp_cov.svh @@ -3,37 +3,19 @@ class int2fp_cov extends uvm_component; uvm_analysis_imp_cmd #(bb_blink_cmd_item, int2fp_cov) cmd_imp; - covergroup cmd_cg; - coverpoint cur_mode {bins fp32 = {0}; bins i8 = {1};} - coverpoint cur_iter {bins it1 = {1}; bins it2 = {2}; bins it4 = {4}; bins it16 = {16};} - endgroup - - int unsigned cur_mode; - int unsigned cur_iter; int cmd_count; function new(string name, uvm_component parent); super.new(name, parent); cmd_imp = new("cmd_imp", this); - cmd_cg = new(); endfunction function void write_cmd(bb_blink_cmd_item item); - cur_mode = int'(item.special[33:32]); - cur_iter = int'(item.iter); cmd_count++; - cmd_cg.sample(); endfunction function void check_phase(uvm_phase phase); super.check_phase(phase); if (cmd_count == 0) `uvm_fatal("COV", "no cmd observed") - if (cmd_cg.get_coverage() != 100.0) begin - `uvm_fatal( - "COV", - $sformatf( - "int2fp_cov coverage incomplete: %0.2f%% (need mode {fp32,i8} and iter {1,2,4,16})", - cmd_cg.get_coverage())) - end endfunction endclass diff --git a/examples/balls/int2fp/verify/src/env/int2fp_env.svh b/examples/balls/int2fp/verify/src/env/int2fp_env.svh index 7aa974720..66a249530 100644 --- a/examples/balls/int2fp/verify/src/env/int2fp_env.svh +++ b/examples/balls/int2fp/verify/src/env/int2fp_env.svh @@ -21,6 +21,7 @@ class int2fp_env extends bb_blink_env #(1, 1); cmd_agent.cmd_ap.connect(scb.cmd_imp); cmd_agent.cmd_ap.connect(icov.cmd_imp); read_mon.read_ap.connect(scb.read_imp); + mmio_read_mon.read_ap.connect(scb.mmio_read_imp); write_mon.write_ap.connect(scb.write_imp); resp_mon.resp_ap.connect(scb.resp_imp); endfunction diff --git a/examples/balls/int2fp/verify/src/env/int2fp_scoreboard.svh b/examples/balls/int2fp/verify/src/env/int2fp_scoreboard.svh index d0888ff79..768569d96 100644 --- a/examples/balls/int2fp/verify/src/env/int2fp_scoreboard.svh +++ b/examples/balls/int2fp/verify/src/env/int2fp_scoreboard.svh @@ -6,33 +6,34 @@ class int2fp_scoreboard extends uvm_scoreboard; uvm_analysis_imp_read #(bb_blink_read_item, int2fp_scoreboard) read_imp; uvm_analysis_imp_write #(bb_blink_write_item, int2fp_scoreboard) write_imp; uvm_analysis_imp_resp #(bb_blink_resp_item, int2fp_scoreboard) resp_imp; + uvm_analysis_imp_mmio_read #(bb_mmio_read_item, int2fp_scoreboard) mmio_read_imp; bb_blink_mem_model #(1, 1) mem_model; int2fp_cmd_item stim_q[$]; - bit [127:0] expected_words[INT2FP_MAX_ITER]; + bit [127:0] expected_words[INT2FP_MAX_WORDS]; int unsigned expected_reads; int unsigned expected_writes; int unsigned cmd_count; int unsigned read_count; int unsigned write_count; int unsigned resp_count; - int unsigned expect_group; + int unsigned mmio_read_count; function new(string name, uvm_component parent); super.new(name, parent); - stim_imp = new("stim_imp", this); - cmd_imp = new("cmd_imp", this); - read_imp = new("read_imp", this); + stim_imp = new("stim_imp", this); + cmd_imp = new("cmd_imp", this); + read_imp = new("read_imp", this); write_imp = new("write_imp", this); - resp_imp = new("resp_imp", this); + resp_imp = new("resp_imp", this); + mmio_read_imp = new("mmio_read_imp", this); endfunction function void write_stim(bb_blink_cmd_item item); int2fp_cmd_item fitem; int2fp_cmd_item clone; int i; - int row; int group; if (stim_q.size() != 0) begin @@ -50,45 +51,35 @@ class int2fp_scoreboard extends uvm_scoreboard; `uvm_fatal("SCB", "mem_model handle not set") end mem_model.clear_mem(); - if (clone.is_i8()) begin - for (row = 0; row < clone.iter; row++) begin - for (group = 0; group < INT2FP_NUM_GROUPS; group++) begin - mem_model.load_word_g(int'(clone.op1_bank), group, row, - clone.input_words[row*INT2FP_NUM_GROUPS+group]); - end + mem_model.load_mmio_word(int'(clone.da_addr), clone.da_bits); + if (clone.per_channel) begin + for (i = 0; i < clone.op1_col * 4; i++) begin + mem_model.load_mmio_word(int'(clone.dw_addr) + i * 4, clone.dw_bits[i]); end end else begin - for (i = 0; i < clone.iter; i++) begin - mem_model.load_word(int'(clone.op1_bank), i, clone.input_words[i]); + mem_model.load_mmio_word(int'(clone.dw_addr), clone.dw_bits[0]); + end + for (i = 0; i < clone.iter; i++) begin + for (group = 0; group < clone.op1_col; group++) begin + mem_model.load_word_g(int'(clone.op1_bank), group, i, + clone.input_words[i*clone.op1_col+group]); end end mem_model.arm(); build_expected(clone); - expected_writes = clone.iter; - expected_reads = clone.is_i8() ? (clone.iter * INT2FP_NUM_GROUPS) : clone.iter; - expect_group = 0; + expected_writes = clone.iter * clone.op1_col; + expected_reads = clone.iter * clone.op1_col; endfunction function void build_expected(int2fp_cmd_item item); - if (item.is_i8()) begin - for (int row = 0; row < item.iter; row++) begin - bit [127:0] packed_word = '0; - for (int group = 0; group < INT2FP_NUM_GROUPS; group++) begin - bit [127:0] src = item.input_words[row*INT2FP_NUM_GROUPS+group]; - for (int lane = 0; lane < 4; lane++) begin - int signed v = $signed(src[lane*32+:32]); - bit [7:0] q = int2fp_ref_i8(v, item.scale_bits); - packed_word[group*32+lane*8+:8] = q; - end - end - expected_words[row] = packed_word; - end - end else begin - for (int w = 0; w < item.iter; w++) begin + for (int w = 0; w < item.iter; w++) begin + for (int group = 0; group < item.op1_col; group++) begin for (int e = 0; e < 4; e++) begin - int signed v = $signed(item.input_words[w][e*32+:32]); - expected_words[w][e*32+:32] = int2fp_ref_fp32(v, item.scale_bits); + int signed v = $signed(item.input_words[w*item.op1_col+group][e*32+:32]); + int scale_index = item.per_channel ? group * 4 + e : 0; + expected_words[w*item.op1_col+group][e*32+:32] = + int2fp_ref_fp32(v, item.da_bits, item.dw_bits[scale_index]); end end end @@ -108,12 +99,12 @@ class int2fp_scoreboard extends uvm_scoreboard; `uvm_fatal("CMD", $sformatf("funct7 mismatch: got %0d exp %0d", got.funct7, exp.funct7)) if (got.iter !== exp.iter) `uvm_fatal("CMD", $sformatf("iter mismatch: got %0d exp %0d", got.iter, exp.iter)) - if (got.special[31:0] !== exp.scale_bits) - `uvm_fatal("CMD", $sformatf( - "scale mismatch: got 0x%08h exp 0x%08h", got.special[31:0], exp.scale_bits)) - if (got.special[33:32] !== exp.output_mode[1:0]) - `uvm_fatal("CMD", $sformatf( - "output_mode mismatch: got %0d exp %0d", got.special[33:32], exp.output_mode)) + if (got.special[12:0] !== exp.da_addr || + got.special[25:13] !== exp.dw_addr || got.special[63:26] !== '0) + `uvm_fatal("CMD", "Int2Fp special must contain only MMIO scale metadata") + if (got.funct7 !== (exp.per_channel ? INT2FP_CHANNEL_CORE_FUNCT7[6:0] + : INT2FP_TENSOR_CORE_FUNCT7[6:0])) + `uvm_fatal("CMD", "Int2Fp funct7 does not match scale mode") if (got.op1_bank !== exp.op1_bank || got.wr_bank !== exp.wr_bank) `uvm_fatal("CMD", "bank field mismatch") if (got.op1_col !== exp.op1_col || got.wr_col !== exp.wr_col) @@ -132,20 +123,8 @@ class int2fp_scoreboard extends uvm_scoreboard; if (item.rob_id !== stim.rob_id) `uvm_fatal("READ", $sformatf("rob_id mismatch: got %0d exp %0d", item.rob_id, stim.rob_id)) - if (stim.is_i8()) begin - expect_addr = read_count / INT2FP_NUM_GROUPS; - if (item.group_id !== expect_group[4:0]) - `uvm_fatal("READ", $sformatf("group mismatch: got %0d exp %0d", item.group_id, expect_group - )) - if (item.addr !== expect_addr[9:0]) - `uvm_fatal("READ", $sformatf("addr mismatch: got %0d exp %0d", item.addr, expect_addr)) - expect_group = (expect_group + 1) % INT2FP_NUM_GROUPS; - end else begin - if (item.group_id !== 5'd0) - `uvm_fatal("READ", $sformatf("group mismatch: got %0d exp 0", item.group_id)) - if (item.addr !== read_count[9:0]) - `uvm_fatal("READ", $sformatf("addr mismatch: got %0d exp %0d", item.addr, read_count)) - end + if (item.group_id !== (read_count % stim.op1_col)) `uvm_fatal("READ", "group mismatch") + if (item.addr !== (read_count / stim.op1_col)) `uvm_fatal("READ", "addr mismatch") read_count++; endfunction @@ -158,22 +137,31 @@ class int2fp_scoreboard extends uvm_scoreboard; `uvm_fatal("WRITE", $sformatf("bank mismatch: got %0d exp %0d", item.bank_id, stim.wr_bank)) if (item.rob_id !== stim.rob_id) `uvm_fatal("WRITE", $sformatf("rob_id mismatch: got %0d exp %0d", item.rob_id, stim.rob_id)) - if (item.group_id !== '0) - `uvm_fatal("WRITE", $sformatf("group_id mismatch: got %0d exp 0", item.group_id)) - if (item.addr !== write_count[9:0]) - `uvm_fatal("WRITE", $sformatf("addr mismatch: got %0d exp %0d", item.addr, write_count)) + if (item.group_id !== (write_count % stim.op1_col)) `uvm_fatal("WRITE", "group mismatch") + if (item.addr !== (write_count / stim.op1_col)) `uvm_fatal("WRITE", "addr mismatch") if (item.mask !== 16'hFFFF) `uvm_fatal("WRITE", $sformatf("mask mismatch: got 0x%04h", item.mask)) - if (item.data !== expected_words[item.addr]) + if (item.data !== expected_words[item.addr*stim.op1_col+item.group_id]) `uvm_fatal("SCB", $sformatf( "data mismatch at addr %0d: got 0x%032h exp 0x%032h", item.addr, item.data, - expected_words[item.addr] + expected_words[item.addr*stim.op1_col+item.group_id] )) write_count++; endfunction + function void write_mmio_read(bb_mmio_read_item item); + int2fp_cmd_item stim; + int unsigned expected_addr; + stim = current_stim("MMIO_READ"); + if (mmio_read_count < 4) expected_addr = stim.da_addr + mmio_read_count; + else if (!stim.per_channel) expected_addr = stim.dw_addr + (mmio_read_count - 4); + else expected_addr = stim.dw_addr + ((mmio_read_count - 4) % (stim.op1_col * 16)); + if (item.addr !== expected_addr) `uvm_fatal("MMIO", "scale read address mismatch") + mmio_read_count++; + endfunction + function void write_resp(bb_blink_resp_item item); int2fp_cmd_item stim; @@ -198,7 +186,8 @@ class int2fp_scoreboard extends uvm_scoreboard; return cmd_count == 1 && read_count == expected_reads && write_count == expected_writes && - resp_count == 1; + resp_count == 1 && mmio_read_count == + (stim_q[0].per_channel ? 4 + expected_reads * 16 : 8); endfunction function void reset_counters(); @@ -207,9 +196,9 @@ class int2fp_scoreboard extends uvm_scoreboard; read_count = 0; write_count = 0; resp_count = 0; + mmio_read_count = 0; expected_reads = 0; expected_writes = 0; - expect_group = 0; endfunction function void check_phase(uvm_phase phase); diff --git a/examples/balls/int2fp/verify/src/pkg/int2fp_pkg.sv b/examples/balls/int2fp/verify/src/pkg/int2fp_pkg.sv index 6e97aeb10..ee390406b 100644 --- a/examples/balls/int2fp/verify/src/pkg/int2fp_pkg.sv +++ b/examples/balls/int2fp/verify/src/pkg/int2fp_pkg.sv @@ -9,5 +9,10 @@ package int2fp_pkg; `include "cov/int2fp_cov.svh" `include "env/int2fp_scoreboard.svh" `include "env/int2fp_env.svh" - `include "tests/int2fp_test.svh" + `include "tests/int2fp_case_test.svh" + `include "tests/int2fp_tensor_rows_test.svh" + `include "tests/int2fp_channel_lanes_test.svh" + `include "tests/int2fp_tensor_groups_test.svh" + `include "tests/int2fp_channel_base_test.svh" + `include "tests/int2fp_channel_two_rows_test.svh" endpackage diff --git a/examples/balls/int2fp/verify/src/seq/int2fp_sequences.svh b/examples/balls/int2fp/verify/src/seq/int2fp_sequences.svh index 023f9b2f0..0bea8e3e2 100644 --- a/examples/balls/int2fp/verify/src/seq/int2fp_sequences.svh +++ b/examples/balls/int2fp/verify/src/seq/int2fp_sequences.svh @@ -2,19 +2,17 @@ class int2fp_basic_seq extends uvm_sequence #(bb_blink_cmd_item); `uvm_object_utils(int2fp_basic_seq) int unsigned case_index; - int unsigned seed; int unsigned bid; function new(string name = "int2fp_basic_seq"); super.new(name); - seed = INT2FP_SEED; endfunction task body(); int2fp_cmd_item item; item = int2fp_cmd_item::type_id::create("item"); start_item(item); - item.load_rust_case(seed, case_index, bid); + item.load_rust_case(case_index, bid); finish_item(item); endtask endclass diff --git a/examples/balls/int2fp/verify/src/tb_top.sv b/examples/balls/int2fp/verify/src/tb_top.sv index 0d57159c8..82fb6bf69 100644 --- a/examples/balls/int2fp/verify/src/tb_top.sv +++ b/examples/balls/int2fp/verify/src/tb_top.sv @@ -71,9 +71,38 @@ module tb_top; .io_bankWrite_0_io_resp_valid(intf.bank_write_resp_valid[0]), .io_bankWrite_0_io_resp_bits_ok(intf.bank_write_resp_ok[0]), .io_subRobReq_ready(intf.sub_rob_req_ready), - .io_mmioRead_req_ready(intf.mmio_read_req_ready), - .io_mmioRead_resp_valid(intf.mmio_read_resp_valid), - .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data) + .io_mmioRead_0_ball_id(), + .io_mmioRead_0_rob_id(), + .io_mmioRead_0_req_ready(intf.mmio_read_req_ready[0]), + .io_mmioRead_0_req_valid(intf.mmio_read_req_valid[0]), + .io_mmioRead_0_req_bits_addr(intf.mmio_read_req_addr[0]), + .io_mmioRead_0_resp_ready(intf.mmio_read_resp_ready[0]), + .io_mmioRead_0_resp_valid(intf.mmio_read_resp_valid[0]), + .io_mmioRead_0_resp_bits_data(intf.mmio_read_resp_bits_data[0]), + .io_mmioRead_1_ball_id(), + .io_mmioRead_1_rob_id(), + .io_mmioRead_1_req_ready(intf.mmio_read_req_ready[1]), + .io_mmioRead_1_req_valid(intf.mmio_read_req_valid[1]), + .io_mmioRead_1_req_bits_addr(intf.mmio_read_req_addr[1]), + .io_mmioRead_1_resp_ready(intf.mmio_read_resp_ready[1]), + .io_mmioRead_1_resp_valid(intf.mmio_read_resp_valid[1]), + .io_mmioRead_1_resp_bits_data(intf.mmio_read_resp_bits_data[1]), + .io_mmioRead_2_ball_id(), + .io_mmioRead_2_rob_id(), + .io_mmioRead_2_req_ready(intf.mmio_read_req_ready[2]), + .io_mmioRead_2_req_valid(intf.mmio_read_req_valid[2]), + .io_mmioRead_2_req_bits_addr(intf.mmio_read_req_addr[2]), + .io_mmioRead_2_resp_ready(intf.mmio_read_resp_ready[2]), + .io_mmioRead_2_resp_valid(intf.mmio_read_resp_valid[2]), + .io_mmioRead_2_resp_bits_data(intf.mmio_read_resp_bits_data[2]), + .io_mmioRead_3_ball_id(), + .io_mmioRead_3_rob_id(), + .io_mmioRead_3_req_ready(intf.mmio_read_req_ready[3]), + .io_mmioRead_3_req_valid(intf.mmio_read_req_valid[3]), + .io_mmioRead_3_req_bits_addr(intf.mmio_read_req_addr[3]), + .io_mmioRead_3_resp_ready(intf.mmio_read_resp_ready[3]), + .io_mmioRead_3_resp_valid(intf.mmio_read_resp_valid[3]), + .io_mmioRead_3_resp_bits_data(intf.mmio_read_resp_bits_data[3]) ); initial begin @@ -83,6 +112,6 @@ module tb_top; initial begin uvm_config_db#(virtual bb_blink_if #(1, 1))::set(null, "*", "vif", intf); - run_test("int2fp_ball_test"); + run_test(); end endmodule diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_case_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_case_test.svh new file mode 100644 index 000000000..e988d5311 --- /dev/null +++ b/examples/balls/int2fp/verify/src/tests/int2fp_case_test.svh @@ -0,0 +1,45 @@ +virtual class int2fp_case_test extends uvm_test; + typedef virtual bb_blink_if #(1, 1) vif_t; + vif_t vif; + int2fp_env env; + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function void build_phase(uvm_phase phase); + super.build_phase(phase); + if (!uvm_config_db#(vif_t)::get(this, "", "vif", vif)) + `uvm_fatal("NOVIF", "bb_blink_if not found") + env = int2fp_env::type_id::create("env", this); + endfunction + + pure virtual function int unsigned case_index(); + pure virtual function string case_label(); + + task run_phase(uvm_phase phase); + int2fp_basic_seq seq; + int cycles; + + phase.raise_objection(this); + vif.reset = 1'b1; + repeat (5) @(posedge vif.clock); + vif.reset = 1'b0; + repeat (2) @(posedge vif.clock); + env.scb.reset_counters(); + seq = int2fp_basic_seq::type_id::create("seq"); + seq.case_index = case_index(); + seq.bid = int2fp_require_bid(); + seq.start(env.cmd_agent.seqr); + + cycles = 0; + while (!env.scb.done()) begin + @(posedge vif.clock); + cycles++; + if (cycles > INT2FP_TIMEOUT_CYCLES) + `uvm_fatal("TIMEOUT", $sformatf("Int2FpBall %s timeout", case_label())) + end + `uvm_info("INT2FP", $sformatf("Int2FpBall %s passed", case_label()), UVM_LOW) + phase.drop_objection(this); + endtask +endclass diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_channel_base_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_channel_base_test.svh new file mode 100644 index 000000000..a59d75dc4 --- /dev/null +++ b/examples/balls/int2fp/verify/src/tests/int2fp_channel_base_test.svh @@ -0,0 +1,15 @@ +class int2fp_channel_base_test extends int2fp_case_test; + `uvm_component_utils(int2fp_channel_base_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 3; + endfunction + + function string case_label(); + return "CHANNEL_BASE_64"; + endfunction +endclass diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_channel_lanes_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_channel_lanes_test.svh new file mode 100644 index 000000000..4245584c8 --- /dev/null +++ b/examples/balls/int2fp/verify/src/tests/int2fp_channel_lanes_test.svh @@ -0,0 +1,15 @@ +class int2fp_channel_lanes_test extends int2fp_case_test; + `uvm_component_utils(int2fp_channel_lanes_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 1; + endfunction + + function string case_label(); + return "CHANNEL_4_GROUPS"; + endfunction +endclass diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_channel_two_rows_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_channel_two_rows_test.svh new file mode 100644 index 000000000..edd62ef40 --- /dev/null +++ b/examples/balls/int2fp/verify/src/tests/int2fp_channel_two_rows_test.svh @@ -0,0 +1,15 @@ +class int2fp_channel_two_rows_test extends int2fp_case_test; + `uvm_component_utils(int2fp_channel_two_rows_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 4; + endfunction + + function string case_label(); + return "CHANNEL_TWO_ROWS"; + endfunction +endclass diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_tensor_groups_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_tensor_groups_test.svh new file mode 100644 index 000000000..0ea37787a --- /dev/null +++ b/examples/balls/int2fp/verify/src/tests/int2fp_tensor_groups_test.svh @@ -0,0 +1,15 @@ +class int2fp_tensor_groups_test extends int2fp_case_test; + `uvm_component_utils(int2fp_tensor_groups_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 2; + endfunction + + function string case_label(); + return "TENSOR_4_GROUPS"; + endfunction +endclass diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_tensor_rows_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_tensor_rows_test.svh new file mode 100644 index 000000000..bc9bb195c --- /dev/null +++ b/examples/balls/int2fp/verify/src/tests/int2fp_tensor_rows_test.svh @@ -0,0 +1,15 @@ +class int2fp_tensor_rows_test extends int2fp_case_test; + `uvm_component_utils(int2fp_tensor_rows_test) + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function int unsigned case_index(); + return 0; + endfunction + + function string case_label(); + return "TENSOR_16_ROWS"; + endfunction +endclass diff --git a/examples/balls/int2fp/verify/src/tests/int2fp_test.svh b/examples/balls/int2fp/verify/src/tests/int2fp_test.svh deleted file mode 100644 index 7f248961c..000000000 --- a/examples/balls/int2fp/verify/src/tests/int2fp_test.svh +++ /dev/null @@ -1,77 +0,0 @@ -class int2fp_ball_test extends uvm_test; - `uvm_component_utils(int2fp_ball_test) - - typedef virtual bb_blink_if #(1, 1) vif_t; - vif_t vif; - int2fp_env env; - - function new(string name, uvm_component parent); - super.new(name, parent); - endfunction - - function void build_phase(uvm_phase phase); - super.build_phase(phase); - if (!uvm_config_db#(vif_t)::get(this, "", "vif", vif)) begin - `uvm_fatal("NOVIF", "bb_blink_if not found") - end - env = int2fp_env::type_id::create("env", this); - endfunction - - task run_phase(uvm_phase phase); - int unsigned bid; - bid = int2fp_require_bid(); - phase.raise_objection(this); - run_directed(bid); - run_random(bid); - phase.drop_objection(this); - endtask - - task run_directed(int unsigned bid); - run_case(0, "DIR_FP32", bid); - run_case(1, "DIR_I8", bid); - endtask - - task run_random(int unsigned bid); - int i; - for (i = 2; i <= 21; i++) begin - run_case(i, $sformatf("RND_%0d", i), bid); - end - endtask - - task run_case(int unsigned idx, string label, int unsigned bid); - int2fp_basic_seq seq; - int cycles; - - apply_reset(); - env.scb.reset_counters(); - - seq = int2fp_basic_seq::type_id::create("seq"); - seq.case_index = idx; - seq.bid = bid; - seq.start(env.cmd_agent.seqr); - - cycles = 0; - while (!env.scb.done()) begin - @(posedge vif.clock); - cycles++; - if (cycles > INT2FP_TIMEOUT_CYCLES) begin - `uvm_fatal( - "TIMEOUT", - $sformatf( - "Int2FpBall %s (idx=%0d) timeout: cmds=%0d reads=%0d/%0d writes=%0d/%0d resp=%0d", - label, idx, env.scb.cmd_count, env.scb.read_count, env.scb.expected_reads, - env.scb.write_count, env.scb.expected_writes, env.scb.resp_count)) - end - end - - `uvm_info("INT2FP", $sformatf("Int2FpBall %s (idx=%0d) passed in %0d cycles", label, idx, - cycles), UVM_LOW) - endtask - - task apply_reset(); - vif.reset = 1'b1; - repeat (5) @(posedge vif.clock); - vif.reset = 1'b0; - repeat (2) @(posedge vif.clock); - endtask -endclass diff --git a/examples/balls/int2fp/workloads/ctests/CMakeLists.txt b/examples/balls/int2fp/workloads/ctests/CMakeLists.txt index dabae42e8..27a32c4b6 100644 --- a/examples/balls/int2fp/workloads/ctests/CMakeLists.txt +++ b/examples/balls/int2fp/workloads/ctests/CMakeLists.txt @@ -1,12 +1,8 @@ add_buckyball_ctests( - int2fp_int32_to_fp32_test.c - int2fp_int8_to_fp32_test.c - int2fp_int32_to_int8_test.c - int2fp_int32_to_fp32_col2_test.c - int2fp_int32_to_fp32_col4_test.c - int2fp_bank_int32_to_fp32_test.c - int2fp_bank_int8_to_fp32_test.c - int2fp_bank_int32_to_int8_test.c - int2fp_bank_int32_to_fp32_col2_test.c - int2fp_bank_int32_to_fp32_col4_test.c + int2fp_tensor_values_test.c + int2fp_tensor_groups_test.c + int2fp_tensor_rows_test.c + int2fp_channel_lanes_test.c + int2fp_channel_base_test.c + int2fp_channel_two_rows_test.c ) diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_col2_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_col2_test.c deleted file mode 100644 index fb5595edc..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_col2_test.c +++ /dev/null @@ -1,72 +0,0 @@ -#include "buckyball.h" -#include "int2fp_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 8 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_1_0 0x3F800000U -#define SEED 0x55555555U - -static int32_t input_i32[NELEM] __attribute__((aligned(64))); -static float output_fp32[NELEM] __attribute__((aligned(64))); -static uint32_t expected_bits[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_1_0); - - for (int i = 0; i < NELEM; ++i) { - int32_t v = (int32_t)(((uint32_t)rand() << 16) ^ (uint32_t)rand()); - input_i32[i] = v; - expected_bits[i] = int2fp_fp32(v, SCALE_1_0); - output_fp32[i] = 0.0f; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 2); - bb_mem_alloc(dst, BANK_LINES, 2); - bb_mvin((uintptr_t)input_i32, src, BANK_LINES, 1); - bb_int2fp(src, dst, BANK_LINES, SCALE_1_0); - bb_mvout((uintptr_t)output_fp32, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - uint32_t got = f32_to_bits(output_fp32[i]); - if (got != expected_bits[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, got, - expected_bits[i]); - passed = 0; - } - double ideal = (double)input_i32[i] * scale; - double err = (double)output_fp32[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Int2Fp bank INT32-to-FP32 col2 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_col4_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_col4_test.c deleted file mode 100644 index 260086c48..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_col4_test.c +++ /dev/null @@ -1,72 +0,0 @@ -#include "buckyball.h" -#include "int2fp_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 16 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_1_0 0x3F800000U -#define SEED 0x44444444U - -static int32_t input_i32[NELEM] __attribute__((aligned(64))); -static float output_fp32[NELEM] __attribute__((aligned(64))); -static uint32_t expected_bits[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_1_0); - - for (int i = 0; i < NELEM; ++i) { - int32_t v = (int32_t)(((uint32_t)rand() << 16) ^ (uint32_t)rand()); - input_i32[i] = v; - expected_bits[i] = int2fp_fp32(v, SCALE_1_0); - output_fp32[i] = 0.0f; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 4); - bb_mem_alloc(dst, BANK_LINES, 4); - bb_mvin((uintptr_t)input_i32, src, BANK_LINES, 1); - bb_int2fp(src, dst, BANK_LINES, SCALE_1_0); - bb_mvout((uintptr_t)output_fp32, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - uint32_t got = f32_to_bits(output_fp32[i]); - if (got != expected_bits[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, got, - expected_bits[i]); - passed = 0; - } - double ideal = (double)input_i32[i] * scale; - double err = (double)output_fp32[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Int2Fp bank INT32-to-FP32 col4 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_test.c deleted file mode 100644 index f54a19fe1..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_fp32_test.c +++ /dev/null @@ -1,72 +0,0 @@ -#include "buckyball.h" -#include "int2fp_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 4 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_1_0 0x3F800000U -#define SEED 0x11111111U - -static int32_t input_i32[NELEM] __attribute__((aligned(64))); -static float output_fp32[NELEM] __attribute__((aligned(64))); -static uint32_t expected_bits[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_1_0); - - for (int i = 0; i < NELEM; ++i) { - int32_t v = (int32_t)(((uint32_t)rand() << 16) ^ (uint32_t)rand()); - input_i32[i] = v; - expected_bits[i] = int2fp_fp32(v, SCALE_1_0); - output_fp32[i] = 0.0f; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 1); - bb_mem_alloc(dst, BANK_LINES, 1); - bb_mvin((uintptr_t)input_i32, src, BANK_LINES, 1); - bb_int2fp(src, dst, BANK_LINES, SCALE_1_0); - bb_mvout((uintptr_t)output_fp32, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - uint32_t got = f32_to_bits(output_fp32[i]); - if (got != expected_bits[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, got, - expected_bits[i]); - passed = 0; - } - double ideal = (double)input_i32[i] * scale; - double err = (double)output_fp32[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Int2Fp bank INT32-to-FP32 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_int8_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_int8_test.c deleted file mode 100644 index 37785e867..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int32_to_int8_test.c +++ /dev/null @@ -1,71 +0,0 @@ -#include "buckyball.h" -#include "int2fp_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 16 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_0_5 0x3F000000U -#define SEED 0x33333333U - -static int32_t input_i32[NELEM] __attribute__((aligned(64))); -static int8_t output_i8[NELEM] __attribute__((aligned(64))); -static int8_t expected_i8[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_0_5); - - for (int i = 0; i < NELEM; ++i) { - int32_t v = (int32_t)(rand() % 1024) - 512; - input_i32[i] = v; - expected_i8[i] = int2fp_i8(v, SCALE_0_5); - output_i8[i] = (int8_t)0x5a; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 4); - bb_mem_alloc(dst, BANK_LINES, 1); - bb_mvin((uintptr_t)input_i32, src, BANK_LINES, 1); - bb_int32_to_int8(src, dst, BANK_LINES, SCALE_0_5); - bb_mvout((uintptr_t)output_i8, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - if (output_i8[i] != expected_i8[i]) { - printf("MISMATCH at [%d]: got %d, expected %d\n", i, output_i8[i], - expected_i8[i]); - passed = 0; - } - double ideal = (double)input_i32[i] * scale; - double err = (double)output_i8[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Int2Fp bank INT32-to-INT8 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int8_to_fp32_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_bank_int8_to_fp32_test.c deleted file mode 100644 index 176ff30f9..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_bank_int8_to_fp32_test.c +++ /dev/null @@ -1,72 +0,0 @@ -#include "buckyball.h" -#include "int2fp_ref.h" -#include -#include -#include -#include -#include - -#define ELEMS_PER_ROW 16 -#define NELEM (BANK_LINES * ELEMS_PER_ROW) -#define SCALE_1_0 0x3F800000U -#define SEED 0x22222222U - -static int8_t input_i8[NELEM] __attribute__((aligned(64))); -static float output_fp32[NELEM] __attribute__((aligned(64))); -static uint32_t expected_bits[NELEM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - srand(SEED); - double scale = (double)bits_to_f32(SCALE_1_0); - - for (int i = 0; i < NELEM; ++i) { - int8_t v = (int8_t)((rand() % 256) - 128); - input_i8[i] = v; - expected_bits[i] = int2fp_fp32((int32_t)v, SCALE_1_0); - output_fp32[i] = 0.0f; - } - - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, BANK_LINES, 1); - bb_mem_alloc(dst, BANK_LINES, 4); - bb_mvin((uintptr_t)input_i8, src, BANK_LINES, 1); - bb_int2fp(src, dst, BANK_LINES, SCALE_1_0); - bb_mvout((uintptr_t)output_fp32, dst, BANK_LINES, 1); - bb_fence(); - - int passed = 1; - double max_abs = 0.0; - double sum_abs = 0.0; - double max_rel = 0.0; - for (int i = 0; i < NELEM; ++i) { - uint32_t got = f32_to_bits(output_fp32[i]); - if (got != expected_bits[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, got, - expected_bits[i]); - passed = 0; - } - double ideal = (double)input_i8[i] * scale; - double err = (double)output_fp32[i] - ideal; - if (err < 0.0) - err = -err; - if (err > max_abs) - max_abs = err; - sum_abs += err; - double denom = ideal < 0.0 ? -ideal : ideal; - if (denom > 1.0e-12) { - double rel = err / denom; - if (rel > max_rel) - max_rel = rel; - } - } - - printf("LOSS n=%d max_abs=%.6f mean_abs=%.6f max_rel=%.6f\n", NELEM, max_abs, - sum_abs / (double)NELEM, max_rel); - printf("Int2Fp bank INT8-to-FP32 %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_channel_base_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_channel_base_test.c new file mode 100644 index 000000000..2184a1a36 --- /dev/null +++ b/examples/balls/int2fp/workloads/ctests/int2fp_channel_base_test.c @@ -0,0 +1,44 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static int32_t input[16] __attribute__((aligned(64))) = { + 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8}; +static float output[16] __attribute__((aligned(64))); +static float da[4] __attribute__((aligned(64))) = {0.5f}; +static float dw[16] __attribute__((aligned(64))) = { + 0.25f, 0.5f, 0.75f, 1.0f, 1.25f, 1.5f, 1.75f, 2.0f, + 0.125f, 0.25f, 0.375f, 0.5f, 2.25f, 2.5f, 2.75f, 3.0f, +}; +static const float expected[16] = { + 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, + 0.5f, 1.0f, 1.5f, 2.0f, 9.0f, 10.0f, 11.0f, 12.0f, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 64, 4, 16); + bb_mem_alloc(0, 1, 4); + bb_mem_alloc(1, 1, 4); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_int2fp_channel(0, 1, 1, 0, 64); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("int2fp_channel_base i=%d got=%f exp=%f\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("int2fp_channel_base PASS\n"); + return 0; +} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_channel_lanes_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_channel_lanes_test.c new file mode 100644 index 000000000..5a48269b5 --- /dev/null +++ b/examples/balls/int2fp/workloads/ctests/int2fp_channel_lanes_test.c @@ -0,0 +1,44 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static int32_t input[16] __attribute__((aligned(64))) = { + 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8}; +static float output[16] __attribute__((aligned(64))); +static float da[4] __attribute__((aligned(64))) = {0.5f}; +static float dw[16] __attribute__((aligned(64))) = { + 0.25f, 0.5f, 0.75f, 1.0f, 1.25f, 1.5f, 1.75f, 2.0f, + 0.125f, 0.25f, 0.375f, 0.5f, 2.25f, 2.5f, 2.75f, 3.0f, +}; +static const float expected[16] = { + 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, + 0.5f, 1.0f, 1.5f, 2.0f, 9.0f, 10.0f, 11.0f, 12.0f, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 4, 16); + bb_mem_alloc(0, 1, 4); + bb_mem_alloc(1, 1, 4); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_int2fp_channel(0, 1, 1, 0, 16); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("int2fp_channel_lanes i=%d got=%f exp=%f\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("int2fp_channel_lanes PASS\n"); + return 0; +} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_channel_two_rows_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_channel_two_rows_test.c new file mode 100644 index 000000000..a014215de --- /dev/null +++ b/examples/balls/int2fp/workloads/ctests/int2fp_channel_two_rows_test.c @@ -0,0 +1,46 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static int32_t input[32] __attribute__((aligned(64))) = { + 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, +}; +static float output[32] __attribute__((aligned(64))); +static const float da[4] __attribute__((aligned(64))) = {0.5f}; +static const float dw[16] __attribute__((aligned(64))) = { + 0.25f, 0.5f, 0.75f, 1.0f, 1.25f, 1.5f, 1.75f, 2.0f, + 0.125f, 0.25f, 0.375f, 0.5f, 2.25f, 2.5f, 2.75f, 3.0f, +}; +static const float expected[32] = { + 1, 2, 3, 4, 5, 6, 7, 8, 0.5f, 1, 1.5f, 2, 9, 10, 11, 12, + 2, 4, 6, 8, 10, 12, 14, 16, 1, 2, 3, 4, 18, 20, 22, 24, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 4, 16); + bb_mem_alloc(0, 2, 4); + bb_mem_alloc(1, 2, 4); + bb_mvin((uintptr_t)input, 0, 2, 1); + bb_int2fp_channel(0, 1, 2, 0, 16); + bb_mvout((uintptr_t)output, 1, 2, 1); + bb_fence(); + for (int i = 0; i < 32; ++i) { + if (output[i] != expected[i]) { + printf("int2fp_channel_two_rows i=%d got=%f exp=%f\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("int2fp_channel_two_rows PASS\n"); + return 0; +} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_col2_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_col2_test.c deleted file mode 100644 index 3f9499f1b..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_col2_test.c +++ /dev/null @@ -1,49 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include - -#define ROWS 2 -#define LANES 8 -#define SCALE_1_0 0x3F800000U - -static int32_t int32_input[ROWS * LANES] __attribute__((aligned(64))) = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, -}; - -static uint32_t expected_fp32[ROWS * LANES] __attribute__((aligned(64))) = { - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, - 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, -}; - -static uint32_t output_fp32[ROWS * LANES] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - const uint32_t src_bank = 0; - const uint32_t dst_bank = 1; - - bb_mem_alloc(src_bank, ROWS, 2); - bb_mem_alloc(dst_bank, ROWS, 2); - bb_mvin((uintptr_t)int32_input, src_bank, ROWS, 1); - bb_int2fp(src_bank, dst_bank, ROWS, SCALE_1_0); - bb_mvout((uintptr_t)output_fp32, dst_bank, ROWS, 1); - bb_fence(); - - int passed = 1; - for (int i = 0; i < ROWS * LANES; ++i) { - if (output_fp32[i] != expected_fp32[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, - output_fp32[i], expected_fp32[i]); - passed = 0; - } - } - - printf("Int2Fp INT32-to-FP32 col2 test %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_col4_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_col4_test.c deleted file mode 100644 index 930346b73..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_col4_test.c +++ /dev/null @@ -1,53 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include - -#define ROWS 2 -#define LANES 16 -#define SCALE_1_0 0x3F800000U - -static int32_t int32_input[ROWS * LANES] __attribute__((aligned(64))) = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, - -64, -33, -17, -9, -5, -2, 0, 4, 6, 8, 10, 12, 14, 16, 32, 64, -}; - -static uint32_t expected_fp32[ROWS * LANES] __attribute__((aligned(64))) = { - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, - 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, 0xC2800000, 0xC2040000, - 0xC1880000, 0xC1100000, 0xC0A00000, 0xC0000000, 0x00000000, 0x40800000, - 0x40C00000, 0x41000000, 0x41200000, 0x41400000, 0x41600000, 0x41800000, - 0x42000000, 0x42800000, -}; - -static uint32_t output_fp32[ROWS * LANES] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - const uint32_t src_bank = 0; - const uint32_t dst_bank = 1; - - bb_mem_alloc(src_bank, ROWS, 4); - bb_mem_alloc(dst_bank, ROWS, 4); - bb_mvin((uintptr_t)int32_input, src_bank, ROWS, 1); - bb_int2fp(src_bank, dst_bank, ROWS, SCALE_1_0); - bb_mvout((uintptr_t)output_fp32, dst_bank, ROWS, 1); - bb_fence(); - - int passed = 1; - for (int i = 0; i < ROWS * LANES; ++i) { - if (output_fp32[i] != expected_fp32[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, - output_fp32[i], expected_fp32[i]); - passed = 0; - } - } - - printf("Int2Fp INT32-to-FP32 col4 test %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_test.c deleted file mode 100644 index 45948b5f8..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_fp32_test.c +++ /dev/null @@ -1,87 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include -#include - -#define DIM 16 - -// INT32 input data (4 elements per SRAM word, 16 words) -static int32_t int32_input[DIM * 4] __attribute__((aligned(64))) = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, -8, -}; - -// Expected FP32 output as bit patterns: int32_val * 1.0 = float(int32_val) -// 1.0=0x3F800000, 2.0=0x40000000, 3.0=0x40400000, -1.0=0xBF800000 -// -2.0=0xC0000000, 0.0=0x00000000, 4.0=0x40800000, 5.0=0x40A00000 -// 10.0=0x41200000, -10.0=0xC1200000, 7.0=0x40E00000, 100.0=0x42C80000 -// -100.0=0xC2C80000, 8.0=0x41000000, 16.0=0x41800000, -8.0=0xC1000000 -static uint32_t expected_fp32[DIM * 4] __attribute__((aligned(64))) = { - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, - 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, 0x3F800000, 0x40000000, - 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, 0x40800000, 0x40A00000, - 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, 0xC2C80000, 0x41000000, - 0x41800000, 0xC1000000, 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, - 0xC0000000, 0x00000000, 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, - 0x40E00000, 0x42C80000, 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, - 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, -}; - -static uint32_t output_fp32[DIM * 4] __attribute__((aligned(64))); - -// FP32 bit pattern for scale = 1.0 -#define SCALE_1_0 0x3F800000U - -void hw_int2fp(int32_t *input, uint32_t *output, int num_words) { - uint32_t op1_bank_id = 0; - uint32_t wr_bank_id = 1; - - bb_mem_alloc(op1_bank_id, 1, 1); - bb_mem_alloc(wr_bank_id, 1, 1); - - bb_mvin((uintptr_t)input, op1_bank_id, num_words, 1); - - bb_int2fp(op1_bank_id, wr_bank_id, num_words, SCALE_1_0); - - bb_mvout((uintptr_t)output, wr_bank_id, num_words, 1); - bb_fence(); -} - -int main() { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - for (int i = 0; i < DIM * 4; i++) { - output_fp32[i] = 0; - } - - hw_int2fp(int32_input, output_fp32, DIM); - - int passed = 1; - for (int i = 0; i < DIM * 4; i++) { - if (output_fp32[i] != expected_fp32[i]) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, - output_fp32[i], expected_fp32[i]); - passed = 0; - } - } - - if (passed) { - printf("Int2Fp INT32-to-FP32 test PASSED\n"); - } else { - printf("Int2Fp INT32-to-FP32 test FAILED\n"); - } - return (!passed); - -#ifdef MULTICORE - exit(0); -#endif -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_int8_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_int8_test.c deleted file mode 100644 index 8028a3ebb..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_int32_to_int8_test.c +++ /dev/null @@ -1,49 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include - -#define ROWS 2 -#define LANES 16 -#define SCALE_0_5 0x3F000000U - -static int32_t int32_input[ROWS * LANES] __attribute__((aligned(64))) = { - -1000, -257, -255, -5, -3, -1, 0, 1, 3, 5, 127, 253, 255, 257, 1000, 2, - -999, -511, -259, -9, -7, -3, 2, 4, 6, 9, 125, 251, 254, 258, 511, 999, -}; - -static int8_t expected_int8[ROWS * LANES] __attribute__((aligned(64))) = { - -128, -128, -128, -2, -2, 0, 0, 0, 2, 2, 64, 126, 127, 127, 127, 1, - -128, -128, -128, -4, -4, -2, 1, 2, 3, 4, 62, 126, 127, 127, 127, 127, -}; - -static int8_t output_int8[ROWS * LANES] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - const uint32_t src_bank = 0; - const uint32_t dst_bank = 1; - - bb_mem_alloc(src_bank, ROWS, 4); - bb_mem_alloc(dst_bank, ROWS, 1); - bb_mvin((uintptr_t)int32_input, src_bank, ROWS, 1); - bb_int32_to_int8(src_bank, dst_bank, ROWS, SCALE_0_5); - bb_mvout((uintptr_t)output_int8, dst_bank, ROWS, 1); - bb_fence(); - - int passed = 1; - for (int i = 0; i < ROWS * LANES; ++i) { - if (output_int8[i] != expected_int8[i]) { - printf("MISMATCH at [%d]: got %d, expected %d\n", i, output_int8[i], - expected_int8[i]); - passed = 0; - } - } - - printf("Int2Fp INT32-to-INT8 test %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_int8_to_fp32_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_int8_to_fp32_test.c deleted file mode 100644 index 87b7787fa..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_int8_to_fp32_test.c +++ /dev/null @@ -1,60 +0,0 @@ -#include "buckyball.h" -#include -#include -#include -#include -#include - -#define ROWS 2 -#define LANES 16 -#define SCALE_0_25 0x3E800000U - -static int8_t int8_input[ROWS * LANES] __attribute__((aligned(64))) = { - -128, -127, -31, -15, -7, -3, -1, 0, 1, 2, 3, 5, 7, 15, 31, 127, - -64, -33, -17, -9, -5, -2, 0, 4, 6, 8, 10, 12, 14, 16, 32, 64, -}; - -static float expected_fp32[ROWS * LANES] __attribute__((aligned(64))) = { - -32.0f, -31.75f, -7.75f, -3.75f, -1.75f, -0.75f, -0.25f, 0.0f, - 0.25f, 0.5f, 0.75f, 1.25f, 1.75f, 3.75f, 7.75f, 31.75f, - -16.0f, -8.25f, -4.25f, -2.25f, -1.25f, -0.5f, 0.0f, 1.0f, - 1.5f, 2.0f, 2.5f, 3.0f, 3.5f, 4.0f, 8.0f, 16.0f, -}; - -static float output_fp32[ROWS * LANES] __attribute__((aligned(64))); - -static uint32_t fp32_bits(float value) { - uint32_t bits; - memcpy(&bits, &value, sizeof(bits)); - return bits; -} - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - - const uint32_t src_bank = 0; - const uint32_t dst_bank = 1; - - bb_mem_alloc(src_bank, ROWS, 1); - bb_mem_alloc(dst_bank, ROWS, 4); - bb_mvin((uintptr_t)int8_input, src_bank, ROWS, 1); - bb_int2fp(src_bank, dst_bank, ROWS, SCALE_0_25); - bb_mvout((uintptr_t)output_fp32, dst_bank, ROWS, 1); - bb_fence(); - - int passed = 1; - for (int i = 0; i < ROWS * LANES; ++i) { - const uint32_t actual = fp32_bits(output_fp32[i]); - const uint32_t expected = fp32_bits(expected_fp32[i]); - if (actual != expected) { - printf("MISMATCH at [%d]: got 0x%08X, expected 0x%08X\n", i, actual, - expected); - passed = 0; - } - } - - printf("Int2Fp INT8-to-FP32 test %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; -} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_ref.h b/examples/balls/int2fp/workloads/ctests/int2fp_ref.h deleted file mode 100644 index 1cebf2b44..000000000 --- a/examples/balls/int2fp/workloads/ctests/int2fp_ref.h +++ /dev/null @@ -1,158 +0,0 @@ -#ifndef INT2FP_REF_H -#define INT2FP_REF_H - -#include - -static uint32_t fp32_mul(uint32_t a, uint32_t b) { - uint32_t a_sign = (a >> 31) & 1; - uint32_t b_sign = (b >> 31) & 1; - uint32_t a_exp = (a >> 23) & 0xff; - uint32_t b_exp = (b >> 23) & 0xff; - uint32_t a_frac = a & 0x7fffff; - uint32_t b_frac = b & 0x7fffff; - uint64_t a_mant = (1ull << 23) | a_frac; - uint64_t b_mant = (1ull << 23) | b_frac; - int a_zero = a_exp == 0 && a_frac == 0; - int b_zero = b_exp == 0 && b_frac == 0; - uint64_t prod = a_mant * b_mant; - uint32_t sig, exp_adjust; - int guard, roundb, sticky; - if ((prod >> 47) & 1) { - sig = (uint32_t)(prod >> 24); - guard = ((prod >> 23) & 1) != 0; - roundb = ((prod >> 22) & 1) != 0; - sticky = (prod & ((1ull << 22) - 1)) != 0; - exp_adjust = 1; - } else { - sig = (uint32_t)(prod >> 23); - guard = ((prod >> 22) & 1) != 0; - roundb = ((prod >> 21) & 1) != 0; - sticky = (prod & ((1ull << 21) - 1)) != 0; - exp_adjust = 0; - } - int increment = guard && (roundb || sticky || (sig & 1) != 0); - uint64_t rounded = (uint64_t)sig + (uint64_t)increment; - uint32_t carry = (uint32_t)((rounded >> 24) & 1); - uint32_t final_sig = (uint32_t)(carry != 0 ? (rounded >> 1) : rounded); - uint32_t exp_wide = (a_exp + b_exp + exp_adjust + carry - 127) & 0x3ff; - - if (a_zero || b_zero) - return 0; - if (exp_wide & 0x200) - return 0; - if (exp_wide & 0x100) - return ((a_sign ^ b_sign) << 31) | (0xffu << 23); - return ((a_sign ^ b_sign) << 31) | ((exp_wide & 0xff) << 23) | - (final_sig & 0x7fffff); -} - -static int32_t fp32_to_i32(uint32_t fp) { - int sign = ((fp >> 31) & 1) != 0; - int32_t exponent = (int32_t)((fp >> 23) & 0xff); - uint32_t frac = fp & 0x7fffff; - uint32_t mantissa = (1u << 23) | frac; - int is_zero = exponent == 0 && frac == 0; - int32_t exp_val = exponent - 127; - - if (exponent == 0xff && frac != 0) - return 0; - if (is_zero) - return 0; - - uint64_t magnitude; - if (exp_val >= 31) { - magnitude = 0x80000000ull; - } else if (exp_val >= 23) { - magnitude = (uint64_t)mantissa << (exp_val - 23); - } else if (exp_val >= -1) { - uint32_t right_shift = (uint32_t)(23 - exp_val); - uint64_t truncated = (uint64_t)mantissa >> right_shift; - uint64_t half = 1ull << (right_shift - 1); - uint64_t remainder = (uint64_t)mantissa & ((1ull << right_shift) - 1); - int round_up = - remainder > half || (remainder == half && (truncated & 1) != 0); - magnitude = truncated + (uint64_t)round_up; - } else { - magnitude = 0; - } - - if (sign) { - if (magnitude >= 0x80000000ull) - return (int32_t)0x80000000; - return -(int32_t)magnitude; - } - if (magnitude > 0x7fffffffull) - return 0x7fffffff; - return (int32_t)magnitude; -} - -static uint32_t i32_to_fp32(int32_t value) { - if (value == 0) - return 0; - - uint32_t sign = ((uint32_t)value >> 31) & 1; - uint32_t absv; - if (value == (int32_t)0x80000000) - absv = 0x80000000u; - else if (value < 0) - absv = (uint32_t)(-value); - else - absv = (uint32_t)value; - - uint32_t leading_one = 31; - while ((absv & (1u << leading_one)) == 0) - leading_one--; - uint32_t exponent = leading_one + 127; - uint32_t significand; - if (leading_one > 23) { - uint32_t right_shift = leading_one - 23; - uint64_t abs_wide = absv; - uint64_t truncated = abs_wide >> right_shift; - uint64_t half = 1ull << (right_shift - 1); - uint64_t remainder = abs_wide & ((1ull << right_shift) - 1); - int round_up = - remainder > half || (remainder == half && (truncated & 1) != 0); - uint64_t rounded = truncated + (uint64_t)round_up; - if ((rounded >> 24) & 1) { - exponent = leading_one + 128; - significand = (uint32_t)(rounded >> 1); - } else { - significand = (uint32_t)rounded; - } - } else { - significand = absv << (23 - leading_one); - } - - return (sign << 31) | ((exponent & 0xff) << 23) | (significand & 0x7fffff); -} - -static uint32_t int2fp_fp32(int32_t value, uint32_t scale_bits) { - return fp32_mul(i32_to_fp32(value), scale_bits); -} - -static int8_t int2fp_i8(int32_t value, uint32_t scale_bits) { - int32_t v = fp32_to_i32(int2fp_fp32(value, scale_bits)); - if (v > 127) - return 127; - if (v < -128) - return -128; - return (int8_t)v; -} - -static float bits_to_f32(uint32_t bits) { - union { - uint32_t u; - float f; - } x = {.u = bits}; - return x.f; -} - -static uint32_t f32_to_bits(float f) { - union { - float f; - uint32_t u; - } x = {.f = f}; - return x.u; -} - -#endif diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_tensor_groups_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_tensor_groups_test.c new file mode 100644 index 000000000..5d87dfbec --- /dev/null +++ b/examples/balls/int2fp/workloads/ctests/int2fp_tensor_groups_test.c @@ -0,0 +1,42 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static int32_t input[16] __attribute__((aligned(64))) = { + 2, -2, 4, -4, 6, -6, 8, -8, 10, -10, 12, -12, 14, -14, 16, -16, +}; +static float output[16] __attribute__((aligned(64))); +static float da[4] __attribute__((aligned(64))) = {0.5f}; +static float dw[4] __attribute__((aligned(64))) = {0.25f}; +static const float expected[16] = { + 0.25f, -0.25f, 0.5f, -0.5f, 0.75f, -0.75f, 1.0f, -1.0f, + 1.25f, -1.25f, 1.5f, -1.5f, 1.75f, -1.75f, 2.0f, -2.0f, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 1, 4); + bb_mem_alloc(0, 1, 4); + bb_mem_alloc(1, 1, 4); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_int2fp_tensor(0, 1, 1, 0, 16); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 16; ++i) { + if (output[i] != expected[i]) { + printf("int2fp_tensor_groups i=%d got=%f exp=%f\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("int2fp_tensor_groups PASS\n"); + return 0; +} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_tensor_rows_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_tensor_rows_test.c new file mode 100644 index 000000000..ede948156 --- /dev/null +++ b/examples/balls/int2fp/workloads/ctests/int2fp_tensor_rows_test.c @@ -0,0 +1,39 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static int32_t input[8] + __attribute__((aligned(64))) = {4, -4, 8, -8, 12, -12, 16, -16}; +static float output[8] __attribute__((aligned(64))); +static float da[4] __attribute__((aligned(64))) = {0.5f}; +static float dw[4] __attribute__((aligned(64))) = {0.5f}; +static const float expected[8] = {1.0f, -1.0f, 2.0f, -2.0f, + 3.0f, -3.0f, 4.0f, -4.0f}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 1, 4); + bb_mem_alloc(0, 2, 1); + bb_mem_alloc(1, 2, 1); + bb_mvin((uintptr_t)input, 0, 2, 1); + bb_int2fp_tensor(0, 1, 2, 0, 16); + bb_mvout((uintptr_t)output, 1, 2, 1); + bb_fence(); + for (int i = 0; i < 8; ++i) { + if (output[i] != expected[i]) { + printf("int2fp_tensor_rows i=%d got=%f exp=%f\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("int2fp_tensor_rows PASS\n"); + return 0; +} diff --git a/examples/balls/int2fp/workloads/ctests/int2fp_tensor_values_test.c b/examples/balls/int2fp/workloads/ctests/int2fp_tensor_values_test.c new file mode 100644 index 000000000..c501609ab --- /dev/null +++ b/examples/balls/int2fp/workloads/ctests/int2fp_tensor_values_test.c @@ -0,0 +1,47 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +static int32_t input[4] __attribute__((aligned(64))) = {8, -4, 16, 0}; +static uint32_t output[4] __attribute__((aligned(64))) = { + 0xdeadbeef, + 0xdeadbeef, + 0xdeadbeef, + 0xdeadbeef, +}; +static float da[4] __attribute__((aligned(64))) = {0.5f}; +static float dw[4] __attribute__((aligned(64))) = {0.25f}; +static const uint32_t expected[4] = { + 0x3f800000, + 0xbf000000, + 0x40000000, + 0x00000000, +}; + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 1, 4); + bb_mem_alloc(0, 1, 1); + bb_mem_alloc(1, 1, 1); + bb_mvin((uintptr_t)input, 0, 1, 1); + bb_int2fp_tensor(0, 1, 1, 0, 16); + bb_mvout((uintptr_t)output, 1, 1, 1); + bb_fence(); + for (int i = 0; i < 4; ++i) { + if (output[i] != expected[i]) { + printf("int2fp_tensor_values i=%d got=%08x exp=%08x\n", i, output[i], + expected[i]); + return 1; + } + } + bb_mem_release(0); + bb_mem_release(1); + printf("int2fp_tensor_values PASS\n"); + return 0; +} diff --git a/examples/balls/int2fp/workloads/isa/int2fp.h b/examples/balls/int2fp/workloads/isa/int2fp.h new file mode 100644 index 000000000..bc4fad659 --- /dev/null +++ b/examples/balls/int2fp/workloads/isa/int2fp.h @@ -0,0 +1,27 @@ +#ifndef _BB_INT2FP_H_ +#define _BB_INT2FP_H_ + +#include +#include + +// Dequantize INT32 accumulators with scales in absolute MMIO byte addresses. +// rs2[12:0] = activation Da address, rs2[25:13] = weight Dw address. +#define BB_INT2FP_RS1(bank_id, wr_bank_id, iter) \ + (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)) +#define BB_INT2FP_RS2(act_scale_addr, weight_scale_addr) \ + (FIELD((uint64_t)(act_scale_addr), 0, 12) | \ + FIELD((uint64_t)(weight_scale_addr), 13, 25)) + +#define bb_int2fp_tensor(bank_id, wr_bank_id, iter, act_scale_addr, \ + weight_scale_addr) \ + BUCKYBALL_INSTRUCTION_R_R(BB_INT2FP_RS1(bank_id, wr_bank_id, iter), \ + BB_INT2FP_RS2(act_scale_addr, weight_scale_addr), \ + BB_FUNC7(INT2FP_TENSOR)) + +#define bb_int2fp_channel(bank_id, wr_bank_id, iter, act_scale_addr, \ + weight_scale_addr) \ + BUCKYBALL_INSTRUCTION_R_R(BB_INT2FP_RS1(bank_id, wr_bank_id, iter), \ + BB_INT2FP_RS2(act_scale_addr, weight_scale_addr), \ + BB_FUNC7(INT2FP_CHANNEL)) + +#endif // _BB_INT2FP_H_ diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/CMakeLists.txt b/examples/balls/int2fp/workloads/mlir_tests/ball/CMakeLists.txt index cf88ef314..5e0677d27 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/ball/CMakeLists.txt +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/CMakeLists.txt @@ -1,13 +1,18 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX ball) add_buckyball_mlir_contract(int2fp_4x4_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) +add_buckyball_mlir_contract(int2fp_channel_4x4_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} PASSES - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} ${BUCKYBALL_LOWER_BUCKYBALL}) -add_buckyball_mlir_optest(int2fp_4x4) -add_buckyball_mlir_optest(int32_to_int8_2x16) +add_buckyball_mlir_test(int2fp_4x4 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) +add_buckyball_mlir_test(int2fp_channel_4x4 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4.mlir b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4.mlir index 65048db75..155daa7ba 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4.mlir +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4.mlir @@ -1,5 +1,6 @@ -// Already-lowered ball Op: buckyball.int2fp (physical-bank form). +// Already-lowered ball Op: tensor int2fp form. +func.func private @prepare_scales() -> () func.func private @check_result(memref<4x4xf32>) -> () memref.global "private" constant @i32_in : memref<4x4xi32> = @@ -13,18 +14,22 @@ func.func @main() -> i8 { %depth = arith.constant 4 : i64 %stride = arith.constant 1 : i64 %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 + %dw_addr = arith.constant 16 : i64 + + func.call @prepare_scales() : () -> () %input = memref.get_global @i32_in : memref<4x4xi32> - %output = memref.alloc() {alignment = 64 : i64} : memref<4x4xf32> + %output = memref.alloc() alignment = 64 : memref<4x4xf32> %bin = buckyball.bank_alloc %bout = buckyball.bank_alloc %loaded = buckyball.bank_mvin %input %bin %depth %stride : memref<4x4xi32> i64 i64 i64 - buckyball.int2fp %loaded, %bout, %iter, %scale : i64 + buckyball.int2fp_tensor %loaded, %bout, %iter, %da_addr, %dw_addr : i64 %stored = buckyball.bank_mvout %output %bout %depth %stride : memref<4x4xf32> i64 i64 i64 + buckyball.fence buckyball.bank_release %loaded : i64 buckyball.bank_release %stored : i64 diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_contract.mlir b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_contract.mlir index 75673a171..ced4d212f 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_contract.mlir +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_contract.mlir @@ -1,22 +1,15 @@ // CHECK: buckyball.intr.custom -// CHECK-SAME: funct7 = 52 -// CHECK: arith.constant 4294967296 -// CHECK-NOT: buckyball.int2fp -// CHECK-NOT: buckyball.int32_to_int8 +// CHECK-NOT: buckyball.int2fp_tensor func.func @main() -> i8 { %z = arith.constant 0 : i8 %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 + %dw_addr = arith.constant 16 : i64 %in0 = buckyball.bank_alloc %out0 = buckyball.bank_alloc - %in1 = buckyball.bank_alloc - %out1 = buckyball.bank_alloc - buckyball.int2fp %in0, %out0, %iter, %scale : i64 - buckyball.int32_to_int8 %in1, %out1, %iter, %scale : i64 + buckyball.int2fp_tensor %in0, %out0, %iter, %da_addr, %dw_addr : i64 buckyball.bank_release %in0 : i64 buckyball.bank_release %out0 : i64 - buckyball.bank_release %in1 : i64 - buckyball.bank_release %out1 : i64 return %z : i8 } diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_main.cpp b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_main.cpp index a6f61702c..b5f28f981 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_main.cpp +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_4x4_main.cpp @@ -1,12 +1,13 @@ +#include #include #include #include #include static const uint32_t expected_bits[16] = { - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, - 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, + 0x3E000000, 0x3E800000, 0x3EC00000, 0xBE000000, 0xBE800000, 0x00000000, + 0x3F000000, 0x3F200000, 0x3FA00000, 0xBFA00000, 0x3F600000, 0x41480000, + 0xC1480000, 0x3F800000, 0x40000000, 0xBF800000, }; static void fail(void) { @@ -26,6 +27,14 @@ static uint32_t fp_bits(float v) { return bits; } +extern "C" void prepare_scales() { + static float da[4] __attribute__((aligned(64))) = {0.5f, 0.0f, 0.0f, 0.0f}; + static float dw[4] __attribute__((aligned(64))) = {0.25f, 0.0f, 0.0f, 0.0f}; + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 1, 4); + bb_fence(); +} + #ifdef __cplusplus extern "C" #endif diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4.mlir b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4.mlir new file mode 100644 index 000000000..182a7cf4e --- /dev/null +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4.mlir @@ -0,0 +1,24 @@ +func.func private @prepare_scales() -> () +func.func private @check_result(memref<1x16xf32>) -> () +memref.global "private" constant @input : memref<1x16xi32> = dense<8> + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %one = arith.constant 1 : i64 + %zero = arith.constant 0 : i64 + %sixteen = arith.constant 16 : i64 + func.call @prepare_scales() : () -> () + %input = memref.get_global @input : memref<1x16xi32> + %output = memref.alloc() alignment = 64 : memref<1x16xf32> + %in = buckyball.bank_alloc {col = 4 : i64} + %out = buckyball.bank_alloc {col = 4 : i64} + %loaded = buckyball.bank_mvin %input %in %one %one : memref<1x16xi32> i64 i64 i64 + buckyball.int2fp_channel %loaded, %out, %one, %zero, %sixteen : i64 + %stored = buckyball.bank_mvout %output %out %one %one : memref<1x16xf32> i64 i64 i64 + buckyball.fence + buckyball.bank_release %loaded : i64 + buckyball.bank_release %stored : i64 + func.call @check_result(%output) : (memref<1x16xf32>) -> () + memref.dealloc %output : memref<1x16xf32> + return %z : i8 +} diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4_contract.mlir b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4_contract.mlir new file mode 100644 index 000000000..351f86d12 --- /dev/null +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4_contract.mlir @@ -0,0 +1,15 @@ +// CHECK: buckyball.intr.custom +// CHECK-NOT: buckyball.int2fp_channel + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %iter = arith.constant 1 : i64 + %da = arith.constant 0 : i64 + %dw = arith.constant 16 : i64 + %in = buckyball.bank_alloc {col = 4 : i64} + %out = buckyball.bank_alloc {col = 4 : i64} + buckyball.int2fp_channel %in, %out, %iter, %da, %dw : i64 + buckyball.bank_release %in : i64 + buckyball.bank_release %out : i64 + return %z : i8 +} diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4_main.cpp b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4_main.cpp new file mode 100644 index 000000000..50920dee7 --- /dev/null +++ b/examples/balls/int2fp/workloads/mlir_tests/ball/int2fp_channel_4x4_main.cpp @@ -0,0 +1,41 @@ +#include +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + *(volatile uint32_t *)0x60000000 = 1; + while (1) { + } +#else + exit(1); +#endif +} + +static const float expected[16] = { + 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, + 0.5f, 1.0f, 1.5f, 2.0f, 9.0f, 10.0f, 11.0f, 12.0f, +}; + +extern "C" void prepare_scales() { + static float da[4] __attribute__((aligned(64))) = {0.5f}; + static float dw[16] __attribute__((aligned(64))) = { + 0.25f, 0.5f, 0.75f, 1.0f, 1.25f, 1.5f, 1.75f, 2.0f, + 0.125f, 0.25f, 0.375f, 0.5f, 2.25f, 2.5f, 2.75f, 3.0f, + }; + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 4, 16); + bb_fence(); +} + +extern "C" void check_result(float *, float *out, int64_t offset, int64_t rows, + int64_t cols, int64_t stride0, int64_t stride1) { + if (rows != 1 || cols != 16 || stride0 != 16 || stride1 != 1) + fail(); + for (int i = 0; i < 16; ++i) { + if (out[offset + i] != expected[i]) + fail(); + } + printf("PASSED: int2fp_channel_4x4 ball\n"); +} diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int32_to_int8_2x16.mlir b/examples/balls/int2fp/workloads/mlir_tests/ball/int32_to_int8_2x16.mlir deleted file mode 100644 index 240fc466b..000000000 --- a/examples/balls/int2fp/workloads/mlir_tests/ball/int32_to_int8_2x16.mlir +++ /dev/null @@ -1,30 +0,0 @@ -func.func private @check_result(memref<2x16xi8>) -> () - -memref.global "private" constant @input_i32 : memref<2x16xi32> = dense<[ - [-1000, -257, -255, -5, -3, -1, 0, 1, 3, 5, 127, 253, 255, 257, 1000, 2], - [-999, -511, -259, -9, -7, -3, 2, 4, 6, 9, 125, 251, 254, 258, 511, 999] -]> - -func.func @main() -> i8 { - %zero_i8 = arith.constant 0 : i8 - %depth = arith.constant 2 : i64 - %stride = arith.constant 1 : i64 - %scale = arith.constant 1056964608 : i64 - %input = memref.get_global @input_i32 : memref<2x16xi32> - %output = memref.alloc() {alignment = 64 : i64} : memref<2x16xi8> - - %in = buckyball.bank_alloc {col = 4 : i64} - %out = buckyball.bank_alloc - %loaded = buckyball.bank_mvin %input %in %depth %stride - : memref<2x16xi32> i64 i64 i64 - buckyball.int32_to_int8 %loaded, %out, %depth, %scale : i64 - %stored = buckyball.bank_mvout %output %out %depth %stride - : memref<2x16xi8> i64 i64 i64 - buckyball.fence - buckyball.bank_release %loaded : i64 - buckyball.bank_release %stored : i64 - - func.call @check_result(%output) : (memref<2x16xi8>) -> () - memref.dealloc %output : memref<2x16xi8> - return %zero_i8 : i8 -} diff --git a/examples/balls/int2fp/workloads/mlir_tests/ball/int32_to_int8_2x16_main.cpp b/examples/balls/int2fp/workloads/mlir_tests/ball/int32_to_int8_2x16_main.cpp deleted file mode 100644 index afa19344a..000000000 --- a/examples/balls/int2fp/workloads/mlir_tests/ball/int32_to_int8_2x16_main.cpp +++ /dev/null @@ -1,39 +0,0 @@ -#include -#include -#include - -static void fail(void) { -#ifdef BAREMETAL - volatile uint32_t *sim_exit = (volatile uint32_t *)0x60000000; - *sim_exit = 1; - while (1) { - } -#else - exit(1); -#endif -} - -#ifdef __cplusplus -extern "C" -#endif - void check_result(int8_t *allocated, int8_t *aligned, int64_t offset, - int64_t size0, int64_t size1, int64_t stride0, - int64_t stride1) { - static const int8_t expected[32] = { - -128, -128, -128, -2, -2, 0, 0, 0, 2, 2, 64, 126, 127, 127, 127, 1, - -128, -128, -128, -4, -4, -2, 1, 2, 3, 4, 62, 126, 127, 127, 127, 127, - }; - (void)allocated; - if (size0 != 2 || size1 != 16 || stride0 != 16 || stride1 != 1) { - fail(); - } - int8_t *out = aligned + offset; - for (int i = 0; i < 2; ++i) { - for (int j = 0; j < 16; ++j) { - if (out[i * stride0 + j * stride1] != expected[i * 16 + j]) { - fail(); - } - } - } - printf("PASSED: int2fp ball_int32_to_int8 2x16\n"); -} diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/int2fp/workloads/mlir_tests/bank/CMakeLists.txt index f988a8028..eaa486012 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/bank/CMakeLists.txt +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/CMakeLists.txt @@ -1,10 +1,12 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX bank) add_buckyball_mlir_contract(int2fp_4x4_contract - PASSES "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}") + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) +add_buckyball_mlir_contract(int2fp_channel_4x4_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) -add_buckyball_mlir_optest(int2fp_4x4) -add_buckyball_mlir_optest(int32_to_int8_2x16) +add_buckyball_mlir_test(int2fp_4x4 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) +add_buckyball_mlir_test(int2fp_channel_4x4 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4.mlir b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4.mlir index d2eee7292..4917ed968 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4.mlir +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4.mlir @@ -1,6 +1,7 @@ -// Bank-level Op: buckyball.bank_int2fp -// Lowering: assign-physical-banks -> bank_int2fp becomes int2fp. +// Bank-level Op: buckyball.bank_int2fp_tensor +// Lowering: assign-physical-banks -> tensor int2fp. +func.func private @prepare_scales() -> () func.func private @check_result(memref<4x4xf32>) -> () memref.global "private" constant @i32_in : memref<4x4xi32> = @@ -14,19 +15,23 @@ func.func @main() -> i8 { %depth = arith.constant 4 : i64 %stride = arith.constant 1 : i64 %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 + %dw_addr = arith.constant 16 : i64 + + func.call @prepare_scales() : () -> () %input = memref.get_global @i32_in : memref<4x4xi32> - %output = memref.alloc() {alignment = 64 : i64} : memref<4x4xf32> + %output = memref.alloc() alignment = 64 : memref<4x4xf32> %bin = buckyball.bank_alloc %bout = buckyball.bank_alloc %loaded = buckyball.bank_mvin %input %bin %depth %stride : memref<4x4xi32> i64 i64 i64 - %q = buckyball.bank_int2fp %loaded %bout %iter %scale - : i64 i64 i64 i64 + %q = buckyball.bank_int2fp_tensor %loaded %bout %iter %da_addr %dw_addr + : i64 i64 i64 i64 i64 %stored = buckyball.bank_mvout %output %q %depth %stride : memref<4x4xf32> i64 i64 i64 + buckyball.fence buckyball.bank_release %loaded : i64 buckyball.bank_release %stored : i64 diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_contract.mlir b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_contract.mlir index 17ad57370..3fcadda0a 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_contract.mlir +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_contract.mlir @@ -1,23 +1,16 @@ -// CHECK: buckyball.int2fp -// CHECK: buckyball.int32_to_int8 -// CHECK-NOT: buckyball.bank_int2fp -// CHECK-NOT: buckyball.bank_int32_to_int8 +// CHECK: buckyball.int2fp_tensor +// CHECK-NOT: buckyball.bank_int2fp_tensor func.func @main() -> i8 { %z = arith.constant 0 : i8 %iter = arith.constant 4 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 + %dw_addr = arith.constant 16 : i64 %in0 = buckyball.bank_alloc %out0 = buckyball.bank_alloc - %in1 = buckyball.bank_alloc - %out1 = buckyball.bank_alloc - %r0 = buckyball.bank_int2fp %in0 %out0 %iter %scale - : i64 i64 i64 i64 - %r1 = buckyball.bank_int32_to_int8 %in1 %out1 %iter %scale - : i64 i64 i64 i64 + %r0 = buckyball.bank_int2fp_tensor %in0 %out0 %iter %da_addr %dw_addr + : i64 i64 i64 i64 i64 buckyball.bank_release %in0 : i64 buckyball.bank_release %r0 : i64 - buckyball.bank_release %in1 : i64 - buckyball.bank_release %r1 : i64 return %z : i8 } diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_main.cpp b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_main.cpp index 14b03f54a..8546f2755 100644 --- a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_main.cpp +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_4x4_main.cpp @@ -1,12 +1,13 @@ +#include #include #include #include #include static const uint32_t expected_bits[16] = { - 0x3F800000, 0x40000000, 0x40400000, 0xBF800000, 0xC0000000, 0x00000000, - 0x40800000, 0x40A00000, 0x41200000, 0xC1200000, 0x40E00000, 0x42C80000, - 0xC2C80000, 0x41000000, 0x41800000, 0xC1000000, + 0x3E000000, 0x3E800000, 0x3EC00000, 0xBE000000, 0xBE800000, 0x00000000, + 0x3F000000, 0x3F200000, 0x3FA00000, 0xBFA00000, 0x3F600000, 0x41480000, + 0xC1480000, 0x3F800000, 0x40000000, 0xBF800000, }; static void fail(void) { @@ -26,6 +27,14 @@ static uint32_t fp_bits(float v) { return bits; } +extern "C" void prepare_scales() { + static float da[4] __attribute__((aligned(64))) = {0.5f, 0.0f, 0.0f, 0.0f}; + static float dw[4] __attribute__((aligned(64))) = {0.25f, 0.0f, 0.0f, 0.0f}; + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 1, 4); + bb_fence(); +} + #ifdef __cplusplus extern "C" #endif diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4.mlir b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4.mlir new file mode 100644 index 000000000..1963db20e --- /dev/null +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4.mlir @@ -0,0 +1,24 @@ +func.func private @prepare_scales() -> () +func.func private @check_result(memref<1x16xf32>) -> () +memref.global "private" constant @input : memref<1x16xi32> = dense<8> + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %one = arith.constant 1 : i64 + %zero = arith.constant 0 : i64 + %sixteen = arith.constant 16 : i64 + func.call @prepare_scales() : () -> () + %input = memref.get_global @input : memref<1x16xi32> + %output = memref.alloc() alignment = 64 : memref<1x16xf32> + %in = buckyball.bank_alloc {col = 4 : i64} + %out = buckyball.bank_alloc {col = 4 : i64} + %loaded = buckyball.bank_mvin %input %in %one %one : memref<1x16xi32> i64 i64 i64 + %q = buckyball.bank_int2fp_channel %loaded %out %one %zero %sixteen : i64 i64 i64 i64 i64 + %stored = buckyball.bank_mvout %output %q %one %one : memref<1x16xf32> i64 i64 i64 + buckyball.fence + buckyball.bank_release %loaded : i64 + buckyball.bank_release %stored : i64 + func.call @check_result(%output) : (memref<1x16xf32>) -> () + memref.dealloc %output : memref<1x16xf32> + return %z : i8 +} diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4_contract.mlir b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4_contract.mlir new file mode 100644 index 000000000..a04734930 --- /dev/null +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4_contract.mlir @@ -0,0 +1,16 @@ +// CHECK: buckyball.int2fp_channel +// CHECK-NOT: buckyball.bank_int2fp_channel + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %iter = arith.constant 1 : i64 + %da = arith.constant 0 : i64 + %dw = arith.constant 16 : i64 + %in = buckyball.bank_alloc {col = 4 : i64} + %out = buckyball.bank_alloc {col = 4 : i64} + %r = buckyball.bank_int2fp_channel %in %out %iter %da %dw + : i64 i64 i64 i64 i64 + buckyball.bank_release %in : i64 + buckyball.bank_release %r : i64 + return %z : i8 +} diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4_main.cpp b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4_main.cpp new file mode 100644 index 000000000..3155b5d4a --- /dev/null +++ b/examples/balls/int2fp/workloads/mlir_tests/bank/int2fp_channel_4x4_main.cpp @@ -0,0 +1,41 @@ +#include +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + *(volatile uint32_t *)0x60000000 = 1; + while (1) { + } +#else + exit(1); +#endif +} + +static const float expected[16] = { + 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, + 0.5f, 1.0f, 1.5f, 2.0f, 9.0f, 10.0f, 11.0f, 12.0f, +}; + +extern "C" void prepare_scales() { + static float da[4] __attribute__((aligned(64))) = {0.5f}; + static float dw[16] __attribute__((aligned(64))) = { + 0.25f, 0.5f, 0.75f, 1.0f, 1.25f, 1.5f, 1.75f, 2.0f, + 0.125f, 0.25f, 0.375f, 0.5f, 2.25f, 2.5f, 2.75f, 3.0f, + }; + bb_mvin_mmio((uintptr_t)da, 0, 1, 4); + bb_mvin_mmio((uintptr_t)dw, 16, 4, 16); + bb_fence(); +} + +extern "C" void check_result(float *, float *out, int64_t offset, int64_t rows, + int64_t cols, int64_t stride0, int64_t stride1) { + if (rows != 1 || cols != 16 || stride0 != 16 || stride1 != 1) + fail(); + for (int i = 0; i < 16; ++i) { + if (out[offset + i] != expected[i]) + fail(); + } + printf("PASSED: int2fp_channel_4x4 bank\n"); +} diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int32_to_int8_2x16.mlir b/examples/balls/int2fp/workloads/mlir_tests/bank/int32_to_int8_2x16.mlir deleted file mode 100644 index 607cf55c1..000000000 --- a/examples/balls/int2fp/workloads/mlir_tests/bank/int32_to_int8_2x16.mlir +++ /dev/null @@ -1,31 +0,0 @@ -func.func private @check_result(memref<2x16xi8>) -> () - -memref.global "private" constant @input_i32 : memref<2x16xi32> = dense<[ - [-1000, -257, -255, -5, -3, -1, 0, 1, 3, 5, 127, 253, 255, 257, 1000, 2], - [-999, -511, -259, -9, -7, -3, 2, 4, 6, 9, 125, 251, 254, 258, 511, 999] -]> - -func.func @main() -> i8 { - %zero_i8 = arith.constant 0 : i8 - %depth = arith.constant 2 : i64 - %stride = arith.constant 1 : i64 - %scale = arith.constant 1056964608 : i64 - %input = memref.get_global @input_i32 : memref<2x16xi32> - %output = memref.alloc() {alignment = 64 : i64} : memref<2x16xi8> - - %in = buckyball.bank_alloc {col = 4 : i64} - %out = buckyball.bank_alloc - %loaded = buckyball.bank_mvin %input %in %depth %stride - : memref<2x16xi32> i64 i64 i64 - %converted = buckyball.bank_int32_to_int8 %loaded %out %depth %scale - : i64 i64 i64 i64 - %stored = buckyball.bank_mvout %output %converted %depth %stride - : memref<2x16xi8> i64 i64 i64 - buckyball.fence - buckyball.bank_release %loaded : i64 - buckyball.bank_release %stored : i64 - - func.call @check_result(%output) : (memref<2x16xi8>) -> () - memref.dealloc %output : memref<2x16xi8> - return %zero_i8 : i8 -} diff --git a/examples/balls/int2fp/workloads/mlir_tests/bank/int32_to_int8_2x16_main.cpp b/examples/balls/int2fp/workloads/mlir_tests/bank/int32_to_int8_2x16_main.cpp deleted file mode 100644 index efb9fbf4f..000000000 --- a/examples/balls/int2fp/workloads/mlir_tests/bank/int32_to_int8_2x16_main.cpp +++ /dev/null @@ -1,39 +0,0 @@ -#include -#include -#include - -static void fail(void) { -#ifdef BAREMETAL - volatile uint32_t *sim_exit = (volatile uint32_t *)0x60000000; - *sim_exit = 1; - while (1) { - } -#else - exit(1); -#endif -} - -#ifdef __cplusplus -extern "C" -#endif - void check_result(int8_t *allocated, int8_t *aligned, int64_t offset, - int64_t size0, int64_t size1, int64_t stride0, - int64_t stride1) { - static const int8_t expected[32] = { - -128, -128, -128, -2, -2, 0, 0, 0, 2, 2, 64, 126, 127, 127, 127, 1, - -128, -128, -128, -4, -4, -2, 1, 2, 3, 4, 62, 126, 127, 127, 127, 127, - }; - (void)allocated; - if (size0 != 2 || size1 != 16 || stride0 != 16 || stride1 != 1) { - fail(); - } - int8_t *out = aligned + offset; - for (int i = 0; i < 2; ++i) { - for (int j = 0; j < 16; ++j) { - if (out[i * stride0 + j * stride1] != expected[i * 16 + j]) { - fail(); - } - } - } - printf("PASSED: int2fp bank_int32_to_int8 2x16\n"); -} diff --git a/examples/balls/matadd/README.md b/examples/balls/matadd/README.md new file mode 100644 index 000000000..217a9e913 --- /dev/null +++ b/examples/balls/matadd/README.md @@ -0,0 +1,16 @@ +# MatAddBall + +`MatAddBall` adds INT32 data from two virtual banks into a third virtual bank. + +``` +MATADD(a, b, c, iter) +``` + +`a`, `b`, and `c` must be different allocated vbank IDs. Their `col` values +must match. `iter` is the number of 128-bit physical lines in every group. +The ball processes group 0 through the final group in order. For each line it +reads one 128-bit word from `a` and `b`, adds four INT32 lanes modulo 2^32, +then writes one 128-bit word to `c`. + +The ball has two SRAM read ports and one SRAM write port. It has no internal +SRAM. It holds all three bank channels until the command response is accepted. diff --git a/examples/balls/matadd/arch/src/main/scala/MatAdd.scala b/examples/balls/matadd/arch/src/main/scala/MatAdd.scala new file mode 100644 index 000000000..a7bc26f56 --- /dev/null +++ b/examples/balls/matadd/arch/src/main/scala/MatAdd.scala @@ -0,0 +1,199 @@ +package examples.balls.matadd + +import chisel3._ +import chisel3.experimental.hierarchy.{instantiable, public} +import chisel3.util._ +import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} +import framework.balldomain.rs.{BallRsComplete, BallRsIssue} +import framework.top.GlobalConfig + +@instantiable +class MatAdd(val b: GlobalConfig) extends Module { + private val addressWidth = log2Up(b.memDomain.bankEntries) + private val countWidth = log2Up(b.memDomain.bankEntries + 1) + private val bankIdWidth = log2Up(b.memDomain.bankNum) + + private val mapping = b.ballDomain.ballIdMappings + .find(_.ballName == "MatAddBall") + .getOrElse(throw new IllegalArgumentException("MatAddBall not found in config")) + + private val funct = b.ballDomain.ballISA + .find(_.mnemonic == "MATADD") + .map(_.funct7) + .getOrElse(throw new IllegalArgumentException("MATADD not found in ballISA")) + + require(mapping.inBW == 2, "MatAddBall requires two SRAM read ports") + require(mapping.outBW == 1, "MatAddBall requires one SRAM write port") + require(b.memDomain.bankWidth == 128, "MatAddBall requires 128-bit SRAM rows") + require(b.memDomain.bankMaskLen == 16, "MatAddBall requires sixteen byte enables") + + @public + val io = IO(new Bundle { + val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) + val cmdResp = Decoupled(new BallRsComplete(b)) + val channelReady = Input(Bool()) + val bankRead = Vec(2, Flipped(new BankRead(b))) + val bankWrite = Vec(1, Flipped(new BankWrite(b))) + val status = new BallStatus + }) + + val Seq(idle, waitForChannels, readRequest, readResponse, writeRequest, writeResponse, complete) = Enum(7) + val state = RegInit(idle) + + val robId = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) + val isSub = RegInit(false.B) + val subRobId = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) + val aBank = RegInit(0.U(bankIdWidth.W)) + val bBank = RegInit(0.U(bankIdWidth.W)) + val cBank = RegInit(0.U(bankIdWidth.W)) + val groups = RegInit(0.U(5.W)) + val group = RegInit(0.U(5.W)) + val iter = RegInit(0.U(countWidth.W)) + val line = RegInit(0.U(addressWidth.W)) + val aWord = Reg(UInt(128.W)) + val bWord = Reg(UInt(128.W)) + val aRequestSent = RegInit(false.B) + val bRequestSent = RegInit(false.B) + val aReceived = RegInit(false.B) + val bReceived = RegInit(false.B) + + val sum = Wire(Vec(4, UInt(32.W))) + for (lane <- 0 until 4) { + sum(lane) := (aWord(32 * lane + 31, 32 * lane) +& + bWord(32 * lane + 31, 32 * lane))(31, 0) + } + + io.cmdReq.ready := state === idle + io.cmdResp.valid := state === complete + io.cmdResp.bits.rob_id := robId + io.cmdResp.bits.is_sub := isSub + io.cmdResp.bits.sub_rob_id := subRobId + + for (port <- 0 until 2) { + io.bankRead(port).rob_id := robId + io.bankRead(port).ball_id := 0.U + io.bankRead(port).bank_id := (if (port == 0) aBank else bBank) + io.bankRead(port).group_id := group + io.bankRead(port).io.req.valid := false.B + io.bankRead(port).io.req.bits.addr := line + io.bankRead(port).io.resp.ready := false.B + } + + io.bankWrite(0).rob_id := robId + io.bankWrite(0).ball_id := 0.U + io.bankWrite(0).bank_id := cBank + io.bankWrite(0).group_id := group + io.bankWrite(0).io.req.valid := false.B + io.bankWrite(0).io.req.bits.addr := line + io.bankWrite(0).io.req.bits.data := Cat(sum.reverse) + io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(16)(true.B)) + io.bankWrite(0).io.resp.ready := false.B + + switch(state) { + is(idle) { + when(io.cmdReq.fire) { + val command = io.cmdReq.bits.cmd + assert(command.funct7 === funct.U(7.W), "MatAddBall funct7 must be MATADD") + assert(command.op1_en && command.op2_en && command.wr_spad_en, "MatAddBall requires two inputs and one output") + assert( + command.op1_bank =/= command.op2_bank && command.op1_bank =/= command.wr_bank && + command.op2_bank =/= command.wr_bank, + "MatAddBall banks must be distinct" + ) + assert( + command.op1_col =/= 0.U && command.op1_col === command.op2_col && + command.op1_col === command.wr_col, + "MatAddBall bank groups must match" + ) + assert(command.op1_col <= b.memDomain.bankNum.U, "MatAddBall bank groups must fit in physical banks") + assert( + command.iter =/= 0.U && command.iter <= b.memDomain.bankEntries.U, + "MatAddBall iter must fit in one physical bank" + ) + assert(command.special === 0.U, "MatAddBall reserves rs2") + + robId := io.cmdReq.bits.rob_id + isSub := io.cmdReq.bits.is_sub + subRobId := io.cmdReq.bits.sub_rob_id + aBank := command.op1_bank + bBank := command.op2_bank + cBank := command.wr_bank + groups := command.op1_col + group := 0.U + iter := command.iter + line := 0.U + aRequestSent := false.B + bRequestSent := false.B + aReceived := false.B + bReceived := false.B + state := waitForChannels + } + } + + is(waitForChannels) { + when(io.channelReady)(state := readRequest) + } + + is(readRequest) { + io.bankRead(0).io.req.valid := !aRequestSent + io.bankRead(1).io.req.valid := !bRequestSent + when(io.bankRead(0).io.req.fire)(aRequestSent := true.B) + when(io.bankRead(1).io.req.fire)(bRequestSent := true.B) + when((aRequestSent || io.bankRead(0).io.req.fire) && + (bRequestSent || io.bankRead(1).io.req.fire)) { + state := readResponse + } + } + + is(readResponse) { + io.bankRead(0).io.resp.ready := !aReceived + io.bankRead(1).io.resp.ready := !bReceived + when(io.bankRead(0).io.resp.fire) { + aWord := io.bankRead(0).io.resp.bits.data + aReceived := true.B + } + when(io.bankRead(1).io.resp.fire) { + bWord := io.bankRead(1).io.resp.bits.data + bReceived := true.B + } + when((aReceived || io.bankRead(0).io.resp.fire) && + (bReceived || io.bankRead(1).io.resp.fire)) { + state := writeRequest + } + } + + is(writeRequest) { + io.bankWrite(0).io.req.valid := true.B + when(io.bankWrite(0).io.req.fire)(state := writeResponse) + } + + is(writeResponse) { + io.bankWrite(0).io.resp.ready := true.B + when(io.bankWrite(0).io.resp.fire) { + aRequestSent := false.B + bRequestSent := false.B + aReceived := false.B + bReceived := false.B + when(line +& 1.U === iter) { + line := 0.U + when(group +& 1.U === groups) { + state := complete + }.otherwise { + group := group + 1.U + state := readRequest + } + }.otherwise { + line := line + 1.U + state := readRequest + } + } + } + + is(complete) { + when(io.cmdResp.fire)(state := idle) + } + } + + io.status.idle := state === idle + io.status.running := state =/= idle +} diff --git a/examples/balls/matadd/arch/src/main/scala/MatAddBall.scala b/examples/balls/matadd/arch/src/main/scala/MatAddBall.scala new file mode 100644 index 000000000..4193186da --- /dev/null +++ b/examples/balls/matadd/arch/src/main/scala/MatAddBall.scala @@ -0,0 +1,34 @@ +package examples.balls.matadd + +import chisel3._ +import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} +import framework.balldomain.blink.{BlinkIO, HasBallStatus, HasBlink, SubRobRow} +import framework.balldomain.blink.mmio.MmioRead +import framework.top.GlobalConfig + +@instantiable +class MatAddBall(val b: GlobalConfig) extends Module with HasBlink with HasBallStatus { + + private val mapping = b.ballDomain.ballIdMappings + .find(_.ballName == "MatAddBall") + .getOrElse(throw new IllegalArgumentException("MatAddBall not found in config")) + + @public + val io = IO(new BlinkIO(b, mapping.inBW, mapping.outBW)) + + def blink: BlinkIO = io + def status = io.status + dontTouch(io) + + val unit: Instance[MatAdd] = Instantiate(new MatAdd(b)) + unit.io.cmdReq <> io.cmdReq + unit.io.cmdResp <> io.cmdResp + unit.io.channelReady := io.channelReady + unit.io.bankRead(0) <> io.bankRead(0) + unit.io.bankRead(1) <> io.bankRead(1) + unit.io.bankWrite(0) <> io.bankWrite(0) + io.status <> unit.io.status + io.subRobReq.valid := false.B + io.subRobReq.bits := SubRobRow.tieOff(b) + MmioRead.tieOff(io.mmioRead) +} diff --git a/examples/balls/matadd/compiler/src/Backend/include/IntrinsicsRISCVMatAddBall.td b/examples/balls/matadd/compiler/src/Backend/include/IntrinsicsRISCVMatAddBall.td new file mode 100644 index 000000000..125b1e629 --- /dev/null +++ b/examples/balls/matadd/compiler/src/Backend/include/IntrinsicsRISCVMatAddBall.td @@ -0,0 +1 @@ +// MatAddBall uses CustomIntrOp with funct7 from the target ballISA. diff --git a/examples/balls/matadd/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/matadd/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp new file mode 100644 index 000000000..6040e0a1f --- /dev/null +++ b/examples/balls/matadd/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -0,0 +1,34 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace mlir::buddy { +void populateMatAddBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +} + +namespace { +class BankMatAddPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(BankMatAddOp op, + PatternRewriter &rewriter) const override { + rewriter.create(op.getLoc(), op.getABank(), op.getBBank(), + op.getCBank(), op.getIter()); + rewriter.replaceOp(op, op.getCBank()); + return success(); + } +}; +} // namespace + +void mlir::buddy::populateMatAddBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { + (void)state; + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/matadd/compiler/src/Dialect/Buckyball/MatAddBall.td b/examples/balls/matadd/compiler/src/Dialect/Buckyball/MatAddBall.td new file mode 100644 index 000000000..62e34bd00 --- /dev/null +++ b/examples/balls/matadd/compiler/src/Dialect/Buckyball/MatAddBall.td @@ -0,0 +1,21 @@ +#ifndef MATADDBALL_OPS +#define MATADDBALL_OPS + +include "Buckyball.td" + +def MatAddOp : Buckyball_Op<"matadd"> { + let summary = "Add matching INT32 physical-bank lines"; + let arguments = (ins I64:$aBankId, I64:$bBankId, I64:$cBankId, I64:$iter); + let assemblyFormat = "$aBankId `,` $bBankId `,` $cBankId `,` $iter attr-dict `:` " + "type($aBankId)"; +} + +def BankMatAddOp : Buckyball_Op<"bank_matadd"> { + let summary = "Add matching INT32 virtual-bank lines"; + let arguments = (ins I64:$aBank, I64:$bBank, I64:$cBank, I64:$iter); + let results = (outs I64:$cBankOut); + let assemblyFormat = "$aBank $bBank $cBank $iter attr-dict `:` " + "type($aBank) type($bBank) type($cBank) type($iter)"; +} + +#endif // MATADDBALL_OPS diff --git a/examples/balls/matadd/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/matadd/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..a3208d929 --- /dev/null +++ b/examples/balls/matadd/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,45 @@ +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/Pattern.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" + +using namespace mlir; +using namespace buddy::buckyball; +using namespace buddy::buckyball::legalize; + +namespace { +struct MatAddLowering : public ConvertOpToLLVMPattern { + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; + + LogicalResult + matchAndRewrite(MatAddOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("MatAddBall"); + Location loc = op.getLoc(); + Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getABankId(), + adaptor.getBBankId(), adaptor.getCBankId(), + adaptor.getIter()); + rewriter.replaceOpWithNewOp( + op, rs1, cstI64(rewriter, loc, 0), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("MATADD"))); + return success(); + } +}; +} // namespace + +namespace mlir::buddy::buckyball { +void populateMatAddBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool, int64_t, + bool) { + patterns.add(converter); +} + +void configureMatAddBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool) { + target.addIllegalOp(); +} +} // namespace mlir::buddy::buckyball diff --git a/examples/balls/matadd/configs/default.toml b/examples/balls/matadd/configs/default.toml new file mode 100644 index 000000000..c61b39cb6 --- /dev/null +++ b/examples/balls/matadd/configs/default.toml @@ -0,0 +1 @@ +[ball] diff --git a/examples/balls/matadd/emu/src/64_matadd.rs b/examples/balls/matadd/emu/src/64_matadd.rs new file mode 100644 index 000000000..66949fd6a --- /dev/null +++ b/examples/balls/matadd/emu/src/64_matadd.rs @@ -0,0 +1,65 @@ +use super::super::bank::{bank_num, bank_size}; +use super::decode::{pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; +use super::instruction::{BallInstruction, ExecContext}; + +pub struct MatAdd; + +impl BallInstruction for MatAdd { + fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { + let a_bank = rs1_b0(xs1); + let b_bank = rs1_b1(xs1); + let c_bank = rs1_b2(xs1); + let iter = rs1_iter(xs1) as usize; + + if xs2 != 0 { + panic!("matadd: rs2 must be zero"); + } + if a_bank >= bank_num() as u64 || b_bank >= bank_num() as u64 || c_bank >= bank_num() as u64 { + panic!("matadd: invalid bank id"); + } + if a_bank == b_bank || a_bank == c_bank || b_bank == c_bank { + panic!("matadd: banks must be distinct"); + } + if iter == 0 || iter * 16 > bank_size() { + panic!("matadd: iter must fit in one physical bank"); + } + + let a = &ctx.cfgs[a_bank as usize]; + let b = &ctx.cfgs[b_bank as usize]; + let c = &ctx.cfgs[c_bank as usize]; + if !a.allocated || !b.allocated || !c.allocated { + panic!("matadd: all banks must be allocated"); + } + if a.cols == 0 || a.cols != b.cols || a.cols != c.cols { + panic!("matadd: bank groups must match"); + } + + for group in 0..a.cols as usize { + let a_pbank = pbank_group(ctx.bank_map, a_bank, group as u64); + let b_pbank = pbank_group(ctx.bank_map, b_bank, group as u64); + let c_pbank = pbank_group(ctx.bank_map, c_bank, group as u64); + for line in 0..iter { + let offset = line * 16; + for lane in 0..4 { + let byte = offset + lane * 4; + let a_value = u32::from_le_bytes(ctx.banks[a_pbank][byte..byte + 4].try_into().unwrap()); + let b_value = u32::from_le_bytes(ctx.banks[b_pbank][byte..byte + 4].try_into().unwrap()); + ctx.banks[c_pbank][byte..byte + 4] + .copy_from_slice(&a_value.wrapping_add(b_value).to_le_bytes()); + } + } + } + 0 + } + + fn latency(xs1: u64, xs2: u64) -> u64 { + if xs2 != 0 { + panic!("matadd: rs2 must be zero"); + } + let iter = rs1_iter(xs1); + if iter == 0 || iter * 16 > bank_size() as u64 { + panic!("matadd: iter must fit in one physical bank"); + } + iter * 4 + } +} diff --git a/examples/balls/matadd/emu/src/lib.rs b/examples/balls/matadd/emu/src/lib.rs new file mode 100644 index 000000000..67b2b7072 --- /dev/null +++ b/examples/balls/matadd/emu/src/lib.rs @@ -0,0 +1,34 @@ +pub(crate) use crate::inst::{decode, instruction}; + +use crate::inst::instruction::{BallInstruction, ExecContext}; + +#[path = "64_matadd.rs"] +mod f64_matadd; + +const BALL_CLASS: &str = "examples.balls.matadd.MatAddBall"; + +pub fn execute_known( + ball_class: &str, + funct: u32, + xs1: u64, + xs2: u64, + ctx: &mut ExecContext, +) -> Option { + if ball_class != BALL_CLASS { + return None; + } + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("MATADD") => Some(f64_matadd::MatAdd::exec(xs1, xs2, ctx)), + Some(_) | None => None, + } +} + +pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> Option { + if ball_class != BALL_CLASS { + return None; + } + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("MATADD") => Some(f64_matadd::MatAdd::latency(xs1, xs2)), + Some(_) | None => None, + } +} diff --git a/examples/balls/matrix/verify/casegen/Cargo.lock b/examples/balls/matadd/verify/casegen/Cargo.lock similarity index 84% rename from examples/balls/matrix/verify/casegen/Cargo.lock rename to examples/balls/matadd/verify/casegen/Cargo.lock index ab32229d5..ec2bf5090 100644 --- a/examples/balls/matrix/verify/casegen/Cargo.lock +++ b/examples/balls/matadd/verify/casegen/Cargo.lock @@ -3,5 +3,5 @@ version = 4 [[package]] -name = "matrix-casegen" +name = "matadd-casegen" version = "0.1.0" diff --git a/examples/balls/matadd/verify/casegen/Cargo.toml b/examples/balls/matadd/verify/casegen/Cargo.toml new file mode 100644 index 000000000..9dfe0fa47 --- /dev/null +++ b/examples/balls/matadd/verify/casegen/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "matadd-casegen" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["rlib", "cdylib"] + +[lints.rust] +unsafe_code = "forbid" diff --git a/examples/balls/matadd/verify/casegen/src/lib.rs b/examples/balls/matadd/verify/casegen/src/lib.rs new file mode 100644 index 000000000..45278f224 --- /dev/null +++ b/examples/balls/matadd/verify/casegen/src/lib.rs @@ -0,0 +1 @@ +#![forbid(unsafe_code)] diff --git a/examples/balls/matadd/verify/filelists/matadd_ball.f b/examples/balls/matadd/verify/filelists/matadd_ball.f new file mode 100644 index 000000000..436e5c31f --- /dev/null +++ b/examples/balls/matadd/verify/filelists/matadd_ball.f @@ -0,0 +1,22 @@ ++incdir+@UVM@/src ++incdir+@UVM@/src/agents/cmd ++incdir+@UVM@/src/agents/mem ++incdir+@UVM@/src/agents/resp ++incdir+@UVM@/src/cov ++incdir+@UVM@/src/env ++incdir+src ++incdir+src/pkg ++incdir+src/common ++incdir+src/seq ++incdir+src/env ++incdir+src/tests ++incdir+@RTL@ + +@UVM@/src/bb_blink_if.sv +@UVM@/src/bb_uvm_pkg.sv + +@RTL@/MatAdd.sv +@RTL@/MatAddBall.sv + +src/pkg/matadd_pkg.sv +src/tb_top.sv diff --git a/examples/balls/matadd/verify/src/common/matadd_defs.svh b/examples/balls/matadd/verify/src/common/matadd_defs.svh new file mode 100644 index 000000000..5750adb01 --- /dev/null +++ b/examples/balls/matadd/verify/src/common/matadd_defs.svh @@ -0,0 +1,15 @@ +`ifndef MATADD_FUNCT7 +`error "MATADD_FUNCT7 must be provided by the selected Core ballISA" +`endif + +localparam int MATADD_CORE_FUNCT7 = `MATADD_FUNCT7; +localparam int MATADD_MAX_LINES = 1024; +localparam int MATADD_TIMEOUT_CYCLES = 100000; + +function automatic int unsigned matadd_require_bid(); + int unsigned bid; + if (!$value$plusargs("BID=%d", bid)) begin + `uvm_fatal("BID", "missing required plusarg +BID=") + end + return bid; +endfunction diff --git a/examples/balls/matadd/verify/src/common/matadd_items.svh b/examples/balls/matadd/verify/src/common/matadd_items.svh new file mode 100644 index 000000000..4ca129f43 --- /dev/null +++ b/examples/balls/matadd/verify/src/common/matadd_items.svh @@ -0,0 +1,97 @@ +class matadd_cmd_item extends bb_blink_cmd_item; + `uvm_object_utils(matadd_cmd_item) + + int unsigned lines; + int unsigned group_count; + int unsigned case_index; + + constraint legal_c { + funct7 == MATADD_CORE_FUNCT7[6:0]; + op1_en == 1'b1; + op2_en == 1'b1; + wr_spad_en == 1'b1; + op1_from_spad == 1'b1; + op2_from_spad == 1'b1; + meta_bank == 5'd0; + rs1 == 64'd0; + rs2 == 64'd0; + special == 64'd0; + is_sub == 1'b0; + sub_rob_id == 8'd0; + } + + function new(string name = "matadd_cmd_item"); + super.new(name); + endfunction + + function void load_case(int unsigned index, int unsigned ball_id); + if (index > 2) `uvm_fatal("CASE", $sformatf("unsupported case %0d", index)) + case_index = index; + this.bid = ball_id[4:0]; + funct7 = MATADD_CORE_FUNCT7[6:0]; + iter = (index == 2) ? MATADD_MAX_LINES : 16; + lines = iter; + group_count = (index == 0) ? 1 : 2; + op1_bank = 5'd0; + op2_bank = 5'd1; + wr_bank = 5'd2; + op1_col = group_count[4:0]; + op2_col = group_count[4:0]; + wr_col = group_count[4:0]; + rob_id = 4 + index; + op1_en = 1'b1; + op2_en = 1'b1; + wr_spad_en = 1'b1; + op1_from_spad = 1'b1; + op2_from_spad = 1'b1; + meta_bank = 5'd0; + rs1 = 64'd0; + rs2 = 64'd0; + special = 64'd0; + is_sub = 1'b0; + sub_rob_id = 8'd0; + endfunction + + function automatic bit [31:0] a_lane(int unsigned group, int unsigned line, int unsigned lane); + int unsigned offset; + offset = (group * lines + line) * 4 + lane; + if (case_index == 2) return 32'hffff_ff00 + offset; + return 32'h1000_0000 + offset * 32'h0001_0011; + endfunction + + function automatic bit [31:0] b_lane(int unsigned group, int unsigned line, int unsigned lane); + int unsigned offset; + offset = (group * lines + line) * 4 + lane; + if (case_index == 2) return 32'h0000_0200 + offset * 5; + return 32'he000_0000 + offset * 32'h0000_0103; + endfunction + + function automatic bit [127:0] a_word(int unsigned group, int unsigned line); + bit [127:0] word; + for (int lane = 0; lane < 4; lane++) word[lane*32+:32] = a_lane(group, line, lane); + return word; + endfunction + + function automatic bit [127:0] b_word(int unsigned group, int unsigned line); + bit [127:0] word; + for (int lane = 0; lane < 4; lane++) word[lane*32+:32] = b_lane(group, line, lane); + return word; + endfunction + + function automatic bit [127:0] sum_word(int unsigned group, int unsigned line); + bit [127:0] word; + for (int lane = 0; lane < 4; lane++) begin + word[lane*32+:32] = a_lane(group, line, lane) + b_lane(group, line, lane); + end + return word; + endfunction + + function void do_copy(uvm_object rhs); + matadd_cmd_item rhs_item; + super.do_copy(rhs); + if (!$cast(rhs_item, rhs)) `uvm_fatal("COPY", "rhs is not matadd_cmd_item") + lines = rhs_item.lines; + group_count = rhs_item.group_count; + case_index = rhs_item.case_index; + endfunction +endclass diff --git a/examples/balls/matadd/verify/src/env/matadd_env.svh b/examples/balls/matadd/verify/src/env/matadd_env.svh new file mode 100644 index 000000000..fe0e42c79 --- /dev/null +++ b/examples/balls/matadd/verify/src/env/matadd_env.svh @@ -0,0 +1,24 @@ +class matadd_env extends bb_blink_env #(2, 1); + `uvm_component_utils(matadd_env) + + matadd_scoreboard scb; + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function void build_phase(uvm_phase phase); + super.build_phase(phase); + scb = matadd_scoreboard::type_id::create("scb", this); + endfunction + + function void connect_phase(uvm_phase phase); + super.connect_phase(phase); + scb.mem_model = mem_model; + cmd_agent.stim_ap.connect(scb.stim_imp); + cmd_agent.cmd_ap.connect(scb.cmd_imp); + read_mon.read_ap.connect(scb.read_imp); + write_mon.write_ap.connect(scb.write_imp); + resp_mon.resp_ap.connect(scb.resp_imp); + endfunction +endclass diff --git a/examples/balls/matadd/verify/src/env/matadd_scoreboard.svh b/examples/balls/matadd/verify/src/env/matadd_scoreboard.svh new file mode 100644 index 000000000..5efeca2b3 --- /dev/null +++ b/examples/balls/matadd/verify/src/env/matadd_scoreboard.svh @@ -0,0 +1,151 @@ +class matadd_scoreboard extends uvm_scoreboard; + `uvm_component_utils(matadd_scoreboard) + + uvm_analysis_imp_stim #(bb_blink_cmd_item, matadd_scoreboard) stim_imp; + uvm_analysis_imp_cmd #(bb_blink_cmd_item, matadd_scoreboard) cmd_imp; + uvm_analysis_imp_read #(bb_blink_read_item, matadd_scoreboard) read_imp; + uvm_analysis_imp_write #(bb_blink_write_item, matadd_scoreboard) write_imp; + uvm_analysis_imp_resp #(bb_blink_resp_item, matadd_scoreboard) resp_imp; + + bb_blink_mem_model #(2, 1) mem_model; + matadd_cmd_item stim; + int unsigned read_count[2]; + int unsigned write_count; + int unsigned cmd_count; + int unsigned resp_count; + + function new(string name, uvm_component parent); + super.new(name, parent); + stim_imp = new("stim_imp", this); + cmd_imp = new("cmd_imp", this); + read_imp = new("read_imp", this); + write_imp = new("write_imp", this); + resp_imp = new("resp_imp", this); + endfunction + + function void write_stim(bb_blink_cmd_item item); + matadd_cmd_item source; + matadd_cmd_item clone; + if (stim != null) `uvm_fatal("SCB", "single outstanding command supported") + if (!$cast(source, item)) `uvm_fatal("SCB", "stimulus is not matadd_cmd_item") + if (!$cast(clone, source.clone())) `uvm_fatal("SCB", "failed to clone stimulus") + if (mem_model == null) `uvm_fatal("SCB", "mem_model handle not set") + stim = clone; + mem_model.clear_mem(); + for (int group = 0; group < stim.group_count; group++) begin + for (int line = 0; line < stim.lines; line++) begin + mem_model.load_word_g(stim.op1_bank, group, line, stim.a_word(group, line)); + mem_model.load_word_g(stim.op2_bank, group, line, stim.b_word(group, line)); + end + end + mem_model.arm(); + endfunction + + function matadd_cmd_item current_stim(string source); + if (stim == null) `uvm_fatal("SCB", $sformatf("%s observed before stimulus", source)) + return stim; + endfunction + + function int unsigned total_lines(); + return current_stim("COUNT").lines * current_stim("COUNT").group_count; + endfunction + + function void write_cmd(bb_blink_cmd_item item); + matadd_cmd_item expected; + expected = current_stim("CMD"); + if (item.bid !== expected.bid) `uvm_fatal("CMD", "bid mismatch") + if (item.funct7 !== expected.funct7) `uvm_fatal("CMD", "funct7 mismatch") + if (item.iter !== expected.iter) `uvm_fatal("CMD", "iter mismatch") + if (item.op1_bank !== expected.op1_bank || item.op2_bank !== expected.op2_bank || + item.wr_bank !== expected.wr_bank) + `uvm_fatal("CMD", "bank field mismatch") + if (item.op1_col !== expected.group_count || item.op2_col !== expected.group_count || + item.wr_col !== expected.group_count) + `uvm_fatal("CMD", "group field mismatch") + if (item.rob_id !== expected.rob_id) `uvm_fatal("CMD", "rob_id mismatch") + if (item.op1_en !== 1'b1 || item.op2_en !== 1'b1 || item.wr_spad_en !== 1'b1 || + item.op1_from_spad !== 1'b1 || item.op2_from_spad !== 1'b1) + `uvm_fatal("CMD", "enable field mismatch") + if (item.meta_bank !== 0 || item.rs1 !== 0 || item.rs2 !== 0 || item.special !== 0 || + item.is_sub !== 0 || item.sub_rob_id !== 0) + `uvm_fatal("CMD", "reserved field mismatch") + if (cmd_count != 0) `uvm_fatal("CMD", "extra command observed") + cmd_count++; + endfunction + + function void write_read(bb_blink_read_item item); + matadd_cmd_item expected; + int unsigned index; + int unsigned group; + int unsigned line; + expected = current_stim("READ"); + if (item.port < 0 || item.port > 1) `uvm_fatal("READ", "invalid read port") + index = read_count[item.port]; + if (index >= total_lines()) `uvm_fatal("READ", "extra read observed") + group = index / expected.lines; + line = index % expected.lines; + if (item.rob_id !== expected.rob_id) `uvm_fatal("READ", "rob_id mismatch") + if (item.bank_id !== (item.port == 0 ? expected.op1_bank : expected.op2_bank)) + `uvm_fatal("READ", "bank mismatch") + if (item.group_id !== group || item.addr !== line) + `uvm_fatal("READ", $sformatf( + "address mismatch port=%0d group=%0d line=%0d", item.port, item.group_id, item.addr + )) + read_count[item.port]++; + endfunction + + function void write_write(bb_blink_write_item item); + matadd_cmd_item expected; + int unsigned group; + int unsigned line; + expected = current_stim("WRITE"); + if (write_count >= total_lines()) `uvm_fatal("WRITE", "extra write observed") + group = write_count / expected.lines; + line = write_count % expected.lines; + if (item.port != 0 || item.bank_id !== expected.wr_bank || item.rob_id !== expected.rob_id) + `uvm_fatal("WRITE", "port, bank, or rob_id mismatch") + if (item.group_id !== group || item.addr !== line) + `uvm_fatal("WRITE", $sformatf("address mismatch group=%0d line=%0d", item.group_id, item.addr + )) + if (item.mask !== 16'hffff) `uvm_fatal("WRITE", "mask mismatch") + if (item.data !== expected.sum_word(group, line)) + `uvm_fatal("WRITE", $sformatf("data mismatch group=%0d line=%0d", group, line)) + write_count++; + endfunction + + function void write_resp(bb_blink_resp_item item); + matadd_cmd_item expected; + expected = current_stim("RESP"); + if (resp_count != 0) `uvm_fatal("RESP", "extra response observed") + if (item.rob_id !== expected.rob_id || item.is_sub !== 0 || item.sub_rob_id !== 0) + `uvm_fatal("RESP", "response mismatch") + resp_count++; + endfunction + + function bit done(); + return cmd_count == 1 && read_count[0] == total_lines() && read_count[1] == total_lines() && + write_count == total_lines() && resp_count == 1; + endfunction + + function void reset_counters(); + stim = null; + read_count[0] = 0; + read_count[1] = 0; + write_count = 0; + cmd_count = 0; + resp_count = 0; + endfunction + + function void check_phase(uvm_phase phase); + super.check_phase(phase); + if (!done()) + `uvm_fatal("SCB", $sformatf( + "incomplete cmds=%0d reads=%0d/%0d writes=%0d resp=%0d", + cmd_count, + read_count[0], + read_count[1], + write_count, + resp_count + )) + endfunction +endclass diff --git a/examples/balls/matadd/verify/src/pkg/matadd_pkg.sv b/examples/balls/matadd/verify/src/pkg/matadd_pkg.sv new file mode 100644 index 000000000..74dc74775 --- /dev/null +++ b/examples/balls/matadd/verify/src/pkg/matadd_pkg.sv @@ -0,0 +1,12 @@ +package matadd_pkg; + import uvm_pkg::*; + import bb_uvm_pkg::*; + `include "uvm_macros.svh" + + `include "common/matadd_defs.svh" + `include "common/matadd_items.svh" + `include "seq/matadd_sequences.svh" + `include "env/matadd_scoreboard.svh" + `include "env/matadd_env.svh" + `include "tests/matadd_test.svh" +endpackage diff --git a/examples/balls/matadd/verify/src/seq/matadd_sequences.svh b/examples/balls/matadd/verify/src/seq/matadd_sequences.svh new file mode 100644 index 000000000..e621df916 --- /dev/null +++ b/examples/balls/matadd/verify/src/seq/matadd_sequences.svh @@ -0,0 +1,18 @@ +class matadd_seq extends uvm_sequence #(bb_blink_cmd_item); + `uvm_object_utils(matadd_seq) + + int unsigned case_index; + int unsigned bid; + + function new(string name = "matadd_seq"); + super.new(name); + endfunction + + task body(); + matadd_cmd_item item; + item = matadd_cmd_item::type_id::create("item"); + start_item(item); + item.load_case(case_index, bid); + finish_item(item); + endtask +endclass diff --git a/examples/balls/matadd/verify/src/tb_top.sv b/examples/balls/matadd/verify/src/tb_top.sv new file mode 100644 index 000000000..677b20a9e --- /dev/null +++ b/examples/balls/matadd/verify/src/tb_top.sv @@ -0,0 +1,96 @@ +module tb_top; + import uvm_pkg::*; + import matadd_pkg::*; + + bb_blink_if #(2, 1) intf (); + + MatAddBall dut ( + .clock(intf.clock), + .reset(intf.reset), + .io_channelReady(1'b1), + .io_cmdReq_ready(intf.cmd_req_ready), + .io_cmdReq_valid(intf.cmd_req_valid), + .io_cmdReq_bits_cmd_bid(intf.cmd_req_bits_cmd_bid), + .io_cmdReq_bits_cmd_funct7(intf.cmd_req_bits_cmd_funct7), + .io_cmdReq_bits_cmd_iter(intf.cmd_req_bits_cmd_iter), + .io_cmdReq_bits_cmd_op1_en(intf.cmd_req_bits_cmd_op1_en), + .io_cmdReq_bits_cmd_op2_en(intf.cmd_req_bits_cmd_op2_en), + .io_cmdReq_bits_cmd_wr_spad_en(intf.cmd_req_bits_cmd_wr_spad_en), + .io_cmdReq_bits_cmd_op1_from_spad(intf.cmd_req_bits_cmd_op1_from_spad), + .io_cmdReq_bits_cmd_op2_from_spad(intf.cmd_req_bits_cmd_op2_from_spad), + .io_cmdReq_bits_cmd_special(intf.cmd_req_bits_cmd_special), + .io_cmdReq_bits_cmd_op1_bank(intf.cmd_req_bits_cmd_op1_bank[3:0]), + .io_cmdReq_bits_cmd_op2_bank(intf.cmd_req_bits_cmd_op2_bank[3:0]), + .io_cmdReq_bits_cmd_wr_bank(intf.cmd_req_bits_cmd_wr_bank[3:0]), + .io_cmdReq_bits_cmd_op1_col(intf.cmd_req_bits_cmd_op1_col), + .io_cmdReq_bits_cmd_op2_col(intf.cmd_req_bits_cmd_op2_col), + .io_cmdReq_bits_cmd_wr_col(intf.cmd_req_bits_cmd_wr_col), + .io_cmdReq_bits_cmd_meta_bank(intf.cmd_req_bits_cmd_meta_bank[3:0]), + .io_cmdReq_bits_cmd_rs1(intf.cmd_req_bits_cmd_rs1), + .io_cmdReq_bits_cmd_rs2(intf.cmd_req_bits_cmd_rs2), + .io_cmdReq_bits_rob_id(intf.cmd_req_bits_rob_id), + .io_cmdReq_bits_is_sub(intf.cmd_req_bits_is_sub), + .io_cmdReq_bits_sub_rob_id(intf.cmd_req_bits_sub_rob_id), + .io_cmdResp_ready(intf.cmd_resp_ready), + .io_cmdResp_valid(intf.cmd_resp_valid), + .io_cmdResp_bits_rob_id(intf.cmd_resp_bits_rob_id), + .io_cmdResp_bits_is_sub(intf.cmd_resp_bits_is_sub), + .io_cmdResp_bits_sub_rob_id(intf.cmd_resp_bits_sub_rob_id), + .io_bankRead_0_bank_id(intf.bank_read_bank_id[0][3:0]), + .io_bankRead_0_rob_id(intf.bank_read_rob_id[0]), + .io_bankRead_0_group_id(intf.bank_read_group_id[0][3:0]), + .io_bankRead_0_io_req_ready(intf.bank_read_req_ready[0]), + .io_bankRead_0_io_req_valid(intf.bank_read_req_valid[0]), + .io_bankRead_0_io_req_bits_addr(intf.bank_read_req_addr[0]), + .io_bankRead_0_io_resp_ready(intf.bank_read_resp_ready[0]), + .io_bankRead_0_io_resp_valid(intf.bank_read_resp_valid[0]), + .io_bankRead_0_io_resp_bits_data(intf.bank_read_resp_data[0]), + .io_bankRead_1_bank_id(intf.bank_read_bank_id[1][3:0]), + .io_bankRead_1_rob_id(intf.bank_read_rob_id[1]), + .io_bankRead_1_group_id(intf.bank_read_group_id[1][3:0]), + .io_bankRead_1_io_req_ready(intf.bank_read_req_ready[1]), + .io_bankRead_1_io_req_valid(intf.bank_read_req_valid[1]), + .io_bankRead_1_io_req_bits_addr(intf.bank_read_req_addr[1]), + .io_bankRead_1_io_resp_ready(intf.bank_read_resp_ready[1]), + .io_bankRead_1_io_resp_valid(intf.bank_read_resp_valid[1]), + .io_bankRead_1_io_resp_bits_data(intf.bank_read_resp_data[1]), + .io_bankWrite_0_bank_id(intf.bank_write_bank_id[0][3:0]), + .io_bankWrite_0_rob_id(intf.bank_write_rob_id[0]), + .io_bankWrite_0_group_id(intf.bank_write_group_id[0][3:0]), + .io_bankWrite_0_io_req_ready(intf.bank_write_req_ready[0]), + .io_bankWrite_0_io_req_valid(intf.bank_write_req_valid[0]), + .io_bankWrite_0_io_req_bits_addr(intf.bank_write_req_addr[0]), + .io_bankWrite_0_io_req_bits_mask_0(intf.bank_write_req_mask[0][0]), + .io_bankWrite_0_io_req_bits_mask_1(intf.bank_write_req_mask[0][1]), + .io_bankWrite_0_io_req_bits_mask_2(intf.bank_write_req_mask[0][2]), + .io_bankWrite_0_io_req_bits_mask_3(intf.bank_write_req_mask[0][3]), + .io_bankWrite_0_io_req_bits_mask_4(intf.bank_write_req_mask[0][4]), + .io_bankWrite_0_io_req_bits_mask_5(intf.bank_write_req_mask[0][5]), + .io_bankWrite_0_io_req_bits_mask_6(intf.bank_write_req_mask[0][6]), + .io_bankWrite_0_io_req_bits_mask_7(intf.bank_write_req_mask[0][7]), + .io_bankWrite_0_io_req_bits_mask_8(intf.bank_write_req_mask[0][8]), + .io_bankWrite_0_io_req_bits_mask_9(intf.bank_write_req_mask[0][9]), + .io_bankWrite_0_io_req_bits_mask_10(intf.bank_write_req_mask[0][10]), + .io_bankWrite_0_io_req_bits_mask_11(intf.bank_write_req_mask[0][11]), + .io_bankWrite_0_io_req_bits_mask_12(intf.bank_write_req_mask[0][12]), + .io_bankWrite_0_io_req_bits_mask_13(intf.bank_write_req_mask[0][13]), + .io_bankWrite_0_io_req_bits_mask_14(intf.bank_write_req_mask[0][14]), + .io_bankWrite_0_io_req_bits_mask_15(intf.bank_write_req_mask[0][15]), + .io_bankWrite_0_io_req_bits_data(intf.bank_write_req_data[0]), + .io_bankWrite_0_io_resp_ready(intf.bank_write_resp_ready[0]), + .io_bankWrite_0_io_resp_valid(intf.bank_write_resp_valid[0]), + .io_bankWrite_0_io_resp_bits_ok(intf.bank_write_resp_ok[0]), + .io_subRobReq_ready(intf.sub_rob_req_ready) + ); + + initial begin + intf.clock = 1'b0; + intf.reset = 1'b1; + forever #5 intf.clock = ~intf.clock; + end + + initial begin + uvm_config_db#(virtual bb_blink_if #(2, 1))::set(null, "*", "vif", intf); + run_test("matadd_ball_test"); + end +endmodule diff --git a/examples/balls/matadd/verify/src/tests/matadd_test.svh b/examples/balls/matadd/verify/src/tests/matadd_test.svh new file mode 100644 index 000000000..1115ec042 --- /dev/null +++ b/examples/balls/matadd/verify/src/tests/matadd_test.svh @@ -0,0 +1,62 @@ +class matadd_ball_test extends uvm_test; + `uvm_component_utils(matadd_ball_test) + + typedef virtual bb_blink_if #(2, 1) vif_t; + vif_t vif; + matadd_env env; + + function new(string name, uvm_component parent); + super.new(name, parent); + endfunction + + function void build_phase(uvm_phase phase); + super.build_phase(phase); + if (!uvm_config_db#(vif_t)::get(this, "", "vif", vif)) + `uvm_fatal("NOVIF", "bb_blink_if not found") + env = matadd_env::type_id::create("env", this); + endfunction + + task run_phase(uvm_phase phase); + int unsigned bid; + bid = matadd_require_bid(); + phase.raise_objection(this); + run_case(0, "ONE_GROUP_16_LINES", bid); + run_case(1, "TWO_GROUPS_16_LINES", bid); + run_case(2, "TWO_GROUPS_1024_LINES", bid); + phase.drop_objection(this); + endtask + + task run_case(int unsigned index, string name, int unsigned bid); + matadd_seq seq; + int cycles; + apply_reset(); + env.scb.reset_counters(); + seq = matadd_seq::type_id::create("seq"); + seq.case_index = index; + seq.bid = bid; + seq.start(env.cmd_agent.seqr); + cycles = 0; + while (!env.scb.done()) begin + @(posedge vif.clock); + cycles++; + if (cycles > MATADD_TIMEOUT_CYCLES) + `uvm_fatal("TIMEOUT", $sformatf( + "%s timed out reads=%0d/%0d writes=%0d resp=%0d", + name, + env.scb.read_count[0], + env.scb.read_count[1], + env.scb.write_count, + env.scb.resp_count + )) + end + `uvm_info("MATADD", $sformatf("%s passed in %0d cycles", name, cycles), UVM_LOW) + endtask + + task apply_reset(); + vif.reset = 1'b1; + @(posedge vif.clock); + repeat (4) @(posedge vif.clock); + vif.reset = 1'b0; + repeat (2) @(posedge vif.clock); + endtask +endclass diff --git a/examples/balls/matadd/workloads/ctests/CMakeLists.txt b/examples/balls/matadd/workloads/ctests/CMakeLists.txt new file mode 100644 index 000000000..985208347 --- /dev/null +++ b/examples/balls/matadd/workloads/ctests/CMakeLists.txt @@ -0,0 +1,4 @@ +add_buckyball_ctests( + matadd_test.c + matadd_bank_1024x8_i32_test.c +) diff --git a/examples/balls/matadd/workloads/ctests/matadd_bank_1024x8_i32_test.c b/examples/balls/matadd/workloads/ctests/matadd_bank_1024x8_i32_test.c new file mode 100644 index 000000000..2464974a6 --- /dev/null +++ b/examples/balls/matadd/workloads/ctests/matadd_bank_1024x8_i32_test.c @@ -0,0 +1,48 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +#define LINES BANK_LINES +#define GROUPS 2 +#define VALUES (LINES * GROUPS * 4) + +static int32_t a[VALUES] __attribute__((aligned(64))); +static int32_t b[VALUES] __attribute__((aligned(64))); +static int32_t c[VALUES] __attribute__((aligned(64))); + +int main(void) { + const uint32_t a_bank = 0; + const uint32_t b_bank = 1; + const uint32_t c_bank = 2; + + for (int i = 0; i < VALUES; ++i) { + a[i] = (i * 73) - 300000; + b[i] = 200000 - (i * 41); + } + + bb_mem_alloc(a_bank, 1, GROUPS); + bb_mem_alloc(b_bank, 1, GROUPS); + bb_mem_alloc(c_bank, 1, GROUPS); + bb_mvin((uintptr_t)a, a_bank, LINES, 1); + bb_mvin((uintptr_t)b, b_bank, LINES, 1); + bb_matadd(a_bank, b_bank, c_bank, LINES); + bb_mvout((uintptr_t)c, c_bank, LINES, 1); + bb_fence(); + bb_mem_release(a_bank); + bb_mem_release(b_bank); + bb_mem_release(c_bank); + + for (int i = 0; i < VALUES; ++i) { + uint32_t expected = (uint32_t)a[i] + (uint32_t)b[i]; + if ((uint32_t)c[i] != expected) { + printf("matadd bank mismatch at %d: got %d expected %d\n", i, c[i], + (int32_t)expected); + return 1; + } + } + printf("matadd_bank_1024x8_i32 PASSED\n"); + return 0; +} diff --git a/examples/balls/matadd/workloads/ctests/matadd_test.c b/examples/balls/matadd/workloads/ctests/matadd_test.c new file mode 100644 index 000000000..22a39b945 --- /dev/null +++ b/examples/balls/matadd/workloads/ctests/matadd_test.c @@ -0,0 +1,48 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include + +#define LINES 16 +#define GROUPS 2 +#define VALUES (LINES * GROUPS * 4) + +static int32_t a[VALUES] __attribute__((aligned(64))); +static int32_t b[VALUES] __attribute__((aligned(64))); +static int32_t c[VALUES] __attribute__((aligned(64))); + +int main(void) { + const uint32_t a_bank = 0; + const uint32_t b_bank = 1; + const uint32_t c_bank = 2; + + for (int i = 0; i < VALUES; ++i) { + a[i] = (i * 23) - 700; + b[i] = 900 - (i * 17); + } + + bb_mem_alloc(a_bank, 1, GROUPS); + bb_mem_alloc(b_bank, 1, GROUPS); + bb_mem_alloc(c_bank, 1, GROUPS); + bb_mvin((uintptr_t)a, a_bank, LINES, 1); + bb_mvin((uintptr_t)b, b_bank, LINES, 1); + bb_matadd(a_bank, b_bank, c_bank, LINES); + bb_mvout((uintptr_t)c, c_bank, LINES, 1); + bb_fence(); + bb_mem_release(a_bank); + bb_mem_release(b_bank); + bb_mem_release(c_bank); + + for (int i = 0; i < VALUES; ++i) { + uint32_t expected = (uint32_t)a[i] + (uint32_t)b[i]; + if ((uint32_t)c[i] != expected) { + printf("matadd mismatch at %d: got %d expected %d\n", i, c[i], + (int32_t)expected); + return 1; + } + } + printf("matadd_test PASSED\n"); + return 0; +} diff --git a/examples/balls/matadd/workloads/isa/matadd.h b/examples/balls/matadd/workloads/isa/matadd.h new file mode 100644 index 000000000..08b701742 --- /dev/null +++ b/examples/balls/matadd/workloads/isa/matadd.h @@ -0,0 +1,12 @@ +#ifndef _BB_MATADD_H_ +#define _BB_MATADD_H_ + +#include +#include + +#define bb_matadd(a_bank, b_bank, c_bank, iter) \ + BUCKYBALL_INSTRUCTION_R_R((BB_BANK0(a_bank) | BB_BANK1(b_bank) | \ + BB_BANK2(c_bank) | BB_ITER(iter)), \ + 0, BB_FUNC7(MATADD)) + +#endif // _BB_MATADD_H_ diff --git a/examples/balls/matadd/workloads/mlir_tests/CMakeLists.txt b/examples/balls/matadd/workloads/mlir_tests/CMakeLists.txt new file mode 100644 index 000000000..5275add61 --- /dev/null +++ b/examples/balls/matadd/workloads/mlir_tests/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(bank) diff --git a/examples/balls/matadd/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/matadd/workloads/mlir_tests/bank/CMakeLists.txt new file mode 100644 index 000000000..84ce626b8 --- /dev/null +++ b/examples/balls/matadd/workloads/mlir_tests/bank/CMakeLists.txt @@ -0,0 +1,8 @@ +set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) +set(BUCKYBALL_MLIR_TEST_PREFIX bank) + +add_buckyball_mlir_contract(matadd_bank_1024x8_i32_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) + +add_buckyball_mlir_test(matadd_bank_1024x8_i32 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32.mlir b/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32.mlir new file mode 100644 index 000000000..bd36091ce --- /dev/null +++ b/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32.mlir @@ -0,0 +1,44 @@ +func.func private @check_result(memref<1024x8xi32>) -> () + +func.func @main() -> i8 { + %zeroI8 = arith.constant 0 : i8 + %zero = arith.constant 0 : index + %one = arith.constant 1 : index + %lines = arith.constant 1024 : i64 + %stride = arith.constant 1 : i64 + %limit = arith.constant 1024 : index + %columns = arith.constant 8 : index + %a = memref.alloc() alignment = 64 : memref<1024x8xi32> + %b = memref.alloc() alignment = 64 : memref<1024x8xi32> + %c = memref.alloc() alignment = 64 : memref<1024x8xi32> + + scf.for %line = %zero to %limit step %one { + scf.for %column = %zero to %columns step %one { + %aValue = arith.index_cast %line : index to i32 + %bValue = arith.index_cast %column : index to i32 + memref.store %aValue, %a[%line, %column] : memref<1024x8xi32> + memref.store %bValue, %b[%line, %column] : memref<1024x8xi32> + } + } + + %aBank = buckyball.bank_alloc {col = 2 : i64} + %bBank = buckyball.bank_alloc {col = 2 : i64} + %cBank = buckyball.bank_alloc {col = 2 : i64} + %aLoaded = buckyball.bank_mvin %a %aBank %lines %stride + : memref<1024x8xi32> i64 i64 i64 + %bLoaded = buckyball.bank_mvin %b %bBank %lines %stride + : memref<1024x8xi32> i64 i64 i64 + %result = buckyball.bank_matadd %aLoaded %bLoaded %cBank %lines : i64 i64 i64 i64 + %stored = buckyball.bank_mvout %c %result %lines %stride + : memref<1024x8xi32> i64 i64 i64 + buckyball.fence + buckyball.bank_release %aLoaded : i64 + buckyball.bank_release %bLoaded : i64 + buckyball.bank_release %stored : i64 + + func.call @check_result(%c) : (memref<1024x8xi32>) -> () + memref.dealloc %a : memref<1024x8xi32> + memref.dealloc %b : memref<1024x8xi32> + memref.dealloc %c : memref<1024x8xi32> + return %zeroI8 : i8 +} diff --git a/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32_contract.mlir b/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32_contract.mlir new file mode 100644 index 000000000..bc9d32e1a --- /dev/null +++ b/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32_contract.mlir @@ -0,0 +1,15 @@ +// CHECK: buckyball.matadd +// CHECK-NOT: buckyball.bank_matadd + +func.func @main() -> i8 { + %zero = arith.constant 0 : i8 + %lines = arith.constant 1024 : i64 + %a = buckyball.bank_alloc {col = 2 : i64} + %b = buckyball.bank_alloc {col = 2 : i64} + %c = buckyball.bank_alloc {col = 2 : i64} + %result = buckyball.bank_matadd %a %b %c %lines : i64 i64 i64 i64 + buckyball.bank_release %a : i64 + buckyball.bank_release %b : i64 + buckyball.bank_release %result : i64 + return %zero : i8 +} diff --git a/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32_main.cpp b/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32_main.cpp new file mode 100644 index 000000000..c91731640 --- /dev/null +++ b/examples/balls/matadd/workloads/mlir_tests/bank/matadd_bank_1024x8_i32_main.cpp @@ -0,0 +1,39 @@ +#include +#include +#include + +static void fail() { +#ifdef BAREMETAL + volatile uint32_t *sim_exit = + reinterpret_cast(0x60000000); + *sim_exit = 1; + while (true) { + } +#else + std::exit(1); +#endif +} + +extern "C" void check_result(int32_t *allocated, int32_t *aligned, + int64_t offset, int64_t size0, int64_t size1, + int64_t stride0, int64_t stride1) { + (void)allocated; + if (size0 != 1024 || size1 != 8 || stride0 != 8 || stride1 != 1) { + std::printf("FAILED: matadd bank shape %ldx%ld\n", size0, size1); + fail(); + } + int32_t *c = aligned + offset; + for (int64_t line = 0; line < size0; ++line) { + for (int64_t column = 0; column < size1; ++column) { + size_t index = static_cast(line * stride0 + column * stride1); + uint32_t expected = + static_cast(line) + static_cast(column); + if (static_cast(c[index]) != expected) { + std::printf("FAILED: matadd bank[%ld][%ld] expected %d got %d\n", line, + column, static_cast(expected), c[index]); + fail(); + } + } + } + std::printf("PASSED: matadd bank 1024x8\n"); +} diff --git a/examples/balls/matrix/README.md b/examples/balls/matrix/README.md deleted file mode 100644 index e750d9f81..000000000 --- a/examples/balls/matrix/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# MatrixBall - -Pebble MatrixBall contract (fail-hard). Stack order: ctest → bemu → compiler → mlir_tests → RTL → UVM. - -## Instruction (funct7 = 65) - -| Field | Bits | Meaning | -| --- | --- | --- | -| `op1_bank` | `rs1[9:0]` | A bank | -| `op2_bank` | `rs1[19:10]` | B bank | -| `wr_bank` | `rs1[29:20]` | C bank (`cols=4`) | -| `op1_base` | `rs1[36:30]` | A start row | -| `op2_base` | `rs1[43:37]` | B start row | -| `wr_base` | `rs1[50:44]` | C start block | -| `M` | `rs2[11:0]` | rows of A/C | -| `N` | `rs2[23:12]` | cols of B/C | -| `K` | `rs2[35:24]` | cols of A / rows of B | -| `mode` | `rs2[36]` | `0` OS, `1` WS | - -C macros: `bb_matrix_mnk_mode`, `bb_matrix_mnk`, `bb_matrix_os`, `bb_matrix_ws`. -No cross-instruction ACC phases. - -## Storage - -- A/B: `int8`, C: `int32`. Bank row = 128 bit (16×i8 or 4×i32). -- A: M-tile → K-tile; each tile 16×16; pad zeros. -- B: N-tile → K-tile; each tile 16×16; pad zeros. -- C: each block is one row of one N-tile (≤16 i32), striped across `groupBase+0..3` at the same address. Final layout matches OS M-tile / N-tile / row order (WS remaps to the same addresses). - -## CTests - -Under `workloads/ctests/`. Pack helpers in `matrix_test_common.h`. Every test compares CPU golden. - -- Small tests: hand-written / deterministic inputs; in bemu and Verilator regression. -- Bank tests (`matrix_bank_*`, ≥1024 A rows): random inputs; bemu regression only (not Verilator). diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixBall.scala b/examples/balls/matrix/arch/src/main/scala/MatrixBall.scala deleted file mode 100644 index f21aaed61..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixBall.scala +++ /dev/null @@ -1,45 +0,0 @@ -package examples.balls.matrix - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} -import framework.balldomain.blink.{BallStatus, BlinkIO, HasBallStatus, HasBlink, SubRobRow} -import framework.balldomain.blink.mmio.MmioRead -import framework.top.GlobalConfig - -@instantiable -class MatrixBall(val b: GlobalConfig) extends Module with HasBlink with HasBallStatus { - - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "MatrixBall") - .getOrElse(throw new IllegalArgumentException("MatrixBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW - - @public - val io = IO(new BlinkIO(b, inBW, outBW)) - - def blink: BlinkIO = io - def status: BallStatus = io.status - dontTouch(io) - - val matrixArrayUnit: Instance[MatrixUnit] = Instantiate(new MatrixUnit(b)) - - matrixArrayUnit.io.cmdReq <> io.cmdReq - matrixArrayUnit.io.cmdResp <> io.cmdResp - - for (i <- 0 until inBW) { - matrixArrayUnit.io.bankRead(i) <> io.bankRead(i) - } - - for (i <- 0 until outBW) { - matrixArrayUnit.io.bankWrite(i) <> io.bankWrite(i) - } - - io.status <> matrixArrayUnit.io.status - - io.subRobReq.valid := false.B - io.subRobReq.bits := SubRobRow.tieOff(b) - - // MMIO: this Ball does not consume MMIO metadata - MmioRead.tieOff(io.mmioRead) -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixCtrl.scala b/examples/balls/matrix/arch/src/main/scala/MatrixCtrl.scala deleted file mode 100644 index cd7c09aab..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixCtrl.scala +++ /dev/null @@ -1,534 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import framework.balldomain.rs.{BallRsComplete, BallRsIssue} -import framework.top.GlobalConfig - -@instantiable -class SystolicArrayCtrl(val b: GlobalConfig) extends Module { - /** - * 脉动阵列控制器。 - * - * Ctrl 将一条 MxNxK 矩阵乘命令切分为 16x16x16 tile：一侧向 Load 发出 - * 数据/元数据请求，另一侧在 Store 请求写入时给出 C 行的目标地址。只有 - * 全部 tile 已发射且全部结果行写完后，才向前端返回该命令的完成响应。 - */ - private val tile = SystolicArrayConst.Tile - private val abRows = b.memDomain.bankEntries - private val bankWidth = log2Up(b.memDomain.bankNum) - private val groupWidth = log2Up(b.memDomain.bankNum) - private val addrWidth = log2Up(b.memDomain.bankEntries) - private val countWidth = 32 - private val wsReuseTiles = SystolicArrayConst.wsReuseTiles(b) - - require( - b.memDomain.bankEntries % tile == 0, - "SystolicArrayCtrl expects bankEntries to be an integer number of 16-row A/B tiles" - ) - - @public - val io = IO(new Bundle { - // 前端命令入口与完成出口。cmdReq 可排队，cmdResp_o 仅对应当前 task。 - val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) - val cmdResp_o = Decoupled(new BallRsComplete(b)) - - // Load 接收一个完整 tile 的读地址、有效范围和 EX 元数据。 - val ctrl_ld_o = Decoupled(new SystolicCtrlLoadReq(b)) - - // Ctrl 主动按结果顺序发送写回描述符；Store 以 ready 反馈其两项地址 FIFO 空位。 - val store_ctrl_resp_o = Decoupled(new SystolicStoreCtrlResp(b)) - val store_done_i = Input(Bool()) - - val busy_o = Output(Bool()) - }) - - // 将可能宽于或窄于目标字段的地址/计数截断或零扩展到目标宽度。 - private def fitTo(x: UInt, width: Int): UInt = - if (x.getWidth >= width) x(width - 1, 0) else x.pad(width) - - // 维度按 16 向上取整，得到 M/N/K 方向的 tile 个数。 - private def ceilDiv16(x: UInt): UInt = - ((x.pad(countWidth) +& (tile - 1).U) >> log2Ceil(tile)).pad(countWidth) - - // A/B 的每个 tile 固定占用 16 行。先计算相对起始行，再拆成 bank group 和 - // group 内地址；包含补零行的固定 tile 布局仍可跨越 group 边界。 - private def abGroup(base: UInt, rowOffset: UInt): UInt = { - val linear = base.pad(countWidth) + rowOffset.pad(countWidth) - fitTo(linear / abRows.U, groupWidth) - } - - private def abRow(base: UInt, rowOffset: UInt): UInt = { - val linear = base.pad(countWidth) + rowOffset.pad(countWidth) - fitTo(linear % abRows.U, addrWidth) - } - - // C 的四个 128-bit beat 条带化到连续四个 group。一个线性 C block 对应一行 - // 16 个 int32：groupBase+0..3 分别保存列 0..3、4..7、8..11、12..15；同一 - // group quartet 的地址相同。每个物理 group 写满后，下一 quartet 从 group+4 - // 开始，因此四路写会命中四块独立的单端口 SRAM。 - private def cGroupBase(base: UInt, blockCount: UInt): UInt = { - val linearBlock = base.pad(countWidth) + blockCount.pad(countWidth) - val groupPage = linearBlock / b.memDomain.bankEntries.U - fitTo(groupPage << log2Ceil(SystolicArrayConst.StoreWritePorts), groupWidth) - } - - private def cRowAddr(base: UInt, blockCount: UInt): UInt = { - val linearBlock = base.pad(countWidth) + blockCount.pad(countWidth) - fitTo(linearBlock % b.memDomain.bankEntries.U, addrWidth) - } - - // 返回边缘 tile 的真实长度；完整 tile 为 16，最后一个 tile 可能更短。 - private def validTileExtent(dim: UInt, tileIndex: UInt): UInt = { - val remaining = dim.pad(countWidth) - (tileIndex << log2Ceil(tile)) - Mux(remaining >= tile.U, tile.U(5.W), fitTo(remaining, 5)) - } - - // 命令队列接收前端任务；退休队列按同一顺序保留每个任务的 Store 地址状态。 - // 两者分离后，新任务的 Load tile 不必等待旧任务写回完成。 - val issueQ = Module(new Queue(new BallRsIssue(b), entries = 4)) - issueQ.io.enq.valid := io.cmdReq.valid - issueQ.io.enq.bits := io.cmdReq.bits - io.cmdReq.ready := issueQ.io.enq.ready - - val retireQ = Module(new Queue(new SystolicRetireMeta(b), entries = 4)) - val storeMetaQ = Module(new Queue(new SystolicStoreMeta(b), entries = 4)) - val completeQ = Module(new Queue(new BallRsComplete(b), entries = 4)) - val completionReservations = RegInit(0.U(log2Ceil(4 + 1).W)) - - // task 与其 tile 游标只描述当前发射任务。它的最后一个 tile 被 Load 接收后， - // 可在下一个周期直接装入 issueQ 的后继任务，而旧任务仍停留在 retireQ 队首。 - val issueActive = RegInit(false.B) - val task = RegInit(0.U.asTypeOf(new SystolicTask(b))) - - // 当前发射任务的三个逻辑维度 tile 总数和 (M,N,K) 发射坐标。 - val mTiles = RegInit(0.U(countWidth.W)) - val nTiles = RegInit(0.U(countWidth.W)) - val kTiles = RegInit(0.U(countWidth.W)) - - val mt = RegInit(0.U(countWidth.W)) - val nt = RegInit(0.U(countWidth.W)) - val kt = RegInit(0.U(countWidth.W)) - // WS 一批最多复用 wsReuseTiles 个 M tile；wsBatchBase 是该批第一个 M tile 的下标。 - val wsBatchBase = RegInit(0.U(countWidth.W)) - - // WS 下 wsLoadPeB=true 表示本次请求要以 B 更新 PE 权重；false 时仅读 A。 - val wsLoadPeB = RegInit(false.B) - val wsWeightGeneration = RegInit(false.B) - val wsPrefetchedWeights = RegInit(false.B) - - // 完成游标只随 store_done_i 前进，用于判断 retireQ 队首任务何时真正写完。 - val storeDoneMt = RegInit(0.U(countWidth.W)) - val storeDoneNt = RegInit(0.U(countWidth.W)) - val storeDoneRowIdx = RegInit(0.U(5.W)) - - // 发放游标在描述符进入 Ctrl 的响应队列时前进。它可领先完成游标，使 Store - // 能连续取得后续行的地址信息而不等待任何 bank 写响应。 - val storeMetaMt = RegInit(0.U(countWidth.W)) - val storeMetaNt = RegInit(0.U(countWidth.W)) - val storeMetaRowIdx = RegInit(0.U(5.W)) - val storeMetaBlockCount = RegInit(0.U(countWidth.W)) - - // 描述符直接进入 Store 持有的两项地址 FIFO；这里仅记录已被 Store 接收、 - // 但尚未完成真实 bank 写回的行数。 - val storeOutstanding = RegInit(0.U(5.W)) - - // 指令编码：rs2 携带 M/N/K/mode，rs1 携带 A/B/C 的 bank 与起始地址。 - val rs1 = issueQ.io.deq.bits.cmd.rs1 - val rs2 = issueQ.io.deq.bits.cmd.rs2 - - val issueM = rs2(11, 0) - val issueN = rs2(23, 12) - val issueK = rs2(35, 24) - val issueMode = rs2(36) - - // issueQ 队首命令的组合解码；仅在开始或替换当前发射任务时锁存到 task。 - val nextTask = WireDefault(0.U.asTypeOf(new SystolicTask(b))) - nextTask.mode := issueMode - nextTask.m := issueM - nextTask.n := issueN - nextTask.k := issueK - nextTask.op1_bank := fitTo(rs1(9, 0), bankWidth) - nextTask.op2_bank := fitTo(rs1(19, 10), bankWidth) - nextTask.wr_bank := fitTo(rs1(29, 20), bankWidth) - nextTask.op1_base := fitTo(rs1(36, 30), addrWidth) - nextTask.op2_base := fitTo(rs1(43, 37), addrWidth) - nextTask.wr_base := fitTo(rs1(50, 44), addrWidth) - nextTask.rob_id := issueQ.io.deq.bits.rob_id - nextTask.is_sub := issueQ.io.deq.bits.is_sub - nextTask.sub_rob_id := issueQ.io.deq.bits.sub_rob_id - - val currentKind = Wire(UInt(2.W)) - // EX 根据该标记选择新建累加上下文或继续已有的 K 维累加链。 - val currentKTileKind = Mux( - kTiles === 1.U, - SystolicKTileKind.DIRECT, - Mux(kt === 0.U, SystolicKTileKind.FIRST, - Mux(kt === (kTiles - 1.U), SystolicKTileKind.LAST, SystolicKTileKind.MIDDLE)) - ) - val currentValidM = validTileExtent(task.m, mt) - val currentValidN = validTileExtent(task.n, nt) - val currentValidK = validTileExtent(task.k, kt) - // A 按 (M tile, K tile) 排列，B 按 (N tile, K tile) 排列；无论边缘 tile - // 有多少有效行，每个 tile 的起始地址都固定间隔 16 行。 - val aRowOffset = - (mt.pad(countWidth) * kTiles.pad(countWidth) + kt.pad(countWidth)) << log2Ceil(tile) - val bRowOffset = - (nt.pad(countWidth) * kTiles.pad(countWidth) + kt.pad(countWidth)) << log2Ceil(tile) - // 当前 WS batch 的开区间上界，尾批可能不足 wsReuseTiles 个 M tile。 - val wsBatchLimit = Mux( - wsBatchBase + wsReuseTiles.U < mTiles, - wsBatchBase + wsReuseTiles.U, - mTiles) - val wsReuseCount = wsBatchLimit - wsBatchBase - val wsHasNextWeight = kt + 1.U < kTiles || - wsBatchBase + wsReuseTiles.U < mTiles || nt + 1.U < nTiles - val wsPrefetchB = task.mode && !wsLoadPeB && wsReuseCount >= 3.U && - mt + 2.U === wsBatchLimit && wsHasNextWeight - val prefetchNt = Wire(UInt(countWidth.W)) - val prefetchKt = Wire(UInt(countWidth.W)) - prefetchNt := nt - prefetchKt := 0.U - when(kt + 1.U < kTiles) { - prefetchKt := kt + 1.U - }.elsewhen(wsBatchBase + wsReuseTiles.U < mTiles) { - prefetchKt := 0.U - }.otherwise { - prefetchNt := nt + 1.U - } - val prefetchValidN = validTileExtent(task.n, prefetchNt) - val prefetchValidK = validTileExtent(task.k, prefetchKt) - val prefetchBRowOffset = - (prefetchNt.pad(countWidth) * kTiles.pad(countWidth) + prefetchKt.pad(countWidth)) << log2Ceil(tile) - val selectedBRowOffset = Mux(wsPrefetchB, prefetchBRowOffset, bRowOffset) - currentKind := Mux( - task.mode, - Mux(wsLoadPeB, SystolicCtrlLoadReqKind.READ_A_B_PE, - Mux(wsPrefetchB, SystolicCtrlLoadReqKind.READ_A_B_BUF, SystolicCtrlLoadReqKind.READ_A_ONLY)), - SystolicCtrlLoadReqKind.READ_AB - ) - // retireQ 队首是当前唯一可完成的任务。storeMetaQ 独立保存地址发放顺序， - // 允许后一任务的描述符在前一任务写回完成前进入 Store，同时保持全局保序。 - val storeTask = retireQ.io.deq.bits - val storeMTiles = ceilDiv16(storeTask.m) - val storeNTiles = ceilDiv16(storeTask.n) - val storeMetaTask = storeMetaQ.io.deq.bits - val storeMetaMTiles = ceilDiv16(storeMetaTask.m) - val storeMetaNTiles = ceilDiv16(storeMetaTask.n) - - // 发放游标决定描述符中的地址。OS 使用连续的 16 元素 C block 计数；WS 按 - // 最终 C 布局重映射当前 (M tile, N tile, tile 内行)。这两类地址均不能等待 - // store_done_i 才前进，否则预取的下一行会重复前一行地址。 - val storeMetaRemainingCols = storeMetaTask.n.pad(countWidth) - (storeMetaNt << log2Ceil(tile)) - val storeMetaValidCols = Mux( - storeMetaRemainingCols >= tile.U, - tile.U(5.W), - fitTo(storeMetaRemainingCols, 5)) - val storeMetaValidRows = validTileExtent(storeMetaTask.m, storeMetaMt) - val storeMetaWsCompactBlockBase = - (storeMetaMt.pad(countWidth) << log2Ceil(tile)) * storeMetaNTiles.pad(countWidth) + - storeMetaNt.pad(countWidth) * storeMetaValidRows.pad(countWidth) + - storeMetaRowIdx.pad(countWidth) - val storeMetaBlockBase = Mux( - storeMetaTask.mode, - storeMetaWsCompactBlockBase, - storeMetaBlockCount) - - // 完成游标只用于消费实际完成的行以及生成命令完成响应。它保持落后于发放游标， - // 因而不会影响已经预取的地址。 - val storeDoneValidRows = validTileExtent(storeTask.m, storeDoneMt) - val storeDoneLastTile = storeDoneMt === (storeMTiles - 1.U) && - storeDoneNt === (storeNTiles - 1.U) - val storeDoneLastRow = storeDoneRowIdx + 1.U >= storeDoneValidRows - val storeDoneTaskDone = storeDoneLastTile && storeDoneLastRow - - // 一次 Store 行请求对应的地址元数据。发放游标只遍历合法 M 行，因此每个 - // 描述符都对应一次真实 bank 写；N 维尾部由 row_valid_elems 的 byte mask 处理。 - val nextStoreResp = WireDefault(0.U.asTypeOf(new SystolicStoreCtrlResp(b))) - nextStoreResp.row_valid_elems := storeMetaValidCols - nextStoreResp.rob_id := storeMetaTask.rob_id - nextStoreResp.wr_bank := storeMetaTask.wr_bank - nextStoreResp.wr_group_base := cGroupBase(storeMetaTask.wr_base, storeMetaBlockBase) - nextStoreResp.wr_row_addr := cRowAddr(storeMetaTask.wr_base, storeMetaBlockBase) - - val storeMetaLastTile = storeMetaMt === (storeMetaMTiles - 1.U) && - storeMetaNt === (storeMetaNTiles - 1.U) - val storeMetaLastRow = storeMetaRowIdx + 1.U >= storeMetaValidRows - val storeMetaTaskDone = storeMetaLastTile && storeMetaLastRow - - // 当前发射任务的最后一个 tile 被 Load 接收时，若 issueQ 已有后继任务， - // 同一时钟沿装入后继任务，下一周期立即发送它的第一个 tile。 - val issueLastTile = issueActive && mt === (mTiles - 1.U) && - nt === (nTiles - 1.U) && kt === (kTiles - 1.U) - val queuedTaskReady = issueQ.io.deq.valid && retireQ.io.enq.ready && storeMetaQ.io.enq.ready - val canStartIssueTask = !issueActive && queuedTaskReady - val replaceIssueTask = io.ctrl_ld_o.fire && issueLastTile && queuedTaskReady - val acceptIssueTask = canStartIssueTask || replaceIssueTask - - issueQ.io.deq.ready := acceptIssueTask - retireQ.io.enq.valid := acceptIssueTask - retireQ.io.enq.bits.mode := nextTask.mode - retireQ.io.enq.bits.m := nextTask.m - retireQ.io.enq.bits.n := nextTask.n - retireQ.io.enq.bits.rob_id := nextTask.rob_id - retireQ.io.enq.bits.is_sub := nextTask.is_sub - retireQ.io.enq.bits.sub_rob_id := nextTask.sub_rob_id - storeMetaQ.io.enq.valid := acceptIssueTask - storeMetaQ.io.enq.bits.mode := nextTask.mode - storeMetaQ.io.enq.bits.m := nextTask.m - storeMetaQ.io.enq.bits.n := nextTask.n - storeMetaQ.io.enq.bits.wr_bank := nextTask.wr_bank - storeMetaQ.io.enq.bits.wr_base := nextTask.wr_base - storeMetaQ.io.enq.bits.rob_id := nextTask.rob_id - - // 发射阶段只依赖 Load 的反压，不依赖 retireQ 队首任务的 Store 进度。 - io.ctrl_ld_o.valid := issueActive - io.ctrl_ld_o.bits.req_kind := currentKind - io.ctrl_ld_o.bits.k_tile_kind := currentKTileKind - io.ctrl_ld_o.bits.acc_slot := Mux( - task.mode, - fitTo(mt - wsBatchBase, log2Ceil(wsReuseTiles)), - 0.U) - io.ctrl_ld_o.bits.valid_m := currentValidM - io.ctrl_ld_o.bits.valid_n := currentValidN - io.ctrl_ld_o.bits.valid_k := currentValidK - io.ctrl_ld_o.bits.b_valid_n := Mux(wsPrefetchB, prefetchValidN, currentValidN) - io.ctrl_ld_o.bits.b_valid_k := Mux(wsPrefetchB, prefetchValidK, currentValidK) - io.ctrl_ld_o.bits.weight_generation := wsWeightGeneration - io.ctrl_ld_o.bits.op1_bank := task.op1_bank - io.ctrl_ld_o.bits.op1_group := abGroup(task.op1_base, aRowOffset) - io.ctrl_ld_o.bits.op1_row_base := abRow(task.op1_base, aRowOffset) - io.ctrl_ld_o.bits.op2_bank := task.op2_bank - io.ctrl_ld_o.bits.op2_group := abGroup(task.op2_base, selectedBRowOffset) - io.ctrl_ld_o.bits.op2_row_base := abRow(task.op2_base, selectedBRowOffset) - - // 任务只在其最后一行已经实际写完后进入完成队列；前端对 cmdResp 的背压 - // 不会阻塞 issueTask 的 tile 发射，只会在完成队列满时停住 Store 的最后一行。 - io.cmdResp_o.valid := completeQ.io.deq.valid - io.cmdResp_o.bits := completeQ.io.deq.bits - completeQ.io.deq.ready := io.cmdResp_o.ready - completeQ.io.enq.valid := false.B - completeQ.io.enq.bits.rob_id := storeTask.rob_id - completeQ.io.enq.bits.is_sub := storeTask.is_sub - completeQ.io.enq.bits.sub_rob_id := storeTask.sub_rob_id - retireQ.io.deq.ready := false.B - - io.busy_o := issueActive || issueQ.io.deq.valid || retireQ.io.deq.valid || - completeQ.io.deq.valid || storeOutstanding =/= 0.U - // Ctrl 不等待 Store 完成：只要元数据任务队列仍有下一行描述符，就保持 valid。 - // 最后一行 fire 后立即切换到下一任务，Store 依靠 FIFO 顺序和逐行 rob_id 保序。 - val completionSlotAvailable = completionReservations < 4.U || completeQ.io.deq.fire - val storeCanEnqueue = storeMetaQ.io.deq.valid && - (!storeMetaTaskDone || completionSlotAvailable) - io.store_ctrl_resp_o.valid := storeCanEnqueue - io.store_ctrl_resp_o.bits := nextStoreResp - - val storeMetaIssue = io.store_ctrl_resp_o.fire - val reserveCompletion = storeMetaIssue && storeMetaTaskDone - storeMetaQ.io.deq.ready := storeMetaIssue && storeMetaTaskDone - switch(Cat(reserveCompletion, completeQ.io.deq.fire)) { - is("b10".U) { completionReservations := completionReservations + 1.U } - is("b01".U) { completionReservations := completionReservations - 1.U } - } - when(storeMetaIssue) { - // 描述符已被 Store 接收，推进独立的地址发放游标。完成游标仍只响应 - // store_done_i，因此地址发放可以领先多个严格保序的任务。 - when(storeMetaTaskDone) { - storeMetaMt := 0.U - storeMetaNt := 0.U - storeMetaRowIdx := 0.U - storeMetaBlockCount := 0.U - }.otherwise { - storeMetaBlockCount := storeMetaBlockCount + 1.U - - storeMetaRowIdx := storeMetaRowIdx + 1.U - when(storeMetaLastRow) { - storeMetaRowIdx := 0.U - - when(storeMetaTask.mode) { - when(storeMetaMt + 1.U < storeMetaMTiles) { - storeMetaMt := storeMetaMt + 1.U - }.otherwise { - storeMetaMt := 0.U - when(storeMetaNt + 1.U < storeMetaNTiles) { - storeMetaNt := storeMetaNt + 1.U - } - } - }.otherwise { - when(storeMetaNt + 1.U < storeMetaNTiles) { - storeMetaNt := storeMetaNt + 1.U - }.otherwise { - storeMetaNt := 0.U - when(storeMetaMt + 1.U < storeMetaMTiles) { - storeMetaMt := storeMetaMt + 1.U - } - } - } - } - } - } - assert(completionReservations <= 4.U, - "SystolicArrayCtrl: completion reservation overflow") - - // 每个被 Store 接收的描述符，最终都必须对应一个按顺序到达的 store_done_i。 - // Store 的地址 FIFO 和 Unit 的写飞行队列允许多行已发行但仍在 bank 中飞行。 - switch(Cat(io.store_ctrl_resp_o.fire, io.store_done_i)) { - is("b10".U) { storeOutstanding := storeOutstanding + 1.U } - is("b01".U) { storeOutstanding := storeOutstanding - 1.U } - } - - when(io.store_done_i) { - assert(storeOutstanding =/= 0.U, "SystolicArrayCtrl: store_done without an issued descriptor") - - when(storeDoneTaskDone) { - assert(completeQ.io.enq.ready, - "SystolicArrayCtrl: final store row completed without completion queue space") - completeQ.io.enq.valid := true.B - retireQ.io.deq.ready := completeQ.io.enq.ready - storeDoneMt := 0.U - storeDoneNt := 0.U - storeDoneRowIdx := 0.U - }.otherwise { - storeDoneRowIdx := storeDoneRowIdx + 1.U - - // 一个输出 tile 的有效行完成后，按模式选择下一个 tile 的遍历顺序： - // OS 为 N 内层；WS 为 M 内层。两者最终都映射到相同的 C 存储布局。 - when(storeDoneLastRow) { - storeDoneRowIdx := 0.U - - when(storeTask.mode) { - when(storeDoneMt + 1.U < storeMTiles) { - storeDoneMt := storeDoneMt + 1.U - }.otherwise { - storeDoneMt := 0.U - when(storeDoneNt + 1.U < storeNTiles) { - storeDoneNt := storeDoneNt + 1.U - } - } - }.otherwise { - when(storeDoneNt + 1.U < storeNTiles) { - storeDoneNt := storeDoneNt + 1.U - }.otherwise { - storeDoneNt := 0.U - when(storeDoneMt + 1.U < storeMTiles) { - storeDoneMt := storeDoneMt + 1.U - } - } - } - } - } - } - assert(storeOutstanding <= 8.U, "SystolicArrayCtrl: Store completion pipeline overflow") - - // 防御性检查：每次实际发射时 K-tile 分类必须与 kt/kTiles 一致。 - when(io.ctrl_ld_o.fire) { - when(kTiles === 1.U) { - assert(currentKTileKind === SystolicKTileKind.DIRECT) - }.elsewhen(kt === 0.U) { - assert(currentKTileKind === SystolicKTileKind.FIRST) - }.elsewhen(kt === (kTiles - 1.U)) { - assert(currentKTileKind === SystolicKTileKind.LAST) - }.otherwise { - assert(currentKTileKind === SystolicKTileKind.MIDDLE) - } - - when(issueLastTile) { - when(!replaceIssueTask) { - issueActive := false.B - } - }.otherwise { - when(task.mode) { - advanceIssueWsLoad() - }.otherwise { - advanceIssueOsTile() - } - } - } - - /** OS 发射顺序为 K 内层、N 次之、M 外层：每个输出 tile 完成整条 K 累加链。 */ - private def advanceIssueOsTile(): Unit = { - when(kt + 1.U < kTiles) { - kt := kt + 1.U - }.otherwise { - kt := 0.U - when(nt + 1.U < nTiles) { - nt := nt + 1.U - }.otherwise { - nt := 0.U - when(mt + 1.U < mTiles) { - mt := mt + 1.U - }.otherwise { - assert(false.B, "SystolicArrayCtrl: advanced past final OS tile") - } - } - } - } - - /** - * WS 发射顺序。 - * - * 对固定 (N,K) 与一个最多 wsReuseTiles 个 M tile 的 batch，先为 batch 首个 M tile - * 发送 READ_A_B_PE 装载权重，随后为其余 M tile 发送 READ_A_ONLY。所有 K - * tile 完成后才切换下一 batch 或下一 N tile。 - */ - private def advanceIssueWsLoad(): Unit = { - when(mt + 1.U < wsBatchLimit) { - mt := mt + 1.U - wsLoadPeB := false.B - when(wsPrefetchB) { - wsPrefetchedWeights := true.B - } - }.otherwise { - mt := wsBatchBase - when(kt + 1.U < kTiles) { - kt := kt + 1.U - wsLoadPeB := !wsPrefetchedWeights - wsWeightGeneration := !wsWeightGeneration - wsPrefetchedWeights := false.B - }.otherwise { - kt := 0.U - when(wsBatchBase + wsReuseTiles.U < mTiles) { - wsBatchBase := wsBatchBase + wsReuseTiles.U - mt := wsBatchBase + wsReuseTiles.U - wsLoadPeB := !wsPrefetchedWeights - wsWeightGeneration := !wsWeightGeneration - wsPrefetchedWeights := false.B - }.otherwise { - wsBatchBase := 0.U - mt := 0.U - when(nt + 1.U < nTiles) { - nt := nt + 1.U - wsLoadPeB := !wsPrefetchedWeights - wsWeightGeneration := !wsWeightGeneration - wsPrefetchedWeights := false.B - }.otherwise { - assert(false.B, "SystolicArrayCtrl: advanced past final WS tile") - } - } - } - } - } - - /** 接受队首命令并仅复位 Load 发射游标；Store 地址与完成游标独立推进。 */ - private def startIssueTask(): Unit = { - task := nextTask - mTiles := ceilDiv16(issueM) - nTiles := ceilDiv16(issueN) - kTiles := ceilDiv16(issueK) - mt := 0.U - nt := 0.U - kt := 0.U - wsBatchBase := 0.U - wsLoadPeB := issueMode - wsWeightGeneration := false.B - wsPrefetchedWeights := false.B - issueActive := true.B - - assert(issueM =/= 0.U && issueN =/= 0.U && issueK =/= 0.U, "SystolicArrayCtrl: M/N/K must be non-zero") - } - - when(acceptIssueTask) { - startIssueTask() - } -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixEX.scala b/examples/balls/matrix/arch/src/main/scala/MatrixEX.scala deleted file mode 100644 index 0d0601099..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixEX.scala +++ /dev/null @@ -1,288 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import framework.top.GlobalConfig - -@instantiable -class SystolicArrayEX(b: GlobalConfig) extends SystolicArrayEXDatapath(b) { - val slotLaunching = Wire(Vec(operandSlotCount, Bool())) - for (slot <- 0 until operandSlotCount) { - val slotContextAge = contextAge(slotContext(slot)) - val slotIsWs = isWsKind(slotReqKind(slot)) - val slotContextActive = contextActive(slotContext(slot)) && - (!slotIsWs || contextActiveSlot(slotContext(slot)) === slot.U) - val maxExtent = Mux(slotValidM(slot) >= slotValidN(slot), - slotValidM(slot), slotValidN(slot)) - val osLastUseCycle = slotKBase(slot) + slotValidK(slot).pad(progressWidth) - 1.U + - maxExtent.pad(progressWidth) - 1.U - val wsLastUseCycle = slotValidK(slot) - 1.U + slotValidM(slot) - 1.U - val lastUseCycle = Mux(slotIsWs, wsLastUseCycle, osLastUseCycle) - val reachedLastUse = pipelineAdvance && slotContextActive && slotContextAge >= lastUseCycle - - when(slotOccupied(slot) && reachedLastUse) { - slotUseDone(slot) := true.B - } - when(slotOccupied(slot) && slotInputComplete(slot) && - (slotUseDone(slot) || reachedLastUse) && !slotLaunchPending(slot) && - !slotLaunching(slot)) { - slotOccupied(slot) := false.B - slotInputComplete(slot) := false.B - slotUseDone(slot) := false.B - slotARowsReceived(slot) := 0.U - slotBRowsReceived(slot) := 0.U - for (row <- 0 until tile) { - aRowValid(slot)(row) := false.B - bRowValid(slot)(row) := false.B - } - } - } - - val segmentContext = segmentOrder.io.deq.bits( - contextWidth + operandSlotWidth - 1, operandSlotWidth) - val segmentSlot = segmentOrder.io.deq.bits(operandSlotWidth - 1, 0) - val segmentIsWs = isWsKind(slotReqKind(segmentSlot)) - val activeContextCount = PopCount(contextActive) - val activeOsInputsDrained = (0 until contextCount).map { context => - !contextActive(context) || (!contextWsMode(context) && contextFinalSeen(context) && - contextAge(context) + 1.U >= contextTotalK(context)) - }.reduce(_ && _) - val canOverlapOs = anyContextActive && !segmentIsWs && activeContextCount < 2.U && - activeOsInputsDrained && pipelineAdvance - val canOverlapWs = anyContextActive && - isWsKind(slotReqKind(segmentSlot)) && pipelineAdvance && - (0 until contextCount).map { context => - !contextActive(context) || - (contextWsMode(context) && - contextAge(context) + 1.U >= contextValidM(context).pad(progressWidth)) - }.reduce(_ && _) - val launchAllowed = !anyContextActive || canOverlapOs || canOverlapWs - val segmentWeightBank = slotWeightGeneration(segmentSlot).asUInt - val segmentResidentWeightsReady = wsWeightBankValid(segmentWeightBank) && - wsWeightBankValidN(segmentWeightBank) === slotValidN(segmentSlot) && - wsWeightBankValidK(segmentWeightBank) === slotValidK(segmentSlot) - val segmentPrefetchLaunchReady = - slotReqKind(segmentSlot) === SystolicCtrlLoadReqKind.READ_A_B_BUF && - slotARowsReceived(segmentSlot) >= slotValidM(segmentSlot) && - segmentResidentWeightsReady - val segmentOsStreamLaunchReady = !segmentIsWs && - slotARowsReceived(segmentSlot) =/= 0.U && - slotBRowsReceived(segmentSlot) =/= 0.U - val segmentInputReady = slotInputComplete(segmentSlot) || - segmentPrefetchLaunchReady || segmentOsStreamLaunchReady - val launchSegment = segmentOrder.io.deq.valid && launchAllowed && - slotOccupied(segmentSlot) && segmentInputReady && - !contextActive(segmentContext) && - (!segmentIsWs || !contextWsSegmentPending(segmentContext)) && - contextPendingStart(segmentContext) - segmentOrder.io.deq.ready := launchSegment - slotLaunching := VecInit((0 until operandSlotCount).map(slot => - launchSegment && segmentSlot === slot.U)) - - when(launchSegment) { - assert(slotContext(segmentSlot) === segmentContext, - "SystolicArrayEX: segment queue metadata does not match its operand slot") - slotUseDone(segmentSlot) := false.B - slotLaunchPending(segmentSlot) := false.B - contextPendingStart(segmentContext) := false.B - contextActive(segmentContext) := true.B - contextActiveSlot(segmentContext) := segmentSlot - contextWsMode(segmentContext) := isWsKind(slotReqKind(segmentSlot)) - contextWeightGeneration(segmentContext) := slotWeightGeneration(segmentSlot) - contextAge(segmentContext) := 0.U - when(segmentIsWs) { - val launchWeightBank = slotWeightGeneration(segmentSlot).asUInt - assert(wsWeightBankValid(launchWeightBank), - "SystolicArrayEX: WS launched without a valid weight bank") - assert(wsWeightBankValidN(launchWeightBank) === slotValidN(segmentSlot) && - wsWeightBankValidK(launchWeightBank) === slotValidK(segmentSlot), - "SystolicArrayEX: WS launched with mismatched weight metadata") - contextTotalK(segmentContext) := slotValidK(segmentSlot).pad(progressWidth) - contextFinalSeen(segmentContext) := - slotKTileKind(segmentSlot) === SystolicKTileKind.DIRECT || - slotKTileKind(segmentSlot) === SystolicKTileKind.LAST - contextWsSegmentPending(segmentContext) := true.B - contextWsRowsCommitted(segmentContext) := 0.U - contextWsKTileKind(segmentContext) := slotKTileKind(segmentSlot) - }.otherwise { - wsWeightBankValid(0) := false.B - } - - when(!anyContextActive) { - for (row <- 0 until tile) { - for (col <- 0 until tile) { - aPipeValid(row)(col) := false.B - wsPsumValid(row)(col) := false.B - when(!segmentIsWs) { - bPipeValid(row)(col) := false.B - } - } - } - } - } - - val outputContext = outputOrder.io.deq.bits - val outputRowsComplete = contextRowsComplete(outputContext) - val outputSendRow = contextSendRow(outputContext) - val outputValidM = contextValidM(outputContext) - val outputRowIndex = outputSendRow(rowIndexWidth - 1, 0) - val outputOsAccBank = contextOsAccBank(outputContext) - val osPackedRows = Wire(Vec(osAccBankCount, Vec(tile, UInt(accRowBits.W)))) - for (bank <- 0 until osAccBankCount) { - for (row <- 0 until tile) { - osPackedRows(bank)(row) := resultRowBitsFrom(osAcc(bank)(row)) - } - } - val osSelectedBankRows = Wire(Vec(tile, UInt(accRowBits.W))) - for (row <- 0 until tile) { - osSelectedBankRows(row) := Mux1H((0 until osAccBankCount).map { bank => - (outputOsAccBank === bank.U) -> osPackedRows(bank)(row) - }) - } - val osOutputResult = Mux1H((0 until tile).map { row => - (outputRowIndex === row.U) -> osSelectedBankRows(row) - }) - val outputIsWs = outputOrder.io.deq.valid && contextWsMode(outputContext) - - val wsBufferedResult = wsResultBuffer.io.deq.bits - val wsBufferedFinal = wsBufferedResult.kTileKind === SystolicKTileKind.DIRECT || - wsBufferedResult.kTileKind === SystolicKTileKind.LAST - val wsBufferedMatchesOutput = outputOrder.io.deq.valid && outputIsWs && - wsBufferedResult.context === outputContext && wsBufferedResult.row === outputRowIndex - val wsOutputValid = wsResultBuffer.io.deq.valid && wsBufferedFinal && - wsBufferedMatchesOutput - val osOutputValid = outputOrder.io.deq.valid && !outputIsWs && - outputRowsComplete > outputSendRow - - - io.ex_st_o.valid := Mux(outputIsWs, wsOutputValid, osOutputValid) - io.ex_st_o.bits.data := Mux(outputIsWs, wsBufferedResult.data, osOutputResult) - - - wsResultBuffer.io.deq.ready := Mux(wsBufferedFinal, - wsBufferedMatchesOutput && io.ex_st_o.ready, true.B) - val wsResultCommit = wsResultBuffer.io.deq.fire - when(wsResultCommit) { - val commitContext = wsBufferedResult.context - val commitRow = wsBufferedResult.row - val committedRows = contextWsRowsCommitted(commitContext) - val committingLastRow = committedRows + 1.U >= contextValidM(commitContext) - - - assert(contextAllocated(commitContext) && contextWsMode(commitContext), - "SystolicArrayEX: WS accumulator result refers to an invalid context") - assert(contextWsSegmentPending(commitContext), - "SystolicArrayEX: WS accumulator result has no pending segment") - assert(commitRow === committedRows(rowIndexWidth - 1, 0), - "SystolicArrayEX: WS accumulator rows committed out of order") - - when(!wsBufferedFinal) { - wsAccMem.write(wsAccAddress(commitContext, commitRow), wsBufferedResult.data) - }.otherwise { - contextRowsComplete(commitContext) := contextRowsComplete(commitContext) + 1.U - } - - contextWsRowsCommitted(commitContext) := committedRows + 1.U - when(committingLastRow) { - contextWsSegmentPending(commitContext) := false.B - } - } - - val finishingOutput = io.ex_st_o.fire && outputSendRow + 1.U >= outputValidM - outputOrder.io.deq.ready := finishingOutput - - when(io.ex_st_o.fire) { - when(finishingOutput) { - assert(!contextActive(outputContext), - "SystolicArrayEX: context released before its final MAC completed") - contextAllocated(outputContext) := false.B - contextFinalSeen(outputContext) := false.B - contextTotalK(outputContext) := 0.U - contextRowsComplete(outputContext) := 0.U - contextSendRow(outputContext) := 0.U - contextWsSegmentPending(outputContext) := false.B - contextWsRowsCommitted(outputContext) := 0.U - when(!contextWsMode(outputContext)) { - assert(osAccBankAllocated(outputOsAccBank), - "SystolicArrayEX: released an unallocated OS accumulator bank") - assert(osAccBankOwner(outputOsAccBank) === outputContext, - "SystolicArrayEX: released an OS accumulator bank owned by another context") - osAccBankAllocated(outputOsAccBank) := false.B - } - }.otherwise { - contextSendRow(outputContext) := outputSendRow + 1.U - } - } - - when(firstReceiveEvent) { - assert(io.load_ex_valid_m >= 1.U && io.load_ex_valid_m <= tile.U) - assert(io.load_ex_valid_n >= 1.U && io.load_ex_valid_n <= tile.U) - assert(io.load_ex_valid_k >= 1.U && io.load_ex_valid_k <= tile.U) - when(needOp2(io.load_ex_req_kind)) { - assert(io.load_ex_b_valid_n >= 1.U && io.load_ex_b_valid_n <= tile.U) - assert(io.load_ex_b_valid_k >= 1.U && io.load_ex_b_valid_k <= tile.U) - } - when(io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_A_B_PE) { - assert(io.load_ex_b_valid_n === io.load_ex_valid_n && - io.load_ex_b_valid_k === io.load_ex_valid_k, - "SystolicArrayEX: direct PE weights must match the current tile extent") - } - when(firstIsContinuation && io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_AB) { - assert(chainValid, "SystolicArrayEX: continuation arrived without an active K-tile chain") - } - } - - when(receiveActive) { - assert(slotARowsReceived(receiveSlot) <= slotValidM(receiveSlot), - "SystolicArrayEX: A receive row index overflow") - assert(slotBRowsReceived(receiveSlot) <= - Mux(needOp2(slotReqKind(receiveSlot)), slotBValidK(receiveSlot), 0.U), - "SystolicArrayEX: B receive row index overflow") - } - for (context <- 0 until contextCount) { - assert(contextRowsComplete(context) <= contextValidM(context), - "SystolicArrayEX: completed row count overflow") - when(contextActive(context) || contextPendingStart(context)) { - assert(contextAllocated(context), - "SystolicArrayEX: active context is not allocated") - } - when(contextAllocated(context) && !contextWsMode(context)) { - val bank = contextOsAccBank(context) - assert(osAccBankAllocated(bank), - "SystolicArrayEX: allocated OS context has no accumulator bank") - assert(osAccBankOwner(bank) === context.U, - "SystolicArrayEX: allocated OS context does not own its accumulator bank") - } - when(contextWsSegmentPending(context)) { - assert(contextAllocated(context) && contextWsMode(context), - "SystolicArrayEX: pending WS segment refers to an invalid context") - assert(contextWsRowsCommitted(context) < contextValidM(context), - "SystolicArrayEX: pending WS segment already committed every row") - } - } - for (slot <- 0 until operandSlotCount) { - assert(slotARowsReceived(slot) <= slotValidM(slot), - "SystolicArrayEX: A operand slot row count overflow") - assert(slotBRowsReceived(slot) <= Mux(needOp2(slotReqKind(slot)), slotBValidK(slot), 0.U), - "SystolicArrayEX: B operand slot row count overflow") - when(slotOccupied(slot) && slotInputComplete(slot)) { - assert(slotARowsReceived(slot) === slotValidM(slot), - "SystolicArrayEX: completed slot is missing A rows") - assert(!needOp2(slotReqKind(slot)) || slotBRowsReceived(slot) === slotBValidK(slot), - "SystolicArrayEX: completed slot is missing B rows") - } - when(slotOccupied(slot) && !slotUseDone(slot)) { - assert(contextAllocated(slotContext(slot)), - "SystolicArrayEX: operand slot refers to a free context") - } - when(slotUseDone(slot)) { - assert(slotOccupied(slot), - "SystolicArrayEX: retired operand slot is not occupied") - } - } - when(outputOrder.io.deq.valid) { - assert(contextAllocated(outputOrder.io.deq.bits), - "SystolicArrayEX: output queue refers to a free context") - } -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixEXBase.scala b/examples/balls/matrix/arch/src/main/scala/MatrixEXBase.scala deleted file mode 100644 index d248239a2..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixEXBase.scala +++ /dev/null @@ -1,157 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import framework.top.GlobalConfig - -@instantiable -abstract class SystolicArrayEXBase(val b: GlobalConfig) extends Module { - protected val tile = SystolicArrayConst.Tile - protected val opElemBits = SystolicArrayConst.OpElemBits - protected val accElemBits = SystolicArrayConst.AccElemBits - protected val wsPsumBits = 2 * opElemBits + log2Ceil(tile) - protected val opRowBits = SystolicArrayConst.OpRowBits - protected val contextCount = SystolicArrayConst.wsReuseTiles(b) - protected val operandSlotCount = 3 - protected val osAccBankCount = 2 - protected val contextWidth = log2Ceil(contextCount) - protected val operandSlotWidth = log2Ceil(operandSlotCount) - protected val osAccBankWidth = log2Ceil(osAccBankCount) - protected val rowIndexWidth = log2Ceil(tile) - protected val accRowBits = tile * accElemBits - protected val wsPsumRowBits = tile * wsPsumBits - protected val progressWidth = 13 - - @public - val io = IO(new Bundle { - val load_ex_req_kind = Input(UInt(2.W)) - val load_ex_k_tile_kind = Input(UInt(2.W)) - val load_ex_acc_slot = Input(UInt(contextWidth.W)) - val load_ex_valid_m = Input(UInt(5.W)) - val load_ex_valid_n = Input(UInt(5.W)) - val load_ex_valid_k = Input(UInt(5.W)) - val load_ex_b_valid_n = Input(UInt(5.W)) - val load_ex_b_valid_k = Input(UInt(5.W)) - val load_ex_weight_generation = Input(Bool()) - val load_ex_op1_i = Flipped(Decoupled(UInt(opRowBits.W))) - val load_ex_op2_i = Flipped(Decoupled(UInt(opRowBits.W))) - - val ex_st_o = Decoupled(new SystolicResultRow) - - }) - - protected def needOp2(kind: UInt): Bool = - kind === SystolicCtrlLoadReqKind.READ_AB || kind === SystolicCtrlLoadReqKind.READ_A_B_PE || - kind === SystolicCtrlLoadReqKind.READ_A_B_BUF - - protected def isWsKind(kind: UInt): Bool = - kind === SystolicCtrlLoadReqKind.READ_A_ONLY || kind === SystolicCtrlLoadReqKind.READ_A_B_PE || - kind === SystolicCtrlLoadReqKind.READ_A_B_BUF - - protected def isNewOutputTile(kTileKind: UInt): Bool = - kTileKind === SystolicKTileKind.DIRECT || kTileKind === SystolicKTileKind.FIRST - - protected def isContinuationTile(kTileKind: UInt): Bool = - kTileKind === SystolicKTileKind.MIDDLE || kTileKind === SystolicKTileKind.LAST - - protected def rowByte(row: UInt, idx: Int): UInt = - row((idx + 1) * opElemBits - 1, idx * opElemBits) - - protected def dynamicRowByte(row: UInt, idx: UInt): UInt = - (row >> (idx * opElemBits.U))(opElemBits - 1, 0) - - protected def resultRowBitsFrom(row: Vec[UInt]): UInt = - Cat((0 until tile).reverse.map(index => row(index))) - - val contextAllocated = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextPendingStart = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextActive = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextFinalSeen = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextValidM = RegInit(VecInit(Seq.fill(contextCount)(tile.U(5.W)))) - val contextValidN = RegInit(VecInit(Seq.fill(contextCount)(tile.U(5.W)))) - val contextTotalK = RegInit(VecInit(Seq.fill(contextCount)(0.U(progressWidth.W)))) - val contextAge = RegInit(VecInit(Seq.fill(contextCount)(0.U(progressWidth.W)))) - val contextRowsComplete = RegInit(VecInit(Seq.fill(contextCount)(0.U(5.W)))) - val contextSendRow = RegInit(VecInit(Seq.fill(contextCount)(0.U(5.W)))) - val contextActiveSlot = RegInit(VecInit(Seq.fill(contextCount)(0.U(operandSlotWidth.W)))) - val contextWsMode = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextWeightGeneration = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextPendingWeightGeneration = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val wsContextMap = RegInit(VecInit(Seq.tabulate(contextCount)(_.U(contextWidth.W)))) - val wsContextMapValid = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextOsAccBank = RegInit(VecInit(Seq.fill(contextCount)(0.U(osAccBankWidth.W)))) - val contextWsSegmentPending = RegInit(VecInit(Seq.fill(contextCount)(false.B))) - val contextWsRowsCommitted = RegInit(VecInit(Seq.fill(contextCount)(0.U(5.W)))) - val contextWsKTileKind = RegInit(VecInit(Seq.fill(contextCount)(SystolicKTileKind.DIRECT))) - - // One 512-bit row per address; separate read/write calls infer a 1R1W memory. - val wsAccMem = SyncReadMem(contextCount * tile, UInt(accRowBits.W)) - - // OS only overlaps two output tiles, so each PE needs two local accumulators. - val osAcc = RegInit(VecInit(Seq.tabulate(osAccBankCount)(_ => - VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(accElemBits.W)))))))) - val osAccBankAllocated = RegInit(VecInit(Seq.fill(osAccBankCount)(false.B))) - val osAccBankOwner = RegInit(VecInit(Seq.fill(osAccBankCount)(0.U(contextWidth.W)))) - - val slotOccupied = RegInit(VecInit(Seq.fill(operandSlotCount)(false.B))) - val slotInputComplete = RegInit(VecInit(Seq.fill(operandSlotCount)(false.B))) - val slotUseDone = RegInit(VecInit(Seq.fill(operandSlotCount)(false.B))) - val slotLaunchPending = RegInit(VecInit(Seq.fill(operandSlotCount)(false.B))) - val slotContext = RegInit(VecInit(Seq.fill(operandSlotCount)(0.U(contextWidth.W)))) - val slotReqKind = RegInit(VecInit(Seq.fill(operandSlotCount)(SystolicCtrlLoadReqKind.READ_AB))) - val slotValidM = RegInit(VecInit(Seq.fill(operandSlotCount)(tile.U(5.W)))) - val slotValidN = RegInit(VecInit(Seq.fill(operandSlotCount)(tile.U(5.W)))) - val slotValidK = RegInit(VecInit(Seq.fill(operandSlotCount)(tile.U(5.W)))) - val slotBValidK = RegInit(VecInit(Seq.fill(operandSlotCount)(tile.U(5.W)))) - val slotARowsReceived = RegInit(VecInit(Seq.fill(operandSlotCount)(0.U(5.W)))) - val slotBRowsReceived = RegInit(VecInit(Seq.fill(operandSlotCount)(0.U(5.W)))) - val slotKTileKind = RegInit(VecInit(Seq.fill(operandSlotCount)(SystolicKTileKind.DIRECT))) - val slotKBase = RegInit(VecInit(Seq.fill(operandSlotCount)(0.U(progressWidth.W)))) - val slotWeightGeneration = RegInit(VecInit(Seq.fill(operandSlotCount)(false.B))) - val aRowBuf = Reg(Vec(operandSlotCount, Vec(tile, UInt(opRowBits.W)))) - val bRowBuf = Reg(Vec(operandSlotCount, Vec(tile, UInt(opRowBits.W)))) - val aRowValid = RegInit(VecInit(Seq.tabulate(operandSlotCount)(_ => - VecInit(Seq.fill(tile)(false.B))))) - val bRowValid = RegInit(VecInit(Seq.tabulate(operandSlotCount)(_ => - VecInit(Seq.fill(tile)(false.B))))) - - val bPipeData = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(opElemBits.W)))))) - val bPipeValid = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(false.B))))) - val bPipeContext = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(contextWidth.W)))))) - - val wsBBuffer = Reg(Vec(tile, UInt(opRowBits.W))) - val wsWeightBankValid = RegInit(VecInit(Seq.fill(2)(false.B))) - val wsWeightBankValidN = RegInit(VecInit(Seq.fill(2)(tile.U(5.W)))) - val wsWeightBankValidK = RegInit(VecInit(Seq.fill(2)(tile.U(5.W)))) - val wsFinalReusePending = RegInit(false.B) - val wsPrefetchGeneration = RegInit(false.B) - - val chainValid = RegInit(false.B) - val chainContext = RegInit(0.U(contextWidth.W)) - - val receiveActive = RegInit(false.B) - val receiveSlot = RegInit(0.U(operandSlotWidth.W)) - val receiveWeightGeneration = RegInit(false.B) - val receiveBValidN = RegInit(tile.U(5.W)) - - val outputOrder = Module(new Queue(UInt(contextWidth.W), contextCount)) - val segmentOrder = Module(new Queue(UInt((contextWidth + operandSlotWidth).W), operandSlotCount)) - - val hasFreeSlot = !slotOccupied.asUInt.andR - val freeSlot = PriorityEncoder(~slotOccupied.asUInt) - val hasFreeContext = !contextAllocated.asUInt.andR - val freeContext = PriorityEncoder(~contextAllocated.asUInt) - val hasFreeOsAccBank = !osAccBankAllocated.asUInt.andR - val freeOsAccBank = PriorityEncoder(~osAccBankAllocated.asUInt) - val anyContextActive = contextActive.asUInt.orR - val activeContext = PriorityEncoder(contextActive.asUInt) - val pipelineAdvance = WireDefault(false.B) - - protected def wsAccAddress(context: UInt, row: UInt): UInt = - Cat(context, row(rowIndexWidth - 1, 0)) -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixEXDatapath.scala b/examples/balls/matrix/arch/src/main/scala/MatrixEXDatapath.scala deleted file mode 100644 index 8591da546..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixEXDatapath.scala +++ /dev/null @@ -1,501 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import framework.top.GlobalConfig - -class SystolicWsAccumReadEntry( - contextWidth: Int, - rowIndexWidth: Int, - accRowBits: Int, - partialRowBits: Int) extends Bundle { - val oldRow = UInt(accRowBits.W) - val partialRow = UInt(partialRowBits.W) - val context = UInt(contextWidth.W) - val row = UInt(rowIndexWidth.W) - val kTileKind = UInt(2.W) -} - -class SystolicWsAccumResultEntry( - contextWidth: Int, - rowIndexWidth: Int, - accRowBits: Int) extends Bundle { - val data = UInt(accRowBits.W) - val context = UInt(contextWidth.W) - val row = UInt(rowIndexWidth.W) - val kTileKind = UInt(2.W) -} - -abstract class SystolicArrayEXDatapath(b: GlobalConfig) extends SystolicArrayEXInput(b) { - val contextInputsReady = Wire(Vec(contextCount, Bool())) - val aInjectValid = Wire(Vec(tile, Bool())) - val aInjectData = Wire(Vec(tile, UInt(opElemBits.W))) - val aInjectContext = Wire(Vec(tile, UInt(contextWidth.W))) - val aInjectMRow = Wire(Vec(tile, UInt(5.W))) - val bInjectValid = Wire(Vec(tile, Bool())) - val bInjectData = Wire(Vec(tile, UInt(opElemBits.W))) - val bInjectContext = Wire(Vec(tile, UInt(contextWidth.W))) - - for (context <- 0 until contextCount) { - contextInputsReady(context) := true.B - } - - for (row <- 0 until tile) { - val contextHit = Wire(Vec(contextCount, Bool())) - val contextData = Wire(Vec(contextCount, UInt(opElemBits.W))) - val contextMRow = Wire(Vec(contextCount, UInt(5.W))) - - for (context <- 0 until contextCount) { - val logicalIndex = contextAge(context) - row.U - val expectsOsA = contextActive(context) && !contextWsMode(context) && - row.U < contextValidM(context) && contextAge(context) >= row.U && - logicalIndex < contextTotalK(context) - val expectsWsA = contextActive(context) && contextWsMode(context) && - row.U < contextTotalK(context) && contextAge(context) >= row.U && - logicalIndex < contextValidM(context) - val expectsA = expectsOsA || expectsWsA - val waitsForOsSegment = contextActive(context) && !contextWsMode(context) && - !contextFinalSeen(context) && row.U < contextValidM(context) && - contextAge(context) >= row.U && logicalIndex >= contextTotalK(context) - val sourceFound = WireDefault(false.B) - val sourceReady = WireDefault(false.B) - val sourceData = WireDefault(0.U(opElemBits.W)) - val sourceRowsReceived = WireDefault(false.B) - val sourceRowValid = WireDefault(false.B) - val sourceOwnerMatches = WireDefault(false.B) - val sourceWeightsReady = WireDefault(true.B) - val contextWeightBank = contextWeightGeneration(context).asUInt - val weightsReady = wsWeightBankValid(contextWeightBank) && - wsWeightBankValidN(contextWeightBank) === contextValidN(context) && - wsWeightBankValidK(contextWeightBank).pad(progressWidth) === contextTotalK(context) - - for (slot <- 0 until operandSlotCount) { - val slotEndK = slotKBase(slot) + slotValidK(slot).pad(progressWidth) - val osMatches = logicalIndex >= slotKBase(slot) && logicalIndex < slotEndK - val wsMatches = contextActiveSlot(context) === slot.U - val matches = slotOccupied(slot) && slotContext(slot) === context.U && - Mux(contextWsMode(context), wsMatches, osMatches) - when(matches) { - sourceFound := true.B - when(contextWsMode(context)) { - val aRowIndex = logicalIndex(rowIndexWidth - 1, 0) - sourceRowsReceived := slotARowsReceived(slot) > logicalIndex - sourceRowValid := aRowValid(slot)(aRowIndex) - sourceOwnerMatches := true.B - sourceWeightsReady := weightsReady - sourceReady := sourceRowsReceived && sourceRowValid && - sourceOwnerMatches && sourceWeightsReady - sourceData := rowByte( - aRowBuf(slot)(aRowIndex), - row) - }.otherwise { - val localK = logicalIndex - slotKBase(slot) - sourceRowsReceived := slotARowsReceived(slot) > row.U - sourceRowValid := aRowValid(slot)(row) - sourceOwnerMatches := true.B - sourceReady := sourceRowsReceived && sourceRowValid && sourceOwnerMatches - sourceData := dynamicRowByte(aRowBuf(slot)(row), localK) - } - } - } - - when(waitsForOsSegment || (expectsA && !(sourceFound && sourceReady))) { - contextInputsReady(context) := false.B - } - contextHit(context) := expectsA && sourceFound && sourceReady - contextData(context) := sourceData - contextMRow(context) := Mux(contextWsMode(context), logicalIndex(4, 0), row.U) - } - - assert(PopCount(contextHit) <= 1.U, - "SystolicArrayEX: multiple contexts attempted to inject into one A row") - aInjectValid(row) := contextHit.asUInt.orR - aInjectData(row) := Mux1H(contextHit, contextData) - aInjectContext(row) := PriorityEncoder(contextHit.asUInt) - aInjectMRow(row) := Mux1H(contextHit, contextMRow) - } - - for (col <- 0 until tile) { - val contextHit = Wire(Vec(contextCount, Bool())) - val contextData = Wire(Vec(contextCount, UInt(opElemBits.W))) - - for (context <- 0 until contextCount) { - val logicalK = contextAge(context) - col.U - val expectsB = contextActive(context) && !contextWsMode(context) && - col.U < contextValidN(context) && - contextAge(context) >= col.U && logicalK < contextTotalK(context) - val waitsForOsSegment = contextActive(context) && !contextWsMode(context) && - !contextFinalSeen(context) && col.U < contextValidN(context) && - contextAge(context) >= col.U && logicalK >= contextTotalK(context) - val sourceFound = WireDefault(false.B) - val sourceReady = WireDefault(false.B) - val sourceData = WireDefault(0.U(opElemBits.W)) - val sourceRowsReceived = WireDefault(false.B) - val sourceRowValid = WireDefault(false.B) - val sourceOwnerMatches = WireDefault(false.B) - - for (slot <- 0 until operandSlotCount) { - val slotEndK = slotKBase(slot) + slotValidK(slot).pad(progressWidth) - val localK = logicalK - slotKBase(slot) - val matches = slotOccupied(slot) && slotContext(slot) === context.U && - logicalK >= slotKBase(slot) && logicalK < slotEndK - val bRowIndex = localK(rowIndexWidth - 1, 0) - val bRow = bRowBuf(slot)(bRowIndex) - when(matches) { - sourceFound := true.B - sourceRowsReceived := slotBRowsReceived(slot) > localK - sourceRowValid := bRowValid(slot)(bRowIndex) - sourceOwnerMatches := true.B - sourceReady := sourceRowsReceived && sourceRowValid && sourceOwnerMatches - sourceData := rowByte(bRow, col) - } - } - - when(waitsForOsSegment || (expectsB && !(sourceFound && sourceReady))) { - contextInputsReady(context) := false.B - } - contextHit(context) := expectsB && sourceFound && sourceReady - contextData(context) := sourceData - } - - assert(PopCount(contextHit) <= 1.U, - "SystolicArrayEX: multiple contexts attempted to inject into one B column") - bInjectValid(col) := contextHit.asUInt.orR - bInjectData(col) := Mux1H(contextHit, contextData) - bInjectContext(col) := PriorityEncoder(contextHit.asUInt) - } - - val aPipeData = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(opElemBits.W)))))) - val aPipeValid = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(false.B))))) - val aPipeContext = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(contextWidth.W)))))) - val aPipeMRow = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(5.W)))))) - val aStepData = Wire(Vec(tile, Vec(tile, UInt(opElemBits.W)))) - val aStepValid = Wire(Vec(tile, Vec(tile, Bool()))) - val aStepContext = Wire(Vec(tile, Vec(tile, UInt(contextWidth.W)))) - val aStepMRow = Wire(Vec(tile, Vec(tile, UInt(5.W)))) - val bStepData = Wire(Vec(tile, Vec(tile, UInt(opElemBits.W)))) - val bStepValid = Wire(Vec(tile, Vec(tile, Bool()))) - val bStepContext = Wire(Vec(tile, Vec(tile, UInt(contextWidth.W)))) - val wsPsumData = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(wsPsumBits.W)))))) - val wsPsumValid = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(false.B))))) - val wsPsumContext = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(contextWidth.W)))))) - val wsPsumMRow = RegInit(VecInit(Seq.tabulate(tile)(_ => - VecInit(Seq.fill(tile)(0.U(5.W)))))) - - for (row <- 0 until tile) { - for (col <- 0 until tile) { - if (col == 0) { - aStepData(row)(col) := aInjectData(row) - aStepValid(row)(col) := aInjectValid(row) - aStepContext(row)(col) := aInjectContext(row) - aStepMRow(row)(col) := aInjectMRow(row) - } else { - val sourceContext = aPipeContext(row)(col - 1) - val sourceValidN = contextValidN(sourceContext) - aStepData(row)(col) := aPipeData(row)(col - 1) - aStepValid(row)(col) := aPipeValid(row)(col - 1) && col.U < sourceValidN - aStepContext(row)(col) := sourceContext - aStepMRow(row)(col) := aPipeMRow(row)(col - 1) - } - - if (row == 0) { - bStepData(row)(col) := bInjectData(col) - bStepValid(row)(col) := bInjectValid(col) - bStepContext(row)(col) := bInjectContext(col) - } else { - val sourceContext = bPipeContext(row - 1)(col) - val sourceValidM = contextValidM(sourceContext) - bStepData(row)(col) := bPipeData(row - 1)(col) - bStepValid(row)(col) := bPipeValid(row - 1)(col) && row.U < sourceValidM - bStepContext(row)(col) := sourceContext - } - } - } - - val wsPartialSum = Wire(Vec(tile, Vec(tile, UInt(wsPsumBits.W)))) - val wsComputeValid = Wire(Vec(tile, Vec(tile, Bool()))) - val wsBypassValid = Wire(Vec(tile, Vec(tile, Bool()))) - val wsStageData = Wire(Vec(tile, Vec(tile, UInt(wsPsumBits.W)))) - val wsStageValid = Wire(Vec(tile, Vec(tile, Bool()))) - val wsStageContext = Wire(Vec(tile, Vec(tile, UInt(contextWidth.W)))) - val wsStageMRow = Wire(Vec(tile, Vec(tile, UInt(5.W)))) - for (row <- 0 until tile) { - for (col <- 0 until tile) { - val targetContext = aStepContext(row)(col) - val weightData = Mux(contextWeightGeneration(targetContext), - rowByte(wsBBuffer(row), col), bPipeData(row)(col)) - val product = aStepData(row)(col).asSInt * weightData.asSInt - wsComputeValid(row)(col) := aStepValid(row)(col) && contextWsMode(targetContext) - if (row == 0) { - wsPartialSum(row)(col) := product.pad(wsPsumBits).asUInt - wsBypassValid(row)(col) := false.B - wsStageData(row)(col) := wsPartialSum(row)(col) - wsStageContext(row)(col) := targetContext - wsStageMRow(row)(col) := aStepMRow(row)(col) - } else { - val upstreamContext = wsPsumContext(row - 1)(col) - wsPartialSum(row)(col) := (wsPsumData(row - 1)(col).asSInt + - product.pad(wsPsumBits)).asUInt - wsBypassValid(row)(col) := wsPsumValid(row - 1)(col) && - contextWsMode(upstreamContext) && - row.U >= contextTotalK(upstreamContext) - wsStageData(row)(col) := Mux(wsComputeValid(row)(col), - wsPartialSum(row)(col), wsPsumData(row - 1)(col)) - wsStageContext(row)(col) := Mux(wsComputeValid(row)(col), - targetContext, upstreamContext) - wsStageMRow(row)(col) := Mux(wsComputeValid(row)(col), - aStepMRow(row)(col), wsPsumMRow(row - 1)(col)) - } - wsStageValid(row)(col) := wsComputeValid(row)(col) || wsBypassValid(row)(col) - } - } - - val wsTerminalValid = Wire(Vec(tile, Bool())) - val wsTerminalData = Wire(Vec(tile, UInt(wsPsumBits.W))) - val wsTerminalContext = Wire(Vec(tile, UInt(contextWidth.W))) - val wsTerminalMRow = Wire(Vec(tile, UInt(5.W))) - val wsTerminalKTileKind = Wire(Vec(tile, UInt(2.W))) - for (col <- 0 until tile) { - wsTerminalValid(col) := wsStageValid(tile - 1)(col) - wsTerminalData(col) := wsStageData(tile - 1)(col) - wsTerminalContext(col) := wsStageContext(tile - 1)(col) - wsTerminalMRow(col) := wsStageMRow(tile - 1)(col) - wsTerminalKTileKind(col) := contextWsKTileKind(wsTerminalContext(col)) - } - - // Column c completes c cycles after column 0. Delay it by 15-c cycles so a - // complete 512-bit M row reaches the accumulator pipeline in one cycle. - val wsDeskewData = Seq.tabulate(tile) { col => - RegInit(VecInit(Seq.fill(math.max(tile - 1 - col, 1))(0.U(wsPsumBits.W)))) - } - val wsDeskewValid = Seq.tabulate(tile) { col => - RegInit(VecInit(Seq.fill(math.max(tile - 1 - col, 1))(false.B))) - } - val wsDeskewContext = Seq.tabulate(tile) { col => - RegInit(VecInit(Seq.fill(math.max(tile - 1 - col, 1))(0.U(contextWidth.W)))) - } - val wsDeskewMRow = Seq.tabulate(tile) { col => - RegInit(VecInit(Seq.fill(math.max(tile - 1 - col, 1))(0.U(5.W)))) - } - val wsDeskewKTileKind = Seq.tabulate(tile) { col => - RegInit(VecInit(Seq.fill(math.max(tile - 1 - col, 1))(SystolicKTileKind.DIRECT))) - } - - val wsAlignedValid = Wire(Vec(tile, Bool())) - val wsAlignedData = Wire(Vec(tile, UInt(wsPsumBits.W))) - val wsAlignedContext = Wire(Vec(tile, UInt(contextWidth.W))) - val wsAlignedMRow = Wire(Vec(tile, UInt(5.W))) - val wsAlignedKTileKind = Wire(Vec(tile, UInt(2.W))) - for (col <- 0 until tile) { - val delay = tile - 1 - col - if (delay == 0) { - wsAlignedValid(col) := wsTerminalValid(col) - wsAlignedData(col) := wsTerminalData(col) - wsAlignedContext(col) := wsTerminalContext(col) - wsAlignedMRow(col) := wsTerminalMRow(col) - wsAlignedKTileKind(col) := wsTerminalKTileKind(col) - } else { - wsAlignedValid(col) := wsDeskewValid(col)(delay - 1) - wsAlignedData(col) := wsDeskewData(col)(delay - 1) - wsAlignedContext(col) := wsDeskewContext(col)(delay - 1) - wsAlignedMRow(col) := wsDeskewMRow(col)(delay - 1) - wsAlignedKTileKind(col) := wsDeskewKTileKind(col)(delay - 1) - } - } - - val wsAlignedRowValid = wsAlignedValid(0) - val wsAlignedRowContext = wsAlignedContext(0) - val wsAlignedRowIndex = wsAlignedMRow(0)(rowIndexWidth - 1, 0) - val wsAlignedRowKTileKind = wsAlignedKTileKind(0) - val wsAlignedPartialRow = Cat((0 until tile).reverse.map(col => wsAlignedData(col))) - val wsDeskewPending = (0 until tile - 1).map(col => - wsDeskewValid(col).asUInt.orR).reduce(_ || _) - - val wsReadQueue = Module(new Queue(new SystolicWsAccumReadEntry( - contextWidth, rowIndexWidth, accRowBits, wsPsumRowBits), 2)) - val wsResultBuffer = Module(new Queue(new SystolicWsAccumResultEntry( - contextWidth, rowIndexWidth, accRowBits), 1, pipe = true)) - - val wsReadIssue = Wire(Bool()) - val wsAccReadData = wsAccMem.read( - wsAccAddress(wsAlignedRowContext, wsAlignedRowIndex), wsReadIssue) - val wsReadPartialRow = RegEnable(wsAlignedPartialRow, wsReadIssue) - val wsReadContext = RegEnable(wsAlignedRowContext, wsReadIssue) - val wsReadRow = RegEnable(wsAlignedRowIndex, wsReadIssue) - val wsReadKTileKind = RegEnable(wsAlignedRowKTileKind, wsReadIssue) - val wsReadPending = RegNext(wsReadIssue, false.B) - - wsReadQueue.io.enq.valid := wsReadPending - wsReadQueue.io.enq.bits.oldRow := wsAccReadData - wsReadQueue.io.enq.bits.partialRow := wsReadPartialRow - wsReadQueue.io.enq.bits.context := wsReadContext - wsReadQueue.io.enq.bits.row := wsReadRow - wsReadQueue.io.enq.bits.kTileKind := wsReadKTileKind - - val wsIgnoreOld = wsReadQueue.io.deq.bits.kTileKind === SystolicKTileKind.DIRECT || - wsReadQueue.io.deq.bits.kTileKind === SystolicKTileKind.FIRST - val wsAccumulatedElements = Wire(Vec(tile, UInt(accElemBits.W))) - for (col <- 0 until tile) { - val oldValue = wsReadQueue.io.deq.bits.oldRow( - (col + 1) * accElemBits - 1, col * accElemBits).asSInt - val partialValue = wsReadQueue.io.deq.bits.partialRow( - (col + 1) * wsPsumBits - 1, col * wsPsumBits).asSInt.pad(accElemBits) - wsAccumulatedElements(col) := Mux(wsIgnoreOld, - partialValue, oldValue + partialValue).asUInt - } - - wsResultBuffer.io.enq.valid := wsReadQueue.io.deq.valid - wsResultBuffer.io.enq.bits.data := Cat(wsAccumulatedElements.reverse) - wsResultBuffer.io.enq.bits.context := wsReadQueue.io.deq.bits.context - wsResultBuffer.io.enq.bits.row := wsReadQueue.io.deq.bits.row - wsResultBuffer.io.enq.bits.kTileKind := wsReadQueue.io.deq.bits.kTileKind - wsReadQueue.io.deq.ready := wsResultBuffer.io.enq.ready - - - val wsReadReservations = wsReadQueue.io.count +& wsReadPending.asUInt - val wsReadCanReserve = wsReadReservations < 2.U || wsReadQueue.io.deq.fire - val wsAccumCanAdvance = !wsAlignedRowValid || wsReadCanReserve - val activeInputsReady = (0 until contextCount).map(context => - !contextActive(context) || contextInputsReady(context)).reduce(_ && _) - pipelineAdvance := (anyContextActive || wsDeskewPending) && activeInputsReady && - wsAccumCanAdvance - wsReadIssue := pipelineAdvance && wsAlignedRowValid - - - when(pipelineAdvance) { - for (col <- 0 until tile) { - val delay = tile - 1 - col - if (delay > 0) { - wsDeskewData(col)(0) := wsTerminalData(col) - wsDeskewValid(col)(0) := wsTerminalValid(col) - wsDeskewContext(col)(0) := wsTerminalContext(col) - wsDeskewMRow(col)(0) := wsTerminalMRow(col) - wsDeskewKTileKind(col)(0) := wsTerminalKTileKind(col) - for (stage <- 1 until delay) { - wsDeskewData(col)(stage) := wsDeskewData(col)(stage - 1) - wsDeskewValid(col)(stage) := wsDeskewValid(col)(stage - 1) - wsDeskewContext(col)(stage) := wsDeskewContext(col)(stage - 1) - wsDeskewMRow(col)(stage) := wsDeskewMRow(col)(stage - 1) - wsDeskewKTileKind(col)(stage) := wsDeskewKTileKind(col)(stage - 1) - } - } - } - - for (row <- 0 until tile) { - for (col <- 0 until tile) { - aPipeData(row)(col) := aStepData(row)(col) - aPipeValid(row)(col) := aStepValid(row)(col) - aPipeContext(row)(col) := aStepContext(row)(col) - aPipeMRow(row)(col) := aStepMRow(row)(col) - - when(anyContextActive && !contextWsMode(activeContext)) { - bPipeData(row)(col) := bStepData(row)(col) - bPipeValid(row)(col) := bStepValid(row)(col) - bPipeContext(row)(col) := bStepContext(row)(col) - } - - wsPsumValid(row)(col) := false.B - - assert(!(wsComputeValid(row)(col) && wsBypassValid(row)(col)), - "SystolicArrayEX: WS compute and bypass collided in one PE") - - when(wsStageValid(row)(col)) { - val targetContext = aStepContext(row)(col) - when(wsComputeValid(row)(col)) { - val targetWeightBank = contextWeightGeneration(targetContext).asUInt - - assert(wsWeightBankValid(targetWeightBank), - "SystolicArrayEX: WS used an invalid weight bank") - assert(wsWeightBankValidN(targetWeightBank) === contextValidN(targetContext) && - wsWeightBankValidK(targetWeightBank).pad(progressWidth) === - contextTotalK(targetContext), - "SystolicArrayEX: WS weight bank metadata does not match its context") - - if (row == 0) { - // No upstream partial sum exists in the first physical PE row. - } else { - assert(wsPsumValid(row - 1)(col), - "SystolicArrayEX: WS partial sum did not arrive from the previous PE row") - assert(wsPsumContext(row - 1)(col) === targetContext, - "SystolicArrayEX: WS partial-sum context changed between PE rows") - assert(wsPsumMRow(row - 1)(col) === aStepMRow(row)(col), - "SystolicArrayEX: WS partial-sum M row changed between PE rows") - } - } - - wsPsumData(row)(col) := wsStageData(row)(col) - wsPsumValid(row)(col) := true.B - wsPsumContext(row)(col) := wsStageContext(row)(col) - wsPsumMRow(row)(col) := wsStageMRow(row)(col) - }.elsewhen(aStepValid(row)(col) && bStepValid(row)(col)) { - assert(aStepContext(row)(col) === bStepContext(row)(col), - "SystolicArrayEX: A/B context tags do not match") - val product = aStepData(row)(col).asSInt * bStepData(row)(col).asSInt - val targetContext = aStepContext(row)(col) - val targetOsAccBank = contextOsAccBank(targetContext) - assert(osAccBankAllocated(targetOsAccBank), - "SystolicArrayEX: OS context used an unallocated accumulator bank") - assert(osAccBankOwner(targetOsAccBank) === targetContext, - "SystolicArrayEX: OS accumulator bank owner mismatch") - osAcc(targetOsAccBank)(row)(col) := - osAcc(targetOsAccBank)(row)(col) + product.pad(accElemBits).asUInt - } - } - } - - for (context <- 0 until contextCount) { - when(contextActive(context)) { - val firstCompleteCycle = contextValidN(context).pad(progressWidth) + - contextTotalK(context) - 2.U - val osLastCycle = contextValidM(context).pad(progressWidth) + - contextValidN(context).pad(progressWidth) + contextTotalK(context) - 3.U - val wsLastCycle = contextValidM(context).pad(progressWidth) + - contextValidN(context).pad(progressWidth) + tile.U(progressWidth.W) - 3.U - val lastCycle = Mux(contextWsMode(context), wsLastCycle, osLastCycle) - - when(!contextWsMode(context) && contextFinalSeen(context) && - contextAge(context) >= firstCompleteCycle && - contextRowsComplete(context) < contextValidM(context)) { - contextRowsComplete(context) := contextRowsComplete(context) + 1.U - } - - when(contextAge(context) >= lastCycle) { - contextActive(context) := false.B - contextAge(context) := 0.U - }.otherwise { - contextAge(context) := contextAge(context) + 1.U - } - } - } - - } - - when(pipelineAdvance) { - when(wsAlignedRowValid) { - for (col <- 0 until tile) { - when(col.U < contextValidN(wsAlignedRowContext)) { - assert(wsAlignedValid(col), - "SystolicArrayEX: WS de-skew lost a valid result column") - assert(wsAlignedContext(col) === wsAlignedRowContext && - wsAlignedMRow(col) === wsAlignedMRow(0) && - wsAlignedKTileKind(col) === wsAlignedRowKTileKind, - "SystolicArrayEX: WS de-skew mixed rows or contexts") - }.otherwise { - assert(!wsAlignedValid(col), - "SystolicArrayEX: WS de-skew produced a column outside valid N") - } - } - } - } - when(wsReadPending) { - assert(wsReadQueue.io.enq.ready, - "SystolicArrayEX: WS accumulator read-response queue overflow") - } -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixEXInput.scala b/examples/balls/matrix/arch/src/main/scala/MatrixEXInput.scala deleted file mode 100644 index efb78b1e3..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixEXInput.scala +++ /dev/null @@ -1,280 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import framework.top.GlobalConfig - -abstract class SystolicArrayEXInput(b: GlobalConfig) extends SystolicArrayEXBase(b) { - // Load 保证同一 tile 的两条行流不会交错。EX 在第一条 A 或 B 真正握手时占用 - // receiveSlot，之后 A/B 使用各自行号写入同一个 slot，直到两条流都完成。 - val firstReceiveRow = !receiveActive - val activeReqKind = Mux(firstReceiveRow, io.load_ex_req_kind, slotReqKind(receiveSlot)) - val activeValidM = Mux(firstReceiveRow, io.load_ex_valid_m, slotValidM(receiveSlot)) - val activeBValidN = Mux(firstReceiveRow, io.load_ex_b_valid_n, receiveBValidN) - val activeBValidK = Mux(firstReceiveRow, io.load_ex_b_valid_k, slotBValidK(receiveSlot)) - val activeARowLimit = activeValidM - val activeBRowLimit = Mux(needOp2(activeReqKind), activeBValidK, 0.U(5.W)) - val activeARowsReceived = Mux(firstReceiveRow, 0.U(5.W), slotARowsReceived(receiveSlot)) - val activeBRowsReceived = Mux(firstReceiveRow, 0.U(5.W), slotBRowsReceived(receiveSlot)) - val firstLoadsPeWeights = io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_A_B_PE - val firstPrefetchesWeights = io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_A_B_BUF - val firstIsNewOutput = isNewOutputTile(io.load_ex_k_tile_kind) - val firstIsContinuation = isContinuationTile(io.load_ex_k_tile_kind) - val firstUsesExplicitContext = isWsKind(io.load_ex_req_kind) - val firstWeightGeneration = io.load_ex_weight_generation - val firstWritesWeightBank = firstLoadsPeWeights || firstPrefetchesWeights - val firstBWeightGeneration = Mux(firstPrefetchesWeights, - !firstWeightGeneration, firstWeightGeneration) - val firstWsMappedContext = wsContextMap(io.load_ex_acc_slot) - val firstWsMappingValid = wsContextMapValid(io.load_ex_acc_slot) && - contextAllocated(firstWsMappedContext) && contextWsMode(firstWsMappedContext) - val firstTargetContext = Mux( - firstUsesExplicitContext, - Mux(firstIsContinuation, firstWsMappedContext, freeContext), - Mux(firstIsContinuation, chainContext, freeContext)) - val firstNeedsLaunchQueue = firstIsNewOutput || (firstIsContinuation && firstUsesExplicitContext) - val firstNewContextResourcesAvailable = hasFreeContext && - (firstUsesExplicitContext || hasFreeOsAccBank) - val firstContextAvailable = Mux( - firstIsContinuation, - Mux(firstUsesExplicitContext, - firstWsMappingValid && !contextPendingStart(firstTargetContext), - contextAllocated(firstTargetContext) && chainValid), - firstNewContextResourcesAvailable) - def weightBankInUse(generation: Bool): Bool = (0 until contextCount).map { context => - contextWsMode(context) && ( - (contextActive(context) && contextWeightGeneration(context) === generation) || - (contextPendingStart(context) && contextPendingWeightGeneration(context) === generation)) - }.reduce(_ || _) - val firstWeightBankInUse = weightBankInUse(firstBWeightGeneration) - val anyOsContextReserved = (0 until contextCount).map { context => - (contextActive(context) || contextPendingStart(context)) && !contextWsMode(context) - }.reduce(_ || _) - val firstModeSafe = !firstWritesWeightBank || - (!anyOsContextReserved && (!firstWeightBankInUse || firstPrefetchesWeights)) - val firstUsesResidentWeights = io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_A_ONLY || - firstPrefetchesWeights - val firstResidentBankSafe = firstWeightGeneration || !anyOsContextReserved - val firstResidentWeightsMatch = wsWeightBankValid(firstWeightGeneration.asUInt) && - wsWeightBankValidN(firstWeightGeneration.asUInt) === io.load_ex_valid_n && - wsWeightBankValidK(firstWeightGeneration.asUInt) === io.load_ex_valid_k && - firstResidentBankSafe - val firstBaseCanReceive = hasFreeSlot && - firstContextAvailable && (!firstNeedsLaunchQueue || segmentOrder.io.enq.ready) && - (!firstIsNewOutput || outputOrder.io.enq.ready) && - firstModeSafe && - (!firstUsesResidentWeights || firstResidentWeightsMatch) - val firstPeRowsSafe = true.B - val firstCanReceive = firstBaseCanReceive && (!firstLoadsPeWeights || firstPeRowsSafe) - val activeNeedOp2 = needOp2(activeReqKind) - val activeLoadsPeWeights = activeReqKind === SystolicCtrlLoadReqKind.READ_A_B_PE - val activePrefetchesWeights = activeReqKind === SystolicCtrlLoadReqKind.READ_A_B_BUF - val activeWeightGeneration = Mux(firstReceiveRow, - firstWeightGeneration, receiveWeightGeneration) - val activeBWeightGeneration = Mux(activePrefetchesWeights, - !activeWeightGeneration, activeWeightGeneration) - val activeWeightBankWriteSafe = - (activeLoadsPeWeights || !weightBankInUse(activeBWeightGeneration)) && - (activeBWeightGeneration || !anyOsContextReserved) - val activePeRowSafe = true.B - val receiveAllowed = Mux(firstReceiveRow, firstCanReceive, true.B) - val activeARowIndex = activeARowsReceived(rowIndexWidth - 1, 0) - val activeBRowIndex = activeBRowsReceived(rowIndexWidth - 1, 0) - val currentSlot = Mux(firstReceiveRow, freeSlot, receiveSlot) - val activeARowOverwriteSafe = !aRowValid(currentSlot)(activeARowIndex) - val activeBRowOverwriteSafe = !bRowValid(currentSlot)(activeBRowIndex) - - io.load_ex_op1_i.ready := receiveAllowed && - activeARowsReceived < activeARowLimit && activeARowOverwriteSafe - io.load_ex_op2_i.ready := receiveAllowed && activeNeedOp2 && - activeBRowsReceived < activeBRowLimit && - Mux(activeLoadsPeWeights || activePrefetchesWeights, - activePeRowSafe && activeWeightBankWriteSafe, activeBRowOverwriteSafe) - - - val op1Fire = io.load_ex_op1_i.fire - val op2Fire = io.load_ex_op2_i.fire - val receiveEvent = op1Fire || op2Fire - val firstReceiveEvent = firstReceiveRow && receiveEvent - val currentARowIndex = activeARowIndex - val currentBRowIndex = activeBRowIndex - val aFinishesThisCycle = op1Fire && activeARowsReceived + 1.U >= activeARowLimit - val bFinishesThisCycle = op2Fire && activeBRowsReceived + 1.U >= activeBRowLimit - val aDoneNext = activeARowsReceived === activeARowLimit || aFinishesThisCycle - val bDoneNext = !activeNeedOp2 || activeBRowsReceived === activeBRowLimit || bFinishesThisCycle - val inputCompleteNext = aDoneNext && bDoneNext - outputOrder.io.enq.valid := firstReceiveEvent && firstIsNewOutput - outputOrder.io.enq.bits := firstTargetContext - segmentOrder.io.enq.valid := firstReceiveEvent && firstNeedsLaunchQueue - segmentOrder.io.enq.bits := Cat(firstTargetContext, freeSlot) - - when(op1Fire) { - aRowBuf(currentSlot)(currentARowIndex) := io.load_ex_op1_i.bits - aRowValid(currentSlot)(currentARowIndex) := true.B - } - - when(op2Fire) { - when(activeLoadsPeWeights || activePrefetchesWeights) { - when(activeBWeightGeneration) { - wsBBuffer(currentBRowIndex) := io.load_ex_op2_i.bits - }.otherwise { - for (col <- 0 until tile) { - bPipeData(currentBRowIndex)(col) := rowByte(io.load_ex_op2_i.bits, col) - } - } - }.otherwise { - bRowBuf(currentSlot)(currentBRowIndex) := io.load_ex_op2_i.bits - bRowValid(currentSlot)(currentBRowIndex) := true.B - } - } - - when(firstReceiveEvent) { - slotOccupied(freeSlot) := true.B - slotInputComplete(freeSlot) := inputCompleteNext - slotUseDone(freeSlot) := false.B - slotLaunchPending(freeSlot) := firstNeedsLaunchQueue - slotContext(freeSlot) := firstTargetContext - slotReqKind(freeSlot) := io.load_ex_req_kind - slotValidM(freeSlot) := io.load_ex_valid_m - slotValidN(freeSlot) := io.load_ex_valid_n - slotValidK(freeSlot) := io.load_ex_valid_k - slotBValidK(freeSlot) := io.load_ex_b_valid_k - slotARowsReceived(freeSlot) := Mux(op1Fire, 1.U, 0.U) - slotBRowsReceived(freeSlot) := Mux(op2Fire, 1.U, 0.U) - slotKTileKind(freeSlot) := io.load_ex_k_tile_kind - slotKBase(freeSlot) := Mux( - firstUsesExplicitContext || firstIsNewOutput, - 0.U, - contextTotalK(firstTargetContext)) - slotWeightGeneration(freeSlot) := firstWeightGeneration - when(firstUsesExplicitContext) { - contextPendingWeightGeneration(firstTargetContext) := firstWeightGeneration - } - receiveWeightGeneration := firstWeightGeneration - receiveBValidN := io.load_ex_b_valid_n - when(firstNeedsLaunchQueue) { - contextPendingStart(firstTargetContext) := true.B - } - - when(firstIsNewOutput) { - when(firstUsesExplicitContext) { - for (logicalSlot <- 0 until contextCount) { - when(wsContextMapValid(logicalSlot) && - wsContextMap(logicalSlot) === firstTargetContext) { - wsContextMapValid(logicalSlot) := false.B - } - } - wsContextMap(io.load_ex_acc_slot) := firstTargetContext - wsContextMapValid(io.load_ex_acc_slot) := true.B - } - contextAllocated(firstTargetContext) := true.B - contextFinalSeen(firstTargetContext) := - io.load_ex_k_tile_kind === SystolicKTileKind.DIRECT - contextValidM(firstTargetContext) := io.load_ex_valid_m - contextValidN(firstTargetContext) := io.load_ex_valid_n - contextTotalK(firstTargetContext) := Mux( - firstUsesExplicitContext, - 0.U, - io.load_ex_valid_k.pad(progressWidth)) - contextAge(firstTargetContext) := 0.U - contextRowsComplete(firstTargetContext) := 0.U - contextSendRow(firstTargetContext) := 0.U - contextWsMode(firstTargetContext) := firstUsesExplicitContext - contextWsSegmentPending(firstTargetContext) := false.B - contextWsRowsCommitted(firstTargetContext) := 0.U - - when(!firstUsesExplicitContext) { - contextOsAccBank(firstTargetContext) := freeOsAccBank - osAccBankAllocated(freeOsAccBank) := true.B - osAccBankOwner(freeOsAccBank) := firstTargetContext - for (bank <- 0 until osAccBankCount) { - when(freeOsAccBank === bank.U) { - for (row <- 0 until tile) { - for (col <- 0 until tile) { - osAcc(bank)(row)(col) := 0.U - } - } - } - } - } - - when(io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_AB && - io.load_ex_k_tile_kind === SystolicKTileKind.FIRST) { - chainValid := true.B - chainContext := firstTargetContext - } - }.otherwise { - assert(firstIsContinuation, "SystolicArrayEX: invalid continuation request") - assert(contextWsMode(firstTargetContext) === firstUsesExplicitContext, - "SystolicArrayEX: dataflow mode changed inside a K-tile chain") - assert(contextValidM(firstTargetContext) === io.load_ex_valid_m, - "SystolicArrayEX: M extent changed inside a K-tile chain") - assert(contextValidN(firstTargetContext) === io.load_ex_valid_n, - "SystolicArrayEX: N extent changed inside a K-tile chain") - when(io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_AB && - io.load_ex_k_tile_kind === SystolicKTileKind.LAST) { - chainValid := false.B - } - when(!firstUsesExplicitContext) { - contextTotalK(firstTargetContext) := - contextTotalK(firstTargetContext) + io.load_ex_valid_k.pad(progressWidth) - when(io.load_ex_k_tile_kind === SystolicKTileKind.LAST) { - contextFinalSeen(firstTargetContext) := true.B - } - } - } - - when(firstUsesResidentWeights) { - assert(wsWeightBankValid(firstWeightGeneration.asUInt), - "SystolicArrayEX: WS request arrived without resident weights") - assert(wsWeightBankValidN(firstWeightGeneration.asUInt) === io.load_ex_valid_n && - wsWeightBankValidK(firstWeightGeneration.asUInt) === io.load_ex_valid_k, - "SystolicArrayEX: WS request metadata does not match PE weights") - } - when(firstPrefetchesWeights) { - wsPrefetchGeneration := !firstWeightGeneration - } - } - - when(!firstReceiveRow && op1Fire) { - slotARowsReceived(receiveSlot) := slotARowsReceived(receiveSlot) + 1.U - } - when(!firstReceiveRow && op2Fire) { - slotBRowsReceived(receiveSlot) := slotBRowsReceived(receiveSlot) + 1.U - } - - when(firstReceiveEvent && firstPrefetchesWeights) { - assert(!wsFinalReusePending, - "SystolicArrayEX: started a new WS prefetch before the final old-weight reuse arrived") - wsFinalReusePending := true.B - }.elsewhen(firstReceiveEvent && wsFinalReusePending) { - assert(io.load_ex_req_kind === SystolicCtrlLoadReqKind.READ_A_ONLY && - firstWeightGeneration =/= wsPrefetchGeneration, - "SystolicArrayEX: WS prefetch was not followed by the final old-weight reuse") - wsFinalReusePending := false.B - } - - when(firstReceiveEvent) { - receiveSlot := freeSlot - receiveActive := !inputCompleteNext - }.elsewhen(!firstReceiveRow && receiveEvent) { - when(inputCompleteNext) { - slotInputComplete(receiveSlot) := true.B - } - receiveActive := !inputCompleteNext - } - - when(op2Fire && activeBRowsReceived === 0.U && - (activeLoadsPeWeights || activePrefetchesWeights)) { - wsWeightBankValid(activeBWeightGeneration.asUInt) := false.B - wsWeightBankValidN(activeBWeightGeneration.asUInt) := activeBValidN - wsWeightBankValidK(activeBWeightGeneration.asUInt) := activeBValidK - } - - when(receiveEvent && inputCompleteNext && - (activeLoadsPeWeights || activePrefetchesWeights)) { - wsWeightBankValid(activeBWeightGeneration.asUInt) := true.B - wsWeightBankValidN(activeBWeightGeneration.asUInt) := activeBValidN - wsWeightBankValidK(activeBWeightGeneration.asUInt) := activeBValidK - } -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixLoad.scala b/examples/balls/matrix/arch/src/main/scala/MatrixLoad.scala deleted file mode 100644 index 12fdc7802..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixLoad.scala +++ /dev/null @@ -1,297 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import framework.memdomain.backend.banks.{SramReadReq, SramReadResp} -import framework.top.GlobalConfig - -@instantiable -class SystolicArrayLoad(val b: GlobalConfig) extends Module { - private val tile = SystolicArrayConst.Tile - private val opRowBits = SystolicArrayConst.OpRowBits - private val abRows = b.memDomain.bankEntries - private val bankWidth = log2Up(b.memDomain.bankNum) - private val groupWidth = log2Up(b.memDomain.bankNum) - private val addrWidth = log2Up(b.memDomain.bankEntries) - private val rowIndexWidth = log2Ceil(tile) - private val wsReuseTiles = SystolicArrayConst.wsReuseTiles(b) - private val slotCount = 3 - private val slotWidth = log2Ceil(slotCount) - private val tagQueueDepth = 16 - private val rowFifoDepth = 2 - - private val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "MatrixBall") - .getOrElse(throw new IllegalArgumentException("MatrixBall not found in config")) - private val inBW = ballMapping.inBW - - require(inBW >= 2, "SystolicArrayLoad requires at least two read ports") - require(b.memDomain.bankWidth == opRowBits, "SystolicArrayLoad expects one 16xi8 row per 128-bit bank read") - require( - b.memDomain.bankEntries % tile == 0, - "SystolicArrayLoad expects bankEntries to be an integer number of 16-row A/B tiles" - ) - - @public - val io = IO(new Bundle { - val bankReadReq = Vec(inBW, Decoupled(new SramReadReq(b))) - val bankReadResp = Vec(inBW, Flipped(Decoupled(new SramReadResp(b)))) - - val ctrl_ld_i = Flipped(Decoupled(new SystolicCtrlLoadReq(b))) - - val load_ex_req_kind = Output(UInt(2.W)) - val load_ex_k_tile_kind = Output(UInt(2.W)) - val load_ex_acc_slot = Output(UInt(log2Ceil(wsReuseTiles).W)) - val load_ex_valid_m = Output(UInt(5.W)) - val load_ex_valid_n = Output(UInt(5.W)) - val load_ex_valid_k = Output(UInt(5.W)) - val load_ex_b_valid_n = Output(UInt(5.W)) - val load_ex_b_valid_k = Output(UInt(5.W)) - val load_ex_weight_generation = Output(Bool()) - val load_ex_op1_o = Decoupled(UInt(opRowBits.W)) - val load_ex_op2_o = Decoupled(UInt(opRowBits.W)) - - val op1_rd_bank_o = Output(UInt(bankWidth.W)) - val op1_rd_group_o = Output(UInt(groupWidth.W)) - val op2_rd_bank_o = Output(UInt(bankWidth.W)) - val op2_rd_group_o = Output(UInt(groupWidth.W)) - }) - - private def fitTo(x: UInt, width: Int): UInt = - if (x.getWidth >= width) x(width - 1, 0) else x.pad(width) - - private def rowGroup(baseGroup: UInt, rowBase: UInt, rowIdx: UInt): UInt = { - val rowLinear = rowBase.pad(16) + rowIdx.pad(16) - fitTo(baseGroup.pad(16) + (rowLinear / abRows.U), groupWidth) - } - - private def rowAddr(rowBase: UInt, rowIdx: UInt): UInt = { - val rowLinear = rowBase.pad(16) + rowIdx.pad(16) - fitTo(rowLinear % abRows.U, addrWidth) - } - - private def needOp2(kind: UInt): Bool = - kind === SystolicCtrlLoadReqKind.READ_AB || kind === SystolicCtrlLoadReqKind.READ_A_B_PE || - kind === SystolicCtrlLoadReqKind.READ_A_B_BUF - - private def nextSlot(slot: UInt): UInt = - Mux(slot === (slotCount - 1).U, 0.U, slot + 1.U) - - class LoadSlot extends Bundle { - val req_kind = UInt(2.W) - val k_tile_kind = UInt(2.W) - val acc_slot = UInt(log2Ceil(wsReuseTiles).W) - val valid_m = UInt(5.W) - val valid_n = UInt(5.W) - val valid_k = UInt(5.W) - val b_valid_n = UInt(5.W) - val b_valid_k = UInt(5.W) - val weight_generation = Bool() - val op1_bank = UInt(bankWidth.W) - val op1_group = UInt(groupWidth.W) - val op1_row_base = UInt(addrWidth.W) - val op2_bank = UInt(bankWidth.W) - val op2_group = UInt(groupWidth.W) - val op2_row_base = UInt(addrWidth.W) - } - - class ReadTag extends Bundle { - val slot = UInt(slotWidth.W) - val row = UInt(rowIndexWidth.W) - } - - class RowData extends Bundle { - val slot = UInt(slotWidth.W) - val row = UInt(rowIndexWidth.W) - val data = UInt(opRowBits.W) - } - - val slotReq = RegInit(VecInit(Seq.fill(slotCount)(0.U.asTypeOf(new LoadSlot)))) - val slotOccupied = RegInit(VecInit(Seq.fill(slotCount)(false.B))) - val slotAReqCount = RegInit(VecInit(Seq.fill(slotCount)(0.U(5.W)))) - val slotBReqCount = RegInit(VecInit(Seq.fill(slotCount)(0.U(5.W)))) - - val allocSlot = RegInit(0.U(slotWidth.W)) - val aReqSlot = RegInit(0.U(slotWidth.W)) - val bReqSlot = RegInit(0.U(slotWidth.W)) - val sendSlot = RegInit(0.U(slotWidth.W)) - - io.ctrl_ld_i.ready := !slotOccupied(allocSlot) - when(io.ctrl_ld_i.fire) { - val input = io.ctrl_ld_i.bits - slotReq(allocSlot).req_kind := input.req_kind - slotReq(allocSlot).k_tile_kind := input.k_tile_kind - slotReq(allocSlot).acc_slot := input.acc_slot - slotReq(allocSlot).valid_m := input.valid_m - slotReq(allocSlot).valid_n := input.valid_n - slotReq(allocSlot).valid_k := input.valid_k - slotReq(allocSlot).b_valid_n := input.b_valid_n - slotReq(allocSlot).b_valid_k := input.b_valid_k - slotReq(allocSlot).weight_generation := input.weight_generation - slotReq(allocSlot).op1_bank := input.op1_bank - slotReq(allocSlot).op1_group := input.op1_group - slotReq(allocSlot).op1_row_base := input.op1_row_base - slotReq(allocSlot).op2_bank := input.op2_bank - slotReq(allocSlot).op2_group := input.op2_group - slotReq(allocSlot).op2_row_base := input.op2_row_base - slotOccupied(allocSlot) := true.B - slotAReqCount(allocSlot) := 0.U - slotBReqCount(allocSlot) := 0.U - allocSlot := nextSlot(allocSlot) - - assert(input.valid_m >= 1.U && input.valid_m <= tile.U) - assert(input.valid_n >= 1.U && input.valid_n <= tile.U) - assert(input.valid_k >= 1.U && input.valid_k <= tile.U) - assert(!needOp2(input.req_kind) || input.b_valid_k >= 1.U) - } - - val sendReq = slotReq(sendSlot) - val sendNeedOp2 = needOp2(sendReq.req_kind) - val sendATotalRows = sendReq.valid_m - val sendBTotalRows = Mux(sendNeedOp2, sendReq.b_valid_k, 0.U(5.W)) - val sendARow = RegInit(0.U(5.W)) - val sendBRow = RegInit(0.U(5.W)) - val sendADone = RegInit(false.B) - val sendBDone = RegInit(false.B) - val sendActive = slotOccupied(sendSlot) - - io.load_ex_req_kind := sendReq.req_kind - io.load_ex_k_tile_kind := sendReq.k_tile_kind - io.load_ex_acc_slot := sendReq.acc_slot - io.load_ex_valid_m := sendReq.valid_m - io.load_ex_valid_n := sendReq.valid_n - io.load_ex_valid_k := sendReq.valid_k - io.load_ex_b_valid_n := sendReq.b_valid_n - io.load_ex_b_valid_k := sendReq.b_valid_k - io.load_ex_weight_generation := sendReq.weight_generation - - val aTagQ = Module(new Queue(new ReadTag, entries = tagQueueDepth)) - val bTagQ = Module(new Queue(new ReadTag, entries = tagQueueDepth)) - val aRowQ = Module(new Queue(new RowData, entries = rowFifoDepth)) - val bRowQ = Module(new Queue(new RowData, entries = rowFifoDepth)) - - val aReq = slotReq(aReqSlot) - val bReq = slotReq(bReqSlot) - val bNeed = needOp2(bReq.req_kind) - val aTotalRows = aReq.valid_m - val bTotalRows = Mux(bNeed, bReq.b_valid_k, 0.U(5.W)) - val aRequestsDone = slotAReqCount(aReqSlot) >= aTotalRows - val bRequestsDone = !bNeed || slotBReqCount(bReqSlot) >= bTotalRows - val aCanRequest = slotOccupied(aReqSlot) && !aRequestsDone - val bCanRequest = slotOccupied(bReqSlot) && !bRequestsDone - - for (i <- 0 until inBW) { - io.bankReadReq(i).valid := false.B - io.bankReadReq(i).bits.addr := 0.U - io.bankReadResp(i).ready := false.B - } - - io.op1_rd_bank_o := aReq.op1_bank - io.op1_rd_group_o := rowGroup(aReq.op1_group, aReq.op1_row_base, slotAReqCount(aReqSlot)) - io.op2_rd_bank_o := bReq.op2_bank - io.op2_rd_group_o := rowGroup(bReq.op2_group, bReq.op2_row_base, slotBReqCount(bReqSlot)) - - io.bankReadReq(0).valid := aCanRequest && aTagQ.io.enq.ready - io.bankReadReq(0).bits.addr := rowAddr(aReq.op1_row_base, slotAReqCount(aReqSlot)) - aTagQ.io.enq.valid := io.bankReadReq(0).fire - aTagQ.io.enq.bits.slot := aReqSlot - aTagQ.io.enq.bits.row := slotAReqCount(aReqSlot)(rowIndexWidth - 1, 0) - - io.bankReadReq(1).valid := bCanRequest && bTagQ.io.enq.ready - io.bankReadReq(1).bits.addr := rowAddr(bReq.op2_row_base, slotBReqCount(bReqSlot)) - bTagQ.io.enq.valid := io.bankReadReq(1).fire - bTagQ.io.enq.bits.slot := bReqSlot - bTagQ.io.enq.bits.row := slotBReqCount(bReqSlot)(rowIndexWidth - 1, 0) - - when(io.bankReadReq(0).fire) { - slotAReqCount(aReqSlot) := slotAReqCount(aReqSlot) + 1.U - when(slotAReqCount(aReqSlot) + 1.U >= aTotalRows) { - aReqSlot := nextSlot(aReqSlot) - } - }.elsewhen(slotOccupied(aReqSlot) && aRequestsDone) { - aReqSlot := nextSlot(aReqSlot) - } - - when(io.bankReadReq(1).fire) { - slotBReqCount(bReqSlot) := slotBReqCount(bReqSlot) + 1.U - when(slotBReqCount(bReqSlot) + 1.U >= bTotalRows) { - bReqSlot := nextSlot(bReqSlot) - } - }.elsewhen(slotOccupied(bReqSlot) && bRequestsDone) { - bReqSlot := nextSlot(bReqSlot) - } - - val aResponseReady = aTagQ.io.deq.valid && aRowQ.io.enq.ready - val bResponseReady = bTagQ.io.deq.valid && bRowQ.io.enq.ready - io.bankReadResp(0).ready := aResponseReady - io.bankReadResp(1).ready := bResponseReady - aTagQ.io.deq.ready := io.bankReadResp(0).fire - bTagQ.io.deq.ready := io.bankReadResp(1).fire - aRowQ.io.enq.valid := io.bankReadResp(0).fire - bRowQ.io.enq.valid := io.bankReadResp(1).fire - aRowQ.io.enq.bits.slot := aTagQ.io.deq.bits.slot - aRowQ.io.enq.bits.row := aTagQ.io.deq.bits.row - aRowQ.io.enq.bits.data := io.bankReadResp(0).bits.data - bRowQ.io.enq.bits.slot := bTagQ.io.deq.bits.slot - bRowQ.io.enq.bits.row := bTagQ.io.deq.bits.row - bRowQ.io.enq.bits.data := io.bankReadResp(1).bits.data - - val aHeadMatches = aRowQ.io.deq.valid && - aRowQ.io.deq.bits.slot === sendSlot && - aRowQ.io.deq.bits.row === sendARow(rowIndexWidth - 1, 0) - val bHeadMatches = bRowQ.io.deq.valid && - bRowQ.io.deq.bits.slot === sendSlot && - bRowQ.io.deq.bits.row === sendBRow(rowIndexWidth - 1, 0) - - io.load_ex_op1_o.valid := sendActive && !sendADone && aHeadMatches - io.load_ex_op1_o.bits := aRowQ.io.deq.bits.data - io.load_ex_op2_o.valid := sendActive && sendNeedOp2 && !sendBDone && bHeadMatches - io.load_ex_op2_o.bits := bRowQ.io.deq.bits.data - aRowQ.io.deq.ready := io.load_ex_op1_o.fire - bRowQ.io.deq.ready := io.load_ex_op2_o.fire - - val op1Fire = io.load_ex_op1_o.fire - val op2Fire = io.load_ex_op2_o.fire - val aFinishesThisCycle = op1Fire && sendARow + 1.U >= sendATotalRows - val bFinishesThisCycle = op2Fire && sendBRow + 1.U >= sendBTotalRows - val aDoneNext = sendADone || aFinishesThisCycle - val bDoneNext = !sendNeedOp2 || sendBDone || bFinishesThisCycle - - when(op1Fire) { - when(aFinishesThisCycle) { - sendADone := true.B - }.otherwise { - sendARow := sendARow + 1.U - } - } - when(op2Fire) { - when(bFinishesThisCycle) { - sendBDone := true.B - }.otherwise { - sendBRow := sendBRow + 1.U - } - } - - when(sendActive && aDoneNext && bDoneNext) { - slotOccupied(sendSlot) := false.B - sendSlot := nextSlot(sendSlot) - sendARow := 0.U - sendBRow := 0.U - sendADone := false.B - sendBDone := false.B - } - - for (slot <- 0 until slotCount) { - val slotNeedB = needOp2(slotReq(slot).req_kind) - when(slotOccupied(slot)) { - assert(slotAReqCount(slot) <= slotReq(slot).valid_m) - assert(slotBReqCount(slot) <= Mux(slotNeedB, slotReq(slot).b_valid_k, 0.U)) - } - } - when(io.bankReadResp(0).valid) { - assert(aTagQ.io.deq.valid, "SystolicArrayLoad: A response arrived without an outstanding request") - } - when(io.bankReadResp(1).valid) { - assert(bTagQ.io.deq.valid, "SystolicArrayLoad: B response arrived without an outstanding request") - } -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixStore.scala b/examples/balls/matrix/arch/src/main/scala/MatrixStore.scala deleted file mode 100644 index af7fd1497..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixStore.scala +++ /dev/null @@ -1,109 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import framework.top.GlobalConfig - -@instantiable -class SystolicArrayStore(val b: GlobalConfig) extends Module { - private val tile = SystolicArrayConst.Tile - private val resultRowBits = SystolicArrayConst.ResultRowBits - private val bufferRows = 2 - private val descriptorRows = 2 - private val descriptorCountWidth = log2Ceil(descriptorRows + 1) - - @public - val io = IO(new Bundle { - val ex_st_i = Flipped(Decoupled(new SystolicResultRow)) - - // Ctrl 主动按结果行顺序预发写回描述符；Store 只在本地地址 FIFO 有空位时接收。 - val store_ctrl_resp_i = Flipped(Decoupled(new SystolicStoreCtrlResp(b))) - val store_done_o = Output(Bool()) - - val wr_o = Decoupled(new SystolicStoreWriteReq(b)) - val wr_done_i = Input(Bool()) - }) - - val resultBuffer = Reg(Vec(bufferRows, UInt(resultRowBits.W))) - val readPtr = RegInit(0.U(log2Ceil(bufferRows).W)) - val writePtr = RegInit(0.U(log2Ceil(bufferRows).W)) - val bufferCount = RegInit(0.U(log2Ceil(bufferRows + 1).W)) - - // 四路 Unit 已在接收某行后持有该行的完整数据；Store 因而在 wr_o.fire 时 - // 即可释放本行，继续把下一行送入 Unit，而不是等待 bank 响应。 - val bufferEnq = io.ex_st_i.fire - - when(bufferEnq) { - resultBuffer(writePtr) := io.ex_st_i.bits.data - writePtr := Mux(writePtr === (bufferRows - 1).U, 0.U, writePtr + 1.U) - } - // 结果行和写回描述符分别以 FIFO 顺序保存。两者的队首总是同一逻辑行， - // 因而 resultBuffer 的物理环形回绕不会改变数据与地址的配对关系。 - val descriptorBuffer = - RegInit(VecInit(Seq.fill(descriptorRows)(0.U.asTypeOf(new SystolicStoreCtrlResp(b))))) - val descriptorReadPtr = RegInit(0.U(log2Ceil(descriptorRows).W)) - val descriptorWritePtr = RegInit(0.U(log2Ceil(descriptorRows).W)) - val descriptorCount = RegInit(0.U(descriptorCountWidth.W)) - val descriptorEnq = io.store_ctrl_resp_i.fire - val currentRow = resultBuffer(readPtr) - val currentDescriptor = descriptorBuffer(descriptorReadPtr) - val pairAvailable = bufferCount =/= 0.U && descriptorCount =/= 0.U - - when(descriptorEnq) { - descriptorBuffer(descriptorWritePtr) := io.store_ctrl_resp_i.bits - descriptorWritePtr := Mux( - descriptorWritePtr === (descriptorRows - 1).U, - 0.U, - descriptorWritePtr + 1.U) - } - io.wr_o.bits.rob_id := currentDescriptor.rob_id - io.wr_o.bits.wr_bank := currentDescriptor.wr_bank - io.wr_o.bits.wr_group_base := currentDescriptor.wr_group_base - io.wr_o.bits.wr_row_addr := currentDescriptor.wr_row_addr - io.wr_o.bits.valid_elems := currentDescriptor.row_valid_elems - // Unit 依据 valid_elems 生成 byte mask；被 mask 的尾列不会写入 bank，数据无需再 - // 在 Store 中重复清零。 - io.wr_o.bits.data := currentRow - - // Ctrl 只提供真实结果行的描述符；结果和描述符队首同时存在时即可交给 Unit。 - io.wr_o.valid := pairAvailable - val descriptorDeq = io.wr_o.fire - val bufferDeq = descriptorDeq - - io.store_done_o := io.wr_done_i - - when(bufferDeq) { - readPtr := Mux(readPtr === (bufferRows - 1).U, 0.U, readPtr + 1.U) - } - switch(Cat(bufferEnq, bufferDeq)) { - is("b10".U) { bufferCount := bufferCount + 1.U } - is("b01".U) { bufferCount := bufferCount - 1.U } - } - io.ex_st_i.ready := bufferCount < bufferRows.U || bufferDeq - - when(descriptorDeq) { - descriptorReadPtr := Mux( - descriptorReadPtr === (descriptorRows - 1).U, - 0.U, - descriptorReadPtr + 1.U) - } - switch(Cat(descriptorEnq, descriptorDeq)) { - is("b10".U) { descriptorCount := descriptorCount + 1.U } - is("b01".U) { descriptorCount := descriptorCount - 1.U } - } - - // 地址描述符与结果数据独立预取：即使 EX 尚未交付任何结果，只要 FIFO 有空位 - // 就接收 Ctrl 的下一行地址。descriptorDeq 允许同周期“消费一项、补入一项”。 - io.store_ctrl_resp_i.ready := descriptorCount < descriptorRows.U || descriptorDeq - - when(io.wr_o.valid) { - assert(io.wr_o.bits.valid_elems <= tile.U, "SystolicArrayStore: row valid element count exceeds TILE") - } - assert(bufferCount <= bufferRows.U, "SystolicArrayStore: result buffer overflow") - assert(descriptorCount <= descriptorRows.U, "SystolicArrayStore: descriptor buffer overflow") - when(descriptorDeq) { - assert(bufferCount =/= 0.U, "SystolicArrayStore: completed without a result row") - assert(descriptorCount =/= 0.U, "SystolicArrayStore: completed without a write descriptor") - } -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixTypes.scala b/examples/balls/matrix/arch/src/main/scala/MatrixTypes.scala deleted file mode 100644 index 9a484b161..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixTypes.scala +++ /dev/null @@ -1,140 +0,0 @@ -package framework.balldomain.prototype.systolicarray - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public} -import examples.balls.matrix.configs.MatrixBallParam -import framework.balldomain.rs.{BallRsComplete, BallRsIssue} -import framework.top.GlobalConfig - -object SystolicArrayConst { - // Ctrl、Load、EX、Store 共享的固定阵列参数：一个 tile 对应 16x16 的子矩阵。 - val Tile = 16 - // A/B 操作数按 int8 传输；C 的累加结果按 int32 传输。 - val OpElemBits = 8 - val AccElemBits = 32 - val OpRowBits = Tile * OpElemBits - val ResultRowBits = Tile * AccElemBits - // Store 将一行 16 个 int32 结果拆成最多 4 个 128-bit bank 写 beat。 - val StoreWritePorts = 4 - val StorePortElemCount = Tile / StoreWritePorts - // WS 模式一次保留一组 B 权重，供一批 M tile 复用；具体批大小由 ball TOML 配置。 - def wsReuseTiles(b: GlobalConfig): Int = { - val value = MatrixBallParam(b).wsReuseTiles - require(value >= 2, s"MatrixBall wsReuseTiles must be at least 2, got $value") - value - } -} - -object SystolicCtrlLoadReqKind { - // OS：为当前 tile 同时读取 A 与 B，EX 使用 K-tile 链完成累加。 - val READ_AB = 0.U(2.W) - // WS：B 已驻留在 PE 中，只读取当前 M tile 的 A。 - val READ_A_ONLY = 1.U(2.W) - // WS：读取 A，同时用 B 刷新 PE 权重；B 只发送当前 K tile 的真实行数。 - val READ_A_B_PE = 2.U(2.W) - // WS：当前 A 仍复用 PE 权重，同时预取下一套 B 到 B buffer。 - val READ_A_B_BUF = 3.U(2.W) -} - -object SystolicKTileKind { - // K 维只有一个 tile，无需与其他 K tile 串联累加。 - val DIRECT = 0.U(2.W) - // 多个 K tile 时，分别标记累加链的起始、中间与结束 tile。 - val FIRST = 1.U(2.W) - val MIDDLE = 2.U(2.W) - val LAST = 3.U(2.W) -} - -/** Ctrl 发给 Load 的单个 tile 读请求及其计算元数据。 */ -class SystolicCtrlLoadReq(b: GlobalConfig) extends Bundle { - // 决定读取 A/B 的组合方式，以及 EX 如何解释本请求。 - val req_kind = UInt(2.W) - // 描述当前 tile 在同一输出 tile 的 K 维累加链中的位置。 - val k_tile_kind = UInt(2.W) - // WS 模式的累加上下文槽；OS 不使用该字段，固定发送 0。 - val acc_slot = UInt(log2Ceil(SystolicArrayConst.wsReuseTiles(b)).W) - // 边缘 tile 的真实 M/N/K 尺寸，Load/EX 用它们完成补零和结果裁剪。 - val valid_m = UInt(5.W) - val valid_n = UInt(5.W) - val valid_k = UInt(5.W) - // B buffer 的有效 N/K；普通请求与 valid_n/valid_k 相同。 - val b_valid_n = UInt(5.W) - val b_valid_k = UInt(5.W) - // 当前 A tile 计算时应选用的 PE 权重代际。 - val weight_generation = Bool() - // A、B 各自的 bank、逻辑 group 与 group 内起始行地址。 - val op1_bank = UInt(log2Up(b.memDomain.bankNum).W) - val op1_group = UInt(log2Up(b.memDomain.bankNum).W) - val op1_row_base = UInt(log2Up(b.memDomain.bankEntries).W) - val op2_bank = UInt(log2Up(b.memDomain.bankNum).W) - val op2_group = UInt(log2Up(b.memDomain.bankNum).W) - val op2_row_base = UInt(log2Up(b.memDomain.bankEntries).W) -} - -/** EX 产出的一整行 C 结果；实际写 bank 前由 Store 拆分为若干 beat。 */ -class SystolicResultRow extends Bundle { - val data = UInt(SystolicArrayConst.ResultRowBits.W) -} - -/** Ctrl 对一次 Store 行请求的地址与有效元素数响应。 */ -class SystolicStoreCtrlResp(b: GlobalConfig) extends Bundle { - // 本行真实列数（1..16）；Store 据此产生对应数量的写 beat。 - val row_valid_elems = UInt(5.W) - // 这行结果所属的前端任务；Store 将其保留到实际 bank 写完成。 - val rob_id = UInt(log2Up(b.frontend.rob_entries).W) - val wr_bank = UInt(log2Up(b.memDomain.bankNum).W) - val wr_group_base = UInt(log2Up(b.memDomain.bankNum).W) - val wr_row_addr = UInt(log2Up(b.memDomain.bankEntries).W) -} - -/** Store 发给 bank 的单个写请求格式。Ctrl 不直接生成该请求，仅定义共享类型。 */ -class SystolicStoreWriteReq(b: GlobalConfig) extends Bundle { - val rob_id = UInt(log2Up(b.frontend.rob_entries).W) - val wr_bank = UInt(log2Up(b.memDomain.bankNum).W) - val wr_group_base = UInt(log2Up(b.memDomain.bankNum).W) - val wr_row_addr = UInt(log2Up(b.memDomain.bankEntries).W) - val valid_elems = UInt(5.W) - val data = UInt(SystolicArrayConst.ResultRowBits.W) -} - -/** - * 从一条发射命令中锁存的任务描述。 - * - * 命令可以在 Ctrl 执行期间继续进入 issueQ；这里的寄存器保证当前任务的 - * 维度、bank 地址和 ROB 信息在整个执行周期保持不变。 - */ -class SystolicTask(b: GlobalConfig) extends Bundle { - // mode=false 为 OS，mode=true 为 WS。 - val mode = Bool() - val m = UInt(12.W) - val n = UInt(12.W) - val k = UInt(12.W) - val op1_bank = UInt(log2Up(b.memDomain.bankNum).W) - val op2_bank = UInt(log2Up(b.memDomain.bankNum).W) - val wr_bank = UInt(log2Up(b.memDomain.bankNum).W) - val op1_base = UInt(log2Up(b.memDomain.bankEntries).W) - val op2_base = UInt(log2Up(b.memDomain.bankEntries).W) - val wr_base = UInt(log2Up(b.memDomain.bankEntries).W) - val rob_id = UInt(log2Up(b.frontend.rob_entries).W) - val is_sub = Bool() - val sub_rob_id = UInt(log2Up(b.frontend.sub_rob_depth * 4).W) -} - -class SystolicRetireMeta(b: GlobalConfig) extends Bundle { - val mode = Bool() - val m = UInt(12.W) - val n = UInt(12.W) - val rob_id = UInt(log2Up(b.frontend.rob_entries).W) - val is_sub = Bool() - val sub_rob_id = UInt(log2Up(b.frontend.sub_rob_depth * 4).W) -} - -class SystolicStoreMeta(b: GlobalConfig) extends Bundle { - val mode = Bool() - val m = UInt(12.W) - val n = UInt(12.W) - val wr_bank = UInt(log2Up(b.memDomain.bankNum).W) - val wr_base = UInt(log2Up(b.memDomain.bankEntries).W) - val rob_id = UInt(log2Up(b.frontend.rob_entries).W) -} diff --git a/examples/balls/matrix/arch/src/main/scala/MatrixUnit.scala b/examples/balls/matrix/arch/src/main/scala/MatrixUnit.scala deleted file mode 100644 index c176cd723..000000000 --- a/examples/balls/matrix/arch/src/main/scala/MatrixUnit.scala +++ /dev/null @@ -1,305 +0,0 @@ -package examples.balls.matrix - -import chisel3._ -import chisel3.util._ -import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} -import framework.balldomain.rs.{BallRsComplete, BallRsIssue} -import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} -import framework.top.GlobalConfig -import framework.balldomain.prototype.systolicarray.{ - SystolicArrayConst, - SystolicArrayCtrl, - SystolicArrayEX, - SystolicArrayLoad, - SystolicArrayStore, - SystolicStoreWriteReq -} - -@instantiable -class MatrixUnit(val b: GlobalConfig) extends Module { - private val accElemBits = SystolicArrayConst.AccElemBits - private val writePorts = SystolicArrayConst.StoreWritePorts - private val elemsPerPort = SystolicArrayConst.StorePortElemCount - private val resultRowBits = SystolicArrayConst.ResultRowBits - private val groupWidth = log2Up(b.memDomain.bankNum) - private val writeDataEntries = 2 - private val writeDataIdxWidth = log2Ceil(writeDataEntries) - private val writeDataCountWidth = log2Ceil(writeDataEntries + 1) - private val writeTrackEntries = 4 - private val writeTrackIdxWidth = log2Ceil(writeTrackEntries) - private val writeTrackCountWidth = log2Ceil(writeTrackEntries + 1) - - private def portMask(validElems: UInt): UInt = - VecInit((0 until writePorts).map(port => - (port * elemsPerPort).U < validElems)).asUInt - - private val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "MatrixBall") - .getOrElse(throw new IllegalArgumentException("MatrixBall not found in config")) - private val inBW = ballMapping.inBW - private val outBW = ballMapping.outBW - - require(inBW >= 2, "MatrixUnit requires at least two read ports for op1/op2") - require(outBW >= writePorts, "MatrixUnit requires four write ports for one 16xi32 C row per Store write") - require(b.memDomain.bankWidth == 128, "MatrixUnit expects 128-bit physical bank rows") - require(b.memDomain.bankMaskLen == 16, "MatrixUnit expects byte write masks on 128-bit bank rows") - require(resultRowBits == writePorts * b.memDomain.bankWidth) - - @public - val io = IO(new Bundle { - val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) - val cmdResp = Decoupled(new BallRsComplete(b)) - val bankRead = Vec(inBW, Flipped(new BankRead(b))) - val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) - val status = new BallStatus - }) - val resetActive = reset.asBool - - val ctrl: Instance[SystolicArrayCtrl] = Instantiate(new SystolicArrayCtrl(b)) - val load: Instance[SystolicArrayLoad] = Instantiate(new SystolicArrayLoad(b)) - val ex: Instance[SystolicArrayEX] = Instantiate(new SystolicArrayEX(b)) - val store: Instance[SystolicArrayStore] = Instantiate(new SystolicArrayStore(b)) - - ctrl.io.cmdReq.valid := io.cmdReq.valid && !resetActive - ctrl.io.cmdReq.bits := io.cmdReq.bits - io.cmdReq.ready := ctrl.io.cmdReq.ready && !resetActive - - io.cmdResp.valid := ctrl.io.cmdResp_o.valid && !resetActive - io.cmdResp.bits := ctrl.io.cmdResp_o.bits - ctrl.io.cmdResp_o.ready := io.cmdResp.ready && !resetActive - - load.io.ctrl_ld_i.valid := ctrl.io.ctrl_ld_o.valid - load.io.ctrl_ld_i.bits := ctrl.io.ctrl_ld_o.bits - ctrl.io.ctrl_ld_o.ready := load.io.ctrl_ld_i.ready - - ex.io.load_ex_req_kind := load.io.load_ex_req_kind - ex.io.load_ex_k_tile_kind := load.io.load_ex_k_tile_kind - ex.io.load_ex_acc_slot := load.io.load_ex_acc_slot - ex.io.load_ex_valid_m := load.io.load_ex_valid_m - ex.io.load_ex_valid_n := load.io.load_ex_valid_n - ex.io.load_ex_valid_k := load.io.load_ex_valid_k - ex.io.load_ex_b_valid_n := load.io.load_ex_b_valid_n - ex.io.load_ex_b_valid_k := load.io.load_ex_b_valid_k - ex.io.load_ex_weight_generation := load.io.load_ex_weight_generation - ex.io.load_ex_op1_i.valid := load.io.load_ex_op1_o.valid - ex.io.load_ex_op1_i.bits := load.io.load_ex_op1_o.bits - load.io.load_ex_op1_o.ready := ex.io.load_ex_op1_i.ready - ex.io.load_ex_op2_i.valid := load.io.load_ex_op2_o.valid - ex.io.load_ex_op2_i.bits := load.io.load_ex_op2_o.bits - load.io.load_ex_op2_o.ready := ex.io.load_ex_op2_i.ready - - store.io.ex_st_i.valid := ex.io.ex_st_o.valid - store.io.ex_st_i.bits := ex.io.ex_st_o.bits - ex.io.ex_st_o.ready := store.io.ex_st_i.ready - store.io.store_ctrl_resp_i.valid := ctrl.io.store_ctrl_resp_o.valid - store.io.store_ctrl_resp_i.bits := ctrl.io.store_ctrl_resp_o.bits - ctrl.io.store_ctrl_resp_o.ready := store.io.store_ctrl_resp_i.ready - ctrl.io.store_done_i := store.io.store_done_o - - for (i <- 0 until inBW) { - io.bankRead(i).rob_id := 0.U - io.bankRead(i).ball_id := 0.U - } - - io.bankRead(0).bank_id := load.io.op1_rd_bank_o - io.bankRead(0).group_id := load.io.op1_rd_group_o - io.bankRead(0).io.req.valid := load.io.bankReadReq(0).valid && !resetActive - io.bankRead(0).io.req.bits := load.io.bankReadReq(0).bits - load.io.bankReadReq(0).ready := io.bankRead(0).io.req.ready && !resetActive - load.io.bankReadResp(0).valid := io.bankRead(0).io.resp.valid - load.io.bankReadResp(0).bits := io.bankRead(0).io.resp.bits - io.bankRead(0).io.resp.ready := load.io.bankReadResp(0).ready && !resetActive - - io.bankRead(1).bank_id := load.io.op2_rd_bank_o - io.bankRead(1).group_id := load.io.op2_rd_group_o - io.bankRead(1).io.req.valid := load.io.bankReadReq(1).valid && !resetActive - io.bankRead(1).io.req.bits := load.io.bankReadReq(1).bits - load.io.bankReadReq(1).ready := io.bankRead(1).io.req.ready && !resetActive - load.io.bankReadResp(1).valid := io.bankRead(1).io.resp.valid - load.io.bankReadResp(1).bits := io.bankRead(1).io.resp.bits - io.bankRead(1).io.resp.ready := load.io.bankReadResp(1).ready && !resetActive - - for (i <- 2 until inBW) { - io.bankRead(i).bank_id := 0.U - io.bankRead(i).group_id := 0.U - io.bankRead(i).io.req.valid := false.B - io.bankRead(i).io.req.bits.addr := 0.U - io.bankRead(i).io.resp.ready := false.B - load.io.bankReadReq(i).ready := false.B - load.io.bankReadResp(i).valid := false.B - load.io.bankReadResp(i).bits := 0.U.asTypeOf(load.io.bankReadResp(i).bits) - } - - // 两项数据队列只保留尚未完全发出的 C 行；请求全部发出后立即释放 512-bit 数据。 - // 四项轻量 tracker 继续等待 bank response，从而保持原有的最大在途行数。 - val writeData = Reg(Vec(writeDataEntries, new SystolicStoreWriteReq(b))) - val writeDataTrack = Reg(Vec(writeDataEntries, UInt(writeTrackIdxWidth.W))) - val writeDataReadPtr = RegInit(0.U(writeDataIdxWidth.W)) - val writeDataWritePtr = RegInit(0.U(writeDataIdxWidth.W)) - val writeDataCount = RegInit(0.U(writeDataCountWidth.W)) - - val writeTrackValid = RegInit(VecInit(Seq.fill(writeTrackEntries)(false.B))) - val writeTrackRequiredMask = RegInit(VecInit(Seq.fill(writeTrackEntries)(0.U(writePorts.W)))) - val writeTrackIssuedMask = RegInit(VecInit(Seq.fill(writeTrackEntries)(0.U(writePorts.W)))) - val writeTrackAckMask = RegInit(VecInit(Seq.fill(writeTrackEntries)(0.U(writePorts.W)))) - val writeTrackReadPtr = RegInit(0.U(writeTrackIdxWidth.W)) - val writeTrackWritePtr = RegInit(0.U(writeTrackIdxWidth.W)) - val writeTrackCount = RegInit(0.U(writeTrackCountWidth.W)) - - val issueEntry = writeData(writeDataReadPtr) - val issueEntryValid = writeDataCount =/= 0.U - val issueTrackIdx = writeDataTrack(writeDataReadPtr) - val issuePortMask = writeTrackRequiredMask(issueTrackIdx) - - // bank response 不携带 Unit 自定义的行标签；对每个写口，从最早 entry 开始寻找 - // 该口尚未确认的请求，即可按该口的 FIFO 顺序正确归属 response。 - val responseTarget = Wire(Vec(writePorts, UInt(writeTrackIdxWidth.W))) - val responseTargetValid = Wire(Vec(writePorts, Bool())) - for (port <- 0 until writePorts) { - responseTarget(port) := writeTrackReadPtr - var targetFound = false.B - for (offset <- 0 until writeTrackEntries) { - val candidateIdx = (writeTrackReadPtr + offset.U)(writeTrackIdxWidth - 1, 0) - val candidateWaitsForPort = writeTrackValid(candidateIdx) && - writeTrackIssuedMask(candidateIdx)(port) && !writeTrackAckMask(candidateIdx)(port) - when(!targetFound && candidateWaitsForPort) { - responseTarget(port) := candidateIdx - } - targetFound = targetFound || candidateWaitsForPort - } - responseTargetValid(port) := targetFound - } - - for (i <- 0 until outBW) { - io.bankWrite(i).rob_id := 0.U - io.bankWrite(i).ball_id := 0.U - io.bankWrite(i).bank_id := 0.U - io.bankWrite(i).group_id := 0.U - io.bankWrite(i).io.req.valid := false.B - io.bankWrite(i).io.req.bits.addr := 0.U - io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(false.B)) - io.bankWrite(i).io.resp.ready := false.B - } - - for (port <- 0 until writePorts) { - val byteBase = port * elemsPerPort - val groupWithPort = issueEntry.wr_group_base +& port.U(groupWidth.W) - - io.bankWrite(port).rob_id := issueEntry.rob_id - io.bankWrite(port).bank_id := issueEntry.wr_bank - io.bankWrite(port).group_id := groupWithPort(groupWidth - 1, 0) - io.bankWrite(port).io.req.valid := issueEntryValid && issuePortMask(port) && - !writeTrackIssuedMask(issueTrackIdx)(port) && !resetActive - io.bankWrite(port).io.req.bits.addr := issueEntry.wr_row_addr - io.bankWrite(port).io.req.bits.data := issueEntry.data( - (port + 1) * b.memDomain.bankWidth - 1, - port * b.memDomain.bankWidth) - - val mask = Wire(Vec(b.memDomain.bankMaskLen, Bool())) - for (byte <- 0 until b.memDomain.bankMaskLen) { - val logicalElem = (byteBase + byte / (accElemBits / 8)).U - mask(byte) := logicalElem < issueEntry.valid_elems - } - io.bankWrite(port).io.req.bits.mask := mask - io.bankWrite(port).io.resp.ready := responseTargetValid(port) && !resetActive - } - - val issueFireMask = VecInit((0 until writePorts).map(port => - io.bankWrite(port).io.req.fire)).asUInt - val issueMaskAfterFire = writeTrackIssuedMask(issueTrackIdx) | issueFireMask - val issueEntryFinished = issueEntryValid && - (writeTrackIssuedMask(issueTrackIdx) & issuePortMask) =/= issuePortMask && - (issueMaskAfterFire & issuePortMask) === issuePortMask - - val responseFire = Wire(Vec(writePorts, Bool())) - for (port <- 0 until writePorts) { - responseFire(port) := responseTargetValid(port) && io.bankWrite(port).io.resp.valid && !resetActive - } - val responseAckMask = Wire(Vec(writeTrackEntries, UInt(writePorts.W))) - for (entry <- 0 until writeTrackEntries) { - responseAckMask(entry) := VecInit((0 until writePorts).map { port => - responseFire(port) && responseTarget(port) === entry.U(writeTrackIdxWidth.W) - }).asUInt - } - - // 只允许队首在其所需 port 都返回 response 后退休，因而 Store/Ctrl 看到的 - // wr_done_i 仍是一行一次且严格保序的完成脉冲。 - val headPortMask = writeTrackRequiredMask(writeTrackReadPtr) - val headAckAfterResponse = writeTrackAckMask(writeTrackReadPtr) | - responseAckMask(writeTrackReadPtr) - val headComplete = writeTrackValid(writeTrackReadPtr) && - (writeTrackIssuedMask(writeTrackReadPtr) & headPortMask) === headPortMask && - (headAckAfterResponse & headPortMask) === headPortMask - val canEnqueueData = writeDataCount < writeDataEntries.U || issueEntryFinished - val canEnqueueTrack = writeTrackCount < writeTrackEntries.U || headComplete - - store.io.wr_o.ready := canEnqueueData && canEnqueueTrack - store.io.wr_done_i := headComplete - val writeEnqueue = store.io.wr_o.fire - - for (entry <- 0 until writeTrackEntries) { - when(writeTrackValid(entry) && responseAckMask(entry).orR) { - writeTrackAckMask(entry) := writeTrackAckMask(entry) | responseAckMask(entry) - } - } - - when(issueFireMask.orR) { - writeTrackIssuedMask(issueTrackIdx) := issueMaskAfterFire - } - when(issueEntryFinished) { - writeDataReadPtr := writeDataReadPtr + 1.U - } - when(headComplete) { - writeTrackValid(writeTrackReadPtr) := false.B - writeTrackReadPtr := writeTrackReadPtr + 1.U - } - when(writeEnqueue) { - writeData(writeDataWritePtr) := store.io.wr_o.bits - writeDataTrack(writeDataWritePtr) := writeTrackWritePtr - writeDataWritePtr := writeDataWritePtr + 1.U - - writeTrackRequiredMask(writeTrackWritePtr) := portMask(store.io.wr_o.bits.valid_elems) - writeTrackIssuedMask(writeTrackWritePtr) := 0.U - writeTrackAckMask(writeTrackWritePtr) := 0.U - writeTrackValid(writeTrackWritePtr) := true.B - writeTrackWritePtr := writeTrackWritePtr + 1.U - } - switch(Cat(writeEnqueue, issueEntryFinished)) { - is("b10".U) { writeDataCount := writeDataCount + 1.U } - is("b01".U) { writeDataCount := writeDataCount - 1.U } - } - switch(Cat(writeEnqueue, headComplete)) { - is("b10".U) { writeTrackCount := writeTrackCount + 1.U } - is("b01".U) { writeTrackCount := writeTrackCount - 1.U } - } - - when(writeEnqueue) { - assert(portMask(store.io.wr_o.bits.valid_elems).orR, - "MatrixUnit: write row has no valid port") - } - when(issueEntryValid) { - assert(writeTrackValid(issueTrackIdx), - "MatrixUnit: write data references an invalid response tracker") - } - assert(writeDataCount <= writeDataEntries.U, - "MatrixUnit: write data queue overflow") - assert(writeTrackCount <= writeTrackEntries.U, - "MatrixUnit: write response tracker overflow") - - val hasInput = RegInit(false.B) - val hasOutput = RegInit(false.B) - - when(io.cmdReq.fire) { - hasInput := true.B - } - when(io.cmdResp.fire) { - hasOutput := false.B - hasInput := false.B - } - when(io.cmdResp.valid && !hasOutput) { - hasOutput := true.B - } - - io.status.idle := resetActive || (!hasInput && !hasOutput && !ctrl.io.busy_o) - io.status.running := !resetActive && ctrl.io.busy_o -} diff --git a/examples/balls/matrix/compiler/src/Backend/include/IntrinsicsRISCVMatrixBall.td b/examples/balls/matrix/compiler/src/Backend/include/IntrinsicsRISCVMatrixBall.td deleted file mode 100644 index 7adaa3682..000000000 --- a/examples/balls/matrix/compiler/src/Backend/include/IntrinsicsRISCVMatrixBall.td +++ /dev/null @@ -1,7 +0,0 @@ -//===- IntrinsicsRISCVMatrixBall.td - MatrixBall intrinsics --------------===// -// -// Stable LLVM intrinsic for MatrixBall is not wired yet. Pebble uses -// CustomIntrOp(funct7=65). Keep this file as a placeholder for a future -// int_riscv_bb_matrix addition to the RISC-V backend. -// -//===----------------------------------------------------------------------===// diff --git a/examples/balls/matrix/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/matrix/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp deleted file mode 100644 index 522508165..000000000 --- a/examples/balls/matrix/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp +++ /dev/null @@ -1,38 +0,0 @@ -//===- AssignPhysicalBankPatterns.cpp - Matrix bank assignment patterns ---===// - -#include "Conversion/LowerBuckyball/LowerBuckyball.h" - -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" - -using namespace mlir; -using namespace ::buddy::buckyball; - -namespace mlir::buddy { -void populateMatrixAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -} // namespace mlir::buddy - -namespace { - -class BankMatrixPattern : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(BankMatrixOp op, - PatternRewriter &rewriter) const override { - rewriter.create(op.getLoc(), op.getOp1Bank(), op.getOp2Bank(), - op.getWrBank(), op.getConfig()); - rewriter.replaceOp(op, op.getWrBank()); - return success(); - } -}; - -} // namespace - -void mlir::buddy::populateMatrixAssignPhysicalBankPatterns( - RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { - (void)state; - patterns.add(patterns.getContext()); -} diff --git a/examples/balls/matrix/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp b/examples/balls/matrix/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp deleted file mode 100644 index 07da00ddc..000000000 --- a/examples/balls/matrix/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp +++ /dev/null @@ -1,336 +0,0 @@ -//===- TileMatMulPatterns.cpp - tile.matmul -> buckyball.matrix_matmul ----===// -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// -//===----------------------------------------------------------------------===// - -#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" - -#include "mlir/Dialect/Arith/IR/Arith.h" -#include "mlir/Dialect/Linalg/IR/Linalg.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Dialect/SCF/IR/SCF.h" -#include "mlir/IR/BuiltinTypes.h" -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" -#include "Tile/TileOps.h" - -using namespace mlir; -using namespace ::buddy::buckyball; -using namespace ::buddy::tile; -using mlir::buddy::aMvinDepthLines; -using mlir::buddy::bMvinDepthLines; -using mlir::buddy::ceilDiv; -using mlir::buddy::cMvoutDepthLines; -using mlir::buddy::kMatmulTile; -using mlir::buddy::kMaxAccMvoutDepthLines; -using mlir::buddy::kMaxI8MvinDepthLines; - -namespace { - -class TileMatMulLowering : public OpRewritePattern { - size_t computeBankRows(size_t mTileLen, size_t nTileLen, - size_t kTileLen) const { - return mTileLen * kTileLen + kTileLen * nTileLen; - } - -public: - explicit TileMatMulLowering(MLIRContext *context, int64_t /*bankWidthBytes*/, - int64_t bankDepth, int64_t /*bankNum*/) - : OpRewritePattern(context), bankDepth(bankDepth) {} - - LogicalResult matchAndRewrite(TileMatMulOp tileMatMulOp, - PatternRewriter &rewriter) const override { - Location loc = tileMatMulOp.getLoc(); - - Value aMemArray = tileMatMulOp.getAMemArray(); - Value bMemArray = tileMatMulOp.getBMemArray(); - Value cMemArray = tileMatMulOp.getCMemArray(); - - auto aType = cast(aMemArray.getType()); - auto bType = cast(bMemArray.getType()); - auto cType = cast(cMemArray.getType()); - - auto aShape = aType.getShape(); - auto bShape = bType.getShape(); - auto cShape = cType.getShape(); - size_t M = aShape[aShape.size() - 2]; - size_t K = aShape[aShape.size() - 1]; - size_t N = bShape[bShape.size() - 1]; - - if (bShape[bShape.size() - 2] != (int64_t)K || - cShape[cShape.size() - 2] != (int64_t)M || - cShape[cShape.size() - 1] != (int64_t)N) - return tileMatMulOp.emitError("matmul input/output shapes mismatch"); - - const bool isWideFloat = - aType.getElementType().isF32() || aType.getElementType().isBF16(); - if (isWideFloat && (bType.getElementType() != aType.getElementType() || - cType.getElementType() != aType.getElementType())) - return tileMatMulOp.emitError( - "floating-point matmul requires matching A/B/C element types"); - - size_t M_pad = ceilDiv(M, 16) * 16; - size_t K_pad = ceilDiv(K, 16) * 16; - size_t N_pad = ceilDiv(N, 16) * 16; - bool needPadding = (M_pad != M) || (K_pad != K) || (N_pad != N); - - Value aMemArrayPadded = aMemArray; - Value bMemArrayPadded = bMemArray; - Value cMemArrayPadded = cMemArray; - - if (needPadding) { - auto elemType = aType.getElementType(); - - auto aPadType = - MemRefType::get({(int64_t)M_pad, (int64_t)K_pad}, elemType); - auto bPadType = - MemRefType::get({(int64_t)K_pad, (int64_t)N_pad}, elemType); - auto cPadType = - MemRefType::get({(int64_t)M_pad, (int64_t)N_pad}, elemType); - - aMemArrayPadded = rewriter.create(loc, aPadType); - bMemArrayPadded = rewriter.create(loc, bPadType); - cMemArrayPadded = rewriter.create(loc, cPadType); - - Value zero = rewriter.create( - loc, elemType, rewriter.getZeroAttr(elemType)); - rewriter.create(loc, zero, aMemArrayPadded); - rewriter.create(loc, zero, bMemArrayPadded); - rewriter.create(loc, zero, cMemArrayPadded); - - Value aView = rewriter.create( - loc, aMemArrayPadded, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(M), - rewriter.getIndexAttr(K)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, aMemArray, aView); - - Value bView = rewriter.create( - loc, bMemArrayPadded, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(K), - rewriter.getIndexAttr(N)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, bMemArray, bView); - } - - size_t M_tiling = needPadding ? M_pad : M; - size_t K_tiling = needPadding ? K_pad : K; - size_t N_tiling = needPadding ? N_pad : N; - - const size_t mMeta = kMatmulTile; - const size_t nMeta = kMatmulTile; - const size_t kMeta = kMatmulTile; - - const size_t mPad = ceilDiv(M_tiling, mMeta) * mMeta; - const size_t nPad = ceilDiv(N_tiling, nMeta) * nMeta; - const size_t kPad = ceilDiv(K_tiling, kMeta) * kMeta; - - size_t mTileLen = 1, nTileLen = 1, kTileLen = 1; - - for (size_t cand = kTileLen + 1; cand * kMeta <= kPad; ++cand) { - size_t candSize = cand * kMeta; - if (computeBankRows(1, 1, cand) > (size_t)bankDepth || - aMvinDepthLines(mMeta, candSize) > kMaxI8MvinDepthLines || - bMvinDepthLines(candSize, nMeta) > kMaxI8MvinDepthLines) - break; - if (kPad % candSize == 0) - kTileLen = cand; - } - - const size_t kTileSize = kTileLen * kMeta; - - for (size_t cand = nTileLen + 1; !isWideFloat && cand * nMeta <= nPad; - ++cand) { - size_t candSize = cand * nMeta; - if (computeBankRows(1, cand, kTileLen) > (size_t)bankDepth || - cMvoutDepthLines(mMeta, candSize) > kMaxAccMvoutDepthLines || - bMvinDepthLines(kTileSize, candSize) > kMaxI8MvinDepthLines) - break; - if (nPad % candSize == 0) - nTileLen = cand; - } - - for (size_t cand = mTileLen + 1; !isWideFloat && cand * mMeta <= mPad; - ++cand) { - size_t candSize = cand * mMeta; - if (computeBankRows(cand, nTileLen, kTileLen) > (size_t)bankDepth || - cMvoutDepthLines(candSize, nTileLen * nMeta) > - kMaxAccMvoutDepthLines || - aMvinDepthLines(candSize, kTileSize) > kMaxI8MvinDepthLines) - break; - if (mPad % candSize == 0) - mTileLen = cand; - } - - const size_t mTileSize = mTileLen * mMeta; - const size_t nTileSize = nTileLen * nMeta; - const size_t kTileNum = ceilDiv(kPad, kTileSize); - - if (mPad % mTileSize != 0 || nPad % nTileSize != 0 || - kPad % kTileSize != 0) { - return tileMatMulOp.emitError() - << "padded dims (m=" << mPad << ", n=" << nPad << ", k=" << kPad - << ") must be multiples of tile sizes (m=" << mTileSize - << ", n=" << nTileSize << ", k=" << kTileSize - << "); partial tiles not yet supported"; - } - - OpBuilder::InsertionGuard guard(rewriter); - - Value zeroIdx = rewriter.create(loc, 0); - Value mStepVal = rewriter.create(loc, mTileSize); - Value nStepVal = rewriter.create(loc, nTileSize); - Value kStepVal = rewriter.create(loc, kTileSize); - Value mUpperVal = rewriter.create(loc, mPad); - Value nUpperVal = rewriter.create(loc, nPad); - Value kUpperVal = rewriter.create(loc, kPad); - Operation *outerLoop = nullptr; - - if (kTileNum == 1) { - auto kLoop = - rewriter.create(loc, zeroIdx, kUpperVal, kStepVal); - outerLoop = kLoop; - rewriter.setInsertionPointToStart(kLoop.getBody()); - Value kIv = kLoop.getInductionVar(); - - auto mLoop = - rewriter.create(loc, zeroIdx, mUpperVal, mStepVal); - rewriter.setInsertionPointToStart(mLoop.getBody()); - Value mIv = mLoop.getInductionVar(); - - auto nLoop = - rewriter.create(loc, zeroIdx, nUpperVal, nStepVal); - rewriter.setInsertionPointToStart(nLoop.getBody()); - Value nIv = nLoop.getInductionVar(); - - Value aTile = rewriter.create( - loc, aMemArrayPadded, SmallVector{mIv, kIv}, - SmallVector{rewriter.getIndexAttr(mTileSize), - rewriter.getIndexAttr(kTileSize)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value bTile = rewriter.create( - loc, bMemArrayPadded, SmallVector{kIv, nIv}, - SmallVector{rewriter.getIndexAttr(kTileSize), - rewriter.getIndexAttr(nTileSize)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value cTile = rewriter.create( - loc, cMemArrayPadded, SmallVector{mIv, nIv}, - SmallVector{rewriter.getIndexAttr(mTileSize), - rewriter.getIndexAttr(nTileSize)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - - // matrix_matmul accumulates into C; clear tile first. - auto cElemType = cType.getElementType(); - Value zero = rewriter.create( - loc, cElemType, rewriter.getZeroAttr(cElemType)); - rewriter.create(loc, zero, cTile); - rewriter.create(loc, aTile, bTile, cTile); - } else { - auto mLoop = - rewriter.create(loc, zeroIdx, mUpperVal, mStepVal); - outerLoop = mLoop; - rewriter.setInsertionPointToStart(mLoop.getBody()); - Value mIv = mLoop.getInductionVar(); - - auto nLoop = - rewriter.create(loc, zeroIdx, nUpperVal, nStepVal); - rewriter.setInsertionPointToStart(nLoop.getBody()); - Value nIv = nLoop.getInductionVar(); - - Value cTile = rewriter.create( - loc, cMemArrayPadded, SmallVector{mIv, nIv}, - SmallVector{rewriter.getIndexAttr(mTileSize), - rewriter.getIndexAttr(nTileSize)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - - auto cElemType = cType.getElementType(); - if (!isa(cElemType) && !isa(cElemType)) - return tileMatMulOp.emitError( - "multi-K tile.matmul accumulate requires integer or float C " - "element type"); - - Value zero = rewriter.create( - loc, cElemType, rewriter.getZeroAttr(cElemType)); - rewriter.create(loc, zero, cTile); - - auto kLoop = - rewriter.create(loc, zeroIdx, kUpperVal, kStepVal); - rewriter.setInsertionPointToStart(kLoop.getBody()); - Value kIv = kLoop.getInductionVar(); - - Value aTile = rewriter.create( - loc, aMemArrayPadded, SmallVector{mIv, kIv}, - SmallVector{rewriter.getIndexAttr(mTileSize), - rewriter.getIndexAttr(kTileSize)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value bTile = rewriter.create( - loc, bMemArrayPadded, SmallVector{kIv, nIv}, - SmallVector{rewriter.getIndexAttr(kTileSize), - rewriter.getIndexAttr(nTileSize)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - - // Matrix Ball accumulates i8 products into i32 C across K tiles. - rewriter.create(loc, aTile, bTile, cTile); - - rewriter.setInsertionPointAfter(kLoop); - } - - rewriter.setInsertionPointAfter(outerLoop); - - if (needPadding) { - Value cView = rewriter.create( - loc, cMemArrayPadded, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(M), - rewriter.getIndexAttr(N)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, cView, cMemArray); - - rewriter.create(loc, aMemArrayPadded); - rewriter.create(loc, bMemArrayPadded); - rewriter.create(loc, cMemArrayPadded); - } - - rewriter.eraseOp(tileMatMulOp); - return success(); - } - -private: - int64_t bankDepth; -}; - -} // namespace - -void mlir::buddy::populateMatrixTileMatMulPatterns(RewritePatternSet &patterns, - int64_t bankWidthBytes, - int64_t bankDepth, - int64_t bankNum) { - patterns.add(patterns.getContext(), bankWidthBytes, - bankDepth, bankNum); -} diff --git a/examples/balls/matrix/compiler/src/Dialect/Buckyball/MatrixBall.td b/examples/balls/matrix/compiler/src/Dialect/Buckyball/MatrixBall.td deleted file mode 100644 index 75490cb1c..000000000 --- a/examples/balls/matrix/compiler/src/Dialect/Buckyball/MatrixBall.td +++ /dev/null @@ -1,47 +0,0 @@ -//===-- MatrixBall.td - MatrixBall ops -------*- tablegen -*-===// - -#ifndef MATRIXARRAYBALL_OPS -#define MATRIXARRAYBALL_OPS - -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" - -def MatrixMatmulOp : Buckyball_Op<"matrix_matmul"> { - let summary = "Logical matmul lowered to MatrixBall (funct7=65)"; - let description = [{ - Performs C[M,N] = A[M,K] x B[K,N] with int8 inputs / int32 output. - Memrefs must already match the MatrixBall bank tile layout for the - given shape (single 16x16 tile for the default tile.matmul path). - `ws = true` selects weight-stationary mode. - }]; - let arguments = (ins MemRefRankOf<[AnyType], [2]>:$aMemArray, - MemRefRankOf<[AnyType], [2]>:$bMemArray, - MemRefRankOf<[AnyType], [2]>:$cMemArray, - DefaultValuedAttr:$ws); - let assemblyFormat = [{$aMemArray $bMemArray $cMemArray attr-dict `:` - type($aMemArray) type($bMemArray) type($cMemArray)}]; -} - -def MatrixOp : Buckyball_Op<"matrix"> { - let summary = "MatrixBall compute (funct7=65)"; - let description = [{ - ISA: bb_matrix_mnk_mode(op1, op2, wr, m, n, k, mode). - `config` is the full rs2 encoding (M/N/K/mode). - }]; - let arguments = (ins I64:$op1BankId, I64:$op2BankId, - I64:$resultBankId, I64:$config); - let assemblyFormat = "$op1BankId `,` $op2BankId `,` $resultBankId `,` " - "$config attr-dict `:` " - "type($op1BankId)"; -} - -def BankMatrixOp : Buckyball_Op<"bank_matrix"> { - let summary = "MatrixBall compute with virtual bank handles"; - let arguments = (ins I64:$op1Bank, I64:$op2Bank, I64:$wrBank, I64:$config); - let results = (outs I64:$wrBankOut); - let assemblyFormat = - "$op1Bank $op2Bank $wrBank $config attr-dict `:` " - "type($op1Bank) type($op2Bank) type($wrBank) type($config)"; -} - -#endif // MATRIXARRAYBALL_OPS diff --git a/examples/balls/matrix/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/matrix/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp deleted file mode 100644 index 5d77ef9f1..000000000 --- a/examples/balls/matrix/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ /dev/null @@ -1,187 +0,0 @@ -//===- LegalizeForLLVMExport.cpp - MatrixBall LLVM lowering --------------===// - -#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" -#include "mlir/Conversion/LLVMCommon/Pattern.h" -#include "mlir/Dialect/LLVMIR/LLVMDialect.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/IR/BuiltinTypes.h" -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" -#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" - -using namespace mlir; -using namespace buddy::buckyball; -using namespace buddy::buckyball::legalize; - -namespace { - -uint64_t mnkConfig(uint64_t m, uint64_t n, uint64_t k, uint64_t mode) { - return fieldBits(m, 0, 11) | fieldBits(n, 12, 23) | fieldBits(k, 24, 35) | - fieldBits(mode, 36, 36); -} - -uint64_t matrixRs1(uint64_t op1, uint64_t op2, uint64_t wr) { - return fieldBits(op1, 0, 9) | fieldBits(op2, 10, 19) | fieldBits(wr, 20, 29); -} - -struct MatrixMatmulLowering : public ConvertOpToLLVMPattern { - MatrixMatmulLowering(LLVMTypeConverter &converter, bool /*stable*/, - bool rushB) - : ConvertOpToLLVMPattern(converter), rushB(rushB) {} - - LogicalResult - matchAndRewrite(MatrixMatmulOp op, OpAdaptor, - ConversionPatternRewriter &rewriter) const override { - Location loc = op.getLoc(); - Value aMem = op.getAMemArray(); - Value bMem = op.getBMemArray(); - Value cMem = op.getCMemArray(); - - auto aTy = dyn_cast(aMem.getType()); - auto bTy = dyn_cast(bMem.getType()); - auto cTy = dyn_cast(cMem.getType()); - if (!aTy || !bTy || !cTy || !aTy.hasStaticShape() || - !bTy.hasStaticShape() || !cTy.hasStaticShape()) - return rewriter.notifyMatchFailure( - op, "buckyball.matrix_matmul requires static memref shapes"); - - uint64_t m = aTy.getShape()[0]; - uint64_t k = aTy.getShape()[1]; - uint64_t kb = bTy.getShape()[0]; - uint64_t n = bTy.getShape()[1]; - if (k != kb || cTy.getShape()[0] != (int64_t)m || - cTy.getShape()[1] != (int64_t)n) - return rewriter.notifyMatchFailure(op, "matmul shapes mismatch"); - if (m == 0 || n == 0 || k == 0 || m > 16 || n > 16 || k > 16) - return rewriter.notifyMatchFailure( - op, "matrix_matmul legalize supports one tile with M,N,K in 1..16"); - if (!aTy.getElementType().isInteger(8) || - !bTy.getElementType().isInteger(8) || - !cTy.getElementType().isInteger(32)) - return rewriter.notifyMatchFailure( - op, "Matrix Ball supports only i8 A/B with i32 C"); - - const uint64_t aBank = 0; - const uint64_t bBank = 1; - const uint64_t cBank = 2; - uint64_t depthA = 16; - uint64_t depthB = 16; - uint64_t depthC = m; - - emitMset(rewriter, loc, aBank, 1, 1, 1); - emitMset(rewriter, loc, bBank, 1, 1, 1); - emitMset(rewriter, loc, cBank, 1, 4, 1); - - Value aPtr = extractPtr(rewriter, loc, aMem); - Value bPtr = extractPtr(rewriter, loc, bMem); - Value cPtr = extractPtr(rewriter, loc, cMem); - - Value rs1A = packRs1BankIter(rewriter, loc, cstI64(rewriter, loc, aBank), - cstI64(rewriter, loc, depthA)); - Value rs2A = - packRs2MemStride(rewriter, loc, aPtr, cstI64(rewriter, loc, 1)); - if (rushB) { - Type ptrType = LLVM::LLVMPointerType::get(rewriter.getContext()); - rewriter.create( - loc, rs1A, rs2A, - LLVM::IntToPtrOp::create(rewriter, loc, ptrType, aPtr)); - } else { - rewriter.create(loc, rs1A, rs2A); - } - - Value rs1B = packRs1BankIter(rewriter, loc, cstI64(rewriter, loc, bBank), - cstI64(rewriter, loc, depthB)); - Value rs2B = - packRs2MemStride(rewriter, loc, bPtr, cstI64(rewriter, loc, 1)); - if (rushB) { - Type ptrType = LLVM::LLVMPointerType::get(rewriter.getContext()); - rewriter.create( - loc, rs1B, rs2B, - LLVM::IntToPtrOp::create(rewriter, loc, ptrType, bPtr)); - } else { - rewriter.create(loc, rs1B, rs2B); - } - - uint64_t mode = op.getWs() ? 1ull : 0ull; - rewriter.create( - loc, cstI64(rewriter, loc, matrixRs1(aBank, bBank, cBank)), - cstI64(rewriter, loc, mnkConfig(m, n, k, mode)), - rewriter.getI32IntegerAttr(65)); - - Value rs1C = packRs1BankIter(rewriter, loc, cstI64(rewriter, loc, cBank), - cstI64(rewriter, loc, depthC)); - Value rs2C = - packRs2MemStride(rewriter, loc, cPtr, cstI64(rewriter, loc, 1)); - if (rushB) { - Type ptrType = LLVM::LLVMPointerType::get(rewriter.getContext()); - rewriter.create( - loc, rs1C, rs2C, - LLVM::IntToPtrOp::create(rewriter, loc, ptrType, cPtr)); - } else { - rewriter.create(loc, rs1C, rs2C); - } - - Value zero = cstI64(rewriter, loc, 0); - rewriter.create(loc, zero, zero); - emitMset(rewriter, loc, aBank, 0, 0, 0); - emitMset(rewriter, loc, bBank, 0, 0, 0); - emitMset(rewriter, loc, cBank, 0, 0, 0); - - rewriter.eraseOp(op); - return success(); - } - -private: - bool rushB; -}; - -struct MatrixLowering : public ConvertOpToLLVMPattern { - MatrixLowering(LLVMTypeConverter &converter, bool stable) - : ConvertOpToLLVMPattern(converter), stable(stable) {} - - LogicalResult - matchAndRewrite(MatrixOp op, OpAdaptor adaptor, - ConversionPatternRewriter &rewriter) const override { - if (stable) - return rewriter.notifyMatchFailure( - op, "MatrixBall stable intrinsic is not available; use custom path"); - - Location loc = op.getLoc(); - Value rs1 = packRs1BanksIter( - rewriter, loc, adaptor.getOp1BankId(), adaptor.getOp2BankId(), - adaptor.getResultBankId(), cstI64(rewriter, loc, 0)); - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getConfig(), - rewriter.getI32IntegerAttr(65)); - return success(); - } - -private: - bool stable = false; -}; - -} // namespace - -namespace mlir::buddy::buckyball { -void populateMatrixMatmulLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, - bool rushB) { - patterns.add(converter, stable, rushB); -} - -void configureMatrixMatmulLegalizeForExportTarget(LLVMConversionTarget &target, - bool /*stable*/) { - target.addIllegalOp(); -} - -void populateMatrixLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable) { - patterns.add(converter, stable); -} - -void configureMatrixLegalizeForExportTarget(LLVMConversionTarget &target, - bool /*stable*/) { - target.addIllegalOp(); -} -} // namespace mlir::buddy::buckyball diff --git a/examples/balls/matrix/emu/src/65_matrix.rs b/examples/balls/matrix/emu/src/65_matrix.rs deleted file mode 100644 index e6110103a..000000000 --- a/examples/balls/matrix/emu/src/65_matrix.rs +++ /dev/null @@ -1,215 +0,0 @@ -use super::super::bank::{bank_lines, bank_num, bank_row_bytes}; -use super::bank_matrix::read_i8_k_rows; -use super::decode::{pbank, pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; - -const TILE: usize = 16; - -pub struct Matrix; - -fn ceil_div(x: usize, d: usize) -> usize { - (x + d - 1) / d -} - -fn i8_at(bank: &[u8], row: usize, lane: usize) -> i8 { - let row_bytes = bank_row_bytes(); - if lane >= row_bytes { - panic!("matrix: i8 lane {lane} out of row_bytes={row_bytes}"); - } - let off = row * row_bytes + lane; - if off >= bank.len() { - panic!("matrix: i8 read OOB row={row} lane={lane}"); - } - bank[off] as i8 -} - -fn write_i32(bank: &mut [u8], row: usize, lane: usize, value: i32) { - let row_bytes = bank_row_bytes(); - let off = row * row_bytes + lane * 4; - if off + 4 > bank.len() { - panic!("matrix: i32 write OOB row={row} lane={lane}"); - } - bank[off..off + 4].copy_from_slice(&value.to_le_bytes()); -} - -fn c_block(row: usize, n_tile: usize, m: usize, n: usize) -> usize { - let n_tiles = ceil_div(n, TILE); - let mt = row / TILE; - let mr = row % TILE; - let mut block = 0usize; - for t in 0..mt { - let rows = (m - t * TILE).min(TILE); - block += rows * n_tiles; - } - let rows = (m - mt * TILE).min(TILE); - block + n_tile * rows + mr -} - -impl Instruction for Matrix { - const FUNCT: u32 = 65; - - fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - let op1 = rs1_b0(xs1); - let op2 = rs1_b1(xs1); - let wr = rs1_b2(xs1); - let encoded_m = (xs2 & 0xfff) as usize; - let encoded_n = ((xs2 >> 12) & 0xfff) as usize; - let encoded_k = ((xs2 >> 24) & 0xfff) as usize; - let legacy = encoded_m == 0 && encoded_n == 0 && encoded_k == 0; - let iter = rs1_iter(xs1) as usize; - let (m, n, k) = if legacy { - (iter, iter, iter) - } else { - (encoded_m, encoded_n, encoded_k) - }; - let op1_base = if legacy { - 0 - } else { - ((xs1 >> 30) & 0x7f) as usize - }; - let op2_base = if legacy { - 0 - } else { - ((xs1 >> 37) & 0x7f) as usize - }; - let wr_base = if legacy { - 0 - } else { - ((xs1 >> 44) & 0x7f) as usize - }; - if !legacy && xs1 >> 51 != 0 { - panic!("matrix: rs1[63:51] must be 0"); - } - - let mode = (xs2 >> 36) & 1; - if xs2 >> 37 != 0 { - panic!("matrix: rs2[63:37] must be 0"); - } - let _ws = mode == 1; - - if op1 >= bank_num() as u64 || op2 >= bank_num() as u64 || wr >= bank_num() as u64 { - panic!("matrix: invalid bank_id"); - } - if !ctx.cfgs[op1 as usize].allocated - || !ctx.cfgs[op2 as usize].allocated - || !ctx.cfgs[wr as usize].allocated - { - panic!("matrix: bank not allocated"); - } - if ctx.cfgs[op1 as usize].cols != 1 || ctx.cfgs[op2 as usize].cols != 1 { - panic!( - "matrix: op banks must have cols=1 (op1={} op2={})", - ctx.cfgs[op1 as usize].cols, ctx.cfgs[op2 as usize].cols - ); - } - if ctx.cfgs[wr as usize].cols != 4 { - panic!( - "matrix: wr bank must be acc cols=4, got {}", - ctx.cfgs[wr as usize].cols - ); - } - if m == 0 || n == 0 || k == 0 { - panic!("matrix: M/N/K must be non-zero"); - } - - let k_tiles = ceil_div(k, TILE); - let n_tiles = ceil_div(n, TILE); - let a_rows = ceil_div(m, TILE) * k_tiles * TILE; - let b_rows = n_tiles * k_tiles * TILE; - let c_blocks = m * n_tiles; - let lines = bank_lines(); - - if op1_base + a_rows > lines { - panic!("matrix: A range OOB base={op1_base} rows={a_rows} lines={lines}"); - } - if op2_base + b_rows > lines { - panic!("matrix: B range OOB base={op2_base} rows={b_rows} lines={lines}"); - } - if wr_base + c_blocks > lines { - panic!("matrix: C range OOB base={wr_base} blocks={c_blocks} lines={lines}"); - } - - let p1 = pbank(ctx.bank_map, op1); - let p2 = pbank(ctx.bank_map, op2); - let pw: Vec<_> = (0..4).map(|g| pbank_group(ctx.bank_map, wr, g)).collect(); - if legacy { - // The former BFP API encoded a square dimension in BB_ITER and - // used the original row-major bank layout. - let a = read_i8_k_rows(&ctx.banks, p1, m, k); - let b = read_i8_k_rows(&ctx.banks, p2, k, n); - for i in 0..m { - for j in 0..n { - let mut acc = 0i32; - for kk in 0..k { - acc += a[i][kk] as i32 * b[kk][j] as i32; - } - let group = j / 4; - let lane = j % 4; - let off = i * 16 + lane * 4; - ctx.banks[pw[group]][off..off + 4].copy_from_slice(&acc.to_le_bytes()); - } - } - return 0; - } - - let mut c = vec![vec![0i32; n]; m]; - for i in 0..m { - for j in 0..n { - let mut acc = 0i32; - for kk in 0..k { - let mt = i / TILE; - let mr = i % TILE; - let kt = kk / TILE; - let kl = kk % TILE; - let a_row = op1_base + (mt * k_tiles + kt) * TILE + mr; - let a = i8_at(&ctx.banks[p1], a_row, kl); - - let nt = j / TILE; - let nl = j % TILE; - let bkt = kk / TILE; - let bkr = kk % TILE; - let b_row = op2_base + (nt * k_tiles + bkt) * TILE + bkr; - let b = i8_at(&ctx.banks[p2], b_row, nl); - - acc += a as i32 * b as i32; - } - c[i][j] = acc; - } - } - - for i in 0..m { - for j in 0..n { - let nti = j / TILE; - let lane = j % TILE; - let block = wr_base + c_block(i, nti, m, n); - let group = lane / 4; - let sub = lane % 4; - write_i32(&mut ctx.banks[pw[group]], block, sub, c[i][j]); - } - } - 0 - } - - fn latency(xs1: u64, xs2: u64) -> u64 { - let encoded_m = xs2 & 0xfff; - let encoded_n = (xs2 >> 12) & 0xfff; - let encoded_k = (xs2 >> 24) & 0xfff; - let legacy = encoded_m == 0 && encoded_n == 0 && encoded_k == 0; - let (m, n, k) = if legacy { - let iter = rs1_iter(xs1).clamp(1, 64); - (iter, iter.min(16), iter) - } else { - (encoded_m.max(1), encoded_n.max(1), encoded_k.max(1)) - }; - let mode = (xs2 >> 36) & 1; - if legacy { - return m.saturating_mul(n).saturating_mul(k) / 4 + m.saturating_mul(n); - } - let body = m.saturating_mul(n).saturating_mul(k) / (TILE as u64); - if mode == 1 { - body + m.saturating_mul(n) - } else { - body + m.saturating_mul(n) / 2 - } - } -} diff --git a/examples/balls/matrix/emu/src/lib.rs b/examples/balls/matrix/emu/src/lib.rs deleted file mode 100644 index cf9d709a9..000000000 --- a/examples/balls/matrix/emu/src/lib.rs +++ /dev/null @@ -1,38 +0,0 @@ -pub(crate) use crate::inst::{bank_matrix, decode, instruction}; - -use crate::inst::instruction::{ExecContext, Instruction}; - -#[path = "65_matrix.rs"] -mod f65_matrix; - -const BALL_CLASS: &str = "examples.balls.matrix.MatrixBall"; - -pub fn execute_known( - ball_class: &str, - funct: u32, - xs1: u64, - xs2: u64, - ctx: &mut ExecContext, -) -> Option { - if ball_class != BALL_CLASS { - return None; - } - match funct { - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, - } -} - -pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> Option { - if ball_class != BALL_CLASS { - return None; - } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, - } -} diff --git a/examples/balls/matrix/verify/README.md b/examples/balls/matrix/verify/README.md deleted file mode 100644 index fe01b2a81..000000000 --- a/examples/balls/matrix/verify/README.md +++ /dev/null @@ -1,88 +0,0 @@ -# MatrixBall UVM Verification - -Verifies the generated `MatrixBall` module using the shared Blink UVM framework. - -## Structure - -- `../../../../verify/uvm/src/bb_uvm_pkg.sv`: common Blink UVM agents and base env -- `src/common/matrix_defs.svh`: DPI imports, `matrix_require_bid`, timeouts -- `src/common/matrix_items.svh`: `matrix_cmd_item` with `load_rust_case` -- `src/seq/matrix_sequences.svh`: one-case sequence -- `src/cov/matrix_cov.svh`: mode {OS,WS} and M/N/K shape bins -- `src/env/matrix_scoreboard.svh`: preload A/B, compare writes vs DPI expected list -- `src/env/matrix_env.svh`: extends `bb_blink_env#(2,4)` -- `src/tests/matrix_test.svh`: directed 0..2, random 3..22 -- `src/pkg/matrix_pkg.sv`: package entry -- `src/tb_top.sv`: `bb_blink_if#(2,4)` + `MatrixBall` (2 read, 4 write ports) -- `filelists/matrix_ball.f`: VCS filelist - -## Test plan - -`+UVM_TESTNAME=matrix_ball_test` runs: - -- case 0: OS 4x4x4 (banks 0,1,2) -- case 1: WS 4x4x4 -- case 2: OS 16x16x16 -- cases 3..22: random M/N/K in {1,2,4,8,16}, mode 0/1, distinct banks 0..7 - -Each case: reset, preload A/B into mem model, drive one command, wait for -`scb.done()` or timeout at 100000 cycles. Scoreboard compares writes -(group/addr/data/mask) in DPI emission order. - -## BID (required) - -`+BID=` is mandatory. Missing it fatals at test start. No default bid. - -| Config | Plusarg | -|--------|---------| -| `sims.verilator.BuckyballPebbleVerilatorConfig` | `+BID=1` | -| `sims.verilator.BuckyballToyVerilatorConfig` | `+BID=4` | - -```console -./simv +UVM_TESTNAME=matrix_ball_test +BID=4 -``` - -## Build - -```console -nix develop ../../../../verify -cargo build --manifest-path casegen/Cargo.toml -``` - -Compile from this directory: - -```console -vcs -full64 -sverilog -timescale=1ns/1ps \ - $VCS_UVM_ARGS \ - -sv_lib casegen/target/debug/libmatrix_casegen \ - -f filelists/matrix_ball.f -``` - -Run: - -```console -./simv +UVM_TESTNAME=matrix_ball_test +BID=4 -``` - -Or via bbdev: - -```console -bbdev uvm --build '--ball=matrix' -bbdev uvm --run '--ball=matrix' -- '+BID=4' -``` - -## Known DUT issue - -After mem-model multi-port fix (VCS no longer segfaults), pebble `+BID=1` -still hangs on directed OS 4x4: reads complete (`r0=4 r1=4`) but only -`wr=2/4` store rows appear and `cmd_resp` never fires. TIMEOUT fatals with -those counts. Treat as DUT/RTL investigation; UVM stack is intentionally -fail-hard so the hang is visible. - -## Acceptance - -- [x] Layout matches Blink common + ball-local split; filelist uses `@UVM@` / `@RTL@` -- [x] casegen whole-case DPI API; `cargo test` in casegen -- [x] VCS build with `matrix_ball.f` (no segfault on Instantiate DUT) -- [x] `+BID=` required; pebble `+BID=1` / toy `+BID=4` -- [ ] 23/23 functional pass — blocked on DUT hang above (`wr=2/4`) diff --git a/examples/balls/matrix/verify/casegen/src/casegen.rs b/examples/balls/matrix/verify/casegen/src/casegen.rs deleted file mode 100644 index 021aaa6db..000000000 --- a/examples/balls/matrix/verify/casegen/src/casegen.rs +++ /dev/null @@ -1,339 +0,0 @@ -use crate::model; - -pub const FUNCT7: u32 = 65; -pub const MAX_WORDS: usize = 16; - -#[repr(C)] -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct MatrixCmd { - pub bid: u32, - pub funct7: u32, - pub m: u32, - pub n: u32, - pub k: u32, - pub mode: u32, - pub op1_bank: u32, - pub op2_bank: u32, - pub wr_bank: u32, - pub rob_id: u32, - pub rs1_lo: u32, - pub rs1_hi: u32, - pub rs2_lo: u32, - pub rs2_hi: u32, - pub num_a_words: u32, - pub num_b_words: u32, - pub num_writes: u32, -} - -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct MatrixCase { - pub cmd: MatrixCmd, - pub a: Vec, - pub b: Vec, - pub writes: Vec, -} - -impl MatrixCase { - pub fn a_word_lo(&self, word_index: usize) -> u64 { - word_lo(&self.a, word_index) - } - pub fn a_word_hi(&self, word_index: usize) -> u64 { - word_hi(&self.a, word_index) - } - pub fn b_word_lo(&self, word_index: usize) -> u64 { - word_lo(&self.b, word_index) - } - pub fn b_word_hi(&self, word_index: usize) -> u64 { - word_hi(&self.b, word_index) - } -} - -fn word_lo(data: &[u8], word_index: usize) -> u64 { - let off = word_index * model::BANK_ROW_BYTES; - let mut buf = [0u8; 8]; - buf.copy_from_slice(&data[off..off + 8]); - u64::from_le_bytes(buf) -} - -fn word_hi(data: &[u8], word_index: usize) -> u64 { - let off = word_index * model::BANK_ROW_BYTES + 8; - let mut buf = [0u8; 8]; - buf.copy_from_slice(&data[off..off + 8]); - u64::from_le_bytes(buf) -} - -pub fn gen_case(seed: u32, index: u32, bid: u32) -> MatrixCase { - match index { - 0 => directed_4x4(bid, 0), - 1 => directed_4x4(bid, 1), - 2 => directed_16x16_os(bid), - _ => random_case(seed, index, bid), - } -} - -fn build_case( - bid: u32, - m: usize, - n: usize, - k: usize, - mode: u32, - op1_bank: u32, - op2_bank: u32, - wr_bank: u32, - rob_id: u32, - a_flat: &[i8], - b_flat: &[i8], -) -> MatrixCase { - if op1_bank == op2_bank || op1_bank == wr_bank || op2_bank == wr_bank { - panic!("build_case: banks must be distinct"); - } - if op1_bank > 7 || op2_bank > 7 || wr_bank > 7 { - panic!("build_case: banks must be in 0..7"); - } - if a_flat.len() != m * k || b_flat.len() != k * n { - panic!("build_case: flat buffer size mismatch"); - } - - let a = model::pack_a(a_flat, m, k); - let b = model::pack_b(b_flat, k, n); - let c = model::matmul(a_flat, b_flat, m, n, k); - let writes = model::emit_writes(&c, m, n); - - let na = model::words_from_rows(model::a_rows(m, k)) as u32; - let nb = model::words_from_rows(model::b_rows(n, k)) as u32; - if na as usize > MAX_WORDS || nb as usize > MAX_WORDS { - panic!("build_case: word count out of range a={na} b={nb}"); - } - - let rs1 = model::encode_rs1(op1_bank, op2_bank, wr_bank); - let rs2 = model::encode_rs2(m as u32, n as u32, k as u32, mode); - - MatrixCase { - cmd: MatrixCmd { - bid, - funct7: FUNCT7, - m: m as u32, - n: n as u32, - k: k as u32, - mode, - op1_bank, - op2_bank, - wr_bank, - rob_id, - rs1_lo: (rs1 & 0xffff_ffff) as u32, - rs1_hi: (rs1 >> 32) as u32, - rs2_lo: (rs2 & 0xffff_ffff) as u32, - rs2_hi: (rs2 >> 32) as u32, - num_a_words: na, - num_b_words: nb, - num_writes: writes.len() as u32, - }, - a, - b, - writes, - } -} - -fn directed_4x4(bid: u32, mode: u32) -> MatrixCase { - let m = 4usize; - let n = 4usize; - let k = 4usize; - let a: Vec = (1..=16).map(|x| x as i8).collect(); - let mut b = vec![0i8; k * n]; - for i in 0..k { - b[i * n + i] = 1; - } - build_case(bid, m, n, k, mode, 0, 1, 2, 3, &a, &b) -} - -fn directed_16x16_os(bid: u32) -> MatrixCase { - let m = 16usize; - let n = 16usize; - let k = 16usize; - let a: Vec = (0..m * k).map(|i| ((i % 127) + 1) as i8).collect(); - let mut b = vec![0i8; k * n]; - for i in 0..k { - for j in 0..n { - b[i * n + j] = if i == j { 1 } else { 0 }; - } - } - build_case(bid, m, n, k, 0, 0, 1, 2, 7, &a, &b) -} - -fn random_case(seed: u32, index: u32, bid: u32) -> MatrixCase { - let mut rng = Rng::new(seed, index); - let pool = [1usize, 2, 4, 8, 16]; - let m = pool[(rng.next() as usize) % pool.len()]; - let n = pool[(rng.next() as usize) % pool.len()]; - let k = pool[(rng.next() as usize) % pool.len()]; - let mode = rng.next() & 1; - - let op1_bank = rng.range(0, 7); - let mut op2_bank = rng.range(0, 7); - while op2_bank == op1_bank { - op2_bank = rng.range(0, 7); - } - let mut wr_bank = rng.range(0, 7); - while wr_bank == op1_bank || wr_bank == op2_bank { - wr_bank = rng.range(0, 7); - } - - let a: Vec = (0..m * k) - .map(|_| (rng.next() & 0x7f) as i8) - .collect(); - let b: Vec = (0..k * n) - .map(|_| (rng.next() & 0x7f) as i8) - .collect(); - - build_case( - bid, - m, - n, - k, - mode, - op1_bank, - op2_bank, - wr_bank, - rng.range(0, 15), - &a, - &b, - ) -} - -struct Rng { - state: u64, -} - -impl Rng { - fn new(seed: u32, index: u32) -> Self { - let state = (u64::from(seed) << 32) ^ u64::from(index) ^ 0x9E37_79B9_7F4A_7C15; - Self { state } - } - fn next(&mut self) -> u32 { - self.state ^= self.state >> 12; - self.state ^= self.state << 25; - self.state ^= self.state >> 27; - ((self.state.wrapping_mul(0x2545_F491_4F6C_DD1D)) >> 32) as u32 - } - fn range(&mut self, lo: u32, hi: u32) -> u32 { - lo + (self.next() % (hi - lo + 1)) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::model; - - #[test] - fn pack_a_matches_ctest_layout() { - let m = 4; - let k = 4; - let a: Vec = (1..=16).map(|x| x as i8).collect(); - let pa = model::pack_a(&a, m, k); - assert_eq!(pa.len(), model::a_rows(m, k) * model::TILE); - assert_eq!(pa[0], 1); - assert_eq!(pa[1], 2); - assert_eq!(pa[2], 3); - assert_eq!(pa[3], 4); - } - - #[test] - fn pack_b_identity() { - let k = 4; - let n = 4; - let mut b = vec![0i8; k * n]; - for i in 0..k { - b[i * n + i] = 1; - } - let pb = model::pack_b(&b, k, n); - assert_eq!(pb[0], 1); - assert_eq!(pb[17], 1); - } - - #[test] - fn matmul_4x4_identity_b() { - let a: Vec = (1..=16).map(|x| x as i8).collect(); - let mut b = vec![0i8; 16]; - for i in 0..4 { - b[i * 4 + i] = 1; - } - let c = model::matmul(&a, &b, 4, 4, 4); - for r in 0..4 { - for col in 0..4 { - assert_eq!(c[r][col], a[r * 4 + col] as i32); - } - } - } - - #[test] - fn directed_os_4x4_shape() { - let case = gen_case(0, 0, 1); - assert_eq!(case.cmd.funct7, FUNCT7); - assert_eq!(case.cmd.m, 4); - assert_eq!(case.cmd.n, 4); - assert_eq!(case.cmd.k, 4); - assert_eq!(case.cmd.mode, 0); - assert_eq!(case.cmd.op1_bank, 0); - assert_eq!(case.cmd.op2_bank, 1); - assert_eq!(case.cmd.wr_bank, 2); - assert_eq!(case.cmd.num_a_words, 16); - assert_eq!(case.cmd.num_b_words, 16); - assert_eq!(case.cmd.num_writes, 4); - assert_eq!(case.writes.len(), 4); - for w in &case.writes { - assert_eq!(w.group, 0); - assert_eq!(w.mask, 0xFFFF); - } - } - - #[test] - fn directed_ws_4x4_mode() { - let case = gen_case(0, 1, 1); - assert_eq!(case.cmd.mode, 1); - } - - #[test] - fn directed_16x16_os_writes() { - let case = gen_case(0, 2, 1); - assert_eq!(case.cmd.m, 16); - assert_eq!(case.cmd.num_writes, 64); - assert!(case.cmd.num_a_words < 128); - assert!(case.cmd.num_b_words < 128); - } - - #[test] - fn random_deterministic() { - let a = gen_case(0xCAFE_BABE, 5, 4); - let b = gen_case(0xCAFE_BABE, 5, 4); - assert_eq!(a, b); - } - - #[test] - fn bid_required_arg() { - let c0 = gen_case(0, 0, 1); - let c1 = gen_case(0, 0, 4); - assert_eq!(c0.cmd.bid, 1); - assert_eq!(c1.cmd.bid, 4); - } - - #[test] - fn rs1_rs2_encoding() { - let case = gen_case(0, 0, 1); - let rs1 = (u64::from(case.cmd.rs1_hi) << 32) | u64::from(case.cmd.rs1_lo); - let rs2 = (u64::from(case.cmd.rs2_hi) << 32) | u64::from(case.cmd.rs2_lo); - assert_eq!(rs1, model::encode_rs1(0, 1, 2)); - assert_eq!(rs2, model::encode_rs2(4, 4, 4, 0)); - } - - #[test] - fn write_mask_tail_n5() { - let a = vec![1i8; 5]; - let b = vec![1i8; 25]; - let c = model::matmul(&a, &b, 1, 5, 5); - let writes = model::emit_writes(&c, 1, 5); - assert_eq!(writes.len(), 2); - assert_eq!(writes[0].mask, 0xFFFF); - assert_eq!(writes[1].mask, 0x000F); - } -} diff --git a/examples/balls/matrix/verify/casegen/src/model.rs b/examples/balls/matrix/verify/casegen/src/model.rs deleted file mode 100644 index a25a9d5a1..000000000 --- a/examples/balls/matrix/verify/casegen/src/model.rs +++ /dev/null @@ -1,159 +0,0 @@ -pub const TILE: usize = 16; -pub const BANK_ROW_BYTES: usize = 16; -pub const WRITE_PORTS: usize = 4; -pub const ELEMS_PER_PORT: usize = 4; - -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct WriteExp { - pub group: u32, - pub addr: u32, - pub data: [u8; 16], - pub mask: u16, -} - -pub fn ceil_div(x: usize, d: usize) -> usize { - (x + d - 1) / d -} - -pub fn a_rows(m: usize, k: usize) -> usize { - ceil_div(m, TILE) * ceil_div(k, TILE) * TILE -} - -pub fn b_rows(n: usize, k: usize) -> usize { - ceil_div(n, TILE) * ceil_div(k, TILE) * TILE -} - -pub fn c_blocks(m: usize, n: usize) -> usize { - m * ceil_div(n, TILE) -} - -pub fn c_block(row: usize, n_tile: usize, m: usize, n: usize) -> usize { - let n_tiles = ceil_div(n, TILE); - let mt = row / TILE; - let mr = row % TILE; - let mut block = 0usize; - for t in 0..mt { - let rows = (m - t * TILE).min(TILE); - block += rows * n_tiles; - } - let rows = (m - mt * TILE).min(TILE); - block + n_tile * rows + mr -} - -pub fn pack_a(src: &[i8], m: usize, k: usize) -> Vec { - let kt = ceil_div(k, TILE); - let rows = a_rows(m, k); - let mut dst = vec![0u8; rows * TILE]; - for r in 0..m { - for c in 0..k { - let mt = r / TILE; - let mr = r % TILE; - let kti = c / TILE; - let lane = c % TILE; - let bank_row = (mt * kt + kti) * TILE + mr; - dst[bank_row * TILE + lane] = src[r * k + c] as u8; - } - } - dst -} - -pub fn pack_b(src: &[i8], k: usize, n: usize) -> Vec { - let kt = ceil_div(k, TILE); - let rows = b_rows(n, k); - let mut dst = vec![0u8; rows * TILE]; - for r in 0..k { - for c in 0..n { - let nt = c / TILE; - let lane = c % TILE; - let kti = r / TILE; - let kr = r % TILE; - let bank_row = (nt * kt + kti) * TILE + kr; - dst[bank_row * TILE + lane] = src[r * n + c] as u8; - } - } - dst -} - -pub fn matmul(a: &[i8], b: &[i8], m: usize, n: usize, k: usize) -> Vec> { - let mut c = vec![vec![0i32; n]; m]; - for i in 0..m { - for j in 0..n { - let mut acc = 0i32; - for kk in 0..k { - acc += a[i * k + kk] as i32 * b[kk * n + j] as i32; - } - c[i][j] = acc; - } - } - c -} - -pub fn emit_writes(c: &[Vec], m: usize, n: usize) -> Vec { - let n_tiles = ceil_div(n, TILE); - let mut out = Vec::new(); - for row in 0..m { - for nti in 0..n_tiles { - let block = c_block(row, nti, m, n); - let valid_elems = (n - nti * TILE).min(TILE); - for port in 0..WRITE_PORTS { - if port * ELEMS_PER_PORT >= valid_elems { - continue; - } - let mut data = [0u8; 16]; - let mut mask = 0u16; - for sub in 0..ELEMS_PER_PORT { - let elem = port * ELEMS_PER_PORT + sub; - let off = sub * 4; - if elem < valid_elems { - let col = nti * TILE + elem; - let v = c[row][col]; - data[off..off + 4].copy_from_slice(&v.to_le_bytes()); - for b in 0..4 { - mask |= 1u16 << (off + b); - } - } - } - out.push(WriteExp { - group: port as u32, - addr: block as u32, - data, - mask, - }); - } - } - } - out -} - -pub fn num_words(total_bytes: usize) -> usize { - assert_eq!( - total_bytes % BANK_ROW_BYTES, - 0, - "total_bytes not a multiple of bank row" - ); - total_bytes / BANK_ROW_BYTES -} - -pub fn words_from_rows(rows: usize) -> usize { - num_words(rows * TILE) -} - -pub fn encode_rs1(op1: u32, op2: u32, wr: u32) -> u64 { - if op1 >= 1024 || op2 >= 1024 || wr >= 1024 { - panic!("encode_rs1: bank out of 10-bit range"); - } - u64::from(op1) | (u64::from(op2) << 10) | (u64::from(wr) << 20) -} - -pub fn encode_rs2(m: u32, n: u32, k: u32, mode: u32) -> u64 { - if m == 0 || n == 0 || k == 0 { - panic!("encode_rs2: M/N/K must be non-zero"); - } - if m > 0xfff || n > 0xfff || k > 0xfff { - panic!("encode_rs2: M/N/K out of 12-bit range"); - } - if mode > 1 { - panic!("encode_rs2: mode must be 0 or 1"); - } - u64::from(m) | (u64::from(n) << 12) | (u64::from(k) << 24) | (u64::from(mode) << 36) -} diff --git a/examples/balls/matrix/verify/filelists/matrix_ball.f b/examples/balls/matrix/verify/filelists/matrix_ball.f deleted file mode 100644 index e289dbbef..000000000 --- a/examples/balls/matrix/verify/filelists/matrix_ball.f +++ /dev/null @@ -1,47 +0,0 @@ -+incdir+@UVM@/src -+incdir+@UVM@/src/agents/cmd -+incdir+@UVM@/src/agents/mem -+incdir+@UVM@/src/agents/resp -+incdir+@UVM@/src/cov -+incdir+@UVM@/src/env -+incdir+src -+incdir+src/pkg -+incdir+src/common -+incdir+src/seq -+incdir+src/cov -+incdir+src/env -+incdir+src/tests -+incdir+@RTL@ - -@UVM@/src/bb_blink_if.sv -@UVM@/src/bb_uvm_pkg.sv - -@RTL@/ram_4x287.sv -@RTL@/Queue4_BallRsIssue.sv -@RTL@/ram_4x38.sv -@RTL@/Queue4_SystolicRetireMeta.sv -@RTL@/ram_4x43.sv -@RTL@/Queue4_SystolicStoreMeta.sv -@RTL@/ram_4x13.sv -@RTL@/Queue4_BallRsComplete.sv -@RTL@/SystolicArrayCtrl.sv -@RTL@/ram_16x6.sv -@RTL@/Queue16_ReadTag.sv -@RTL@/ram_2x134.sv -@RTL@/Queue2_RowData.sv -@RTL@/SystolicArrayLoad.sv -@RTL@/mem_8x3.sv -@RTL@/Queue8_UInt3.sv -@RTL@/ram_3x5.sv -@RTL@/Queue3_UInt5.sv -@RTL@/ram_2x841.sv -@RTL@/Queue2_SystolicWsAccumReadEntry.sv -@RTL@/Queue1_SystolicWsAccumResultEntry.sv -@RTL@/wsAccMem_128x512.sv -@RTL@/SystolicArrayEX.sv -@RTL@/SystolicArrayStore.sv -@RTL@/MatrixUnit.sv -@RTL@/MatrixBall.sv - -src/pkg/matrix_pkg.sv -src/tb_top.sv diff --git a/examples/balls/matrix/verify/src/common/matrix_defs.svh b/examples/balls/matrix/verify/src/common/matrix_defs.svh deleted file mode 100644 index c0ddef3d4..000000000 --- a/examples/balls/matrix/verify/src/common/matrix_defs.svh +++ /dev/null @@ -1,49 +0,0 @@ -typedef struct { - int unsigned bid; - int unsigned funct7; - int unsigned m; - int unsigned n; - int unsigned k; - int unsigned mode; - int unsigned op1_bank; - int unsigned op2_bank; - int unsigned wr_bank; - int unsigned rob_id; - int unsigned rs1_lo; - int unsigned rs1_hi; - int unsigned rs2_lo; - int unsigned rs2_hi; - int unsigned num_a_words; - int unsigned num_b_words; - int unsigned num_writes; -} matrix_cmd_dpi_t; - -import "DPI-C" function int matrix_case_load( - input int unsigned seed, - input int unsigned index, - input int unsigned bid -); -import "DPI-C" function void matrix_case_cmd(output matrix_cmd_dpi_t cmd); -import "DPI-C" function longint unsigned matrix_case_a_word_lo(input int unsigned word_index); -import "DPI-C" function longint unsigned matrix_case_a_word_hi(input int unsigned word_index); -import "DPI-C" function longint unsigned matrix_case_b_word_lo(input int unsigned word_index); -import "DPI-C" function longint unsigned matrix_case_b_word_hi(input int unsigned word_index); -import "DPI-C" function int unsigned matrix_case_num_writes(); -import "DPI-C" function int unsigned matrix_case_write_group(input int unsigned i); -import "DPI-C" function int unsigned matrix_case_write_addr(input int unsigned i); -import "DPI-C" function longint unsigned matrix_case_write_data_lo(input int unsigned i); -import "DPI-C" function longint unsigned matrix_case_write_data_hi(input int unsigned i); -import "DPI-C" function int unsigned matrix_case_write_mask(input int unsigned i); - -localparam int MATRIX_FUNCT7 = 7'd65; -localparam int MATRIX_MAX_WORDS = 16; -localparam int MATRIX_TIMEOUT_CYCLES = 100000; -localparam int MATRIX_SEED = 32'hCAFE_BABE; - -function automatic int unsigned matrix_require_bid(); - int unsigned bid; - if (!$value$plusargs("BID=%d", bid)) begin - `uvm_fatal("BID", "missing required plusarg +BID=") - end - return bid; -endfunction diff --git a/examples/balls/matrix/verify/src/pkg/matrix_pkg.sv b/examples/balls/matrix/verify/src/pkg/matrix_pkg.sv deleted file mode 100644 index 94a1de54a..000000000 --- a/examples/balls/matrix/verify/src/pkg/matrix_pkg.sv +++ /dev/null @@ -1,13 +0,0 @@ -package matrix_pkg; - import uvm_pkg::*; - import bb_uvm_pkg::*; - `include "uvm_macros.svh" - - `include "common/matrix_defs.svh" - `include "common/matrix_items.svh" - `include "seq/matrix_sequences.svh" - `include "cov/matrix_cov.svh" - `include "env/matrix_scoreboard.svh" - `include "env/matrix_env.svh" - `include "tests/matrix_test.svh" -endpackage diff --git a/examples/balls/matrix/workloads/ctests/CMakeLists.txt b/examples/balls/matrix/workloads/ctests/CMakeLists.txt deleted file mode 100644 index 5f3015a9f..000000000 --- a/examples/balls/matrix/workloads/ctests/CMakeLists.txt +++ /dev/null @@ -1,20 +0,0 @@ -add_buckyball_ctests( - matrix_os_4x4_test.c - matrix_ws_4x4_test.c - matrix_os_16x16_test.c - matrix_ws_16x16_test.c - matrix_os_32x16_test.c - matrix_ws_32x16_test.c - matrix_os_48x16_test.c - matrix_os_48x16x25_test.c - matrix_os_64x16x25_test.c - matrix_os_64x16x25_slim_test.c - matrix_os_64x16x25_b051_test.c - matrix_os_seq_n6_n16_test.c - matrix_os_16x16_ones.c - matrix_os_32x32_ones.c - matrix_os_tail_5x7x3_test.c - matrix_mode_switch_test.c - matrix_bank_os_1024x16x16_test.c - matrix_bank_ws_1024x16x16_test.c -) diff --git a/examples/balls/matrix/workloads/ctests/matrix_bank_ws_1024x16x16_test.c b/examples/balls/matrix/workloads/ctests/matrix_bank_ws_1024x16x16_test.c deleted file mode 100644 index 89c8696e4..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_bank_ws_1024x16x16_test.c +++ /dev/null @@ -1,39 +0,0 @@ -#include "matrix_test_common.h" -#include - -#define M BANK_LINES -#define N MATRIX_TILE -#define K MATRIX_TILE -#define SEED_A 0x62 -#define SEED_B 0xB4 - -_Static_assert(M >= BANK_LINES && K == MATRIX_TILE, - "bank WS test must cover a full bank of A rows"); - -static elem_t a[M * K] __attribute__((aligned(64))); -static elem_t b[K * N] __attribute__((aligned(64))); -static elem_t pa[BANK_LINES * MATRIX_TILE] __attribute__((aligned(64))); -static elem_t pb[MATRIX_TILE * MATRIX_TILE] __attribute__((aligned(64))); -static result_t pc[BANK_LINES * MATRIX_ACC_LANES] __attribute__((aligned(64))); -static result_t out[M * N] __attribute__((aligned(64))); -static result_t exp_[M * N] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - init_u8_random_matrix(a, M, K, SEED_A); - init_u8_random_matrix(b, K, N, SEED_B); - cpu_matmul(a, b, exp_, M, N, K); - matrix_pack_a(a, pa, M, K); - matrix_pack_b(b, pb, K, N); - clear_u32_matrix(pc, matrix_c_blocks(M, N), MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 1); - matrix_unpack_c(pc, out, M, N); - if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_bank_ws_1024x16x16 FAILED\n"); - return 1; - } - printf("matrix_bank_ws_1024x16x16 PASSED\n"); - return 0; -} diff --git a/examples/balls/matrix/workloads/ctests/matrix_mode_switch_test.c b/examples/balls/matrix/workloads/ctests/matrix_mode_switch_test.c deleted file mode 100644 index b1ee4c768..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_mode_switch_test.c +++ /dev/null @@ -1,46 +0,0 @@ -#include "matrix_test_common.h" - -#define DIM 4 - -static elem_t a0[DIM * DIM] __attribute__((aligned(64))) = { - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, -}; -static elem_t b0[DIM * DIM] __attribute__((aligned(64))) = { - 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, -}; -static elem_t a1[DIM * DIM] __attribute__((aligned(64))) = { - 2, 0, 0, 0, 0, 2, 0, 0, 0, 0, 2, 0, 0, 0, 0, 2, -}; -static elem_t b1[DIM * DIM] __attribute__((aligned(64))) = { - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -}; -static elem_t pa[16 * MATRIX_TILE] __attribute__((aligned(64))); -static elem_t pb[16 * MATRIX_TILE] __attribute__((aligned(64))); -static result_t pc[DIM * MATRIX_ACC_LANES] __attribute__((aligned(64))); -static result_t out[DIM * DIM] __attribute__((aligned(64))); -static result_t exp_[DIM * DIM] __attribute__((aligned(64))); - -static int run_one(const char *name, elem_t *a, elem_t *b, int ws) { - cpu_matmul(a, b, exp_, DIM, DIM, DIM); - matrix_pack_a(a, pa, DIM, DIM); - matrix_pack_b(b, pb, DIM, DIM); - clear_u32_matrix(pc, DIM, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, DIM, DIM, DIM, ws); - matrix_unpack_c(pc, out, DIM, DIM); - if (!compare_u32_matrices(out, exp_, DIM, DIM)) { - printf("%s FAILED\n", name); - return 0; - } - printf("%s PASSED\n", name); - return 1; -} - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - int ok = run_one("mode_switch_os", a0, b0, 0); - ok = run_one("mode_switch_ws", a1, b1, 1) && ok; - ok = run_one("mode_switch_os2", a0, b1, 0) && ok; - return ok ? 0 : 1; -} diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_16x16_test.c b/examples/balls/matrix/workloads/ctests/matrix_os_16x16_test.c deleted file mode 100644 index 1197d818d..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_os_16x16_test.c +++ /dev/null @@ -1,69 +0,0 @@ -#include "matrix_test_common.h" - -#define DIM 16 - -static elem_t a[DIM * DIM] __attribute__((aligned(64))) = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, 16, - -8, 9, -3, 11, 6, -7, 12, -4, 13, 14, -15, 2, -9, 17, -18, - 19, -20, -5, 21, 0, 22, -23, 3, 24, -25, 26, 1, -27, 28, -6, - 29, -30, 31, 32, -33, 34, 4, -35, 36, -37, 5, 38, -39, 40, -41, - 42, 6, -43, 44, -11, 45, 46, -47, 7, 48, -49, 50, -12, 51, 52, - -53, 8, 54, -55, 56, 57, -13, 58, 9, -59, 60, -61, 62, 10, -63, - 64, -14, 65, 66, -67, 11, -68, 69, 12, -70, 71, -15, 72, 73, -74, - 13, 75, -76, 77, -16, 78, 14, 79, -80, 15, 81, -82, 83, -17, 84, - 85, -86, 16, 87, -88, 89, -18, 90, 91, -19, 92, 17, -93, 94, -95, - 18, 96, -97, 98, -20, 99, -21, 22, -24, 25, -26, 27, -28, 29, -31, - 33, -32, 35, -34, 37, -36, 39, -38, 41, -40, 43, -42, 47, -44, 49, - -45, 53, -46, 55, -48, 59, -50, 61, -51, 63, -52, -54, 67, -56, 70, - -57, 74, -58, 76, -60, 80, -62, 82, -64, 86, -65, 88, -66, 93, -69, - 95, -71, 97, -72, 99, -73, 1, -75, 2, -77, 3, -78, 4, -79, 5, - -81, 6, -83, 7, -84, 8, -85, 9, -87, 10, -89, 11, -90, 12, -91, - 13, -92, 14, -94, 15, -96, 16, -98, 17, -99, 18, 19, -1, 20, -2, - 21, -3, 23, -4, 24, -5, 26, -6, 28, -7, 30, -8, 31, -9, 32, - -10, -}; - -static elem_t b[DIM * DIM] __attribute__((aligned(64))) = { - 1, 0, -1, 2, 3, -2, 4, -3, 5, -4, 6, -5, 7, -6, 8, - -7, -8, 9, -9, 10, 0, 11, -11, 12, -12, 13, -13, 14, -14, 15, - -15, 1, 2, -16, 3, 16, -17, 4, 17, -18, 5, 18, -19, 6, 19, - -20, 7, 20, -21, 8, 21, -22, 9, 22, 0, 23, -23, 10, 24, -24, - 11, 25, -25, 12, 26, -26, 13, 27, -27, 14, 28, -28, 15, 29, -29, - 1, 30, -30, 2, 31, -31, 3, 32, -32, 4, 33, -33, 5, 34, -34, - 6, 35, -35, 7, 36, -36, 8, 37, -37, 9, 38, -38, 10, 39, -39, - 11, 40, -40, 12, 0, 41, -41, 13, 42, -42, 14, 43, -43, 15, 44, - -44, 1, 45, -45, 2, 46, -46, 3, 47, -47, 4, 48, -48, 5, 49, - -49, 6, 50, -50, 7, 0, 8, -1, 9, -2, 10, -3, 11, -4, 12, - -5, 13, -6, 14, -7, 15, -8, 1, -9, 2, -10, 3, -11, 4, -12, - 5, -13, 6, -14, 7, -15, 8, 16, -16, 17, 0, 18, -18, 19, -19, - 20, -20, 21, -21, 22, -22, 23, -23, 24, -24, 25, -25, 9, -26, 10, - 26, -27, 11, 27, -28, 12, 28, -29, 13, 29, -30, 14, 30, -31, 15, - 31, 0, 32, -1, -32, 1, 2, -2, 3, -3, 4, -4, 5, -5, 6, - -6, 7, -7, 8, -8, 9, -9, 10, -10, 11, -11, 12, -12, 13, 0, - 14, -13, 15, -14, 1, -15, 2, 16, -16, 3, 17, -17, 4, 18, -18, - 5, -}; - -static elem_t pa[DIM * MATRIX_TILE] __attribute__((aligned(64))); -static elem_t pb[DIM * MATRIX_TILE] __attribute__((aligned(64))); -static result_t pc[DIM * MATRIX_ACC_LANES] __attribute__((aligned(64))); -static result_t out[DIM * DIM] __attribute__((aligned(64))); -static result_t exp_[DIM * DIM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - cpu_matmul(a, b, exp_, DIM, DIM, DIM); - matrix_pack_a(a, pa, DIM, DIM); - matrix_pack_b(b, pb, DIM, DIM); - clear_u32_matrix(pc, DIM, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, DIM, DIM, DIM, 0); - matrix_unpack_c(pc, out, DIM, DIM); - if (!compare_u32_matrices(out, exp_, DIM, DIM)) { - printf("matrix_os_16x16 FAILED\n"); - return 1; - } - printf("matrix_os_16x16 PASSED\n"); - return 0; -} diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_32x32_ones.c b/examples/balls/matrix/workloads/ctests/matrix_os_32x32_ones.c deleted file mode 100644 index 803734160..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_os_32x32_ones.c +++ /dev/null @@ -1,32 +0,0 @@ -#include "matrix_test_common.h" -#include - -#define DIM 32 - -static elem_t a[DIM * DIM] __attribute__((aligned(64))); -static elem_t b[DIM * DIM] __attribute__((aligned(64))); -static elem_t pa[64 * MATRIX_TILE] __attribute__((aligned(64))); -static elem_t pb[64 * MATRIX_TILE] __attribute__((aligned(64))); -static result_t pc[64 * MATRIX_ACC_LANES] __attribute__((aligned(64))); -static result_t out[DIM * DIM] __attribute__((aligned(64))); -static result_t exp_[DIM * DIM] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - init_ones_matrix(a, DIM, DIM); - init_ones_matrix(b, DIM, DIM); - cpu_matmul(a, b, exp_, DIM, DIM, DIM); - matrix_pack_a(a, pa, DIM, DIM); - matrix_pack_b(b, pb, DIM, DIM); - clear_u32_matrix(pc, 64, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, DIM, DIM, DIM, 0); - matrix_unpack_c(pc, out, DIM, DIM); - if (!compare_u32_matrices(out, exp_, DIM, DIM)) { - printf("matrix_os_32x32_ones FAILED\n"); - return 1; - } - printf("matrix_os_32x32_ones PASSED\n"); - return 0; -} diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_4x4_test.c b/examples/balls/matrix/workloads/ctests/matrix_os_4x4_test.c deleted file mode 100644 index 11a5ef858..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_os_4x4_test.c +++ /dev/null @@ -1,35 +0,0 @@ -#include "matrix_test_common.h" - -#define M 4 -#define N 4 -#define K 4 - -static elem_t a[M * K] __attribute__((aligned(64))) = { - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, -}; -static elem_t b[K * N] __attribute__((aligned(64))) = { - 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, -}; -static elem_t pa[16 * MATRIX_TILE] __attribute__((aligned(64))); -static elem_t pb[16 * MATRIX_TILE] __attribute__((aligned(64))); -static result_t pc[M * MATRIX_ACC_LANES] __attribute__((aligned(64))); -static result_t out[M * N] __attribute__((aligned(64))); -static result_t exp_[M * N] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - cpu_matmul(a, b, exp_, M, N, K); - matrix_pack_a(a, pa, M, K); - matrix_pack_b(b, pb, K, N); - clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); - matrix_unpack_c(pc, out, M, N); - if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_os_4x4 FAILED\n"); - return 1; - } - printf("matrix_os_4x4 PASSED\n"); - return 0; -} diff --git a/examples/balls/matrix/workloads/ctests/matrix_test_common.h b/examples/balls/matrix/workloads/ctests/matrix_test_common.h deleted file mode 100644 index 6f4b12fdb..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_test_common.h +++ /dev/null @@ -1,110 +0,0 @@ -#ifndef MATRIX_TEST_COMMON_H -#define MATRIX_TEST_COMMON_H - -#include "buckyball.h" -#include -#include -#include - -#define MATRIX_TILE 16 -#define MATRIX_ACC_LANES 16 - -static inline int matrix_ceil_div(int x, int d) { return (x + d - 1) / d; } - -static inline int matrix_a_rows(int m, int k) { - return matrix_ceil_div(m, MATRIX_TILE) * matrix_ceil_div(k, MATRIX_TILE) * - MATRIX_TILE; -} - -static inline int matrix_b_rows(int n, int k) { - return matrix_ceil_div(n, MATRIX_TILE) * matrix_ceil_div(k, MATRIX_TILE) * - MATRIX_TILE; -} - -static inline int matrix_c_blocks(int m, int n) { - return m * matrix_ceil_div(n, MATRIX_TILE); -} - -static inline void matrix_pack_a(const elem_t *src, elem_t *dst, int m, int k) { - int kt = matrix_ceil_div(k, MATRIX_TILE); - int rows = matrix_a_rows(m, k); - for (int i = 0; i < rows * MATRIX_TILE; ++i) - dst[i] = 0; - for (int r = 0; r < m; ++r) { - for (int c = 0; c < k; ++c) { - int mt = r / MATRIX_TILE; - int mr = r % MATRIX_TILE; - int kti = c / MATRIX_TILE; - int lane = c % MATRIX_TILE; - int bank_row = (mt * kt + kti) * MATRIX_TILE + mr; - dst[bank_row * MATRIX_TILE + lane] = src[r * k + c]; - } - } -} - -static inline void matrix_pack_b(const elem_t *src, elem_t *dst, int k, int n) { - int kt = matrix_ceil_div(k, MATRIX_TILE); - int rows = matrix_b_rows(n, k); - for (int i = 0; i < rows * MATRIX_TILE; ++i) - dst[i] = 0; - for (int r = 0; r < k; ++r) { - for (int c = 0; c < n; ++c) { - int nt = c / MATRIX_TILE; - int lane = c % MATRIX_TILE; - int kti = r / MATRIX_TILE; - int kr = r % MATRIX_TILE; - int bank_row = (nt * kt + kti) * MATRIX_TILE + kr; - dst[bank_row * MATRIX_TILE + lane] = src[r * n + c]; - } - } -} - -static inline int matrix_c_block(int row, int n_tile, int m, int n) { - int n_tiles = matrix_ceil_div(n, MATRIX_TILE); - int mt = row / MATRIX_TILE; - int mr = row % MATRIX_TILE; - int block = 0; - for (int t = 0; t < mt; ++t) { - int rows = m - t * MATRIX_TILE; - if (rows > MATRIX_TILE) - rows = MATRIX_TILE; - block += rows * n_tiles; - } - int rows = m - mt * MATRIX_TILE; - if (rows > MATRIX_TILE) - rows = MATRIX_TILE; - return block + n_tile * rows + mr; -} - -static inline void matrix_unpack_c(const result_t *src, result_t *dst, int m, - int n) { - for (int r = 0; r < m; ++r) { - for (int c = 0; c < n; ++c) { - int nti = c / MATRIX_TILE; - int lane = c % MATRIX_TILE; - int block = matrix_c_block(r, nti, m, n); - dst[r * n + c] = src[block * MATRIX_ACC_LANES + lane]; - } - } -} - -static inline void matrix_hw_mnk(const elem_t *packed_a, const elem_t *packed_b, - result_t *packed_c, int m, int n, int k, - int ws) { - uint32_t op1 = 0, op2 = 1, wr = 2; - int a_rows = matrix_a_rows(m, k); - int b_rows = matrix_b_rows(n, k); - int c_blocks = matrix_c_blocks(m, n); - - bb_mem_alloc(op1, 1, 1); - bb_mem_alloc(op2, 1, 1); - bb_mem_alloc(wr, 1, 4); - bb_mvin((uintptr_t)packed_a, op1, a_rows, 1); - bb_mvin((uintptr_t)packed_b, op2, b_rows, 1); - bb_matrix_mnk_mode(op1, op2, wr, m, n, k, - ws ? BB_MATRIX_MODE_WS : BB_MATRIX_MODE_OS); - bb_mvout((uintptr_t)packed_c, wr, c_blocks, 1); - bb_fence(); -} - -#endif diff --git a/examples/balls/matrix/workloads/ctests/matrix_ws_32x16_test.c b/examples/balls/matrix/workloads/ctests/matrix_ws_32x16_test.c deleted file mode 100644 index e0aace0f1..000000000 --- a/examples/balls/matrix/workloads/ctests/matrix_ws_32x16_test.c +++ /dev/null @@ -1,90 +0,0 @@ -#include "matrix_test_common.h" - -#define M 32 -#define N 16 -#define K 16 - -static elem_t a[M * K] __attribute__((aligned(64))) = { - 1, 2, 3, -1, -2, 0, 4, 5, 10, -10, 7, 100, -100, 8, - 16, -8, 9, -3, 11, 6, -7, 12, -4, 13, 14, -15, 2, -9, - 17, -18, 19, -20, -5, 21, 0, 22, -23, 3, 24, -25, 26, 1, - -27, 28, -6, 29, -30, 31, 32, -33, 34, 4, -35, 36, -37, 5, - 38, -39, 40, -41, 42, 6, -43, 44, -11, 45, 46, -47, 7, 48, - -49, 50, -12, 51, 52, -53, 8, 54, -55, 56, 57, -13, 58, 9, - -59, 60, -61, 62, 10, -63, 64, -14, 65, 66, -67, 11, -68, 69, - 12, -70, 71, -15, 72, 73, -74, 13, 75, -76, 77, -16, 78, 14, - 79, -80, 15, 81, -82, 83, -17, 84, 85, -86, 16, 87, -88, 89, - -18, 90, 91, -19, 92, 17, -93, 94, -95, 18, 96, -97, 98, -20, - 99, -21, 22, -24, 25, -26, 27, -28, 29, -31, 33, -32, 35, -34, - 37, -36, 39, -38, 41, -40, 43, -42, 47, -44, 49, -45, 53, -46, - 55, -48, 59, -50, 61, -51, 63, -52, -54, 67, -56, 70, -57, 74, - -58, 76, -60, 80, -62, 82, -64, 86, -65, 88, -66, 93, -69, 95, - -71, 97, -72, 99, -73, 1, -75, 2, -77, 3, -78, 4, -79, 5, - -81, 6, -83, 7, -84, 8, -85, 9, -87, 10, -89, 11, -90, 12, - -91, 13, -92, 14, -94, 15, -96, 16, -98, 17, -99, 18, 19, -1, - 20, -2, 21, -3, 23, -4, 24, -5, 26, -6, 28, -7, 30, -8, - 31, -9, 32, -10, 100, -99, 50, -50, 25, -25, 12, -12, 6, -6, - 3, -3, 1, 0, -1, 2, -100, 99, -50, 50, -25, 25, -12, 12, - -6, 6, -3, 3, -1, 0, 1, -2, 7, -14, 21, -28, 35, -42, - 49, -56, 63, -70, 77, -84, 91, -98, 8, -15, 16, 0, -16, 32, - -32, 48, -48, 64, -64, 80, -80, 96, -96, 11, -22, 33, 1, -7, - 13, -19, 25, -31, 37, -43, 49, -55, 61, -67, 73, -79, 85, -91, - 90, 80, 70, 60, 50, 40, 30, 20, 10, 0, -10, -20, -30, -40, - -50, -60, -70, -80, -90, 15, 27, 39, 51, 63, 75, 87, 99, -11, - -23, -35, -47, -59, 4, 8, 16, 32, 64, -4, -8, -16, -32, -64, - 5, 10, 20, 40, 80, -5, 0, 1, 0, 2, 0, 3, 0, -1, - 0, -2, 0, -3, 0, 4, 0, -4, 127, -128, 64, -64, 32, -32, - 16, -16, 8, -8, 4, -4, 2, -2, 1, -1, 3, 1, 4, 1, - 5, 9, 2, 6, 5, 3, 5, -8, -9, 7, -9, 3, -17, 34, - -51, 68, -85, 102, -119, 12, -29, 46, -63, 80, -97, 18, -41, 58, - 9, 0, 8, 0, 7, 0, 6, 0, 5, 0, 4, 0, 3, 0, - 2, 0, -9, 1, -8, 2, -7, 3, -6, 4, -5, 5, -4, 6, - -3, 7, -2, 8, 11, 22, 33, 44, 55, 66, 77, 88, 99, -11, - -22, -33, -44, -55, -66, -77, 18, -27, 36, -45, 54, -63, 72, -81, - 90, -99, 12, -21, 30, -39, 48, -57, -}; - -static elem_t b[K * N] __attribute__((aligned(64))) = { - 1, 0, -1, 2, 3, -2, 4, -3, 5, -4, 6, -5, 7, -6, 8, - -7, -8, 9, -9, 10, 0, 11, -11, 12, -12, 13, -13, 14, -14, 15, - -15, 1, 2, -16, 3, 16, -17, 4, 17, -18, 5, 18, -19, 6, 19, - -20, 7, 20, -21, 8, 21, -22, 9, 22, 0, 23, -23, 10, 24, -24, - 11, 25, -25, 12, 26, -26, 13, 27, -27, 14, 28, -28, 15, 29, -29, - 1, 30, -30, 2, 31, -31, 3, 32, -32, 4, 33, -33, 5, 34, -34, - 6, 35, -35, 7, 36, -36, 8, 37, -37, 9, 38, -38, 10, 39, -39, - 11, 40, -40, 12, 0, 41, -41, 13, 42, -42, 14, 43, -43, 15, 44, - -44, 1, 45, -45, 2, 46, -46, 3, 47, -47, 4, 48, -48, 5, 49, - -49, 6, 50, -50, 7, 0, 8, -1, 9, -2, 10, -3, 11, -4, 12, - -5, 13, -6, 14, -7, 15, -8, 1, -9, 2, -10, 3, -11, 4, -12, - 5, -13, 6, -14, 7, -15, 8, 16, -16, 17, 0, 18, -18, 19, -19, - 20, -20, 21, -21, 22, -22, 23, -23, 24, -24, 25, -25, 9, -26, 10, - 26, -27, 11, 27, -28, 12, 28, -29, 13, 29, -30, 14, 30, -31, 15, - 31, 0, 32, -1, -32, 1, 2, -2, 3, -3, 4, -4, 5, -5, 6, - -6, 7, -7, 8, -8, 9, -9, 10, -10, 11, -11, 12, -12, 13, 0, - 14, -13, 15, -14, 1, -15, 2, 16, -16, 3, 17, -17, 4, 18, -18, - 5, -}; - -static elem_t pa[32 * MATRIX_TILE] __attribute__((aligned(64))); -static elem_t pb[16 * MATRIX_TILE] __attribute__((aligned(64))); -static result_t pc[M * MATRIX_ACC_LANES] __attribute__((aligned(64))); -static result_t out[M * N] __attribute__((aligned(64))); -static result_t exp_[M * N] __attribute__((aligned(64))); - -int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif - cpu_matmul(a, b, exp_, M, N, K); - matrix_pack_a(a, pa, M, K); - matrix_pack_b(b, pb, K, N); - clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 1); - matrix_unpack_c(pc, out, M, N); - if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_ws_32x16 FAILED\n"); - return 1; - } - printf("matrix_ws_32x16 PASSED\n"); - return 0; -} diff --git a/examples/balls/matrix/workloads/mlir_tests/ball/CMakeLists.txt b/examples/balls/matrix/workloads/mlir_tests/ball/CMakeLists.txt deleted file mode 100644 index 8c2d2215f..000000000 --- a/examples/balls/matrix/workloads/mlir_tests/ball/CMakeLists.txt +++ /dev/null @@ -1,13 +0,0 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - -set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) -set(BUCKYBALL_MLIR_TEST_PREFIX ball) - -add_buckyball_mlir_contract(matrix_16x16_i8_contract - PASSES - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" - ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} - ${BUCKYBALL_LOWER_BUCKYBALL}) - -add_buckyball_mlir_optest(matrix_16x16_i8) -add_buckyball_mlir_optest(matrix_matmul_16x16_i8) diff --git a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8.mlir b/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8.mlir deleted file mode 100644 index 491a9edf6..000000000 --- a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8.mlir +++ /dev/null @@ -1,40 +0,0 @@ -// Already-lowered ball Op: buckyball.matrix (physical-bank form). - -func.func private @check_result(memref<16x16xi32>) -> () - -func.func @main() -> i8 { - %zero_i8 = arith.constant 0 : i8 - %one_i8 = arith.constant 1 : i8 - %zero_i32 = arith.constant 0 : i32 - %depth = arith.constant 16 : i64 - %stride = arith.constant 1 : i64 - %cfg = arith.constant 268501008 : i64 - - %a = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %b = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %c = memref.alloc() {alignment = 64 : i64} : memref<16x16xi32> - - linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) - linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x16xi8>) - linalg.fill ins(%zero_i32 : i32) outs(%c : memref<16x16xi32>) - - %ba = buckyball.bank_alloc - %bb = buckyball.bank_alloc - %bc = buckyball.bank_alloc {col = 4 : i64} - %la = buckyball.bank_mvin %a %ba %depth %stride - : memref<16x16xi8> i64 i64 i64 - %lb = buckyball.bank_mvin %b %bb %depth %stride - : memref<16x16xi8> i64 i64 i64 - buckyball.matrix %la, %lb, %bc, %cfg : i64 - %stored = buckyball.bank_mvout %c %bc %depth %stride - : memref<16x16xi32> i64 i64 i64 - buckyball.bank_release %la : i64 - buckyball.bank_release %lb : i64 - buckyball.bank_release %stored : i64 - - func.call @check_result(%c) : (memref<16x16xi32>) -> () - memref.dealloc %a : memref<16x16xi8> - memref.dealloc %b : memref<16x16xi8> - memref.dealloc %c : memref<16x16xi32> - return %zero_i8 : i8 -} diff --git a/examples/balls/matrix/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/matrix/workloads/mlir_tests/bank/CMakeLists.txt deleted file mode 100644 index 6477d1110..000000000 --- a/examples/balls/matrix/workloads/mlir_tests/bank/CMakeLists.txt +++ /dev/null @@ -1,9 +0,0 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - -set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) -set(BUCKYBALL_MLIR_TEST_PREFIX bank) - -add_buckyball_mlir_contract(matrix_16x16_i8_contract - PASSES "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}") - -add_buckyball_mlir_optest(matrix_16x16_i8) diff --git a/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8.mlir b/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8.mlir deleted file mode 100644 index 58bdd36b2..000000000 --- a/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8.mlir +++ /dev/null @@ -1,42 +0,0 @@ -// Bank-level Op: buckyball.bank_matrix -// Lowering: assign-physical-banks -> bank_matrix becomes matrix. - -func.func private @check_result(memref<16x16xi32>) -> () - -func.func @main() -> i8 { - %zero_i8 = arith.constant 0 : i8 - %one_i8 = arith.constant 1 : i8 - %zero_i32 = arith.constant 0 : i32 - %depth = arith.constant 16 : i64 - %stride = arith.constant 1 : i64 - %cfg = arith.constant 268501008 : i64 - - %a = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %b = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %c = memref.alloc() {alignment = 64 : i64} : memref<16x16xi32> - - linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) - linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x16xi8>) - linalg.fill ins(%zero_i32 : i32) outs(%c : memref<16x16xi32>) - - %ba = buckyball.bank_alloc - %bb = buckyball.bank_alloc - %bc = buckyball.bank_alloc {col = 4 : i64} - %la = buckyball.bank_mvin %a %ba %depth %stride - : memref<16x16xi8> i64 i64 i64 - %lb = buckyball.bank_mvin %b %bb %depth %stride - : memref<16x16xi8> i64 i64 i64 - %r = buckyball.bank_matrix %la %lb %bc %cfg - : i64 i64 i64 i64 - %stored = buckyball.bank_mvout %c %r %depth %stride - : memref<16x16xi32> i64 i64 i64 - buckyball.bank_release %la : i64 - buckyball.bank_release %lb : i64 - buckyball.bank_release %stored : i64 - - func.call @check_result(%c) : (memref<16x16xi32>) -> () - memref.dealloc %a : memref<16x16xi8> - memref.dealloc %b : memref<16x16xi8> - memref.dealloc %c : memref<16x16xi32> - return %zero_i8 : i8 -} diff --git a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2Int.scala b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2Int.scala index d0f8060c9..165ab53a7 100644 --- a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2Int.scala +++ b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2Int.scala @@ -41,23 +41,34 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { val elemBits = decoder.elemBits val bankWidth = b.memDomain.bankWidth - require(bankWidth == 128, s"Mxfp2IntBall requires bankWidth = 128, got $bankWidth") + require( + bankWidth == 128, + s"Mxfp2IntBall requires bankWidth = 128, got $bankWidth" + ) val wordsPerBlock = ballConfig.mxfpFormat match { case "MXFP4" => 1 - case "MXFP6" => 2 // padded: 192 bits data + 64 bits padding = 256 bits = 2 words + case "MXFP6" => + 2 // padded: 192 bits data + 64 bits padding = 256 bits = 2 words case "MXFP8" => 2 - case _ => throw new IllegalArgumentException(s"Unsupported mxfpFormat: ${ballConfig.mxfpFormat}") + case _ => + throw new IllegalArgumentException( + s"Unsupported mxfpFormat: ${ballConfig.mxfpFormat}" + ) } val FP_PER_BLOCK = 32 val INT8_PER_WORD = bankWidth / 8 // 16 val OUT_WORDS_PER_BLOCK = FP_PER_BLOCK / INT8_PER_WORD // 2 - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "Mxfp2IntBall") - .getOrElse(throw new IllegalArgumentException("Mxfp2IntBall not found in config")) - val inBW = ballMapping.inBW - val outBW = ballMapping.outBW + val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "Mxfp2IntBall") + .getOrElse( + throw new IllegalArgumentException("Mxfp2IntBall not found in config") + ) + + val inBW = ballMapping.inBW + val outBW = ballMapping.outBW @public val io = IO(new Bundle { @@ -81,8 +92,9 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { } // FSM - val idle :: sReadReq :: sReadResp :: sMmioReq :: sMmioResp :: sWriteLo :: sWriteHi :: sComplete :: Nil = Enum(8) - val state = RegInit(idle) + val idle :: sReadReq :: sReadResp :: sMmioReq :: sMmioResp :: sWriteLoReq :: sWriteLoResp :: sWriteHiReq :: sWriteHiResp :: sComplete :: Nil = + Enum(10) + val state = RegInit(idle) // Per-instruction registers val raddr_reg = RegInit(0.U(b.frontend.iter_len.W)) @@ -93,8 +105,10 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { val block_idx_reg = RegInit(0.U(b.frontend.iter_len.W)) val word_idx_reg = RegInit(0.U(2.W)) // 0..wordsPerBlock-1 - val inputWordsReg = RegInit(VecInit(Seq.fill(2)(0.U(bankWidth.W)))) // max 2 words for MXFP6/8 - val scaleReg = RegInit(0.U(8.W)) + val inputWordsReg = RegInit( + VecInit(Seq.fill(2)(0.U(bankWidth.W))) + ) // max 2 words for MXFP6/8 + val scaleReg = RegInit(0.U(8.W)) // Extract 32 FP elements from inputWordsReg based on format val fpElements = VecInit(Seq.tabulate(FP_PER_BLOCK) { i => @@ -107,17 +121,25 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { // Cross-word boundary (only for MXFP6 if packed, but we use padded so this won't happen) val lowBits = bankWidth - bitInWord val highBits = elemBits - lowBits - Cat(inputWordsReg(wordIdx + 1)(highBits - 1, 0), inputWordsReg(wordIdx)(bankWidth - 1, bitInWord)) + Cat( + inputWordsReg(wordIdx + 1)(highBits - 1, 0), + inputWordsReg(wordIdx)(bankWidth - 1, bitInWord) + ) } else { 0.U(elemBits.W) } }) // Combinationally compute all 32 INT8 values from the latched elements + scale - val int8Vec = VecInit(fpElements.map(raw => decoder.dequant(raw, scaleReg).asUInt)) + val int8Vec = VecInit( + fpElements.map(raw => decoder.dequant(raw, scaleReg).asUInt) + ) val outWordLo = Cat((0 until INT8_PER_WORD).map(i => int8Vec(i)).reverse) - val outWordHi = Cat((INT8_PER_WORD until FP_PER_BLOCK).map(i => int8Vec(i)).reverse) + + val outWordHi = Cat( + (INT8_PER_WORD until FP_PER_BLOCK).map(i => int8Vec(i)).reverse + ) // === Default IO === for (i <- 0 until inBW) { @@ -133,7 +155,9 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { io.bankWrite(i).io.req.valid := false.B io.bankWrite(i).io.req.bits.addr := 0.U io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W))) + io.bankWrite(i).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W)) + ) io.bankWrite(i).io.resp.ready := false.B io.bankWrite(i).rob_id := rob_id_reg io.bankWrite(i).ball_id := 0.U @@ -141,17 +165,16 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { io.bankWrite(i).group_id := 0.U } - io.mmioRead.req.valid := false.B - io.mmioRead.req.bits.rel_addr := 0.U - io.mmioRead.resp.ready := false.B - io.mmioRead.meta_bank := wbank_reg - io.mmioRead.rob_id := rob_id_reg - io.mmioRead.ball_id := 0.U - io.cmdReq.ready := state === idle - io.cmdResp.valid := false.B - io.cmdResp.bits.rob_id := rob_id_reg - io.cmdResp.bits.is_sub := is_sub_reg - io.cmdResp.bits.sub_rob_id := sub_rob_id_reg + io.mmioRead.req.valid := false.B + io.mmioRead.req.bits.addr := 0.U + io.mmioRead.resp.ready := false.B + io.mmioRead.rob_id := rob_id_reg + io.mmioRead.ball_id := 0.U + io.cmdReq.ready := state === idle + io.cmdResp.valid := false.B + io.cmdResp.bits.rob_id := rob_id_reg + io.cmdResp.bits.is_sub := is_sub_reg + io.cmdResp.bits.sub_rob_id := sub_rob_id_reg // === FSM === switch(state) { @@ -173,8 +196,12 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { } is(sReadReq) { - io.bankRead(0).io.req.valid := true.B - io.bankRead(0).io.req.bits.addr := raddr_reg + block_idx_reg * wordsPerBlock.U + word_idx_reg + io.bankRead(0).io.req.valid := true.B + io.bankRead(0) + .io + .req + .bits + .addr := raddr_reg + block_idx_reg * wordsPerBlock.U + word_idx_reg when(io.bankRead(0).io.req.fire) { state := sReadResp } @@ -195,8 +222,8 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { } is(sMmioReq) { - io.mmioRead.req.valid := true.B - io.mmioRead.req.bits.rel_addr := block_idx_reg + io.mmioRead.req.valid := true.B + io.mmioRead.req.bits.addr := block_idx_reg when(io.mmioRead.req.fire) { state := sMmioResp } @@ -206,28 +233,44 @@ class PipelinedMxfp2Int(val b: GlobalConfig) extends Module { io.mmioRead.resp.ready := true.B when(io.mmioRead.resp.fire) { scaleReg := io.mmioRead.resp.bits.data(7, 0) - state := sWriteLo + state := sWriteLoReq } } - is(sWriteLo) { + is(sWriteLoReq) { io.bankWrite(0).io.req.valid := true.B io.bankWrite(0).io.req.bits.addr := waddr_reg + (block_idx_reg << 1) io.bankWrite(0).io.req.bits.data := outWordLo - io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(1.U(1.W))) - io.bankWrite(0).io.resp.ready := true.B + io.bankWrite(0).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(1.U(1.W)) + ) when(io.bankWrite(0).io.req.fire) { - state := sWriteHi + state := sWriteLoResp + } + } + + is(sWriteLoResp) { + io.bankWrite(0).io.resp.ready := true.B + when(io.bankWrite(0).io.resp.fire) { + state := sWriteHiReq } } - is(sWriteHi) { + is(sWriteHiReq) { io.bankWrite(0).io.req.valid := true.B io.bankWrite(0).io.req.bits.addr := waddr_reg + (block_idx_reg << 1) + 1.U io.bankWrite(0).io.req.bits.data := outWordHi - io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(1.U(1.W))) - io.bankWrite(0).io.resp.ready := true.B + io.bankWrite(0).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(1.U(1.W)) + ) when(io.bankWrite(0).io.req.fire) { + state := sWriteHiResp + } + } + + is(sWriteHiResp) { + io.bankWrite(0).io.resp.ready := true.B + when(io.bankWrite(0).io.resp.fire) { when(block_idx_reg === iter_reg - 1.U) { state := sComplete }.otherwise { diff --git a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2IntBall.scala b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2IntBall.scala index 3f4ca2014..266958b4d 100644 --- a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2IntBall.scala +++ b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp2IntBall.scala @@ -17,19 +17,26 @@ import framework.top.GlobalConfig @instantiable class Mxfp2IntBall(val b: GlobalConfig) extends Module with HasBlink { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "Mxfp2IntBall") - .getOrElse(throw new IllegalArgumentException("Mxfp2IntBall not found in config")) + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "Mxfp2IntBall") + .getOrElse( + throw new IllegalArgumentException("Mxfp2IntBall not found in config") + ) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW + val mmioReadBW = ballCommonConfig.mmioReadBW + require(mmioReadBW == 1, "Mxfp2IntBall requires exactly one MMIO read line") @public - val io = IO(new BlinkIO(b, inBW, outBW)) + val io = IO(new BlinkIO(b, inBW, outBW, mmioReadBW)) def blink: BlinkIO = io dontTouch(io) - val mxfp2intUnit: Instance[PipelinedMxfp2Int] = Instantiate(new PipelinedMxfp2Int(b)) + val mxfp2intUnit: Instance[PipelinedMxfp2Int] = Instantiate( + new PipelinedMxfp2Int(b) + ) mxfp2intUnit.io.cmdReq <> io.cmdReq mxfp2intUnit.io.cmdResp <> io.cmdResp @@ -47,5 +54,5 @@ class Mxfp2IntBall(val b: GlobalConfig) extends Module with HasBlink { io.subRobReq.valid := false.B io.subRobReq.bits := SubRobRow.tieOff(b) - io.mmioRead <> mxfp2intUnit.io.mmioRead + io.mmioRead(0) <> mxfp2intUnit.io.mmioRead } diff --git a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp4Decoder.scala b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp4Decoder.scala index 72178b0c9..292815bc7 100644 --- a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp4Decoder.scala +++ b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp4Decoder.scala @@ -15,7 +15,10 @@ object Mxfp4Decoder extends MxfpDecoder { val elemBits = 4 def dequant(raw: UInt, scale: UInt): SInt = { - require(raw.getWidth == elemBits, s"Mxfp4Decoder expects $elemBits-bit raw, got ${raw.getWidth}") + require( + raw.getWidth == elemBits, + s"Mxfp4Decoder expects $elemBits-bit raw, got ${raw.getWidth}" + ) val signedVal = raw.zext - 8.S(6.W) // SInt range -8..+7 val shifted = MxfpDecoder.scaleShift(signedVal, scale, 6) MxfpDecoder.saturateInt8(shifted) diff --git a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp6Decoder.scala b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp6Decoder.scala index a07ecd62b..942dbaeb5 100644 --- a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp6Decoder.scala +++ b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp6Decoder.scala @@ -15,7 +15,10 @@ object Mxfp6Decoder extends MxfpDecoder { val elemBits = 6 def dequant(raw: UInt, scale: UInt): SInt = { - require(raw.getWidth == elemBits, s"Mxfp6Decoder expects $elemBits-bit raw, got ${raw.getWidth}") + require( + raw.getWidth == elemBits, + s"Mxfp6Decoder expects $elemBits-bit raw, got ${raw.getWidth}" + ) val signedVal = raw.zext - 32.S(8.W) // SInt range -32..+31 val shifted = MxfpDecoder.scaleShift(signedVal, scale, 8) MxfpDecoder.saturateInt8(shifted) diff --git a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp8Decoder.scala b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp8Decoder.scala index 142a36b53..25134c470 100644 --- a/examples/balls/mxfp2int/arch/src/main/scala/Mxfp8Decoder.scala +++ b/examples/balls/mxfp2int/arch/src/main/scala/Mxfp8Decoder.scala @@ -15,7 +15,10 @@ object Mxfp8Decoder extends MxfpDecoder { val elemBits = 8 def dequant(raw: UInt, scale: UInt): SInt = { - require(raw.getWidth == elemBits, s"Mxfp8Decoder expects $elemBits-bit raw, got ${raw.getWidth}") + require( + raw.getWidth == elemBits, + s"Mxfp8Decoder expects $elemBits-bit raw, got ${raw.getWidth}" + ) val signedVal = raw.zext - 128.S(10.W) // SInt range -128..+127 val shifted = MxfpDecoder.scaleShift(signedVal, scale, 10) MxfpDecoder.saturateInt8(shifted) diff --git a/examples/balls/mxfp2int/arch/src/main/scala/MxfpDecoder.scala b/examples/balls/mxfp2int/arch/src/main/scala/MxfpDecoder.scala index f6b2058cc..19fb356f7 100644 --- a/examples/balls/mxfp2int/arch/src/main/scala/MxfpDecoder.scala +++ b/examples/balls/mxfp2int/arch/src/main/scala/MxfpDecoder.scala @@ -52,7 +52,11 @@ object MxfpDecoder { val shift = scale.zext - 127.S(10.W) val absShift = Mux(shift >= 0.S, shift, -shift)(4, 0) val widened = value.pad(32) - Mux(shift >= 0.S, (widened << absShift).asSInt, (widened >> absShift).asSInt) + Mux( + shift >= 0.S, + (widened << absShift).asSInt, + (widened >> absShift).asSInt + ) } /** @@ -62,7 +66,8 @@ object MxfpDecoder { case "MXFP4" => Mxfp4Decoder case "MXFP6" => Mxfp6Decoder case "MXFP8" => Mxfp8Decoder - case _ => throw new IllegalArgumentException(s"Unsupported MXFP format: $format") + case _ => + throw new IllegalArgumentException(s"Unsupported MXFP format: $format") } } diff --git a/examples/balls/mxfp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/mxfp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp new file mode 100644 index 000000000..4a5c3f6d8 --- /dev/null +++ b/examples/balls/mxfp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -0,0 +1,34 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace mlir::buddy { +void populateMxfp2IntBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +} // namespace mlir::buddy + +namespace { +class BankMxfp2IntPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(BankMxfp2IntOp op, + PatternRewriter &rewriter) const override { + rewriter.create(op.getLoc(), op.getInBank(), op.getOutBank(), + op.getIter(), op.getSpecial()); + rewriter.replaceOp(op, op.getOutBank()); + return success(); + } +}; +} // namespace + +void mlir::buddy::populateMxfp2IntBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { + (void)state; + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/mxfp2int/compiler/src/Dialect/Buckyball/Mxfp2IntBall.td b/examples/balls/mxfp2int/compiler/src/Dialect/Buckyball/Mxfp2IntBall.td new file mode 100644 index 000000000..75e4b813c --- /dev/null +++ b/examples/balls/mxfp2int/compiler/src/Dialect/Buckyball/Mxfp2IntBall.td @@ -0,0 +1,26 @@ +//===-- Mxfp2IntBall.td - MXFP to INT8 dequantization operations --------===// + +#ifndef MXFP2INT_BALL_OPS +#define MXFP2INT_BALL_OPS + +include "Buckyball.td" + +def Mxfp2IntOp : Buckyball_Op<"mxfp2int"> { + let summary = "Dequantize MXFP blocks to INT8"; + let description = [{ + Reads MXFP blocks from inputBankId, writes two INT8 rows per block to + outputBankId, and uses special as rs2 for format and rounding controls. + }]; + let arguments = (ins I64:$inputBankId, I64:$outputBankId, I64:$iter, + I64:$special); + let assemblyFormat = "$inputBankId `,` $outputBankId `,` $iter `,` $special attr-dict `:` type($inputBankId)"; +} + +def BankMxfp2IntOp : Buckyball_Op<"bank_mxfp2int"> { + let summary = "MXFP to INT8 dequantization using bank-SSA handles"; + let arguments = (ins I64:$inBank, I64:$outBank, I64:$iter, I64:$special); + let results = (outs I64:$outBankOut); + let assemblyFormat = "$inBank $outBank $iter $special attr-dict `:` type($inBank) type($outBank) type($iter) type($special)"; +} + +#endif // MXFP2INT_BALL_OPS diff --git a/examples/balls/mxfp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/mxfp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..94dc2f494 --- /dev/null +++ b/examples/balls/mxfp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,50 @@ +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/Pattern.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" + +using namespace mlir; +using namespace buddy::buckyball; +using namespace buddy::buckyball::legalize; + +namespace { +class Mxfp2IntLowering : public ConvertOpToLLVMPattern { +public: + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; + + LogicalResult + matchAndRewrite(Mxfp2IntOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("Mxfp2IntBall"); + Location loc = op.getLoc(); + Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInputBankId(), + cstI64(rewriter, loc, 0), + adaptor.getOutputBankId(), adaptor.getIter()); + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getSpecial(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("MXFP2INT"))); + return success(); + } +}; +} // namespace + +namespace mlir::buddy::buckyball { +void populateMxfp2IntBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB) { + (void)stable; + (void)bankDepth; + (void)rushB; + patterns.add(converter); +} + +void configureMxfp2IntBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { + (void)stable; + target.addIllegalOp(); +} +} // namespace mlir::buddy::buckyball diff --git a/examples/balls/mxfp2int/configs/Mxfp2IntBallParam.scala b/examples/balls/mxfp2int/configs/Mxfp2IntBallParam.scala index 111d2a611..b8d2a71e7 100644 --- a/examples/balls/mxfp2int/configs/Mxfp2IntBallParam.scala +++ b/examples/balls/mxfp2int/configs/Mxfp2IntBallParam.scala @@ -4,8 +4,7 @@ import framework.balldomain.configs.BallParamLoader import framework.top.GlobalConfig case class Mxfp2IntBallParam( - mxfpFormat: String -) + mxfpFormat: String) object Mxfp2IntBallParam { private val ballName = "Mxfp2IntBall" @@ -14,4 +13,5 @@ object Mxfp2IntBallParam { val tbl = BallParamLoader.ballTable(b, ballName) Mxfp2IntBallParam(mxfpFormat = BallParamLoader.str(tbl, "mxfpFormat")) } + } diff --git a/examples/balls/mxfp2int/emu/src/55_mxfp2int.rs b/examples/balls/mxfp2int/emu/src/55_mxfp2int.rs index 82dc9587d..d7e5ecfa4 100644 --- a/examples/balls/mxfp2int/emu/src/55_mxfp2int.rs +++ b/examples/balls/mxfp2int/emu/src/55_mxfp2int.rs @@ -7,20 +7,17 @@ // rs1[63:30]: iter (number of MXFP blocks to process) // // For MXFP4: each block = 32 x 4-bit FP = 16 bytes input → 32 bytes INT8 output -// Scale: per-block E8M0 (8-bit) from MMIO, indexed by block_idx -// MMIO region must be bound to wr_bank (BANK2) via bb_mmio_set beforehand. +// Scale: per-block E8M0 (8-bit) from MMIO, indexed by block_idx. // //===-----------------------------------------------------------------===// use super::super::bank::{bank_width, mmio_read_byte}; use super::decode::{rs1_b0, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct Mxfp2Int; -impl Instruction for Mxfp2Int { - const FUNCT: u32 = 55; - +impl BallInstruction for Mxfp2Int { fn exec(xs1: u64, _xs2: u64, ctx: &mut ExecContext) -> u64 { let in_bank = rs1_b0(xs1) as u32; let out_bank = rs1_b2(xs1) as u32; @@ -40,13 +37,8 @@ impl Instruction for Mxfp2Int { const BYTES_PER_OUTPUT_BLOCK: usize = 32; // 32 x INT8 for block_idx in 0..iter { - // Read E8M0 scale from MMIO (meta_bank = out_bank, rel_addr = block_idx) - let scale_e8m0 = mmio_read_byte( - ctx.mmio_banks, - ctx.mmio_region_table, - out_bank as usize, - block_idx, - ); + // Read E8M0 scale from the globally encoded MMIO address space. + let scale_e8m0 = mmio_read_byte(ctx.mmio_banks, block_idx); // Read input MXFP4 block (16 bytes = 1 bank row) let in_row_addr = block_idx; diff --git a/examples/balls/mxfp2int/emu/src/lib.rs b/examples/balls/mxfp2int/emu/src/lib.rs index aa2815ff2..ccd5becf5 100644 --- a/examples/balls/mxfp2int/emu/src/lib.rs +++ b/examples/balls/mxfp2int/emu/src/lib.rs @@ -1,6 +1,6 @@ pub(crate) use crate::inst::{decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::{BallInstruction, ExecContext}; #[path = "55_mxfp2int.rs"] mod f55_mxfp2int; @@ -17,11 +17,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("MXFP2INT") => Some(f55_mxfp2int::Mxfp2Int::exec(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -29,10 +27,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("MXFP2INT") => Some(f55_mxfp2int::Mxfp2Int::latency(xs1, xs2)), + Some(_) | None => None, } } diff --git a/examples/balls/mxfp2int/workloads/ctests/mxfp2int_test.c b/examples/balls/mxfp2int/workloads/ctests/mxfp2int_test.c index 4d409d7d6..5e8cc30cb 100644 --- a/examples/balls/mxfp2int/workloads/ctests/mxfp2int_test.c +++ b/examples/balls/mxfp2int/workloads/ctests/mxfp2int_test.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -58,7 +59,6 @@ int main(void) { const uint32_t out_bank = 1; bb_mem_alloc(in_bank, 1, 1); bb_mem_alloc(out_bank, 1, 1); - bb_mmio_set(out_bank, mmio_addr, 1); bb_mvin_mmio((uintptr_t)scales_e8m0, mmio_addr, 1, NUM_BLOCKS); bb_mvin((uintptr_t)input_mxfp4, in_bank, NUM_BLOCKS, 1); bb_mxfp2int(in_bank, out_bank, NUM_BLOCKS); @@ -77,7 +77,6 @@ int main(void) { } } - bb_mmio_set(out_bank, 0, 0); bb_mem_release(in_bank); bb_mem_release(out_bank); mmio_allocator_free(&mmio_alloc, mmio_addr, 1); diff --git a/bb-tests/workloads/lib/bbhw/isa/55_mxfp2int.c b/examples/balls/mxfp2int/workloads/isa/mxfp2int.h similarity index 87% rename from bb-tests/workloads/lib/bbhw/isa/55_mxfp2int.c rename to examples/balls/mxfp2int/workloads/isa/mxfp2int.h index 2843ac0ee..51d40cf2a 100644 --- a/bb-tests/workloads/lib/bbhw/isa/55_mxfp2int.c +++ b/examples/balls/mxfp2int/workloads/isa/mxfp2int.h @@ -1,9 +1,8 @@ #ifndef _BB_MXFP2INT_H_ #define _BB_MXFP2INT_H_ -#include "isa.h" - -#define BB_MXFP2INT_FUNC7 55 +#include +#include // Basic version: // rs1 = bank0(read) | bank2(write) | iter @@ -11,7 +10,7 @@ #define bb_mxfp2int(bank_id, wr_bank_id, iter) \ BUCKYBALL_INSTRUCTION_R_R( \ (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), 0, \ - BB_MXFP2INT_FUNC7) + BB_FUNC7(MXFP2INT)) // Extended version: // rs2 carries user-defined special field. @@ -19,6 +18,6 @@ #define bb_mxfp2int_ex(bank_id, wr_bank_id, iter, special) \ BUCKYBALL_INSTRUCTION_R_R( \ (BB_BANK0(bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), (special), \ - BB_MXFP2INT_FUNC7) + BB_FUNC7(MXFP2INT)) #endif // _BB_MXFP2INT_H_ diff --git a/examples/balls/relu/README.md b/examples/balls/relu/README.md new file mode 100644 index 000000000..f4b81faa7 --- /dev/null +++ b/examples/balls/relu/README.md @@ -0,0 +1,26 @@ +# ReluBall + +`ReluBall` computes signed INT32 `max(x, 0)` in place in one physical pbank. +It has no internal SRAM: a 128-bit register holds one pbank line between its +read and write requests. + +## Command + +`RELU(vbank, group, iter, stride)` selects physical pbank `group` in virtual +bank `vbank`. `group` is a pbank number, never a data-layout term. + +`iter` is the number of 128-bit physical lines processed in each segment. +`stride` is the distance between segments. Both are positive; `iter` is a +multiple of 16, `iter <= stride`, and `stride` divides the pbank depth. The +command processes `iter` lines at every `stride` boundary until it reaches the +end of the selected pbank. + +For every line the Ball performs one single-port sequence: + +```text +pbank read -> 4 signed int32 ReLUs -> pbank write +``` + +The selected group must be allocated. There is no output vbank, no fallback +layout, and no tail path. Tile lowering pads logical matrices to 16-aligned +shapes; compiler packing fills the final pbank chunk before issuing ReLU. diff --git a/examples/balls/relu/arch/src/main/scala/README.md b/examples/balls/relu/arch/src/main/scala/README.md deleted file mode 100644 index 3d25b1c4f..000000000 --- a/examples/balls/relu/arch/src/main/scala/README.md +++ /dev/null @@ -1,213 +0,0 @@ -# ReLU Activation Function Accelerator - -## Overview - -This directory implements Buckyball's ReLU (Rectified Linear Unit) activation accelerator, located at `arch/src/main/scala/prototype/relu`. The module performs element-wise ReLU processing on Scratchpad data by tiles (`veclane × veclane`) in a vectorized manner and writes results back. - -Core components: -- **Relu.scala**: ReLU accelerator main implementation - -## Code Structure - -``` -relu/ -└── Relu.scala - ReLU accelerator implementation -``` - -### Module Responsibilities - -**Relu.scala** (Accelerator implementation layer) -- Reads a `veclane × veclane` tile of data from Scratchpad -- Performs signed comparison-based ReLU operation on each element (negative values set to 0) -- Writes back same-sized tile with full mask write -- Provides Ball domain command interface and returns completion response/status - -## Module Description - -### Relu.scala - -**Main functionality**: - -Read input tile by tile (`veclane × veclane`) → Execute element-wise ReLU → Write output back row by row; supports `iter`-driven batch processing and pipelined workflow. - -**State machine definition**: - -```scala -val idle :: sRead :: sWrite :: complete :: Nil = Enum(4) -val state = RegInit(idle) -``` - -**Key registers**: - -```scala -// Data cache: veclane × veclane, each element width is inputType.getWidth -val regArray = RegInit( - VecInit(Seq.fill(b.veclane)( - VecInit(Seq.fill(b.veclane)(0.U(b.inputType.getWidth.W))) - )) -) - -// Counters -val readCounter = RegInit(0.U(log2Ceil(b.veclane + 1).W)) // Read request row count -val respCounter = RegInit(0.U(log2Ceil(b.veclane + 1).W)) // Read response row count -val writeCounter = RegInit(0.U(log2Ceil(b.veclane + 1).W)) // Write-back row count - -// Instruction field registers -val robid_reg = RegInit(0.U(10.W)) // Command ROB ID -val waddr_reg = RegInit(0.U(10.W)) // Write-back start row address -val wbank_reg = RegInit(0.U(log2Up(b.sp_banks).W)) // Write-back target Scratchpad bank selection -val raddr_reg = RegInit(0.U(10.W)) // Read start row address -val rbank_reg = RegInit(0.U(log2Up(b.sp_banks).W)) // Read source Scratchpad bank selection -val iter_reg = RegInit(0.U(10.W)) // Processing row count/length specified in command -val cycle_reg = RegInit(0.U(6.W)) // Tile round count (derived from iter) -val iterCnt = RegInit(0.U(32.W)) // Completed batch count - -// Write-back data and mask -val spad_w = b.veclane * b.inputType.getWidth // Packed data width per row -val writeDataReg = Reg(UInt(spad_w.W)) // Packed data to write back per row -val writeMaskReg = Reg(Vec(b.spad_mask_len, UInt(1.W))) // Write-back mask vector -``` - -**Command parsing**: - -```scala -when(io.cmdReq.fire) { - // Enter read phase and initialize round counters - state := sRead - readCounter := 0.U // Clear read request row count - writeCounter := 0.U // Clear write-back row count - - // Record command identifier - robid_reg := io.cmdReq.bits.rob_id // ROB ID (for completion response matching) - - // Output (write-back) target address: use wr_* fields - waddr_reg := io.cmdReq.bits.cmd.wr_bank_addr // Write-back start row address - wbank_reg := io.cmdReq.bits.cmd.wr_bank // Write-back target bank - - // Input (read) source address: use op1_* fields - raddr_reg := io.cmdReq.bits.cmd.op1_bank_addr // Read start row address - rbank_reg := io.cmdReq.bits.cmd.op1_bank // Read source bank - - // Iteration and rounds - iter_reg := io.cmdReq.bits.cmd.iter // Total rows to process (iteration count) - // Calculate required tile rounds for this batch: each round processes veclane rows - // cycle_reg = ceil(iter / veclane) - 1, decrements after each read/write round completes - cycle_reg := (io.cmdReq.bits.cmd.iter +& (b.veclane.U - 1.U)) / b.veclane.U - 1.U -} -``` - -**Data conversion logic (ReLU)**: - -- Read returns a packed data row of width `spad_w = veclane × inputWidth`; -- Split into `veclane` elements and perform signed comparison: `x < 0 ? 0 : x`; - -```scala -// Split + ReLU (by column) -val dataWord = io.sramRead(rbank_reg).resp.bits.data -for (col <- 0 until b.veclane) { - val hi = (col + 1) * b.inputType.getWidth - 1 - val lo = col * b.inputType.getWidth - val raw = dataWord(hi, lo) - val signed = raw.asSInt - val relu = Mux(signed < 0.S, 0.S(b.inputType.getWidth.W), signed) - regArray(respCounter)(col) := relu.asUInt -} -``` - -When writing back, repack a full row of `veclane` elements: - -```scala -// Pack regArray(rowIdx) into one row for write-back -writeDataReg := Cat((0 until b.veclane).reverse.map(j => regArray(rowIdx)(j))) -// Full mask write -for (i <- 0 until b.spad_mask_len) { writeMaskReg(i) := 1.U } -``` - -**SRAM interface**: - -```scala -val io = IO(new Bundle { - val cmdReq = Flipped(Decoupled(new BallRsIssue)) - val cmdResp = Decoupled(new BallRsComplete) - val sramRead = Vec(b.sp_banks, Flipped(new SramReadIO(b.spad_bank_entries, spad_w))) - val sramWrite = Vec(b.sp_banks, Flipped(new SramWriteIO(b.spad_bank_entries, spad_w, b.spad_mask_len))) - val status = new Status -}) -``` - -**Processing flow**: - -1. **idle**: Wait for command; parse input/output address bank/addr, iteration count `iter`, and calculate `cycle_reg` accordingly. -2. **sRead**: Issue consecutive read requests row by row for input bank/addr; after receiving data, perform element-wise ReLU and fill into `regArray`; enter write phase after accumulating `veclane` rows. -3. **sWrite**: Pack and write back row by row to consecutive addresses in `wbank` with full mask write; enter complete phase after writing `veclane` rows. -4. **complete**: When all rounds complete (`cycle_reg == 0`), issue `cmdResp` completion response; then return to `idle`. - -**Inputs/Outputs**: - -- Input: Ball domain commands (`wr_bank/wr_bank_addr`, `op1_bank/op1_bank_addr`, `iter`, etc.) -- Output: ReLU result tile after write-back, `cmdResp` completion notification -- Boundaries and constraints: - - Each round processes `veclane` rows, iteration round count derived from `iter`; - - Data elements perform signed comparison based on `b.inputType`; - - Write-back uses full mask (can be extended for partial write as needed). - -## ISA Structure - -The Ball instruction corresponding to this module performs element-wise ReLU on Scratchpad data and writes back. - -**Function**: Execute element-wise ReLU on input matrix (negative values set to 0), write back row by row to target address. - -**func7**: `0100110` 38 (corresponds to `DISA.RELU`) - -**Format**: `bb_relu rs1, rs2` - -**Operands**: - -- `rs1[spAddrLen-1:0]`: Source operand Scratchpad address (op1_spaddr) -- `rs2[spAddrLen-1:0]`: Result write-back Scratchpad address (wr_spaddr) -- `rs2[spAddrLen+9:spAddrLen]`: Iteration count (iter, row count) -- `rs2[63:spAddrLen+10]`: special/reserved field (not used by current ReLU) - -Address note: Local address of `spAddrLen` width is further split into bank and row in hardware (see LocalAddr), no need to explicitly distinguish at ISA level. - -**Operation**: Read data from Scratchpad address specified by `rs1`, perform element-wise ReLU, then write results back row by row to address specified by `rs2`, loop `iter` times. - -rs1 (input address): - -``` -┌──────────────────────────────────────────────────────┐ -│ op1_spaddr │ -│ (spAddrLen bits) │ -├──────────────────────────────────────────────────────┤ -│ [spAddrLen-1:0] │ -└──────────────────────────────────────────────────────┘ -``` - -rs2 (write-back address and iteration count): - -``` -┌──────────────────────────────────┬────────────────────┐ -│ iter (rows) │ wr_spaddr │ -│ (10 bits) │ (spAddrLen bits) │ -├──────────────────────────────────┼────────────────────┤ -│ [spAddrLen+9: spAddrLen] │ [spAddrLen-1:0] │ -└──────────────────────────────────┴────────────────────┘ -``` - -Note: During decode, `op1_spaddr` comes from `rs1`, `wr_spaddr` and `iter` come from `rs2`, remaining `special` high bits can be reserved for extension. - -## Usage - -- Place source data at Scratchpad starting position specified by `op1_bank/op1_bank_addr`, ensure each row width is `veclane × inputWidth`; -- Configure output `wr_bank/wr_bank_addr`, and element row count `iter` to process; -- After sending Ball command, wait for `cmdResp` completion; -- Can poll `status`: `ready/valid/idle/init/running/complete/iter` to get runtime information. - -### Notes - -1. **Signed comparison**: ReLU uses `asSInt` for negative value detection, negative values set to 0; ensure `b.inputType` matches upstream data convention (fixed-point/two's complement). -2. **Bandwidth and alignment**: Each read/write is one packed row (`spad_w` bits), addresses need to be row-aligned and increment consecutively. -3. **Mask strategy**: Current implementation uses full mask write; if sparse/partial write needed, extend `writeMaskReg` generation logic. -4. **Iteration and chunking**: When `iter` is not a multiple of `veclane`, `cycle_reg` handles remaining rows with ceiling rounding; pad 0 or trim at boundaries if necessary. -5. **Submodule interaction**: `sramRead/Write` ready/valid handshake needs to be consistent with Scratchpad controller; if out-of-order/multi-cycle delay exists, protect out-of-order cases in `respCounter` logic. -6. **Reset behavior**: Reset clears `regArray`, `writeDataReg`, `writeMaskReg`, facilitating reproducible simulation. diff --git a/examples/balls/relu/arch/src/main/scala/Relu.scala b/examples/balls/relu/arch/src/main/scala/Relu.scala index 54542e211..ef08bae33 100644 --- a/examples/balls/relu/arch/src/main/scala/Relu.scala +++ b/examples/balls/relu/arch/src/main/scala/Relu.scala @@ -1,211 +1,163 @@ package examples.balls.relu import chisel3._ -import chisel3.util._ -import chisel3.stage._ import chisel3.experimental.hierarchy.{instantiable, public} - -import examples.balls.vector._ -import framework.balldomain.rs.{BallRsComplete, BallRsIssue} +import chisel3.util._ import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} +import framework.balldomain.rs.{BallRsComplete, BallRsIssue} import framework.top.GlobalConfig -import examples.balls.relu.configs.ReluBallParam @instantiable -class PipelinedRelu(val b: GlobalConfig) extends Module { - val ballConfig = ReluBallParam(b) - val InputNum = ballConfig.InputNum - val inputWidth = ballConfig.inputWidth - val bankWidth = b.memDomain.bankWidth - - // Get bandwidth from config - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "ReluBall") +class Relu(val b: GlobalConfig) extends Module { + private val addressWidth = log2Up(b.memDomain.bankEntries) + private val countWidth = log2Up(b.memDomain.bankEntries + 1) + private val bankWidth = log2Up(b.memDomain.bankNum) + + private val mapping = b.ballDomain.ballIdMappings + .find(_.ballName == "ReluBall") .getOrElse(throw new IllegalArgumentException("ReluBall not found in config")) - val inBW = ballMapping.inBW - val outBW = ballMapping.outBW + + private val funct = b.ballDomain.ballISA + .find(_.mnemonic == "RELU") + .map(_.funct7) + .getOrElse(throw new IllegalArgumentException("RELU not found in ballISA")) + + require(mapping.inBW == 1, "ReluBall requires one SRAM read port") + require(mapping.outBW == 1, "ReluBall requires one SRAM write port") + require(b.memDomain.bankWidth == 128, "ReluBall requires 128-bit SRAM rows") + require(b.memDomain.bankMaskLen == 16, "ReluBall requires sixteen byte enables") @public val io = IO(new Bundle { - val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) - val cmdResp = Decoupled(new BallRsComplete(b)) - val bankRead = Vec(inBW, Flipped(new BankRead(b))) - val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) - val status = new BallStatus + val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) + val cmdResp = Decoupled(new BallRsComplete(b)) + val channelReady = Input(Bool()) + val bankRead = Vec(1, Flipped(new BankRead(b))) + val bankWrite = Vec(1, Flipped(new BankWrite(b))) + val status = new BallStatus }) - val rob_id_reg = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) - val is_sub_reg = RegInit(false.B) - val sub_rob_id_reg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) - when(io.cmdReq.fire) { - rob_id_reg := io.cmdReq.bits.rob_id - is_sub_reg := io.cmdReq.bits.is_sub - sub_rob_id_reg := io.cmdReq.bits.sub_rob_id - } - - for (i <- 0 until inBW) { - io.bankRead(i).rob_id := rob_id_reg - io.bankRead(i).ball_id := 0.U - } - for (i <- 0 until outBW) { - io.bankWrite(i).rob_id := rob_id_reg - io.bankWrite(i).ball_id := 0.U - } - - val idle :: sRead :: sWrite :: complete :: Nil = Enum(4) - val state = RegInit(idle) - - val regArray = RegInit( - VecInit(Seq.fill(InputNum)( - VecInit(Seq.fill(InputNum)(0.U(inputWidth.W))) - )) - ) - - val readCounter = RegInit(0.U(log2Ceil(InputNum + 1).W)) - val respCounter = RegInit(0.U(log2Ceil(InputNum + 1).W)) - val writeCounter = RegInit(0.U(log2Ceil(InputNum + 1).W)) - - val waddr_reg = RegInit(0.U(b.frontend.iter_len.W)) - val wbank_reg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val raddr_reg = RegInit(0.U(b.frontend.iter_len.W)) - val rbank_reg = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val iter_reg = RegInit(0.U(b.frontend.iter_len.W)) - val cycle_reg = RegInit(0.U(6.W)) - val iterCnt = RegInit(0.U(32.W)) - val writeDataReg = Reg(UInt(bankWidth.W)) - val writeMaskReg = Reg(Vec(b.memDomain.bankMaskLen, UInt(1.W))) - - for (i <- 0 until inBW) { - io.bankRead(i).io.req.valid := false.B - io.bankRead(i).io.req.bits.addr := 0.U - io.bankRead(i).io.resp.ready := false.B - } - - for (i <- 0 until outBW) { - io.bankWrite(i).io.req.valid := false.B - io.bankWrite(i).io.req.bits.addr := 0.U - io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W))) - io.bankWrite(i).io.resp.ready := false.B - } - - for (i <- 0 until inBW) { - io.bankRead(i).bank_id := rbank_reg - io.bankRead(i).group_id := 0.U - } - for (i <- 0 until outBW) { - io.bankWrite(i).bank_id := wbank_reg - io.bankWrite(i).group_id := 0.U + val Seq(idle, waitForChannels, readRequest, readResponse, writeRequest, writeResponse, complete) = Enum(7) + val state = RegInit(idle) + + val robId = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) + val isSub = RegInit(false.B) + val subRobId = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) + val bank = RegInit(0.U(bankWidth.W)) + val group = RegInit(0.U(bankWidth.W)) + val iter = RegInit(0.U(countWidth.W)) + val stride = RegInit(0.U(countWidth.W)) + val segment = RegInit(0.U(addressWidth.W)) + val line = RegInit(0.U(addressWidth.W)) + val word = Reg(UInt(128.W)) + + val relu = Wire(Vec(4, UInt(32.W))) + for (lane <- 0 until 4) { + val value = word(32 * lane + 31, 32 * lane).asSInt + relu(lane) := Mux(value < 0.S, 0.U, value.asUInt) } + val address = segment + line io.cmdReq.ready := state === idle - io.cmdResp.valid := false.B - io.cmdResp.bits.rob_id := rob_id_reg - io.cmdResp.bits.is_sub := is_sub_reg - io.cmdResp.bits.sub_rob_id := sub_rob_id_reg + io.cmdResp.valid := state === complete + io.cmdResp.bits.rob_id := robId + io.cmdResp.bits.is_sub := isSub + io.cmdResp.bits.sub_rob_id := subRobId + + io.bankRead(0).rob_id := robId + io.bankRead(0).ball_id := 0.U + io.bankRead(0).bank_id := bank + io.bankRead(0).group_id := group + io.bankRead(0).io.req.valid := false.B + io.bankRead(0).io.req.bits.addr := address + io.bankRead(0).io.resp.ready := false.B + + io.bankWrite(0).rob_id := robId + io.bankWrite(0).ball_id := 0.U + io.bankWrite(0).bank_id := bank + io.bankWrite(0).group_id := group + io.bankWrite(0).io.req.valid := false.B + io.bankWrite(0).io.req.bits.addr := address + io.bankWrite(0).io.req.bits.data := Cat(relu.reverse) + io.bankWrite(0).io.req.bits.mask := VecInit(Seq.fill(16)(true.B)) + io.bankWrite(0).io.resp.ready := false.B switch(state) { is(idle) { when(io.cmdReq.fire) { - state := sRead - readCounter := 0.U - respCounter := 0.U - writeCounter := 0.U - waddr_reg := 0.U - wbank_reg := io.cmdReq.bits.cmd.wr_bank - raddr_reg := 0.U - rbank_reg := io.cmdReq.bits.cmd.op1_bank - iter_reg := io.cmdReq.bits.cmd.iter - cycle_reg := (io.cmdReq.bits.cmd.iter +& (InputNum.U - 1.U)) / InputNum.U - 1.U - } - when(cycle_reg =/= 0.U) { - state := sRead - readCounter := 0.U - writeCounter := 0.U - respCounter := 0.U - waddr_reg := waddr_reg + InputNum.U - raddr_reg := raddr_reg + InputNum.U - cycle_reg := cycle_reg - 1.U + val command = io.cmdReq.bits.cmd + assert(command.funct7 === funct.U(7.W), "ReluBall funct7 must be RELU") + assert(command.wr_bank === 0.U, "ReluBall bank2 must be zero") + assert(command.op1_col =/= 0.U && command.op2_bank < command.op1_col, "ReluBall group is not allocated in bank") + assert(command.iter =/= 0.U && command.iter(3, 0) === 0.U, "ReluBall iter must be a positive multiple of 16") + assert( + command.special =/= 0.U && command.special <= b.memDomain.bankEntries.U, + "ReluBall stride must fit in one physical bank" + ) + assert( + b.memDomain.bankEntries.U % command.special === 0.U, + "ReluBall stride must divide the physical bank depth" + ) + assert(command.iter <= command.special, "ReluBall iter must not exceed stride") + + robId := io.cmdReq.bits.rob_id + isSub := io.cmdReq.bits.is_sub + subRobId := io.cmdReq.bits.sub_rob_id + bank := command.op1_bank + group := command.op2_bank + iter := command.iter + stride := command.special + segment := 0.U + line := 0.U + state := waitForChannels } } - is(sRead) { - io.bankRead(0).io.resp.ready := true.B - - io.bankRead(0).io.req.valid := (readCounter < InputNum.U) - io.bankRead(0).io.req.bits.addr := raddr_reg + readCounter - - when(io.bankRead(0).io.req.fire) { - readCounter := readCounter + 1.U - } + is(waitForChannels) { + when(io.channelReady)(state := readRequest) + } - val dataWord = io.bankRead(0).io.resp.bits.data + is(readRequest) { + io.bankRead(0).io.req.valid := true.B + when(io.bankRead(0).io.req.fire)(state := readResponse) + } + is(readResponse) { + io.bankRead(0).io.resp.ready := true.B when(io.bankRead(0).io.resp.fire) { - for (col <- 0 until InputNum) { - val hi = (col + 1) * inputWidth - 1 - val lo = col * inputWidth - val raw = dataWord(hi, lo) - val signed = raw.asSInt - val relu = Mux(signed < 0.S, 0.S(inputWidth.W), signed) - regArray(respCounter)(col) := relu.asUInt - } - respCounter := respCounter + 1.U - } - - when(io.bankRead(0).io.resp.fire && respCounter === (InputNum - 1).U) { - state := sWrite - writeDataReg := Cat((0 until InputNum).reverse.map(j => regArray(0)(j))) - for (i <- 0 until b.memDomain.bankMaskLen) { - writeMaskReg(i) := 1.U(1.W) - } + word := io.bankRead(0).io.resp.bits.data + state := writeRequest } } - is(sWrite) { - val hasMore = writeCounter < InputNum.U - - io.bankWrite(0).io.req.valid := hasMore - io.bankWrite(0).io.req.bits.addr := waddr_reg + writeCounter - io.bankWrite(0).io.req.bits.data := writeDataReg - io.bankWrite(0).io.req.bits.mask := writeMaskReg - io.bankWrite(0).io.resp.ready := true.B + is(writeRequest) { + io.bankWrite(0).io.req.valid := true.B + when(io.bankWrite(0).io.req.fire)(state := writeResponse) + } - when(io.bankWrite(0).io.req.fire) { - when(writeCounter === (InputNum - 1).U) { - state := complete + is(writeResponse) { + io.bankWrite(0).io.resp.ready := true.B + when(io.bankWrite(0).io.resp.fire) { + when(line +& 1.U === iter) { + when(segment +& stride === b.memDomain.bankEntries.U) { + state := complete + }.otherwise { + segment := segment + stride + line := 0.U + state := readRequest + } }.otherwise { - val nextCnt = writeCounter + 1.U - writeCounter := nextCnt - writeDataReg := Cat((0 until InputNum).reverse.map(j => regArray(nextCnt)(j))) + line := line + 1.U + state := readRequest } } } is(complete) { - io.bankWrite(0).io.resp.ready := true.B - when(cycle_reg === 0.U) { - io.cmdResp.valid := true.B - io.cmdResp.bits.rob_id := rob_id_reg - when(io.cmdResp.fire) { - iterCnt := iterCnt + 1.U - } - } - state := idle + when(io.cmdResp.fire)(state := idle) } } - io.status.idle := (state === idle) - io.status.running := (state === sRead) || (state === sWrite) - - when(reset.asBool) { - for (i <- 0 until InputNum) { - for (j <- 0 until InputNum) { - regArray(i)(j) := 0.U - } - } - writeDataReg := 0.U - for (i <- 0 until b.memDomain.bankMaskLen) { - writeMaskReg(i) := 0.U - } - } + io.status.idle := state === idle + io.status.running := state =/= idle } diff --git a/examples/balls/relu/arch/src/main/scala/ReluBall.scala b/examples/balls/relu/arch/src/main/scala/ReluBall.scala index 17ad605c3..91024b258 100644 --- a/examples/balls/relu/arch/src/main/scala/ReluBall.scala +++ b/examples/balls/relu/arch/src/main/scala/ReluBall.scala @@ -1,50 +1,33 @@ package examples.balls.relu import chisel3._ -import chisel3.util._ import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} -import framework.balldomain.blink.{BallStatus, BlinkIO, HasBallStatus, HasBlink, SubRobRow} +import framework.balldomain.blink.{BlinkIO, HasBallStatus, HasBlink, SubRobRow} import framework.balldomain.blink.mmio.MmioRead -import examples.balls.relu.PipelinedRelu import framework.top.GlobalConfig -/** - * ReluBall - A ReLU computation Ball that complies with the blink protocol. - * Behavior: Read data from Scratchpad, perform element-wise ReLU (set negative values to 0), - * then write back to Scratchpad. - */ @instantiable -class ReluBall(val b: GlobalConfig) extends Module with HasBlink { +class ReluBall(val b: GlobalConfig) extends Module with HasBlink with HasBallStatus { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "ReluBall") + private val mapping = b.ballDomain.ballIdMappings + .find(_.ballName == "ReluBall") .getOrElse(throw new IllegalArgumentException("ReluBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW @public - val io = IO(new BlinkIO(b, inBW, outBW)) + val io = IO(new BlinkIO(b, mapping.inBW, mapping.outBW)) def blink: BlinkIO = io + def status = io.status dontTouch(io) - val reluUnit: Instance[PipelinedRelu] = Instantiate(new PipelinedRelu(b)) - - reluUnit.io.cmdReq <> io.cmdReq - reluUnit.io.cmdResp <> io.cmdResp - - for (i <- 0 until inBW) { - reluUnit.io.bankRead(i) <> io.bankRead(i) - } - - for (i <- 0 until outBW) { - reluUnit.io.bankWrite(i) <> io.bankWrite(i) - } - - io.status <> reluUnit.io.status - - io.subRobReq.valid := false.B - io.subRobReq.bits := SubRobRow.tieOff(b) - - // MMIO: this Ball does not consume MMIO metadata + val unit: Instance[Relu] = Instantiate(new Relu(b)) + unit.io.cmdReq <> io.cmdReq + unit.io.cmdResp <> io.cmdResp + unit.io.channelReady := io.channelReady + unit.io.bankRead(0) <> io.bankRead(0) + unit.io.bankWrite(0) <> io.bankWrite(0) + io.status <> unit.io.status + io.subRobReq.valid := false.B + io.subRobReq.bits := SubRobRow.tieOff(b) MmioRead.tieOff(io.mmioRead) } diff --git a/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp new file mode 100644 index 000000000..8ecc276ff --- /dev/null +++ b/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -0,0 +1,34 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace mlir::buddy { +void populateReluBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +} + +namespace { +class BankReluPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(BankReluOp op, + PatternRewriter &rewriter) const override { + rewriter.create(op.getLoc(), op.getBank(), op.getGroup(), + op.getIter(), op.getStride()); + rewriter.replaceOp(op, op.getBank()); + return success(); + } +}; +} // namespace + +void mlir::buddy::populateReluBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { + (void)state; + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp b/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp new file mode 100644 index 000000000..431cb3f69 --- /dev/null +++ b/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp @@ -0,0 +1,113 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Target/BuckyballTargetRegistry.h" +#include "Utils/BankUtils.h" + +#include + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace { + +constexpr int64_t kValuesPerLine = 4; + +class ReluMatrixToBankSSAPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(ReluMatrixOp op, + PatternRewriter &b) const override { + auto inputType = dyn_cast(op.getInput().getType()); + auto outputType = dyn_cast(op.getOutput().getType()); + if (!inputType || !outputType || !inputType.hasStaticShape() || + inputType != outputType || !inputType.getElementType().isInteger(32)) + return op.emitError("requires matching static memref"); + + int64_t rows = inputType.getShape()[0]; + int64_t columns = inputType.getShape()[1]; + if (rows <= 0 || columns <= 0 || rows % 16 || columns % 16) + return op.emitError("requires positive 16-aligned dimensions"); + + const auto &target = buckyball_target::getBuckyballTarget(); + if (target.bankWidthBits != 128 || target.bankDepth <= 0 || + target.bankDepth % 16) + return op.emitError( + "ReluBall requires a 128-bit bank with 16-aligned depth"); + + Location loc = op.getLoc(); + int64_t lines = target.bankDepth; + int64_t values = lines * kValuesPerLine; + int64_t total = rows * columns; + Value bank = allocBank(b, loc, 1, 1); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value four = b.create(loc, kValuesPerLine); + Value columnsValue = b.create(loc, columns); + auto packType = MemRefType::get({lines, kValuesPerLine}, b.getI32Type()); + + for (int64_t base = 0; base < total; base += values) { + int64_t count = std::min(values, total - base); + Value inputPack = b.create(loc, packType); + Value outputPack = b.create(loc, packType); + Value zeroValue = b.create(loc, 0, 32); + b.create(loc, zeroValue, inputPack); + + Value countValue = b.create(loc, count); + Value baseValue = b.create(loc, base); + auto copyIn = b.create(loc, zero, countValue, one); + b.setInsertionPointToStart(copyIn.getBody()); + Value index = copyIn.getInductionVar(); + Value source = b.create(loc, baseValue, index); + Value row = b.create(loc, source, columnsValue); + Value column = b.create(loc, source, columnsValue); + Value line = b.create(loc, index, four); + Value lane = b.create(loc, index, four); + Value value = + b.create(loc, op.getInput(), ValueRange{row, column}); + b.create(loc, value, inputPack, ValueRange{line, lane}); + b.setInsertionPointAfter(copyIn); + + Value loaded = mvinBank(b, loc, inputPack, bank, lines); + Value result = b.create( + loc, bank.getType(), loaded, createI64Const(b, loc, 0), + createI64Const(b, loc, lines), createI64Const(b, loc, lines)); + bank = mvoutBank(b, loc, outputPack, result, lines); + b.create(loc); + + auto copyOut = b.create(loc, zero, countValue, one); + b.setInsertionPointToStart(copyOut.getBody()); + index = copyOut.getInductionVar(); + source = b.create(loc, baseValue, index); + row = b.create(loc, source, columnsValue); + column = b.create(loc, source, columnsValue); + line = b.create(loc, index, four); + lane = b.create(loc, index, four); + value = b.create(loc, outputPack, ValueRange{line, lane}); + b.create(loc, value, op.getOutput(), + ValueRange{row, column}); + b.setInsertionPointAfter(copyOut); + b.create(loc, inputPack); + b.create(loc, outputPack); + } + + releaseBank(b, loc, bank); + b.eraseOp(op); + return success(); + } +}; + +} // namespace + +void mlir::buddy::populateReluBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns) { + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/relu/compiler/src/Conversion/LowerTileToBuckyball/TileReluPatterns.cpp b/examples/balls/relu/compiler/src/Conversion/LowerTileToBuckyball/TileReluPatterns.cpp new file mode 100644 index 000000000..9c49d9ec5 --- /dev/null +++ b/examples/balls/relu/compiler/src/Conversion/LowerTileToBuckyball/TileReluPatterns.cpp @@ -0,0 +1,91 @@ +#include "Buckyball/BuckyballOps.h" +#include "Tile/TileOps.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" + +using namespace mlir; +using namespace ::buddy::buckyball; +namespace tile = ::buddy::tile; + +namespace { +class TileReluLowering : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(tile::TileReluOp op, + PatternRewriter &b) const override { + Value input = op.getInput(); + Value output = op.getOutput(); + auto inputType = dyn_cast(input.getType()); + auto outputType = dyn_cast(output.getType()); + if (!inputType || !outputType || !inputType.hasStaticShape() || + inputType != outputType || inputType.getRank() != 2 || + !inputType.getElementType().isInteger(32)) + return op.emitError("requires matching static memref"); + + int64_t rows = inputType.getShape()[0]; + int64_t columns = inputType.getShape()[1]; + if (rows <= 0 || columns <= 0) + return op.emitError("requires positive dimensions"); + + int64_t paddedRows = llvm::alignTo(rows, 16); + int64_t paddedColumns = llvm::alignTo(columns, 16); + Location loc = op.getLoc(); + if (paddedRows == rows && paddedColumns == columns) { + b.create(loc, input, output); + b.eraseOp(op); + return success(); + } + + auto paddedType = + MemRefType::get({paddedRows, paddedColumns}, b.getI32Type()); + Value paddedInput = b.create(loc, paddedType); + Value paddedOutput = b.create(loc, paddedType); + Value zeroValue = b.create(loc, 0, 32); + b.create(loc, zeroValue, paddedInput); + + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value rowLimit = b.create(loc, rows); + Value columnLimit = b.create(loc, columns); + auto copyInputRows = b.create(loc, zero, rowLimit, one); + b.setInsertionPointToStart(copyInputRows.getBody()); + Value row = copyInputRows.getInductionVar(); + auto copyInputColumns = b.create(loc, zero, columnLimit, one); + b.setInsertionPointToStart(copyInputColumns.getBody()); + Value column = copyInputColumns.getInductionVar(); + Value value = b.create(loc, input, ValueRange{row, column}); + b.create(loc, value, paddedInput, ValueRange{row, column}); + b.setInsertionPointAfter(copyInputRows); + + b.create(loc, paddedInput, paddedOutput); + + auto copyOutputRows = b.create(loc, zero, rowLimit, one); + b.setInsertionPointToStart(copyOutputRows.getBody()); + row = copyOutputRows.getInductionVar(); + auto copyOutputColumns = b.create(loc, zero, columnLimit, one); + b.setInsertionPointToStart(copyOutputColumns.getBody()); + column = copyOutputColumns.getInductionVar(); + value = + b.create(loc, paddedOutput, ValueRange{row, column}); + b.create(loc, value, output, ValueRange{row, column}); + b.setInsertionPointAfter(copyOutputRows); + + b.create(loc, paddedInput); + b.create(loc, paddedOutput); + b.eraseOp(op); + return success(); + } +}; +} // namespace + +namespace mlir::buddy { +void populateReluBallTileLoweringPatterns(RewritePatternSet &patterns, int64_t, + int64_t, int64_t) { + patterns.add(patterns.getContext()); +} +} // namespace mlir::buddy diff --git a/examples/balls/relu/compiler/src/Dialect/Buckyball/ReluBall.td b/examples/balls/relu/compiler/src/Dialect/Buckyball/ReluBall.td index cd155c95d..70fafee38 100644 --- a/examples/balls/relu/compiler/src/Dialect/Buckyball/ReluBall.td +++ b/examples/balls/relu/compiler/src/Dialect/Buckyball/ReluBall.td @@ -1,34 +1,31 @@ //===-- ReluBall.td - ReluBall operations ------------------*- tablegen -*-===// -// -// ReluBall operations for ReLU activation function. -// -//===----------------------------------------------------------------------===// #ifndef RELUBALL_OPS #define RELUBALL_OPS -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" +include "Buckyball.td" -//===----------------------------------------------------------------------===// -// ReluBall operations -//===----------------------------------------------------------------------===// +def ReluMatrixOp : Buckyball_Op<"relu_matrix"> { + let summary = "Logical INT32 ReLU matrix tile"; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$input, + MemRefRankOf<[AnyType], [2]>:$output); + let assemblyFormat = [{$input $output attr-dict `:` + type($input) type($output)}]; +} def ReluOp : Buckyball_Op<"relu"> { - let summary = "ReLU activation function"; - let description = [{ - Applies ReLU (Rectified Linear Unit) activation: max(0, x). - Corresponds to bb_relu instruction (funct7=50). - Belongs to ReluBall (bid=1). - }]; - let arguments = (ins I64:$inputBankId, I64:$outputBankId, - I64:$depth, I64:$stride); - let assemblyFormat = "$inputBankId `,` $outputBankId `,` " - "$depth `,` $stride attr-dict `:` " - "type($inputBankId)"; + let summary = "In-place INT32 ReLU on one physical bank"; + let arguments = (ins I64:$bankId, I64:$group, I64:$iter, I64:$stride); + let assemblyFormat = "$bankId `,` $group `,` $iter `,` $stride attr-dict `:` " + "type($bankId)"; } -def ReluIntrOp : Buckyball_IntrOpBase<"relu">, - Arguments<(ins LLVM_Type, LLVM_Type)>; +def BankReluOp : Buckyball_Op<"bank_relu"> { + let summary = "In-place INT32 ReLU with a virtual bank handle"; + let arguments = (ins I64:$bank, I64:$group, I64:$iter, I64:$stride); + let results = (outs I64:$bankOut); + let assemblyFormat = "$bank $group $iter $stride attr-dict `:` " + "type($bank) type($group) type($iter) type($stride)"; +} #endif // RELUBALL_OPS diff --git a/examples/balls/relu/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/relu/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index a911a75de..20a47c724 100644 --- a/examples/balls/relu/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/balls/relu/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -4,6 +4,7 @@ #include "Buckyball/BuckyballOps.h" #include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; @@ -11,43 +12,42 @@ using namespace buddy::buckyball::legalize; namespace { struct ReluLowering : public ConvertOpToLLVMPattern { - ReluLowering(LLVMTypeConverter &converter, bool stable) - : ConvertOpToLLVMPattern(converter), stable(stable) {} + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; LogicalResult matchAndRewrite(ReluOp op, OpAdaptor adaptor, ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("ReluBall"); Location loc = op.getLoc(); - Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInputBankId(), - cstI64(rewriter, loc, 0), - adaptor.getOutputBankId(), adaptor.getDepth()); - if (stable) { - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getStride()); - return success(); - } - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getStride(), - rewriter.getI32IntegerAttr(50)); + Value rs1 = + packRs1BanksIter(rewriter, loc, adaptor.getBankId(), adaptor.getGroup(), + cstI64(rewriter, loc, 0), adaptor.getIter()); + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getStride(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("RELU"))); return success(); } - -private: - bool stable = false; }; } // namespace namespace mlir::buddy::buckyball { -void populateReluLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable) { - patterns.add(converter, stable); +void populateReluBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, + RewritePatternSet &patterns, + bool, int64_t, bool) { + patterns.add(converter); +} + +// Toy still calls the hook directly with its older three-argument shape. +void populateReluBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, + RewritePatternSet &patterns, + bool stable) { + populateReluBallLegalizeForLLVMExportPatterns(converter, patterns, stable, 0, + false); } -void configureReluLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable) { - if (stable) - target.addLegalOp(); - else - target.addIllegalOp(); - target.addIllegalOp(); +void configureReluBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool) { + target.addIllegalOp(); } } // namespace mlir::buddy::buckyball diff --git a/examples/balls/relu/configs/ReluBallParam.scala b/examples/balls/relu/configs/ReluBallParam.scala deleted file mode 100644 index 5aa3e4c03..000000000 --- a/examples/balls/relu/configs/ReluBallParam.scala +++ /dev/null @@ -1,24 +0,0 @@ -package examples.balls.relu.configs - -import framework.balldomain.configs.BallParamLoader -import framework.top.GlobalConfig -import upickle.default._ - -case class ReluBallParam( - InputNum: Int, - inputWidth: Int -) - -object ReluBallParam { - implicit val rw: ReadWriter[ReluBallParam] = macroRW - - private val ballName = "ReluBall" - - def apply(b: GlobalConfig): ReluBallParam = { - val tbl = BallParamLoader.ballTable(b, ballName) - ReluBallParam( - InputNum = BallParamLoader.int(tbl, "InputNum"), - inputWidth = BallParamLoader.int(tbl, "inputWidth") - ) - } -} diff --git a/examples/balls/relu/configs/default.toml b/examples/balls/relu/configs/default.toml index bd743daeb..80321fad9 100644 --- a/examples/balls/relu/configs/default.toml +++ b/examples/balls/relu/configs/default.toml @@ -1,5 +1,2 @@ -# Relu Ball Configuration - +# ReluBall has no implementation-specific parameters. [ball] -InputNum = 16 -inputWidth = 8 diff --git a/examples/balls/relu/emu/src/50_relu.rs b/examples/balls/relu/emu/src/50_relu.rs index 044cf45b6..2f1e11143 100644 --- a/examples/balls/relu/emu/src/50_relu.rs +++ b/examples/balls/relu/emu/src/50_relu.rs @@ -1,69 +1,61 @@ -//===- 50_relu.rs - RELU instruction ----------------------------------------===// - use super::super::bank::{bank_num, bank_size}; -use super::decode::{pbank, rs1_b0, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; +use super::decode::{pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; +use super::instruction::{BallInstruction, ExecContext}; pub struct Relu; -impl Instruction for Relu { - const FUNCT: u32 = 50; - - fn exec(xs1: u64, _xs2: u64, ctx: &mut ExecContext) -> u64 { - let src = rs1_b0(xs1); - let dst = rs1_b2(xs1); - let depth = rs1_iter(xs1) as usize; +impl BallInstruction for Relu { + fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { + let bank = rs1_b0(xs1); + let group = rs1_b1(xs1); + let iter = rs1_iter(xs1) as usize; + let stride = xs2 as usize; - if src >= bank_num() as u64 || dst >= bank_num() as u64 { - panic!("relu: invalid bank_id"); + if rs1_b2(xs1) != 0 { + panic!("relu: bank2 must be zero"); } - - let sc = ctx.cfgs[src as usize]; - let dc = ctx.cfgs[dst as usize]; - if !sc.allocated || !dc.allocated { + if bank >= bank_num() as u64 { + panic!("relu: invalid bank id"); + } + let config = ctx.cfgs[bank as usize]; + if !config.allocated { panic!("relu: bank not allocated"); } - - let ps = pbank(ctx.bank_map, src); - let pd = pbank(ctx.bank_map, dst); - - if sc.cols == 1 && dc.cols == 1 { - for i in 0..depth { - let base = i * 16; - if base + 16 > bank_size() { - panic!("relu: out of range"); - } - for j in 0..16 { - let v = ctx.banks[ps][base + j] as i8; - ctx.banks[pd][base + j] = if v < 0 { 0 } else { v as u8 }; - } - } - return 0; + if group >= config.cols as u64 { + panic!("relu: group is not allocated in bank"); + } + let lines = bank_size() / 16; + if iter == 0 || iter % 16 != 0 || iter > stride { + panic!("relu: iter must be a 16-aligned positive value no larger than stride"); + } + if stride == 0 || lines % stride != 0 { + panic!("relu: stride must divide the physical bank depth"); } - if sc.cols == 4 && dc.cols == 4 { - for i in 0..depth { - let base = i * 64; - if base + 64 > bank_size() { - panic!("relu: out of range"); - } - for j in 0..16 { - let off = base + j * 4; - let v = i32::from_le_bytes(ctx.banks[ps][off..off + 4].try_into().unwrap()); - let o = if v < 0 { 0 } else { v }; - ctx.banks[pd][off..off + 4].copy_from_slice(&o.to_le_bytes()); + let physical = pbank_group(ctx.bank_map, bank, group); + for segment in (0..lines).step_by(stride) { + for line in 0..iter { + let offset = (segment + line) * 16; + for lane in 0..4 { + let byte = offset + lane * 4; + let value = i32::from_le_bytes( + ctx.banks[physical][byte..byte + 4].try_into().unwrap(), + ); + ctx.banks[physical][byte..byte + 4] + .copy_from_slice(&value.max(0).to_le_bytes()); } } - return 0; } - - panic!( - "relu: unsupported layout src_cols={} dst_cols={}", - sc.cols, dc.cols - ); + 0 } - fn latency(xs1: u64, _xs2: u64) -> u64 { - rs1_iter(xs1).max(1) + fn latency(xs1: u64, xs2: u64) -> u64 { + let iter = rs1_iter(xs1); + let stride = xs2; + let lines = bank_size() as u64 / 16; + if iter == 0 || iter % 16 != 0 || iter > stride || stride == 0 || lines % stride != 0 { + panic!("relu: invalid iter or stride"); + } + lines / stride * iter * 4 } } diff --git a/examples/balls/relu/emu/src/lib.rs b/examples/balls/relu/emu/src/lib.rs index 907232237..d81401312 100644 --- a/examples/balls/relu/emu/src/lib.rs +++ b/examples/balls/relu/emu/src/lib.rs @@ -1,6 +1,6 @@ pub(crate) use crate::inst::{decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::{BallInstruction, ExecContext}; #[path = "50_relu.rs"] mod f50_relu; @@ -17,11 +17,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::exec(xs1, xs2, ctx)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("RELU") => Some(f50_relu::Relu::exec(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -29,10 +27,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("RELU") => Some(f50_relu::Relu::latency(xs1, xs2)), + Some(_) | None => None, } } diff --git a/examples/balls/relu/workloads/ctests/CMakeLists.txt b/examples/balls/relu/workloads/ctests/CMakeLists.txt index 7d4ee4962..e2c3c36db 100644 --- a/examples/balls/relu/workloads/ctests/CMakeLists.txt +++ b/examples/balls/relu/workloads/ctests/CMakeLists.txt @@ -1,3 +1,4 @@ add_buckyball_ctests( relu_test.c + relu_bank_256x16_test.c ) diff --git a/examples/balls/relu/workloads/ctests/relu_bank_256x16_test.c b/examples/balls/relu/workloads/ctests/relu_bank_256x16_test.c new file mode 100644 index 000000000..138091714 --- /dev/null +++ b/examples/balls/relu/workloads/ctests/relu_bank_256x16_test.c @@ -0,0 +1,57 @@ +#include "buckyball.h" +#include +#include +#include +#include + +#define ROWS 256 +#define COLUMNS 16 +#define ITER 1024 + +static result_t input[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t packed_input[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t packed_output[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t output[ROWS * COLUMNS] __attribute__((aligned(64))); + +static void pack(const result_t *source, result_t *destination) { + for (int row = 0; row < ROWS; ++row) + for (int column = 0; column < COLUMNS; ++column) { + int line = row * 4 + column / 4; + destination[line * 4 + column % 4] = source[row * COLUMNS + column]; + } +} + +static void unpack(const result_t *source, result_t *destination) { + for (int row = 0; row < ROWS; ++row) + for (int column = 0; column < COLUMNS; ++column) { + int line = row * 4 + column / 4; + destination[row * COLUMNS + column] = source[line * 4 + column % 4]; + } +} + +int main(void) { + const uint32_t bank = 0; + + for (int i = 0; i < ROWS * COLUMNS; ++i) + input[i] = (i * 97 % 401) - 200; + + pack(input, packed_input); + bb_mem_alloc(bank, 1, 1); + bb_mvin((uintptr_t)packed_input, bank, ITER, 1); + bb_relu(bank, 0, ITER, ITER); + bb_mvout((uintptr_t)packed_output, bank, ITER, 1); + bb_fence(); + bb_mem_release(bank); + unpack(packed_output, output); + + for (int i = 0; i < ROWS * COLUMNS; ++i) { + result_t expected = input[i] < 0 ? 0 : input[i]; + if (output[i] != expected) { + printf("relu_bank_256x16 mismatch at %d: got %d expected %d\n", i, + output[i], expected); + return 1; + } + } + printf("relu_bank_256x16 PASSED\n"); + return 0; +} diff --git a/examples/balls/relu/workloads/ctests/relu_test.c b/examples/balls/relu/workloads/ctests/relu_test.c index 88e664e68..3d564b98f 100644 --- a/examples/balls/relu/workloads/ctests/relu_test.c +++ b/examples/balls/relu/workloads/ctests/relu_test.c @@ -1,40 +1,59 @@ #include "buckyball.h" #include #include +#include #include -#define ROWS 4 -#define COLS 16 +#define ROWS 16 +#define COLUMNS 16 +#define ITER (ROWS * COLUMNS / 4) -static elem_t input[ROWS * COLS] __attribute__((aligned(64))) = { - -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, - -17, -16, -15, -14, -13, -12, -11, 10, 11, 12, 13, 14, 15, 16, 17, 18, - -27, -26, -25, -24, -23, -22, -21, 20, 21, 22, 23, 24, 25, 26, 27, 28, - -37, -36, -35, -34, -33, -32, -31, 30, 31, 32, 33, 34, 35, 36, 37, 38, -}; +static result_t input[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t packed_input[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t packed_output[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t output[ROWS * COLUMNS] __attribute__((aligned(64))); -static elem_t expected[ROWS * COLS] __attribute__((aligned(64))); -static elem_t output[ROWS * COLS] __attribute__((aligned(64))); +static void pack(const result_t *source, result_t *destination) { + for (int row = 0; row < ROWS; ++row) { + for (int column = 0; column < COLUMNS; ++column) { + int line = row * 4 + column / 4; + destination[line * 4 + column % 4] = source[row * COLUMNS + column]; + } + } +} + +static void unpack(const result_t *source, result_t *destination) { + for (int row = 0; row < ROWS; ++row) { + for (int column = 0; column < COLUMNS; ++column) { + int line = row * 4 + column / 4; + destination[row * COLUMNS + column] = source[line * 4 + column % 4]; + } + } +} int main(void) { -#ifdef MULTICORE - multicore(MULTICORE); -#endif + const uint32_t bank = 0; - for (int i = 0; i < ROWS * COLS; i++) { - expected[i] = input[i] < 0 ? 0 : input[i]; - } + for (int i = 0; i < ROWS * COLUMNS; ++i) + input[i] = (i * 97 % 401) - 200; - const uint32_t src = 0; - const uint32_t dst = 1; - bb_mem_alloc(src, 1, 1); - bb_mem_alloc(dst, 1, 1); - bb_mvin((uintptr_t)input, src, ROWS, 1); - bb_relu(src, dst, ROWS); - bb_mvout((uintptr_t)output, dst, ROWS, 1); + pack(input, packed_input); + bb_mem_alloc(bank, 1, 1); + bb_mvin((uintptr_t)packed_input, bank, ITER, 1); + bb_relu(bank, 0, ITER, ITER); + bb_mvout((uintptr_t)packed_output, bank, ITER, 1); bb_fence(); + bb_mem_release(bank); + unpack(packed_output, output); - int passed = compare_i8_matrices(output, expected, ROWS, COLS); - printf("ReLU test %s\n", passed ? "PASSED" : "FAILED"); - return passed ? 0 : 1; + for (int i = 0; i < ROWS * COLUMNS; ++i) { + result_t expected = input[i] < 0 ? 0 : input[i]; + if (output[i] != expected) { + printf("relu mismatch at %d: got %d expected %d\n", i, output[i], + expected); + return 1; + } + } + printf("relu_test PASSED\n"); + return 0; } diff --git a/examples/balls/relu/workloads/isa/relu.h b/examples/balls/relu/workloads/isa/relu.h new file mode 100644 index 000000000..7f6589b67 --- /dev/null +++ b/examples/balls/relu/workloads/isa/relu.h @@ -0,0 +1,12 @@ +#ifndef _BB_RELU_H_ +#define _BB_RELU_H_ + +#include +#include + +#define bb_relu(bank_id, group, iter, stride) \ + BUCKYBALL_INSTRUCTION_R_R( \ + (BB_BANK0(bank_id) | BB_BANK1(group) | BB_ITER(iter)), (stride), \ + BB_FUNC7(RELU)) + +#endif // _BB_RELU_H_ diff --git a/examples/balls/relu/workloads/mlir_tests/CMakeLists.txt b/examples/balls/relu/workloads/mlir_tests/CMakeLists.txt new file mode 100644 index 000000000..5275add61 --- /dev/null +++ b/examples/balls/relu/workloads/mlir_tests/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(bank) diff --git a/examples/balls/relu/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/relu/workloads/mlir_tests/bank/CMakeLists.txt new file mode 100644 index 000000000..8d755b482 --- /dev/null +++ b/examples/balls/relu/workloads/mlir_tests/bank/CMakeLists.txt @@ -0,0 +1,8 @@ +set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) +set(BUCKYBALL_MLIR_TEST_PREFIX bank) + +add_buckyball_mlir_contract(relu_bank_256x16_i32_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) + +add_buckyball_mlir_test(relu_bank_256x16_i32 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32.mlir b/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32.mlir new file mode 100644 index 000000000..fbf2ff510 --- /dev/null +++ b/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32.mlir @@ -0,0 +1,45 @@ +// Full physical-bank ReLU test. One group holds 1024 128-bit lines. + +func.func private @check_result(memref<1024x4xi32>) -> () + +func.func @main() -> i8 { + %zeroI8 = arith.constant 0 : i8 + %zero = arith.constant 0 : index + %one = arith.constant 1 : index + %group = arith.constant 0 : i64 + %lines = arith.constant 1024 : i64 + %stride = arith.constant 1 : i64 + %limit = arith.constant 1024 : index + %lanes = arith.constant 4 : index + %four = arith.constant 4 : index + %seventeen = arith.constant 17 : i32 + %bias = arith.constant 8 : i32 + + %input = memref.alloc() alignment = 64 : memref<1024x4xi32> + %output = memref.alloc() alignment = 64 : memref<1024x4xi32> + + scf.for %line = %zero to %limit step %one { + scf.for %lane = %zero to %lanes step %one { + %base = arith.muli %line, %four : index + %index = arith.addi %base, %lane : index + %value = arith.index_cast %index : index to i32 + %remainder = arith.remui %value, %seventeen : i32 + %signed = arith.subi %remainder, %bias : i32 + memref.store %signed, %input[%line, %lane] : memref<1024x4xi32> + } + } + + %bank = buckyball.bank_alloc {col = 1 : i64} + %loaded = buckyball.bank_mvin %input %bank %lines %stride + : memref<1024x4xi32> i64 i64 i64 + %result = buckyball.bank_relu %loaded %group %lines %lines : i64 i64 i64 i64 + %stored = buckyball.bank_mvout %output %result %lines %stride + : memref<1024x4xi32> i64 i64 i64 + buckyball.fence + buckyball.bank_release %stored : i64 + + func.call @check_result(%output) : (memref<1024x4xi32>) -> () + memref.dealloc %input : memref<1024x4xi32> + memref.dealloc %output : memref<1024x4xi32> + return %zeroI8 : i8 +} diff --git a/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32_contract.mlir b/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32_contract.mlir new file mode 100644 index 000000000..3e30deca0 --- /dev/null +++ b/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32_contract.mlir @@ -0,0 +1,12 @@ +// CHECK: buckyball.relu +// CHECK-NOT: buckyball.bank_relu + +func.func @main() -> i8 { + %zero = arith.constant 0 : i8 + %group = arith.constant 0 : i64 + %lines = arith.constant 1024 : i64 + %bank = buckyball.bank_alloc {col = 1 : i64} + %result = buckyball.bank_relu %bank %group %lines %lines : i64 i64 i64 i64 + buckyball.bank_release %result : i64 + return %zero : i8 +} diff --git a/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32_main.cpp b/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32_main.cpp new file mode 100644 index 000000000..4c6c039c6 --- /dev/null +++ b/examples/balls/relu/workloads/mlir_tests/bank/relu_bank_256x16_i32_main.cpp @@ -0,0 +1,43 @@ +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + volatile uint32_t *sim_exit = (volatile uint32_t *)0x60000000; + *sim_exit = 1; + while (1) { + } +#else + exit(1); +#endif +} + +#ifdef __cplusplus +extern "C" +#endif + void check_result(int32_t *allocated, int32_t *aligned, int64_t offset, + int64_t size0, int64_t size1, int64_t stride0, + int64_t stride1) { + (void)allocated; + if (size0 != 1024 || size1 != 4 || stride0 != 4 || stride1 != 1) { + printf("FAILED: relu bank shape %dx%d stride %dx%d\n", (int)size0, + (int)size1, (int)stride0, (int)stride1); + fail(); + } + + int32_t *output = aligned + offset; + for (int line = 0; line < 1024; ++line) { + for (int lane = 0; lane < 4; ++lane) { + int32_t input = (line * 4 + lane) % 17 - 8; + int32_t expected = input < 0 ? 0 : input; + int32_t actual = output[line * stride0 + lane * stride1]; + if (actual != expected) { + printf("FAILED: relu bank[%d][%d] expected %d got %d\n", line, lane, + expected, actual); + fail(); + } + } + } + printf("PASSED: relu bank 256x16\n"); +} diff --git a/examples/balls/smatmul/README.md b/examples/balls/smatmul/README.md new file mode 100644 index 000000000..b66a79d7b --- /dev/null +++ b/examples/balls/smatmul/README.md @@ -0,0 +1,40 @@ +# SMatMulBall + +`SMatMulBall` computes signed INT8 matrix products with signed INT32 output. +The array is 16x16, SRAM rows are 128 bits, and every SRAM is single-port. + +## Commands + +Both commands use `iter` for three bank-relative base rows: A, B, then C. +The remaining `iter` bits and `rs2[63:36]` must be zero. A, B, and C must be +different allocated virtual banks. + +`SMATMUL_OS` is output-stationary. `rows` and `k` are positive multiples of +16, `cols` is exactly 16. A and B use `cols=1`; C uses the SMatMulBall +`outBW` pbank groups from the chip configuration. A and B are provided as +16-wide K tiles; the Ball accumulates all K tiles before writing C. + +`SMATMUL_WS` is weight-stationary. It computes one `16x16` A tile against a +`16xN` B tile. A and B use `cols=1`; B is panel-major in one pbank. C uses the +SMatMulBall `outBW` pbank groups from the chip configuration. A is read once +and remains resident while B panels are processed in increasing line order. + +## Layout + +Let `rounds = 4 / outBW`. Every result row has four 128-bit words. Word +`w` is written to C pbank `group = w % outBW` at `round = w / outBW`. + +For OS, logical output row `r` uses `rounds` consecutive physical lines: + +```text +line r*rounds + round +``` + +For WS, panel `p`, row `r`, and write round use line +`base + p*(16*rounds) + r*rounds + round`. With Pebble `outBW=2`, C group 0 +holds columns `0..3` then `8..11` on adjacent lines; C group 1 holds columns +`4..7` then `12..15`. The two groups are written together in two rounds. + +The Ball owns all three virtual banks until completion. It does not interleave +rows from different groups, and it does not special-case padding; software +must provide 16-aligned shapes. diff --git a/examples/balls/smatmul/arch/src/main/scala/Array.scala b/examples/balls/smatmul/arch/src/main/scala/Array.scala new file mode 100644 index 000000000..6cf185eb6 --- /dev/null +++ b/examples/balls/smatmul/arch/src/main/scala/Array.scala @@ -0,0 +1,69 @@ +package examples.balls.smatmul + +import chisel3._ +import chisel3.util._ +import chisel3.experimental.hierarchy.{instantiable, public, Instantiate} + +@instantiable +class Array extends Module { + private val tile = 16 + + @public + val io = IO(new Bundle { + val start = Input(Bool()) + val aRows = Input(Vec(tile, UInt(128.W))) + val bRows = Input(Vec(tile, UInt(128.W))) + val done = Output(Bool()) + val result = Output(Vec(tile, UInt(512.W))) + }) + + val running = RegInit(false.B) + val cycle = RegInit(0.U(6.W)) + val pe = Seq.tabulate(tile, tile)((_, _) => Instantiate(new PE)) + + when(io.start) { + assert(!running, "Array: start while array is running") + running := true.B + cycle := 0.U + }.elsewhen(running) { + when(cycle === 45.U) { + running := false.B + }.otherwise { + cycle := cycle + 1.U + } + } + + for (row <- 0 until tile) { + val aCycle = cycle - row.U + val aWord = (io.aRows(row) >> (aCycle << 3))(7, 0).asSInt + val aValid = running && cycle >= row.U && cycle < (row + tile).U + + for (column <- 0 until tile) { + pe(row)(column).io.clear := io.start + if (column == 0) { + pe(row)(column).io.aIn := aWord + pe(row)(column).io.aInValid := aValid + } else { + pe(row)(column).io.aIn := pe(row)(column - 1).io.aOut + pe(row)(column).io.aInValid := pe(row)(column - 1).io.aOutValid + } + + val bCycle = cycle - column.U + val bRow = bCycle(3, 0) + val bWord = (io.bRows(bRow)(8 * column + 7, 8 * column)).asSInt + val bValid = running && cycle >= column.U && cycle < (column + tile).U + if (row == 0) { + pe(row)(column).io.bIn := bWord + pe(row)(column).io.bInValid := bValid + } else { + pe(row)(column).io.bIn := pe(row - 1)(column).io.bOut + pe(row)(column).io.bInValid := pe(row - 1)(column).io.bOutValid + } + } + } + + for (row <- 0 until tile) { + io.result(row) := Cat((0 until tile).reverse.map(column => pe(row)(column).io.sum.asUInt)) + } + io.done := running && cycle === 45.U +} diff --git a/examples/balls/smatmul/arch/src/main/scala/PE.scala b/examples/balls/smatmul/arch/src/main/scala/PE.scala new file mode 100644 index 000000000..f0fc50b7c --- /dev/null +++ b/examples/balls/smatmul/arch/src/main/scala/PE.scala @@ -0,0 +1,48 @@ +package examples.balls.smatmul + +import chisel3._ +import chisel3.experimental.hierarchy.{instantiable, public} + +@instantiable +class PE extends Module { + + @public + val io = IO(new Bundle { + val aIn = Input(SInt(8.W)) + val aInValid = Input(Bool()) + val bIn = Input(SInt(8.W)) + val bInValid = Input(Bool()) + val clear = Input(Bool()) + val aOut = Output(SInt(8.W)) + val aOutValid = Output(Bool()) + val bOut = Output(SInt(8.W)) + val bOutValid = Output(Bool()) + val sum = Output(SInt(32.W)) + }) + + val accumulator = RegInit(0.S(32.W)) + val aPipe = RegInit(0.S(8.W)) + val aPipeValid = RegInit(false.B) + val bPipe = RegInit(0.S(8.W)) + val bPipeValid = RegInit(false.B) + + when(io.clear) { + accumulator := 0.S + aPipeValid := false.B + bPipeValid := false.B + }.elsewhen(io.aInValid && io.bInValid) { + accumulator := accumulator + (io.aIn * io.bIn) + } + when(!io.clear) { + aPipe := io.aIn + aPipeValid := io.aInValid + bPipe := io.bIn + bPipeValid := io.bInValid + } + + io.aOut := aPipe + io.aOutValid := aPipeValid + io.bOut := bPipe + io.bOutValid := bPipeValid + io.sum := accumulator +} diff --git a/examples/balls/smatmul/arch/src/main/scala/SMatMulBall.scala b/examples/balls/smatmul/arch/src/main/scala/SMatMulBall.scala new file mode 100644 index 000000000..d4b4970e5 --- /dev/null +++ b/examples/balls/smatmul/arch/src/main/scala/SMatMulBall.scala @@ -0,0 +1,50 @@ +package examples.balls.smatmul + +import chisel3._ +import chisel3.util._ +import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} +import framework.balldomain.blink.{BallStatus, BlinkIO, HasBallStatus, HasBlink, SubRobRow} +import framework.balldomain.blink.mmio.MmioRead +import framework.top.GlobalConfig + +@instantiable +class SMatMulBall(val b: GlobalConfig) extends Module with HasBlink with HasBallStatus { + + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "SMatMulBall") + .getOrElse( + throw new IllegalArgumentException("SMatMulBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW + + @public + val io = IO(new BlinkIO(b, inBW, outBW)) + + def blink: BlinkIO = io + def status: BallStatus = io.status + dontTouch(io) + + val smatmulArrayUnit: Instance[SMatMulUnit] = Instantiate(new SMatMulUnit(b)) + + smatmulArrayUnit.io.cmdReq <> io.cmdReq + smatmulArrayUnit.io.cmdResp <> io.cmdResp + smatmulArrayUnit.io.channelReady := io.channelReady + + for (i <- 0 until inBW) { + smatmulArrayUnit.io.bankRead(i) <> io.bankRead(i) + } + + for (i <- 0 until outBW) { + smatmulArrayUnit.io.bankWrite(i) <> io.bankWrite(i) + } + + io.status <> smatmulArrayUnit.io.status + + io.subRobReq.valid := false.B + io.subRobReq.bits := SubRobRow.tieOff(b) + + // MMIO: this Ball does not consume MMIO metadata + MmioRead.tieOff(io.mmioRead) +} diff --git a/examples/balls/smatmul/arch/src/main/scala/SMatMulUnit.scala b/examples/balls/smatmul/arch/src/main/scala/SMatMulUnit.scala new file mode 100644 index 000000000..729e445f8 --- /dev/null +++ b/examples/balls/smatmul/arch/src/main/scala/SMatMulUnit.scala @@ -0,0 +1,349 @@ +package examples.balls.smatmul + +import chisel3._ +import chisel3.util._ +import chisel3.experimental.hierarchy.{instantiable, public, Instance, Instantiate} +import framework.balldomain.rs.{BallRsComplete, BallRsIssue} +import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} +import framework.top.GlobalConfig + +@instantiable +class SMatMulUnit(val b: GlobalConfig) extends Module { + private val tile = 16 + private val addressWidth = log2Up(b.memDomain.bankEntries) + private val bankWidth = log2Up(b.memDomain.bankNum) + + private val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "SMatMulBall") + .getOrElse(throw new IllegalArgumentException("SMatMulBall not found in config")) + + private val inBW = ballMapping.inBW + private val outBW = ballMapping.outBW + private val rowWords = 4 + private val outputRounds = rowWords / outBW + + require(inBW >= 2, "SMatMulBall requires SRAM read ports for A and B") + require( + outBW > 0 && outBW <= rowWords && rowWords % outBW == 0, + "SMatMulBall outBW must divide four 128-bit result words" + ) + require(b.memDomain.bankWidth == 128, "SMatMulBall requires 128-bit SRAM rows") + require(b.memDomain.bankMaskLen == 16, "SMatMulBall requires sixteen byte enables") + require(3 * addressWidth <= b.frontend.iter_len, "SMatMulBall iter cannot hold three base lines") + + @public + val io = IO(new Bundle { + val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) + val cmdResp = Decoupled(new BallRsComplete(b)) + val bankRead = Vec(inBW, Flipped(new BankRead(b))) + val bankWrite = Vec(outBW, Flipped(new BankWrite(b))) + val channelReady = Input(Bool()) + val status = new BallStatus + }) + + val Seq( + idle, + waitForChannels, + clearAccumulator, + loadTile, + runArray, + readAccumulator, + writeAccumulator, + readResult, + holdResult, + writeResult, + waitForCWrite, + complete + ) = Enum(12) + + val state = RegInit(idle) + + val robId = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) + val isSub = RegInit(false.B) + val subRobId = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) + val aBank = RegInit(0.U(bankWidth.W)) + val bBank = RegInit(0.U(bankWidth.W)) + val cBank = RegInit(0.U(bankWidth.W)) + val aBaseLine = RegInit(0.U(addressWidth.W)) + val bBaseLine = RegInit(0.U(addressWidth.W)) + val cBaseLine = RegInit(0.U(addressWidth.W)) + val outputTileCount = RegInit(0.U(12.W)) + val panelCount = RegInit(1.U(7.W)) + val panelIndex = RegInit(0.U(7.W)) + val isWs = RegInit(false.B) + val reductionTileCount = RegInit(0.U(12.W)) + val outputTile = RegInit(0.U(12.W)) + val reductionTile = RegInit(0.U(12.W)) + val accumulatorRow = RegInit(0.U(4.W)) + val resultRow = RegInit(0.U(4.W)) + val outputRound = RegInit(0.U(math.max(1, log2Up(outputRounds)).W)) + val aRowsRequested = RegInit(0.U(5.W)) + val aRowsStored = RegInit(0.U(5.W)) + val bRowsRequested = RegInit(0.U(5.W)) + val bRowsStored = RegInit(0.U(5.W)) + val cRowData = Reg(UInt(512.W)) + + val aRows = Reg(Vec(tile, UInt(128.W))) + val bRows = Reg(Vec(tile, UInt(128.W))) + val accumulator = SyncReadMem(tile, UInt(512.W)) + val array: Instance[Array] = Instantiate(new Array) + + val accumulatorRead = state === readAccumulator || state === readResult + val accumulatorWrite = state === clearAccumulator || state === writeAccumulator + val accumulatorAddress = Mux(state === readResult, resultRow, accumulatorRow) + val accumulatorData = accumulator.read(accumulatorAddress, accumulatorRead) + assert(!(accumulatorRead && accumulatorWrite), "SMatMulBall accumulator SRAM is single-port") + + val accumulatedResult = Wire(Vec(tile, UInt(32.W))) + for (column <- 0 until tile) { + val oldValue = accumulatorData(32 * column + 31, 32 * column).asSInt + val newValue = array.io.result(accumulatorRow)(32 * column + 31, 32 * column).asSInt + accumulatedResult(column) := (oldValue + newValue).asUInt + } + val accumulatedRow = Cat(accumulatedResult.reverse) + + val aTileLine = aBaseLine.pad(32) + ((outputTile * reductionTileCount + reductionTile) << 4) + val bTileLine = bBaseLine.pad(32) + Mux(isWs, panelIndex << 4, reductionTile << 4) + val wsLine = panelIndex * (tile * outputRounds).U + resultRow * outputRounds.U + outputRound + val osLine = (outputTile * tile.U + resultRow) * outputRounds.U + outputRound + val cLine = cBaseLine.pad(32) + Mux(isWs, wsLine, osLine) + val aReadLine = aTileLine + aRowsRequested + val bReadLine = bTileLine + bRowsRequested + + for (port <- 0 until inBW) { + io.bankRead(port).rob_id := robId + io.bankRead(port).ball_id := 0.U + io.bankRead(port).bank_id := Mux(port.U === 0.U, aBank, bBank) + io.bankRead(port).group_id := 0.U + io.bankRead(port).io.req.valid := false.B + io.bankRead(port).io.req.bits.addr := 0.U + io.bankRead(port).io.resp.ready := false.B + } + io.bankRead(0).group_id := 0.U + io.bankRead(0).io.req.valid := state === loadTile && (!isWs || panelIndex === 0.U) && aRowsRequested < tile.U + io.bankRead(0).io.req.bits.addr := aReadLine(addressWidth - 1, 0) + io.bankRead(0).io.resp.ready := state === loadTile && (!isWs || panelIndex === 0.U) && aRowsStored < tile.U + io.bankRead(1).group_id := 0.U + io.bankRead(1).io.req.valid := state === loadTile && bRowsRequested < tile.U + io.bankRead(1).io.req.bits.addr := bReadLine(addressWidth - 1, 0) + io.bankRead(1).io.resp.ready := state === loadTile && bRowsStored < tile.U + + val cWords = cRowData.asTypeOf(Vec(rowWords, UInt(128.W))) + for (port <- 0 until outBW) { + io.bankWrite(port).rob_id := robId + io.bankWrite(port).ball_id := 0.U + io.bankWrite(port).bank_id := cBank + io.bankWrite(port).group_id := port.U + io.bankWrite(port).io.req.bits.addr := cLine(addressWidth - 1, 0) + io.bankWrite(port).io.req.bits.data := cWords(outputRound * outBW.U + port.U) + io.bankWrite(port).io.req.bits.mask := VecInit(Seq.fill(16)(true.B)) + } + for (port <- 0 until outBW) { + io.bankWrite(port).io.req.valid := state === writeResult + } + val allCWriteResponses = io.bankWrite.map(_.io.resp.valid).reduce(_ && _) + for (port <- 0 until outBW) { + io.bankWrite(port).io.resp.ready := state === waitForCWrite && allCWriteResponses + } + + array.io.start := state === loadTile && aRowsStored === tile.U && bRowsStored === tile.U + array.io.aRows := aRows + array.io.bRows := bRows + + io.cmdReq.ready := state === idle + io.cmdResp.valid := state === complete + io.cmdResp.bits.rob_id := robId + io.cmdResp.bits.is_sub := isSub + io.cmdResp.bits.sub_rob_id := subRobId + + when(state === clearAccumulator) { + accumulator.write(accumulatorRow, 0.U) + when(accumulatorRow === 15.U) { + accumulatorRow := 0.U + reductionTile := 0.U + aRowsRequested := Mux(isWs && panelIndex =/= 0.U, tile.U, 0.U) + aRowsStored := Mux(isWs && panelIndex =/= 0.U, tile.U, 0.U) + bRowsRequested := 0.U + bRowsStored := 0.U + state := loadTile + }.otherwise { + accumulatorRow := accumulatorRow + 1.U + } + } + + when(state === loadTile) { + when(io.bankRead(0).io.req.fire)(aRowsRequested := aRowsRequested + 1.U) + when(io.bankRead(1).io.req.fire)(bRowsRequested := bRowsRequested + 1.U) + when(io.bankRead(0).io.resp.fire) { + aRows(aRowsStored(3, 0)) := io.bankRead(0).io.resp.bits.data + aRowsStored := aRowsStored + 1.U + } + when(io.bankRead(1).io.resp.fire) { + bRows(bRowsStored(3, 0)) := io.bankRead(1).io.resp.bits.data + bRowsStored := bRowsStored + 1.U + } + when(aRowsStored === tile.U && bRowsStored === tile.U) { + state := runArray + } + } + + when(state === runArray && array.io.done) { + accumulatorRow := 0.U + state := readAccumulator + } + + when(state === readAccumulator) { + state := writeAccumulator + } + + when(state === writeAccumulator) { + accumulator.write(accumulatorRow, accumulatedRow) + when(accumulatorRow === 15.U) { + when(reductionTile + 1.U === reductionTileCount) { + resultRow := 0.U + state := readResult + }.otherwise { + reductionTile := reductionTile + 1.U + aRowsRequested := 0.U + aRowsStored := 0.U + bRowsRequested := 0.U + bRowsStored := 0.U + state := loadTile + } + }.otherwise { + accumulatorRow := accumulatorRow + 1.U + state := readAccumulator + } + } + + when(state === readResult) { + outputRound := 0.U + state := holdResult + } + + when(state === holdResult) { + cRowData := accumulatorData + state := writeResult + } + + when(state === writeResult) { + val allCWriteRequests = io.bankWrite.map(_.io.req.ready).reduce(_ && _) + assert( + io.bankWrite.map(_.io.req.ready).map(_ === io.bankWrite(0).io.req.ready).reduce(_ && _), + "SMatMulBall C channels must be ready together" + ) + when(allCWriteRequests) { + state := waitForCWrite + } + } + + when(state === waitForCWrite) { + assert( + io.bankWrite.map(_.io.resp.valid).map(_ === io.bankWrite(0).io.resp.valid).reduce(_ && _), + "SMatMulBall C channels must respond together" + ) + when(allCWriteResponses) { + when(outputRound =/= (outputRounds - 1).U) { + outputRound := outputRound + 1.U + state := writeResult + }.otherwise { + when(resultRow === 15.U) { + when(isWs && panelIndex + 1.U < panelCount) { + panelIndex := panelIndex + 1.U + accumulatorRow := 0.U + resultRow := 0.U + bRowsRequested := 0.U + bRowsStored := 0.U + state := clearAccumulator + }.elsewhen(outputTile + 1.U === outputTileCount) { + state := complete + }.otherwise { + outputTile := outputTile + 1.U + accumulatorRow := 0.U + state := clearAccumulator + } + }.otherwise { + resultRow := resultRow + 1.U + state := readResult + } + } + } + } + + when(state === complete && io.cmdResp.fire) { + state := idle + } + + when(io.cmdReq.fire) { + val command = io.cmdReq.bits.cmd + val rows = command.rs2(11, 0) + val columns = command.rs2(23, 12) + val reduction = command.rs2(35, 24) + val commandIter = command.iter + robId := io.cmdReq.bits.rob_id + isSub := io.cmdReq.bits.is_sub + subRobId := io.cmdReq.bits.sub_rob_id + aBank := command.op1_bank + bBank := command.op2_bank + cBank := command.wr_bank + aBaseLine := commandIter(addressWidth - 1, 0) + bBaseLine := commandIter(2 * addressWidth - 1, addressWidth) + cBaseLine := commandIter(3 * addressWidth - 1, 2 * addressWidth) + outputTileCount := rows >> 4 + panelCount := columns >> 4 + panelIndex := 0.U + reductionTileCount := reduction >> 4 + outputTile := 0.U + accumulatorRow := 0.U + resultRow := 0.U + isWs := command.funct7 === 68.U + assert(rows =/= 0.U && rows(3, 0) === 0.U, "SMatMulBall rows must be a non-zero multiple of 16") + assert(command.funct7 === 65.U || command.funct7 === 68.U, "SMatMulBall funct7 must select OS or WS") + when(command.funct7 === 68.U) { + assert( + rows === 16.U && reduction === 16.U && columns(3, 0) === 0.U && + columns * outputRounds.U <= b.memDomain.bankEntries.U, + "SMATMUL_WS requires rows=k=16 and C to fit in its output groups" + ) + assert( + command.op1_col === 1.U && command.op2_col === 1.U && command.wr_col === outBW.U, + "SMATMUL_WS bank groups mismatch" + ) + }.otherwise { + assert(columns === 16.U, "SMATMUL_OS columns must be exactly 16") + assert( + command.op1_col === 1.U && command.op2_col === 1.U && command.wr_col === outBW.U, + "SMATMUL_OS bank groups mismatch" + ) + } + assert(reduction =/= 0.U && reduction(3, 0) === 0.U, "SMatMulBall reduction must be a non-zero multiple of 16") + assert(command.rs2(63, 36) === 0.U, "SMatMulBall rs2[63:36] must be zero") + assert(commandIter(b.frontend.iter_len - 1, 3 * addressWidth) === 0.U, "SMatMulBall iter high bits must be zero") + assert( + command.op1_bank =/= command.op2_bank && command.op1_bank =/= command.wr_bank && command.op2_bank =/= command.wr_bank, + "SMatMulBall A, B, and C must use different SRAM banks" + ) + assert( + commandIter(addressWidth - 1, 0) + (rows >> 4) * (reduction >> 4) * tile.U <= b.memDomain.bankEntries.U, + "SMatMulBall A does not fit in one bank" + ) + assert( + commandIter(2 * addressWidth - 1, addressWidth) + + Mux(command.funct7 === 68.U, columns, reduction) <= b.memDomain.bankEntries.U, + "SMatMulBall B does not fit in one bank" + ) + assert( + commandIter(3 * addressWidth - 1, 2 * addressWidth) + + Mux(command.funct7 === 68.U, columns * outputRounds.U, rows * outputRounds.U) <= b.memDomain.bankEntries.U, + "SMatMulBall C does not fit in its physical banks" + ) + state := waitForChannels + } + + when(state === waitForChannels && io.channelReady) { + state := clearAccumulator + } + + io.status.idle := state === idle + io.status.running := state =/= idle +} diff --git a/examples/balls/smatmul/compiler/src/Backend/include/IntrinsicsRISCVSMatMulBall.td b/examples/balls/smatmul/compiler/src/Backend/include/IntrinsicsRISCVSMatMulBall.td new file mode 100644 index 000000000..6b92a9730 --- /dev/null +++ b/examples/balls/smatmul/compiler/src/Backend/include/IntrinsicsRISCVSMatMulBall.td @@ -0,0 +1,7 @@ +//===- IntrinsicsRISCVSMatMulBall.td - SMatMulBall intrinsics --------------===// +// +// Stable LLVM intrinsic for SMatMulBall is not wired yet. Pebble uses +// CustomIntrOp(funct7 from Core ballISA SMATMUL). Keep this file as a placeholder for a future +// int_riscv_bb_smatmul addition to the RISC-V backend. +// +//===----------------------------------------------------------------------===// diff --git a/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp new file mode 100644 index 000000000..f31ff776d --- /dev/null +++ b/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -0,0 +1,41 @@ +//===- AssignPhysicalBankPatterns.cpp - Matrix bank assignment patterns ---===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace mlir::buddy { +void populateSMatMulBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +} // namespace mlir::buddy + +namespace { + +class BankSMatMulPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(BankSMatMulOp op, + PatternRewriter &rewriter) const override { + auto matrix = rewriter.create( + op.getLoc(), op.getOp1Bank(), op.getOp2Bank(), op.getWrBank(), + op.getConfig(), rewriter.getI64IntegerAttr(op.getOp1Base()), + rewriter.getI64IntegerAttr(op.getOp2Base()), + rewriter.getI64IntegerAttr(op.getWrBase()), op.getWsAttr()); + rewriter.replaceOp(op, op.getWrBank()); + return success(); + } +}; + +} // namespace + +void mlir::buddy::populateSMatMulBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { + (void)state; + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp b/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp new file mode 100644 index 000000000..3e20be7c4 --- /dev/null +++ b/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp @@ -0,0 +1,267 @@ +//===- SMatMulMatmulToBankSSAPatterns.cpp - f32 matmul -> BankMatrix +//-------===// +// +// Bank unit: all matrix dimensions are multiples of 16. +// One activation chunk is packed and quantized once before its N panels run. +// +//===----------------------------------------------------------------------===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Target/BuckyballTargetRegistry.h" +#include "Utils/BankUtils.h" + +#include + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace { + +constexpr int64_t kLane = 16; +constexpr int64_t kBankDepth = 1024; +constexpr int64_t kMmioBytes = 5 * 1024; + +struct OutputPack { + Value value; + int64_t m0; + int64_t mChunk; + int64_t n0; + int64_t nChunk; +}; + +class MatrixMatmulToBankSSAPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(SMatMulMatmulOp op, + PatternRewriter &b) const override { + Location loc = op.getLoc(); + auto aTy = dyn_cast(op.getAMemArray().getType()); + auto bTy = dyn_cast(op.getBMemArray().getType()); + auto cTy = dyn_cast(op.getCMemArray().getType()); + if (!aTy || !bTy || !cTy || !aTy.hasStaticShape() || + !bTy.hasStaticShape() || !cTy.hasStaticShape()) + return b.notifyMatchFailure(op, "requires static memrefs"); + + Type elem = aTy.getElementType(); + if (isa(elem) && bTy.getElementType() == elem && + cTy.getElementType() == elem) { + int64_t M = aTy.getShape()[0], K = aTy.getShape()[1]; + int64_t Kb = bTy.getShape()[0], N = bTy.getShape()[1]; + if (K != Kb || cTy.getShape()[0] != M || cTy.getShape()[1] != N) + return op.emitError("matmul shape mismatch"); + b.create( + loc, ValueRange{op.getAMemArray(), op.getBMemArray()}, + ValueRange{op.getCMemArray()}); + b.eraseOp(op); + return success(); + } + + if (!aTy.getElementType().isF32() || !bTy.getElementType().isInteger(8) || + !cTy.getElementType().isF32()) + return b.notifyMatchFailure( + op, + "requires FP32 activations, offline INT8 weights, and FP32 output"); + + int64_t dwAddrBase = op.getDwAddr(); + int64_t dwBytes = op.getDwBytes(); + bool perChannel = op.getPerChannel(); + if (dwAddrBase < 16 || dwAddrBase % 4) + return op.emitError( + "requires a 4-byte-aligned Dw MMIO byte address from RAX metadata"); + + int64_t M = aTy.getShape()[0], K = aTy.getShape()[1]; + int64_t Kb = bTy.getShape()[0], N = bTy.getShape()[1]; + if (K != Kb || cTy.getShape()[0] != M || cTy.getShape()[1] != N) + return op.emitError("matmul shape mismatch"); + if (M % kLane || N % kLane || K % kLane) + return op.emitError("SMatMul requires M/N/K to be 16-aligned"); + if (M <= 0 || K <= 0 || N <= 0) + return op.emitError("M/N/K must be positive"); + const int64_t outputGroups = + buckyball_target::getBuckyballBallMapping("SMatMulBall").outBW; + if (outputGroups <= 0 || outputGroups > 4 || 4 % outputGroups) + return op.emitError("SMatMulBall outBW must divide four result blocks"); + const int64_t outputRounds = 4 / outputGroups; + int64_t requiredDwBytes = perChannel ? N * 4 : 4; + if (dwBytes < requiredDwBytes || dwAddrBase + requiredDwBytes > kMmioBytes) + return op.emitError("Dw scale range exceeds Pebble MMIO space"); + const int64_t kTiles = K / kLane; + const int64_t aRowsPerMTile = K; + const int64_t maxMChunk = (kBankDepth / aRowsPerMTile) * kLane; + if (maxMChunk == 0) + return op.emitError("K exceeds activation-bank packing capacity"); + + SmallVector activationPacks; + SmallVector weightPacks; + SmallVector outputPacks; + Value aF = allocBank(b, loc, 1, 4); + Value aI = allocBank(b, loc, 1, 1); + Value bI = allocBank(b, loc, 1, 1); + Value cB = allocBank(b, loc, 1, outputGroups); + Value cF = allocBank(b, loc, 1, outputGroups); + + // Quantize each activation chunk once, then reuse its Da and INT8 bank + // across every output panel. The panel loop only changes the weight/Dw + // slice and the destination tile. + for (int64_t m0 = 0; m0 < M;) { + int64_t mChunk = std::min(maxMChunk, M - m0); + if (mChunk > 4095) + return op.emitError("matmul M chunk exceeds matrix cfg field"); + + const int64_t aRows = mChunk / kLane * aRowsPerMTile; + Value daAddr = createI64Const(b, loc, 0); + auto aPackType = MemRefType::get({aRows, kLane}, b.getF32Type()); + Value aPack = b.create(loc, aPackType); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value sixteen = b.create(loc, kLane); + Value mUpper = b.create(loc, mChunk / kLane); + Value kUpper = b.create(loc, kTiles); + Value mBase = b.create(loc, m0); + Value kTilesValue = b.create(loc, kTiles); + auto mLoop = b.create(loc, zero, mUpper, one); + b.setInsertionPointToStart(mLoop.getBody()); + Value m = mLoop.getInductionVar(); + auto kLoop = b.create(loc, zero, kUpper, one); + b.setInsertionPointToStart(kLoop.getBody()); + Value k = kLoop.getInductionVar(); + auto rowLoop = b.create(loc, zero, sixteen, one); + b.setInsertionPointToStart(rowLoop.getBody()); + Value row = rowLoop.getInductionVar(); + auto columnLoop = b.create(loc, zero, sixteen, one); + b.setInsertionPointToStart(columnLoop.getBody()); + Value column = columnLoop.getInductionVar(); + Value sourceRow = b.create( + loc, mBase, + b.create(loc, b.create(loc, m, sixteen), + row)); + Value sourceColumn = b.create( + loc, b.create(loc, k, sixteen), column); + Value tile = b.create( + loc, b.create(loc, m, kTilesValue), k); + Value packedRow = b.create( + loc, b.create(loc, tile, sixteen), row); + Value value = b.create( + loc, op.getAMemArray(), ValueRange{sourceRow, sourceColumn}); + b.create(loc, value, aPack, + ValueRange{packedRow, column}); + b.setInsertionPointAfter(mLoop); + activationPacks.push_back(aPack); + Value aL = mvinBank(b, loc, aPack, aF, aRows, 1); + Value aQ = b.create(loc, aI.getType(), aL, aI, + createI64Const(b, loc, aRows), daAddr); + + for (int64_t n0 = 0; n0 < N; n0 += kLane) { + int64_t dwAddr = dwAddrBase + (perChannel ? n0 * 4 : 0); + Value dwAddrValue = createI64Const(b, loc, dwAddr); + uint64_t cfg = matrixRs2((uint64_t)mChunk, 16, (uint64_t)K); + auto bPackType = MemRefType::get({K, kLane}, bTy.getElementType()); + Value bPack = b.create(loc, bPackType); + Value bZero = b.create(loc, 0); + Value bOne = b.create(loc, 1); + Value bK = b.create(loc, K); + Value bN = b.create(loc, n0); + Value bSixteen = b.create(loc, kLane); + auto bRowLoop = b.create(loc, bZero, bK, bOne); + b.setInsertionPointToStart(bRowLoop.getBody()); + Value bRow = bRowLoop.getInductionVar(); + auto bColumnLoop = b.create(loc, bZero, bSixteen, bOne); + b.setInsertionPointToStart(bColumnLoop.getBody()); + Value bColumn = bColumnLoop.getInductionVar(); + Value bSourceColumn = b.create(loc, bN, bColumn); + Value bValue = b.create( + loc, op.getBMemArray(), ValueRange{bRow, bSourceColumn}); + b.create(loc, bValue, bPack, + ValueRange{bRow, bColumn}); + b.setInsertionPointAfter(bRowLoop); + weightPacks.push_back(bPack); + Value bL = mvinBank(b, loc, bPack, bI, aRowsPerMTile); + Value cO = createBankSMatMul(b, loc, cB.getType(), aQ, bL, cB, + createI64Const(b, loc, (int64_t)cfg)); + Value fp = perChannel + ? b.create( + loc, cF.getType(), cO, cF, + createI64Const(b, loc, outputRounds * mChunk), + daAddr, dwAddrValue) + .getResult() + : b.create( + loc, cF.getType(), cO, cF, + createI64Const(b, loc, outputRounds * mChunk), + daAddr, dwAddrValue) + .getResult(); + auto cPackType = MemRefType::get( + {outputRounds * mChunk, outputGroups * 4}, b.getF32Type()); + Value cPack = b.create(loc, cPackType); + mvoutBank(b, loc, cPack, fp, outputRounds * mChunk, 1); + outputPacks.push_back({cPack, m0, mChunk, n0, kLane}); + } + m0 += mChunk; + } + + releaseBank(b, loc, aF); + releaseBank(b, loc, aI); + releaseBank(b, loc, bI); + releaseBank(b, loc, cB); + releaseBank(b, loc, cF); + + b.create(loc); + for (const OutputPack &pack : outputPacks) { + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value rounds = b.create(loc, outputRounds); + Value groups = b.create(loc, outputGroups); + Value four = b.create(loc, 4); + Value mUpper = b.create(loc, pack.mChunk); + Value nUpper = b.create(loc, pack.nChunk); + Value mBase = b.create(loc, pack.m0); + Value nBase = b.create(loc, pack.n0); + auto mLoop = b.create(loc, zero, mUpper, one); + b.setInsertionPointToStart(mLoop.getBody()); + Value m = mLoop.getInductionVar(); + auto nLoop = b.create(loc, zero, nUpper, one); + b.setInsertionPointToStart(nLoop.getBody()); + Value n = nLoop.getInductionVar(); + Value block = b.create(loc, n, four); + Value packedRow = + b.create(loc, b.create(loc, m, rounds), + b.create(loc, block, groups)); + Value packedColumn = b.create( + loc, + b.create( + loc, b.create(loc, block, groups), four), + b.create(loc, n, four)); + Value value = b.create( + loc, pack.value, ValueRange{packedRow, packedColumn}); + Value row = b.create(loc, mBase, m); + Value column = b.create(loc, nBase, n); + b.create(loc, value, op.getCMemArray(), + ValueRange{row, column}); + b.setInsertionPointAfter(mLoop); + } + for (Value pack : activationPacks) + b.create(loc, pack); + for (Value pack : weightPacks) + b.create(loc, pack); + for (const OutputPack &pack : outputPacks) + b.create(loc, pack.value); + b.eraseOp(op); + return success(); + } +}; + +} // namespace + +void mlir::buddy::populateSMatMulBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns) { + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/smatmul/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp b/examples/balls/smatmul/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp new file mode 100644 index 000000000..17b80e9e1 --- /dev/null +++ b/examples/balls/smatmul/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp @@ -0,0 +1,324 @@ +//===- TileMatMulPatterns.cpp - tile.matmul -> buckyball.smatmul_matmul +//----===// +// +// Non-quantized matmul is split into bank-sized panels. Quantized matmul +// retains its full N dimension so bank lowering can quantize one activation +// chunk and hand it to every output panel. +// +//===----------------------------------------------------------------------===// + +#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Tile/TileOps.h" + +#include + +using namespace mlir; +using namespace ::buddy::buckyball; +using namespace ::buddy::tile; +using mlir::buddy::aMvinDepthLines; +using mlir::buddy::bMvinDepthLines; +using mlir::buddy::ceilDiv; +using mlir::buddy::cMvoutDepthLines; +using mlir::buddy::kBankLane; +using mlir::buddy::kMaxMvinDepth8b; +using mlir::buddy::kMaxMvoutDepth32b; + +namespace { + +class TileMatMulLowering : public OpRewritePattern { +public: + explicit TileMatMulLowering(MLIRContext *context, int64_t /*bankWidthBytes*/, + int64_t bankDepth, int64_t /*bankNum*/) + : OpRewritePattern(context), bankDepth(bankDepth) {} + + LogicalResult matchAndRewrite(TileMatMulOp tileMatMulOp, + PatternRewriter &rewriter) const override { + Location loc = tileMatMulOp.getLoc(); + + Value aMemArray = tileMatMulOp.getAMemArray(); + Value bMemArray = tileMatMulOp.getBMemArray(); + Value cMemArray = tileMatMulOp.getCMemArray(); + + auto aType = cast(aMemArray.getType()); + auto bType = cast(bMemArray.getType()); + auto cType = cast(cMemArray.getType()); + + auto aShape = aType.getShape(); + auto bShape = bType.getShape(); + auto cShape = cType.getShape(); + size_t M = aShape[aShape.size() - 2]; + size_t K = aShape[aShape.size() - 1]; + size_t N = bShape[bShape.size() - 1]; + + if (bShape[bShape.size() - 2] != (int64_t)K || + cShape[cShape.size() - 2] != (int64_t)M || + cShape[cShape.size() - 1] != (int64_t)N) + return tileMatMulOp.emitError("matmul input/output shapes mismatch"); + + const bool isQuantized = aType.getElementType().isF32() && + bType.getElementType().isInteger(8) && + cType.getElementType().isF32(); + const bool isWideFloat = + (aType.getElementType().isF32() || aType.getElementType().isBF16()) && + bType.getElementType() == aType.getElementType() && + cType.getElementType() == aType.getElementType(); + const bool isInt8 = aType.getElementType().isInteger(8) && + bType.getElementType().isInteger(8) && + cType.getElementType().isInteger(32); + if (!isQuantized && !isWideFloat && !isInt8) + return tileMatMulOp.emitError( + "requires matching floating-point types, FP32 x INT8 -> FP32, " + "or INT8 x INT8 -> INT32"); + auto dwAddrAttr = tileMatMulOp->getAttrOfType("dw_addr"); + auto dwBytesAttr = tileMatMulOp->getAttrOfType("dw_bytes"); + auto perChannelAttr = tileMatMulOp->getAttrOfType("per_channel"); + if (isQuantized && (!dwAddrAttr || !dwBytesAttr || !perChannelAttr || + dwAddrAttr.getInt() < 16)) + return tileMatMulOp.emitError( + "quantized matmul requires RAX Dw metadata"); + + size_t M_pad = ceilDiv(M, kBankLane) * kBankLane; + size_t K_pad = ceilDiv(K, kBankLane) * kBankLane; + size_t N_pad = ceilDiv(N, kBankLane) * kBankLane; + bool needPadding = (M_pad != M) || (K_pad != K) || (N_pad != N); + + Value aMemArrayPadded = aMemArray; + Value bMemArrayPadded = bMemArray; + Value cMemArrayPadded = cMemArray; + + if (needPadding) { + auto aPadType = MemRefType::get({(int64_t)M_pad, (int64_t)K_pad}, + aType.getElementType()); + auto bPadType = MemRefType::get({(int64_t)K_pad, (int64_t)N_pad}, + bType.getElementType()); + auto cPadType = MemRefType::get({(int64_t)M_pad, (int64_t)N_pad}, + cType.getElementType()); + + aMemArrayPadded = rewriter.create(loc, aPadType); + bMemArrayPadded = rewriter.create(loc, bPadType); + cMemArrayPadded = rewriter.create(loc, cPadType); + + Value aZero = rewriter.create( + loc, aType.getElementType(), + rewriter.getZeroAttr(aType.getElementType())); + Value bZero = rewriter.create( + loc, bType.getElementType(), + rewriter.getZeroAttr(bType.getElementType())); + Value cZero = rewriter.create( + loc, cType.getElementType(), + rewriter.getZeroAttr(cType.getElementType())); + rewriter.create(loc, aZero, aMemArrayPadded); + rewriter.create(loc, bZero, bMemArrayPadded); + rewriter.create(loc, cZero, cMemArrayPadded); + + Value aView = rewriter.create( + loc, aMemArrayPadded, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(M), + rewriter.getIndexAttr(K)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + rewriter.create(loc, aMemArray, aView); + + Value bView = rewriter.create( + loc, bMemArrayPadded, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(K), + rewriter.getIndexAttr(N)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + rewriter.create(loc, bMemArray, bView); + } + + size_t M_tiling = needPadding ? M_pad : M; + size_t K_tiling = needPadding ? K_pad : K; + size_t N_tiling = needPadding ? N_pad : N; + + if (bankDepth <= 0 || (size_t)bankDepth % kBankLane != 0) + return tileMatMulOp.emitError( + "bankDepth must be a positive multiple of bank lane"); + if (M_tiling % kBankLane || K_tiling % kBankLane || N_tiling % kBankLane) + return tileMatMulOp.emitError("M/N/K must be multiples of bank lane"); + if (isQuantized) { + const int64_t dwRequired = perChannelAttr.getValue() ? N_tiling * 4 : 4; + if (dwBytesAttr.getInt() < dwRequired) + return tileMatMulOp.emitError("Dw scale image does not cover padded N"); + } + + // Bank unit on M (depth) and N (lane). Quantized matmul carries one + // logical K into SMatMul so FP2INT computes one activation scale. + const size_t mTileSize = std::min((size_t)bankDepth, M_tiling); + const size_t nTileSize = kBankLane; + const size_t kTileSize = + (isWideFloat || isQuantized) ? K_tiling : kBankLane; + if (M_tiling % mTileSize != 0) + return tileMatMulOp.emitError("M does not split into bank-depth units"); + + if (!isWideFloat && !isQuantized) { + if (aMvinDepthLines(mTileSize, kTileSize) > kMaxMvinDepth8b || + bMvinDepthLines(kTileSize, nTileSize) > kMaxMvinDepth8b || + cMvoutDepthLines(mTileSize, nTileSize) > kMaxMvoutDepth32b) + return tileMatMulOp.emitError( + "bank-unit matmul exceeds 8-bit mvin or 32-bit mvout depth"); + } + + const size_t kTileNum = ceilDiv(K_tiling, kTileSize); + + OpBuilder::InsertionGuard guard(rewriter); + Value zeroIdx = rewriter.create(loc, 0); + Value mStepVal = rewriter.create(loc, mTileSize); + Value kStepVal = rewriter.create(loc, kTileSize); + Value mUpperVal = rewriter.create(loc, M_tiling); + Value kUpperVal = rewriter.create(loc, K_tiling); + + auto mLoop = rewriter.create(loc, zeroIdx, mUpperVal, mStepVal); + rewriter.setInsertionPointToStart(mLoop.getBody()); + Value mIv = mLoop.getInductionVar(); + + if (isQuantized) { + Value aTile = rewriter.create( + loc, aMemArrayPadded, + SmallVector{mIv, rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(mTileSize), + rewriter.getIndexAttr(K_tiling)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + Value bTile = rewriter.create( + loc, bMemArrayPadded, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(K_tiling), + rewriter.getIndexAttr(N_tiling)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + Value cTile = rewriter.create( + loc, cMemArrayPadded, + SmallVector{mIv, rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(mTileSize), + rewriter.getIndexAttr(N_tiling)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + Value zero = rewriter.create( + loc, cType.getElementType(), + rewriter.getZeroAttr(cType.getElementType())); + rewriter.create(loc, zero, cTile); + auto smatmul = rewriter.create(loc, aTile, bTile, cTile); + smatmul->setAttr("dwAddr", dwAddrAttr); + smatmul->setAttr("dwBytes", dwBytesAttr); + smatmul->setAttr("perChannel", perChannelAttr); + } else { + // N is static. Non-quantized matmul remains one physical N panel per op. + for (size_t n0 = 0; n0 < N_tiling; n0 += nTileSize) { + Value cTile = rewriter.create( + loc, cMemArrayPadded, + SmallVector{mIv, rewriter.getIndexAttr(n0)}, + SmallVector{rewriter.getIndexAttr(mTileSize), + rewriter.getIndexAttr(nTileSize)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + + auto cElemType = cType.getElementType(); + Value zero = rewriter.create( + loc, cElemType, rewriter.getZeroAttr(cElemType)); + rewriter.create(loc, zero, cTile); + + if (kTileNum == 1) { + Value aTile = rewriter.create( + loc, aMemArrayPadded, + SmallVector{mIv, rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(mTileSize), + rewriter.getIndexAttr(kTileSize)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + Value bTile = rewriter.create( + loc, bMemArrayPadded, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(n0)}, + SmallVector{rewriter.getIndexAttr(kTileSize), + rewriter.getIndexAttr(nTileSize)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + auto smatmul = + rewriter.create(loc, aTile, bTile, cTile); + } else { + auto kLoop = + rewriter.create(loc, zeroIdx, kUpperVal, kStepVal); + rewriter.setInsertionPointToStart(kLoop.getBody()); + Value kIv = kLoop.getInductionVar(); + Value aTile = rewriter.create( + loc, aMemArrayPadded, SmallVector{mIv, kIv}, + SmallVector{rewriter.getIndexAttr(mTileSize), + rewriter.getIndexAttr(kTileSize)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + Value bTile = rewriter.create( + loc, bMemArrayPadded, + SmallVector{kIv, rewriter.getIndexAttr(n0)}, + SmallVector{rewriter.getIndexAttr(kTileSize), + rewriter.getIndexAttr(nTileSize)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + if (isWideFloat) + return tileMatMulOp.emitError( + "FP32 matmul must pass full K in one smatmul_matmul"); + // 8-bit legalize overwrites C; accumulate via host partial. + auto partialTy = MemRefType::get( + {(int64_t)mTileSize, (int64_t)nTileSize}, cElemType); + Value partial = rewriter.create(loc, partialTy); + rewriter.create(loc, zero, partial); + rewriter.create(loc, aTile, bTile, partial); + rewriter.create(loc, ValueRange{cTile, partial}, + ValueRange{cTile}); + rewriter.create(loc, partial); + rewriter.setInsertionPointAfter(kLoop); + } + } + } + + rewriter.setInsertionPointAfter(mLoop); + + if (needPadding) { + Value cView = rewriter.create( + loc, cMemArrayPadded, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(M), + rewriter.getIndexAttr(N)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + rewriter.create(loc, cView, cMemArray); + rewriter.create(loc, aMemArrayPadded); + rewriter.create(loc, bMemArrayPadded); + rewriter.create(loc, cMemArrayPadded); + } + + rewriter.eraseOp(tileMatMulOp); + return success(); + } + +private: + int64_t bankDepth; +}; + +} // namespace + +namespace mlir::buddy { +void populateSMatMulBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, + int64_t bankNum) { + patterns.add(patterns.getContext(), bankWidthBytes, + bankDepth, bankNum); +} +} // namespace mlir::buddy diff --git a/examples/balls/smatmul/compiler/src/Dialect/Buckyball/SMatMulBall.td b/examples/balls/smatmul/compiler/src/Dialect/Buckyball/SMatMulBall.td new file mode 100644 index 000000000..925e0d53e --- /dev/null +++ b/examples/balls/smatmul/compiler/src/Dialect/Buckyball/SMatMulBall.td @@ -0,0 +1,55 @@ +//===-- SMatMulBall.td - SMatMulBall ops -------*- tablegen -*-===// + +#ifndef SMATMULBALL_OPS +#define SMATMULBALL_OPS + +include "Buckyball.td" + +def SMatMulMatmulOp : Buckyball_Op<"smatmul_matmul"> { + let summary = "Logical matmul lowered to SMatMulBall"; + let description = [{ + Performs C[M,N] = A[M,K] x B[K,N] with int8 inputs / int32 output. + Memrefs must already match the SMatMulBall bank tile layout. Logical N + may exceed 16; bank lowering emits one 16-lane C panel at a time. + }]; + let arguments = (ins MemRefRankOf<[AnyType], [2]>:$aMemArray, + MemRefRankOf<[AnyType], [2]>:$bMemArray, + MemRefRankOf<[AnyType], [2]>:$cMemArray, + DefaultValuedAttr:$dwAddr, + DefaultValuedAttr:$dwBytes, + DefaultValuedAttr:$perChannel); + let assemblyFormat = [{$aMemArray $bMemArray $cMemArray attr-dict `:` + type($aMemArray) type($bMemArray) type($cMemArray)}]; +} + +def SMatMulOp : Buckyball_Op<"smatmul"> { + let summary = "SMatMulBall compute"; + let description = [{ + ISA: SMATMUL_OS or SMATMUL_WS, selected by the `ws` attribute. + `config` is rs2 (rows/cols/k). + }]; + let arguments = (ins I64:$op1BankId, I64:$op2BankId, + I64:$resultBankId, I64:$config, + DefaultValuedAttr:$op1Base, + DefaultValuedAttr:$op2Base, + DefaultValuedAttr:$wrBase, + DefaultValuedAttr:$ws); + let assemblyFormat = "$op1BankId `,` $op2BankId `,` $resultBankId `,` " + "$config attr-dict `:` " + "type($op1BankId)"; +} + +def BankSMatMulOp : Buckyball_Op<"bank_smatmul"> { + let summary = "SMatMulBall compute with virtual bank handles"; + let arguments = (ins I64:$op1Bank, I64:$op2Bank, I64:$wrBank, I64:$config, + DefaultValuedAttr:$op1Base, + DefaultValuedAttr:$op2Base, + DefaultValuedAttr:$wrBase, + DefaultValuedAttr:$ws); + let results = (outs I64:$wrBankOut); + let assemblyFormat = + "$op1Bank $op2Bank $wrBank $config attr-dict `:` " + "type($op1Bank) type($op2Bank) type($wrBank) type($config)"; +} + +#endif // SMATMULBALL_OPS diff --git a/examples/balls/smatmul/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/smatmul/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..a849dabc3 --- /dev/null +++ b/examples/balls/smatmul/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,235 @@ +//===- LegalizeForLLVMExport.cpp - SMatMulBall LLVM lowering --------------===// + +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/Pattern.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/LLVMIR/LLVMDialect.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" +#include "Utils/BankUtils.h" + +#include "llvm/Support/ErrorHandling.h" + +using namespace mlir; +using namespace buddy::buckyball; +using namespace buddy::buckyball::legalize; + +namespace { + +uint64_t matrixCfg(uint64_t rows, uint64_t cols, uint64_t k) { + if (rows == 0 || cols == 0 || k == 0 || rows > 0xfff || k > 0xfff || + cols > 16) + llvm::report_fatal_error("matrix cfg: rows/k in 1..4095, cols in 1..16"); + return fieldBits(rows, 0, 11) | fieldBits(cols, 12, 23) | + fieldBits(k, 24, 35); +} + +uint64_t matrixRs1(uint64_t op1, uint64_t op2, uint64_t wr) { + return fieldBits(op1, 0, 9) | fieldBits(op2, 10, 19) | fieldBits(wr, 20, 29); +} + +struct SMatMulMatmulLowering : public ConvertOpToLLVMPattern { + SMatMulMatmulLowering(LLVMTypeConverter &converter, bool /*stable*/, + bool rushB) + : ConvertOpToLLVMPattern(converter), rushB(rushB) {} + + LogicalResult + matchAndRewrite(SMatMulMatmulOp op, OpAdaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("SMatMulBall"); + Location loc = op.getLoc(); + Value aMem = op.getAMemArray(); + Value bMem = op.getBMemArray(); + Value cMem = op.getCMemArray(); + + auto aTy = dyn_cast(aMem.getType()); + auto bTy = dyn_cast(bMem.getType()); + auto cTy = dyn_cast(cMem.getType()); + if (!aTy || !bTy || !cTy || !aTy.hasStaticShape() || + !bTy.hasStaticShape() || !cTy.hasStaticShape()) + return rewriter.notifyMatchFailure( + op, "buckyball.smatmul_matmul requires static memref shapes"); + + uint64_t m = aTy.getShape()[0]; + uint64_t k = aTy.getShape()[1]; + uint64_t kb = bTy.getShape()[0]; + uint64_t n = bTy.getShape()[1]; + if (k != kb || cTy.getShape()[0] != (int64_t)m || + cTy.getShape()[1] != (int64_t)n) + return rewriter.notifyMatchFailure(op, "matmul shapes mismatch"); + // Bank-unit 8-bit path: N/K are one bank lane; M fills bank depth. + if (m == 0 || n != 16 || k != 16 || m > 1024 || m % 16 != 0) + return rewriter.notifyMatchFailure( + op, "SMatMul requires M/K exactly 16 and N exactly 16"); + if (!aTy.getElementType().isInteger(8) || + !bTy.getElementType().isInteger(8) || + !cTy.getElementType().isInteger(32)) + return rewriter.notifyMatchFailure( + op, "SMatMul Ball supports only 8-bit A/B with 32-bit C"); + + const uint64_t aBank = 0; + const uint64_t bBank = 1; + const uint64_t cBank = 2; + uint64_t depthA = m; // K <= 16 => one bank row per M row + uint64_t depthB = k; + uint64_t depthC = 2 * m; + + emitMset(rewriter, loc, aBank, 1, 1, 1); + emitMset(rewriter, loc, bBank, 1, 1, 1); + emitMset(rewriter, loc, cBank, 1, 2, 1); + + Value aPtr = extractPtr(rewriter, loc, aMem); + Value bPtr = extractPtr(rewriter, loc, bMem); + auto packedType = MemRefType::get({static_cast(depthC), 8}, + cTy.getElementType()); + Value packed = rewriter.create(loc, packedType); + Value packedPtr = extractPtr(rewriter, loc, packed); + + Value rs1A = packRs1BankIter(rewriter, loc, cstI64(rewriter, loc, aBank), + cstI64(rewriter, loc, depthA)); + Value rs2A = + packRs2MemStride(rewriter, loc, aPtr, cstI64(rewriter, loc, 1)); + emitDmaCacheFlush(rewriter, loc); + if (rushB) { + Type ptrType = LLVM::LLVMPointerType::get(rewriter.getContext()); + rewriter.create( + loc, rs1A, rs2A, + LLVM::IntToPtrOp::create(rewriter, loc, ptrType, aPtr)); + } else { + rewriter.create(loc, rs1A, rs2A); + } + + Value rs1B = packRs1BankIter(rewriter, loc, cstI64(rewriter, loc, bBank), + cstI64(rewriter, loc, depthB)); + Value rs2B = + packRs2MemStride(rewriter, loc, bPtr, cstI64(rewriter, loc, 1)); + emitDmaCacheFlush(rewriter, loc); + if (rushB) { + Type ptrType = LLVM::LLVMPointerType::get(rewriter.getContext()); + rewriter.create( + loc, rs1B, rs2B, + LLVM::IntToPtrOp::create(rewriter, loc, ptrType, bPtr)); + } else { + rewriter.create(loc, rs1B, rs2B); + } + + rewriter.create( + loc, cstI64(rewriter, loc, matrixRs1(aBank, bBank, cBank)), + cstI64(rewriter, loc, matrixCfg(m, n, k)), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("SMATMUL_OS"))); + + Value rs1C = packRs1BankIter(rewriter, loc, cstI64(rewriter, loc, cBank), + cstI64(rewriter, loc, depthC)); + Value rs2C = + packRs2MemStride(rewriter, loc, packedPtr, cstI64(rewriter, loc, 1)); + emitDmaCacheFlush(rewriter, loc); + if (rushB) { + Type ptrType = LLVM::LLVMPointerType::get(rewriter.getContext()); + rewriter.create( + loc, rs1C, rs2C, + LLVM::IntToPtrOp::create(rewriter, loc, ptrType, packedPtr)); + } else { + rewriter.create(loc, rs1C, rs2C); + } + + Value zero = cstI64(rewriter, loc, 0); + rewriter.create(loc, zero, zero); + emitDmaCacheFence(rewriter, loc); + + Value indexZero = rewriter.create(loc, 0); + Value indexOne = rewriter.create(loc, 1); + Value indexTwo = rewriter.create(loc, 2); + Value indexEight = rewriter.create(loc, 8); + Value indexM = rewriter.create(loc, m); + Value indexN = rewriter.create(loc, n); + auto rowLoop = + rewriter.create(loc, indexZero, indexM, indexOne); + rewriter.setInsertionPointToStart(rowLoop.getBody()); + Value row = rowLoop.getInductionVar(); + auto columnLoop = + rewriter.create(loc, indexZero, indexN, indexOne); + rewriter.setInsertionPointToStart(columnLoop.getBody()); + Value column = columnLoop.getInductionVar(); + Value packedRow = rewriter.create( + loc, rewriter.create(loc, row, indexTwo), + rewriter.create(loc, column, indexEight)); + Value packedColumn = + rewriter.create(loc, column, indexEight); + Value value = rewriter.create( + loc, packed, ValueRange{packedRow, packedColumn}); + rewriter.create(loc, value, cMem, ValueRange{row, column}); + rewriter.setInsertionPointAfter(rowLoop); + rewriter.create(loc, packed); + + emitMset(rewriter, loc, aBank, 0, 0, 0); + emitMset(rewriter, loc, bBank, 0, 0, 0); + emitMset(rewriter, loc, cBank, 0, 0, 0); + + rewriter.eraseOp(op); + return success(); + } + +private: + bool rushB; +}; + +struct SMatMulLowering : public ConvertOpToLLVMPattern { + SMatMulLowering(LLVMTypeConverter &converter, bool, int64_t bankDepth) + : ConvertOpToLLVMPattern(converter), bankDepth(bankDepth) {} + + LogicalResult + matchAndRewrite(SMatMulOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("SMatMulBall"); + if (bankDepth <= 1) + return op.emitError("smatmul lowering requires bank_depth > 1"); + int64_t aw = addrBitsForDepth(bankDepth); + if (aw < 1 || 3 * aw > 34) + return op.emitError( + "smatmul iter cannot hold three bases from bank_depth"); + uint64_t mask = (1ULL << aw) - 1; + if ((uint64_t)op.getOp1Base() > mask || (uint64_t)op.getOp2Base() > mask || + (uint64_t)op.getWrBase() > mask) + return op.emitError("smatmul base exceeds log2(bank_depth)"); + + Location loc = op.getLoc(); + Value rs1 = packRs1BanksIter( + rewriter, loc, adaptor.getOp1BankId(), adaptor.getOp2BankId(), + adaptor.getResultBankId(), + cstI64(rewriter, loc, + smatmulIterBits(op.getOp1Base(), op.getOp2Base(), op.getWrBase(), + aw))); + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getConfig(), + rewriter.getI32IntegerAttr(buckyball_target::getBuckyballFunct7( + op.getWs() ? "SMATMUL_WS" : "SMATMUL_OS"))); + return success(); + } + +private: + int64_t bankDepth = 0; +}; + +} // namespace + +namespace mlir::buddy::buckyball { +void populateSMatMulBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB) { + patterns.add(converter, stable, rushB); + patterns.add(converter, stable, bankDepth); +} + +void configureSMatMulBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool /*stable*/) { + target.addIllegalOp(); + target.addIllegalOp(); +} +} // namespace mlir::buddy::buckyball diff --git a/examples/balls/matrix/configs/MatrixBallParam.scala b/examples/balls/smatmul/configs/SMatMulBallParam.scala similarity index 75% rename from examples/balls/matrix/configs/MatrixBallParam.scala rename to examples/balls/smatmul/configs/SMatMulBallParam.scala index 20c167b3b..907fee4d4 100644 --- a/examples/balls/matrix/configs/MatrixBallParam.scala +++ b/examples/balls/smatmul/configs/SMatMulBallParam.scala @@ -1,24 +1,23 @@ -package examples.balls.matrix.configs +package examples.balls.smatmul.configs import framework.balldomain.configs.BallParamLoader import framework.top.GlobalConfig -case class MatrixBallParam( +case class SMatMulBallParam( InputNum: Int, inputWidth: Int, lane: Int, wsReuseTiles: Int, outputWidth: Int, numMulThreads: Int, - numCasThreads: Int -) + numCasThreads: Int) -object MatrixBallParam { - private val ballName = "MatrixBall" +object SMatMulBallParam { + private val ballName = "SMatMulBall" - def apply(b: GlobalConfig): MatrixBallParam = { + def apply(b: GlobalConfig): SMatMulBallParam = { val tbl = BallParamLoader.ballTable(b, ballName) - MatrixBallParam( + SMatMulBallParam( InputNum = BallParamLoader.int(tbl, "InputNum"), inputWidth = BallParamLoader.int(tbl, "inputWidth"), lane = BallParamLoader.int(tbl, "lane"), @@ -28,4 +27,5 @@ object MatrixBallParam { numCasThreads = BallParamLoader.int(tbl, "numCasThreads") ) } + } diff --git a/examples/balls/matrix/configs/default.toml b/examples/balls/smatmul/configs/default.toml similarity index 71% rename from examples/balls/matrix/configs/default.toml rename to examples/balls/smatmul/configs/default.toml index b144cd35f..11764ebb2 100644 --- a/examples/balls/matrix/configs/default.toml +++ b/examples/balls/smatmul/configs/default.toml @@ -1,4 +1,4 @@ -# Matrix Ball Configuration (default, lane=16) +# SMatMul Ball Configuration (default, lane=16) [ball] InputNum = 16 diff --git a/examples/balls/matrix/configs/lane16.toml b/examples/balls/smatmul/configs/lane16.toml similarity index 75% rename from examples/balls/matrix/configs/lane16.toml rename to examples/balls/smatmul/configs/lane16.toml index 0c2d490e5..9f47b0252 100644 --- a/examples/balls/matrix/configs/lane16.toml +++ b/examples/balls/smatmul/configs/lane16.toml @@ -1,4 +1,4 @@ -# Matrix Ball Configuration (lane=16) +# SMatMul Ball Configuration (lane=16) [ball] InputNum = 16 diff --git a/examples/balls/matrix/configs/lane32.toml b/examples/balls/smatmul/configs/lane32.toml similarity index 75% rename from examples/balls/matrix/configs/lane32.toml rename to examples/balls/smatmul/configs/lane32.toml index cfa99a92a..888773fa7 100644 --- a/examples/balls/matrix/configs/lane32.toml +++ b/examples/balls/smatmul/configs/lane32.toml @@ -1,4 +1,4 @@ -# Matrix Ball Configuration (lane=32) +# SMatMul Ball Configuration (lane=32) [ball] InputNum = 32 diff --git a/examples/balls/matrix/configs/lane4.toml b/examples/balls/smatmul/configs/lane4.toml similarity index 75% rename from examples/balls/matrix/configs/lane4.toml rename to examples/balls/smatmul/configs/lane4.toml index 1475d41dd..8b1491499 100644 --- a/examples/balls/matrix/configs/lane4.toml +++ b/examples/balls/smatmul/configs/lane4.toml @@ -1,4 +1,4 @@ -# Matrix Ball Configuration (lane=4) +# SMatMul Ball Configuration (lane=4) [ball] InputNum = 4 diff --git a/examples/balls/matrix/configs/lane8.toml b/examples/balls/smatmul/configs/lane8.toml similarity index 75% rename from examples/balls/matrix/configs/lane8.toml rename to examples/balls/smatmul/configs/lane8.toml index 14c2c17b5..ac69ef2a4 100644 --- a/examples/balls/matrix/configs/lane8.toml +++ b/examples/balls/smatmul/configs/lane8.toml @@ -1,4 +1,4 @@ -# Matrix Ball Configuration (lane=8) +# SMatMul Ball Configuration (lane=8) [ball] InputNum = 8 diff --git a/examples/balls/smatmul/emu/src/lib.rs b/examples/balls/smatmul/emu/src/lib.rs new file mode 100644 index 000000000..3a042b551 --- /dev/null +++ b/examples/balls/smatmul/emu/src/lib.rs @@ -0,0 +1,35 @@ +pub(crate) use crate::inst::{decode, instruction}; + +use crate::inst::instruction::ExecContext; + +mod smatmul; + +const BALL_CLASS: &str = "examples.balls.smatmul.SMatMulBall"; + +pub fn execute_known( + ball_class: &str, + funct: u32, + xs1: u64, + xs2: u64, + ctx: &mut ExecContext, +) -> Option { + if ball_class != BALL_CLASS { + return None; + } + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("SMATMUL_OS") => Some(smatmul::exec_smatmul(false, xs1, xs2, ctx)), + Some("SMATMUL_WS") => Some(smatmul::exec_smatmul(true, xs1, xs2, ctx)), + _ => None, + } +} + +pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> Option { + if ball_class != BALL_CLASS { + return None; + } + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("SMATMUL_OS") => Some(smatmul::latency(false, xs1, xs2)), + Some("SMATMUL_WS") => Some(smatmul::latency(true, xs1, xs2)), + _ => None, + } +} diff --git a/examples/balls/smatmul/emu/src/smatmul.rs b/examples/balls/smatmul/emu/src/smatmul.rs new file mode 100644 index 000000000..9c495280b --- /dev/null +++ b/examples/balls/smatmul/emu/src/smatmul.rs @@ -0,0 +1,215 @@ +use super::super::bank::{bank_lines, bank_num, bank_row_bytes}; +use super::decode::{pbank, pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; +use super::instruction::ExecContext; + +const TILE: usize = 16; + +fn log2_up(n: usize) -> u32 { + if n <= 1 { + panic!("matrix: bank_lines must be > 1, got {n}"); + } + usize::BITS - (n - 1).leading_zeros() +} + +fn i8_at(bank: &[u8], row: usize, lane: usize) -> i8 { + let row_bytes = bank_row_bytes(); + if lane >= row_bytes { + panic!("matrix: i8 lane {lane} out of row_bytes={row_bytes}"); + } + let off = row * row_bytes + lane; + if off >= bank.len() { + panic!("matrix: i8 read OOB row={row} lane={lane}"); + } + bank[off] as i8 +} + +fn write_i32(bank: &mut [u8], row: usize, lane: usize, value: i32) { + let row_bytes = bank_row_bytes(); + let off = row * row_bytes + lane * 4; + if off + 4 > bank.len() { + panic!("matrix: i32 write OOB row={row} lane={lane}"); + } + bank[off..off + 4].copy_from_slice(&value.to_le_bytes()); +} + +pub(crate) fn exec_smatmul(ws: bool, xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { + let op1 = rs1_b0(xs1); + let op2 = rs1_b1(xs1); + let wr = rs1_b2(xs1); + let rows = (xs2 & 0xfff) as usize; + let cols = ((xs2 >> 12) & 0xfff) as usize; + let k = ((xs2 >> 24) & 0xfff) as usize; + if xs2 >> 36 != 0 { + panic!("matrix: rs2[63:36] must be 0"); + } + if rows == 0 || cols == 0 || k == 0 || rows % TILE != 0 || k % TILE != 0 { + panic!("matrix: rows/cols/k must be non-zero"); + } + if ws { + if rows != TILE || k != TILE || cols % TILE != 0 { + panic!("matrix WS requires rows=k=16 and 16-aligned cols"); + } + } else if cols != TILE { + panic!("matrix OS requires cols exactly {TILE}, got {cols}"); + } + + let lines = bank_lines(); + let output_groups = crate::config::ball_domain::out_bw("examples.balls.smatmul.SMatMulBall"); + if output_groups == 0 || output_groups > 4 || 4 % output_groups != 0 { + panic!("matrix: SMatMulBall outBW must divide four result blocks"); + } + let output_rounds = 4 / output_groups; + let addr_bits = log2_up(lines); + if 3 * addr_bits > 34 { + panic!("matrix: iter cannot hold 3 bases of {addr_bits} bits"); + } + let iter = rs1_iter(xs1); + let base_mask = (1u64 << addr_bits) - 1; + let op1_base = (iter & base_mask) as usize; + let op2_base = ((iter >> addr_bits) & base_mask) as usize; + let wr_base = ((iter >> (2 * addr_bits)) & base_mask) as usize; + if iter >> (3 * addr_bits) != 0 { + panic!("matrix: iter unused high bits must be 0"); + } + + if op1 >= bank_num() as u64 || op2 >= bank_num() as u64 || wr >= bank_num() as u64 { + panic!("matrix: invalid bank_id"); + } + if !ctx.cfgs[op1 as usize].allocated + || !ctx.cfgs[op2 as usize].allocated + || !ctx.cfgs[wr as usize].allocated + { + panic!("matrix: bank not allocated"); + } + if ctx.cfgs[op1 as usize].cols != 1 { + panic!("matrix: A bank must have cols=1"); + } + if ws { + if cols * output_rounds > lines { + panic!("matrix WS C lines={} exceed bank depth={lines}", cols * output_rounds); + } + if ctx.cfgs[op2 as usize].cols != 1 || ctx.cfgs[wr as usize].cols != output_groups as u64 { + panic!("matrix WS bank groups mismatch"); + } + } else { + if ctx.cfgs[op2 as usize].cols != 1 || ctx.cfgs[wr as usize].cols != output_groups as u64 { + panic!("matrix OS bank groups mismatch"); + } + } + + if op1 == op2 || op1 == wr || op2 == wr { + panic!("matrix: A, B, and C must be different banks"); + } + + let k_tiles = k / TILE; + let a_rows = (rows / TILE) * k_tiles * TILE; + let b_lines = if ws { cols } else { k_tiles * TILE }; + let c_lines = if ws { + cols * output_rounds + } else { + rows * output_rounds + }; + + if op1_base + a_rows > lines { + panic!("matrix: A range OOB base={op1_base} rows={a_rows} lines={lines}"); + } + if op2_base + b_lines > lines { + panic!("matrix: B range OOB base={op2_base} rows={b_lines} lines={lines}"); + } + if wr_base + c_lines > lines { + panic!("matrix: C range OOB base={wr_base} lines={c_lines} lines={lines}"); + } + + let p1 = pbank(ctx.bank_map, op1); + let p2 = pbank(ctx.bank_map, op2); + + let mut c = vec![vec![0i32; cols]; rows]; + for i in 0..rows { + for j in 0..cols { + let mut acc = 0i32; + for kk in 0..k { + let mt = i / TILE; + let mr = i % TILE; + let kt = kk / TILE; + let kl = kk % TILE; + let a_row = op1_base + (mt * k_tiles + kt) * TILE + mr; + let a = i8_at(&ctx.banks[p1], a_row, kl); + + let bkr = kk % TILE; + let bkt = kk / TILE; + let b_row = if ws { + op2_base + (j / TILE) * TILE + bkr + } else { + op2_base + bkt * TILE + bkr + }; + let b_physical = p2; + let b = if ws { + i8_at(&ctx.banks[b_physical], b_row, j % TILE) + } else { + i8_at(&ctx.banks[b_physical], b_row, j) + }; + + acc += a as i32 * b as i32; + } + c[i][j] = acc; + } + } + + for i in 0..rows { + for j in 0..cols { + if ws { + let panel = j / TILE; + let block = (j % TILE) / 4; + let line = wr_base + panel * TILE * output_rounds + i * output_rounds + block / output_groups; + let group = block % output_groups; + let lane = j % 4; + write_i32( + &mut ctx.banks[pbank_group(ctx.bank_map, wr, group as u64)], + line, + lane, + c[i][j], + ); + } else { + let block = j / 4; + let line = wr_base + i * output_rounds + block / output_groups; + let group = block % output_groups; + let lane = j % 4; + write_i32( + &mut ctx.banks[pbank_group(ctx.bank_map, wr, group as u64)], + line, + lane, + c[i][j], + ); + } + } + } + 0 +} + +pub(crate) fn latency(ws: bool, _xs1: u64, xs2: u64) -> u64 { + let rows = (xs2 & 0xfff) as u64; + let cols = ((xs2 >> 12) & 0xfff) as u64; + let k = ((xs2 >> 24) & 0xfff) as u64; + if xs2 >> 36 != 0 { + panic!("matrix: rs2[63:36] must be 0"); + } + if rows == 0 || cols == 0 || k == 0 || rows % TILE as u64 != 0 || k % TILE as u64 != 0 { + panic!("matrix: rows/cols/k must be non-zero"); + } + if ws { + if rows != TILE as u64 || k != TILE as u64 || cols % TILE as u64 != 0 { + panic!("matrix WS requires rows=k=16 and 16-aligned cols"); + } + } else if cols != TILE as u64 { + panic!("matrix OS requires cols exactly {TILE}"); + } + let output_groups = crate::config::ball_domain::out_bw("examples.balls.smatmul.SMatMulBall") as u64; + if output_groups == 0 || output_groups > 4 || 4 % output_groups != 0 { + panic!("matrix: SMatMulBall outBW must divide four result blocks"); + } + if ws && cols * (4 / output_groups) > bank_lines() as u64 { + panic!("matrix WS C does not fit in its output groups"); + } + let body = rows.saturating_mul(cols).saturating_mul(k) / (TILE as u64); + body + rows.saturating_mul(cols) / 2 +} diff --git a/examples/balls/smatmul/verify/README.md b/examples/balls/smatmul/verify/README.md new file mode 100644 index 000000000..6165f95cc --- /dev/null +++ b/examples/balls/smatmul/verify/README.md @@ -0,0 +1,81 @@ +# SMatMulBall UVM Verification + +Verifies the generated `SMatMulBall` module using the shared Blink UVM framework. + +## Structure + +- `../../../../verify/uvm/src/bb_uvm_pkg.sv`: common Blink UVM agents and base env +- `src/common/smatmul_defs.svh`: DPI imports, `matrix_require_bid`, timeouts +- `src/common/smatmul_items.svh`: `smatmul_cmd_item` with `load_rust_case` +- `src/seq/smatmul_sequences.svh`: one-case sequence +- `src/cov/smatmul_cov.svh`: M/N/K shape bins `{1,2,4,8,16}` +- `src/env/smatmul_scoreboard.svh`: preload A/B, compare writes vs DPI expected list +- `src/env/smatmul_env.svh`: extends `bb_blink_env#(2,4)` +- `src/tests/smatmul_test.svh`: directed 0..3, random 4..23 +- `src/pkg/smatmul_pkg.sv`: package entry +- `src/tb_top.sv`: `bb_blink_if#(2,2)` + `SMatMulBall` (2 read, 2 write ports) +- `filelists/smatmul_ball.f`: VCS filelist + +## Test plan + +`+UVM_TESTNAME=smatmul_ball_test` runs: + +- case 0: 4x4x4 OS (banks 0,1,2) +- case 1: 5x7x3 OS tail +- case 2: 16x16x16 OS +- case 3: 32x16x16 WS +- cases 4..23: random M in {1,2,4,8,16,32}, N/K in {1,2,4,8,16}, distinct banks 0..7 + +Each case: reset, preload A/B into mem model, drive one command, wait for +`scb.done()` or timeout at 100000 cycles. Scoreboard compares writes +(group/addr/data/mask) in DPI emission order. + +## BID (required) + +`+BID=` is mandatory. Missing it fatals at test start. No default bid. + +| Config | Plusarg | +|--------|---------| +| `sims.verilator.BuckyballPebbleVerilatorConfig` | `+BID=1` | +| `sims.verilator.BuckyballToyVerilatorConfig` | `+BID=4` | + +```console +./simv +UVM_TESTNAME=smatmul_ball_test +BID=4 +``` + +## Build + +```console +nix develop ../../../../verify +cargo build --manifest-path casegen/Cargo.toml +``` + +Compile from this directory: + +```console +vcs -full64 -sverilog -timescale=1ns/1ps \ + $VCS_UVM_ARGS \ + -sv_lib casegen/target/debug/libmatrix_casegen \ + -f filelists/smatmul_ball.f +``` + +Run: + +```console +./simv +UVM_TESTNAME=smatmul_ball_test +BID=4 +``` + +Or via bbdev: + +```console +bbdev uvm --build '--ball=matrix' +bbdev uvm --run '--ball=matrix' -- '+BID=4' +``` + +## Acceptance + +- [x] Layout matches Blink common + ball-local split; filelist uses `@UVM@` / `@RTL@` +- [x] casegen whole-case DPI API; `cargo test` in casegen +- [x] VCS build with `smatmul_ball.f` (no segfault on Instantiate DUT) +- [x] `+BID=` required; pebble `+BID=1` / toy `+BID=4` +- [ ] 23/23 functional pass diff --git a/examples/balls/int2fp/emu/Cargo.lock b/examples/balls/smatmul/verify/casegen/Cargo.lock similarity index 83% rename from examples/balls/int2fp/emu/Cargo.lock rename to examples/balls/smatmul/verify/casegen/Cargo.lock index b7c7f7684..ab2798cfb 100644 --- a/examples/balls/int2fp/emu/Cargo.lock +++ b/examples/balls/smatmul/verify/casegen/Cargo.lock @@ -3,5 +3,5 @@ version = 4 [[package]] -name = "int2fp-emu-model" +name = "smatmul-casegen" version = "0.1.0" diff --git a/examples/balls/matrix/verify/casegen/Cargo.toml b/examples/balls/smatmul/verify/casegen/Cargo.toml similarity index 83% rename from examples/balls/matrix/verify/casegen/Cargo.toml rename to examples/balls/smatmul/verify/casegen/Cargo.toml index 3ba1bc988..688208365 100644 --- a/examples/balls/matrix/verify/casegen/Cargo.toml +++ b/examples/balls/smatmul/verify/casegen/Cargo.toml @@ -1,5 +1,5 @@ [package] -name = "matrix-casegen" +name = "smatmul-casegen" version = "0.1.0" edition = "2021" diff --git a/examples/balls/smatmul/verify/casegen/src/casegen.rs b/examples/balls/smatmul/verify/casegen/src/casegen.rs new file mode 100644 index 000000000..2c54fad2e --- /dev/null +++ b/examples/balls/smatmul/verify/casegen/src/casegen.rs @@ -0,0 +1,126 @@ +use crate::model; + +pub const MAX_WORDS: usize = 32; + +#[repr(C)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct MatrixCmd { + pub bid: u32, + pub ws: u32, + pub m: u32, + pub n: u32, + pub k: u32, + pub op1_bank: u32, + pub op2_bank: u32, + pub wr_bank: u32, + pub rob_id: u32, + pub rs1_lo: u32, + pub rs1_hi: u32, + pub rs2_lo: u32, + pub rs2_hi: u32, + pub num_a_words: u32, + pub num_b_words: u32, + pub num_writes: u32, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct MatrixCase { + pub cmd: MatrixCmd, + pub a: Vec, + pub b: Vec, + pub writes: Vec, +} + +impl MatrixCase { + pub fn a_word_lo(&self, index: usize) -> u64 { word_lo(&self.a, index) } + pub fn a_word_hi(&self, index: usize) -> u64 { word_hi(&self.a, index) } + pub fn b_word_lo(&self, index: usize) -> u64 { word_lo(&self.b, index) } + pub fn b_word_hi(&self, index: usize) -> u64 { word_hi(&self.b, index) } +} + +fn word_lo(data: &[u8], index: usize) -> u64 { + let offset = index * model::BANK_ROW_BYTES; + u64::from_le_bytes(data[offset..offset + 8].try_into().unwrap()) +} + +fn word_hi(data: &[u8], index: usize) -> u64 { + let offset = index * model::BANK_ROW_BYTES + 8; + u64::from_le_bytes(data[offset..offset + 8].try_into().unwrap()) +} + +pub fn gen_case(seed: u32, index: u32, bid: u32, out_bw: usize) -> MatrixCase { + match index { + 0 => build_case(bid, false, 16, 16, 16, out_bw, seed, 7), + 1 => build_case(bid, false, 32, 16, 16, out_bw, seed, 8), + 2 => build_case(bid, true, 16, 16, 16, out_bw, seed, 9), + 3 => build_case(bid, true, 16, 32, 16, out_bw, seed, 10), + _ => random_case(seed, index, bid, out_bw), + } +} + +fn build_case(bid: u32, ws: bool, rows: usize, columns: usize, k: usize, + out_bw: usize, seed: u32, rob_id: u32) -> MatrixCase { + assert!(matches!(out_bw, 1 | 2 | 4)); + assert_eq!(rows % model::TILE, 0); + assert_eq!(columns % model::TILE, 0); + assert_eq!(k % model::TILE, 0); + if ws { assert_eq!(rows, model::TILE); assert_eq!(k, model::TILE); } + let mut rng = Rng::new(seed, rows as u32 ^ columns as u32 ^ k as u32); + let a: Vec = (0..rows * k).map(|_| rng.value()).collect(); + let b: Vec = (0..k * columns).map(|_| rng.value()).collect(); + let a_packed = model::pack_a(&a, rows, k); + let b_packed = if ws { model::pack_b_ws(&b, k, columns) } else { model::pack_b_os(&b, k) }; + let writes = model::emit_writes(&model::matmul(&a, &b, rows, columns, k), ws, out_bw); + assert!(model::words(&a_packed) <= MAX_WORDS); + assert!(model::words(&b_packed) <= MAX_WORDS); + MatrixCase { + cmd: MatrixCmd { + bid, ws: ws as u32, m: rows as u32, n: columns as u32, k: k as u32, + op1_bank: 0, op2_bank: 1, wr_bank: 2, rob_id, + rs1_lo: model::encode_rs1(0, 1, 2) as u32, + rs1_hi: (model::encode_rs1(0, 1, 2) >> 32) as u32, + rs2_lo: model::encode_rs2(rows as u32, columns as u32, k as u32) as u32, + rs2_hi: (model::encode_rs2(rows as u32, columns as u32, k as u32) >> 32) as u32, + num_a_words: model::words(&a_packed) as u32, + num_b_words: model::words(&b_packed) as u32, + num_writes: writes.len() as u32, + }, + a: a_packed, b: b_packed, writes, + } +} + +fn random_case(seed: u32, index: u32, bid: u32, out_bw: usize) -> MatrixCase { + let mut rng = Rng::new(seed, index); + let ws = rng.next() & 1 != 0; + let rows = if ws { 16 } else if rng.next() & 1 == 0 { 16 } else { 32 }; + let columns = if ws && rng.next() & 1 != 0 { 32 } else { 16 }; + build_case(bid, ws, rows, columns, 16, out_bw, rng.next(), rng.next() & 15) +} + +struct Rng { state: u64 } +impl Rng { + fn new(seed: u32, salt: u32) -> Self { Self { state: u64::from(seed) << 32 | u64::from(salt) } } + fn next(&mut self) -> u32 { self.state ^= self.state << 13; self.state ^= self.state >> 7; self.state ^= self.state << 17; self.state as u32 } + fn value(&mut self) -> i8 { (self.next() % 23) as i8 - 11 } +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn os_uses_two_compact_rounds() { + let case = gen_case(1, 0, 1, 2); + assert_eq!(case.cmd.ws, 0); + assert_eq!(case.cmd.num_writes, 32); + assert_eq!(case.writes[0].group, 0); + assert_eq!(case.writes[1].group, 1); + assert_eq!(case.writes[2].addr, 1); + } + #[test] + fn ws_second_panel_follows_first() { + let case = gen_case(1, 3, 1, 2); + assert_eq!(case.cmd.ws, 1); + assert_eq!(case.cmd.num_b_words, 32); + assert_eq!(case.writes[32].addr, 32); + } +} diff --git a/examples/balls/matrix/verify/casegen/src/lib.rs b/examples/balls/smatmul/verify/casegen/src/lib.rs similarity index 59% rename from examples/balls/matrix/verify/casegen/src/lib.rs rename to examples/balls/smatmul/verify/casegen/src/lib.rs index 6d534fee6..70f960be5 100644 --- a/examples/balls/matrix/verify/casegen/src/lib.rs +++ b/examples/balls/smatmul/verify/casegen/src/lib.rs @@ -10,12 +10,12 @@ thread_local! { } #[no_mangle] -pub extern "C" fn matrix_case_load(seed: u32, index: u32, bid: u32) -> i32 { - let case = casegen::gen_case(seed, index, bid); +pub extern "C" fn smatmul_case_load(seed: u32, index: u32, bid: u32, out_bw: u32) -> i32 { + let case = casegen::gen_case(seed, index, bid, out_bw as usize); let na = case.cmd.num_a_words as usize; let nb = case.cmd.num_b_words as usize; if na > MAX_WORDS || nb > MAX_WORDS { - panic!("matrix_case_load: word count out of range a={na} b={nb}"); + panic!("smatmul_case_load: word count out of range a={na} b={nb}"); } CURRENT.with(|c| *c.borrow_mut() = Some(case)); 0 @@ -24,107 +24,107 @@ pub extern "C" fn matrix_case_load(seed: u32, index: u32, bid: u32) -> i32 { fn current_case R, R>(f: F) -> R { CURRENT.with(|c| match c.borrow().as_ref() { Some(case) => f(case), - None => panic!("matrix_case: no case loaded; call matrix_case_load first"), + None => panic!("matrix_case: no case loaded; call smatmul_case_load first"), }) } #[no_mangle] -pub extern "C" fn matrix_case_cmd(out_ptr: *mut MatrixCmd) { +pub extern "C" fn smatmul_case_cmd(out_ptr: *mut MatrixCmd) { current_case(|case| unsafe { if out_ptr.is_null() { - panic!("matrix_case_cmd: null out_ptr"); + panic!("smatmul_case_cmd: null out_ptr"); } *out_ptr = case.cmd; }); } #[no_mangle] -pub extern "C" fn matrix_case_a_word_lo(word_index: u32) -> u64 { +pub extern "C" fn smatmul_case_a_word_lo(word_index: u32) -> u64 { if word_index as usize >= MAX_WORDS { - panic!("matrix_case_a_word_lo: word_index out of range"); + panic!("smatmul_case_a_word_lo: word_index out of range"); } current_case(|case| case.a_word_lo(word_index as usize)) } #[no_mangle] -pub extern "C" fn matrix_case_a_word_hi(word_index: u32) -> u64 { +pub extern "C" fn smatmul_case_a_word_hi(word_index: u32) -> u64 { if word_index as usize >= MAX_WORDS { - panic!("matrix_case_a_word_hi: word_index out of range"); + panic!("smatmul_case_a_word_hi: word_index out of range"); } current_case(|case| case.a_word_hi(word_index as usize)) } #[no_mangle] -pub extern "C" fn matrix_case_b_word_lo(word_index: u32) -> u64 { +pub extern "C" fn smatmul_case_b_word_lo(word_index: u32) -> u64 { if word_index as usize >= MAX_WORDS { - panic!("matrix_case_b_word_lo: word_index out of range"); + panic!("smatmul_case_b_word_lo: word_index out of range"); } current_case(|case| case.b_word_lo(word_index as usize)) } #[no_mangle] -pub extern "C" fn matrix_case_b_word_hi(word_index: u32) -> u64 { +pub extern "C" fn smatmul_case_b_word_hi(word_index: u32) -> u64 { if word_index as usize >= MAX_WORDS { - panic!("matrix_case_b_word_hi: word_index out of range"); + panic!("smatmul_case_b_word_hi: word_index out of range"); } current_case(|case| case.b_word_hi(word_index as usize)) } #[no_mangle] -pub extern "C" fn matrix_case_num_writes() -> u32 { +pub extern "C" fn smatmul_case_num_writes() -> u32 { current_case(|case| case.cmd.num_writes) } #[no_mangle] -pub extern "C" fn matrix_case_write_group(i: u32) -> u32 { +pub extern "C" fn smatmul_case_write_group(i: u32) -> u32 { current_case(|case| { let idx = i as usize; if idx >= case.writes.len() { - panic!("matrix_case_write_group: index out of range"); + panic!("smatmul_case_write_group: index out of range"); } case.writes[idx].group }) } #[no_mangle] -pub extern "C" fn matrix_case_write_addr(i: u32) -> u32 { +pub extern "C" fn smatmul_case_write_addr(i: u32) -> u32 { current_case(|case| { let idx = i as usize; if idx >= case.writes.len() { - panic!("matrix_case_write_addr: index out of range"); + panic!("smatmul_case_write_addr: index out of range"); } case.writes[idx].addr }) } #[no_mangle] -pub extern "C" fn matrix_case_write_data_lo(i: u32) -> u64 { +pub extern "C" fn smatmul_case_write_data_lo(i: u32) -> u64 { current_case(|case| { let idx = i as usize; if idx >= case.writes.len() { - panic!("matrix_case_write_data_lo: index out of range"); + panic!("smatmul_case_write_data_lo: index out of range"); } u64::from_le_bytes(case.writes[idx].data[0..8].try_into().unwrap()) }) } #[no_mangle] -pub extern "C" fn matrix_case_write_data_hi(i: u32) -> u64 { +pub extern "C" fn smatmul_case_write_data_hi(i: u32) -> u64 { current_case(|case| { let idx = i as usize; if idx >= case.writes.len() { - panic!("matrix_case_write_data_hi: index out of range"); + panic!("smatmul_case_write_data_hi: index out of range"); } u64::from_le_bytes(case.writes[idx].data[8..16].try_into().unwrap()) }) } #[no_mangle] -pub extern "C" fn matrix_case_write_mask(i: u32) -> u32 { +pub extern "C" fn smatmul_case_write_mask(i: u32) -> u32 { current_case(|case| { let idx = i as usize; if idx >= case.writes.len() { - panic!("matrix_case_write_mask: index out of range"); + panic!("smatmul_case_write_mask: index out of range"); } case.writes[idx].mask as u32 }) @@ -136,14 +136,13 @@ mod tests { #[test] fn dpi_load_then_cmd_and_words() { - assert_eq!(matrix_case_load(0x1234, 0, 1), 0); + assert_eq!(smatmul_case_load(0x1234, 0, 1, 2), 0); let mut cmd = MatrixCmd { bid: 0, - funct7: 0, + ws: 0, m: 0, n: 0, k: 0, - mode: 0, op1_bank: 0, op2_bank: 0, wr_bank: 0, @@ -156,12 +155,12 @@ mod tests { num_b_words: 0, num_writes: 0, }; - matrix_case_cmd(&mut cmd as *mut MatrixCmd); - assert_eq!(cmd.funct7, 65); + smatmul_case_cmd(&mut cmd as *mut MatrixCmd); + assert_eq!(cmd.ws, 0); assert_eq!(cmd.num_writes, 4); - let _ = matrix_case_a_word_lo(0); - let _ = matrix_case_b_word_lo(0); - assert_eq!(matrix_case_write_group(0), 0); + let _ = smatmul_case_a_word_lo(0); + let _ = smatmul_case_b_word_lo(0); + assert_eq!(smatmul_case_write_group(0), 0); } #[test] diff --git a/examples/balls/smatmul/verify/casegen/src/model.rs b/examples/balls/smatmul/verify/casegen/src/model.rs new file mode 100644 index 000000000..abe6a6192 --- /dev/null +++ b/examples/balls/smatmul/verify/casegen/src/model.rs @@ -0,0 +1,108 @@ +pub const TILE: usize = 16; +pub const BANK_ROW_BYTES: usize = 16; +pub const VALUES_PER_WORD: usize = 4; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct WriteExp { + pub group: u32, + pub addr: u32, + pub data: [u8; 16], + pub mask: u16, +} + +pub fn pack_a(src: &[i8], rows: usize, k: usize) -> Vec { + assert_eq!(rows % TILE, 0); + assert_eq!(k % TILE, 0); + let mut packed = vec![0; rows * k]; + for row in 0..rows { + for column in 0..k { + let line = (row / TILE * (k / TILE) + column / TILE) * TILE + row % TILE; + packed[line * TILE + column % TILE] = src[row * k + column] as u8; + } + } + packed +} + +pub fn pack_b_os(src: &[i8], k: usize) -> Vec { + assert_eq!(k % TILE, 0); + let mut packed = vec![0; k * TILE]; + for row in 0..k { + for column in 0..TILE { + packed[row * TILE + column] = src[row * TILE + column] as u8; + } + } + packed +} + +pub fn pack_b_ws(src: &[i8], k: usize, columns: usize) -> Vec { + assert_eq!(k, TILE); + assert_eq!(columns % TILE, 0); + let mut packed = vec![0; k * columns]; + for panel in 0..columns / TILE { + for row in 0..k { + for column in 0..TILE { + packed[(panel * TILE + row) * TILE + column] = + src[row * columns + panel * TILE + column] as u8; + } + } + } + packed +} + +pub fn matmul(a: &[i8], b: &[i8], rows: usize, columns: usize, k: usize) -> Vec> { + let mut c = vec![vec![0; columns]; rows]; + for row in 0..rows { + for column in 0..columns { + for reduction in 0..k { + c[row][column] += a[row * k + reduction] as i32 * b[reduction * columns + column] as i32; + } + } + } + c +} + +pub fn emit_writes(c: &[Vec], ws: bool, out_bw: usize) -> Vec { + assert!(matches!(out_bw, 1 | 2 | 4)); + let rows = c.len(); + let columns = c[0].len(); + assert_eq!(columns % TILE, 0); + let rounds = VALUES_PER_WORD / out_bw; + let mut writes = Vec::new(); + for panel in 0..columns / TILE { + for row in 0..rows { + for round in 0..rounds { + let address = if ws { + panel * TILE * rounds + row * rounds + round + } else { + (row / TILE * TILE + row % TILE) * rounds + round + }; + for group in 0..out_bw { + let word = round * out_bw + group; + let mut data = [0; BANK_ROW_BYTES]; + for lane in 0..VALUES_PER_WORD { + let value = c[row][panel * TILE + word * VALUES_PER_WORD + lane]; + data[lane * 4..lane * 4 + 4].copy_from_slice(&value.to_le_bytes()); + } + writes.push(WriteExp { group: group as u32, addr: address as u32, data, mask: 0xffff }); + } + } + } + } + writes +} + +pub fn words(data: &[u8]) -> usize { + assert_eq!(data.len() % BANK_ROW_BYTES, 0); + data.len() / BANK_ROW_BYTES +} + +pub fn encode_rs1(op1: u32, op2: u32, wr: u32) -> u64 { + assert!(op1 < 1024 && op2 < 1024 && wr < 1024); + u64::from(op1) | (u64::from(op2) << 10) | (u64::from(wr) << 20) +} + +pub fn encode_rs2(rows: u32, columns: u32, k: u32) -> u64 { + assert!(rows > 0 && columns > 0 && k > 0); + assert!(rows <= 0xfff && columns <= 0xfff && k <= 0xfff); + u64::from(rows) | (u64::from(columns) << 12) | (u64::from(k) << 24) +} diff --git a/examples/balls/smatmul/verify/filelists/smatmul_ball.f b/examples/balls/smatmul/verify/filelists/smatmul_ball.f new file mode 100644 index 000000000..f367f7fca --- /dev/null +++ b/examples/balls/smatmul/verify/filelists/smatmul_ball.f @@ -0,0 +1,26 @@ ++incdir+@UVM@/src ++incdir+@UVM@/src/agents/cmd ++incdir+@UVM@/src/agents/mem ++incdir+@UVM@/src/agents/resp ++incdir+@UVM@/src/cov ++incdir+@UVM@/src/env ++incdir+src ++incdir+src/pkg ++incdir+src/common ++incdir+src/seq ++incdir+src/cov ++incdir+src/env ++incdir+src/tests ++incdir+@RTL@ + +@UVM@/src/bb_blink_if.sv +@UVM@/src/bb_uvm_pkg.sv + +@RTL@/PE.sv +@RTL@/Array.sv +@RTL@/SMatMulUnit.sv +@RTL@/SMatMulBall.sv +@RTL@/accumulator_16x512.sv + +src/pkg/smatmul_pkg.sv +src/tb_top.sv diff --git a/examples/balls/smatmul/verify/src/common/smatmul_defs.svh b/examples/balls/smatmul/verify/src/common/smatmul_defs.svh new file mode 100644 index 000000000..920c1a418 --- /dev/null +++ b/examples/balls/smatmul/verify/src/common/smatmul_defs.svh @@ -0,0 +1,60 @@ +typedef struct { + int unsigned bid; + int unsigned ws; + int unsigned m; + int unsigned n; + int unsigned k; + int unsigned op1_bank; + int unsigned op2_bank; + int unsigned wr_bank; + int unsigned rob_id; + int unsigned rs1_lo; + int unsigned rs1_hi; + int unsigned rs2_lo; + int unsigned rs2_hi; + int unsigned num_a_words; + int unsigned num_b_words; + int unsigned num_writes; +} matrix_cmd_dpi_t; + +import "DPI-C" function int smatmul_case_load( + input int unsigned seed, + input int unsigned index, + input int unsigned bid, + input int unsigned out_bw +); +import "DPI-C" function void smatmul_case_cmd(output matrix_cmd_dpi_t cmd); +import "DPI-C" function longint unsigned smatmul_case_a_word_lo(input int unsigned word_index); +import "DPI-C" function longint unsigned smatmul_case_a_word_hi(input int unsigned word_index); +import "DPI-C" function longint unsigned smatmul_case_b_word_lo(input int unsigned word_index); +import "DPI-C" function longint unsigned smatmul_case_b_word_hi(input int unsigned word_index); +import "DPI-C" function int unsigned smatmul_case_num_writes(); +import "DPI-C" function int unsigned smatmul_case_write_group(input int unsigned i); +import "DPI-C" function int unsigned smatmul_case_write_addr(input int unsigned i); +import "DPI-C" function longint unsigned smatmul_case_write_data_lo(input int unsigned i); +import "DPI-C" function longint unsigned smatmul_case_write_data_hi(input int unsigned i); +import "DPI-C" function int unsigned smatmul_case_write_mask(input int unsigned i); + +`ifndef SMATMUL_OS_FUNCT7 +`error "SMATMUL_OS_FUNCT7 must be provided by the selected Core ballISA" +`endif +`ifndef SMATMUL_WS_FUNCT7 +`error "SMATMUL_WS_FUNCT7 must be provided by the selected Core ballISA" +`endif +`ifndef SMATMUL_OUT_BW +`error "SMATMUL_OUT_BW must be provided by the selected Core ball mapping" +`endif +localparam int SMATMUL_OS_CORE_FUNCT7 = `SMATMUL_OS_FUNCT7; +localparam int SMATMUL_WS_CORE_FUNCT7 = `SMATMUL_WS_FUNCT7; +localparam int SMATMUL_OUT_BW = `SMATMUL_OUT_BW; +localparam int MATRIX_MAX_WORDS = 32; +localparam int MATRIX_TIMEOUT_CYCLES = 100000; +localparam int MATRIX_SEED = 32'hCAFE_BABE; + +function automatic int unsigned matrix_require_bid(); + int unsigned bid; + if (!$value$plusargs("BID=%d", bid)) begin + `uvm_fatal("BID", "missing required plusarg +BID=") + end + return bid; +endfunction diff --git a/examples/balls/matrix/verify/src/common/matrix_items.svh b/examples/balls/smatmul/verify/src/common/smatmul_items.svh similarity index 73% rename from examples/balls/matrix/verify/src/common/matrix_items.svh rename to examples/balls/smatmul/verify/src/common/smatmul_items.svh index b7216c6e2..a87dd85d8 100644 --- a/examples/balls/matrix/verify/src/common/matrix_items.svh +++ b/examples/balls/smatmul/verify/src/common/smatmul_items.svh @@ -1,10 +1,9 @@ -class matrix_cmd_item extends bb_blink_cmd_item; - `uvm_object_utils(matrix_cmd_item) +class smatmul_cmd_item extends bb_blink_cmd_item; + `uvm_object_utils(smatmul_cmd_item) bit [31:0] m; bit [31:0] n; bit [31:0] k; - bit [31:0] mode; bit [31:0] num_a_words; bit [31:0] num_b_words; bit [31:0] num_writes; @@ -12,7 +11,7 @@ class matrix_cmd_item extends bb_blink_cmd_item; bit [127:0] b_words[MATRIX_MAX_WORDS]; constraint legal_c { - funct7 == MATRIX_FUNCT7[6:0]; + funct7 inside {SMATMUL_OS_CORE_FUNCT7[6:0], SMATMUL_WS_CORE_FUNCT7[6:0]}; op1_en == 1'b1; op2_en == 1'b1; wr_spad_en == 1'b1; @@ -20,7 +19,7 @@ class matrix_cmd_item extends bb_blink_cmd_item; op2_from_spad == 1'b1; op1_col == 5'd1; op2_col == 5'd1; - wr_col == 5'd4; + wr_col == SMATMUL_OUT_BW; meta_bank == 5'd0; iter == 34'd0; special == 64'd0; @@ -28,7 +27,7 @@ class matrix_cmd_item extends bb_blink_cmd_item; sub_rob_id == 8'h00; } - function new(string name = "matrix_cmd_item"); + function new(string name = "smatmul_cmd_item"); super.new(name); endfunction @@ -39,24 +38,26 @@ class matrix_cmd_item extends bb_blink_cmd_item; longint unsigned w_hi; int i; - rc = matrix_case_load(seed, index, bid); + rc = smatmul_case_load(seed, index, bid, SMATMUL_OUT_BW); if (rc != 0) begin - `uvm_fatal("CASE", $sformatf("matrix_case_load returned %0d for index %0d", rc, index)) + `uvm_fatal("CASE", $sformatf("smatmul_case_load returned %0d for index %0d", rc, index)) end - matrix_case_cmd(cmd); + smatmul_case_cmd(cmd); - bid = cmd.bid[4:0]; - funct7 = cmd.funct7[6:0]; + bid = cmd.bid[4:0]; + if (cmd.ws > 1) begin + `uvm_fatal("CASE", $sformatf("invalid ws selector %0d", cmd.ws)) + end + funct7 = cmd.ws ? SMATMUL_WS_CORE_FUNCT7[6:0] : SMATMUL_OS_CORE_FUNCT7[6:0]; m = cmd.m; n = cmd.n; k = cmd.k; - mode = cmd.mode; op1_bank = cmd.op1_bank[4:0]; op2_bank = cmd.op2_bank[4:0]; wr_bank = cmd.wr_bank[4:0]; op1_col = 5'd1; op2_col = 5'd1; - wr_col = 5'd4; + wr_col = SMATMUL_OUT_BW; rob_id = cmd.rob_id[3:0]; rs1 = {cmd.rs1_hi, cmd.rs1_lo}; rs2 = {cmd.rs2_hi, cmd.rs2_lo}; @@ -86,27 +87,26 @@ class matrix_cmd_item extends bb_blink_cmd_item; end for (i = 0; i < num_a_words; i++) begin - w_lo = matrix_case_a_word_lo(i); - w_hi = matrix_case_a_word_hi(i); + w_lo = smatmul_case_a_word_lo(i); + w_hi = smatmul_case_a_word_hi(i); a_words[i] = {w_hi, w_lo}; end for (i = 0; i < num_b_words; i++) begin - w_lo = matrix_case_b_word_lo(i); - w_hi = matrix_case_b_word_hi(i); + w_lo = smatmul_case_b_word_lo(i); + w_hi = smatmul_case_b_word_hi(i); b_words[i] = {w_hi, w_lo}; end endfunction function void do_copy(uvm_object rhs); - matrix_cmd_item rhs_; + smatmul_cmd_item rhs_; super.do_copy(rhs); if (!$cast(rhs_, rhs)) begin - `uvm_fatal("COPY", "rhs is not matrix_cmd_item") + `uvm_fatal("COPY", "rhs is not smatmul_cmd_item") end m = rhs_.m; n = rhs_.n; k = rhs_.k; - mode = rhs_.mode; num_a_words = rhs_.num_a_words; num_b_words = rhs_.num_b_words; num_writes = rhs_.num_writes; diff --git a/examples/balls/matrix/verify/src/cov/matrix_cov.svh b/examples/balls/smatmul/verify/src/cov/smatmul_cov.svh similarity index 59% rename from examples/balls/matrix/verify/src/cov/matrix_cov.svh rename to examples/balls/smatmul/verify/src/cov/smatmul_cov.svh index 231d879dc..3007a4502 100644 --- a/examples/balls/matrix/verify/src/cov/matrix_cov.svh +++ b/examples/balls/smatmul/verify/src/cov/smatmul_cov.svh @@ -1,10 +1,8 @@ -class matrix_cov extends uvm_component; - `uvm_component_utils(matrix_cov) +class smatmul_cov extends uvm_component; + `uvm_component_utils(smatmul_cov) - uvm_analysis_imp_cmd #(bb_blink_cmd_item, matrix_cov) cmd_imp; + uvm_analysis_imp_cmd #(bb_blink_cmd_item, smatmul_cov) cmd_imp; - bit saw_os; - bit saw_ws; bit saw_m[int]; bit saw_n[int]; bit saw_k[int]; @@ -16,16 +14,16 @@ class matrix_cov extends uvm_component; endfunction function void write_cmd(bb_blink_cmd_item item); - int unsigned mode; int unsigned m; int unsigned n; int unsigned k; - mode = int'(item.rs2[36]); m = int'(item.rs2[11:0]); n = int'(item.rs2[23:12]); k = int'(item.rs2[35:24]); - if (mode == 0) saw_os = 1'b1; - if (mode == 1) saw_ws = 1'b1; + if (item.rs2[63:36] != 0) `uvm_fatal("COV", "rs2[63:36] must be 0") + if (m != 16 && m != 32) `uvm_fatal("COV", $sformatf("rows must be 16 or 32: %0d", m)) + if (n != 16 && n != 32) `uvm_fatal("COV", $sformatf("cols must be 16 or 32: %0d", n)) + if (k != 16) `uvm_fatal("COV", $sformatf("reduction must be 16: %0d", k)) saw_m[m] = 1'b1; saw_n[n] = 1'b1; saw_k[k] = 1'b1; @@ -33,18 +31,16 @@ class matrix_cov extends uvm_component; endfunction function void check_phase(uvm_phase phase); - int unsigned need[5] = '{1, 2, 4, 8, 16}; + int unsigned need[2] = '{16, 32}; int i; super.check_phase(phase); if (cmd_count == 0) `uvm_fatal("COV", "no cmd observed") - if (!saw_os || !saw_ws) `uvm_fatal("COV", "mode bins incomplete (need OS and WS)") - for (i = 0; i < 5; i++) begin + for (i = 0; i < 2; i++) begin if (!saw_m.exists(need[i]) || !saw_m[need[i]]) `uvm_fatal("COV", $sformatf("missing M bin %0d", need[i])) if (!saw_n.exists(need[i]) || !saw_n[need[i]]) `uvm_fatal("COV", $sformatf("missing N bin %0d", need[i])) - if (!saw_k.exists(need[i]) || !saw_k[need[i]]) - `uvm_fatal("COV", $sformatf("missing K bin %0d", need[i])) end + if (!saw_k.exists(16) || !saw_k[16]) `uvm_fatal("COV", "missing K bin 16") endfunction endclass diff --git a/examples/balls/matrix/verify/src/env/matrix_env.svh b/examples/balls/smatmul/verify/src/env/smatmul_env.svh similarity index 70% rename from examples/balls/matrix/verify/src/env/matrix_env.svh rename to examples/balls/smatmul/verify/src/env/smatmul_env.svh index cd88befc5..633615ee4 100644 --- a/examples/balls/matrix/verify/src/env/matrix_env.svh +++ b/examples/balls/smatmul/verify/src/env/smatmul_env.svh @@ -1,8 +1,8 @@ -class matrix_env extends bb_blink_env #(2, 4); - `uvm_component_utils(matrix_env) +class smatmul_env extends bb_blink_env #(2, 2); + `uvm_component_utils(smatmul_env) - matrix_scoreboard scb; - matrix_cov mcov; + smatmul_scoreboard scb; + smatmul_cov mcov; function new(string name, uvm_component parent); super.new(name, parent); @@ -10,8 +10,8 @@ class matrix_env extends bb_blink_env #(2, 4); function void build_phase(uvm_phase phase); super.build_phase(phase); - scb = matrix_scoreboard::type_id::create("scb", this); - mcov = matrix_cov::type_id::create("mcov", this); + scb = smatmul_scoreboard::type_id::create("scb", this); + mcov = smatmul_cov::type_id::create("mcov", this); endfunction function void connect_phase(uvm_phase phase); diff --git a/examples/balls/matrix/verify/src/env/matrix_scoreboard.svh b/examples/balls/smatmul/verify/src/env/smatmul_scoreboard.svh similarity index 84% rename from examples/balls/matrix/verify/src/env/matrix_scoreboard.svh rename to examples/balls/smatmul/verify/src/env/smatmul_scoreboard.svh index 2a8a9153f..c05010cf0 100644 --- a/examples/balls/matrix/verify/src/env/matrix_scoreboard.svh +++ b/examples/balls/smatmul/verify/src/env/smatmul_scoreboard.svh @@ -1,15 +1,15 @@ -class matrix_scoreboard extends uvm_scoreboard; - `uvm_component_utils(matrix_scoreboard) +class smatmul_scoreboard extends uvm_scoreboard; + `uvm_component_utils(smatmul_scoreboard) - uvm_analysis_imp_stim #(bb_blink_cmd_item, matrix_scoreboard) stim_imp; - uvm_analysis_imp_cmd #(bb_blink_cmd_item, matrix_scoreboard) cmd_imp; - uvm_analysis_imp_read #(bb_blink_read_item, matrix_scoreboard) read_imp; - uvm_analysis_imp_write #(bb_blink_write_item, matrix_scoreboard) write_imp; - uvm_analysis_imp_resp #(bb_blink_resp_item, matrix_scoreboard) resp_imp; + uvm_analysis_imp_stim #(bb_blink_cmd_item, smatmul_scoreboard) stim_imp; + uvm_analysis_imp_cmd #(bb_blink_cmd_item, smatmul_scoreboard) cmd_imp; + uvm_analysis_imp_read #(bb_blink_read_item, smatmul_scoreboard) read_imp; + uvm_analysis_imp_write #(bb_blink_write_item, smatmul_scoreboard) write_imp; + uvm_analysis_imp_resp #(bb_blink_resp_item, smatmul_scoreboard) resp_imp; - bb_blink_mem_model #(2, 4) mem_model; + bb_blink_mem_model #(2, 2) mem_model; - matrix_cmd_item stim_q[$]; + smatmul_cmd_item stim_q[$]; int unsigned exp_group[$]; int unsigned exp_addr[$]; bit [127:0] exp_data[$]; @@ -31,8 +31,8 @@ class matrix_scoreboard extends uvm_scoreboard; endfunction function void write_stim(bb_blink_cmd_item item); - matrix_cmd_item mitem; - matrix_cmd_item clone; + smatmul_cmd_item mitem; + smatmul_cmd_item clone; int i; longint unsigned d_lo; longint unsigned d_hi; @@ -41,7 +41,7 @@ class matrix_scoreboard extends uvm_scoreboard; `uvm_fatal("SCB", "single outstanding command supported") end if (!$cast(mitem, item)) begin - `uvm_fatal("SCB", "stim item is not matrix_cmd_item") + `uvm_fatal("SCB", "stim item is not smatmul_cmd_item") end if (!$cast(clone, mitem.clone())) begin `uvm_fatal("SCB", "failed to clone stimulus item") @@ -66,25 +66,25 @@ class matrix_scoreboard extends uvm_scoreboard; exp_mask.delete(); exp_seen.delete(); for (i = 0; i < clone.num_writes; i++) begin - exp_group.push_back(matrix_case_write_group(i)); - exp_addr.push_back(matrix_case_write_addr(i)); - d_lo = matrix_case_write_data_lo(i); - d_hi = matrix_case_write_data_hi(i); + exp_group.push_back(smatmul_case_write_group(i)); + exp_addr.push_back(smatmul_case_write_addr(i)); + d_lo = smatmul_case_write_data_lo(i); + d_hi = smatmul_case_write_data_hi(i); exp_data.push_back({d_hi, d_lo}); - exp_mask.push_back(matrix_case_write_mask(i) [15:0]); + exp_mask.push_back(smatmul_case_write_mask(i) [15:0]); exp_seen.push_back(1'b0); end expected_writes = clone.num_writes; endfunction function void write_cmd(bb_blink_cmd_item item); - matrix_cmd_item stim; + smatmul_cmd_item stim; stim = current_stim("CMD"); check_cmd(item, stim); cmd_count++; endfunction - function void check_cmd(bb_blink_cmd_item got, matrix_cmd_item exp); + function void check_cmd(bb_blink_cmd_item got, smatmul_cmd_item exp); if (got.bid !== exp.bid) `uvm_fatal("CMD", $sformatf("bid mismatch: got %0d exp %0d", got.bid, exp.bid)) if (got.funct7 !== exp.funct7) @@ -108,7 +108,7 @@ class matrix_scoreboard extends uvm_scoreboard; endfunction function void write_read(bb_blink_read_item item); - matrix_cmd_item stim; + smatmul_cmd_item stim; stim = current_stim("READ"); if (item.bank_id == stim.op1_bank) begin if (item.addr >= stim.num_a_words[9:0]) @@ -129,7 +129,7 @@ class matrix_scoreboard extends uvm_scoreboard; endfunction function void write_write(bb_blink_write_item item); - matrix_cmd_item stim; + smatmul_cmd_item stim; int i; bit found; stim = current_stim("WRITE"); @@ -162,7 +162,7 @@ class matrix_scoreboard extends uvm_scoreboard; endfunction function void write_resp(bb_blink_resp_item item); - matrix_cmd_item stim; + smatmul_cmd_item stim; stim = current_stim("RESP"); if (item.rob_id !== stim.rob_id) `uvm_fatal("RESP", $sformatf("rob_id mismatch: got %0d exp %0d", item.rob_id, stim.rob_id)) @@ -170,7 +170,7 @@ class matrix_scoreboard extends uvm_scoreboard; resp_count++; endfunction - function matrix_cmd_item current_stim(string tag); + function smatmul_cmd_item current_stim(string tag); if (stim_q.size() == 0) begin `uvm_fatal("SCB", $sformatf("%s observed before stimulus", tag)) return null; diff --git a/examples/balls/smatmul/verify/src/pkg/smatmul_pkg.sv b/examples/balls/smatmul/verify/src/pkg/smatmul_pkg.sv new file mode 100644 index 000000000..400d1f37f --- /dev/null +++ b/examples/balls/smatmul/verify/src/pkg/smatmul_pkg.sv @@ -0,0 +1,13 @@ +package smatmul_pkg; + import uvm_pkg::*; + import bb_uvm_pkg::*; + `include "uvm_macros.svh" + + `include "common/smatmul_defs.svh" + `include "common/smatmul_items.svh" + `include "seq/smatmul_sequences.svh" + `include "cov/smatmul_cov.svh" + `include "env/smatmul_scoreboard.svh" + `include "env/smatmul_env.svh" + `include "tests/smatmul_test.svh" +endpackage diff --git a/examples/balls/matrix/verify/src/seq/matrix_sequences.svh b/examples/balls/smatmul/verify/src/seq/smatmul_sequences.svh similarity index 83% rename from examples/balls/matrix/verify/src/seq/matrix_sequences.svh rename to examples/balls/smatmul/verify/src/seq/smatmul_sequences.svh index 8e951615f..9f7638ffb 100644 --- a/examples/balls/matrix/verify/src/seq/matrix_sequences.svh +++ b/examples/balls/smatmul/verify/src/seq/smatmul_sequences.svh @@ -11,8 +11,8 @@ class matrix_basic_seq extends uvm_sequence #(bb_blink_cmd_item); endfunction task body(); - matrix_cmd_item item; - item = matrix_cmd_item::type_id::create("item"); + smatmul_cmd_item item; + item = smatmul_cmd_item::type_id::create("item"); start_item(item); item.load_rust_case(seed, case_index, bid); finish_item(item); diff --git a/examples/balls/matrix/verify/src/tb_top.sv b/examples/balls/smatmul/verify/src/tb_top.sv similarity index 62% rename from examples/balls/matrix/verify/src/tb_top.sv rename to examples/balls/smatmul/verify/src/tb_top.sv index 515585153..cf87984af 100644 --- a/examples/balls/matrix/verify/src/tb_top.sv +++ b/examples/balls/smatmul/verify/src/tb_top.sv @@ -1,12 +1,13 @@ module tb_top; import uvm_pkg::*; - import matrix_pkg::*; + import smatmul_pkg::*; - bb_blink_if #(2, 4) intf (); + bb_blink_if #(2, 2) intf (); - MatrixBall dut ( + SMatMulBall dut ( .clock(intf.clock), .reset(intf.reset), + .io_channelReady(1'b1), .io_cmdReq_ready(intf.cmd_req_ready), .io_cmdReq_valid(intf.cmd_req_valid), .io_cmdReq_bits_cmd_bid(intf.cmd_req_bits_cmd_bid), @@ -105,62 +106,7 @@ module tb_top; .io_bankWrite_1_io_resp_ready(intf.bank_write_resp_ready[1]), .io_bankWrite_1_io_resp_valid(intf.bank_write_resp_valid[1]), .io_bankWrite_1_io_resp_bits_ok(intf.bank_write_resp_ok[1]), - .io_bankWrite_2_bank_id(intf.bank_write_bank_id[2][3:0]), - .io_bankWrite_2_rob_id(intf.bank_write_rob_id[2]), - .io_bankWrite_2_group_id(intf.bank_write_group_id[2][3:0]), - .io_bankWrite_2_io_req_ready(intf.bank_write_req_ready[2]), - .io_bankWrite_2_io_req_valid(intf.bank_write_req_valid[2]), - .io_bankWrite_2_io_req_bits_addr(intf.bank_write_req_addr[2]), - .io_bankWrite_2_io_req_bits_mask_0(intf.bank_write_req_mask[2][0]), - .io_bankWrite_2_io_req_bits_mask_1(intf.bank_write_req_mask[2][1]), - .io_bankWrite_2_io_req_bits_mask_2(intf.bank_write_req_mask[2][2]), - .io_bankWrite_2_io_req_bits_mask_3(intf.bank_write_req_mask[2][3]), - .io_bankWrite_2_io_req_bits_mask_4(intf.bank_write_req_mask[2][4]), - .io_bankWrite_2_io_req_bits_mask_5(intf.bank_write_req_mask[2][5]), - .io_bankWrite_2_io_req_bits_mask_6(intf.bank_write_req_mask[2][6]), - .io_bankWrite_2_io_req_bits_mask_7(intf.bank_write_req_mask[2][7]), - .io_bankWrite_2_io_req_bits_mask_8(intf.bank_write_req_mask[2][8]), - .io_bankWrite_2_io_req_bits_mask_9(intf.bank_write_req_mask[2][9]), - .io_bankWrite_2_io_req_bits_mask_10(intf.bank_write_req_mask[2][10]), - .io_bankWrite_2_io_req_bits_mask_11(intf.bank_write_req_mask[2][11]), - .io_bankWrite_2_io_req_bits_mask_12(intf.bank_write_req_mask[2][12]), - .io_bankWrite_2_io_req_bits_mask_13(intf.bank_write_req_mask[2][13]), - .io_bankWrite_2_io_req_bits_mask_14(intf.bank_write_req_mask[2][14]), - .io_bankWrite_2_io_req_bits_mask_15(intf.bank_write_req_mask[2][15]), - .io_bankWrite_2_io_req_bits_data(intf.bank_write_req_data[2]), - .io_bankWrite_2_io_resp_ready(intf.bank_write_resp_ready[2]), - .io_bankWrite_2_io_resp_valid(intf.bank_write_resp_valid[2]), - .io_bankWrite_2_io_resp_bits_ok(intf.bank_write_resp_ok[2]), - .io_bankWrite_3_bank_id(intf.bank_write_bank_id[3][3:0]), - .io_bankWrite_3_rob_id(intf.bank_write_rob_id[3]), - .io_bankWrite_3_group_id(intf.bank_write_group_id[3][3:0]), - .io_bankWrite_3_io_req_ready(intf.bank_write_req_ready[3]), - .io_bankWrite_3_io_req_valid(intf.bank_write_req_valid[3]), - .io_bankWrite_3_io_req_bits_addr(intf.bank_write_req_addr[3]), - .io_bankWrite_3_io_req_bits_mask_0(intf.bank_write_req_mask[3][0]), - .io_bankWrite_3_io_req_bits_mask_1(intf.bank_write_req_mask[3][1]), - .io_bankWrite_3_io_req_bits_mask_2(intf.bank_write_req_mask[3][2]), - .io_bankWrite_3_io_req_bits_mask_3(intf.bank_write_req_mask[3][3]), - .io_bankWrite_3_io_req_bits_mask_4(intf.bank_write_req_mask[3][4]), - .io_bankWrite_3_io_req_bits_mask_5(intf.bank_write_req_mask[3][5]), - .io_bankWrite_3_io_req_bits_mask_6(intf.bank_write_req_mask[3][6]), - .io_bankWrite_3_io_req_bits_mask_7(intf.bank_write_req_mask[3][7]), - .io_bankWrite_3_io_req_bits_mask_8(intf.bank_write_req_mask[3][8]), - .io_bankWrite_3_io_req_bits_mask_9(intf.bank_write_req_mask[3][9]), - .io_bankWrite_3_io_req_bits_mask_10(intf.bank_write_req_mask[3][10]), - .io_bankWrite_3_io_req_bits_mask_11(intf.bank_write_req_mask[3][11]), - .io_bankWrite_3_io_req_bits_mask_12(intf.bank_write_req_mask[3][12]), - .io_bankWrite_3_io_req_bits_mask_13(intf.bank_write_req_mask[3][13]), - .io_bankWrite_3_io_req_bits_mask_14(intf.bank_write_req_mask[3][14]), - .io_bankWrite_3_io_req_bits_mask_15(intf.bank_write_req_mask[3][15]), - .io_bankWrite_3_io_req_bits_data(intf.bank_write_req_data[3]), - .io_bankWrite_3_io_resp_ready(intf.bank_write_resp_ready[3]), - .io_bankWrite_3_io_resp_valid(intf.bank_write_resp_valid[3]), - .io_bankWrite_3_io_resp_bits_ok(intf.bank_write_resp_ok[3]), - .io_subRobReq_ready(intf.sub_rob_req_ready), - .io_mmioRead_req_ready(intf.mmio_read_req_ready), - .io_mmioRead_resp_valid(intf.mmio_read_resp_valid), - .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data) + .io_subRobReq_ready(intf.sub_rob_req_ready) ); initial begin @@ -170,7 +116,7 @@ module tb_top; end initial begin - uvm_config_db#(virtual bb_blink_if #(2, 4))::set(null, "*", "vif", intf); - run_test("matrix_ball_test"); + uvm_config_db#(virtual bb_blink_if #(2, 2))::set(null, "*", "vif", intf); + run_test("smatmul_ball_test"); end endmodule diff --git a/examples/balls/matrix/verify/src/tests/matrix_test.svh b/examples/balls/smatmul/verify/src/tests/smatmul_test.svh similarity index 64% rename from examples/balls/matrix/verify/src/tests/matrix_test.svh rename to examples/balls/smatmul/verify/src/tests/smatmul_test.svh index 0e611d646..8fdeaf6bd 100644 --- a/examples/balls/matrix/verify/src/tests/matrix_test.svh +++ b/examples/balls/smatmul/verify/src/tests/smatmul_test.svh @@ -1,9 +1,9 @@ -class matrix_ball_test extends uvm_test; - `uvm_component_utils(matrix_ball_test) +class smatmul_ball_test extends uvm_test; + `uvm_component_utils(smatmul_ball_test) - typedef virtual bb_blink_if #(2, 4) vif_t; + typedef virtual bb_blink_if #(2, 2) vif_t; vif_t vif; - matrix_env env; + smatmul_env env; function new(string name, uvm_component parent); super.new(name, parent); @@ -14,7 +14,7 @@ class matrix_ball_test extends uvm_test; if (!uvm_config_db#(vif_t)::get(this, "", "vif", vif)) begin `uvm_fatal("NOVIF", "bb_blink_if not found") end - env = matrix_env::type_id::create("env", this); + env = smatmul_env::type_id::create("env", this); endfunction task run_phase(uvm_phase phase); @@ -27,14 +27,15 @@ class matrix_ball_test extends uvm_test; endtask task run_directed(int unsigned bid); - run_case(0, "DIR_OS_4x4", bid); - run_case(1, "DIR_WS_4x4", bid); - run_case(2, "DIR_OS_16x16", bid); + run_case(0, "DIR_OS_16x16", bid); + run_case(1, "DIR_OS_32x16", bid); + run_case(2, "DIR_WS_16x16", bid); + run_case(3, "DIR_WS_16x32", bid); endtask task run_random(int unsigned bid); int i; - for (i = 3; i <= 22; i++) begin + for (i = 4; i <= 23; i++) begin run_case(i, $sformatf("RND_%0d", i), bid); end endtask @@ -57,13 +58,14 @@ class matrix_ball_test extends uvm_test; if (cycles > MATRIX_TIMEOUT_CYCLES) begin `uvm_fatal( "TIMEOUT", - $sformatf("MatrixBall %s (idx=%0d) timeout: cmds=%0d r0=%0d r1=%0d wr=%0d/%0d resp=%0d", - label, idx, env.scb.cmd_count, env.scb.read_count[0], env.scb.read_count[1], - env.scb.write_count, env.scb.expected_writes, env.scb.resp_count)) + $sformatf( + "SMatMulBall %s (idx=%0d) timeout: cmds=%0d r0=%0d r1=%0d wr=%0d/%0d resp=%0d", + label, idx, env.scb.cmd_count, env.scb.read_count[0], env.scb.read_count[1], + env.scb.write_count, env.scb.expected_writes, env.scb.resp_count)) end end - `uvm_info("MATRIX", $sformatf("MatrixBall %s (idx=%0d) passed in %0d cycles", label, idx, + `uvm_info("MATRIX", $sformatf("SMatMulBall %s (idx=%0d) passed in %0d cycles", label, idx, cycles), UVM_LOW) endtask diff --git a/examples/balls/smatmul/workloads/ctests/CMakeLists.txt b/examples/balls/smatmul/workloads/ctests/CMakeLists.txt new file mode 100644 index 000000000..914dc0c7f --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/CMakeLists.txt @@ -0,0 +1,6 @@ +add_buckyball_ctests( + smatmul_16x16_test.c + smatmul_32x16x32_test.c + smatmul_ws_16x16_test.c + smatmul_ws_16x512_test.c +) diff --git a/examples/balls/smatmul/workloads/ctests/smatmul_16x16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_16x16_test.c new file mode 100644 index 000000000..7644b8f9d --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/smatmul_16x16_test.c @@ -0,0 +1,28 @@ +#include "smatmul_test_common.h" + +#define ROWS 16 +#define K 16 + +static elem_t a[ROWS * K] __attribute__((aligned(64))); +static elem_t b[K * MATRIX_TILE] __attribute__((aligned(64))); +static elem_t packed_a[ROWS * K] __attribute__((aligned(64))); +static elem_t packed_b[K * MATRIX_TILE] __attribute__((aligned(64))); +static result_t packed_c[ROWS * MATRIX_TILE] __attribute__((aligned(64))); +static result_t expected[ROWS * MATRIX_TILE] __attribute__((aligned(64))); +static result_t actual[ROWS * MATRIX_TILE] __attribute__((aligned(64))); + +int main(void) { + for (int i = 0; i < ROWS * K; ++i) + a[i] = (i * 7 % 23) - 11; + for (int i = 0; i < K * MATRIX_TILE; ++i) + b[i] = (i * 5 % 19) - 9; + cpu_matmul(a, b, expected, ROWS, MATRIX_TILE, K); + matrix_pack_a(a, packed_a, ROWS, K); + matrix_pack_b(b, packed_b, K); + matrix_hw(packed_a, packed_b, packed_c, ROWS, K); + matrix_unpack_c(packed_c, actual, ROWS); + if (!compare_u32_matrices(actual, expected, ROWS, MATRIX_TILE)) + return 1; + printf("smatmul_16x16 PASSED\n"); + return 0; +} diff --git a/examples/balls/smatmul/workloads/ctests/smatmul_32x16x32_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_32x16x32_test.c new file mode 100644 index 000000000..08c73d24d --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/smatmul_32x16x32_test.c @@ -0,0 +1,28 @@ +#include "smatmul_test_common.h" + +#define ROWS 32 +#define K 32 + +static elem_t a[ROWS * K] __attribute__((aligned(64))); +static elem_t b[K * MATRIX_TILE] __attribute__((aligned(64))); +static elem_t packed_a[ROWS * K] __attribute__((aligned(64))); +static elem_t packed_b[K * MATRIX_TILE] __attribute__((aligned(64))); +static result_t packed_c[ROWS * MATRIX_TILE] __attribute__((aligned(64))); +static result_t expected[ROWS * MATRIX_TILE] __attribute__((aligned(64))); +static result_t actual[ROWS * MATRIX_TILE] __attribute__((aligned(64))); + +int main(void) { + for (int i = 0; i < ROWS * K; ++i) + a[i] = (i * 11 % 29) - 14; + for (int i = 0; i < K * MATRIX_TILE; ++i) + b[i] = (i * 3 % 31) - 15; + cpu_matmul(a, b, expected, ROWS, MATRIX_TILE, K); + matrix_pack_a(a, packed_a, ROWS, K); + matrix_pack_b(b, packed_b, K); + matrix_hw(packed_a, packed_b, packed_c, ROWS, K); + matrix_unpack_c(packed_c, actual, ROWS); + if (!compare_u32_matrices(actual, expected, ROWS, MATRIX_TILE)) + return 1; + printf("smatmul_32x16x32 PASSED\n"); + return 0; +} diff --git a/examples/balls/matrix/workloads/ctests/matrix_bank_os_1024x16x16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_bank_os_1024x16x16_test.c similarity index 86% rename from examples/balls/matrix/workloads/ctests/matrix_bank_os_1024x16x16_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_bank_os_1024x16x16_test.c index 37c418ae9..3388da330 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_bank_os_1024x16x16_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_bank_os_1024x16x16_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #include #define M BANK_LINES @@ -28,12 +28,12 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, matrix_c_blocks(M, N), MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); + matrix_hw_ws(pa, pb, pc, M, N, K); matrix_unpack_c(pc, out, M, N); if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_bank_os_1024x16x16 FAILED\n"); + printf("smatmul_bank_os_1024x16x16 FAILED\n"); return 1; } - printf("matrix_bank_os_1024x16x16 PASSED\n"); + printf("smatmul_bank_os_1024x16x16 PASSED\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_16x16_ones.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_16x16_ones.c similarity index 84% rename from examples/balls/matrix/workloads/ctests/matrix_os_16x16_ones.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_16x16_ones.c index 54ddcd1a8..4244cb40d 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_16x16_ones.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_16x16_ones.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #include #define DIM 16 @@ -21,12 +21,12 @@ int main(void) { matrix_pack_a(a, pa, DIM, DIM); matrix_pack_b(b, pb, DIM, DIM); clear_u32_matrix(pc, DIM, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, DIM, DIM, DIM, 0); + matrix_hw_os(pa, pb, pc, DIM, DIM, DIM); matrix_unpack_c(pc, out, DIM, DIM); if (!compare_u32_matrices(out, exp_, DIM, DIM)) { - printf("matrix_os_16x16_ones FAILED\n"); + printf("smatmul_os_16x16_ones FAILED\n"); return 1; } - printf("matrix_os_16x16_ones PASSED\n"); + printf("smatmul_os_16x16_ones PASSED\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_ws_16x16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_16x16_test.c similarity index 95% rename from examples/balls/matrix/workloads/ctests/matrix_ws_16x16_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_16x16_test.c index 2a3e10171..013d4c33f 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_ws_16x16_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_16x16_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define DIM 16 @@ -58,12 +58,12 @@ int main(void) { matrix_pack_a(a, pa, DIM, DIM); matrix_pack_b(b, pb, DIM, DIM); clear_u32_matrix(pc, DIM, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, DIM, DIM, DIM, 1); + matrix_hw_os(pa, pb, pc, DIM, DIM, DIM); matrix_unpack_c(pc, out, DIM, DIM); if (!compare_u32_matrices(out, exp_, DIM, DIM)) { - printf("matrix_ws_16x16 FAILED\n"); + printf("smatmul_os_16x16 FAILED\n"); return 1; } - printf("matrix_ws_16x16 PASSED\n"); + printf("smatmul_os_16x16 PASSED\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_32x16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_32x16_test.c similarity index 97% rename from examples/balls/matrix/workloads/ctests/matrix_os_32x16_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_32x16_test.c index 29e09e92c..12981cf54 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_32x16_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_32x16_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 32 #define N 16 @@ -79,12 +79,12 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); + matrix_hw_os(pa, pb, pc, M, N, K); matrix_unpack_c(pc, out, M, N); if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_os_32x16 FAILED\n"); + printf("smatmul_os_32x16 FAILED\n"); return 1; } - printf("matrix_os_32x16 PASSED\n"); + printf("smatmul_os_32x16 PASSED\n"); return 0; } diff --git a/examples/balls/smatmul/workloads/ctests/smatmul_os_32x32_ones.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_32x32_ones.c new file mode 100644 index 000000000..a286d3ad4 --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_32x32_ones.c @@ -0,0 +1,42 @@ +#include "smatmul_test_common.h" + +#define DIM 32 +#define PANEL 16 + +static elem_t a[DIM * DIM] __attribute__((aligned(64))); +static elem_t b[DIM * DIM] __attribute__((aligned(64))); +static elem_t pa[64 * MATRIX_TILE] __attribute__((aligned(64))); +static elem_t bpanel[DIM * PANEL] __attribute__((aligned(64))); +static elem_t pb[32 * MATRIX_TILE] __attribute__((aligned(64))); +static result_t pc[DIM * MATRIX_ACC_LANES] __attribute__((aligned(64))); +static result_t panel_out[DIM * PANEL] __attribute__((aligned(64))); +static result_t out[DIM * DIM] __attribute__((aligned(64))); +static result_t exp_[DIM * DIM] __attribute__((aligned(64))); + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + init_ones_matrix(a, DIM, DIM); + init_ones_matrix(b, DIM, DIM); + cpu_matmul(a, b, exp_, DIM, DIM, DIM); + matrix_pack_a(a, pa, DIM, DIM); + for (int n0 = 0; n0 < DIM; n0 += PANEL) { + for (int r = 0; r < DIM; ++r) + for (int c = 0; c < PANEL; ++c) + bpanel[r * PANEL + c] = b[r * DIM + n0 + c]; + matrix_pack_b(bpanel, pb, DIM, PANEL); + clear_u32_matrix(pc, DIM, MATRIX_ACC_LANES); + matrix_hw_ws(pa, pb, pc, DIM, PANEL, DIM); + matrix_unpack_c(pc, panel_out, DIM, PANEL); + for (int r = 0; r < DIM; ++r) + for (int c = 0; c < PANEL; ++c) + out[r * DIM + n0 + c] = panel_out[r * PANEL + c]; + } + if (!compare_u32_matrices(out, exp_, DIM, DIM)) { + printf("smatmul_os_32x32_ones FAILED\n"); + return 1; + } + printf("smatmul_os_32x32_ones PASSED\n"); + return 0; +} diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_48x16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_48x16_test.c similarity index 80% rename from examples/balls/matrix/workloads/ctests/matrix_os_48x16_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_48x16_test.c index b1bce7247..1d7c6bba4 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_48x16_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_48x16_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 48 #define N 16 @@ -19,8 +19,8 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - printf("matrix_os_48x16 start\n"); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); - printf("matrix_os_48x16 DONE\n"); + printf("smatmul_os_48x16 start\n"); + matrix_hw_ws(pa, pb, pc, M, N, K); + printf("smatmul_os_48x16 DONE\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_48x16x25_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_48x16x25_test.c similarity index 79% rename from examples/balls/matrix/workloads/ctests/matrix_os_48x16x25_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_48x16x25_test.c index cdbcdce45..efc76454e 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_48x16x25_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_48x16x25_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 48 #define N 16 @@ -19,8 +19,8 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - printf("matrix_os_48x16x25 start\n"); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); - printf("matrix_os_48x16x25 DONE\n"); + printf("smatmul_os_48x16x25 start\n"); + matrix_hw_ws(pa, pb, pc, M, N, K); + printf("smatmul_os_48x16x25 DONE\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_ws_4x4_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_4x4_test.c similarity index 86% rename from examples/balls/matrix/workloads/ctests/matrix_ws_4x4_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_4x4_test.c index 76d8faf1e..475b71911 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_ws_4x4_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_4x4_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 4 #define N 4 @@ -24,12 +24,12 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 1); + matrix_hw_os(pa, pb, pc, M, N, K); matrix_unpack_c(pc, out, M, N); if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_ws_4x4 FAILED\n"); + printf("smatmul_os_4x4 FAILED\n"); return 1; } - printf("matrix_ws_4x4 PASSED\n"); + printf("smatmul_os_4x4 PASSED\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_b051_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_b051_test.c similarity index 82% rename from examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_b051_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_b051_test.c index c9fe48258..87dbf70b5 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_b051_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_b051_test.c @@ -1,4 +1,5 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" +#include #define M 64 #define N 16 @@ -22,9 +23,12 @@ static void matrix_hw_banks(const elem_t *packed_a, const elem_t *packed_b, bb_mem_alloc(wr, 1, 4); bb_mvin((uintptr_t)packed_a, op1, a_rows, 1); bb_mvin((uintptr_t)packed_b, op2, b_rows, 1); - bb_matrix_mnk_mode(op1, op2, wr, m, n, k, BB_MATRIX_MODE_OS); + bb_smatmul_ws(op1, op2, wr, m, n, k); bb_mvout((uintptr_t)packed_c, wr, c_blocks, 1); bb_fence(); + bb_mem_release(op1); + bb_mem_release(op2); + bb_mem_release(wr); } int main(void) { @@ -36,8 +40,8 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - printf("matrix_os_64x16x25_b051 start\n"); + printf("smatmul_os_64x16x25_b051 start\n"); matrix_hw_banks(pa, pb, pc, M, N, K, 0, 5, 1); - printf("matrix_os_64x16x25_b051 DONE\n"); + printf("smatmul_os_64x16x25_b051 DONE\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_slim_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_slim_test.c similarity index 78% rename from examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_slim_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_slim_test.c index a3368df33..8887413a4 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_slim_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_slim_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 64 #define N 16 @@ -19,8 +19,8 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - printf("matrix_os_64x16x25_slim start\n"); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); - printf("matrix_os_64x16x25_slim DONE\n"); + printf("smatmul_os_64x16x25_slim start\n"); + matrix_hw_ws(pa, pb, pc, M, N, K); + printf("smatmul_os_64x16x25_slim DONE\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_test.c similarity index 81% rename from examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_test.c index 7102d0883..d0b3bad65 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_64x16x25_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_64x16x25_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 64 #define N 16 @@ -22,13 +22,13 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - printf("matrix_os_64x16x25 start\n"); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); + printf("smatmul_os_64x16x25 start\n"); + matrix_hw_ws(pa, pb, pc, M, N, K); matrix_unpack_c(pc, out, M, N); if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_os_64x16x25 FAILED\n"); + printf("smatmul_os_64x16x25 FAILED\n"); return 1; } - printf("matrix_os_64x16x25 PASSED\n"); + printf("smatmul_os_64x16x25 PASSED\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_seq_n6_n16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_seq_n6_n16_test.c similarity index 86% rename from examples/balls/matrix/workloads/ctests/matrix_os_seq_n6_n16_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_seq_n6_n16_test.c index 82d0ae931..5e1d5b9f8 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_seq_n6_n16_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_seq_n6_n16_test.c @@ -1,4 +1,5 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" +#include #define M1 48 #define N1 6 @@ -31,9 +32,12 @@ static void matrix_hw_banks(const elem_t *packed_a, const elem_t *packed_b, bb_mem_alloc(wr, 1, 4); bb_mvin((uintptr_t)packed_a, op1, a_rows, 1); bb_mvin((uintptr_t)packed_b, op2, b_rows, 1); - bb_matrix_mnk_mode(op1, op2, wr, m, n, k, BB_MATRIX_MODE_OS); + bb_smatmul_ws(op1, op2, wr, m, n, k); bb_mvout((uintptr_t)packed_c, wr, c_blocks, 1); bb_fence(); + bb_mem_release(op1); + bb_mem_release(op2); + bb_mem_release(wr); } int main(void) { @@ -52,10 +56,10 @@ int main(void) { matrix_pack_b(b2, pb2, K2, N2); clear_u32_matrix(pc2, M2, MATRIX_ACC_LANES); - printf("matrix_os_seq_n6_n16 start\n"); + printf("smatmul_os_seq_n6_n16 start\n"); matrix_hw_banks(pa1, pb1, pc1, M1, N1, K1, 0, 5, 1); - printf("matrix_os_seq_n6_n16 after_n6\n"); + printf("smatmul_os_seq_n6_n16 after_n6\n"); matrix_hw_banks(pa2, pb2, pc2, M2, N2, K2, 0, 5, 1); - printf("matrix_os_seq_n6_n16 DONE\n"); + printf("smatmul_os_seq_n6_n16 DONE\n"); return 0; } diff --git a/examples/balls/matrix/workloads/ctests/matrix_os_tail_5x7x3_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_os_tail_5x7x3_test.c similarity index 85% rename from examples/balls/matrix/workloads/ctests/matrix_os_tail_5x7x3_test.c rename to examples/balls/smatmul/workloads/ctests/smatmul_os_tail_5x7x3_test.c index 73c36c80d..b2699e6b3 100644 --- a/examples/balls/matrix/workloads/ctests/matrix_os_tail_5x7x3_test.c +++ b/examples/balls/smatmul/workloads/ctests/smatmul_os_tail_5x7x3_test.c @@ -1,4 +1,4 @@ -#include "matrix_test_common.h" +#include "smatmul_test_common.h" #define M 5 #define N 7 @@ -24,12 +24,12 @@ int main(void) { matrix_pack_a(a, pa, M, K); matrix_pack_b(b, pb, K, N); clear_u32_matrix(pc, M, MATRIX_ACC_LANES); - matrix_hw_mnk(pa, pb, pc, M, N, K, 0); + matrix_hw_os(pa, pb, pc, M, N, K); matrix_unpack_c(pc, out, M, N); if (!compare_u32_matrices(out, exp_, M, N)) { - printf("matrix_os_tail_5x7x3 FAILED\n"); + printf("smatmul_os_tail_5x7x3 FAILED\n"); return 1; } - printf("matrix_os_tail_5x7x3 PASSED\n"); + printf("smatmul_os_tail_5x7x3 PASSED\n"); return 0; } diff --git a/examples/balls/smatmul/workloads/ctests/smatmul_test_common.h b/examples/balls/smatmul/workloads/ctests/smatmul_test_common.h new file mode 100644 index 000000000..2f63fb7e5 --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/smatmul_test_common.h @@ -0,0 +1,114 @@ +#ifndef SMATMUL_TEST_COMMON_H +#define SMATMUL_TEST_COMMON_H + +#include "buckyball.h" +#include +#include +#include +#include +#include + +#define MATRIX_TILE 16 +#define MATRIX_OUTPUT_GROUPS 2 +#define MATRIX_OUTPUT_ROUNDS (4 / MATRIX_OUTPUT_GROUPS) + +static inline void matrix_require_shape(int rows, int k) { + if (rows <= 0 || k <= 0 || rows % MATRIX_TILE || k % MATRIX_TILE) { + printf("smatmul requires non-zero 16-aligned rows and k\n"); + exit(1); + } +} + +static inline int matrix_a_lines(int rows, int k) { + return rows * (k / MATRIX_TILE); +} + +static inline void matrix_pack_a(const elem_t *src, elem_t *dst, int rows, + int k) { + int reduction_tiles = k / MATRIX_TILE; + for (int row = 0; row < rows; ++row) { + for (int column = 0; column < k; ++column) { + int line = (row / MATRIX_TILE * reduction_tiles + column / MATRIX_TILE) * + MATRIX_TILE + + row % MATRIX_TILE; + dst[line * MATRIX_TILE + column % MATRIX_TILE] = src[row * k + column]; + } + } +} + +static inline void matrix_pack_b(const elem_t *src, elem_t *dst, int k) { + for (int row = 0; row < k; ++row) + for (int column = 0; column < MATRIX_TILE; ++column) + dst[row * MATRIX_TILE + column] = src[row * MATRIX_TILE + column]; +} + +static inline void matrix_pack_b_ws(const elem_t *src, elem_t *dst, int k, + int n) { + for (int panel = 0; panel < n / MATRIX_TILE; ++panel) + for (int row = 0; row < k; ++row) + for (int column = 0; column < MATRIX_TILE; ++column) + dst[(panel * k + row) * MATRIX_TILE + column] = + src[row * n + panel * MATRIX_TILE + column]; +} + +static inline void matrix_unpack_c(const result_t *src, result_t *dst, + int rows) { + for (int row = 0; row < rows; ++row) { + for (int column = 0; column < MATRIX_TILE; ++column) { + int line = 2 * row + column / 8; + dst[row * MATRIX_TILE + column] = src[line * 8 + column % 8]; + } + } +} + +static inline void matrix_unpack_c_ws(const result_t *src, result_t *dst, + int rows, int columns) { + for (int panel = 0; panel < columns / MATRIX_TILE; ++panel) { + for (int row = 0; row < rows; ++row) { + for (int column = 0; column < MATRIX_TILE; ++column) { + int line = panel * MATRIX_TILE * MATRIX_OUTPUT_ROUNDS + + row * MATRIX_OUTPUT_ROUNDS + column / 8; + dst[row * columns + panel * MATRIX_TILE + column] = + src[line * 8 + column % 8]; + } + } + } +} + +static inline void matrix_hw_ws(const elem_t *a, const elem_t *b, result_t *c, + int rows, int columns, int k) { + if (rows != MATRIX_TILE || columns <= 0 || columns % MATRIX_TILE || + columns > 512 || k != MATRIX_TILE) + exit(1); + const uint32_t a_bank = 0, b_bank = 1, c_bank = 2; + bb_mem_alloc(a_bank, 1, 1); + bb_mem_alloc(b_bank, 1, 1); + bb_mem_alloc(c_bank, 1, MATRIX_OUTPUT_GROUPS); + bb_mvin((uintptr_t)a, a_bank, rows, 1); + bb_mvin((uintptr_t)b, b_bank, columns, 1); + bb_smatmul_ws(a_bank, b_bank, c_bank, rows, columns, k); + bb_mvout((uintptr_t)c, c_bank, columns * MATRIX_OUTPUT_ROUNDS, 1); + bb_fence(); + bb_mem_release(a_bank); + bb_mem_release(b_bank); + bb_mem_release(c_bank); +} + +static inline void matrix_hw(const elem_t *a, const elem_t *b, result_t *c, + int rows, int k) { + const uint32_t a_bank = 0, b_bank = 1, c_bank = 2; + matrix_require_shape(rows, k); + bb_mem_alloc(a_bank, 1, 1); + bb_mem_alloc(b_bank, 1, 1); + bb_mem_alloc(c_bank, 1, 2); + bb_mvin((uintptr_t)a, a_bank, matrix_a_lines(rows, k), 1); + bb_mvin((uintptr_t)b, b_bank, k, 1); + bb_smatmul_os(a_bank, b_bank, c_bank, rows, MATRIX_TILE, k); + bb_mvout((uintptr_t)c, c_bank, 2 * rows, 1); + bb_fence(); + bb_mem_release(a_bank); + bb_mem_release(b_bank); + bb_mem_release(c_bank); +} + +#endif diff --git a/examples/balls/smatmul/workloads/ctests/smatmul_ws_16x16_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_ws_16x16_test.c new file mode 100644 index 000000000..279d98003 --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/smatmul_ws_16x16_test.c @@ -0,0 +1,29 @@ +#include "smatmul_test_common.h" + +#define ROWS 16 +#define COLUMNS 16 +#define K 16 + +static elem_t a[ROWS * K] __attribute__((aligned(64))); +static elem_t b[K * COLUMNS] __attribute__((aligned(64))); +static elem_t packed_a[ROWS * K] __attribute__((aligned(64))); +static elem_t packed_b[COLUMNS * MATRIX_TILE] __attribute__((aligned(64))); +static result_t packed_c[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t expected[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t actual[ROWS * COLUMNS] __attribute__((aligned(64))); + +int main(void) { + for (int i = 0; i < ROWS * K; ++i) + a[i] = (i * 7 % 23) - 11; + for (int i = 0; i < K * COLUMNS; ++i) + b[i] = (i * 5 % 19) - 9; + cpu_matmul(a, b, expected, ROWS, COLUMNS, K); + matrix_pack_a(a, packed_a, ROWS, K); + matrix_pack_b_ws(b, packed_b, K, COLUMNS); + matrix_hw_ws(packed_a, packed_b, packed_c, ROWS, COLUMNS, K); + matrix_unpack_c_ws(packed_c, actual, ROWS, COLUMNS); + if (!compare_u32_matrices(actual, expected, ROWS, COLUMNS)) + return 1; + printf("smatmul_ws_16x16 PASSED\n"); + return 0; +} diff --git a/examples/balls/smatmul/workloads/ctests/smatmul_ws_16x512_test.c b/examples/balls/smatmul/workloads/ctests/smatmul_ws_16x512_test.c new file mode 100644 index 000000000..4d391bc24 --- /dev/null +++ b/examples/balls/smatmul/workloads/ctests/smatmul_ws_16x512_test.c @@ -0,0 +1,29 @@ +#include "smatmul_test_common.h" + +#define ROWS 16 +#define COLUMNS 512 +#define K 16 + +static elem_t a[ROWS * K] __attribute__((aligned(64))); +static elem_t b[K * COLUMNS] __attribute__((aligned(64))); +static elem_t packed_a[ROWS * K] __attribute__((aligned(64))); +static elem_t packed_b[COLUMNS * MATRIX_TILE] __attribute__((aligned(64))); +static result_t packed_c[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t expected[ROWS * COLUMNS] __attribute__((aligned(64))); +static result_t actual[ROWS * COLUMNS] __attribute__((aligned(64))); + +int main(void) { + for (int i = 0; i < ROWS * K; ++i) + a[i] = (i * 7 % 23) - 11; + for (int i = 0; i < K * COLUMNS; ++i) + b[i] = (i * 5 % 19) - 9; + cpu_matmul(a, b, expected, ROWS, COLUMNS, K); + matrix_pack_a(a, packed_a, ROWS, K); + matrix_pack_b_ws(b, packed_b, K, COLUMNS); + matrix_hw_ws(packed_a, packed_b, packed_c, ROWS, COLUMNS, K); + matrix_unpack_c_ws(packed_c, actual, ROWS, COLUMNS); + if (!compare_u32_matrices(actual, expected, ROWS, COLUMNS)) + return 1; + printf("smatmul_ws_16x512 PASSED\n"); + return 0; +} diff --git a/examples/balls/smatmul/workloads/isa/smatmul.h b/examples/balls/smatmul/workloads/isa/smatmul.h new file mode 100644 index 000000000..4964ea243 --- /dev/null +++ b/examples/balls/smatmul/workloads/isa/smatmul.h @@ -0,0 +1,24 @@ +#ifndef _BB_SMATMUL_H_ +#define _BB_SMATMUL_H_ + +#include +#include + +#define BB_SMATMUL_CFG(rows, cols, k) \ + (FIELD((rows), 0, 11) | FIELD((cols), 12, 23) | FIELD((k), 24, 35)) + +#define BB_SMATMUL_RS1(op1_bank_id, op2_bank_id, wr_bank_id) \ + (BB_BANK0(op1_bank_id) | BB_BANK1(op2_bank_id) | BB_BANK2(wr_bank_id) | \ + BB_ITER(0)) + +#define bb_smatmul_os(op1_bank_id, op2_bank_id, wr_bank_id, rows, cols, k) \ + BUCKYBALL_INSTRUCTION_R_R( \ + BB_SMATMUL_RS1(op1_bank_id, op2_bank_id, wr_bank_id), \ + BB_SMATMUL_CFG(rows, cols, k), BB_FUNC7(SMATMUL_OS)) + +#define bb_smatmul_ws(op1_bank_id, op2_bank_id, wr_bank_id, rows, cols, k) \ + BUCKYBALL_INSTRUCTION_R_R( \ + BB_SMATMUL_RS1(op1_bank_id, op2_bank_id, wr_bank_id), \ + BB_SMATMUL_CFG(rows, cols, k), BB_FUNC7(SMATMUL_WS)) + +#endif // _BB_SMATMUL_H_ diff --git a/examples/balls/matrix/workloads/mlir_tests/CMakeLists.txt b/examples/balls/smatmul/workloads/mlir_tests/CMakeLists.txt similarity index 100% rename from examples/balls/matrix/workloads/mlir_tests/CMakeLists.txt rename to examples/balls/smatmul/workloads/mlir_tests/CMakeLists.txt diff --git a/examples/balls/smatmul/workloads/mlir_tests/ball/CMakeLists.txt b/examples/balls/smatmul/workloads/mlir_tests/ball/CMakeLists.txt new file mode 100644 index 000000000..d1c1046a1 --- /dev/null +++ b/examples/balls/smatmul/workloads/mlir_tests/ball/CMakeLists.txt @@ -0,0 +1,12 @@ +set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) +set(BUCKYBALL_MLIR_TEST_PREFIX ball) + +add_buckyball_mlir_contract(smatmul_16x16_i8_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) + +add_buckyball_mlir_test(smatmul_16x16_i8 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) +add_buckyball_mlir_test(smatmul_matmul_16x16_i8 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8.mlir b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8.mlir new file mode 100644 index 000000000..5d1ec18c6 --- /dev/null +++ b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8.mlir @@ -0,0 +1,42 @@ +// Already-lowered ball Op: buckyball.smatmul (physical-bank form). + +func.func private @check_result(memref<32x8xi32>) -> () + +func.func @main() -> i8 { + %zero_i8 = arith.constant 0 : i8 + %one_i8 = arith.constant 1 : i8 + %zero_i32 = arith.constant 0 : i32 + %input_depth = arith.constant 16 : i64 + %output_depth = arith.constant 32 : i64 + %stride = arith.constant 1 : i64 + %cfg = arith.constant 268501008 : i64 + + %a = memref.alloc() alignment = 64 : memref<16x16xi8> + %b = memref.alloc() alignment = 64 : memref<16x16xi8> + %c = memref.alloc() alignment = 64 : memref<32x8xi32> + + linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) + linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x16xi8>) + linalg.fill ins(%zero_i32 : i32) outs(%c : memref<32x8xi32>) + + %ba = buckyball.bank_alloc + %bb = buckyball.bank_alloc + %bc = buckyball.bank_alloc {col = 2 : i64} + %la = buckyball.bank_mvin %a %ba %input_depth %stride + : memref<16x16xi8> i64 i64 i64 + %lb = buckyball.bank_mvin %b %bb %input_depth %stride + : memref<16x16xi8> i64 i64 i64 + buckyball.smatmul %la, %lb, %bc, %cfg : i64 + %stored = buckyball.bank_mvout %c %bc %output_depth %stride + : memref<32x8xi32> i64 i64 i64 + buckyball.fence + buckyball.bank_release %la : i64 + buckyball.bank_release %lb : i64 + buckyball.bank_release %stored : i64 + + func.call @check_result(%c) : (memref<32x8xi32>) -> () + memref.dealloc %a : memref<16x16xi8> + memref.dealloc %b : memref<16x16xi8> + memref.dealloc %c : memref<32x8xi32> + return %zero_i8 : i8 +} diff --git a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8_contract.mlir b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8_contract.mlir similarity index 67% rename from examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8_contract.mlir rename to examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8_contract.mlir index 973b1f19c..10b545ffd 100644 --- a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8_contract.mlir +++ b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8_contract.mlir @@ -1,14 +1,14 @@ // CHECK: buckyball.intr.custom -// CHECK-SAME: funct7 = 65 -// CHECK-NOT: buckyball.matrix +// CHECK-SAME: funct7 = +// CHECK-NOT: buckyball.smatmul func.func @main() -> i8 { %z = arith.constant 0 : i8 %cfg = arith.constant 268501008 : i64 %a = buckyball.bank_alloc %b = buckyball.bank_alloc - %c = buckyball.bank_alloc {col = 4 : i64} - buckyball.matrix %a, %b, %c, %cfg : i64 + %c = buckyball.bank_alloc {col = 2 : i64} + buckyball.smatmul %a, %b, %c, %cfg : i64 buckyball.bank_release %a : i64 buckyball.bank_release %b : i64 buckyball.bank_release %c : i64 diff --git a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8_main.cpp b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8_main.cpp similarity index 86% rename from examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8_main.cpp rename to examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8_main.cpp index c52d60930..31151b1b0 100644 --- a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_16x16_i8_main.cpp +++ b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_16x16_i8_main.cpp @@ -20,15 +20,15 @@ extern "C" int64_t size0, int64_t size1, int64_t stride0, int64_t stride1) { (void)allocated; - if (size0 != 16 || size1 != 16 || stride0 != 16 || stride1 != 1) { + if (size0 != 32 || size1 != 8 || stride0 != 8 || stride1 != 1) { printf("FAILED: ball_matrix shape %dx%d stride %dx%d\n", (int)size0, (int)size1, (int)stride0, (int)stride1); fail(); } int32_t *out = aligned + offset; - for (int i = 0; i < 16; ++i) { - for (int j = 0; j < 16; ++j) { + for (int i = 0; i < 32; ++i) { + for (int j = 0; j < 8; ++j) { int32_t got = out[i * stride0 + j * stride1]; if (got != 16) { printf("FAILED: ball_matrix out[%d][%d] exp=16 got=%d\n", i, j, diff --git a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_matmul_16x16_i8.mlir b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_matmul_16x16_i8.mlir similarity index 73% rename from examples/balls/matrix/workloads/mlir_tests/ball/matrix_matmul_16x16_i8.mlir rename to examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_matmul_16x16_i8.mlir index 576e5d785..f3eb03d73 100644 --- a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_matmul_16x16_i8.mlir +++ b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_matmul_16x16_i8.mlir @@ -4,14 +4,14 @@ func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 %one_i8 = arith.constant 1 : i8 %zero_i32 = arith.constant 0 : i32 - %a = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %b = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %c = memref.alloc() {alignment = 64 : i64} : memref<16x16xi32> + %a = memref.alloc() alignment = 64 : memref<16x16xi8> + %b = memref.alloc() alignment = 64 : memref<16x16xi8> + %c = memref.alloc() alignment = 64 : memref<16x16xi32> linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x16xi8>) linalg.fill ins(%zero_i32 : i32) outs(%c : memref<16x16xi32>) - buckyball.matrix_matmul %a %b %c + buckyball.smatmul_matmul %a %b %c : memref<16x16xi8> memref<16x16xi8> memref<16x16xi32> func.call @check_result(%c) : (memref<16x16xi32>) -> () diff --git a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_matmul_16x16_i8_main.cpp b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_matmul_16x16_i8_main.cpp similarity index 92% rename from examples/balls/matrix/workloads/mlir_tests/ball/matrix_matmul_16x16_i8_main.cpp rename to examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_matmul_16x16_i8_main.cpp index 0b32b6bb3..d385cfbef 100644 --- a/examples/balls/matrix/workloads/mlir_tests/ball/matrix_matmul_16x16_i8_main.cpp +++ b/examples/balls/smatmul/workloads/mlir_tests/ball/smatmul_matmul_16x16_i8_main.cpp @@ -31,5 +31,5 @@ extern "C" } } } - printf("PASSED: matrix ball_matrix_matmul 16x16 i8\n"); + printf("PASSED: matrix ball_smatmul_matmul 16x16 i8\n"); } diff --git a/examples/balls/smatmul/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/smatmul/workloads/mlir_tests/bank/CMakeLists.txt new file mode 100644 index 000000000..402c568a5 --- /dev/null +++ b/examples/balls/smatmul/workloads/mlir_tests/bank/CMakeLists.txt @@ -0,0 +1,8 @@ +set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) +set(BUCKYBALL_MLIR_TEST_PREFIX bank) + +add_buckyball_mlir_contract(smatmul_ws_16x512_i8_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) + +add_buckyball_mlir_test(smatmul_ws_16x512_i8 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8.mlir b/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8.mlir new file mode 100644 index 000000000..da867f341 --- /dev/null +++ b/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8.mlir @@ -0,0 +1,45 @@ +// Bank-level SMatMul WS capacity test. +// Two C groups each occupy all 1024 SRAM rows. + +func.func private @check_result(memref<1024x8xi32>) -> () + +func.func @main() -> i8 { + %zero_i8 = arith.constant 0 : i8 + %one_i8 = arith.constant 1 : i8 + %zero_i32 = arith.constant 0 : i32 + %a_depth = arith.constant 16 : i64 + %b_depth = arith.constant 512 : i64 + %c_depth = arith.constant 1024 : i64 + %stride = arith.constant 1 : i64 + %cfg = arith.constant 270532624 : i64 + + %a = memref.alloc() alignment = 64 : memref<16x16xi8> + %b = memref.alloc() alignment = 64 : memref<16x512xi8> + %c = memref.alloc() alignment = 64 : memref<1024x8xi32> + + linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) + linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x512xi8>) + linalg.fill ins(%zero_i32 : i32) outs(%c : memref<1024x8xi32>) + + %ba = buckyball.bank_alloc + %bb = buckyball.bank_alloc + %bc = buckyball.bank_alloc {col = 2 : i64} + %la = buckyball.bank_mvin %a %ba %a_depth %stride + : memref<16x16xi8> i64 i64 i64 + %lb = buckyball.bank_mvin %b %bb %b_depth %stride + : memref<16x512xi8> i64 i64 i64 + %r = buckyball.bank_smatmul %la %lb %bc %cfg {ws = true} + : i64 i64 i64 i64 + %stored = buckyball.bank_mvout %c %r %c_depth %stride + : memref<1024x8xi32> i64 i64 i64 + buckyball.fence + buckyball.bank_release %la : i64 + buckyball.bank_release %lb : i64 + buckyball.bank_release %stored : i64 + + func.call @check_result(%c) : (memref<1024x8xi32>) -> () + memref.dealloc %a : memref<16x16xi8> + memref.dealloc %b : memref<16x512xi8> + memref.dealloc %c : memref<1024x8xi32> + return %zero_i8 : i8 +} diff --git a/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8_contract.mlir b/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8_contract.mlir similarity index 55% rename from examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8_contract.mlir rename to examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8_contract.mlir index a09568679..de60eaa59 100644 --- a/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8_contract.mlir +++ b/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8_contract.mlir @@ -1,13 +1,13 @@ -// CHECK: buckyball.matrix -// CHECK-NOT: buckyball.bank_matrix +// CHECK: buckyball.smatmul +// CHECK-NOT: buckyball.bank_smatmul func.func @main() -> i8 { %z = arith.constant 0 : i8 - %cfg = arith.constant 268501008 : i64 + %cfg = arith.constant 270532624 : i64 %a = buckyball.bank_alloc %b = buckyball.bank_alloc - %c = buckyball.bank_alloc {col = 4 : i64} - %r = buckyball.bank_matrix %a %b %c %cfg + %c = buckyball.bank_alloc {col = 2 : i64} + %r = buckyball.bank_smatmul %a %b %c %cfg {ws = true} : i64 i64 i64 i64 buckyball.bank_release %a : i64 buckyball.bank_release %b : i64 diff --git a/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8_main.cpp b/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8_main.cpp similarity index 68% rename from examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8_main.cpp rename to examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8_main.cpp index a53277436..da4cc4acf 100644 --- a/examples/balls/matrix/workloads/mlir_tests/bank/matrix_16x16_i8_main.cpp +++ b/examples/balls/smatmul/workloads/mlir_tests/bank/smatmul_ws_16x512_i8_main.cpp @@ -20,22 +20,22 @@ extern "C" int64_t size0, int64_t size1, int64_t stride0, int64_t stride1) { (void)allocated; - if (size0 != 16 || size1 != 16 || stride0 != 16 || stride1 != 1) { - printf("FAILED: bank_matrix shape %dx%d stride %dx%d\n", (int)size0, + if (size0 != 1024 || size1 != 8 || stride0 != 8 || stride1 != 1) { + printf("FAILED: bank_smatmul shape %dx%d stride %dx%d\n", (int)size0, (int)size1, (int)stride0, (int)stride1); fail(); } int32_t *out = aligned + offset; - for (int i = 0; i < 16; ++i) { - for (int j = 0; j < 16; ++j) { + for (int i = 0; i < 1024; ++i) { + for (int j = 0; j < 8; ++j) { int32_t got = out[i * stride0 + j * stride1]; if (got != 16) { - printf("FAILED: bank_matrix out[%d][%d] exp=16 got=%d\n", i, j, + printf("FAILED: bank_smatmul out[%d][%d] exp=16 got=%d\n", i, j, (int)got); fail(); } } } - printf("PASSED: matrix bank_matrix 16x16 ones\n"); + printf("PASSED: bank_smatmul WS 16x512\n"); } diff --git a/examples/balls/trace/arch/src/main/scala/README.md b/examples/balls/trace/arch/src/main/scala/README.md index 713df5fe0..c7d8393c7 100644 --- a/examples/balls/trace/arch/src/main/scala/README.md +++ b/examples/balls/trace/arch/src/main/scala/README.md @@ -13,9 +13,10 @@ All DPI-C hooks live inside TraceBall only and do not affect other modules. ## Instruction encoding -TraceBall uses **two funct7 encodings**. +TraceBall uses two Core-owned `ballISA` mnemonics. The selected Core assigns +their `funct7` values at compile time. -### Instruction 1: `bdb_counter` (funct7 = 48, 0x30) +### Instruction 1: `bdb_counter` Cycle counter control. **Does not touch SRAM, needs no bank ports, completes in 1 cycle.** @@ -44,7 +45,7 @@ DPI-C trace format (written to bdb.log): [CTRACE] CTR_READ ctr=0 current=200 cycle=10242 ``` -### Instruction 2: `bdb_backdoor` (funct7 = 49, 0x31) +### Instruction 2: `bdb_backdoor` SRAM backdoor read/write; **all parameters (bank_id, row, data) come from DPI-C**. **Requires bank ports (inBW=1, outBW=1).** @@ -79,7 +80,7 @@ DPI-C trace format: ```c // One matmul region bdb_counter_start(0, 0xA001); // counter 0, tag=matmul -bb_mul_warp16(A, B, C, 16); +bb_vecmat16(A, B, C, 16); bb_fence(); bdb_counter_stop(0); // prints elapsed @@ -91,7 +92,7 @@ bdb_counter_start(0, 0xB001); // outer: whole conv bdb_counter_stop(1); bdb_counter_start(2, 0xB003); // inner: matmul - bb_mul_warp16(...); + bb_vecmat16(...); bb_fence(); bdb_counter_stop(2); bdb_counter_stop(0); // outer end diff --git a/examples/balls/trace/arch/src/main/scala/Trace.scala b/examples/balls/trace/arch/src/main/scala/Trace.scala index 2a4904aeb..78f3acda6 100644 --- a/examples/balls/trace/arch/src/main/scala/Trace.scala +++ b/examples/balls/trace/arch/src/main/scala/Trace.scala @@ -18,7 +18,9 @@ class Trace(val b: GlobalConfig) extends Module { val ballMapping = b.ballDomain.ballIdMappings .find(_.ballName == "TraceBall") - .getOrElse(throw new IllegalArgumentException("TraceBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("TraceBall not found in config") + ) val inBW = ballMapping.inBW val outBW = ballMapping.outBW @@ -81,7 +83,9 @@ class Trace(val b: GlobalConfig) extends Module { privBank.io.sramWrite.req.valid := false.B privBank.io.sramWrite.req.bits.addr := 0.U privBank.io.sramWrite.req.bits.data := 0.U - privBank.io.sramWrite.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) + privBank.io.sramWrite.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(true.B) + ) privBank.io.sramWrite.resp.ready := true.B // ============================================================ @@ -118,7 +122,9 @@ class Trace(val b: GlobalConfig) extends Module { io.bankWrite(i).io.req.valid := false.B io.bankWrite(i).io.req.bits.addr := 0.U io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W))) + io.bankWrite(i).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W)) + ) io.bankWrite(i).io.resp.ready := false.B } diff --git a/examples/balls/trace/arch/src/main/scala/TraceBall.scala b/examples/balls/trace/arch/src/main/scala/TraceBall.scala index a6f3831b6..112fa2938 100644 --- a/examples/balls/trace/arch/src/main/scala/TraceBall.scala +++ b/examples/balls/trace/arch/src/main/scala/TraceBall.scala @@ -10,15 +10,16 @@ import framework.top.GlobalConfig /** * TraceBall — Debug trace Ball providing cycle counters. * - * Uses funct7 encoding: - * - bdb_counter (funct7=4): cycle counter management + * The Core ballISA assigns the funct7 encoding for bdb_counter. */ @instantiable class TraceBall(val b: GlobalConfig) extends Module with HasBlink { val ballCommonConfig = b.ballDomain.ballIdMappings .find(_.ballName == "TraceBall") - .getOrElse(throw new IllegalArgumentException("TraceBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("TraceBall not found in config") + ) val inBW = ballCommonConfig.inBW val outBW = ballCommonConfig.outBW diff --git a/examples/balls/trace/compiler/src/Dialect/Buckyball/TraceBall.td b/examples/balls/trace/compiler/src/Dialect/Buckyball/TraceBall.td new file mode 100644 index 000000000..398293258 --- /dev/null +++ b/examples/balls/trace/compiler/src/Dialect/Buckyball/TraceBall.td @@ -0,0 +1,18 @@ +//===-- TraceBall.td - Trace Ball operations ----------------------------===// + +#ifndef TRACE_BALL_OPS +#define TRACE_BALL_OPS + +include "Buckyball.td" + +def BdbCounterOp : Buckyball_Op<"bdb_counter"> { + let summary = "Start, stop, or read a Buckyball debug counter"; + let description = [{ + Encodes BDB_COUNTER rs2 as subcommand[3:0], counterId[7:4], and + payload[63:8]. rs1 is always zero. + }]; + let arguments = (ins I64:$subcommand, I64:$counterId, I64:$payload); + let assemblyFormat = "$subcommand `,` $counterId `,` $payload attr-dict `:` type($subcommand)"; +} + +#endif // TRACE_BALL_OPS diff --git a/examples/balls/trace/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/trace/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..270d80eb1 --- /dev/null +++ b/examples/balls/trace/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,55 @@ +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/Pattern.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" + +using namespace mlir; +using namespace buddy::buckyball; +using namespace buddy::buckyball::legalize; + +namespace { +class BdbCounterLowering : public ConvertOpToLLVMPattern { +public: + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; + + LogicalResult + matchAndRewrite(BdbCounterOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("TraceBall"); + Location loc = op.getLoc(); + Value counter = rewriter.create(loc, adaptor.getCounterId(), + cstI64(rewriter, loc, 4)); + Value payload = rewriter.create(loc, adaptor.getPayload(), + cstI64(rewriter, loc, 8)); + Value rs2 = + rewriter.create(loc, adaptor.getSubcommand(), counter); + rs2 = rewriter.create(loc, rs2, payload); + rewriter.replaceOpWithNewOp( + op, cstI64(rewriter, loc, 0), rs2, + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("BDB_COUNTER"))); + return success(); + } +}; +} // namespace + +namespace mlir::buddy::buckyball { +void populateTraceBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB) { + (void)stable; + (void)bankDepth; + (void)rushB; + patterns.add(converter); +} + +void configureTraceBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { + (void)stable; + target.addIllegalOp(); +} +} // namespace mlir::buddy::buckyball diff --git a/examples/balls/trace/configs/default.toml b/examples/balls/trace/configs/default.toml new file mode 100644 index 000000000..6f2c952cd --- /dev/null +++ b/examples/balls/trace/configs/default.toml @@ -0,0 +1,3 @@ +# Trace Ball Configuration + +[ball] diff --git a/examples/balls/trace/emu/src/04_bdb_counter.rs b/examples/balls/trace/emu/src/04_bdb_counter.rs index 43e5e6165..4fd1f9437 100644 --- a/examples/balls/trace/emu/src/04_bdb_counter.rs +++ b/examples/balls/trace/emu/src/04_bdb_counter.rs @@ -1,12 +1,10 @@ //===- 04_bdb_counter.rs - BDB_COUNTER instruction -------------------------===// -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; pub struct BdbCounter; -impl Instruction for BdbCounter { - const FUNCT: u32 = 4; - +impl BallInstruction for BdbCounter { fn exec(_xs1: u64, _xs2: u64, _ctx: &mut ExecContext) -> u64 { 0 } diff --git a/examples/balls/trace/emu/src/lib.rs b/examples/balls/trace/emu/src/lib.rs index 299714514..855e863ff 100644 --- a/examples/balls/trace/emu/src/lib.rs +++ b/examples/balls/trace/emu/src/lib.rs @@ -1,6 +1,6 @@ pub(crate) use crate::inst::instruction; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::{BallInstruction, ExecContext}; #[path = "04_bdb_counter.rs"] mod f04_bdb_counter; @@ -17,11 +17,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("BDB_COUNTER") => Some(f04_bdb_counter::BdbCounter::exec(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -29,10 +27,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("BDB_COUNTER") => Some(f04_bdb_counter::BdbCounter::latency(xs1, xs2)), + Some(_) | None => None, } } diff --git a/examples/balls/trace/workloads/ctests/bdb_counter_test.c b/examples/balls/trace/workloads/ctests/bdb_counter_test.c index 5acc8693a..a8dd3b452 100644 --- a/examples/balls/trace/workloads/ctests/bdb_counter_test.c +++ b/examples/balls/trace/workloads/ctests/bdb_counter_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #include #include @@ -21,7 +22,8 @@ int main() { bdb_counter_start(0, 0xA001); // Do some work to burn cycles volatile int x = 0; - for (int i = 0; i < 10; i++) x += i; + for (int i = 0; i < 10; i++) + x += i; bdb_counter_stop(0); printf("Test 1 PASSED\n"); @@ -29,19 +31,21 @@ int main() { printf("Test 2: start/read/stop\n"); bdb_counter_start(1, 0xA002); volatile int y = 0; - for (int i = 0; i < 5; i++) y += i; + for (int i = 0; i < 5; i++) + y += i; bdb_counter_read(1); bdb_counter_stop(1); printf("Test 2 PASSED\n"); // Test 3: nested counters (two levels) printf("Test 3: nested counters\n"); - bdb_counter_start(0, 0xB001); // outer - bdb_counter_start(1, 0xB002); // inner - volatile int z = 0; - for (int i = 0; i < 5; i++) z += i; - bdb_counter_stop(1); // inner done - bdb_counter_stop(0); // outer done + bdb_counter_start(0, 0xB001); // outer + bdb_counter_start(1, 0xB002); // inner + volatile int z = 0; + for (int i = 0; i < 5; i++) + z += i; + bdb_counter_stop(1); // inner done + bdb_counter_stop(0); // outer done printf("Test 3 PASSED\n"); // Test 4: multiple independent counters diff --git a/bb-tests/workloads/lib/bbhw/isa/04_bdb_counter.c b/examples/balls/trace/workloads/isa/bdb_counter.h similarity index 83% rename from bb-tests/workloads/lib/bbhw/isa/04_bdb_counter.c rename to examples/balls/trace/workloads/isa/bdb_counter.h index 449a6a8e6..0f37555a7 100644 --- a/bb-tests/workloads/lib/bbhw/isa/04_bdb_counter.c +++ b/examples/balls/trace/workloads/isa/bdb_counter.h @@ -1,9 +1,8 @@ #ifndef _BDB_COUNTER_H_ #define _BDB_COUNTER_H_ -#include "isa.h" - -#define BDB_COUNTER_FUNC7 4 +#include +#include // subcmd values #define BDB_CTR_START 0 @@ -18,16 +17,16 @@ // Start counter ctr_id with user tag #define bdb_counter_start(ctr_id, tag) \ BUCKYBALL_INSTRUCTION_R_R(0, BDB_CTR_RS2(BDB_CTR_START, ctr_id, tag), \ - BDB_COUNTER_FUNC7) + BB_FUNC7(BDB_COUNTER)) // Stop counter ctr_id, output elapsed to trace #define bdb_counter_stop(ctr_id) \ BUCKYBALL_INSTRUCTION_R_R(0, BDB_CTR_RS2(BDB_CTR_STOP, ctr_id, 0), \ - BDB_COUNTER_FUNC7) + BB_FUNC7(BDB_COUNTER)) // Read counter ctr_id current value (non-destructive), output to trace #define bdb_counter_read(ctr_id) \ BUCKYBALL_INSTRUCTION_R_R(0, BDB_CTR_RS2(BDB_CTR_READ, ctr_id, 0), \ - BDB_COUNTER_FUNC7) + BB_FUNC7(BDB_COUNTER)) #endif // _BDB_COUNTER_H_ diff --git a/examples/balls/transpose/arch/src/main/scala/Transpose.scala b/examples/balls/transpose/arch/src/main/scala/Transpose.scala index ba4ffe781..f9e96b0e7 100644 --- a/examples/balls/transpose/arch/src/main/scala/Transpose.scala +++ b/examples/balls/transpose/arch/src/main/scala/Transpose.scala @@ -17,7 +17,9 @@ class Transpose(val b: GlobalConfig) extends Module { val ballMapping = b.ballDomain.ballIdMappings .find(_.ballName == "TransposeBall") - .getOrElse(throw new IllegalArgumentException("TransposeBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("TransposeBall not found in config") + ) val inBW = ballMapping.inBW val outBW = ballMapping.outBW @@ -83,7 +85,9 @@ class Transpose(val b: GlobalConfig) extends Module { io.bankWrite(i).io.req.valid := false.B io.bankWrite(i).io.req.bits.addr := 0.U io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W))) + io.bankWrite(i).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(0.U(1.W)) + ) io.bankWrite(i).io.resp.ready := (state =/= idle) } @@ -93,9 +97,9 @@ class Transpose(val b: GlobalConfig) extends Module { io.cmdResp.bits.is_sub := is_sub_reg io.cmdResp.bits.sub_rob_id := sub_rob_id_reg - val elemBits = elem_reg + val elemBits = elem_reg val elemBytes = elemBits >> 3 - val epg = (rowBytes.U / elemBytes) + val epg = rowBytes.U / elemBytes val wElems = ncol_reg * epg val total = iter_reg * wElems @@ -135,12 +139,20 @@ class Transpose(val b: GlobalConfig) extends Module { cached := false.B assert(cmd.iter > 0.U, "Transpose iter must be > 0") assert(cmd.op1_bank =/= cmd.wr_bank, "Transpose op1 and wr must differ") - assert(cmd.op1_col === cmd.wr_col && cmd.op1_col =/= 0.U, "Transpose cols mismatch") + assert( + cmd.op1_col === cmd.wr_col && cmd.op1_col =/= 0.U, + "Transpose cols mismatch" + ) assert(cmd.rs2(63, 8) === 0.U, "Transpose rs2[63:8] must be 0") - assert(cmd.rs2(7, 0) === 8.U || cmd.rs2(7, 0) === 32.U, "Transpose elem_bits") - assert(bankWidth.U % cmd.rs2(7, 0) === 0.U, - "Transpose bankWidth not divisible by elem_bits") - state := sRead + assert( + cmd.rs2(7, 0) === 8.U || cmd.rs2(7, 0) === 32.U, + "Transpose elem_bits" + ) + assert( + bankWidth.U % cmd.rs2(7, 0) === 0.U, + "Transpose bankWidth not divisible by elem_bits" + ) + state := sRead } } @@ -171,15 +183,15 @@ class Transpose(val b: GlobalConfig) extends Module { wrGroup := dstGroup wrAddr := dstVirtRow } - wrData := base | (elem << wsh) + wrData := base | (elem << wsh) beatFill := beatFill + 1.U dstIdx := dstIdx + 1.U val beatDone = (beatFill + 1.U === epg) || (dstIdx + 1.U === total) when(beatDone) { wrMask.foreach(_ := 1.U) - wrValid := true.B - state := sWrite + wrValid := true.B + state := sWrite } } } diff --git a/examples/balls/transpose/arch/src/main/scala/TransposeBall.scala b/examples/balls/transpose/arch/src/main/scala/TransposeBall.scala index 955a0e661..41a18a43e 100644 --- a/examples/balls/transpose/arch/src/main/scala/TransposeBall.scala +++ b/examples/balls/transpose/arch/src/main/scala/TransposeBall.scala @@ -11,10 +11,14 @@ import framework.top.GlobalConfig @instantiable class TransposeBall(val b: GlobalConfig) extends Module with HasBlink { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "TransposeBall") - .getOrElse(throw new IllegalArgumentException("TransposeBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "TransposeBall") + .getOrElse( + throw new IllegalArgumentException("TransposeBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW @public val io = IO(new BlinkIO(b, inBW, outBW)) diff --git a/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp index 655cfa48d..20819a12c 100644 --- a/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp +++ b/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -10,8 +10,8 @@ using namespace mlir; using namespace ::buddy::buckyball; namespace mlir::buddy { -void populateTransposeAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateTransposeBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, PhysicalBankState &state); } // namespace mlir::buddy namespace { @@ -31,7 +31,7 @@ class BankTransposePattern : public OpRewritePattern { } // namespace -void mlir::buddy::populateTransposeAssignPhysicalBankPatterns( +void mlir::buddy::populateTransposeBallAssignPhysicalBankPatterns( RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { (void)state; patterns.add(patterns.getContext()); diff --git a/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp b/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp new file mode 100644 index 000000000..880276a9a --- /dev/null +++ b/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp @@ -0,0 +1,52 @@ +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/Utils/StructuredOpsUtils.h" +#include "mlir/IR/AffineMap.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace mlir::buddy { +void populateTransposeBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns); +} // namespace mlir::buddy + +namespace { +class MemTransposeToLinalgPattern : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(MemTransposeOp op, + PatternRewriter &rewriter) const override { + auto input = dyn_cast(op.getInput().getType()); + auto output = dyn_cast(op.getOutput().getType()); + if (!input || !output || !input.hasStaticShape() || + !output.hasStaticShape() || input.getRank() != 2 || + output.getRank() != 2 || output.getShape()[0] != input.getShape()[1] || + output.getShape()[1] != input.getShape()[0]) + return op.emitError("requires matching static rank-2 transpose memrefs"); + MLIRContext *context = rewriter.getContext(); + AffineMap identity = AffineMap::getMultiDimIdentityMap(2, context); + AffineMap transpose = AffineMap::get( + 2, 0, {rewriter.getAffineDimExpr(1), rewriter.getAffineDimExpr(0)}, + context); + SmallVector iterators(2, + utils::IteratorType::parallel); + rewriter.create( + op.getLoc(), TypeRange{}, ValueRange{op.getInput()}, + ValueRange{op.getOutput()}, ArrayRef{identity, transpose}, + iterators, [](OpBuilder &builder, Location location, ValueRange args) { + builder.create(location, args[0]); + }); + rewriter.eraseOp(op); + return success(); + } +}; +} // namespace + +void mlir::buddy::populateTransposeBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns) { + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/transpose/compiler/src/Conversion/LowerTileToBuckyball/TileTransposePatterns.cpp b/examples/balls/transpose/compiler/src/Conversion/LowerTileToBuckyball/TileTransposePatterns.cpp new file mode 100644 index 000000000..8b0a81c2a --- /dev/null +++ b/examples/balls/transpose/compiler/src/Conversion/LowerTileToBuckyball/TileTransposePatterns.cpp @@ -0,0 +1,50 @@ +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/IR/PatternMatch.h" + +#include "Buckyball/BuckyballOps.h" +#include "Tile/TileOps.h" + +using namespace mlir; +using namespace ::buddy::buckyball; +namespace tile = ::buddy::tile; + +namespace mlir::buddy { +void populateTransposeBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, + int64_t bankNum); +} // namespace mlir::buddy + +namespace { +class TileTransposeLowering : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(tile::TileTransposeOp op, + PatternRewriter &rewriter) const override { + auto inputType = dyn_cast(op.getAMemArray().getType()); + auto outputType = dyn_cast(op.getBMemArray().getType()); + if (!inputType || !outputType || !inputType.hasStaticShape() || + !outputType.hasStaticShape() || inputType.getRank() != 2 || + outputType.getRank() != 2) + return op.emitError("requires static rank-2 input and output memrefs"); + if (outputType.getShape()[0] != inputType.getShape()[1] || + outputType.getShape()[1] != inputType.getShape()[0] || + inputType.getElementType() != outputType.getElementType()) + return op.emitError("requires a matching transposed output memref"); + rewriter.create(op.getLoc(), op.getAMemArray(), + op.getBMemArray()); + rewriter.eraseOp(op); + return success(); + } +}; +} // namespace + +void mlir::buddy::populateTransposeBallTileLoweringPatterns( + RewritePatternSet &patterns, int64_t bankWidthBytes, int64_t bankDepth, + int64_t bankNum) { + (void)bankWidthBytes; + (void)bankDepth; + (void)bankNum; + patterns.add(patterns.getContext()); +} diff --git a/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index b41150a03..9ccc0ce2c 100644 --- a/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -4,6 +4,7 @@ #include "Buckyball/BuckyballOps.h" #include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; @@ -17,6 +18,7 @@ struct TransposeLowering : public ConvertOpToLLVMPattern { LogicalResult matchAndRewrite(TransposeOp op, OpAdaptor adaptor, ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("TransposeBall"); Location loc = op.getLoc(); Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInputBankId(), cstI64(rewriter, loc, 0), @@ -26,8 +28,37 @@ struct TransposeLowering : public ConvertOpToLLVMPattern { adaptor.getElemBits()); return success(); } - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getElemBits(), - rewriter.getI32IntegerAttr(49)); + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getElemBits(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("TRANSPOSE"))); + return success(); + } + +private: + bool stable = false; +}; +struct BankTransposeLowering : public ConvertOpToLLVMPattern { + BankTransposeLowering(LLVMTypeConverter &converter, bool stable) + : ConvertOpToLLVMPattern(converter), stable(stable) {} + + LogicalResult + matchAndRewrite(BankTransposeOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("TransposeBall"); + Location loc = op.getLoc(); + Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getInBank(), + cstI64(rewriter, loc, 0), adaptor.getOutBank(), + adaptor.getIter()); + if (stable) { + rewriter.replaceOpWithNewOp(op, rs1, + adaptor.getElemBits()); + return success(); + } + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getElemBits(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("TRANSPOSE"))); return success(); } @@ -37,13 +68,15 @@ struct TransposeLowering : public ConvertOpToLLVMPattern { } // namespace namespace mlir::buddy::buckyball { -void populateTransposeLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable) { +void populateTransposeBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t, bool) { patterns.add(converter, stable); + patterns.add(converter, stable); } -void configureTransposeLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable) { +void configureTransposeBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { if (stable) target.addLegalOp(); else diff --git a/examples/balls/transpose/compiler/src/Dialect/Buckyball/TransposeBall.td b/examples/balls/transpose/compiler/src/Dialect/Buckyball/TransposeBall.td index 4179f4a2c..f25c3b5f3 100644 --- a/examples/balls/transpose/compiler/src/Dialect/Buckyball/TransposeBall.td +++ b/examples/balls/transpose/compiler/src/Dialect/Buckyball/TransposeBall.td @@ -7,10 +7,9 @@ #ifndef TRANSPOSEBALL_OPS #define TRANSPOSEBALL_OPS -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" +include "Buckyball.td" -// Memref-level transpose (like matrix_matmul). Chip bank-ssa decides +// Memref-level transpose (like smatmul_matmul). Chip bank-ssa decides // pad / tile / Bank* composition; this op carries no bank policy. def MemTransposeOp : Buckyball_Op<"mem_transpose"> { let summary = "Logical matrix transpose on memrefs"; @@ -28,7 +27,7 @@ def TransposeOp : Buckyball_Op<"transpose"> { let summary = "Matrix transpose operation"; let description = [{ Transposes iter x W to W x iter in scratchpad banks. - funct7=49. ISA: bb_transpose(op1_bank, wr_bank, iter, elem_bits) + ISA: bb_transpose(op1_bank, wr_bank, iter, elem_bits); funct7 from Core ballISA. W = (bankWidth * cols) / elem_bits. }]; let arguments = (ins I64:$inputBankId, I64:$outputBankId, diff --git a/examples/balls/transpose/configs/TransposeBallParam.scala b/examples/balls/transpose/configs/TransposeBallParam.scala index 5d3a4c500..1bb46a731 100644 --- a/examples/balls/transpose/configs/TransposeBallParam.scala +++ b/examples/balls/transpose/configs/TransposeBallParam.scala @@ -5,8 +5,7 @@ import framework.top.GlobalConfig case class TransposeBallParam( InputNum: Int, - inputWidth: Int -) + inputWidth: Int) object TransposeBallParam { private val ballName = "TransposeBall" @@ -18,4 +17,5 @@ object TransposeBallParam { inputWidth = BallParamLoader.int(tbl, "inputWidth") ) } + } diff --git a/examples/balls/transpose/emu/src/49_transpose.rs b/examples/balls/transpose/emu/src/49_transpose.rs deleted file mode 100644 index cbd020197..000000000 --- a/examples/balls/transpose/emu/src/49_transpose.rs +++ /dev/null @@ -1,107 +0,0 @@ -use super::super::bank::{bank_num, bank_row_bytes, bank_size, bank_width}; -use super::decode::{pbank_group, rs1_b0, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; - -pub struct Transpose; - -impl Instruction for Transpose { - const FUNCT: u32 = 49; - - fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { - let op1 = rs1_b0(xs1); - let wr = rs1_b2(xs1); - let iter = rs1_iter(xs1) as usize; - let elem_bits = (xs2 & 0xff) as usize; - - if xs2 >> 8 != 0 { - panic!("transpose: rs2[63:8] must be 0"); - } - if op1 >= bank_num() as u64 || wr >= bank_num() as u64 { - panic!("transpose: invalid bank_id"); - } - if op1 == wr { - panic!("transpose: op1 and wr must differ"); - } - if iter == 0 { - panic!("transpose: iter must be > 0"); - } - if elem_bits != 8 && elem_bits != 32 { - panic!("transpose: unsupported elem_bits={elem_bits}"); - } - if bank_width() % elem_bits != 0 { - panic!( - "transpose: bank_width={} not divisible by elem_bits={}", - bank_width(), - elem_bits - ); - } - - let c1 = ctx.cfgs[op1 as usize].cols as usize; - let cw = ctx.cfgs[wr as usize].cols as usize; - if !ctx.cfgs[op1 as usize].allocated || !ctx.cfgs[wr as usize].allocated { - panic!("transpose: bank not allocated"); - } - if c1 == 0 || c1 != cw { - panic!("transpose: cols mismatch op1_cols={c1} wr_cols={cw}"); - } - - let n = c1; - let row_bytes = bank_row_bytes(); - let elem_bytes = elem_bits / 8; - let epg = row_bytes / elem_bytes; - let w = n * epg; - let total = iter * w; - - if iter * row_bytes > bank_size() { - panic!("transpose: src depth out of range iter={iter}"); - } - // Dest packs W×iter densely into virt rows of width W → depth = iter. - if iter * row_bytes > bank_size() { - panic!("transpose: dst depth out of range iter={iter}"); - } - - let mut src = vec![0u8; total * elem_bytes]; - for r in 0..iter { - for g in 0..n { - let p = pbank_group(ctx.bank_map, op1, g as u64); - let bank_off = r * row_bytes; - let flat_off = (r * w + g * epg) * elem_bytes; - src[flat_off..flat_off + row_bytes] - .copy_from_slice(&ctx.banks[p][bank_off..bank_off + row_bytes]); - } - } - - let mut dst = vec![0u8; total * elem_bytes]; - for r in 0..iter { - for c in 0..w { - let s = (r * w + c) * elem_bytes; - let d = (c * iter + r) * elem_bytes; - dst[d..d + elem_bytes].copy_from_slice(&src[s..s + elem_bytes]); - } - } - - for idx in 0..total { - let byte_off = idx * elem_bytes; - let virt_row = idx / w; - let virt_col = idx % w; - let g = virt_col / epg; - let lane = virt_col % epg; - let p = pbank_group(ctx.bank_map, wr, g as u64); - let bank_off = virt_row * row_bytes + lane * elem_bytes; - if bank_off + elem_bytes > bank_size() { - panic!("transpose: dst bank range idx={idx}"); - } - ctx.banks[p][bank_off..bank_off + elem_bytes] - .copy_from_slice(&dst[byte_off..byte_off + elem_bytes]); - } - - 0 - } - - fn latency(xs1: u64, xs2: u64) -> u64 { - let iter = rs1_iter(xs1).max(1); - let elem_bits = (xs2 & 0xff).max(1); - let epg = ((bank_width() as u64) / elem_bits).max(1); - iter.saturating_mul(epg).saturating_mul(2) - } -} diff --git a/examples/balls/transpose/emu/src/lib.rs b/examples/balls/transpose/emu/src/lib.rs index ad7e81e54..0b26a27ea 100644 --- a/examples/balls/transpose/emu/src/lib.rs +++ b/examples/balls/transpose/emu/src/lib.rs @@ -1,9 +1,8 @@ pub(crate) use crate::inst::{decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::ExecContext; -#[path = "49_transpose.rs"] -mod f49_transpose; +mod transpose; const BALL_CLASS: &str = "examples.balls.transpose.TransposeBall"; @@ -17,11 +16,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("TRANSPOSE") => Some(transpose::exec_transpose(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -29,10 +26,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => { - Some(::latency(xs1, xs2)) - } - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("TRANSPOSE") => Some(transpose::latency(xs1, xs2)), + Some(_) | None => None, } } diff --git a/examples/balls/transpose/emu/src/transpose.rs b/examples/balls/transpose/emu/src/transpose.rs new file mode 100644 index 000000000..5b1c01a39 --- /dev/null +++ b/examples/balls/transpose/emu/src/transpose.rs @@ -0,0 +1,101 @@ +use super::super::bank::{bank_num, bank_row_bytes, bank_size, bank_width}; +use super::decode::{pbank_group, rs1_b0, rs1_b2, rs1_iter}; +use super::instruction::ExecContext; + +pub(crate) fn exec_transpose(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { + let op1 = rs1_b0(xs1); + let wr = rs1_b2(xs1); + let iter = rs1_iter(xs1) as usize; + let elem_bits = (xs2 & 0xff) as usize; + + if xs2 >> 8 != 0 { + panic!("transpose: rs2[63:8] must be 0"); + } + if op1 >= bank_num() as u64 || wr >= bank_num() as u64 { + panic!("transpose: invalid bank_id"); + } + if op1 == wr { + panic!("transpose: op1 and wr must differ"); + } + if iter == 0 { + panic!("transpose: iter must be > 0"); + } + if elem_bits != 8 && elem_bits != 32 { + panic!("transpose: unsupported elem_bits={elem_bits}"); + } + if bank_width() % elem_bits != 0 { + panic!( + "transpose: bank_width={} not divisible by elem_bits={}", + bank_width(), + elem_bits + ); + } + + let c1 = ctx.cfgs[op1 as usize].cols as usize; + let cw = ctx.cfgs[wr as usize].cols as usize; + if !ctx.cfgs[op1 as usize].allocated || !ctx.cfgs[wr as usize].allocated { + panic!("transpose: bank not allocated"); + } + if c1 == 0 || c1 != cw { + panic!("transpose: cols mismatch op1_cols={c1} wr_cols={cw}"); + } + + let n = c1; + let row_bytes = bank_row_bytes(); + let elem_bytes = elem_bits / 8; + let epg = row_bytes / elem_bytes; + let w = n * epg; + let total = iter * w; + + if iter * row_bytes > bank_size() { + panic!("transpose: src depth out of range iter={iter}"); + } + // Dest packs W×iter densely into virt rows of width W → depth = iter. + if iter * row_bytes > bank_size() { + panic!("transpose: dst depth out of range iter={iter}"); + } + + let mut src = vec![0u8; total * elem_bytes]; + for r in 0..iter { + for g in 0..n { + let p = pbank_group(ctx.bank_map, op1, g as u64); + let bank_off = r * row_bytes; + let flat_off = (r * w + g * epg) * elem_bytes; + src[flat_off..flat_off + row_bytes] + .copy_from_slice(&ctx.banks[p][bank_off..bank_off + row_bytes]); + } + } + + let mut dst = vec![0u8; total * elem_bytes]; + for r in 0..iter { + for c in 0..w { + let s = (r * w + c) * elem_bytes; + let d = (c * iter + r) * elem_bytes; + dst[d..d + elem_bytes].copy_from_slice(&src[s..s + elem_bytes]); + } + } + + for idx in 0..total { + let byte_off = idx * elem_bytes; + let virt_row = idx / w; + let virt_col = idx % w; + let g = virt_col / epg; + let lane = virt_col % epg; + let p = pbank_group(ctx.bank_map, wr, g as u64); + let bank_off = virt_row * row_bytes + lane * elem_bytes; + if bank_off + elem_bytes > bank_size() { + panic!("transpose: dst bank range idx={idx}"); + } + ctx.banks[p][bank_off..bank_off + elem_bytes] + .copy_from_slice(&dst[byte_off..byte_off + elem_bytes]); + } + + 0 +} + +pub(crate) fn latency(xs1: u64, xs2: u64) -> u64 { + let iter = rs1_iter(xs1).max(1); + let elem_bits = (xs2 & 0xff).max(1); + let epg = ((bank_width() as u64) / elem_bits).max(1); + iter.saturating_mul(epg).saturating_mul(2) +} diff --git a/examples/balls/transpose/verify/casegen/src/casegen.rs b/examples/balls/transpose/verify/casegen/src/casegen.rs index 4c7dd1e4b..38018e500 100644 --- a/examples/balls/transpose/verify/casegen/src/casegen.rs +++ b/examples/balls/transpose/verify/casegen/src/casegen.rs @@ -1,6 +1,5 @@ use crate::model; -pub const FUNCT7: u32 = 49; pub const BANK_ROW_BYTES: usize = 16; pub const MAX_ITER: usize = 16; pub const MAX_WORDS: usize = MAX_ITER; @@ -9,7 +8,6 @@ pub const MAX_WORDS: usize = MAX_ITER; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct TransposeCmd { pub bid: u32, - pub funct7: u32, pub iter: u32, pub op1_bank: u32, pub wr_bank: u32, @@ -77,7 +75,6 @@ fn directed_i8(bid: u32) -> TransposeCase { TransposeCase { cmd: TransposeCmd { bid, - funct7: FUNCT7, iter: iter as u32, op1_bank: 0, wr_bank: 1, @@ -99,13 +96,15 @@ fn directed_i32(bid: u32) -> TransposeCase { let cols = 1usize; let w = cols * (BANK_ROW_BYTES / elem_bytes); let total = iter * w; - let src: Vec = (0..total).map(|i| i as u32).flat_map(|v| v.to_le_bytes()).collect(); + let src: Vec = (0..total) + .map(|i| i as u32) + .flat_map(|v| v.to_le_bytes()) + .collect(); let dst = model::transpose_bytes(&src, iter, w, elem_bytes); let nwords = model::num_words(total * elem_bytes) as u32; TransposeCase { cmd: TransposeCmd { bid, - funct7: FUNCT7, iter: iter as u32, op1_bank: 0, wr_bank: 1, @@ -148,7 +147,6 @@ fn random_case(seed: u32, index: u32, bid: u32) -> TransposeCase { TransposeCase { cmd: TransposeCmd { bid, - funct7: FUNCT7, iter: iter as u32, op1_bank, wr_bank, @@ -205,7 +203,10 @@ mod tests { fn model_transpose_i32_roundtrip() { let iter = 8; let w = 4; - let src: Vec = (0..iter * w).map(|i| i as u32).flat_map(|v| v.to_le_bytes()).collect(); + let src: Vec = (0..iter * w) + .map(|i| i as u32) + .flat_map(|v| v.to_le_bytes()) + .collect(); let dst = model::transpose_bytes(&src, iter, w, 4); for r in 0..iter { for c in 0..w { @@ -220,7 +221,6 @@ mod tests { fn directed_i8_matches_ctest_shape() { let case = gen_case(0x1234, 0, 0); assert_eq!(case.cmd.bid, 0); - assert_eq!(case.cmd.funct7, FUNCT7); assert_eq!(case.cmd.iter, 16); assert_eq!(case.cmd.elem_bits, 8); assert_eq!(case.cmd.op1_bank, 0); diff --git a/examples/balls/transpose/verify/casegen/src/lib.rs b/examples/balls/transpose/verify/casegen/src/lib.rs index 7a7e8ee1e..7f09fb927 100644 --- a/examples/balls/transpose/verify/casegen/src/lib.rs +++ b/examples/balls/transpose/verify/casegen/src/lib.rs @@ -77,12 +77,18 @@ mod tests { fn dpi_load_then_cmd_and_words() { assert_eq!(transpose_case_load(0x1234, 0, 0), 0); let mut cmd = TransposeCmd { - bid: 0, funct7: 0, iter: 0, op1_bank: 0, wr_bank: 0, - op1_col: 0, wr_col: 0, rob_id: 0, elem_bits: 0, - num_src_words: 0, num_dst_words: 0, + bid: 0, + iter: 0, + op1_bank: 0, + wr_bank: 0, + op1_col: 0, + wr_col: 0, + rob_id: 0, + elem_bits: 0, + num_src_words: 0, + num_dst_words: 0, }; transpose_case_cmd(&mut cmd as *mut TransposeCmd); - assert_eq!(cmd.funct7, 49); assert_eq!(cmd.iter, 16); assert_eq!(cmd.elem_bits, 8); assert_eq!(cmd.num_src_words, 16); diff --git a/examples/balls/transpose/verify/casegen/src/model.rs b/examples/balls/transpose/verify/casegen/src/model.rs index 13acf8f5d..7d6ce7d8a 100644 --- a/examples/balls/transpose/verify/casegen/src/model.rs +++ b/examples/balls/transpose/verify/casegen/src/model.rs @@ -15,6 +15,10 @@ pub fn transpose_bytes(src: &[u8], iter: usize, w: usize, elem_bytes: usize) -> } pub fn num_words(total_bytes: usize) -> usize { - assert_eq!(total_bytes % BANK_ROW_BYTES, 0, "total_bytes not a multiple of bank row"); + assert_eq!( + total_bytes % BANK_ROW_BYTES, + 0, + "total_bytes not a multiple of bank row" + ); total_bytes / BANK_ROW_BYTES } diff --git a/examples/balls/transpose/verify/src/common/transpose_defs.svh b/examples/balls/transpose/verify/src/common/transpose_defs.svh index 75f1cb01b..5b2313d78 100644 --- a/examples/balls/transpose/verify/src/common/transpose_defs.svh +++ b/examples/balls/transpose/verify/src/common/transpose_defs.svh @@ -1,6 +1,5 @@ typedef struct { int unsigned bid; - int unsigned funct7; int unsigned iter; int unsigned op1_bank; int unsigned wr_bank; @@ -23,7 +22,10 @@ import "DPI-C" function longint unsigned transpose_case_src_word_hi(input int un import "DPI-C" function longint unsigned transpose_case_dst_word_lo(input int unsigned word_index); import "DPI-C" function longint unsigned transpose_case_dst_word_hi(input int unsigned word_index); -localparam int TRANSPOSE_FUNCT7 = 7'd49; +`ifndef TRANSPOSE_FUNCT7 +`error "TRANSPOSE_FUNCT7 must be provided by the selected Core ballISA" +`endif +localparam int TRANSPOSE_CORE_FUNCT7 = `TRANSPOSE_FUNCT7; localparam int TRANSPOSE_MAX_WORDS = 16; localparam int TRANSPOSE_TIMEOUT_CYCLES = 2000; localparam int TRANSPOSE_SEED = 32'hCAFE_BABE; diff --git a/examples/balls/transpose/verify/src/common/transpose_items.svh b/examples/balls/transpose/verify/src/common/transpose_items.svh index 789780800..19f26044e 100644 --- a/examples/balls/transpose/verify/src/common/transpose_items.svh +++ b/examples/balls/transpose/verify/src/common/transpose_items.svh @@ -8,7 +8,7 @@ class transpose_cmd_item extends bb_blink_cmd_item; bit [127:0] dst_words[TRANSPOSE_MAX_WORDS]; constraint legal_c { - funct7 == TRANSPOSE_FUNCT7[6:0]; + funct7 == TRANSPOSE_CORE_FUNCT7[6:0]; op1_en == 1'b1; op2_en == 1'b0; wr_spad_en == 1'b1; @@ -41,7 +41,7 @@ class transpose_cmd_item extends bb_blink_cmd_item; transpose_case_cmd(cmd); bid = cmd.bid[4:0]; - funct7 = cmd.funct7[6:0]; + funct7 = TRANSPOSE_CORE_FUNCT7[6:0]; iter = cmd.iter; op1_bank = cmd.op1_bank[4:0]; wr_bank = cmd.wr_bank[4:0]; diff --git a/examples/balls/transpose/verify/src/tb_top.sv b/examples/balls/transpose/verify/src/tb_top.sv index d71672fcc..d2327f0f2 100644 --- a/examples/balls/transpose/verify/src/tb_top.sv +++ b/examples/balls/transpose/verify/src/tb_top.sv @@ -71,9 +71,9 @@ module tb_top; .io_bankWrite_0_io_resp_valid(intf.bank_write_resp_valid[0]), .io_bankWrite_0_io_resp_bits_ok(intf.bank_write_resp_ok[0]), .io_subRobReq_ready(intf.sub_rob_req_ready), - .io_mmioRead_req_ready(intf.mmio_read_req_ready), - .io_mmioRead_resp_valid(intf.mmio_read_resp_valid), - .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data) + .io_mmioRead_req_ready(intf.mmio_read_req_ready[0]), + .io_mmioRead_resp_valid(intf.mmio_read_resp_valid[0]), + .io_mmioRead_resp_bits_data(intf.mmio_read_resp_bits_data[0]) ); initial begin diff --git a/examples/balls/transpose/workloads/ctests/transpose_bank_i32_n4_1024x16_test.c b/examples/balls/transpose/workloads/ctests/transpose_bank_i32_n4_1024x16_test.c index ac94dd529..39e1f36f7 100644 --- a/examples/balls/transpose/workloads/ctests/transpose_bank_i32_n4_1024x16_test.c +++ b/examples/balls/transpose/workloads/ctests/transpose_bank_i32_n4_1024x16_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #include #include @@ -37,6 +38,8 @@ int main(void) { bb_transpose(src, dst, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, dst, ROWS, 1); bb_fence(); + bb_mem_release(src); + bb_mem_release(dst); for (int i = 0; i < COLS * ROWS; ++i) { if (output_matrix[i] != expected_matrix[i]) { diff --git a/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n1_1024x16_test.c b/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n1_1024x16_test.c index 4a3ba0513..ac6dfb0d3 100644 --- a/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n1_1024x16_test.c +++ b/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n1_1024x16_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define ROWS BANK_LINES @@ -31,6 +32,8 @@ int main(void) { bb_transpose(src, dst, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, dst, ROWS, 1); bb_fence(); + bb_mem_release(src); + bb_mem_release(dst); if (compare_i8_matrices(output_matrix, expected_matrix, COLS, ROWS)) { printf("Transpose bank i8 n1 1024x16 PASSED\n"); diff --git a/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n2_1024x32_test.c b/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n2_1024x32_test.c index 77cda6454..c6d94c78d 100644 --- a/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n2_1024x32_test.c +++ b/examples/balls/transpose/workloads/ctests/transpose_bank_i8_n2_1024x32_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define ROWS BANK_LINES @@ -31,6 +32,8 @@ int main(void) { bb_transpose(src, dst, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, dst, ROWS, 1); bb_fence(); + bb_mem_release(src); + bb_mem_release(dst); if (compare_i8_matrices(output_matrix, expected_matrix, COLS, ROWS)) { printf("Transpose bank i8 n2 1024x32 PASSED\n"); diff --git a/examples/balls/transpose/workloads/ctests/transpose_i32_8x16_test.c b/examples/balls/transpose/workloads/ctests/transpose_i32_8x16_test.c index 3bdbf1d23..c6e1ba711 100644 --- a/examples/balls/transpose/workloads/ctests/transpose_i32_8x16_test.c +++ b/examples/balls/transpose/workloads/ctests/transpose_i32_8x16_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #include #include @@ -46,6 +47,8 @@ int main(void) { bb_transpose(src, dst, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, dst, ROWS, 1); bb_fence(); + bb_mem_release(src); + bb_mem_release(dst); for (int i = 0; i < COLS * ROWS; ++i) { if (output_matrix[i] != expected_matrix[i]) { diff --git a/examples/balls/transpose/workloads/ctests/transpose_i8_16x16_test.c b/examples/balls/transpose/workloads/ctests/transpose_i8_16x16_test.c index b8baa170a..72de43e4c 100644 --- a/examples/balls/transpose/workloads/ctests/transpose_i8_16x16_test.c +++ b/examples/balls/transpose/workloads/ctests/transpose_i8_16x16_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define ROWS 16 @@ -52,6 +53,8 @@ int main(void) { bb_transpose(src, dst, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, dst, ROWS, 1); bb_fence(); + bb_mem_release(src); + bb_mem_release(dst); if (compare_u8_matrices(output_matrix, expected_matrix, COLS, ROWS)) { printf("Transpose i8 16x16 PASSED\n"); diff --git a/examples/balls/transpose/workloads/ctests/transpose_i8_16x32_test.c b/examples/balls/transpose/workloads/ctests/transpose_i8_16x32_test.c index 2f47d671e..eb05d2414 100644 --- a/examples/balls/transpose/workloads/ctests/transpose_i8_16x32_test.c +++ b/examples/balls/transpose/workloads/ctests/transpose_i8_16x32_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define ROWS 16 @@ -57,6 +58,8 @@ int main(void) { bb_transpose(src, dst, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, dst, ROWS, 1); bb_fence(); + bb_mem_release(src); + bb_mem_release(dst); if (compare_u8_matrices(output_matrix, expected_matrix, COLS, ROWS)) { printf("Transpose i8 16x32 PASSED\n"); diff --git a/bb-tests/workloads/lib/bbhw/isa/49_transpose.c b/examples/balls/transpose/workloads/isa/transpose.h similarity index 78% rename from bb-tests/workloads/lib/bbhw/isa/49_transpose.c rename to examples/balls/transpose/workloads/isa/transpose.h index 75d9af638..4c219ae72 100644 --- a/bb-tests/workloads/lib/bbhw/isa/49_transpose.c +++ b/examples/balls/transpose/workloads/isa/transpose.h @@ -1,9 +1,8 @@ #ifndef _BB_TRANSPOSE_H_ #define _BB_TRANSPOSE_H_ -#include "isa.h" - -#define BB_TRANSPOSE_FUNC7 49 +#include +#include // bb_transpose(op1_bank_id, wr_bank_id, iter, elem_bits) // rs1 = banks | iter @@ -11,6 +10,6 @@ #define bb_transpose(op1_bank_id, wr_bank_id, iter, elem_bits) \ BUCKYBALL_INSTRUCTION_R_R( \ (BB_BANK0(op1_bank_id) | BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ - (FIELD((uint64_t)(elem_bits), 0, 7)), BB_TRANSPOSE_FUNC7) + (FIELD((uint64_t)(elem_bits), 0, 7)), BB_FUNC7(TRANSPOSE)) #endif // _BB_TRANSPOSE_H_ diff --git a/examples/balls/transpose/workloads/mlir_tests/ball/CMakeLists.txt b/examples/balls/transpose/workloads/mlir_tests/ball/CMakeLists.txt index 21cbf815d..99f3d6907 100644 --- a/examples/balls/transpose/workloads/mlir_tests/ball/CMakeLists.txt +++ b/examples/balls/transpose/workloads/mlir_tests/ball/CMakeLists.txt @@ -1,12 +1,11 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX ball) add_buckyball_mlir_contract(transpose_16x16_i8_contract + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} PASSES - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} ${BUCKYBALL_LOWER_BUCKYBALL}) -add_buckyball_mlir_optest(transpose_16x16_i8) +add_buckyball_mlir_test(transpose_16x16_i8 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8.mlir b/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8.mlir index cb589ef9e..3bdc770ae 100644 --- a/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8.mlir +++ b/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8.mlir @@ -13,8 +13,8 @@ func.func @main() -> i8 { %iter = arith.constant 16 : i64 %elem_bits = arith.constant 8 : i64 - %input = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<16x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> scf.for %i = %c0 to %n step %c1 { scf.for %j = %c0 to %n step %c1 { @@ -34,6 +34,7 @@ func.func @main() -> i8 { buckyball.transpose %loaded, %bout, %iter, %elem_bits : i64 %stored = buckyball.bank_mvout %output %bout %depth %stride : memref<16x16xi8> i64 i64 i64 + buckyball.fence buckyball.bank_release %loaded : i64 buckyball.bank_release %stored : i64 diff --git a/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8_contract.mlir b/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8_contract.mlir index a220e54e5..8cf849a15 100644 --- a/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8_contract.mlir +++ b/examples/balls/transpose/workloads/mlir_tests/ball/transpose_16x16_i8_contract.mlir @@ -1,5 +1,5 @@ // CHECK: buckyball.intr.custom -// CHECK-SAME: funct7 = 49 +// CHECK-SAME: funct7 = 49 : i32 // CHECK-NOT: buckyball.transpose func.func @main() -> i8 { diff --git a/examples/balls/transpose/workloads/mlir_tests/bank/CMakeLists.txt b/examples/balls/transpose/workloads/mlir_tests/bank/CMakeLists.txt index 048f2d09a..a0574da7e 100644 --- a/examples/balls/transpose/workloads/mlir_tests/bank/CMakeLists.txt +++ b/examples/balls/transpose/workloads/mlir_tests/bank/CMakeLists.txt @@ -1,9 +1,8 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET balls-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX bank) add_buckyball_mlir_contract(transpose_16x16_i8_contract - PASSES "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}") + TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET} + PASSES -assign-physical-banks) -add_buckyball_mlir_optest(transpose_16x16_i8) +add_buckyball_mlir_test(transpose_16x16_i8 TARGET ${BUCKYBALL_MLIR_ACTIVE_TARGET}) diff --git a/examples/balls/transpose/workloads/mlir_tests/bank/transpose_16x16_i8.mlir b/examples/balls/transpose/workloads/mlir_tests/bank/transpose_16x16_i8.mlir index 4f9b93ae3..c88dee47f 100644 --- a/examples/balls/transpose/workloads/mlir_tests/bank/transpose_16x16_i8.mlir +++ b/examples/balls/transpose/workloads/mlir_tests/bank/transpose_16x16_i8.mlir @@ -14,8 +14,8 @@ func.func @main() -> i8 { %iter = arith.constant 16 : i64 %elem_bits = arith.constant 8 : i64 - %input = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<16x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> scf.for %i = %c0 to %n step %c1 { scf.for %j = %c0 to %n step %c1 { @@ -36,6 +36,7 @@ func.func @main() -> i8 { : i64 i64 i64 i64 %stored = buckyball.bank_mvout %output %t %depth %stride : memref<16x16xi8> i64 i64 i64 + buckyball.fence buckyball.bank_release %loaded : i64 buckyball.bank_release %stored : i64 diff --git a/examples/balls/vector/arch/src/main/scala/VecBall.scala b/examples/balls/vector/arch/src/main/scala/VecBall.scala index 1e948d360..7cf6bc243 100644 --- a/examples/balls/vector/arch/src/main/scala/VecBall.scala +++ b/examples/balls/vector/arch/src/main/scala/VecBall.scala @@ -14,10 +14,14 @@ import framework.top.GlobalConfig @instantiable class VecBall(val b: GlobalConfig) extends Module with HasBlink with HasBallStatus { - val ballCommonConfig = b.ballDomain.ballIdMappings.find(_.ballName == "VecBall") - .getOrElse(throw new IllegalArgumentException("VecBall not found in config")) - val inBW = ballCommonConfig.inBW - val outBW = ballCommonConfig.outBW + val ballCommonConfig = b.ballDomain.ballIdMappings + .find(_.ballName == "VecBall") + .getOrElse( + throw new IllegalArgumentException("VecBall not found in config") + ) + + val inBW = ballCommonConfig.inBW + val outBW = ballCommonConfig.outBW @public val io = IO(new BlinkIO(b, inBW, outBW)) diff --git a/examples/balls/vector/arch/src/main/scala/VecCtrlUnit.scala b/examples/balls/vector/arch/src/main/scala/VecCtrlUnit.scala index 125ca451c..07ab558f3 100644 --- a/examples/balls/vector/arch/src/main/scala/VecCtrlUnit.scala +++ b/examples/balls/vector/arch/src/main/scala/VecCtrlUnit.scala @@ -19,7 +19,9 @@ class VecCtrlUnit(val b: GlobalConfig) extends Module { val ctrl_st_o = Decoupled(new ctrl_st_req(b)) val ctrl_ex_o = Decoupled(new ctrl_ex_req(b)) - val cmdResp_i = Flipped(Valid(new Bundle { val commit = Bool() })) // from store unit + val cmdResp_i = Flipped(Valid(new Bundle { + val commit = Bool() + })) // from store unit }) val rob_id_reg = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) @@ -27,13 +29,21 @@ class VecCtrlUnit(val b: GlobalConfig) extends Module { val sub_rob_id_reg = RegInit(0.U(log2Up(b.frontend.sub_rob_depth * 4).W)) val iter = RegInit(0.U(b.frontend.iter_len.W)) val op1_bank = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val op1_bank_addr = RegInit(0.U(12.W)) // New ISA: always 0, but keep for compatibility - val op2_bank_addr = RegInit(0.U(12.W)) // New ISA: always 0, but keep for compatibility - val op2_bank = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val wr_bank = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) - val wr_bank_addr = RegInit(0.U(12.W)) // New ISA: always 0, but keep for compatibility - val has_send = RegInit(false.B) - val mode = RegInit(0.U(1.W)) + + val op1_bank_addr = RegInit( + 0.U(12.W) + ) // New ISA: always 0, but keep for compatibility + val op2_bank_addr = RegInit( + 0.U(12.W) + ) // New ISA: always 0, but keep for compatibility + val op2_bank = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + val wr_bank = RegInit(0.U(log2Up(b.memDomain.bankNum).W)) + + val wr_bank_addr = RegInit( + 0.U(12.W) + ) // New ISA: always 0, but keep for compatibility + val has_send = RegInit(false.B) + val mode = RegInit(0.U(1.W)) val idle :: busy :: Nil = Enum(2) val state = RegInit(idle) diff --git a/examples/balls/vector/arch/src/main/scala/VecEXUnit.scala b/examples/balls/vector/arch/src/main/scala/VecEXUnit.scala index 648d4d0a0..484a3f940 100644 --- a/examples/balls/vector/arch/src/main/scala/VecEXUnit.scala +++ b/examples/balls/vector/arch/src/main/scala/VecEXUnit.scala @@ -64,7 +64,11 @@ class VecEXUnit(val b: GlobalConfig) extends Module { state := busy } when(io.ld_ex_i.fire) { - threadId := Mux(threadId === (config.numMulThreads - 1).U, 0.U, threadId + 1.U) + threadId := Mux( + threadId === (config.numMulThreads - 1).U, + 0.U, + threadId + 1.U + ) } // ----------------------------------------------------------------------------- diff --git a/examples/balls/vector/arch/src/main/scala/VecLoadUnit.scala b/examples/balls/vector/arch/src/main/scala/VecLoadUnit.scala index 678b68f5e..5ec9b796d 100644 --- a/examples/balls/vector/arch/src/main/scala/VecLoadUnit.scala +++ b/examples/balls/vector/arch/src/main/scala/VecLoadUnit.scala @@ -26,9 +26,13 @@ class VecLoadUnit(val b: GlobalConfig) extends Module { val rob_id_width = log2Up(b.frontend.rob_entries) // Get bandwidth from config (use first VecBall mapping) - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "VecBall") - .getOrElse(throw new IllegalArgumentException("VecBall not found in config")) - val inBW = ballMapping.inBW + val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "VecBall") + .getOrElse( + throw new IllegalArgumentException("VecBall not found in config") + ) + + val inBW = ballMapping.inBW @public val io = IO(new Bundle { @@ -97,7 +101,11 @@ class VecLoadUnit(val b: GlobalConfig) extends Module { when(state === busy && io.ld_ex_o.ready && !wait1_reg) { io.bankReadReq(0).valid := op1_iter_counter < iter io.bankReadReq(0).bits.addr := op1_addr + op1_iter_counter - op1_iter_counter := Mux(io.bankReadReq(0).ready, op1_iter_counter + 1.U, op1_iter_counter) + op1_iter_counter := Mux( + io.bankReadReq(0).ready, + op1_iter_counter + 1.U, + op1_iter_counter + ) wait1_reg := Mux((op1_iter_counter + 1.U) % 16.U === 0.U, 1.U, 0.U) when(io.bankReadReq(0).fire) {} } @@ -105,7 +113,11 @@ class VecLoadUnit(val b: GlobalConfig) extends Module { when(state === busy && io.ld_ex_o.ready && !wait2_reg) { io.bankReadReq(1).valid := op2_iter_counter < iter io.bankReadReq(1).bits.addr := op2_addr + op2_iter_counter - op2_iter_counter := Mux(io.bankReadReq(1).ready, op2_iter_counter + 1.U, op2_iter_counter) + op2_iter_counter := Mux( + io.bankReadReq(1).ready, + op2_iter_counter + 1.U, + op2_iter_counter + ) wait2_reg := Mux((op2_iter_counter + 1.U) % 16.U === 0.U, 1.U, 0.U) when(io.bankReadReq(1).fire) {} } @@ -133,8 +145,10 @@ class VecLoadUnit(val b: GlobalConfig) extends Module { io.ld_ex_o.valid := both_valid when(both_valid) { - io.ld_ex_o.bits.op1 := bankRespQueue0.io.deq.bits.data.asTypeOf(Vec(InputNum, UInt(inputWidth.W))) - io.ld_ex_o.bits.op2 := bankRespQueue1.io.deq.bits.data.asTypeOf(Vec(InputNum, UInt(inputWidth.W))) + io.ld_ex_o.bits.op1 := bankRespQueue0.io.deq.bits.data + .asTypeOf(Vec(InputNum, UInt(inputWidth.W))) + io.ld_ex_o.bits.op2 := bankRespQueue1.io.deq.bits.data + .asTypeOf(Vec(InputNum, UInt(inputWidth.W))) io.ld_ex_o.bits.iter := ld_ex_iter_reg }.otherwise { io.ld_ex_o.bits.iter := 0.U diff --git a/examples/balls/vector/arch/src/main/scala/MulWarp16.scala b/examples/balls/vector/arch/src/main/scala/VecMat16.scala similarity index 61% rename from examples/balls/vector/arch/src/main/scala/MulWarp16.scala rename to examples/balls/vector/arch/src/main/scala/VecMat16.scala index ddf07edb3..33ca65d25 100644 --- a/examples/balls/vector/arch/src/main/scala/MulWarp16.scala +++ b/examples/balls/vector/arch/src/main/scala/VecMat16.scala @@ -6,13 +6,19 @@ import framework.balldomain.blink.{BallStatus, BankRead, BankWrite} import framework.balldomain.rs.{BallRsComplete, BallRsIssue} import framework.top.GlobalConfig -class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, accW: Int) +class VecMat16( + val b: GlobalConfig, + lane: Int, + inBW: Int, + outBW: Int, + inW: Int, + accW: Int) extends Module { - require(lane == 16, "MulWarp16 requires lane=16") - require(inW == 8, "MulWarp16 requires int8 inputs") - require(accW == 32, "MulWarp16 requires int32 accumulators") - require(inBW >= 2, "MulWarp16 requires two read ports") - require(outBW >= 4, "MulWarp16 requires four write ports") + require(lane == 16, "VecMat16 requires lane=16") + require(inW == 8, "VecMat16 requires int8 inputs") + require(accW == 32, "VecMat16 requires int32 accumulators") + require(inBW >= 2, "VecMat16 requires two read ports") + require(outBW >= 4, "VecMat16 requires four write ports") val io = IO(new Bundle { val cmdReq = Flipped(Decoupled(new BallRsIssue(b))) @@ -22,8 +28,9 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, val status = new BallStatus }) - val sIdle :: sLoadAcc :: sReadReq :: sReadResp :: sWrite :: sDone :: Nil = Enum(6) - val state = RegInit(sIdle) + val sIdle :: sLoadAcc :: sReadReq :: sReadResp :: sWrite :: sDone :: Nil = + Enum(6) + val state = RegInit(sIdle) val robIdReg = RegInit(0.U(log2Up(b.frontend.rob_entries).W)) val isSubReg = RegInit(false.B) @@ -41,8 +48,8 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, val bWord = RegInit(0.U(b.memDomain.bankWidth.W)) val acc = Reg(Vec(lane, Vec(lane, UInt(accW.W)))) - val accLoad = Module(new MulWarp16AccLoad(b, lane, accW)) - val store = Module(new MulWarp16Store(b, lane, accW)) + val accLoad = Module(new VecMat16AccLoad(b, lane, accW)) + val store = Module(new VecMat16Store(b, lane, accW)) accLoad.io.start := false.B accLoad.io.wrBank := wrBank accLoad.io.robId := robIdReg @@ -68,29 +75,29 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, io.cmdResp.bits.sub_rob_id := subRobIdReg io.status.idle := state === sIdle io.status.running := state =/= sIdle && state =/= sDone - MulWarp16Tie.offRead(io.bankRead, robIdReg) - MulWarp16Tie.offWrite(io.bankWrite, robIdReg, wrBank, b) + VecMat16Tie.offRead(io.bankRead, robIdReg) + VecMat16Tie.offWrite(io.bankWrite, robIdReg, wrBank, b) switch(state) { is(sIdle) { when(io.cmdReq.fire) { val cmd = io.cmdReq.bits.cmd - robIdReg := io.cmdReq.bits.rob_id - isSubReg := io.cmdReq.bits.is_sub - subRobIdReg := io.cmdReq.bits.sub_rob_id - op1Bank := cmd.op1_bank - op2Bank := cmd.op2_bank - wrBank := cmd.wr_bank - iterReg := cmd.iter - readRow := 0.U - aReqDone := false.B - bReqDone := false.B - aRespDone := false.B - bRespDone := false.B - assert(cmd.iter > 0.U, "MulWarp16 iter must be > 0") + robIdReg := io.cmdReq.bits.rob_id + isSubReg := io.cmdReq.bits.is_sub + subRobIdReg := io.cmdReq.bits.sub_rob_id + op1Bank := cmd.op1_bank + op2Bank := cmd.op2_bank + wrBank := cmd.wr_bank + iterReg := cmd.iter + readRow := 0.U + aReqDone := false.B + bReqDone := false.B + aRespDone := false.B + bRespDone := false.B + assert(cmd.iter > 0.U, "VecMat16 iter must be > 0") assert( cmd.op1_col === 1.U && cmd.op2_col === 1.U && cmd.wr_col === 4.U, - "MulWarp16 unsupported bank layout" + "VecMat16 unsupported bank layout" ) accLoad.io.start := true.B state := sLoadAcc @@ -98,7 +105,9 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, } is(sLoadAcc) { - for (p <- 0 until 2) { MulWarp16Tie.takeRead(io.bankRead(p), accLoad.io.bankRead(p)) } + for (p <- 0 until 2) { + VecMat16Tie.takeRead(io.bankRead(p), accLoad.io.bankRead(p)) + } when(accLoad.io.done) { acc := accLoad.io.acc aReqDone := false.B @@ -108,15 +117,21 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, } is(sReadReq) { - io.bankRead(0).bank_id := op1Bank - io.bankRead(0).io.req.valid := !aReqDone - io.bankRead(0).io.req.bits.addr := readRow - io.bankRead(1).bank_id := op2Bank - io.bankRead(1).io.req.valid := !bReqDone - io.bankRead(1).io.req.bits.addr := readRow - when(io.bankRead(0).io.req.fire) { aReqDone := true.B } - when(io.bankRead(1).io.req.fire) { bReqDone := true.B } - when((aReqDone || io.bankRead(0).io.req.fire) && (bReqDone || io.bankRead(1).io.req.fire)) { + io.bankRead(0).bank_id := op1Bank + io.bankRead(0).io.req.valid := !aReqDone + io.bankRead(0).io.req.bits.addr := readRow + io.bankRead(1).bank_id := op2Bank + io.bankRead(1).io.req.valid := !bReqDone + io.bankRead(1).io.req.bits.addr := readRow + when(io.bankRead(0).io.req.fire)(aReqDone := true.B) + when(io.bankRead(1).io.req.fire)(bReqDone := true.B) + when( + (aReqDone || io + .bankRead(0) + .io + .req + .fire) && (bReqDone || io.bankRead(1).io.req.fire) + ) { aRespDone := false.B bRespDone := false.B state := sReadResp @@ -134,15 +149,20 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, val nextB = Mux(bFire, io.bankRead(1).io.resp.bits.data, bWord) val haveA = aRespDone || aFire val haveB = bRespDone || bFire - when(aFire) { aWord := io.bankRead(0).io.resp.bits.data } - when(bFire) { bWord := io.bankRead(1).io.resp.bits.data } - when(haveA) { aRespDone := true.B } - when(haveB) { bRespDone := true.B } + when(aFire)(aWord := io.bankRead(0).io.resp.bits.data) + when(bFire)(bWord := io.bankRead(1).io.resp.bits.data) + when(haveA)(aRespDone := true.B) + when(haveB)(bRespDone := true.B) when(haveA && haveB) { val aElems = nextA.asTypeOf(Vec(lane, UInt(inW.W))) val bElems = nextB.asTypeOf(Vec(lane, UInt(inW.W))) - for (i <- 0 until lane; j <- 0 until lane) { - acc(i)(j) := (acc(i)(j).asSInt + aElems(i).asSInt * bElems(j).asSInt).asUInt + for { + i <- 0 until lane + j <- 0 until lane + } { + acc(i)(j) := (acc(i)(j).asSInt + aElems(i).asSInt * bElems( + j + ).asSInt).asUInt } when(readRow === iterReg - 1.U) { store.io.start := true.B @@ -157,12 +177,14 @@ class MulWarp16(val b: GlobalConfig, lane: Int, inBW: Int, outBW: Int, inW: Int, } is(sWrite) { - for (g <- 0 until 4) { MulWarp16Tie.takeWrite(io.bankWrite(g), store.io.bankWrite(g)) } - when(store.io.done) { state := sDone } + for (g <- 0 until 4) { + VecMat16Tie.takeWrite(io.bankWrite(g), store.io.bankWrite(g)) + } + when(store.io.done)(state := sDone) } is(sDone) { - when(io.cmdResp.fire) { state := sIdle } + when(io.cmdResp.fire)(state := sIdle) } } } diff --git a/examples/balls/vector/arch/src/main/scala/MulWarp16AccLoad.scala b/examples/balls/vector/arch/src/main/scala/VecMat16AccLoad.scala similarity index 85% rename from examples/balls/vector/arch/src/main/scala/MulWarp16AccLoad.scala rename to examples/balls/vector/arch/src/main/scala/VecMat16AccLoad.scala index 8ee887525..21494c1fd 100644 --- a/examples/balls/vector/arch/src/main/scala/MulWarp16AccLoad.scala +++ b/examples/balls/vector/arch/src/main/scala/VecMat16AccLoad.scala @@ -5,7 +5,8 @@ import chisel3.util._ import framework.balldomain.blink.BankRead import framework.top.GlobalConfig -class MulWarp16AccLoad(val b: GlobalConfig, lane: Int, accWidth: Int) extends Module { +class VecMat16AccLoad(val b: GlobalConfig, lane: Int, accWidth: Int) extends Module { + val io = IO(new Bundle { val start = Input(Bool()) val wrBank = Input(UInt(log2Up(b.memDomain.bankNum).W)) @@ -17,7 +18,7 @@ class MulWarp16AccLoad(val b: GlobalConfig, lane: Int, accWidth: Int) extends Mo }) val sIdle :: sReq :: sResp :: sFinish :: Nil = Enum(4) - val state = RegInit(sIdle) + val state = RegInit(sIdle) val loadRow = RegInit(0.U(log2Ceil(lane + 1).W)) val loadPhase = RegInit(0.U(1.W)) @@ -63,7 +64,11 @@ class MulWarp16AccLoad(val b: GlobalConfig, lane: Int, accWidth: Int) extends Mo reqDone(p) := true.B } } - when((0 until 2).map(p => reqDone(p) || io.bankRead(p).io.req.fire).reduce(_ && _)) { + when( + (0 until 2) + .map(p => reqDone(p) || io.bankRead(p).io.req.fire) + .reduce(_ && _) + ) { respDone := clr2() state := sResp } @@ -76,7 +81,8 @@ class MulWarp16AccLoad(val b: GlobalConfig, lane: Int, accWidth: Int) extends Mo io.bankRead(p).io.resp.ready := !respDone(p) when(io.bankRead(p).io.resp.fire) { val g = loadGroup(p) - val elems = io.bankRead(p).io.resp.bits.data.asTypeOf(Vec(4, UInt(accWidth.W))) + val elems = + io.bankRead(p).io.resp.bits.data.asTypeOf(Vec(4, UInt(accWidth.W))) val row = loadRow(log2Ceil(lane) - 1, 0) for (i <- 0 until 4) { acc(row)(g * 4.U + i.U) := elems(i) @@ -84,7 +90,11 @@ class MulWarp16AccLoad(val b: GlobalConfig, lane: Int, accWidth: Int) extends Mo respDone(p) := true.B } } - when((0 until 2).map(p => respDone(p) || io.bankRead(p).io.resp.fire).reduce(_ && _)) { + when( + (0 until 2) + .map(p => respDone(p) || io.bankRead(p).io.resp.fire) + .reduce(_ && _) + ) { when(loadPhase === 0.U) { loadPhase := 1.U reqDone := clr2() diff --git a/examples/balls/vector/arch/src/main/scala/MulWarp16Store.scala b/examples/balls/vector/arch/src/main/scala/VecMat16Store.scala similarity index 81% rename from examples/balls/vector/arch/src/main/scala/MulWarp16Store.scala rename to examples/balls/vector/arch/src/main/scala/VecMat16Store.scala index 938374c42..d044359ce 100644 --- a/examples/balls/vector/arch/src/main/scala/MulWarp16Store.scala +++ b/examples/balls/vector/arch/src/main/scala/VecMat16Store.scala @@ -5,7 +5,8 @@ import chisel3.util._ import framework.balldomain.blink.BankWrite import framework.top.GlobalConfig -class MulWarp16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Module { +class VecMat16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Module { + val io = IO(new Bundle { val start = Input(Bool()) val wrBank = Input(UInt(log2Up(b.memDomain.bankNum).W)) @@ -17,7 +18,7 @@ class MulWarp16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Modu }) val sIdle :: sReq :: sResp :: Nil = Enum(3) - val state = RegInit(sIdle) + val state = RegInit(sIdle) val writeRow = RegInit(0.U(log2Ceil(lane + 1).W)) val writeIssued = RegInit(VecInit(Seq.fill(4)(false.B))) @@ -28,6 +29,7 @@ class MulWarp16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Modu val elems = (0 until 4).map(i => io.acc(row)(group * 4 + i)) Cat(elems.reverse) } + def clr4() = VecInit(Seq.fill(4)(false.B)) io.done := false.B @@ -41,7 +43,9 @@ class MulWarp16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Modu io.bankWrite(i).io.req.valid := false.B io.bankWrite(i).io.req.bits.addr := 0.U io.bankWrite(i).io.req.bits.data := 0.U - io.bankWrite(i).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(false.B)) + io.bankWrite(i).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(false.B) + ) io.bankWrite(i).io.resp.ready := false.B } @@ -62,12 +66,18 @@ class MulWarp16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Modu io.bankWrite(g).io.req.valid := !writeIssued(g) io.bankWrite(g).io.req.bits.addr := writeRow io.bankWrite(g).io.req.bits.data := writeData(g) - io.bankWrite(g).io.req.bits.mask := VecInit(Seq.fill(b.memDomain.bankMaskLen)(true.B)) + io.bankWrite(g).io.req.bits.mask := VecInit( + Seq.fill(b.memDomain.bankMaskLen)(true.B) + ) when(io.bankWrite(g).io.req.fire) { writeIssued(g) := true.B } } - when((0 until 4).map(g => writeIssued(g) || io.bankWrite(g).io.req.fire).reduce(_ && _)) { + when( + (0 until 4) + .map(g => writeIssued(g) || io.bankWrite(g).io.req.fire) + .reduce(_ && _) + ) { writeAcked := clr4() state := sResp } @@ -82,7 +92,11 @@ class MulWarp16Store(val b: GlobalConfig, lane: Int, accWidth: Int) extends Modu writeAcked(g) := true.B } } - when((0 until 4).map(g => writeAcked(g) || io.bankWrite(g).io.resp.fire).reduce(_ && _)) { + when( + (0 until 4) + .map(g => writeAcked(g) || io.bankWrite(g).io.resp.fire) + .reduce(_ && _) + ) { when(writeRow === (lane - 1).U) { io.done := true.B state := sIdle diff --git a/examples/balls/vector/arch/src/main/scala/MulWarp16Tie.scala b/examples/balls/vector/arch/src/main/scala/VecMat16Tie.scala similarity index 57% rename from examples/balls/vector/arch/src/main/scala/MulWarp16Tie.scala rename to examples/balls/vector/arch/src/main/scala/VecMat16Tie.scala index 0ce1ff9d8..a4d0f4dae 100644 --- a/examples/balls/vector/arch/src/main/scala/MulWarp16Tie.scala +++ b/examples/balls/vector/arch/src/main/scala/VecMat16Tie.scala @@ -5,7 +5,8 @@ import chisel3.util._ import framework.balldomain.blink.{BankRead, BankWrite} import framework.top.GlobalConfig -object MulWarp16Tie { +object VecMat16Tie { + def offRead(ports: Vec[BankRead], robId: UInt): Unit = { for (p <- ports) { p.rob_id := robId @@ -19,10 +20,10 @@ object MulWarp16Tie { } def offWrite( - ports: Vec[BankWrite], - robId: UInt, - wrBank: UInt, - b: GlobalConfig + ports: Vec[BankWrite], + robId: UInt, + wrBank: UInt, + b: GlobalConfig ): Unit = { for ((p, i) <- ports.zipWithIndex) { p.rob_id := robId @@ -38,22 +39,23 @@ object MulWarp16Tie { } def takeRead(parent: BankRead, child: BankRead): Unit = { - parent.rob_id := child.rob_id - parent.ball_id := child.ball_id - parent.bank_id := child.bank_id - parent.group_id := child.group_id - parent.io.req.valid := child.io.req.valid - parent.io.req.bits := child.io.req.bits - parent.io.resp.ready := child.io.resp.ready + parent.rob_id := child.rob_id + parent.ball_id := child.ball_id + parent.bank_id := child.bank_id + parent.group_id := child.group_id + parent.io.req.valid := child.io.req.valid + parent.io.req.bits := child.io.req.bits + parent.io.resp.ready := child.io.resp.ready } def takeWrite(parent: BankWrite, child: BankWrite): Unit = { - parent.rob_id := child.rob_id - parent.ball_id := child.ball_id - parent.bank_id := child.bank_id - parent.group_id := child.group_id - parent.io.req.valid := child.io.req.valid - parent.io.req.bits := child.io.req.bits - parent.io.resp.ready := child.io.resp.ready + parent.rob_id := child.rob_id + parent.ball_id := child.ball_id + parent.bank_id := child.bank_id + parent.group_id := child.group_id + parent.io.req.valid := child.io.req.valid + parent.io.req.bits := child.io.req.bits + parent.io.resp.ready := child.io.resp.ready } + } diff --git a/examples/balls/vector/arch/src/main/scala/VecStoreUnit.scala b/examples/balls/vector/arch/src/main/scala/VecStoreUnit.scala index b1ba3c5ca..7d31eaa33 100644 --- a/examples/balls/vector/arch/src/main/scala/VecStoreUnit.scala +++ b/examples/balls/vector/arch/src/main/scala/VecStoreUnit.scala @@ -35,9 +35,13 @@ class VecStoreUnit(val b: GlobalConfig) extends Module { val accWidth = config.outputWidth // Get bandwidth from config (use first VecBall mapping) - val ballMapping = b.ballDomain.ballIdMappings.find(_.ballName == "VecBall") - .getOrElse(throw new IllegalArgumentException("VecBall not found in config")) - val outBW = ballMapping.outBW + val ballMapping = b.ballDomain.ballIdMappings + .find(_.ballName == "VecBall") + .getOrElse( + throw new IllegalArgumentException("VecBall not found in config") + ) + + val outBW = ballMapping.outBW @public val io = IO(new Bundle { @@ -55,7 +59,9 @@ class VecStoreUnit(val b: GlobalConfig) extends Module { val idle :: busy :: Nil = Enum(2) val state = RegInit(idle) - val writeQueues = VecInit(Seq.fill(outBW)(Module(new Queue(new BankWriteEntry(b), 16)).io)) + val writeQueues = VecInit( + Seq.fill(outBW)(Module(new Queue(new BankWriteEntry(b), 16)).io) + ) // ----------------------------------------------------------------------------- // Set registers when Ctrl instruction arrives diff --git a/examples/balls/vector/arch/src/main/scala/VecUnit.scala b/examples/balls/vector/arch/src/main/scala/VecUnit.scala index b3d0b4002..ebdb188ff 100644 --- a/examples/balls/vector/arch/src/main/scala/VecUnit.scala +++ b/examples/balls/vector/arch/src/main/scala/VecUnit.scala @@ -11,22 +11,25 @@ import framework.top.GlobalConfig @instantiable class VecUnit(val b: GlobalConfig) extends Module { - val cfg = VectorBallParam(b) + val cfg = VectorBallParam(b) val lane = cfg.lane val inW = cfg.inputWidth val accW = cfg.outputWidth - require(lane == 16, "VecUnit mul_warp16 requires lane=16") - require(inW == 8, "VecUnit mul_warp16 requires int8 inputs") - require(accW == 32, "VecUnit mul_warp16 requires int32 accumulators") + require(lane == 16, "VecUnit vecmat16 requires lane=16") + require(inW == 8, "VecUnit vecmat16 requires int8 inputs") + require(accW == 32, "VecUnit vecmat16 requires int32 accumulators") val map = b.ballDomain.ballIdMappings .find(_.ballName == "VecBall") - .getOrElse(throw new IllegalArgumentException("VecBall not found in config")) + .getOrElse( + throw new IllegalArgumentException("VecBall not found in config") + ) + val inBW = map.inBW val outBW = map.outBW - require(inBW >= 2, "VecUnit mul_warp16 requires two read ports") - require(outBW >= 4, "VecUnit mul_warp16 requires four write ports") + require(inBW >= 2, "VecUnit vecmat16 requires two read ports") + require(outBW >= 4, "VecUnit vecmat16 requires four write ports") @public val io = IO(new Bundle { @@ -37,7 +40,7 @@ class VecUnit(val b: GlobalConfig) extends Module { val status = new BallStatus }) - val core = Module(new MulWarp16(b, lane, inBW, outBW, inW, accW)) + val core = Module(new VecMat16(b, lane, inBW, outBW, inW, accW)) core.io.cmdReq <> io.cmdReq core.io.cmdResp <> io.cmdResp for (i <- 0 until inBW) { core.io.bankRead(i) <> io.bankRead(i) } diff --git a/examples/balls/vector/arch/src/main/scala/bond/vvv.scala b/examples/balls/vector/arch/src/main/scala/bond/vvv.scala index 569f7fd7f..d63bb25ff 100644 --- a/examples/balls/vector/arch/src/main/scala/bond/vvv.scala +++ b/examples/balls/vector/arch/src/main/scala/bond/vvv.scala @@ -12,7 +12,11 @@ import examples.balls.vector.configs.VectorBallParam import chisel3.experimental.hierarchy.{instantiable, public} @instantiable -class VVV(val config: VectorBallParam, val inputWidth: Int, val outputWidth: Int) extends Bundle { +class VVV( + val config: VectorBallParam, + val inputWidth: Int, + val outputWidth: Int) + extends Bundle { val lane = config.lane @public diff --git a/examples/balls/vector/arch/src/main/scala/thread/CasThread.scala b/examples/balls/vector/arch/src/main/scala/thread/CasThread.scala index 87e9b0f2d..d2c4a6926 100644 --- a/examples/balls/vector/arch/src/main/scala/thread/CasThread.scala +++ b/examples/balls/vector/arch/src/main/scala/thread/CasThread.scala @@ -8,7 +8,10 @@ import examples.balls.vector.op.CascadeOp class CasThread(config: VectorBallParam) extends BaseThread(config, "cascade") { val cascadeOp = Module(new CascadeOp(this.config)) - val vvvBond = IO(new VVV(this.config, this.config.outputWidth, this.config.outputWidth)) + + val vvvBond = IO( + new VVV(this.config, this.config.outputWidth, this.config.outputWidth) + ) // Connect CascadeOp and VVVBond cascadeOp.io.in <> vvvBond.in diff --git a/examples/balls/vector/arch/src/main/scala/thread/MulThread.scala b/examples/balls/vector/arch/src/main/scala/thread/MulThread.scala index 8e0990f2f..3ef6dd26f 100644 --- a/examples/balls/vector/arch/src/main/scala/thread/MulThread.scala +++ b/examples/balls/vector/arch/src/main/scala/thread/MulThread.scala @@ -7,8 +7,11 @@ import examples.balls.vector.bond.VVV import examples.balls.vector.op.MulOp class MulThread(config: VectorBallParam) extends BaseThread(config, "mul") { - val mulOp = Module(new MulOp(this.config)) - val vvvBond = IO(new VVV(this.config, this.config.inputWidth, this.config.outputWidth)) + val mulOp = Module(new MulOp(this.config)) + + val vvvBond = IO( + new VVV(this.config, this.config.inputWidth, this.config.outputWidth) + ) // Connect MulOp and VVVBond mulOp.io.in <> vvvBond.in diff --git a/examples/balls/vector/arch/src/main/scala/warp/MeshWarp.scala b/examples/balls/vector/arch/src/main/scala/warp/MeshWarp.scala index e06b0cf2e..13ebaf4d9 100644 --- a/examples/balls/vector/arch/src/main/scala/warp/MeshWarp.scala +++ b/examples/balls/vector/arch/src/main/scala/warp/MeshWarp.scala @@ -45,7 +45,9 @@ class MeshWarp(val config: VectorBallParam) extends Module { // Connect cascade thread's second input and output ready signal if (i == 0) { - casBond.in.bits.in2 := VecInit(Seq.fill(config.lane)(0.U(config.outputWidth.W))) + casBond.in.bits.in2 := VecInit( + Seq.fill(config.lane)(0.U(config.outputWidth.W)) + ) // First cascade thread's valid is determined by mulBond's output valid casBond.in.valid := mulBond.out.valid // First cascade thread's output ready is connected to next cascade thread's input ready @@ -56,7 +58,9 @@ class MeshWarp(val config: VectorBallParam) extends Module { // Directly connect output to input casBond.in.bits.in2 := casThreads(i - 1).vvvBond.out.bits.out // casBond's valid is jointly determined by previous casBond's output valid and current mulBond's output valid - casBond.in.valid := casThreads(i - 1).vvvBond.out.valid || mulBond.out.valid + casBond.in.valid := casThreads( + i - 1 + ).vvvBond.out.valid || mulBond.out.valid // Middle cascade thread's output ready is connected to next cascade thread's input ready if (i < config.numCasThreads - 1) { casBond.out.ready := casThreads(i + 1).vvvBond.in.ready @@ -71,8 +75,12 @@ class MeshWarp(val config: VectorBallParam) extends Module { io.in.ready := mulBond.in.ready }.otherwise { mulBond.in.valid := false.B - mulBond.in.bits.in1 := VecInit(Seq.fill(config.lane)(0.U(config.inputWidth.W))) - mulBond.in.bits.in2 := VecInit(Seq.fill(config.lane)(0.U(config.inputWidth.W))) + mulBond.in.bits.in1 := VecInit( + Seq.fill(config.lane)(0.U(config.inputWidth.W)) + ) + mulBond.in.bits.in2 := VecInit( + Seq.fill(config.lane)(0.U(config.inputWidth.W)) + ) } } diff --git a/examples/balls/vector/compiler/src/Backend/include/IntrinsicsRISCVVecBall.td b/examples/balls/vector/compiler/src/Backend/include/IntrinsicsRISCVVecBall.td index 3a906f0bd..5d4fe7537 100644 --- a/examples/balls/vector/compiler/src/Backend/include/IntrinsicsRISCVVecBall.td +++ b/examples/balls/vector/compiler/src/Backend/include/IntrinsicsRISCVVecBall.td @@ -1,4 +1,4 @@ //===- IntrinsicsRISCVVecBall.td - VecBall intrinsics --------------------===// let TargetPrefix = "riscv" in -def int_riscv_bb_mul_warp16 : Intrinsic<[], [llvm_i64_ty, llvm_i64_ty], []>; +def int_riscv_bb_vecmat16 : Intrinsic<[], [llvm_i64_ty, llvm_i64_ty], []>; diff --git a/examples/balls/vector/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp b/examples/balls/vector/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp index 60735d142..42b048874 100644 --- a/examples/balls/vector/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp +++ b/examples/balls/vector/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp @@ -10,20 +10,20 @@ using namespace mlir; using namespace ::buddy::buckyball; namespace mlir::buddy { -void populateVectorAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateVecBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); } // namespace mlir::buddy namespace { -class BankMulWarp16Pattern : public OpRewritePattern { +class BankVecMat16Pattern : public OpRewritePattern { public: - using OpRewritePattern::OpRewritePattern; + using OpRewritePattern::OpRewritePattern; - LogicalResult matchAndRewrite(BankMulWarp16Op op, + LogicalResult matchAndRewrite(BankVecMat16Op op, PatternRewriter &rewriter) const override { - rewriter.create(op.getLoc(), op.getOp1Bank(), op.getOp2Bank(), - op.getWrBank(), op.getIter(), op.getMode()); + rewriter.create(op.getLoc(), op.getOp1Bank(), op.getOp2Bank(), + op.getWrBank(), op.getIter(), op.getMode()); rewriter.replaceOp(op, op.getWrBank()); return success(); } @@ -31,8 +31,8 @@ class BankMulWarp16Pattern : public OpRewritePattern { } // namespace -void mlir::buddy::populateVectorAssignPhysicalBankPatterns( +void mlir::buddy::populateVecBallAssignPhysicalBankPatterns( RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { (void)state; - patterns.add(patterns.getContext()); + patterns.add(patterns.getContext()); } diff --git a/examples/balls/vector/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/balls/vector/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index cd3ad2e39..8cfe5cd1f 100644 --- a/examples/balls/vector/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/balls/vector/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -4,50 +4,44 @@ #include "Buckyball/BuckyballOps.h" #include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" +#include "Target/BuckyballTargetRegistry.h" using namespace mlir; using namespace buddy::buckyball; using namespace buddy::buckyball::legalize; namespace { -struct MulWarp16Lowering : public ConvertOpToLLVMPattern { - MulWarp16Lowering(LLVMTypeConverter &converter, bool stable) - : ConvertOpToLLVMPattern(converter), stable(stable) {} +struct VecMat16Lowering : public ConvertOpToLLVMPattern { + using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern; LogicalResult - matchAndRewrite(MulWarp16Op op, OpAdaptor adaptor, + matchAndRewrite(VecMat16Op op, OpAdaptor adaptor, ConversionPatternRewriter &rewriter) const override { + buckyball_target::requireBuckyballBall("VecBall"); Location loc = op.getLoc(); Value rs1 = packRs1BanksIter(rewriter, loc, adaptor.getOp1BankId(), adaptor.getOp2BankId(), adaptor.getWrBankId(), adaptor.getIter()); - if (stable) { - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getMode()); - return success(); - } - rewriter.replaceOpWithNewOp(op, rs1, adaptor.getMode(), - rewriter.getI32IntegerAttr(64)); + rewriter.replaceOpWithNewOp( + op, rs1, adaptor.getMode(), + rewriter.getI32IntegerAttr( + buckyball_target::getBuckyballFunct7("VECMAT16"))); return success(); } - -private: - bool stable = false; }; } // namespace namespace mlir::buddy::buckyball { -void populateVectorLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable) { - patterns.add(converter, stable); +void populateVecBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, + RewritePatternSet &patterns, + bool stable, int64_t, bool) { + (void)stable; + patterns.add(converter); } -void configureVectorLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable) { - if (stable) - target.addLegalOp(); - else - target.addIllegalOp(); - target.addIllegalOp(); +void configureVecBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable) { + (void)stable; + target.addIllegalOp(); } } // namespace mlir::buddy::buckyball diff --git a/examples/balls/vector/compiler/src/Dialect/Buckyball/VecBall.td b/examples/balls/vector/compiler/src/Dialect/Buckyball/VecBall.td index ceea9cfc5..b3fdec455 100644 --- a/examples/balls/vector/compiler/src/Dialect/Buckyball/VecBall.td +++ b/examples/balls/vector/compiler/src/Dialect/Buckyball/VecBall.td @@ -7,18 +7,17 @@ #ifndef VECBALL_OPS #define VECBALL_OPS -include "BuckyballBase.td" -include "BuckyballIntrinsics.td" +include "Buckyball.td" //===----------------------------------------------------------------------===// // VecBall operations //===----------------------------------------------------------------------===// -def MulWarp16Op : Buckyball_Op<"mul_warp16"> { +def VecMat16Op : Buckyball_Op<"vecmat16"> { let summary = "Warp-16 matrix multiplication"; let description = [{ Performs matrix multiplication using 16x16 matrix array. - Corresponds to bb_mul_warp16 instruction (funct7=64). + Corresponds to bb_vecmat16 instruction (funct7 from Core ballISA). Belongs to VecBall (bid=0). }]; let arguments = (ins I64:$op1BankId, I64:$op2BankId, @@ -28,7 +27,7 @@ def MulWarp16Op : Buckyball_Op<"mul_warp16"> { "type($op1BankId)"; } -def BankMulWarp16Op : Buckyball_Op<"bank_mul_warp16"> { +def BankVecMat16Op : Buckyball_Op<"bank_vecmat16"> { let summary = "Warp-16 matmul using virtual bank handles"; let arguments = (ins I64:$op1Bank, I64:$op2Bank, I64:$wrBank, I64:$iter, I64:$mode); @@ -37,7 +36,4 @@ def BankMulWarp16Op : Buckyball_Op<"bank_mul_warp16"> { "$op1Bank $op2Bank $wrBank $iter $mode attr-dict `:` type($op1Bank) type($op2Bank) type($wrBank) type($iter) type($mode)"; } -def MulWarp16IntrOp : Buckyball_IntrOpBase<"mul.warp16">, - Arguments<(ins LLVM_Type, LLVM_Type)>; - #endif // VECBALL_OPS diff --git a/examples/balls/vector/configs/VectorBallParam.scala b/examples/balls/vector/configs/VectorBallParam.scala index 71ef52f8d..0eaf9d325 100644 --- a/examples/balls/vector/configs/VectorBallParam.scala +++ b/examples/balls/vector/configs/VectorBallParam.scala @@ -10,8 +10,7 @@ case class VectorBallParam( lane: Int, outputWidth: Int, numMulThreads: Int, - numCasThreads: Int -) + numCasThreads: Int) object VectorBallParam { implicit val rw: ReadWriter[VectorBallParam] = macroRW @@ -29,4 +28,5 @@ object VectorBallParam { numCasThreads = BallParamLoader.int(tbl, "numCasThreads") ) } + } diff --git a/examples/balls/vector/emu/src/64_mul_warp16.rs b/examples/balls/vector/emu/src/64_vecmat16.rs similarity index 79% rename from examples/balls/vector/emu/src/64_mul_warp16.rs rename to examples/balls/vector/emu/src/64_vecmat16.rs index 9d5b60ea2..207439cb7 100644 --- a/examples/balls/vector/emu/src/64_mul_warp16.rs +++ b/examples/balls/vector/emu/src/64_vecmat16.rs @@ -1,18 +1,16 @@ -//===- 64_mul_warp16.rs - MUL_WARP16 instruction ---------------------------===// +//===- 64_vecmat16.rs - MUL_WARP16 instruction ---------------------------===// use super::super::bank::bank_num; use super::bank_matrix::{read_i32_nn_groups, read_i8_k_rows, write_i32_nn_groups}; use super::decode::{pbank, pbank_group, rs1_b0, rs1_b1, rs1_b2, rs1_iter}; -use super::instruction::{ExecContext, Instruction}; +use super::instruction::{BallInstruction, ExecContext}; const WARP_M: usize = 16; const WARP_N: usize = 16; -pub struct MulWarp16; - -impl Instruction for MulWarp16 { - const FUNCT: u32 = 64; +pub struct VecMat16; +impl BallInstruction for VecMat16 { fn exec(xs1: u64, xs2: u64, ctx: &mut ExecContext) -> u64 { let op1 = rs1_b0(xs1); let op2 = rs1_b1(xs1); @@ -21,14 +19,14 @@ impl Instruction for MulWarp16 { let _ = xs2; if op1 >= bank_num() as u64 || op2 >= bank_num() as u64 || wr >= bank_num() as u64 { - panic!("mul_warp16: invalid bank_id"); + panic!("vecmat16: invalid bank_id"); } let c1 = ctx.cfgs[op1 as usize].cols; let c2 = ctx.cfgs[op2 as usize].cols; let cw = ctx.cfgs[wr as usize].cols; if c1 != 1 || c2 != 1 || cw != 4 { - panic!("mul_warp16: unsupported bank layout op1_cols={c1} op2_cols={c2} wr_cols={cw}"); + panic!("vecmat16: unsupported bank layout op1_cols={c1} op2_cols={c2} wr_cols={cw}"); } let p1 = pbank(ctx.bank_map, op1); @@ -39,12 +37,12 @@ impl Instruction for MulWarp16 { let kin = iter as usize; if kin == 0 { - panic!("mul_warp16: iter must be > 0"); + panic!("vecmat16: iter must be > 0"); } let need = kin * WARP_N; if need > ctx.banks[p1].len() || need > ctx.banks[p2].len() { - panic!("mul_warp16: iter too large for bank"); + panic!("vecmat16: iter too large for bank"); } let a_t = read_i8_k_rows(&ctx.banks, p1, kin, WARP_N); diff --git a/examples/balls/vector/emu/src/lib.rs b/examples/balls/vector/emu/src/lib.rs index 6584d21a4..89313ebba 100644 --- a/examples/balls/vector/emu/src/lib.rs +++ b/examples/balls/vector/emu/src/lib.rs @@ -1,9 +1,9 @@ pub(crate) use crate::inst::{bank_matrix, decode, instruction}; -use crate::inst::instruction::{ExecContext, Instruction}; +use crate::inst::instruction::{BallInstruction, ExecContext}; -#[path = "64_mul_warp16.rs"] -mod f64_mul_warp16; +#[path = "64_vecmat16.rs"] +mod f64_vecmat16; const BALL_CLASS: &str = "examples.balls.vector.VecBall"; @@ -17,11 +17,9 @@ pub fn execute_known( if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::exec(xs1, xs2, ctx), - ), - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("VECMAT16") => Some(f64_vecmat16::VecMat16::exec(xs1, xs2, ctx)), + Some(_) | None => None, } } @@ -29,10 +27,8 @@ pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> O if ball_class != BALL_CLASS { return None; } - match funct { - ::FUNCT => Some( - ::latency(xs1, xs2), - ), - _ => None, + match crate::config::ball_domain::mnemonic_for_funct(funct).as_deref() { + Some("VECMAT16") => Some(f64_vecmat16::VecMat16::latency(xs1, xs2)), + Some(_) | None => None, } } diff --git a/examples/balls/vector/workloads/ctests/CMakeLists.txt b/examples/balls/vector/workloads/ctests/CMakeLists.txt index 17220341e..99477a3f5 100644 --- a/examples/balls/vector/workloads/ctests/CMakeLists.txt +++ b/examples/balls/vector/workloads/ctests/CMakeLists.txt @@ -1,3 +1,3 @@ add_buckyball_ctests( - vector_mul_warp16_test.c + vector_vecmat16_test.c ) diff --git a/examples/balls/vector/workloads/ctests/vector_mul_warp16_test.c b/examples/balls/vector/workloads/ctests/vector_vecmat16_test.c similarity index 70% rename from examples/balls/vector/workloads/ctests/vector_mul_warp16_test.c rename to examples/balls/vector/workloads/ctests/vector_vecmat16_test.c index 29526163c..0ea21edb0 100644 --- a/examples/balls/vector/workloads/ctests/vector_mul_warp16_test.c +++ b/examples/balls/vector/workloads/ctests/vector_vecmat16_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #include @@ -11,7 +12,7 @@ static elem_t input_matrix_b[DIM * DIM] __attribute__((aligned(64))); static result_t output_matrix[DIM * DIM] __attribute__((aligned(64))); static result_t expected_matrix[DIM * DIM] __attribute__((aligned(64))); -static void hw_vector_mul_warp16(elem_t *a, elem_t *b, result_t *c, int size) { +static void hw_vector_vecmat16(elem_t *a, elem_t *b, result_t *c, int size) { uint32_t op1_bank_id = 0; uint32_t op2_bank_id = 1; int acc_bank_id = 2; @@ -23,9 +24,12 @@ static void hw_vector_mul_warp16(elem_t *a, elem_t *b, result_t *c, int size) { bb_mvin((uintptr_t)a, op1_bank_id, DIM, 1); bb_mvin((uintptr_t)b, op2_bank_id, DIM, 1); - bb_mul_warp16(op1_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(op1_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size << 2, 1); bb_fence(); + bb_mem_release(op1_bank_id); + bb_mem_release(op2_bank_id); + bb_mem_release(acc_bank_id); } static int run_boot_init_test(void) { @@ -33,14 +37,14 @@ static int run_boot_init_test(void) { init_sequence_matrix(input_matrix_b, DIM, DIM); cpu_matmul(input_matrix_a, input_matrix_b, expected_matrix, DIM, DIM, DIM); - hw_vector_mul_warp16(input_matrix_a, input_matrix_b, output_matrix, DIM); + hw_vector_vecmat16(input_matrix_a, input_matrix_b, output_matrix, DIM); if (compare_u32_matrices(output_matrix, expected_matrix, DIM, DIM)) { - printf("vector_mul_warp16_test matmul PASSED\n"); + printf("vector_vecmat16_test matmul PASSED\n"); return 1; } - printf("vector_mul_warp16_test matmul FAILED\n"); + printf("vector_vecmat16_test matmul FAILED\n"); return 0; } @@ -52,11 +56,11 @@ int main() { int passed = run_boot_init_test(); if (passed) { - printf("vector_mul_warp16_test PASSED\n"); + printf("vector_vecmat16_test PASSED\n"); return 0; } - printf("vector_mul_warp16_test FAILED\n"); + printf("vector_vecmat16_test FAILED\n"); return 1; #ifdef MULTICORE diff --git a/examples/balls/vector/workloads/isa/vecmat16.h b/examples/balls/vector/workloads/isa/vecmat16.h new file mode 100644 index 000000000..8c802d967 --- /dev/null +++ b/examples/balls/vector/workloads/isa/vecmat16.h @@ -0,0 +1,12 @@ +#ifndef _BB_VECMAT16_H_ +#define _BB_VECMAT16_H_ + +#include +#include + +#define bb_vecmat16(op1_bank_id, op2_bank_id, wr_bank_id, iter, mode) \ + BUCKYBALL_INSTRUCTION_R_R((BB_BANK0(op1_bank_id) | BB_BANK1(op2_bank_id) | \ + BB_BANK2(wr_bank_id) | BB_ITER(iter)), \ + (FIELD(mode, 0, 63)), BB_FUNC7(VECMAT16)) + +#endif // _BB_VECMAT16_H_ diff --git a/examples/chips/cmake/BuckyballCtestLimits.cmake b/examples/chips/cmake/BuckyballCtestLimits.cmake deleted file mode 100644 index c1f2ebd09..000000000 --- a/examples/chips/cmake/BuckyballCtestLimits.cmake +++ /dev/null @@ -1,32 +0,0 @@ -# Shared by all chips (and ball ctests pulled in via chip workloads). -set(BUCKYBALL_CTEST_MAX_LINES 100) - -function(buckyball_enforce_ctest_line_limit SOURCE_FILE) - if(IS_ABSOLUTE "${SOURCE_FILE}") - set(_src "${SOURCE_FILE}") - else() - set(_src "${CMAKE_CURRENT_SOURCE_DIR}/${SOURCE_FILE}") - endif() - if(NOT EXISTS "${_src}") - message(FATAL_ERROR "ctest source does not exist: ${_src}") - endif() - - file(READ "${_src}" _content) - if(_content STREQUAL "") - set(_n 0) - else() - string(REGEX MATCHALL "\n" _newlines "${_content}") - list(LENGTH _newlines _n) - string(REGEX MATCH "[^\n]$" _no_trailing_nl "${_content}") - if(_no_trailing_nl) - math(EXPR _n "${_n} + 1") - endif() - endif() - - if(_n GREATER BUCKYBALL_CTEST_MAX_LINES) - message(FATAL_ERROR - "ctest exceeds ${BUCKYBALL_CTEST_MAX_LINES} lines: ${_src} (${_n} lines). " - "Split it into smaller focused ctests. " - "Do not evade this check by moving functional code into headers.") - endif() -endfunction() diff --git a/examples/chips/goban/arch/src/main/scala/CustomConfigs.scala b/examples/chips/goban/arch/src/main/scala/CustomConfigs.scala index f50fb248f..518c4c5bf 100644 --- a/examples/chips/goban/arch/src/main/scala/CustomConfigs.scala +++ b/examples/chips/goban/arch/src/main/scala/CustomConfigs.scala @@ -5,21 +5,24 @@ import org.chipsalliance.cde.config.Config import freechips.rocketchip.tile.MaxHartIdBits import framework.system.tile.WithBuckyballTiles -class WithGobanHiddenHartIdBits(nTiles: Int, nCoresPerTile: Int, hiddenHartBase: Int) - extends Config((site, here, up) => { - case MaxHartIdBits => log2Ceil(hiddenHartBase + nTiles * (nCoresPerTile - 1)) +class WithGobanHiddenHartIdBits( + nTiles: Int, + nCoresPerTile: Int, + hiddenHartBase: Int) + extends Config((site, here, up) => { case MaxHartIdBits => + log2Ceil(hiddenHartBase + nTiles * (nCoresPerTile - 1)) }) class WithGobanHartIdBits(nTiles: Int, nCoresPerTile: Int) - extends Config((site, here, up) => { - case MaxHartIdBits => log2Ceil(nTiles * nCoresPerTile) + extends Config((site, here, up) => { case MaxHartIdBits => + log2Ceil(nTiles * nCoresPerTile) }) /** 1 BBTile × 2 Buckyball cores. */ class BuckyballGoban2CoreConfig extends Config( new WithGobanHartIdBits(nTiles = 1, nCoresPerTile = 2) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/1t2c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/1t2c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -28,7 +31,7 @@ class BuckyballGoban2CoreConfig class BuckyballGoban4CoreConfig extends Config( new WithGobanHartIdBits(nTiles = 1, nCoresPerTile = 4) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/1t4c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/1t4c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -37,7 +40,7 @@ class BuckyballGoban4CoreConfig class BuckyballGoban8CoreConfig extends Config( new WithGobanHartIdBits(nTiles = 1, nCoresPerTile = 8) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/1t8c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/1t8c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -46,7 +49,7 @@ class BuckyballGoban8CoreConfig class BuckyballGoban16CoreConfig extends Config( new WithGobanHartIdBits(nTiles = 1, nCoresPerTile = 16) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/1t16c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/1t16c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -55,7 +58,7 @@ class BuckyballGoban16CoreConfig class BuckyballGoban32CoreConfig extends Config( new WithGobanHartIdBits(nTiles = 1, nCoresPerTile = 32) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/1t32c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/1t32c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -64,7 +67,7 @@ class BuckyballGoban32CoreConfig class BuckyballGoban64CoreConfig extends Config( new WithGobanHartIdBits(nTiles = 1, nCoresPerTile = 64) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/1t64c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/1t64c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -73,7 +76,7 @@ class BuckyballGoban64CoreConfig // this config is incoherent class BuckyballGoban4Tile8CoreConfig extends Config( - new WithBuckyballTiles("../examples/chips/goban/configs/4t16c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/4t16c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -82,7 +85,7 @@ class BuckyballGoban4Tile8CoreConfig // this config is incoherent class BuckyballGoban4Tile16CoreConfig extends Config( - new WithBuckyballTiles("../examples/chips/goban/configs/4t16c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/4t16c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -91,7 +94,7 @@ class BuckyballGoban4Tile16CoreConfig // this config is incoherent class BuckyballGoban8Tile8CoreConfig extends Config( - new WithBuckyballTiles("../examples/chips/goban/configs/8t8c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/8t8c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -99,7 +102,7 @@ class BuckyballGoban8Tile8CoreConfig /** 24 BBTiles × 16 Buckyball cores each = 384 total. */ class BuckyballGoban24Tile16CoreConfig extends Config( - new WithBuckyballTiles("../examples/chips/goban/configs/24t16c.toml") ++ + new WithBuckyballTiles("../examples/chips/goban/configs/generated/24t16c.pb") ++ new chipyard.config.WithSystemBusWidth(256) ++ new sims.base.BuckyballBaseConfig ) @@ -107,8 +110,15 @@ class BuckyballGoban24Tile16CoreConfig /** 2 BBTiles × 4 Rocket cores each = 8 harts; core 0 in each tile has Buckyball. */ class BuckyballGoban2Tile4CoreConfig extends Config( - new WithGobanHiddenHartIdBits(nTiles = 2, nCoresPerTile = 4, hiddenHartBase = 2) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/2t4c-private.toml", hiddenHartBase = Some(2)) ++ + new WithGobanHiddenHartIdBits( + nTiles = 2, + nCoresPerTile = 4, + hiddenHartBase = 2 + ) ++ + new WithBuckyballTiles( + "../examples/chips/goban/configs/generated/2t4c-private.pb", + hiddenHartBase = Some(2) + ) ++ new chipyard.config.WithSystemBusWidth(256) ++ new sims.base.BuckyballBaseConfig ) @@ -116,8 +126,15 @@ class BuckyballGoban2Tile4CoreConfig /** 64 BBTiles × 4 Rocket cores each = 256 harts; core 0 in each tile has Buckyball. */ class BuckyballGoban64Tile4CoreConfig extends Config( - new WithGobanHiddenHartIdBits(nTiles = 64, nCoresPerTile = 4, hiddenHartBase = 64) ++ - new WithBuckyballTiles("../examples/chips/goban/configs/64t4c-private.toml", hiddenHartBase = Some(64)) ++ + new WithGobanHiddenHartIdBits( + nTiles = 64, + nCoresPerTile = 4, + hiddenHartBase = 64 + ) ++ + new WithBuckyballTiles( + "../examples/chips/goban/configs/generated/64t4c-private.pb", + hiddenHartBase = Some(64) + ) ++ new chipyard.config.WithSystemBusWidth(256) ++ new sims.base.BuckyballBaseConfig ) @@ -125,7 +142,10 @@ class BuckyballGoban64Tile4CoreConfig /** Rocket-only variant of the default goban topology (no Buckyball). */ class RocketOnlyGobanConfig extends Config( - new WithBuckyballTiles("../examples/chips/goban/configs/goban.toml", withBuckyball = false) ++ + new WithBuckyballTiles( + "../examples/chips/goban/configs/generated/chip.pb", + withBuckyball = false + ) ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) diff --git a/examples/chips/goban/chip.toml b/examples/chips/goban/chip.toml deleted file mode 100644 index 446b89974..000000000 --- a/examples/chips/goban/chip.toml +++ /dev/null @@ -1,10 +0,0 @@ -[chip] -name = "goban" -topology = "configs/goban.toml" -compilerCore = "goban" - -[runtime] -bemu = "emu/Cargo.toml" -bemuCore = "../../cores/goban/emu/Cargo.toml" -bemuTile = "configs/tiles/default.toml" -workloads = "workloads" diff --git a/examples/chips/goban/configs/64t4c-private.toml b/examples/chips/goban/configs/64t4c-private.toml deleted file mode 100644 index 60ea06972..000000000 --- a/examples/chips/goban/configs/64t4c-private.toml +++ /dev/null @@ -1,8 +0,0 @@ -# Goban - 64 tiles x 4 cores each (256 harts total) - -[top] -nTiles = 64 - -[tileTemplate] -include = "tiles/4core-private-1bb.toml" -count = 64 diff --git a/examples/chips/goban/configs/chip.toml b/examples/chips/goban/configs/chip.toml new file mode 100644 index 000000000..19c985795 --- /dev/null +++ b/examples/chips/goban/configs/chip.toml @@ -0,0 +1,6 @@ +[designs] +include = "designs/goban.toml" + +[sims] +verilator = "sims.verilator.BuckyballGoban2CoreVerilatorConfig" +p2e = "sims.p2e.BuckyballGoban2CoreP2EConfig" diff --git a/examples/chips/goban/configs/1t16c.toml b/examples/chips/goban/configs/designs/1t16c.toml similarity index 88% rename from examples/chips/goban/configs/1t16c.toml rename to examples/chips/goban/configs/designs/1t16c.toml index 44a6d85f2..18ca104f3 100644 --- a/examples/chips/goban/configs/1t16c.toml +++ b/examples/chips/goban/configs/designs/1t16c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/16core.toml" diff --git a/examples/chips/goban/configs/1t2c.toml b/examples/chips/goban/configs/designs/1t2c.toml similarity index 87% rename from examples/chips/goban/configs/1t2c.toml rename to examples/chips/goban/configs/designs/1t2c.toml index 1e03331f3..55353353a 100644 --- a/examples/chips/goban/configs/1t2c.toml +++ b/examples/chips/goban/configs/designs/1t2c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/2core.toml" diff --git a/examples/chips/goban/configs/1t32c.toml b/examples/chips/goban/configs/designs/1t32c.toml similarity index 88% rename from examples/chips/goban/configs/1t32c.toml rename to examples/chips/goban/configs/designs/1t32c.toml index 210614aa4..e395ac1c2 100644 --- a/examples/chips/goban/configs/1t32c.toml +++ b/examples/chips/goban/configs/designs/1t32c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/32core.toml" diff --git a/examples/chips/goban/configs/1t4c.toml b/examples/chips/goban/configs/designs/1t4c.toml similarity index 88% rename from examples/chips/goban/configs/1t4c.toml rename to examples/chips/goban/configs/designs/1t4c.toml index 2539cb4f1..f5f257c4d 100644 --- a/examples/chips/goban/configs/1t4c.toml +++ b/examples/chips/goban/configs/designs/1t4c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/default.toml" diff --git a/examples/chips/goban/configs/1t64c.toml b/examples/chips/goban/configs/designs/1t64c.toml similarity index 88% rename from examples/chips/goban/configs/1t64c.toml rename to examples/chips/goban/configs/designs/1t64c.toml index 91b2691a3..dcc2133b3 100644 --- a/examples/chips/goban/configs/1t64c.toml +++ b/examples/chips/goban/configs/designs/1t64c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/64core.toml" diff --git a/examples/chips/goban/configs/1t8c.toml b/examples/chips/goban/configs/designs/1t8c.toml similarity index 87% rename from examples/chips/goban/configs/1t8c.toml rename to examples/chips/goban/configs/designs/1t8c.toml index 2a234308a..0174108e6 100644 --- a/examples/chips/goban/configs/1t8c.toml +++ b/examples/chips/goban/configs/designs/1t8c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/8core.toml" diff --git a/examples/chips/goban/configs/24t16c.toml b/examples/chips/goban/configs/designs/24t16c.toml similarity index 58% rename from examples/chips/goban/configs/24t16c.toml rename to examples/chips/goban/configs/designs/24t16c.toml index 2d99f1df1..366e8af54 100644 --- a/examples/chips/goban/configs/24t16c.toml +++ b/examples/chips/goban/configs/designs/24t16c.toml @@ -4,5 +4,6 @@ nTiles = 24 [tileTemplate] +tile_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] include = "tiles/16core-incoherent.toml" count = 24 diff --git a/examples/chips/goban/configs/2t4c-private.toml b/examples/chips/goban/configs/designs/2t4c-private.toml similarity index 88% rename from examples/chips/goban/configs/2t4c-private.toml rename to examples/chips/goban/configs/designs/2t4c-private.toml index 8112ca367..77805e549 100644 --- a/examples/chips/goban/configs/2t4c-private.toml +++ b/examples/chips/goban/configs/designs/2t4c-private.toml @@ -4,5 +4,6 @@ nTiles = 2 [tileTemplate] +tile_ids = [0, 1] include = "tiles/4core-private-1bb.toml" count = 2 diff --git a/examples/chips/goban/configs/4t16c.toml b/examples/chips/goban/configs/designs/4t16c.toml similarity index 82% rename from examples/chips/goban/configs/4t16c.toml rename to examples/chips/goban/configs/designs/4t16c.toml index 18700ad6b..6c5e58136 100644 --- a/examples/chips/goban/configs/4t16c.toml +++ b/examples/chips/goban/configs/designs/4t16c.toml @@ -4,5 +4,6 @@ nTiles = 4 [tileTemplate] +tile_ids = [0, 1, 2, 3] include = "tiles/16core-incoherent.toml" count = 4 diff --git a/examples/chips/goban/configs/4t8c.toml b/examples/chips/goban/configs/designs/4t8c.toml similarity index 82% rename from examples/chips/goban/configs/4t8c.toml rename to examples/chips/goban/configs/designs/4t8c.toml index 7d60a7bc5..30d4018cf 100644 --- a/examples/chips/goban/configs/4t8c.toml +++ b/examples/chips/goban/configs/designs/4t8c.toml @@ -4,5 +4,6 @@ nTiles = 4 [tileTemplate] +tile_ids = [0, 1, 2, 3] include = "tiles/8core-incoherent.toml" count = 4 diff --git a/examples/chips/goban/configs/designs/64t4c-private.toml b/examples/chips/goban/configs/designs/64t4c-private.toml new file mode 100644 index 000000000..a2cf16d34 --- /dev/null +++ b/examples/chips/goban/configs/designs/64t4c-private.toml @@ -0,0 +1,9 @@ +# Goban - 64 tiles x 4 cores each (256 harts total) + +[top] +nTiles = 64 + +[tileTemplate] +tile_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63] +include = "tiles/4core-private-1bb.toml" +count = 64 diff --git a/examples/chips/goban/configs/8t8c.toml b/examples/chips/goban/configs/designs/8t8c.toml similarity index 75% rename from examples/chips/goban/configs/8t8c.toml rename to examples/chips/goban/configs/designs/8t8c.toml index 6918e46dc..a92c5c320 100644 --- a/examples/chips/goban/configs/8t8c.toml +++ b/examples/chips/goban/configs/designs/8t8c.toml @@ -4,5 +4,6 @@ nTiles = 8 [tileTemplate] +tile_ids = [0, 1, 2, 3, 4, 5, 6, 7] include = "tiles/8core-incoherent.toml" count = 8 diff --git a/examples/chips/goban/configs/goban.toml b/examples/chips/goban/configs/designs/goban.toml similarity index 89% rename from examples/chips/goban/configs/goban.toml rename to examples/chips/goban/configs/designs/goban.toml index 7262e74c0..c7c883781 100644 --- a/examples/chips/goban/configs/goban.toml +++ b/examples/chips/goban/configs/designs/goban.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/default.toml" diff --git a/examples/chips/goban/configs/tiles/16core-incoherent.toml b/examples/chips/goban/configs/designs/tiles/16core-incoherent.toml similarity index 69% rename from examples/chips/goban/configs/tiles/16core-incoherent.toml rename to examples/chips/goban/configs/designs/tiles/16core-incoherent.toml index e281c3503..aceef646d 100644 --- a/examples/chips/goban/configs/tiles/16core-incoherent.toml +++ b/examples/chips/goban/configs/designs/tiles/16core-incoherent.toml @@ -18,5 +18,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] +include = "../../../../../cores/goban/configs/default.toml" count = 16 diff --git a/examples/chips/goban/configs/tiles/16core.toml b/examples/chips/goban/configs/designs/tiles/16core.toml similarity index 69% rename from examples/chips/goban/configs/tiles/16core.toml rename to examples/chips/goban/configs/designs/tiles/16core.toml index faf06c130..57648b82b 100644 --- a/examples/chips/goban/configs/tiles/16core.toml +++ b/examples/chips/goban/configs/designs/tiles/16core.toml @@ -18,5 +18,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] +include = "../../../../../cores/goban/configs/default.toml" count = 16 diff --git a/examples/chips/goban/configs/tiles/2core.toml b/examples/chips/goban/configs/designs/tiles/2core.toml similarity index 78% rename from examples/chips/goban/configs/tiles/2core.toml rename to examples/chips/goban/configs/designs/tiles/2core.toml index 6bfb0e745..8f4bf42eb 100644 --- a/examples/chips/goban/configs/tiles/2core.toml +++ b/examples/chips/goban/configs/designs/tiles/2core.toml @@ -18,5 +18,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1] +include = "../../../../../cores/goban/configs/default.toml" count = 2 diff --git a/examples/chips/goban/configs/tiles/32core.toml b/examples/chips/goban/configs/designs/tiles/32core.toml similarity index 60% rename from examples/chips/goban/configs/tiles/32core.toml rename to examples/chips/goban/configs/designs/tiles/32core.toml index d6f1ae11b..62a4faa3f 100644 --- a/examples/chips/goban/configs/tiles/32core.toml +++ b/examples/chips/goban/configs/designs/tiles/32core.toml @@ -18,5 +18,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31] +include = "../../../../../cores/goban/configs/default.toml" count = 32 diff --git a/examples/chips/goban/configs/tiles/4core-private-1bb.toml b/examples/chips/goban/configs/designs/tiles/4core-private-1bb.toml similarity index 53% rename from examples/chips/goban/configs/tiles/4core-private-1bb.toml rename to examples/chips/goban/configs/designs/tiles/4core-private-1bb.toml index e6a2b244b..f324a3766 100644 --- a/examples/chips/goban/configs/tiles/4core-private-1bb.toml +++ b/examples/chips/goban/configs/designs/tiles/4core-private-1bb.toml @@ -18,13 +18,17 @@ defaultGroupCount = 1 memCycles = 10 [[cores]] -include = "../../../../cores/goban/core.toml" +core_id = 0 +include = "../../../../../cores/goban/configs/default.toml" [[cores]] -include = "../../../../cores/rocket/core.toml" +core_id = 1 +include = "../../../../../cores/rocket/configs/default.toml" [[cores]] -include = "../../../../cores/rocket/core.toml" +core_id = 2 +include = "../../../../../cores/rocket/configs/default.toml" [[cores]] -include = "../../../../cores/rocket/core.toml" +core_id = 3 +include = "../../../../../cores/rocket/configs/default.toml" diff --git a/examples/chips/goban/configs/designs/tiles/64core.toml b/examples/chips/goban/configs/designs/tiles/64core.toml new file mode 100644 index 000000000..f46dd4bf1 --- /dev/null +++ b/examples/chips/goban/configs/designs/tiles/64core.toml @@ -0,0 +1,23 @@ +# Goban Tile (16 identical cores) + +memBallChannelNum = 6 + +[privateDCache] +enable = false +ways = 4 +capacityKB = 256 +writeBytes = 8 +portFactor = 2 +memCycles = 10 + +[sharedMem] +enable = true +entries = 16384 +inputChannels = 64 +defaultGroupCount = 1 +memCycles = 10 + +[coreTemplate] +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63] +include = "../../../../../cores/goban/configs/default.toml" +count = 64 diff --git a/examples/chips/goban/configs/tiles/8core-incoherent.toml b/examples/chips/goban/configs/designs/tiles/8core-incoherent.toml similarity index 74% rename from examples/chips/goban/configs/tiles/8core-incoherent.toml rename to examples/chips/goban/configs/designs/tiles/8core-incoherent.toml index 74f99fd6f..90d1f5c8a 100644 --- a/examples/chips/goban/configs/tiles/8core-incoherent.toml +++ b/examples/chips/goban/configs/designs/tiles/8core-incoherent.toml @@ -18,5 +18,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7] +include = "../../../../../cores/goban/configs/default.toml" count = 8 diff --git a/examples/chips/goban/configs/tiles/8core.toml b/examples/chips/goban/configs/designs/tiles/8core.toml similarity index 74% rename from examples/chips/goban/configs/tiles/8core.toml rename to examples/chips/goban/configs/designs/tiles/8core.toml index 2759a9a81..dfb6f9411 100644 --- a/examples/chips/goban/configs/tiles/8core.toml +++ b/examples/chips/goban/configs/designs/tiles/8core.toml @@ -18,5 +18,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7] +include = "../../../../../cores/goban/configs/default.toml" count = 8 diff --git a/examples/chips/goban/configs/tiles/default.toml b/examples/chips/goban/configs/designs/tiles/default.toml similarity index 78% rename from examples/chips/goban/configs/tiles/default.toml rename to examples/chips/goban/configs/designs/tiles/default.toml index 8f796eed5..fc5bd61bc 100644 --- a/examples/chips/goban/configs/tiles/default.toml +++ b/examples/chips/goban/configs/designs/tiles/default.toml @@ -19,5 +19,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/goban/core.toml" +core_ids = [0, 1, 2, 3] +include = "../../../../../cores/goban/configs/default.toml" count = 4 diff --git a/examples/chips/goban/configs/tiles/64core.toml b/examples/chips/goban/configs/tiles/64core.toml deleted file mode 100644 index 598db6695..000000000 --- a/examples/chips/goban/configs/tiles/64core.toml +++ /dev/null @@ -1,22 +0,0 @@ -# Goban Tile (16 identical cores) - -memBallChannelNum = 6 - -[privateDCache] -enable = false -ways = 4 -capacityKB = 256 -writeBytes = 8 -portFactor = 2 -memCycles = 10 - -[sharedMem] -enable = true -entries = 16384 -inputChannels = 64 -defaultGroupCount = 1 -memCycles = 10 - -[coreTemplate] -include = "../../../../cores/goban/core.toml" -count = 64 diff --git a/examples/chips/goban/emu/Cargo.lock b/examples/chips/goban/emu/Cargo.lock deleted file mode 100644 index 92d9cb40f..000000000 --- a/examples/chips/goban/emu/Cargo.lock +++ /dev/null @@ -1,935 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bemu-goban" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "camino", - "cc", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "log", - "nix 0.29.0", - "once_cell", - "serde", - "serde_json", - "snafu", - "toml", -] - -[[package]] -name = "bemu-goban-chip" -version = "0.1.0" -dependencies = [ - "bemu-goban", - "clap", - "snafu", - "toml", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "clap" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "find-msvc-tools" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de" - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "portable-atomic" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "thiserror" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/chips/goban/emu/Cargo.toml b/examples/chips/goban/emu/Cargo.toml index e463e4f55..a0745805c 100644 --- a/examples/chips/goban/emu/Cargo.toml +++ b/examples/chips/goban/emu/Cargo.toml @@ -1,20 +1,12 @@ [package] -name = "bemu-goban-chip" +name = "bebop-chip-goban" version = "0.1.0" edition = "2021" [[bin]] -name = "bebop-goban" +name = "bebop-chip-goban" path = "src/main.rs" -[[test]] -name = "test_verilator" -path = "../../../../bebop/tests/test_verilator.rs" -harness = false -required-features = ["verilator"] - [dependencies] +bebop-bemu = { path = "../configs/generated/bemu" } clap = { version = "4", features = ["derive"] } -snafu = "0.8" -toml = "0.8" -core_bemu = { package = "bemu-goban", path = "../../../cores/goban/emu" } diff --git a/examples/chips/goban/emu/src/main.rs b/examples/chips/goban/emu/src/main.rs index 3deb5f44e..da035cb37 100644 --- a/examples/chips/goban/emu/src/main.rs +++ b/examples/chips/goban/emu/src/main.rs @@ -1,21 +1,17 @@ use clap::Parser; -use core_bemu::{BemuInstance, SharedMemory, TraceConfig}; +use bebop_bemu::{tile_topology, BemuInstance, SharedMemory, TraceConfig}; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, AtomicI32, Ordering}; use std::sync::{mpsc, Arc, Condvar, Mutex}; use std::thread; -#[allow(dead_code)] -#[path = "../../../../../bebop/src/nodes/bemu/build_support/config_loader.rs"] -mod config_loader; - const DRAM_SIZE: usize = 1 << 30; #[derive(Parser, Debug)] struct Args { - /// Tile TOML. The file must contain [[cores]] or [coreTemplate]. - #[arg(long)] - tile: PathBuf, + /// Tile index in the chip bundle. + #[arg(long, default_value_t = 0)] + tile_index: usize, #[arg(long)] elf: PathBuf, #[arg(long)] @@ -63,16 +59,15 @@ fn main() { } fn run(args: Args) -> Result<(), String> { - let tile = absolute(&args.tile)?; let elf = absolute(&args.elf)?; let log_dir = absolute(&args.log_dir)?; - let topology = config_loader::parse_tile_config(&tile); + let topology = tile_topology(args.tile_index); if topology.cores.is_empty() { - return Err(format!("tile has no Core instances: {}", tile.display())); + return Err(format!("tile {} has no Core instances", args.tile_index)); } eprintln!( - "[INFO] Goban Chip BEMU: tile={} cores={} shared_dram={}MiB", - tile.display(), + "[INFO] Goban Chip BEMU: tile_index={} cores={} shared_dram={}MiB", + args.tile_index, topology.cores.len(), DRAM_SIZE >> 20 ); @@ -87,7 +82,7 @@ fn run(args: Args) -> Result<(), String> { let (prepared_tx, prepared_rx) = mpsc::channel(); let mut workers = Vec::with_capacity(topology.cores.len()); - for (hart_id, (core_name, core_path, _)) in topology.cores.into_iter().enumerate() { + for (hart_id, (core_name, core_index)) in topology.cores.into_iter().enumerate() { let elf = elf.clone(); let worker_log = log_dir.join(format!("hart-{hart_id}")); let memory = Arc::clone(&memory); @@ -106,7 +101,7 @@ fn run(args: Args) -> Result<(), String> { run_core( hart_id, &core_name, - &core_path, + core_index, &elf, &worker_log, pk, @@ -153,7 +148,7 @@ fn run(args: Args) -> Result<(), String> { fn run_core( hart_id: usize, core_name: &str, - core_path: &Path, + core_index: usize, elf: &Path, log_dir: &Path, pk: bool, @@ -168,8 +163,7 @@ fn run_core( virtual_bank_count: usize, ) -> Result<(), String> { eprintln!( - "[INFO] starting Core worker hart={hart_id} core={core_name} config={}", - core_path.display() + "[INFO] starting Core worker hart={hart_id} core={core_name} core_index={core_index}" ); let prepared_bemu = (|| { let _turn = schedule.lock().map_err(|_| "BEMU scheduler poisoned".to_string())?; @@ -178,7 +172,7 @@ fn run_core( TraceConfig::new(false, false), disasm, tool_profile, - core_path, + core_index, hart_id, Some(Arc::clone(&memory)), Some(virtual_bank_count), diff --git a/examples/chips/goban/workloads/CMakeLists.txt b/examples/chips/goban/workloads/CMakeLists.txt index 8cbcf0766..f3e04f693 100644 --- a/examples/chips/goban/workloads/CMakeLists.txt +++ b/examples/chips/goban/workloads/CMakeLists.txt @@ -1,6 +1,11 @@ +set(BUCKYBALL_CTEST_TARGET goban) + add_subdirectory(ctests) +add_custom_target(chip-ctests-build DEPENDS goban-ctests-build) +add_custom_target(chip-mlirtests-build) + add_custom_target(chip-workloads-build ALL - DEPENDS goban-ctests-build + DEPENDS chip-ctests-build COMMENT "Building goban workloads" VERBATIM) diff --git a/examples/chips/goban/workloads/cmake/RushB.cmake b/examples/chips/goban/workloads/cmake/RushB.cmake deleted file mode 100644 index 8208e28cd..000000000 --- a/examples/chips/goban/workloads/cmake/RushB.cmake +++ /dev/null @@ -1,8 +0,0 @@ -# Goban's host-native rushB backend is owned by the Goban Core. The chip layer -# supplies the selected Core runtime to model workload CMake files. -set(BUCKYBALL_RUSHB_BEMU_MANIFEST - ${BUCKYBALL_REPO_ROOT}/examples/cores/goban/emu/Cargo.toml) -set(BUCKYBALL_RUSHB_BEMU_LIBRARY - ${BUCKYBALL_REPO_ROOT}/examples/cores/goban/emu/target/release/libbebop_bemu.so) -set(BUCKYBALL_RUSHB_VERILATOR_LIBRARY - ${BUCKYBALL_REPO_ROOT}/bebop/target/release/deps/libbebop_verilator.so) diff --git a/examples/chips/goban/workloads/ctests/CMakeLists.txt b/examples/chips/goban/workloads/ctests/CMakeLists.txt index 4f260f6ce..02bb8a556 100644 --- a/examples/chips/goban/workloads/ctests/CMakeLists.txt +++ b/examples/chips/goban/workloads/ctests/CMakeLists.txt @@ -2,9 +2,20 @@ if(NOT DEFINED ELF_CC) set(ELF_CC "riscv64-unknown-elf-gcc") endif() +if(NOT DEFINED BUCKYBALL_CTEST_TARGET OR BUCKYBALL_CTEST_TARGET STREQUAL "") + message(FATAL_ERROR "BUCKYBALL_CTEST_TARGET is required for goban ctests") +endif() +list(FIND BUCKYBALL_TARGETS "${BUCKYBALL_CTEST_TARGET}" _goban_target_index) +if(_goban_target_index EQUAL -1) + message(FATAL_ERROR "unknown C test target: ${BUCKYBALL_CTEST_TARGET}") +endif() +if(NOT EXISTS "${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET}/ballISA.h") + message(FATAL_ERROR + "missing target ISA header ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET}/ballISA.h") +endif() + get_filename_component(_BUCKYBALL_REPO_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/../../../../.." ABSOLUTE) -include(${_BUCKYBALL_REPO_ROOT}/examples/chips/cmake/BuckyballCtestLimits.cmake) set(GOBAN_CTEST_DIR ${CMAKE_CURRENT_SOURCE_DIR}) set(GOBAN_LD ${GOBAN_CTEST_DIR}/goban.ld) @@ -17,6 +28,7 @@ set(GOBAN_C_FLAGS -DBANK_NUM=64 -Wl,-T,${GOBAN_LD} -I${WORKLOAD_LIB_DIR} + -I${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET} -I${GOBAN_CTEST_DIR} ) @@ -43,6 +55,7 @@ function(add_goban_test_target STEM SOURCE_FILE) ${GOBAN_CTEST_DIR}/scu.h ${GOBAN_MEM_RUNTIME} ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.h + ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET}/ballISA.h COMMENT "Building goban multicore baremetal: ${EXECUTABLE}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) @@ -78,6 +91,7 @@ function(add_goban_64t4c_test_target STEM SOURCE_FILE) ${GOBAN_CTEST_DIR}/scu.h ${GOBAN_MEM_RUNTIME} ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.h + ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET}/ballISA.h COMMENT "Building goban 64t4c baremetal: ${EXECUTABLE}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) @@ -114,6 +128,7 @@ function(add_goban_2t4c_test_target STEM SOURCE_FILE) ${GOBAN_CTEST_DIR}/scu.h ${GOBAN_MEM_RUNTIME} ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.h + ${BUCKYBALL_ISA_DIR}/${BUCKYBALL_CTEST_TARGET}/ballISA.h COMMENT "Building goban 2t4c baremetal: ${EXECUTABLE}" WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) diff --git a/examples/chips/goban/workloads/ctests/barrier_mvin_test.c b/examples/chips/goban/workloads/ctests/barrier_mvin_test.c index d67fe3f79..d2abfbfc4 100644 --- a/examples/chips/goban/workloads/ctests/barrier_mvin_test.c +++ b/examples/chips/goban/workloads/ctests/barrier_mvin_test.c @@ -21,13 +21,13 @@ #include #include -#define DIM 16 +#define DIM 16 #define NCORES 4 /* Per-core scratchpad buffers — compiler places these in BSS (all harts share the same virtual address space, but each core writes its own slot). */ -static elem_t src[NCORES][DIM * DIM] __attribute__((aligned(128))); -static elem_t dst[NCORES][DIM * DIM] __attribute__((aligned(128))); +static elem_t src[NCORES][DIM * DIM] __attribute__((aligned(128))); +static elem_t dst[NCORES][DIM * DIM] __attribute__((aligned(128))); static volatile int core_ok[NCORES]; int main(void) { @@ -36,13 +36,13 @@ int main(void) { printf("[core %d] starting mvin/mvout\n", cid); /* ---- Step 1: fill src with a core-specific pattern ---- */ - elem_t pat = (elem_t)(cid + 1); /* core 0 → 1, core 1 → 2, … */ + elem_t pat = (elem_t)(cid + 1); /* core 0 → 1, core 1 → 2, … */ for (int i = 0; i < DIM * DIM; i++) { src[cid][i] = pat; } /* ---- Step 2: mvin → shared bank ---- */ - int bank = cid; /* each core uses its own bank, no conflict */ + int bank = cid; /* each core uses its own bank, no conflict */ bb_mem_alloc(bank, 1, 1); bb_mvin((uintptr_t)src[cid], bank, DIM, 1); @@ -55,8 +55,8 @@ int main(void) { int ok = 1; for (int i = 0; i < DIM * DIM; i++) { if (dst[cid][i] != pat) { - printf("[core %d] ERROR at [%d]: got %d, expected %d\n", - cid, i, (int)dst[cid][i], (int)pat); + printf("[core %d] ERROR at [%d]: got %d, expected %d\n", cid, i, + (int)dst[cid][i], (int)pat); ok = 0; break; } diff --git a/examples/chips/goban/workloads/ctests/goban.h b/examples/chips/goban/workloads/ctests/goban.h index c30d33f29..d78366a9a 100644 --- a/examples/chips/goban/workloads/ctests/goban.h +++ b/examples/chips/goban/workloads/ctests/goban.h @@ -17,6 +17,7 @@ #endif #define GOBAN_MAX_HARTS (GOBAN_CORES_PER_TILE * GOBAN_MAX_TILES) +/* Must match frontend sharedBankIdBase in goban's frontend TOML. */ #define GOBAN_SHARED_BANK_BASE 32 static inline int bb_get_hart_id(void) { diff --git a/examples/chips/goban/workloads/ctests/shared_multigroup_mvin_test.c b/examples/chips/goban/workloads/ctests/shared_multigroup_mvin_test.c index 10e2a5b19..867454ee7 100644 --- a/examples/chips/goban/workloads/ctests/shared_multigroup_mvin_test.c +++ b/examples/chips/goban/workloads/ctests/shared_multigroup_mvin_test.c @@ -66,7 +66,9 @@ int main(void) { } } core_ok[cid] = ok; - log_msg(hart, cid, ok ? "multi-group shared test PASSED" : "multi-group shared test FAILED"); + log_msg(hart, cid, + ok ? "multi-group shared test PASSED" + : "multi-group shared test FAILED"); bb_barrier(); diff --git a/examples/chips/goban/workloads/ctests/shared_same_vbank_test.c b/examples/chips/goban/workloads/ctests/shared_same_vbank_test.c index 516a6921d..53f40bc15 100644 --- a/examples/chips/goban/workloads/ctests/shared_same_vbank_test.c +++ b/examples/chips/goban/workloads/ctests/shared_same_vbank_test.c @@ -65,7 +65,9 @@ int main(void) { } } core_ok[cid] = ok; - log_msg(hart, cid, ok ? "same-vbank shared test PASSED" : "same-vbank shared test FAILED"); + log_msg(hart, cid, + ok ? "same-vbank shared test PASSED" + : "same-vbank shared test FAILED"); bb_barrier(); diff --git a/examples/chips/multi-rocket/arch/src/main/scala/CustomConfigs.scala b/examples/chips/multi-rocket/arch/src/main/scala/CustomConfigs.scala index 91e7870fc..11188342c 100644 --- a/examples/chips/multi-rocket/arch/src/main/scala/CustomConfigs.scala +++ b/examples/chips/multi-rocket/arch/src/main/scala/CustomConfigs.scala @@ -6,15 +6,17 @@ import freechips.rocketchip.tile.MaxHartIdBits import framework.system.tile.WithBuckyballTiles class WithMultiRocketHartIdBits(nTiles: Int, nCoresPerTile: Int) - extends Config((site, here, up) => { - case MaxHartIdBits => log2Ceil(nTiles * nCoresPerTile) + extends Config((site, here, up) => { case MaxHartIdBits => + log2Ceil(nTiles * nCoresPerTile) }) /** 1 tile × 32 Rocket-only cores (no Buckyball). */ class MultiRocket32CoreConfig extends Config( new WithMultiRocketHartIdBits(nTiles = 1, nCoresPerTile = 32) ++ - new WithBuckyballTiles("../examples/chips/multi-rocket/configs/multi-rocket.toml") ++ + new WithBuckyballTiles( + "../examples/chips/multi-rocket/configs/generated/chip.pb" + ) ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -23,7 +25,9 @@ class MultiRocket32CoreConfig class MultiRocket48CoreConfig extends Config( new WithMultiRocketHartIdBits(nTiles = 1, nCoresPerTile = 48) ++ - new WithBuckyballTiles("../examples/chips/multi-rocket/configs/1t48c.toml") ++ + new WithBuckyballTiles( + "../examples/chips/multi-rocket/configs/generated/1t48c.pb" + ) ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) diff --git a/examples/chips/multi-rocket/chip.toml b/examples/chips/multi-rocket/chip.toml deleted file mode 100644 index acd8a6edc..000000000 --- a/examples/chips/multi-rocket/chip.toml +++ /dev/null @@ -1,10 +0,0 @@ -[chip] -name = "multi-rocket" -topology = "configs/multi-rocket.toml" -compilerCore = "rocket" - -[runtime] -bemu = "emu/Cargo.toml" -bemuCore = "../../cores/goban/emu/Cargo.toml" -bemuTile = "configs/tiles/32core.toml" -workloads = "workloads" diff --git a/examples/chips/multi-rocket/configs/chip.toml b/examples/chips/multi-rocket/configs/chip.toml new file mode 100644 index 000000000..770bd1bc4 --- /dev/null +++ b/examples/chips/multi-rocket/configs/chip.toml @@ -0,0 +1,6 @@ +[designs] +include = "designs/multi-rocket.toml" + +[sims] +verilator = "sims.verilator.MultiRocket32CoreVerilatorConfig" +p2e = "sims.p2e.MultiRocket32CoreP2EConfig" diff --git a/examples/chips/multi-rocket/configs/1t48c.toml b/examples/chips/multi-rocket/configs/designs/1t48c.toml similarity index 89% rename from examples/chips/multi-rocket/configs/1t48c.toml rename to examples/chips/multi-rocket/configs/designs/1t48c.toml index 6ce04b7b6..df7de5647 100644 --- a/examples/chips/multi-rocket/configs/1t48c.toml +++ b/examples/chips/multi-rocket/configs/designs/1t48c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/48core.toml" diff --git a/examples/chips/multi-rocket/configs/multi-rocket.toml b/examples/chips/multi-rocket/configs/designs/multi-rocket.toml similarity index 89% rename from examples/chips/multi-rocket/configs/multi-rocket.toml rename to examples/chips/multi-rocket/configs/designs/multi-rocket.toml index 7269df5c4..e5b2aa3b6 100644 --- a/examples/chips/multi-rocket/configs/multi-rocket.toml +++ b/examples/chips/multi-rocket/configs/designs/multi-rocket.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/32core.toml" diff --git a/examples/chips/multi-rocket/configs/tiles/32core.toml b/examples/chips/multi-rocket/configs/designs/tiles/32core.toml similarity index 58% rename from examples/chips/multi-rocket/configs/tiles/32core.toml rename to examples/chips/multi-rocket/configs/designs/tiles/32core.toml index bb20bd9f3..53f185086 100644 --- a/examples/chips/multi-rocket/configs/tiles/32core.toml +++ b/examples/chips/multi-rocket/configs/designs/tiles/32core.toml @@ -16,5 +16,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/rocket/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31] +include = "../../../../../cores/rocket/configs/default.toml" count = 32 diff --git a/examples/chips/multi-rocket/configs/tiles/48core.toml b/examples/chips/multi-rocket/configs/designs/tiles/48core.toml similarity index 51% rename from examples/chips/multi-rocket/configs/tiles/48core.toml rename to examples/chips/multi-rocket/configs/designs/tiles/48core.toml index 248dbdf02..0dcd6c8b7 100644 --- a/examples/chips/multi-rocket/configs/tiles/48core.toml +++ b/examples/chips/multi-rocket/configs/designs/tiles/48core.toml @@ -16,5 +16,6 @@ defaultGroupCount = 1 memCycles = 10 [coreTemplate] -include = "../../../../cores/rocket/core.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47] +include = "../../../../../cores/rocket/configs/default.toml" count = 48 diff --git a/examples/chips/multi-rocket/emu/Cargo.lock b/examples/chips/multi-rocket/emu/Cargo.lock deleted file mode 100644 index c848016a8..000000000 --- a/examples/chips/multi-rocket/emu/Cargo.lock +++ /dev/null @@ -1,935 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bemu-goban" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "camino", - "cc", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "log", - "nix 0.29.0", - "once_cell", - "serde", - "serde_json", - "snafu", - "toml", -] - -[[package]] -name = "bemu-multi-rocket-chip" -version = "0.1.0" -dependencies = [ - "bemu-goban", - "clap", - "snafu", - "toml", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "clap" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "find-msvc-tools" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de" - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "portable-atomic" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "thiserror" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/chips/multi-rocket/emu/Cargo.toml b/examples/chips/multi-rocket/emu/Cargo.toml index 988ed57e0..fd64af93b 100644 --- a/examples/chips/multi-rocket/emu/Cargo.toml +++ b/examples/chips/multi-rocket/emu/Cargo.toml @@ -1,14 +1,12 @@ [package] -name = "bemu-multi-rocket-chip" +name = "bebop-chip-multi-rocket" version = "0.1.0" edition = "2021" [[bin]] -name = "bebop-multi-rocket" +name = "bebop-chip-multi-rocket" path = "src/main.rs" [dependencies] +bebop-bemu = { path = "../configs/generated/bemu" } clap = { version = "4", features = ["derive"] } -snafu = "0.8" -toml = "0.8" -core_bemu = { package = "bemu-goban", path = "../../../cores/goban/emu" } diff --git a/examples/chips/multi-rocket/emu/src/main.rs b/examples/chips/multi-rocket/emu/src/main.rs index 6d386c4c2..82761471c 100644 --- a/examples/chips/multi-rocket/emu/src/main.rs +++ b/examples/chips/multi-rocket/emu/src/main.rs @@ -1 +1,241 @@ -include!("../../../goban/emu/src/main.rs"); +use clap::Parser; +use bebop_bemu::{tile_topology, BemuInstance, SharedMemory, TraceConfig}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, AtomicI32, Ordering}; +use std::sync::{mpsc, Arc, Condvar, Mutex}; +use std::thread; + +const DRAM_SIZE: usize = 1 << 30; + +#[derive(Parser, Debug)] +struct Args { + /// Tile index in the chip bundle. + #[arg(long, default_value_t = 0)] + tile_index: usize, + #[arg(long)] + elf: PathBuf, + #[arg(long)] + log_dir: PathBuf, + #[arg(long)] + pk: bool, + #[arg(long)] + disasm: bool, + #[arg(long = "tool-profile")] + tool_profile: bool, +} + +struct StartGate { + result: Mutex>>, + ready: Condvar, +} + +impl StartGate { + fn new() -> Self { + Self { + result: Mutex::new(None), + ready: Condvar::new(), + } + } + + fn wait(&self) -> Result<(), String> { + let mut result = self.result.lock().map_err(|_| "BEMU start gate poisoned".to_string())?; + while result.is_none() { + result = self.ready.wait(result).map_err(|_| "BEMU start gate poisoned".to_string())?; + } + result.as_ref().expect("BEMU start gate was set").clone() + } + + fn release(&self, result: Result<(), String>) { + *self.result.lock().expect("BEMU start gate poisoned") = Some(result); + self.ready.notify_all(); + } +} + +fn main() { + if let Err(error) = run(Args::parse()) { + eprintln!("error: {error}"); + std::process::exit(1); + } +} + +fn run(args: Args) -> Result<(), String> { + let elf = absolute(&args.elf)?; + let log_dir = absolute(&args.log_dir)?; + let topology = tile_topology(args.tile_index); + if topology.cores.is_empty() { + return Err(format!("tile {} has no Core instances", args.tile_index)); + } + eprintln!( + "[INFO] Multi-Rocket Chip BEMU: tile_index={} cores={} shared_dram={}MiB", + args.tile_index, + topology.cores.len(), + DRAM_SIZE >> 20 + ); + + let core_count = topology.cores.len(); + let virtual_bank_count = topology.virtual_bank_count; + let memory = SharedMemory::new(DRAM_SIZE, core_count); + let schedule = Arc::new(Mutex::new(())); + let start = Arc::new(StartGate::new()); + let done = Arc::new(AtomicBool::new(false)); + let exit_code = Arc::new(AtomicI32::new(0)); + let (prepared_tx, prepared_rx) = mpsc::channel(); + let mut workers = Vec::with_capacity(topology.cores.len()); + + for (hart_id, (core_name, core_index)) in topology.cores.into_iter().enumerate() { + let elf = elf.clone(); + let worker_log = log_dir.join(format!("hart-{hart_id}")); + let memory = Arc::clone(&memory); + let schedule = Arc::clone(&schedule); + let start = Arc::clone(&start); + let done = Arc::clone(&done); + let exit_code = Arc::clone(&exit_code); + let prepared_tx = prepared_tx.clone(); + let pk = args.pk; + let disasm = args.disasm; + let tool_profile = args.tool_profile; + workers.push( + thread::Builder::new() + .name(format!("core-{hart_id}-{core_name}")) + .spawn(move || { + run_core( + hart_id, + &core_name, + core_index, + &elf, + &worker_log, + pk, + disasm, + tool_profile, + memory, + schedule, + start, + done, + exit_code, + prepared_tx, + virtual_bank_count, + ) + }) + .map_err(|error| format!("failed to spawn Core worker {hart_id}: {error}"))?, + ); + } + drop(prepared_tx); + + let mut preparation_error = None; + for _ in 0..workers.len() { + if let Err(error) = prepared_rx.recv().map_err(|_| "Core worker exited before initialization".to_string())? { + preparation_error.get_or_insert(error); + } + } + start.release(preparation_error.map_or(Ok(()), Err)); + + let mut first_error = None; + for (hart_id, worker) in workers.into_iter().enumerate() { + match worker.join() { + Ok(Ok(())) => {} + Ok(Err(error)) => { + first_error.get_or_insert(format!("Core worker {hart_id}: {error}")); + } + Err(_) => { + first_error.get_or_insert(format!("Core worker {hart_id} panicked")); + } + }; + } + first_error.map_or(Ok(()), Err) +} + +#[allow(clippy::too_many_arguments)] +fn run_core( + hart_id: usize, + core_name: &str, + core_index: usize, + elf: &Path, + log_dir: &Path, + pk: bool, + disasm: bool, + tool_profile: bool, + memory: Arc, + schedule: Arc>, + start: Arc, + done: Arc, + exit_code: Arc, + prepared: mpsc::Sender>, + virtual_bank_count: usize, +) -> Result<(), String> { + eprintln!( + "[INFO] starting Core worker hart={hart_id} core={core_name} core_index={core_index}" + ); + let prepared_bemu = (|| { + let _turn = schedule.lock().map_err(|_| "BEMU scheduler poisoned".to_string())?; + let mut bemu = BemuInstance::new_with_core_hart( + log_dir, + TraceConfig::new(false, false), + disasm, + tool_profile, + core_index, + hart_id, + Some(Arc::clone(&memory)), + Some(virtual_bank_count), + ) + .map_err(|error| error.to_string())?; + bemu.load_elf(elf).map_err(|error| error.to_string())?; + bemu.init_hart(pk).map_err(|error| error.to_string())?; + Ok::(bemu) + })(); + + let mut bemu = match prepared_bemu { + Ok(bemu) => { + let _ = prepared.send(Ok(())); + bemu + } + Err(error) => { + let _ = prepared.send(Err(error.clone())); + return Err(error); + } + }; + start.wait()?; + + loop { + let barrier_hit = { + let _turn = schedule.lock().map_err(|_| "BEMU scheduler poisoned".to_string())?; + if done.load(Ordering::Acquire) { + bemu.stop(exit_code.load(Ordering::Acquire)); + break; + } + if let Err(error) = bemu.step() { + exit_code.store(1, Ordering::Release); + done.store(true, Ordering::Release); + memory.abort_barrier(); + return Err(error.to_string()); + } + bemu.barrier_hit() + }; + if barrier_hit { + memory.wait_barrier(hart_id); + } + if bemu.finished() { + let code = bemu.exit_code().unwrap_or(1); + exit_code.store(code, Ordering::Release); + done.store(true, Ordering::Release); + memory.abort_barrier(); + break; + } + } + + let code = exit_code.load(Ordering::Acquire); + eprintln!("[INFO] stopped Core worker hart={hart_id} core={core_name} exit={code}"); + if code == 0 { + Ok(()) + } else { + Err(format!("guest exited with code {code}")) + } +} + +fn absolute(path: &Path) -> Result { + if path.is_absolute() { + return Ok(path.to_path_buf()); + } + std::env::current_dir() + .map(|cwd| cwd.join(path)) + .map_err(|error| format!("failed to resolve {}: {error}", path.display())) +} diff --git a/examples/chips/multi-rocket/regression/batch/bemu/workloads-elf.toml b/examples/chips/multi-rocket/regression/batch/bemu/workloads-elf.toml new file mode 100644 index 000000000..bf2704972 --- /dev/null +++ b/examples/chips/multi-rocket/regression/batch/bemu/workloads-elf.toml @@ -0,0 +1,22 @@ +[workloads] +search_path = "workloads/src" +tests = [ + "embench_aha-mont64-baremetal", + "embench_crc32-baremetal", + "embench_edn-baremetal", + "embench_huffbench-baremetal", + "embench_matmult-int-baremetal", + "embench_minver-baremetal", + "embench_nbody-baremetal", + "embench_nettle-aes-baremetal", + "embench_nettle-sha256-baremetal", + "embench_nsichneu-baremetal", + "embench_picojpeg-baremetal", + "embench_qrduino-baremetal", + "embench_sglib-combined-baremetal", + "embench_slre-baremetal", + "embench_st-baremetal", + "embench_statemate-baremetal", + "embench_ud-baremetal", + "embench_wikisort-baremetal", +] diff --git a/examples/chips/multi-rocket/regression/batch/verilator/workloads-elf.toml b/examples/chips/multi-rocket/regression/batch/verilator/workloads-elf.toml new file mode 100644 index 000000000..e8dccb50d --- /dev/null +++ b/examples/chips/multi-rocket/regression/batch/verilator/workloads-elf.toml @@ -0,0 +1,22 @@ +[workloads] +search_path = "workloads/src" +tests = [ + "embench_aha-mont64-baremetal", + "embench_crc32-baremetal", + "embench_edn-baremetal", + # "embench_huffbench-baremetal", + "embench_matmult-int-baremetal", + "embench_minver-baremetal", + "embench_nbody-baremetal", + "embench_nettle-aes-baremetal", + "embench_nettle-sha256-baremetal", + "embench_nsichneu-baremetal", + # "embench_picojpeg-baremetal", + # "embench_qrduino-baremetal", + "embench_sglib-combined-baremetal", + "embench_slre-baremetal", + "embench_st-baremetal", + "embench_statemate-baremetal", + "embench_ud-baremetal", + # "embench_wikisort-baremetal", +] diff --git a/examples/chips/multi-rocket/workloads/CMakeLists.txt b/examples/chips/multi-rocket/workloads/CMakeLists.txt index 9c6f1fe7c..aec81ab12 100644 --- a/examples/chips/multi-rocket/workloads/CMakeLists.txt +++ b/examples/chips/multi-rocket/workloads/CMakeLists.txt @@ -1,3 +1,7 @@ +add_custom_target(chip-ctests-build) +add_custom_target(chip-mlirtests-build) + add_custom_target(chip-workloads-build ALL + DEPENDS chip-ctests-build chip-mlirtests-build COMMENT "multi-rocket has no chip-local workloads" VERBATIM) diff --git a/examples/chips/multi-rocket/workloads/cmake/RushB.cmake b/examples/chips/multi-rocket/workloads/cmake/RushB.cmake deleted file mode 100644 index 7a69cb98c..000000000 --- a/examples/chips/multi-rocket/workloads/cmake/RushB.cmake +++ /dev/null @@ -1,6 +0,0 @@ -set(BUCKYBALL_RUSHB_BEMU_MANIFEST - ${BUCKYBALL_REPO_ROOT}/examples/cores/goban/emu/Cargo.toml) -set(BUCKYBALL_RUSHB_BEMU_LIBRARY - ${BUCKYBALL_REPO_ROOT}/examples/cores/goban/emu/target/release/libbebop_bemu.so) -set(BUCKYBALL_RUSHB_VERILATOR_LIBRARY - ${BUCKYBALL_REPO_ROOT}/bebop/target/release/deps/libbebop_verilator.so) diff --git a/examples/chips/pebble/arch/src/main/scala/CustomConfigs.scala b/examples/chips/pebble/arch/src/main/scala/CustomConfigs.scala index bcb343523..d8ab3b94c 100644 --- a/examples/chips/pebble/arch/src/main/scala/CustomConfigs.scala +++ b/examples/chips/pebble/arch/src/main/scala/CustomConfigs.scala @@ -4,18 +4,21 @@ import org.chipsalliance.cde.config.Config import framework.system.tile.WithBuckyballTiles /** - * Pebble example: one tile with TransposeBall and MatrixBall. + * Pebble example: one tile with TransposeBall and SMatMulBall. */ class BuckyballPebbleConfig extends Config( - new WithBuckyballTiles("../examples/chips/pebble/configs/pebble.toml") ++ + new WithBuckyballTiles("../examples/chips/pebble/configs/generated/chip.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) class RocketOnlyPebbleConfig extends Config( - new WithBuckyballTiles("../examples/chips/pebble/configs/pebble.toml", withBuckyball = false) ++ + new WithBuckyballTiles( + "../examples/chips/pebble/configs/generated/chip.pb", + withBuckyball = false + ) ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) diff --git a/examples/chips/pebble/chip.toml b/examples/chips/pebble/chip.toml deleted file mode 100644 index 33e9bc49f..000000000 --- a/examples/chips/pebble/chip.toml +++ /dev/null @@ -1,10 +0,0 @@ -[chip] -name = "pebble" -topology = "configs/pebble.toml" -compilerCore = "pebble" - -[runtime] -bemu = "../../cores/pebble/emu/Cargo.toml" -kernel = "kernel" -workloads = "workloads" -regression = "regression" diff --git a/examples/chips/pebble/compiler/src/Conversion/LowerBuckyball/MatrixMatmulToBankSSAPatterns.cpp b/examples/chips/pebble/compiler/src/Conversion/LowerBuckyball/MatrixMatmulToBankSSAPatterns.cpp deleted file mode 100644 index 83911f36e..000000000 --- a/examples/chips/pebble/compiler/src/Conversion/LowerBuckyball/MatrixMatmulToBankSSAPatterns.cpp +++ /dev/null @@ -1,64 +0,0 @@ -//===- MatrixMatmulToBankSSAPatterns.cpp - float matmul -------------------===// - -#include "Conversion/LowerBuckyball/LowerBuckyball.h" - -#include "mlir/Dialect/Linalg/IR/Linalg.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/IR/BuiltinTypes.h" -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" - -using namespace mlir; -using namespace ::buddy::buckyball; - -namespace { - -// Float matrix_matmul: keep IR compact with linalg.matmul (accumulates into C). -// Full bank-SSA quantize expansion blows up LLM-sized graphs and crashes -// convert-scf-to-cf (MLIR IRRewrite assert on huge modules). -class MatrixMatmulToBankSSAPattern : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(MatrixMatmulOp op, - PatternRewriter &b) const override { - Location loc = op.getLoc(); - auto aTy = dyn_cast(op.getAMemArray().getType()); - auto bTy = dyn_cast(op.getBMemArray().getType()); - auto cTy = dyn_cast(op.getCMemArray().getType()); - if (!aTy || !bTy || !cTy || !aTy.hasStaticShape() || - !bTy.hasStaticShape() || !cTy.hasStaticShape()) - return b.notifyMatchFailure(op, "requires static memrefs"); - Type elem = aTy.getElementType(); - if (!isa(elem) || bTy.getElementType() != elem || - cTy.getElementType() != elem) - return b.notifyMatchFailure(op, - "pebble bank-ssa compact path is float only"); - - int64_t M = aTy.getShape()[0], K = aTy.getShape()[1]; - int64_t Kb = bTy.getShape()[0], N = bTy.getShape()[1]; - if (K != Kb || cTy.getShape()[0] != M || cTy.getShape()[1] != N) - return op.emitError("matmul shape mismatch"); - - b.create(loc, - ValueRange{op.getAMemArray(), op.getBMemArray()}, - ValueRange{op.getCMemArray()}); - b.eraseOp(op); - return success(); - } -}; - -} // namespace - -void mlir::buddy::populatePebbleMatrixMatmulToBankSSAPatterns( - RewritePatternSet &patterns) { - patterns.add(patterns.getContext()); -} - -void mlir::buddy::populatePebbleLowerBuckyballToBankSSAPatterns( - RewritePatternSet &patterns) { - populatePebbleMatrixMatmulToBankSSAPatterns(patterns); - populatePebbleIm2colMatmulToBankSSAPatterns(patterns); - populatePebbleMemTransposeToBankSSAPatterns(patterns); -} diff --git a/examples/chips/pebble/configs/chip.toml b/examples/chips/pebble/configs/chip.toml new file mode 100644 index 000000000..d8ebb849d --- /dev/null +++ b/examples/chips/pebble/configs/chip.toml @@ -0,0 +1,6 @@ +[designs] +include = "designs/pebble.toml" + +[sims] +verilator = "sims.verilator.BuckyballPebbleVerilatorConfig" +p2e = "sims.p2e.P2EPebbleLinuxConfig" diff --git a/examples/chips/pebble/configs/pebble.toml b/examples/chips/pebble/configs/designs/pebble.toml similarity index 88% rename from examples/chips/pebble/configs/pebble.toml rename to examples/chips/pebble/configs/designs/pebble.toml index ac8a4d13e..8d6f59dc0 100644 --- a/examples/chips/pebble/configs/pebble.toml +++ b/examples/chips/pebble/configs/designs/pebble.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/default.toml" diff --git a/examples/chips/pebble/configs/tiles/default.toml b/examples/chips/pebble/configs/designs/tiles/default.toml similarity index 77% rename from examples/chips/pebble/configs/tiles/default.toml rename to examples/chips/pebble/configs/designs/tiles/default.toml index c74eb6db1..e91d7b9b6 100644 --- a/examples/chips/pebble/configs/tiles/default.toml +++ b/examples/chips/pebble/configs/designs/tiles/default.toml @@ -17,4 +17,5 @@ inputChannels = 7 defaultGroupCount = 1 [[cores]] -include = "../../../../cores/pebble/core.toml" +core_id = 0 +include = "../../../../../cores/pebble/configs/default.toml" diff --git a/examples/chips/pebble/kernel/overlay/init b/examples/chips/pebble/kernel/overlay/init index 14d4d200b..1eba0cad3 100755 --- a/examples/chips/pebble/kernel/overlay/init +++ b/examples/chips/pebble/kernel/overlay/init @@ -36,6 +36,8 @@ fail_exit() { } count=0 +failed=0 +fails="" for t in /root/*-linux; do [ -e "$t" ] || continue name=${t##*/} @@ -44,16 +46,24 @@ for t in /root/*-linux; do "$t" rc=$? set -e + count=$((count + 1)) if [ "$rc" -ne 0 ]; then - fail_exit "FAIL $name rc=$rc" + echo "FAIL $name rc=$rc" + failed=$((failed + 1)) + fails="$fails $name" + else + echo "PASS $name" fi - echo "PASS $name" - count=$((count + 1)) done if [ "$count" -eq 0 ]; then fail_exit "ERROR: no *-linux tests installed under /root" fi +echo "SUMMARY total=$count passed=$((count - failed)) failed=$failed" +if [ "$failed" -ne 0 ]; then + fail_exit "FAILED_TESTS:$fails" +fi + echo "ALL_PASS count=$count" poweroff -f diff --git a/examples/chips/pebble/regression/batch/bemu/workloads-elf.toml b/examples/chips/pebble/regression/batch/bemu/workloads-elf.toml index 063d95c2a..1ad28a34e 100644 --- a/examples/chips/pebble/regression/batch/bemu/workloads-elf.toml +++ b/examples/chips/pebble/regression/batch/bemu/workloads-elf.toml @@ -1,103 +1,87 @@ [workloads] search_path = "workloads/src" tests = [ - "pebble_mvin_mvout_test-singlecore-baremetal", - "pebble_mvin_mvout_stride_test-singlecore-baremetal", - "pebble_optest_base_mvin_mvout_16x16_i8_singlecore-baremetal", - "pebble_optest_base_bank_mvin_mvout_16x16_i8_singlecore-baremetal", - "pebble_transpose_i8_16x16_test-singlecore-baremetal", - "pebble_transpose_i8_16x32_test-singlecore-baremetal", - "pebble_transpose_i32_8x16_test-singlecore-baremetal", - "pebble_transpose_bank_i8_n1_1024x16_test-singlecore-baremetal", - "pebble_transpose_bank_i8_n2_1024x32_test-singlecore-baremetal", - "pebble_transpose_bank_i32_n4_1024x16_test-singlecore-baremetal", - "pebble_optest_bank_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_bank_matrix_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_matrix_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_matrix_matmul_16x16_i8_singlecore-baremetal", - "pebble_matrix_os_4x4_test-singlecore-baremetal", - "pebble_matrix_ws_4x4_test-singlecore-baremetal", - "pebble_matrix_os_16x16_test-singlecore-baremetal", - "pebble_matrix_ws_16x16_test-singlecore-baremetal", - "pebble_matrix_os_32x16_test-singlecore-baremetal", - "pebble_matrix_ws_32x16_test-singlecore-baremetal", - "pebble_matrix_os_16x16_ones-singlecore-baremetal", - "pebble_matrix_os_32x32_ones-singlecore-baremetal", - "pebble_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "pebble_matrix_mode_switch_test-singlecore-baremetal", - "pebble_matrix_bank_os_1024x16x16_test-singlecore-baremetal", - "pebble_matrix_bank_ws_1024x16x16_test-singlecore-baremetal", - "pebble_optest_linalg_matmul_16x16_i8_singlecore-baremetal", - "pebble_optest_linalg_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_linalg_conv_2d_nhwc_hwcf_6x6_k3_singlecore-baremetal", - "pebble_optest_tile_matmul_16x16_i8_singlecore-baremetal", - "pebble_optest_tile_matmul_16x16_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_1x256x120_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_1x576x1024_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_1x1024x1000_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_5x3072x768_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_tile_transpose_80x80_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_120x256_f32_singlecore-baremetal", + "pebble-pebble-ctest-mvin_mvout_test-baremetal", + "pebble-pebble-ctest-mvin_mvout_stride_test-baremetal", + "pebble-pebble-mlirtest-base_mvin_mvout_16x16_i8-baremetal", + "pebble-pebble-mlirtest-base_bank_mvin_mvout_16x16_i8-baremetal", + "pebble-pebble-ctest-transpose_i8_16x16_test-baremetal", + "pebble-pebble-ctest-transpose_i8_16x32_test-baremetal", + "pebble-pebble-ctest-transpose_i32_8x16_test-baremetal", + "pebble-pebble-ctest-transpose_bank_i8_n1_1024x16_test-baremetal", + "pebble-pebble-ctest-transpose_bank_i8_n2_1024x32_test-baremetal", + "pebble-pebble-ctest-transpose_bank_i32_n4_1024x16_test-baremetal", + "pebble-pebble-mlirtest-bank_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-ball_transpose_16x16_i8-baremetal", + "pebble-pebble-ctest-smatmul_16x16_test-baremetal", + "pebble-pebble-ctest-smatmul_32x16x32_test-baremetal", + "pebble-pebble-ctest-smatmul_ws_16x16_test-baremetal", + "pebble-pebble-ctest-smatmul_ws_16x512_test-baremetal", + "pebble-pebble-mlirtest-bank_smatmul_ws_16x512_i8-baremetal", + "pebble-pebble-mlirtest-ball_smatmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-ball_smatmul_matmul_16x16_i8-baremetal", + "pebble-pebble-ctest-relu_test-baremetal", + "pebble-pebble-ctest-relu_bank_256x16_test-baremetal", + "pebble-pebble-mlirtest-bank_relu_bank_256x16_i32-baremetal", + "pebble-pebble-mlirtest-linalg_relu_513x17_i32-baremetal", + "pebble-pebble-ctest-matadd_test-baremetal", + "pebble-pebble-ctest-matadd_bank_1024x8_i32_test-baremetal", + "pebble-pebble-mlirtest-bank_matadd_bank_1024x8_i32-baremetal", + "pebble-pebble-mlirtest-linalg_matmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-linalg_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-linalg_conv_2d_nhwc_hwcf_6x6_k3-baremetal", + "pebble-pebble-mlirtest-tile_matmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-tile_matmul_16x16_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_1x256x120_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_1x576x1024_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_1x1024x1000_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_5x3072x768_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-tile_transpose_80x80_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_120x256_f32-baremetal", - "pebble_optest_tile_transpose_128x128_dynstride_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_1024x576_dynstride_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_matmul_120x256_f32_singlecore-baremetal", - "pebble_optest_tile_conv2d_6x6_k3_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc8_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc24_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c240_oc64_singlecore-baremetal", - "pebble_optest_tile_conv2d_8x8_1x1_c16_oc24_singlecore-baremetal", - "pebble_optest_tile_conv2d_8x8_k3_c2_oc2_singlecore-baremetal", - "pebble_optest_tile_conv2d_12x12_k5_singlecore-baremetal", - "pebble_optest_tile_conv2d_28x28_k5_singlecore-baremetal", - "pebble_optest_bank_im2col_6x6_k3_singlecore-baremetal", - "pebble_optest_ball_im2col_6x6_k3_singlecore-baremetal", - "pebble_optest_bank_fp2int_4x4_singlecore-baremetal", - "pebble_optest_ball_fp2int_4x4_singlecore-baremetal", - "pebble_optest_bank_int2fp_4x4_singlecore-baremetal", - "pebble_optest_ball_int2fp_4x4_singlecore-baremetal", - "pebble_optest_bank_int32_to_int8_2x16_singlecore-baremetal", - "pebble_optest_ball_int32_to_int8_2x16_singlecore-baremetal", - "pebble_im2col_k1_test-singlecore-baremetal", - "pebble_im2col_k3_test-singlecore-baremetal", - "pebble_im2col_k3_partial_test-singlecore-baremetal", - "pebble_im2col_k3_pad_test-singlecore-baremetal", - "pebble_im2col_k5_test-singlecore-baremetal", - "pebble_im2col_k7_test-singlecore-baremetal", - "pebble_im2col_bank_k1_test-singlecore-baremetal", - "pebble_im2col_bank_k3_test-singlecore-baremetal", - "pebble_im2col_bank_k5_test-singlecore-baremetal", - "pebble_im2col_bank_k7_test-singlecore-baremetal", - "pebble_conv_im2col_test-singlecore-baremetal", - "pebble_conv_transpose_test-singlecore-baremetal", - "pebble_conv_matmul_test-singlecore-baremetal", - "pebble_quant_fp32_to_int8_test-singlecore-baremetal", - "pebble_quant_im2col_test-singlecore-baremetal", - "pebble_quant_transpose_test-singlecore-baremetal", - "pebble_quant_gemm_test-singlecore-baremetal", - "pebble_quant_int8_to_fp32_test-singlecore-baremetal", - "pebble_conv_im2col_bank_test-singlecore-baremetal", - "pebble_conv_transpose_bank_test-singlecore-baremetal", - "pebble_conv_matmul_bank_test-singlecore-baremetal", - "pebble_quant_fp32_to_int8_bank_test-singlecore-baremetal", - "pebble_quant_im2col_bank_test-singlecore-baremetal", - "pebble_quant_transpose_bank_test-singlecore-baremetal", - "pebble_quant_gemm_bank_test-singlecore-baremetal", - "pebble_quant_int8_to_fp32_bank_test-singlecore-baremetal", - "pebble_fp2int_fp32_to_int32_test-singlecore-baremetal", - "pebble_fp2int_fp32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_int8_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_int32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", - "pebble_fp2int_bank_fp32_to_int32_test-singlecore-baremetal", - "pebble_fp2int_bank_fp32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_bank_int8_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_fp32_col2_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_fp32_col4_test-singlecore-baremetal", + "pebble-pebble-mlirtest-tile_transpose_128x128_dynstride_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_1024x576_dynstride_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_matmul_120x256_f32-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_6x6_k3-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc8-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc24-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c240_oc64-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_8x8_1x1_c16_oc24-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_8x8_k3_c2_oc2-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_12x12_k5-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_28x28_k5-baremetal", + "pebble-pebble-mlirtest-bank_im2col_6x6_k3-baremetal", + "pebble-pebble-mlirtest-ball_im2col_6x6_k3-baremetal", + "pebble-pebble-mlirtest-bank_fp2int_4x4-baremetal", + "pebble-pebble-mlirtest-ball_fp2int_4x4-baremetal", + "pebble-pebble-mlirtest-bank_int2fp_4x4-baremetal", + "pebble-pebble-mlirtest-ball_int2fp_4x4-baremetal", + "pebble-pebble-mlirtest-bank_int2fp_channel_4x4-baremetal", + "pebble-pebble-mlirtest-ball_int2fp_channel_4x4-baremetal", + "pebble-pebble-ctest-fp2int_i8_signed_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_rounding_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_zero_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_rows_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_scale_rows_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_values_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_groups_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_rows_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_lanes_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_base_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_two_rows_test-baremetal", + "pebble-pebble-ctest-im2col_k1_test-baremetal", + "pebble-pebble-ctest-im2col_k3_test-baremetal", + "pebble-pebble-ctest-im2col_k3_partial_test-baremetal", + "pebble-pebble-ctest-im2col_k3_pad_test-baremetal", + "pebble-pebble-ctest-im2col_k5_test-baremetal", + "pebble-pebble-ctest-im2col_k7_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k1_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k3_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k5_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k7_test-baremetal", + "pebble-pebble-ctest-conv_im2col_test-baremetal", + "pebble-pebble-ctest-conv_transpose_test-baremetal", + "pebble-pebble-ctest-conv_im2col_bank_test-baremetal", + "pebble-pebble-ctest-conv_transpose_bank_test-baremetal", ] diff --git a/examples/chips/pebble/regression/batch/bemu/workloads-pk-mvout-fence.toml b/examples/chips/pebble/regression/batch/bemu/workloads-pk-mvout-fence.toml index 9f953b552..7b9d6926c 100644 --- a/examples/chips/pebble/regression/batch/bemu/workloads-pk-mvout-fence.toml +++ b/examples/chips/pebble/regression/batch/bemu/workloads-pk-mvout-fence.toml @@ -1,5 +1,5 @@ [workloads] search_path = "workloads/src" tests = [ - "pebble_mvout_fence_loop_test-linux", + "pebble-pebble-ctest-mvout_fence_loop_test-linux", ] diff --git a/examples/chips/pebble/regression/batch/bemu/workloads-pk.toml b/examples/chips/pebble/regression/batch/bemu/workloads-pk.toml index 1012ec61d..7c49c7141 100644 --- a/examples/chips/pebble/regression/batch/bemu/workloads-pk.toml +++ b/examples/chips/pebble/regression/batch/bemu/workloads-pk.toml @@ -1,104 +1,89 @@ [workloads] search_path = "workloads/src" tests = [ - "pebble_mvin_mvout_test-linux", - "pebble_mvin_mvout_stride_test-linux", - "pebble_optest_base_mvin_mvout_16x16_i8-linux", - "pebble_optest_base_bank_mvin_mvout_16x16_i8-linux", - "pebble_transpose_i8_16x16_test-linux", - "pebble_transpose_i8_16x32_test-linux", - "pebble_transpose_i32_8x16_test-linux", - "pebble_transpose_bank_i8_n1_1024x16_test-linux", - "pebble_transpose_bank_i8_n2_1024x32_test-linux", - "pebble_transpose_bank_i32_n4_1024x16_test-linux", - "pebble_optest_bank_transpose_16x16_i8-linux", - "pebble_optest_ball_transpose_16x16_i8-linux", - "pebble_optest_bank_matrix_16x16_i8-linux", - "pebble_optest_ball_matrix_16x16_i8-linux", - "pebble_optest_ball_matrix_matmul_16x16_i8-linux", - "pebble_matrix_os_4x4_test-linux", - "pebble_matrix_ws_4x4_test-linux", - "pebble_matrix_os_16x16_test-linux", - "pebble_matrix_ws_16x16_test-linux", - "pebble_matrix_os_32x16_test-linux", - "pebble_matrix_ws_32x16_test-linux", - "pebble_matrix_os_16x16_ones-linux", - "pebble_matrix_os_32x32_ones-linux", - "pebble_matrix_os_tail_5x7x3_test-linux", - "pebble_matrix_mode_switch_test-linux", - "pebble_matrix_bank_os_1024x16x16_test-linux", - "pebble_matrix_bank_ws_1024x16x16_test-linux", - "pebble_optest_linalg_matmul_16x16_i8-linux", - "pebble_optest_linalg_transpose_16x16_i8-linux", - "pebble_optest_linalg_conv_2d_nhwc_hwcf_6x6_k3-linux", - "pebble_optest_tile_matmul_16x16_i8-linux", - "pebble_optest_tile_matmul_16x16_f32-linux", - "pebble_optest_tile_matmul_1x256x120_f32-linux", - "pebble_optest_tile_matmul_1x576x1024_f32-linux", - "pebble_optest_tile_matmul_1x1024x1000_f32-linux", - "pebble_optest_tile_matmul_5x3072x768_f32-linux", - "pebble_optest_tile_transpose_16x16_i8-linux", - "pebble_optest_tile_transpose_80x80_f32-linux", - "pebble_optest_tile_transpose_120x256_f32-linux", - - "pebble_optest_tile_transpose_128x128_dynstride_f32-linux", - "pebble_optest_tile_transpose_1024x576_dynstride_f32-linux", - "pebble_optest_tile_transpose_matmul_120x256_f32-linux", - "pebble_optest_tile_conv2d_6x6_k3-linux", - "pebble_optest_tile_conv2d_1x1_c16_oc8-linux", - "pebble_optest_tile_conv2d_1x1_c16_oc24-linux", - "pebble_optest_tile_conv2d_1x1_c240_oc64-linux", - "pebble_optest_tile_conv2d_8x8_1x1_c16_oc24-linux", - "pebble_optest_tile_conv2d_8x8_k3_c2_oc2-linux", - "pebble_optest_tile_conv2d_12x12_k5-linux", - "pebble_optest_tile_conv2d_28x28_k5-linux", - "pebble_optest_bank_im2col_6x6_k3-linux", - "pebble_optest_ball_im2col_6x6_k3-linux", - "pebble_optest_bank_fp2int_4x4-linux", - "pebble_optest_ball_fp2int_4x4-linux", - "pebble_optest_bank_int2fp_4x4-linux", - "pebble_optest_ball_int2fp_4x4-linux", - "pebble_optest_bank_int32_to_int8_2x16-linux", - "pebble_optest_ball_int32_to_int8_2x16-linux", - "pebble_im2col_k1_test-linux", - "pebble_im2col_k3_test-linux", - "pebble_im2col_k3_partial_test-linux", - "pebble_im2col_k3_pad_test-linux", - "pebble_im2col_k5_test-linux", - "pebble_im2col_k7_test-linux", - "pebble_im2col_bank_k1_test-linux", - "pebble_im2col_bank_k3_test-linux", - "pebble_im2col_bank_k5_test-linux", - "pebble_im2col_bank_k7_test-linux", - "pebble_conv_im2col_test-linux", - "pebble_conv_transpose_test-linux", - "pebble_conv_matmul_test-linux", - "pebble_quant_fp32_to_int8_test-linux", - "pebble_quant_im2col_test-linux", - "pebble_quant_transpose_test-linux", - "pebble_quant_gemm_test-linux", - "pebble_quant_int8_to_fp32_test-linux", - "pebble_conv_im2col_bank_test-linux", - "pebble_conv_transpose_bank_test-linux", - "pebble_conv_matmul_bank_test-linux", - "pebble_mvout_fence_loop_test-linux", - "pebble_quant_fp32_to_int8_bank_test-linux", - "pebble_quant_im2col_bank_test-linux", - "pebble_quant_transpose_bank_test-linux", - "pebble_quant_gemm_bank_test-linux", - "pebble_quant_int8_to_fp32_bank_test-linux", - "pebble_fp2int_fp32_to_int32_test-linux", - "pebble_fp2int_fp32_to_int8_test-linux", - "pebble_int2fp_int32_to_fp32_test-linux", - "pebble_int2fp_int8_to_fp32_test-linux", - "pebble_int2fp_int32_to_int8_test-linux", - "pebble_int2fp_int32_to_fp32_col2_test-linux", - "pebble_int2fp_int32_to_fp32_col4_test-linux", - "pebble_fp2int_bank_fp32_to_int32_test-linux", - "pebble_fp2int_bank_fp32_to_int8_test-linux", - "pebble_int2fp_bank_int32_to_fp32_test-linux", - "pebble_int2fp_bank_int8_to_fp32_test-linux", - "pebble_int2fp_bank_int32_to_int8_test-linux", - "pebble_int2fp_bank_int32_to_fp32_col2_test-linux", - "pebble_int2fp_bank_int32_to_fp32_col4_test-linux", + "pebble-pebble-ctest-conv_im2col_bank_test-linux", + "pebble-pebble-ctest-conv_im2col_test-linux", + "pebble-pebble-ctest-conv_transpose_bank_test-linux", + "pebble-pebble-ctest-conv_transpose_test-linux", + "pebble-pebble-ctest-im2col_bank_k1_test-linux", + "pebble-pebble-ctest-im2col_bank_k3_test-linux", + "pebble-pebble-ctest-im2col_bank_k5_test-linux", + "pebble-pebble-ctest-im2col_bank_k7_test-linux", + "pebble-pebble-ctest-im2col_k1_test-linux", + "pebble-pebble-ctest-im2col_k3_pad_test-linux", + "pebble-pebble-ctest-im2col_k3_asym_pad_test-linux", + "pebble-pebble-ctest-im2col_k3_partial_test-linux", + "pebble-pebble-ctest-im2col_k3_test-linux", + "pebble-pebble-ctest-im2col_k5_test-linux", + "pebble-pebble-ctest-im2col_k7_test-linux", + "pebble-pebble-ctest-fp2int_i8_signed_test-linux", + "pebble-pebble-ctest-fp2int_i8_rounding_test-linux", + "pebble-pebble-ctest-fp2int_i8_zero_test-linux", + "pebble-pebble-ctest-fp2int_i8_rows_test-linux", + "pebble-pebble-ctest-fp2int_i8_scale_rows_test-linux", + "pebble-pebble-ctest-int2fp_tensor_values_test-linux", + "pebble-pebble-ctest-int2fp_tensor_groups_test-linux", + "pebble-pebble-ctest-int2fp_tensor_rows_test-linux", + "pebble-pebble-ctest-int2fp_channel_lanes_test-linux", + "pebble-pebble-ctest-int2fp_channel_base_test-linux", + "pebble-pebble-ctest-int2fp_channel_two_rows_test-linux", + "pebble-pebble-mlirtest-ball_fp2int_4x4-linux", + "pebble-pebble-mlirtest-bank_fp2int_4x4-linux", + "pebble-pebble-mlirtest-ball_int2fp_4x4-linux", + "pebble-pebble-mlirtest-bank_int2fp_4x4-linux", + "pebble-pebble-mlirtest-ball_int2fp_channel_4x4-linux", + "pebble-pebble-mlirtest-bank_int2fp_channel_4x4-linux", + "pebble-pebble-ctest-smatmul_16x16_test-linux", + "pebble-pebble-ctest-smatmul_32x16x32_test-linux", + "pebble-pebble-ctest-smatmul_ws_16x16_test-linux", + "pebble-pebble-ctest-smatmul_ws_16x512_test-linux", + "pebble-pebble-mlirtest-bank_smatmul_ws_16x512_i8-linux", + "pebble-pebble-mlirtest-ball_smatmul_16x16_i8-linux", + "pebble-pebble-mlirtest-ball_smatmul_matmul_16x16_i8-linux", + "pebble-pebble-ctest-relu_test-linux", + "pebble-pebble-ctest-relu_bank_256x16_test-linux", + "pebble-pebble-mlirtest-bank_relu_bank_256x16_i32-linux", + "pebble-pebble-mlirtest-linalg_relu_513x17_i32-linux", + "pebble-pebble-ctest-matadd_test-linux", + "pebble-pebble-ctest-matadd_bank_1024x8_i32_test-linux", + "pebble-pebble-mlirtest-bank_matadd_bank_1024x8_i32-linux", + "pebble-pebble-ctest-mvin_mvout_acc_test-linux", + "pebble-pebble-ctest-mvin_mvout_stride_test-linux", + "pebble-pebble-ctest-mvin_mvout_test-linux", + "pebble-pebble-ctest-mvin_mvout_unaligned_test-linux", + "pebble-pebble-ctest-mvin_unaligned_test-linux", + "pebble-pebble-mlirtest-ball_im2col_6x6_k3-linux", + "pebble-pebble-mlirtest-ball_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-bank_im2col_6x6_k3-linux", + "pebble-pebble-mlirtest-bank_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-base_bank_mvin_mvout_16x16_i8-linux", + "pebble-pebble-mlirtest-base_mvin_mvout_16x16_i8-linux", + "pebble-pebble-mlirtest-linalg_conv_2d_nhwc_hwcf_6x6_k3-linux", + "pebble-pebble-mlirtest-linalg_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-tile_conv2d_12x12_k5-linux", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc16-linux", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc24-linux", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc8-linux", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c240_oc64-linux", + "pebble-pebble-mlirtest-tile_conv2d_28x28_k5-linux", + "pebble-pebble-mlirtest-tile_conv2d_6x6_k3-linux", + "pebble-pebble-mlirtest-tile_conv2d_8x8_1x1_c16_oc24-linux", + "pebble-pebble-mlirtest-tile_conv2d_8x8_k3_c2_oc2-linux", + "pebble-pebble-mlirtest-tile_matmul_16x16_f32-linux", + "pebble-pebble-mlirtest-tile_matmul_1x1024x1000_f32-linux", + "pebble-pebble-mlirtest-tile_matmul_1x256x120_f32-linux", + "pebble-pebble-mlirtest-tile_matmul_1x576x1024_f32-linux", + "pebble-pebble-mlirtest-tile_matmul_5x3072x768_f32-linux", + "pebble-pebble-mlirtest-tile_transpose_1024x576_dynstride_f32-linux", + "pebble-pebble-mlirtest-tile_transpose_120x256_f32-linux", + "pebble-pebble-mlirtest-tile_transpose_128x128_dynstride_f32-linux", + "pebble-pebble-mlirtest-tile_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-tile_transpose_80x80_f32-linux", + "pebble-pebble-mlirtest-tile_transpose_matmul_120x256_f32-linux", + "pebble-pebble-ctest-transpose_bank_i32_n4_1024x16_test-linux", + "pebble-pebble-ctest-transpose_bank_i8_n1_1024x16_test-linux", + "pebble-pebble-ctest-transpose_bank_i8_n2_1024x32_test-linux", + "pebble-pebble-ctest-transpose_i32_8x16_test-linux", + "pebble-pebble-ctest-transpose_i8_16x16_test-linux", + "pebble-pebble-ctest-transpose_i8_16x32_test-linux", ] diff --git a/examples/chips/pebble/regression/batch/p2e/workloads-elf.toml b/examples/chips/pebble/regression/batch/p2e/workloads-elf.toml index d7bab0156..22e4ce474 100644 --- a/examples/chips/pebble/regression/batch/p2e/workloads-elf.toml +++ b/examples/chips/pebble/regression/batch/p2e/workloads-elf.toml @@ -1,106 +1,72 @@ [workloads] search_path = "workloads/src" tests = [ - "pebble_mvin_mvout_test-singlecore-baremetal", - "pebble_mvin_mvout_stride_test-singlecore-baremetal", - "pebble_optest_base_mvin_mvout_16x16_i8_singlecore-baremetal", - "pebble_optest_base_bank_mvin_mvout_16x16_i8_singlecore-baremetal", - "pebble_transpose_i8_16x16_test-singlecore-baremetal", - "pebble_transpose_i8_16x32_test-singlecore-baremetal", - "pebble_transpose_i32_8x16_test-singlecore-baremetal", - "pebble_transpose_bank_i8_n1_1024x16_test-singlecore-baremetal", - "pebble_transpose_bank_i8_n2_1024x32_test-singlecore-baremetal", - "pebble_transpose_bank_i32_n4_1024x16_test-singlecore-baremetal", - "pebble_optest_bank_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_bank_matrix_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_matrix_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_matrix_matmul_16x16_i8_singlecore-baremetal", - "pebble_matrix_os_4x4_test-singlecore-baremetal", - "pebble_matrix_ws_4x4_test-singlecore-baremetal", - "pebble_matrix_os_16x16_test-singlecore-baremetal", - "pebble_matrix_ws_16x16_test-singlecore-baremetal", - "pebble_matrix_os_32x16_test-singlecore-baremetal", - "pebble_matrix_ws_32x16_test-singlecore-baremetal", - "pebble_matrix_os_16x16_ones-singlecore-baremetal", - "pebble_matrix_os_32x32_ones-singlecore-baremetal", - "pebble_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "pebble_matrix_mode_switch_test-singlecore-baremetal", - "pebble_matrix_bank_os_1024x16x16_test-singlecore-baremetal", - "pebble_matrix_bank_ws_1024x16x16_test-singlecore-baremetal", - "pebble_optest_linalg_matmul_16x16_i8_singlecore-baremetal", - "pebble_optest_linalg_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_linalg_conv_2d_nhwc_hwcf_6x6_k3_singlecore-baremetal", - "pebble_optest_tile_matmul_16x16_i8_singlecore-baremetal", - "pebble_optest_tile_matmul_16x16_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_1x256x120_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_1x576x1024_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_1x1024x1000_f32_singlecore-baremetal", - "pebble_optest_tile_matmul_5x3072x768_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_tile_transpose_80x80_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_120x256_f32_singlecore-baremetal", + "pebble-pebble-ctest-mvin_mvout_test-baremetal", + "pebble-pebble-ctest-mvin_mvout_stride_test-baremetal", + "pebble-pebble-mlirtest-base_mvin_mvout_16x16_i8-baremetal", + "pebble-pebble-mlirtest-base_bank_mvin_mvout_16x16_i8-baremetal", + "pebble-pebble-ctest-transpose_i8_16x16_test-baremetal", + "pebble-pebble-ctest-transpose_i8_16x32_test-baremetal", + "pebble-pebble-ctest-transpose_i32_8x16_test-baremetal", + "pebble-pebble-ctest-transpose_bank_i8_n1_1024x16_test-baremetal", + "pebble-pebble-ctest-transpose_bank_i8_n2_1024x32_test-baremetal", + "pebble-pebble-ctest-transpose_bank_i32_n4_1024x16_test-baremetal", + "pebble-pebble-mlirtest-bank_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-ball_transpose_16x16_i8-baremetal", + "pebble-pebble-ctest-smatmul_16x16_test-baremetal", + "pebble-pebble-ctest-smatmul_32x16x32_test-baremetal", + "pebble-pebble-ctest-relu_test-baremetal", + "pebble-pebble-mlirtest-linalg_matmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-linalg_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-linalg_conv_2d_nhwc_hwcf_6x6_k3-baremetal", + "pebble-pebble-mlirtest-tile_matmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-tile_matmul_16x16_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_1x256x120_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_1x576x1024_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_1x1024x1000_f32-baremetal", + "pebble-pebble-mlirtest-tile_matmul_5x3072x768_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-tile_transpose_80x80_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_120x256_f32-baremetal", - "pebble_optest_tile_transpose_128x128_dynstride_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_1024x576_dynstride_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_matmul_120x256_f32_singlecore-baremetal", - "pebble_optest_tile_conv2d_6x6_k3_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc8_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc16_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc24_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c240_oc64_singlecore-baremetal", - "pebble_optest_tile_conv2d_8x8_1x1_c16_oc24_singlecore-baremetal", - "pebble_optest_tile_conv2d_8x8_k3_c2_oc2_singlecore-baremetal", - "pebble_optest_tile_conv2d_12x12_k5_singlecore-baremetal", - "pebble_optest_tile_conv2d_28x28_k5_singlecore-baremetal", - "pebble_optest_bank_im2col_6x6_k3_singlecore-baremetal", - "pebble_optest_ball_im2col_6x6_k3_singlecore-baremetal", - "pebble_optest_bank_fp2int_4x4_singlecore-baremetal", - "pebble_optest_ball_fp2int_4x4_singlecore-baremetal", - "pebble_optest_bank_fp2int_im2col_k1_1x4_singlecore-baremetal", - "pebble_optest_bank_int2fp_4x4_singlecore-baremetal", - "pebble_optest_ball_int2fp_4x4_singlecore-baremetal", - "pebble_optest_bank_int32_to_int8_2x16_singlecore-baremetal", - "pebble_optest_ball_int32_to_int8_2x16_singlecore-baremetal", - "pebble_im2col_k1_test-singlecore-baremetal", - "pebble_im2col_k3_test-singlecore-baremetal", - "pebble_im2col_k3_partial_test-singlecore-baremetal", - "pebble_im2col_k3_pad_test-singlecore-baremetal", - "pebble_im2col_k5_test-singlecore-baremetal", - "pebble_im2col_k7_test-singlecore-baremetal", - "pebble_im2col_bank_k1_test-singlecore-baremetal", - "pebble_im2col_bank_k3_test-singlecore-baremetal", - "pebble_im2col_bank_k5_test-singlecore-baremetal", - "pebble_im2col_bank_k7_test-singlecore-baremetal", - "pebble_conv_im2col_test-singlecore-baremetal", - "pebble_conv_transpose_test-singlecore-baremetal", - "pebble_conv_matmul_test-singlecore-baremetal", - "pebble_quant_fp32_to_int8_test-singlecore-baremetal", - "pebble_quant_im2col_test-singlecore-baremetal", - "pebble_quant_transpose_test-singlecore-baremetal", - "pebble_quant_gemm_test-singlecore-baremetal", - "pebble_quant_int8_to_fp32_test-singlecore-baremetal", - "pebble_conv_im2col_bank_test-singlecore-baremetal", - "pebble_conv_transpose_bank_test-singlecore-baremetal", - "pebble_conv_matmul_bank_test-singlecore-baremetal", - "pebble_mvout_fence_loop_test-singlecore-baremetal", - "pebble_quant_fp32_to_int8_bank_test-singlecore-baremetal", - "pebble_quant_im2col_bank_test-singlecore-baremetal", - "pebble_quant_transpose_bank_test-singlecore-baremetal", - "pebble_quant_gemm_bank_test-singlecore-baremetal", - "pebble_quant_int8_to_fp32_bank_test-singlecore-baremetal", - "pebble_fp2int_fp32_to_int32_test-singlecore-baremetal", - "pebble_fp2int_fp32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_int8_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_int32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", - "pebble_fp2int_bank_fp32_to_int32_test-singlecore-baremetal", - "pebble_fp2int_bank_fp32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_bank_int8_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_fp32_col2_test-singlecore-baremetal", - "pebble_int2fp_bank_int32_to_fp32_col4_test-singlecore-baremetal", + "pebble-pebble-mlirtest-tile_transpose_128x128_dynstride_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_1024x576_dynstride_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_matmul_120x256_f32-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_6x6_k3-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc8-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc16-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc24-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c240_oc64-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_8x8_1x1_c16_oc24-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_8x8_k3_c2_oc2-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_12x12_k5-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_28x28_k5-baremetal", + "pebble-pebble-mlirtest-bank_im2col_6x6_k3-baremetal", + "pebble-pebble-mlirtest-ball_im2col_6x6_k3-baremetal", + "pebble-pebble-ctest-im2col_k1_test-baremetal", + "pebble-pebble-ctest-im2col_k3_test-baremetal", + "pebble-pebble-ctest-im2col_k3_partial_test-baremetal", + "pebble-pebble-ctest-im2col_k3_pad_test-baremetal", + "pebble-pebble-ctest-im2col_k5_test-baremetal", + "pebble-pebble-ctest-im2col_k7_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k1_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k3_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k5_test-baremetal", + "pebble-pebble-ctest-im2col_bank_k7_test-baremetal", + "pebble-pebble-ctest-conv_im2col_test-baremetal", + "pebble-pebble-ctest-conv_transpose_test-baremetal", + "pebble-pebble-ctest-conv_im2col_bank_test-baremetal", + "pebble-pebble-ctest-conv_transpose_bank_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_signed_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_rounding_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_zero_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_rows_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_scale_rows_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_values_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_groups_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_rows_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_lanes_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_base_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_two_rows_test-baremetal", + "pebble-pebble-ctest-quant_online_gemm_test-baremetal", ] diff --git a/examples/chips/pebble/regression/batch/verilator/workloads-elf.toml b/examples/chips/pebble/regression/batch/verilator/workloads-elf.toml index 51cf61ef3..155ed0e03 100644 --- a/examples/chips/pebble/regression/batch/verilator/workloads-elf.toml +++ b/examples/chips/pebble/regression/batch/verilator/workloads-elf.toml @@ -1,65 +1,54 @@ [workloads] search_path = "workloads/src" tests = [ - "pebble_mvin_mvout_test-singlecore-baremetal", - "pebble_mvin_mvout_stride_test-singlecore-baremetal", - "pebble_optest_base_mvin_mvout_16x16_i8_singlecore-baremetal", - "pebble_optest_base_bank_mvin_mvout_16x16_i8_singlecore-baremetal", - "pebble_transpose_i8_16x16_test-singlecore-baremetal", - "pebble_transpose_i8_16x32_test-singlecore-baremetal", - "pebble_transpose_i32_8x16_test-singlecore-baremetal", - "pebble_optest_bank_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_bank_matrix_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_matrix_16x16_i8_singlecore-baremetal", - "pebble_optest_ball_matrix_matmul_16x16_i8_singlecore-baremetal", - "pebble_matrix_os_4x4_test-singlecore-baremetal", - "pebble_matrix_ws_4x4_test-singlecore-baremetal", - "pebble_matrix_os_16x16_test-singlecore-baremetal", - "pebble_matrix_ws_16x16_test-singlecore-baremetal", - "pebble_matrix_os_32x16_test-singlecore-baremetal", - "pebble_matrix_ws_32x16_test-singlecore-baremetal", - "pebble_matrix_os_16x16_ones-singlecore-baremetal", - "pebble_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "pebble_matrix_mode_switch_test-singlecore-baremetal", - "pebble_optest_linalg_matmul_16x16_i8_singlecore-baremetal", - "pebble_optest_linalg_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_linalg_conv_2d_nhwc_hwcf_6x6_k3_singlecore-baremetal", - "pebble_optest_tile_matmul_16x16_i8_singlecore-baremetal", - "pebble_optest_tile_matmul_16x16_f32_singlecore-baremetal", - "pebble_optest_tile_transpose_16x16_i8_singlecore-baremetal", - "pebble_optest_tile_conv2d_6x6_k3_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc8_singlecore-baremetal", - "pebble_optest_tile_conv2d_1x1_c16_oc24_singlecore-baremetal", - "pebble_optest_tile_conv2d_8x8_k3_c2_oc2_singlecore-baremetal", - "pebble_optest_tile_conv2d_12x12_k5_singlecore-baremetal", - "pebble_optest_bank_im2col_6x6_k3_singlecore-baremetal", - "pebble_optest_ball_im2col_6x6_k3_singlecore-baremetal", - "pebble_optest_bank_fp2int_4x4_singlecore-baremetal", - "pebble_optest_ball_fp2int_4x4_singlecore-baremetal", - "pebble_optest_bank_int2fp_4x4_singlecore-baremetal", - "pebble_optest_ball_int2fp_4x4_singlecore-baremetal", - "pebble_optest_bank_int32_to_int8_2x16_singlecore-baremetal", - "pebble_optest_ball_int32_to_int8_2x16_singlecore-baremetal", - "pebble_im2col_k1_test-singlecore-baremetal", - "pebble_im2col_k3_test-singlecore-baremetal", - "pebble_im2col_k3_partial_test-singlecore-baremetal", - "pebble_im2col_k3_pad_test-singlecore-baremetal", - "pebble_im2col_k5_test-singlecore-baremetal", - "pebble_im2col_k7_test-singlecore-baremetal", - "pebble_conv_im2col_test-singlecore-baremetal", - "pebble_conv_transpose_test-singlecore-baremetal", - "pebble_conv_matmul_test-singlecore-baremetal", - "pebble_quant_fp32_to_int8_test-singlecore-baremetal", - "pebble_quant_im2col_test-singlecore-baremetal", - "pebble_quant_transpose_test-singlecore-baremetal", - "pebble_quant_gemm_test-singlecore-baremetal", - "pebble_quant_int8_to_fp32_test-singlecore-baremetal", - "pebble_fp2int_fp32_to_int32_test-singlecore-baremetal", - "pebble_fp2int_fp32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_int8_to_fp32_test-singlecore-baremetal", - "pebble_int2fp_int32_to_int8_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "pebble_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", + "pebble-pebble-ctest-mvin_mvout_test-baremetal", + "pebble-pebble-ctest-mvin_mvout_stride_test-baremetal", + "pebble-pebble-mlirtest-base_mvin_mvout_16x16_i8-baremetal", + "pebble-pebble-mlirtest-base_bank_mvin_mvout_16x16_i8-baremetal", + "pebble-pebble-ctest-transpose_i8_16x16_test-baremetal", + "pebble-pebble-ctest-transpose_i8_16x32_test-baremetal", + "pebble-pebble-ctest-transpose_i32_8x16_test-baremetal", + "pebble-pebble-mlirtest-bank_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-ball_transpose_16x16_i8-baremetal", + "pebble-pebble-ctest-smatmul_16x16_test-baremetal", + "pebble-pebble-ctest-smatmul_32x16x32_test-baremetal", + "pebble-pebble-ctest-relu_test-baremetal", + "pebble-pebble-ctest-matadd_test-baremetal", + "pebble-pebble-mlirtest-linalg_matmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-linalg_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-linalg_conv_2d_nhwc_hwcf_6x6_k3-baremetal", + "pebble-pebble-mlirtest-tile_matmul_16x16_i8-baremetal", + "pebble-pebble-mlirtest-tile_matmul_16x16_f32-baremetal", + "pebble-pebble-mlirtest-tile_transpose_16x16_i8-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_6x6_k3-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc8-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc24-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_8x8_k3_c2_oc2-baremetal", + "pebble-pebble-mlirtest-tile_conv2d_12x12_k5-baremetal", + "pebble-pebble-mlirtest-bank_im2col_6x6_k3-baremetal", + "pebble-pebble-mlirtest-ball_im2col_6x6_k3-baremetal", + "pebble-pebble-ctest-im2col_k1_test-baremetal", + "pebble-pebble-ctest-im2col_k3_test-baremetal", + "pebble-pebble-ctest-im2col_k3_partial_test-baremetal", + "pebble-pebble-ctest-im2col_k3_pad_test-baremetal", + "pebble-pebble-ctest-im2col_k3_asym_pad_test-baremetal", + "pebble-pebble-ctest-im2col_k5_test-baremetal", + "pebble-pebble-ctest-im2col_k7_test-baremetal", + "pebble-pebble-ctest-conv_im2col_test-baremetal", + "pebble-pebble-ctest-conv_transpose_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_signed_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_rounding_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_zero_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_rows_test-baremetal", + "pebble-pebble-ctest-fp2int_i8_scale_rows_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_values_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_groups_test-baremetal", + "pebble-pebble-ctest-int2fp_tensor_rows_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_lanes_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_base_test-baremetal", + "pebble-pebble-ctest-int2fp_channel_two_rows_test-baremetal", + "pebble-pebble-ctest-quant_online_gemm_test-baremetal", + "pebble-pebble-mlirtest-ball_fp2int_4x4-baremetal", + "pebble-pebble-mlirtest-ball_int2fp_4x4-baremetal", + "pebble-pebble-mlirtest-ball_int2fp_channel_4x4-baremetal", ] diff --git a/examples/chips/pebble/regression/batch/verilator/workloads-pk.toml b/examples/chips/pebble/regression/batch/verilator/workloads-pk.toml index e6e979625..6b832bd86 100644 --- a/examples/chips/pebble/regression/batch/verilator/workloads-pk.toml +++ b/examples/chips/pebble/regression/batch/verilator/workloads-pk.toml @@ -1,65 +1,50 @@ [workloads] search_path = "workloads/src" tests = [ - "pebble_mvin_mvout_test-linux", - "pebble_mvin_mvout_stride_test-linux", - "pebble_optest_base_mvin_mvout_16x16_i8-linux", - "pebble_optest_base_bank_mvin_mvout_16x16_i8-linux", - "pebble_transpose_i8_16x16_test-linux", - "pebble_transpose_i8_16x32_test-linux", - "pebble_transpose_i32_8x16_test-linux", - "pebble_optest_bank_transpose_16x16_i8-linux", - "pebble_optest_ball_transpose_16x16_i8-linux", - "pebble_optest_bank_matrix_16x16_i8-linux", - "pebble_optest_ball_matrix_16x16_i8-linux", - "pebble_optest_ball_matrix_matmul_16x16_i8-linux", - "pebble_matrix_os_4x4_test-linux", - "pebble_matrix_ws_4x4_test-linux", - "pebble_matrix_os_16x16_test-linux", - "pebble_matrix_ws_16x16_test-linux", - "pebble_matrix_os_32x16_test-linux", - "pebble_matrix_ws_32x16_test-linux", - "pebble_matrix_os_16x16_ones-linux", - "pebble_matrix_os_tail_5x7x3_test-linux", - "pebble_matrix_mode_switch_test-linux", - "pebble_optest_linalg_matmul_16x16_i8-linux", - "pebble_optest_linalg_transpose_16x16_i8-linux", - "pebble_optest_linalg_conv_2d_nhwc_hwcf_6x6_k3-linux", - "pebble_optest_tile_matmul_16x16_i8-linux", - "pebble_optest_tile_matmul_16x16_f32-linux", - "pebble_optest_tile_transpose_16x16_i8-linux", - "pebble_optest_tile_conv2d_6x6_k3-linux", - "pebble_optest_tile_conv2d_1x1_c16_oc8-linux", - "pebble_optest_tile_conv2d_1x1_c16_oc24-linux", - "pebble_optest_tile_conv2d_8x8_k3_c2_oc2-linux", - "pebble_optest_tile_conv2d_12x12_k5-linux", - "pebble_optest_bank_im2col_6x6_k3-linux", - "pebble_optest_ball_im2col_6x6_k3-linux", - "pebble_optest_bank_fp2int_4x4-linux", - "pebble_optest_ball_fp2int_4x4-linux", - "pebble_optest_bank_int2fp_4x4-linux", - "pebble_optest_ball_int2fp_4x4-linux", - "pebble_optest_bank_int32_to_int8_2x16-linux", - "pebble_optest_ball_int32_to_int8_2x16-linux", - "pebble_im2col_k1_test-linux", - "pebble_im2col_k3_test-linux", - "pebble_im2col_k3_partial_test-linux", - "pebble_im2col_k3_pad_test-linux", - "pebble_im2col_k5_test-linux", - "pebble_im2col_k7_test-linux", - "pebble_conv_im2col_test-linux", - "pebble_conv_transpose_test-linux", - "pebble_conv_matmul_test-linux", - "pebble_quant_fp32_to_int8_test-linux", - "pebble_quant_im2col_test-linux", - "pebble_quant_transpose_test-linux", - "pebble_quant_gemm_test-linux", - "pebble_quant_int8_to_fp32_test-linux", - "pebble_fp2int_fp32_to_int32_test-linux", - "pebble_fp2int_fp32_to_int8_test-linux", - "pebble_int2fp_int32_to_fp32_test-linux", - "pebble_int2fp_int8_to_fp32_test-linux", - "pebble_int2fp_int32_to_int8_test-linux", - "pebble_int2fp_int32_to_fp32_col2_test-linux", - "pebble_int2fp_int32_to_fp32_col4_test-linux", + "pebble-pebble-ctest-mvin_mvout_test-linux", + "pebble-pebble-ctest-mvin_mvout_stride_test-linux", + "pebble-pebble-mlirtest-base_mvin_mvout_16x16_i8-linux", + "pebble-pebble-mlirtest-base_bank_mvin_mvout_16x16_i8-linux", + "pebble-pebble-ctest-transpose_i8_16x16_test-linux", + "pebble-pebble-ctest-transpose_i8_16x32_test-linux", + "pebble-pebble-ctest-transpose_i32_8x16_test-linux", + "pebble-pebble-mlirtest-bank_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-ball_transpose_16x16_i8-linux", + "pebble-pebble-ctest-smatmul_16x16_test-linux", + "pebble-pebble-ctest-smatmul_32x16x32_test-linux", + "pebble-pebble-ctest-relu_test-linux", + "pebble-pebble-ctest-matadd_test-linux", + "pebble-pebble-mlirtest-linalg_matmul_16x16_i8-linux", + "pebble-pebble-mlirtest-linalg_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-linalg_conv_2d_nhwc_hwcf_6x6_k3-linux", + "pebble-pebble-mlirtest-tile_matmul_16x16_i8-linux", + "pebble-pebble-mlirtest-tile_matmul_16x16_f32-linux", + "pebble-pebble-mlirtest-tile_transpose_16x16_i8-linux", + "pebble-pebble-mlirtest-tile_conv2d_6x6_k3-linux", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc8-linux", + "pebble-pebble-mlirtest-tile_conv2d_1x1_c16_oc24-linux", + "pebble-pebble-mlirtest-tile_conv2d_8x8_k3_c2_oc2-linux", + "pebble-pebble-mlirtest-tile_conv2d_12x12_k5-linux", + "pebble-pebble-mlirtest-bank_im2col_6x6_k3-linux", + "pebble-pebble-mlirtest-ball_im2col_6x6_k3-linux", + "pebble-pebble-ctest-im2col_k1_test-linux", + "pebble-pebble-ctest-im2col_k3_test-linux", + "pebble-pebble-ctest-im2col_k3_partial_test-linux", + "pebble-pebble-ctest-im2col_k3_pad_test-linux", + "pebble-pebble-ctest-im2col_k5_test-linux", + "pebble-pebble-ctest-im2col_k7_test-linux", + "pebble-pebble-ctest-conv_im2col_test-linux", + "pebble-pebble-ctest-conv_transpose_test-linux", + "pebble-pebble-ctest-fp2int_i8_signed_test-linux", + "pebble-pebble-ctest-fp2int_i8_rounding_test-linux", + "pebble-pebble-ctest-fp2int_i8_zero_test-linux", + "pebble-pebble-ctest-fp2int_i8_rows_test-linux", + "pebble-pebble-ctest-fp2int_i8_scale_rows_test-linux", + "pebble-pebble-ctest-int2fp_tensor_values_test-linux", + "pebble-pebble-ctest-int2fp_tensor_groups_test-linux", + "pebble-pebble-ctest-int2fp_tensor_rows_test-linux", + "pebble-pebble-ctest-int2fp_channel_lanes_test-linux", + "pebble-pebble-ctest-int2fp_channel_base_test-linux", + "pebble-pebble-ctest-int2fp_channel_two_rows_test-linux", + "pebble-pebble-ctest-quant_online_gemm_test-linux", ] diff --git a/examples/chips/pebble/regression/eval/README.md b/examples/chips/pebble/regression/eval/README.md index e69de29bb..6b13db42c 100644 --- a/examples/chips/pebble/regression/eval/README.md +++ b/examples/chips/pebble/regression/eval/README.md @@ -0,0 +1,36 @@ +# pebble regression eval + +`models.toml` lists models for `--eval-performance` and per-model accuracy specs. + +## Accuracy + +Each model hardcodes its dataset relative path under `/root`. Kernel only packs +that directory from the host tree. No CLI flags. + +| model | metric | relative path | +|-------|--------|---------------| +| lenet | top1 | `./datasets/MNIST` | +| mobilenet | top1 | `./datasets/ImageNet` | +| resnet | top1 | `./datasets/ImageNet` | +| yolo | map | `./datasets/COCO` | + +Host source (copied to `/root/datasets/...`): + +```text +bb-tests/workloads/src/ModelTest/e2e/datasets/ + MNIST/raw/t10k-*-ubyte + ImageNet/labels.csv # relpath,class_id + ImageNet/images/... + COCO/boxes.csv # file,class_id,x1,y1,x2,y2 + COCO/images/... +``` + +Sample count = files/entries in that directory. Put a subset there for faster +eval; put the full set for full numbers. + +UART: + +- top1: `top1=/` +- map: `map=<0..1>` + +Top-level `accuracy` = mean of per-model accuracies. diff --git a/examples/chips/pebble/regression/eval/models.toml b/examples/chips/pebble/regression/eval/models.toml new file mode 100644 index 000000000..45960249b --- /dev/null +++ b/examples/chips/pebble/regression/eval/models.toml @@ -0,0 +1,22 @@ +[eval] +models = ["lenet", "mobilenet", "resnet", "yolo"] + +# dataset: path relative to /root (host: e2e/datasets/). +# metric: top1 -> uart "top1=c/n"; map -> uart "map=" +# Sample count = whatever is in that relative dataset dir. + +[eval.accuracy.lenet] +metric = "top1" +dataset = "datasets/MNIST" + +[eval.accuracy.mobilenet] +metric = "top1" +dataset = "datasets/ImageNet" + +[eval.accuracy.resnet] +metric = "top1" +dataset = "datasets/ImageNet" + +[eval.accuracy.yolo] +metric = "map" +dataset = "datasets/COCO" diff --git a/examples/chips/pebble/workloads/CMakeLists.txt b/examples/chips/pebble/workloads/CMakeLists.txt index 4b22e33fb..04095e6c6 100644 --- a/examples/chips/pebble/workloads/CMakeLists.txt +++ b/examples/chips/pebble/workloads/CMakeLists.txt @@ -1,33 +1,26 @@ get_filename_component(BUCKYBALL_REPO_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../../../.. ABSOLUTE) -set(BUCKYBALL_TOY_COMMON_DIR - ${BUCKYBALL_REPO_ROOT}/examples/chips/toy/workloads/common) set(BUCKYBALL_CHIP_COMMON_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/common/pebble.c) set(BUCKYBALL_CHIP_COMMON_HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/common/pebble.h) set(BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/common) -include(${BUCKYBALL_REPO_ROOT}/examples/chips/toy/workloads/cmake/BuckyballWorkloads.cmake) +set(BUCKYBALL_CTEST_TARGET pebble) add_custom_target(pebble-ctests-build) set(BUCKYBALL_CTEST_GROUP_TARGET pebble-ctests-build) add_subdirectory(ctests) add_subdirectory(mlir_tests) +buckyball_add_ball_workload_subdirs(ctests) -set(BUCKYBALL_BALL_CTEST_DIRS - ${BUCKYBALL_REPO_ROOT}/examples/balls/transpose/workloads/ctests - ${BUCKYBALL_REPO_ROOT}/examples/balls/matrix/workloads/ctests - ${BUCKYBALL_REPO_ROOT}/examples/balls/im2col/workloads/ctests - ${BUCKYBALL_REPO_ROOT}/examples/balls/fp2int/workloads/ctests - ${BUCKYBALL_REPO_ROOT}/examples/balls/int2fp/workloads/ctests -) -buckyball_add_ball_ctest_subdirs() +add_custom_target(chip-ctests-build DEPENDS pebble-ctests-build) +add_custom_target(chip-mlirtests-build DEPENDS pebble-mlir-tests-build) add_custom_target(chip-workloads-build ALL DEPENDS - pebble-ctests-build - pebble-mlir-tests-build + chip-ctests-build + chip-mlirtests-build COMMENT "Building pebble workloads" VERBATIM) diff --git a/examples/chips/pebble/workloads/cmake/RushB.cmake b/examples/chips/pebble/workloads/cmake/RushB.cmake deleted file mode 100644 index dce3c156a..000000000 --- a/examples/chips/pebble/workloads/cmake/RushB.cmake +++ /dev/null @@ -1,8 +0,0 @@ -# Pebble's host-native rushB backends. Model CMake files consume these names -# through the chip-neutral BUCKYBALL_RUSHB_* interface. -set(BUCKYBALL_RUSHB_BEMU_MANIFEST - ${BUCKYBALL_REPO_ROOT}/examples/cores/pebble/emu/Cargo.toml) -set(BUCKYBALL_RUSHB_BEMU_LIBRARY - ${BUCKYBALL_REPO_ROOT}/examples/cores/pebble/emu/target/release/libbebop_bemu.so) -set(BUCKYBALL_RUSHB_VERILATOR_LIBRARY - ${BUCKYBALL_REPO_ROOT}/bebop/target/release/deps/libbebop_verilator.so) diff --git a/examples/chips/pebble/workloads/ctests/CMakeLists.txt b/examples/chips/pebble/workloads/ctests/CMakeLists.txt index e784cb962..6ecac2cc1 100644 --- a/examples/chips/pebble/workloads/ctests/CMakeLists.txt +++ b/examples/chips/pebble/workloads/ctests/CMakeLists.txt @@ -1,19 +1,20 @@ add_buckyball_ctests( conv_im2col_test.c conv_transpose_test.c - conv_matmul_test.c - quant_fp32_to_int8_test.c - quant_im2col_test.c - quant_transpose_test.c - quant_gemm_test.c - quant_int8_to_fp32_test.c conv_im2col_bank_test.c conv_transpose_bank_test.c + conv_matmul_test.c conv_matmul_bank_test.c + mvout_fence_loop_test.c + quant_online_gemm_test.c + quant_gemm_test.c + quant_gemm_bank_test.c + quant_fp32_to_int8_test.c quant_fp32_to_int8_bank_test.c + quant_int8_to_fp32_test.c + quant_int8_to_fp32_bank_test.c + quant_im2col_test.c quant_im2col_bank_test.c + quant_transpose_test.c quant_transpose_bank_test.c - quant_gemm_bank_test.c - quant_int8_to_fp32_bank_test.c - mvout_fence_loop_test.c ) diff --git a/examples/chips/pebble/workloads/ctests/conv_im2col_bank_test.c b/examples/chips/pebble/workloads/ctests/conv_im2col_bank_test.c index 60fe5fbb8..efcd3f1fa 100644 --- a/examples/chips/pebble/workloads/ctests/conv_im2col_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/conv_im2col_bank_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include /* 34x34 k3 -> exactly one bank of im2col rows. */ @@ -56,6 +57,8 @@ int main(void) { bb_im2col(0, 1, ITER, K, STRIDE, PAD); bb_mvout((uintptr_t)out, 1, OUT_ROWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int ok = compare_i8_matrices(out, exp, OUT_ROWS, LANES); printf("conv_im2col_bank %s\n", ok ? "PASS" : "FAIL"); return ok ? 0 : 1; diff --git a/examples/chips/pebble/workloads/ctests/conv_im2col_test.c b/examples/chips/pebble/workloads/ctests/conv_im2col_test.c index fd974d3e2..b948a94c2 100644 --- a/examples/chips/pebble/workloads/ctests/conv_im2col_test.c +++ b/examples/chips/pebble/workloads/ctests/conv_im2col_test.c @@ -1,6 +1,7 @@ #include "pebble.h" #include #include +#include #include #ifndef TEST_CASE_NAME @@ -57,6 +58,8 @@ int main(void) { bb_im2col(0, 1, t->input_h, t->kernel_h, t->stride, t->padding); bb_mvout((uintptr_t)actual, 1, rows, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int row = 0; row < m_tiles * PEBBLE_INT8_LANES; ++row) { diff --git a/examples/chips/pebble/workloads/ctests/conv_matmul_bank_test.c b/examples/chips/pebble/workloads/ctests/conv_matmul_bank_test.c index 467868d2e..e5f7727f7 100644 --- a/examples/chips/pebble/workloads/ctests/conv_matmul_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/conv_matmul_bank_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include /* One bank of A rows: M=1024, N=1, K=16. */ @@ -43,9 +44,12 @@ int main(void) { bb_mvin((uintptr_t)a, 0, M, 1); bb_mvin((uintptr_t)b, 1, K, 1); bb_mvin((uintptr_t)zero, 2, M, 1); - bb_matrix_mnk(0, 1, 2, M, N, K); + bb_smatmul_ws(0, 1, 2, M, N, K); bb_mvout((uintptr_t)actual, 2, M, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); + bb_mem_release(2); int passed = 1; for (int i = 0; i < M; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/conv_matmul_test.c b/examples/chips/pebble/workloads/ctests/conv_matmul_test.c index f4ea1a01d..63e342dfb 100644 --- a/examples/chips/pebble/workloads/ctests/conv_matmul_test.c +++ b/examples/chips/pebble/workloads/ctests/conv_matmul_test.c @@ -1,6 +1,7 @@ #include "pebble.h" #include #include +#include #include #ifndef TEST_CASE_NAME @@ -62,9 +63,12 @@ int main(void) { bb_mem_alloc(2, 1, 4); bb_mvin((uintptr_t)matrix_a, 0, a_rows, 1); bb_mvin((uintptr_t)matrix_b, 1, k_elems, 1); - bb_matrix_mnk(0, 1, 2, wins, 1, k_elems); + bb_smatmul_os(0, 1, 2, wins, 1, k_elems); bb_mvout((uintptr_t)packed_c, 2, wins, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); + bb_mem_release(2); int passed = 1; for (int i = 0; i < wins; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/conv_transpose_bank_test.c b/examples/chips/pebble/workloads/ctests/conv_transpose_bank_test.c index 37bd08d32..b90c62477 100644 --- a/examples/chips/pebble/workloads/ctests/conv_transpose_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/conv_transpose_bank_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include #define ROWS BANK_LINES @@ -26,6 +27,8 @@ int main(void) { bb_transpose(0, 1, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, 1, ROWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int ok = compare_i8_matrices(output_matrix, expected_matrix, COLS, ROWS); printf("conv_transpose_bank %s\n", ok ? "PASS" : "FAIL"); return ok ? 0 : 1; diff --git a/examples/chips/pebble/workloads/ctests/conv_transpose_test.c b/examples/chips/pebble/workloads/ctests/conv_transpose_test.c index 1d3fc172a..5e3287eb8 100644 --- a/examples/chips/pebble/workloads/ctests/conv_transpose_test.c +++ b/examples/chips/pebble/workloads/ctests/conv_transpose_test.c @@ -1,6 +1,7 @@ #include "pebble.h" #include #include +#include #include #ifndef TEST_CASE_NAME @@ -42,6 +43,8 @@ int main(void) { bb_transpose(0, 1, PEBBLE_INT8_LANES, 8); bb_mvout((uintptr_t)actual, 1, k_elems, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int row = 0; row < k_elems; ++row) { diff --git a/examples/chips/pebble/workloads/ctests/mvout_fence_loop_test.c b/examples/chips/pebble/workloads/ctests/mvout_fence_loop_test.c index 829eae3f2..a955d4d65 100644 --- a/examples/chips/pebble/workloads/ctests/mvout_fence_loop_test.c +++ b/examples/chips/pebble/workloads/ctests/mvout_fence_loop_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include /* Closer to MobileNet tile epilogue: compute, mvout, fence, bank @@ -39,7 +40,7 @@ int main(void) { bb_mvin((uintptr_t)a, 0, M, 1); bb_mvin((uintptr_t)b, 1, K, 1); bb_mvin((uintptr_t)zero, 2, M, 1); - bb_matrix_mnk(0, 1, 2, M, N, K); + bb_smatmul_os(0, 1, 2, M, N, K); bb_mvout((uintptr_t)out, 2, M, 1); bb_fence(); diff --git a/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_bank_test.c b/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_bank_test.c index 5864f4ecc..96eb2e36a 100644 --- a/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_bank_test.c @@ -2,12 +2,12 @@ #include #include #include +#include #include #include #define LANES 16 #define NELEM (BANK_LINES * LANES) -#define SCALE 0x3F800000U /* 1.0f */ #define SEED 0xC5 static float input_fp32[NELEM] __attribute__((aligned(64))); @@ -22,15 +22,34 @@ int main(void) { for (int i = 0; i < NELEM; ++i) { int v = (rand() % 201) - 100; /* integer-valued float in [-100,100] */ input_fp32[i] = (float)v; - expected[i] = (int8_t)v; actual[i] = (int8_t)0x5a; } + int max_abs = 0; + for (int i = 0; i < NELEM; ++i) { + int value = (int)input_fp32[i]; + int magnitude = value < 0 ? -value : value; + if (magnitude > max_abs) + max_abs = magnitude; + } + for (int i = 0; i < NELEM; ++i) { + int value = (int)input_fp32[i]; + int magnitude = value < 0 ? -value : value; + int numerator = magnitude * 127; + int quantized = numerator / max_abs; + int remainder = numerator % max_abs; + if (2 * remainder > max_abs || + (2 * remainder == max_abs && (quantized & 1))) + ++quantized; + expected[i] = (int8_t)(value < 0 ? -quantized : quantized); + } bb_mem_alloc(0, BANK_LINES, 4); bb_mem_alloc(1, BANK_LINES, 1); bb_mvin((uintptr_t)input_fp32, 0, BANK_LINES, 1); - bb_fp2int(0, 1, BANK_LINES, SCALE); + bb_fp2int(0, 1, BANK_LINES, 0); bb_mvout((uintptr_t)actual, 1, BANK_LINES, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int i = 0; i < NELEM; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_test.c b/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_test.c index 55320ec94..bb54a5f87 100644 --- a/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_fp32_to_int8_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define INPUT_N 36 @@ -15,8 +16,9 @@ static const float input_fp32[INPUT_N] = { }; static const int8_t expected[INPUT_N] = { - 1, 3, 5, 7, 9, 11, 3, 1, -1, -3, -5, -7, 5, -1, -5, 3, -3, 7, - 7, -3, 3, -5, 1, -3, 9, -5, -3, 1, 5, -1, 11, -7, 7, -3, -1, 3, + 12, 35, 58, 81, 104, 127, 35, 12, -12, -35, -58, -81, + 58, -12, -58, 35, -35, 81, 81, -35, 35, -58, 12, -35, + 104, -58, -35, 12, 58, -12, 127, -81, 81, -35, -12, 35, }; static float packed[ROWS * 16] __attribute__((aligned(64))); @@ -36,9 +38,11 @@ int main(void) { bb_mem_alloc(0, 1, 4); bb_mem_alloc(1, 1, 1); bb_mvin((uintptr_t)packed, 0, ROWS, 1); - bb_fp2int(0, 1, ROWS, SCALE); + bb_fp2int(0, 1, ROWS, 0); bb_mvout((uintptr_t)actual, 1, ROWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int i = 0; i < INPUT_N; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/quant_gemm_bank_test.c b/examples/chips/pebble/workloads/ctests/quant_gemm_bank_test.c index 749134987..5c3c64c1a 100644 --- a/examples/chips/pebble/workloads/ctests/quant_gemm_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_gemm_bank_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include #define M BANK_LINES @@ -43,9 +44,12 @@ int main(void) { bb_mvin((uintptr_t)a, 0, M, 1); bb_mvin((uintptr_t)b, 1, K, 1); bb_mvin((uintptr_t)zero, 2, M, 1); - bb_matrix_mnk(0, 1, 2, M, N, K); + bb_smatmul_ws(0, 1, 2, M, N, K); bb_mvout((uintptr_t)actual, 2, M, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); + bb_mem_release(2); int passed = 1; for (int i = 0; i < M; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/quant_gemm_test.c b/examples/chips/pebble/workloads/ctests/quant_gemm_test.c index 8630132eb..d1e39291e 100644 --- a/examples/chips/pebble/workloads/ctests/quant_gemm_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_gemm_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define NUM_WINDOWS 4 @@ -36,9 +37,12 @@ int main(void) { bb_mvin((uintptr_t)matrix_a, 0, NUM_WINDOWS, 1); bb_mvin((uintptr_t)matrix_b, 1, KERNEL_ELEMS, 1); bb_mvin((uintptr_t)zero, 2, NUM_WINDOWS, 1); - bb_matrix_mnk(0, 1, 2, NUM_WINDOWS, 1, KERNEL_ELEMS); + bb_smatmul_os(0, 1, 2, NUM_WINDOWS, 1, KERNEL_ELEMS); bb_mvout((uintptr_t)actual, 2, NUM_WINDOWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); + bb_mem_release(2); int passed = 1; for (int i = 0; i < NUM_WINDOWS; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/quant_im2col_bank_test.c b/examples/chips/pebble/workloads/ctests/quant_im2col_bank_test.c index b45fa41ff..b7952973e 100644 --- a/examples/chips/pebble/workloads/ctests/quant_im2col_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_im2col_bank_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include /* Same bank-fill shape as conv_im2col_bank; different seed. */ @@ -56,6 +57,8 @@ int main(void) { bb_im2col(0, 1, ITER, K, STRIDE, PAD); bb_mvout((uintptr_t)out, 1, OUT_ROWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int ok = compare_i8_matrices(out, exp, OUT_ROWS, LANES); printf("quant_im2col_bank %s\n", ok ? "PASS" : "FAIL"); return ok ? 0 : 1; diff --git a/examples/chips/pebble/workloads/ctests/quant_im2col_test.c b/examples/chips/pebble/workloads/ctests/quant_im2col_test.c index bac2a3b8a..c33b35e54 100644 --- a/examples/chips/pebble/workloads/ctests/quant_im2col_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_im2col_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include /* 4x4 input, 3x3 kernel -> 2x2 windows; keeps golden small. */ @@ -40,6 +41,8 @@ int main(void) { bb_im2col(0, 1, INPUT_H, KERNEL_H, STRIDE, PADDING); bb_mvout((uintptr_t)actual, 1, NUM_WINDOWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int row = 0; row < NUM_WINDOWS; ++row) { diff --git a/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_bank_test.c b/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_bank_test.c index 937471693..3bd803824 100644 --- a/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_bank_test.c @@ -2,18 +2,22 @@ #include #include #include +#include #include #include #include #define LANES 16 #define NELEM (BANK_LINES * LANES) -#define SCALE 0x3F800000U /* 1.0f */ +#define DA 1.0f +#define DW 1.0f #define SEED 0xCA -static int8_t input_i8[NELEM] __attribute__((aligned(64))); +static int32_t input_i32[NELEM] __attribute__((aligned(64))); static float actual[NELEM] __attribute__((aligned(64))); static float expected[NELEM]; +static const float da_scale = DA; +static const float dw_scale = DW; int main(void) { #ifdef MULTICORE @@ -22,16 +26,20 @@ int main(void) { srand(SEED); for (int i = 0; i < NELEM; ++i) { int8_t v = (int8_t)((rand() % 256) - 128); - input_i8[i] = v; + input_i32[i] = v; expected[i] = (float)v; actual[i] = 0.0f; } + bb_mvin_mmio((uintptr_t)(&da_scale), 0, 1, 4); + bb_mvin_mmio((uintptr_t)(&dw_scale), 16, 1, 4); bb_mem_alloc(0, BANK_LINES, 1); bb_mem_alloc(1, BANK_LINES, 4); - bb_mvin((uintptr_t)input_i8, 0, BANK_LINES, 1); - bb_int2fp(0, 1, BANK_LINES, SCALE); + bb_mvin((uintptr_t)input_i32, 0, BANK_LINES, 1); + bb_int2fp_tensor(0, 1, BANK_LINES, 0, 16); bb_mvout((uintptr_t)actual, 1, BANK_LINES, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int i = 0; i < NELEM; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_test.c b/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_test.c index 3bab2bac3..1a80cf32f 100644 --- a/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_int8_to_fp32_test.c @@ -1,13 +1,15 @@ #include "buckyball.h" #include #include +#include #include #include #define N 16 -#define SCALE 0x3E800000U /* 0.25f */ +#define DA 1.0f +#define DW 0.25f -static const int8_t input[N] __attribute__((aligned(64))) = { +static const int32_t input[N] __attribute__((aligned(64))) = { 7, 0, 1, 0, 14, -1, 1, -3, 15, -3, -3, -3, 16, -7, -3, -2, }; static const float expected[N] = { @@ -15,6 +17,8 @@ static const float expected[N] = { 3.75f, -0.75f, -0.75f, -0.75f, 4.00f, -1.75f, -0.75f, -0.50f, }; static float actual[N] __attribute__((aligned(64))); +static const float da_scale = DA; +static const float dw_scale = DW; int main(void) { #ifdef MULTICORE @@ -22,12 +26,16 @@ int main(void) { #endif bb_dma_touch(actual, sizeof(actual)); + bb_mvin_mmio((uintptr_t)(&da_scale), 0, 1, 4); + bb_mvin_mmio((uintptr_t)(&dw_scale), 16, 1, 4); bb_mem_alloc(0, 1, 1); bb_mem_alloc(1, 1, 4); bb_mvin((uintptr_t)input, 0, 1, 1); - bb_int2fp(0, 1, 1, SCALE); + bb_int2fp_tensor(0, 1, 1, 0, 16); bb_mvout((uintptr_t)actual, 1, 1, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int i = 0; i < N; ++i) { diff --git a/examples/chips/pebble/workloads/ctests/quant_online_gemm_test.c b/examples/chips/pebble/workloads/ctests/quant_online_gemm_test.c new file mode 100644 index 000000000..0421e69e3 --- /dev/null +++ b/examples/chips/pebble/workloads/ctests/quant_online_gemm_test.c @@ -0,0 +1,68 @@ +#include "buckyball.h" +#include +#include +#include +#include +#include +#include +#include +#include + +#define TILE_ROWS 32 +#define TILE_COLS 16 +#define OUTPUT_ROUNDS 2 + +static float activation[TILE_ROWS * TILE_COLS] __attribute__((aligned(64))); +static int8_t weight[16 * 16] __attribute__((aligned(64))); +static int32_t zero[TILE_ROWS * TILE_COLS] __attribute__((aligned(64))); +static float output[TILE_ROWS * TILE_COLS] __attribute__((aligned(64))); +static float dw[4] __attribute__((aligned(64))) = {0.25f, 0.0f, 0.0f, 0.0f}; + +static uint32_t f32_bits(float value) { + union { + float f; + uint32_t u; + } bits = {.f = value}; + return bits.u; +} + +int main(void) { +#ifdef MULTICORE + multicore(MULTICORE); +#endif + for (int i = 0; i < TILE_ROWS * TILE_COLS; ++i) + activation[i] = 8.0f; + for (int i = 0; i < 16 * 16; ++i) + weight[i] = 0; + for (int k = 0; k < 16; ++k) + weight[k * 16] = 1; + bb_mvin_mmio((uintptr_t)dw, 16, 1, 4); + bb_mem_alloc(0, TILE_ROWS, 4); + bb_mem_alloc(1, TILE_ROWS, 1); + bb_mem_alloc(2, TILE_COLS, 1); + bb_mem_alloc(3, TILE_ROWS * OUTPUT_ROUNDS, 2); + bb_mem_alloc(4, TILE_ROWS * OUTPUT_ROUNDS, 2); + bb_mvin((uintptr_t)activation, 0, TILE_ROWS, 1); + bb_mvin((uintptr_t)weight, 2, TILE_COLS, 1); + bb_mvin((uintptr_t)zero, 3, TILE_ROWS * OUTPUT_ROUNDS, 1); + bb_fp2int(0, 1, TILE_ROWS, 0); + bb_smatmul_os(1, 2, 3, TILE_ROWS, TILE_COLS, TILE_COLS); + bb_int2fp_tensor(3, 4, TILE_ROWS * OUTPUT_ROUNDS, 0, 16); + bb_mvout((uintptr_t)output, 4, TILE_ROWS * OUTPUT_ROUNDS, 1); + bb_fence(); + int ok = 1; + for (int row = 0; row < TILE_ROWS; ++row) { + if (fabsf(output[row * TILE_COLS] - 32.0f) > 1e-5f) { + printf("quant_online_gemm row=%d expected=0x42000000 actual=0x%08x\n", + row, f32_bits(output[row * TILE_COLS])); + ok = 0; + } + } + bb_mem_release(0); + bb_mem_release(1); + bb_mem_release(2); + bb_mem_release(3); + bb_mem_release(4); + printf("quant_online_gemm %s\n", ok ? "PASS" : "FAIL"); + return ok ? 0 : 1; +} diff --git a/examples/chips/pebble/workloads/ctests/quant_transpose_bank_test.c b/examples/chips/pebble/workloads/ctests/quant_transpose_bank_test.c index 49f9d0778..2b613996b 100644 --- a/examples/chips/pebble/workloads/ctests/quant_transpose_bank_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_transpose_bank_test.c @@ -2,6 +2,7 @@ #include #include #include +#include #include #define ROWS BANK_LINES @@ -26,6 +27,8 @@ int main(void) { bb_transpose(0, 1, ROWS, ELEM_BITS); bb_mvout((uintptr_t)output_matrix, 1, ROWS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int ok = compare_i8_matrices(output_matrix, expected_matrix, COLS, ROWS); printf("quant_transpose_bank %s\n", ok ? "PASS" : "FAIL"); return ok ? 0 : 1; diff --git a/examples/chips/pebble/workloads/ctests/quant_transpose_test.c b/examples/chips/pebble/workloads/ctests/quant_transpose_test.c index 66f9cba59..0e123bee5 100644 --- a/examples/chips/pebble/workloads/ctests/quant_transpose_test.c +++ b/examples/chips/pebble/workloads/ctests/quant_transpose_test.c @@ -1,6 +1,7 @@ #include "buckyball.h" #include #include +#include #include #define KERNEL_ELEMS 9 @@ -27,6 +28,8 @@ int main(void) { bb_transpose(0, 1, LANES, 8); bb_mvout((uintptr_t)actual, 1, KERNEL_ELEMS, 1); bb_fence(); + bb_mem_release(0); + bb_mem_release(1); int passed = 1; for (int row = 0; row < KERNEL_ELEMS; ++row) { diff --git a/examples/chips/pebble/workloads/mlir_tests/bank/CMakeLists.txt b/examples/chips/pebble/workloads/mlir_tests/bank/CMakeLists.txt index 8a4c692c1..302202a77 100644 --- a/examples/chips/pebble/workloads/mlir_tests/bank/CMakeLists.txt +++ b/examples/chips/pebble/workloads/mlir_tests/bank/CMakeLists.txt @@ -1,6 +1,4 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET pebble-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX bank) -add_buckyball_mlir_optest(fp2int_im2col_k1_1x4) +add_buckyball_mlir_test(fp2int_im2col_k1_1x4 TARGET pebble) diff --git a/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4.mlir b/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4.mlir index 435f582c1..268d4f1e2 100644 --- a/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4.mlir +++ b/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4.mlir @@ -1,10 +1,11 @@ // Chip integration: MobileNet tile0-like bank SSA chain // (fp2int col=4 -> im2col k=1 -> mvout). Not a Ball unit test. -func.func private @check_result(memref<16x16xi32>) -> () +func.func private @check_result(memref<16x16xi8>) -> () -memref.global "private" constant @fp_in : memref<1x4xf32> = - dense<[[1.0, 2.0, 3.0, 4.0]]> +memref.global "private" constant @fp_in : memref<1x16xf32> = + dense<[[1.0, 2.0, 3.0, 4.0, 0.0, 0.0, 0.0, 0.0, + 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]> func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 @@ -12,29 +13,29 @@ func.func @main() -> i8 { %depth_out = arith.constant 16 : i64 %stride = arith.constant 1 : i64 %iter = arith.constant 1 : i64 - %scale = arith.constant 1065353216 : i64 + %da_addr = arith.constant 0 : i64 %ksize = arith.constant 1 : i64 %pad = arith.constant 0 : i64 - %input = memref.get_global @fp_in : memref<1x4xf32> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi32> + %input = memref.get_global @fp_in : memref<1x16xf32> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> - %bin = buckyball.bank_alloc + %bin = buckyball.bank_alloc {col = 4 : i64} %bq = buckyball.bank_alloc %bout = buckyball.bank_alloc %loaded = buckyball.bank_mvin %input %bin %depth_in %stride - : memref<1x4xf32> i64 i64 i64 - %q = buckyball.bank_fp2int %loaded %bq %iter %scale + : memref<1x16xf32> i64 i64 i64 + %q = buckyball.bank_fp2int %loaded %bq %iter %da_addr : i64 i64 i64 i64 %im = buckyball.bank_im2col %q %bout %iter %ksize %stride %pad : i64 i64 i64 i64 i64 i64 %stored = buckyball.bank_mvout %output %im %depth_out %stride - : memref<16x16xi32> i64 i64 i64 + : memref<16x16xi8> i64 i64 i64 buckyball.bank_release %loaded : i64 buckyball.bank_release %q : i64 buckyball.bank_release %stored : i64 - func.call @check_result(%output) : (memref<16x16xi32>) -> () - memref.dealloc %output : memref<16x16xi32> + func.call @check_result(%output) : (memref<16x16xi8>) -> () + memref.dealloc %output : memref<16x16xi8> return %zero_i8 : i8 } diff --git a/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4_main.cpp b/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4_main.cpp index 00102a7b0..919c6e330 100644 --- a/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4_main.cpp +++ b/examples/chips/pebble/workloads/mlir_tests/bank/fp2int_im2col_k1_1x4_main.cpp @@ -16,7 +16,7 @@ static void fail(void) { #ifdef __cplusplus extern "C" #endif - void check_result(int32_t *allocated, int32_t *aligned, int64_t offset, + void check_result(int8_t *allocated, int8_t *aligned, int64_t offset, int64_t size0, int64_t size1, int64_t stride0, int64_t stride1) { (void)allocated; @@ -26,10 +26,10 @@ extern "C" fail(); } - int32_t *out = aligned + offset; + int8_t *out = aligned + offset; int32_t got = out[0 * stride0 + 0 * stride1]; - if (got != 1) { - printf("FAILED: bank_fp2int_im2col_k1_1x4 out[0][0] exp=1 got=%d\n", + if (got != 32) { + printf("FAILED: bank_fp2int_im2col_k1_1x4 out[0][0] exp=32 got=%d\n", (int)got); fail(); } diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/CMakeLists.txt b/examples/chips/pebble/workloads/mlir_tests/linalg/CMakeLists.txt index 76b0138c9..8174b5784 100644 --- a/examples/chips/pebble/workloads/mlir_tests/linalg/CMakeLists.txt +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/CMakeLists.txt @@ -1,14 +1,80 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET pebble-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX linalg) -add_buckyball_mlir_optest(linalg_matmul_16x16_i8 +add_buckyball_mlir_contract(quant_matmul_f32_i8_contract + TARGET pebble + PASSES + -convert-tile-to-buckyball + -lower-buckyball-to-bank-ssa + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) + +add_buckyball_mlir_contract(quant_matmul_f32_i8_edge_contract + TARGET pebble + PASSES + -convert-tile-to-buckyball + -lower-buckyball-to-bank-ssa + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) + +add_buckyball_mlir_contract(quant_matmul_f32_i8_tensor_contract + TARGET pebble + PASSES + -convert-tile-to-buckyball + -lower-buckyball-to-bank-ssa + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) + +add_buckyball_mlir_contract(quantized_generic_f32_i8_contract + TARGET pebble + PASSES + -convert-linalg-to-tile + -convert-tile-to-buckyball + -lower-buckyball-to-bank-ssa + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) + +add_buckyball_mlir_contract(linalg_relu_513x17_i32_contract + TARGET pebble + PASSES + -convert-linalg-to-tile + -convert-tile-to-buckyball + -lower-buckyball-to-bank-ssa + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + ${BUCKYBALL_LOWER_BUCKYBALL}) + +add_buckyball_mlir_test(linalg_relu_513x17_i32 + TARGET pebble + PASSES + -convert-linalg-to-tile + -convert-tile-to-buckyball + -lower-buckyball-to-bank-ssa + -assign-physical-banks + ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} + -convert-linalg-to-loops + -expand-strided-metadata + -lower-affine + -convert-scf-to-cf + -convert-cf-to-llvm + ${BUCKYBALL_LOWER_BUCKYBALL} + -convert-arith-to-llvm + -convert-math-to-llvm + -finalize-memref-to-llvm + -convert-func-to-llvm + -reconcile-unrealized-casts) + +add_buckyball_mlir_test(linalg_matmul_16x16_i8 + TARGET pebble PASSES -convert-linalg-to-tile -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -22,12 +88,13 @@ add_buckyball_mlir_optest(linalg_matmul_16x16_i8 -convert-func-to-llvm -reconcile-unrealized-casts) -add_buckyball_mlir_optest(linalg_transpose_16x16_i8 +add_buckyball_mlir_test(linalg_transpose_16x16_i8 + TARGET pebble PASSES -convert-linalg-to-tile -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -41,12 +108,13 @@ add_buckyball_mlir_optest(linalg_transpose_16x16_i8 -convert-func-to-llvm -reconcile-unrealized-casts) -add_buckyball_mlir_optest(linalg_conv_2d_nhwc_hwcf_6x6_k3 +add_buckyball_mlir_test(linalg_conv_2d_nhwc_hwcf_6x6_k3 + TARGET pebble PASSES -convert-linalg-to-tile -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_matmul_16x16_i8.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_matmul_16x16_i8.mlir index f415f5d15..ddc417e97 100644 --- a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_matmul_16x16_i8.mlir +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_matmul_16x16_i8.mlir @@ -4,9 +4,9 @@ func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 %one_i8 = arith.constant 1 : i8 %zero_i32 = arith.constant 0 : i32 - %a = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %b = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %c = memref.alloc() {alignment = 64 : i64} : memref<16x16xi32> + %a = memref.alloc() alignment = 64 : memref<16x16xi8> + %b = memref.alloc() alignment = 64 : memref<16x16xi8> + %c = memref.alloc() alignment = 64 : memref<16x16xi32> linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x16xi8>) diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32.mlir new file mode 100644 index 000000000..6480fca6d --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32.mlir @@ -0,0 +1,44 @@ +memref.global "private" constant @zero : memref<513x17xi32> = dense<0> + +func.func private @check_result(memref<513x17xi32>) -> () + +func.func @main() -> i8 { + %zeroI8 = arith.constant 0 : i8 + %zero = arith.constant 0 : index + %one = arith.constant 1 : index + %rows = arith.constant 513 : index + %columns = arith.constant 17 : index + %thirtySeven = arith.constant 37 : i32 + %eighteen = arith.constant 18 : i32 + %input = memref.alloc() alignment = 64 : memref<513x17xi32> + %output = memref.alloc() alignment = 64 : memref<513x17xi32> + %zeroInput = memref.get_global @zero : memref<513x17xi32> + + scf.for %row = %zero to %rows step %one { + scf.for %column = %zero to %columns step %one { + %rowI32 = arith.index_cast %row : index to i32 + %columnI32 = arith.index_cast %column : index to i32 + %value = arith.addi %rowI32, %columnI32 : i32 + %remainder = arith.remui %value, %thirtySeven : i32 + %signed = arith.subi %remainder, %eighteen : i32 + memref.store %signed, %input[%row, %column] : memref<513x17xi32> + } + } + + linalg.generic { + indexing_maps = [affine_map<(row, column) -> (row, column)>, + affine_map<(row, column) -> (row, column)>, + affine_map<(row, column) -> (row, column)>], + iterator_types = ["parallel", "parallel"]} + ins(%input, %zeroInput : memref<513x17xi32>, memref<513x17xi32>) + outs(%output : memref<513x17xi32>) { + ^bb0(%value: i32, %zeroValue: i32, %old: i32): + %relu = arith.maxsi %value, %zeroValue : i32 + linalg.yield %relu : i32 + } + + func.call @check_result(%output) : (memref<513x17xi32>) -> () + memref.dealloc %input : memref<513x17xi32> + memref.dealloc %output : memref<513x17xi32> + return %zeroI8 : i8 +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32_contract.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32_contract.mlir new file mode 100644 index 000000000..965b8f645 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32_contract.mlir @@ -0,0 +1,25 @@ +// 513x17 is padded to 528x32, then packed linearly into five physical pbank +// chunks. Every ReLU command owns one pbank in place. +// CHECK: funct7 = 50 +// CHECK: funct7 = 50 +// CHECK: funct7 = 50 +// CHECK: funct7 = 50 +// CHECK: funct7 = 50 + +memref.global "private" constant @zero : memref<513x17xi32> = dense<0> + +func.func @main(%input: memref<513x17xi32>, %output: memref<513x17xi32>) { + %zero = memref.get_global @zero : memref<513x17xi32> + linalg.generic { + indexing_maps = [affine_map<(row, column) -> (row, column)>, + affine_map<(row, column) -> (row, column)>, + affine_map<(row, column) -> (row, column)>], + iterator_types = ["parallel", "parallel"]} + ins(%input, %zero : memref<513x17xi32>, memref<513x17xi32>) + outs(%output : memref<513x17xi32>) { + ^bb0(%value: i32, %zeroValue: i32, %old: i32): + %relu = arith.maxsi %value, %zeroValue : i32 + linalg.yield %relu : i32 + } + return +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32_main.cpp b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32_main.cpp new file mode 100644 index 000000000..8a0626a87 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_relu_513x17_i32_main.cpp @@ -0,0 +1,33 @@ +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + *(volatile uint32_t *)0x60000000 = 1; + while (1) { + } +#else + exit(1); +#endif +} + +#ifdef __cplusplus +extern "C" +#endif + void check_result(int32_t *allocated, int32_t *aligned, int64_t offset, + int64_t rows, int64_t columns, int64_t rowStride, + int64_t columnStride) { + (void)allocated; + if (rows != 513 || columns != 17 || rowStride != 17 || columnStride != 1) + fail(); + for (int row = 0; row < 513; ++row) { + for (int column = 0; column < 17; ++column) { + int32_t value = (row + column) % 37 - 18; + int32_t expected = value < 0 ? 0 : value; + if (aligned[offset + row * rowStride + column * columnStride] != expected) + fail(); + } + } + printf("PASSED: linalg.relu 513x17 i32\n"); +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_16x16_i8.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_16x16_i8.mlir index 1c8fa3bd3..0143407de 100644 --- a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_16x16_i8.mlir +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_16x16_i8.mlir @@ -5,8 +5,8 @@ func.func @main() -> i8 { %c0 = arith.constant 0 : index %c1 = arith.constant 1 : index %c16 = arith.constant 16 : index - %input = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<16x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> scf.for %i = %c0 to %c16 step %c1 { scf.for %j = %c0 to %c16 step %c1 { %ii = arith.index_cast %i : index to i8 diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_detect_1x84x100_f32.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_detect_1x84x100_f32.mlir new file mode 100644 index 000000000..ed2074d81 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_detect_1x84x100_f32.mlir @@ -0,0 +1,26 @@ +func.func private @check_result(memref<1x100x84xf32>) -> () + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c84 = arith.constant 84 : index + %c100 = arith.constant 100 : index + %in = memref.alloc() : memref<1x84x100xf32> + %out = memref.alloc() : memref<1x100x84xf32> + scf.for %a = %c0 to %c84 step %c1 { + scf.for %b = %c0 to %c100 step %c1 { + %ai = arith.index_cast %a : index to i32 + %bi = arith.index_cast %b : index to i32 + %s = arith.addi %ai, %bi : i32 + %v = arith.sitofp %s : i32 to f32 + memref.store %v, %in[%c0, %a, %b] : memref<1x84x100xf32> + } + } + linalg.transpose ins(%in : memref<1x84x100xf32>) + outs(%out : memref<1x100x84xf32>) permutation = [0, 2, 1] + func.call @check_result(%out) : (memref<1x100x84xf32>) -> () + memref.dealloc %in : memref<1x84x100xf32> + memref.dealloc %out : memref<1x100x84xf32> + return %z : i8 +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_detect_1x84x100_f32_main.cpp b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_detect_1x84x100_f32_main.cpp new file mode 100644 index 000000000..95663b90d --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_detect_1x84x100_f32_main.cpp @@ -0,0 +1,34 @@ +#include +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + *(volatile uint32_t *)0x60000000 = 1; + while (1) { + } +#else + exit(1); +#endif +} + +#ifdef __cplusplus +extern "C" +#endif + void check_result(float *allocated, float *aligned, int64_t offset, + int64_t s0, int64_t s1, int64_t s2, int64_t st0, + int64_t st1, int64_t st2) { + (void)allocated; + if (s0 != 1 || s1 != 100 || s2 != 84) + fail(); + float *out = aligned + offset; + for (int b = 0; b < 100; ++b) + for (int a = 0; a < 84; ++a) { + float got = out[b * st1 + a * st2]; + float exp = (float)(a + b); + if (fabsf(got - exp) > 1e-5f) + fail(); + } + printf("PASSED: linalg.transpose detect 1x84x100 f32\n"); +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_nchw_nhwc_1x3x8x8_f32.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_nchw_nhwc_1x3x8x8_f32.mlir new file mode 100644 index 000000000..fc8bfa8aa --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_nchw_nhwc_1x3x8x8_f32.mlir @@ -0,0 +1,30 @@ +func.func private @check_result(memref<1x8x8x3xf32>) -> () + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c3 = arith.constant 3 : index + %c8 = arith.constant 8 : index + %in = memref.alloc() : memref<1x3x8x8xf32> + %out = memref.alloc() : memref<1x8x8x3xf32> + scf.for %c = %c0 to %c3 step %c1 { + scf.for %h = %c0 to %c8 step %c1 { + scf.for %w = %c0 to %c8 step %c1 { + %cf = arith.index_cast %c : index to i32 + %hf = arith.index_cast %h : index to i32 + %wf = arith.index_cast %w : index to i32 + %s = arith.addi %cf, %hf : i32 + %s2 = arith.addi %s, %wf : i32 + %v = arith.sitofp %s2 : i32 to f32 + memref.store %v, %in[%c0, %c, %h, %w] : memref<1x3x8x8xf32> + } + } + } + linalg.transpose ins(%in : memref<1x3x8x8xf32>) + outs(%out : memref<1x8x8x3xf32>) permutation = [0, 2, 3, 1] + func.call @check_result(%out) : (memref<1x8x8x3xf32>) -> () + memref.dealloc %in : memref<1x3x8x8xf32> + memref.dealloc %out : memref<1x8x8x3xf32> + return %z : i8 +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_nchw_nhwc_1x3x8x8_f32_main.cpp b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_nchw_nhwc_1x3x8x8_f32_main.cpp new file mode 100644 index 000000000..465041d3a --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_nchw_nhwc_1x3x8x8_f32_main.cpp @@ -0,0 +1,35 @@ +#include +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + *(volatile uint32_t *)0x60000000 = 1; + while (1) { + } +#else + exit(1); +#endif +} + +#ifdef __cplusplus +extern "C" +#endif + void check_result(float *allocated, float *aligned, int64_t offset, + int64_t s0, int64_t s1, int64_t s2, int64_t s3, + int64_t st0, int64_t st1, int64_t st2, int64_t st3) { + (void)allocated; + if (s0 != 1 || s1 != 8 || s2 != 8 || s3 != 3) + fail(); + float *out = aligned + offset; + for (int h = 0; h < 8; ++h) + for (int w = 0; w < 8; ++w) + for (int c = 0; c < 3; ++c) { + float got = out[h * st1 + w * st2 + c * st3]; + float exp = (float)(c + h + w); + if (fabsf(got - exp) > 1e-5f) + fail(); + } + printf("PASSED: linalg.transpose nchw->nhwc 1x3x8x8 f32\n"); +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_oihw_ohwi_4x8x3x3_f32.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_oihw_ohwi_4x8x3x3_f32.mlir new file mode 100644 index 000000000..c370a5a3b --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_oihw_ohwi_4x8x3x3_f32.mlir @@ -0,0 +1,35 @@ +func.func private @check_result(memref<4x3x3x8xf32>) -> () + +func.func @main() -> i8 { + %z = arith.constant 0 : i8 + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c3 = arith.constant 3 : index + %c4 = arith.constant 4 : index + %c8 = arith.constant 8 : index + %in = memref.alloc() : memref<4x8x3x3xf32> + %out = memref.alloc() : memref<4x3x3x8xf32> + scf.for %o = %c0 to %c4 step %c1 { + scf.for %i = %c0 to %c8 step %c1 { + scf.for %h = %c0 to %c3 step %c1 { + scf.for %w = %c0 to %c3 step %c1 { + %oi = arith.index_cast %o : index to i32 + %ii = arith.index_cast %i : index to i32 + %hi = arith.index_cast %h : index to i32 + %wi = arith.index_cast %w : index to i32 + %s = arith.addi %oi, %ii : i32 + %s2 = arith.addi %s, %hi : i32 + %s3 = arith.addi %s2, %wi : i32 + %v = arith.sitofp %s3 : i32 to f32 + memref.store %v, %in[%o, %i, %h, %w] : memref<4x8x3x3xf32> + } + } + } + } + linalg.transpose ins(%in : memref<4x8x3x3xf32>) + outs(%out : memref<4x3x3x8xf32>) permutation = [0, 2, 3, 1] + func.call @check_result(%out) : (memref<4x3x3x8xf32>) -> () + memref.dealloc %in : memref<4x8x3x3xf32> + memref.dealloc %out : memref<4x3x3x8xf32> + return %z : i8 +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_oihw_ohwi_4x8x3x3_f32_main.cpp b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_oihw_ohwi_4x8x3x3_f32_main.cpp new file mode 100644 index 000000000..bab64d128 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/linalg_transpose_oihw_ohwi_4x8x3x3_f32_main.cpp @@ -0,0 +1,36 @@ +#include +#include +#include +#include + +static void fail(void) { +#ifdef BAREMETAL + *(volatile uint32_t *)0x60000000 = 1; + while (1) { + } +#else + exit(1); +#endif +} + +#ifdef __cplusplus +extern "C" +#endif + void check_result(float *allocated, float *aligned, int64_t offset, + int64_t s0, int64_t s1, int64_t s2, int64_t s3, + int64_t st0, int64_t st1, int64_t st2, int64_t st3) { + (void)allocated; + if (s0 != 4 || s1 != 3 || s2 != 3 || s3 != 8) + fail(); + float *out = aligned + offset; + for (int o = 0; o < 4; ++o) + for (int h = 0; h < 3; ++h) + for (int w = 0; w < 3; ++w) + for (int i = 0; i < 8; ++i) { + float got = out[o * st0 + h * st1 + w * st2 + i * st3]; + float exp = (float)(o + i + h + w); + if (fabsf(got - exp) > 1e-5f) + fail(); + } + printf("PASSED: linalg.transpose oihw->ohwi 4x8x3x3 f32\n"); +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_contract.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_contract.mlir new file mode 100644 index 000000000..912dab44e --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_contract.mlir @@ -0,0 +1,15 @@ +// One activation chunk is quantized once; each N panel does SMATMUL + INT2FP. +// Pebble owns these funct7 assignments. +// CHECK: funct7 = 51 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 + +func.func @main(%a: memref<16x32xf32>, %b: memref<32x32xi8>, + %c: memref<16x32xf32>) { + tile.tile_matmul %a %b %c {dw_addr = 16 : i64, dw_bytes = 128 : i64, + per_channel = true} + : memref<16x32xf32> memref<32x32xi8> memref<16x32xf32> + return +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_edge_contract.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_edge_contract.mlir new file mode 100644 index 000000000..af59bb8b1 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_edge_contract.mlir @@ -0,0 +1,27 @@ +// Edge tiles are padded in lowering: one FP2INT, eight SMATMUL and eight +// channel INT2FP instructions. Pebble owns these funct7 assignments. +// CHECK: funct7 = 51 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 + +func.func @main(%a: memref<1x256xf32>, %b: memref<256x120xi8>, + %c: memref<1x120xf32>) { + tile.tile_matmul %a %b %c {dw_addr = 16 : i64, dw_bytes = 512 : i64, + per_channel = true} + : memref<1x256xf32> memref<256x120xi8> memref<1x120xf32> + return +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_tensor_contract.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_tensor_contract.mlir new file mode 100644 index 000000000..396baff81 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/quant_matmul_f32_i8_tensor_contract.mlir @@ -0,0 +1,15 @@ +// One activation chunk feeds two N panels with a single tensor Dw at byte 16. +// Pebble owns these funct7 assignments. +// CHECK: funct7 = 51 +// CHECK: funct7 = 65 +// CHECK: funct7 = 52 +// CHECK: funct7 = 65 +// CHECK: funct7 = 52 + +func.func @main(%a: memref<16x32xf32>, %b: memref<32x32xi8>, + %c: memref<16x32xf32>) { + tile.tile_matmul %a %b %c {dw_addr = 16 : i64, dw_bytes = 4 : i64, + per_channel = false} + : memref<16x32xf32> memref<32x32xi8> memref<16x32xf32> + return +} diff --git a/examples/chips/pebble/workloads/mlir_tests/linalg/quantized_generic_f32_i8_contract.mlir b/examples/chips/pebble/workloads/mlir_tests/linalg/quantized_generic_f32_i8_contract.mlir new file mode 100644 index 000000000..7b596e5a5 --- /dev/null +++ b/examples/chips/pebble/workloads/mlir_tests/linalg/quantized_generic_f32_i8_contract.mlir @@ -0,0 +1,22 @@ +// CHECK: funct7 = 51 +// CHECK: funct7 = 65 +// CHECK: funct7 = 54 + +func.func @main(%a: memref<16x32xf32>, %b: memref<32x16xi8>, + %c: memref<16x16xf32>) { + linalg.generic {buckyball.quantized = true, dw_addr = 16 : i64, + dw_bytes = 64 : i64, per_channel = true, + indexing_maps = [affine_map<(m, n, k) -> (m, k)>, + affine_map<(m, n, k) -> (k, n)>, + affine_map<(m, n, k) -> (m, n)>], + iterator_types = ["parallel", "parallel", "reduction"]} + ins(%a, %b : memref<16x32xf32>, memref<32x16xi8>) + outs(%c : memref<16x16xf32>) { + ^bb0(%x: f32, %w: i8, %sum: f32): + %wf = arith.sitofp %w : i8 to f32 + %product = arith.mulf %x, %wf : f32 + %next = arith.addf %sum, %product : f32 + linalg.yield %next : f32 + } + return +} diff --git a/examples/chips/pebble/workloads/mlir_tests/tile/CMakeLists.txt b/examples/chips/pebble/workloads/mlir_tests/tile/CMakeLists.txt index 244cc70bf..018615c44 100644 --- a/examples/chips/pebble/workloads/mlir_tests/tile/CMakeLists.txt +++ b/examples/chips/pebble/workloads/mlir_tests/tile/CMakeLists.txt @@ -1,12 +1,10 @@ -include(${OPTEST_WORKLOAD_DIR}/cmake/BuckyballMlirTests.cmake) - set(BUCKYBALL_MLIR_GROUP_TARGET pebble-mlir-tests-build) set(BUCKYBALL_MLIR_TEST_PREFIX tile) set(PEBBLE_TILE_TO_ELF_PASSES -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -20,25 +18,25 @@ set(PEBBLE_TILE_TO_ELF_PASSES -convert-func-to-llvm -reconcile-unrealized-casts) -add_buckyball_mlir_optest(tile_matmul_16x16_i8 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_matmul_16x16_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_matmul_1x256x120_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_matmul_1x576x1024_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_matmul_1x1024x1000_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_matmul_5x3072x768_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_transpose_16x16_i8 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_transpose_80x80_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_transpose_120x256_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_matmul_16x16_i8 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_matmul_16x16_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_matmul_1x256x120_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_matmul_1x576x1024_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_matmul_1x1024x1000_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_matmul_5x3072x768_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_transpose_16x16_i8 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_transpose_80x80_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_transpose_120x256_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_transpose_128x128_dynstride_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_transpose_1024x576_dynstride_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_transpose_matmul_120x256_f32 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_6x6_k3 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_1x1_c16_oc8 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_1x1_c16_oc16 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_1x1_c16_oc24 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_1x1_c240_oc64 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_8x8_1x1_c16_oc24 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_8x8_k3_c2_oc2 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_12x12_k5 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) -add_buckyball_mlir_optest(tile_conv2d_28x28_k5 PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_transpose_128x128_dynstride_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_transpose_1024x576_dynstride_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_transpose_matmul_120x256_f32 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_6x6_k3 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_1x1_c16_oc8 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_1x1_c16_oc16 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_1x1_c16_oc24 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_1x1_c240_oc64 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_8x8_1x1_c16_oc24 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_8x8_k3_c2_oc2 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_12x12_k5 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) +add_buckyball_mlir_test(tile_conv2d_28x28_k5 TARGET pebble PASSES ${PEBBLE_TILE_TO_ELF_PASSES}) diff --git a/examples/chips/pebble/workloads/mlir_tests/tile/tile_matmul_16x16_i8.mlir b/examples/chips/pebble/workloads/mlir_tests/tile/tile_matmul_16x16_i8.mlir index 7ead0d596..909991451 100644 --- a/examples/chips/pebble/workloads/mlir_tests/tile/tile_matmul_16x16_i8.mlir +++ b/examples/chips/pebble/workloads/mlir_tests/tile/tile_matmul_16x16_i8.mlir @@ -4,9 +4,9 @@ func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 %one_i8 = arith.constant 1 : i8 %zero_i32 = arith.constant 0 : i32 - %a = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %b = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %c = memref.alloc() {alignment = 64 : i64} : memref<16x16xi32> + %a = memref.alloc() alignment = 64 : memref<16x16xi8> + %b = memref.alloc() alignment = 64 : memref<16x16xi8> + %c = memref.alloc() alignment = 64 : memref<16x16xi32> linalg.fill ins(%one_i8 : i8) outs(%a : memref<16x16xi8>) linalg.fill ins(%one_i8 : i8) outs(%b : memref<16x16xi8>) linalg.fill ins(%zero_i32 : i32) outs(%c : memref<16x16xi32>) diff --git a/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_16x16_i8.mlir b/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_16x16_i8.mlir index 90831d528..568c0730f 100644 --- a/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_16x16_i8.mlir +++ b/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_16x16_i8.mlir @@ -5,8 +5,8 @@ func.func @main() -> i8 { %c0 = arith.constant 0 : index %c1 = arith.constant 1 : index %c16 = arith.constant 16 : index - %input = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> - %output = memref.alloc() {alignment = 64 : i64} : memref<16x16xi8> + %input = memref.alloc() alignment = 64 : memref<16x16xi8> + %output = memref.alloc() alignment = 64 : memref<16x16xi8> scf.for %i = %c0 to %c16 step %c1 { scf.for %j = %c0 to %c16 step %c1 { %ii = arith.index_cast %i : index to i8 diff --git a/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32.mlir b/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32.mlir index 6b6626b90..874026e4f 100644 --- a/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32.mlir +++ b/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32.mlir @@ -1,15 +1,25 @@ -func.func private @check_result(memref<80x80xf32>) -> () +func.func private @check_result(memref<80x80xf32>, memref<80x80xf32>) -> () func.func @main() -> i8 { %zero_i8 = arith.constant 0 : i8 %zero = arith.constant 0.0 : f32 - %one = arith.constant 1.0 : f32 + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c80 = arith.constant 80 : index %w = memref.alloc() : memref<80x80xf32> %wt = memref.alloc() : memref<80x80xf32> - linalg.fill ins(%one : f32) outs(%w : memref<80x80xf32>) linalg.fill ins(%zero : f32) outs(%wt : memref<80x80xf32>) + scf.for %i = %c0 to %c80 step %c1 { + scf.for %j = %c0 to %c80 step %c1 { + %ii = arith.index_cast %i : index to i32 + %jj = arith.index_cast %j : index to i32 + %s = arith.addi %ii, %jj : i32 + %v = arith.sitofp %s : i32 to f32 + memref.store %v, %w[%i, %j] : memref<80x80xf32> + } + } tile.tile_transpose %w %wt : memref<80x80xf32> memref<80x80xf32> - func.call @check_result(%wt) : (memref<80x80xf32>) -> () + func.call @check_result(%w, %wt) : (memref<80x80xf32>, memref<80x80xf32>) -> () memref.dealloc %w : memref<80x80xf32> memref.dealloc %wt : memref<80x80xf32> return %zero_i8 : i8 diff --git a/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32_main.cpp b/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32_main.cpp index 8ef34f23a..e510b7111 100644 --- a/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32_main.cpp +++ b/examples/chips/pebble/workloads/mlir_tests/tile/tile_transpose_80x80_f32_main.cpp @@ -15,16 +15,25 @@ static void fail(void) { #ifdef __cplusplus extern "C" #endif - void check_result(float *allocated, float *aligned, int64_t offset, - int64_t m, int64_t n, int64_t sm, int64_t sn) { - (void)allocated; - (void)sm; - (void)sn; - if (m != 80 || n != 80) + void check_result(float *w_alloc, float *w_align, int64_t w_off, int64_t wm, + int64_t wn, int64_t wsm, int64_t wsn, float *wt_alloc, + float *wt_align, int64_t wt_off, int64_t wtm, int64_t wtn, + int64_t wtsm, int64_t wtsn) { + (void)w_alloc; + (void)wt_alloc; + (void)wsm; + (void)wsn; + (void)wtsm; + (void)wtsn; + if (wm != 80 || wn != 80 || wtm != 80 || wtn != 80) fail(); - for (int i = 0; i < 80 * 80; ++i) { - if (aligned[offset + i] != 1.0f) - fail(); + for (int i = 0; i < 80; ++i) { + for (int j = 0; j < 80; ++j) { + float src = w_align[w_off + i * 80 + j]; + float dst = wt_align[wt_off + j * 80 + i]; + if (dst != src) + fail(); + } } printf("PASSED: tile.tile_transpose 80x80 f32\n"); } diff --git a/examples/chips/poly/arch/src/main/scala/CustomConfigs.scala b/examples/chips/poly/arch/src/main/scala/CustomConfigs.scala index 6ea3ae807..648d74374 100644 --- a/examples/chips/poly/arch/src/main/scala/CustomConfigs.scala +++ b/examples/chips/poly/arch/src/main/scala/CustomConfigs.scala @@ -5,15 +5,13 @@ import org.chipsalliance.cde.config.Config import freechips.rocketchip.tile.MaxHartIdBits import framework.system.tile.WithBuckyballTiles -class WithPolyHartIdBits extends Config((site, here, up) => { - case MaxHartIdBits => log2Ceil(5) -}) - -/** One Poly Tile: 3 prefill Cores followed by 2 decode Cores. */ +/** Four Poly tiles: 3 prefill + 2 decode Cores per tile (20 harts). */ class BuckyballPolyConfig extends Config( - new WithPolyHartIdBits ++ - new WithBuckyballTiles("../examples/chips/poly/configs/poly.toml") ++ + new Config((site, here, up) => { case MaxHartIdBits => + log2Ceil(20) + }) ++ + new WithBuckyballTiles("../examples/chips/poly/configs/generated/chip.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) diff --git a/examples/chips/poly/chip.toml b/examples/chips/poly/chip.toml deleted file mode 100644 index fc3c53853..000000000 --- a/examples/chips/poly/chip.toml +++ /dev/null @@ -1,10 +0,0 @@ -[chip] -name = "poly" -topology = "configs/poly.toml" -compilerCore = "prefill" - -[runtime] -bemu = "emu/Cargo.toml" -bemuCore = "../../cores/prefill/emu/Cargo.toml" -bemuTile = "configs/tiles/default.toml" -workloads = "workloads" diff --git a/examples/chips/poly/configs/chip.toml b/examples/chips/poly/configs/chip.toml new file mode 100644 index 000000000..eef111c38 --- /dev/null +++ b/examples/chips/poly/configs/chip.toml @@ -0,0 +1,6 @@ +[designs] +include = "designs/poly.toml" + +[sims] +verilator = "sims.verilator.BuckyballPolyVerilatorConfig" +p2e = "sims.p2e.BuckyballPolyP2EConfig" diff --git a/examples/chips/poly/configs/designs/poly.toml b/examples/chips/poly/configs/designs/poly.toml new file mode 100644 index 000000000..ceabacf1c --- /dev/null +++ b/examples/chips/poly/configs/designs/poly.toml @@ -0,0 +1,9 @@ +# Poly Chip: 4 homogeneous tiles, each 3 prefill + 2 decode Cores. + +[top] +nTiles = 4 + +[tileTemplate] +tile_ids = [0, 1, 2, 3] +include = "tiles/default.toml" +count = 4 diff --git a/examples/chips/poly/configs/tiles/default.toml b/examples/chips/poly/configs/designs/tiles/default.toml similarity index 53% rename from examples/chips/poly/configs/tiles/default.toml rename to examples/chips/poly/configs/designs/tiles/default.toml index 86ae10066..fdf1973e7 100644 --- a/examples/chips/poly/configs/tiles/default.toml +++ b/examples/chips/poly/configs/designs/tiles/default.toml @@ -18,21 +18,26 @@ defaultGroupCount = 1 memCycles = 10 [[cores]] +core_id = 0 name = "prefill" -include = "../../../../cores/prefill/core.toml" +include = "../../../../../cores/prefill/configs/default.toml" [[cores]] +core_id = 1 name = "prefill" -include = "../../../../cores/prefill/core.toml" +include = "../../../../../cores/prefill/configs/default.toml" [[cores]] +core_id = 2 name = "prefill" -include = "../../../../cores/prefill/core.toml" +include = "../../../../../cores/prefill/configs/default.toml" [[cores]] +core_id = 3 name = "decode" -include = "../../../../cores/decode/core.toml" +include = "../../../../../cores/decode/configs/default.toml" [[cores]] +core_id = 4 name = "decode" -include = "../../../../cores/decode/core.toml" +include = "../../../../../cores/decode/configs/default.toml" diff --git a/examples/chips/poly/configs/poly.toml b/examples/chips/poly/configs/poly.toml deleted file mode 100644 index 1d721430d..000000000 --- a/examples/chips/poly/configs/poly.toml +++ /dev/null @@ -1,7 +0,0 @@ -# Poly Chip: higher levels replicate a homogeneous 3-prefill/2-decode Tile. - -[top] -nTiles = 1 - -[[tiles]] -include = "tiles/default.toml" diff --git a/examples/chips/poly/emu/Cargo.lock b/examples/chips/poly/emu/Cargo.lock index 5cea2f8ed..2658eb909 100644 --- a/examples/chips/poly/emu/Cargo.lock +++ b/examples/chips/poly/emu/Cargo.lock @@ -2,15 +2,6 @@ # It is not intended for manual editing. version = 4 -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - [[package]] name = "anstream" version = "1.0.0" @@ -47,7 +38,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -58,7 +49,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -72,73 +63,17 @@ dependencies = [ ] [[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bemu-goban" +name = "bebop-bemu" version = "0.1.0" dependencies = [ "bebop-bank-hash", "bebop-bemu-profile", - "bebop-dasm", "bebop-dtb", "bebop-elf", - "bebop-fd-redirect", - "bebop-rtl-trace", "bebop-rushb", "bebop-syscall", "bebop-uart", - "camino", "cc", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "log", - "nix 0.29.0", "once_cell", "serde", "serde_json", @@ -147,54 +82,45 @@ dependencies = [ ] [[package]] -name = "bemu-poly-chip" +name = "bebop-bemu-profile" +version = "0.1.0" + +[[package]] +name = "bebop-chip-poly" version = "0.1.0" dependencies = [ - "bemu-prefill", + "bebop-bemu", "clap", - "snafu", - "toml", ] [[package]] -name = "bemu-prefill" +name = "bebop-dtb" version = "0.1.0" -dependencies = [ - "bemu-goban", -] [[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" +name = "bebop-elf" +version = "0.1.0" [[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" +name = "bebop-rushb" +version = "0.1.0" [[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" +name = "bebop-syscall" +version = "0.1.0" dependencies = [ - "objc2", + "once_cell", ] [[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" +name = "bebop-uart" +version = "0.1.0" [[package]] name = "cc" -version = "1.4.2" +version = "1.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e" +checksum = "0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273" dependencies = [ "find-msvc-tools", "jobserver", @@ -208,12 +134,6 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - [[package]] name = "clap" version = "4.6.6" @@ -260,116 +180,17 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - [[package]] name = "equivalent" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - [[package]] name = "find-msvc-tools" -version = "0.1.10" +version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de" +checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890" [[package]] name = "getrandom" @@ -416,42 +237,6 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "jobserver" version = "0.1.35" @@ -468,57 +253,12 @@ version = "0.2.189" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - [[package]] name = "memchr" version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - [[package]] name = "once_cell" version = "1.21.4" @@ -531,31 +271,6 @@ version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "portable-atomic" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - [[package]] name = "proc-macro2" version = "1.0.107" @@ -580,35 +295,6 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - [[package]] name = "serde" version = "1.0.229" @@ -661,54 +347,12 @@ dependencies = [ "serde", ] -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - [[package]] name = "shlex" version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - [[package]] name = "snafu" version = "0.8.9" @@ -758,26 +402,6 @@ dependencies = [ "unicode-ident", ] -[[package]] -name = "thiserror" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - [[package]] name = "toml" version = "0.8.23" @@ -837,15 +461,6 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - [[package]] name = "windows-sys" version = "0.61.2" @@ -855,71 +470,6 @@ dependencies = [ "windows-link", ] -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - [[package]] name = "winnow" version = "0.7.15" diff --git a/examples/chips/poly/emu/Cargo.toml b/examples/chips/poly/emu/Cargo.toml index 73a79a447..5f64c1352 100644 --- a/examples/chips/poly/emu/Cargo.toml +++ b/examples/chips/poly/emu/Cargo.toml @@ -1,14 +1,12 @@ [package] -name = "bemu-poly-chip" +name = "bebop-chip-poly" version = "0.1.0" edition = "2021" [[bin]] -name = "bebop-poly" +name = "bebop-chip-poly" path = "src/main.rs" [dependencies] +bebop-bemu = { path = "../configs/generated/bemu" } clap = { version = "4", features = ["derive"] } -snafu = "0.8" -toml = "0.8" -core_bemu = { package = "bemu-prefill", path = "../../../cores/prefill/emu" } diff --git a/examples/chips/poly/emu/src/main.rs b/examples/chips/poly/emu/src/main.rs index 6d386c4c2..7cdae9500 100644 --- a/examples/chips/poly/emu/src/main.rs +++ b/examples/chips/poly/emu/src/main.rs @@ -1 +1,241 @@ -include!("../../../goban/emu/src/main.rs"); +use clap::Parser; +use bebop_bemu::{tile_topology, BemuInstance, SharedMemory, TraceConfig}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, AtomicI32, Ordering}; +use std::sync::{mpsc, Arc, Condvar, Mutex}; +use std::thread; + +const DRAM_SIZE: usize = 1 << 30; + +#[derive(Parser, Debug)] +struct Args { + /// Tile index in the chip bundle. + #[arg(long, default_value_t = 0)] + tile_index: usize, + #[arg(long)] + elf: PathBuf, + #[arg(long)] + log_dir: PathBuf, + #[arg(long)] + pk: bool, + #[arg(long)] + disasm: bool, + #[arg(long = "tool-profile")] + tool_profile: bool, +} + +struct StartGate { + result: Mutex>>, + ready: Condvar, +} + +impl StartGate { + fn new() -> Self { + Self { + result: Mutex::new(None), + ready: Condvar::new(), + } + } + + fn wait(&self) -> Result<(), String> { + let mut result = self.result.lock().map_err(|_| "BEMU start gate poisoned".to_string())?; + while result.is_none() { + result = self.ready.wait(result).map_err(|_| "BEMU start gate poisoned".to_string())?; + } + result.as_ref().expect("BEMU start gate was set").clone() + } + + fn release(&self, result: Result<(), String>) { + *self.result.lock().expect("BEMU start gate poisoned") = Some(result); + self.ready.notify_all(); + } +} + +fn main() { + if let Err(error) = run(Args::parse()) { + eprintln!("error: {error}"); + std::process::exit(1); + } +} + +fn run(args: Args) -> Result<(), String> { + let elf = absolute(&args.elf)?; + let log_dir = absolute(&args.log_dir)?; + let topology = tile_topology(args.tile_index); + if topology.cores.is_empty() { + return Err(format!("tile {} has no Core instances", args.tile_index)); + } + eprintln!( + "[INFO] Poly Chip BEMU: tile_index={} cores={} shared_dram={}MiB", + args.tile_index, + topology.cores.len(), + DRAM_SIZE >> 20 + ); + + let core_count = topology.cores.len(); + let virtual_bank_count = topology.virtual_bank_count; + let memory = SharedMemory::new(DRAM_SIZE, core_count); + let schedule = Arc::new(Mutex::new(())); + let start = Arc::new(StartGate::new()); + let done = Arc::new(AtomicBool::new(false)); + let exit_code = Arc::new(AtomicI32::new(0)); + let (prepared_tx, prepared_rx) = mpsc::channel(); + let mut workers = Vec::with_capacity(topology.cores.len()); + + for (hart_id, (core_name, core_index)) in topology.cores.into_iter().enumerate() { + let elf = elf.clone(); + let worker_log = log_dir.join(format!("hart-{hart_id}")); + let memory = Arc::clone(&memory); + let schedule = Arc::clone(&schedule); + let start = Arc::clone(&start); + let done = Arc::clone(&done); + let exit_code = Arc::clone(&exit_code); + let prepared_tx = prepared_tx.clone(); + let pk = args.pk; + let disasm = args.disasm; + let tool_profile = args.tool_profile; + workers.push( + thread::Builder::new() + .name(format!("core-{hart_id}-{core_name}")) + .spawn(move || { + run_core( + hart_id, + &core_name, + core_index, + &elf, + &worker_log, + pk, + disasm, + tool_profile, + memory, + schedule, + start, + done, + exit_code, + prepared_tx, + virtual_bank_count, + ) + }) + .map_err(|error| format!("failed to spawn Core worker {hart_id}: {error}"))?, + ); + } + drop(prepared_tx); + + let mut preparation_error = None; + for _ in 0..workers.len() { + if let Err(error) = prepared_rx.recv().map_err(|_| "Core worker exited before initialization".to_string())? { + preparation_error.get_or_insert(error); + } + } + start.release(preparation_error.map_or(Ok(()), Err)); + + let mut first_error = None; + for (hart_id, worker) in workers.into_iter().enumerate() { + match worker.join() { + Ok(Ok(())) => {} + Ok(Err(error)) => { + first_error.get_or_insert(format!("Core worker {hart_id}: {error}")); + } + Err(_) => { + first_error.get_or_insert(format!("Core worker {hart_id} panicked")); + } + }; + } + first_error.map_or(Ok(()), Err) +} + +#[allow(clippy::too_many_arguments)] +fn run_core( + hart_id: usize, + core_name: &str, + core_index: usize, + elf: &Path, + log_dir: &Path, + pk: bool, + disasm: bool, + tool_profile: bool, + memory: Arc, + schedule: Arc>, + start: Arc, + done: Arc, + exit_code: Arc, + prepared: mpsc::Sender>, + virtual_bank_count: usize, +) -> Result<(), String> { + eprintln!( + "[INFO] starting Core worker hart={hart_id} core={core_name} core_index={core_index}" + ); + let prepared_bemu = (|| { + let _turn = schedule.lock().map_err(|_| "BEMU scheduler poisoned".to_string())?; + let mut bemu = BemuInstance::new_with_core_hart( + log_dir, + TraceConfig::new(false, false), + disasm, + tool_profile, + core_index, + hart_id, + Some(Arc::clone(&memory)), + Some(virtual_bank_count), + ) + .map_err(|error| error.to_string())?; + bemu.load_elf(elf).map_err(|error| error.to_string())?; + bemu.init_hart(pk).map_err(|error| error.to_string())?; + Ok::(bemu) + })(); + + let mut bemu = match prepared_bemu { + Ok(bemu) => { + let _ = prepared.send(Ok(())); + bemu + } + Err(error) => { + let _ = prepared.send(Err(error.clone())); + return Err(error); + } + }; + start.wait()?; + + loop { + let barrier_hit = { + let _turn = schedule.lock().map_err(|_| "BEMU scheduler poisoned".to_string())?; + if done.load(Ordering::Acquire) { + bemu.stop(exit_code.load(Ordering::Acquire)); + break; + } + if let Err(error) = bemu.step() { + exit_code.store(1, Ordering::Release); + done.store(true, Ordering::Release); + memory.abort_barrier(); + return Err(error.to_string()); + } + bemu.barrier_hit() + }; + if barrier_hit { + memory.wait_barrier(hart_id); + } + if bemu.finished() { + let code = bemu.exit_code().unwrap_or(1); + exit_code.store(code, Ordering::Release); + done.store(true, Ordering::Release); + memory.abort_barrier(); + break; + } + } + + let code = exit_code.load(Ordering::Acquire); + eprintln!("[INFO] stopped Core worker hart={hart_id} core={core_name} exit={code}"); + if code == 0 { + Ok(()) + } else { + Err(format!("guest exited with code {code}")) + } +} + +fn absolute(path: &Path) -> Result { + if path.is_absolute() { + return Ok(path.to_path_buf()); + } + std::env::current_dir() + .map(|cwd| cwd.join(path)) + .map_err(|error| format!("failed to resolve {}: {error}", path.display())) +} diff --git a/examples/chips/poly/workloads/CMakeLists.txt b/examples/chips/poly/workloads/CMakeLists.txt index 84715c2a5..4e5d0c3d6 100644 --- a/examples/chips/poly/workloads/CMakeLists.txt +++ b/examples/chips/poly/workloads/CMakeLists.txt @@ -1,42 +1,16 @@ -# Poly reuses the common workload definitions, but owns its Core placement. -# Derive the type -> physical Core IDs directly from the tile TOML so adding a -# third Core type only requires editing the chip configuration. -get_filename_component(_BUCKYBALL_REPO_ROOT - "${CMAKE_CURRENT_SOURCE_DIR}/../../../.." ABSOLUTE) -set(_POLY_TILE_TOML "${CMAKE_CURRENT_SOURCE_DIR}/../configs/tiles/default.toml") -file(STRINGS "${_POLY_TILE_TOML}" _POLY_CORE_LINES REGEX "^name[ \\t]*=") -set(_POLY_CORE_INDEX 0) -foreach(_POLY_CORE_LINE IN LISTS _POLY_CORE_LINES) - string(REGEX REPLACE "^name[ \\t]*=[ \\t]*\"([^\"]+)\".*$" "\\1" - _POLY_CORE_TYPE "${_POLY_CORE_LINE}") - if(DEFINED _POLY_CORE_IDS_${_POLY_CORE_TYPE}) - set(_POLY_CORE_IDS_${_POLY_CORE_TYPE} - "${_POLY_CORE_IDS_${_POLY_CORE_TYPE}};${_POLY_CORE_INDEX}") - else() - set(_POLY_CORE_IDS_${_POLY_CORE_TYPE} "${_POLY_CORE_INDEX}") - endif() - math(EXPR _POLY_CORE_INDEX "${_POLY_CORE_INDEX} + 1") -endforeach() -set(BUCKYBALL_RUSHB_CORE_TYPES) -foreach(_POLY_CORE_LINE IN LISTS _POLY_CORE_LINES) - string(REGEX REPLACE "^name[ \\t]*=[ \\t]*\"([^\"]+)\".*$" "\\1" - _POLY_CORE_TYPE "${_POLY_CORE_LINE}") - list(FIND BUCKYBALL_RUSHB_CORE_TYPES "${_POLY_CORE_TYPE}" _POLY_TYPE_SEEN) - if(_POLY_TYPE_SEEN EQUAL -1) - list(APPEND BUCKYBALL_RUSHB_CORE_TYPES "${_POLY_CORE_TYPE}") - endif() -endforeach() -set(BUCKYBALL_RUSHB_PLACEMENT_STRICT ON) +add_custom_target(poly-ctests-build) +set(BUCKYBALL_CTEST_GROUP_TARGET poly-ctests-build) -# Poly selects the prefill Core as the host BEMU ABI provider. The runtime -# still reads the tile TOML and creates one worker per physical Core instance. -set(BUCKYBALL_RUSHB_BEMU_MANIFEST - ${_BUCKYBALL_REPO_ROOT}/examples/cores/prefill/emu/Cargo.toml) -set(BUCKYBALL_RUSHB_BEMU_LIBRARY - ${_BUCKYBALL_REPO_ROOT}/examples/cores/prefill/emu/target/release/libbemu_prefill.so) -set(BUCKYBALL_RUSHB_VERILATOR_LIBRARY - ${_BUCKYBALL_REPO_ROOT}/bebop/target/release/deps/libbebop_verilator.so) -add_subdirectory( - ${_BUCKYBALL_REPO_ROOT}/examples/chips/goban/workloads - ${CMAKE_CURRENT_BINARY_DIR}/goban-workloads -) +add_subdirectory(ctests) +add_subdirectory(mlir_tests) +buckyball_add_ball_workload_subdirs(ctests) + +add_custom_target(chip-ctests-build DEPENDS poly-ctests-build) +add_custom_target(chip-mlirtests-build DEPENDS poly-mlir-tests-build) + +add_custom_target(chip-workloads-build ALL + DEPENDS + chip-ctests-build + chip-mlirtests-build + COMMENT "Building poly workloads" + VERBATIM) diff --git a/examples/chips/poly/workloads/cmake/RushB.cmake b/examples/chips/poly/workloads/cmake/RushB.cmake deleted file mode 100644 index 27d8eac67..000000000 --- a/examples/chips/poly/workloads/cmake/RushB.cmake +++ /dev/null @@ -1,26 +0,0 @@ -# Poly owns a heterogeneous tile but uses the common RushB host ABI. -set(BUCKYBALL_RUSHB_BEMU_MANIFEST - ${BUCKYBALL_REPO_ROOT}/examples/cores/prefill/emu/Cargo.toml) -set(BUCKYBALL_RUSHB_BEMU_LIBRARY - ${BUCKYBALL_REPO_ROOT}/examples/cores/prefill/emu/target/release/libbemu_prefill.so) -set(BUCKYBALL_RUSHB_VERILATOR_LIBRARY - ${BUCKYBALL_REPO_ROOT}/bebop/target/release/deps/libbebop_verilator.so) - -# Build a type -> physical Core-ID map from the tile configuration. The -# workload placement code consumes these variables for typed subgraphs. -set(_POLY_TILE_TOML - ${BUCKYBALL_REPO_ROOT}/examples/chips/poly/configs/tiles/default.toml) -file(STRINGS ${_POLY_TILE_TOML} _POLY_CORE_LINES REGEX "^name[ \t]*=") -set(_POLY_CORE_INDEX 0) -foreach(_POLY_CORE_LINE IN LISTS _POLY_CORE_LINES) - string(REGEX REPLACE "^name[ \t]*=[ \t]*\"([^\"]+)\".*$" "\\1" - _POLY_CORE_TYPE "${_POLY_CORE_LINE}") - if(DEFINED _POLY_CORE_IDS_${_POLY_CORE_TYPE}) - set(_POLY_CORE_IDS_${_POLY_CORE_TYPE} - "${_POLY_CORE_IDS_${_POLY_CORE_TYPE}};${_POLY_CORE_INDEX}") - else() - set(_POLY_CORE_IDS_${_POLY_CORE_TYPE} "${_POLY_CORE_INDEX}") - endif() - math(EXPR _POLY_CORE_INDEX "${_POLY_CORE_INDEX} + 1") -endforeach() -set(BUCKYBALL_RUSHB_PLACEMENT_STRICT ON) diff --git a/examples/chips/poly/workloads/ctests/CMakeLists.txt b/examples/chips/poly/workloads/ctests/CMakeLists.txt new file mode 100644 index 000000000..e69de29bb diff --git a/examples/chips/poly/workloads/mlir_tests/CMakeLists.txt b/examples/chips/poly/workloads/mlir_tests/CMakeLists.txt new file mode 100644 index 000000000..b01519570 --- /dev/null +++ b/examples/chips/poly/workloads/mlir_tests/CMakeLists.txt @@ -0,0 +1 @@ +add_custom_target(poly-mlir-tests-build) diff --git a/examples/chips/toy/arch/src/main/scala/CustomConfigs.scala b/examples/chips/toy/arch/src/main/scala/CustomConfigs.scala index 8675c0a5a..566a642c5 100644 --- a/examples/chips/toy/arch/src/main/scala/CustomConfigs.scala +++ b/examples/chips/toy/arch/src/main/scala/CustomConfigs.scala @@ -6,8 +6,8 @@ import freechips.rocketchip.tile.MaxHartIdBits import framework.system.tile.WithBuckyballTiles class WithToyHartIdBits(nTiles: Int, nCoresPerTile: Int) - extends Config((site, here, up) => { - case MaxHartIdBits => log2Ceil(nTiles * nCoresPerTile) + extends Config((site, here, up) => { case MaxHartIdBits => + log2Ceil(nTiles * nCoresPerTile) }) /** @@ -19,7 +19,7 @@ class WithToyHartIdBits(nTiles: Int, nCoresPerTile: Int) */ class BuckyballToyConfig extends Config( - new WithBuckyballTiles("../examples/chips/toy/configs/toy.toml") ++ + new WithBuckyballTiles("../examples/chips/toy/configs/generated/chip.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -28,7 +28,7 @@ class BuckyballToyConfig class BuckyballToy4CoreConfig extends Config( new WithToyHartIdBits(nTiles = 1, nCoresPerTile = 4) ++ - new WithBuckyballTiles("../examples/chips/toy/configs/1t4c.toml") ++ + new WithBuckyballTiles("../examples/chips/toy/configs/generated/1t4c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -37,7 +37,7 @@ class BuckyballToy4CoreConfig class BuckyballToy8CoreConfig extends Config( new WithToyHartIdBits(nTiles = 1, nCoresPerTile = 8) ++ - new WithBuckyballTiles("../examples/chips/toy/configs/1t8c.toml") ++ + new WithBuckyballTiles("../examples/chips/toy/configs/generated/1t8c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -46,7 +46,7 @@ class BuckyballToy8CoreConfig class BuckyballToy16CoreConfig extends Config( new WithToyHartIdBits(nTiles = 1, nCoresPerTile = 16) ++ - new WithBuckyballTiles("../examples/chips/toy/configs/1t16c.toml") ++ + new WithBuckyballTiles("../examples/chips/toy/configs/generated/1t16c.pb") ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) @@ -57,7 +57,10 @@ class BuckyballToy16CoreConfig */ class RocketOnlyToyConfig extends Config( - new WithBuckyballTiles("../examples/chips/toy/configs/toy.toml", withBuckyball = false) ++ + new WithBuckyballTiles( + "../examples/chips/toy/configs/generated/chip.pb", + withBuckyball = false + ) ++ new chipyard.config.WithSystemBusWidth(128) ++ new sims.base.BuckyballBaseConfig ) diff --git a/examples/chips/toy/chip.toml b/examples/chips/toy/chip.toml deleted file mode 100644 index 8eff6a9b4..000000000 --- a/examples/chips/toy/chip.toml +++ /dev/null @@ -1,10 +0,0 @@ -[chip] -name = "toy" -topology = "configs/toy.toml" -compilerCore = "toy" - -[runtime] -bemu = "../../cores/toy/emu/Cargo.toml" -kernel = "kernel" -workloads = "workloads" -regression = "regression" diff --git a/examples/chips/toy/configs/chip.toml b/examples/chips/toy/configs/chip.toml new file mode 100644 index 000000000..6c69a9b68 --- /dev/null +++ b/examples/chips/toy/configs/chip.toml @@ -0,0 +1,6 @@ +[designs] +include = "designs/toy.toml" + +[sims] +verilator = "sims.verilator.BuckyballToyVerilatorConfig" +p2e = "sims.p2e.P2EToyConfig" diff --git a/examples/chips/toy/configs/1t16c.toml b/examples/chips/toy/configs/designs/1t16c.toml similarity index 89% rename from examples/chips/toy/configs/1t16c.toml rename to examples/chips/toy/configs/designs/1t16c.toml index ede82911b..1dcdad670 100644 --- a/examples/chips/toy/configs/1t16c.toml +++ b/examples/chips/toy/configs/designs/1t16c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/16core.toml" diff --git a/examples/chips/toy/configs/1t4c.toml b/examples/chips/toy/configs/designs/1t4c.toml similarity index 89% rename from examples/chips/toy/configs/1t4c.toml rename to examples/chips/toy/configs/designs/1t4c.toml index 1860b5503..013a1d688 100644 --- a/examples/chips/toy/configs/1t4c.toml +++ b/examples/chips/toy/configs/designs/1t4c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/4core.toml" diff --git a/examples/chips/toy/configs/1t8c.toml b/examples/chips/toy/configs/designs/1t8c.toml similarity index 89% rename from examples/chips/toy/configs/1t8c.toml rename to examples/chips/toy/configs/designs/1t8c.toml index 340ddabe5..68f228678 100644 --- a/examples/chips/toy/configs/1t8c.toml +++ b/examples/chips/toy/configs/designs/1t8c.toml @@ -4,4 +4,5 @@ nTiles = 1 [[tiles]] +tile_id = 0 include = "tiles/8core.toml" diff --git a/examples/chips/toy/configs/tiles/16core.toml b/examples/chips/toy/configs/designs/tiles/16core.toml similarity index 71% rename from examples/chips/toy/configs/tiles/16core.toml rename to examples/chips/toy/configs/designs/tiles/16core.toml index db9dba73d..33db43e84 100644 --- a/examples/chips/toy/configs/tiles/16core.toml +++ b/examples/chips/toy/configs/designs/tiles/16core.toml @@ -17,5 +17,6 @@ inputChannels = 16 defaultGroupCount = 1 [coreTemplate] -include = "../../../../cores/toy/configs/demo.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] +include = "../../../../../cores/toy/configs/demo.toml" count = 16 diff --git a/examples/chips/toy/configs/tiles/4core.toml b/examples/chips/toy/configs/designs/tiles/4core.toml similarity index 78% rename from examples/chips/toy/configs/tiles/4core.toml rename to examples/chips/toy/configs/designs/tiles/4core.toml index d3fa4bfc6..71e4fd902 100644 --- a/examples/chips/toy/configs/tiles/4core.toml +++ b/examples/chips/toy/configs/designs/tiles/4core.toml @@ -17,5 +17,6 @@ inputChannels = 4 defaultGroupCount = 1 [coreTemplate] -include = "../../../../cores/toy/configs/demo.toml" +core_ids = [0, 1, 2, 3] +include = "../../../../../cores/toy/configs/demo.toml" count = 4 diff --git a/examples/chips/toy/configs/tiles/8core.toml b/examples/chips/toy/configs/designs/tiles/8core.toml similarity index 76% rename from examples/chips/toy/configs/tiles/8core.toml rename to examples/chips/toy/configs/designs/tiles/8core.toml index 19e4622a0..2db67672c 100644 --- a/examples/chips/toy/configs/tiles/8core.toml +++ b/examples/chips/toy/configs/designs/tiles/8core.toml @@ -17,5 +17,6 @@ inputChannels = 8 defaultGroupCount = 1 [coreTemplate] -include = "../../../../cores/toy/configs/demo.toml" +core_ids = [0, 1, 2, 3, 4, 5, 6, 7] +include = "../../../../../cores/toy/configs/demo.toml" count = 8 diff --git a/examples/chips/toy/configs/tiles/default.toml b/examples/chips/toy/configs/designs/tiles/default.toml similarity index 78% rename from examples/chips/toy/configs/tiles/default.toml rename to examples/chips/toy/configs/designs/tiles/default.toml index 87b57262c..d83ed3031 100644 --- a/examples/chips/toy/configs/tiles/default.toml +++ b/examples/chips/toy/configs/designs/tiles/default.toml @@ -17,4 +17,5 @@ inputChannels = 7 defaultGroupCount = 1 [[cores]] -include = "../../../../cores/toy/core.toml" +core_id = 0 +include = "../../../../../cores/toy/configs/default.toml" diff --git a/examples/chips/toy/configs/designs/toy.toml b/examples/chips/toy/configs/designs/toy.toml new file mode 100644 index 000000000..f0f25b045 --- /dev/null +++ b/examples/chips/toy/configs/designs/toy.toml @@ -0,0 +1,8 @@ +# Toy: virtual single-core that mounts leftover balls. + +[top] +nTiles = 1 + +[[tiles]] +tile_id = 0 +include = "tiles/default.toml" diff --git a/examples/chips/toy/configs/toy.toml b/examples/chips/toy/configs/toy.toml deleted file mode 100644 index e40e1bfd7..000000000 --- a/examples/chips/toy/configs/toy.toml +++ /dev/null @@ -1,8 +0,0 @@ -# Toy Example - Top Level Configuration -# Single tile with 1 Buckyball core - -[top] -nTiles = 1 - -[[tiles]] -include = "tiles/default.toml" diff --git a/examples/chips/toy/regression/batch/bemu/workloads-elf-rushB.toml b/examples/chips/toy/regression/batch/bemu/workloads-elf-rushB.toml index c5a860572..687e22068 100644 --- a/examples/chips/toy/regression/batch/bemu/workloads-elf-rushB.toml +++ b/examples/chips/toy/regression/batch/bemu/workloads-elf-rushB.toml @@ -1,74 +1,12 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-rushB-bemu-run", - "toy_mvin_mvout_stride_test-rushB-bemu-run", - "toy_vecunit_matmul_16xn_ones-rushB-bemu-run", - "toy_vecunit_matmul_16xn_random1-rushB-bemu-run", - "toy_vecunit_matmul_16xn_random2-rushB-bemu-run", - "toy_vecunit_matmul_16xn_random3-rushB-bemu-run", - "toy_vecunit_matmul_16xn_zero_random-rushB-bemu-run", - "toy_vecunit_matmul_col_row_vector-rushB-bemu-run", - "toy_vecunit_matmul_identity_random-rushB-bemu-run", - "toy_vecunit_matmul_ones-rushB-bemu-run", - "toy_vecunit_matmul_random1-rushB-bemu-run", - "toy_vecunit_matmul_random2-rushB-bemu-run", - "toy_vecunit_matmul_zero_random-rushB-bemu-run", - "toy_relu_test-rushB-bemu-run", - "toy_transpose_i8_16x16_test-rushB-bemu-run", - "toy_transpose_i8_16x32_test-rushB-bemu-run", - "toy_transpose_i32_8x16_test-rushB-bemu-run", - "toy_transpose_bank_i8_n1_1024x16_test-rushB-bemu-run", - "toy_transpose_bank_i8_n2_1024x32_test-rushB-bemu-run", - "toy_transpose_bank_i32_n4_1024x16_test-rushB-bemu-run", - "toy_im2col_k1_test-rushB-bemu-run", - "toy_im2col_k3_test-rushB-bemu-run", - "toy_im2col_k3_partial_test-rushB-bemu-run", - "toy_im2col_k3_pad_test-rushB-bemu-run", - "toy_im2col_k5_test-rushB-bemu-run", - "toy_im2col_k7_test-rushB-bemu-run", - "toy_im2col_bank_k1_test-rushB-bemu-run", - "toy_im2col_bank_k3_test-rushB-bemu-run", - "toy_im2col_bank_k5_test-rushB-bemu-run", - "toy_im2col_bank_k7_test-rushB-bemu-run", - "toy_bdb_counter_test-rushB-bemu-run", - "toy_matrix_os_4x4_test-rushB-bemu-run", - "toy_matrix_ws_4x4_test-rushB-bemu-run", - "toy_matrix_os_16x16_test-rushB-bemu-run", - "toy_matrix_ws_16x16_test-rushB-bemu-run", - "toy_matrix_os_32x16_test-rushB-bemu-run", - "toy_matrix_ws_32x16_test-rushB-bemu-run", - "toy_matrix_os_16x16_ones-rushB-bemu-run", - "toy_matrix_os_32x32_ones-rushB-bemu-run", - "toy_matrix_os_tail_5x7x3_test-rushB-bemu-run", - "toy_matrix_mode_switch_test-rushB-bemu-run", - "toy_matrix_bank_os_1024x16x16_test-rushB-bemu-run", - "toy_matrix_bank_ws_1024x16x16_test-rushB-bemu-run", - "toy_gemmini_os_risc_atranspose_test-rushB-bemu-run", - "toy_gemmini_os_risc_btranspose_test-rushB-bemu-run", - "toy_gemmini_os_risc_abtranspose_test-rushB-bemu-run", - "toy_gemmini_os_risc_shift_test-rushB-bemu-run", - "toy_gemmini_ws_risc_basic_test-rushB-bemu-run", - "toy_fp2int_fp32_to_int32_test-rushB-bemu-run", - "toy_fp2int_fp32_to_int8_test-rushB-bemu-run", - "toy_int2fp_int32_to_fp32_test-rushB-bemu-run", - "toy_int2fp_int8_to_fp32_test-rushB-bemu-run", - "toy_int2fp_int32_to_int8_test-rushB-bemu-run", - "toy_int2fp_int32_to_fp32_col2_test-rushB-bemu-run", - "toy_int2fp_int32_to_fp32_col4_test-rushB-bemu-run", - "toy_fp2int_bank_fp32_to_int32_test-rushB-bemu-run", - "toy_fp2int_bank_fp32_to_int8_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_fp32_test-rushB-bemu-run", - "toy_int2fp_bank_int8_to_fp32_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_int8_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_fp32_col2_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_fp32_col4_test-rushB-bemu-run", - "toy_optest_buckyball_matmul_16x64_64x16-rushB-bemu-run", - "toy_optest_buckyball_matmul_64x16_16x64-rushB-bemu-run", - "toy_optest_buckyball_matmul_1024x16_16x1024-rushB-bemu-run", - "toy_optest_tile_matmul_127x17_17x127-rushB-bemu-run", - "toy_optest_tile_matmul_1x84_84x10-rushB-bemu-run", - "toy_optest_linalg_conv2d_nhwc_hwcf-rushB-bemu-run", - "toy_optest_linalg_conv2d_nhwc_fhwc-rushB-bemu-run", - "toy_optest_linalg_conv2d_nchw_fchw-rushB-bemu-run", + "toy-toy-ctest-mvin_mvout_test-rushB-bemu-run", + "toy-toy-ctest-mvin_mvout_stride_test-rushB-bemu-run", + "toy-toy-ctest-bdb_counter_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_shift_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_ws_risc_basic_test-rushB-bemu-run", ] diff --git a/examples/chips/toy/regression/batch/bemu/workloads-elf.toml b/examples/chips/toy/regression/batch/bemu/workloads-elf.toml index 4f9034e0b..2e6792d3f 100644 --- a/examples/chips/toy/regression/batch/bemu/workloads-elf.toml +++ b/examples/chips/toy/regression/batch/bemu/workloads-elf.toml @@ -1,81 +1,19 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-singlecore-baremetal", - "toy_mvin_mvout_stride_test-singlecore-baremetal", - "toy_vecunit_matmul_16xn_ones-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random1-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random2-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random3-singlecore-baremetal", - "toy_vecunit_matmul_16xn_zero_random-singlecore-baremetal", - "toy_vecunit_matmul_col_row_vector-singlecore-baremetal", - "toy_vecunit_matmul_identity_random-singlecore-baremetal", - "toy_vecunit_matmul_ones-singlecore-baremetal", - "toy_vecunit_matmul_random1-singlecore-baremetal", - "toy_vecunit_matmul_random2-singlecore-baremetal", - "toy_vecunit_matmul_zero_random-singlecore-baremetal", - "toy_relu_test-singlecore-baremetal", - "toy_transpose_i8_16x16_test-singlecore-baremetal", - "toy_transpose_i8_16x32_test-singlecore-baremetal", - "toy_transpose_i32_8x16_test-singlecore-baremetal", - "toy_transpose_bank_i8_n1_1024x16_test-singlecore-baremetal", - "toy_transpose_bank_i8_n2_1024x32_test-singlecore-baremetal", - "toy_transpose_bank_i32_n4_1024x16_test-singlecore-baremetal", - "toy_im2col_k1_test-singlecore-baremetal", - "toy_im2col_k3_test-singlecore-baremetal", - "toy_im2col_k3_partial_test-singlecore-baremetal", - "toy_im2col_k3_pad_test-singlecore-baremetal", - "toy_im2col_k5_test-singlecore-baremetal", - "toy_im2col_k7_test-singlecore-baremetal", - "toy_im2col_bank_k1_test-singlecore-baremetal", - "toy_im2col_bank_k3_test-singlecore-baremetal", - "toy_im2col_bank_k5_test-singlecore-baremetal", - "toy_im2col_bank_k7_test-singlecore-baremetal", - "toy_bdb_counter_test-singlecore-baremetal", - "toy_matrix_os_4x4_test-singlecore-baremetal", - "toy_matrix_ws_4x4_test-singlecore-baremetal", - "toy_matrix_os_16x16_test-singlecore-baremetal", - "toy_matrix_ws_16x16_test-singlecore-baremetal", - "toy_matrix_os_32x16_test-singlecore-baremetal", - "toy_matrix_ws_32x16_test-singlecore-baremetal", - "toy_matrix_os_16x16_ones-singlecore-baremetal", - "toy_matrix_os_32x32_ones-singlecore-baremetal", - "toy_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "toy_matrix_mode_switch_test-singlecore-baremetal", - "toy_matrix_bank_os_1024x16x16_test-singlecore-baremetal", - "toy_matrix_bank_ws_1024x16x16_test-singlecore-baremetal", - "toy_gemmini_os_cisc_basic_test-singlecore-baremetal", - "toy_gemmini_os_cisc_atranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_btranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_shift_test-singlecore-baremetal", - "toy_gemmini_os_risc_atranspose_test-singlecore-baremetal", - "toy_gemmini_os_risc_btranspose_test-singlecore-baremetal", - "toy_gemmini_os_risc_abtranspose_test-singlecore-baremetal", - "toy_gemmini_os_risc_shift_test-singlecore-baremetal", - "toy_gemmini_ws_cisc_conv_test-singlecore-baremetal", - "toy_gemmini_ws_risc_basic_test-singlecore-baremetal", - "toy_mmio_minimal_test-singlecore-baremetal", - "toy_mxfp2int_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int32_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int8_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", - "toy_fp2int_bank_fp32_to_int32_test-singlecore-baremetal", - "toy_fp2int_bank_fp32_to_int8_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_fp32_test-singlecore-baremetal", - "toy_int2fp_bank_int8_to_fp32_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_int8_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_fp32_col2_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_fp32_col4_test-singlecore-baremetal", - "toy_optest_buckyball_matmul_16x64_64x16_singlecore-baremetal", - "toy_optest_buckyball_matmul_64x16_16x64_singlecore-baremetal", - "toy_optest_buckyball_matmul_1024x16_16x1024_singlecore-baremetal", - "toy_optest_tile_matmul_127x17_17x127_singlecore-baremetal", - "toy_optest_tile_matmul_1x84_84x10_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_hwcf_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_fhwc_singlecore-baremetal", - "toy_optest_linalg_conv2d_nchw_fchw_singlecore-baremetal", + "toy-toy-ctest-mvin_mvout_test-baremetal", + "toy-toy-ctest-mvin_mvout_stride_test-baremetal", + "toy-toy-ctest-bdb_counter_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_basic_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_atranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_btranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_shift_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_shift_test-baremetal", + "toy-toy-ctest-gemmini_ws_cisc_conv_test-baremetal", + "toy-toy-ctest-gemmini_ws_risc_basic_test-baremetal", + "toy-toy-ctest-mmio_minimal_test-baremetal", + "toy-toy-ctest-mxfp2int_test-baremetal", ] diff --git a/examples/chips/toy/regression/batch/bemu/workloads-pk-rushB.toml b/examples/chips/toy/regression/batch/bemu/workloads-pk-rushB.toml index c5a860572..687e22068 100644 --- a/examples/chips/toy/regression/batch/bemu/workloads-pk-rushB.toml +++ b/examples/chips/toy/regression/batch/bemu/workloads-pk-rushB.toml @@ -1,74 +1,12 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-rushB-bemu-run", - "toy_mvin_mvout_stride_test-rushB-bemu-run", - "toy_vecunit_matmul_16xn_ones-rushB-bemu-run", - "toy_vecunit_matmul_16xn_random1-rushB-bemu-run", - "toy_vecunit_matmul_16xn_random2-rushB-bemu-run", - "toy_vecunit_matmul_16xn_random3-rushB-bemu-run", - "toy_vecunit_matmul_16xn_zero_random-rushB-bemu-run", - "toy_vecunit_matmul_col_row_vector-rushB-bemu-run", - "toy_vecunit_matmul_identity_random-rushB-bemu-run", - "toy_vecunit_matmul_ones-rushB-bemu-run", - "toy_vecunit_matmul_random1-rushB-bemu-run", - "toy_vecunit_matmul_random2-rushB-bemu-run", - "toy_vecunit_matmul_zero_random-rushB-bemu-run", - "toy_relu_test-rushB-bemu-run", - "toy_transpose_i8_16x16_test-rushB-bemu-run", - "toy_transpose_i8_16x32_test-rushB-bemu-run", - "toy_transpose_i32_8x16_test-rushB-bemu-run", - "toy_transpose_bank_i8_n1_1024x16_test-rushB-bemu-run", - "toy_transpose_bank_i8_n2_1024x32_test-rushB-bemu-run", - "toy_transpose_bank_i32_n4_1024x16_test-rushB-bemu-run", - "toy_im2col_k1_test-rushB-bemu-run", - "toy_im2col_k3_test-rushB-bemu-run", - "toy_im2col_k3_partial_test-rushB-bemu-run", - "toy_im2col_k3_pad_test-rushB-bemu-run", - "toy_im2col_k5_test-rushB-bemu-run", - "toy_im2col_k7_test-rushB-bemu-run", - "toy_im2col_bank_k1_test-rushB-bemu-run", - "toy_im2col_bank_k3_test-rushB-bemu-run", - "toy_im2col_bank_k5_test-rushB-bemu-run", - "toy_im2col_bank_k7_test-rushB-bemu-run", - "toy_bdb_counter_test-rushB-bemu-run", - "toy_matrix_os_4x4_test-rushB-bemu-run", - "toy_matrix_ws_4x4_test-rushB-bemu-run", - "toy_matrix_os_16x16_test-rushB-bemu-run", - "toy_matrix_ws_16x16_test-rushB-bemu-run", - "toy_matrix_os_32x16_test-rushB-bemu-run", - "toy_matrix_ws_32x16_test-rushB-bemu-run", - "toy_matrix_os_16x16_ones-rushB-bemu-run", - "toy_matrix_os_32x32_ones-rushB-bemu-run", - "toy_matrix_os_tail_5x7x3_test-rushB-bemu-run", - "toy_matrix_mode_switch_test-rushB-bemu-run", - "toy_matrix_bank_os_1024x16x16_test-rushB-bemu-run", - "toy_matrix_bank_ws_1024x16x16_test-rushB-bemu-run", - "toy_gemmini_os_risc_atranspose_test-rushB-bemu-run", - "toy_gemmini_os_risc_btranspose_test-rushB-bemu-run", - "toy_gemmini_os_risc_abtranspose_test-rushB-bemu-run", - "toy_gemmini_os_risc_shift_test-rushB-bemu-run", - "toy_gemmini_ws_risc_basic_test-rushB-bemu-run", - "toy_fp2int_fp32_to_int32_test-rushB-bemu-run", - "toy_fp2int_fp32_to_int8_test-rushB-bemu-run", - "toy_int2fp_int32_to_fp32_test-rushB-bemu-run", - "toy_int2fp_int8_to_fp32_test-rushB-bemu-run", - "toy_int2fp_int32_to_int8_test-rushB-bemu-run", - "toy_int2fp_int32_to_fp32_col2_test-rushB-bemu-run", - "toy_int2fp_int32_to_fp32_col4_test-rushB-bemu-run", - "toy_fp2int_bank_fp32_to_int32_test-rushB-bemu-run", - "toy_fp2int_bank_fp32_to_int8_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_fp32_test-rushB-bemu-run", - "toy_int2fp_bank_int8_to_fp32_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_int8_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_fp32_col2_test-rushB-bemu-run", - "toy_int2fp_bank_int32_to_fp32_col4_test-rushB-bemu-run", - "toy_optest_buckyball_matmul_16x64_64x16-rushB-bemu-run", - "toy_optest_buckyball_matmul_64x16_16x64-rushB-bemu-run", - "toy_optest_buckyball_matmul_1024x16_16x1024-rushB-bemu-run", - "toy_optest_tile_matmul_127x17_17x127-rushB-bemu-run", - "toy_optest_tile_matmul_1x84_84x10-rushB-bemu-run", - "toy_optest_linalg_conv2d_nhwc_hwcf-rushB-bemu-run", - "toy_optest_linalg_conv2d_nhwc_fhwc-rushB-bemu-run", - "toy_optest_linalg_conv2d_nchw_fchw-rushB-bemu-run", + "toy-toy-ctest-mvin_mvout_test-rushB-bemu-run", + "toy-toy-ctest-mvin_mvout_stride_test-rushB-bemu-run", + "toy-toy-ctest-bdb_counter_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_os_risc_shift_test-rushB-bemu-run", + "toy-toy-ctest-gemmini_ws_risc_basic_test-rushB-bemu-run", ] diff --git a/examples/chips/toy/regression/batch/bemu/workloads-pk.toml b/examples/chips/toy/regression/batch/bemu/workloads-pk.toml index 115f53d67..2099a31a0 100644 --- a/examples/chips/toy/regression/batch/bemu/workloads-pk.toml +++ b/examples/chips/toy/regression/batch/bemu/workloads-pk.toml @@ -1,81 +1,19 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-linux", - "toy_mvin_mvout_stride_test-linux", - "toy_vecunit_matmul_16xn_ones-linux", - "toy_vecunit_matmul_16xn_random1-linux", - "toy_vecunit_matmul_16xn_random2-linux", - "toy_vecunit_matmul_16xn_random3-linux", - "toy_vecunit_matmul_16xn_zero_random-linux", - "toy_vecunit_matmul_col_row_vector-linux", - "toy_vecunit_matmul_identity_random-linux", - "toy_vecunit_matmul_ones-linux", - "toy_vecunit_matmul_random1-linux", - "toy_vecunit_matmul_random2-linux", - "toy_vecunit_matmul_zero_random-linux", - "toy_relu_test-linux", - "toy_transpose_i8_16x16_test-linux", - "toy_transpose_i8_16x32_test-linux", - "toy_transpose_i32_8x16_test-linux", - "toy_transpose_bank_i8_n1_1024x16_test-linux", - "toy_transpose_bank_i8_n2_1024x32_test-linux", - "toy_transpose_bank_i32_n4_1024x16_test-linux", - "toy_im2col_k1_test-linux", - "toy_im2col_k3_test-linux", - "toy_im2col_k3_partial_test-linux", - "toy_im2col_k3_pad_test-linux", - "toy_im2col_k5_test-linux", - "toy_im2col_k7_test-linux", - "toy_im2col_bank_k1_test-linux", - "toy_im2col_bank_k3_test-linux", - "toy_im2col_bank_k5_test-linux", - "toy_im2col_bank_k7_test-linux", - "toy_bdb_counter_test-linux", - "toy_matrix_os_4x4_test-linux", - "toy_matrix_ws_4x4_test-linux", - "toy_matrix_os_16x16_test-linux", - "toy_matrix_ws_16x16_test-linux", - "toy_matrix_os_32x16_test-linux", - "toy_matrix_ws_32x16_test-linux", - "toy_matrix_os_16x16_ones-linux", - "toy_matrix_os_32x32_ones-linux", - "toy_matrix_os_tail_5x7x3_test-linux", - "toy_matrix_mode_switch_test-linux", - "toy_matrix_bank_os_1024x16x16_test-linux", - "toy_matrix_bank_ws_1024x16x16_test-linux", - "toy_gemmini_os_cisc_basic_test-linux", - "toy_gemmini_os_cisc_atranspose_test-linux", - "toy_gemmini_os_cisc_btranspose_test-linux", - "toy_gemmini_os_cisc_shift_test-linux", - "toy_gemmini_os_risc_atranspose_test-linux", - "toy_gemmini_os_risc_btranspose_test-linux", - "toy_gemmini_os_risc_abtranspose_test-linux", - "toy_gemmini_os_risc_shift_test-linux", - "toy_gemmini_ws_cisc_conv_test-linux", - "toy_gemmini_ws_risc_basic_test-linux", - "toy_mmio_minimal_test-linux", - "toy_mxfp2int_test-linux", - "toy_fp2int_fp32_to_int32_test-linux", - "toy_fp2int_fp32_to_int8_test-linux", - "toy_int2fp_int32_to_fp32_test-linux", - "toy_int2fp_int8_to_fp32_test-linux", - "toy_int2fp_int32_to_int8_test-linux", - "toy_int2fp_int32_to_fp32_col2_test-linux", - "toy_int2fp_int32_to_fp32_col4_test-linux", - "toy_fp2int_bank_fp32_to_int32_test-linux", - "toy_fp2int_bank_fp32_to_int8_test-linux", - "toy_int2fp_bank_int32_to_fp32_test-linux", - "toy_int2fp_bank_int8_to_fp32_test-linux", - "toy_int2fp_bank_int32_to_int8_test-linux", - "toy_int2fp_bank_int32_to_fp32_col2_test-linux", - "toy_int2fp_bank_int32_to_fp32_col4_test-linux", - "toy_optest_buckyball_matmul_16x64_64x16-linux", - "toy_optest_buckyball_matmul_64x16_16x64-linux", - "toy_optest_buckyball_matmul_1024x16_16x1024-linux", - "toy_optest_tile_matmul_127x17_17x127-linux", - "toy_optest_tile_matmul_1x84_84x10-linux", - "toy_optest_linalg_conv2d_nhwc_hwcf-linux", - "toy_optest_linalg_conv2d_nhwc_fhwc-linux", - "toy_optest_linalg_conv2d_nchw_fchw-linux", + "toy-toy-ctest-mvin_mvout_test-linux", + "toy-toy-ctest-mvin_mvout_stride_test-linux", + "toy-toy-ctest-bdb_counter_test-linux", + "toy-toy-ctest-gemmini_os_cisc_basic_test-linux", + "toy-toy-ctest-gemmini_os_cisc_atranspose_test-linux", + "toy-toy-ctest-gemmini_os_cisc_btranspose_test-linux", + "toy-toy-ctest-gemmini_os_cisc_shift_test-linux", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-linux", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-linux", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-linux", + "toy-toy-ctest-gemmini_os_risc_shift_test-linux", + "toy-toy-ctest-gemmini_ws_cisc_conv_test-linux", + "toy-toy-ctest-gemmini_ws_risc_basic_test-linux", + "toy-toy-ctest-mmio_minimal_test-linux", + "toy-toy-ctest-mxfp2int_test-linux", ] diff --git a/examples/chips/toy/regression/batch/p2e/workloads-elf.toml b/examples/chips/toy/regression/batch/p2e/workloads-elf.toml index 4f9034e0b..2e6792d3f 100644 --- a/examples/chips/toy/regression/batch/p2e/workloads-elf.toml +++ b/examples/chips/toy/regression/batch/p2e/workloads-elf.toml @@ -1,81 +1,19 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-singlecore-baremetal", - "toy_mvin_mvout_stride_test-singlecore-baremetal", - "toy_vecunit_matmul_16xn_ones-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random1-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random2-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random3-singlecore-baremetal", - "toy_vecunit_matmul_16xn_zero_random-singlecore-baremetal", - "toy_vecunit_matmul_col_row_vector-singlecore-baremetal", - "toy_vecunit_matmul_identity_random-singlecore-baremetal", - "toy_vecunit_matmul_ones-singlecore-baremetal", - "toy_vecunit_matmul_random1-singlecore-baremetal", - "toy_vecunit_matmul_random2-singlecore-baremetal", - "toy_vecunit_matmul_zero_random-singlecore-baremetal", - "toy_relu_test-singlecore-baremetal", - "toy_transpose_i8_16x16_test-singlecore-baremetal", - "toy_transpose_i8_16x32_test-singlecore-baremetal", - "toy_transpose_i32_8x16_test-singlecore-baremetal", - "toy_transpose_bank_i8_n1_1024x16_test-singlecore-baremetal", - "toy_transpose_bank_i8_n2_1024x32_test-singlecore-baremetal", - "toy_transpose_bank_i32_n4_1024x16_test-singlecore-baremetal", - "toy_im2col_k1_test-singlecore-baremetal", - "toy_im2col_k3_test-singlecore-baremetal", - "toy_im2col_k3_partial_test-singlecore-baremetal", - "toy_im2col_k3_pad_test-singlecore-baremetal", - "toy_im2col_k5_test-singlecore-baremetal", - "toy_im2col_k7_test-singlecore-baremetal", - "toy_im2col_bank_k1_test-singlecore-baremetal", - "toy_im2col_bank_k3_test-singlecore-baremetal", - "toy_im2col_bank_k5_test-singlecore-baremetal", - "toy_im2col_bank_k7_test-singlecore-baremetal", - "toy_bdb_counter_test-singlecore-baremetal", - "toy_matrix_os_4x4_test-singlecore-baremetal", - "toy_matrix_ws_4x4_test-singlecore-baremetal", - "toy_matrix_os_16x16_test-singlecore-baremetal", - "toy_matrix_ws_16x16_test-singlecore-baremetal", - "toy_matrix_os_32x16_test-singlecore-baremetal", - "toy_matrix_ws_32x16_test-singlecore-baremetal", - "toy_matrix_os_16x16_ones-singlecore-baremetal", - "toy_matrix_os_32x32_ones-singlecore-baremetal", - "toy_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "toy_matrix_mode_switch_test-singlecore-baremetal", - "toy_matrix_bank_os_1024x16x16_test-singlecore-baremetal", - "toy_matrix_bank_ws_1024x16x16_test-singlecore-baremetal", - "toy_gemmini_os_cisc_basic_test-singlecore-baremetal", - "toy_gemmini_os_cisc_atranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_btranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_shift_test-singlecore-baremetal", - "toy_gemmini_os_risc_atranspose_test-singlecore-baremetal", - "toy_gemmini_os_risc_btranspose_test-singlecore-baremetal", - "toy_gemmini_os_risc_abtranspose_test-singlecore-baremetal", - "toy_gemmini_os_risc_shift_test-singlecore-baremetal", - "toy_gemmini_ws_cisc_conv_test-singlecore-baremetal", - "toy_gemmini_ws_risc_basic_test-singlecore-baremetal", - "toy_mmio_minimal_test-singlecore-baremetal", - "toy_mxfp2int_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int32_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int8_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", - "toy_fp2int_bank_fp32_to_int32_test-singlecore-baremetal", - "toy_fp2int_bank_fp32_to_int8_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_fp32_test-singlecore-baremetal", - "toy_int2fp_bank_int8_to_fp32_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_int8_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_fp32_col2_test-singlecore-baremetal", - "toy_int2fp_bank_int32_to_fp32_col4_test-singlecore-baremetal", - "toy_optest_buckyball_matmul_16x64_64x16_singlecore-baremetal", - "toy_optest_buckyball_matmul_64x16_16x64_singlecore-baremetal", - "toy_optest_buckyball_matmul_1024x16_16x1024_singlecore-baremetal", - "toy_optest_tile_matmul_127x17_17x127_singlecore-baremetal", - "toy_optest_tile_matmul_1x84_84x10_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_hwcf_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_fhwc_singlecore-baremetal", - "toy_optest_linalg_conv2d_nchw_fchw_singlecore-baremetal", + "toy-toy-ctest-mvin_mvout_test-baremetal", + "toy-toy-ctest-mvin_mvout_stride_test-baremetal", + "toy-toy-ctest-bdb_counter_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_basic_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_atranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_btranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_shift_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_risc_shift_test-baremetal", + "toy-toy-ctest-gemmini_ws_cisc_conv_test-baremetal", + "toy-toy-ctest-gemmini_ws_risc_basic_test-baremetal", + "toy-toy-ctest-mmio_minimal_test-baremetal", + "toy-toy-ctest-mxfp2int_test-baremetal", ] diff --git a/examples/chips/toy/regression/batch/verilator/workloads-elf-diff.toml b/examples/chips/toy/regression/batch/verilator/workloads-elf-diff.toml index d95b284dd..8f0cb514c 100644 --- a/examples/chips/toy/regression/batch/verilator/workloads-elf-diff.toml +++ b/examples/chips/toy/regression/batch/verilator/workloads-elf-diff.toml @@ -3,58 +3,14 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-singlecore-baremetal", - "toy_vecunit_matmul_16xn_ones-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random1-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random2-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random3-singlecore-baremetal", - "toy_vecunit_matmul_16xn_zero_random-singlecore-baremetal", - "toy_vecunit_matmul_col_row_vector-singlecore-baremetal", - "toy_vecunit_matmul_identity_random-singlecore-baremetal", - "toy_vecunit_matmul_ones-singlecore-baremetal", - "toy_vecunit_matmul_random1-singlecore-baremetal", - "toy_vecunit_matmul_random2-singlecore-baremetal", - "toy_vecunit_matmul_zero_random-singlecore-baremetal", - "toy_relu_test-singlecore-baremetal", - "toy_transpose_i8_16x16_test-singlecore-baremetal", - "toy_transpose_i8_16x32_test-singlecore-baremetal", - "toy_transpose_i32_8x16_test-singlecore-baremetal", - "toy_im2col_k1_test-singlecore-baremetal", - "toy_im2col_k3_test-singlecore-baremetal", - "toy_im2col_k3_partial_test-singlecore-baremetal", - "toy_im2col_k3_pad_test-singlecore-baremetal", - "toy_im2col_k5_test-singlecore-baremetal", - "toy_im2col_k7_test-singlecore-baremetal", - "toy_bdb_counter_test-singlecore-baremetal", - "toy_matrix_os_4x4_test-singlecore-baremetal", - "toy_matrix_ws_4x4_test-singlecore-baremetal", - "toy_matrix_os_16x16_test-singlecore-baremetal", - "toy_matrix_ws_16x16_test-singlecore-baremetal", - "toy_matrix_os_32x16_test-singlecore-baremetal", - "toy_matrix_ws_32x16_test-singlecore-baremetal", - "toy_matrix_os_16x16_ones-singlecore-baremetal", - "toy_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "toy_matrix_mode_switch_test-singlecore-baremetal", - "toy_gemmini_os_cisc_basic_test-singlecore-baremetal", - "toy_gemmini_os_cisc_atranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_btranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_shift_test-singlecore-baremetal", - "toy_gemmini_ws_cisc_conv_test-singlecore-baremetal", - "toy_mmio_minimal_test-singlecore-baremetal", - "toy_mxfp2int_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int32_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int8_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", - "toy_mvin_mvout_stride_test-singlecore-baremetal", - "toy_transpose_matmul-singlecore-baremetal", - "toy_optest_buckyball_matmul_16x64_64x16_singlecore-baremetal", - "toy_optest_buckyball_matmul_64x16_16x64_singlecore-baremetal", - "toy_optest_tile_matmul_1x84_84x10_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_hwcf_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_fhwc_singlecore-baremetal", - "toy_optest_linalg_conv2d_nchw_fchw_singlecore-baremetal", + "toy-toy-ctest-mvin_mvout_test-baremetal", + "toy-toy-ctest-bdb_counter_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_basic_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_atranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_btranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_shift_test-baremetal", + "toy-toy-ctest-gemmini_ws_cisc_conv_test-baremetal", + "toy-toy-ctest-mmio_minimal_test-baremetal", + "toy-toy-ctest-mxfp2int_test-baremetal", + "toy-toy-ctest-mvin_mvout_stride_test-baremetal", ] diff --git a/examples/chips/toy/regression/batch/verilator/workloads-elf-rushB.toml b/examples/chips/toy/regression/batch/verilator/workloads-elf-rushB.toml index 4b2efde5f..fac23af3b 100644 --- a/examples/chips/toy/regression/batch/verilator/workloads-elf-rushB.toml +++ b/examples/chips/toy/regression/batch/verilator/workloads-elf-rushB.toml @@ -1,51 +1,7 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-rushB-verilator-run", - "toy_vecunit_matmul_16xn_ones-rushB-verilator-run", - "toy_vecunit_matmul_16xn_random1-rushB-verilator-run", - "toy_vecunit_matmul_16xn_random2-rushB-verilator-run", - "toy_vecunit_matmul_16xn_random3-rushB-verilator-run", - "toy_vecunit_matmul_16xn_zero_random-rushB-verilator-run", - "toy_vecunit_matmul_col_row_vector-rushB-verilator-run", - "toy_vecunit_matmul_identity_random-rushB-verilator-run", - "toy_vecunit_matmul_ones-rushB-verilator-run", - "toy_vecunit_matmul_random1-rushB-verilator-run", - "toy_vecunit_matmul_random2-rushB-verilator-run", - "toy_vecunit_matmul_zero_random-rushB-verilator-run", - "toy_relu_test-rushB-verilator-run", - "toy_transpose_i8_16x16_test-rushB-verilator-run", - "toy_transpose_i8_16x32_test-rushB-verilator-run", - "toy_transpose_i32_8x16_test-rushB-verilator-run", - "toy_im2col_k1_test-rushB-verilator-run", - "toy_im2col_k3_test-rushB-verilator-run", - "toy_im2col_k3_partial_test-rushB-verilator-run", - "toy_im2col_k3_pad_test-rushB-verilator-run", - "toy_im2col_k5_test-rushB-verilator-run", - "toy_im2col_k7_test-rushB-verilator-run", - "toy_bdb_counter_test-rushB-verilator-run", - "toy_matrix_os_4x4_test-rushB-verilator-run", - "toy_matrix_ws_4x4_test-rushB-verilator-run", - "toy_matrix_os_16x16_test-rushB-verilator-run", - "toy_matrix_ws_16x16_test-rushB-verilator-run", - "toy_matrix_os_32x16_test-rushB-verilator-run", - "toy_matrix_ws_32x16_test-rushB-verilator-run", - "toy_matrix_os_16x16_ones-rushB-verilator-run", - "toy_matrix_os_tail_5x7x3_test-rushB-verilator-run", - "toy_matrix_mode_switch_test-rushB-verilator-run", - "toy_fp2int_fp32_to_int32_test-rushB-verilator-run", - "toy_fp2int_fp32_to_int8_test-rushB-verilator-run", - "toy_int2fp_int32_to_fp32_test-rushB-verilator-run", - "toy_int2fp_int8_to_fp32_test-rushB-verilator-run", - "toy_int2fp_int32_to_int8_test-rushB-verilator-run", - "toy_int2fp_int32_to_fp32_col2_test-rushB-verilator-run", - "toy_int2fp_int32_to_fp32_col4_test-rushB-verilator-run", - "toy_mvin_mvout_stride_test-rushB-verilator-run", - "toy_transpose_matmul-rushB-verilator-run", - "toy_optest_buckyball_matmul_16x64_64x16-rushB-verilator-run", - "toy_optest_buckyball_matmul_64x16_16x64-rushB-verilator-run", - "toy_optest_tile_matmul_1x84_84x10-rushB-verilator-run", - "toy_optest_linalg_conv2d_nhwc_hwcf-rushB-verilator-run", - "toy_optest_linalg_conv2d_nhwc_fhwc-rushB-verilator-run", - "toy_optest_linalg_conv2d_nchw_fchw-rushB-verilator-run", + "toy-toy-ctest-mvin_mvout_test-rushB-verilator-run", + "toy-toy-ctest-bdb_counter_test-rushB-verilator-run", + "toy-toy-ctest-mvin_mvout_stride_test-rushB-verilator-run", ] diff --git a/examples/chips/toy/regression/batch/verilator/workloads-elf.toml b/examples/chips/toy/regression/batch/verilator/workloads-elf.toml index 200ad99fa..0d3e5b513 100644 --- a/examples/chips/toy/regression/batch/verilator/workloads-elf.toml +++ b/examples/chips/toy/regression/batch/verilator/workloads-elf.toml @@ -1,65 +1,21 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-singlecore-baremetal", - "toy_vecunit_matmul_16xn_ones-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random1-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random2-singlecore-baremetal", - "toy_vecunit_matmul_16xn_random3-singlecore-baremetal", - "toy_vecunit_matmul_16xn_zero_random-singlecore-baremetal", - "toy_vecunit_matmul_col_row_vector-singlecore-baremetal", - "toy_vecunit_matmul_identity_random-singlecore-baremetal", - "toy_vecunit_matmul_ones-singlecore-baremetal", - "toy_vecunit_matmul_random1-singlecore-baremetal", - "toy_vecunit_matmul_random2-singlecore-baremetal", - "toy_vecunit_matmul_zero_random-singlecore-baremetal", - "toy_relu_test-singlecore-baremetal", - "toy_transpose_i8_16x16_test-singlecore-baremetal", - "toy_transpose_i8_16x32_test-singlecore-baremetal", - "toy_transpose_i32_8x16_test-singlecore-baremetal", - "toy_im2col_k1_test-singlecore-baremetal", - "toy_im2col_k3_test-singlecore-baremetal", - "toy_im2col_k3_partial_test-singlecore-baremetal", - "toy_im2col_k3_pad_test-singlecore-baremetal", - "toy_im2col_k5_test-singlecore-baremetal", - "toy_im2col_k7_test-singlecore-baremetal", - "toy_bdb_counter_test-singlecore-baremetal", - "toy_matrix_os_4x4_test-singlecore-baremetal", - "toy_matrix_ws_4x4_test-singlecore-baremetal", - "toy_matrix_os_16x16_test-singlecore-baremetal", - "toy_matrix_ws_16x16_test-singlecore-baremetal", - "toy_matrix_os_32x16_test-singlecore-baremetal", - "toy_matrix_ws_32x16_test-singlecore-baremetal", - "toy_matrix_os_16x16_ones-singlecore-baremetal", - "toy_matrix_os_tail_5x7x3_test-singlecore-baremetal", - "toy_matrix_mode_switch_test-singlecore-baremetal", - "toy_gemmini_os_cisc_basic_test-singlecore-baremetal", - "toy_gemmini_os_cisc_atranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_btranspose_test-singlecore-baremetal", - "toy_gemmini_os_cisc_shift_test-singlecore-baremetal", - "toy_gemmini_ws_cisc_conv_test-singlecore-baremetal", - "toy_mmio_minimal_test-singlecore-baremetal", - "toy_mxfp2int_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int32_test-singlecore-baremetal", - "toy_fp2int_fp32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int8_to_fp32_test-singlecore-baremetal", - "toy_int2fp_int32_to_int8_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col2_test-singlecore-baremetal", - "toy_int2fp_int32_to_fp32_col4_test-singlecore-baremetal", - "toy_tlb_bare_mode_test-singlecore-baremetal", - "toy_tlb_sv39_identity_test-singlecore-baremetal", - "toy_tlb_sfence_flush_test-singlecore-baremetal", - "toy_tlb_repeated_sfence_test-singlecore-baremetal", - "toy_tlb_sv39_nonidentity_test-singlecore-baremetal", - "toy_tlb_sv39_4k_alias_test-singlecore-baremetal", - "toy_tlb_sfence_remap_test-singlecore-baremetal", - "toy_mvin_mvout_stride_test-singlecore-baremetal", - "toy_transpose_matmul-singlecore-baremetal", - "toy_optest_buckyball_matmul_16x64_64x16_singlecore-baremetal", - "toy_optest_buckyball_matmul_64x16_16x64_singlecore-baremetal", - "toy_optest_tile_matmul_1x84_84x10_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_hwcf_singlecore-baremetal", - "toy_optest_linalg_conv2d_nhwc_fhwc_singlecore-baremetal", - "toy_optest_linalg_conv2d_nchw_fchw_singlecore-baremetal", + "toy-toy-ctest-mvin_mvout_test-baremetal", + "toy-toy-ctest-bdb_counter_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_basic_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_atranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_btranspose_test-baremetal", + "toy-toy-ctest-gemmini_os_cisc_shift_test-baremetal", + "toy-toy-ctest-gemmini_ws_cisc_conv_test-baremetal", + "toy-toy-ctest-mmio_minimal_test-baremetal", + "toy-toy-ctest-mxfp2int_test-baremetal", + "toy-toy-ctest-tlb_bare_mode_test-baremetal", + "toy-toy-ctest-tlb_sv39_identity_test-baremetal", + "toy-toy-ctest-tlb_sfence_flush_test-baremetal", + "toy-toy-ctest-tlb_repeated_sfence_test-baremetal", + "toy-toy-ctest-tlb_sv39_nonidentity_test-baremetal", + "toy-toy-ctest-tlb_sv39_4k_alias_test-baremetal", + "toy-toy-ctest-tlb_sfence_remap_test-baremetal", + "toy-toy-ctest-mvin_mvout_stride_test-baremetal", ] diff --git a/examples/chips/toy/regression/batch/verilator/workloads-pk-rushB.toml b/examples/chips/toy/regression/batch/verilator/workloads-pk-rushB.toml index cbc5e0341..22656f30d 100644 --- a/examples/chips/toy/regression/batch/verilator/workloads-pk-rushB.toml +++ b/examples/chips/toy/regression/batch/verilator/workloads-pk-rushB.toml @@ -1,55 +1,12 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-rushB-verilator-run", - "toy_mvin_mvout_stride_test-rushB-verilator-run", - "toy_vecunit_matmul_16xn_ones-rushB-verilator-run", - "toy_vecunit_matmul_16xn_random1-rushB-verilator-run", - "toy_vecunit_matmul_16xn_random2-rushB-verilator-run", - "toy_vecunit_matmul_16xn_random3-rushB-verilator-run", - "toy_vecunit_matmul_16xn_zero_random-rushB-verilator-run", - "toy_vecunit_matmul_col_row_vector-rushB-verilator-run", - "toy_vecunit_matmul_identity_random-rushB-verilator-run", - "toy_vecunit_matmul_ones-rushB-verilator-run", - "toy_vecunit_matmul_random1-rushB-verilator-run", - "toy_vecunit_matmul_random2-rushB-verilator-run", - "toy_vecunit_matmul_zero_random-rushB-verilator-run", - "toy_relu_test-rushB-verilator-run", - "toy_transpose_i8_16x16_test-rushB-verilator-run", - "toy_transpose_i8_16x32_test-rushB-verilator-run", - "toy_transpose_i32_8x16_test-rushB-verilator-run", - "toy_im2col_k1_test-rushB-verilator-run", - "toy_im2col_k3_test-rushB-verilator-run", - "toy_im2col_k3_partial_test-rushB-verilator-run", - "toy_im2col_k3_pad_test-rushB-verilator-run", - "toy_im2col_k5_test-rushB-verilator-run", - "toy_im2col_k7_test-rushB-verilator-run", - "toy_bdb_counter_test-rushB-verilator-run", - "toy_matrix_os_4x4_test-rushB-verilator-run", - "toy_matrix_ws_4x4_test-rushB-verilator-run", - "toy_matrix_os_16x16_test-rushB-verilator-run", - "toy_matrix_ws_16x16_test-rushB-verilator-run", - "toy_matrix_os_32x16_test-rushB-verilator-run", - "toy_matrix_ws_32x16_test-rushB-verilator-run", - "toy_matrix_os_16x16_ones-rushB-verilator-run", - "toy_matrix_os_tail_5x7x3_test-rushB-verilator-run", - "toy_matrix_mode_switch_test-rushB-verilator-run", - "toy_gemmini_os_risc_atranspose_test-rushB-verilator-run", - "toy_gemmini_os_risc_btranspose_test-rushB-verilator-run", - "toy_gemmini_os_risc_abtranspose_test-rushB-verilator-run", - "toy_gemmini_os_risc_shift_test-rushB-verilator-run", - "toy_gemmini_ws_risc_basic_test-rushB-verilator-run", - "toy_fp2int_fp32_to_int32_test-rushB-verilator-run", - "toy_fp2int_fp32_to_int8_test-rushB-verilator-run", - "toy_int2fp_int32_to_fp32_test-rushB-verilator-run", - "toy_int2fp_int8_to_fp32_test-rushB-verilator-run", - "toy_int2fp_int32_to_int8_test-rushB-verilator-run", - "toy_int2fp_int32_to_fp32_col2_test-rushB-verilator-run", - "toy_int2fp_int32_to_fp32_col4_test-rushB-verilator-run", - "toy_optest_buckyball_matmul_16x64_64x16-rushB-verilator-run", - "toy_optest_buckyball_matmul_64x16_16x64-rushB-verilator-run", - "toy_optest_tile_matmul_1x84_84x10-rushB-verilator-run", - "toy_optest_linalg_conv2d_nhwc_hwcf-rushB-verilator-run", - "toy_optest_linalg_conv2d_nhwc_fhwc-rushB-verilator-run", - "toy_optest_linalg_conv2d_nchw_fchw-rushB-verilator-run", + "toy-toy-ctest-mvin_mvout_test-rushB-verilator-run", + "toy-toy-ctest-mvin_mvout_stride_test-rushB-verilator-run", + "toy-toy-ctest-bdb_counter_test-rushB-verilator-run", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-rushB-verilator-run", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-rushB-verilator-run", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-rushB-verilator-run", + "toy-toy-ctest-gemmini_os_risc_shift_test-rushB-verilator-run", + "toy-toy-ctest-gemmini_ws_risc_basic_test-rushB-verilator-run", ] diff --git a/examples/chips/toy/regression/batch/verilator/workloads-pk.toml b/examples/chips/toy/regression/batch/verilator/workloads-pk.toml index 75beb1843..2099a31a0 100644 --- a/examples/chips/toy/regression/batch/verilator/workloads-pk.toml +++ b/examples/chips/toy/regression/batch/verilator/workloads-pk.toml @@ -1,62 +1,19 @@ [workloads] search_path = "workloads/src" tests = [ - "toy_mvin_mvout_test-linux", - "toy_mvin_mvout_stride_test-linux", - "toy_vecunit_matmul_16xn_ones-linux", - "toy_vecunit_matmul_16xn_random1-linux", - "toy_vecunit_matmul_16xn_random2-linux", - "toy_vecunit_matmul_16xn_random3-linux", - "toy_vecunit_matmul_16xn_zero_random-linux", - "toy_vecunit_matmul_col_row_vector-linux", - "toy_vecunit_matmul_identity_random-linux", - "toy_vecunit_matmul_ones-linux", - "toy_vecunit_matmul_random1-linux", - "toy_vecunit_matmul_random2-linux", - "toy_vecunit_matmul_zero_random-linux", - "toy_relu_test-linux", - "toy_transpose_i8_16x16_test-linux", - "toy_transpose_i8_16x32_test-linux", - "toy_transpose_i32_8x16_test-linux", - "toy_im2col_k1_test-linux", - "toy_im2col_k3_test-linux", - "toy_im2col_k3_partial_test-linux", - "toy_im2col_k3_pad_test-linux", - "toy_im2col_k5_test-linux", - "toy_im2col_k7_test-linux", - "toy_bdb_counter_test-linux", - "toy_matrix_os_4x4_test-linux", - "toy_matrix_ws_4x4_test-linux", - "toy_matrix_os_16x16_test-linux", - "toy_matrix_ws_16x16_test-linux", - "toy_matrix_os_32x16_test-linux", - "toy_matrix_ws_32x16_test-linux", - "toy_matrix_os_16x16_ones-linux", - "toy_matrix_os_tail_5x7x3_test-linux", - "toy_matrix_mode_switch_test-linux", - "toy_gemmini_os_cisc_basic_test-linux", - "toy_gemmini_os_cisc_atranspose_test-linux", - "toy_gemmini_os_cisc_btranspose_test-linux", - "toy_gemmini_os_cisc_shift_test-linux", - "toy_gemmini_os_risc_atranspose_test-linux", - "toy_gemmini_os_risc_btranspose_test-linux", - "toy_gemmini_os_risc_abtranspose_test-linux", - "toy_gemmini_os_risc_shift_test-linux", - "toy_gemmini_ws_cisc_conv_test-linux", - "toy_gemmini_ws_risc_basic_test-linux", - "toy_mmio_minimal_test-linux", - "toy_mxfp2int_test-linux", - "toy_fp2int_fp32_to_int32_test-linux", - "toy_fp2int_fp32_to_int8_test-linux", - "toy_int2fp_int32_to_fp32_test-linux", - "toy_int2fp_int8_to_fp32_test-linux", - "toy_int2fp_int32_to_int8_test-linux", - "toy_int2fp_int32_to_fp32_col2_test-linux", - "toy_int2fp_int32_to_fp32_col4_test-linux", - "toy_optest_buckyball_matmul_16x64_64x16-linux", - "toy_optest_buckyball_matmul_64x16_16x64-linux", - "toy_optest_tile_matmul_1x84_84x10-linux", - "toy_optest_linalg_conv2d_nhwc_hwcf-linux", - "toy_optest_linalg_conv2d_nhwc_fhwc-linux", - "toy_optest_linalg_conv2d_nchw_fchw-linux", + "toy-toy-ctest-mvin_mvout_test-linux", + "toy-toy-ctest-mvin_mvout_stride_test-linux", + "toy-toy-ctest-bdb_counter_test-linux", + "toy-toy-ctest-gemmini_os_cisc_basic_test-linux", + "toy-toy-ctest-gemmini_os_cisc_atranspose_test-linux", + "toy-toy-ctest-gemmini_os_cisc_btranspose_test-linux", + "toy-toy-ctest-gemmini_os_cisc_shift_test-linux", + "toy-toy-ctest-gemmini_os_risc_atranspose_test-linux", + "toy-toy-ctest-gemmini_os_risc_btranspose_test-linux", + "toy-toy-ctest-gemmini_os_risc_abtranspose_test-linux", + "toy-toy-ctest-gemmini_os_risc_shift_test-linux", + "toy-toy-ctest-gemmini_ws_cisc_conv_test-linux", + "toy-toy-ctest-gemmini_ws_risc_basic_test-linux", + "toy-toy-ctest-mmio_minimal_test-linux", + "toy-toy-ctest-mxfp2int_test-linux", ] diff --git a/examples/chips/toy/workloads/CMakeLists.txt b/examples/chips/toy/workloads/CMakeLists.txt index adb64a481..8261d4d40 100644 --- a/examples/chips/toy/workloads/CMakeLists.txt +++ b/examples/chips/toy/workloads/CMakeLists.txt @@ -1,22 +1,19 @@ get_filename_component(BUCKYBALL_REPO_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../../../.. ABSOLUTE) -set(BUCKYBALL_TOY_COMMON_DIR ${CMAKE_CURRENT_SOURCE_DIR}/common) -include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/BuckyballWorkloads.cmake) -include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/MlirRushB.cmake) - add_custom_target(toy-ctests-build) set(BUCKYBALL_CTEST_GROUP_TARGET toy-ctests-build) +set(BUCKYBALL_CTEST_TARGET toy) add_subdirectory(ctests) add_subdirectory(mlir_tests) +buckyball_add_ball_workload_subdirs(ctests) -file(GLOB BUCKYBALL_BALL_CTEST_DIRS CONFIGURE_DEPENDS - "${BUCKYBALL_REPO_ROOT}/examples/balls/*/workloads/ctests") -buckyball_add_ball_ctest_subdirs() +add_custom_target(chip-ctests-build DEPENDS toy-ctests-build) +add_custom_target(chip-mlirtests-build DEPENDS toy-mlir-tests-build) add_custom_target(chip-workloads-build ALL DEPENDS - toy-ctests-build - toy-mlir-tests-build + chip-ctests-build + chip-mlirtests-build COMMENT "Building toy workloads" VERBATIM) diff --git a/examples/chips/toy/workloads/cmake/BuckyballWorkloads.cmake b/examples/chips/toy/workloads/cmake/BuckyballWorkloads.cmake deleted file mode 100644 index 9e0466d9c..000000000 --- a/examples/chips/toy/workloads/cmake/BuckyballWorkloads.cmake +++ /dev/null @@ -1,314 +0,0 @@ -if(NOT DEFINED ELF_CC) - set(ELF_CC "riscv64-unknown-elf-gcc") -endif() - -if(NOT DEFINED LINUX_CC) - set(LINUX_CC "riscv64-unknown-linux-gnu-gcc") -endif() - -if(NOT DEFINED BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS) - set(BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS ${BUCKYBALL_TOY_COMMON_DIR}) -endif() - -set(BBSIM_LD ${BBSW_BAREMETAL_DIR}/bbsim.ld) -set(BUDDY_CYCLE_TRACE_RUNTIME_SRC ${BBSW_BAREMETAL_DIR}/buddy_cycle_trace.c) -set(BUCKYBALL_CTEST_C_FLAGS - -g -fno-common -O2 -static -march=rv64gc -mcmodel=medany - -fno-builtin-printf -specs=nano.specs -specs=nosys.specs -nostartfiles - -Wl,-T,${BBSIM_LD} - -I${BUCKYBALL_TOY_COMMON_DIR} -) - -set(CMAKE_C_COMPILER ${LINUX_CC}) -set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -march=rv64gc") -set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static -Wl,--no-dynamic-linker") - -function(add_buckyball_rushb_native TARGET_NAME) - cmake_parse_arguments(ARG "CXX" "OUTPUT_SUBDIR" "SOURCES;INCLUDE_DIRS;DEPENDS" ${ARGN}) - - if(NOT ARG_SOURCES) - message(FATAL_ERROR "${TARGET_NAME}: rushB runner has no sources") - endif() - if(NOT ARG_OUTPUT_SUBDIR) - message(FATAL_ERROR "${TARGET_NAME}: rushB runner has no output directory") - endif() - - if(NOT DEFINED BUILD_BIN_DIR OR NOT DEFINED OUTPUT_BIN_DIR) - message(FATAL_ERROR - "add_buckyball_rushb_native requires BUILD_BIN_DIR and OUTPUT_BIN_DIR") - endif() - - # Install under both trees: batch discovery reads OUTPUT, while sync-bin - # rsync --delete mirrors BUILD→OUTPUT and would otherwise wipe rushB-only - # artifacts that never appear under BUILD. - set(RUSHB_OUTPUT_ROOT ${OUTPUT_BIN_DIR}/${ARG_OUTPUT_SUBDIR}) - set(RUSHB_SYNC_ROOT ${BUILD_BIN_DIR}/${ARG_OUTPUT_SUBDIR}) - string(MAKE_C_IDENTIFIER "${ARG_OUTPUT_SUBDIR}" RUSHB_OUTPUT_ROOT_ID) - set(RUSHB_OUTPUT_ROOT_TARGET rushB-output-${RUSHB_OUTPUT_ROOT_ID}) - if(NOT TARGET ${RUSHB_OUTPUT_ROOT_TARGET}) - add_custom_target(${RUSHB_OUTPUT_ROOT_TARGET} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_OUTPUT_ROOT} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_SYNC_ROOT} - COMMENT "Creating rushB output directory: ${ARG_OUTPUT_SUBDIR}" - VERBATIM) - endif() - - if(ARG_CXX) - set(RUSHB_COMPILER c++) - set(RUSHB_STANDARD -std=c++17) - else() - set(RUSHB_COMPILER cc) - set(RUSHB_STANDARD -std=gnu17) - endif() - - foreach(RUSHB_BACKEND bemu verilator) - set(RUSHB_TARGET ${TARGET_NAME}-rushB-${RUSHB_BACKEND}-run) - set(RUSHB_BUILD_DIR ${CMAKE_CURRENT_BINARY_DIR}/rushB/${RUSHB_TARGET}) - set(RUSHB_BINARY ${RUSHB_BUILD_DIR}/${RUSHB_TARGET}.bin) - set(RUSHB_OUTPUT_DIR ${RUSHB_OUTPUT_ROOT}/${RUSHB_TARGET}) - set(RUSHB_SYNC_DIR ${RUSHB_SYNC_ROOT}/${RUSHB_TARGET}) - if(RUSHB_BACKEND STREQUAL "bemu") - set(RUSHB_LIBRARY ${BUCKYBALL_RUSHB_BEMU_LIBRARY}) - set(RUSHB_LIBRARY_NAME bebop_bemu) - set(RUSHB_BUILD_RUNTIME - COMMAND cargo build --release --manifest-path ${BUCKYBALL_RUSHB_BEMU_MANIFEST} --lib) - set(RUSHB_RUNTIME_DEPENDENCY ${BUCKYBALL_RUSHB_BEMU_MANIFEST}) - else() - set(RUSHB_LIBRARY ${BUCKYBALL_RUSHB_VERILATOR_LIBRARY}) - set(RUSHB_LIBRARY_NAME bebop_verilator) - set(RUSHB_BUILD_RUNTIME) - set(RUSHB_RUNTIME_DEPENDENCY ${BUCKYBALL_RUSHB_VERILATOR_LIBRARY}) - endif() - if(NOT RUSHB_LIBRARY) - message(FATAL_ERROR - "${TARGET_NAME}: BUCKYBALL_RUSHB_${RUSHB_BACKEND} library is unset. " - "Include examples/chips//workloads/cmake/RushB.cmake first.") - endif() - - set(RUSHB_RUNTIME_OUTPUT - ${RUSHB_OUTPUT_ROOT}/lib${RUSHB_LIBRARY_NAME}.so) - set(RUSHB_RUNTIME_SYNC - ${RUSHB_SYNC_ROOT}/lib${RUSHB_LIBRARY_NAME}.so) - set(RUSHB_RUNTIME_TARGET - rushB-runtime-${RUSHB_OUTPUT_ROOT_ID}-${RUSHB_BACKEND}) - if(NOT TARGET ${RUSHB_RUNTIME_TARGET}) - add_custom_command( - OUTPUT ${RUSHB_RUNTIME_OUTPUT} ${RUSHB_RUNTIME_SYNC} - ${RUSHB_BUILD_RUNTIME} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_OUTPUT_ROOT} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_SYNC_ROOT} - COMMAND ${CMAKE_COMMAND} -E copy_if_different - ${RUSHB_LIBRARY} ${RUSHB_RUNTIME_OUTPUT} - COMMAND ${CMAKE_COMMAND} -E copy_if_different - ${RUSHB_LIBRARY} ${RUSHB_RUNTIME_SYNC} - DEPENDS ${RUSHB_RUNTIME_DEPENDENCY} - COMMENT "Installing rushB ${RUSHB_BACKEND} runtime: ${ARG_OUTPUT_SUBDIR}" - VERBATIM) - add_custom_target(${RUSHB_RUNTIME_TARGET} - DEPENDS ${RUSHB_RUNTIME_OUTPUT} ${RUSHB_RUNTIME_SYNC}) - add_dependencies(${RUSHB_RUNTIME_TARGET} ${RUSHB_OUTPUT_ROOT_TARGET}) - endif() - - get_filename_component(RUSHB_LIBRARY_DIR ${RUSHB_LIBRARY} DIRECTORY) - set(RUSHB_INCLUDE_ARGS -I${BUCKYBALL_REPO_ROOT}/compiler/include) - foreach(RUSHB_INCLUDE_DIR ${ARG_INCLUDE_DIRS}) - list(APPEND RUSHB_INCLUDE_ARGS -I${RUSHB_INCLUDE_DIR}) - endforeach() - set(RUSHB_INSTALLED ${RUSHB_OUTPUT_DIR}/${RUSHB_TARGET}) - set(RUSHB_SYNCED ${RUSHB_SYNC_DIR}/${RUSHB_TARGET}) - add_custom_command( - OUTPUT ${RUSHB_BINARY} ${RUSHB_INSTALLED} ${RUSHB_SYNCED} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_BUILD_DIR} - COMMAND ${RUSHB_COMPILER} -no-pie ${RUSHB_STANDARD} -O2 -DBUCKYBALL_RUSHB - ${RUSHB_INCLUDE_ARGS} - ${ARG_SOURCES} ${BUCKYBALL_REPO_ROOT}/compiler/lib/RushBRuntime.c - -L${RUSHB_LIBRARY_DIR} -l${RUSHB_LIBRARY_NAME} - -Wl,-rpath,${RUSHB_OUTPUT_ROOT} -o ${RUSHB_BINARY} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_OUTPUT_DIR} - COMMAND ${CMAKE_COMMAND} -E make_directory ${RUSHB_SYNC_DIR} - COMMAND ${CMAKE_COMMAND} -E copy_if_different ${RUSHB_BINARY} - ${RUSHB_INSTALLED} - COMMAND ${CMAKE_COMMAND} -E copy_if_different ${RUSHB_BINARY} - ${RUSHB_SYNCED} - DEPENDS ${ARG_SOURCES} ${ARG_DEPENDS} - ${BUCKYBALL_REPO_ROOT}/compiler/include/buckyball/rushb.h - ${BUCKYBALL_REPO_ROOT}/compiler/lib/RushBRuntime.c - COMMENT "Building rushB ${RUSHB_BACKEND}: ${TARGET_NAME}" - VERBATIM) - add_custom_target(${RUSHB_TARGET} - DEPENDS ${RUSHB_BINARY} ${RUSHB_INSTALLED} ${RUSHB_SYNCED}) - add_dependencies(${RUSHB_TARGET} ${RUSHB_OUTPUT_ROOT_TARGET}) - add_dependencies(${RUSHB_TARGET} ${RUSHB_RUNTIME_TARGET}) - add_dependencies(rushB-${RUSHB_BACKEND}-workloads-build ${RUSHB_TARGET}) - endforeach() -endfunction() -set(BUCKYBALL_BBHW_MEM_C ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.c) - -function(buckyball_ctest_deps OUT_DEPS SOURCE_DIR SOURCE_FILE) - file(GLOB BUCKYBALL_ISA_DEPS CONFIGURE_DEPENDS - ${WORKLOAD_LIB_DIR}/bbhw/isa/*) - set(DEPS - ${SOURCE_DIR}/${SOURCE_FILE} - ${BUCKYBALL_TOY_COMMON_DIR}/buckyball.c - ${BUCKYBALL_TOY_COMMON_DIR}/buckyball.h - ${BUCKYBALL_BBHW_MEM_C} - ${WORKLOAD_LIB_DIR}/bbhw/mem/mem.h - ${BUCKYBALL_CHIP_COMMON_SOURCES} - ${BUCKYBALL_CHIP_COMMON_HEADERS} - ${BUCKYBALL_ISA_DEPS}) - - if(SOURCE_FILE MATCHES "^tlb_.*\\.c$") - list(APPEND DEPS ${BUCKYBALL_TOY_COMMON_DIR}/tlb_common.h) - endif() - - set(${OUT_DEPS} ${DEPS} PARENT_SCOPE) -endfunction() - -function(add_buckyball_linux_ctest TEST_NAME SOURCE_DIR SOURCE_FILE) - set(EXECUTABLE "${TEST_NAME}-linux") - - add_executable(${EXECUTABLE} - ${SOURCE_DIR}/${SOURCE_FILE} - ${BUCKYBALL_TOY_COMMON_DIR}/buckyball.c - ${BUCKYBALL_BBHW_MEM_C} - ${BUCKYBALL_CHIP_COMMON_SOURCES}) - target_include_directories(${EXECUTABLE} PRIVATE - ${WORKLOAD_LIB_DIR} - ${BUCKYBALL_TOY_COMMON_DIR} - ${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS} - ${SOURCE_DIR}) - set_target_properties(${EXECUTABLE} PROPERTIES LINKER_LANGUAGE C) - - add_custom_target(${TEST_NAME}-linux-build - DEPENDS ${EXECUTABLE}) -endfunction() - -function(add_buckyball_multicore_ctest TEST_NAME SOURCE_DIR SOURCE_FILE) - set(EXECUTABLE "${TEST_NAME}-multicore-baremetal") - buckyball_ctest_deps(TEST_DEPS ${SOURCE_DIR} ${SOURCE_FILE}) - - add_custom_command( - OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} - COMMAND ${ELF_CC} ${BUCKYBALL_CTEST_C_FLAGS} - -o ${EXECUTABLE} - ${BBSW_BAREMETAL_DIR}/start.S - -DMULTICORE=3 - ${BUCKYBALL_TOY_COMMON_DIR}/buckyball.c - ${BUCKYBALL_BBHW_MEM_C} - ${BUCKYBALL_CHIP_COMMON_SOURCES} - ${SOURCE_DIR}/${SOURCE_FILE} - -I${WORKLOAD_LIB_DIR} - -I${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS} - -I${SOURCE_DIR} - DEPENDS - ${TEST_DEPS} - ${BBSW_BAREMETAL_DIR}/start.S - COMMENT "Building multicore baremetal executable: ${EXECUTABLE}" - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} - ) - - add_custom_target(${TEST_NAME}-multicore-baremetal-build - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) -endfunction() - -function(add_buckyball_singlecore_ctest TEST_NAME SOURCE_DIR SOURCE_FILE) - set(EXECUTABLE "${TEST_NAME}-singlecore-baremetal") - buckyball_ctest_deps(TEST_DEPS ${SOURCE_DIR} ${SOURCE_FILE}) - - add_custom_command( - OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE} - COMMAND ${ELF_CC} ${BUCKYBALL_CTEST_C_FLAGS} - -o ${EXECUTABLE} - ${BBSW_BAREMETAL_DIR}/crt0.S - ${BUCKYBALL_TOY_COMMON_DIR}/buckyball.c - ${BUCKYBALL_BBHW_MEM_C} - ${BUCKYBALL_CHIP_COMMON_SOURCES} - ${SOURCE_DIR}/${SOURCE_FILE} - -I${WORKLOAD_LIB_DIR} - -I${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS} - -I${SOURCE_DIR} - DEPENDS - ${TEST_DEPS} - ${BBSW_BAREMETAL_DIR}/crt0.S - COMMENT "Building singlecore baremetal executable: ${EXECUTABLE}" - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} - ) - - add_custom_target(${TEST_NAME}-singlecore-baremetal-build - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${EXECUTABLE}) -endfunction() - -get_filename_component(_BUCKYBALL_CHIPS_CMAKE_DIR - "${CMAKE_CURRENT_LIST_DIR}/../../../cmake" ABSOLUTE) -include(${_BUCKYBALL_CHIPS_CMAKE_DIR}/BuckyballCtestLimits.cmake) - -function(add_buckyball_ctest SOURCE_FILE) - if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") - message(FATAL_ERROR - "BUCKYBALL_WORKLOAD_CHIP must be set before add_buckyball_ctest()") - endif() - get_filename_component(_stem ${SOURCE_FILE} NAME_WE) - set(TEST_NAME "${BUCKYBALL_WORKLOAD_CHIP}_${_stem}") - set(SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}) - - buckyball_enforce_ctest_line_limit(${SOURCE_FILE}) - buckyball_ctest_deps(TEST_DEPS ${SOURCE_DIR} ${SOURCE_FILE}) - - add_buckyball_linux_ctest(${TEST_NAME} ${SOURCE_DIR} ${SOURCE_FILE}) - add_buckyball_multicore_ctest(${TEST_NAME} ${SOURCE_DIR} ${SOURCE_FILE}) - add_buckyball_singlecore_ctest(${TEST_NAME} ${SOURCE_DIR} ${SOURCE_FILE}) - set(_rushb_ok TRUE) - if(_stem MATCHES "^tlb_" OR _stem STREQUAL "test_vsetvli") - set(_rushb_ok FALSE) - endif() - if(_rushb_ok AND - DEFINED BUCKYBALL_RUSHB_BEMU_MANIFEST AND - DEFINED BUCKYBALL_RUSHB_VERILATOR_LIBRARY) - add_buckyball_rushb_native(${TEST_NAME} - OUTPUT_SUBDIR src/CTest/rushB - SOURCES - ${SOURCE_DIR}/${SOURCE_FILE} - ${BUCKYBALL_TOY_COMMON_DIR}/buckyball.c - ${BUCKYBALL_BBHW_MEM_C} - ${BUCKYBALL_CHIP_COMMON_SOURCES} - INCLUDE_DIRS - ${WORKLOAD_LIB_DIR} - ${BUCKYBALL_TOY_COMMON_DIR} - ${BUCKYBALL_CHIP_COMMON_INCLUDE_DIRS} - ${SOURCE_DIR} - DEPENDS ${TEST_DEPS}) - endif() - - add_custom_target(${TEST_NAME}-ctest-build - DEPENDS - ${TEST_NAME}-linux-build - ${TEST_NAME}-multicore-baremetal-build - ${TEST_NAME}-singlecore-baremetal-build - COMMENT "Building CTest workload ${TEST_NAME}" - ) - - if(DEFINED BUCKYBALL_CTEST_GROUP_TARGET) - add_dependencies(${BUCKYBALL_CTEST_GROUP_TARGET} ${TEST_NAME}-ctest-build) - endif() -endfunction() - -function(add_buckyball_ctests) - foreach(SOURCE_FILE ${ARGV}) - add_buckyball_ctest(${SOURCE_FILE}) - endforeach() -endfunction() - -function(buckyball_add_ball_ctest_subdirs) - if(NOT DEFINED BUCKYBALL_BALL_CTEST_DIRS) - message(FATAL_ERROR "BUCKYBALL_BALL_CTEST_DIRS must be set before buckyball_add_ball_ctest_subdirs()") - endif() - - foreach(BUCKYBALL_BALL_CTEST_DIR ${BUCKYBALL_BALL_CTEST_DIRS}) - get_filename_component(BUCKYBALL_BALL_WORKLOADS_DIR ${BUCKYBALL_BALL_CTEST_DIR} DIRECTORY) - get_filename_component(BUCKYBALL_BALL_DIR ${BUCKYBALL_BALL_WORKLOADS_DIR} DIRECTORY) - get_filename_component(BUCKYBALL_BALL_NAME ${BUCKYBALL_BALL_DIR} NAME) - add_subdirectory( - ${BUCKYBALL_BALL_CTEST_DIR} - ${CMAKE_CURRENT_BINARY_DIR}/balls/${BUCKYBALL_BALL_NAME}/ctests) - endforeach() -endfunction() diff --git a/examples/chips/toy/workloads/cmake/MlirRushB.cmake b/examples/chips/toy/workloads/cmake/MlirRushB.cmake deleted file mode 100644 index 35185bff2..000000000 --- a/examples/chips/toy/workloads/cmake/MlirRushB.cmake +++ /dev/null @@ -1,62 +0,0 @@ -# Shared rushB object + host runner wiring for toy chip MLIR optests. -# Requires BuckyballWorkloads.cmake (add_buckyball_rushb_native) and -# BUCKYBALL_LOWER_*_RUSHB from bb-tests/workloads/CMakeLists.txt. - -function(add_toy_mlir_rushb) - cmake_parse_arguments(ARG "CXX" "NAME;TARGET_STEM" - "FRONT_PASSES;SOURCES;INCLUDE_DIRS;DEPENDS" ${ARGN}) - - if(NOT ARG_NAME OR NOT ARG_TARGET_STEM) - message(FATAL_ERROR "add_toy_mlir_rushb requires NAME and TARGET_STEM") - endif() - if(NOT DEFINED BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS OR - NOT DEFINED BUCKYBALL_LOWER_BUCKYBALL_RUSHB) - message(FATAL_ERROR - "add_toy_mlir_rushb requires BUCKYBALL_LOWER_*_RUSHB " - "(define them in bb-tests/workloads/CMakeLists.txt)") - endif() - if(NOT DEFINED BUCKYBALL_RUSHB_BEMU_MANIFEST OR - NOT DEFINED BUCKYBALL_RUSHB_VERILATOR_LIBRARY) - return() - endif() - - set(MLIR_SRC ${CMAKE_CURRENT_SOURCE_DIR}/${ARG_NAME}.mlir) - set(RUSHB_OBJ ${CMAKE_CURRENT_BINARY_DIR}/${ARG_NAME}-rushB.o) - - add_custom_command( - OUTPUT ${RUSHB_OBJ} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} - ${ARG_FRONT_PASSES} - "--assign-physical-banks=bank_num=${BUCKYBALL_MLIR_BANK_NUM}" - ${BUCKYBALL_LOWER_BANK_SSA_TO_RUSHB_INTRINSICS} - -convert-linalg-to-loops - -expand-strided-metadata - -lower-affine - -convert-scf-to-cf - -convert-cf-to-llvm - ${BUCKYBALL_LOWER_BUCKYBALL_RUSHB} - -convert-arith-to-llvm - -convert-math-to-llvm - -finalize-memref-to-llvm - -convert-func-to-llvm - -lower-buckyball-intrinsics-to-rushb - -reconcile-unrealized-casts | - ${BUDDY_TRANSLATE} --buddy-to-llvmir | - ${BUDDY_LLC} -filetype=obj -mtriple=x86_64 -O2 -o ${RUSHB_OBJ} - DEPENDS ${MLIR_SRC} ${BUDDY_OPT} ${BUDDY_TRANSLATE} ${BUDDY_LLC} - ${ARG_DEPENDS} - COMMENT "Building rushB ${ARG_NAME}.o from Toy MLIR" - VERBATIM) - - set(RUSHB_CXX_FLAG) - if(ARG_CXX) - set(RUSHB_CXX_FLAG CXX) - endif() - - add_buckyball_rushb_native(${ARG_TARGET_STEM} - ${RUSHB_CXX_FLAG} - OUTPUT_SUBDIR src/CTest/rushB - SOURCES ${ARG_SOURCES} ${RUSHB_OBJ} - INCLUDE_DIRS ${ARG_INCLUDE_DIRS} - DEPENDS ${RUSHB_OBJ} ${ARG_DEPENDS}) -endfunction() diff --git a/examples/chips/toy/workloads/cmake/RushB.cmake b/examples/chips/toy/workloads/cmake/RushB.cmake deleted file mode 100644 index 8848c7b09..000000000 --- a/examples/chips/toy/workloads/cmake/RushB.cmake +++ /dev/null @@ -1,8 +0,0 @@ -# Toy's host-native rushB backends. Model CMake files consume these names -# through the chip-neutral BUCKYBALL_RUSHB_* interface. -set(BUCKYBALL_RUSHB_BEMU_MANIFEST - ${BUCKYBALL_REPO_ROOT}/examples/cores/toy/emu/Cargo.toml) -set(BUCKYBALL_RUSHB_BEMU_LIBRARY - ${BUCKYBALL_REPO_ROOT}/examples/cores/toy/emu/target/release/libbebop_bemu.so) -set(BUCKYBALL_RUSHB_VERILATOR_LIBRARY - ${BUCKYBALL_REPO_ROOT}/bebop/target/release/deps/libbebop_verilator.so) diff --git a/examples/chips/toy/workloads/ctests/CMakeLists.txt b/examples/chips/toy/workloads/ctests/CMakeLists.txt index faad1d7fd..214e42267 100644 --- a/examples/chips/toy/workloads/ctests/CMakeLists.txt +++ b/examples/chips/toy/workloads/ctests/CMakeLists.txt @@ -1,27 +1,5 @@ add_buckyball_ctests( - aligned_matmul.c mmio_minimal_test.c scu_uart_rx_echo_test.c test_vsetvli.c - tlb_bare_mode_test.c - tlb_repeated_sfence_test.c - tlb_sfence_flush_test.c - tlb_sfence_remap_test.c - tlb_sv39_4k_alias_test.c - tlb_sv39_identity_test.c - tlb_sv39_nonidentity_test.c - transpose_matmul.c - vecunit_matmul_16xn_ones.c - vecunit_matmul_16xn_random1.c - vecunit_matmul_16xn_random2.c - vecunit_matmul_16xn_random3.c - vecunit_matmul_16xn_zero_random.c - vecunit_matmul_col_row_vector.c - vecunit_matmul_identity_random.c - vecunit_matmul_ones.c - vecunit_matmul_random1.c - vecunit_matmul_random2.c - vecunit_matmul_row_col_vector.c - vecunit_matmul_zero_random.c - vecunit_tiled_matmul.c ) diff --git a/examples/chips/toy/workloads/ctests/aligned_matmul.c b/examples/chips/toy/workloads/ctests/aligned_matmul.c index 64a60b145..575dd3902 100644 --- a/examples/chips/toy/workloads/ctests/aligned_matmul.c +++ b/examples/chips/toy/workloads/ctests/aligned_matmul.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -33,7 +35,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, size, 1); bb_mvin((uintptr_t)c, op2_bank_id, DIM, 1); bb_transpose(op2_bank_id, op1_bank_id, size, 8); - bb_mul_warp16(op1_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(op1_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, op2_bank_id, DIM, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/mmio_minimal_test.c b/examples/chips/toy/workloads/ctests/mmio_minimal_test.c index fc0705ad5..2d3706278 100644 --- a/examples/chips/toy/workloads/ctests/mmio_minimal_test.c +++ b/examples/chips/toy/workloads/ctests/mmio_minimal_test.c @@ -1,4 +1,4 @@ -// Minimal MMIO test - just test mmio_set and mvin_mmio +// Minimal MMIO test - mvin_mmio into MMIO SRAM #include "buckyball.h" #include @@ -37,23 +37,14 @@ int main(void) { bb_mem_alloc(bank_id, 1, 1); printf("Allocated bank %d\n", bank_id); - // 3. Bind bank to MMIO - printf("Calling bb_mmio_set...\n"); - bb_mmio_set(bank_id, mmio_addr, 1); - printf("bb_mmio_set done\n"); - - // 4. Load data to MMIO printf("Calling bb_mvin_mmio...\n"); bb_mvin_mmio((uintptr_t)test_data, mmio_addr, 1, 16); printf("bb_mvin_mmio done\n"); - // 5. Fence printf("Calling bb_fence...\n"); bb_fence(); printf("bb_fence done\n"); - // 6. Cleanup - bb_mmio_set(bank_id, 0, 0); bb_mem_release(bank_id); mmio_allocator_free(&mmio_alloc, mmio_addr, 1); diff --git a/examples/chips/toy/workloads/ctests/transpose_matmul.c b/examples/chips/toy/workloads/ctests/transpose_matmul.c index 59d6f2587..766ebfd72 100644 --- a/examples/chips/toy/workloads/ctests/transpose_matmul.c +++ b/examples/chips/toy/workloads/ctests/transpose_matmul.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include #include @@ -41,7 +43,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(b + k0 * DIM), op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); } bb_mvout((uintptr_t)c, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_ones.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_ones.c index 683aa5c8c..85862f144 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_ones.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_ones.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include #include @@ -39,7 +41,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(b + k0 * DIM), op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); } bb_mvout((uintptr_t)c, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random1.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random1.c index a49efbfe7..536686930 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random1.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random1.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include #include @@ -39,7 +41,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(b + k0 * DIM), op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); } bb_mvout((uintptr_t)c, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random2.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random2.c index 02206b511..2d99cb737 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random2.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random2.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include #include @@ -39,7 +41,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(b + k0 * DIM), op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); } bb_mvout((uintptr_t)c, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random3.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random3.c index d51885af2..461703fac 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random3.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_random3.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include #include @@ -39,7 +41,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(b + k0 * DIM), op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); } bb_mvout((uintptr_t)c, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_zero_random.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_zero_random.c index cfe3bdff2..2351caac0 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_zero_random.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_16xn_zero_random.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include #include @@ -39,7 +41,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(b + k0 * DIM), op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); } bb_mvout((uintptr_t)c, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_col_row_vector.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_col_row_vector.c index 7b221b029..4f414a4b2 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_col_row_vector.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_col_row_vector.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -32,7 +34,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, size, 1); bb_transpose(op1_bank_id, a_transposed_bank_id, size, 8); - bb_mul_warp16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_identity_random.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_identity_random.c index cba2e0dde..4ee482c8b 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_identity_random.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_identity_random.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -31,7 +33,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, size, 1); bb_transpose(op1_bank_id, a_transposed_bank_id, size, 8); - bb_mul_warp16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_ones.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_ones.c index 834fe3fd9..e566ec7cf 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_ones.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_ones.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -31,7 +33,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, DIM, 1); bb_transpose(op1_bank_id, a_transposed_bank_id, size, 8); - bb_mul_warp16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_random1.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_random1.c index 8b3723d7e..a2ab655dd 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_random1.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_random1.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #define DIM 16 @@ -34,7 +36,7 @@ int main(void) { bb_mvin((uintptr_t)input_a, op1, DIM, 1); bb_mvin((uintptr_t)input_b, op2, DIM, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, DIM, 0); + bb_vecmat16(a_t, op2, acc, DIM, 0); bb_mvout((uintptr_t)output, acc, DIM, 1); bb_fence(); diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_random2.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_random2.c index c67cc5864..09d4dba8b 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_random2.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_random2.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -31,7 +33,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, DIM, 1); bb_transpose(op1_bank_id, a_transposed_bank_id, size, 8); - bb_mul_warp16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_row_col_vector.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_row_col_vector.c index dedd8a153..06d0a5e86 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_row_col_vector.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_row_col_vector.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -31,7 +33,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, size, 1); bb_transpose(op1_bank_id, a_transposed_bank_id, size, 8); - bb_mul_warp16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/vecunit_matmul_zero_random.c b/examples/chips/toy/workloads/ctests/vecunit_matmul_zero_random.c index 98931eec7..4d301e295 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_matmul_zero_random.c +++ b/examples/chips/toy/workloads/ctests/vecunit_matmul_zero_random.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -31,7 +33,7 @@ void hw_matmul(const char *test_name, elem_t *a, elem_t *b, result_t *c, bb_mvin((uintptr_t)b, op2_bank_id, size, 1); bb_transpose(op1_bank_id, a_transposed_bank_id, size, 8); - bb_mul_warp16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); + bb_vecmat16(a_transposed_bank_id, op2_bank_id, acc_bank_id, size, 0); bb_mvout((uintptr_t)c, acc_bank_id, size, 1); bb_fence(); } diff --git a/examples/chips/toy/workloads/ctests/vecunit_tiled_matmul.c b/examples/chips/toy/workloads/ctests/vecunit_tiled_matmul.c index 2d0a758f0..7d8c9561c 100644 --- a/examples/chips/toy/workloads/ctests/vecunit_tiled_matmul.c +++ b/examples/chips/toy/workloads/ctests/vecunit_tiled_matmul.c @@ -1,6 +1,8 @@ #include "buckyball.h" #include #include +#include +#include #include #include @@ -51,7 +53,7 @@ int main(void) { bb_mvin((uintptr_t)tile_a, op1, DIM, 1); bb_mvin((uintptr_t)(input_b + k0 * DIM), op2, KTILE, 1); bb_transpose(op1, a_t, DIM, 8); - bb_mul_warp16(a_t, op2, acc, KTILE, 0); + bb_vecmat16(a_t, op2, acc, KTILE, 0); } bb_mvout((uintptr_t)output, acc, DIM, 1); diff --git a/examples/chips/toy/workloads/mlir_tests/buckyball/CMakeLists.txt b/examples/chips/toy/workloads/mlir_tests/buckyball/CMakeLists.txt index a0e81f12b..390704077 100644 --- a/examples/chips/toy/workloads/mlir_tests/buckyball/CMakeLists.txt +++ b/examples/chips/toy/workloads/mlir_tests/buckyball/CMakeLists.txt @@ -17,29 +17,29 @@ function(add_toy_mlir_matmul_test name) if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP must be set before mlir ctests") endif() - set(BAREMETAL_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_buckyball_${name}_singlecore-baremetal) - set(BAREMETAL_TARGET optest_buckyball_${name}_singlecore_baremetal) - set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_buckyball_${name}-linux) - set(LINUX_TARGET optest_buckyball_${name}_linux) + set(BAREMETAL_BIN ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_buckyball_${name}-baremetal) + set(BAREMETAL_TARGET mlirtest_buckyball_${name}_baremetal) + set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_buckyball_${name}-linux) + set(LINUX_TARGET mlirtest_buckyball_${name}_linux) add_custom_command( OUTPUT ${OBJ} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -lower-buckyball-to-bank-ssa > ${CMAKE_CURRENT_BINARY_DIR}/${name}_1_bank_ssa.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks > ${CMAKE_CURRENT_BINARY_DIR}/${name}_2_physical_banks.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} ${BUCKYBALL_LOWER_BUCKYBALL} > ${CMAKE_CURRENT_BINARY_DIR}/${name}_3_buckyball_intrinsics.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -53,9 +53,9 @@ function(add_toy_mlir_matmul_test name) -convert-func-to-llvm -reconcile-unrealized-casts > ${CMAKE_CURRENT_BINARY_DIR}/${name}_4_llvm.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -70,9 +70,9 @@ function(add_toy_mlir_matmul_test name) -reconcile-unrealized-casts | ${BUDDY_TRANSLATE} --buddy-to-llvmir > ${CMAKE_CURRENT_BINARY_DIR}/${name}_5.ll - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -141,12 +141,13 @@ function(add_toy_mlir_matmul_test name) DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${BAREMETAL_BIN}) add_custom_target(${LINUX_TARGET} DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${LINUX_BIN}) - add_custom_target(optest_buckyball_${name} + add_custom_target(mlirtest_buckyball_${name} DEPENDS ${BAREMETAL_TARGET} ${LINUX_TARGET}) add_toy_mlir_rushb( NAME ${name} - TARGET_STEM ${BUCKYBALL_WORKLOAD_CHIP}_optest_buckyball_${name} + TARGET_STEM ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_buckyball_${name} + TARGET toy CXX FRONT_PASSES -lower-buckyball-to-bank-ssa SOURCES ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} ${MAIN_SRC} @@ -159,7 +160,7 @@ endfunction() function(add_toy_mlir_matmul_contract_test name) add_toy_mlir_matmul_test(${name}) - add_custom_target(optest_buckyball_${name}_contract + add_custom_target(mlirtest_buckyball_${name}_contract COMMAND ${FILECHECK} ${CMAKE_CURRENT_SOURCE_DIR}/${name}.mlir --input-file=${CMAKE_CURRENT_BINARY_DIR}/${name}_1_bank_ssa.mlir @@ -168,44 +169,6 @@ function(add_toy_mlir_matmul_contract_test name) VERBATIM) endfunction() -add_toy_mlir_matmul_contract_test(matmul_16x64_64x16) -add_toy_mlir_matmul_test(matmul_64x16_16x64) -add_toy_mlir_matmul_test(matmul_1024x16_16x1024) - -set(INT_CONVERT_MLIR ${CMAKE_CURRENT_SOURCE_DIR}/int_convert_ops.mlir) -set(INT_CONVERT_PHYSICAL ${CMAKE_CURRENT_BINARY_DIR}/int_convert_ops_physical.mlir) -set(INT_CONVERT_INTRIN ${CMAKE_CURRENT_BINARY_DIR}/int_convert_ops_intrinsics.mlir) - -add_custom_command( - OUTPUT ${INT_CONVERT_PHYSICAL} ${INT_CONVERT_INTRIN} - COMMAND ${BUDDY_OPT} ${INT_CONVERT_MLIR} - "--assign-physical-banks=bank_num=16" - > ${INT_CONVERT_PHYSICAL} - COMMAND ${BUDDY_OPT} ${INT_CONVERT_MLIR} - "--assign-physical-banks=bank_num=16" - ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} - ${BUCKYBALL_LOWER_BUCKYBALL} - > ${INT_CONVERT_INTRIN} - DEPENDS ${INT_CONVERT_MLIR} ${BUDDY_OPT} - COMMENT "Lowering int_convert_ops.mlir" - VERBATIM -) - -add_custom_target(optest_buckyball_int_convert_ops_contract - COMMAND ${FILECHECK} ${INT_CONVERT_MLIR} - --check-prefix=CHECK-PHYSICAL - --input-file=${INT_CONVERT_PHYSICAL} - COMMAND ${FILECHECK} ${INT_CONVERT_MLIR} - --check-prefix=CHECK-INTRIN - --input-file=${INT_CONVERT_INTRIN} - DEPENDS ${INT_CONVERT_PHYSICAL} ${INT_CONVERT_INTRIN} - COMMENT "Checking Int2Fp convert ISA contract" - VERBATIM) - -add_custom_target(buckyball-mlir-tests-build ALL DEPENDS - optest_buckyball_matmul_16x64_64x16 - optest_buckyball_matmul_64x16_16x64 - optest_buckyball_matmul_1024x16_16x1024 - optest_buckyball_int_convert_ops_contract +add_custom_target(buckyball-mlir-tests-build ALL COMMENT "Building Toy Buckyball MLIR tests" VERBATIM) diff --git a/examples/chips/toy/workloads/mlir_tests/buckyball/int_convert_ops.mlir b/examples/chips/toy/workloads/mlir_tests/buckyball/int_convert_ops.mlir index b36cb24ec..7eb268998 100644 --- a/examples/chips/toy/workloads/mlir_tests/buckyball/int_convert_ops.mlir +++ b/examples/chips/toy/workloads/mlir_tests/buckyball/int_convert_ops.mlir @@ -1,19 +1,14 @@ // Int2FpBall convert op contract: -// - bank_int2fp lowers to int2fp (output_mode=0) -// - bank_int32_to_int8 lowers to int32_to_int8 (output_mode=1) +// - bank_int2fp_tensor lowers to int2fp_tensor // // RUN stages are driven by CMake FileCheck targets. // -// CHECK-PHYSICAL: buckyball.int2fp -// CHECK-PHYSICAL: buckyball.int32_to_int8 +// CHECK-PHYSICAL: buckyball.int2fp_tensor // CHECK-INTRIN: buckyball.intr.custom // CHECK-INTRIN: funct7 = 52 -// CHECK-INTRIN: arith.constant 4294967296 -func.func @int_convert_ops(%in0: i64, %out0: i64, %in1: i64, %out1: i64, %iter: i64, %scale: i64) { - %r0 = buckyball.bank_int2fp %in0 %out0 %iter %scale - : i64 i64 i64 i64 - %r1 = buckyball.bank_int32_to_int8 %in1 %out1 %iter %scale - : i64 i64 i64 i64 +func.func @int_convert_ops(%in0: i64, %out0: i64, %iter: i64, %da: i64, %dw: i64) { + %r0 = buckyball.bank_int2fp_tensor %in0 %out0 %iter %da %dw + : i64 i64 i64 i64 i64 return } diff --git a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1024x16_16x1024.mlir b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1024x16_16x1024.mlir index be2ab9ef5..0438b4cc1 100644 --- a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1024x16_16x1024.mlir +++ b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1024x16_16x1024.mlir @@ -7,15 +7,15 @@ func.func @main() -> i8 { %one_f32 = arith.constant 1.0 : f32 %zero_f32 = arith.constant 0.0 : f32 - %a = memref.alloc() {alignment = 16 : i64} : memref<1024x16xf32> - %b = memref.alloc() {alignment = 16 : i64} : memref<16x1024xf32> - %c = memref.alloc() {alignment = 16 : i64} : memref<1024x1024xf32> + %a = memref.alloc() : memref<1024x16xf32> + %b = memref.alloc() : memref<16x1024xf32> + %c = memref.alloc() : memref<1024x1024xf32> linalg.fill ins(%one_f32 : f32) outs(%a : memref<1024x16xf32>) linalg.fill ins(%one_f32 : f32) outs(%b : memref<16x1024xf32>) linalg.fill ins(%zero_f32 : f32) outs(%c : memref<1024x1024xf32>) - buckyball.matrix_matmul %a %b %c + buckyball.smatmul_matmul %a %b %c : memref<1024x16xf32> memref<16x1024xf32> memref<1024x1024xf32> func.call @check_result(%c) : (memref<1024x1024xf32>) -> () diff --git a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_16x64_64x16.mlir b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_16x64_64x16.mlir index 2a22dd04d..5c46afefe 100644 --- a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_16x64_64x16.mlir +++ b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_16x64_64x16.mlir @@ -1,7 +1,7 @@ // Buckyball Dialect matmul test: 16x64 @ 64x16 // Matrix: 16x64 (fp32) @ 64x16 (fp32) -> 16x16 (fp32) // This locks the matrix ISA contract where fp2int, transpose, and -// mul_warp16 process K rows in one instruction. +// vecmat16 process K rows in one instruction. // // CHECK-DAG: %[[C64:.*]] = arith.constant 64 : i64 // CHECK-DAG: %[[C16:.*]] = arith.constant 16 : i64 @@ -12,8 +12,8 @@ // CHECK: buckyball.bank_fp2int {{.*}} %[[C64]] // CHECK: buckyball.bank_fp2int {{.*}} %[[C64]] // CHECK: buckyball.bank_transpose {{.*}} %[[C64]] %[[C8]] -// CHECK: buckyball.bank_mul_warp16 {{.*}} %[[C64]] -// CHECK: buckyball.bank_int2fp {{.*}} %[[C16]] +// CHECK: buckyball.bank_vecmat16 {{.*}} %[[C64]] +// CHECK: buckyball.bank_int2fp_tensor {{.*}} %[[C16]] // CHECK: buckyball.bank_mvout {{.*}} %[[C16]] %[[C1]] func.func private @check_result(memref<16x16xf32>) -> () @@ -23,15 +23,15 @@ func.func @main() -> i8 { %one_f32 = arith.constant 1.0 : f32 %zero_f32 = arith.constant 0.0 : f32 - %a = memref.alloc() {alignment = 16 : i64} : memref<16x64xf32> - %b = memref.alloc() {alignment = 16 : i64} : memref<64x16xf32> - %c = memref.alloc() {alignment = 16 : i64} : memref<16x16xf32> + %a = memref.alloc() : memref<16x64xf32> + %b = memref.alloc() : memref<64x16xf32> + %c = memref.alloc() : memref<16x16xf32> linalg.fill ins(%one_f32 : f32) outs(%a : memref<16x64xf32>) linalg.fill ins(%one_f32 : f32) outs(%b : memref<64x16xf32>) linalg.fill ins(%zero_f32 : f32) outs(%c : memref<16x16xf32>) - buckyball.matrix_matmul %a %b %c + buckyball.smatmul_matmul %a %b %c : memref<16x64xf32> memref<64x16xf32> memref<16x16xf32> func.call @check_result(%c) : (memref<16x16xf32>) -> () diff --git a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1x128_128x96.mlir b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1x128_128x96.mlir index 18d70d53b..7393527a0 100644 --- a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1x128_128x96.mlir +++ b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_1x128_128x96.mlir @@ -9,15 +9,15 @@ func.func @main() -> i8 { %one_f32 = arith.constant 1.0 : f32 %zero_f32 = arith.constant 0.0 : f32 - %a = memref.alloc() {alignment = 16 : i64} : memref<1x128xf32> - %b = memref.alloc() {alignment = 16 : i64} : memref<128x96xf32> - %c = memref.alloc() {alignment = 16 : i64} : memref<1x96xf32> + %a = memref.alloc() : memref<1x128xf32> + %b = memref.alloc() : memref<128x96xf32> + %c = memref.alloc() : memref<1x96xf32> linalg.fill ins(%one_f32 : f32) outs(%a : memref<1x128xf32>) linalg.fill ins(%one_f32 : f32) outs(%b : memref<128x96xf32>) linalg.fill ins(%zero_f32 : f32) outs(%c : memref<1x96xf32>) - buckyball.matrix_matmul %a %b %c + buckyball.smatmul_matmul %a %b %c : memref<1x128xf32> memref<128x96xf32> memref<1x96xf32> func.call @check_result(%c) : (memref<1x96xf32>) -> () diff --git a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_64x16_16x64.mlir b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_64x16_16x64.mlir index d2196fe4e..89cde7869 100644 --- a/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_64x16_16x64.mlir +++ b/examples/chips/toy/workloads/mlir_tests/buckyball/matmul_64x16_16x64.mlir @@ -1,6 +1,6 @@ // Buckyball Dialect matmul test: 64x16 @ 16x64 // Matrix: 64x16 (fp32) @ 16x64 (fp32) -> 64x64 (fp32) -// Tests buckyball.matrix_matmul lowering: fp32 -> quant -> mul_warp16 -> dequant +// Tests buckyball.smatmul_matmul lowering: fp32 -> quant -> vecmat16 -> dequant // Verification done in C wrapper to avoid fp constant pool (.LCPI) which // triggers R_RISCV_HI20 relocation truncation under newlib. @@ -11,15 +11,15 @@ func.func @main() -> i8 { %one_f32 = arith.constant 1.0 : f32 %zero_f32 = arith.constant 0.0 : f32 - %a = memref.alloc() {alignment = 16 : i64} : memref<64x16xf32> - %b = memref.alloc() {alignment = 16 : i64} : memref<16x64xf32> - %c = memref.alloc() {alignment = 16 : i64} : memref<64x64xf32> + %a = memref.alloc() : memref<64x16xf32> + %b = memref.alloc() : memref<16x64xf32> + %c = memref.alloc() : memref<64x64xf32> linalg.fill ins(%one_f32 : f32) outs(%a : memref<64x16xf32>) linalg.fill ins(%one_f32 : f32) outs(%b : memref<16x64xf32>) linalg.fill ins(%zero_f32 : f32) outs(%c : memref<64x64xf32>) - buckyball.matrix_matmul %a %b %c + buckyball.smatmul_matmul %a %b %c : memref<64x16xf32> memref<16x64xf32> memref<64x64xf32> func.call @check_result(%c) : (memref<64x64xf32>) -> () diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/CMakeLists.txt b/examples/chips/toy/workloads/mlir_tests/linalg/CMakeLists.txt index fcc057386..3384e3c92 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/CMakeLists.txt +++ b/examples/chips/toy/workloads/mlir_tests/linalg/CMakeLists.txt @@ -18,7 +18,7 @@ function(add_linalg_contract_test name) add_custom_command( OUTPUT ${OUT} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-linalg-to-tile > ${OUT} DEPENDS ${MLIR_SRC} ${BUDDY_OPT} @@ -26,7 +26,7 @@ function(add_linalg_contract_test name) VERBATIM ) - add_custom_target(optest_linalg_${name}_contract + add_custom_target(mlirtest_linalg_${name}_contract COMMAND ${FILECHECK} ${MLIR_SRC} --input-file=${OUT} DEPENDS ${OUT} COMMENT "Checking Linalg contract for ${name}" @@ -43,16 +43,16 @@ function(add_linalg_conv2d_test name main_src) if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP must be set before mlir ctests") endif() - set(BAREMETAL_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_linalg_${name}_singlecore-baremetal) - set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_linalg_${name}-linux) + set(BAREMETAL_BIN ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_linalg_${name}-baremetal) + set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_linalg_${name}-linux) add_custom_command( OUTPUT ${OBJ} ${LLVM_MLIR} ${LLVM_IR} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-linalg-to-tile -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -124,21 +124,22 @@ function(add_linalg_conv2d_test name main_src) VERBATIM ) - add_custom_target(optest_linalg_${name}_singlecore_baremetal + add_custom_target(mlirtest_linalg_${name}_baremetal DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${BAREMETAL_BIN}) - add_custom_target(optest_linalg_${name}_linux + add_custom_target(mlirtest_linalg_${name}_linux DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${LINUX_BIN}) - add_custom_target(optest_linalg_${name} - DEPENDS optest_linalg_${name}_linux - optest_linalg_${name}_singlecore_baremetal - COMMENT "Building optest_linalg_${name} for all platforms" + add_custom_target(mlirtest_linalg_${name} + DEPENDS mlirtest_linalg_${name}_linux + mlirtest_linalg_${name}_baremetal + COMMENT "Building mlirtest_linalg_${name} for all platforms" ) add_toy_mlir_rushb( NAME ${name} - TARGET_STEM ${BUCKYBALL_WORKLOAD_CHIP}_optest_linalg_${name} + TARGET_STEM ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_linalg_${name} + TARGET toy CXX FRONT_PASSES -convert-linalg-to-tile @@ -160,70 +161,6 @@ function(add_linalg_conv2d_test name main_src) ${BBHW_MEM_C}) endfunction() -# Compile matmul.mlir -> matmul.o -add_custom_command( - OUTPUT matmul.o - COMMAND ${BUDDY_OPT} ${CMAKE_CURRENT_SOURCE_DIR}/matmul.mlir - -convert-linalg-to-tile - -convert-tile-to-buckyball - -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" - ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} - -convert-linalg-to-loops - -expand-strided-metadata - -lower-affine - -convert-scf-to-cf - -convert-cf-to-llvm - ${BUCKYBALL_LOWER_BUCKYBALL} - -convert-arith-to-llvm - -convert-math-to-llvm - -finalize-memref-to-llvm - -convert-func-to-llvm - -reconcile-unrealized-casts - > ${CMAKE_CURRENT_BINARY_DIR}/matmul_llvm.mlir - COMMAND ${BUDDY_TRANSLATE} --buddy-to-llvmir - ${CMAKE_CURRENT_BINARY_DIR}/matmul_llvm.mlir - > ${CMAKE_CURRENT_BINARY_DIR}/matmul.ll - COMMAND ${BUDDY_LLC} ${CMAKE_CURRENT_BINARY_DIR}/matmul.ll - -filetype=obj -mtriple=riscv64 -O2 -code-model=medium - -mattr=${BUCKYBALL_RISCV_MATTR} -float-abi=hard - -o ${CMAKE_CURRENT_BINARY_DIR}/matmul.o - DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/matmul.mlir - ${BUDDY_OPT} ${BUDDY_TRANSLATE} ${BUDDY_LLC} - COMMENT "Building matmul.o from Linalg MLIR" - VERBATIM -) - -if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") - message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP must be set before mlir ctests") -endif() -set(LINALG_MATMUL_BAREMETAL_BIN - ${BUCKYBALL_WORKLOAD_CHIP}_optest_linalg_matmul_singlecore-baremetal) - -add_custom_command( - OUTPUT ${LINALG_MATMUL_BAREMETAL_BIN} - COMMAND ${ELF_CC} ${C_FLAGS} - -I${WORKLOAD_LIB_DIR} - -o ${CMAKE_CURRENT_BINARY_DIR}/${LINALG_MATMUL_BAREMETAL_BIN} - ${BBSW_BAREMETAL_DIR}/crt0.S - ${BBSW_BAREMETAL_DIR}/syscalls.c - ${BBHW_MEM_C} - ${CMAKE_CURRENT_SOURCE_DIR}/matmul_main.cpp - ${CMAKE_CURRENT_BINARY_DIR}/matmul.o - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/matmul.o - ${BBHW_MEM_C} - ${CMAKE_CURRENT_SOURCE_DIR}/matmul_main.cpp - ${BBSW_BAREMETAL_DIR}/crt0.S - ${BBSW_BAREMETAL_DIR}/syscalls.c - COMMENT "Linking ${LINALG_MATMUL_BAREMETAL_BIN}" - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} - VERBATIM -) - -add_custom_target(optest_linalg_matmul_singlecore_baremetal ALL - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${LINALG_MATMUL_BAREMETAL_BIN} -) - add_linalg_contract_test(conv2d_nhwc_hwcf_contract) add_linalg_contract_test(conv2d_nhwc_fhwc_contract) add_linalg_contract_test(conv2d_nchw_fchw_contract) @@ -233,9 +170,8 @@ add_linalg_conv2d_test(conv2d_nhwc_fhwc conv2d_nhwc_main.cpp) add_linalg_conv2d_test(conv2d_nchw_fchw conv2d_nchw_main.cpp) add_custom_target(linalg-mlir-tests-build ALL DEPENDS - optest_linalg_matmul_singlecore_baremetal - optest_linalg_conv2d_nhwc_hwcf - optest_linalg_conv2d_nhwc_fhwc - optest_linalg_conv2d_nchw_fchw + mlirtest_linalg_conv2d_nhwc_hwcf + mlirtest_linalg_conv2d_nhwc_fhwc + mlirtest_linalg_conv2d_nchw_fchw COMMENT "Building all Linalg MLIR tests" VERBATIM) diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw.mlir b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw.mlir index 72b07d36d..a4d3b200d 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw.mlir +++ b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw.mlir @@ -5,9 +5,9 @@ func.func @main() -> i8 { %zero_f32 = arith.constant 0.0 : f32 %one_f32 = arith.constant 1.0 : f32 - %input = memref.alloc() {alignment = 16 : i64} : memref<1x8x4x1xf32> - %filter = memref.alloc() {alignment = 16 : i64} : memref<6x8x1x1xf32> - %output = memref.alloc() {alignment = 16 : i64} : memref<1x6x4x1xf32> + %input = memref.alloc() : memref<1x8x4x1xf32> + %filter = memref.alloc() : memref<6x8x1x1xf32> + %output = memref.alloc() : memref<1x6x4x1xf32> linalg.fill ins(%one_f32 : f32) outs(%input : memref<1x8x4x1xf32>) linalg.fill ins(%one_f32 : f32) outs(%filter : memref<6x8x1x1xf32>) diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw_contract.mlir b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw_contract.mlir index b62b96417..e1efb9730 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw_contract.mlir +++ b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nchw_fchw_contract.mlir @@ -12,9 +12,9 @@ func.func @main() -> i8 { %zero_f32 = arith.constant 0.0 : f32 %one_f32 = arith.constant 1.0 : f32 - %input = memref.alloc() {alignment = 16 : i64} : memref<1x8x4x1xf32> - %filter = memref.alloc() {alignment = 16 : i64} : memref<6x8x1x1xf32> - %output = memref.alloc() {alignment = 16 : i64} : memref<1x6x4x1xf32> + %input = memref.alloc() : memref<1x8x4x1xf32> + %filter = memref.alloc() : memref<6x8x1x1xf32> + %output = memref.alloc() : memref<1x6x4x1xf32> linalg.fill ins(%one_f32 : f32) outs(%input : memref<1x8x4x1xf32>) linalg.fill ins(%one_f32 : f32) outs(%filter : memref<6x8x1x1xf32>) diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc.mlir b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc.mlir index 7a9fea4e8..1c8c39948 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc.mlir +++ b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc.mlir @@ -5,9 +5,9 @@ func.func @main() -> i8 { %zero_f32 = arith.constant 0.0 : f32 %one_f32 = arith.constant 1.0 : f32 - %input = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x8xf32> - %filter = memref.alloc() {alignment = 16 : i64} : memref<6x1x1x8xf32> - %output = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x6xf32> + %input = memref.alloc() : memref<1x4x1x8xf32> + %filter = memref.alloc() : memref<6x1x1x8xf32> + %output = memref.alloc() : memref<1x4x1x6xf32> linalg.fill ins(%one_f32 : f32) outs(%input : memref<1x4x1x8xf32>) linalg.fill ins(%one_f32 : f32) outs(%filter : memref<6x1x1x8xf32>) diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc_contract.mlir b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc_contract.mlir index aa755d299..807b207e6 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc_contract.mlir +++ b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_fhwc_contract.mlir @@ -11,9 +11,9 @@ func.func @main() -> i8 { %zero_f32 = arith.constant 0.0 : f32 %one_f32 = arith.constant 1.0 : f32 - %input = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x8xf32> - %filter = memref.alloc() {alignment = 16 : i64} : memref<6x1x1x8xf32> - %output = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x6xf32> + %input = memref.alloc() : memref<1x4x1x8xf32> + %filter = memref.alloc() : memref<6x1x1x8xf32> + %output = memref.alloc() : memref<1x4x1x6xf32> linalg.fill ins(%one_f32 : f32) outs(%input : memref<1x4x1x8xf32>) linalg.fill ins(%one_f32 : f32) outs(%filter : memref<6x1x1x8xf32>) diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf.mlir b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf.mlir index 07f888607..cca9db045 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf.mlir +++ b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf.mlir @@ -5,9 +5,9 @@ func.func @main() -> i8 { %zero_f32 = arith.constant 0.0 : f32 %one_f32 = arith.constant 1.0 : f32 - %input = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x8xf32> - %filter = memref.alloc() {alignment = 16 : i64} : memref<1x1x8x6xf32> - %output = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x6xf32> + %input = memref.alloc() : memref<1x4x1x8xf32> + %filter = memref.alloc() : memref<1x1x8x6xf32> + %output = memref.alloc() : memref<1x4x1x6xf32> linalg.fill ins(%one_f32 : f32) outs(%input : memref<1x4x1x8xf32>) linalg.fill ins(%one_f32 : f32) outs(%filter : memref<1x1x8x6xf32>) diff --git a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf_contract.mlir b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf_contract.mlir index bbb593579..76df6dc03 100644 --- a/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf_contract.mlir +++ b/examples/chips/toy/workloads/mlir_tests/linalg/conv2d_nhwc_hwcf_contract.mlir @@ -6,9 +6,9 @@ func.func @main() -> i8 { %zero_f32 = arith.constant 0.0 : f32 %one_f32 = arith.constant 1.0 : f32 - %input = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x8xf32> - %filter = memref.alloc() {alignment = 16 : i64} : memref<1x1x8x6xf32> - %output = memref.alloc() {alignment = 16 : i64} : memref<1x4x1x6xf32> + %input = memref.alloc() : memref<1x4x1x8xf32> + %filter = memref.alloc() : memref<1x1x8x6xf32> + %output = memref.alloc() : memref<1x4x1x6xf32> linalg.fill ins(%one_f32 : f32) outs(%input : memref<1x4x1x8xf32>) linalg.fill ins(%one_f32 : f32) outs(%filter : memref<1x1x8x6xf32>) diff --git a/examples/chips/toy/workloads/mlir_tests/tile/CMakeLists.txt b/examples/chips/toy/workloads/mlir_tests/tile/CMakeLists.txt index 107c31699..624553561 100644 --- a/examples/chips/toy/workloads/mlir_tests/tile/CMakeLists.txt +++ b/examples/chips/toy/workloads/mlir_tests/tile/CMakeLists.txt @@ -17,34 +17,34 @@ function(add_tile_matmul_test name) if(BUCKYBALL_WORKLOAD_CHIP STREQUAL "") message(FATAL_ERROR "BUCKYBALL_WORKLOAD_CHIP must be set before mlir ctests") endif() - set(BAREMETAL_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_tile_${name}_singlecore-baremetal) - set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_optest_tile_${name}-linux) + set(BAREMETAL_BIN ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_tile_${name}-baremetal) + set(LINUX_BIN ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_tile_${name}-linux) add_custom_command( OUTPUT ${OBJ} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball > ${CMAKE_CURRENT_BINARY_DIR}/${name}_1_buckyball.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa > ${CMAKE_CURRENT_BINARY_DIR}/${name}_2_bank_ssa.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks > ${CMAKE_CURRENT_BINARY_DIR}/${name}_3_physical_banks.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} ${BUCKYBALL_LOWER_BUCKYBALL} > ${CMAKE_CURRENT_BINARY_DIR}/${name}_4_buckyball_intrinsics.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -58,10 +58,10 @@ function(add_tile_matmul_test name) -convert-func-to-llvm -reconcile-unrealized-casts > ${CMAKE_CURRENT_BINARY_DIR}/${name}_5_llvm.mlir - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -76,10 +76,10 @@ function(add_tile_matmul_test name) -reconcile-unrealized-casts | ${BUDDY_TRANSLATE} --buddy-to-llvmir > ${CMAKE_CURRENT_BINARY_DIR}/${name}_6.ll - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa - "--assign-physical-banks=bank_num=16" + -assign-physical-banks ${BUCKYBALL_LOWER_BANK_SSA_TO_INTRINSICS} -convert-linalg-to-loops -expand-strided-metadata @@ -144,21 +144,22 @@ function(add_tile_matmul_test name) VERBATIM ) - add_custom_target(optest_tile_${name}_singlecore_baremetal + add_custom_target(mlirtest_tile_${name}_baremetal DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${BAREMETAL_BIN}) - add_custom_target(optest_tile_${name}_linux + add_custom_target(mlirtest_tile_${name}_linux DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${LINUX_BIN}) - add_custom_target(optest_tile_${name} - DEPENDS optest_tile_${name}_linux - optest_tile_${name}_singlecore_baremetal - COMMENT "Building optest_tile_${name} for all platforms" + add_custom_target(mlirtest_tile_${name} + DEPENDS mlirtest_tile_${name}_linux + mlirtest_tile_${name}_baremetal + COMMENT "Building mlirtest_tile_${name} for all platforms" ) add_toy_mlir_rushb( NAME ${name} - TARGET_STEM ${BUCKYBALL_WORKLOAD_CHIP}_optest_tile_${name} + TARGET_STEM ${BUCKYBALL_WORKLOAD_CHIP}_mlirtest_tile_${name} + TARGET toy CXX FRONT_PASSES -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa SOURCES ${CRUNNER_UTILS_SRC} ${BBHW_MEM_C} ${MAIN_SRC} @@ -175,7 +176,7 @@ function(add_tile_matmul_contract_test name) add_custom_command( OUTPUT ${OUT} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy "-convert-tile-to-buckyball=bank_depth=128" > ${OUT} DEPENDS ${MLIR_SRC} ${BUDDY_OPT} @@ -183,7 +184,7 @@ function(add_tile_matmul_contract_test name) VERBATIM ) - add_custom_target(optest_tile_${name}_contract + add_custom_target(mlirtest_tile_${name}_contract COMMAND ${FILECHECK} ${MLIR_SRC} --input-file=${OUT} DEPENDS ${OUT} COMMENT "Checking Tile matmul contract for ${name}" @@ -197,7 +198,7 @@ function(add_tile_contract_test name) add_custom_command( OUTPUT ${OUT} - COMMAND ${BUDDY_OPT} ${MLIR_SRC} + COMMAND ${BUDDY_OPT} ${MLIR_SRC} --target=toy -convert-tile-to-buckyball -lower-buckyball-to-bank-ssa > ${OUT} @@ -206,7 +207,7 @@ function(add_tile_contract_test name) VERBATIM ) - add_custom_target(optest_tile_${name}_contract + add_custom_target(mlirtest_tile_${name}_contract COMMAND ${FILECHECK} ${MLIR_SRC} --input-file=${OUT} DEPENDS ${OUT} COMMENT "Checking Tile contract for ${name}" @@ -214,14 +215,9 @@ function(add_tile_contract_test name) ) endfunction() -add_tile_matmul_test(matmul_127x17_17x127) -add_tile_matmul_test(matmul_1x84_84x10) -add_tile_matmul_contract_test(matmul_16x2048_2048x16_contract) add_tile_contract_test(conv2d_nhwc_hwcf_contract) add_tile_contract_test(conv2d_nhwc_hwcf_pad_contract) -add_custom_target(tile-mlir-tests-build ALL DEPENDS - optest_tile_matmul_127x17_17x127 - optest_tile_matmul_1x84_84x10 +add_custom_target(tile-mlir-tests-build ALL COMMENT "Building all Tile MLIR tests" VERBATIM) diff --git a/examples/chips/toy/workloads/mlir_tests/tile/matmul_127x17_17x127.mlir b/examples/chips/toy/workloads/mlir_tests/tile/matmul_127x17_17x127.mlir index 3ebb929d3..5b039b3a7 100644 --- a/examples/chips/toy/workloads/mlir_tests/tile/matmul_127x17_17x127.mlir +++ b/examples/chips/toy/workloads/mlir_tests/tile/matmul_127x17_17x127.mlir @@ -1,6 +1,6 @@ // Tile Dialect matmul test // Matrix: 127x17 (fp32) @ 17x127 (fp32) -> 127x127 (fp32) -// Tests tile-level padding before lowering to buckyball.matrix_matmul. +// Tests tile-level padding before lowering to buckyball.smatmul_matmul. func.func private @check_result(memref<127x127xf32>) -> () diff --git a/examples/chips/toy/workloads/mlir_tests/tile/matmul_16x2048_2048x16_contract.mlir b/examples/chips/toy/workloads/mlir_tests/tile/matmul_16x2048_2048x16_contract.mlir index c6eccbe57..af69d0155 100644 --- a/examples/chips/toy/workloads/mlir_tests/tile/matmul_16x2048_2048x16_contract.mlir +++ b/examples/chips/toy/workloads/mlir_tests/tile/matmul_16x2048_2048x16_contract.mlir @@ -10,7 +10,7 @@ // CHECK: %[[PARTIAL:.*]] = memref.alloc() : memref<16x16xf32> // CHECK: linalg.fill // CHECK: scf.for {{.*}} = %[[C0]] to %[[C2048]] step %[[C1024]] { -// CHECK: buckyball.matrix_matmul {{.*}} %[[PARTIAL]] +// CHECK: buckyball.smatmul_matmul {{.*}} %[[PARTIAL]] // CHECK: scf.for {{.*}} = %[[C0]] to {{.*}} step {{.*}} { // CHECK: scf.for {{.*}} = %[[C0]] to {{.*}} step {{.*}} { // CHECK: memref.load %[[C_TILE]] diff --git a/examples/cores/README.md b/examples/cores/README.md index 8d441d5fc..79eee9887 100644 --- a/examples/cores/README.md +++ b/examples/cores/README.md @@ -5,7 +5,16 @@ owns its Rocket/frontend parameters, BallDomain, private MemDomain, and the compiler package for its ISA. It does not own a tile topology, device model, kernel, or regression suite. -Every Core has a `manifest.toml` and a `core.toml`. A Buckyball Core also has a -`compiler/` directory. Chip topology files instantiate Cores through relative -`include` paths; Chips own all tile-shared memory, runtime, and multi-core +Every Core has a `configs/` tree and `isa/` (`ballISA.h` from balldomain). +Buckyball Cores also have `compiler/` (buddy-mlir dialect plugin; CMake is the +C++ recipe, not the product entry) and non-empty `ballISA`. Rocket-only +cores keep `configs/default.toml` without a `balldomain` key so tiles stay +Rocket-only; empty `configs/balldomains/` + `isa/` still exist for Bazel / +`compilerCore` consumers. BEMU lives in bebop; chip-level tile runners are +`examples/chips//emu/src/main.rs`. Ball functional models are +`examples/balls//emu/`. + +Chip topology files instantiate a concrete Core TOML through relative +`include` paths; that include is the config entry (not a fixed +`configs/default.toml`). Chips own tile-shared memory, runtime, and multi-core behavior. diff --git a/examples/cores/decode/compiler/CMakeLists.txt b/examples/cores/decode/compiler/CMakeLists.txt index 090379e7c..5657b38eb 100644 --- a/examples/cores/decode/compiler/CMakeLists.txt +++ b/examples/cores/decode/compiler/CMakeLists.txt @@ -4,4 +4,8 @@ endif() set(BB_CORE_NAME decode) set(BB_CORE_TARGET_PREFIX "") -add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/../../goban/compiler/src ${CMAKE_CURRENT_BINARY_DIR}/goban-src) + +set(BB_CORE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..) +get_filename_component(_BB_REPO_ROOT "${BB_CORE_DIR}/../../.." ABSOLUTE) +include_directories(${_BB_REPO_ROOT}/examples/balls) +add_subdirectory(src) diff --git a/examples/cores/decode/compiler/src/CMakeLists.txt b/examples/cores/decode/compiler/src/CMakeLists.txt new file mode 100644 index 000000000..7ed4fa960 --- /dev/null +++ b/examples/cores/decode/compiler/src/CMakeLists.txt @@ -0,0 +1,60 @@ +if(NOT COMMAND add_mlir_library OR NOT COMMAND add_mlir_dialect) + message(FATAL_ERROR "MLIR CMake helpers must be loaded. Build as part of buddy-mlir.") +endif() + +message(STATUS "Building Decode compiler Core package") +message(STATUS " Source: ${CMAKE_CURRENT_SOURCE_DIR}") +message(STATUS " Binary: ${CMAKE_CURRENT_BINARY_DIR}") + +get_filename_component(BB_REPO_ROOT ${CMAKE_CURRENT_LIST_DIR}/../../../../.. ABSOLUTE) +set(BB_COMPILER_COMMON_DIR ${BB_REPO_ROOT}/compiler/src) + +if(NOT EXISTS ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Buckyball.td) + message(FATAL_ERROR "Missing Buckyball compiler common dir: ${BB_COMPILER_COMMON_DIR}") +endif() + +include_directories(${BB_COMPILER_COMMON_DIR}) +include_directories(${BB_COMPILER_COMMON_DIR}/Dialect) +include_directories(${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball) +include_directories(${CMAKE_CURRENT_SOURCE_DIR}) +include_directories(${CMAKE_CURRENT_BINARY_DIR}) +include_directories(${CMAKE_CURRENT_SOURCE_DIR}/Dialect) +include_directories(${CMAKE_CURRENT_BINARY_DIR}/Dialect) +include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}/Dialect/Buckyball) + +set(DECODE_BALL_COMPILER_DIALECT_DIRS + ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/vector/compiler/src/Dialect/Buckyball +) + +set(BB_BALL_BACKEND_INTRINSICS_DIRS) +foreach(DECODE_BALL_COMPILER_DIALECT_DIR ${DECODE_BALL_COMPILER_DIALECT_DIRS}) + if(NOT EXISTS ${DECODE_BALL_COMPILER_DIALECT_DIR}) + message(FATAL_ERROR + "Missing decode ball compiler dialect dir: ${DECODE_BALL_COMPILER_DIALECT_DIR}") + endif() + include_directories(${DECODE_BALL_COMPILER_DIALECT_DIR}) + get_filename_component(DECODE_BALL_COMPILER_DIR + ${DECODE_BALL_COMPILER_DIALECT_DIR}/../.. + ABSOLUTE) + list(APPEND BB_BALL_BACKEND_INTRINSICS_DIRS + ${DECODE_BALL_COMPILER_DIR}/Backend/include) +endforeach() + +set(BB_BALL_BACKEND_INTRINSICS_DIRS ${BB_BALL_BACKEND_INTRINSICS_DIRS} + CACHE INTERNAL "Buckyball ball backend intrinsic directories") + +add_subdirectory(Dialect) +add_subdirectory(Conversion) +add_subdirectory(Target) + +add_custom_target(buckyball-compiler-${BB_CORE_NAME} + DEPENDS + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballToLLVMIRTranslation + ${BB_CORE_TARGET_PREFIX}LowerBuckyballPass + ${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass +) diff --git a/examples/cores/decode/compiler/src/Conversion/CMakeLists.txt b/examples/cores/decode/compiler/src/Conversion/CMakeLists.txt new file mode 100644 index 000000000..9ca69b1a2 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/CMakeLists.txt @@ -0,0 +1,2 @@ +add_subdirectory(LowerBuckyball) +add_subdirectory(LowerTileToBuckyball) diff --git a/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp new file mode 100644 index 000000000..c33e08f07 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp @@ -0,0 +1,82 @@ +//===- AssignPhysicalBanksPass.cpp - Decode physical bank assignment ------===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/WalkPatternRewriteDriver.h" + +#include "Buckyball/BuckyballDialect.h" + +using namespace mlir; + +namespace mlir::buddy { +void populateTransposeBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, PhysicalBankState &state); +void populateFp2IntBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateInt2FpBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateVecBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +} // namespace mlir::buddy + +namespace { + +class AssignPhysicalBanksPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(AssignPhysicalBanksPass) + AssignPhysicalBanksPass() = default; + AssignPhysicalBanksPass(const AssignPhysicalBanksPass &) {} + + StringRef getArgument() const final { return "assign-physical-banks"; } + StringRef getDescription() const final { + return "Assign physical banks for Decode bank-SSA ops."; + } + + Option bankNum{*this, "bank_num", + llvm::cl::desc("Number of physical banks."), + llvm::cl::init(20)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (bankNum <= 0) { + func.emitError("assign-physical-banks: bank_num must be > 0"); + signalPassFailure(); + return; + } + + mlir::buddy::PhysicalBankState state(bankNum); + RewritePatternSet patterns(&getContext()); + mlir::buddy::addBaseAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateFp2IntBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateInt2FpBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateTransposeBallAssignPhysicalBankPatterns(patterns, + state); + mlir::buddy::populateVecBallAssignPhysicalBankPatterns(patterns, state); + + walkAndApplyPatterns(func, std::move(patterns)); + if (failed(mlir::buddy::verifyNoBankSSAOps(func))) { + signalPassFailure(); + return; + } + + if (!state.empty()) { + func.emitError("assign-physical-banks: leaked virtual bank handles"); + signalPassFailure(); + } + } +}; + +} // namespace + +void mlir::buddy::registerAssignPhysicalBanksPass() { + PassRegistration(); +} diff --git a/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt new file mode 100644 index 000000000..e3c8842f1 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt @@ -0,0 +1,16 @@ +add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerBuckyballPass + ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp + ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/PhysicalBankState.cpp + ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/ReportBankUsagePass.cpp + ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/vector/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + AssignPhysicalBanksPass.cpp + LowerBuckyballPass.cpp + LowerBuckyballToBankSSAPass.cpp + + LINK_LIBS PUBLIC + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms + MLIRLinalgDialect +) diff --git a/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp new file mode 100644 index 000000000..c09bde647 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp @@ -0,0 +1,259 @@ +//===- LowerBuckyballPass.cpp - Decode Buckyball lowering pass ------------===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/TypeConverter.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/ControlFlow/IR/ControlFlowOps.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/LLVMIR/LLVMDialect.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/Pass/Pass.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" +#include "Buckyball/Transform.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace { + +class LowerBuckyballToLLVMPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBuckyballToLLVMPass) + LowerBuckyballToLLVMPass() = default; + LowerBuckyballToLLVMPass(const LowerBuckyballToLLVMPass &) {} + + StringRef getArgument() const final { return "lower-buckyball"; } + StringRef getDescription() const final { + return "Lower Decode Buckyball dialect ops."; + } + + Option bankWidthBytes{ + *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), + llvm::cl::init(16)}; + Option bankDepth{*this, "bank_depth", + llvm::cl::desc("Depth of each bank."), + llvm::cl::init(1024)}; + Option bankNum{*this, "bank_num", llvm::cl::desc("Number of banks."), + llvm::cl::init(20)}; + Option stable{*this, "stable", + llvm::cl::desc("Use stable LLVM Buckyball intrinsics."), + llvm::cl::init(false)}; + Option rushB{ + *this, "rushb", + llvm::cl::desc("Lower DMA operations to the rushB host ABI."), + llvm::cl::init(false)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + MLIRContext *context = &getContext(); + ModuleOp module = getOperation(); + LLVMTypeConverter converter(context); + RewritePatternSet patterns(context); + LLVMConversionTarget target(*context); + + configureBuckyballLegalizeForExportTarget(target, stable); + target.addLegalDialect(); + populateBuckyballLegalizeForLLVMExportPatterns( + converter, patterns, bankWidthBytes, bankDepth, bankNum, + /*includeFuncOperandForwarding=*/false, stable, rushB); + + ConversionConfig config; + config.allowPatternRollback = false; + if (failed(applyPartialConversion(module, target, std::move(patterns), + config))) + signalPassFailure(); + } +}; + +class LowerBankSSAToIntrinsicsPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBankSSAToIntrinsicsPass) + LowerBankSSAToIntrinsicsPass() = default; + LowerBankSSAToIntrinsicsPass(const LowerBankSSAToIntrinsicsPass &) {} + + StringRef getArgument() const final { return "lower-bank-ssa-to-intrinsics"; } + StringRef getDescription() const final { + return "Lower Decode bank-SSA and Buckyball ops to intrinsic ops."; + } + + Option bankWidthBytes{ + *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), + llvm::cl::init(16)}; + Option bankDepth{*this, "bank_depth", + llvm::cl::desc("Depth of each bank."), + llvm::cl::init(1024)}; + Option bankNum{*this, "bank_num", llvm::cl::desc("Number of banks."), + llvm::cl::init(20)}; + Option stable{*this, "stable", + llvm::cl::desc("Use stable LLVM Buckyball intrinsics."), + llvm::cl::init(false)}; + Option rushB{ + *this, "rushb", + llvm::cl::desc("Lower DMA operations to the rushB host ABI."), + llvm::cl::init(false)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + MLIRContext *context = &getContext(); + ModuleOp module = getOperation(); + LLVMTypeConverter converter(context); + RewritePatternSet patterns(context); + LLVMConversionTarget target(*context); + + configureBuckyballLegalizeForExportTarget(target, stable); + target.addLegalDialect(); + populateBuckyballLegalizeForLLVMExportPatterns( + converter, patterns, bankWidthBytes, bankDepth, bankNum, + /*includeFuncOperandForwarding=*/false, stable, rushB); + + ConversionConfig config; + config.allowPatternRollback = false; + if (failed(applyPartialConversion(module, target, std::move(patterns), + config))) + signalPassFailure(); + } +}; + +static FlatSymbolRefAttr getOrInsertRushBFunction(OpBuilder &builder, + ModuleOp module, + StringRef name, + LLVM::LLVMFunctionType type) { + if (module.lookupSymbol(name)) + return FlatSymbolRefAttr::get(builder.getContext(), name); + + OpBuilder::InsertionGuard guard(builder); + builder.setInsertionPointToEnd(module.getBody()); + LLVM::LLVMFuncOp::create(builder, module.getLoc(), name, type, + LLVM::Linkage::External, false, LLVM::CConv::C); + return FlatSymbolRefAttr::get(builder.getContext(), name); +} + +class LowerBuckyballIntrinsicsToRushBPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID( + LowerBuckyballIntrinsicsToRushBPass) + LowerBuckyballIntrinsicsToRushBPass() = default; + LowerBuckyballIntrinsicsToRushBPass( + const LowerBuckyballIntrinsicsToRushBPass &) {} + + StringRef getArgument() const final { + return "lower-buckyball-intrinsics-to-rushb"; + } + StringRef getDescription() const final { + return "Lower Decode Buckyball intrinsic ops to the rushB host ABI."; + } + + Option coreId{ + *this, "core_id", + llvm::cl::desc("Bind generated rushB calls to a tile Core."), + llvm::cl::init(-1)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry.insert(); + } + + void runOnOperation() override { + ModuleOp module = getOperation(); + SmallVector intrinsicOps; + module.walk([&](Operation *op) { + if (isa( + op)) + intrinsicOps.push_back(op); + }); + + OpBuilder builder(&getContext()); + Type i32Type = IntegerType::get(&getContext(), 32); + Type voidType = LLVM::LLVMVoidType::get(&getContext()); + FlatSymbolRefAttr selectCallee; + if (coreId >= 0) { + auto selectType = + LLVM::LLVMFunctionType::get(voidType, {i32Type, i32Type}); + selectCallee = getOrInsertRushBFunction( + builder, module, "rushb_select_accelerator", selectType); + } + llvm::SmallPtrSet boundFunctions; + for (Operation *op : intrinsicOps) { + if (coreId >= 0) { + if (auto function = op->getParentOfType(); + function && boundFunctions.insert(function).second) { + OpBuilder::InsertionGuard guard(builder); + builder.setInsertionPointToStart(&function.front()); + auto selected = LLVM::ConstantOp::create( + builder, function.getLoc(), i32Type, + builder.getI32IntegerAttr(static_cast(coreId))); + auto chip = + LLVM::ConstantOp::create(builder, function.getLoc(), i32Type, + builder.getI32IntegerAttr(0)); + LLVM::CallOp::create(builder, function.getLoc(), TypeRange{}, + selectCallee, ValueRange{selected, chip}); + } + } + builder.setInsertionPoint(op); + SmallVector operands; + StringRef name; + if (isa(op)) { + name = "rushb_mset"; + operands.append(op->getOperands().begin(), op->getOperands().end()); + } else if (isa(op) || isa(op)) { + name = isa(op) ? "rushb_mvin" : "rushb_mvout"; + operands.append(op->getOperands().begin(), op->getOperands().end()); + } else { + name = "rushb_custom"; + if (auto custom = dyn_cast(op)) { + operands.append(custom.getOperands().begin(), + custom.getOperands().end()); + operands.push_back(LLVM::ConstantOp::create( + builder, op->getLoc(), i32Type, + builder.getI32IntegerAttr(custom.getFunct7()))); + } else { + auto fence = cast(op); + operands.append(fence->getOperands().begin(), + fence->getOperands().end()); + operands.push_back(LLVM::ConstantOp::create( + builder, op->getLoc(), i32Type, builder.getI32IntegerAttr(0))); + } + } + + SmallVector argumentTypes; + for (Value operand : operands) + argumentTypes.push_back(operand.getType()); + auto type = LLVM::LLVMFunctionType::get(voidType, argumentTypes); + auto callee = getOrInsertRushBFunction(builder, module, name, type); + LLVM::CallOp::create(builder, op->getLoc(), TypeRange{}, callee, + operands); + op->erase(); + } + } +}; + +} // namespace + +void mlir::buddy::registerLowerBuckyballPass() { + PassRegistration(); +} + +void mlir::buddy::registerLowerBankSSAToIntrinsicsPass() { + PassRegistration(); + PassRegistration(); +} diff --git a/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp new file mode 100644 index 000000000..54b98d005 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp @@ -0,0 +1,90 @@ +//===- LowerBuckyballToBankSSAPass.cpp - Decode bank-SSA lowering ---------===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/Dialect/Utils/StructuredOpsUtils.h" +#include "mlir/IR/AffineMap.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/GreedyPatternRewriteDriver.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; + +namespace { + +class MemTransposeToLinalgPattern + : public OpRewritePattern<::buddy::buckyball::MemTransposeOp> { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(::buddy::buckyball::MemTransposeOp op, + PatternRewriter &rewriter) const override { + auto input = dyn_cast(op.getInput().getType()); + auto output = dyn_cast(op.getOutput().getType()); + if (!input || !output || !input.hasStaticShape() || + !output.hasStaticShape() || input.getRank() != 2 || + output.getRank() != 2 || output.getShape()[0] != input.getShape()[1] || + output.getShape()[1] != input.getShape()[0]) + return op.emitError("requires matching static rank-2 transpose memrefs"); + + MLIRContext *context = rewriter.getContext(); + AffineMap identity = AffineMap::getMultiDimIdentityMap(2, context); + AffineMap transpose = AffineMap::get( + 2, 0, {rewriter.getAffineDimExpr(1), rewriter.getAffineDimExpr(0)}, + context); + SmallVector iterators(2, + utils::IteratorType::parallel); + rewriter.create( + op.getLoc(), TypeRange{}, ValueRange{op.getInput()}, + ValueRange{op.getOutput()}, ArrayRef{identity, transpose}, + iterators, [](OpBuilder &builder, Location location, ValueRange args) { + builder.create(location, args[0]); + }); + rewriter.eraseOp(op); + return success(); + } +}; + +} // namespace + +namespace { + +class LowerBuckyballToBankSSAPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBuckyballToBankSSAPass) + + StringRef getArgument() const final { return "lower-buckyball-to-bank-ssa"; } + StringRef getDescription() const final { + return "Lower Decode Buckyball ops to explicit bank-SSA ops."; + } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry.insert(); + } + + void runOnOperation() override { + RewritePatternSet patterns(&getContext()); + patterns.add(&getContext()); + if (failed(applyPatternsGreedily(getOperation(), std::move(patterns)))) + signalPassFailure(); + } +}; + +} // namespace + +void mlir::buddy::registerLowerBuckyballToBankSSAPass() { + PassRegistration(); +} diff --git a/examples/cores/decode/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt b/examples/cores/decode/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt new file mode 100644 index 000000000..82b282ef1 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt @@ -0,0 +1,8 @@ +add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass + LowerTileToBuckyball.cpp + + LINK_LIBS PUBLIC + BuddyTile + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball + MLIRLinalgDialect +) diff --git a/examples/cores/decode/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp b/examples/cores/decode/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp new file mode 100644 index 000000000..0dc5d06a0 --- /dev/null +++ b/examples/cores/decode/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp @@ -0,0 +1,110 @@ +//===- LowerTileToBuckyball.cpp - Decode tile pass registration -----------===// +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +//===----------------------------------------------------------------------===// + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/DialectConversion.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" +#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" +#include "Tile/TileDialect.h" +#include "Tile/TileOps.h" +#include "Tile/Transform.h" + +using namespace mlir; +using namespace ::buddy::buckyball; +namespace tile = ::buddy::tile; + +namespace { +class TileTransposeLowering : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(tile::TileTransposeOp op, + PatternRewriter &rewriter) const override { + auto inputType = dyn_cast(op.getAMemArray().getType()); + auto outputType = dyn_cast(op.getBMemArray().getType()); + if (!inputType || !outputType || !inputType.hasStaticShape() || + !outputType.hasStaticShape() || inputType.getRank() != 2 || + outputType.getRank() != 2) + return op.emitError("requires static rank-2 input and output memrefs"); + if (outputType.getShape()[0] != inputType.getShape()[1] || + outputType.getShape()[1] != inputType.getShape()[0] || + inputType.getElementType() != outputType.getElementType()) + return op.emitError("requires a matching transposed output memref"); + + rewriter.create(op.getLoc(), op.getAMemArray(), + op.getBMemArray()); + rewriter.eraseOp(op); + return success(); + } +}; + +class LowerTileToBuckyballPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerTileToBuckyballPass) + StringRef getArgument() const final { return "convert-tile-to-buckyball"; } + StringRef getDescription() const final { + return "Convert Tile operations for the Decode Core."; + } + LowerTileToBuckyballPass() = default; + LowerTileToBuckyballPass(const LowerTileToBuckyballPass &) {} + + Option bankWidthBytes{ + *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), + llvm::cl::init(16)}; + Option bankDepth{*this, "bank_depth", + llvm::cl::desc("Bank depth (rows per bank)."), + llvm::cl::init(1024)}; + Option bankNum{*this, "bank_num", + llvm::cl::desc("Number of physical banks."), + llvm::cl::init(20)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + MLIRContext *context = &getContext(); + ConversionTarget target(*context); + target.addLegalDialect<::buddy::buckyball::BuckyballDialect, + memref::MemRefDialect, arith::ArithDialect, + scf::SCFDialect, func::FuncDialect, + linalg::LinalgDialect>(); + target.addIllegalOp(); + + RewritePatternSet patterns(context); + patterns.add(context); + if (failed(applyPartialConversion(getOperation(), target, + std::move(patterns)))) + signalPassFailure(); + } +}; +} // namespace + +void mlir::buddy::registerLowerTileToBuckyballPass() { + PassRegistration(); +} diff --git a/examples/cores/decode/compiler/src/Dialect/Buckyball/CMakeLists.txt b/examples/cores/decode/compiler/src/Dialect/Buckyball/CMakeLists.txt new file mode 100644 index 000000000..20c3cfc78 --- /dev/null +++ b/examples/cores/decode/compiler/src/Dialect/Buckyball/CMakeLists.txt @@ -0,0 +1,17 @@ +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) +mlir_tablegen(Buckyball.h.inc -gen-op-decls) +mlir_tablegen(Buckyball.cpp.inc -gen-op-defs) +mlir_tablegen(BuckyballTypes.h.inc -gen-typedef-decls -typedefs-dialect=buckyball) +mlir_tablegen(BuckyballTypes.cpp.inc -gen-typedef-defs -typedefs-dialect=buckyball) +mlir_tablegen(BuckyballDialect.h.inc -gen-dialect-decls -dialect=buckyball) +mlir_tablegen(BuckyballDialect.cpp.inc -gen-dialect-defs -dialect=buckyball) +add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}MLIRBuckyballIncGen) +add_mlir_doc(DecodeBuckyball ${BB_CORE_TARGET_PREFIX}Buckyball Dialects/ + -gen-dialect-doc) + +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) +mlir_tablegen(BuckyballConversions.inc -gen-llvmir-conversions) +add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen) + +add_subdirectory(IR) +add_subdirectory(Transforms) diff --git a/examples/cores/decode/compiler/src/Dialect/Buckyball/DecodeBuckyball.td b/examples/cores/decode/compiler/src/Dialect/Buckyball/DecodeBuckyball.td new file mode 100644 index 000000000..056d785ab --- /dev/null +++ b/examples/cores/decode/compiler/src/Dialect/Buckyball/DecodeBuckyball.td @@ -0,0 +1,26 @@ +//===-- DecodeBuckyball.td - Decode Buckyball dialect ops -*- tablegen -*-===// +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +//===----------------------------------------------------------------------===// + +#ifndef DECODE_BUCKYBALL_OPS +#define DECODE_BUCKYBALL_OPS + +include "Buckyball.td" +include "Fp2IntBall.td" +include "Int2FpBall.td" +include "TransposeBall.td" +include "VecBall.td" + +#endif // DECODE_BUCKYBALL_OPS diff --git a/examples/cores/decode/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt b/examples/cores/decode/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt new file mode 100644 index 000000000..a2e7abc2e --- /dev/null +++ b/examples/cores/decode/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt @@ -0,0 +1,16 @@ +add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyball + ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/IR/BuckyballDialect.cpp + ${BB_TARGET_REGISTRY_SOURCE} + + LINK_LIBS PUBLIC + MLIRIR + MLIRLLVMDialect +) + +target_include_directories(${BB_CORE_TARGET_PREFIX}BuddyBuckyball PUBLIC + ${BB_COMPILER_COMMON_DIR} + ${BB_COMPILER_COMMON_DIR}/Dialect + ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball + ${CMAKE_CURRENT_BINARY_DIR}/.. + ${CMAKE_CURRENT_BINARY_DIR}/../../.. +) diff --git a/examples/cores/decode/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt b/examples/cores/decode/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt new file mode 100644 index 000000000..668792218 --- /dev/null +++ b/examples/cores/decode/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt @@ -0,0 +1,14 @@ +add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms + ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp + ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/vector/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + LegalizeForLLVMExport.cpp + + LINK_LIBS PUBLIC + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball + MLIRIR + MLIRLLVMCommonConversion + MLIRLLVMDialect +) diff --git a/examples/cores/decode/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/cores/decode/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..69168364b --- /dev/null +++ b/examples/cores/decode/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,78 @@ +//===- LegalizeForLLVMExport.cpp - Decode Buckyball LLVM lowering ---------===// +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +//===----------------------------------------------------------------------===// + +#include "Buckyball/Transform.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" + +using namespace mlir; +using namespace buddy::buckyball::legalize; + +namespace mlir::buddy::buckyball { +void populateTransposeBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureTransposeBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateFp2IntBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureFp2IntBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateInt2FpBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureInt2FpBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateVecBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, + RewritePatternSet &patterns, + bool stable, + int64_t bankDepth, + bool rushB); +void configureVecBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +} // namespace mlir::buddy::buckyball + +void mlir::populateBuckyballLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, + int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum, + bool includeFuncOperandForwarding, bool stable, bool rushB) { + (void)bankWidthBytes; + (void)bankNum; + + populateBaseLegalizeForLLVMExportPatterns( + converter, patterns, includeFuncOperandForwarding, rushB); + mlir::buddy::buckyball::populateFp2IntBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateInt2FpBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateTransposeBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateVecBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); +} + +void mlir::configureBuckyballLegalizeForExportTarget( + LLVMConversionTarget &target, bool stable) { + configureBaseLegalizeForExportTarget(target); + mlir::buddy::buckyball::configureFp2IntBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureInt2FpBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureTransposeBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureVecBallLegalizeForExportTarget(target, + stable); +} diff --git a/examples/cores/decode/compiler/src/Dialect/CMakeLists.txt b/examples/cores/decode/compiler/src/Dialect/CMakeLists.txt new file mode 100644 index 000000000..ec181148e --- /dev/null +++ b/examples/cores/decode/compiler/src/Dialect/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(Buckyball) diff --git a/examples/cores/decode/compiler/src/Target/CMakeLists.txt b/examples/cores/decode/compiler/src/Target/CMakeLists.txt new file mode 100644 index 000000000..39d31dc9b --- /dev/null +++ b/examples/cores/decode/compiler/src/Target/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(LLVMIR) diff --git a/examples/cores/decode/compiler/src/Target/LLVMIR/CMakeLists.txt b/examples/cores/decode/compiler/src/Target/LLVMIR/CMakeLists.txt new file mode 100644 index 000000000..0ca0f41c5 --- /dev/null +++ b/examples/cores/decode/compiler/src/Target/LLVMIR/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(Dialect) diff --git a/examples/cores/decode/compiler/src/Target/LLVMIR/Dialect/Buckyball/CMakeLists.txt b/examples/cores/decode/compiler/src/Target/LLVMIR/Dialect/Buckyball/CMakeLists.txt new file mode 100644 index 000000000..06b99171b --- /dev/null +++ b/examples/cores/decode/compiler/src/Target/LLVMIR/Dialect/Buckyball/CMakeLists.txt @@ -0,0 +1,15 @@ +add_mlir_translation_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballToLLVMIRTranslation + ${BB_COMPILER_COMMON_DIR}/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp + + DEPENDS + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen + intrinsics_gen + + LINK_COMPONENTS + Core + + LINK_LIBS PUBLIC + MLIRIR + MLIRSupport + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball +) diff --git a/examples/cores/decode/compiler/src/Target/LLVMIR/Dialect/CMakeLists.txt b/examples/cores/decode/compiler/src/Target/LLVMIR/Dialect/CMakeLists.txt new file mode 100644 index 000000000..ec181148e --- /dev/null +++ b/examples/cores/decode/compiler/src/Target/LLVMIR/Dialect/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(Buckyball) diff --git a/examples/cores/decode/configs/balldomains/default.toml b/examples/cores/decode/configs/balldomains/default.toml index 4d0de9df4..8adfb9d11 100644 --- a/examples/cores/decode/configs/balldomains/default.toml +++ b/examples/cores/decode/configs/balldomains/default.toml @@ -1,19 +1,22 @@ # Poly Decode Core Ball Domain Configuration +# +# Decode is dominated by token-sized matrix-vector work. It exposes VecMat16 +# and deliberately omits the batch-oriented systolic and image-convolution +# Balls used by other targets. -ballNum = 5 +ballNum = 4 ballIdMappings = [ { ballId = 0, ballName = "VecBall", ballClass = "examples.balls.vector.VecBall", config = "../../../../balls/vector/configs/default.toml", inBW = 2, outBW = 4 }, { ballId = 1, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1 }, { ballId = 2, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 3, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 4, ballName = "Im2colBall", ballClass = "examples.balls.im2col.Im2colBall", config = "../../../../balls/im2col/configs/default.toml", inBW = 1, outBW = 1 } + { ballId = 3, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 } ] ballISA = [ - { mnemonic = "MATMUL_WARP16", funct7 = 64, bid = 0 }, + { mnemonic = "VECMAT16", funct7 = 64, bid = 0 }, { mnemonic = "FP2INT", funct7 = 51, bid = 1 }, - { mnemonic = "INT2FP", funct7 = 52, bid = 2 }, - { mnemonic = "TRANSPOSE", funct7 = 49, bid = 3 }, - { mnemonic = "IM2COL", funct7 = 48, bid = 4 } + { mnemonic = "INT2FP_TENSOR", funct7 = 52, bid = 2 }, + { mnemonic = "INT2FP_CHANNEL", funct7 = 54, bid = 2 }, + { mnemonic = "TRANSPOSE", funct7 = 49, bid = 3 } ] diff --git a/examples/cores/decode/core.toml b/examples/cores/decode/configs/default.toml similarity index 66% rename from examples/cores/decode/core.toml rename to examples/cores/decode/configs/default.toml index 404c1c16f..67e6e1f9e 100644 --- a/examples/cores/decode/core.toml +++ b/examples/cores/decode/configs/default.toml @@ -1,10 +1,10 @@ # Decode Core: Goban accelerator with sub-ROB scheduling for short steps. -balldomain = "configs/balldomains/default.toml" -memdomain = "configs/memdomains/default.toml" -frontend = "configs/frontends/default.toml" -gpdomain = "configs/gpdomains/default.toml" -core = "configs/core/default.toml" +balldomain = "balldomains/default.toml" +memdomain = "memdomains/default.toml" +frontend = "frontends/default.toml" +gpdomain = "gpdomains/default.toml" +core = "core/default.toml" [rocketCore] xLen = 64 @@ -13,6 +13,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = false [rocketCore.mulDiv] enable = true diff --git a/examples/cores/decode/configs/frontends/default.toml b/examples/cores/decode/configs/frontends/default.toml index 57e9ec42c..0d0cb8b84 100644 --- a/examples/cores/decode/configs/frontends/default.toml +++ b/examples/cores/decode/configs/frontends/default.toml @@ -1,7 +1,8 @@ robEntries = 16 rsOutOfOrderResponse = true bankIdLen = 10 -vbankIdUpperBound = 15 +vbankIdUpperBound = 7 +sharedBankIdBase = 32 iterLen = 34 subRobEnable = true subRobDepth = 64 diff --git a/examples/cores/decode/configs/memdomains/default.toml b/examples/cores/decode/configs/memdomains/default.toml index 53707516e..17a59adeb 100644 --- a/examples/cores/decode/configs/memdomains/default.toml +++ b/examples/cores/decode/configs/memdomains/default.toml @@ -1,16 +1,17 @@ # Poly Decode Core Memory Domain Configuration # Private (per-core) memory parameters. Shared memory lives in the tile TOML. +# Token-at-a-time decode favors a smaller, lower-footprint private scratchpad. [bank] -num = 16 +num = 8 width = 128 -entries = 1024 +entries = 512 maskLen = 16 channel = 7 [dma] nXacts = 8 -burstMaxBytes = 64 +burstMaxBytes = 1024 busWidth = 128 maxInFlightMemReqs = 16 diff --git a/examples/cores/decode/emu/Cargo.lock b/examples/cores/decode/emu/Cargo.lock deleted file mode 100644 index 63a87c730..000000000 --- a/examples/cores/decode/emu/Cargo.lock +++ /dev/null @@ -1,932 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bemu-decode" -version = "0.1.0" -dependencies = [ - "bemu-goban", -] - -[[package]] -name = "bemu-goban" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "camino", - "cc", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "log", - "nix 0.29.0", - "once_cell", - "serde", - "serde_json", - "snafu", - "toml", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "clap" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "find-msvc-tools" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de" - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "portable-atomic" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "thiserror" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/cores/decode/emu/Cargo.toml b/examples/cores/decode/emu/Cargo.toml deleted file mode 100644 index b21423e3f..000000000 --- a/examples/cores/decode/emu/Cargo.toml +++ /dev/null @@ -1,13 +0,0 @@ -[package] -name = "bemu-decode" -version = "0.1.0" -edition = "2021" -build = "build.rs" - -[lib] -name = "bemu_decode" -path = "src/lib.rs" -crate-type = ["rlib", "cdylib"] - -[dependencies] -core_bemu = { package = "bemu-goban", path = "../../goban/emu" } diff --git a/examples/cores/decode/emu/build.rs b/examples/cores/decode/emu/build.rs deleted file mode 100644 index 627f300a0..000000000 --- a/examples/cores/decode/emu/build.rs +++ /dev/null @@ -1,14 +0,0 @@ -use std::{env, fs, path::PathBuf}; - -fn main() { - let manifest = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); - let build_root = manifest.join("target/release/build"); - let Ok(entries) = fs::read_dir(build_root) else { return }; - for entry in entries.flatten() { - let lib_dir = entry.path().join("out/spike_install/lib"); - if lib_dir.join("libriscv.so").is_file() { - println!("cargo:rustc-link-arg-cdylib=-Wl,-rpath,{}", lib_dir.display()); - break; - } - } -} diff --git a/examples/cores/decode/emu/src/lib.rs b/examples/cores/decode/emu/src/lib.rs deleted file mode 100644 index c11148620..000000000 --- a/examples/cores/decode/emu/src/lib.rs +++ /dev/null @@ -1 +0,0 @@ -pub use core_bemu::*; diff --git a/examples/cores/decode/manifest.toml b/examples/cores/decode/manifest.toml deleted file mode 100644 index 8c5bbe875..000000000 --- a/examples/cores/decode/manifest.toml +++ /dev/null @@ -1,5 +0,0 @@ -[core] -name = "decode" -kind = "buckyball" -config = "core.toml" -compiler = "compiler" diff --git a/examples/cores/goban/compiler/CMakeLists.txt b/examples/cores/goban/compiler/CMakeLists.txt index a88098c2b..cf5f19d4c 100644 --- a/examples/cores/goban/compiler/CMakeLists.txt +++ b/examples/cores/goban/compiler/CMakeLists.txt @@ -5,4 +5,8 @@ endif() set(BB_CORE_NAME goban) set(BB_CORE_TARGET_PREFIX "") +set(BB_CORE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..) +get_filename_component(_BB_REPO_ROOT "${BB_CORE_DIR}/../../.." ABSOLUTE) +include_directories(${_BB_REPO_ROOT}/examples/balls) + add_subdirectory(src) diff --git a/examples/cores/goban/compiler/src/CMakeLists.txt b/examples/cores/goban/compiler/src/CMakeLists.txt index 3926baf21..6f67f6255 100644 --- a/examples/cores/goban/compiler/src/CMakeLists.txt +++ b/examples/cores/goban/compiler/src/CMakeLists.txt @@ -26,7 +26,7 @@ set(GOBAN_BALL_COMPILER_DIALECT_DIRS ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Dialect/Buckyball ) foreach(GOBAN_BALL_COMPILER_DIALECT_DIR ${GOBAN_BALL_COMPILER_DIALECT_DIRS}) diff --git a/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp b/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp index 54b599a6f..bb3f3d4ee 100644 --- a/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp +++ b/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp @@ -12,12 +12,14 @@ using namespace mlir; namespace mlir::buddy { -void populateTransposeAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateFp2IntAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateInt2FpAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateTransposeBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, PhysicalBankState &state); +void populateFp2IntBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateInt2FpBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateSMatMulBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); } // namespace mlir::buddy namespace { @@ -54,9 +56,11 @@ class AssignPhysicalBanksPass mlir::buddy::PhysicalBankState state(bankNum); RewritePatternSet patterns(&getContext()); mlir::buddy::addBaseAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateFp2IntAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateInt2FpAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateTransposeAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateFp2IntBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateInt2FpBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateTransposeBallAssignPhysicalBankPatterns(patterns, + state); + mlir::buddy::populateSMatMulBallAssignPhysicalBankPatterns(patterns, state); walkAndApplyPatterns(func, std::move(patterns)); if (failed(mlir::buddy::verifyNoBankSSAOps(func))) { diff --git a/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt b/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt index d76498cd8..3880ea4f1 100644 --- a/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt +++ b/examples/cores/goban/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt @@ -5,6 +5,7 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerBuckyballPass ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp AssignPhysicalBanksPass.cpp LowerBuckyballPass.cpp LowerBuckyballToBankSSAPass.cpp diff --git a/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt b/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt index 61a092d94..6b73e8283 100644 --- a/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt +++ b/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt @@ -1,6 +1,6 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass LowerTileToBuckyball.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp LINK_LIBS PUBLIC BuddyTile diff --git a/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp b/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp index de1081138..e71f84fad 100644 --- a/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp +++ b/examples/cores/goban/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp @@ -34,6 +34,13 @@ using namespace mlir; using namespace ::buddy::buckyball; namespace tile = ::buddy::tile; +namespace mlir::buddy { +void populateSMatMulBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, + int64_t bankNum); +} // namespace mlir::buddy + namespace { class TileTransposeLowering : public OpRewritePattern { public: @@ -98,8 +105,8 @@ class LowerTileToBuckyballPass target.addIllegalOp(); RewritePatternSet patterns(context); - mlir::buddy::populateMatrixTileMatMulPatterns(patterns, bankWidthBytes, - bankDepth, bankNum); + mlir::buddy::populateSMatMulBallTileLoweringPatterns( + patterns, bankWidthBytes, bankDepth, bankNum); patterns.add(context); if (failed(applyPartialConversion(getOperation(), target, std::move(patterns)))) diff --git a/examples/cores/goban/compiler/src/Dialect/Buckyball/CMakeLists.txt b/examples/cores/goban/compiler/src/Dialect/Buckyball/CMakeLists.txt index f6b8a6915..004ad362c 100644 --- a/examples/cores/goban/compiler/src/Dialect/Buckyball/CMakeLists.txt +++ b/examples/cores/goban/compiler/src/Dialect/Buckyball/CMakeLists.txt @@ -1,4 +1,4 @@ -set(LLVM_TARGET_DEFINITIONS GobanBuckyball.td) +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) mlir_tablegen(Buckyball.h.inc -gen-op-decls) mlir_tablegen(Buckyball.cpp.inc -gen-op-defs) mlir_tablegen(BuckyballTypes.h.inc -gen-typedef-decls -typedefs-dialect=buckyball) @@ -9,7 +9,7 @@ add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}MLIRBuckyballIncGen) add_mlir_doc(GobanBuckyball ${BB_CORE_TARGET_PREFIX}Buckyball Dialects/ -gen-dialect-doc) -set(LLVM_TARGET_DEFINITIONS GobanBuckyball.td) +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) mlir_tablegen(BuckyballConversions.inc -gen-llvmir-conversions) add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen) diff --git a/examples/cores/goban/compiler/src/Dialect/Buckyball/GobanBuckyball.td b/examples/cores/goban/compiler/src/Dialect/Buckyball/GobanBuckyball.td index 988f2af59..53f4e3c7d 100644 --- a/examples/cores/goban/compiler/src/Dialect/Buckyball/GobanBuckyball.td +++ b/examples/cores/goban/compiler/src/Dialect/Buckyball/GobanBuckyball.td @@ -21,6 +21,6 @@ include "Buckyball.td" include "Fp2IntBall.td" include "Int2FpBall.td" include "TransposeBall.td" -include "MatrixBall.td" +include "SMatMulBall.td" #endif // GOBAN_BUCKYBALL_OPS diff --git a/examples/cores/goban/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt b/examples/cores/goban/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt index da76ba536..a2e7abc2e 100644 --- a/examples/cores/goban/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt +++ b/examples/cores/goban/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt @@ -1,5 +1,6 @@ add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyball ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/IR/BuckyballDialect.cpp + ${BB_TARGET_REGISTRY_SOURCE} LINK_LIBS PUBLIC MLIRIR diff --git a/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt b/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt index 486fe9f3c..715c53bd2 100644 --- a/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt +++ b/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt @@ -3,7 +3,7 @@ add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp LegalizeForLLVMExport.cpp LINK_LIBS PUBLIC diff --git a/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index 07f8a705f..e5b23d4f8 100644 --- a/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/cores/goban/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -21,30 +21,23 @@ using namespace mlir; using namespace buddy::buckyball::legalize; namespace mlir::buddy::buckyball { -void populateTransposeLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable); -void configureTransposeLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateMatrixLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureMatrixLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateMatrixMatmulLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, - bool rushB); -void configureMatrixMatmulLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateFp2IntLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureFp2IntLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateInt2FpLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureInt2FpLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); +void populateTransposeBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &, + RewritePatternSet &, + bool, int64_t, bool); +void configureTransposeBallLegalizeForExportTarget(LLVMConversionTarget &, + bool); +void populateSMatMulBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &, + RewritePatternSet &, bool, + int64_t, bool); +void configureSMatMulBallLegalizeForExportTarget(LLVMConversionTarget &, bool); +void populateFp2IntBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &, + RewritePatternSet &, bool, + int64_t, bool); +void configureFp2IntBallLegalizeForExportTarget(LLVMConversionTarget &, bool); +void populateInt2FpBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &, + RewritePatternSet &, bool, + int64_t, bool); +void configureInt2FpBallLegalizeForExportTarget(LLVMConversionTarget &, bool); } // namespace mlir::buddy::buckyball void mlir::populateBuckyballLegalizeForLLVMExportPatterns( @@ -52,34 +45,29 @@ void mlir::populateBuckyballLegalizeForLLVMExportPatterns( int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum, bool includeFuncOperandForwarding, bool stable, bool rushB) { (void)bankWidthBytes; - (void)bankDepth; (void)bankNum; populateBaseLegalizeForLLVMExportPatterns( converter, patterns, includeFuncOperandForwarding, rushB); - mlir::buddy::buckyball::populateFp2IntLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateInt2FpLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateTransposeLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateMatrixLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateMatrixMatmulLegalizeForLLVMExportPatterns( - converter, patterns, stable, rushB); + mlir::buddy::buckyball::populateFp2IntBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateInt2FpBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateTransposeBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateSMatMulBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); } void mlir::configureBuckyballLegalizeForExportTarget( LLVMConversionTarget &target, bool stable) { configureBaseLegalizeForExportTarget(target); - mlir::buddy::buckyball::configureFp2IntLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureInt2FpLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureTransposeLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureMatrixLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureMatrixMatmulLegalizeForExportTarget(target, - stable); + mlir::buddy::buckyball::configureFp2IntBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureInt2FpBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureTransposeBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureSMatMulBallLegalizeForExportTarget(target, + stable); } diff --git a/examples/cores/goban/configs/balldomains/default.toml b/examples/cores/goban/configs/balldomains/default.toml index a93be423f..0e9b704dc 100644 --- a/examples/cores/goban/configs/balldomains/default.toml +++ b/examples/cores/goban/configs/balldomains/default.toml @@ -3,16 +3,17 @@ ballNum = 4 ballIdMappings = [ - { ballId = 0, ballName = "MatrixBall", ballClass = "examples.balls.matrix.MatrixBall", config = "../../../../balls/matrix/configs/lane16.toml", inBW = 2, outBW = 4 }, + { ballId = 0, ballName = "SMatMulBall", ballClass = "examples.balls.smatmul.SMatMulBall", config = "../../../../balls/smatmul/configs/lane16.toml", inBW = 2, outBW = 4 }, { ballId = 1, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1 }, { ballId = 2, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1 }, { ballId = 3, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 } ] ballISA = [ - { mnemonic = "MATRIX", funct7 = 65, bid = 0 }, - { mnemonic = "MATRIX_F32", funct7 = 66, bid = 0 }, + { mnemonic = "SMATMUL_OS", funct7 = 65, bid = 0 }, + { mnemonic = "SMATMUL_WS", funct7 = 68, bid = 0 }, { mnemonic = "FP2INT", funct7 = 51, bid = 1 }, - { mnemonic = "INT2FP", funct7 = 52, bid = 2 }, + { mnemonic = "INT2FP_TENSOR", funct7 = 52, bid = 2 }, + { mnemonic = "INT2FP_CHANNEL", funct7 = 54, bid = 2 }, { mnemonic = "TRANSPOSE", funct7 = 49, bid = 3 } ] diff --git a/examples/cores/goban/core.toml b/examples/cores/goban/configs/default.toml similarity index 64% rename from examples/cores/goban/core.toml rename to examples/cores/goban/configs/default.toml index 612c1642a..851d3e514 100644 --- a/examples/cores/goban/core.toml +++ b/examples/cores/goban/configs/default.toml @@ -1,10 +1,10 @@ # Goban Core Configuration (default) -balldomain = "configs/balldomains/default.toml" -memdomain = "configs/memdomains/default.toml" -frontend = "configs/frontends/default.toml" -gpdomain = "configs/gpdomains/default.toml" -core = "configs/core/default.toml" +balldomain = "balldomains/default.toml" +memdomain = "memdomains/default.toml" +frontend = "frontends/default.toml" +gpdomain = "gpdomains/default.toml" +core = "core/default.toml" [rocketCore] xLen = 64 @@ -13,6 +13,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = false [rocketCore.mulDiv] enable = true diff --git a/examples/cores/goban/configs/frontends/default.toml b/examples/cores/goban/configs/frontends/default.toml index 7adaefebd..41cd684cc 100644 --- a/examples/cores/goban/configs/frontends/default.toml +++ b/examples/cores/goban/configs/frontends/default.toml @@ -2,6 +2,7 @@ robEntries = 16 rsOutOfOrderResponse = true bankIdLen = 10 vbankIdUpperBound = 15 +sharedBankIdBase = 32 iterLen = 34 subRobEnable = false subRobDepth = 64 diff --git a/examples/cores/goban/configs/memdomains/default.toml b/examples/cores/goban/configs/memdomains/default.toml index a939ffa91..ef9a70dbc 100644 --- a/examples/cores/goban/configs/memdomains/default.toml +++ b/examples/cores/goban/configs/memdomains/default.toml @@ -10,7 +10,7 @@ channel = 7 [dma] nXacts = 8 -burstMaxBytes = 64 +burstMaxBytes = 1024 busWidth = 128 maxInFlightMemReqs = 16 diff --git a/examples/cores/goban/emu/Cargo.lock b/examples/cores/goban/emu/Cargo.lock deleted file mode 100644 index 9de9b09bd..000000000 --- a/examples/cores/goban/emu/Cargo.lock +++ /dev/null @@ -1,1317 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "android_system_properties" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" -dependencies = [ - "libc", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "assert_cmd" -version = "2.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2aa3a22042e45de04255c7bf3626e239f450200fd0493c1e382263544b20aea6" -dependencies = [ - "anstyle", - "bstr", - "libc", - "predicates", - "predicates-core", - "predicates-tree", - "wait-timeout", -] - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-p2e" -version = "0.1.0" -dependencies = [ - "bebop-rtl-trace", - "bebop-uart", - "duct", - "log", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bebop-verilator" -version = "0.1.0" -dependencies = [ - "bebop-dasm", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-uart", - "cc", - "ctrlc", - "libc", - "snafu", -] - -[[package]] -name = "bemu-goban" -version = "0.1.0" -dependencies = [ - "assert_cmd", - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-p2e", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "bebop-verilator", - "camino", - "cc", - "chrono", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "libtest-mimic", - "log", - "nix 0.29.0", - "once_cell", - "predicates", - "serde", - "serde_json", - "snafu", - "toml", - "walkdir", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "bstr" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f7dc094d718f2e1c1559ad110e27eeaae14a5465d3d56dd6dbd793079fbd530" -dependencies = [ - "memchr", - "regex-automata", - "serde_core", -] - -[[package]] -name = "bumpalo" -version = "3.20.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "chrono" -version = "0.4.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" -dependencies = [ - "iana-time-zone", - "num-traits", - "windows-link", -] - -[[package]] -name = "clap" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91e0c145792ef73a6ad36d27c75ac09f1832222a3c209689d90f534685ee5b7" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f09628afdcc538b57f3c6341e9c8e9970f18e4a481690a64974d7023bd33548b" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "difflib" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6184e33543162437515c2e2b48714794e37845ec9851711914eec9d308f6ebe8" - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "escape8259" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5692dd7b5a1978a5aeb0ce83b7655c58ca8efdcb79d21036ea249da95afec2c6" - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "float-cmp" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b09cf3155332e944990140d967ff5eceb70df778b34f77d8075db46e4704e6d8" -dependencies = [ - "num-traits", -] - -[[package]] -name = "futures-core" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" - -[[package]] -name = "futures-task" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" - -[[package]] -name = "futures-util" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" -dependencies = [ - "futures-core", - "futures-task", - "pin-project-lite", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "iana-time-zone" -version = "0.1.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "log", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.103" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" -dependencies = [ - "cfg-if", - "futures-util", - "wasm-bindgen", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "libtest-mimic" -version = "0.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "14e6ba06f0ade6e504aff834d7c34298e5155c6baca353cc6a4aaff2f9fd7f33" -dependencies = [ - "anstream", - "anstyle", - "clap", - "escape8259", -] - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "normalize-line-endings" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61807f77802ff30975e01f4f071c8ba10c022052f98b3294119f3e615d13e5be" - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "portable-atomic" -version = "1.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "predicates" -version = "3.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ada8f2932f28a27ee7b70dd6c1c39ea0675c55a36879ab92f3a715eaa1e63cfe" -dependencies = [ - "anstyle", - "difflib", - "float-cmp", - "normalize-line-endings", - "predicates-core", - "regex", -] - -[[package]] -name = "predicates-core" -version = "1.0.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cad38746f3166b4031b1a0d39ad9f954dd291e7854fcc0eed52ee41a0b50d144" - -[[package]] -name = "predicates-tree" -version = "1.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0de1b847b39c8131db0467e9df1ff60e6d0562ab8e9a16e568ad0fdb372e2f2" -dependencies = [ - "predicates-core", - "termtree", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "rustversion" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" - -[[package]] -name = "same-file" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "termtree" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f50febec83f5ee1df3015341d8bd429f2d1cc62bcba7ea2076759d315084683" - -[[package]] -name = "thiserror" -version = "2.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "wait-timeout" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" -dependencies = [ - "libc", -] - -[[package]] -name = "walkdir" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" -dependencies = [ - "same-file", - "winapi-util", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" -dependencies = [ - "cfg-if", - "once_cell", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn 2.0.119", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "winapi-util" -version = "0.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-link", - "windows-result", - "windows-strings", -] - -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/cores/goban/emu/Cargo.toml b/examples/cores/goban/emu/Cargo.toml deleted file mode 100644 index bc229789a..000000000 --- a/examples/cores/goban/emu/Cargo.toml +++ /dev/null @@ -1,67 +0,0 @@ -[package] -name = "bemu-goban" -version = "0.1.0" -edition = "2021" -build = "../../../../bebop/src/nodes/bemu/build.rs" - -[lib] -name = "bebop_bemu" -path = "src/lib.rs" -crate-type = ["rlib", "cdylib"] - -[[bin]] -name = "bebop" -path = "../../../../bebop/src/main.rs" - -[[test]] -name = "test_bemu" -path = "../../../../bebop/tests/test_bemu.rs" -harness = false - -[dependencies] -snafu = "0.8" -once_cell = "1" -toml = "0.8" -bebop-dasm = { path = "../../../../bebop/src/nodes/lib/dasm" } -bebop-bank-hash = { path = "../../../../bebop/src/nodes/lib/bank-hash" } -bebop-bemu-profile = { path = "../../../../bebop/src/nodes/lib/bemu-profile" } -bebop-rushb = { path = "../../../../bebop/src/nodes/lib/rushB" } -bebop-fd-redirect = { path = "../../../../bebop/src/nodes/lib/fd-redirect" } -bebop-rtl-trace = { path = "../../../../bebop/src/nodes/lib/rtl-trace" } -bebop-uart = { path = "../../../../bebop/src/nodes/lib/uart" } -bebop-dtb = { path = "../../../../bebop/src/nodes/lib/dtb" } -bebop-elf = { path = "../../../../bebop/src/nodes/lib/elf" } -bebop-syscall = { path = "../../../../bebop/src/nodes/lib/syscall" } -bebop-verilator = { path = "../../../../bebop/src/nodes/verilator", optional = true } -bebop-p2e = { path = "../../../../bebop/src/nodes/p2e", optional = true } -clap = { version = "4", features = ["derive"] } -libc = "0.2" -log = "0.4" -env_logger = "0.11" -nix = { version = "0.29", features = ["fs", "mman", "signal", "process"] } -ctrlc = "3" -camino = "1.1" -serde = { version = "1", features = ["derive"] } -serde_json = "1" -duct = "0.13" - -[features] -default = ["bemu"] -bemu = [] -verilator = ["dep:bebop-verilator"] -p2e = ["dep:bebop-p2e"] - -[build-dependencies] -cc = { version = "1", features = ["parallel"] } -toml = "0.8" - -[dev-dependencies] -clap = { version = "4", features = ["derive"] } -libtest-mimic = "0.8" -assert_cmd = "2" -walkdir = "2" -predicates = "3" -chrono = { version = "0.4", default-features = false, features = ["clock"] } -serde = { version = "1", features = ["derive"] } -serde_json = "1" -toml = "0.8" diff --git a/examples/cores/goban/emu/src/chip.rs b/examples/cores/goban/emu/src/chip.rs deleted file mode 100644 index ba4d16312..000000000 --- a/examples/cores/goban/emu/src/chip.rs +++ /dev/null @@ -1,38 +0,0 @@ -pub(crate) mod bank { - pub(crate) use crate::bank::*; -} - -use crate::inst::instruction::ExecContext; - -#[path = "../../../../balls/fp2int/emu/src/lib.rs"] -mod fp2int; -#[path = "../../../../balls/int2fp/emu/src/lib.rs"] -mod int2fp; -#[path = "../../../../balls/matrix/emu/src/lib.rs"] -mod matrix; -#[path = "../../../../balls/transpose/emu/src/lib.rs"] -mod transpose; - -pub fn execute_known( - ball_class: &str, - funct: u32, - xs1: u64, - xs2: u64, - ctx: &mut ExecContext, -) -> u64 { - fp2int::execute_known(ball_class, funct, xs1, xs2, ctx) - .or_else(|| int2fp::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| matrix::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| transpose::execute_known(ball_class, funct, xs1, xs2, ctx)) - .unwrap_or_else(|| { - panic!("no goban BEMU ball implementation for ballClass={ball_class} funct7={funct}") - }) -} - -pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> u64 { - fp2int::cycles_after_issue(ball_class, funct, xs1, xs2) - .or_else(|| int2fp::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| matrix::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| transpose::cycles_after_issue(ball_class, funct, xs1, xs2)) - .unwrap_or_else(|| panic!("no goban BEMU ball latency implementation for ballClass={ball_class} funct7={funct}")) -} diff --git a/examples/cores/goban/emu/src/lib.rs b/examples/cores/goban/emu/src/lib.rs deleted file mode 100644 index 6a4f91897..000000000 --- a/examples/cores/goban/emu/src/lib.rs +++ /dev/null @@ -1,9 +0,0 @@ -pub const BEMU_TOP_CONFIG: &str = "../../core.toml"; - -pub fn bemu_top_config() -> &'static str { - BEMU_TOP_CONFIG -} - -mod chip; - -include!("../../../../../bebop/src/nodes/bemu/src/lib.rs"); diff --git a/examples/cores/goban/manifest.toml b/examples/cores/goban/manifest.toml deleted file mode 100644 index 21a759fe6..000000000 --- a/examples/cores/goban/manifest.toml +++ /dev/null @@ -1,5 +0,0 @@ -[core] -name = "goban" -kind = "buckyball" -config = "core.toml" -compiler = "compiler" diff --git a/examples/cores/pebble/compiler/CMakeLists.txt b/examples/cores/pebble/compiler/CMakeLists.txt index 03257c436..2551caa37 100644 --- a/examples/cores/pebble/compiler/CMakeLists.txt +++ b/examples/cores/pebble/compiler/CMakeLists.txt @@ -5,4 +5,8 @@ endif() set(BB_CORE_NAME pebble) set(BB_CORE_TARGET_PREFIX "") +set(BB_CORE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..) +get_filename_component(_BB_REPO_ROOT "${BB_CORE_DIR}/../../.." ABSOLUTE) +include_directories(${_BB_REPO_ROOT}/examples/balls) + add_subdirectory(src) diff --git a/examples/cores/pebble/compiler/src/CMakeLists.txt b/examples/cores/pebble/compiler/src/CMakeLists.txt index 9fa68e269..70d88d569 100644 --- a/examples/cores/pebble/compiler/src/CMakeLists.txt +++ b/examples/cores/pebble/compiler/src/CMakeLists.txt @@ -24,20 +24,36 @@ include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}/Dialect/Buckyball) set(PEBBLE_BALL_COMPILER_DIALECT_DIRS ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Dialect/Buckyball ${BB_REPO_ROOT}/examples/balls/im2col/compiler/src/Dialect/Buckyball ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/matadd/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Dialect/Buckyball ) +set(BB_CORE_BACKEND_INTRINSICS_DIR + ${CMAKE_CURRENT_SOURCE_DIR}/Backend/include + CACHE INTERNAL "Buckyball chip backend intrinsic directory") + +set(BB_BALL_BACKEND_INTRINSICS_DIRS) + foreach(PEBBLE_BALL_COMPILER_DIALECT_DIR ${PEBBLE_BALL_COMPILER_DIALECT_DIRS}) if(NOT EXISTS ${PEBBLE_BALL_COMPILER_DIALECT_DIR}) message(FATAL_ERROR "Missing pebble ball compiler dialect dir: ${PEBBLE_BALL_COMPILER_DIALECT_DIR}") endif() include_directories(${PEBBLE_BALL_COMPILER_DIALECT_DIR}) + get_filename_component(PEBBLE_BALL_COMPILER_DIR + ${PEBBLE_BALL_COMPILER_DIALECT_DIR}/../.. + ABSOLUTE) + list(APPEND BB_BALL_BACKEND_INTRINSICS_DIRS + ${PEBBLE_BALL_COMPILER_DIR}/Backend/include) endforeach() +set(BB_BALL_BACKEND_INTRINSICS_DIRS ${BB_BALL_BACKEND_INTRINSICS_DIRS} + CACHE INTERNAL "Buckyball ball backend intrinsic directories") + add_subdirectory(Dialect) add_subdirectory(Conversion) add_subdirectory(Target) diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp index 848cf6c06..b9d83cce1 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp @@ -12,16 +12,20 @@ using namespace mlir; namespace mlir::buddy { -void populateTransposeAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateMatrixAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateIm2colAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateFp2IntAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateInt2FpAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); +void populateTransposeBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, PhysicalBankState &state); +void populateSMatMulBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateIm2colBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateFp2IntBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateInt2FpBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateMatAddBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateReluBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); } // namespace mlir::buddy namespace { @@ -58,11 +62,14 @@ class AssignPhysicalBanksPass mlir::buddy::PhysicalBankState state(bankNum); RewritePatternSet patterns(&getContext()); mlir::buddy::addBaseAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateTransposeAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateMatrixAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateIm2colAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateFp2IntAssignPhysicalBankPatterns(patterns, state); - mlir::buddy::populateInt2FpAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateTransposeBallAssignPhysicalBankPatterns(patterns, + state); + mlir::buddy::populateSMatMulBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateIm2colBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateFp2IntBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateInt2FpBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateMatAddBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateReluBallAssignPhysicalBankPatterns(patterns, state); walkAndApplyPatterns(func, std::move(patterns)); if (failed(mlir::buddy::verifyNoBankSSAOps(func))) { diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt index ddd2bf4bf..186fc5b4d 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt @@ -3,14 +3,18 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerBuckyballPass ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/PhysicalBankState.cpp ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/ReportBankUsagePass.cpp ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp ${BB_REPO_ROOT}/examples/balls/im2col/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/matadd/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp + CoreBankSSALowering.cpp AssignPhysicalBanksPass.cpp LowerBuckyballPass.cpp LowerBuckyballToBankSSAPass.cpp - MatrixMatmulToBankSSAPatterns.cpp Im2colMatmulToBankSSAPatterns.cpp MemTransposeToBankSSAPatterns.cpp diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CoreBankSSALowering.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CoreBankSSALowering.cpp new file mode 100644 index 000000000..0f7cc09b2 --- /dev/null +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/CoreBankSSALowering.cpp @@ -0,0 +1,19 @@ +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +using namespace mlir; + +namespace mlir::buddy { +void populateSMatMulBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns); +void populateReluBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns); +void populatePebbleIm2colMatmulToBankSSAPatterns(RewritePatternSet &patterns); +void populatePebbleMemTransposeToBankSSAPatterns(RewritePatternSet &patterns); + +void populatePebbleCoreBankSSALoweringPatterns(RewritePatternSet &patterns) { + populateSMatMulBallLowerBuckyballToBankSSAPatterns(patterns); + populateReluBallLowerBuckyballToBankSSAPatterns(patterns); + populatePebbleIm2colMatmulToBankSSAPatterns(patterns); + populatePebbleMemTransposeToBankSSAPatterns(patterns); +} +} // namespace mlir::buddy diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/Im2colMatmulToBankSSAPatterns.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/Im2colMatmulToBankSSAPatterns.cpp index d44c4dcd2..f75996214 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/Im2colMatmulToBankSSAPatterns.cpp +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/Im2colMatmulToBankSSAPatterns.cpp @@ -5,12 +5,12 @@ #include "mlir/Dialect/Arith/IR/Arith.h" #include "mlir/Dialect/Linalg/IR/Linalg.h" #include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" #include "mlir/IR/BuiltinTypes.h" #include "mlir/IR/PatternMatch.h" #include "Buckyball/BuckyballOps.h" #include "Utils/BankUtils.h" -#include "Utils/QuantUtils.h" using namespace mlir; using namespace ::buddy::buckyball; @@ -18,9 +18,25 @@ using namespace ::buddy::buckyball; namespace { constexpr int64_t kLane = 16; +constexpr int64_t kBankDepth = 1024; +constexpr int64_t kMmioBytes = 5 * 1024; static int64_t cdiv(int64_t a, int64_t b) { return (a + b - 1) / b; } +static LogicalResult rowStrideDivLane(MemRefType ty, int64_t &out) { + SmallVector strides; + int64_t offset = 0; + if (failed(ty.getStridesAndOffset(strides, offset)) || strides.size() < 2) + return failure(); + if (ShapedType::isDynamic(strides[0]) || strides[0] <= 0 || + strides[0] % kLane != 0) + return failure(); + if (ShapedType::isDynamic(strides[1]) || strides[1] != 1) + return failure(); + out = strides[0] / kLane; + return success(); +} + class Im2colMatmulToBankSSAPattern : public OpRewritePattern { public: using OpRewritePattern::OpRewritePattern; @@ -34,96 +50,570 @@ class Im2colMatmulToBankSSAPattern : public OpRewritePattern { if (!inTy || !fTy || !oTy || !inTy.hasStaticShape() || !fTy.hasStaticShape() || !oTy.hasStaticShape()) return b.notifyMatchFailure(op, "requires static memrefs"); - if (!inTy.getElementType().isF32() || !fTy.getElementType().isF32() || + if (!inTy.getElementType().isF32() || !fTy.getElementType().isInteger(8) || !oTy.getElementType().isF32()) - return b.notifyMatchFailure(op, "pebble bank-ssa path is f32 only"); + return b.notifyMatchFailure( + op, + "requires FP32 activations, offline INT8 weights, and FP32 output"); + int64_t dwAddrBase = op.getDwAddr(); + int64_t dwBytes = op.getDwBytes(); + bool perChannel = op.getPerChannel(); + if (dwAddrBase < 16 || dwAddrBase % 4) + return op.emitError( + "requires a 4-byte-aligned Dw MMIO byte address from RAX metadata"); int64_t inSize = op.getInSize(); int64_t ksize = op.getKsize(); int64_t n = op.getN(); int64_t stride = op.getStride(); int64_t padding = op.getPadding(); - if (stride != 1 || padding != 0) - return op.emitError("only stride=1 padding=0 supported"); - if (ksize < 1 || n < 1 || n > kLane) + int64_t startRow = op.getStartRow(); + int64_t startCol = op.getStartCol(); + if (stride < 1 || padding < 0 || startRow < 0 || startCol < 0) + return op.emitError("stride/padding/start out of range"); + if (startRow > padding || startCol > padding) + return op.emitError("startRow/Col must be <= padding"); + if (ksize < 1 || n < kLane || n > 4096 || n % kLane != 0) return op.emitError("ksize/n out of range"); + int64_t requiredDwBytes = perChannel ? n * 4 : 4; + if (dwBytes < requiredDwBytes || dwAddrBase + requiredDwBytes > kMmioBytes) + return op.emitError("Dw scale range exceeds Pebble MMIO space"); int64_t kElems = ksize * ksize; - int64_t tile = inSize - ksize + 1; - if (tile < 1) - return op.emitError("inSize/ksize yield empty output tile"); + int64_t padded = inSize + 2 * padding; + if (padded < ksize + startRow || padded < ksize + startCol) + return op.emitError("kernel+start larger than padded input"); + if ((padded - ksize - startRow) % stride != 0 || + (padded - ksize - startCol) % stride != 0) + return op.emitError("inSize/pad/start/stride yield non-integer tile"); + int64_t tileH = (padded - ksize - startRow) / stride + 1; + int64_t tileW = (padded - ksize - startCol) / stride + 1; + if (tileH != tileW || tileH < 1) + return op.emitError( + "im2col_matmul requires square non-empty output tile"); + int64_t tile = tileH; int64_t wins = tile * tile; int64_t inRows = cdiv(inSize * inSize, kLane); - int64_t bRows = cdiv(kElems, kLane) * kLane; + int64_t paddedWins = cdiv(wins, kLane) * kLane; + int64_t paddedK = cdiv(kElems, kLane) * kLane; + int64_t bRows = paddedK; + int64_t aRows = (paddedWins / kLane) * paddedK; + if (aRows > kBankDepth) + return op.emitError("im2col A layout exceeds bank depth"); + if (wins > kBankDepth) + return op.emitError("im2col C rows exceed bank depth"); if (inTy.getShape()[0] != inRows || inTy.getShape()[1] != kLane || - fTy.getShape()[0] != bRows || fTy.getShape()[1] != kLane || - oTy.getShape()[0] != wins || oTy.getShape()[1] != kLane) + fTy.getShape()[0] != bRows || fTy.getShape()[1] != n || + oTy.getShape()[0] != wins || oTy.getShape()[1] != n) return op.emitError("im2col_matmul packed shape mismatch"); - uint64_t cfg = - packBits(wins, 0, 11) | packBits(n, 12, 23) | packBits(kElems, 24, 35); + int64_t strideF = 0; + if (failed(rowStrideDivLane(fTy, strideF))) + return op.emitError("filter needs static strided<[row,1]> row%16==0"); - Value scaleAF = - quantScale(b, loc, absMaxF32(b, loc, op.getInput(), inRows, kLane)); - Value scaleBF = - quantScale(b, loc, absMaxF32(b, loc, op.getFilter(), bRows, kLane)); - Value scaleA = packF32BitsAsI64(b, loc, scaleAF); - Value scaleB = packF32BitsAsI64(b, loc, scaleBF); - Value scaleD = - packF32BitsAsI64(b, loc, dequantScale(b, loc, scaleAF, scaleBF)); - - Value tmpI = b.create( - loc, MemRefType::get({wins, kLane}, b.getI32Type())); + Value daAddr = createI64Const(b, loc, 0); Value inFB = allocBank(b, loc, 1, 4); Value inIB = allocBank(b, loc, 1, 1); Value loaded = mvinBank(b, loc, op.getInput(), inFB, inRows); Value quant = b.create(loc, inIB.getType(), loaded, inIB, - createI64Const(b, loc, inRows), scaleA); + createI64Const(b, loc, inRows), daAddr); releaseBank(b, loc, loaded); Value patches = allocBank(b, loc, 1, 1); Value patch = b.create( loc, patches.getType(), quant, patches, createI64Const(b, loc, inSize), createI64Const(b, loc, ksize), createI64Const(b, loc, stride), - createI64Const(b, loc, padding)); + createI64Const(b, loc, padding), b.getI64IntegerAttr(startRow), + b.getI64IntegerAttr(startCol)); releaseBank(b, loc, quant); - Value fFB = allocBank(b, loc, 1, 4); Value fIB = allocBank(b, loc, 1, 1); - Value fLoaded = mvinBank(b, loc, op.getFilter(), fFB, bRows); - Value fQuant = - b.create(loc, fIB.getType(), fLoaded, fIB, - createI64Const(b, loc, bRows), scaleB); - releaseBank(b, loc, fLoaded); - - Value accB = allocBank(b, loc, 1, 4); - Value computed = - b.create(loc, accB.getType(), patch, fQuant, accB, - createI64Const(b, loc, (int64_t)cfg)); + Value accB = allocBank(b, loc, 1, 2); + Value outB = allocBank(b, loc, 1, 2); + uint64_t cfg = matrixRs2((uint64_t)paddedWins, 16, (uint64_t)paddedK); + + Value zero = b.create(loc, 0); + Value panelStep = b.create(loc, kLane); + Value panelEnd = b.create(loc, n); + auto panelLoop = b.create(loc, zero, panelEnd, panelStep); + b.setInsertionPointToStart(panelLoop.getBody()); + Value n0 = panelLoop.getInductionVar(); + { + Value bTile = b.create( + loc, op.getFilter(), SmallVector{b.getIndexAttr(0), n0}, + SmallVector{b.getIndexAttr(bRows), + b.getIndexAttr(kLane)}, + SmallVector{b.getIndexAttr(1), b.getIndexAttr(1)}); + Value fLoaded = mvinBank(b, loc, bTile, fIB, bRows, strideF); + Value gemm = + createBankSMatMul(b, loc, accB.getType(), patch, fLoaded, accB, + createI64Const(b, loc, (int64_t)cfg)); + Value dwAddr = createI64Const(b, loc, dwAddrBase); + if (perChannel) { + Value n0I64 = b.create(loc, b.getI64Type(), n0); + dwAddr = b.create( + loc, dwAddr, + b.create(loc, n0I64, createI64Const(b, loc, 4))); + } + Value fp = + perChannel + ? b.create( + loc, outB.getType(), gemm, outB, + createI64Const(b, loc, 2 * paddedWins), daAddr, dwAddr) + .getResult() + : b.create( + loc, outB.getType(), gemm, outB, + createI64Const(b, loc, 2 * paddedWins), daAddr, dwAddr) + .getResult(); + Value packed = b.create( + loc, MemRefType::get({2 * paddedWins, 8}, b.getF32Type())); + mvoutBank(b, loc, packed, fp, 2 * paddedWins); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value two = b.create(loc, 2); + Value eight = b.create(loc, 8); + Value sixteen = b.create(loc, kLane); + Value winsV = b.create(loc, wins); + Value n0V = n0; + auto rowLoop = b.create(loc, zero, winsV, one); + b.setInsertionPointToStart(rowLoop.getBody()); + Value row = rowLoop.getInductionVar(); + auto colLoop = b.create(loc, zero, sixteen, one); + b.setInsertionPointToStart(colLoop.getBody()); + Value col = colLoop.getInductionVar(); + Value packedRow = + b.create(loc, b.create(loc, row, two), + b.create(loc, col, eight)); + Value packedCol = b.create(loc, col, eight); + Value value = b.create(loc, packed, + ValueRange{packedRow, packedCol}); + Value outputCol = b.create(loc, n0V, col); + b.create(loc, value, op.getOutput(), + ValueRange{row, outputCol}); + b.setInsertionPointAfter(rowLoop); + b.create(loc, packed); + } + b.setInsertionPointAfter(panelLoop); + releaseBank(b, loc, patch); - releaseBank(b, loc, fQuant); - mvoutBank(b, loc, tmpI, computed, wins); - releaseBank(b, loc, computed); - - Value tF = allocBank(b, loc, 1, 4); - Value tL = mvinBank(b, loc, tmpI, tF, wins); - Value fp = b.create(loc, tL.getType(), tL, tL, - createI64Const(b, loc, wins), scaleD); - mvoutBank(b, loc, op.getOutput(), fp, wins); - releaseBank(b, loc, fp); - b.create(loc, tmpI); + releaseBank(b, loc, fIB); + releaseBank(b, loc, accB); + releaseBank(b, loc, outB); + b.create(loc); b.eraseOp(op); return success(); } }; +class Im2colFatMatmulToBankSSAPattern + : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(Im2colFatMatmulOp op, + PatternRewriter &b) const override { + Location loc = op.getLoc(); + auto inTy = dyn_cast(op.getInput().getType()); + auto fTy = dyn_cast(op.getFilter().getType()); + auto oTy = dyn_cast(op.getOutput().getType()); + if (!inTy || !fTy || !oTy || !inTy.hasStaticShape() || + !fTy.hasStaticShape() || !oTy.hasStaticShape()) + return b.notifyMatchFailure(op, "requires static memrefs"); + if (!inTy.getElementType().isF32() || !fTy.getElementType().isInteger(8) || + !oTy.getElementType().isF32()) + return b.notifyMatchFailure( + op, + "requires FP32 activations, offline INT8 weights, and FP32 output"); + int64_t dwAddrBase = op.getDwAddr(); + int64_t dwBytes = op.getDwBytes(); + bool perChannel = op.getPerChannel(); + if (dwAddrBase < 16 || dwAddrBase % 4) + return op.emitError( + "requires a 4-byte-aligned Dw MMIO byte address from RAX metadata"); + + int64_t inSize = op.getInSize(); + int64_t ksize = op.getKsize(); + int64_t n = op.getN(); + int64_t nCin = op.getNCin(); + int64_t stride = op.getStride(); + int64_t padding = op.getPadding(); + int64_t startRow = op.getStartRow(); + int64_t startCol = op.getStartCol(); + if (stride < 1 || padding < 0 || startRow < 0 || startCol < 0) + return op.emitError("stride/padding/start out of range"); + if (startRow > padding || startCol > padding) + return op.emitError("startRow/Col must be <= padding"); + if (ksize < 1 || n < kLane || n > 4096 || n % kLane != 0) + return op.emitError("ksize/n out of range"); + int64_t requiredDwBytes = perChannel ? n * 4 : 4; + if (dwBytes < requiredDwBytes || dwAddrBase + requiredDwBytes > kMmioBytes) + return op.emitError("Dw scale range exceeds Pebble MMIO space"); + if (nCin < 1 || nCin > 256) + return op.emitError("nCin out of range"); + int64_t kElems = ksize * ksize; + int64_t padded = inSize + 2 * padding; + if (padded < ksize + startRow || padded < ksize + startCol) + return op.emitError("kernel+start larger than padded input"); + if ((padded - ksize - startRow) % stride != 0 || + (padded - ksize - startCol) % stride != 0) + return op.emitError("inSize/pad/start/stride yield non-integer tile"); + int64_t tileH = (padded - ksize - startRow) / stride + 1; + int64_t tileW = (padded - ksize - startCol) / stride + 1; + if (tileH != tileW || tileH < 1) + return op.emitError("im2col_fat_matmul requires square output tile"); + int64_t wins = tileH * tileW; + int64_t inRows = cdiv(inSize * inSize, kLane); + int64_t paddedWins = cdiv(wins, kLane) * kLane; + int64_t paddedK = cdiv(kElems, kLane) * kLane; + int64_t bRows = paddedK; + int64_t aRowsSingle = (paddedWins / kLane) * paddedK; + if (aRowsSingle > kBankDepth) + return op.emitError("im2col A layout exceeds bank depth"); + if (wins > kBankDepth) + return op.emitError("im2col C rows exceed bank depth"); + + if (inTy.getShape()[0] != nCin * inRows || inTy.getShape()[1] != kLane || + fTy.getShape()[0] != nCin * bRows || fTy.getShape()[1] != n || + oTy.getShape()[0] != wins || oTy.getShape()[1] != n) + return op.emitError("im2col_fat_matmul packed shape mismatch"); + + int64_t strideF = 0; + if (failed(rowStrideDivLane(fTy, strideF))) + return op.emitError("filter needs static strided<[row,1]> row%16==0"); + + Value daAddr = createI64Const(b, loc, 0); + + Value f0 = b.create(loc, b.getF32Type(), + b.getF32FloatAttr(0.0f)); + Value partial = b.create( + loc, MemRefType::get({wins, n}, b.getF32Type())); + + Value inFB = allocBank(b, loc, 1, 4); + Value inIB = allocBank(b, loc, 1, 1); + Value patches = allocBank(b, loc, 1, 1); + Value fIB = allocBank(b, loc, 1, 1); + Value accB = allocBank(b, loc, 1, 2); + Value outB = allocBank(b, loc, 1, 2); + + b.create(loc, f0, op.getOutput()); + uint64_t cfg = matrixRs2((uint64_t)paddedWins, 16, (uint64_t)paddedK); + Value channelZero = b.create(loc, 0); + Value channelEnd = b.create(loc, nCin); + Value channelStep = b.create(loc, 1); + auto channelLoop = + b.create(loc, channelZero, channelEnd, channelStep); + b.setInsertionPointToStart(channelLoop.getBody()); + Value channel = channelLoop.getInductionVar(); + { + Value inputOffset = b.create( + loc, channel, b.create(loc, inRows)); + Value plane = b.create( + loc, op.getInput(), + ArrayRef{inputOffset, b.getIndexAttr(0)}, + ArrayRef{b.getIndexAttr(inRows), b.getIndexAttr(kLane)}, + ArrayRef{b.getIndexAttr(1), b.getIndexAttr(1)}); + + Value loaded = mvinBank(b, loc, plane, inFB, inRows); + Value quant = + b.create(loc, inIB.getType(), loaded, inIB, + createI64Const(b, loc, inRows), daAddr); + Value patch = b.create( + loc, patches.getType(), quant, patches, + createI64Const(b, loc, inSize), createI64Const(b, loc, ksize), + createI64Const(b, loc, stride), createI64Const(b, loc, padding), + b.getI64IntegerAttr(startRow), b.getI64IntegerAttr(startCol)); + + Value panelZero = b.create(loc, 0); + Value panelStep = b.create(loc, kLane); + Value panelEnd = b.create(loc, n); + auto panelLoop = + b.create(loc, panelZero, panelEnd, panelStep); + b.setInsertionPointToStart(panelLoop.getBody()); + Value n0 = panelLoop.getInductionVar(); + { + Value filterOffset = b.create( + loc, channel, b.create(loc, bRows)); + Value bTile = b.create( + loc, op.getFilter(), SmallVector{filterOffset, n0}, + SmallVector{b.getIndexAttr(bRows), + b.getIndexAttr(kLane)}, + SmallVector{b.getIndexAttr(1), b.getIndexAttr(1)}); + Value fLoaded = mvinBank(b, loc, bTile, fIB, bRows, strideF); + Value gemm = + createBankSMatMul(b, loc, accB.getType(), patch, fLoaded, accB, + createI64Const(b, loc, (int64_t)cfg)); + Value dwAddr = createI64Const(b, loc, dwAddrBase); + if (perChannel) { + Value n0I64 = b.create(loc, b.getI64Type(), n0); + dwAddr = b.create( + loc, dwAddr, + b.create(loc, n0I64, createI64Const(b, loc, 4))); + } + Value fp = + perChannel + ? b.create( + loc, outB.getType(), gemm, outB, + createI64Const(b, loc, 2 * paddedWins), daAddr, dwAddr) + .getResult() + : b.create( + loc, outB.getType(), gemm, outB, + createI64Const(b, loc, 2 * paddedWins), daAddr, dwAddr) + .getResult(); + Value packed = b.create( + loc, MemRefType::get({2 * paddedWins, 8}, b.getF32Type())); + mvoutBank(b, loc, packed, fp, 2 * paddedWins); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value two = b.create(loc, 2); + Value eight = b.create(loc, 8); + Value winsV = b.create(loc, wins); + auto rowLoop = b.create(loc, zero, winsV, one); + b.setInsertionPointToStart(rowLoop.getBody()); + Value row = rowLoop.getInductionVar(); + auto colLoop = b.create(loc, zero, panelStep, one); + b.setInsertionPointToStart(colLoop.getBody()); + Value col = colLoop.getInductionVar(); + Value packedRow = + b.create(loc, b.create(loc, row, two), + b.create(loc, col, eight)); + Value packedCol = b.create(loc, col, eight); + Value value = b.create( + loc, packed, ValueRange{packedRow, packedCol}); + Value outputCol = b.create(loc, n0, col); + b.create(loc, value, partial, + ValueRange{row, outputCol}); + b.setInsertionPointAfter(rowLoop); + b.create(loc, packed); + } + b.setInsertionPointAfter(panelLoop); + + b.create(loc); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value winsV = b.create(loc, wins); + Value nV = b.create(loc, n); + auto rowLoop = b.create(loc, zero, winsV, one); + b.setInsertionPointToStart(rowLoop.getBody()); + auto colLoop = b.create(loc, zero, nV, one); + b.setInsertionPointToStart(colLoop.getBody()); + Value row = rowLoop.getInductionVar(); + Value col = colLoop.getInductionVar(); + Value accumulated = + b.create(loc, op.getOutput(), ValueRange{row, col}); + Value addend = + b.create(loc, partial, ValueRange{row, col}); + b.create( + loc, b.create(loc, accumulated, addend), + op.getOutput(), ValueRange{row, col}); + b.setInsertionPointAfter(rowLoop); + } + b.setInsertionPointAfter(channelLoop); + + releaseBank(b, loc, inFB); + releaseBank(b, loc, inIB); + releaseBank(b, loc, patches); + releaseBank(b, loc, fIB); + releaseBank(b, loc, accB); + releaseBank(b, loc, outB); + b.create(loc); + + b.create(loc, partial); + b.eraseOp(op); + return success(); + } +}; + +class Im2colDepthwiseMatmulToBankSSAPattern + : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(Im2colDepthwiseMatmulOp op, + PatternRewriter &b) const override { + Location loc = op.getLoc(); + auto inTy = dyn_cast(op.getInput().getType()); + auto fTy = dyn_cast(op.getFilter().getType()); + auto oTy = dyn_cast(op.getOutput().getType()); + if (!inTy || !fTy || !oTy || !inTy.hasStaticShape() || + !fTy.hasStaticShape() || !oTy.hasStaticShape()) + return b.notifyMatchFailure(op, "requires static memrefs"); + if (!inTy.getElementType().isF32() || !fTy.getElementType().isInteger(8) || + !oTy.getElementType().isF32()) + return b.notifyMatchFailure( + op, + "requires FP32 activations, offline INT8 weights, and FP32 output"); + int64_t dwAddrBase = op.getDwAddr(); + int64_t dwBytes = op.getDwBytes(); + bool perChannel = op.getPerChannel(); + if (dwAddrBase < 16 || dwAddrBase % 4) + return op.emitError( + "requires a 4-byte-aligned Dw MMIO byte address from RAX metadata"); + + int64_t inSize = op.getInSize(); + int64_t ksize = op.getKsize(); + int64_t n = op.getN(); + int64_t stride = op.getStride(); + int64_t padding = op.getPadding(); + int64_t startRow = op.getStartRow(); + int64_t startCol = op.getStartCol(); + if (stride < 1 || padding < 0 || startRow < 0 || startCol < 0) + return op.emitError("stride/padding/start out of range"); + if (startRow > padding || startCol > padding) + return op.emitError("startRow/Col must be <= padding"); + if (ksize < 1 || n < 1 || n > kLane) + return op.emitError("ksize/n out of range"); + int64_t requiredDwBytes = perChannel ? n * 4 : 4; + if (dwBytes < requiredDwBytes || dwAddrBase + requiredDwBytes > kMmioBytes) + return op.emitError("Dw scale range exceeds Pebble MMIO space"); + + int64_t kElems = ksize * ksize; + int64_t padded = inSize + 2 * padding; + if (padded < ksize + startRow || padded < ksize + startCol) + return op.emitError("kernel+start larger than padded input"); + if ((padded - ksize - startRow) % stride != 0 || + (padded - ksize - startCol) % stride != 0) + return op.emitError("inSize/pad/start/stride yield non-integer tile"); + int64_t tileH = (padded - ksize - startRow) / stride + 1; + int64_t tileW = (padded - ksize - startCol) / stride + 1; + if (tileH != tileW || tileH < 1) + return op.emitError("requires square non-empty output tile"); + int64_t tile = tileH; + int64_t wins = tile * tile; + int64_t inRows = cdiv(inSize * inSize, kLane); + int64_t paddedWins = cdiv(wins, kLane) * kLane; + int64_t paddedK = cdiv(kElems, kLane) * kLane; + int64_t bRows = paddedK; + if (wins > kBankDepth) + return op.emitError("im2col_depthwise_matmul output exceeds bank depth"); + + if (inTy.getShape()[0] != n * inRows || inTy.getShape()[1] != kLane || + fTy.getShape()[0] != n * bRows || fTy.getShape()[1] != kLane || + oTy.getShape()[0] != wins || oTy.getShape()[1] != kLane) + return op.emitError("im2col_depthwise_matmul packed shape mismatch"); + + uint64_t cfg = matrixRs2((uint64_t)paddedWins, 16, (uint64_t)paddedK); + + Value daAddr = createI64Const(b, loc, 0); + + Value f0 = + b.create(loc, b.getI8Type(), b.getI8IntegerAttr(0)); + Value fOne = b.create( + loc, MemRefType::get({bRows, kLane}, b.getI8Type())); + Value tmpOut = b.create( + loc, MemRefType::get({2 * paddedWins, 8}, b.getF32Type())); + + Value inFB = allocBank(b, loc, 1, 4); + Value inIB = allocBank(b, loc, 1, 1); + Value patches = allocBank(b, loc, 1, 1); + Value fIB = allocBank(b, loc, 1, 1); + Value accB = allocBank(b, loc, 1, 2); + Value outB = allocBank(b, loc, 1, 2); + + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value winsV = b.create(loc, wins); + + Value channelEnd = b.create(loc, n); + auto channelLoop = b.create(loc, zero, channelEnd, one); + b.setInsertionPointToStart(channelLoop.getBody()); + Value channel = channelLoop.getInductionVar(); + { + Value inputOffset = b.create( + loc, channel, b.create(loc, inRows)); + Value plane = b.create( + loc, op.getInput(), + ArrayRef{inputOffset, b.getIndexAttr(0)}, + ArrayRef{b.getIndexAttr(inRows), b.getIndexAttr(kLane)}, + ArrayRef{b.getIndexAttr(1), b.getIndexAttr(1)}); + + b.create(loc, f0, fOne); + for (int64_t k = 0; k < kElems; ++k) { + Value rowV = b.create(loc, k); + Value srcRow = b.create( + loc, + b.create( + loc, channel, b.create(loc, bRows)), + rowV); + Value wt = b.create(loc, op.getFilter(), + ValueRange{srcRow, channel}); + b.create(loc, wt, fOne, ValueRange{rowV, zero}); + } + + Value loaded = mvinBank(b, loc, plane, inFB, inRows); + Value quant = + b.create(loc, inIB.getType(), loaded, inIB, + createI64Const(b, loc, inRows), daAddr); + Value patch = b.create( + loc, patches.getType(), quant, patches, + createI64Const(b, loc, inSize), createI64Const(b, loc, ksize), + createI64Const(b, loc, stride), createI64Const(b, loc, padding), + b.getI64IntegerAttr(startRow), b.getI64IntegerAttr(startCol)); + + Value fLoaded = mvinBank(b, loc, fOne, fIB, bRows); + Value computed = + createBankSMatMul(b, loc, accB.getType(), patch, fLoaded, accB, + createI64Const(b, loc, (int64_t)cfg)); + Value dwAddr = createI64Const(b, loc, dwAddrBase); + if (perChannel) { + Value channelI64 = + b.create(loc, b.getI64Type(), channel); + dwAddr = b.create( + loc, dwAddr, + b.create(loc, channelI64, + createI64Const(b, loc, 4))); + } + Value fp = + perChannel + ? b.create( + loc, outB.getType(), computed, outB, + createI64Const(b, loc, 2 * paddedWins), daAddr, dwAddr) + .getResult() + : b.create( + loc, outB.getType(), computed, outB, + createI64Const(b, loc, 2 * paddedWins), daAddr, dwAddr) + .getResult(); + mvoutBank(b, loc, tmpOut, fp, 2 * paddedWins); + b.create(loc); + + auto rL = b.create(loc, zero, winsV, one); + b.setInsertionPointToStart(rL.getBody()); + Value r = rL.getInductionVar(); + Value two = b.create(loc, 2); + Value packedRow = b.create(loc, r, two); + Value v = + b.create(loc, tmpOut, ValueRange{packedRow, zero}); + b.create(loc, v, op.getOutput(), ValueRange{r, channel}); + b.setInsertionPointAfter(rL); + } + b.setInsertionPointAfter(channelLoop); + + releaseBank(b, loc, inFB); + releaseBank(b, loc, inIB); + releaseBank(b, loc, patches); + releaseBank(b, loc, fIB); + releaseBank(b, loc, accB); + releaseBank(b, loc, outB); + b.create(loc); + + b.create(loc, fOne); + b.create(loc, tmpOut); + b.eraseOp(op); + return success(); + } +}; + } // namespace -void mlir::buddy::populatePebbleIm2colMatmulToBankSSAPatterns( - RewritePatternSet &patterns) { +namespace mlir::buddy { + +void populatePebbleIm2colMatmulToBankSSAPatterns(RewritePatternSet &patterns) { patterns.add(patterns.getContext()); + patterns.add(patterns.getContext()); + patterns.add(patterns.getContext()); } + +} // namespace mlir::buddy diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp index d084972cb..937e08225 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp @@ -151,6 +151,9 @@ class LowerBuckyballIntrinsicsToRushBPass public: MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID( LowerBuckyballIntrinsicsToRushBPass) + LowerBuckyballIntrinsicsToRushBPass() = default; + LowerBuckyballIntrinsicsToRushBPass( + const LowerBuckyballIntrinsicsToRushBPass &) {} StringRef getArgument() const final { return "lower-buckyball-intrinsics-to-rushb"; @@ -159,6 +162,11 @@ class LowerBuckyballIntrinsicsToRushBPass return "Lower Buckyball intrinsic ops to the rushB host ABI."; } + Option coreId{ + *this, "core_id", + llvm::cl::desc("Bind generated rushB calls to a tile Core."), + llvm::cl::init(-1)}; + void getDependentDialects(DialectRegistry ®istry) const override { registry.insert(); } @@ -175,10 +183,57 @@ class LowerBuckyballIntrinsicsToRushBPass OpBuilder builder(&getContext()); Type i32Type = IntegerType::get(&getContext(), 32); Type voidType = LLVM::LLVMVoidType::get(&getContext()); + FlatSymbolRefAttr selectCallee; + if (coreId >= 0) { + auto selectType = + LLVM::LLVMFunctionType::get(voidType, {i32Type, i32Type}); + selectCallee = getOrInsertRushBFunction( + builder, module, "rushb_select_accelerator", selectType); + } + llvm::SmallPtrSet boundFunctions; + for (Operation *op : intrinsicOps) { + if (coreId >= 0) { + if (auto function = op->getParentOfType(); + function && boundFunctions.insert(function).second) { + OpBuilder::InsertionGuard guard(builder); + builder.setInsertionPointToStart(&function.front()); + auto selected = LLVM::ConstantOp::create( + builder, function.getLoc(), i32Type, + builder.getI32IntegerAttr(static_cast(coreId))); + auto chip = + LLVM::ConstantOp::create(builder, function.getLoc(), i32Type, + builder.getI32IntegerAttr(0)); + LLVM::CallOp::create(builder, function.getLoc(), TypeRange{}, + selectCallee, ValueRange{selected, chip}); + } + } + builder.setInsertionPoint(op); + SmallVector operands; + StringRef name; + if (isa(op)) { + name = "rushb_mset"; + operands.append(op->getOperands().begin(), op->getOperands().end()); + } else if (isa(op)) { + name = isa(op) ? "rushb_mvin" : "rushb_mvout"; + operands.append(op->getOperands().begin(), op->getOperands().end()); + } else { + name = "rushb_custom"; + if (auto custom = dyn_cast(op)) { + operands.append(custom.getOperands().begin(), + custom.getOperands().end()); + operands.push_back(LLVM::ConstantOp::create( + builder, op->getLoc(), i32Type, + builder.getI32IntegerAttr(custom.getFunct7()))); + } else { + auto fence = cast(op); + operands.append(fence->getOperands().begin(), + fence->getOperands().end()); + operands.push_back(LLVM::ConstantOp::create( + builder, op->getLoc(), i32Type, builder.getI32IntegerAttr(0))); + } + } - auto call = [&](Operation *op, StringRef name, ValueRange operands) { SmallVector argumentTypes; - argumentTypes.reserve(operands.size()); for (Value operand : operands) argumentTypes.push_back(operand.getType()); auto type = LLVM::LLVMFunctionType::get(voidType, argumentTypes); @@ -186,34 +241,6 @@ class LowerBuckyballIntrinsicsToRushBPass LLVM::CallOp::create(builder, op->getLoc(), TypeRange{}, callee, operands); op->erase(); - }; - - for (Operation *op : intrinsicOps) { - builder.setInsertionPoint(op); - if (isa(op)) { - call(op, "rushb_mset", op->getOperands()); - continue; - } - if (isa(op)) { - call(op, isa(op) ? "rushb_mvin" : "rushb_mvout", - op->getOperands()); - continue; - } - if (auto custom = dyn_cast(op)) { - Value funct7 = LLVM::ConstantOp::create( - builder, op->getLoc(), i32Type, - builder.getI32IntegerAttr(custom.getFunct7())); - SmallVector operands{custom.getRs1(), custom.getRs2(), funct7}; - call(op, "rushb_custom", operands); - continue; - } - if (auto fence = dyn_cast(op)) { - Value funct7 = LLVM::ConstantOp::create(builder, op->getLoc(), i32Type, - builder.getI32IntegerAttr(0)); - SmallVector operands{fence->getOperand(0), fence->getOperand(1), - funct7}; - call(op, "rushb_custom", operands); - } } } }; diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp index 7947e92e6..49401adb8 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp @@ -15,6 +15,10 @@ using namespace mlir; +namespace mlir::buddy { +void populatePebbleCoreBankSSALoweringPatterns(RewritePatternSet &patterns); +} + namespace { class LowerBuckyballToBankSSAPass @@ -36,7 +40,7 @@ class LowerBuckyballToBankSSAPass void runOnOperation() override { RewritePatternSet patterns(&getContext()); - mlir::buddy::populatePebbleLowerBuckyballToBankSSAPatterns(patterns); + mlir::buddy::populatePebbleCoreBankSSALoweringPatterns(patterns); if (failed(applyPatternsGreedily(getOperation(), std::move(patterns)))) signalPassFailure(); } diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MatrixMatmulToBankSSAPatterns.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MatrixMatmulToBankSSAPatterns.cpp deleted file mode 100644 index 83911f36e..000000000 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MatrixMatmulToBankSSAPatterns.cpp +++ /dev/null @@ -1,64 +0,0 @@ -//===- MatrixMatmulToBankSSAPatterns.cpp - float matmul -------------------===// - -#include "Conversion/LowerBuckyball/LowerBuckyball.h" - -#include "mlir/Dialect/Linalg/IR/Linalg.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/IR/BuiltinTypes.h" -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" - -using namespace mlir; -using namespace ::buddy::buckyball; - -namespace { - -// Float matrix_matmul: keep IR compact with linalg.matmul (accumulates into C). -// Full bank-SSA quantize expansion blows up LLM-sized graphs and crashes -// convert-scf-to-cf (MLIR IRRewrite assert on huge modules). -class MatrixMatmulToBankSSAPattern : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(MatrixMatmulOp op, - PatternRewriter &b) const override { - Location loc = op.getLoc(); - auto aTy = dyn_cast(op.getAMemArray().getType()); - auto bTy = dyn_cast(op.getBMemArray().getType()); - auto cTy = dyn_cast(op.getCMemArray().getType()); - if (!aTy || !bTy || !cTy || !aTy.hasStaticShape() || - !bTy.hasStaticShape() || !cTy.hasStaticShape()) - return b.notifyMatchFailure(op, "requires static memrefs"); - Type elem = aTy.getElementType(); - if (!isa(elem) || bTy.getElementType() != elem || - cTy.getElementType() != elem) - return b.notifyMatchFailure(op, - "pebble bank-ssa compact path is float only"); - - int64_t M = aTy.getShape()[0], K = aTy.getShape()[1]; - int64_t Kb = bTy.getShape()[0], N = bTy.getShape()[1]; - if (K != Kb || cTy.getShape()[0] != M || cTy.getShape()[1] != N) - return op.emitError("matmul shape mismatch"); - - b.create(loc, - ValueRange{op.getAMemArray(), op.getBMemArray()}, - ValueRange{op.getCMemArray()}); - b.eraseOp(op); - return success(); - } -}; - -} // namespace - -void mlir::buddy::populatePebbleMatrixMatmulToBankSSAPatterns( - RewritePatternSet &patterns) { - patterns.add(patterns.getContext()); -} - -void mlir::buddy::populatePebbleLowerBuckyballToBankSSAPatterns( - RewritePatternSet &patterns) { - populatePebbleMatrixMatmulToBankSSAPatterns(patterns); - populatePebbleIm2colMatmulToBankSSAPatterns(patterns); - populatePebbleMemTransposeToBankSSAPatterns(patterns); -} diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp index 20f20e7bc..9147a4f08 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp +++ b/examples/cores/pebble/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp @@ -1,28 +1,45 @@ -//===- MemTransposeToBankSSAPatterns.cpp - mem_transpose -> Bank*/linalg --===// +//===- MemTransposeToBankSSAPatterns.cpp - mem_transpose -> Bank* ---------===// +// +// Bank-unit transpose (same model as transpose ctests): +// pad to bank-row width -> contiguous mvin(rows,stride=1) +// -> transpose(rows) -> mvout(rows,stride=1) +// Dest bank holds transposed data as stacked bank rows of width +// W = groups * (bankWidthBytes / elemBytes). +// That "16" for i8 is elems-per-bank-row from bank width, not a 16x16 tile. +// +// Packing: iter is always the mvin row count. For fat [R,C] with R<=maxW, +// pack long dim C as iter (up to bankDepth) so we do not emit iter=R tiny ops. +// +//===----------------------------------------------------------------------===// #include "Conversion/LowerBuckyball/LowerBuckyball.h" #include "mlir/Dialect/Arith/IR/Arith.h" #include "mlir/Dialect/Linalg/IR/Linalg.h" #include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Dialect/Utils/StructuredOpsUtils.h" -#include "mlir/IR/AffineMap.h" +#include "mlir/Dialect/SCF/IR/SCF.h" #include "mlir/IR/BuiltinTypes.h" #include "mlir/IR/PatternMatch.h" #include "Buckyball/BuckyballOps.h" #include "Utils/BankUtils.h" +#include + using namespace mlir; using namespace ::buddy::buckyball; namespace { constexpr int64_t kBankWidthBytes = 16; -constexpr int64_t kRowTile = 16; -// Above this, bank-SSA tile expansion blows up (Bert 768x768 → -// convert-scf-to-cf crash). Use compact linalg.generic instead. -constexpr int64_t kMaxBankTransposeElems = 64 * 64; +constexpr int64_t kBankDepth = 1024; +constexpr int64_t kMaxGroups = 4; + +static int64_t alignUp(int64_t x, int64_t a) { + if (a <= 0) + return x; + return (x + a - 1) / a * a; +} static size_t elemsPerBankRow(Type elemType) { unsigned bitWidth = elemType.getIntOrFloatBitWidth(); @@ -31,35 +48,74 @@ static size_t elemsPerBankRow(Type elemType) { return kBankWidthBytes / (bitWidth / 8); } -static LogicalResult rowStrideDiv16(MemRefType ty, int64_t &out) { - SmallVector strides; - int64_t offset = 0; - if (failed(ty.getStridesAndOffset(strides, offset)) || strides.size() < 2) - return failure(); - if (ShapedType::isDynamic(strides[0]) || strides[0] <= 0 || - strides[0] % 16 != 0) - return failure(); - if (ShapedType::isDynamic(strides[1]) || strides[1] != 1) - return failure(); - out = strides[0] / 16; - return success(); +static Value zeroConst(OpBuilder &b, Location loc, Type elemTy) { + if (auto it = dyn_cast(elemTy)) + return b.create(loc, b.getIntegerAttr(it, 0)); + if (isa(elemTy)) + return b.create(loc, b.getF32FloatAttr(0.0f)); + return {}; } -static int64_t cdiv(int64_t a, int64_t b) { return (a + b - 1) / b; } - -static void emitLinalgTranspose(PatternRewriter &rewriter, Location loc, - Value input, Value output) { - MLIRContext *ctx = rewriter.getContext(); - AffineMap id = AffineMap::getMultiDimIdentityMap(2, ctx); - AffineMap tr = AffineMap::get( - 2, 0, {rewriter.getAffineDimExpr(1), rewriter.getAffineDimExpr(0)}, ctx); - SmallVector iters(2, utils::IteratorType::parallel); - rewriter.create( - loc, TypeRange{}, ValueRange{input}, ValueRange{output}, - ArrayRef{id, tr}, iters, - [](OpBuilder &b, Location loc, ValueRange args) { - b.create(loc, args[0]); - }); +static void emitBankTranspose(OpBuilder &b, Location loc, Value inContig, + Value outContig, int64_t rows, int64_t groups, + int64_t elemBits) { + // CPU packed inContig before mvin; CPU reads outContig after mvout. + // FPGA RoCC is async — fence both edges (bemu is sync so hid this). + b.create(loc); + Value src = allocBank(b, loc, 1, groups); + Value dst = allocBank(b, loc, 1, groups); + Value loaded = mvinBank(b, loc, inContig, src, rows, /*stride=*/1); + Value transposed = b.create( + loc, dst.getType(), loaded, dst, createI64Const(b, loc, rows), + createI64Const(b, loc, elemBits)); + mvoutBank(b, loc, outContig, transposed, rows, /*stride=*/1); + b.create(loc); + releaseBank(b, loc, loaded); + releaseBank(b, loc, transposed); +} + +// dst[i,j] = src[j, c0+i]; dst:[h,rows] src:[rows,cols] +static void gatherCols(OpBuilder &b, Location loc, Value src, Value dst, + int64_t rows, int64_t c0, int64_t h) { + Value c0v = b.create(loc, c0); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value hUb = b.create(loc, h); + Value rUb = b.create(loc, rows); + auto iLoop = b.create(loc, zero, hUb, one); + { + OpBuilder::InsertionGuard g(b); + b.setInsertionPointToStart(iLoop.getBody()); + Value i = iLoop.getInductionVar(); + Value col = b.create(loc, c0v, i); + auto jLoop = b.create(loc, zero, rUb, one); + b.setInsertionPointToStart(jLoop.getBody()); + Value j = jLoop.getInductionVar(); + Value v = b.create(loc, src, ValueRange{j, col}); + b.create(loc, v, dst, ValueRange{i, j}); + } +} + +// dst[c0+i, j] = src[j, i]; src:[rows,h] dst:[cols,rows] +static void scatterCols(OpBuilder &b, Location loc, Value src, Value dst, + int64_t rows, int64_t c0, int64_t h) { + Value c0v = b.create(loc, c0); + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value hUb = b.create(loc, h); + Value rUb = b.create(loc, rows); + auto iLoop = b.create(loc, zero, hUb, one); + { + OpBuilder::InsertionGuard g(b); + b.setInsertionPointToStart(iLoop.getBody()); + Value i = iLoop.getInductionVar(); + Value row = b.create(loc, c0v, i); + auto jLoop = b.create(loc, zero, rUb, one); + b.setInsertionPointToStart(jLoop.getBody()); + Value j = jLoop.getInductionVar(); + Value v = b.create(loc, src, ValueRange{j, i}); + b.create(loc, v, dst, ValueRange{row, j}); + } } class MemTransposeToBankSSAPattern : public OpRewritePattern { @@ -77,117 +133,123 @@ class MemTransposeToBankSSAPattern : public OpRewritePattern { !outputType.hasStaticShape()) return op.emitError("requires static input and output memrefs"); - auto inShape = inputType.getShape(); - auto outShape = outputType.getShape(); - int64_t rows = inShape[0]; - int64_t cols = inShape[1]; - if (outShape[0] != cols || outShape[1] != rows) + int64_t rows = inputType.getShape()[0]; + int64_t cols = inputType.getShape()[1]; + if (rows <= 0 || cols <= 0) + return op.emitError("transpose requires positive rows/cols"); + if (outputType.getShape()[0] != cols || outputType.getShape()[1] != rows) return op.emitError("output shape must transpose the input shape"); Type elemTy = inputType.getElementType(); int64_t elemBits = elemTy.getIntOrFloatBitWidth(); - if (elemBits == 0 || elemBits % 8 != 0) - return op.emitError("unsupported transpose element type"); - // HW bank transpose only handles i8/f32; bf16 and oversized mats use - // compact linalg (same Bert IR-size escape hatch). - bool hwElem = elemBits == 8 || elemBits == 32; - if (!hwElem || rows * cols > kMaxBankTransposeElems) { - emitLinalgTranspose(rewriter, loc, input, output); - rewriter.eraseOp(op); - return success(); - } + if (elemBits != 8 && elemBits != 32) + return op.emitError("transpose only supports 8-bit or 32-bit elements"); - size_t elemsPerRow = elemsPerBankRow(elemTy); - if (elemsPerRow == 0) + size_t epr = elemsPerBankRow(elemTy); + if (epr == 0) return op.emitError("unsupported transpose element type"); - size_t maxColTile = elemsPerRow * 4; - size_t colsAlign = - ((size_t)cols + elemsPerRow - 1) / elemsPerRow * elemsPerRow; - size_t colTile = colsAlign <= maxColTile ? colsAlign : maxColTile; - if (colTile == 0) - return op.emitError("tile width is smaller than one bank row"); - - int64_t rowsPad = cdiv(rows, kRowTile) * kRowTile; - int64_t colsPad = cdiv(cols, (int64_t)colTile) * (int64_t)colTile; - bool pad = rowsPad != rows || colsPad != cols; - int64_t dummyStride = 0; - bool inContig = succeeded(rowStrideDiv16(inputType, dummyStride)); - bool outContig = succeeded(rowStrideDiv16(outputType, dummyStride)); - bool materialize = pad || !inContig || !outContig; - - Value inBuf = input; - Value outBuf = output; - if (materialize) { - auto inPadTy = MemRefType::get({rowsPad, colsPad}, elemTy); - auto outPadTy = MemRefType::get({colsPad, rowsPad}, elemTy); - inBuf = rewriter.create(loc, inPadTy); - outBuf = rewriter.create(loc, outPadTy); - Value z = rewriter.create( - loc, elemTy, rewriter.getZeroAttr(elemTy)); - rewriter.create(loc, z, inBuf); - rewriter.create(loc, z, outBuf); - Value inView = rewriter.create( - loc, inBuf, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(rows), - rewriter.getIndexAttr(cols)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, input, inView); - } + Value zero = zeroConst(rewriter, loc, elemTy); + if (!zero) + return op.emitError("unsupported transpose element type for pad"); + + int64_t eprI = (int64_t)epr; + int64_t rowsPad = alignUp(rows, eprI); + int64_t colsPad = alignUp(cols, eprI); + int64_t maxW = eprI * kMaxGroups; + + auto inPadTy = MemRefType::get({rowsPad, colsPad}, elemTy); + auto outPadTy = MemRefType::get({colsPad, rowsPad}, elemTy); + Value inPad = rewriter.create(loc, inPadTy); + Value outPad = rewriter.create(loc, outPadTy); + rewriter.create(loc, ValueRange{zero}, ValueRange{inPad}); + rewriter.create(loc, ValueRange{zero}, ValueRange{outPad}); - int64_t groups = (int64_t)(colTile / elemsPerRow); - int64_t strideIn = colsPad / 16; - int64_t strideOut = rowsPad / 16; - for (int64_t r0 = 0; r0 < rowsPad; r0 += kRowTile) { - for (int64_t c0 = 0; c0 < colsPad; c0 += (int64_t)colTile) { - Value inTile = rewriter.create( - loc, inBuf, - SmallVector{rewriter.getIndexAttr(r0), - rewriter.getIndexAttr(c0)}, - SmallVector{rewriter.getIndexAttr(kRowTile), - rewriter.getIndexAttr((int64_t)colTile)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value outTile = rewriter.create( - loc, outBuf, - SmallVector{rewriter.getIndexAttr(c0), - rewriter.getIndexAttr(r0)}, - SmallVector{rewriter.getIndexAttr((int64_t)colTile), - rewriter.getIndexAttr(kRowTile)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - - Value src = allocBank(rewriter, loc, 1, groups); - Value dst = allocBank(rewriter, loc, 1, groups); - Value loaded = mvinBank(rewriter, loc, inTile, src, kRowTile, strideIn); - Value transposed = rewriter.create( - loc, dst.getType(), loaded, dst, - createI64Const(rewriter, loc, kRowTile), - createI64Const(rewriter, loc, elemBits)); - mvoutBank(rewriter, loc, outTile, transposed, (int64_t)colTile, - strideOut); - releaseBank(rewriter, loc, loaded); - releaseBank(rewriter, loc, transposed); + Value inView = rewriter.create( + loc, inPad, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(rows), + rewriter.getIndexAttr(cols)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + rewriter.create(loc, input, inView); + + // Fat [R,C] with R fitting in bank width: pack C as iter (long side). + bool fat = rowsPad <= maxW && colsPad > rowsPad; + if (fat) { + if (rowsPad % eprI != 0) + return op.emitError("fat transpose rowsPad not bank-row aligned"); + int64_t groups = rowsPad / eprI; + for (int64_t c0 = 0; c0 < colsPad;) { + int64_t h = std::min(kBankDepth, colsPad - c0); + auto inContigTy = MemRefType::get({h, rowsPad}, elemTy); + auto outContigTy = MemRefType::get({rowsPad, h}, elemTy); + Value inContig = rewriter.create(loc, inContigTy); + Value outContig = rewriter.create(loc, outContigTy); + gatherCols(rewriter, loc, inPad, inContig, rowsPad, c0, h); + emitBankTranspose(rewriter, loc, inContig, outContig, h, groups, + elemBits); + scatterCols(rewriter, loc, outContig, outPad, rowsPad, c0, h); + rewriter.create(loc, inContig); + rewriter.create(loc, outContig); + c0 += h; } - } + } else { + for (int64_t r0 = 0; r0 < rowsPad;) { + int64_t h = std::min(kBankDepth, rowsPad - r0); + for (int64_t c0 = 0; c0 < colsPad;) { + int64_t w = std::min(maxW, colsPad - c0); + if (w % eprI != 0) + return op.emitError("transpose tile width not bank-row aligned"); + int64_t groups = w / eprI; - if (materialize) { - Value outView = rewriter.create( - loc, outBuf, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(cols), - rewriter.getIndexAttr(rows)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, outView, output); - rewriter.create(loc, inBuf); - rewriter.create(loc, outBuf); + Value inSub = rewriter.create( + loc, inPad, + SmallVector{rewriter.getIndexAttr(r0), + rewriter.getIndexAttr(c0)}, + SmallVector{rewriter.getIndexAttr(h), + rewriter.getIndexAttr(w)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + auto inContigTy = MemRefType::get({h, w}, elemTy); + Value inContig = rewriter.create(loc, inContigTy); + rewriter.create(loc, inSub, inContig); + + auto outContigTy = MemRefType::get({w, h}, elemTy); + Value outContig = rewriter.create(loc, outContigTy); + emitBankTranspose(rewriter, loc, inContig, outContig, h, groups, + elemBits); + + Value outSub = rewriter.create( + loc, outPad, + SmallVector{rewriter.getIndexAttr(c0), + rewriter.getIndexAttr(r0)}, + SmallVector{rewriter.getIndexAttr(w), + rewriter.getIndexAttr(h)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + rewriter.create(loc, outContig, outSub); + rewriter.create(loc, inContig); + rewriter.create(loc, outContig); + c0 += w; + } + r0 += h; + } } + Value outView = rewriter.create( + loc, outPad, + SmallVector{rewriter.getIndexAttr(0), + rewriter.getIndexAttr(0)}, + SmallVector{rewriter.getIndexAttr(cols), + rewriter.getIndexAttr(rows)}, + SmallVector{rewriter.getIndexAttr(1), + rewriter.getIndexAttr(1)}); + rewriter.create(loc, outView, output); + rewriter.create(loc, inPad); + rewriter.create(loc, outPad); + rewriter.create(loc); rewriter.eraseOp(op); return success(); } @@ -195,7 +257,10 @@ class MemTransposeToBankSSAPattern : public OpRewritePattern { } // namespace -void mlir::buddy::populatePebbleMemTransposeToBankSSAPatterns( - RewritePatternSet &patterns) { +namespace mlir::buddy { + +void populatePebbleMemTransposeToBankSSAPatterns(RewritePatternSet &patterns) { patterns.add(patterns.getContext()); } + +} // namespace mlir::buddy diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt b/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt index 61a092d94..4664c50e1 100644 --- a/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt +++ b/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt @@ -1,6 +1,8 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass - LowerTileToBuckyball.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp + CoreTileLowering.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Conversion/LowerTileToBuckyball/TileTransposePatterns.cpp + ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Conversion/LowerTileToBuckyball/TileReluPatterns.cpp LINK_LIBS PUBLIC BuddyTile diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CoreTileLowering.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CoreTileLowering.cpp new file mode 100644 index 000000000..b93befca7 --- /dev/null +++ b/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/CoreTileLowering.cpp @@ -0,0 +1,843 @@ +//===- LowerTileToBuckyball.cpp - Pebble tile->buckyball pass -------------===// +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +//===----------------------------------------------------------------------===// + +#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/DialectConversion.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" +#include "Target/BuckyballTargetRegistry.h" +#include "Tile/TileDialect.h" +#include "Tile/TileOps.h" +#include "Tile/Transform.h" + +#include + +using namespace mlir; +using namespace ::buddy::buckyball; +namespace tile = ::buddy::tile; + +namespace mlir::buddy { +void populateSMatMulBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, + int64_t bankNum); +void populateTransposeBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, + int64_t bankNum); +void populateReluBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, int64_t bankNum); +} // namespace mlir::buddy + +namespace { + +// Pebble im2col: square HxW, per-cin plane, K=k^2. +constexpr int64_t kMaxIter = 34; +constexpr int64_t kMaxK = 7; +constexpr int64_t kBankLines = 1024; +constexpr int64_t kLane = 16; + +static int64_t cdiv(int64_t a, int64_t b) { return (a + b - 1) / b; } + +static int64_t pickTile(int64_t outDim, int64_t ksize, int64_t stride) { + int64_t kElems = ksize * ksize; + for (int64_t t = outDim; t >= 1; --t) { + int64_t inSize = (t - 1) * stride + ksize; + if (inSize > kMaxIter) + continue; + int64_t rows = cdiv(t * t, kLane) * cdiv(kElems, kLane) * kLane; + if (rows <= kBankLines) + return t; + } + return 0; +} + +static int64_t pickFatCin(int64_t kElems, int64_t wins) { + for (int64_t n = 256; n >= 1; --n) { + int64_t kFat = n * kElems; + if (kFat > 4095) + continue; + int64_t aRows = cdiv(wins, kLane) * cdiv(kFat, kLane) * kLane; + if (aRows <= kBankLines) + return n; + } + return 1; +} + +static int64_t pickTileForConv(int64_t outDim, int64_t ksize, int64_t stride, + int64_t cin) { + int64_t kElems = ksize * ksize; + int64_t bestT = 0; + int64_t bestScore = INT64_MAX; + for (int64_t t = outDim; t >= 1; --t) { + int64_t inSize = (t - 1) * stride + ksize; + if (inSize > kMaxIter) + continue; + int64_t wins = t * t; + int64_t rows = cdiv(wins, kLane) * cdiv(kElems, kLane) * kLane; + if (rows > kBankLines) + continue; + int64_t fat = pickFatCin(kElems, wins); + if (fat < 1) + continue; + int64_t spatial = cdiv(outDim, t) * cdiv(outDim, t); + int64_t cinSteps = cdiv(cin, fat); + int64_t score = spatial * cinSteps; + if (score < bestScore) { + bestScore = score; + bestT = t; + } + } + return bestT; +} + +static void copyPlaneRows(OpBuilder &b, Location loc, Value plane, Value inPack, + int64_t lc, int64_t inRows, Value zero, Value one, + Value sixteen) { + Value inRowsV = b.create(loc, inRows); + Value base = b.create(loc, lc * inRows); + auto rL = b.create(loc, zero, inRowsV, one); + b.setInsertionPointToStart(rL.getBody()); + Value r = rL.getInductionVar(); + Value rDst = b.create(loc, base, r); + auto cL = b.create(loc, zero, sixteen, one); + b.setInsertionPointToStart(cL.getBody()); + Value col = cL.getInductionVar(); + Value v = b.create(loc, plane, ValueRange{r, col}); + b.create(loc, v, inPack, ValueRange{rDst, col}); + b.setInsertionPointAfter(rL); +} + +static void packInPlane(OpBuilder &b, Location loc, Value inPack, Value input, + Value nV, Value cV, Value ih0V, Value iw0V, + int64_t inSize, int64_t inRows, int64_t H, int64_t W, + int64_t padLow, Value zero, Value one, Value sixteen, + Value f0) { + b.create(loc, f0, inPack); + Value inSizeV = b.create(loc, inSize); + Value nElems = b.create(loc, inSize * inSize); + Value hV = b.create(loc, H); + Value wV = b.create(loc, W); + Value padV = b.create(loc, padLow); + auto rL = b.create( + loc, zero, b.create(loc, inRows), one); + b.setInsertionPointToStart(rL.getBody()); + auto cL = b.create(loc, zero, sixteen, one); + b.setInsertionPointToStart(cL.getBody()); + Value flat = b.create( + loc, b.create(loc, rL.getInductionVar(), sixteen), + cL.getInductionVar()); + Value inBound = + b.create(loc, arith::CmpIPredicate::ult, flat, nElems); + Value flatC = b.create(loc, inBound, flat, zero); + Value ih = b.create(loc, flatC, inSizeV); + Value iw = b.create(loc, flatC, inSizeV); + Value ihAbs = b.create(loc, ih0V, ih); + Value iwAbs = b.create(loc, iw0V, iw); + Value ihReal = b.create(loc, ihAbs, padV); + Value iwReal = b.create(loc, iwAbs, padV); + Value ihGe0 = + b.create(loc, arith::CmpIPredicate::sge, ihReal, zero); + Value iwGe0 = + b.create(loc, arith::CmpIPredicate::sge, iwReal, zero); + Value ihLt = + b.create(loc, arith::CmpIPredicate::slt, ihReal, hV); + Value iwLt = + b.create(loc, arith::CmpIPredicate::slt, iwReal, wV); + Value inImg = + b.create(loc, b.create(loc, ihGe0, iwGe0), + b.create(loc, ihLt, iwLt)); + Value ok = b.create(loc, inBound, inImg); + Value ihSafe = b.create(loc, ok, ihReal, zero); + Value iwSafe = b.create(loc, ok, iwReal, zero); + SmallVector inIdx{nV, ihSafe, iwSafe, cV}; + Value v = b.create(loc, input, inIdx); + v = b.create(loc, ok, v, f0); + b.create( + loc, v, inPack, ValueRange{rL.getInductionVar(), cL.getInductionVar()}); + b.setInsertionPointAfter(rL); +} + +static void packFatFilter(OpBuilder &b, Location loc, Value filter, Value fPack, + Value c0V, int64_t nCin, int64_t KH, int64_t KW, + int64_t bRows, Value ocEnd, Value zero, Value one) { + for (int64_t lc = 0; lc < nCin; ++lc) { + Value lcV = b.create(loc, lc); + Value cV = b.create(loc, c0V, lcV); + for (int64_t kr = 0; kr < KH; ++kr) { + for (int64_t kc = 0; kc < KW; ++kc) { + int64_t k = kr * KH + kc; + Value krV = b.create(loc, kr); + Value kcV = b.create(loc, kc); + Value rowV = b.create(loc, lc * bRows + k); + auto oL = b.create(loc, zero, ocEnd, one); + b.setInsertionPointToStart(oL.getBody()); + Value oV = oL.getInductionVar(); + Value wt = + b.create(loc, filter, ValueRange{krV, kcV, cV, oV}); + b.create(loc, wt, fPack, ValueRange{rowV, oV}); + b.setInsertionPointAfter(oL); + } + } + } +} + +static void packDwFilter(OpBuilder &b, Location loc, Value filter, Value fPack, + Value cV, int64_t lc, int64_t KH, int64_t KW, + int64_t bRows) { + for (int64_t kr = 0; kr < KH; ++kr) { + for (int64_t kc = 0; kc < KW; ++kc) { + int64_t k = kr * KH + kc; + Value krV = b.create(loc, kr); + Value kcV = b.create(loc, kc); + Value rowV = b.create(loc, lc * bRows + k); + Value colV = b.create(loc, lc); + Value zeroC = b.create(loc, 0); + Value wt = b.create(loc, filter, + ValueRange{krV, kcV, cV, zeroC}); + b.create(loc, wt, fPack, ValueRange{rowV, colV}); + } + } +} + +static void scatterOut(OpBuilder &b, Location loc, Value src, Value output, + Value nV, Value oh0, Value ow0, Value c0V, Value nChV, + Value tileV, Value ohLim, Value owLim, Value zero, + Value one) { + auto ohL = b.create(loc, zero, tileV, one); + b.setInsertionPointToStart(ohL.getBody()); + auto owL = b.create(loc, zero, tileV, one); + b.setInsertionPointToStart(owL.getBody()); + Value loh = ohL.getInductionVar(); + Value low = owL.getInductionVar(); + Value oh = b.create(loc, oh0, loh); + Value ow = b.create(loc, ow0, low); + Value win = b.create( + loc, b.create(loc, loh, tileV), low); + Value ohOk = + b.create(loc, arith::CmpIPredicate::ult, oh, ohLim); + Value owOk = + b.create(loc, arith::CmpIPredicate::ult, ow, owLim); + Value ok = b.create(loc, ohOk, owOk); + auto ifOp = b.create(loc, ok, /*withElseRegion=*/false); + b.setInsertionPointToStart(&ifOp.getThenRegion().front()); + auto cL = b.create(loc, zero, nChV, one); + b.setInsertionPointToStart(cL.getBody()); + Value ov = + b.create(loc, src, ValueRange{win, cL.getInductionVar()}); + Value cGlobal = b.create(loc, c0V, cL.getInductionVar()); + SmallVector outIdx{nV, oh, ow, cGlobal}; + Value cur = b.create(loc, output, outIdx); + b.create(loc, b.create(loc, cur, ov), output, + outIdx); + b.setInsertionPointAfter(ohL); +} + +static void scatterMap(OpBuilder &b, Location loc, Value src, Value output, + Value nV, Value c0V, Value nChV, Value ohLim, + Value owLim, Value zero, Value one) { + auto ohL = b.create(loc, zero, ohLim, one); + b.setInsertionPointToStart(ohL.getBody()); + auto owL = b.create(loc, zero, owLim, one); + b.setInsertionPointToStart(owL.getBody()); + Value oh = ohL.getInductionVar(); + Value ow = owL.getInductionVar(); + Value win = + b.create(loc, b.create(loc, oh, owLim), ow); + auto cL = b.create(loc, zero, nChV, one); + b.setInsertionPointToStart(cL.getBody()); + Value ov = + b.create(loc, src, ValueRange{win, cL.getInductionVar()}); + Value cGlobal = b.create(loc, c0V, cL.getInductionVar()); + SmallVector outIdx{nV, oh, ow, cGlobal}; + Value cur = b.create(loc, output, outIdx); + b.create(loc, b.create(loc, cur, ov), output, + outIdx); + b.setInsertionPointAfter(ohL); +} + +class TileConv2dLowering : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(tile::TileConv2dOp op, + PatternRewriter &b) const override { + Location loc = op.getLoc(); + auto inTy = dyn_cast(op.getInput().getType()); + auto fTy = dyn_cast(op.getFilter().getType()); + auto oTy = dyn_cast(op.getOutput().getType()); + if (!inTy || !fTy || !oTy || !inTy.hasStaticShape() || + !fTy.hasStaticShape() || !oTy.hasStaticShape()) + return op.emitError("requires static memrefs"); + + Type elem = inTy.getElementType(); + if (isa(elem) && fTy.getElementType() == elem && + oTy.getElementType() == elem) { + auto is = inTy.getShape(), fs = fTy.getShape(), os = oTy.getShape(); + int64_t H = is[1], W = is[2]; + int64_t KH = fs[0], KW = fs[1]; + int64_t OH = os[1], OW = os[2]; + int64_t padLow = 0, padHigh = 0; + if (auto a = op->getAttrOfType("pad_low")) + padLow = a.getInt(); + if (auto a = op->getAttrOfType("pad_high")) + padHigh = a.getInt(); + if (padLow != 0 || padHigh != 0) + return op.emitError("float compact conv path requires pad=0"); + if (H != W || OH != OW || KH != KW) + return op.emitError("float compact conv requires square H/W/K"); + int64_t stride; + if (OH == 1) + stride = H - KH + 1; + else { + if ((H - KH) % (OH - 1) != 0) + return op.emitError("cannot infer integer stride from shapes"); + stride = (H - KH) / (OH - 1); + } + if (stride < 1 || (H - KH) / stride + 1 != OH) + return op.emitError("pad/stride/shape mismatch"); + auto ones = b.getI64TensorAttr({stride, stride}); + auto dilations = b.getI64TensorAttr({1, 1}); + b.create( + loc, TypeRange{}, ValueRange{op.getInput(), op.getFilter()}, + ValueRange{op.getOutput()}, ones, dilations); + b.eraseOp(op); + return success(); + } + + if (!inTy.getElementType().isF32() || !fTy.getElementType().isInteger(8) || + !oTy.getElementType().isF32()) + return op.emitError( + "requires static FP32 activation, INT8 filter, FP32 output"); + + auto dwAddrAttr = op->getAttrOfType("dw_addr"); + auto dwBytesAttr = op->getAttrOfType("dw_bytes"); + auto perChannelAttr = op->getAttrOfType("per_channel"); + if (!dwAddrAttr || !dwBytesAttr || !perChannelAttr || + dwAddrAttr.getInt() < 16) + return op.emitError("quantized conv requires RAX Dw metadata"); + + auto is = inTy.getShape(), fs = fTy.getShape(), os = oTy.getShape(); + int64_t N = is[0], H = is[1], W = is[2], C = is[3]; + int64_t KH = fs[0], KW = fs[1], FC = fs[2], OC = fs[3]; + int64_t OH = os[1], OW = os[2]; + int64_t padLow = 0, padHigh = 0; + if (auto a = op->getAttrOfType("pad_low")) + padLow = a.getInt(); + if (auto a = op->getAttrOfType("pad_high")) + padHigh = a.getInt(); + if (N != os[0] || C != FC || OC != os[3] || H != W || OH != OW || KH != KW) + return op.emitError("shape mismatch or non-square H/W/K"); + const int64_t ocPadForDw = cdiv(OC, kLane) * kLane; + const int64_t dwRequired = perChannelAttr.getValue() ? ocPadForDw * 4 : 4; + if (dwBytesAttr.getInt() < dwRequired) + return op.emitError( + "Dw scale image does not cover padded output channels"); + if (KH < 1 || KH > kMaxK) + return op.emitError("ksize out of range"); + if (padLow < 0 || padHigh < 0 || padLow > 7 || padHigh > 7) + return op.emitError("pad out of range"); + int64_t padded = H + padLow + padHigh; + if (padded < KH) + return op.emitError("padded input smaller than kernel"); + int64_t stride; + if (OH == 1) + stride = padded - KH + 1; + else { + if ((padded - KH) % (OH - 1) != 0) + return op.emitError("cannot infer integer stride from shapes"); + stride = (padded - KH) / (OH - 1); + } + if (stride < 1 || (padded - KH) / stride + 1 != OH) + return op.emitError("pad/stride/shape mismatch"); + + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value f0 = b.create(loc, b.getF32Type(), + b.getF32FloatAttr(0.0f)); + Value i0 = b.create(loc, fTy.getElementType(), + b.getZeroAttr(fTy.getElementType())); + + if (KH == 1) { + if (padLow != 0 || padHigh != 0) + return op.emitError("1x1 conv requires pad=0"); + if ((OH - 1) * stride + 1 != H || (OW - 1) * stride + 1 != W) + return op.emitError("1x1 conv stride/shape mismatch"); + int64_t M = OH * OW; + int64_t mPad = cdiv(M, kLane) * kLane; + int64_t kPad = cdiv(C, kLane) * kLane; + Value ohEnd = b.create(loc, OH); + Value owEnd = b.create(loc, OW); + Value cEnd = b.create(loc, C); + Value strideV = b.create(loc, stride); + Value aPack = b.create( + loc, MemRefType::get({mPad, kPad}, b.getF32Type())); + Value bPack = b.create( + loc, MemRefType::get({kPad, kLane}, fTy.getElementType())); + Value cPack = b.create( + loc, MemRefType::get({mPad, kLane}, b.getF32Type())); + + for (int64_t n = 0; n < N; ++n) { + Value nV = b.create(loc, n); + b.create(loc, f0, aPack); + auto ohL = b.create(loc, zero, ohEnd, one); + b.setInsertionPointToStart(ohL.getBody()); + auto owL = b.create(loc, zero, owEnd, one); + b.setInsertionPointToStart(owL.getBody()); + Value oh = ohL.getInductionVar(); + Value ow = owL.getInductionVar(); + Value ih = b.create(loc, oh, strideV); + Value iw = b.create(loc, ow, strideV); + Value win = b.create( + loc, b.create(loc, oh, owEnd), ow); + auto cL = b.create(loc, zero, cEnd, one); + b.setInsertionPointToStart(cL.getBody()); + Value cV = cL.getInductionVar(); + Value v = b.create(loc, op.getInput(), + ValueRange{nV, ih, iw, cV}); + b.create(loc, v, aPack, ValueRange{win, cV}); + b.setInsertionPointAfter(ohL); + + for (int64_t oc0 = 0; oc0 < OC; oc0 += kLane) { + int64_t ocTile = OC - oc0 < kLane ? OC - oc0 : kLane; + Value oc0V = b.create(loc, oc0); + Value ocTileV = b.create(loc, ocTile); + const int64_t panelDwOffset = perChannelAttr.getValue() ? oc0 * 4 : 0; + auto panelDwAddr = + b.getI64IntegerAttr(dwAddrAttr.getInt() + panelDwOffset); + auto panelDwBytes = + b.getI64IntegerAttr(dwBytesAttr.getInt() - panelDwOffset); + b.create(loc, i0, bPack); + auto cPackL = b.create(loc, zero, cEnd, one); + b.setInsertionPointToStart(cPackL.getBody()); + Value cPv = cPackL.getInductionVar(); + auto oPackL = b.create(loc, zero, ocTileV, one); + b.setInsertionPointToStart(oPackL.getBody()); + Value oLocal = oPackL.getInductionVar(); + Value oGlobal = b.create(loc, oc0V, oLocal); + Value wt = b.create( + loc, op.getFilter(), ValueRange{zero, zero, cPv, oGlobal}); + b.create(loc, wt, bPack, ValueRange{cPv, oLocal}); + b.setInsertionPointAfter(cPackL); + + if (mPad <= kBankLines) { + b.create(loc, f0, cPack); + auto matmul = b.create(loc, aPack, bPack, cPack); + matmul->setAttr("dwAddr", panelDwAddr); + matmul->setAttr("dwBytes", panelDwBytes); + matmul->setAttr("perChannel", perChannelAttr); + } else { + int64_t aligned = (mPad / kBankLines) * kBankLines; + int64_t rem = mPad - aligned; + Value step = b.create(loc, kBankLines); + Value alignedV = b.create(loc, aligned); + auto mL = b.create(loc, zero, alignedV, step); + b.setInsertionPointToStart(mL.getBody()); + Value aTile = b.create( + loc, aPack, + SmallVector{mL.getInductionVar(), + b.getIndexAttr(0)}, + SmallVector{b.getIndexAttr(kBankLines), + b.getIndexAttr(kPad)}, + SmallVector{b.getIndexAttr(1), + b.getIndexAttr(1)}); + Value cTile = b.create( + loc, cPack, + SmallVector{mL.getInductionVar(), + b.getIndexAttr(0)}, + SmallVector{b.getIndexAttr(kBankLines), + b.getIndexAttr(kLane)}, + SmallVector{b.getIndexAttr(1), + b.getIndexAttr(1)}); + auto matmul = b.create(loc, aTile, bPack, cTile); + matmul->setAttr("dwAddr", panelDwAddr); + matmul->setAttr("dwBytes", panelDwBytes); + matmul->setAttr("perChannel", perChannelAttr); + b.setInsertionPointAfter(mL); + if (rem > 0) { + Value aRem = b.create( + loc, aPack, + SmallVector{b.getIndexAttr(aligned), + b.getIndexAttr(0)}, + SmallVector{b.getIndexAttr(rem), + b.getIndexAttr(kPad)}, + SmallVector{b.getIndexAttr(1), + b.getIndexAttr(1)}); + Value cRem = b.create( + loc, cPack, + SmallVector{b.getIndexAttr(aligned), + b.getIndexAttr(0)}, + SmallVector{b.getIndexAttr(rem), + b.getIndexAttr(kLane)}, + SmallVector{b.getIndexAttr(1), + b.getIndexAttr(1)}); + auto matmul = b.create(loc, aRem, bPack, cRem); + matmul->setAttr("dwAddr", panelDwAddr); + matmul->setAttr("dwBytes", panelDwBytes); + matmul->setAttr("perChannel", perChannelAttr); + } + } + scatterMap(b, loc, cPack, op.getOutput(), nV, oc0V, ocTileV, ohEnd, + owEnd, zero, one); + } + } + + b.create(loc, aPack); + b.create(loc, bPack); + b.create(loc, cPack); + b.eraseOp(op); + return success(); + } + + int64_t tile = pickTileForConv(OH, KH, stride, C); + if (tile == 0) + return op.emitError("no tile fits im2col bank capacity"); + + int64_t kElems = KH * KH; + int64_t wins = tile * tile; + int64_t inSize = (tile - 1) * stride + KH; + int64_t inRows = cdiv(inSize * inSize, kLane); + int64_t bRows = cdiv(kElems, kLane) * kLane; + if (wins > kBankLines) + return op.emitError("conv M pad exceeds bank depth"); + + Value sixteen = b.create(loc, kLane); + Value tileV = b.create(loc, tile); + Value strideV = b.create(loc, stride); + Value ohEnd = b.create(loc, OH); + Value owEnd = b.create(loc, OW); + + int64_t ocPad = cdiv(OC, kLane) * kLane; + if (ocPad < kLane || ocPad > 4096) + return op.emitError("OC pad out of range"); + + int64_t fatCinMax = pickFatCin(kElems, wins); + if (fatCinMax < 1) + return op.emitError("no fat cin batch fits im2col bank capacity"); + + Value inPack = b.create( + loc, MemRefType::get({fatCinMax * inRows, kLane}, b.getF32Type())); + Value fPack = b.create( + loc, MemRefType::get({fatCinMax * bRows, ocPad}, fTy.getElementType())); + Value tmpF = b.create( + loc, MemRefType::get({wins, ocPad}, b.getF32Type())); + Value accF = b.create( + loc, MemRefType::get({wins, ocPad}, b.getF32Type())); + Value plane = b.create( + loc, MemRefType::get({inRows, kLane}, b.getF32Type())); + Value ocPadV = b.create(loc, ocPad); + Value ocEnd = b.create(loc, OC); + Value oc0V = zero; + Value fatCinV = b.create(loc, fatCinMax); + + auto emitFatBlock = [&](Value nV, Value ih0V, Value iw0V, Value c0V, + int64_t nCin) { + b.create(loc, i0, fPack); + for (int64_t lc = 0; lc < nCin; ++lc) { + Value cV = b.create( + loc, c0V, b.create(loc, lc)); + packInPlane(b, loc, plane, op.getInput(), nV, cV, ih0V, iw0V, inSize, + inRows, H, W, padLow, zero, one, sixteen, f0); + copyPlaneRows(b, loc, plane, inPack, lc, inRows, zero, one, sixteen); + } + packFatFilter(b, loc, op.getFilter(), fPack, c0V, nCin, KH, KW, bRows, + ocEnd, zero, one); + Value inSub = b.create( + loc, inPack, + ArrayRef{b.getIndexAttr(0), b.getIndexAttr(0)}, + ArrayRef{b.getIndexAttr(nCin * inRows), + b.getIndexAttr(kLane)}, + ArrayRef{b.getIndexAttr(1), b.getIndexAttr(1)}); + Value fSub = b.create( + loc, fPack, + ArrayRef{b.getIndexAttr(0), b.getIndexAttr(0)}, + ArrayRef{b.getIndexAttr(nCin * bRows), + b.getIndexAttr(ocPad)}, + ArrayRef{b.getIndexAttr(1), b.getIndexAttr(1)}); + auto matmul = b.create( + loc, inSub, fSub, tmpF, b.getI64IntegerAttr(inSize), + b.getI64IntegerAttr(KH), b.getI64IntegerAttr(ocPad), + b.getI64IntegerAttr(nCin), b.getI64IntegerAttr(stride), + b.getI64IntegerAttr(0)); + matmul->setAttr("dwAddr", dwAddrAttr); + matmul->setAttr("dwBytes", dwBytesAttr); + matmul->setAttr("perChannel", perChannelAttr); + Value zI = b.create(loc, 0); + Value nRows = b.create(loc, wins); + auto rAdd = b.create(loc, zI, nRows, one); + b.setInsertionPointToStart(rAdd.getBody()); + auto cAdd = b.create(loc, zI, ocPadV, one); + b.setInsertionPointToStart(cAdd.getBody()); + Value aa = b.create( + loc, accF, + ValueRange{rAdd.getInductionVar(), cAdd.getInductionVar()}); + Value tt = b.create( + loc, tmpF, + ValueRange{rAdd.getInductionVar(), cAdd.getInductionVar()}); + b.create( + loc, b.create(loc, aa, tt), accF, + ValueRange{rAdd.getInductionVar(), cAdd.getInductionVar()}); + b.setInsertionPointAfter(rAdd); + }; + + for (int64_t n = 0; n < N; ++n) { + Value nV = b.create(loc, n); + auto oh0L = b.create(loc, zero, ohEnd, tileV); + b.setInsertionPointToStart(oh0L.getBody()); + Value oh0 = oh0L.getInductionVar(); + auto ow0L = b.create(loc, zero, owEnd, tileV); + b.setInsertionPointToStart(ow0L.getBody()); + Value ow0 = ow0L.getInductionVar(); + Value ih0V = b.create(loc, oh0, strideV); + Value iw0V = b.create(loc, ow0, strideV); + + b.create(loc, f0, accF); + int64_t fullCin = (C / fatCinMax) * fatCinMax; + if (fullCin) { + Value fullCinV = b.create(loc, fullCin); + auto c0L = b.create(loc, zero, fullCinV, fatCinV); + b.setInsertionPointToStart(c0L.getBody()); + emitFatBlock(nV, ih0V, iw0V, c0L.getInductionVar(), fatCinMax); + b.setInsertionPointAfter(c0L); + } + if (fullCin != C) + emitFatBlock(nV, ih0V, iw0V, + b.create(loc, fullCin), + C - fullCin); + + scatterOut(b, loc, accF, op.getOutput(), nV, oh0, ow0, oc0V, ocEnd, tileV, + ohEnd, owEnd, zero, one); + b.setInsertionPointAfter(oh0L); + } + + b.create(loc, inPack); + b.create(loc, fPack); + b.create(loc, tmpF); + b.create(loc, accF); + b.create(loc, plane); + b.eraseOp(op); + return success(); + } +}; + +// Depthwise: channel tiles of 16 -> one fat im2col+GEMM (block-diag B), never +// GEMV. +class TileDepthwiseConv2dLowering + : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(tile::TileDepthwiseConv2dOp op, + PatternRewriter &b) const override { + Location loc = op.getLoc(); + auto inTy = dyn_cast(op.getInput().getType()); + auto fTy = dyn_cast(op.getFilter().getType()); + auto oTy = dyn_cast(op.getOutput().getType()); + if (!inTy || !fTy || !oTy || !inTy.hasStaticShape() || + !fTy.hasStaticShape() || !oTy.hasStaticShape() || + !inTy.getElementType().isF32() || !fTy.getElementType().isInteger(8) || + !oTy.getElementType().isF32()) + return op.emitError( + "requires static FP32 activations, INT8 weights, and FP32 output"); + + auto dwAddr = op->getAttrOfType("dw_addr"); + auto dwBytes = op->getAttrOfType("dw_bytes"); + auto perChannel = op->getAttrOfType("per_channel"); + if (!dwAddr || !dwBytes || !perChannel) + return op.emitError("requires Dw RAX metadata"); + + auto is = inTy.getShape(), fs = fTy.getShape(), os = oTy.getShape(); + int64_t N = is[0], H = is[1], W = is[2], C = is[3]; + int64_t KH = fs[0], KW = fs[1], FC = fs[2], mult = fs[3]; + int64_t OH = os[1], OW = os[2], OC = os[3]; + int64_t padLow = 0, padHigh = 0; + if (auto a = op->getAttrOfType("pad_low")) + padLow = a.getInt(); + if (auto a = op->getAttrOfType("pad_high")) + padHigh = a.getInt(); + if (N != os[0] || C != FC || OC != C || mult != 1 || H != W || OH != OW || + KH != KW) + return op.emitError("depthwise shape mismatch"); + const int64_t cPadForDw = cdiv(C, kLane) * kLane; + const int64_t dwRequired = perChannel.getValue() ? cPadForDw * 4 : 4; + if (dwBytes.getInt() < dwRequired) + return op.emitError( + "Dw scale image does not cover padded output channels"); + if (KH < 1 || KH > kMaxK) + return op.emitError("ksize out of range"); + if (padLow < 0 || padHigh < 0 || padLow > 7 || padHigh > 7) + return op.emitError("pad out of range"); + int64_t padded = H + padLow + padHigh; + if (padded < KH) + return op.emitError("padded input smaller than kernel"); + int64_t stride; + if (OH == 1) + stride = padded - KH + 1; + else { + if ((padded - KH) % (OH - 1) != 0) + return op.emitError("cannot infer integer stride from shapes"); + stride = (padded - KH) / (OH - 1); + } + if (stride < 1 || (padded - KH) / stride + 1 != OH) + return op.emitError("pad/stride/shape mismatch"); + + // Spatial tile by single-plane im2col capacity; channels batched by lane. + int64_t tile = pickTile(OH, KH, stride); + int64_t nBatch = kLane; + if (tile == 0) + return op.emitError("no depthwise tile fits im2col bank capacity"); + + int64_t kElems = KH * KH; + int64_t wins = tile * tile; + int64_t inSize = (tile - 1) * stride + KH; + int64_t inRows = cdiv(inSize * inSize, kLane); + int64_t bRows = cdiv(kElems, kLane) * kLane; + + Value zero = b.create(loc, 0); + Value one = b.create(loc, 1); + Value sixteen = b.create(loc, kLane); + Value tileV = b.create(loc, tile); + Value strideV = b.create(loc, stride); + Value ohEnd = b.create(loc, OH); + Value owEnd = b.create(loc, OW); + Value f0 = b.create(loc, b.getF32Type(), + b.getF32FloatAttr(0.0f)); + Value i0 = + b.create(loc, b.getI8Type(), b.getI8IntegerAttr(0)); + + Value inPack = b.create( + loc, MemRefType::get({nBatch * inRows, kLane}, b.getF32Type())); + Value fPack = b.create( + loc, MemRefType::get({nBatch * bRows, kLane}, b.getI8Type())); + Value outF = b.create( + loc, MemRefType::get({wins, kLane}, b.getF32Type())); + Value plane = b.create( + loc, MemRefType::get({inRows, kLane}, b.getF32Type())); + + for (int64_t n = 0; n < N; ++n) { + Value nV = b.create(loc, n); + auto oh0L = b.create(loc, zero, ohEnd, tileV); + b.setInsertionPointToStart(oh0L.getBody()); + Value oh0 = oh0L.getInductionVar(); + auto ow0L = b.create(loc, zero, owEnd, tileV); + b.setInsertionPointToStart(ow0L.getBody()); + Value ow0 = ow0L.getInductionVar(); + Value ih0V = b.create(loc, oh0, strideV); + Value iw0V = b.create(loc, ow0, strideV); + + for (int64_t c0 = 0; c0 < C; c0 += nBatch) { + int64_t nCh = C - c0 < nBatch ? C - c0 : nBatch; + b.create(loc, f0, inPack); + b.create(loc, i0, fPack); + + for (int64_t lc = 0; lc < nCh; ++lc) { + Value cV = b.create(loc, c0 + lc); + packInPlane(b, loc, plane, op.getInput(), nV, cV, ih0V, iw0V, inSize, + inRows, H, W, padLow, zero, one, sixteen, f0); + copyPlaneRows(b, loc, plane, inPack, lc, inRows, zero, one, sixteen); + packDwFilter(b, loc, op.getFilter(), fPack, cV, lc, KH, KW, bRows); + } + + auto matmul = b.create( + loc, inPack, fPack, outF, b.getI64IntegerAttr(inSize), + b.getI64IntegerAttr(KH), b.getI64IntegerAttr(nBatch), + b.getI64IntegerAttr(stride), b.getI64IntegerAttr(0)); + const int64_t panelDwOffset = perChannel.getValue() ? c0 * 4 : 0; + matmul->setAttr("dwAddr", + b.getI64IntegerAttr(dwAddr.getInt() + panelDwOffset)); + matmul->setAttr("dwBytes", + b.getI64IntegerAttr(dwBytes.getInt() - panelDwOffset)); + matmul->setAttr("perChannel", perChannel); + + Value nChV = b.create(loc, nCh); + Value c0V = b.create(loc, c0); + scatterOut(b, loc, outF, op.getOutput(), nV, oh0, ow0, c0V, nChV, tileV, + ohEnd, owEnd, zero, one); + } + b.setInsertionPointAfter(oh0L); + } + + b.create(loc, inPack); + b.create(loc, fPack); + b.create(loc, outF); + b.create(loc, plane); + b.eraseOp(op); + return success(); + } +}; + +class LowerTileToBuckyballPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerTileToBuckyballPass) + + StringRef getArgument() const final { return "convert-tile-to-buckyball"; } + StringRef getDescription() const final { + return "Convert Tile operations for the Pebble Core."; + } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + const buckyball_target::BuckyballTargetConfig &targetConfig = + buckyball_target::getBuckyballTarget(); + MLIRContext *context = &getContext(); + + ConversionTarget target(*context); + target.addLegalDialect<::buddy::buckyball::BuckyballDialect, + memref::MemRefDialect, arith::ArithDialect, + scf::SCFDialect, func::FuncDialect, + linalg::LinalgDialect>(); + target.addIllegalDialect(); + + RewritePatternSet patterns(context); + mlir::buddy::populateSMatMulBallTileLoweringPatterns( + patterns, targetConfig.bankWidthBits / 8, targetConfig.bankDepth, + targetConfig.bankNum); + mlir::buddy::populateTransposeBallTileLoweringPatterns( + patterns, targetConfig.bankWidthBits / 8, targetConfig.bankDepth, + targetConfig.bankNum); + mlir::buddy::populateReluBallTileLoweringPatterns( + patterns, targetConfig.bankWidthBits / 8, targetConfig.bankDepth, + targetConfig.bankNum); + patterns.add(context); + + if (failed(applyPartialConversion(getOperation(), target, + std::move(patterns)))) + signalPassFailure(); + } +}; + +} // namespace + +void mlir::buddy::registerLowerTileToBuckyballPass() { + PassRegistration(); +} diff --git a/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp b/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp deleted file mode 100644 index b2d853675..000000000 --- a/examples/cores/pebble/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp +++ /dev/null @@ -1,347 +0,0 @@ -//===- LowerTileToBuckyball.cpp - Pebble tile->buckyball pass -------------===// -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// -//===----------------------------------------------------------------------===// - -#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" - -#include "mlir/Dialect/Arith/IR/Arith.h" -#include "mlir/Dialect/Func/IR/FuncOps.h" -#include "mlir/Dialect/Linalg/IR/Linalg.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Dialect/SCF/IR/SCF.h" -#include "mlir/IR/BuiltinOps.h" -#include "mlir/Pass/Pass.h" -#include "mlir/Transforms/DialectConversion.h" - -#include "Buckyball/BuckyballDialect.h" -#include "Buckyball/BuckyballOps.h" -#include "Tile/TileDialect.h" -#include "Tile/TileOps.h" -#include "Tile/Transform.h" - -using namespace mlir; -using namespace ::buddy::buckyball; -namespace tile = ::buddy::tile; -using mlir::buddy::kDefaultBankWidthBytes; -using mlir::buddy::populateMatrixTileMatMulPatterns; - -namespace { - -class TileTransposeLowering : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(tile::TileTransposeOp op, - PatternRewriter &rewriter) const override { - auto inputType = dyn_cast(op.getAMemArray().getType()); - auto outputType = dyn_cast(op.getBMemArray().getType()); - if (!inputType || !outputType || !inputType.hasStaticShape() || - !outputType.hasStaticShape()) - return op.emitError("requires static input and output memrefs"); - if (inputType.getRank() != 2 || outputType.getRank() != 2) - return op.emitError("requires rank-2 memrefs"); - if (outputType.getShape()[0] != inputType.getShape()[1] || - outputType.getShape()[1] != inputType.getShape()[0]) - return op.emitError("output shape must transpose the input shape"); - if (inputType.getElementType() != outputType.getElementType()) - return op.emitError("input/output element types must match"); - - rewriter.create(op.getLoc(), op.getAMemArray(), - op.getBMemArray()); - rewriter.eraseOp(op); - return success(); - } -}; - -// Pebble im2col: square HxW, per-cin plane, K=k^2. -// tile->buckyball: pack + buckyball.im2col_matmul (no quant). pad=0 stride=1. -constexpr int64_t kMaxIter = 34; -constexpr int64_t kMaxK = 7; -constexpr int64_t kBankLines = 1024; -constexpr int64_t kLane = 16; - -static int64_t cdiv(int64_t a, int64_t b) { return (a + b - 1) / b; } - -static int64_t pickTile(int64_t outDim, int64_t ksize) { - int64_t kElems = ksize * ksize; - for (int64_t t = outDim; t >= 1; --t) { - if (outDim % t != 0) - continue; - int64_t inSize = t + ksize - 1; - if (inSize > kMaxIter) - continue; - int64_t rows = cdiv(t * t, kLane) * cdiv(kElems, kLane) * kLane; - if (rows <= kBankLines) - return t; - } - return 0; -} - -static void addF32Pack(OpBuilder &b, Location loc, Value dst, Value src, - int64_t rows) { - Value zero = b.create(loc, 0); - Value one = b.create(loc, 1); - Value nRows = b.create(loc, rows); - Value nCols = b.create(loc, kLane); - auto r = b.create(loc, zero, nRows, one); - b.setInsertionPointToStart(r.getBody()); - auto c = b.create(loc, zero, nCols, one); - b.setInsertionPointToStart(c.getBody()); - Value a = b.create( - loc, dst, ValueRange{r.getInductionVar(), c.getInductionVar()}); - Value t = b.create( - loc, src, ValueRange{r.getInductionVar(), c.getInductionVar()}); - b.create( - loc, b.create(loc, a, t), dst, - ValueRange{r.getInductionVar(), c.getInductionVar()}); - b.setInsertionPointAfter(r); -} - -static void packInPlane(OpBuilder &b, Location loc, Value inPack, Value input, - Value nV, Value cV, Value ih0V, Value iw0V, - int64_t inSize, int64_t inRows, Value zero, Value one, - Value sixteen, Value f0) { - b.create(loc, f0, inPack); - Value inSizeV = b.create(loc, inSize); - Value nElems = b.create(loc, inSize * inSize); - auto rL = b.create( - loc, zero, b.create(loc, inRows), one); - b.setInsertionPointToStart(rL.getBody()); - auto cL = b.create(loc, zero, sixteen, one); - b.setInsertionPointToStart(cL.getBody()); - Value flat = b.create( - loc, b.create(loc, rL.getInductionVar(), sixteen), - cL.getInductionVar()); - Value inBound = - b.create(loc, arith::CmpIPredicate::ult, flat, nElems); - Value flatC = b.create(loc, inBound, flat, zero); - Value ih = b.create(loc, flatC, inSizeV); - Value iw = b.create(loc, flatC, inSizeV); - Value ihAbs = b.create(loc, ih0V, ih); - Value iwAbs = b.create(loc, iw0V, iw); - SmallVector inIdx{nV, ihAbs, iwAbs, cV}; - Value v = b.create(loc, input, inIdx); - v = b.create(loc, inBound, v, f0); - b.create( - loc, v, inPack, ValueRange{rL.getInductionVar(), cL.getInductionVar()}); - b.setInsertionPointAfter(rL); -} - -class TileConv2dLowering : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(tile::TileConv2dOp op, - PatternRewriter &b) const override { - Location loc = op.getLoc(); - auto inTy = dyn_cast(op.getInput().getType()); - auto fTy = dyn_cast(op.getFilter().getType()); - auto oTy = dyn_cast(op.getOutput().getType()); - if (!inTy || !fTy || !oTy || !inTy.hasStaticShape() || - !fTy.hasStaticShape() || !oTy.hasStaticShape() || - !inTy.getElementType().isF32() || !fTy.getElementType().isF32() || - !oTy.getElementType().isF32()) - return op.emitError("requires static f32 memrefs"); - - auto is = inTy.getShape(), fs = fTy.getShape(), os = oTy.getShape(); - int64_t N = is[0], H = is[1], W = is[2], C = is[3]; - int64_t KH = fs[0], KW = fs[1], FC = fs[2], OC = fs[3]; - int64_t OH = os[1], OW = os[2]; - if (N != os[0] || C != FC || OC != os[3] || H != W || OH != OW || KH != KW) - return op.emitError("shape mismatch or non-square H/W/K"); - if (KH < 1 || KH > kMaxK) - return op.emitError("ksize out of range"); - if (OH != H - KH + 1) - return op.emitError("only pad=0 stride=1 supported"); - - int64_t tile = pickTile(OH, KH); - if (tile == 0) - return op.emitError("no tile fits im2col bank capacity"); - - int64_t kElems = KH * KH; - int64_t wins = tile * tile; - int64_t inSize = tile + KH - 1; - int64_t inRows = cdiv(inSize * inSize, kLane); - int64_t bRows = cdiv(kElems, kLane) * kLane; - int64_t cBlocks = wins; - - Value zero = b.create(loc, 0); - Value one = b.create(loc, 1); - Value sixteen = b.create(loc, kLane); - Value f0 = b.create(loc, b.getF32Type(), - b.getF32FloatAttr(0.0f)); - Value cEnd = b.create(loc, C); - - Value inPack = b.create( - loc, MemRefType::get({inRows, kLane}, b.getF32Type())); - Value fPack = b.create( - loc, MemRefType::get({bRows, kLane}, b.getF32Type())); - Value tmpF = b.create( - loc, MemRefType::get({cBlocks, kLane}, b.getF32Type())); - Value accF = b.create( - loc, MemRefType::get({cBlocks, kLane}, b.getF32Type())); - - for (int64_t n = 0; n < N; ++n) { - Value nV = b.create(loc, n); - for (int64_t oh0 = 0; oh0 < OH; oh0 += tile) { - for (int64_t ow0 = 0; ow0 < OW; ow0 += tile) { - for (int64_t oc0 = 0; oc0 < OC; oc0 += kLane) { - int64_t ocTile = OC - oc0 < kLane ? OC - oc0 : kLane; - b.create(loc, f0, accF); - - Value ih0V = b.create(loc, oh0); - Value iw0V = b.create(loc, ow0); - auto cinL = b.create(loc, zero, cEnd, one); - b.setInsertionPointToStart(cinL.getBody()); - Value cV = cinL.getInductionVar(); - - packInPlane(b, loc, inPack, op.getInput(), nV, cV, ih0V, iw0V, - inSize, inRows, zero, one, sixteen, f0); - - b.create(loc, f0, fPack); - for (int64_t kr = 0; kr < KH; ++kr) { - for (int64_t kc = 0; kc < KW; ++kc) { - int64_t k = kr * KH + kc; - Value krV = b.create(loc, kr); - Value kcV = b.create(loc, kc); - Value rowV = b.create(loc, k); - for (int64_t o = 0; o < ocTile; ++o) { - Value oLocal = b.create(loc, o); - Value oGlobal = - b.create(loc, oc0 + o); - Value wt = b.create( - loc, op.getFilter(), ValueRange{krV, kcV, cV, oGlobal}); - b.create(loc, wt, fPack, - ValueRange{rowV, oLocal}); - } - } - } - - b.create( - loc, inPack, fPack, tmpF, b.getI64IntegerAttr(inSize), - b.getI64IntegerAttr(KH), b.getI64IntegerAttr(ocTile), - b.getI64IntegerAttr(1), b.getI64IntegerAttr(0)); - addF32Pack(b, loc, accF, tmpF, cBlocks); - b.setInsertionPointAfter(cinL); - - Value tileV = b.create(loc, tile); - Value oh0V = b.create(loc, oh0); - Value ow0V = b.create(loc, ow0); - Value oc0V = b.create(loc, oc0); - auto ohL = b.create(loc, zero, tileV, one); - b.setInsertionPointToStart(ohL.getBody()); - auto owL = b.create(loc, zero, tileV, one); - b.setInsertionPointToStart(owL.getBody()); - Value win = b.create( - loc, b.create(loc, ohL.getInductionVar(), tileV), - owL.getInductionVar()); - auto ocL = b.create( - loc, zero, b.create(loc, ocTile), one); - b.setInsertionPointToStart(ocL.getBody()); - Value ov = b.create( - loc, accF, ValueRange{win, ocL.getInductionVar()}); - Value oGlobal = - b.create(loc, oc0V, ocL.getInductionVar()); - Value oh = - b.create(loc, oh0V, ohL.getInductionVar()); - Value ow = - b.create(loc, ow0V, owL.getInductionVar()); - SmallVector outIdx{nV, oh, ow, oGlobal}; - Value cur = b.create(loc, op.getOutput(), outIdx); - b.create(loc, - b.create(loc, cur, ov), - op.getOutput(), outIdx); - b.setInsertionPointAfter(ohL); - } - } - } - } - - b.create(loc, inPack); - b.create(loc, fPack); - b.create(loc, tmpF); - b.create(loc, accF); - b.eraseOp(op); - return success(); - } -}; - -} // namespace - -void mlir::populateLowerTileToBuckyballConversionPatterns( - RewritePatternSet &patterns, int64_t bankWidthBytes, int64_t bankDepth, - int64_t bankNum) { - populateMatrixTileMatMulPatterns(patterns, bankWidthBytes, bankDepth, - bankNum); - patterns.add(patterns.getContext()); - patterns.add(patterns.getContext()); -} - -namespace { - -class LowerTileToBuckyballPass - : public PassWrapper> { -public: - MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerTileToBuckyballPass) - StringRef getArgument() const final { return "convert-tile-to-buckyball"; } - StringRef getDescription() const final { - return "Convert Tile dialect to Buckyball dialect"; - } - LowerTileToBuckyballPass() = default; - LowerTileToBuckyballPass(const LowerTileToBuckyballPass &) {} - - Option bankWidthBytes{ - *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), - llvm::cl::init(kDefaultBankWidthBytes)}; - Option bankDepth{*this, "bank_depth", - llvm::cl::desc("Bank depth (rows per bank)."), - llvm::cl::init(1024)}; - Option bankNum{*this, "bank_num", llvm::cl::desc("Number of banks."), - llvm::cl::init(8)}; - - void getDependentDialects(DialectRegistry ®istry) const override { - registry.insert<::buddy::tile::TileDialect, - ::buddy::buckyball::BuckyballDialect, func::FuncDialect, - memref::MemRefDialect, arith::ArithDialect, scf::SCFDialect, - linalg::LinalgDialect>(); - } - - void runOnOperation() override { - MLIRContext *context = &getContext(); - ModuleOp module = getOperation(); - - ConversionTarget target(*context); - target.addLegalDialect<::buddy::buckyball::BuckyballDialect, - memref::MemRefDialect, arith::ArithDialect, - scf::SCFDialect, func::FuncDialect, - linalg::LinalgDialect>(); - target.addIllegalOp<::buddy::tile::TileMatMulOp>(); - target.addIllegalOp<::buddy::tile::TileConv2dOp>(); - target.addIllegalOp<::buddy::tile::TileTransposeOp>(); - - RewritePatternSet patterns(context); - populateLowerTileToBuckyballConversionPatterns(patterns, bankWidthBytes, - bankDepth, bankNum); - - if (failed(applyPartialConversion(module, target, std::move(patterns)))) - signalPassFailure(); - } -}; - -} // namespace - -void mlir::buddy::registerLowerTileToBuckyballPass() { - PassRegistration(); -} diff --git a/examples/cores/pebble/compiler/src/Dialect/Buckyball/CMakeLists.txt b/examples/cores/pebble/compiler/src/Dialect/Buckyball/CMakeLists.txt index d87ec0dba..867a85b69 100644 --- a/examples/cores/pebble/compiler/src/Dialect/Buckyball/CMakeLists.txt +++ b/examples/cores/pebble/compiler/src/Dialect/Buckyball/CMakeLists.txt @@ -1,4 +1,4 @@ -set(LLVM_TARGET_DEFINITIONS PebbleBuckyball.td) +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) mlir_tablegen(Buckyball.h.inc -gen-op-decls) mlir_tablegen(Buckyball.cpp.inc -gen-op-defs) mlir_tablegen(BuckyballTypes.h.inc -gen-typedef-decls -typedefs-dialect=buckyball) @@ -9,7 +9,7 @@ add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}MLIRBuckyballIncGen) add_mlir_doc(PebbleBuckyball ${BB_CORE_TARGET_PREFIX}Buckyball Dialects/ -gen-dialect-doc) -set(LLVM_TARGET_DEFINITIONS PebbleBuckyball.td) +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) mlir_tablegen(BuckyballConversions.inc -gen-llvmir-conversions) add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen) diff --git a/examples/cores/pebble/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt b/examples/cores/pebble/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt index da76ba536..a2e7abc2e 100644 --- a/examples/cores/pebble/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt +++ b/examples/cores/pebble/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt @@ -1,5 +1,6 @@ add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyball ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/IR/BuckyballDialect.cpp + ${BB_TARGET_REGISTRY_SOURCE} LINK_LIBS PUBLIC MLIRIR diff --git a/examples/cores/pebble/compiler/src/Dialect/Buckyball/PebbleBuckyball.td b/examples/cores/pebble/compiler/src/Dialect/Buckyball/PebbleBuckyball.td index 7bc1af820..2107cda1d 100644 --- a/examples/cores/pebble/compiler/src/Dialect/Buckyball/PebbleBuckyball.td +++ b/examples/cores/pebble/compiler/src/Dialect/Buckyball/PebbleBuckyball.td @@ -19,9 +19,11 @@ include "Buckyball.td" include "TransposeBall.td" -include "MatrixBall.td" +include "SMatMulBall.td" include "Im2colBall.td" include "Fp2IntBall.td" include "Int2FpBall.td" +include "MatAddBall.td" +include "ReluBall.td" #endif // PEBBLE_BUCKYBALL_OPS diff --git a/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt b/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt index 0ffa37742..7bc98d4ab 100644 --- a/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt +++ b/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt @@ -1,10 +1,12 @@ add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp ${BB_REPO_ROOT}/examples/balls/im2col/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/matadd/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp LegalizeForLLVMExport.cpp LINK_LIBS PUBLIC diff --git a/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index e913d14fe..73a9b4463 100644 --- a/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/cores/pebble/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -21,35 +21,41 @@ using namespace mlir; using namespace buddy::buckyball::legalize; namespace mlir::buddy::buckyball { -void populateTransposeLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable); -void configureTransposeLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateMatrixMatmulLegalizeForLLVMExportPatterns( +void populateTransposeBallLegalizeForLLVMExportPatterns( LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, - bool rushB); -void configureMatrixMatmulLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateMatrixLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureMatrixLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateIm2colLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureIm2colLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateFp2IntLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureFp2IntLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateInt2FpLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, + int64_t bankDepth, bool rushB); +void configureTransposeBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateSMatMulBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureSMatMulBallLegalizeForExportTarget(LLVMConversionTarget &target, bool stable); -void configureInt2FpLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); +void populateIm2colBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureIm2colBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateFp2IntBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureFp2IntBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateInt2FpBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureInt2FpBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateMatAddBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureMatAddBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateReluBallLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, + RewritePatternSet &patterns, + bool stable); +void configureReluBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); } // namespace mlir::buddy::buckyball void mlir::populateBuckyballLegalizeForLLVMExportPatterns( @@ -57,38 +63,41 @@ void mlir::populateBuckyballLegalizeForLLVMExportPatterns( int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum, bool includeFuncOperandForwarding, bool stable, bool rushB) { (void)bankWidthBytes; - (void)bankDepth; (void)bankNum; populateBaseLegalizeForLLVMExportPatterns( converter, patterns, includeFuncOperandForwarding, rushB); - mlir::buddy::buckyball::populateMatrixMatmulLegalizeForLLVMExportPatterns( - converter, patterns, stable, rushB); - mlir::buddy::buckyball::populateTransposeLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateMatrixLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateIm2colLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateFp2IntLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateInt2FpLegalizeForLLVMExportPatterns( + mlir::buddy::buckyball::populateSMatMulBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateTransposeBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateIm2colBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateFp2IntBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateInt2FpBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateMatAddBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateReluBallLegalizeForLLVMExportPatterns( converter, patterns, stable); } void mlir::configureBuckyballLegalizeForExportTarget( LLVMConversionTarget &target, bool stable) { configureBaseLegalizeForExportTarget(target); - mlir::buddy::buckyball::configureMatrixMatmulLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureTransposeLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureMatrixLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureIm2colLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureFp2IntLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureInt2FpLegalizeForExportTarget(target, - stable); + mlir::buddy::buckyball::configureSMatMulBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureTransposeBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureIm2colBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureFp2IntBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureInt2FpBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureMatAddBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureReluBallLegalizeForExportTarget(target, + stable); } diff --git a/examples/cores/pebble/configs/balldomains/balls/matrix/lane16.toml b/examples/cores/pebble/configs/balldomains/balls/smatmul/lane16.toml similarity index 72% rename from examples/cores/pebble/configs/balldomains/balls/matrix/lane16.toml rename to examples/cores/pebble/configs/balldomains/balls/smatmul/lane16.toml index 62327938c..c433ff3fc 100644 --- a/examples/cores/pebble/configs/balldomains/balls/matrix/lane16.toml +++ b/examples/cores/pebble/configs/balldomains/balls/smatmul/lane16.toml @@ -1,4 +1,4 @@ -# Pebble Matrix Ball Configuration (lane=16) +# Pebble SMatMul Ball Configuration (lane=16) [ball] InputNum = 16 diff --git a/examples/cores/pebble/configs/balldomains/default.toml b/examples/cores/pebble/configs/balldomains/default.toml index 74fb660eb..1c842350d 100644 --- a/examples/cores/pebble/configs/balldomains/default.toml +++ b/examples/cores/pebble/configs/balldomains/default.toml @@ -1,19 +1,25 @@ # Pebble ball domain configuration -ballNum = 5 +ballNum = 7 ballIdMappings = [ { ballId = 0, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 1, ballName = "MatrixBall", ballClass = "examples.balls.matrix.MatrixBall", config = "balls/matrix/lane16.toml", inBW = 2, outBW = 4 }, + { ballId = 1, ballName = "SMatMulBall", ballClass = "examples.balls.smatmul.SMatMulBall", config = "../../../../balls/smatmul/configs/lane16.toml", inBW = 2, outBW = 2 }, { ballId = 2, ballName = "Im2colBall", ballClass = "examples.balls.im2col.Im2colBall", config = "../../../../balls/im2col/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 3, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 4, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1 } + { ballId = 3, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1, mmioWriteBW = 4 }, + { ballId = 4, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1, mmioReadBW = 4 }, + { ballId = 5, ballName = "ReluBall", ballClass = "examples.balls.relu.ReluBall", config = "../../../../balls/relu/configs/default.toml", inBW = 1, outBW = 1 }, + { ballId = 6, ballName = "MatAddBall", ballClass = "examples.balls.matadd.MatAddBall", config = "../../../../balls/matadd/configs/default.toml", inBW = 2, outBW = 1 } ] ballISA = [ { mnemonic = "TRANSPOSE", funct7 = 49, bid = 0 }, - { mnemonic = "MATRIX", funct7 = 65, bid = 1 }, + { mnemonic = "SMATMUL_OS", funct7 = 65, bid = 1 }, + { mnemonic = "SMATMUL_WS", funct7 = 68, bid = 1 }, { mnemonic = "IM2COL", funct7 = 48, bid = 2 }, { mnemonic = "FP2INT", funct7 = 51, bid = 3 }, - { mnemonic = "INT2FP", funct7 = 52, bid = 4 } + { mnemonic = "INT2FP_TENSOR", funct7 = 52, bid = 4 }, + { mnemonic = "INT2FP_CHANNEL", funct7 = 54, bid = 4 }, + { mnemonic = "RELU", funct7 = 50, bid = 5 }, + { mnemonic = "MATADD", funct7 = 64, bid = 6 } ] diff --git a/examples/cores/pebble/core.toml b/examples/cores/pebble/configs/default.toml similarity index 64% rename from examples/cores/pebble/core.toml rename to examples/cores/pebble/configs/default.toml index a5caffdbc..82f4e9d48 100644 --- a/examples/cores/pebble/core.toml +++ b/examples/cores/pebble/configs/default.toml @@ -1,10 +1,10 @@ # Pebble core configuration -balldomain = "configs/balldomains/default.toml" -memdomain = "configs/memdomains/default.toml" -frontend = "configs/frontends/default.toml" -gpdomain = "configs/gpdomains/default.toml" -core = "configs/core/default.toml" +balldomain = "balldomains/default.toml" +memdomain = "memdomains/default.toml" +frontend = "frontends/default.toml" +gpdomain = "gpdomains/default.toml" +core = "core/default.toml" [rocketCore] xLen = 64 @@ -13,6 +13,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = true [rocketCore.mulDiv] enable = true diff --git a/examples/cores/pebble/configs/frontends/default.toml b/examples/cores/pebble/configs/frontends/default.toml index f9e95b0cc..e900d1c97 100644 --- a/examples/cores/pebble/configs/frontends/default.toml +++ b/examples/cores/pebble/configs/frontends/default.toml @@ -2,6 +2,7 @@ robEntries = 16 rsOutOfOrderResponse = true bankIdLen = 10 vbankIdUpperBound = 7 +sharedBankIdBase = 32 iterLen = 34 subRobEnable = false subRobDepth = 64 diff --git a/examples/cores/pebble/configs/memdomains/default.toml b/examples/cores/pebble/configs/memdomains/default.toml index a75402804..3c06d94e8 100644 --- a/examples/cores/pebble/configs/memdomains/default.toml +++ b/examples/cores/pebble/configs/memdomains/default.toml @@ -21,10 +21,10 @@ readChannel = 4 writeChannel = 4 [mmio] -enable = false -bankNum = 16 -bankEntries = 64 -bankWidth = 128 +enable = true +bankNum = 5 +bankEntries = 1024 +bankWidth = 8 readWidth = 8 [mem] diff --git a/examples/cores/pebble/emu/Cargo.lock b/examples/cores/pebble/emu/Cargo.lock deleted file mode 100644 index 09c1042df..000000000 --- a/examples/cores/pebble/emu/Cargo.lock +++ /dev/null @@ -1,1317 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "android_system_properties" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" -dependencies = [ - "libc", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "assert_cmd" -version = "2.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2aa3a22042e45de04255c7bf3626e239f450200fd0493c1e382263544b20aea6" -dependencies = [ - "anstyle", - "bstr", - "libc", - "predicates", - "predicates-core", - "predicates-tree", - "wait-timeout", -] - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-p2e" -version = "0.1.0" -dependencies = [ - "bebop-rtl-trace", - "bebop-uart", - "duct", - "log", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bebop-verilator" -version = "0.1.0" -dependencies = [ - "bebop-dasm", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-uart", - "cc", - "ctrlc", - "libc", - "snafu", -] - -[[package]] -name = "bemu-pebble" -version = "0.1.0" -dependencies = [ - "assert_cmd", - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-p2e", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "bebop-verilator", - "camino", - "cc", - "chrono", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "libtest-mimic", - "log", - "nix 0.29.0", - "once_cell", - "predicates", - "serde", - "serde_json", - "snafu", - "toml", - "walkdir", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "bstr" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f7dc094d718f2e1c1559ad110e27eeaae14a5465d3d56dd6dbd793079fbd530" -dependencies = [ - "memchr", - "regex-automata", - "serde_core", -] - -[[package]] -name = "bumpalo" -version = "3.20.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "chrono" -version = "0.4.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" -dependencies = [ - "iana-time-zone", - "num-traits", - "windows-link", -] - -[[package]] -name = "clap" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91e0c145792ef73a6ad36d27c75ac09f1832222a3c209689d90f534685ee5b7" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f09628afdcc538b57f3c6341e9c8e9970f18e4a481690a64974d7023bd33548b" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "difflib" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6184e33543162437515c2e2b48714794e37845ec9851711914eec9d308f6ebe8" - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "escape8259" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5692dd7b5a1978a5aeb0ce83b7655c58ca8efdcb79d21036ea249da95afec2c6" - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "float-cmp" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b09cf3155332e944990140d967ff5eceb70df778b34f77d8075db46e4704e6d8" -dependencies = [ - "num-traits", -] - -[[package]] -name = "futures-core" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" - -[[package]] -name = "futures-task" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" - -[[package]] -name = "futures-util" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" -dependencies = [ - "futures-core", - "futures-task", - "pin-project-lite", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "iana-time-zone" -version = "0.1.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "log", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.103" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" -dependencies = [ - "cfg-if", - "futures-util", - "wasm-bindgen", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "libtest-mimic" -version = "0.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "14e6ba06f0ade6e504aff834d7c34298e5155c6baca353cc6a4aaff2f9fd7f33" -dependencies = [ - "anstream", - "anstyle", - "clap", - "escape8259", -] - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "normalize-line-endings" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61807f77802ff30975e01f4f071c8ba10c022052f98b3294119f3e615d13e5be" - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "portable-atomic" -version = "1.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "predicates" -version = "3.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ada8f2932f28a27ee7b70dd6c1c39ea0675c55a36879ab92f3a715eaa1e63cfe" -dependencies = [ - "anstyle", - "difflib", - "float-cmp", - "normalize-line-endings", - "predicates-core", - "regex", -] - -[[package]] -name = "predicates-core" -version = "1.0.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cad38746f3166b4031b1a0d39ad9f954dd291e7854fcc0eed52ee41a0b50d144" - -[[package]] -name = "predicates-tree" -version = "1.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0de1b847b39c8131db0467e9df1ff60e6d0562ab8e9a16e568ad0fdb372e2f2" -dependencies = [ - "predicates-core", - "termtree", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "rustversion" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" - -[[package]] -name = "same-file" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "termtree" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f50febec83f5ee1df3015341d8bd429f2d1cc62bcba7ea2076759d315084683" - -[[package]] -name = "thiserror" -version = "2.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "wait-timeout" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" -dependencies = [ - "libc", -] - -[[package]] -name = "walkdir" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" -dependencies = [ - "same-file", - "winapi-util", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" -dependencies = [ - "cfg-if", - "once_cell", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn 2.0.119", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "winapi-util" -version = "0.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-link", - "windows-result", - "windows-strings", -] - -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/cores/pebble/emu/Cargo.toml b/examples/cores/pebble/emu/Cargo.toml deleted file mode 100644 index 810b986d6..000000000 --- a/examples/cores/pebble/emu/Cargo.toml +++ /dev/null @@ -1,74 +0,0 @@ -[package] -name = "bemu-pebble" -version = "0.1.0" -edition = "2021" -build = "../../../../bebop/src/nodes/bemu/build.rs" - -[lib] -name = "bebop_bemu" -path = "src/lib.rs" -crate-type = ["rlib", "cdylib"] - -[[bin]] -name = "bebop" -path = "../../../../bebop/src/main.rs" - -[[test]] -name = "test_bemu" -path = "../../../../bebop/tests/test_bemu.rs" -harness = false - -[[test]] -name = "test_verilator" -path = "../../../../bebop/tests/test_verilator.rs" -harness = false -required-features = ["verilator"] - -[dependencies] -snafu = "0.8" -once_cell = "1" -toml = "0.8" -bebop-dasm = { path = "../../../../bebop/src/nodes/lib/dasm" } -bebop-bank-hash = { path = "../../../../bebop/src/nodes/lib/bank-hash" } -bebop-bemu-profile = { path = "../../../../bebop/src/nodes/lib/bemu-profile" } -bebop-rushb = { path = "../../../../bebop/src/nodes/lib/rushB" } -bebop-fd-redirect = { path = "../../../../bebop/src/nodes/lib/fd-redirect" } -bebop-rtl-trace = { path = "../../../../bebop/src/nodes/lib/rtl-trace" } -bebop-uart = { path = "../../../../bebop/src/nodes/lib/uart" } -bebop-dtb = { path = "../../../../bebop/src/nodes/lib/dtb" } -bebop-elf = { path = "../../../../bebop/src/nodes/lib/elf" } -bebop-syscall = { path = "../../../../bebop/src/nodes/lib/syscall" } -bebop-verilator = { path = "../../../../bebop/src/nodes/verilator", optional = true } -bebop-p2e = { path = "../../../../bebop/src/nodes/p2e", optional = true } -clap = { version = "4", features = ["derive"] } -libc = "0.2" -log = "0.4" -env_logger = "0.11" -nix = { version = "0.29", features = ["fs", "mman", "signal", "process"] } -ctrlc = "3" -camino = "1.1" -serde = { version = "1", features = ["derive"] } -serde_json = "1" -duct = "0.13" - -[features] -default = ["bemu"] -bemu = [] -verilator = ["dep:bebop-verilator"] -p2e = ["dep:bebop-p2e"] -difftest = [] - -[build-dependencies] -cc = { version = "1", features = ["parallel"] } -toml = "0.8" - -[dev-dependencies] -clap = { version = "4", features = ["derive"] } -libtest-mimic = "0.8" -assert_cmd = "2" -walkdir = "2" -predicates = "3" -chrono = { version = "0.4", default-features = false, features = ["clock"] } -serde = { version = "1", features = ["derive"] } -serde_json = "1" -toml = "0.8" diff --git a/examples/cores/pebble/emu/src/chip.rs b/examples/cores/pebble/emu/src/chip.rs deleted file mode 100644 index 823c8aba6..000000000 --- a/examples/cores/pebble/emu/src/chip.rs +++ /dev/null @@ -1,42 +0,0 @@ -pub(crate) mod bank { - pub(crate) use crate::bank::*; -} - -use crate::inst::instruction::ExecContext; - -#[path = "../../../../balls/fp2int/emu/src/lib.rs"] -mod fp2int; -#[path = "../../../../balls/im2col/emu/src/lib.rs"] -mod im2col; -#[path = "../../../../balls/int2fp/emu/src/lib.rs"] -mod int2fp; -#[path = "../../../../balls/matrix/emu/src/lib.rs"] -mod matrix; -#[path = "../../../../balls/transpose/emu/src/lib.rs"] -mod transpose; - -pub fn execute_known( - ball_class: &str, - funct: u32, - xs1: u64, - xs2: u64, - ctx: &mut ExecContext, -) -> u64 { - fp2int::execute_known(ball_class, funct, xs1, xs2, ctx) - .or_else(|| im2col::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| int2fp::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| matrix::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| transpose::execute_known(ball_class, funct, xs1, xs2, ctx)) - .unwrap_or_else(|| { - panic!("no pebble BEMU ball implementation for ballClass={ball_class} funct7={funct}") - }) -} - -pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> u64 { - fp2int::cycles_after_issue(ball_class, funct, xs1, xs2) - .or_else(|| im2col::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| int2fp::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| matrix::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| transpose::cycles_after_issue(ball_class, funct, xs1, xs2)) - .unwrap_or_else(|| panic!("no pebble BEMU ball latency implementation for ballClass={ball_class} funct7={funct}")) -} diff --git a/examples/cores/pebble/emu/src/lib.rs b/examples/cores/pebble/emu/src/lib.rs deleted file mode 100644 index e78963189..000000000 --- a/examples/cores/pebble/emu/src/lib.rs +++ /dev/null @@ -1,9 +0,0 @@ -pub const BEMU_TOP_CONFIG: &str = "../../../../chips/pebble/configs/pebble.toml"; - -pub fn bemu_top_config() -> &'static str { - BEMU_TOP_CONFIG -} - -mod chip; - -include!("../../../../../bebop/src/nodes/bemu/src/lib.rs"); diff --git a/examples/cores/pebble/manifest.toml b/examples/cores/pebble/manifest.toml deleted file mode 100644 index 0373a8c69..000000000 --- a/examples/cores/pebble/manifest.toml +++ /dev/null @@ -1,5 +0,0 @@ -[core] -name = "pebble" -kind = "buckyball" -config = "core.toml" -compiler = "compiler" diff --git a/examples/cores/prefill/compiler/CMakeLists.txt b/examples/cores/prefill/compiler/CMakeLists.txt index 69545f11e..a088522f3 100644 --- a/examples/cores/prefill/compiler/CMakeLists.txt +++ b/examples/cores/prefill/compiler/CMakeLists.txt @@ -4,4 +4,8 @@ endif() set(BB_CORE_NAME prefill) set(BB_CORE_TARGET_PREFIX "") -add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/../../goban/compiler/src ${CMAKE_CURRENT_BINARY_DIR}/goban-src) + +set(BB_CORE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..) +get_filename_component(_BB_REPO_ROOT "${BB_CORE_DIR}/../../.." ABSOLUTE) +include_directories(${_BB_REPO_ROOT}/examples/balls) +add_subdirectory(src) diff --git a/examples/cores/prefill/compiler/src/CMakeLists.txt b/examples/cores/prefill/compiler/src/CMakeLists.txt new file mode 100644 index 000000000..74505498b --- /dev/null +++ b/examples/cores/prefill/compiler/src/CMakeLists.txt @@ -0,0 +1,60 @@ +if(NOT COMMAND add_mlir_library OR NOT COMMAND add_mlir_dialect) + message(FATAL_ERROR "MLIR CMake helpers must be loaded. Build as part of buddy-mlir.") +endif() + +message(STATUS "Building Prefill compiler Core package") +message(STATUS " Source: ${CMAKE_CURRENT_SOURCE_DIR}") +message(STATUS " Binary: ${CMAKE_CURRENT_BINARY_DIR}") + +get_filename_component(BB_REPO_ROOT ${CMAKE_CURRENT_LIST_DIR}/../../../../.. ABSOLUTE) +set(BB_COMPILER_COMMON_DIR ${BB_REPO_ROOT}/compiler/src) + +if(NOT EXISTS ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Buckyball.td) + message(FATAL_ERROR "Missing Buckyball compiler common dir: ${BB_COMPILER_COMMON_DIR}") +endif() + +include_directories(${BB_COMPILER_COMMON_DIR}) +include_directories(${BB_COMPILER_COMMON_DIR}/Dialect) +include_directories(${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball) +include_directories(${CMAKE_CURRENT_SOURCE_DIR}) +include_directories(${CMAKE_CURRENT_BINARY_DIR}) +include_directories(${CMAKE_CURRENT_SOURCE_DIR}/Dialect) +include_directories(${CMAKE_CURRENT_BINARY_DIR}/Dialect) +include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}/Dialect/Buckyball) + +set(PREFILL_BALL_COMPILER_DIALECT_DIRS + ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Dialect/Buckyball +) + +set(BB_BALL_BACKEND_INTRINSICS_DIRS) +foreach(PREFILL_BALL_COMPILER_DIALECT_DIR ${PREFILL_BALL_COMPILER_DIALECT_DIRS}) + if(NOT EXISTS ${PREFILL_BALL_COMPILER_DIALECT_DIR}) + message(FATAL_ERROR + "Missing prefill ball compiler dialect dir: ${PREFILL_BALL_COMPILER_DIALECT_DIR}") + endif() + include_directories(${PREFILL_BALL_COMPILER_DIALECT_DIR}) + get_filename_component(PREFILL_BALL_COMPILER_DIR + ${PREFILL_BALL_COMPILER_DIALECT_DIR}/../.. + ABSOLUTE) + list(APPEND BB_BALL_BACKEND_INTRINSICS_DIRS + ${PREFILL_BALL_COMPILER_DIR}/Backend/include) +endforeach() + +set(BB_BALL_BACKEND_INTRINSICS_DIRS ${BB_BALL_BACKEND_INTRINSICS_DIRS} + CACHE INTERNAL "Buckyball ball backend intrinsic directories") + +add_subdirectory(Dialect) +add_subdirectory(Conversion) +add_subdirectory(Target) + +add_custom_target(buckyball-compiler-${BB_CORE_NAME} + DEPENDS + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballToLLVMIRTranslation + ${BB_CORE_TARGET_PREFIX}LowerBuckyballPass + ${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass +) diff --git a/examples/cores/prefill/compiler/src/Conversion/CMakeLists.txt b/examples/cores/prefill/compiler/src/Conversion/CMakeLists.txt new file mode 100644 index 000000000..9ca69b1a2 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/CMakeLists.txt @@ -0,0 +1,2 @@ +add_subdirectory(LowerBuckyball) +add_subdirectory(LowerTileToBuckyball) diff --git a/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp new file mode 100644 index 000000000..afc4f31b0 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBanksPass.cpp @@ -0,0 +1,82 @@ +//===- AssignPhysicalBanksPass.cpp - Prefill physical bank assignment -----===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/WalkPatternRewriteDriver.h" + +#include "Buckyball/BuckyballDialect.h" + +using namespace mlir; + +namespace mlir::buddy { +void populateTransposeBallAssignPhysicalBankPatterns( + RewritePatternSet &patterns, PhysicalBankState &state); +void populateSMatMulBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateFp2IntBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +void populateInt2FpBallAssignPhysicalBankPatterns(RewritePatternSet &patterns, + PhysicalBankState &state); +} // namespace mlir::buddy + +namespace { + +class AssignPhysicalBanksPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(AssignPhysicalBanksPass) + AssignPhysicalBanksPass() = default; + AssignPhysicalBanksPass(const AssignPhysicalBanksPass &) {} + + StringRef getArgument() const final { return "assign-physical-banks"; } + StringRef getDescription() const final { + return "Assign physical banks for Prefill bank-SSA ops."; + } + + Option bankNum{*this, "bank_num", + llvm::cl::desc("Number of physical banks."), + llvm::cl::init(20)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (bankNum <= 0) { + func.emitError("assign-physical-banks: bank_num must be > 0"); + signalPassFailure(); + return; + } + + mlir::buddy::PhysicalBankState state(bankNum); + RewritePatternSet patterns(&getContext()); + mlir::buddy::addBaseAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateFp2IntBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateInt2FpBallAssignPhysicalBankPatterns(patterns, state); + mlir::buddy::populateTransposeBallAssignPhysicalBankPatterns(patterns, + state); + mlir::buddy::populateSMatMulBallAssignPhysicalBankPatterns(patterns, state); + + walkAndApplyPatterns(func, std::move(patterns)); + if (failed(mlir::buddy::verifyNoBankSSAOps(func))) { + signalPassFailure(); + return; + } + + if (!state.empty()) { + func.emitError("assign-physical-banks: leaked virtual bank handles"); + signalPassFailure(); + } + } +}; + +} // namespace + +void mlir::buddy::registerAssignPhysicalBanksPass() { + PassRegistration(); +} diff --git a/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt new file mode 100644 index 000000000..21e037cd9 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt @@ -0,0 +1,17 @@ +add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerBuckyballPass + ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp + ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/PhysicalBankState.cpp + ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/ReportBankUsagePass.cpp + ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPatterns.cpp + AssignPhysicalBanksPass.cpp + LowerBuckyballPass.cpp + LowerBuckyballToBankSSAPass.cpp + + LINK_LIBS PUBLIC + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms + MLIRLinalgDialect +) diff --git a/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp new file mode 100644 index 000000000..6ad04d6fa --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/LowerBuckyballPass.cpp @@ -0,0 +1,259 @@ +//===- LowerBuckyballPass.cpp - Prefill Buckyball lowering pass -----------===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Conversion/LLVMCommon/ConversionTarget.h" +#include "mlir/Conversion/LLVMCommon/TypeConverter.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/ControlFlow/IR/ControlFlowOps.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/LLVMIR/LLVMDialect.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/Pass/Pass.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" +#include "Buckyball/Transform.h" + +using namespace mlir; +using namespace ::buddy::buckyball; + +namespace { + +class LowerBuckyballToLLVMPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBuckyballToLLVMPass) + LowerBuckyballToLLVMPass() = default; + LowerBuckyballToLLVMPass(const LowerBuckyballToLLVMPass &) {} + + StringRef getArgument() const final { return "lower-buckyball"; } + StringRef getDescription() const final { + return "Lower Prefill Buckyball dialect ops."; + } + + Option bankWidthBytes{ + *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), + llvm::cl::init(16)}; + Option bankDepth{*this, "bank_depth", + llvm::cl::desc("Depth of each bank."), + llvm::cl::init(1024)}; + Option bankNum{*this, "bank_num", llvm::cl::desc("Number of banks."), + llvm::cl::init(20)}; + Option stable{*this, "stable", + llvm::cl::desc("Use stable LLVM Buckyball intrinsics."), + llvm::cl::init(false)}; + Option rushB{ + *this, "rushb", + llvm::cl::desc("Lower DMA operations to the rushB host ABI."), + llvm::cl::init(false)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + MLIRContext *context = &getContext(); + ModuleOp module = getOperation(); + LLVMTypeConverter converter(context); + RewritePatternSet patterns(context); + LLVMConversionTarget target(*context); + + configureBuckyballLegalizeForExportTarget(target, stable); + target.addLegalDialect(); + populateBuckyballLegalizeForLLVMExportPatterns( + converter, patterns, bankWidthBytes, bankDepth, bankNum, + /*includeFuncOperandForwarding=*/false, stable, rushB); + + ConversionConfig config; + config.allowPatternRollback = false; + if (failed(applyPartialConversion(module, target, std::move(patterns), + config))) + signalPassFailure(); + } +}; + +class LowerBankSSAToIntrinsicsPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBankSSAToIntrinsicsPass) + LowerBankSSAToIntrinsicsPass() = default; + LowerBankSSAToIntrinsicsPass(const LowerBankSSAToIntrinsicsPass &) {} + + StringRef getArgument() const final { return "lower-bank-ssa-to-intrinsics"; } + StringRef getDescription() const final { + return "Lower Prefill bank-SSA and Buckyball ops to intrinsic ops."; + } + + Option bankWidthBytes{ + *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), + llvm::cl::init(16)}; + Option bankDepth{*this, "bank_depth", + llvm::cl::desc("Depth of each bank."), + llvm::cl::init(1024)}; + Option bankNum{*this, "bank_num", llvm::cl::desc("Number of banks."), + llvm::cl::init(20)}; + Option stable{*this, "stable", + llvm::cl::desc("Use stable LLVM Buckyball intrinsics."), + llvm::cl::init(false)}; + Option rushB{ + *this, "rushb", + llvm::cl::desc("Lower DMA operations to the rushB host ABI."), + llvm::cl::init(false)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + MLIRContext *context = &getContext(); + ModuleOp module = getOperation(); + LLVMTypeConverter converter(context); + RewritePatternSet patterns(context); + LLVMConversionTarget target(*context); + + configureBuckyballLegalizeForExportTarget(target, stable); + target.addLegalDialect(); + populateBuckyballLegalizeForLLVMExportPatterns( + converter, patterns, bankWidthBytes, bankDepth, bankNum, + /*includeFuncOperandForwarding=*/false, stable, rushB); + + ConversionConfig config; + config.allowPatternRollback = false; + if (failed(applyPartialConversion(module, target, std::move(patterns), + config))) + signalPassFailure(); + } +}; + +static FlatSymbolRefAttr getOrInsertRushBFunction(OpBuilder &builder, + ModuleOp module, + StringRef name, + LLVM::LLVMFunctionType type) { + if (module.lookupSymbol(name)) + return FlatSymbolRefAttr::get(builder.getContext(), name); + + OpBuilder::InsertionGuard guard(builder); + builder.setInsertionPointToEnd(module.getBody()); + LLVM::LLVMFuncOp::create(builder, module.getLoc(), name, type, + LLVM::Linkage::External, false, LLVM::CConv::C); + return FlatSymbolRefAttr::get(builder.getContext(), name); +} + +class LowerBuckyballIntrinsicsToRushBPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID( + LowerBuckyballIntrinsicsToRushBPass) + LowerBuckyballIntrinsicsToRushBPass() = default; + LowerBuckyballIntrinsicsToRushBPass( + const LowerBuckyballIntrinsicsToRushBPass &) {} + + StringRef getArgument() const final { + return "lower-buckyball-intrinsics-to-rushb"; + } + StringRef getDescription() const final { + return "Lower Prefill Buckyball intrinsic ops to the rushB host ABI."; + } + + Option coreId{ + *this, "core_id", + llvm::cl::desc("Bind generated rushB calls to a tile Core."), + llvm::cl::init(-1)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry.insert(); + } + + void runOnOperation() override { + ModuleOp module = getOperation(); + SmallVector intrinsicOps; + module.walk([&](Operation *op) { + if (isa( + op)) + intrinsicOps.push_back(op); + }); + + OpBuilder builder(&getContext()); + Type i32Type = IntegerType::get(&getContext(), 32); + Type voidType = LLVM::LLVMVoidType::get(&getContext()); + FlatSymbolRefAttr selectCallee; + if (coreId >= 0) { + auto selectType = + LLVM::LLVMFunctionType::get(voidType, {i32Type, i32Type}); + selectCallee = getOrInsertRushBFunction( + builder, module, "rushb_select_accelerator", selectType); + } + llvm::SmallPtrSet boundFunctions; + for (Operation *op : intrinsicOps) { + if (coreId >= 0) { + if (auto function = op->getParentOfType(); + function && boundFunctions.insert(function).second) { + OpBuilder::InsertionGuard guard(builder); + builder.setInsertionPointToStart(&function.front()); + auto selected = LLVM::ConstantOp::create( + builder, function.getLoc(), i32Type, + builder.getI32IntegerAttr(static_cast(coreId))); + auto chip = + LLVM::ConstantOp::create(builder, function.getLoc(), i32Type, + builder.getI32IntegerAttr(0)); + LLVM::CallOp::create(builder, function.getLoc(), TypeRange{}, + selectCallee, ValueRange{selected, chip}); + } + } + builder.setInsertionPoint(op); + SmallVector operands; + StringRef name; + if (isa(op)) { + name = "rushb_mset"; + operands.append(op->getOperands().begin(), op->getOperands().end()); + } else if (isa(op) || isa(op)) { + name = isa(op) ? "rushb_mvin" : "rushb_mvout"; + operands.append(op->getOperands().begin(), op->getOperands().end()); + } else { + name = "rushb_custom"; + if (auto custom = dyn_cast(op)) { + operands.append(custom.getOperands().begin(), + custom.getOperands().end()); + operands.push_back(LLVM::ConstantOp::create( + builder, op->getLoc(), i32Type, + builder.getI32IntegerAttr(custom.getFunct7()))); + } else { + auto fence = cast(op); + operands.append(fence->getOperands().begin(), + fence->getOperands().end()); + operands.push_back(LLVM::ConstantOp::create( + builder, op->getLoc(), i32Type, builder.getI32IntegerAttr(0))); + } + } + + SmallVector argumentTypes; + for (Value operand : operands) + argumentTypes.push_back(operand.getType()); + auto type = LLVM::LLVMFunctionType::get(voidType, argumentTypes); + auto callee = getOrInsertRushBFunction(builder, module, name, type); + LLVM::CallOp::create(builder, op->getLoc(), TypeRange{}, callee, + operands); + op->erase(); + } + } +}; + +} // namespace + +void mlir::buddy::registerLowerBuckyballPass() { + PassRegistration(); +} + +void mlir::buddy::registerLowerBankSSAToIntrinsicsPass() { + PassRegistration(); + PassRegistration(); +} diff --git a/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp new file mode 100644 index 000000000..f51a201cb --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/LowerBuckyball/LowerBuckyballToBankSSAPass.cpp @@ -0,0 +1,96 @@ +//===- LowerBuckyballToBankSSAPass.cpp - Prefill bank-SSA lowering --------===// + +#include "Conversion/LowerBuckyball/LowerBuckyball.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/Dialect/Utils/StructuredOpsUtils.h" +#include "mlir/IR/AffineMap.h" +#include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/PatternMatch.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/GreedyPatternRewriteDriver.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" + +using namespace mlir; + +namespace mlir::buddy { +void populateSMatMulBallLowerBuckyballToBankSSAPatterns( + RewritePatternSet &patterns); +} // namespace mlir::buddy + +namespace { + +class MemTransposeToLinalgPattern + : public OpRewritePattern<::buddy::buckyball::MemTransposeOp> { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(::buddy::buckyball::MemTransposeOp op, + PatternRewriter &rewriter) const override { + auto input = dyn_cast(op.getInput().getType()); + auto output = dyn_cast(op.getOutput().getType()); + if (!input || !output || !input.hasStaticShape() || + !output.hasStaticShape() || input.getRank() != 2 || + output.getRank() != 2 || output.getShape()[0] != input.getShape()[1] || + output.getShape()[1] != input.getShape()[0]) + return op.emitError("requires matching static rank-2 transpose memrefs"); + + MLIRContext *context = rewriter.getContext(); + AffineMap identity = AffineMap::getMultiDimIdentityMap(2, context); + AffineMap transpose = AffineMap::get( + 2, 0, {rewriter.getAffineDimExpr(1), rewriter.getAffineDimExpr(0)}, + context); + SmallVector iterators(2, + utils::IteratorType::parallel); + rewriter.create( + op.getLoc(), TypeRange{}, ValueRange{op.getInput()}, + ValueRange{op.getOutput()}, ArrayRef{identity, transpose}, + iterators, [](OpBuilder &builder, Location location, ValueRange args) { + builder.create(location, args[0]); + }); + rewriter.eraseOp(op); + return success(); + } +}; + +} // namespace + +namespace { + +class LowerBuckyballToBankSSAPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerBuckyballToBankSSAPass) + + StringRef getArgument() const final { return "lower-buckyball-to-bank-ssa"; } + StringRef getDescription() const final { + return "Lower Prefill Buckyball ops to explicit bank-SSA ops."; + } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry.insert(); + } + + void runOnOperation() override { + RewritePatternSet patterns(&getContext()); + patterns.add(&getContext()); + mlir::buddy::populateSMatMulBallLowerBuckyballToBankSSAPatterns(patterns); + if (failed(applyPatternsGreedily(getOperation(), std::move(patterns)))) + signalPassFailure(); + } +}; + +} // namespace + +void mlir::buddy::registerLowerBuckyballToBankSSAPass() { + PassRegistration(); +} diff --git a/examples/cores/prefill/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt b/examples/cores/prefill/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt new file mode 100644 index 000000000..6b73e8283 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt @@ -0,0 +1,9 @@ +add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass + LowerTileToBuckyball.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp + + LINK_LIBS PUBLIC + BuddyTile + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball + MLIRLinalgDialect +) diff --git a/examples/cores/prefill/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp b/examples/cores/prefill/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp new file mode 100644 index 000000000..8ee394969 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp @@ -0,0 +1,120 @@ +//===- LowerTileToBuckyball.cpp - Prefill tile pass registration ----------===// +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +//===----------------------------------------------------------------------===// + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/Linalg/IR/Linalg.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/BuiltinOps.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/DialectConversion.h" + +#include "Buckyball/BuckyballDialect.h" +#include "Buckyball/BuckyballOps.h" +#include "Conversion/LowerTileToBuckyball/LowerTileToBuckyball.h" +#include "Tile/TileDialect.h" +#include "Tile/TileOps.h" +#include "Tile/Transform.h" + +using namespace mlir; +using namespace ::buddy::buckyball; +namespace tile = ::buddy::tile; + +namespace mlir::buddy { +void populateSMatMulBallTileLoweringPatterns(RewritePatternSet &patterns, + int64_t bankWidthBytes, + int64_t bankDepth, + int64_t bankNum); +} // namespace mlir::buddy + +namespace { +class TileTransposeLowering : public OpRewritePattern { +public: + using OpRewritePattern::OpRewritePattern; + + LogicalResult matchAndRewrite(tile::TileTransposeOp op, + PatternRewriter &rewriter) const override { + auto inputType = dyn_cast(op.getAMemArray().getType()); + auto outputType = dyn_cast(op.getBMemArray().getType()); + if (!inputType || !outputType || !inputType.hasStaticShape() || + !outputType.hasStaticShape() || inputType.getRank() != 2 || + outputType.getRank() != 2) + return op.emitError("requires static rank-2 input and output memrefs"); + if (outputType.getShape()[0] != inputType.getShape()[1] || + outputType.getShape()[1] != inputType.getShape()[0] || + inputType.getElementType() != outputType.getElementType()) + return op.emitError("requires a matching transposed output memref"); + + rewriter.create(op.getLoc(), op.getAMemArray(), + op.getBMemArray()); + rewriter.eraseOp(op); + return success(); + } +}; + +class LowerTileToBuckyballPass + : public PassWrapper> { +public: + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerTileToBuckyballPass) + StringRef getArgument() const final { return "convert-tile-to-buckyball"; } + StringRef getDescription() const final { + return "Convert Tile operations for the Prefill Core."; + } + LowerTileToBuckyballPass() = default; + LowerTileToBuckyballPass(const LowerTileToBuckyballPass &) {} + + Option bankWidthBytes{ + *this, "bank_width", llvm::cl::desc("Physical bank width in bytes."), + llvm::cl::init(16)}; + Option bankDepth{*this, "bank_depth", + llvm::cl::desc("Bank depth (rows per bank)."), + llvm::cl::init(1024)}; + Option bankNum{*this, "bank_num", + llvm::cl::desc("Number of physical banks."), + llvm::cl::init(20)}; + + void getDependentDialects(DialectRegistry ®istry) const override { + registry + .insert(); + } + + void runOnOperation() override { + MLIRContext *context = &getContext(); + ConversionTarget target(*context); + target.addLegalDialect<::buddy::buckyball::BuckyballDialect, + memref::MemRefDialect, arith::ArithDialect, + scf::SCFDialect, func::FuncDialect, + linalg::LinalgDialect>(); + target.addIllegalOp(); + target.addIllegalOp(); + + RewritePatternSet patterns(context); + mlir::buddy::populateSMatMulBallTileLoweringPatterns( + patterns, bankWidthBytes, bankDepth, bankNum); + patterns.add(context); + if (failed(applyPartialConversion(getOperation(), target, + std::move(patterns)))) + signalPassFailure(); + } +}; +} // namespace + +void mlir::buddy::registerLowerTileToBuckyballPass() { + PassRegistration(); +} diff --git a/examples/cores/prefill/compiler/src/Dialect/Buckyball/CMakeLists.txt b/examples/cores/prefill/compiler/src/Dialect/Buckyball/CMakeLists.txt new file mode 100644 index 000000000..c8dacc918 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Dialect/Buckyball/CMakeLists.txt @@ -0,0 +1,17 @@ +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) +mlir_tablegen(Buckyball.h.inc -gen-op-decls) +mlir_tablegen(Buckyball.cpp.inc -gen-op-defs) +mlir_tablegen(BuckyballTypes.h.inc -gen-typedef-decls -typedefs-dialect=buckyball) +mlir_tablegen(BuckyballTypes.cpp.inc -gen-typedef-defs -typedefs-dialect=buckyball) +mlir_tablegen(BuckyballDialect.h.inc -gen-dialect-decls -dialect=buckyball) +mlir_tablegen(BuckyballDialect.cpp.inc -gen-dialect-defs -dialect=buckyball) +add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}MLIRBuckyballIncGen) +add_mlir_doc(PrefillBuckyball ${BB_CORE_TARGET_PREFIX}Buckyball Dialects/ + -gen-dialect-doc) + +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) +mlir_tablegen(BuckyballConversions.inc -gen-llvmir-conversions) +add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen) + +add_subdirectory(IR) +add_subdirectory(Transforms) diff --git a/examples/cores/prefill/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt b/examples/cores/prefill/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt new file mode 100644 index 000000000..a2e7abc2e --- /dev/null +++ b/examples/cores/prefill/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt @@ -0,0 +1,16 @@ +add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyball + ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/IR/BuckyballDialect.cpp + ${BB_TARGET_REGISTRY_SOURCE} + + LINK_LIBS PUBLIC + MLIRIR + MLIRLLVMDialect +) + +target_include_directories(${BB_CORE_TARGET_PREFIX}BuddyBuckyball PUBLIC + ${BB_COMPILER_COMMON_DIR} + ${BB_COMPILER_COMMON_DIR}/Dialect + ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball + ${CMAKE_CURRENT_BINARY_DIR}/.. + ${CMAKE_CURRENT_BINARY_DIR}/../../.. +) diff --git a/compiler/src/Dialect/Buckyball/BuckyballCarrier.td b/examples/cores/prefill/compiler/src/Dialect/Buckyball/PrefillBuckyball.td similarity index 53% rename from compiler/src/Dialect/Buckyball/BuckyballCarrier.td rename to examples/cores/prefill/compiler/src/Dialect/Buckyball/PrefillBuckyball.td index 8ca66ca3e..adb07a766 100644 --- a/compiler/src/Dialect/Buckyball/BuckyballCarrier.td +++ b/examples/cores/prefill/compiler/src/Dialect/Buckyball/PrefillBuckyball.td @@ -1,4 +1,4 @@ -//===-- BuckyballCarrier.td - Generic Buckyball carrier ops -*- tablegen -*-===// +//===-- PrefillBuckyball.td - Prefill Buckyball dialect ops -*- tablegen -*-===// // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. @@ -14,18 +14,13 @@ // //===----------------------------------------------------------------------===// -#ifndef BUCKYBALL_CARRIER -#define BUCKYBALL_CARRIER +#ifndef PREFILL_BUCKYBALL_OPS +#define PREFILL_BUCKYBALL_OPS -def InstOp : Buckyball_Op<"inst"> { - let summary = "Generic Buckyball custom instruction carrier"; - let description = [{ - Emits a Buckyball custom instruction with explicit rs1, rs2 and funct7. - }]; - let arguments = (ins I64:$rs1, I64:$rs2, - ConfinedAttr, IntMaxValue<127>]>:$funct7); - let assemblyFormat = "$rs1 `,` $rs2 attr-dict `:` type($rs1)"; -} +include "Buckyball.td" +include "Fp2IntBall.td" +include "Int2FpBall.td" +include "TransposeBall.td" +include "SMatMulBall.td" -#endif // BUCKYBALL_CARRIER +#endif // PREFILL_BUCKYBALL_OPS diff --git a/examples/cores/prefill/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt b/examples/cores/prefill/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt new file mode 100644 index 000000000..715c53bd2 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt @@ -0,0 +1,14 @@ +add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms + ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp + ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + ${BB_REPO_ROOT}/examples/balls/smatmul/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp + LegalizeForLLVMExport.cpp + + LINK_LIBS PUBLIC + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball + MLIRIR + MLIRLLVMCommonConversion + MLIRLLVMDialect +) diff --git a/examples/cores/prefill/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/cores/prefill/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp new file mode 100644 index 000000000..cd5b981c4 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -0,0 +1,76 @@ +//===- LegalizeForLLVMExport.cpp - Prefill Buckyball LLVM lowering --------===// +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +//===----------------------------------------------------------------------===// + +#include "Buckyball/Transform.h" +#include "Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.h" + +using namespace mlir; +using namespace buddy::buckyball::legalize; + +namespace mlir::buddy::buckyball { +void populateTransposeBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureTransposeBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateSMatMulBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureSMatMulBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateFp2IntBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureFp2IntBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +void populateInt2FpBallLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, + int64_t bankDepth, bool rushB); +void configureInt2FpBallLegalizeForExportTarget(LLVMConversionTarget &target, + bool stable); +} // namespace mlir::buddy::buckyball + +void mlir::populateBuckyballLegalizeForLLVMExportPatterns( + LLVMTypeConverter &converter, RewritePatternSet &patterns, + int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum, + bool includeFuncOperandForwarding, bool stable, bool rushB) { + (void)bankWidthBytes; + (void)bankNum; + + populateBaseLegalizeForLLVMExportPatterns( + converter, patterns, includeFuncOperandForwarding, rushB); + mlir::buddy::buckyball::populateFp2IntBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateInt2FpBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateTransposeBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); + mlir::buddy::buckyball::populateSMatMulBallLegalizeForLLVMExportPatterns( + converter, patterns, stable, bankDepth, rushB); +} + +void mlir::configureBuckyballLegalizeForExportTarget( + LLVMConversionTarget &target, bool stable) { + configureBaseLegalizeForExportTarget(target); + mlir::buddy::buckyball::configureFp2IntBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureInt2FpBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureTransposeBallLegalizeForExportTarget(target, + stable); + mlir::buddy::buckyball::configureSMatMulBallLegalizeForExportTarget(target, + stable); +} diff --git a/examples/cores/prefill/compiler/src/Dialect/CMakeLists.txt b/examples/cores/prefill/compiler/src/Dialect/CMakeLists.txt new file mode 100644 index 000000000..ec181148e --- /dev/null +++ b/examples/cores/prefill/compiler/src/Dialect/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(Buckyball) diff --git a/examples/cores/prefill/compiler/src/Target/CMakeLists.txt b/examples/cores/prefill/compiler/src/Target/CMakeLists.txt new file mode 100644 index 000000000..39d31dc9b --- /dev/null +++ b/examples/cores/prefill/compiler/src/Target/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(LLVMIR) diff --git a/examples/cores/prefill/compiler/src/Target/LLVMIR/CMakeLists.txt b/examples/cores/prefill/compiler/src/Target/LLVMIR/CMakeLists.txt new file mode 100644 index 000000000..0ca0f41c5 --- /dev/null +++ b/examples/cores/prefill/compiler/src/Target/LLVMIR/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(Dialect) diff --git a/examples/cores/prefill/compiler/src/Target/LLVMIR/Dialect/Buckyball/CMakeLists.txt b/examples/cores/prefill/compiler/src/Target/LLVMIR/Dialect/Buckyball/CMakeLists.txt new file mode 100644 index 000000000..06b99171b --- /dev/null +++ b/examples/cores/prefill/compiler/src/Target/LLVMIR/Dialect/Buckyball/CMakeLists.txt @@ -0,0 +1,15 @@ +add_mlir_translation_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballToLLVMIRTranslation + ${BB_COMPILER_COMMON_DIR}/Target/LLVMIR/Dialect/Buckyball/BuckyballToLLVMIRTranslation.cpp + + DEPENDS + ${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen + intrinsics_gen + + LINK_COMPONENTS + Core + + LINK_LIBS PUBLIC + MLIRIR + MLIRSupport + ${BB_CORE_TARGET_PREFIX}BuddyBuckyball +) diff --git a/examples/cores/prefill/compiler/src/Target/LLVMIR/Dialect/CMakeLists.txt b/examples/cores/prefill/compiler/src/Target/LLVMIR/Dialect/CMakeLists.txt new file mode 100644 index 000000000..ec181148e --- /dev/null +++ b/examples/cores/prefill/compiler/src/Target/LLVMIR/Dialect/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(Buckyball) diff --git a/examples/cores/prefill/configs/balldomains/default.toml b/examples/cores/prefill/configs/balldomains/default.toml index abcbc946c..7bd4b310b 100644 --- a/examples/cores/prefill/configs/balldomains/default.toml +++ b/examples/cores/prefill/configs/balldomains/default.toml @@ -1,19 +1,23 @@ # Poly Prefill Core Ball Domain Configuration +# +# Prefill amortizes setup across long prompt batches. Its target therefore +# exposes the systolic matrix Ball and intentionally omits decode-only vector +# and image-convolution operations. -ballNum = 5 +ballNum = 4 ballIdMappings = [ - { ballId = 0, ballName = "VecBall", ballClass = "examples.balls.vector.VecBall", config = "../../../../balls/vector/configs/default.toml", inBW = 2, outBW = 4 }, + { ballId = 0, ballName = "SMatMulBall", ballClass = "examples.balls.smatmul.SMatMulBall", config = "../../../../balls/smatmul/configs/default.toml", inBW = 2, outBW = 4 }, { ballId = 1, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1 }, { ballId = 2, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 3, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 4, ballName = "Im2colBall", ballClass = "examples.balls.im2col.Im2colBall", config = "../../../../balls/im2col/configs/default.toml", inBW = 1, outBW = 1 } + { ballId = 3, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 } ] ballISA = [ - { mnemonic = "MATMUL_WARP16", funct7 = 64, bid = 0 }, + { mnemonic = "SMATMUL_OS", funct7 = 65, bid = 0 }, + { mnemonic = "SMATMUL_WS", funct7 = 68, bid = 0 }, { mnemonic = "FP2INT", funct7 = 51, bid = 1 }, - { mnemonic = "INT2FP", funct7 = 52, bid = 2 }, - { mnemonic = "TRANSPOSE", funct7 = 49, bid = 3 }, - { mnemonic = "IM2COL", funct7 = 48, bid = 4 } + { mnemonic = "INT2FP_TENSOR", funct7 = 52, bid = 2 }, + { mnemonic = "INT2FP_CHANNEL", funct7 = 54, bid = 2 }, + { mnemonic = "TRANSPOSE", funct7 = 49, bid = 3 } ] diff --git a/examples/cores/prefill/core.toml b/examples/cores/prefill/configs/default.toml similarity index 66% rename from examples/cores/prefill/core.toml rename to examples/cores/prefill/configs/default.toml index 9fdcc2211..529cf53e3 100644 --- a/examples/cores/prefill/core.toml +++ b/examples/cores/prefill/configs/default.toml @@ -1,10 +1,10 @@ # Prefill Core: Goban accelerator with a depth-oriented frontend. -balldomain = "configs/balldomains/default.toml" -memdomain = "configs/memdomains/default.toml" -frontend = "configs/frontends/default.toml" -gpdomain = "configs/gpdomains/default.toml" -core = "configs/core/default.toml" +balldomain = "balldomains/default.toml" +memdomain = "memdomains/default.toml" +frontend = "frontends/default.toml" +gpdomain = "gpdomains/default.toml" +core = "core/default.toml" [rocketCore] xLen = 64 @@ -13,6 +13,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = false [rocketCore.mulDiv] enable = true diff --git a/examples/cores/prefill/configs/frontends/default.toml b/examples/cores/prefill/configs/frontends/default.toml index 7adaefebd..418994d6b 100644 --- a/examples/cores/prefill/configs/frontends/default.toml +++ b/examples/cores/prefill/configs/frontends/default.toml @@ -1,7 +1,8 @@ robEntries = 16 rsOutOfOrderResponse = true bankIdLen = 10 -vbankIdUpperBound = 15 +vbankIdUpperBound = 31 +sharedBankIdBase = 32 iterLen = 34 subRobEnable = false subRobDepth = 64 diff --git a/examples/cores/prefill/configs/memdomains/default.toml b/examples/cores/prefill/configs/memdomains/default.toml index 1b9757a6d..962ca3e7b 100644 --- a/examples/cores/prefill/configs/memdomains/default.toml +++ b/examples/cores/prefill/configs/memdomains/default.toml @@ -1,16 +1,17 @@ # Poly Prefill Core Memory Domain Configuration # Private (per-core) memory parameters. Shared memory lives in the tile TOML. +# Long prompt batches need more resident activation and weight tiles. [bank] -num = 16 +num = 32 width = 128 -entries = 1024 +entries = 2048 maskLen = 16 channel = 7 [dma] nXacts = 8 -burstMaxBytes = 64 +burstMaxBytes = 1024 busWidth = 128 maxInFlightMemReqs = 16 diff --git a/examples/cores/prefill/emu/Cargo.lock b/examples/cores/prefill/emu/Cargo.lock deleted file mode 100644 index 7f861ae94..000000000 --- a/examples/cores/prefill/emu/Cargo.lock +++ /dev/null @@ -1,932 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bemu-goban" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "camino", - "cc", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "log", - "nix 0.29.0", - "once_cell", - "serde", - "serde_json", - "snafu", - "toml", -] - -[[package]] -name = "bemu-prefill" -version = "0.1.0" -dependencies = [ - "bemu-goban", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "clap" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "find-msvc-tools" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de" - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "portable-atomic" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "thiserror" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/cores/prefill/emu/Cargo.toml b/examples/cores/prefill/emu/Cargo.toml deleted file mode 100644 index 39f341ba9..000000000 --- a/examples/cores/prefill/emu/Cargo.toml +++ /dev/null @@ -1,13 +0,0 @@ -[package] -name = "bemu-prefill" -version = "0.1.0" -edition = "2021" -build = "build.rs" - -[lib] -name = "bemu_prefill" -path = "src/lib.rs" -crate-type = ["rlib", "cdylib"] - -[dependencies] -core_bemu = { package = "bemu-goban", path = "../../goban/emu" } diff --git a/examples/cores/prefill/emu/build.rs b/examples/cores/prefill/emu/build.rs deleted file mode 100644 index 627f300a0..000000000 --- a/examples/cores/prefill/emu/build.rs +++ /dev/null @@ -1,14 +0,0 @@ -use std::{env, fs, path::PathBuf}; - -fn main() { - let manifest = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); - let build_root = manifest.join("target/release/build"); - let Ok(entries) = fs::read_dir(build_root) else { return }; - for entry in entries.flatten() { - let lib_dir = entry.path().join("out/spike_install/lib"); - if lib_dir.join("libriscv.so").is_file() { - println!("cargo:rustc-link-arg-cdylib=-Wl,-rpath,{}", lib_dir.display()); - break; - } - } -} diff --git a/examples/cores/prefill/emu/src/lib.rs b/examples/cores/prefill/emu/src/lib.rs deleted file mode 100644 index c11148620..000000000 --- a/examples/cores/prefill/emu/src/lib.rs +++ /dev/null @@ -1 +0,0 @@ -pub use core_bemu::*; diff --git a/examples/cores/prefill/manifest.toml b/examples/cores/prefill/manifest.toml deleted file mode 100644 index 3c1eca165..000000000 --- a/examples/cores/prefill/manifest.toml +++ /dev/null @@ -1,5 +0,0 @@ -[core] -name = "prefill" -kind = "buckyball" -config = "core.toml" -compiler = "compiler" diff --git a/examples/cores/rocket/compiler/CMakeLists.txt b/examples/cores/rocket/compiler/CMakeLists.txt new file mode 100644 index 000000000..95412c193 --- /dev/null +++ b/examples/cores/rocket/compiler/CMakeLists.txt @@ -0,0 +1,12 @@ +if(NOT COMMAND add_mlir_library OR NOT COMMAND add_mlir_dialect) + message(FATAL_ERROR "MLIR CMake helpers must be loaded. Build as part of buddy-mlir.") +endif() + +# Rocket has no Ball lowering policy. Reuse the Ball-free compiler base rather +# than importing another Core's dialect or lowering implementation. +set(BB_CORE_NAME rocket) +set(BB_CORE_TARGET_PREFIX "") + +get_filename_component(_BB_REPO_ROOT "${CMAKE_CURRENT_LIST_DIR}/../../../.." ABSOLUTE) +add_subdirectory("${_BB_REPO_ROOT}/examples/cores/toy/compiler/src" + "${CMAKE_CURRENT_BINARY_DIR}/base") diff --git a/examples/cores/rocket/configs/balldomains/default.toml b/examples/cores/rocket/configs/balldomains/default.toml new file mode 100644 index 000000000..a23e23157 --- /dev/null +++ b/examples/cores/rocket/configs/balldomains/default.toml @@ -0,0 +1,9 @@ +# Rocket-only core: no Buckyball balls. + +ballNum = 0 + +ballIdMappings = [ +] + +ballISA = [ +] diff --git a/examples/cores/rocket/core.toml b/examples/cores/rocket/configs/default.toml similarity index 95% rename from examples/cores/rocket/core.toml rename to examples/cores/rocket/configs/default.toml index 84c941cdf..10d75eb8d 100644 --- a/examples/cores/rocket/core.toml +++ b/examples/cores/rocket/configs/default.toml @@ -7,6 +7,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = false [rocketCore.mulDiv] enable = true diff --git a/examples/cores/rocket/configs/memdomains/default.toml b/examples/cores/rocket/configs/memdomains/default.toml new file mode 100644 index 000000000..783da930a --- /dev/null +++ b/examples/cores/rocket/configs/memdomains/default.toml @@ -0,0 +1,32 @@ +# Rocket-only stub memdomain for bank_params / chip select pairing. +# Not referenced by rocket configs/default.toml (tiles stay rocket-only). + +[bank] +num = 1 +width = 128 +entries = 1 +maskLen = 16 +channel = 1 + +[dma] +nXacts = 1 +burstMaxBytes = 1024 +busWidth = 128 +maxInFlightMemReqs = 16 + +[tlb] +size = 1 + +[tma] +readChannel = 1 +writeChannel = 1 + +[mmio] +enable = false +bankNum = 1 +bankEntries = 1 +bankWidth = 128 +readWidth = 8 + +[mem] +addrLen = 39 diff --git a/examples/cores/rocket/manifest.toml b/examples/cores/rocket/manifest.toml deleted file mode 100644 index 1da4fc745..000000000 --- a/examples/cores/rocket/manifest.toml +++ /dev/null @@ -1,4 +0,0 @@ -[core] -name = "rocket" -kind = "rocket" -config = "core.toml" diff --git a/examples/cores/toy/compiler/CMakeLists.txt b/examples/cores/toy/compiler/CMakeLists.txt index 78c8d9a5d..1893a2fc2 100644 --- a/examples/cores/toy/compiler/CMakeLists.txt +++ b/examples/cores/toy/compiler/CMakeLists.txt @@ -5,4 +5,8 @@ endif() set(BB_CORE_NAME toy) set(BB_CORE_TARGET_PREFIX "") +set(BB_CORE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..) +get_filename_component(_BB_REPO_ROOT "${BB_CORE_DIR}/../../.." ABSOLUTE) +include_directories(${_BB_REPO_ROOT}/examples/balls) + add_subdirectory(src) diff --git a/examples/cores/toy/compiler/include/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballPatterns.h b/examples/cores/toy/compiler/include/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballPatterns.h index 09643684f..c07fc3ade 100644 --- a/examples/cores/toy/compiler/include/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballPatterns.h +++ b/examples/cores/toy/compiler/include/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballPatterns.h @@ -15,20 +15,6 @@ void populateToyAssignPhysicalBankPatterns(RewritePatternSet &patterns, void populateToyLowerBuckyballToLLVMPatterns(RewritePatternSet &patterns); void configureToyLowerBuckyballToLLVMTarget(LLVMConversionTarget &target); void populateToyLowerBuckyballToBankSSAPatterns(RewritePatternSet &patterns); -void populateToyMemTransposeToBankSSAPatterns(RewritePatternSet &patterns); - -void populateVectorAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateTransposeAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateMatrixAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateIm2colAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateFp2IntAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); -void populateInt2FpAssignPhysicalBankPatterns(RewritePatternSet &patterns, - PhysicalBankState &state); } // namespace buddy } // namespace mlir diff --git a/examples/cores/toy/compiler/src/Backend/include/IntrinsicsRISCVBuckyball.td b/examples/cores/toy/compiler/src/Backend/include/IntrinsicsRISCVBuckyball.td index e6675ae2d..dd695fb46 100644 --- a/examples/cores/toy/compiler/src/Backend/include/IntrinsicsRISCVBuckyball.td +++ b/examples/cores/toy/compiler/src/Backend/include/IntrinsicsRISCVBuckyball.td @@ -1,11 +1,3 @@ //===- IntrinsicsRISCVBuckyball.td - Buckyball intrinsics ----------------===// include "IntrinsicsRISCVBuckyballBase.td" - -include "IntrinsicsRISCVVecBall.td" -include "IntrinsicsRISCVReluBall.td" -include "IntrinsicsRISCVTransposeBall.td" -include "IntrinsicsRISCVIm2colBall.td" -include "IntrinsicsRISCVMatrixBall.td" -include "IntrinsicsRISCVFp2IntBall.td" -include "IntrinsicsRISCVInt2FpBall.td" diff --git a/examples/cores/toy/compiler/src/CMakeLists.txt b/examples/cores/toy/compiler/src/CMakeLists.txt index fa156b4b5..511255e84 100644 --- a/examples/cores/toy/compiler/src/CMakeLists.txt +++ b/examples/cores/toy/compiler/src/CMakeLists.txt @@ -24,15 +24,7 @@ include_directories(${CMAKE_CURRENT_SOURCE_DIR}/Dialect) include_directories(${CMAKE_CURRENT_BINARY_DIR}/Dialect) include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}/Dialect/Buckyball) -set(TOY_BALL_COMPILER_DIALECT_DIRS - ${BB_REPO_ROOT}/examples/balls/vector/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/im2col/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball - ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball -) +set(TOY_BALL_COMPILER_DIALECT_DIRS) set(BB_CORE_BACKEND_INTRINSICS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/Backend/include @@ -55,7 +47,6 @@ endforeach() set(BB_BALL_BACKEND_INTRINSICS_DIRS ${BB_BALL_BACKEND_INTRINSICS_DIRS} CACHE INTERNAL "Buckyball ball backend intrinsic directories") -# Add subdirectories add_subdirectory(Dialect) add_subdirectory(Conversion) add_subdirectory(Target) diff --git a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt index 3cd3b9926..56c1286ae 100644 --- a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt +++ b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/CMakeLists.txt @@ -2,16 +2,9 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerBuckyballPass ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/BaseAssignPhysicalBankPatterns.cpp ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/PhysicalBankState.cpp ${BB_COMPILER_COMMON_DIR}/Conversion/LowerBuckyball/ReportBankUsagePass.cpp - ${BB_REPO_ROOT}/examples/balls/vector/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp - ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp - ${BB_REPO_ROOT}/examples/balls/im2col/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp - ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp - ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Conversion/LowerBuckyball/AssignPhysicalBankPatterns.cpp AssignPhysicalBanksPass.cpp LowerBuckyballPass.cpp LowerBuckyballToBankSSAPass.cpp - MemTransposeToBankSSAPatterns.cpp Patterns/ToyAssignPhysicalBankPatterns.cpp Patterns/ToyLowerBuckyballToBankSSAPatterns.cpp Patterns/ToyLowerBuckyballToLLVMPatterns.cpp @@ -20,8 +13,6 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerBuckyballPass ${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms ) -# CMake's archive rule appends members, so a chip switch can retain stale -# objects from the previously configured LowerBuckyballPass target. add_custom_command(TARGET ${BB_CORE_TARGET_PREFIX}LowerBuckyballPass PRE_LINK COMMAND ${CMAKE_COMMAND} -E rm -f $) diff --git a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp deleted file mode 100644 index fe94c8eeb..000000000 --- a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/MemTransposeToBankSSAPatterns.cpp +++ /dev/null @@ -1,197 +0,0 @@ -//===- MemTransposeToBankSSAPatterns.cpp - mem_transpose -> Bank*/linalg --===// - -#include "Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballPatterns.h" - -#include "mlir/Dialect/Arith/IR/Arith.h" -#include "mlir/Dialect/Linalg/IR/Linalg.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Dialect/Utils/StructuredOpsUtils.h" -#include "mlir/IR/AffineMap.h" -#include "mlir/IR/BuiltinTypes.h" -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" -#include "Utils/BankUtils.h" - -using namespace mlir; -using namespace ::buddy::buckyball; - -namespace { - -constexpr int64_t kBankWidthBytes = 16; -constexpr int64_t kRowTile = 16; -constexpr int64_t kMaxBankTransposeElems = 64 * 64; - -static size_t elemsPerBankRow(Type elemType) { - unsigned bitWidth = elemType.getIntOrFloatBitWidth(); - if (bitWidth == 0 || bitWidth % 8 != 0) - return 0; - return kBankWidthBytes / (bitWidth / 8); -} - -static LogicalResult rowStrideDiv16(MemRefType ty, int64_t &out) { - SmallVector strides; - int64_t offset = 0; - if (failed(ty.getStridesAndOffset(strides, offset)) || strides.size() < 2) - return failure(); - if (ShapedType::isDynamic(strides[0]) || strides[0] <= 0 || - strides[0] % 16 != 0) - return failure(); - if (ShapedType::isDynamic(strides[1]) || strides[1] != 1) - return failure(); - out = strides[0] / 16; - return success(); -} - -static int64_t cdiv(int64_t a, int64_t b) { return (a + b - 1) / b; } - -static void emitLinalgTranspose(PatternRewriter &rewriter, Location loc, - Value input, Value output) { - MLIRContext *ctx = rewriter.getContext(); - AffineMap id = AffineMap::getMultiDimIdentityMap(2, ctx); - AffineMap tr = AffineMap::get( - 2, 0, {rewriter.getAffineDimExpr(1), rewriter.getAffineDimExpr(0)}, ctx); - SmallVector iters(2, utils::IteratorType::parallel); - rewriter.create( - loc, TypeRange{}, ValueRange{input}, ValueRange{output}, - ArrayRef{id, tr}, iters, - [](OpBuilder &b, Location loc, ValueRange args) { - b.create(loc, args[0]); - }); -} - -class MemTransposeToBankSSAPattern : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(MemTransposeOp op, - PatternRewriter &rewriter) const override { - Location loc = op.getLoc(); - Value input = op.getInput(); - Value output = op.getOutput(); - auto inputType = dyn_cast(input.getType()); - auto outputType = dyn_cast(output.getType()); - if (!inputType || !outputType || !inputType.hasStaticShape() || - !outputType.hasStaticShape()) - return op.emitError("requires static input and output memrefs"); - - auto inShape = inputType.getShape(); - auto outShape = outputType.getShape(); - int64_t rows = inShape[0]; - int64_t cols = inShape[1]; - if (outShape[0] != cols || outShape[1] != rows) - return op.emitError("output shape must transpose the input shape"); - - Type elemTy = inputType.getElementType(); - int64_t elemBits = elemTy.getIntOrFloatBitWidth(); - if (elemBits == 0 || elemBits % 8 != 0) - return op.emitError("unsupported transpose element type"); - bool hwElem = elemBits == 8 || elemBits == 32; - if (!hwElem || rows * cols > kMaxBankTransposeElems) { - emitLinalgTranspose(rewriter, loc, input, output); - rewriter.eraseOp(op); - return success(); - } - - size_t elemsPerRow = elemsPerBankRow(elemTy); - if (elemsPerRow == 0) - return op.emitError("unsupported transpose element type"); - - size_t maxColTile = elemsPerRow * 4; - size_t colsAlign = - ((size_t)cols + elemsPerRow - 1) / elemsPerRow * elemsPerRow; - size_t colTile = colsAlign <= maxColTile ? colsAlign : maxColTile; - if (colTile == 0) - return op.emitError("tile width is smaller than one bank row"); - - int64_t rowsPad = cdiv(rows, kRowTile) * kRowTile; - int64_t colsPad = cdiv(cols, (int64_t)colTile) * (int64_t)colTile; - bool pad = rowsPad != rows || colsPad != cols; - int64_t dummyStride = 0; - bool inContig = succeeded(rowStrideDiv16(inputType, dummyStride)); - bool outContig = succeeded(rowStrideDiv16(outputType, dummyStride)); - bool materialize = pad || !inContig || !outContig; - - Value inBuf = input; - Value outBuf = output; - if (materialize) { - auto inPadTy = MemRefType::get({rowsPad, colsPad}, elemTy); - auto outPadTy = MemRefType::get({colsPad, rowsPad}, elemTy); - inBuf = rewriter.create(loc, inPadTy); - outBuf = rewriter.create(loc, outPadTy); - Value z = rewriter.create( - loc, elemTy, rewriter.getZeroAttr(elemTy)); - rewriter.create(loc, z, inBuf); - rewriter.create(loc, z, outBuf); - Value inView = rewriter.create( - loc, inBuf, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(rows), - rewriter.getIndexAttr(cols)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, input, inView); - } - - int64_t groups = (int64_t)(colTile / elemsPerRow); - int64_t strideIn = colsPad / 16; - int64_t strideOut = rowsPad / 16; - for (int64_t r0 = 0; r0 < rowsPad; r0 += kRowTile) { - for (int64_t c0 = 0; c0 < colsPad; c0 += (int64_t)colTile) { - Value inTile = rewriter.create( - loc, inBuf, - SmallVector{rewriter.getIndexAttr(r0), - rewriter.getIndexAttr(c0)}, - SmallVector{rewriter.getIndexAttr(kRowTile), - rewriter.getIndexAttr((int64_t)colTile)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value outTile = rewriter.create( - loc, outBuf, - SmallVector{rewriter.getIndexAttr(c0), - rewriter.getIndexAttr(r0)}, - SmallVector{rewriter.getIndexAttr((int64_t)colTile), - rewriter.getIndexAttr(kRowTile)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - - Value src = allocBank(rewriter, loc, 1, groups); - Value dst = allocBank(rewriter, loc, 1, groups); - Value loaded = mvinBank(rewriter, loc, inTile, src, kRowTile, strideIn); - Value transposed = rewriter.create( - loc, dst.getType(), loaded, dst, - createI64Const(rewriter, loc, kRowTile), - createI64Const(rewriter, loc, elemBits)); - mvoutBank(rewriter, loc, outTile, transposed, (int64_t)colTile, - strideOut); - releaseBank(rewriter, loc, loaded); - releaseBank(rewriter, loc, transposed); - } - } - - if (materialize) { - Value outView = rewriter.create( - loc, outBuf, - SmallVector{rewriter.getIndexAttr(0), - rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(cols), - rewriter.getIndexAttr(rows)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - rewriter.create(loc, outView, output); - rewriter.create(loc, inBuf); - rewriter.create(loc, outBuf); - } - - rewriter.eraseOp(op); - return success(); - } -}; - -} // namespace - -void mlir::buddy::populateToyMemTransposeToBankSSAPatterns( - RewritePatternSet &patterns) { - patterns.add(patterns.getContext()); -} diff --git a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyAssignPhysicalBankPatterns.cpp b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyAssignPhysicalBankPatterns.cpp index 364f507b3..13bc120b1 100644 --- a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyAssignPhysicalBankPatterns.cpp +++ b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyAssignPhysicalBankPatterns.cpp @@ -6,10 +6,6 @@ using namespace mlir; void mlir::buddy::populateToyAssignPhysicalBankPatterns( RewritePatternSet &patterns, mlir::buddy::PhysicalBankState &state) { - populateVectorAssignPhysicalBankPatterns(patterns, state); - populateTransposeAssignPhysicalBankPatterns(patterns, state); - populateMatrixAssignPhysicalBankPatterns(patterns, state); - populateIm2colAssignPhysicalBankPatterns(patterns, state); - populateFp2IntAssignPhysicalBankPatterns(patterns, state); - populateInt2FpAssignPhysicalBankPatterns(patterns, state); + (void)patterns; + (void)state; } diff --git a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballToBankSSAPatterns.cpp b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballToBankSSAPatterns.cpp index bc16c52c4..f90c9d971 100644 --- a/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballToBankSSAPatterns.cpp +++ b/examples/cores/toy/compiler/src/Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballToBankSSAPatterns.cpp @@ -1,291 +1,10 @@ -//===- ToyLowerBuckyballToBankSSAPatterns.cpp - Toy bank-SSA patterns -----===// +//===- ToyLowerBuckyballToBankSSAPatterns.cpp - Toy bank SSA hooks --------===// #include "Conversion/LowerBuckyball/Patterns/ToyLowerBuckyballPatterns.h" -#include "mlir/Dialect/Arith/IR/Arith.h" -#include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Dialect/SCF/IR/SCF.h" -#include "mlir/IR/BuiltinTypes.h" -#include "mlir/IR/PatternMatch.h" - -#include "Buckyball/BuckyballOps.h" - using namespace mlir; -using namespace ::buddy::buckyball; - -namespace { - -static Value cstI64(OpBuilder &builder, Location loc, uint64_t value) { - return builder.create(loc, builder.getI64Type(), - builder.getI64IntegerAttr(value)); -} - -static Value cstF32(OpBuilder &builder, Location loc, float value) { - return builder.create(loc, builder.getF32Type(), - builder.getF32FloatAttr(value)); -} - -static Value packF32BitsAsI64(OpBuilder &builder, Location loc, Value f32Val) { - Value i32Bits = - builder.create(loc, builder.getI32Type(), f32Val); - return builder.create(loc, builder.getI64Type(), i32Bits); -} - -static Value buildTileAbsMax(PatternRewriter &rewriter, Location loc, Value mem, - uint64_t rows, uint64_t cols) { - auto maxTy = MemRefType::get({1}, rewriter.getF32Type()); - Value maxBuf = rewriter.create(loc, maxTy); - - Value zeroIdx = rewriter.create(loc, 0); - Value oneIdx = rewriter.create(loc, 1); - Value rowsIdx = rewriter.create(loc, rows); - Value colsIdx = rewriter.create(loc, cols); - Value zeroF32 = cstF32(rewriter, loc, 0.0f); - rewriter.create(loc, zeroF32, maxBuf, ValueRange{zeroIdx}); - - auto rowLoop = rewriter.create(loc, zeroIdx, rowsIdx, oneIdx); - rewriter.setInsertionPointToStart(rowLoop.getBody()); - auto colLoop = rewriter.create(loc, zeroIdx, colsIdx, oneIdx); - rewriter.setInsertionPointToStart(colLoop.getBody()); - - Value elem = rewriter.create( - loc, mem, - ValueRange{rowLoop.getInductionVar(), colLoop.getInductionVar()}); - if (elem.getType() != rewriter.getF32Type()) - elem = rewriter.create(loc, rewriter.getF32Type(), elem); - Value neg = rewriter.create(loc, elem); - Value abs = rewriter.create(loc, elem, neg); - Value cur = rewriter.create(loc, maxBuf, ValueRange{zeroIdx}); - Value upd = rewriter.create(loc, cur, abs); - rewriter.create(loc, upd, maxBuf, ValueRange{zeroIdx}); - - rewriter.setInsertionPointAfter(rowLoop); - Value result = - rewriter.create(loc, maxBuf, ValueRange{zeroIdx}); - rewriter.create(loc, maxBuf); - return result; -} - -static Value buildQuantScale(PatternRewriter &rewriter, Location loc, - Value maxAbs) { - Value zeroF32 = cstF32(rewriter, loc, 0.0f); - Value oneF32 = cstF32(rewriter, loc, 1.0f); - Value qmaxF32 = cstF32(rewriter, loc, 127.0f); - Value hasData = rewriter.create(loc, arith::CmpFPredicate::OGT, - maxAbs, zeroF32); - Value scaled = rewriter.create(loc, qmaxF32, maxAbs); - return rewriter.create(loc, hasData, scaled, oneF32); -} - -static LogicalResult getStaticRowStrideDiv16(MemRefType type, uint64_t &out) { - SmallVector strides; - int64_t offset = 0; - if (failed(type.getStridesAndOffset(strides, offset)) || strides.size() < 2) - return failure(); - if (ShapedType::isDynamic(strides[0]) || strides[0] <= 0 || - strides[0] % 16 != 0) - return failure(); - if (ShapedType::isDynamic(strides[1]) || strides[1] != 1) - return failure(); - out = static_cast(strides[0] / 16); - return success(); -} - -class MatrixMatmulToBankSSAPattern : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(MatrixMatmulOp op, - PatternRewriter &rewriter) const override { - Location loc = op.getLoc(); - Value aMem = op.getAMemArray(); - Value bMem = op.getBMemArray(); - Value cMem = op.getCMemArray(); - - auto aTy = dyn_cast(aMem.getType()); - auto bTy = dyn_cast(bMem.getType()); - auto cTy = dyn_cast(cMem.getType()); - if (!aTy || !bTy || !cTy || !aTy.hasStaticShape() || - !bTy.hasStaticShape() || !cTy.hasStaticShape()) - return rewriter.notifyMatchFailure( - op, "bb_matmul requires static rank-2 memrefs"); - - uint64_t m = aTy.getShape()[0]; - uint64_t k = aTy.getShape()[1]; - uint64_t kb = bTy.getShape()[0]; - uint64_t n = bTy.getShape()[1]; - if (k != kb) - return rewriter.notifyMatchFailure(op, "inner dimensions must match"); - if (cTy.getShape()[0] != static_cast(m) || - cTy.getShape()[1] != static_cast(n)) - return rewriter.notifyMatchFailure(op, "output dimensions must match"); - if (m % 16 != 0 || k % 16 != 0 || n % 16 != 0) - return rewriter.notifyMatchFailure( - op, - "buckyball.matrix_matmul requires M, K and N to be multiples of 16"); - - uint64_t strideB = 0; - uint64_t strideC = 0; - if (failed(getStaticRowStrideDiv16(bTy, strideB))) - return rewriter.notifyMatchFailure( - op, "B requires static strided<[row,1]> and row % 16 == 0"); - if (failed(getStaticRowStrideDiv16(cTy, strideC))) - return rewriter.notifyMatchFailure( - op, "C requires static strided<[row,1]> and row % 16 == 0"); - - constexpr uint64_t tile = 16; - uint64_t depthC = tile; - - OpBuilder::InsertionGuard guard(rewriter); - Value zeroIdx = rewriter.create(loc, 0); - Value mUpper = rewriter.create(loc, m); - Value nUpper = rewriter.create(loc, n); - Value kUpper = rewriter.create(loc, k); - Value step = rewriter.create(loc, tile); - - auto mLoop = rewriter.create(loc, zeroIdx, mUpper, step); - rewriter.setInsertionPointToStart(mLoop.getBody()); - Value mIv = mLoop.getInductionVar(); - - auto nLoop = rewriter.create(loc, zeroIdx, nUpper, step); - rewriter.setInsertionPointToStart(nLoop.getBody()); - Value nIv = nLoop.getInductionVar(); - - Value aTile = rewriter.create( - loc, aMem, SmallVector{mIv, rewriter.getIndexAttr(0)}, - SmallVector{rewriter.getIndexAttr(tile), - rewriter.getIndexAttr(k)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value bTile = rewriter.create( - loc, bMem, SmallVector{rewriter.getIndexAttr(0), nIv}, - SmallVector{rewriter.getIndexAttr(k), - rewriter.getIndexAttr(tile)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - Value cTile = rewriter.create( - loc, cMem, SmallVector{mIv, nIv}, - SmallVector{rewriter.getIndexAttr(tile), - rewriter.getIndexAttr(tile)}, - SmallVector{rewriter.getIndexAttr(1), - rewriter.getIndexAttr(1)}); - - Value maxA = buildTileAbsMax(rewriter, loc, aTile, tile, k); - Value maxB = buildTileAbsMax(rewriter, loc, bTile, k, tile); - Value scaleAF32 = buildQuantScale(rewriter, loc, maxA); - Value scaleBF32 = buildQuantScale(rewriter, loc, maxB); - Value scaleABits = packF32BitsAsI64(rewriter, loc, scaleAF32); - Value scaleBBits = packF32BitsAsI64(rewriter, loc, scaleBF32); - Value oneF32 = cstF32(rewriter, loc, 1.0f); - Value scaleProd = rewriter.create(loc, scaleAF32, scaleBF32); - Value dequantScaleF32 = - rewriter.create(loc, oneF32, scaleProd); - Value dequantScaleBits = packF32BitsAsI64(rewriter, loc, dequantScaleF32); - - auto cI32 = rewriter.create(loc, rewriter.getI64Type()); - cI32->setAttr("col", rewriter.getI64IntegerAttr(4)); - - auto zeroI32Ty = - MemRefType::get({(int64_t)tile, (int64_t)tile}, rewriter.getI32Type()); - Value zeroI32Buf = rewriter.create(loc, zeroI32Ty); - Value zeroI32 = rewriter.create( - loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(0)); - Value tileIdx = rewriter.create(loc, tile); - Value oneIdx = rewriter.create(loc, 1); - auto zRow = rewriter.create(loc, zeroIdx, tileIdx, oneIdx); - rewriter.setInsertionPointToStart(zRow.getBody()); - auto zCol = rewriter.create(loc, zeroIdx, tileIdx, oneIdx); - rewriter.setInsertionPointToStart(zCol.getBody()); - rewriter.create( - loc, zeroI32, zeroI32Buf, - ValueRange{zRow.getInductionVar(), zCol.getInductionVar()}); - rewriter.setInsertionPointAfter(zRow); - - auto cZero = rewriter.create( - loc, rewriter.getI64Type(), zeroI32Buf, cI32.getBank(), - cstI64(rewriter, loc, depthC), cstI64(rewriter, loc, 1)); - rewriter.create(loc, zeroI32Buf); - - SmallVector kIterArgs = {cZero.getBankOut()}; - auto kLoop = rewriter.create( - loc, zeroIdx, kUpper, step, kIterArgs, - [&](OpBuilder &b, Location bodyLoc, Value kIv, ValueRange args) { - Value cIn = args[0]; - - Value aKTile = b.create( - bodyLoc, aMem, SmallVector{mIv, kIv}, - SmallVector{b.getIndexAttr(tile), - b.getIndexAttr(tile)}, - SmallVector{b.getIndexAttr(1), b.getIndexAttr(1)}); - Value bKTile = b.create( - bodyLoc, bMem, SmallVector{kIv, nIv}, - SmallVector{b.getIndexAttr(tile), - b.getIndexAttr(tile)}, - SmallVector{b.getIndexAttr(1), b.getIndexAttr(1)}); - - uint64_t strideAK = k / tile; - - auto aFp32 = b.create(bodyLoc, b.getI64Type()); - aFp32->setAttr("col", b.getI64IntegerAttr(4)); - auto bFp32 = b.create(bodyLoc, b.getI64Type()); - bFp32->setAttr("col", b.getI64IntegerAttr(4)); - auto aI8 = b.create(bodyLoc, b.getI64Type()); - auto bI8 = b.create(bodyLoc, b.getI64Type()); - auto aI8T = b.create(bodyLoc, b.getI64Type()); - - auto aLoad = b.create( - bodyLoc, b.getI64Type(), aKTile, aFp32.getBank(), - cstI64(b, bodyLoc, tile), cstI64(b, bodyLoc, strideAK)); - auto bLoad = b.create( - bodyLoc, b.getI64Type(), bKTile, bFp32.getBank(), - cstI64(b, bodyLoc, tile), cstI64(b, bodyLoc, strideB)); - - auto aQuant = b.create( - bodyLoc, b.getI64Type(), aLoad.getBankOut(), aI8.getBank(), - cstI64(b, bodyLoc, tile), scaleABits); - auto bQuant = b.create( - bodyLoc, b.getI64Type(), bLoad.getBankOut(), bI8.getBank(), - cstI64(b, bodyLoc, tile), scaleBBits); - b.create(bodyLoc, aLoad.getBankOut()); - b.create(bodyLoc, bLoad.getBankOut()); - - auto aTrans = b.create( - bodyLoc, b.getI64Type(), aQuant.getOutBankOut(), aI8T.getBank(), - cstI64(b, bodyLoc, tile), cstI64(b, bodyLoc, 8)); - b.create(bodyLoc, aQuant.getOutBankOut()); - - auto cMul = b.create( - bodyLoc, b.getI64Type(), aTrans.getOutBankOut(), - bQuant.getOutBankOut(), cIn, cstI64(b, bodyLoc, tile), - cstI64(b, bodyLoc, 0)); - b.create(bodyLoc, aTrans.getOutBankOut()); - b.create(bodyLoc, bQuant.getOutBankOut()); - b.create(bodyLoc, ValueRange{cMul.getWrBankOut()}); - }); - - Value cAcc = kLoop.getResult(0); - auto cFp32 = rewriter.create(loc, rewriter.getI64Type()); - cFp32->setAttr("col", rewriter.getI64IntegerAttr(4)); - auto cDequant = rewriter.create( - loc, rewriter.getI64Type(), cAcc, cFp32.getBank(), - cstI64(rewriter, loc, depthC), dequantScaleBits); - rewriter.create(loc, cAcc); - - auto cStore = rewriter.create( - loc, rewriter.getI64Type(), cTile, cDequant.getOutBankOut(), - cstI64(rewriter, loc, depthC), cstI64(rewriter, loc, strideC)); - rewriter.create(loc); - rewriter.create(loc, cStore.getBankOut()); - - rewriter.eraseOp(op); - return success(); - } -}; - -} // namespace void mlir::buddy::populateToyLowerBuckyballToBankSSAPatterns( RewritePatternSet &patterns) { - patterns.add(patterns.getContext()); - populateToyMemTransposeToBankSSAPatterns(patterns); + (void)patterns; } diff --git a/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt b/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt index 61a092d94..82b282ef1 100644 --- a/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt +++ b/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/CMakeLists.txt @@ -1,6 +1,5 @@ add_mlir_library(${BB_CORE_TARGET_PREFIX}LowerTileToBuckyballPass LowerTileToBuckyball.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Conversion/LowerTileToBuckyball/TileMatMulPatterns.cpp LINK_LIBS PUBLIC BuddyTile diff --git a/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp b/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp index ee39e7094..c7dcd8dc2 100644 --- a/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp +++ b/examples/cores/toy/compiler/src/Conversion/LowerTileToBuckyball/LowerTileToBuckyball.cpp @@ -27,17 +27,13 @@ #include "mlir/Transforms/DialectConversion.h" #include "Buckyball/BuckyballDialect.h" -#include "Buckyball/BuckyballOps.h" #include "Tile/TileDialect.h" #include "Tile/TileOps.h" #include "Tile/Transform.h" using namespace mlir; -using namespace ::buddy::buckyball; -using namespace ::buddy::tile; namespace tile = ::buddy::tile; using mlir::buddy::kDefaultBankWidthBytes; -using mlir::buddy::populateMatrixTileMatMulPatterns; namespace { @@ -45,84 +41,6 @@ static Value cstF32(OpBuilder &b, Location loc, float v) { return b.create(loc, b.getF32Type(), b.getF32FloatAttr(v)); } -static Value packF32BitsAsI64(OpBuilder &b, Location loc, Value f32Val) { - Value i32Bits = b.create(loc, b.getI32Type(), f32Val); - return b.create(loc, b.getI64Type(), i32Bits); -} - -static Value buildTileAbsMax(PatternRewriter &rewriter, Location loc, Value mem, - uint64_t rows, uint64_t cols) { - auto maxTy = MemRefType::get({1}, rewriter.getF32Type()); - Value maxBuf = rewriter.create(loc, maxTy); - - Value zeroIdx = rewriter.create(loc, 0); - Value oneIdx = rewriter.create(loc, 1); - Value rowsIdx = rewriter.create(loc, rows); - Value colsIdx = rewriter.create(loc, cols); - Value zeroF32 = cstF32(rewriter, loc, 0.0f); - - rewriter.create(loc, zeroF32, maxBuf, ValueRange{zeroIdx}); - - auto rowLoop = rewriter.create(loc, zeroIdx, rowsIdx, oneIdx); - rewriter.setInsertionPointToStart(rowLoop.getBody()); - auto colLoop = rewriter.create(loc, zeroIdx, colsIdx, oneIdx); - rewriter.setInsertionPointToStart(colLoop.getBody()); - - Value elem = rewriter.create( - loc, mem, - ValueRange{rowLoop.getInductionVar(), colLoop.getInductionVar()}); - if (elem.getType() != rewriter.getF32Type()) - elem = rewriter.create(loc, rewriter.getF32Type(), elem); - Value neg = rewriter.create(loc, elem); - Value abs = rewriter.create(loc, elem, neg); - Value cur = rewriter.create(loc, maxBuf, ValueRange{zeroIdx}); - Value upd = rewriter.create(loc, cur, abs); - rewriter.create(loc, upd, maxBuf, ValueRange{zeroIdx}); - - rewriter.setInsertionPointAfter(rowLoop); - Value result = - rewriter.create(loc, maxBuf, ValueRange{zeroIdx}); - rewriter.create(loc, maxBuf); - return result; -} - -static Value buildQuantScale(PatternRewriter &rewriter, Location loc, - Value maxAbs) { - Value zeroF32 = cstF32(rewriter, loc, 0.0f); - Value oneF32 = cstF32(rewriter, loc, 1.0f); - Value qmaxF32 = cstF32(rewriter, loc, 127.0f); - Value hasData = rewriter.create(loc, arith::CmpFPredicate::OGT, - maxAbs, zeroF32); - Value scaled = rewriter.create(loc, qmaxF32, maxAbs); - return rewriter.create(loc, hasData, scaled, oneF32); -} - -class TileTransposeLowering : public OpRewritePattern { -public: - using OpRewritePattern::OpRewritePattern; - - LogicalResult matchAndRewrite(tile::TileTransposeOp op, - PatternRewriter &rewriter) const override { - auto inputType = dyn_cast(op.getAMemArray().getType()); - auto outputType = dyn_cast(op.getBMemArray().getType()); - if (!inputType || !outputType || !inputType.hasStaticShape() || - !outputType.hasStaticShape()) - return op.emitError("requires static input and output memrefs"); - if (inputType.getRank() != 2 || outputType.getRank() != 2) - return op.emitError("requires rank-2 memrefs"); - if (outputType.getShape()[0] != inputType.getShape()[1] || - outputType.getShape()[1] != inputType.getShape()[0]) - return op.emitError("output shape must transpose the input shape"); - if (inputType.getElementType() != outputType.getElementType()) - return op.emitError("input/output element types must match"); - - rewriter.create(op.getLoc(), op.getAMemArray(), - op.getBMemArray()); - rewriter.eraseOp(op); - return success(); - } -}; - class TileConv2dLowering : public OpRewritePattern { public: explicit TileConv2dLowering(MLIRContext *context, int64_t bankWidthBytes, @@ -249,22 +167,13 @@ class TileConv2dLowering : public OpRewritePattern { int64_t bankWidthBytes, bankDepth; }; -void populateToyLocalTilePatterns(RewritePatternSet &patterns, - int64_t bankWidthBytes, int64_t bankDepth, - int64_t bankNum) { - patterns.add(patterns.getContext()); - patterns.add(patterns.getContext(), bankWidthBytes, - bankDepth, bankNum); -} - } // namespace void mlir::populateLowerTileToBuckyballConversionPatterns( RewritePatternSet &patterns, int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum) { - populateMatrixTileMatMulPatterns(patterns, bankWidthBytes, bankDepth, - bankNum); - populateToyLocalTilePatterns(patterns, bankWidthBytes, bankDepth, bankNum); + patterns.add(patterns.getContext(), bankWidthBytes, + bankDepth, bankNum); } namespace { diff --git a/examples/cores/toy/compiler/src/Dialect/Buckyball/CMakeLists.txt b/examples/cores/toy/compiler/src/Dialect/Buckyball/CMakeLists.txt index c8f457a72..aa8da8d59 100644 --- a/examples/cores/toy/compiler/src/Dialect/Buckyball/CMakeLists.txt +++ b/examples/cores/toy/compiler/src/Dialect/Buckyball/CMakeLists.txt @@ -1,5 +1,5 @@ # Generate dialect and ops from the Toy dialect assembly. -set(LLVM_TARGET_DEFINITIONS ToyBuckyball.td) +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) mlir_tablegen(Buckyball.h.inc -gen-op-decls) mlir_tablegen(Buckyball.cpp.inc -gen-op-defs) mlir_tablegen(BuckyballTypes.h.inc -gen-typedef-decls -typedefs-dialect=buckyball) @@ -11,7 +11,7 @@ add_mlir_doc(ToyBuckyball ${BB_CORE_TARGET_PREFIX}Buckyball Dialects/ -gen-dialect-doc) # Generate LLVM IR conversions -set(LLVM_TARGET_DEFINITIONS ToyBuckyball.td) +set(LLVM_TARGET_DEFINITIONS ${BB_BUCKYBALL_DIALECT_TD}) mlir_tablegen(BuckyballConversions.inc -gen-llvmir-conversions) add_public_tablegen_target(${BB_CORE_TARGET_PREFIX}BuddyBuckyballConversionsIncGen) diff --git a/examples/cores/toy/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt b/examples/cores/toy/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt index da76ba536..a2e7abc2e 100644 --- a/examples/cores/toy/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt +++ b/examples/cores/toy/compiler/src/Dialect/Buckyball/IR/CMakeLists.txt @@ -1,5 +1,6 @@ add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyball ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/IR/BuckyballDialect.cpp + ${BB_TARGET_REGISTRY_SOURCE} LINK_LIBS PUBLIC MLIRIR diff --git a/examples/cores/toy/compiler/src/Dialect/Buckyball/ToyBuckyball.td b/examples/cores/toy/compiler/src/Dialect/Buckyball/ToyBuckyball.td index 252e875c8..303415d74 100644 --- a/examples/cores/toy/compiler/src/Dialect/Buckyball/ToyBuckyball.td +++ b/examples/cores/toy/compiler/src/Dialect/Buckyball/ToyBuckyball.td @@ -14,7 +14,8 @@ // //===----------------------------------------------------------------------===// // -// This is the Toy compiler's Buckyball dialect assembly. +// Toy mounts leftover balls: Gemmini, Trace, Mxfp2Int. +// Gemmini/Trace/Mxfp2Int lower outside this dialect (Gemmini dialect / custom). // //===----------------------------------------------------------------------===// @@ -23,13 +24,4 @@ include "Buckyball.td" -// Ball-specific operations -include "VecBall.td" -include "ReluBall.td" -include "TransposeBall.td" -include "Im2colBall.td" -include "MatrixBall.td" -include "Fp2IntBall.td" -include "Int2FpBall.td" - #endif // TOY_BUCKYBALL_OPS diff --git a/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt b/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt index 566c7a2fd..061c8e8b9 100644 --- a/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt +++ b/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/CMakeLists.txt @@ -1,12 +1,5 @@ add_mlir_dialect_library(${BB_CORE_TARGET_PREFIX}BuddyBuckyballTransforms ${BB_COMPILER_COMMON_DIR}/Dialect/Buckyball/Transforms/LegalizeForLLVMExportBase.cpp - ${BB_REPO_ROOT}/examples/balls/vector/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/transpose/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/im2col/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/fp2int/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/int2fp/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/relu/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp - ${BB_REPO_ROOT}/examples/balls/matrix/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp LegalizeForLLVMExport.cpp LINK_LIBS PUBLIC diff --git a/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp b/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp index 67e120d10..c03708cef 100644 --- a/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp +++ b/examples/cores/toy/compiler/src/Dialect/Buckyball/Transforms/LegalizeForLLVMExport.cpp @@ -1,18 +1,4 @@ //===- LegalizeForLLVMExport.cpp - Toy Buckyball LLVM lowering ------------===// -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// -//===----------------------------------------------------------------------===// #include "mlir/Conversion/LLVMCommon/ConversionTarget.h" #include "mlir/Conversion/LLVMCommon/Pattern.h" @@ -24,48 +10,6 @@ using namespace mlir; using namespace buddy::buckyball::legalize; -namespace mlir::buddy::buckyball { -void populateVectorLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureVectorLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateTransposeLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable); -void configureTransposeLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateIm2colLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureIm2colLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateFp2IntLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureFp2IntLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateInt2FpLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureInt2FpLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateReluLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureReluLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateMatrixMatmulLegalizeForLLVMExportPatterns( - LLVMTypeConverter &converter, RewritePatternSet &patterns, bool stable, - bool rushB); -void configureMatrixMatmulLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -void populateMatrixLegalizeForLLVMExportPatterns(LLVMTypeConverter &converter, - RewritePatternSet &patterns, - bool stable); -void configureMatrixLegalizeForExportTarget(LLVMConversionTarget &target, - bool stable); -} // namespace mlir::buddy::buckyball - void mlir::populateBuckyballLegalizeForLLVMExportPatterns( LLVMTypeConverter &converter, RewritePatternSet &patterns, int64_t bankWidthBytes, int64_t bankDepth, int64_t bankNum, @@ -76,40 +20,11 @@ void mlir::populateBuckyballLegalizeForLLVMExportPatterns( populateBaseLegalizeForLLVMExportPatterns( converter, patterns, includeFuncOperandForwarding, rushB); - mlir::buddy::buckyball::populateMatrixMatmulLegalizeForLLVMExportPatterns( - converter, patterns, stable, rushB); - mlir::buddy::buckyball::populateVectorLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateTransposeLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateIm2colLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateFp2IntLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateInt2FpLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateReluLegalizeForLLVMExportPatterns( - converter, patterns, stable); - mlir::buddy::buckyball::populateMatrixLegalizeForLLVMExportPatterns( - converter, patterns, stable); + (void)stable; } void mlir::configureBuckyballLegalizeForExportTarget( LLVMConversionTarget &target, bool stable) { configureBaseLegalizeForExportTarget(target); - mlir::buddy::buckyball::configureMatrixMatmulLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureVectorLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureTransposeLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureIm2colLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureFp2IntLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureInt2FpLegalizeForExportTarget(target, - stable); - mlir::buddy::buckyball::configureReluLegalizeForExportTarget(target, stable); - mlir::buddy::buckyball::configureMatrixLegalizeForExportTarget(target, - stable); + (void)stable; } diff --git a/examples/cores/toy/configs/balldomains/balls/matrix/lane16.toml b/examples/cores/toy/configs/balldomains/balls/smatmul/lane16.toml similarity index 100% rename from examples/cores/toy/configs/balldomains/balls/matrix/lane16.toml rename to examples/cores/toy/configs/balldomains/balls/smatmul/lane16.toml diff --git a/examples/cores/toy/configs/balldomains/default.toml b/examples/cores/toy/configs/balldomains/default.toml index 56bde411e..cc120efbd 100644 --- a/examples/cores/toy/configs/balldomains/default.toml +++ b/examples/cores/toy/configs/balldomains/default.toml @@ -1,23 +1,37 @@ -# Toy Ball Domain Configuration +# Toy: virtual single-core. Mounts leftover balls not used by pebble/poly/goban. -ballNum = 7 +ballNum = 3 ballIdMappings = [ - { ballId = 0, ballName = "VecBall", ballClass = "examples.balls.vector.VecBall", config = "../../../../balls/vector/configs/default.toml", inBW = 2, outBW = 4 }, - { ballId = 1, ballName = "ReluBall", ballClass = "examples.balls.relu.ReluBall", config = "../../../../balls/relu/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 2, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 3, ballName = "Im2colBall", ballClass = "examples.balls.im2col.Im2colBall", config = "../../../../balls/im2col/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 4, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 5, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 6, ballName = "Mxfp2IntBall", ballClass = "examples.balls.mxfp2int.Mxfp2IntBall", config = "../../../../balls/mxfp2int/configs/default.toml", inBW = 1, outBW = 1 }, + { ballId = 0, ballName = "GemminiBall", ballClass = "examples.balls.gemmini.GemminiBall", config = "../../../../balls/gemmini/configs/default.toml", inBW = 2, outBW = 4 }, + { ballId = 1, ballName = "TraceBall", ballClass = "examples.balls.trace.TraceBall", config = "../../../../balls/trace/configs/default.toml", inBW = 1, outBW = 1 }, + { ballId = 2, ballName = "Mxfp2IntBall", ballClass = "examples.balls.mxfp2int.Mxfp2IntBall", config = "../../../../balls/mxfp2int/configs/default.toml", inBW = 1, outBW = 1, mmioReadBW = 1 }, ] ballISA = [ - { mnemonic = "MATMUL_WARP16", funct7 = 64, bid = 0 }, - { mnemonic = "RELU", funct7 = 50, bid = 1 }, - { mnemonic = "TRANSPOSE", funct7 = 49, bid = 2 }, - { mnemonic = "IM2COL", funct7 = 48, bid = 3 }, - { mnemonic = "FP2INT", funct7 = 51, bid = 4 }, - { mnemonic = "INT2FP", funct7 = 52, bid = 5 }, - { mnemonic = "MXFP2INT", funct7 = 55, bid = 6 }, + { mnemonic = "GEMMINI_CONFIG", funct7 = 2, bid = 0 }, + { mnemonic = "GEMMINI_FLUSH", funct7 = 3, bid = 0 }, + { mnemonic = "GEMMINI_PRELOAD", funct7 = 53, bid = 0 }, + { mnemonic = "GEMMINI_COMPUTE_PRELOADED", funct7 = 66, bid = 0 }, + { mnemonic = "GEMMINI_COMPUTE_ACCUMULATED", funct7 = 67, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_BOUNDS", funct7 = 80, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_A", funct7 = 81, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_B", funct7 = 82, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_D", funct7 = 83, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_C", funct7 = 84, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_STRIDES_AB", funct7 = 85, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS_CONFIG_STRIDES_DC", funct7 = 86, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_WS", funct7 = 87, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_1", funct7 = 96, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_2", funct7 = 97, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_3", funct7 = 98, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_4", funct7 = 99, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_5", funct7 = 100, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_6", funct7 = 101, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_7", funct7 = 102, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_8", funct7 = 103, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_9", funct7 = 104, bid = 0 }, + { mnemonic = "GEMMINI_LOOP_CONV_WS", funct7 = 105, bid = 0 }, + { mnemonic = "BDB_COUNTER", funct7 = 4, bid = 1 }, + { mnemonic = "MXFP2INT", funct7 = 55, bid = 2 }, ] diff --git a/examples/cores/toy/configs/balldomains/full.toml b/examples/cores/toy/configs/balldomains/full.toml deleted file mode 100644 index 01eb04e92..000000000 --- a/examples/cores/toy/configs/balldomains/full.toml +++ /dev/null @@ -1,51 +0,0 @@ -# Toy Ball Domain Configuration - -ballNum = 10 - -ballIdMappings = [ - { ballId = 0, ballName = "VecBall", ballClass = "examples.balls.vector.VecBall", config = "../../../../balls/vector/configs/default.toml", inBW = 2, outBW = 4 }, - { ballId = 1, ballName = "ReluBall", ballClass = "examples.balls.relu.ReluBall", config = "../../../../balls/relu/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 2, ballName = "TransposeBall", ballClass = "examples.balls.transpose.TransposeBall", config = "../../../../balls/transpose/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 3, ballName = "Im2colBall", ballClass = "examples.balls.im2col.Im2colBall", config = "../../../../balls/im2col/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 4, ballName = "MatrixBall", ballClass = "examples.balls.matrix.MatrixBall", config = "balls/matrix/lane16.toml", inBW = 2, outBW = 4 }, - { ballId = 5, ballName = "Fp2IntBall", ballClass = "examples.balls.fp2int.Fp2IntBall", config = "../../../../balls/fp2int/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 6, ballName = "Int2FpBall", ballClass = "examples.balls.int2fp.Int2FpBall", config = "../../../../balls/int2fp/configs/default.toml", inBW = 1, outBW = 1 }, - { ballId = 7, ballName = "GemminiBall", ballClass = "examples.balls.gemmini.GemminiBall", config = "../../../../balls/gemmini/configs/default.toml", inBW = 2, outBW = 4 }, - { ballId = 8, ballName = "TraceBall", ballClass = "examples.balls.trace.TraceBall", inBW = 1, outBW = 1 }, - { ballId = 9, ballName = "Mxfp2IntBall", ballClass = "examples.balls.mxfp2int.Mxfp2IntBall", config = "../../../../balls/mxfp2int/configs/default.toml", inBW = 1, outBW = 1 } -] - -ballISA = [ - { mnemonic = "MATMUL_WARP16", funct7 = 64, bid = 0 }, - { mnemonic = "RELU", funct7 = 50, bid = 1 }, - { mnemonic = "TRANSPOSE", funct7 = 49, bid = 2 }, - { mnemonic = "IM2COL", funct7 = 48, bid = 3 }, - { mnemonic = "MATRIX", funct7 = 65, bid = 4 }, - { mnemonic = "FP2INT", funct7 = 51, bid = 5 }, - { mnemonic = "INT2FP", funct7 = 52, bid = 6 }, - { mnemonic = "GEMMINI_CONFIG", funct7 = 2, bid = 7 }, - { mnemonic = "GEMMINI_FLUSH", funct7 = 3, bid = 7 }, - { mnemonic = "GEMMINI_PRELOAD", funct7 = 53, bid = 7 }, - { mnemonic = "GEMMINI_COMPUTE_PRELOADED", funct7 = 66, bid = 7 }, - { mnemonic = "GEMMINI_COMPUTE_ACCUMULATED", funct7 = 67, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_BOUNDS", funct7 = 80, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_A", funct7 = 81, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_B", funct7 = 82, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_D", funct7 = 83, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_ADDR_C", funct7 = 84, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_STRIDES_AB", funct7 = 85, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS_CONFIG_STRIDES_DC", funct7 = 86, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_WS", funct7 = 87, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_1", funct7 = 96, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_2", funct7 = 97, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_3", funct7 = 98, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_4", funct7 = 99, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_5", funct7 = 100, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_6", funct7 = 101, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_7", funct7 = 102, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_8", funct7 = 103, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS_CONFIG_9", funct7 = 104, bid = 7 }, - { mnemonic = "GEMMINI_LOOP_CONV_WS", funct7 = 105, bid = 7 }, - { mnemonic = "BDB_COUNTER", funct7 = 4, bid = 8 }, - { mnemonic = "MXFP2INT", funct7 = 55, bid = 9 } -] diff --git a/examples/cores/toy/core.toml b/examples/cores/toy/configs/default.toml similarity index 65% rename from examples/cores/toy/core.toml rename to examples/cores/toy/configs/default.toml index 8d2dc5dba..bb1f4b686 100644 --- a/examples/cores/toy/core.toml +++ b/examples/cores/toy/configs/default.toml @@ -1,10 +1,10 @@ # Toy Core Configuration (default) -balldomain = "configs/balldomains/full.toml" -memdomain = "configs/memdomains/default.toml" -frontend = "configs/frontends/default.toml" -gpdomain = "configs/gpdomains/default.toml" -core = "configs/core/default.toml" +balldomain = "balldomains/default.toml" +memdomain = "memdomains/default.toml" +frontend = "frontends/default.toml" +gpdomain = "gpdomains/default.toml" +core = "core/default.toml" [rocketCore] xLen = 64 @@ -13,6 +13,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = false [rocketCore.mulDiv] enable = true diff --git a/examples/cores/toy/configs/demo.toml b/examples/cores/toy/configs/demo.toml index 0a899411e..e4ec501df 100644 --- a/examples/cores/toy/configs/demo.toml +++ b/examples/cores/toy/configs/demo.toml @@ -1,6 +1,6 @@ # Toy Core Configuration (default) -balldomain = "balldomains/full.toml" +balldomain = "balldomains/default.toml" memdomain = "memdomains/default.toml" frontend = "frontends/demo.toml" gpdomain = "gpdomains/default.toml" @@ -13,6 +13,7 @@ useVM = true useZba = true useZbb = true useZbs = true +haveCFlush = false [rocketCore.mulDiv] enable = true diff --git a/examples/cores/toy/configs/frontends/default.toml b/examples/cores/toy/configs/frontends/default.toml index 57e9ec42c..b17bc5c5d 100644 --- a/examples/cores/toy/configs/frontends/default.toml +++ b/examples/cores/toy/configs/frontends/default.toml @@ -2,6 +2,7 @@ robEntries = 16 rsOutOfOrderResponse = true bankIdLen = 10 vbankIdUpperBound = 15 +sharedBankIdBase = 32 iterLen = 34 subRobEnable = true subRobDepth = 64 diff --git a/examples/cores/toy/configs/frontends/demo.toml b/examples/cores/toy/configs/frontends/demo.toml index 7adaefebd..41cd684cc 100644 --- a/examples/cores/toy/configs/frontends/demo.toml +++ b/examples/cores/toy/configs/frontends/demo.toml @@ -2,6 +2,7 @@ robEntries = 16 rsOutOfOrderResponse = true bankIdLen = 10 vbankIdUpperBound = 15 +sharedBankIdBase = 32 iterLen = 34 subRobEnable = false subRobDepth = 64 diff --git a/examples/cores/toy/configs/memdomains/default.toml b/examples/cores/toy/configs/memdomains/default.toml index be48b4c94..5ee4d654e 100644 --- a/examples/cores/toy/configs/memdomains/default.toml +++ b/examples/cores/toy/configs/memdomains/default.toml @@ -25,7 +25,7 @@ writeChannel = 4 enable = true bankNum = 16 bankEntries = 64 -bankWidth = 128 +bankWidth = 8 readWidth = 8 [mem] diff --git a/examples/cores/toy/emu/Cargo.lock b/examples/cores/toy/emu/Cargo.lock deleted file mode 100644 index 544d7864e..000000000 --- a/examples/cores/toy/emu/Cargo.lock +++ /dev/null @@ -1,1317 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "android_system_properties" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" -dependencies = [ - "libc", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys 0.61.2", -] - -[[package]] -name = "assert_cmd" -version = "2.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2aa3a22042e45de04255c7bf3626e239f450200fd0493c1e382263544b20aea6" -dependencies = [ - "anstyle", - "bstr", - "libc", - "predicates", - "predicates-core", - "predicates-tree", - "wait-timeout", -] - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "bebop-bank-hash" -version = "0.1.0" -dependencies = [ - "serde", - "serde_json", - "snafu", - "xxhash-rust", -] - -[[package]] -name = "bebop-bemu-profile" -version = "0.1.0" - -[[package]] -name = "bebop-dasm" -version = "0.1.0" - -[[package]] -name = "bebop-dtb" -version = "0.1.0" - -[[package]] -name = "bebop-elf" -version = "0.1.0" - -[[package]] -name = "bebop-fd-redirect" -version = "0.1.0" -dependencies = [ - "libc", -] - -[[package]] -name = "bebop-p2e" -version = "0.1.0" -dependencies = [ - "bebop-rtl-trace", - "bebop-uart", - "duct", - "log", -] - -[[package]] -name = "bebop-rtl-trace" -version = "0.1.0" -dependencies = [ - "bebop-bank-hash", -] - -[[package]] -name = "bebop-rushb" -version = "0.1.0" - -[[package]] -name = "bebop-syscall" -version = "0.1.0" -dependencies = [ - "once_cell", -] - -[[package]] -name = "bebop-uart" -version = "0.1.0" - -[[package]] -name = "bebop-verilator" -version = "0.1.0" -dependencies = [ - "bebop-dasm", - "bebop-fd-redirect", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-uart", - "cc", - "ctrlc", - "libc", - "snafu", -] - -[[package]] -name = "bemu-toy" -version = "0.1.0" -dependencies = [ - "assert_cmd", - "bebop-bank-hash", - "bebop-bemu-profile", - "bebop-dasm", - "bebop-dtb", - "bebop-elf", - "bebop-fd-redirect", - "bebop-p2e", - "bebop-rtl-trace", - "bebop-rushb", - "bebop-syscall", - "bebop-uart", - "bebop-verilator", - "camino", - "cc", - "chrono", - "clap", - "ctrlc", - "duct", - "env_logger", - "libc", - "libtest-mimic", - "log", - "nix 0.29.0", - "once_cell", - "predicates", - "serde", - "serde_json", - "snafu", - "toml", - "walkdir", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "block2" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" -dependencies = [ - "objc2", -] - -[[package]] -name = "bstr" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f7dc094d718f2e1c1559ad110e27eeaae14a5465d3d56dd6dbd793079fbd530" -dependencies = [ - "memchr", - "regex-automata", - "serde_core", -] - -[[package]] -name = "bumpalo" -version = "3.20.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" - -[[package]] -name = "camino" -version = "1.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" - -[[package]] -name = "cc" -version = "1.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "cfg_aliases" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" - -[[package]] -name = "chrono" -version = "0.4.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" -dependencies = [ - "iana-time-zone", - "num-traits", - "windows-link", -] - -[[package]] -name = "clap" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91e0c145792ef73a6ad36d27c75ac09f1832222a3c209689d90f534685ee5b7" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f09628afdcc538b57f3c6341e9c8e9970f18e4a481690a64974d7023bd33548b" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "ctrlc" -version = "3.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" -dependencies = [ - "dispatch2", - "nix 0.31.3", - "windows-sys 0.61.2", -] - -[[package]] -name = "defmt" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" -dependencies = [ - "bitflags 1.3.2", - "defmt-macros", -] - -[[package]] -name = "defmt-macros" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" -dependencies = [ - "defmt-parser", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "defmt-parser" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" -dependencies = [ - "thiserror", -] - -[[package]] -name = "difflib" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6184e33543162437515c2e2b48714794e37845ec9851711914eec9d308f6ebe8" - -[[package]] -name = "dispatch2" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" -dependencies = [ - "bitflags 2.13.1", - "block2", - "libc", - "objc2", -] - -[[package]] -name = "duct" -version = "0.13.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ab5718d1224b63252cd0c6f74f6480f9ffeb117438a2e0f5cf6d9a4798929c" -dependencies = [ - "libc", - "once_cell", - "os_pipe", - "shared_child", -] - -[[package]] -name = "env_filter" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d271a03799a1ee8d1ca9b19893b48ca674a9284fefcfb85f05e74ed314217" -dependencies = [ - "log", - "regex", -] - -[[package]] -name = "env_logger" -version = "0.11.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de671bd27a75a797dc9ae289ba1e77276e75e2026408aab65185384e2d5cd3f6" -dependencies = [ - "anstream", - "anstyle", - "env_filter", - "jiff", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "escape8259" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5692dd7b5a1978a5aeb0ce83b7655c58ca8efdcb79d21036ea249da95afec2c6" - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "float-cmp" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b09cf3155332e944990140d967ff5eceb70df778b34f77d8075db46e4704e6d8" -dependencies = [ - "num-traits", -] - -[[package]] -name = "futures-core" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" - -[[package]] -name = "futures-task" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" - -[[package]] -name = "futures-util" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" -dependencies = [ - "futures-core", - "futures-task", - "pin-project-lite", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "iana-time-zone" -version = "0.1.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "log", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" -dependencies = [ - "defmt", - "jiff-core", - "jiff-static", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", -] - -[[package]] -name = "jiff-core" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" -dependencies = [ - "defmt", -] - -[[package]] -name = "jiff-static" -version = "0.2.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" -dependencies = [ - "jiff-core", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "jobserver" -version = "0.1.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" -dependencies = [ - "getrandom", - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.103" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" -dependencies = [ - "cfg-if", - "futures-util", - "wasm-bindgen", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "libtest-mimic" -version = "0.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "14e6ba06f0ade6e504aff834d7c34298e5155c6baca353cc6a4aaff2f9fd7f33" -dependencies = [ - "anstream", - "anstyle", - "clap", - "escape8259", -] - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "nix" -version = "0.31.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" -dependencies = [ - "bitflags 2.13.1", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "normalize-line-endings" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61807f77802ff30975e01f4f071c8ba10c022052f98b3294119f3e615d13e5be" - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "objc2" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" -dependencies = [ - "objc2-encode", -] - -[[package]] -name = "objc2-encode" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "os_pipe" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "portable-atomic" -version = "1.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "predicates" -version = "3.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ada8f2932f28a27ee7b70dd6c1c39ea0675c55a36879ab92f3a715eaa1e63cfe" -dependencies = [ - "anstyle", - "difflib", - "float-cmp", - "normalize-line-endings", - "predicates-core", - "regex", -] - -[[package]] -name = "predicates-core" -version = "1.0.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cad38746f3166b4031b1a0d39ad9f954dd291e7854fcc0eed52ee41a0b50d144" - -[[package]] -name = "predicates-tree" -version = "1.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0de1b847b39c8131db0467e9df1ff60e6d0562ab8e9a16e568ad0fdb372e2f2" -dependencies = [ - "predicates-core", - "termtree", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "rustversion" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" - -[[package]] -name = "same-file" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "serde_json" -version = "1.0.151" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_spanned" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" -dependencies = [ - "serde", -] - -[[package]] -name = "shared_child" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" -dependencies = [ - "libc", - "sigchld", - "windows-sys 0.60.2", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "sigchld" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" -dependencies = [ - "libc", - "os_pipe", - "signal-hook", -] - -[[package]] -name = "signal-hook" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" -dependencies = [ - "libc", - "signal-hook-registry", -] - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "snafu" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" -dependencies = [ - "snafu-derive", -] - -[[package]] -name = "snafu-derive" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "termtree" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f50febec83f5ee1df3015341d8bd429f2d1cc62bcba7ea2076759d315084683" - -[[package]] -name = "thiserror" -version = "2.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.3", -] - -[[package]] -name = "toml" -version = "0.8.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "toml_write", - "winnow", -] - -[[package]] -name = "toml_write" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "wait-timeout" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" -dependencies = [ - "libc", -] - -[[package]] -name = "walkdir" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" -dependencies = [ - "same-file", - "winapi-util", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" -dependencies = [ - "cfg-if", - "once_cell", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn 2.0.119", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.126" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "winapi-util" -version = "0.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" -dependencies = [ - "windows-sys 0.61.2", -] - -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-link", - "windows-result", - "windows-strings", -] - -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - -[[package]] -name = "winnow" -version = "0.7.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] - -[[package]] -name = "xxhash-rust" -version = "0.8.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/examples/cores/toy/emu/Cargo.toml b/examples/cores/toy/emu/Cargo.toml deleted file mode 100644 index 330dd0418..000000000 --- a/examples/cores/toy/emu/Cargo.toml +++ /dev/null @@ -1,74 +0,0 @@ -[package] -name = "bemu-toy" -version = "0.1.0" -edition = "2021" -build = "../../../../bebop/src/nodes/bemu/build.rs" - -[lib] -name = "bebop_bemu" -path = "src/lib.rs" -crate-type = ["rlib", "cdylib"] - -[[bin]] -name = "bebop" -path = "../../../../bebop/src/main.rs" - -[[test]] -name = "test_bemu" -path = "../../../../bebop/tests/test_bemu.rs" -harness = false - -[[test]] -name = "test_verilator" -path = "../../../../bebop/tests/test_verilator.rs" -harness = false -required-features = ["verilator"] - -[dependencies] -snafu = "0.8" -once_cell = "1" -toml = "0.8" -bebop-dasm = { path = "../../../../bebop/src/nodes/lib/dasm" } -bebop-bank-hash = { path = "../../../../bebop/src/nodes/lib/bank-hash" } -bebop-bemu-profile = { path = "../../../../bebop/src/nodes/lib/bemu-profile" } -bebop-rushb = { path = "../../../../bebop/src/nodes/lib/rushB" } -bebop-fd-redirect = { path = "../../../../bebop/src/nodes/lib/fd-redirect" } -bebop-rtl-trace = { path = "../../../../bebop/src/nodes/lib/rtl-trace" } -bebop-uart = { path = "../../../../bebop/src/nodes/lib/uart" } -bebop-dtb = { path = "../../../../bebop/src/nodes/lib/dtb" } -bebop-elf = { path = "../../../../bebop/src/nodes/lib/elf" } -bebop-syscall = { path = "../../../../bebop/src/nodes/lib/syscall" } -bebop-verilator = { path = "../../../../bebop/src/nodes/verilator", optional = true } -bebop-p2e = { path = "../../../../bebop/src/nodes/p2e", optional = true } -clap = { version = "4", features = ["derive"] } -libc = "0.2" -log = "0.4" -env_logger = "0.11" -nix = { version = "0.29", features = ["fs", "mman", "signal", "process"] } -ctrlc = "3" -camino = "1.1" -serde = { version = "1", features = ["derive"] } -serde_json = "1" -duct = "0.13" - -[features] -default = ["bemu"] -bemu = [] -verilator = ["dep:bebop-verilator"] -p2e = ["dep:bebop-p2e"] -difftest = [] - -[build-dependencies] -cc = { version = "1", features = ["parallel"] } -toml = "0.8" - -[dev-dependencies] -clap = { version = "4", features = ["derive"] } -libtest-mimic = "0.8" -assert_cmd = "2" -walkdir = "2" -predicates = "3" -chrono = { version = "0.4", default-features = false, features = ["clock"] } -serde = { version = "1", features = ["derive"] } -serde_json = "1" -toml = "0.8" diff --git a/examples/cores/toy/emu/src/chip.rs b/examples/cores/toy/emu/src/chip.rs deleted file mode 100644 index f9a534b0c..000000000 --- a/examples/cores/toy/emu/src/chip.rs +++ /dev/null @@ -1,66 +0,0 @@ -pub(crate) mod bank { - pub(crate) use crate::bank::*; -} - -use crate::inst::instruction::ExecContext; - -#[path = "../../../../balls/fp2int/emu/src/lib.rs"] -mod fp2int; -#[path = "../../../../balls/gemmini/emu/src/lib.rs"] -mod gemmini; -#[path = "../../../../balls/im2col/emu/src/lib.rs"] -mod im2col; -#[path = "../../../../balls/int2fp/emu/src/lib.rs"] -mod int2fp; -#[path = "../../../../balls/matrix/emu/src/lib.rs"] -mod matrix; -#[path = "../../../../balls/mxfp2int/emu/src/lib.rs"] -mod mxfp2int; -#[path = "../../../../balls/relu/emu/src/lib.rs"] -mod relu; -#[path = "../../../../balls/trace/emu/src/lib.rs"] -mod trace; -#[path = "../../../../balls/transpose/emu/src/lib.rs"] -mod transpose; -#[path = "../../../../balls/vector/emu/src/lib.rs"] -mod vector; - -pub fn execute_known( - ball_class: &str, - funct: u32, - xs1: u64, - xs2: u64, - ctx: &mut ExecContext, -) -> u64 { - fp2int::execute_known(ball_class, funct, xs1, xs2, ctx) - .or_else(|| gemmini::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| im2col::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| int2fp::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| matrix::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| mxfp2int::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| relu::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| trace::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| transpose::execute_known(ball_class, funct, xs1, xs2, ctx)) - .or_else(|| vector::execute_known(ball_class, funct, xs1, xs2, ctx)) - .unwrap_or_else(|| { - panic!("no toy BEMU ball implementation for ballClass={ball_class} funct7={funct}") - }) -} - -pub fn cycles_after_issue(ball_class: &str, funct: u32, xs1: u64, xs2: u64) -> u64 { - fp2int::cycles_after_issue(ball_class, funct, xs1, xs2) - .or_else(|| gemmini::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| im2col::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| int2fp::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| matrix::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| mxfp2int::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| relu::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| trace::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| transpose::cycles_after_issue(ball_class, funct, xs1, xs2)) - .or_else(|| vector::cycles_after_issue(ball_class, funct, xs1, xs2)) - .unwrap_or_else(|| { - panic!( - "no toy BEMU ball latency implementation for ballClass={ball_class} funct7={funct}" - ) - }) -} diff --git a/examples/cores/toy/emu/src/lib.rs b/examples/cores/toy/emu/src/lib.rs deleted file mode 100644 index 37998770d..000000000 --- a/examples/cores/toy/emu/src/lib.rs +++ /dev/null @@ -1,9 +0,0 @@ -pub const BEMU_TOP_CONFIG: &str = "../../../../chips/toy/configs/toy.toml"; - -pub fn bemu_top_config() -> &'static str { - BEMU_TOP_CONFIG -} - -mod chip; - -include!("../../../../../bebop/src/nodes/bemu/src/lib.rs"); diff --git a/examples/cores/toy/manifest.toml b/examples/cores/toy/manifest.toml deleted file mode 100644 index f590ece12..000000000 --- a/examples/cores/toy/manifest.toml +++ /dev/null @@ -1,5 +0,0 @@ -[core] -name = "toy" -kind = "buckyball" -config = "core.toml" -compiler = "compiler" diff --git a/flake.nix b/flake.nix index aec5ea344..0640cee01 100644 --- a/flake.nix +++ b/flake.nix @@ -42,11 +42,10 @@ compiler.flatbuffers compiler.numactl - mosoo.bun - mosoo.just - mosoo.workerd - rustTools.cargoNextest + + # protoc for bbdev config --install (chip.proto -> chip.pb) + pkgs.protobuf ]; shellHook = '' # Must run with cwd at the git checkout. Store copies from toString ./. @@ -57,6 +56,7 @@ return 1 2>/dev/null || exit 1 fi BB_ROOT="$PWD" + export DSH_HOME="$BB_ROOT/.dsh" if [ -d "$BB_ROOT/result/bin" ]; then export PATH="$BB_ROOT/result/bin:$PATH" else @@ -65,14 +65,18 @@ source "$BB_ROOT/sourceme.sh" - export MINIFLARE_WORKERD_PATH="''${MINIFLARE_WORKERD_PATH:-${pkgs.mosoo.workerd}/bin/workerd}" - # Verilator build acceleration: ccache via OBJCACHE export OBJCACHE=ccache export CUDA_HOME="${pkgs.cuda.cudatoolkit}" export CPATH="''${CUDA_HOME}/include''${CPATH:+:$CPATH}" + export CC="${pkgs.systemTools.clang}/bin/clang" + export CXX="${pkgs.systemTools.clang}/bin/clang++" + + export JAVA_HOME="${pkgs.jdk17}" + export PATH="$JAVA_HOME/bin:$PATH" + # Banner must go to stderr. stdout is reserved for MCP stdio / machine parsers. if [ -z "$NIX_QUIET" ]; then echo "================= Buckyball Environment Activated =========================" >&2 @@ -82,12 +86,13 @@ echo "RISC-V Linux GCC: $(riscv64-unknown-linux-gnu-gcc --version 2>&1 | head -1)" >&2 echo "Mill: $(mill --version 2>&1 | head -1)" >&2 echo "Cargo: $(cargo --version 2>&1 | head -1)" >&2 - echo "npm: $(npm --version 2>&1 | head -1)" >&2 + echo "pnpm: $(pnpm --version 2>&1 | head -1)" >&2 + echo "CXX: $CXX" >&2 echo "bbdev: $(which bbdev)" >&2 echo "RISCV: $RISCV" >&2 echo "Yosys: $(yosys --version 2>&1 | head -1)" >&2 echo "OpenSTA: $(sta -version 2>&1 | head -1)" >&2 - echo "Buddy MLIR: $(which buddy-opt)" >&2 + # echo "Buddy MLIR: $(which buddy-opt)" >&2 echo "===========================================================================" >&2 fi ''; @@ -134,7 +139,8 @@ # python environment python.python3Packages pkgs."pre-commit" - pkgs.clang-tools # clang-format for pre-commit (language: system) + # clang-format for pre-commit (language: system) + pkgs.clang-tools # Rust toolchain rustTools.rustc @@ -192,12 +198,7 @@ systemTools.rsync systemTools.nodejs systemTools.git - # systemTools.npm - - # Mosoo local development tools - mosoo.bun - mosoo.just - mosoo.workerd + systemTools.pnpm # CUDA toolkit (12.8, matches host driver) + host g++-13 cuda.cudatoolkit diff --git a/scripts/claude/README.md b/scripts/claude/README.md index 12c086e0e..c61ff2697 100644 --- a/scripts/claude/README.md +++ b/scripts/claude/README.md @@ -46,14 +46,16 @@ UVM when needed: `bbdev_uvm_build` / `bbdev_uvm_run`. ### bbdev wrappers (all POST APIs) | Tool | API | |------|-----| +| `bbdev_config_install` | `/config/install` | | `bbdev_compiler_build` | `/compiler/build` | | `bbdev_task_status` | State for a submitted `trace_id` | | `bbdev_workload_{clean,build,tohex}` | `/workload/{clean,build,tohex}` | | `bbdev_bemu_{sim,batch}` | `/bebop/bemu/{sim,batch}` | | `bbdev_bebop_verilator_*` | `/bebop/verilator/{clean,verilog,build,sim,run,batch}` | | `bbdev_verilator_*` | `/verilator/{clean,verilog,build,sim,run}` (non-bebop) | +| `bbdev_vcs_*` | `/vcs/{clean,verilog,build,sim,run}` | | `bbdev_bebop_p2e_*` | `/bebop/p2e/{clean,verilog,buildbitstream,runworkload,batch}` | -| `bbdev_uvm_{build,run}` | `/uvm/{build,run}` | +| `bbdev_uvm_{verilog,build,run}` | `/uvm/{verilog,build,run}` | | `bbdev_yosys_{run,verilog,synth}` | `/yosys/{run,verilog,synth}` | | `bbdev_dc_verilog` | `/dc/verilog` | | `bbdev_firesim_*` | `/firesim/{enumeratefpgas,buildbitstream,infrasetup,runworkload}` | diff --git a/scripts/nix/build-all.sh b/scripts/nix/build-all.sh index 6f289a01d..bc543fe06 100755 --- a/scripts/nix/build-all.sh +++ b/scripts/nix/build-all.sh @@ -25,15 +25,12 @@ usage() { echo "Options" echo " --help -h : Display this message" echo " --skip -s N : Skip step N in the list above. Use multiple times to skip multiple steps ('-s N -s M ...')." - echo " --nv : Enable NVPTX + CUDA runner (requires CUDA_HOME. We install CUDA in nix store, and you need to make \ - sure CUDA driver is installed on your system.)" exit "$1" } SKIP_LIST=() VERBOSE_FLAG="" INSTALL_IN_NIX=0 -WITH_NV=0 while [ "$1" != "" ]; do @@ -48,8 +45,6 @@ do SKIP_LIST+=(${1}) ;; --install-in-nix) INSTALL_IN_NIX=1 ;; - --nv) - WITH_NV=1 ;; * ) echo "Error: invalid option $1" >&2 usage 1 ;; @@ -90,7 +85,6 @@ if [ "${INSTALL_IN_NIX}" != "1" ]; then for skip in "${SKIP_LIST[@]}"; do SKIP_ARGS="${SKIP_ARGS} -s ${skip}" done - [ "${WITH_NV}" = "1" ] && SKIP_ARGS="${SKIP_ARGS} --nv" exec nix develop --command bash ${BBDIR}/scripts/nix/build-all.sh --install-in-nix ${SKIP_ARGS} ${VERBOSE_FLAG} fi @@ -101,43 +95,19 @@ if run_step "1"; then cd ${BBDIR}/bbdev/api uv venv .venv --python python3 --seed uv pip install --python .venv/bin/python -r pyproject.toml + + bbdev config --install fi if run_step "2"; then begin_step "2" "Compiler installation" - cd ${BBDIR}/compiler/thirdparty/buddy-mlir - - cmake -G Ninja -S llvm/llvm -B llvm/build \ - -DLLVM_ENABLE_PROJECTS="mlir;clang" \ - -DLLVM_ENABLE_RUNTIMES="openmp" \ - -DLLVM_TARGETS_TO_BUILD="host;RISCV$([ "${WITH_NV}" = "1" ] && echo ';NVPTX')" \ - $([ "${WITH_NV}" = "1" ] && echo "-DMLIR_ENABLE_CUDA_RUNNER=ON -DCMAKE_CUDA_COMPILER=$CUDA_HOME/bin/nvcc -DCUDAToolkit_ROOT=$CUDA_HOME -DCMAKE_CUDA_HOST_COMPILER=$(command -v g++-13)") \ - -DLLVM_ENABLE_ASSERTIONS=ON \ - -DOPENMP_ENABLE_LIBOMPTARGET=OFF \ - -DLIBOMP_LIBFLAGS=-lrt \ - -DCMAKE_BUILD_TYPE=RELEASE \ - -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ - -DPython3_EXECUTABLE="$(which python3)" \ - -DPython_EXECUTABLE="$(which python3)" - ninja -C llvm/build #check-clang check-mlir check-openmp - - cmake -G Ninja -S . -B build \ - -DMLIR_DIR=$PWD/llvm/build/lib/cmake/mlir \ - -DLLVM_DIR=$PWD/llvm/build/lib/cmake/llvm \ - -DLLVM_ENABLE_ASSERTIONS=ON \ - -DCMAKE_BUILD_TYPE=RELEASE \ - -DBUDDY_MLIR_ENABLE_PYTHON_PACKAGES=ON \ - -DBUDDY_EXTERNAL_DIALECTS_DIR=${BBDIR}/examples/cores/toy/compiler \ - -DPython3_EXECUTABLE="$(which python3)" \ - -DPython_EXECUTABLE="$(which python3)" - ninja -C build # check-buddy + cd ${BBDIR} + bbdev compiler --build '--chip toy' fi if run_step "3"; then begin_step "3" "RTL source pre-compile" - - cd ${BBDIR}/arch - bbdev verilator --verilog '--config sims.verilator.BuckyballToyVerilatorConfig' + bbdev verilator --verilog '--chip toy' fi if run_step "4"; then diff --git a/scripts/nix/build-env-clibs.nix b/scripts/nix/build-env-clibs.nix index 6aba3f079..57b57ad69 100644 --- a/scripts/nix/build-env-clibs.nix +++ b/scripts/nix/build-env-clibs.nix @@ -42,7 +42,7 @@ in zlib = pkgs.zlib; # Verilator FST tracing uses lz4.h and links liblz4. lz4-dev = pkgs.lz4.dev; - lz4 = pkgs.lz4; + lz4 = pkgs.lz4.lib; # C libraries needed by bdb debugger readline-dev = pkgs.readline.dev; readline = pkgs.readline; diff --git a/scripts/nix/build-env-mosoo.nix b/scripts/nix/build-env-mosoo.nix deleted file mode 100644 index 4d5886ae9..000000000 --- a/scripts/nix/build-env-mosoo.nix +++ /dev/null @@ -1,43 +0,0 @@ -{ pkgs }: - -let - workerdVersion = "1.20260714.1"; - runtimeLibraryPath = pkgs.lib.makeLibraryPath (with pkgs; [ - glibc - stdenv.cc.cc - ]); - dynamicLinker = pkgs.stdenv.cc.bintools.dynamicLinker; - workerd = pkgs.stdenv.mkDerivation { - pname = "workerd"; - version = workerdVersion; - - src = pkgs.fetchurl { - url = "https://registry.npmjs.org/@cloudflare/workerd-linux-64/-/workerd-linux-64-${workerdVersion}.tgz"; - sha256 = "0z4kijx9bri6jfbfci09ivip7wh4s6x5j9rhlj78iifjnzjdfz9z"; - }; - - nativeBuildInputs = [ pkgs.patchelf ]; - dontBuild = true; - - installPhase = '' - runHook preInstall - - mkdir -p $out/bin - cp bin/workerd $out/bin/workerd - chmod +x $out/bin/workerd - patchelf \ - --set-interpreter ${dynamicLinker} \ - --set-rpath ${runtimeLibraryPath} \ - $out/bin/workerd - - runHook postInstall - ''; - }; -in -{ - # Mosoo local development entry points. Project package tools such as - # wrangler/vite-plus are resolved from mosoo's bun workspace dependencies. - bun = pkgs.bun; - just = pkgs.just; - workerd = workerd; -} diff --git a/scripts/nix/build-env-scala.nix b/scripts/nix/build-env-scala.nix index fc9409944..286db7b2c 100644 --- a/scripts/nix/build-env-scala.nix +++ b/scripts/nix/build-env-scala.nix @@ -13,8 +13,11 @@ let nativeBuildInputs = [ pkgs.makeWrapper ]; installPhase = '' mkdir -p $out/bin - cp $src $out/bin/mill - chmod +x $out/bin/mill + install -m755 $src $out/bin/.mill-wrapped + makeWrapper $out/bin/.mill-wrapped $out/bin/mill \ + --set JAVA_HOME ${pkgs.jdk17} \ + --prefix PATH : ${pkgs.jdk17}/bin \ + --unset LD_LIBRARY_PATH ''; meta = with pkgs.lib; { description = "Mill build tool ${millVersion}"; diff --git a/scripts/nix/build-env-system.nix b/scripts/nix/build-env-system.nix index 53c403f76..1cf94acfd 100644 --- a/scripts/nix/build-env-system.nix +++ b/scripts/nix/build-env-system.nix @@ -4,5 +4,6 @@ rsync = pkgs.rsync; nodejs = pkgs.nodejs; git = pkgs.git; - # npm = pkgs.npm; + pnpm = pkgs.pnpm; + clang = pkgs.clang; } diff --git a/scripts/nix/overlay.nix b/scripts/nix/overlay.nix index 26806d3e3..eca706954 100644 --- a/scripts/nix/overlay.nix +++ b/scripts/nix/overlay.nix @@ -16,6 +16,5 @@ final: prev: opensta = final.tools.opensta; }; kernel = final.callPackage ./build-env-kernel.nix { }; - mosoo = final.callPackage ./build-env-mosoo.nix { }; cuda = final.callPackage ./build-env-cuda.nix { }; } diff --git a/sourceme.sh b/sourceme.sh index 48c2e8137..83fd3d303 100644 --- a/sourceme.sh +++ b/sourceme.sh @@ -19,13 +19,27 @@ RESULT_PATH="${BBDIR}/result" #===----------------------------------------------------------------------------=== # Source Environment Variables #===----------------------------------------------------------------------------=== -export BUDDY_MLIR_BUILD_DIR="${BBDIR}/compiler/thirdparty/buddy-mlir/build" +export BUDDY_MLIR_BUILD_ROOT="${BBDIR}/compiler/thirdparty/buddy-mlir/build" export LLVM_MLIR_BUILD_DIR="${BBDIR}/compiler/thirdparty/buddy-mlir/llvm/build" -export PYTHONPATH="${BBDIR}/compiler/thirdparty/buddy-mlir/llvm/build/tools/mlir/python_packages/mlir_core:${BBDIR}/compiler/thirdparty/buddy-mlir/build/python_packages:$PYTHONPATH" -export BUDDY_BINARY_DIR="${BBDIR}/compiler/thirdparty/buddy-mlir/build/bin" +export PYTHONPATH="${LLVM_MLIR_BUILD_DIR}/tools/mlir/python_packages/mlir_core:${PYTHONPATH}" export RISCV="${BBDIR}/result" export PATH="${BBDIR}/thirdparty/libgloss/install/lib:$PATH" -export PATH="${BUDDY_BINARY_DIR}:${PATH}" +# Per-chip buddy-opt lives at compiler/thirdparty/buddy-mlir/build//bin +export PATH="${BUDDY_MLIR_BUILD_ROOT}/bin:${PATH}" + +# Optional interactive compiler session. Set before sourcing, e.g.: +# export BUCKYBALL_COMPILER_CHIP=toy +# bbdev compiler/workload inject BUDDY_MLIR_BUILD_DIR per subprocess. +if [ -n "${BUCKYBALL_COMPILER_CHIP:-}" ]; then + export BUDDY_MLIR_BUILD_DIR="${BUDDY_MLIR_BUILD_ROOT}/${BUCKYBALL_COMPILER_CHIP}" + export BUDDY_BINARY_DIR="${BUDDY_MLIR_BUILD_DIR}/bin" + if [ ! -d "${BUDDY_MLIR_BUILD_DIR}" ]; then + echo "ERROR: missing compiler build for chip ${BUCKYBALL_COMPILER_CHIP}: ${BUDDY_MLIR_BUILD_DIR}" >&2 + return 1 2>/dev/null || exit 1 + fi + export PYTHONPATH="${BUDDY_MLIR_BUILD_DIR}/python_packages:${PYTHONPATH}" + export PATH="${BUDDY_BINARY_DIR}:${PATH}" +fi #===----------------------------------------------------------------------------=== # Export each submodule's PATH diff --git a/thirdparty/soc-framework b/thirdparty/soc-framework index 693476902..f638dc68b 160000 --- a/thirdparty/soc-framework +++ b/thirdparty/soc-framework @@ -1 +1 @@ -Subproject commit 693476902435fa0366ffe469f1313f8ad61dbb97 +Subproject commit f638dc68bb694e4601f37d741db2702dd5418c7d diff --git a/verify b/verify index eda2f7f70..a5606004d 160000 --- a/verify +++ b/verify @@ -1 +1 @@ -Subproject commit eda2f7f705d01c45a71ff964c382b6526d1efe30 +Subproject commit a5606004de93c7ceae66276ac3193dcc0a304855