diff --git a/README.md b/README.md index 1d470899..06ad3d49 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ featuring advanced flamegraph analysis tools. - [Usage](#usage) - [Managing the stack](#managing-the-stack) - [Local running and development](#local-running-and-development) +- [End-to-end test harness](#end-to-end-test-harness) ## System Overview @@ -55,6 +56,7 @@ same backend services and storage layer: ```mermaid %%{init: { 'theme': 'base', + 'flowchart': { 'defaultRenderer': 'elk' }, 'themeVariables': { 'primaryColor': '#ffffff', 'primaryBorderColor': '#cbd5e1', @@ -388,3 +390,16 @@ To develop the project, it may be useful to run each component locally, see rele - [webapp (backend and frontend)](src/gprofiler/README.md) - [gprofiler_flamedb_rest](src/gprofiler_flamedb_rest/README.md) - [gprofiler_indexer](src/gprofiler_indexer/README.md) + +## End-to-end test harness +To run the full stack **plus a real gProfiler agent** locally (LocalStack S3/SQS +included) and exercise the workload-level profiling flow end to end — with API +acceptance tests (AT-S1..S15) and Playwright UI tests — see +[deploy/E2E_HARNESS.md](deploy/E2E_HARNESS.md). + +```bash +cd deploy +make -f Makefile.e2e e2e-up-src # studio + sample workload + source-built agent +make -f Makefile.e2e e2e-test # API acceptance suite +make -f Makefile.e2e e2e-ui-test # Playwright UI suite +``` diff --git a/deploy/E2E_HARNESS.md b/deploy/E2E_HARNESS.md new file mode 100644 index 00000000..c8d471f9 --- /dev/null +++ b/deploy/E2E_HARNESS.md @@ -0,0 +1,342 @@ +# End-to-end test harness (Performance Studio + real gProfiler agent) + +This harness runs the **full** Performance Studio stack **plus a real gProfiler +agent** locally, so the dynamic/workload-level profiling flow can be exercised +end to end — including the S3 → SQS → indexer → ClickHouse → flamegraph pipeline +that the studio-only setup never touches. + +``` + e2e-sample-app (CPU workload) + ▲ profiled by + gprofiler-agent ──heartbeat/commands──► webapp (backend) ──► Postgres + │ │ + └──upload profile (/api/v2/profiles)──┘ + ▼ + S3 (LocalStack) ──► SQS (LocalStack) + ▼ + ch-indexer ──► ClickHouse (flamedb) + ▼ + UI / flamegraph API (nginx :4433) +``` + +> **Where this fits.** This harness is **Layer 2** of a three-layer test setup +> (fast unit → this compose harness → a kind/minikube Kubernetes sandbox). See +> [The three-layer test harness](#the-three-layer-test-harness) below for how the +> layers differ and how to invoke each. Layer 3 lives in +> [`k8s-sandbox/`](k8s-sandbox/K8S_SANDBOX.md) (architecture: +> [`../docs/K8S_SANDBOX.md`](../docs/K8S_SANDBOX.md)). + +## Files + +| File | Purpose | +|------|---------| +| `docker-compose.e2e.yml` | Overlay adding `e2e-sample-app`, `gprofiler-agent`, `e2e-tests`, `e2e-ui-tests` | +| `Makefile.e2e` | `up / up-src / status / start / stop / flamegraph / test / ui-test / down` targets | +| `e2e/agent-glibc.Dockerfile` | Wraps a source-built (`--fast`) agent exe in a glibc base | +| `../src/tests/e2e/` | In-network pytest API acceptance runner (AT-S1..S15) | +| `../src/tests/playwright/` | Playwright UI acceptance runner | + +## Container architecture + +**It is not one container and not a Kubernetes pod.** It is a set of independent +Docker containers wired together by Docker Compose on a single user-defined +bridge network (`_default`, i.e. `deploy_default`). Each service is its +own container running one process; they find each other by **service name** as a +DNS hostname on that network. There is no orchestrator (no k8s, no pod spec) — +just Compose. + +| Container | Image / build | Role | +|-----------|---------------|------| +| `gprofiler-ps-webapp` | built from `../src` | FastAPI backend + built UI (heartbeat, profile ingest, status APIs) | +| `gprofiler-ps-nginx-load-balancer` | `nginx:1.23.3` | TLS + basic-auth edge; publishes `:4433`/`:8443` to the host | +| `gprofiler-ps-postgres` | `postgres` | heartbeat inventory, profiling requests/commands | +| `gprofiler-ps-clickhouse` | `clickhouse` | `flamedb` sample/metric storage (profile `with-clickhouse`) | +| `gprofiler-ps-ch-rest-service` | built | ClickHouse REST facade for the backend | +| `gprofiler-ps-ch-indexer` | built | **SQS consumer**: reads S3 profiles → writes ClickHouse | +| `gprofiler-ps-agents-logs-backend` | built | agent log ingest | +| `gprofiler-ps-periodic-tasks` | built | background jobs | +| `gprofiler-ps-localstack` | `localstack/localstack:3.0` | **one** container emulating **S3 + SQS** | +| `gprofiler-ps-e2e-sample-app` | `python:3.11-slim` | deterministic CPU workload to profile *(harness)* | +| `gprofiler-ps-e2e-agent` | source-built or `GPROFILER_IMAGE` | real gProfiler agent in heartbeat mode *(harness)* | +| `gprofiler-ps-e2e-tests` | built from `../src/tests/e2e` | pytest API acceptance runner, profile `test` *(harness)* | +| `gprofiler-ps-e2e-ui-tests` | official Playwright image | Playwright UI runner, profile `ui` *(harness)* | + +Only nginx publishes host ports; everything else talks over the internal +network. The agent, for example, reaches the backend at `http://webapp` directly +(no host port, no nginx auth). + +## S3 + SQS via LocalStack (no AWS account needed) + +The `localstack` container runs the S3 and SQS emulators in-process +(`SERVICES=s3,sqs`) and exposes a single gateway on `:4566`. On startup it runs +every script in `deploy/localstack_init/` — `01_init_s3_sqs.sh` creates the +bucket and queue and wires S3 event notifications into SQS. A healthcheck gates +dependents until both services report `running`. + +The pipeline then flows entirely inside the network: + +1. agent uploads a profile to `webapp` (`/api/v2/profiles`) +2. `webapp` writes the object to **S3** (`s3://performance-studio-bucket/...`) +3. S3 emits an event to **SQS** (`performance-studio-queue`) +4. `ch-indexer` consumes the SQS message, reads the object, and writes rows to + **ClickHouse** (`flamedb.samples`) plus the flamegraph HTML back to S3 + +All AWS SDK clients in the stack point at `AWS_ENDPOINT_URL=http://localstack:4566` +with dummy `test`/`test` credentials, so nothing touches real AWS. + +## Prerequisites + +- **Docker + Docker Compose v2** (Linux, or macOS/Windows via Docker Desktop). +- The **`gprofiler` agent repo** checked out as a sibling of this repo + (`../../gprofiler`) — required for `e2e-up-src` (the portable agent path). +- **TLS cert + basic-auth file** for nginx (one-time, in `deploy/`): + +```bash +cd deploy +mkdir -p tls +openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls/key.pem -out tls/cert.pem +openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls/ch_rest_key.pem -out tls/ch_rest_cert.pem +htpasswd -B -C 12 -c .htpasswd admin # set the password to 'admin' to match the defaults +``` + +The harness assumes web creds `admin` / `admin` (override with `AUTH=` on the +Make targets and the `E2E_BASIC_AUTH_*` env vars). + +> **Note:** The `htpasswd` command above uses `-c` which *creates* (or overwrites) +> `.htpasswd`. If you already have a `.htpasswd` with a different username, the +> `admin` entry won't be present and every `make` target that calls the nginx API +> will get a `401`. Either recreate the file with `-c`, or add the entry without +> overwriting: `htpasswd -B -C 12 -b .htpasswd admin admin`. + +## Quick start + +```bash +cd deploy + +# Bring up studio + sample app + agent (mints a valid profiler token for you). +make -f Makefile.e2e e2e-up + +# See the agent's live inventory row (heartbeat, agent version, status). +make -f Makefile.e2e e2e-status + +# Drive the control plane: +make -f Makefile.e2e e2e-start # host-scope START (profiles the sample app) +make -f Makefile.e2e e2e-stop # host-scope STOP (no PIDs at host level) + +# Inspect the artifacts the pipeline produced in (Local)S3. +make -f Makefile.e2e e2e-flamegraph + +# Follow the agent. +make -f Makefile.e2e e2e-agent-logs + +# Remove ONLY the harness services (studio keeps running). +make -f Makefile.e2e e2e-down +``` + +`SERVICE=` overrides the target service (default `e2e-sample-app`) on the +`e2e-start` / `e2e-stop` / `e2e-flamegraph` targets. + +## How the agent is wired + +- **Endpoint / auth.** The agent points at the *internal* `webapp` service + (`--server-host=http://webapp --api-server=http://webapp`), which bypasses the + nginx basic-auth that guards the browser UI. Heartbeat/command endpoints don't + validate the bearer token locally, but the upload/health-check path *does* — so + `e2e-up` mints a real token via `GET /api/api_key` and injects it as + `GPROFILER_TOKEN`. +- **Isolation.** The agent shares only the **sample app's** PID namespace + (`pid: "service:e2e-sample-app"`), so it profiles that workload and never the + host. `perf` is disabled (`--perf-mode=none`); py-spy profiles the Python + workload. +- **Determinism.** `e2e-sample-app` runs a fixed hot loop, so flamegraphs have + stable, recognizable frames. + +## Agent image + +There are two ways to get the agent image; pick per your goal. + +### A) From source, fast (verify your agent changes) — recommended + +Uses the agent repo's **`--fast`** executable build, which skips `staticx` +(the slow bundling step) and finishes in ~1-2 min on a warm Docker cache: + +```bash +make -f Makefile.e2e e2e-up-src # builds exe (--fast) + wraps + brings up +# or just rebuild the image: +make -f Makefile.e2e e2e-agent-build +``` + +Because `--fast` skips staticx, the resulting `build//gprofiler` is +**glibc-dynamic** and will *not* run on the alpine base in the repo's +`container.Dockerfile` (you'd get `exec /gprofiler: No such file or directory`). +The harness therefore wraps it in a glibc base — see `e2e/agent-glibc.Dockerfile`. +Override arch/repo with `AGENT_ARCH=` / `AGENT_REPO=` if needed. + +### B) Prebuilt image (fastest, no build) + +`make -f Makefile.e2e e2e-up` uses `GPROFILER_IMAGE` (default +`intel/gprofiler:latest`, the upstream public image) with the vanilla +`/gprofiler` entrypoint. Good for exercising the studio side when you don't need +agent-source changes. Override it to test a specific build: + +```bash +GPROFILER_IMAGE=intel/gprofiler:1.53.1 make -f Makefile.e2e e2e-up +``` + +Agent-side pure-logic changes (heartbeat inventory, command queue) are also +covered by the fast unit suite in `gprofiler/tests_fast/`. + +## Portability (will it work on any engineer's box?) + +Yes for **Linux/Ubuntu** with Docker + Compose v2 — that's the primary target and +where it's verified. Keep these in mind: + +- **Agent image.** `e2e-up` defaults to the upstream public `intel/gprofiler:latest`. + To verify *your own* agent changes, use `make -f Makefile.e2e e2e-up-src`, which + builds the agent from the sibling `../../gprofiler` checkout. Studio images all + build from source here, so the studio side is portable as-is. +- **CPU architecture.** The source build defaults to `x86_64` + (`build_x86_64_executable.sh`). On arm64 (Apple Silicon, Graviton) use + `AGENT_ARCH=aarch64 make -f Makefile.e2e e2e-up-src` (drives + `build_aarch64_executable.sh`); the glibc wrapper honors `AGENT_ARCH`. +- **Linux vs macOS/Windows.** On Linux the agent's `privileged: true` + shared + PID namespace + py-spy `ptrace` work natively. On Docker Desktop + (macOS/Windows) everything runs inside its Linux VM and works, but profiling + fidelity depends on the VM; `--perf-mode=none` (already set) avoids kernel-perf + issues. `pid: "service:..."` is a Compose feature and is portable. +- **Host ports.** Only nginx publishes ports (`4433`, `8443`). If those clash, + remap them in `docker-compose.yml`. The DB ports are intentionally not + published (tests run in-network), so there's nothing else to collide. +- **Compose network name.** Helper targets use `$(NETWORK)`, derived from the + project dir (`deploy_default`). If you set `COMPOSE_PROJECT_NAME`, pass + `NETWORK=_default`. +- **Prerequisites** above (TLS + `.htpasswd`) are one-time and cross-platform + (need `openssl` + `htpasswd`; `htpasswd` ships with `apache2-utils` on Ubuntu). + +## The three-layer test harness + +Three layers test the workload-level profiling flow at increasing fidelity. The +key difference between layers is **how the workload inventory is produced** — +fabricated in-code, POSTed as synthetic heartbeats, or enumerated for real from a +Kubernetes node's container runtime. + +| Layer | What runs | Inventory source | Proves | Invoke | +|-------|-----------|------------------|--------|--------| +| **1. Fast unit/spec** | no stack | in-code fixtures | backend/agent pure logic (PR gate) | `pytest` / `node --test` | +| **2. Compose harness** (this doc) | full studio via Docker Compose | **synthetic** heartbeats (+ real-agent pipeline) | studio logic + S3→SQS→indexer→ClickHouse→flamegraph | `make -f Makefile.e2e …` | +| **3. kind/minikube sandbox** | full studio on Kubernetes | **real** CRI enumeration by a DaemonSet | real namespace/pod/container/process discovery + scoping | `make -f Makefile.k8s …` | + +### Layer 1 — fast unit/spec (no stack, PR gate) + +No services required; runs in seconds. + +```bash +# studio backend spec (request builder, PMU/capacity, profiling-request logic) +pytest -q gprofiler-performance-studio/src/tests/spec +# studio frontend unit tests +cd gprofiler-performance-studio/src/gprofiler/frontend && node --test +# agent-side fast logic (heartbeat inventory, command queue) — in the agent repo +pytest -q gprofiler/tests_fast +``` + +### Layer 2 — compose harness (this doc) + +Full studio + a real agent profiling a deterministic sample app, plus the +complete artifact pipeline. Three test families: + +- **API acceptance `AT-S1..S15`** — 15 in-network pytest cases (`src/tests/e2e/`) + driving the live stack with **synthetic** heartbeats: inventory/tab-counts + (S1–S4), host/service/process resolution + command creation (S5–S7), + empty-resolution 422 (S8), PMU rejection (S9), continuous-subscription + auto-enrollment (S10–S13), legacy/partial-inventory compatibility (S14–S15). +- **Full-pipeline smoke `AT-A1..A8`** — the real source-built agent proves + S3 → SQS → indexer → ClickHouse → flamegraph. +- **Playwright UI e2e** — drives the console through nginx (basic auth + + self-signed TLS), asserting the start/stop confirmation flow (the STOP case is + the regression test for the host-level-stop bug). + +```bash +cd deploy +make -f Makefile.e2e e2e-up # bring up (or e2e-up-src for source-built agent) +make -f Makefile.e2e e2e-test # AT-S1..S15 API acceptance +make -f Makefile.e2e e2e-start \ + && make -f Makefile.e2e e2e-flamegraph # AT-A1..A8 pipeline smoke (inspect artifacts) +make -f Makefile.e2e e2e-ui-test # Playwright UI acceptance +make -f Makefile.e2e e2e-down # remove harness (studio keeps running) +``` + +### Layer 3 — kind/minikube sandbox (real Kubernetes topology) + +The one thing compose can't do: real namespaces/pods/containers. A gProfiler +agent **DaemonSet** on a real node reads the node's CRI socket and enumerates +genuine workloads, so this layer proves discovery + scope resolution under +Kubernetes. Tests: **`AT-K1..K5`** (agent registers as host; tenant namespaces, +pods, and containers — incl. a 2-container pod — discovered from real CRI; +host-scope start dispatches to the real agent). Full details: +[`k8s-sandbox/K8S_SANDBOX.md`](k8s-sandbox/K8S_SANDBOX.md) (architecture + +rationale: [`../docs/K8S_SANDBOX.md`](../docs/K8S_SANDBOX.md)). + +```bash +cd deploy/k8s-sandbox + +# --- kind (default, recommended: lightweight, containerd-native) --- +make -f Makefile.k8s k8s-all # cluster + build + load + deploy + token +make -f Makefile.k8s k8s-test # AT-K1..K5 real-topology acceptance +make -f Makefile.k8s k8s-status # live inventory across all scopes +make -f Makefile.k8s k8s-url # -> https://localhost:30443 (admin/admin) +make -f Makefile.k8s k8s-down-all # delete the whole cluster +``` + +**Option: minikube instead of kind.** The same manifests, agent DaemonSet, and +`AT-K1..K5` suite run unchanged on minikube — only cluster lifecycle, image +loading, and the URL differ. Select it with `CLUSTER=minikube`, and use a distinct +`CLUSTER_NAME` if a kind cluster is already running (kind binds host port `30443`; +map minikube elsewhere, e.g. `--ports=30444:30443`, to avoid a collision): + +```bash +cd deploy/k8s-sandbox +make -f Makefile.k8s k8s-all CLUSTER=minikube CLUSTER_NAME=gprofiler-mk +make -f Makefile.k8s k8s-test CLUSTER=minikube CLUSTER_NAME=gprofiler-mk +make -f Makefile.k8s k8s-url CLUSTER=minikube CLUSTER_NAME=gprofiler-mk # -> https://:30443 +make -f Makefile.k8s k8s-down-all CLUSTER=minikube CLUSTER_NAME=gprofiler-mk +``` + +> **Caveat (why kind is the default).** With `--container-runtime=containerd` +> minikube gives the same prod-like CRI path, but its **docker driver may fail to +> boot on some hosts** (e.g. Docker 28 + cgroup v2 + newer kernels, where the node +> container can't start systemd), while kind's `kindest/node` boots there fine. If +> minikube won't start, use kind. See the "Challenge" section in +> [`../docs/K8S_SANDBOX.md`](../docs/K8S_SANDBOX.md) for the full write-up. + +## Optional: container/pod inventory + +The agent logs `No container runtime found for heartbeat workload inventory` +because the Docker socket isn't mounted (keeps it isolated from the host). To +exercise namespace/pod/container scope tabs with a real agent, mount +`/var/run/docker.sock:/var/run/docker.sock:ro` into `gprofiler-agent` — note this +gives the agent visibility into all host containers. + +For **real** Kubernetes namespace/pod/container/process inventory (not the Docker +socket workaround), use **Layer 3** — the kind/minikube sandbox — which runs an +agent DaemonSet against a real node's CRI. See +[The three-layer test harness](#the-three-layer-test-harness). + +## Notes + +- `e2e-down` removes only the harness containers (agent, sample app, test/UI + runners); use `e2e-down-all` to tear down the entire studio stack (destructive). +- **Token minting on cold builds.** `e2e-up` polls the studio API for up to 60 s + (30 × 2 s) before minting the agent token. On a cold build the webapp can take + longer, leaving the agent container with an empty `GPROFILER_TOKEN` and causing + it to crash-loop. If this happens the rest of the stack is still healthy — just + re-run the mint + recreate step manually: + ```bash + tok=$(curl -sk https://localhost:4433/api/api_key -u admin:admin \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['apiKey'])") + GPROFILER_TOKEN="$tok" GPROFILER_IMAGE=gprofiler-e2e-agent:src \ + docker compose -f docker-compose.yml -f docker-compose.e2e.yml \ + --profile with-clickhouse up -d --force-recreate gprofiler-agent + ``` +- The compose project name defaults to the directory (`deploy`), so the network + is `deploy_default`. Helper targets read `$(NETWORK)`; override with + `NETWORK=_default` if you set `COMPOSE_PROJECT_NAME`. diff --git a/deploy/Makefile.e2e b/deploy/Makefile.e2e new file mode 100644 index 00000000..12ed6608 --- /dev/null +++ b/deploy/Makefile.e2e @@ -0,0 +1,114 @@ +# E2E harness for Performance Studio + a real gProfiler agent. +# +# Usage: +# make -f Makefile.e2e e2e-up # bring up studio + sample app + agent +# make -f Makefile.e2e e2e-status # show the agent in workload_status +# make -f Makefile.e2e e2e-start # submit a host-scope start (SERVICE/HOST overridable) +# make -f Makefile.e2e e2e-stop # submit a host-scope stop +# make -f Makefile.e2e e2e-agent-logs +# make -f Makefile.e2e e2e-test # run in-network pytest acceptance suite +# make -f Makefile.e2e e2e-down # remove only the agent + sample app (studio stays up) +# make -f Makefile.e2e e2e-down-all # tear the whole stack down (destructive) + +SHELL := /bin/bash + +COMPOSE := docker compose -f docker-compose.yml -f docker-compose.e2e.yml --profile with-clickhouse +NGINX := https://localhost:4433 +AUTH := admin:admin +SERVICE ?= e2e-sample-app +# Compose default network. Derived from the project name (the deploy/ dir), so +# override NETWORK=_default if you set COMPOSE_PROJECT_NAME. +NETWORK ?= $(notdir $(CURDIR))_default + +# Source-build settings (sibling agent repo). +AGENT_REPO ?= ../../gprofiler +AGENT_ARCH ?= x86_64 +AGENT_SRC_IMAGE ?= gprofiler-e2e-agent:src + +# Placeholder token so compose can parse before a real one is minted; the agent +# is re-created with the real token in e2e-up. +export GPROFILER_TOKEN ?= bootstrap + +.PHONY: e2e-up e2e-up-src e2e-agent-build e2e-token e2e-status e2e-start e2e-stop e2e-agent-logs e2e-test e2e-down e2e-down-all e2e-flamegraph + +## Build the agent image FROM SOURCE using the fast (no-staticx) build (~1-2 min +## warm cache). The --fast exe is glibc-dynamic, so it's wrapped in a glibc base. +e2e-agent-build: + @echo ">> building agent executable from source (--fast, no staticx)" + cd $(AGENT_REPO) && scripts/build_$(AGENT_ARCH)_executable.sh --fast + @echo ">> wrapping exe in glibc base -> $(AGENT_SRC_IMAGE)" + docker build -f e2e/agent-glibc.Dockerfile --build-arg ARCH=$(AGENT_ARCH) -t $(AGENT_SRC_IMAGE) $(AGENT_REPO) + +## Build the agent from source, then bring the whole harness up with it. +e2e-up-src: e2e-agent-build + GPROFILER_IMAGE=$(AGENT_SRC_IMAGE) $(MAKE) -f Makefile.e2e e2e-up + +## Mint (or fetch) a valid profiler token from the running studio. +e2e-token: + @curl -sk $(NGINX)/api/api_key -u $(AUTH) \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['apiKey'])" + +## Bring the whole stack up, then re-create the agent with a real token. +e2e-up: + @echo ">> starting studio + sample app + agent (bootstrap token)" + $(COMPOSE) up -d --build + @echo ">> waiting for studio API..." + @for i in $$(seq 1 30); do \ + if curl -skf $(NGINX)/api/api_key -u $(AUTH) >/dev/null 2>&1; then break; fi; \ + sleep 2; \ + done + @tok=$$(curl -sk $(NGINX)/api/api_key -u $(AUTH) | python3 -c "import sys,json;print(json.load(sys.stdin)['apiKey'])"); \ + echo ">> minting token: $$tok"; \ + GPROFILER_TOKEN=$$tok $(COMPOSE) up -d --force-recreate gprofiler-agent + @echo ">> up. Try: make -f Makefile.e2e e2e-status" + +## Show the agent's live inventory row. +e2e-status: + @curl -sk "$(NGINX)/api/metrics/profiling/workload_status" -u $(AUTH) \ + | python3 -m json.tool + +## Submit a host-scope START for $(SERVICE) (auto-discovers the reporting host). +e2e-start: + @host=$$(curl -sk "$(NGINX)/api/metrics/profiling/host_status?service_name=$(SERVICE)" -u $(AUTH) \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['hosts'][0]['hostname'])"); \ + echo ">> start on host=$$host"; \ + curl -sk -X POST "$(NGINX)/api/metrics/profile_request" -u $(AUTH) -H "Content-Type: application/json" \ + -d "{\"service_name\":\"$(SERVICE)\",\"request_type\":\"start\",\"continuous\":false,\"duration\":30,\"frequency\":11,\"profiling_mode\":\"cpu\",\"target_scope\":\"host\",\"target_hosts\":{\"$$host\":[]},\"target_entities\":[{\"service_name\":\"$(SERVICE)\",\"hostname\":\"$$host\"}],\"additional_args\":{}}" \ + | python3 -m json.tool + +## Submit a host-scope STOP for $(SERVICE) (regression check: no PIDs at host level). +e2e-stop: + @host=$$(curl -sk "$(NGINX)/api/metrics/profiling/host_status?service_name=$(SERVICE)" -u $(AUTH) \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['hosts'][0]['hostname'])"); \ + echo ">> stop on host=$$host"; \ + curl -sk -X POST "$(NGINX)/api/metrics/profile_request" -u $(AUTH) -H "Content-Type: application/json" \ + -d "{\"service_name\":\"$(SERVICE)\",\"request_type\":\"stop\",\"continuous\":false,\"duration\":30,\"frequency\":11,\"profiling_mode\":\"cpu\",\"target_scope\":\"host\",\"stop_level\":\"host\",\"target_hosts\":{\"$$host\":[]},\"target_entities\":[{\"service_name\":\"$(SERVICE)\",\"hostname\":\"$$host\"}],\"additional_args\":{}}" \ + | python3 -m json.tool + +## List the flamegraph artifacts produced in (local)S3. +e2e-flamegraph: + @docker run --rm --network $(NETWORK) \ + -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 \ + amazon/aws-cli:2.15.0 --endpoint-url http://localstack:4566 \ + s3 ls s3://performance-studio-bucket/$${S3_PATH_PREFIX:+$${S3_PATH_PREFIX}/}products/$(SERVICE)/ --recursive + +e2e-agent-logs: + @docker logs -f gprofiler-ps-e2e-agent + +## Run the in-network acceptance suite (pytest) against the live stack. +e2e-test: + $(COMPOSE) --profile test build e2e-tests + $(COMPOSE) --profile test run --rm e2e-tests + +## Run the Playwright UI acceptance suite (in-network, official browser image). +e2e-ui-test: + $(COMPOSE) --profile ui build e2e-ui-tests + $(COMPOSE) --profile ui run --rm e2e-ui-tests + +## Remove only the harness-added services; studio keeps running. +e2e-down: + $(COMPOSE) rm -sf gprofiler-agent e2e-sample-app e2e-tests e2e-ui-tests + +## Tear everything down (studio included). Destructive. +e2e-down-all: + $(COMPOSE) down diff --git a/deploy/docker-compose.e2e.yml b/deploy/docker-compose.e2e.yml new file mode 100644 index 00000000..8870acd3 --- /dev/null +++ b/deploy/docker-compose.e2e.yml @@ -0,0 +1,116 @@ +# +# E2E overlay for the Performance Studio + gProfiler agent test harness. +# +# Layer this on top of the base stack: +# +# docker compose -f docker-compose.yml -f docker-compose.e2e.yml \ +# --profile with-clickhouse up -d --build +# +# It adds: +# * e2e-sample-app - a deterministic CPU-burning workload to profile +# * gprofiler-agent - a real gProfiler agent in dynamic/heartbeat mode, pointed +# at the INTERNAL webapp (bypassing nginx basic auth), that +# reports workload inventory and executes start/stop commands +# * e2e-tests - an in-network pytest runner (profile "test"; on demand) +# +# The agent profiles only the sample app's PID namespace, so it never touches the +# host's processes. +# +services: + # A deterministic target the agent can profile (py-spy sees a hot Python frame). + e2e-sample-app: + image: python:3.11-slim + container_name: gprofiler-ps-e2e-sample-app + restart: unless-stopped + command: + - python3 + - -c + - | + import time + def hot_loop(): + total = 0 + for i in range(5_000_000): + total += (i * i) % 7 + return total + while True: + hot_loop() + time.sleep(0.01) + + # gProfiler agent in dynamic profiling (heartbeat) mode. + # Defaults to the upstream public image; override with GPROFILER_IMAGE, or build + # from source with `make -f Makefile.e2e e2e-up-src` (see E2E_HARNESS.md). + gprofiler-agent: + image: ${GPROFILER_IMAGE:-intel/gprofiler:latest} + container_name: gprofiler-ps-e2e-agent + restart: unless-stopped + entrypoint: ["/gprofiler"] + # Point at the internal webapp service (no nginx basic auth on the internal + # port); the backend does not validate the bearer token locally. + command: + # --server-host: profile upload + health check; --api-server: heartbeat/metrics. + # Both are served by the internal webapp (no nginx basic auth on that port). + - --server-host=http://webapp + - --api-server=http://webapp + # A valid profiler token is required for upload/health-check. Mint one from + # the running studio (GET /api/api_key) and export GPROFILER_TOKEN before up. + - "--token=${GPROFILER_TOKEN:?set GPROFILER_TOKEN first - use make e2e-up}" + - --service-name=e2e-sample-app + - --upload-results + - --enable-heartbeat-server + - --heartbeat-interval=10 + - --perf-mode=none + - --output-dir=/tmp/gprofiler_output + - --dont-send-logs + # Agent shares the sample app's PID namespace (not host-init), so skip the + # init-namespace guard. + - --disable-pidns-check + privileged: true + # Share the sample app's PID namespace so the agent profiles only that + # workload (isolated from the host). + pid: "service:e2e-sample-app" + environment: + - GPROFILER_IN_CONTAINER=1 + - POD_NAMESPACE=e2e + - POD_NAME=gprofiler-agent + depends_on: + - webapp + - e2e-sample-app + + # In-network acceptance/UI test runner. Kept behind the "test" profile so it + # only runs on demand (e.g. `docker compose ... --profile test run --rm e2e-tests`). + e2e-tests: + build: + context: ../src + dockerfile: tests/e2e/Dockerfile + container_name: gprofiler-ps-e2e-tests + profiles: ["test"] + environment: + # Reach services by their compose network names. + - E2E_BASE_URL=http://webapp + - E2E_NGINX_URL=https://nginx-load-balancer + - E2E_BASIC_AUTH_USER=admin + - E2E_BASIC_AUTH_PASSWORD=admin + - GPROFILER_POSTGRES_HOST=$POSTGRES_HOST + - GPROFILER_POSTGRES_PORT=$POSTGRES_PORT + - GPROFILER_POSTGRES_USERNAME=$POSTGRES_USER + - GPROFILER_POSTGRES_PASSWORD=$POSTGRES_PASSWORD + - GPROFILER_POSTGRES_DB_NAME=$POSTGRES_DB + depends_on: + - webapp + - db_postgres + + # Playwright UI acceptance runner (profile "ui"; on demand). Reaches the + # console via the internal nginx service with basic auth over self-signed TLS. + e2e-ui-tests: + build: + context: ../src/tests/playwright + dockerfile: Dockerfile + container_name: gprofiler-ps-e2e-ui-tests + profiles: ["ui"] + environment: + - E2E_UI_URL=https://nginx-load-balancer + - E2E_BASIC_AUTH_USER=admin + - E2E_BASIC_AUTH_PASSWORD=admin + - E2E_UI_SERVICE=e2e-sample-app + depends_on: + - nginx-load-balancer diff --git a/deploy/e2e/agent-glibc.Dockerfile b/deploy/e2e/agent-glibc.Dockerfile new file mode 100644 index 00000000..0f2ef8c4 --- /dev/null +++ b/deploy/e2e/agent-glibc.Dockerfile @@ -0,0 +1,19 @@ +# Wraps a locally-built gProfiler executable in a glibc base for the e2e harness. +# +# Build context MUST be the gprofiler agent repo root (so build//gprofiler +# is present). Produce that exe first with the fast (no-staticx) build: +# +# cd ../../gprofiler && scripts/build_x86_64_executable.sh --fast +# +# The --fast build skips staticx, so the exe is glibc-dynamic and needs a glibc +# base (ubuntu) rather than the alpine base in the repo's container.Dockerfile. +ARG ARCH=x86_64 +FROM ubuntu:22.04 + +ARG ARCH +ENV GPROFILER_IN_CONTAINER=1 + +COPY build/${ARCH}/gprofiler /gprofiler +RUN chmod +x /gprofiler + +ENTRYPOINT ["/gprofiler"] diff --git a/deploy/k8s-sandbox/K8S_SANDBOX.md b/deploy/k8s-sandbox/K8S_SANDBOX.md new file mode 100644 index 00000000..50e01b24 --- /dev/null +++ b/deploy/k8s-sandbox/K8S_SANDBOX.md @@ -0,0 +1,208 @@ +# Kubernetes sandbox (real workload/pod/container scoping) + +A self-contained, disposable **Kubernetes** environment that runs the full +Performance Studio stack **plus a real gProfiler agent DaemonSet** and a set of +tenant workloads, so the workload-level profiling flow can be exercised against a +**genuine cluster topology**. + +> **This page is the operational runbook.** For the architecture, the +> "what is kind" primer, and the pods-vs-systemd topology breakdown, see +> [`docs/K8S_SANDBOX.md`](../../docs/K8S_SANDBOX.md). + +This is a *separate layer* from the docker-compose harness in +[`deploy/E2E_HARNESS.md`](../E2E_HARNESS.md) — it does not replace it. Use them +for different jobs: + +| Layer | Orchestrator | Best at | Can't do | +|-------|--------------|---------|----------| +| `deploy/Makefile.e2e` (compose) | Docker Compose | fast API + full S3→SQS→indexer→ClickHouse→flamegraph pipeline smoke | no real namespaces/pods/containers (no CRI) | +| `deploy/k8s-sandbox` (this) | kind / minikube | **real namespace/pod/container/process inventory + scope resolution** | heavier, slower inner loop | + +## Why this exists (the one thing compose cannot do) + +Workload-level profiling resolves **namespace / pod / container / process** +selections. The agent builds that inventory in +`gprofiler/metadata/heartbeat_metadata.py` by asking `granulate_utils`' +`ContainersClient` to enumerate the node's containers and reading the kubelet +labels `io.kubernetes.pod.namespace`, `io.kubernetes.pod.name`, +`io.kubernetes.container.name`. + +`ContainersClient` talks to the **container-runtime socket** — CRI at +`/run/containerd/containerd.sock` (or `/var/run/crio/crio.sock`) and/or Docker — +**not** the Kubernetes API server. Under docker-compose there is no such socket +for the agent, so it logs `No container runtime found for heartbeat workload +inventory` and the pod/container/namespace tabs are exercised only with +*synthetic* heartbeats. On a real node the socket exists, so the inventory is +**real**. + +### How the DaemonSet reaches the socket + +`granulate_utils` resolves the socket path under `HOST_ROOT_PREFIX = /proc/1/root` +(see `granulate_utils/linux/ns.py`). So the mechanism is: + +- **`hostPID: true`** → PID 1 in the pod is the host init, so `/proc/1/root` is + the node root filesystem and `/proc/1/root/run/containerd/containerd.sock` is + the node's real CRI socket. `hostPID` is also what lets the agent see every + node PID to map processes → containers (`get_process_container_id`). +- **`privileged: true`** → grants access to that socket and lets py-spy `ptrace` + the target workloads. + +No bind-mount of the socket is required; the two flags above are the whole trick. +See [`manifests/50-agent-daemonset.yaml`](manifests/50-agent-daemonset.yaml). + +> **Use the source-built agent.** The container-inventory heartbeat is a fork +> feature, so the public `intel/gprofiler:latest` image will **not** populate the +> pod/container tabs. `k8s-agent-build` builds it from the sibling `../../gprofiler` +> checkout (reusing `deploy/e2e/agent-glibc.Dockerfile`). + +## Architecture + +``` + kind / minikube node (containerd) + ┌──────────────────────────────────────────────────────────────────────┐ + │ ns team-a: checkout(x2), web ns team-b: payments, search(app+car) │ + │ ▲ enumerated via CRI (/proc/1/root/run/containerd/...) │ + │ gprofiler-agent DaemonSet ──heartbeat/commands──► webapp ──► postgres │ + │ (hostPID, privileged) │ │ + │ └── upload ──► S3 (LocalStack) │ + │ ▼ │ + │ SQS ──► ch-indexer ──► ClickHouse + │ ▼ │ + │ nginx NodePort :30443 (UI) │ + └──────────────────────────────────────────────────────────────────────┘ +``` + +Everything is in the `perf-studio` namespace except the tenant workloads +(`team-a`, `team-b`). Only the optional nginx edge is published (NodePort 30443); +the agent and the in-cluster test Job reach `http://webapp` directly. + +## Prerequisites + +- **Docker**, plus **`kind`** (recommended) or **`minikube`**, and **`kubectl`**. +- **`python3`** on your host (used to parse the minted profiler token). +- The **agent repo** checked out at `../../../gprofiler` (sibling of the studio + repo) for `k8s-agent-build`. +- TLS certs + `.htpasswd` in `deploy/` (same one-time step as the compose + harness — see [`deploy/E2E_HARNESS.md`](../E2E_HARNESS.md#prerequisites)). + +## Quick start + +```bash +cd deploy/k8s-sandbox + +# One-shot: create cluster, build+load images, deploy stack+workloads+agent, mint token. +make -f Makefile.k8s k8s-all + +# Give the agent ~30-60s to enumerate CRI, then inspect the live inventory. +make -f Makefile.k8s k8s-status + +# Run the real-topology acceptance suite (AT-K1..K5). +make -f Makefile.k8s k8s-test + +# Drive the control plane against the real agent. +make -f Makefile.k8s k8s-start +make -f Makefile.k8s k8s-stop + +# Open the console (basic auth admin/admin). +make -f Makefile.k8s k8s-url + +# Tear the whole sandbox down. +make -f Makefile.k8s k8s-down-all +``` + +Use `CLUSTER=minikube` on any target to use minikube instead of kind: + +```bash +make -f Makefile.k8s k8s-all CLUSTER=minikube +``` + +`make -f Makefile.k8s help` lists all targets. + +## What the acceptance suite proves (AT-K1..K5) + +[`tests/test_k8s_inventory.py`](tests/test_k8s_inventory.py), run in-cluster as a +Job against the live `webapp`: + +- **AT-K1** — the real agent DaemonSet registers as a host under service + `k8s-sandbox`. +- **AT-K2** — tenant **namespaces** (`team-a`, `team-b`) appear in the namespace + scope (discovered from CRI, not fabricated). +- **AT-K3** — **pods** for each workload (`checkout`, `web`, `payments`, + `search`) appear in the pod scope. +- **AT-K4** — **containers** appear, including *both* containers of the + 2-container `search` pod (`search-app`, `search-sidecar`). +- **AT-K5** — a host-scope start/stop resolves the **real** agent host (from live + inventory) and is accepted — command creation against an actual agent. + +The suite reuses the compose suite's HTTP harness (`src/tests/e2e/harness.py`) so +the request/response contract lives in one place, and matches on serialized +inventory (not hard-coded per-scope key casing) so it stays robust as the node +also contains system/studio containers. + +## Files + +| Path | Purpose | +|------|---------| +| `Makefile.k8s` | cluster/build/load/deploy/token/test/start/stop/down targets | +| `kind-config.yaml` | single-node kind cluster; publishes nginx NodePort 30443 | +| `manifests/00-namespaces.yaml` | `perf-studio`, `team-a`, `team-b` | +| `manifests/01-config.yaml` | shared `studio-config` ConfigMap + `studio-secrets` | +| `manifests/10-datastores.yaml` | Postgres + ClickHouse (schema via init ConfigMaps) | +| `manifests/12-localstack.yaml` | S3 + SQS emulator (init from ConfigMap) | +| `manifests/13-ch-rest-service.yaml` | ClickHouse REST facade (TLS from Secret) | +| `manifests/20-webapp.yaml` | FastAPI backend + UI (`http://webapp`) | +| `manifests/21-ch-indexer.yaml` | SQS→ClickHouse indexer | +| `manifests/22-logs-backend.yaml` | agent-logs + periodic-tasks (shared `/logs`) | +| `manifests/30-nginx.yaml` | optional TLS/basic-auth edge (NodePort 30443) | +| `manifests/40-workloads.yaml` | tenant workloads the agent enumerates | +| `manifests/50-agent-daemonset.yaml` | **the real agent (hostPID + privileged)** | +| `tests/` | in-cluster acceptance Job + real-topology tests | + +## Notes & caveats + +- **kind vs minikube.** kind is the default and recommended for CI (lighter, + containerd-native, faster boot). minikube is supported for local use via + `CLUSTER=minikube` (start it with `--container-runtime=containerd`). +- **Inner loop.** Unlike compose (which builds straight from `../src`), a cluster + requires images to be *loaded* in (`kind load` / `minikube image load`) after + each rebuild — re-run `k8s-images && k8s-load` (or `k8s-agent-build && k8s-load`) + when you change code. +- **Disposable.** The cluster is hermetic: LocalStack fakes AWS and nothing + leaves the node. `k8s-down-all` deletes the cluster and leaves no residue. +- **The agent enumerates the whole node**, so inventory also contains + `kube-system`/studio containers — expected. The tests assert on the specific + tenant entities rather than exact totals. +- **Low ports as non-root.** The webapp and agents-logs-backend images run as a + non-root user and bind port 80. Docker permits this via its default + `net.ipv4.ip_unprivileged_port_start=0`; Kubernetes does not, so those pods set + that (safe) sysctl in their `securityContext`. If your kubelet restricts safe + sysctls, allowlist `net.ipv4.ip_unprivileged_port_start` (or run those two as + root). + +## Verified + +Brought up on a single-node **kind v1.30** cluster (containerd 1.7) and confirmed +end to end: + +- The agent DaemonSet connects and heartbeats with **no** `No container runtime + found` error (contrast the compose harness), i.e. it reached the node's CRI + socket via `/proc/1/root` with `hostPID` + `privileged`. +- `workload_status` reported real topology — `tabCounts` `{service:1, host:1, + namespace:5, pod:23, container:25, process:93}` — including the tenant + namespaces `team-a`/`team-b` and the 2-container `search` pod resolved as + distinct `search-app` + `search-sidecar` containers. +- The full acceptance suite passed: `AT-K1..K5` (5 passed). +- **Full artifact pipeline closed in-cluster**: a host-scope start made the agent + profile the real workloads and upload; the webapp wrote the collapsed stacks to + S3 (`products/k8s-sandbox/stacks/...gz`) and enqueued SQS; the indexer consumed + it, inserted 27 rows into `flamedb.samples`, and wrote the rendered + `..._adhoc_flamegraph.html` back to S3 — all against LocalStack, no real AWS. + +Two k8s-specific fixes came out of that run (both committed): + +- **Non-root low-port bind** (webapp, agents-logs-backend) — added the + `net.ipv4.ip_unprivileged_port_start=0` sysctl (above). +- **Indexer startup ordering** — compose gated it on `localstack: service_healthy`; + k8s has no `depends_on` and the indexer resolves the SQS URL once without + retry, so it can boot before LocalStack and never consume. Added an + init-container that waits for LocalStack's SQS to report running. diff --git a/deploy/k8s-sandbox/Makefile.k8s b/deploy/k8s-sandbox/Makefile.k8s new file mode 100644 index 00000000..4a3cf881 --- /dev/null +++ b/deploy/k8s-sandbox/Makefile.k8s @@ -0,0 +1,329 @@ +# Kubernetes sandbox for Performance Studio + a REAL gProfiler agent DaemonSet. +# +# This is the k8s-native counterpart of deploy/Makefile.e2e. Its reason to exist: +# a DaemonSet on a real node reads the node's CRI socket and enumerates genuine +# pods/containers, so it proves the workload/pod/container scope resolution that +# docker-compose cannot (compose has no container runtime -> empty inventory). +# +# One-shot: +# make -f Makefile.k8s k8s-all # cluster + build + load + deploy + token +# make -f Makefile.k8s k8s-test # run AT-K1..K5 real-topology acceptance +# make -f Makefile.k8s spark-demo # one Spark app + per-thread flamegraph +# make -f Makefile.k8s spark-multi-demo # several distinct Spark apps (relative weight) +# make -f Makefile.k8s k8s-down-all # delete the whole cluster +# +# Granular targets are listed in `make -f Makefile.k8s help`. + +SHELL := /bin/bash + +# kind (default, recommended for CI: lighter, containerd-native) or minikube. +CLUSTER ?= kind +CLUSTER_NAME ?= gprofiler-sandbox +NS ?= perf-studio + +# Source-built agent (the container-inventory heartbeat is a fork feature, so the +# public image would leave pod/container tabs empty). Mirrors Makefile.e2e. +AGENT_REPO ?= ../../../gprofiler +AGENT_ARCH ?= x86_64 +AGENT_IMAGE ?= gprofiler-e2e-agent:src + +VITE_DOCUMENTATION_URL ?= http://custom.gprofiler.doc.domain/ + +# Locally-built studio images (referenced by the manifests with the same tags). +STUDIO_IMAGES := \ + gprofiler-ps/webapp:sandbox \ + gprofiler-ps/ch-rest-service:sandbox \ + gprofiler-ps/ch-indexer:sandbox \ + gprofiler-ps/agents-logs-backend:sandbox \ + gprofiler-ps/periodic-tasks:sandbox +TESTS_IMAGE := gprofiler-ps/k8s-tests:sandbox + +KUBECTL := kubectl -n $(NS) + +.PHONY: help k8s-all k8s-cluster k8s-images k8s-agent-build k8s-load k8s-config \ + k8s-deploy k8s-token k8s-up k8s-status k8s-test k8s-start k8s-stop \ + k8s-flamegraph k8s-logs k8s-ui k8s-url k8s-down k8s-down-all + +help: + @grep -E '^## ' $(MAKEFILE_LIST) | sed 's/^## //' + +## k8s-all : cluster + images + agent + load + deploy + token (one-shot) +k8s-all: k8s-cluster k8s-images k8s-agent-build k8s-load k8s-up + @echo ">> sandbox ready. Try: make -f Makefile.k8s k8s-status" + +## k8s-cluster : create the local cluster (CLUSTER=kind|minikube) +k8s-cluster: +ifeq ($(CLUSTER),kind) + @kind get clusters 2>/dev/null | grep -qx $(CLUSTER_NAME) \ + && echo ">> kind cluster $(CLUSTER_NAME) already exists" \ + || kind create cluster --name $(CLUSTER_NAME) --config kind-config.yaml +else ifeq ($(CLUSTER),minikube) + @minikube status -p $(CLUSTER_NAME) >/dev/null 2>&1 \ + && echo ">> minikube profile $(CLUSTER_NAME) already running" \ + || minikube start -p $(CLUSTER_NAME) --container-runtime=containerd --ports=30443:30443 +else + @echo "Unsupported CLUSTER=$(CLUSTER) (use kind or minikube)"; exit 1 +endif + +## k8s-images : build the studio + test images from source +k8s-images: + @echo ">> building studio images from source" + docker build -t gprofiler-ps/webapp:sandbox \ + -f ../../src/gprofiler/Dockerfile \ + --build-arg VITE_DOCUMENTATION_URL=$(VITE_DOCUMENTATION_URL) ../../src + docker build -t gprofiler-ps/ch-rest-service:sandbox ../../src/gprofiler_flamedb_rest + docker build -t gprofiler-ps/ch-indexer:sandbox ../../src/gprofiler_indexer + docker build -t gprofiler-ps/agents-logs-backend:sandbox \ + -f ../../src/gprofiler_logging/Dockerfile ../../src + docker build -t gprofiler-ps/periodic-tasks:sandbox ../periodic_tasks + @echo ">> building k8s acceptance test image (context = studio root)" + docker build -f tests/Dockerfile -t $(TESTS_IMAGE) ../.. + +## k8s-agent-build : build the agent FROM SOURCE (fast, glibc-wrapped) +k8s-agent-build: + @echo ">> building agent executable from source (--fast, no staticx)" + cd $(AGENT_REPO) && scripts/build_$(AGENT_ARCH)_executable.sh --fast + @echo ">> wrapping exe in glibc base -> $(AGENT_IMAGE)" + docker build -f ../e2e/agent-glibc.Dockerfile --build-arg ARCH=$(AGENT_ARCH) -t $(AGENT_IMAGE) $(AGENT_REPO) + +## k8s-load : load all locally-built images into the cluster +k8s-load: + @for img in $(STUDIO_IMAGES) $(TESTS_IMAGE) $(AGENT_IMAGE); do \ + echo ">> loading $$img"; \ + if [ "$(CLUSTER)" = "kind" ]; then \ + kind load docker-image $$img --name $(CLUSTER_NAME); \ + else \ + minikube image load $$img -p $(CLUSTER_NAME); \ + fi; \ + done + +## k8s-config : create ConfigMaps/Secrets from the existing deploy/ files +k8s-config: + $(KUBECTL) apply -f manifests/00-namespaces.yaml + $(KUBECTL) apply -f manifests/01-config.yaml + @echo ">> schema + init ConfigMaps (single source of truth with compose)" + $(KUBECTL) create configmap pg-initdb \ + --from-file=create_scheme.sql=../../scripts/setup/postgres/gprofiler_recreate.sql \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + $(KUBECTL) create configmap ch-initdb \ + --from-file=create_schema.sql=../../src/gprofiler_indexer/sql/create_ch_schema.sql \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + $(KUBECTL) create configmap localstack-init \ + --from-file=../localstack_init \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + @echo ">> TLS + nginx edge inputs" + $(KUBECTL) create secret generic ch-rest-tls \ + --from-file=ch_rest_cert.pem=../tls/ch_rest_cert.pem \ + --from-file=ch_rest_key.pem=../tls/ch_rest_key.pem \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + $(KUBECTL) create configmap nginx-conf \ + --from-file=nginx.conf=../https_nginx.conf \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + $(KUBECTL) create secret generic nginx-tls \ + --from-file=cert.pem=../tls/cert.pem --from-file=key.pem=../tls/key.pem \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + $(KUBECTL) create secret generic nginx-htpasswd \ + --from-file=.htpasswd=../.htpasswd \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + @echo ">> placeholder agent token (replaced by k8s-token)" + $(KUBECTL) create secret generic gprofiler-agent-token \ + --from-literal=token=bootstrap \ + --dry-run=client -o yaml | $(KUBECTL) apply -f - + +## k8s-deploy : apply all manifests (core stack + workloads + agent) +k8s-deploy: + $(KUBECTL) apply -f manifests/ + @echo ">> waiting for the webapp to become ready..." + $(KUBECTL) rollout status deploy/webapp --timeout=300s + +## k8s-token : mint a real profiler token and restart the agent with it +k8s-token: + @echo ">> minting profiler token from the running studio" + @tok=$$($(KUBECTL) run tokfetch-$$RANDOM --image=curlimages/curl:8.7.1 \ + --restart=Never -i --rm --quiet --command -- \ + curl -s http://webapp/api/api_key \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['apiKey'])"); \ + echo ">> token: $$tok"; \ + $(KUBECTL) create secret generic gprofiler-agent-token \ + --from-literal=token=$$tok --dry-run=client -o yaml | $(KUBECTL) apply -f - + $(KUBECTL) rollout restart daemonset/gprofiler-agent + $(KUBECTL) rollout status daemonset/gprofiler-agent --timeout=180s + +## k8s-up : config + deploy + token (assumes cluster up & images loaded) +k8s-up: k8s-config k8s-deploy k8s-token + @echo ">> up. Give the agent ~30-60s to enumerate CRI, then: make -f Makefile.k8s k8s-status" + +## k8s-status : show the live workload inventory (all scopes) +k8s-status: + @echo "== pods ==" && $(KUBECTL) get pods -o wide + @echo "== agent DaemonSet ==" && $(KUBECTL) get ds gprofiler-agent + @echo "== workload_status (host scope) ==" ; \ + $(KUBECTL) run wsq-$$RANDOM --image=curlimages/curl:8.7.1 --restart=Never -i --rm --quiet --command -- \ + curl -s "http://webapp/api/metrics/profiling/workload_status?scope=host" | python3 -m json.tool || true + +## k8s-test : run the real-topology acceptance suite (AT-K1..K5) as a Job +k8s-test: + $(KUBECTL) delete job k8s-acceptance --ignore-not-found + $(KUBECTL) apply -f tests/job-api.yaml + @echo ">> waiting for the acceptance Job..." + @$(KUBECTL) wait --for=condition=complete job/k8s-acceptance --timeout=420s & \ + comp=$$!; \ + $(KUBECTL) wait --for=condition=failed job/k8s-acceptance --timeout=420s & \ + fail=$$!; \ + wait -n $$comp $$fail; \ + $(KUBECTL) logs job/k8s-acceptance + +## k8s-start : host-scope START for the real agent (SERVICE overridable) +SERVICE ?= k8s-sandbox +# Host discovery + JSON parsing run on the HOST (python3), not in the curl pod +# (the curl image has no python3); only the raw HTTP calls run in-cluster. +define resolve_host +$$(kubectl -n $(NS) run hs-$$RANDOM --image=curlimages/curl:8.7.1 --restart=Never -i --rm --quiet --command -- \ + curl -s "http://webapp/api/metrics/profiling/host_status?service_name=$(SERVICE)" \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['hosts'][0]['hostname'])") +endef + +k8s-start: + @host=$(call resolve_host); echo ">> start on host=$$host"; \ + req=$$(python3 -c "import json;print(json.dumps({'service_name':'$(SERVICE)','request_type':'start','continuous':False,'duration':30,'frequency':11,'profiling_mode':'cpu','target_scope':'host','target_hosts':{'$$host':[]},'target_entities':[{'service_name':'$(SERVICE)','hostname':'$$host'}],'additional_args':{}}))"); \ + $(KUBECTL) run rq-$$RANDOM --image=curlimages/curl:8.7.1 --restart=Never -i --rm --quiet --command -- \ + curl -s -X POST "http://webapp/api/metrics/profile_request" -H "Content-Type: application/json" -d "$$req" + +## k8s-stop : host-scope STOP for the real agent +k8s-stop: + @host=$(call resolve_host); echo ">> stop on host=$$host"; \ + req=$$(python3 -c "import json;print(json.dumps({'service_name':'$(SERVICE)','request_type':'stop','continuous':False,'duration':30,'frequency':11,'profiling_mode':'cpu','target_scope':'host','stop_level':'host','target_hosts':{'$$host':[]},'target_entities':[{'service_name':'$(SERVICE)','hostname':'$$host'}],'additional_args':{}}))"); \ + $(KUBECTL) run rq-$$RANDOM --image=curlimages/curl:8.7.1 --restart=Never -i --rm --quiet --command -- \ + curl -s -X POST "http://webapp/api/metrics/profile_request" -H "Content-Type: application/json" -d "$$req" + +## k8s-flamegraph : list flamegraph artifacts produced in (Local)S3 +# NOTE: the amazon/aws-cli image ENTRYPOINT is already `aws`, so pass args +# directly after `--` (no `--command`, which would override the entrypoint). +k8s-flamegraph: + @$(KUBECTL) run s3ls-$$RANDOM --image=amazon/aws-cli:2.15.0 --restart=Never -i --rm --quiet \ + --env=AWS_ACCESS_KEY_ID=test --env=AWS_SECRET_ACCESS_KEY=test --env=AWS_DEFAULT_REGION=us-east-1 \ + -- --endpoint-url http://localstack:4566 s3 ls s3://performance-studio-bucket/ --recursive || true + +## k8s-logs : follow the agent DaemonSet logs +k8s-logs: + $(KUBECTL) logs -f ds/gprofiler-agent + +## k8s-ui : print how to reach the console (via nginx NodePort) +k8s-ui: k8s-url + +## k8s-url : print the console URL (basic auth admin/admin) +k8s-url: + @echo "Console (basic auth from deploy/.htpasswd):" +ifeq ($(CLUSTER),kind) + @echo " https://localhost:30443 (kind extraPortMapping)" +else + @echo " https://$$(minikube ip -p $(CLUSTER_NAME)):30443" +endif + +## k8s-down : remove workloads + agent (studio stack stays up) +k8s-down: + $(KUBECTL) delete -f manifests/40-workloads.yaml --ignore-not-found + $(KUBECTL) delete -f manifests/50-agent-daemonset.yaml --ignore-not-found + +## k8s-down-all : delete the entire cluster (destructive) +k8s-down-all: +ifeq ($(CLUSTER),kind) + kind delete cluster --name $(CLUSTER_NAME) +else + minikube delete -p $(CLUSTER_NAME) +endif + +# --------------------------------------------------------------------------- # +# Spark-on-Kubernetes demo (spark/): a driver + 6 executor JVM pods deliberately +# oversubscribed to be very busy, profiled PER JVM THREAD by the agent (which runs +# with --java-async-profiler-args=threads). This is the workload that shows real +# Spark executor task threads in a gProfiler flamegraph. +# --------------------------------------------------------------------------- # +SPARK_DIR := spark +SPARK_NS := spark +SPARK_IMAGE := spark-cpu-job:sandbox +SPARK_PROFILE_SECONDS ?= 90 +# Distinct workloads to run side by side (each becomes its own appid in gProfiler). +SPARK_MODES ?= agg join regex pyudf +SPARK_MULTI_SECONDS ?= 300 +# Partitions per app => task count => number of per-thread columns. Lower it +# (e.g. SPARK_PARTITIONS=6) for a less crowded per-thread flamegraph. +SPARK_PARTITIONS ?= 24 + +.PHONY: spark-build spark-load spark-submit spark-status spark-profile spark-demo \ + spark-multi spark-multi-demo spark-clean + +## spark-build : build the Spark demo image (job baked into apache/spark) +spark-build: + docker build -t $(SPARK_IMAGE) $(SPARK_DIR) + +## spark-load : load the Spark image into the cluster (CLUSTER=kind|minikube) +spark-load: + @if [ "$(CLUSTER)" = "kind" ]; then \ + kind load docker-image $(SPARK_IMAGE) --name $(CLUSTER_NAME); \ + else \ + minikube image load $(SPARK_IMAGE) -p $(CLUSTER_NAME); \ + fi + +## spark-submit : RBAC + submit (spark-submit cluster mode -> driver + executors) +spark-submit: + kubectl apply -f $(SPARK_DIR)/00-rbac.yaml + -kubectl -n $(SPARK_NS) delete job spark-submit --ignore-not-found --now + -kubectl -n $(SPARK_NS) delete pod spark-cpu-driver --ignore-not-found --now + kubectl apply -f $(SPARK_DIR)/10-submit-job.yaml + @echo ">> submitted. Driver + 6 executor pods appear in ns/$(SPARK_NS) within ~30-40s." + +## spark-status : show the Spark driver + executor pods +spark-status: + kubectl -n $(SPARK_NS) get pods -o wide + +## spark-profile : host-scope profile while Spark runs (SPARK_PROFILE_SECONDS=90) +spark-profile: + @host=$(call resolve_host); echo ">> profiling host=$$host for $(SPARK_PROFILE_SECONDS)s"; \ + req=$$(python3 -c "import json;print(json.dumps({'service_name':'$(SERVICE)','request_type':'start','continuous':False,'duration':$(SPARK_PROFILE_SECONDS),'frequency':11,'profiling_mode':'cpu','target_scope':'host','target_hosts':{'$$host':[]},'target_entities':[{'service_name':'$(SERVICE)','hostname':'$$host'}],'additional_args':{}}))"); \ + $(KUBECTL) run rq-$$RANDOM --image=curlimages/curl:8.7.1 --restart=Never -i --rm --quiet --command -- \ + curl -s -X POST "http://webapp/api/metrics/profile_request" -H "Content-Type: application/json" -d "$$req" + +## spark-demo : build + load + submit + profile + list flamegraph (one-shot) +spark-demo: spark-build spark-load spark-submit + @echo ">> letting executors schedule and get busy (~45s)..."; sleep 45 + @$(MAKE) -f Makefile.k8s spark-status + @$(MAKE) -f Makefile.k8s spark-profile + @echo ">> waiting for the profile to run + upload..."; sleep $$(( $(SPARK_PROFILE_SECONDS) + 45 )) + @$(MAKE) -f Makefile.k8s k8s-flamegraph + @echo ">> done: the newest *_adhoc_flamegraph.html above is the Spark per-thread profile." + +## spark-multi : submit SEVERAL distinct apps at once (SPARK_MODES="agg join regex pyudf") +# Each mode hits a different hot path and carries its own app name, so gProfiler +# shows them under separate appids -> you can compare RELATIVE weight across apps +# instead of staring at four identical per-thread flamegraphs. +spark-multi: spark-build spark-load + kubectl apply -f $(SPARK_DIR)/00-rbac.yaml + @for m in $(SPARK_MODES); do \ + name="spark-$$m"; \ + echo ">> submitting workload=$$m app=$$name"; \ + kubectl -n $(SPARK_NS) delete job spark-submit-$$m --ignore-not-found --now >/dev/null 2>&1; \ + kubectl -n $(SPARK_NS) delete pod spark-$$m-driver --ignore-not-found --now >/dev/null 2>&1; \ + sed -e "s/__SUFFIX__/$$m/g" -e "s/__MODE__/$$m/g" \ + -e "s/__APPNAME__/$$name/g" -e "s/__SECONDS__/$(SPARK_MULTI_SECONDS)/g" \ + -e "s/__PARTS__/$(SPARK_PARTITIONS)/g" \ + $(SPARK_DIR)/11-submit-workload.yaml | kubectl apply -f - ; \ + done + @echo ">> submitted modes: $(SPARK_MODES). Drivers + executors appear in ns/$(SPARK_NS) within ~40s." + +## spark-multi-demo: build+load+submit N apps + profile + list flamegraph (one-shot) +spark-multi-demo: spark-multi + @echo ">> letting all apps schedule and get busy (~60s)..."; sleep 60 + @$(MAKE) -f Makefile.k8s spark-status + @$(MAKE) -f Makefile.k8s spark-profile + @echo ">> waiting for the profile to run + upload..."; sleep $$(( $(SPARK_PROFILE_SECONDS) + 45 )) + @$(MAKE) -f Makefile.k8s k8s-flamegraph + @echo ">> done. In the gProfiler UI, filter by appid (java: spark-agg / spark-join /" + @echo " spark-regex / spark-python-udf) to compare each app's relative weight." + +## spark-clean : remove all Spark jobs/drivers + RBAC (deletes the spark namespace) +spark-clean: + -kubectl -n $(SPARK_NS) delete job -l app=spark-submit --ignore-not-found + -kubectl -n $(SPARK_NS) delete job spark-submit --ignore-not-found + -kubectl -n $(SPARK_NS) delete pod spark-cpu-driver --ignore-not-found + -kubectl delete -f $(SPARK_DIR)/00-rbac.yaml --ignore-not-found diff --git a/deploy/k8s-sandbox/kind-config.yaml b/deploy/k8s-sandbox/kind-config.yaml new file mode 100644 index 00000000..f16ec4ee --- /dev/null +++ b/deploy/k8s-sandbox/kind-config.yaml @@ -0,0 +1,15 @@ +# Single-node kind cluster for the sandbox. kind nodes run containerd, whose CRI +# socket at /run/containerd/containerd.sock is exactly what the agent DaemonSet +# reads (via /proc/1/root with hostPID) to build workload inventory. +# +# extraPortMappings publishes the nginx NodePort (30443) to the host so you can +# open the console at https://localhost:30443 without kubectl port-forward. +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +name: gprofiler-sandbox +nodes: + - role: control-plane + extraPortMappings: + - containerPort: 30443 + hostPort: 30443 + protocol: TCP diff --git a/deploy/k8s-sandbox/manifests/00-namespaces.yaml b/deploy/k8s-sandbox/manifests/00-namespaces.yaml new file mode 100644 index 00000000..d6b2de0c --- /dev/null +++ b/deploy/k8s-sandbox/manifests/00-namespaces.yaml @@ -0,0 +1,30 @@ +# Namespaces for the k8s sandbox. +# perf-studio : the Performance Studio control plane + in-cluster test Jobs +# team-a / team-b : realistic tenant namespaces holding the workloads the +# gProfiler agent DaemonSet enumerates over CRI. Having more +# than one namespace is the whole point: it proves the +# namespace/pod/container scope resolution against a real +# cluster topology (which docker-compose cannot produce). +apiVersion: v1 +kind: Namespace +metadata: + name: perf-studio + labels: + app.kubernetes.io/part-of: gprofiler-performance-studio + gprofiler.sandbox/role: control-plane +--- +apiVersion: v1 +kind: Namespace +metadata: + name: team-a + labels: + app.kubernetes.io/part-of: gprofiler-performance-studio + gprofiler.sandbox/role: workload +--- +apiVersion: v1 +kind: Namespace +metadata: + name: team-b + labels: + app.kubernetes.io/part-of: gprofiler-performance-studio + gprofiler.sandbox/role: workload diff --git a/deploy/k8s-sandbox/manifests/01-config.yaml b/deploy/k8s-sandbox/manifests/01-config.yaml new file mode 100644 index 00000000..728a09dc --- /dev/null +++ b/deploy/k8s-sandbox/manifests/01-config.yaml @@ -0,0 +1,50 @@ +# Shared, non-secret configuration for the studio control plane. +# Mirrors deploy/.env, but with Kubernetes service DNS names instead of the +# docker-compose service names (k8s Service names cannot contain underscores, +# so db_postgres -> postgres and db_clickhouse -> clickhouse). Each app already +# takes its dependency hosts via env, so only these values change. +apiVersion: v1 +kind: ConfigMap +metadata: + name: studio-config + namespace: perf-studio +data: + AWS_REGION: us-east-1 + AWS_DEFAULT_REGION: us-east-1 + # LocalStack gateway (S3 + SQS emulator) reachable in-cluster by Service name. + AWS_ENDPOINT_URL: http://localstack:4566 + S3_ENDPOINT_URL: http://localstack:4566 + SQS_ENDPOINT_URL: http://localstack:4566 + BUCKET_NAME: performance-studio-bucket + S3_PATH_PREFIX: "" + # The indexer takes a bare queue name; the webapp takes a full URL. Path-style + # LocalStack URL works cross-pod (the localhost.localstack.cloud magic DNS the + # compose .env used only resolves on the host). + SQS_INDEXER_QUEUE_URL: performance-studio-queue + SQS_WEBAPP_QUEUE_URL: http://localstack:4566/000000000000/performance-studio-queue + POSTGRES_HOST: postgres + POSTGRES_PORT: "5432" + POSTGRES_DB: performance_studio + POSTGRES_USER: performance_studio + CLICKHOUSE_HOST: clickhouse + REST_USERNAME: user + COMMON_LOGS_DIR: /logs + # No external metrics sink in the sandbox (compose pointed at host.docker.internal). + METRICS_ENABLED: "false" + INDEXER_METRICS_ENABLED: "false" + MAX_SIMULTANEOUS_PROFILING_HOSTS: "100" +--- +apiVersion: v1 +kind: Secret +metadata: + name: studio-secrets + namespace: perf-studio +type: Opaque +stringData: + POSTGRES_PASSWORD: performance_studio_password + CLICKHOUSE_USER: dbuser + CLICKHOUSE_PASSWORD: simplePassword + REST_PASSWORD: pass + # Dummy credentials; all AWS traffic is served by LocalStack. + AWS_ACCESS_KEY_ID: test + AWS_SECRET_ACCESS_KEY: test diff --git a/deploy/k8s-sandbox/manifests/10-datastores.yaml b/deploy/k8s-sandbox/manifests/10-datastores.yaml new file mode 100644 index 00000000..f8effb90 --- /dev/null +++ b/deploy/k8s-sandbox/manifests/10-datastores.yaml @@ -0,0 +1,117 @@ +# Postgres (heartbeat inventory, profiling requests/commands) and ClickHouse +# (flamedb sample/metric storage). Schemas are loaded from init ConfigMaps that +# the Makefile creates from the same SQL files docker-compose mounts, so there is +# a single source of truth for the schema. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: postgres + namespace: perf-studio + labels: { app: postgres } +spec: + replicas: 1 + selector: + matchLabels: { app: postgres } + template: + metadata: + labels: { app: postgres } + spec: + containers: + - name: postgres + image: postgres:15.1 + imagePullPolicy: IfNotPresent + env: + - name: POSTGRES_USER + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_USER } } + - name: POSTGRES_DB + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_DB } } + - name: POSTGRES_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: POSTGRES_PASSWORD } } + ports: + - containerPort: 5432 + volumeMounts: + - name: initdb + mountPath: /docker-entrypoint-initdb.d + - name: data + mountPath: /var/lib/postgresql/data + readinessProbe: + exec: + command: ["sh", "-c", "pg_isready -U $POSTGRES_USER -d $POSTGRES_DB"] + initialDelaySeconds: 10 + periodSeconds: 5 + volumes: + - name: initdb + configMap: + name: pg-initdb + - name: data + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: postgres + namespace: perf-studio +spec: + selector: { app: postgres } + ports: + - port: 5432 + targetPort: 5432 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: clickhouse + namespace: perf-studio + labels: { app: clickhouse } +spec: + replicas: 1 + selector: + matchLabels: { app: clickhouse } + template: + metadata: + labels: { app: clickhouse } + spec: + containers: + - name: clickhouse + image: clickhouse/clickhouse-server:22.8 + imagePullPolicy: IfNotPresent + env: + - name: CLICKHOUSE_USER + valueFrom: { secretKeyRef: { name: studio-secrets, key: CLICKHOUSE_USER } } + - name: CLICKHOUSE_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: CLICKHOUSE_PASSWORD } } + ports: + - containerPort: 8123 + - containerPort: 9000 + volumeMounts: + - name: initdb + mountPath: /docker-entrypoint-initdb.d + - name: data + mountPath: /var/lib/clickhouse + readinessProbe: + httpGet: + path: /ping + port: 8123 + initialDelaySeconds: 15 + periodSeconds: 10 + volumes: + - name: initdb + configMap: + name: ch-initdb + - name: data + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: clickhouse + namespace: perf-studio +spec: + selector: { app: clickhouse } + ports: + - name: http + port: 8123 + targetPort: 8123 + - name: native + port: 9000 + targetPort: 9000 diff --git a/deploy/k8s-sandbox/manifests/12-localstack.yaml b/deploy/k8s-sandbox/manifests/12-localstack.yaml new file mode 100644 index 00000000..bb3edabe --- /dev/null +++ b/deploy/k8s-sandbox/manifests/12-localstack.yaml @@ -0,0 +1,63 @@ +# LocalStack emulates S3 + SQS in-process so nothing touches real AWS. On startup +# it runs every script in /etc/localstack/init/ready.d (mounted from a ConfigMap +# the Makefile builds from deploy/localstack_init/) to create the bucket + queue +# and wire S3 -> SQS event notifications, exactly like the compose harness. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: localstack + namespace: perf-studio + labels: { app: localstack } +spec: + replicas: 1 + selector: + matchLabels: { app: localstack } + template: + metadata: + labels: { app: localstack } + spec: + containers: + - name: localstack + image: localstack/localstack:3.0 + imagePullPolicy: IfNotPresent + env: + - name: SERVICES + value: s3,sqs + - name: DEBUG + value: "1" + - name: AWS_DEFAULT_REGION + value: us-east-1 + - name: AWS_ACCESS_KEY_ID + value: test + - name: AWS_SECRET_ACCESS_KEY + value: test + ports: + - containerPort: 4566 + volumeMounts: + - name: init + mountPath: /etc/localstack/init/ready.d + readinessProbe: + exec: + command: + - bash + - -c + - "curl -s http://localhost:4566/_localstack/health | grep -q '\"s3\": \"running\"' && curl -s http://localhost:4566/_localstack/health | grep -q '\"sqs\": \"running\"'" + initialDelaySeconds: 15 + periodSeconds: 10 + failureThreshold: 12 + volumes: + - name: init + configMap: + name: localstack-init + defaultMode: 0755 +--- +apiVersion: v1 +kind: Service +metadata: + name: localstack + namespace: perf-studio +spec: + selector: { app: localstack } + ports: + - port: 4566 + targetPort: 4566 diff --git a/deploy/k8s-sandbox/manifests/13-ch-rest-service.yaml b/deploy/k8s-sandbox/manifests/13-ch-rest-service.yaml new file mode 100644 index 00000000..3bd16115 --- /dev/null +++ b/deploy/k8s-sandbox/manifests/13-ch-rest-service.yaml @@ -0,0 +1,72 @@ +# ClickHouse REST facade the webapp queries for flamegraph data. Serves HTTPS on +# 4433 with a self-signed cert (mounted from the ch-rest-tls Secret the Makefile +# creates from deploy/tls/); the webapp trusts it via REST_VERIFY_TLS=FALSE. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ch-rest-service + namespace: perf-studio + labels: { app: ch-rest-service } +spec: + replicas: 1 + selector: + matchLabels: { app: ch-rest-service } + template: + metadata: + labels: { app: ch-rest-service } + spec: + securityContext: + runAsUser: 888 + runAsGroup: 888 + fsGroup: 888 + containers: + - name: ch-rest-service + image: gprofiler-ps/ch-rest-service:sandbox + imagePullPolicy: IfNotPresent + env: + - name: CLICKHOUSE_HOST + valueFrom: { configMapKeyRef: { name: studio-config, key: CLICKHOUSE_HOST } } + - name: CLICKHOUSE_USER + valueFrom: { secretKeyRef: { name: studio-secrets, key: CLICKHOUSE_USER } } + - name: CLICKHOUSE_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: CLICKHOUSE_PASSWORD } } + - name: REST_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: REST_PASSWORD } } + # $(VAR) expands against env vars declared earlier in this list. + - name: CLICKHOUSE_ADDR + value: "$(CLICKHOUSE_HOST):9000?username=$(CLICKHOUSE_USER)&password=$(CLICKHOUSE_PASSWORD)" + - name: CLICKHOUSE_STACKS_TABLE + value: flamedb.samples + - name: CLICKHOUSE_METRICS_TABLE + value: flamedb.metrics + - name: BASIC_AUTH_CREDENTIALS + value: "user:$(REST_PASSWORD)" + - name: CERT_FILE_PATH + value: /tls/ch_rest_cert.pem + - name: KEY_FILE_PATH + value: /tls/ch_rest_key.pem + - name: AWS_ACCESS_KEY_ID + valueFrom: { secretKeyRef: { name: studio-secrets, key: AWS_ACCESS_KEY_ID } } + - name: AWS_SECRET_ACCESS_KEY + valueFrom: { secretKeyRef: { name: studio-secrets, key: AWS_SECRET_ACCESS_KEY } } + ports: + - containerPort: 4433 + volumeMounts: + - name: tls + mountPath: /tls + readOnly: true + volumes: + - name: tls + secret: + secretName: ch-rest-tls +--- +apiVersion: v1 +kind: Service +metadata: + name: ch-rest-service + namespace: perf-studio +spec: + selector: { app: ch-rest-service } + ports: + - port: 4433 + targetPort: 4433 diff --git a/deploy/k8s-sandbox/manifests/20-webapp.yaml b/deploy/k8s-sandbox/manifests/20-webapp.yaml new file mode 100644 index 00000000..b6417c88 --- /dev/null +++ b/deploy/k8s-sandbox/manifests/20-webapp.yaml @@ -0,0 +1,86 @@ +# FastAPI backend + built UI: heartbeat ingest, profile upload, workload status +# and profiling request APIs. Serves plain HTTP on :80 internally (TLS is left to +# the optional nginx edge), so the agent and in-cluster tests reach it directly at +# http://webapp without going through basic auth. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: webapp + namespace: perf-studio + labels: { app: webapp } +spec: + replicas: 1 + selector: + matchLabels: { app: webapp } + template: + metadata: + labels: { app: webapp } + spec: + # The webapp image runs as non_root (uid 888) and binds port 80. Docker + # allows low-port binds by non-root via its default + # net.ipv4.ip_unprivileged_port_start=0; Kubernetes does not, so set the + # (safe) sysctl explicitly to reproduce that behavior. + securityContext: + sysctls: + - name: net.ipv4.ip_unprivileged_port_start + value: "0" + containers: + - name: webapp + image: gprofiler-ps/webapp:sandbox + imagePullPolicy: IfNotPresent + envFrom: + - configMapRef: { name: studio-config } + env: + - name: QUERY_API_BASE_URL + value: https://ch-rest-service:4433 + - name: REST_VERIFY_TLS + value: "FALSE" + - name: REST_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: REST_PASSWORD } } + - name: AWS_ACCESS_KEY_ID + valueFrom: { secretKeyRef: { name: studio-secrets, key: AWS_ACCESS_KEY_ID } } + - name: AWS_SECRET_ACCESS_KEY + valueFrom: { secretKeyRef: { name: studio-secrets, key: AWS_SECRET_ACCESS_KEY } } + # webapp reads Postgres via GPROFILER_POSTGRES_* (not the bare names). + - name: GPROFILER_POSTGRES_HOST + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_HOST } } + - name: GPROFILER_POSTGRES_PORT + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_PORT } } + - name: GPROFILER_POSTGRES_DB_NAME + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_DB } } + - name: GPROFILER_POSTGRES_USERNAME + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_USER } } + - name: GPROFILER_POSTGRES_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: POSTGRES_PASSWORD } } + - name: APP_LOG_LEVEL + value: INFO + - name: APP_LOG_FILE_PATH + value: webapp.log + - name: AWS_METADATA_SERVICE_NUM_ATTEMPTS + value: "100" + # TLS off internally: no cert paths -> webapp serves plain HTTP on :80. + - name: GPROFILER_TLS_CERT_PATH + value: "" + - name: GPROFILER_TLS_KEY_PATH + value: "" + - name: GPROFILER_TLS_CA_PATH + value: "" + ports: + - containerPort: 80 + readinessProbe: + tcpSocket: + port: 80 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 30 +--- +apiVersion: v1 +kind: Service +metadata: + name: webapp + namespace: perf-studio +spec: + selector: { app: webapp } + ports: + - port: 80 + targetPort: 80 diff --git a/deploy/k8s-sandbox/manifests/21-ch-indexer.yaml b/deploy/k8s-sandbox/manifests/21-ch-indexer.yaml new file mode 100644 index 00000000..4a0cc990 --- /dev/null +++ b/deploy/k8s-sandbox/manifests/21-ch-indexer.yaml @@ -0,0 +1,85 @@ +# SQS consumer: reads uploaded profiles from S3 (LocalStack) and writes rows to +# ClickHouse (flamedb.samples) plus the flamegraph HTML back to S3. This is the +# tail of the pipeline the compose "studio-only" setup never exercises. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ch-indexer + namespace: perf-studio + labels: { app: ch-indexer } +spec: + replicas: 1 + selector: + matchLabels: { app: ch-indexer } + template: + metadata: + labels: { app: ch-indexer } + spec: + # Compose gated the indexer on `localstack: service_healthy`; k8s has no + # depends_on, and the indexer resolves the SQS queue URL once at startup and + # does not retry. Without this wait it can boot before LocalStack is up, + # fail queue resolution, and silently never consume. Block until the SQS + # emulator reports running. + initContainers: + - name: wait-for-localstack + image: curlimages/curl:8.7.1 + command: + - sh + - -c + - | + until curl -sf http://localstack:4566/_localstack/health | grep -q '"sqs": "running"'; do + echo "waiting for localstack sqs..."; sleep 3; + done + containers: + - name: ch-indexer + image: gprofiler-ps/ch-indexer:sandbox + imagePullPolicy: IfNotPresent + env: + - name: SQS_QUEUE_URL + valueFrom: { configMapKeyRef: { name: studio-config, key: SQS_INDEXER_QUEUE_URL } } + - name: AWS_REGION + valueFrom: { configMapKeyRef: { name: studio-config, key: AWS_REGION } } + - name: AWS_ENDPOINT_URL + valueFrom: { configMapKeyRef: { name: studio-config, key: AWS_ENDPOINT_URL } } + - name: S3_BUCKET + valueFrom: { configMapKeyRef: { name: studio-config, key: BUCKET_NAME } } + - name: S3_PATH_PREFIX + valueFrom: { configMapKeyRef: { name: studio-config, key: S3_PATH_PREFIX } } + - name: CLICKHOUSE_HOST + valueFrom: { configMapKeyRef: { name: studio-config, key: CLICKHOUSE_HOST } } + - name: CLICKHOUSE_USER + valueFrom: { secretKeyRef: { name: studio-secrets, key: CLICKHOUSE_USER } } + - name: CLICKHOUSE_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: CLICKHOUSE_PASSWORD } } + - name: CLICKHOUSE_ADDR + value: "$(CLICKHOUSE_HOST):9000" + - name: CLICKHOUSE_USE_TLS + value: "false" + - name: CLICKHOUSE_STACKS_TABLE + value: flamedb.samples + - name: CLICKHOUSE_METRICS_TABLE + value: flamedb.metrics + - name: CLICKHOUSE_STACKS_BATCH_SIZE + value: "100000" + - name: CLICKHOUSE_METRICS_BATCH_SIZE + value: "1000" + - name: CONCURRENCY + value: "8" + - name: CACHE_SIZE + value: "2048" + - name: INDEXER_METRICS_ENABLED + valueFrom: { configMapKeyRef: { name: studio-config, key: INDEXER_METRICS_ENABLED } } + - name: AWS_ACCESS_KEY_ID + valueFrom: { secretKeyRef: { name: studio-secrets, key: AWS_ACCESS_KEY_ID } } + - name: AWS_SECRET_ACCESS_KEY + valueFrom: { secretKeyRef: { name: studio-secrets, key: AWS_SECRET_ACCESS_KEY } } + - name: GPROFILER_POSTGRES_HOST + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_HOST } } + - name: GPROFILER_POSTGRES_PORT + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_PORT } } + - name: GPROFILER_POSTGRES_DB_NAME + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_DB } } + - name: GPROFILER_POSTGRES_USERNAME + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_USER } } + - name: GPROFILER_POSTGRES_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: POSTGRES_PASSWORD } } diff --git a/deploy/k8s-sandbox/manifests/22-logs-backend.yaml b/deploy/k8s-sandbox/manifests/22-logs-backend.yaml new file mode 100644 index 00000000..ef6a5eac --- /dev/null +++ b/deploy/k8s-sandbox/manifests/22-logs-backend.yaml @@ -0,0 +1,87 @@ +# Agent log ingest + periodic log-rotation. Co-located in one pod so they can +# share the /logs volume (compose used a shared named volume; an emptyDir shared +# by two containers is the pod-native equivalent and avoids needing RWX storage). +apiVersion: apps/v1 +kind: Deployment +metadata: + name: logs-backend + namespace: perf-studio + labels: { app: logs-backend } +spec: + replicas: 1 + selector: + matchLabels: { app: logs-backend } + template: + metadata: + labels: { app: logs-backend } + spec: + securityContext: + runAsUser: 888 + runAsGroup: 888 + fsGroup: 888 + # agents-logs-backend binds port 80 as non-root; allow the low-port bind + # the same way Docker does by default (see webapp for rationale). + sysctls: + - name: net.ipv4.ip_unprivileged_port_start + value: "0" + containers: + - name: agents-logs-backend + image: gprofiler-ps/agents-logs-backend:sandbox + imagePullPolicy: IfNotPresent + env: + - name: ENV + value: open + - name: APP_LOG_FILE_PATH + value: /logs/agents-logs-app.log + - name: LOG_FILE_PATH + value: /logs/agents-logs.log + - name: GPROFILER_POSTGRES_HOST + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_HOST } } + - name: GPROFILER_POSTGRES_PORT + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_PORT } } + - name: GPROFILER_POSTGRES_DB_NAME + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_DB } } + - name: GPROFILER_POSTGRES_USERNAME + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_USER } } + - name: GPROFILER_POSTGRES_PASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: POSTGRES_PASSWORD } } + ports: + - containerPort: 80 + volumeMounts: + - name: logs + mountPath: /logs + - name: periodic-tasks + image: gprofiler-ps/periodic-tasks:sandbox + imagePullPolicy: IfNotPresent + env: + - name: PGHOST + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_HOST } } + - name: PGPORT + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_PORT } } + - name: PGDATABASE + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_DB } } + - name: PGUSER + valueFrom: { configMapKeyRef: { name: studio-config, key: POSTGRES_USER } } + - name: PGPASSWORD + valueFrom: { secretKeyRef: { name: studio-secrets, key: POSTGRES_PASSWORD } } + - name: LOGROTATE_PATTERN + value: /logs/*.log + - name: LOGROTATE_SIZE + value: 15M + volumeMounts: + - name: logs + mountPath: /logs + volumes: + - name: logs + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: agents-logs-backend + namespace: perf-studio +spec: + selector: { app: logs-backend } + ports: + - port: 80 + targetPort: 80 diff --git a/deploy/k8s-sandbox/manifests/30-nginx.yaml b/deploy/k8s-sandbox/manifests/30-nginx.yaml new file mode 100644 index 00000000..b7fcf8d4 --- /dev/null +++ b/deploy/k8s-sandbox/manifests/30-nginx.yaml @@ -0,0 +1,62 @@ +# OPTIONAL TLS + basic-auth edge, only needed for the browser UI / Playwright +# path. The API acceptance suite and the agent talk to http://webapp directly, so +# the core sandbox works without this. The nginx config, .htpasswd and TLS cert +# come from ConfigMap/Secrets the Makefile builds from the existing deploy/ files +# (https_nginx.conf, .htpasswd, tls/), so there is one source of truth. +# +# Exposed as a NodePort so you can reach the console from the host at +# https://:30443 (see `make -f Makefile.k8s k8s-url`). +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx + namespace: perf-studio + labels: { app: nginx } +spec: + replicas: 1 + selector: + matchLabels: { app: nginx } + template: + metadata: + labels: { app: nginx } + spec: + containers: + - name: nginx + image: nginx:1.23.3 + imagePullPolicy: IfNotPresent + ports: + - containerPort: 443 + volumeMounts: + - name: conf + mountPath: /etc/nginx/nginx.conf + subPath: nginx.conf + - name: htpasswd + mountPath: /etc/nginx/.htpasswd + subPath: .htpasswd + - name: tls + mountPath: /etc/nginx/tls + readOnly: true + volumes: + - name: conf + configMap: + name: nginx-conf + - name: htpasswd + secret: + secretName: nginx-htpasswd + - name: tls + secret: + secretName: nginx-tls +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx + namespace: perf-studio +spec: + type: NodePort + selector: { app: nginx } + ports: + - name: https + port: 443 + targetPort: 443 + nodePort: 30443 diff --git a/deploy/k8s-sandbox/manifests/40-workloads.yaml b/deploy/k8s-sandbox/manifests/40-workloads.yaml new file mode 100644 index 00000000..6ba8132b --- /dev/null +++ b/deploy/k8s-sandbox/manifests/40-workloads.yaml @@ -0,0 +1,163 @@ +# Realistic tenant workloads the agent DaemonSet enumerates over CRI. These give +# the inventory genuine namespace/pod/container/process topology: +# team-a: checkout (2 replicas), web +# team-b: payments, search (2-container pod: app + sidecar) +# Each container runs a deterministic CPU hot loop so py-spy produces stable, +# recognizable frames. kubelet stamps every container with the +# io.kubernetes.pod.{namespace,name,uid} + io.kubernetes.container.name labels +# that heartbeat_metadata.py reads, so no in-app cooperation is required. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: checkout + namespace: team-a + labels: { app.kubernetes.io/name: checkout } +spec: + replicas: 2 + selector: + matchLabels: { app.kubernetes.io/name: checkout } + template: + metadata: + labels: { app.kubernetes.io/name: checkout } + spec: + containers: + - name: checkout + image: python:3.11-slim + imagePullPolicy: IfNotPresent + command: ["python3", "-c"] + args: + - | + import time + def hot_loop(): + total = 0 + for i in range(5_000_000): + total += (i * i) % 7 + return total + while True: + hot_loop() + time.sleep(0.01) + resources: + requests: { cpu: 50m, memory: 32Mi } + limits: { cpu: 250m, memory: 128Mi } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web + namespace: team-a + labels: { app.kubernetes.io/name: web } +spec: + replicas: 1 + selector: + matchLabels: { app.kubernetes.io/name: web } + template: + metadata: + labels: { app.kubernetes.io/name: web } + spec: + containers: + - name: web + image: python:3.11-slim + imagePullPolicy: IfNotPresent + command: ["python3", "-c"] + args: + - | + import time + def serve(): + total = 0 + for i in range(3_000_000): + total += (i * i) % 5 + return total + while True: + serve() + time.sleep(0.02) + resources: + requests: { cpu: 50m, memory: 32Mi } + limits: { cpu: 250m, memory: 128Mi } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: payments + namespace: team-b + labels: { app.kubernetes.io/name: payments } +spec: + replicas: 1 + selector: + matchLabels: { app.kubernetes.io/name: payments } + template: + metadata: + labels: { app.kubernetes.io/name: payments } + spec: + containers: + - name: payments + image: python:3.11-slim + imagePullPolicy: IfNotPresent + command: ["python3", "-c"] + args: + - | + import time + def settle(): + total = 0 + for i in range(4_000_000): + total += (i * i) % 11 + return total + while True: + settle() + time.sleep(0.01) + resources: + requests: { cpu: 50m, memory: 32Mi } + limits: { cpu: 250m, memory: 128Mi } +--- +# Multi-container pod: proves container-scope resolution can distinguish two +# containers inside the same pod. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: search + namespace: team-b + labels: { app.kubernetes.io/name: search } +spec: + replicas: 1 + selector: + matchLabels: { app.kubernetes.io/name: search } + template: + metadata: + labels: { app.kubernetes.io/name: search } + spec: + containers: + - name: search-app + image: python:3.11-slim + imagePullPolicy: IfNotPresent + command: ["python3", "-c"] + args: + - | + import time + def index(): + total = 0 + for i in range(4_500_000): + total += (i * i) % 13 + return total + while True: + index() + time.sleep(0.01) + resources: + requests: { cpu: 50m, memory: 32Mi } + limits: { cpu: 250m, memory: 128Mi } + - name: search-sidecar + image: python:3.11-slim + imagePullPolicy: IfNotPresent + command: ["python3", "-c"] + args: + - | + import time + def tail(): + total = 0 + for i in range(1_000_000): + total += (i * i) % 3 + return total + while True: + tail() + time.sleep(0.05) + resources: + requests: { cpu: 25m, memory: 32Mi } + limits: { cpu: 100m, memory: 64Mi } diff --git a/deploy/k8s-sandbox/manifests/50-agent-daemonset.yaml b/deploy/k8s-sandbox/manifests/50-agent-daemonset.yaml new file mode 100644 index 00000000..0d15c0a9 --- /dev/null +++ b/deploy/k8s-sandbox/manifests/50-agent-daemonset.yaml @@ -0,0 +1,82 @@ +# The real gProfiler agent, one per node (prod shape). THIS is what the sandbox +# exists to exercise: unlike docker-compose, a DaemonSet on a real node can read +# the node's CRI socket and enumerate genuine k8s pods/containers. +# +# Why it lights up namespace/pod/container inventory (see the agent code): +# * granulate_utils CriClient talks to /run/containerd/containerd.sock, but it +# resolves that path under HOST_ROOT_PREFIX = /proc/1/root. With hostPID:true +# PID 1 is the host init, so /proc/1/root is the node root fs and the socket +# is reachable. privileged:true grants the access (and lets py-spy ptrace). +# * CRI returns the io.kubernetes.pod.{namespace,name,uid} + container.name +# labels that heartbeat_metadata.py turns into inventory rows. +# * get_process_container_id() maps every host PID to its container; hostPID +# is required to see PIDs outside the agent's own container. +# +# IMPORTANT: use the SOURCE-built agent image (make -f Makefile.k8s agent-build). +# The container-inventory heartbeat is a fork feature; the public intel/gprofiler +# image does not emit it, so the pod/container tabs would stay empty. +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: gprofiler-agent + namespace: perf-studio + labels: { app: gprofiler-agent } +spec: + selector: + matchLabels: { app: gprofiler-agent } + template: + metadata: + labels: { app: gprofiler-agent } + spec: + hostPID: true + containers: + - name: gprofiler-agent + image: gprofiler-e2e-agent:src + imagePullPolicy: IfNotPresent + securityContext: + privileged: true + command: ["/gprofiler"] + args: + - --server-host=http://webapp + - --api-server=http://webapp + - --token=$(GPROFILER_TOKEN) + - --service-name=k8s-sandbox + - --upload-results + - --enable-heartbeat-server + - --heartbeat-interval=10 + - --perf-mode=none + - --output-dir=/tmp/gprofiler_output + - --dont-send-logs + - --disable-pidns-check + # Per-thread Java profiling: appends ',threads' to async-profiler so + # each sample is split by (and prefixed with) its JVM thread name + # (e.g. "Executor task launch worker-0"). This is the async-profiler + # "thread renaming in samples" capability surfaced via gProfiler. + - --java-async-profiler-args=threads + # Tag Spark executor samples with the Spark application name in the + # appid, so different Spark apps are distinguishable in the data. + - --java-collect-spark-app-name-as-appid + env: + - name: GPROFILER_IN_CONTAINER + value: "1" + - name: GPROFILER_TOKEN + valueFrom: + secretKeyRef: + name: gprofiler-agent-token + key: token + # The agent's own identity (downward API), reported alongside inventory. + - name: POD_NAMESPACE + valueFrom: { fieldRef: { fieldPath: metadata.namespace } } + - name: POD_NAME + valueFrom: { fieldRef: { fieldPath: metadata.name } } + - name: NODE_NAME + valueFrom: { fieldRef: { fieldPath: spec.nodeName } } + resources: + requests: { cpu: 100m, memory: 128Mi } + # bumped: per-thread collapsed output + a busy 6-executor Spark app + # produces a much larger profile to hold/merge. + limits: { cpu: "2", memory: 1Gi } + # Tolerate control-plane taints so the DaemonSet also lands on a + # single-node kind/minikube cluster's control-plane node. + tolerations: + - operator: Exists diff --git a/deploy/k8s-sandbox/spark/00-rbac.yaml b/deploy/k8s-sandbox/spark/00-rbac.yaml new file mode 100644 index 00000000..31e66493 --- /dev/null +++ b/deploy/k8s-sandbox/spark/00-rbac.yaml @@ -0,0 +1,35 @@ +# Namespace + service account the Spark driver uses to create executor pods. +apiVersion: v1 +kind: Namespace +metadata: + name: spark +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spark + namespace: spark +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: spark-role + namespace: spark +rules: + - apiGroups: [""] + resources: ["pods", "pods/log", "services", "configmaps", "persistentvolumeclaims"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: spark-role-binding + namespace: spark +subjects: + - kind: ServiceAccount + name: spark + namespace: spark +roleRef: + kind: Role + name: spark-role + apiGroup: rbac.authorization.k8s.io diff --git a/deploy/k8s-sandbox/spark/10-submit-job.yaml b/deploy/k8s-sandbox/spark/10-submit-job.yaml new file mode 100644 index 00000000..cb7ab687 --- /dev/null +++ b/deploy/k8s-sandbox/spark/10-submit-job.yaml @@ -0,0 +1,67 @@ +# A one-shot pod that runs spark-submit in k8s "cluster" deploy mode. spark-submit +# creates the driver pod, which in turn creates the executor pods. Cluster mode is +# used because Spark then wires driver<->executor networking automatically (it +# creates a headless service for the driver), avoiding client-mode host/port setup. +apiVersion: batch/v1 +kind: Job +metadata: + name: spark-submit + namespace: spark +spec: + backoffLimit: 0 + template: + metadata: + labels: + app: spark-submit + spec: + serviceAccountName: spark + restartPolicy: Never + containers: + - name: submit + image: spark-cpu-job:sandbox + imagePullPolicy: IfNotPresent + command: + - /opt/spark/bin/spark-submit + - --master + - k8s://https://kubernetes.default.svc:443 + - --deploy-mode + - cluster + - --name + - spark-cpu + - --conf + - spark.kubernetes.namespace=spark + - --conf + - spark.kubernetes.authenticate.driver.serviceAccountName=spark + - --conf + - spark.kubernetes.authenticate.submission.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt + - --conf + - spark.kubernetes.authenticate.submission.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token + - --conf + - spark.kubernetes.container.image=spark-cpu-job:sandbox + - --conf + - spark.kubernetes.container.image.pullPolicy=IfNotPresent + - --conf + - spark.kubernetes.driver.pod.name=spark-cpu-driver + - --conf + - spark.kubernetes.submission.waitAppCompletion=true + - --conf + - spark.executor.instances=6 + - --conf + - spark.executor.cores=4 + # Decouple task parallelism from the k8s CPU *request* so all 6 executor + # pods schedule on a 16-core node (6x4=24 threads > 16 cores => oversubscribed + # and deliberately very busy), instead of requesting 4 cores each and leaving + # half the executors Pending. + - --conf + - spark.kubernetes.executor.request.cores=1 + - --conf + - spark.executor.memory=1g + - --conf + - spark.driver.cores=1 + - --conf + - spark.driver.memory=1g + # keep executors around briefly after finish so they're easy to inspect + - --conf + - spark.kubernetes.executor.deleteOnTermination=true + - local:///opt/spark_cpu_job.py + - "600" diff --git a/deploy/k8s-sandbox/spark/11-submit-workload.yaml b/deploy/k8s-sandbox/spark/11-submit-workload.yaml new file mode 100644 index 00000000..c6045d5b --- /dev/null +++ b/deploy/k8s-sandbox/spark/11-submit-workload.yaml @@ -0,0 +1,73 @@ +# Templated spark-submit Job for ONE workload mode. The Makefile `spark-multi` +# target sed-substitutes the __PLACEHOLDERS__ and applies one copy per mode, so +# several distinct Spark apps run side by side. Each app is smaller than the +# single-app demo (2 executors x 2 cores) so 3-4 apps fit on the 16-core node. +# +# Placeholders: __SUFFIX__ (mode), __MODE__, __APPNAME__, __SECONDS__ +apiVersion: batch/v1 +kind: Job +metadata: + name: spark-submit-__SUFFIX__ + namespace: spark +spec: + backoffLimit: 0 + template: + metadata: + labels: + app: spark-submit + workload: __SUFFIX__ + spec: + serviceAccountName: spark + restartPolicy: Never + containers: + - name: submit + image: spark-cpu-job:sandbox + imagePullPolicy: IfNotPresent + command: + - /opt/spark/bin/spark-submit + - --master + - k8s://https://kubernetes.default.svc:443 + - --deploy-mode + - cluster + - --name + - __APPNAME__ + - --conf + - spark.kubernetes.namespace=spark + - --conf + - spark.kubernetes.authenticate.driver.serviceAccountName=spark + - --conf + - spark.kubernetes.authenticate.submission.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt + - --conf + - spark.kubernetes.authenticate.submission.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token + - --conf + - spark.kubernetes.container.image=spark-cpu-job:sandbox + - --conf + - spark.kubernetes.container.image.pullPolicy=IfNotPresent + - --conf + - spark.kubernetes.driver.pod.name=spark-__SUFFIX__-driver + # Label the executor pods so `kubectl get pods -l workload=` groups them. + - --conf + - spark.kubernetes.executor.label.workload=__SUFFIX__ + - --conf + - spark.kubernetes.submission.waitAppCompletion=true + # Per-app footprint kept small so several apps schedule concurrently: + # 2 executors x 2 cores = 4 task threads/app, but request only 1 core each. + - --conf + - spark.executor.instances=2 + - --conf + - spark.executor.cores=2 + - --conf + - spark.kubernetes.executor.request.cores=1 + - --conf + - spark.executor.memory=1g + - --conf + - spark.driver.cores=1 + - --conf + - spark.driver.memory=1g + - --conf + - spark.kubernetes.executor.deleteOnTermination=true + - local:///opt/spark_workloads.py + - __MODE__ + - "__SECONDS__" + - __APPNAME__ + - "__PARTS__" diff --git a/deploy/k8s-sandbox/spark/Dockerfile b/deploy/k8s-sandbox/spark/Dockerfile new file mode 100644 index 00000000..8b5271ed --- /dev/null +++ b/deploy/k8s-sandbox/spark/Dockerfile @@ -0,0 +1,6 @@ +# Thin layer over the official Spark image that bakes in the CPU-burn job so the +# driver/executor pods can reference it via local:// (no file upload needed). +FROM apache/spark:3.5.1 + +COPY spark_cpu_job.py /opt/spark_cpu_job.py +COPY spark_workloads.py /opt/spark_workloads.py diff --git a/deploy/k8s-sandbox/spark/README.md b/deploy/k8s-sandbox/spark/README.md new file mode 100644 index 00000000..84b6cfe8 --- /dev/null +++ b/deploy/k8s-sandbox/spark/README.md @@ -0,0 +1,134 @@ +# Spark-on-Kubernetes profiling demos + +Two demos that run Spark as real k8s pods so the gProfiler agent DaemonSet +profiles the executor JVMs per thread: + +| demo | what runs | use it to see | +|------|-----------|---------------| +| **single** (`spark-demo`) | one CPU-burn app: 6 executors x 4 task threads | per-thread Java flamegraph of one busy Spark app | +| **multi** (`spark-multi-demo`) | 4 *different* apps side by side | **relative weight** of different workloads | + +## Why "multi"? (relative weight) + +When every executor runs the same kernel, all the per-thread flamegraphs look +alike. `spark-multi` submits four apps that each hammer a different hot path, so +the profile shows visibly different stacks and lets you compare where CPU goes: + +| mode | app name | dominant frames | +|------|----------|-----------------| +| `agg` | `spark-agg` | `WholeStageCodegen` + transcendental math (`sin/cos/sqrt/log`) | +| `join` | `spark-join` | sort-merge join: `UnsafeExternalSorter`, `ShuffleWriter`, `Sorter` | +| `regex` | `spark-regex` | `java.util.regex` (`Pattern`/`Matcher`) + codegen | +| `pyudf` | `spark-pyudf` | Python UDF: JVM `PythonRunner` + Python worker CPU (`appid: pyspark`) | +| `skew` | `spark-skew` | data skew: ONE task-launch-worker thread dominates, the rest finish fast | + +`agg/join/regex/pyudf` make the *apps* differ. `skew` makes the *threads within one +app* differ -- ~98% of rows land on one partition so a single "Executor task launch +worker" thread is far wider than its peers (vs the uniform widths you get when every +task does equal work). + +**Preferred readable demo** (one app, few partitions, short profile): + +```bash +# 4 partitions keep the per-thread forest small; skew makes one task ~3-4x wider. +make -f Makefile.k8s spark-multi \ + SPARK_MODES="skew" SPARK_PARTITIONS=4 SPARK_MULTI_SECONDS=500 +make -f Makefile.k8s spark-profile SPARK_PROFILE_SECONDS=30 +``` + +Measured on the sandbox (one 30s profile while skew was busy): + +``` +task 1: 53.7% ################################ <- hot partition +task 3: 18.5% ########### +task 2: 15.3% ######### +task 0: 12.5% ####### +``` + +`SPARK_PARTITIONS` (default 24) controls how many task-thread columns appear under +async-profiler's per-sample rename. Lower it when the flamegraph looks like a +forest of equal slivers. + +Example from one 120s host profile (8 executor JVMs, 16-core node): + +``` +spark-pyudf 43.9% of Spark CPU (Python worker dominated) +spark-regex 31.6% +spark-join 14.0% +spark-agg 10.4% +``` + +## Running + +```bash +cd deploy/k8s-sandbox + +make -f Makefile.k8s spark-demo # single app, one-shot +make -f Makefile.k8s spark-multi-demo # 4 distinct apps, one-shot + +# granular +make -f Makefile.k8s spark-multi SPARK_MODES="agg join regex pyudf" SPARK_MULTI_SECONDS=1200 +make -f Makefile.k8s spark-status +make -f Makefile.k8s spark-profile SPARK_PROFILE_SECONDS=120 +make -f Makefile.k8s spark-clean +``` + +## Reading the per-thread flamegraph + +The agent runs async-profiler in `threads` mode, so each stack is prefixed with its +JVM thread name. Spark task threads look like: + +``` +Executor task launch worker for task 1.0 in stage 165.0 (TID 1376) tid=334 + │ │ │ │ └ OS thread id + │ │ │ └ Spark task id (unique within the app) + │ │ └ stage id (within the app) + └────┴ taskId.attempt +``` + +That identifies the task/stage/thread **within one JVM**, but NOT which app -- stage +and TID numbering restart per app. The app is a *separate* frame in every sample: the +executor pod, e.g. `k8s_spark_spark---exec-N_spark_...`. To attribute a +thread to an app: + +- Switch the flamegraph's top grouping from `appid` to **container / process name** + (or filter by pod). Then each thread rolls up under its `spark-` pod. +- Or, in the raw collapsed `.gz`, group by the `spark--...-exec-N` frame. + +Why they all share one `appid: java: org.apache.spark...KubernetesExecutorBackend`: +see gotcha #2 below. If you want the appid dropdown itself to separate the apps, +patch the agent's `_JavaSparkApplicationIdentifier` to also match the k8s executor +backend class (it already reads `spark.app.name`), then `make agent-build` + rollout. + +## Files + +- `spark_cpu_job.py` — single-app CPU burn (used by `spark-demo`). +- `spark_workloads.py` — multi-mode workloads (used by `spark-multi`), dispatched by `` arg. +- `Dockerfile` — thin layer over `apache/spark:3.5.1` baking both scripts in. +- `00-rbac.yaml` — `spark` namespace + ServiceAccount/Role so the driver can create executor pods. +- `10-submit-job.yaml` — single-app `spark-submit` Job (cluster mode). +- `11-submit-workload.yaml` — templated per-mode Job (`__SUFFIX__/__MODE__/__APPNAME__/__SECONDS__/__PARTS__`). + +## Gotchas (learned the hard way) + +1. **Profile while the apps are running.** The Spark Jobs exit after + `SPARK_MULTI_SECONDS`. If they finish before the profile window, async-profiler + has no JVMs to attach to and you get a Python-only capture. `spark-*-demo` + size the run window to cover profiling; if you drive it manually, submit with a + long duration and profile promptly (`spark-status` should show executors + `Running`). +2. **appid does not split Spark apps on k8s 3.5.** All executors report + `appid: java: org.apache.spark...KubernetesExecutorBackend`, so + `--java-collect-spark-app-name-as-appid` won't separate them in the UI. + gProfiler's Spark detector (`_JavaSparkApplicationIdentifier`) matches + `org.apache.spark.executor` in argv, which the k8s executor backend main class + no longer contains. Group by the executor **pod** (its name is a frame in every + sample) or the `workload=` pod label instead. +3. **Uniform workloads + per-thread naming = a forest.** Spark renames the + executor task thread for *every* task (`... for task X in stage Y [TID N]`), + so a busy uniform app fans into one thin column per task. Use `skew` (or + lower `SPARK_PARTITIONS`) when you want a readable per-thread view. +4. **Wedged ad-hoc agent.** If `spark-profile` returns success but no new `.gz` + lands in S3 / the UI stays stale, the agent may be stuck re-skipping a prior + command (`Command ID ... already received, skipping`). Restart it: + `kubectl -n perf-studio rollout restart ds/gprofiler-agent`. diff --git a/deploy/k8s-sandbox/spark/spark_cpu_job.py b/deploy/k8s-sandbox/spark/spark_cpu_job.py new file mode 100644 index 00000000..b09450f6 --- /dev/null +++ b/deploy/k8s-sandbox/spark/spark_cpu_job.py @@ -0,0 +1,44 @@ +"""CPU-heavy Spark job whose hot path stays inside the executor JVMs. + +It uses only Catalyst/DataFrame built-ins (no Python UDFs), so all the arithmetic +runs as whole-stage-codegen'd bytecode on the executor JVM task threads -- exactly +the Java threads we want gProfiler to profile. The Python driver just orchestrates. + +Arg 1 (optional): how many seconds to keep the cluster busy (default 300). +""" +import sys +import time + +from pyspark.sql import SparkSession +from pyspark.sql.functions import col, sqrt, sin, cos, log, expr + +RUN_SECONDS = int(sys.argv[1]) if len(sys.argv) > 1 else 300 +# 48 partitions keeps all executor task threads (6 execs x 4 cores = 24 slots) +# saturated with a backlog, so every JVM worker thread stays busy. +PARTITIONS = 48 +ROWS = 40_000_000 + +spark = ( + SparkSession.builder.appName("spark-cpu-burn") + .getOrCreate() +) +print(f">> spark-cpu-burn starting: run_seconds={RUN_SECONDS} " + f"partitions={PARTITIONS} rows={ROWS}", flush=True) + +deadline = time.time() + RUN_SECONDS +iteration = 0 +while time.time() < deadline: + df = spark.range(0, ROWS, numPartitions=PARTITIONS) + # Stack several transcendental ops so each row costs real CPU in the JVM. + for _ in range(8): + df = ( + df.withColumn("v", sqrt(col("id") + 1.0)) + .withColumn("v", sin(col("v")) * cos(col("v")) + log(col("v") + 2.0)) + .withColumn("id", (col("id") + 1) % ROWS) + ) + total = df.agg(expr("sum(v) as s")).collect()[0]["s"] + iteration += 1 + print(f">> iteration {iteration} done sum={total}", flush=True) + +print(">> spark-cpu-burn finished", flush=True) +spark.stop() diff --git a/deploy/k8s-sandbox/spark/spark_workloads.py b/deploy/k8s-sandbox/spark/spark_workloads.py new file mode 100644 index 00000000..f2678b12 --- /dev/null +++ b/deploy/k8s-sandbox/spark/spark_workloads.py @@ -0,0 +1,136 @@ +"""Several DISTINCT CPU-heavy Spark workloads, each with its own Spark app name. + +Why this exists: when every executor runs the same kernel, the per-thread +flamegraphs all look alike. Running a few *different* apps side by side gives +gProfiler visibly different stacks and lets you compare RELATIVE weight across +apps. + +How to tell the apps apart in the profile: each app's executor pods are named +`spark---exec-N` and that pod identity appears as a frame in every +sample (k8s_spark_spark--...-exec-N_spark_...), so you group/filter by +workload there. NOTE: on Spark 3.5's k8s executors the built-in +--java-collect-spark-app-name-as-appid does NOT split the apps -- they all share +`appid: java: org.apache.spark...KubernetesExecutorBackend` (gProfiler's spark +detector keys off `org.apache.spark.executor` in argv, which the k8s executor +backend main class no longer matches). Group by the executor pod / the +`workload=` label instead. See spark/README.md. + +Each mode stresses a different Catalyst/JVM path: + agg -> whole-stage codegen + transcendental math (HashAggregate, codegen) + join -> shuffle + sort-merge join (ExternalSorter, ShuffleWriter, SMJ) + regex -> string/regex parsing (java.util.regex, UTF8String) + pyudf -> Python UDF round-trips (JVM PythonRunner + socket, Python worker CPU) + skew -> deliberate data skew so ONE task thread dominates the flamegraph + (the others finish fast) -- use it to see unequal per-thread weight + +Usage: spark_workloads.py [app_name] +""" +import math +import sys +import time + +from pyspark.sql import SparkSession +from pyspark.sql import functions as F +from pyspark.sql.types import DoubleType + +MODE = sys.argv[1] if len(sys.argv) > 1 else "agg" +RUN_SECONDS = int(sys.argv[2]) if len(sys.argv) > 2 else 300 +APP_NAME = sys.argv[3] if len(sys.argv) > 3 else f"spark-{MODE}" +# Fewer partitions -> fewer tasks -> fewer per-thread columns in the flamegraph +# (each task becomes its own thread-name frame under per-sample renaming). +PARTITIONS = int(sys.argv[4]) if len(sys.argv) > 4 else 24 + +spark = SparkSession.builder.appName(APP_NAME).getOrCreate() +print(f">> {APP_NAME} starting: mode={MODE} run_seconds={RUN_SECONDS} " + f"partitions={PARTITIONS}", flush=True) + + +def run_agg(): + """Transcendental math folded into whole-stage codegen -> pure JVM CPU.""" + rows = 20_000_000 + df = spark.range(0, rows, numPartitions=PARTITIONS) + for _ in range(8): + df = (df.withColumn("v", F.sqrt(F.col("id") + 1.0)) + .withColumn("v", F.sin(F.col("v")) * F.cos(F.col("v")) + + F.log(F.col("v") + 2.0)) + .withColumn("id", (F.col("id") + 1) % rows)) + return df.agg(F.expr("sum(v) as s")).collect()[0]["s"] + + +def run_join(): + """1:1 sort-merge join across two big frames -> heavy shuffle + external sort.""" + rows = 8_000_000 + left = (spark.range(0, rows, numPartitions=PARTITIONS) + .withColumn("k", (F.col("id") * 2654435761) % rows) + .withColumnRenamed("id", "lid")) + right = (spark.range(0, rows, numPartitions=PARTITIONS) + .withColumn("k", (F.col("id") * 40503) % rows) + .withColumnRenamed("id", "rid")) + joined = left.join(right, on="k", how="inner") + return joined.agg(F.count(F.lit(1)).alias("c")).collect()[0]["c"] + + +def run_regex(): + """Regex extract/replace over synthetic strings -> java.util.regex hot path.""" + rows = 8_000_000 + base = spark.range(0, rows, numPartitions=PARTITIONS) + s = base.withColumn( + "s", + F.concat(F.lit("id-"), F.col("id").cast("string"), + F.lit("-x9y8z7-"), (F.col("id") % 9973).cast("string")), + ) + for _ in range(6): + s = (s.withColumn("d", F.regexp_extract(F.col("s"), r"id-(\d+)-", 1)) + .withColumn("up", F.upper(F.col("s"))) + .withColumn("rep", F.regexp_replace(F.col("s"), r"[0-9]", "#"))) + return s.agg(F.sum(F.length(F.col("rep")))).collect()[0][0] + + +def run_pyudf(): + """Python UDF forces rows through Python workers -> distinct PythonRunner/ + socket stacks in the JVM (plus CPU burned in the python worker processes).""" + rows = 1_500_000 + + @F.udf(DoubleType()) + def churn(x): + v = float(x) + for _ in range(200): + v = math.sin(v) * math.cos(v) + math.sqrt(abs(v) + 1.0) + return v + + df = spark.range(0, rows, numPartitions=PARTITIONS).withColumn("v", churn(F.col("id"))) + return df.agg(F.sum("v")).collect()[0][0] + + +def run_skew(): + """DELIBERATE DATA SKEW: ~98% of rows collapse onto a single partition key, so + after the repartition ONE downstream task gets almost all the rows and its + 'Executor task launch worker' thread dominates the flamegraph while the other + task threads finish quickly. Use this to see one thread far wider than the rest + (vs the uniform-width threads you get when every task does equal work).""" + rows = 12_000_000 + base = spark.range(0, rows, numPartitions=PARTITIONS) + # id % 50 != 0 -> key 0 (98% of rows); else a spread key. One hot partition. + keyed = base.withColumn("p", F.when(F.col("id") % 50 != 0, F.lit(0)).otherwise(F.col("id"))) + skewed = keyed.repartition(PARTITIONS, "p") + # Heavy per-row math in the skewed stage so the hot task actually burns CPU. + for _ in range(6): + skewed = (skewed.withColumn("v", F.sqrt(F.col("id") + 1.0)) + .withColumn("v", F.sin(F.col("v")) * F.cos(F.col("v")) + + F.log(F.col("v") + 2.0))) + return skewed.agg(F.sum("v").alias("s")).collect()[0]["s"] + + +RUNNERS = {"agg": run_agg, "join": run_join, "regex": run_regex, + "pyudf": run_pyudf, "skew": run_skew} +runner = RUNNERS.get(MODE, run_agg) + +deadline = time.time() + RUN_SECONDS +iteration = 0 +while time.time() < deadline: + result = runner() + iteration += 1 + print(f">> {APP_NAME} iter {iteration} result={result}", flush=True) + +print(f">> {APP_NAME} finished", flush=True) +spark.stop() diff --git a/deploy/k8s-sandbox/tests/Dockerfile b/deploy/k8s-sandbox/tests/Dockerfile new file mode 100644 index 00000000..e004a9c6 --- /dev/null +++ b/deploy/k8s-sandbox/tests/Dockerfile @@ -0,0 +1,25 @@ +# In-cluster acceptance runner for the k8s sandbox. +# +# BUILD CONTEXT MUST BE THE STUDIO REPO ROOT (gprofiler-performance-studio/) so +# it can copy both the shared HTTP harness from the compose e2e suite and the +# k8s-specific tests: +# +# docker build -f deploy/k8s-sandbox/tests/Dockerfile -t gprofiler-ps/k8s-tests:sandbox . +FROM python:3.11-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 + +WORKDIR /work + +RUN pip install --no-cache-dir \ + pytest==8.2.0 \ + requests==2.32.3 + +# Shared request/response harness (single source of truth with the compose suite). +COPY src/tests/e2e/harness.py /work/harness.py +# k8s-specific real-topology tests. +COPY deploy/k8s-sandbox/tests/conftest.py /work/conftest.py +COPY deploy/k8s-sandbox/tests/test_k8s_inventory.py /work/test_k8s_inventory.py + +CMD ["pytest", "-q", "-rA", "test_k8s_inventory.py"] diff --git a/deploy/k8s-sandbox/tests/conftest.py b/deploy/k8s-sandbox/tests/conftest.py new file mode 100644 index 00000000..b9e470d9 --- /dev/null +++ b/deploy/k8s-sandbox/tests/conftest.py @@ -0,0 +1,14 @@ +"""Fixtures for the k8s-sandbox acceptance suite. + +Reuses the HTTP harness from the compose e2e suite (copied in at image build +time) so the request/response contract stays in one place. Runs in-cluster as a +Job in the perf-studio namespace, reaching the backend at http://webapp. +""" +import pytest + +from harness import Client + + +@pytest.fixture(scope="session") +def client() -> Client: + return Client() diff --git a/deploy/k8s-sandbox/tests/job-api.yaml b/deploy/k8s-sandbox/tests/job-api.yaml new file mode 100644 index 00000000..85b004b6 --- /dev/null +++ b/deploy/k8s-sandbox/tests/job-api.yaml @@ -0,0 +1,21 @@ +# Runs the real-topology acceptance suite (AT-K1..K5) in-cluster against the live +# webapp Service. Recreate it each run (kubectl delete --ignore-not-found first). +apiVersion: batch/v1 +kind: Job +metadata: + name: k8s-acceptance + namespace: perf-studio +spec: + backoffLimit: 0 + template: + metadata: + labels: { app: k8s-acceptance } + spec: + restartPolicy: Never + containers: + - name: tests + image: gprofiler-ps/k8s-tests:sandbox + imagePullPolicy: IfNotPresent + env: + - name: E2E_BASE_URL + value: http://webapp diff --git a/deploy/k8s-sandbox/tests/test_k8s_inventory.py b/deploy/k8s-sandbox/tests/test_k8s_inventory.py new file mode 100644 index 00000000..f2e41938 --- /dev/null +++ b/deploy/k8s-sandbox/tests/test_k8s_inventory.py @@ -0,0 +1,129 @@ +"""Real-cluster acceptance tests (AT-K1 .. AT-K5). + +Unlike the compose e2e suite (which POSTs *synthetic* heartbeats to fabricate an +inventory), these assert against inventory produced by a REAL gProfiler agent +DaemonSet enumerating the node's CRI socket. That is the capability docker-compose +cannot provide (no container runtime -> empty inventory), so this is the layer +that actually proves namespace/pod/container/process discovery and scope +resolution under Kubernetes. + +Prerequisite: the sandbox is up with the SOURCE-built agent and the tenant +workloads deployed (see deploy/k8s-sandbox/Makefile.k8s): + + make -f Makefile.k8s k8s-up + make -f Makefile.k8s k8s-test + +The agent reports under service_name "k8s-sandbox". Because it enumerates the +whole node, inventory also contains system/studio containers; every assertion +therefore looks for the specific tenant entities we deployed rather than exact +totals, which keeps the suite robust on any cluster. +""" +import json +import time + +import harness as h +import pytest + +AGENT_SERVICE = "k8s-sandbox" + +# What deploy/k8s-sandbox/manifests/40-workloads.yaml creates. +EXPECTED_NAMESPACES = ["team-a", "team-b"] +EXPECTED_PODS = ["checkout", "web", "payments", "search"] +EXPECTED_CONTAINERS = ["checkout", "web", "payments", "search-app", "search-sidecar"] + +# The agent inventory refresh is 30s and heartbeats every 10s; give the first +# full snapshot generous time to propagate on a cold cluster. +DISCOVERY_TIMEOUT_S = 240 +POLL_INTERVAL_S = 5 + + +def _status_blob(client, scope): + """workload_status for a scope, as (parsed, serialized) for tolerant matching. + + Per-scope row field names are intentionally not hard-coded; we match on the + serialized document so the test doesn't break if the response key casing + changes. tabCounts and host rows use the same stable contract as the compose + suite. + """ + status = h.get_workload_status(client, scope=scope) + return status, json.dumps(status) + + +def _wait_for(client, scope, needles): + """Poll workload_status[scope] until every needle appears, or time out.""" + deadline = time.time() + DISCOVERY_TIMEOUT_S + missing = list(needles) + blob = "" + while time.time() < deadline: + _, blob = _status_blob(client, scope) + missing = [n for n in needles if n not in blob] + if not missing: + return blob + time.sleep(POLL_INTERVAL_S) + pytest.fail( + f"scope={scope}: timed out after {DISCOVERY_TIMEOUT_S}s waiting for " + f"{missing}. Is the SOURCE-built agent DaemonSet running with hostPID + " + f"privileged and a reachable containerd socket? Last blob: {blob[:2000]}" + ) + + +def test_at_k1_agent_registers_as_host(client): + """AT-K1: the real agent DaemonSet shows up as a host under its service.""" + deadline = time.time() + DISCOVERY_TIMEOUT_S + hosts = [] + while time.time() < deadline: + status = h.get_workload_status(client, scope="host", service_name=AGENT_SERVICE) + hosts = [r["hostname"] for r in h.rows_for(status, AGENT_SERVICE)] + if hosts: + break + time.sleep(POLL_INTERVAL_S) + assert hosts, f"no host reported for service {AGENT_SERVICE!r} within {DISCOVERY_TIMEOUT_S}s" + + +def test_at_k2_namespaces_from_real_cri(client): + """AT-K2: tenant namespaces are discovered from the node's CRI, not fabricated.""" + blob = _wait_for(client, "namespace", EXPECTED_NAMESPACES) + for ns in EXPECTED_NAMESPACES: + assert ns in blob + + +def test_at_k3_pods_from_real_cri(client): + """AT-K3: pods for each tenant workload appear in the pod scope.""" + # Deployment pods are named --; the agent + # derives the workload name, so matching the workload prefix is sufficient. + blob = _wait_for(client, "pod", EXPECTED_PODS) + for pod in EXPECTED_PODS: + assert pod in blob + + +def test_at_k4_containers_including_multi_container_pod(client): + """AT-K4: every container is discovered, including both in the 2-container pod.""" + blob = _wait_for(client, "container", EXPECTED_CONTAINERS) + for container in EXPECTED_CONTAINERS: + assert container in blob, f"missing container {container!r}" + + +def test_at_k5_host_scope_start_resolves_real_agent(client): + """AT-K5: a host-scope start resolves the real agent's host and creates a command. + + Uses the same start/stop contract the compose suite validates (AT-S5), but the + target host is the *real* DaemonSet host discovered from live inventory, so + this proves the studio resolves and dispatches to an actual agent (not a + synthetic heartbeat). + """ + status = h.get_workload_status(client, scope="host", service_name=AGENT_SERVICE) + hosts = [r["hostname"] for r in h.rows_for(status, AGENT_SERVICE)] + assert hosts, "no real agent host to target" + host = hosts[0] + + started = h.submit( + client, + h.start_request(AGENT_SERVICE, target_scope="host", target_hosts={host: []}), + ) + assert started.status_code == 200, started.text + + stopped = h.submit( + client, + h.stop_request(AGENT_SERVICE, target_scope="host", stop_level="host", target_hosts={host: []}), + ) + assert stopped.status_code == 200, stopped.text diff --git a/deploy/periodic_tasks/Dockerfile b/deploy/periodic_tasks/Dockerfile index 66d567b0..e7219df2 100644 --- a/deploy/periodic_tasks/Dockerfile +++ b/deploy/periodic_tasks/Dockerfile @@ -4,27 +4,33 @@ RUN apk add --no-cache dcron wget rsync ca-certificates postgresql-client logrot COPY crontab /etc/cron.d/my-cron-job COPY aggregations.sh /aggregations.sh +COPY refresh_workload_snapshot.sh /refresh_workload_snapshot.sh COPY logrotate_conf.sh /logrotate_conf.sh RUN chmod +x /logrotate_conf.sh RUN chmod 0644 /etc/cron.d/my-cron-job RUN chmod +x /aggregations.sh +RUN chmod +x /refresh_workload_snapshot.sh RUN crontab /etc/cron.d/my-cron-job -RUN touch /var/log/cron.log +RUN mkdir -p /var/log/cron && touch /var/log/cron/cron.log RUN addgroup -S non_root && adduser -S -G non_root -u 888 non_root && \ chown -R non_root:non_root /aggregations.sh && \ + chown -R non_root:non_root /refresh_workload_snapshot.sh && \ chown -R non_root:non_root /logrotate_conf.sh && \ chown -R non_root:non_root /etc/logrotate.conf && \ chown -R non_root:non_root /etc/cron.d/my-cron-job && \ - chown -R non_root:non_root /var/log/cron.log + chown -R non_root:non_root /var/log/cron RUN echo "non_root ALL=(ALL) NOPASSWD: $(which crond)" >> /etc/sudoers USER non_root -CMD /logrotate_conf.sh && sudo crond && tail -f /var/log/cron.log +# cron jobs do not inherit the container env, so psql couldn't find PGHOST/etc. +# Persist the runtime env to a file that the cron scripts load on each run. +# touch cron.log at runtime: a bind mount over /var/log/cron hides the build-time file. +CMD printenv > /tmp/cron.env && /logrotate_conf.sh && sudo crond && touch /var/log/cron/cron.log && tail -f /var/log/cron/cron.log diff --git a/deploy/periodic_tasks/aggregations.sh b/deploy/periodic_tasks/aggregations.sh index b32ff6c1..f502a7e3 100644 --- a/deploy/periodic_tasks/aggregations.sh +++ b/deploy/periodic_tasks/aggregations.sh @@ -17,4 +17,13 @@ # echo "aggregation started" +# cron jobs do not inherit the container env; load the DB connection vars the +# container start-up persisted (see periodic_tasks/Dockerfile). +if [ -f /tmp/cron.env ]; then + while IFS='=' read -r _k _v; do + case "$_k" in + PGHOST|PGPORT|PGUSER|PGPASSWORD|PGDATABASE) export "$_k=$_v" ;; + esac + done < /tmp/cron.env +fi psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE -c "CALL update_profiler_service_hourly_usages()" diff --git a/deploy/periodic_tasks/crontab b/deploy/periodic_tasks/crontab index 99322f94..6fba05e8 100644 --- a/deploy/periodic_tasks/crontab +++ b/deploy/periodic_tasks/crontab @@ -1,2 +1,3 @@ -10 * * * * /aggregations.sh >> /var/log/cron.log 2>&1 +10 * * * * /aggregations.sh >> /var/log/cron/cron.log 2>&1 +* * * * * flock -n /tmp/workload_snapshot.lock /refresh_workload_snapshot.sh >> /var/log/cron/cron.log 2>&1 */15 * * * * /usr/sbin/logrotate /etc/logrotate.conf diff --git a/deploy/periodic_tasks/refresh_workload_snapshot.sh b/deploy/periodic_tasks/refresh_workload_snapshot.sh new file mode 100644 index 00000000..faf4a6b1 --- /dev/null +++ b/deploy/periodic_tasks/refresh_workload_snapshot.sh @@ -0,0 +1,36 @@ +#!/bin/sh + +# +# Copyright (C) 2023 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Rebuilds the precomputed workload_status Layer 2 (tab counts + coarse-scope +# summaries) and atomically swaps it in. cron invokes this once per minute (~60s +# cadence). The procedure self-guards with an advisory lock, so an in-progress +# build is never queued behind -- a concurrent invocation simply skips. + +# cron jobs do not inherit the container env; load the DB connection vars the +# container start-up persisted (see periodic_tasks/Dockerfile). +if [ -f /tmp/cron.env ]; then + while IFS='=' read -r _k _v; do + case "$_k" in + PGHOST|PGPORT|PGUSER|PGPASSWORD|PGDATABASE) export "$_k=$_v" ;; + esac + done < /tmp/cron.env +fi + +echo "workload snapshot refresh started ($(date -u +%FT%TZ))" +psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" \ + -c "CALL refresh_workload_snapshot()" diff --git a/docs/K8S_SANDBOX.md b/docs/K8S_SANDBOX.md new file mode 100644 index 00000000..b3037d64 --- /dev/null +++ b/docs/K8S_SANDBOX.md @@ -0,0 +1,281 @@ +# Kubernetes sandbox for workload-level profiling + +Architecture and rationale for the local Kubernetes sandbox that validates +workload-level (namespace / pod / container / process) profiling against a +**real cluster topology**. + +> **Operational runbook** (prerequisites, `make` targets, quick start) lives next +> to the code: [`deploy/k8s-sandbox/K8S_SANDBOX.md`](../deploy/k8s-sandbox/K8S_SANDBOX.md). +> This page is the conceptual/architecture companion. + +## Why a Kubernetes sandbox (and not just docker-compose) + +The compose harness ([`deploy/E2E_HARNESS.md`](../deploy/E2E_HARNESS.md)) runs the +full stack fast and exercises the S3→SQS→indexer→ClickHouse→flamegraph pipeline, +but it **cannot** produce real namespaces/pods/containers. Workload-level +profiling resolves those scopes, and the agent builds that inventory in +`gprofiler/metadata/heartbeat_metadata.py` by asking `granulate_utils`' +`ContainersClient` to enumerate the node's containers via the **container-runtime +socket** (CRI at `/run/containerd/containerd.sock`, or Docker) — **not** the +Kubernetes API server. It reads the kubelet labels +`io.kubernetes.pod.namespace`, `io.kubernetes.pod.name`, +`io.kubernetes.container.name` off each container. + +Under docker-compose there is no such socket for the agent, so it logs +`No container runtime found for heartbeat workload inventory` and the +pod/container/namespace tabs are only exercised with *synthetic* heartbeats. On a +real node the socket exists, so the inventory is **real**. That gap is the reason +this sandbox exists. + +| Layer | Orchestrator | Proves | Can't do | +|-------|--------------|--------|----------| +| `deploy/Makefile.e2e` (compose) | Docker Compose | fast API + full artifact pipeline | no real namespaces/pods/containers | +| `deploy/k8s-sandbox` (this) | kind / minikube | **real namespace/pod/container/process inventory + scope resolution** | heavier, slower inner loop | + +## What is a kind cluster? + +**kind = "Kubernetes IN Docker."** The entire cluster *node* is itself a single +Docker container running the `kindest/node` image. Inside that container, an init +system (systemd) boots a container runtime and the kubelet, and Kubernetes then +runs all workloads as pods *inside* the node container. It is nested containers: + +``` +Your Linux host +└─ Docker + └─ kind node (container: kindest/node:v1.30.0) <- "the cluster" + ├─ systemd (PID 1) + │ ├─ containerd.service ← the CRI runtime that runs every pod + │ ├─ kubelet.service ← node agent: talks to the API server, drives containerd + │ └─ systemd-journald + └─ pods (run by containerd, orchestrated by Kubernetes) +``` + +Both kind and minikube give a **hermetic, disposable** cluster: create → test → +destroy, nothing touches real infrastructure. The `Makefile.k8s` supports either +via `CLUSTER=kind|minikube` (see [the runbook](../deploy/k8s-sandbox/K8S_SANDBOX.md)). + +## kind vs. minikube (and why kind is the default) + +Both create a throwaway local Kubernetes cluster; the difference is *how* the node +is run and which container runtime lives inside it — which matters a lot here, +because the whole point of this sandbox is that the agent reads the node's **CRI +socket**. + +| | **kind** | **minikube** | +|---|---|---| +| Name | "Kubernetes **IN D**ocker" | "mini Kubernetes" | +| Node runs as | a Docker container (`kindest/node`) | a VM *or* a container ("drivers": docker, kvm2, virtualbox, none, …) | +| Runtime inside node | **containerd** (native) — socket at `/run/containerd/containerd.sock` | docker by default; containerd only with `--container-runtime=containerd` | +| Install | single static binary, needs Docker | single binary, but drivers add moving parts | +| Weight / speed | very light, fast, CI-standard | heavier; more features (addons, dashboard, LoadBalancer tunnels) | +| Best fit | automated testing / CI | general local dev with extras | + +Why kind is the default for this sandbox: + +1. **containerd-native, prod-like path.** A real production node runs containerd + and exposes the CRI socket the agent reads (`/run/containerd/containerd.sock`). + kind gives exactly that out of the box. minikube's default docker driver would + hand the agent `docker.sock` instead — a different, less prod-like code path — + unless you explicitly pass `--container-runtime=containerd`. +2. **Lighter + fewer choices.** One binary, only needs Docker; no VM/driver + matrix to reason about. +3. **CI-standard.** Easiest to graduate this sandbox into CI later. + +minikube is **not wrong** — with `--container-runtime=containerd` it produces the +same containerd CRI topology and the manifests/tests are identical. It's just +heavier and has more environment-specific setup. + +### Isolation model: VM drivers vs container drivers (and why no VM here) + +A common assumption is "minikube = an isolated VM." That was minikube's original +design, but today its isolation depends on the **driver**: + +- **VM drivers** (`kvm2`, `virtualbox`, `hyperkit`, `hyper-v`, `qemu`) boot a real + virtual machine with its **own kernel** — the strongest isolation, and + minikube's genuine edge over kind *when a hypervisor is available*. +- **Container drivers** (`docker`, `podman`) run the node as a **container sharing + the host kernel** — the same model as kind (which is *always* a container). +- **`none`** runs the kubelet directly on the host (no isolation). + +Either way the *cluster* is hermetic and disposable; what differs is whether the +boundary is a separate kernel (VM) or a shared-kernel container. + +**Why this host only offers the container driver.** The build host is a standard +(non-`.metal`) AWS EC2 instance — itself a guest VM that does **not expose nested +virtualization**: there is no `/dev/kvm`, `/proc/cpuinfo` shows zero `vmx`/`svm` +flags, and `systemd-detect-virt` reports `amazon`. A VM driver needs hardware +virtualization (VT-x/AMD-V) passed through to the guest, so `kvm2`/`virtualbox` +cannot run — only the `docker` (container) driver is viable. That is why minikube +here has the *same* shared-kernel boundary as kind and hits the same +systemd-in-container problem described in the Challenge below. To get a true +minikube VM on AWS you need a **bare-metal instance type** (e.g. `*.metal`), which +exposes the CPU virtualization extensions to the guest. + +## Challenge: minikube's docker driver won't boot on this host + +`CLUSTER=minikube` is fully wired into `Makefile.k8s`, but when validated on the +build host it could **not** bring up a cluster. Documented here so the next person +doesn't burn time rediscovering it. + +**Symptom.** `minikube start --driver=docker --container-runtime=containerd` +creates the `kicbase` node container, which then exits immediately at +`exec /sbin/init`: + +``` ++ exec /sbin/init +Couldn't find an alternative telinit implementation to spawn.: container exited unexpectedly +X Exiting due to GUEST_PROVISION_EXIT_UNEXPECTED: Failed to start host ... +``` + +i.e. systemd cannot come up as PID 1 inside minikube's node container, so the +kubelet/API server never start. + +**What was tried (all reproduced the same failure):** + +| Attempt | Result | +|---------|--------| +| current kicbase `v0.0.50` (Ubuntu 24.04) + containerd | `exec /sbin/init` exits | +| older kicbase `v0.0.44` + `--kubernetes-version=v1.30.0` | same | +| `--force-systemd` | same | + +**Root cause.** minikube's node image can't run systemd-as-PID-1 in this +particular **Docker 28.x + cgroup v2 + AWS `6.8.0` kernel** combination — the +container's init bails before systemd takes over. Notably, kind's +`kindest/node:v1.30.0` boots systemd fine on the *exact same host* (that's what +the running sandbox uses), so this is specific to how minikube constructs/runs its +node container, not a general "systemd-in-container is impossible here" problem. + +**Why the other minikube drivers weren't used.** VM drivers (`kvm2`, +`virtualbox`) need nested virtualization this cloud VM doesn't expose; the `none` +driver installs the kubelet **directly on the host** (invasive, root-level, would +mutate the host) and was intentionally avoided. That leaves the docker driver as +the only in-scope option — and it's the one that fails. + +**Resolution / takeaway.** Use **kind** here (the default). On a laptop or CI +runner where minikube's docker driver boots normally, `CLUSTER=minikube` works +with the *same* manifests and acceptance suite — nothing else changes. This is the +concrete, practical reason kind is the default for this sandbox, beyond the +containerd-native argument above. + +## How to use it (kind or minikube) + +All flow through `deploy/k8s-sandbox/Makefile.k8s`. Pick the cluster tool with the +`CLUSTER` variable (default `kind`): + +```bash +cd gprofiler-performance-studio/deploy/k8s-sandbox + +# --- kind (default, recommended) --- +make -f Makefile.k8s k8s-all # cluster + build + load + deploy + token +make -f Makefile.k8s k8s-test # AT-K1..K5 real-topology acceptance +make -f Makefile.k8s k8s-status # live workload inventory (all scopes) +make -f Makefile.k8s k8s-url # -> https://localhost:30443 (admin/admin) +make -f Makefile.k8s k8s-down-all # delete the whole cluster + +# --- minikube (same manifests/tests; needs a host where its docker driver boots) --- +make -f Makefile.k8s k8s-all CLUSTER=minikube CLUSTER_NAME=gprofiler-mk +make -f Makefile.k8s k8s-test CLUSTER=minikube CLUSTER_NAME=gprofiler-mk +make -f Makefile.k8s k8s-url CLUSTER=minikube CLUSTER_NAME=gprofiler-mk # -> https://:30443 +make -f Makefile.k8s k8s-down-all CLUSTER=minikube CLUSTER_NAME=gprofiler-mk +``` + +The only thing `CLUSTER` changes is cluster lifecycle + image loading (`kind load +docker-image` vs `minikube image load`) and the URL; every manifest, ConfigMap, +Secret, the agent DaemonSet, and the acceptance suite are identical. Use a +distinct `CLUSTER_NAME` (e.g. `gprofiler-mk`) if you want to run minikube +alongside an existing kind cluster, and note kind already binds host port `30443` +— map minikube elsewhere (e.g. `--ports=30444:30443`) to avoid a collision. + +## Topology: what runs as systemd vs. as pods + +This is the key mental model. Inside the kind node, **only three things run under +systemd** — everything else (including Kubernetes' own control plane) runs as +pods managed by the kubelet + containerd: + +**systemd services (inside the node container):** + +| Unit | Role | +|------|------| +| `containerd.service` | container runtime (CRI) that actually runs all pods | +| `kubelet.service` | node agent; `--container-runtime-endpoint=unix:///run/containerd/containerd.sock` | +| `systemd-journald.service` | logging | + +Notably, the kubelet's CRI endpoint is the **exact socket the agent DaemonSet +reads** (via `/proc/1/root`, see below) to enumerate workloads. + +**Everything else is a pod:** + +- **Kubernetes control plane** (`kube-system`, run as *static pods*): + `kube-apiserver`, `etcd`, `kube-scheduler`, `kube-controller-manager`, plus + `kube-proxy`, `kindnet` (CNI), `coredns`, and `local-path-provisioner`. +- **Performance Studio stack** (`perf-studio` namespace): `webapp`, `postgres`, + `clickhouse`, `ch-rest-service`, `ch-indexer`, `logs-backend` (2 containers), + `nginx`, `localstack`, and the **`gprofiler-agent` DaemonSet**. +- **Tenant workloads**: `team-a` (`checkout` ×2, `web`) and `team-b` + (`payments`, `search` — a 2-container pod). + +So even etcd and the API server are pods — a kind design choice. Contrast +docker-compose, where each service is a bare container on a bridge network with +**no kubelet, no containerd-as-CRI, and no pods**, which is exactly why the agent +finds "no container runtime" there but discovers real pods/namespaces/containers +here. + +## How the agent DaemonSet reaches the CRI socket + +`granulate_utils` resolves the socket path under `HOST_ROOT_PREFIX = /proc/1/root` +(`granulate_utils/linux/ns.py`). The mechanism is two pod settings, not a +bind-mount: + +- **`hostPID: true`** → PID 1 in the pod is the host (node) init, so + `/proc/1/root` is the node root filesystem and + `/proc/1/root/run/containerd/containerd.sock` is the node's real CRI socket. + `hostPID` also lets the agent see every node PID to map processes → containers. +- **`privileged: true`** → grants access to that socket and lets py-spy `ptrace` + the target workloads. + +> **Use the source-built agent.** The container-inventory heartbeat is a fork +> feature, so the public `intel/gprofiler:latest` image will **not** populate the +> pod/container tabs. + +## Data flow + +``` + kind node (containerd) + ns team-a: checkout(x2), web ns team-b: payments, search(app+sidecar) + ▲ enumerated via CRI (/proc/1/root/run/containerd/...) + gprofiler-agent DaemonSet ─ heartbeat/commands ─► webapp ─► postgres + (hostPID, privileged) │ + └─ upload ─► S3 (LocalStack) + ▼ + SQS ─► ch-indexer ─► ClickHouse + ▼ + nginx NodePort :30443 (UI) +``` + +## Verified end-to-end + +Brought up on a single-node **kind v1.30** cluster (containerd 1.7) and confirmed: + +- Agent DaemonSet connects and heartbeats with **no** `No container runtime + found` error (contrast compose) — it reached the node CRI socket. +- `workload_status` reported real topology — `tabCounts` `{service:1, host:1, + namespace:5, pod:23, container:25, process:93}` — including tenant namespaces + `team-a`/`team-b` and the 2-container `search` pod resolved as distinct + `search-app` + `search-sidecar`. +- Acceptance suite `AT-K1..K5` passed (5/5). +- **Full artifact pipeline closed in-cluster**: a host-scope start made the agent + profile the real workloads and upload; the webapp wrote collapsed stacks to S3 + (`products/k8s-sandbox/stacks/...gz`) and enqueued SQS; the indexer consumed it, + inserted 27 rows into `flamedb.samples`, and wrote the rendered + `..._adhoc_flamegraph.html` back to S3 — all against LocalStack, no real AWS. + +Two k8s-specific fixes came out of that run: + +- **Non-root low-port bind** (webapp, agents-logs-backend run as non-root and + bind port 80): Docker allows this via its default + `net.ipv4.ip_unprivileged_port_start=0`; Kubernetes does not, so those pods set + that (safe) sysctl explicitly. +- **Indexer startup ordering**: compose gated it on `localstack: service_healthy`; + k8s has no `depends_on` and the indexer resolves the SQS URL once without retry, + so it can boot before LocalStack and never consume. An init-container now waits + for LocalStack's SQS to report running. diff --git a/docs/WEBAPP_DB_CONCURRENCY_TUNING.md b/docs/WEBAPP_DB_CONCURRENCY_TUNING.md new file mode 100644 index 00000000..2a1dd289 --- /dev/null +++ b/docs/WEBAPP_DB_CONCURRENCY_TUNING.md @@ -0,0 +1,164 @@ +# Webapp DB Concurrency Tuning + +Environment variables that control how much database work the webapp +(`gprofiler_frontend` container) can do in parallel, and how to set them in +production. + +## Background: why this matters + +The webapp runs under **gunicorn** with **`uvicorn.workers.UvicornWorker`** +(async workers). The API route handlers are synchronous `def` functions, so +Starlette runs each request in a **per-worker threadpool** (anyio default: 40 +threads). + +Database access goes through `PostgresDB`, which by default keeps **one shared +psycopg2 connection per worker process**, guarded by a process-wide lock. Every +query — read or write — is serialized through that single connection: + +``` +worker process + ├─ ~40 request threads accepted concurrently + └─ 1 DB connection + lock ← all threads queue here +``` + +This is fine at low load, but when an expensive request (e.g. +`POST /api/metrics/heartbeat`) holds the connection, **every other endpoint in +that worker stalls behind it**. Under production heartbeat volume this starves +reads and all endpoints start hitting the 15 s load-balancer timeout — even +ones whose own query takes milliseconds. + +The knobs below let you add real DB concurrency without a code change. + +## The knobs + +| Environment variable | Where read | Default | Effect | +| --- | --- | --- | --- | +| `GUNICORN_PROCESS_COUNT` | `src/gprofiler/run.sh` | `nproc` | Number of gunicorn worker **processes** per replica. Each worker is a separate process with its own DB connection(s). | +| `GPROFILER_POSTGRES_CONN_PER_THREAD` | `src/gprofiler-dev/gprofiler_dev/config.py` | `FALSE` | When `TRUE`, each worker **thread** gets its own DB connection instead of sharing one. This removes the per-worker serialization. | +| `GPROFILER_WEBAPP_THREAD_POOL_SIZE` | `src/gprofiler/backend/config.py` | `0` (keep anyio default of 40) | Per-worker threadpool size for sync route handlers. With `CONN_PER_THREAD=TRUE` this also caps the number of DB connections each worker can open. | + +### `GPROFILER_POSTGRES_CONN_PER_THREAD` — the primary fix + +This is the highest-leverage setting. With it `FALSE` (the default), a worker +can only run **one** DB operation at a time regardless of how many requests it +accepts. Setting it to `TRUE` gives each threadpool thread its own connection, +so a single worker can run up to `threadpool_size` queries in parallel. + +Set this to `TRUE` first — it directly removes the cross-endpoint starvation. + +### `GUNICORN_PROCESS_COUNT` — raw process concurrency + +Each worker is a separate process (and, with `CONN_PER_THREAD=FALSE`, exactly +one DB connection). Increasing workers multiplies DB concurrency **across** +processes. Because the workers are I/O-bound on the database, it is fine to run +more workers than CPU cores. Useful as a throughput multiplier on top of +`CONN_PER_THREAD=TRUE`, or as the only lever if you keep `CONN_PER_THREAD=FALSE`. + +### `GPROFILER_WEBAPP_THREAD_POOL_SIZE` — bound / widen per-worker concurrency + +Controls how many sync requests a worker runs at once. Leave at `0` to keep the +framework default (40). Raise it to allow more in-flight requests per worker; +lower it to **bound the number of DB connections** when +`CONN_PER_THREAD=TRUE` (see the connection math below). + +## Connection math + +Total DB connections opened by the webapp is approximately: + +``` +connections ≈ replicas × GUNICORN_PROCESS_COUNT × threads_per_worker +``` + +where `threads_per_worker` is: + +- `1` when `GPROFILER_POSTGRES_CONN_PER_THREAD=FALSE` (shared connection), or +- `GPROFILER_WEBAPP_THREAD_POOL_SIZE` (or 40 if unset) when `TRUE`. + +Stay under the database's `max_connections`. On the current Aurora cluster +`max_connections = 5000` with only ~70 in use, so there is large headroom. + +**Examples (per the current prod cluster, `nproc = 8`):** + +| Config | Conns/replica | 6 replicas | +| --- | --- | --- | +| Current: `CONN_PER_THREAD=FALSE`, 8 workers | 8 | 48 | +| `CONN_PER_THREAD=TRUE`, 8 workers, pool 40 (default) | 320 | 1920 | +| `CONN_PER_THREAD=TRUE`, 8 workers, pool 20 | 160 | 960 | +| `CONN_PER_THREAD=FALSE`, 24 workers | 24 | 144 | + +## Sizing for load (Little's Law) + +The number of DB operations in flight at once is `L = λ × W`, where `λ` is the +request rate and `W` is how long each request holds a connection. Your total +provisioned concurrency (`replicas × workers × threads_per_worker`) must exceed +`L`, with headroom for bursts. + +The dominant driver here is the agent heartbeat: **30k hosts × 1 per 30 s ≈ +1000 heartbeat QPS**. Each heartbeat is write-heavy — `upsert_host_heartbeat` +(host upsert + container/process diff-sync) plus a profiling-command lookup and +possible status updates — so `W` is on the order of tens of milliseconds: + +| Heartbeat hold time `W` | Busy connections `L` at 1000 QPS | +| --- | --- | +| 30 ms | 30 | +| 50 ms | 50 | +| 100 ms | 100 | +| 150 ms | 150 | + +Add read traffic on top. Coarse read scopes now serve from the store in ~10 ms +(negligible), but the **live fine scopes hold a connection for seconds** +(container ~4 s, process ~8 s), so each concurrent fine-scope request consumes +several connection-seconds. Budget generously for these. + +Target total provisioned concurrency at **2–3× the computed `L`** so bursts and +slow reads don't exhaust the pool. Measure `W` from your own metrics +(`pg_stat_activity`, request latency) and re-derive — the table is a starting +estimate. + +> **Connections enable parallelism, they do not create DB throughput.** 1000 +> write-heavy heartbeats/s is real load on the Aurora writer regardless of how +> many connections you open. If the writer saturates (CPU, lock/WAL), the fix is +> to cut per-heartbeat cost (batch the inventory writes) or scale the DB — not to +> add more connections. + +## Recommended production settings + +Start here, then adjust based on measured `W` and replica count: + +```bash +# Remove the per-worker serialization (primary fix). +GPROFILER_POSTGRES_CONN_PER_THREAD=TRUE + +# Per-worker connection cap. Sized for ~1000 heartbeat QPS with headroom. +GPROFILER_WEBAPP_THREAD_POOL_SIZE=40 + +# Process-level concurrency. Workers are I/O-bound on the DB, so exceeding +# core count is fine; raise if still throughput-bound. +GUNICORN_PROCESS_COUNT=16 +``` + +This gives `16 × 40 = 640` parallel DB slots **per replica**. Across replicas, +check the total against `max_connections`: + +| Config | Conns/replica | 3 replicas | 6 replicas | +| --- | --- | --- | --- | +| `CONN_PER_THREAD=TRUE`, 8 workers, pool 40 | 320 | 960 | 1920 | +| `CONN_PER_THREAD=TRUE`, 16 workers, pool 40 | 640 | 1920 | 3840 | +| `CONN_PER_THREAD=TRUE`, 16 workers, pool 20 | 320 | 960 | 1920 | + +At `max_connections = 5000` even the largest row leaves headroom, but if you +scale replicas hard, keep `replicas × workers × pool` under the limit (with +margin for the periodic tasks and admin connections) — lower +`GPROFILER_WEBAPP_THREAD_POOL_SIZE` or add a pooler (pgbouncer) if you approach +it. + +## Verifying + +Check live connection usage against the limit: + +```sql +SHOW max_connections; +SELECT count(*) AS total, + count(*) FILTER (WHERE state = 'active') AS active +FROM pg_stat_activity; +``` diff --git a/docs/WORKLOAD_STATUS_PERF_FOLLOWUPS.md b/docs/WORKLOAD_STATUS_PERF_FOLLOWUPS.md new file mode 100644 index 00000000..5a941d91 --- /dev/null +++ b/docs/WORKLOAD_STATUS_PERF_FOLLOWUPS.md @@ -0,0 +1,107 @@ +# Workload Status (`GET /profiling/workload_status`) — performance follow-ups + +## Context + +At ~34K active hosts / ~684K total `HostHeartbeats` rows (PROD, Sep 2026) the endpoint +timed out (>30s). Root causes, from prod `EXPLAIN (ANALYZE)`: + +1. **Bad plan from `ORDER BY/GROUP BY service_name`** — the planner did a full scan of + `idx_hostheartbeats_service_name` (cost ~126K) over all 684K rows, applying the + `heartbeat_timestamp > now()-2min` freshness filter as a per-row heap filter, to avoid a + sort. Random I/O over the whole table → timeout. +2. **Table bloat / no retention** — only ~34K of 684K hosts are active; ~650K are stale + decommissioned hosts that are never cleaned up. Every request scans all of them. +3. **`COUNT(*) OVER ()`** (`total_groups`) prevents `LIMIT` from short-circuiting. + +## Shipped in this change (a) + +- Restrict to the active fleet **first**, in a `fresh_hosts AS MATERIALIZED (…)` CTE, so + grouping/sorting can no longer drive the full-index scan. +- Dropped the redundant `latest_commands` window (`ProfilingCommands` is already + `UNIQUE (hostname, service_name)`). + +Measured on PROD (read-only `EXPLAIN ANALYZE`), `scope=host`, page 0: + +| Query | Before | After (a) | +| --- | --- | --- | +| grouped rows | **timeout (>30s)** | **~1.4s** | + +Uniform across all scopes, no schema change, no semantic change (all filters still applied +in `filtered`). + +--- + +## Follow-up (b) — entity-first pagination ✅ shipped (as a hybrid) + +**Original idea:** page the *driving entity* first, then aggregate only the page via +`LATERAL` joins for just those ≤`page_size` entities; get `total_count` from a single +`COUNT(*)` over the driving set and drop `COUNT(*) OVER ()`. + +**What changed since this was written:** the workload agent rolled out, so +`HeartbeatContainers` (~316K) and `HeartbeatProcesses` (~1.08M) are now populated and the +flatten is ~1.08M rows (was ~34K when only hosts existed). Pure entity-first is a big win +for scopes with *many small* entities but *regresses* for scopes with *few large* entities +(a page of 50 services still covers most of the fleet, and each per-entity `LATERAL` +re-scans a whole service). So (b) shipped as a **hybrid**, chosen per scope: + +- **Entity-first** (`host`, `container`, `process` — many small entities): page the key set, + then hydrate only that page via a `LATERAL` that reads `HostHeartbeats` **directly** + (indexed on `service_name`/`hostname`). Only host-level keys are correlated with `=`; + finer keys are NULL-safe residual filters (never pushed onto child tables, or the planner + drives from a global `idx_hb_containers_namespace` scan and explodes on `default`/ + `kube-system`). +- **Single-pass** (`namespace`, `pod` — few/large, PID-aware): one `GROUP BY` over the + flatten, paginated (the guard `namespace/pod_name IS NOT NULL` prunes the fan-out). +- **Two-grain single-pass** (`service`): counts + latest metadata at the container grain + (~316K rows), `process_count` via `COUNT(*) FROM (SELECT DISTINCT …) GROUP BY`, and + `any_active` at the **host** grain — provably identical to the PID-aware value for service + scope (verified 0/460 services differ on prod). Falls back to plain single-pass under a + process-tier filter. + +**Measured (prod, page 0):** grouped `host` ~10s → **~0.6s**; `container` ~2.0s; +`process` ~3.7s; `namespace` ~3.5s; `pod` ~4.1s; `service` ~16s → **~6.4s**. + +**Gate:** query-only, but higher correctness risk than (a) (filter push-down level, +NULL-safe intermediate keys, the service `any_active` equivalence). Validated read-only on +prod against authoritative counts; still gate on the `AT-S1..S17` workload-acceptance tests. + +## Follow-up (c) — targeted `tab_counts` ✅ shipped + +The single 7×`COUNT(DISTINCT …)` pass timed out once the child tables filled. Replaced with +targeted per-tier counts sharing the `fresh_hosts` CTE: + +- `service` / `host` / `active_hosts` → from `fresh_hosts` only. +- `namespace` / `pod` / `container` → `HeartbeatContainers ⋈ fresh_hosts`. +- `process` → 3-way join, distinct on the integer `(host_id, pid)`. + +Each count uses `COUNT(*) FROM (SELECT DISTINCT …)` (hash-distinct) instead of +`COUNT(DISTINCT tuple)` (sort-per-aggregate) — the process count alone went +**13.2s → 0.84s**. All filters still apply at every tier. + +**Measured (prod):** **timeout → ~4s** (tier A ~0.4s + B ~1.1s + C ~0.8s). Short-TTL caching +was *not* added and remains an option if the tabs need to feel instant. + +## Follow-up — retention cleanup (root cause) ⏳ outstanding + +The unbounded growth of `HostHeartbeats` / `HeartbeatContainers` / `HeartbeatProcesses` is +the structural driver: ~95% of `HostHeartbeats` rows are stale, and the child tables now +carry ~1.08M live-plus-stale rows. A periodic cleanup that deletes hosts whose +`heartbeat_timestamp` is older than a retention threshold (child tables cascade via FK) +would shrink the base tables ~20×. + +- The parallel seq scan to extract the fresh set drops from ~100ms to ~10ms. +- Every scan, index, and autovacuum on these tables gets ~20× cheaper. +- It is the single change that would bring the heavier scopes (`service`, `pod`) back to + sub-second and keep them there as the fleet grows. +- Natural home: the `deploy/periodic_tasks` container (cron), matching the existing + aggregation/logrotate jobs. + +## Status / suggested order + +1. **(a)** — shipped (killed the original host-scan timeout; PR #95). +2. **(c) tiered `tab_counts`** — shipped (timeout → ~4s). +3. **(b) hybrid entity-first / single-pass / two-grain** — shipped (default `host` view + ~10s → ~0.6s; all scopes under the timeout). +4. **Retention job** — outstanding; the biggest remaining structural win, benefits every + scope and is what would bring `service` (~6.4s) and `pod` (~4.1s) back to sub-second. + diff --git a/heartbeat_doc/PERFSPECT_DYNAMIC_PROFILING.md b/heartbeat_doc/PERFSPECT_DYNAMIC_PROFILING.md index 7aebd515..75c2ad40 100644 --- a/heartbeat_doc/PERFSPECT_DYNAMIC_PROFILING.md +++ b/heartbeat_doc/PERFSPECT_DYNAMIC_PROFILING.md @@ -73,7 +73,7 @@ This feature addresses the need for comprehensive performance analysis by combin ```bash # Test Case: Create profiling request with PerfSpect enabled curl -X POST http://localhost:8080/api/metrics/profile_request \ - -u "username:password" \ + -u "prashantpatel:password" \ -H "Content-Type: application/json" \ -d '{ "service_name": "test-service-2", diff --git a/heartbeat_doc/README_HEARTBEAT.md b/heartbeat_doc/README_HEARTBEAT.md index baf73e98..8d0f649e 100644 --- a/heartbeat_doc/README_HEARTBEAT.md +++ b/heartbeat_doc/README_HEARTBEAT.md @@ -10,50 +10,155 @@ The heartbeat system enables remote control of gProfiler agents through a simple 2. **Backend responds with profiling commands** (start/stop) when available 3. **Agents execute commands with built-in idempotency** to prevent duplicate execution 4. **Commands are tracked and logged** for audit and debugging +5. **Agents report hardware performance-counter (PMU) capabilities** so the backend can validate event-level profiling requests before dispatch ## Architecture ``` -┌─────────────────┐ heartbeat ┌──────────────────────┐ -│ gProfiler │ ──────────────► │ Performance Studio │ -│ Agent │ │ Backend │ -│ │ ◄────────────── │ │ -└─────────────────┘ commands └──────────────────────┘ - │ │ - │ │ - ▼ ▼ -┌─────────────────┐ ┌──────────────────────┐ -│ Profile Data │ │ PostgreSQL DB │ -│ (S3/Local) │ │ - Host Heartbeats │ -└─────────────────┘ │ - Profiling Cmds │ - └──────────────────────┘ +┌─────────────────────────────┐ +│ gProfiler Agent │ +│ │ +│ ┌───────────────────────┐ │ +│ │ ContinuousProfilerSlot│ │ heartbeat (POST /api/metrics/heartbeat) +│ └───────────────────────┘ │ ──────────────────────────────────────────► +│ ┌───────────────────────┐ │ │ +│ │ AdhocProfilerSlot │ │ ◄──────────────────────────────────────── │ +│ └───────────────────────┘ │ commands + combined_config │ +│ ┌───────────────────────┐ │ │ +│ │ CommandManager │ │ ┌────────────┴─────────────┐ +│ │ (priority queue) │ │ │ Performance Studio │ +│ └───────────────────────┘ │ │ Backend (FastAPI) │ +└─────────────────────────────┘ │ │ + │ │ - PMU event validation │ + │ profile data │ - Slack notifications │ + ▼ │ - Capacity enforcement │ +┌─────────────────┐ └────────────┬─────────────┘ +│ Profile Data │ │ +│ (S3/Local) │ │ +└─────────────────┘ ┌────────────▼─────────────┐ + │ PostgreSQL DB │ + │ - HostHeartbeats │ + │ - ProfilingRequests │ + │ - ProfilingCommands │ + │ - ProfilingExecutions │ + └──────────────────────────┘ ``` ## Database Schema -### Core Tables +### ENUMs -1. **HostHeartbeats** - Track agent status and last seen information -2. **ProfilingRequests** - Store profiling requests from API calls -3. **ProfilingCommands** - Commands sent to agents (merged from multiple requests) -4. **ProfilingExecutions** - Execution history for audit trail +```sql +ProfilingMode = ('cpu', 'allocation', 'none') +ProfilingRequestStatus = ('pending', 'assigned', 'completed', 'failed', 'cancelled') +CommandStatus = ('pending', 'sent', 'completed', 'failed') +HostStatus = ('active', 'idle', 'error', 'offline') +``` -### Key Features -- **Simple DDL** with essential indexes only -- **No stored procedures** - all logic in application code -- **No triggers** - timestamps handled by application -- **Consistent naming** with `idx_` prefix for all indexes +### `HostHeartbeats` + +Tracks agent status and last-seen information. Upserted on `(hostname, service_name)` at every heartbeat. + +| Column | Type | Notes | +|---|---|---| +| `id` | `bigserial PK` | | +| `hostname` | `text NOT NULL` | | +| `ip_address` | `inet NOT NULL` | | +| `service_name` | `text NOT NULL` | | +| `last_command_id` | `uuid NULL` | Last command acknowledged by the agent | +| `received_command_ids` | `uuid[] NULL` | All command IDs the agent has received | +| `executed_command_ids` | `uuid[] NULL` | All command IDs the agent has executed | +| `status` | `HostStatus DEFAULT 'active'` | | +| `heartbeat_timestamp` | `timestamp` | Set by server on upsert | +| `supported_perf_events` | `text[] NULL` | PMU events reported by the agent; used for bulk request validation | +| `created_at` / `updated_at` | `timestamp` | | + +Indexes: `hostname`, `service_name`, `status`, `heartbeat_timestamp`. + +### `ProfilingRequests` + +One row per API-level profiling request. Multiple requests for the same host are merged into a single `ProfilingCommands` row. + +| Column | Type | Notes | +|---|---|---| +| `id` | `bigserial PK` | | +| `request_id` | `uuid NOT NULL UNIQUE` | | +| `service_name` | `text NOT NULL` | | +| `request_type` | `text` | `'start'` or `'stop'` | +| `continuous` | `boolean DEFAULT false` | Whether the profiler should run in continuous mode | +| `duration` | `integer DEFAULT 60` | Seconds | +| `frequency` | `integer DEFAULT 11` | Hz | +| `profiling_mode` | `ProfilingMode DEFAULT 'cpu'` | | +| `target_hostnames` | `text[] NOT NULL` | | +| `pids` | `integer[] NULL` | **Deprecated** — always `NULL`; per-host PIDs are stored in the backend process memory (`DBManager.request_host_pid_mappings`) and lost on restart | +| `stop_level` | `text DEFAULT 'process'` | `'process'` or `'host'` | +| `additional_args` | `jsonb NULL` | Merged flat into `combined_config` | +| `status` | `ProfilingRequestStatus DEFAULT 'pending'` | | +| `estimated_completion_time` | `timestamp NULL` | | +| `created_at` / `updated_at` | `timestamp` | | + +### `ProfilingCommands` + +One active command per `(hostname, service_name)` pair. When a new request arrives for a host that already has a `pending` command, the configs are **merged** (max duration, max frequency, union of PIDs, OR of `continuous`) rather than replaced. + +| Column | Type | Notes | +|---|---|---| +| `id` | `bigserial PK` | | +| `command_id` | `uuid NOT NULL` | | +| `hostname` | `text NOT NULL` | | +| `service_name` | `text NOT NULL` | | +| `command_type` | `text` | `'start'` or `'stop'` | +| `request_ids` | `uuid[] NOT NULL` | All request UUIDs merged into this command | +| `combined_config` | `jsonb NULL` | Merged configuration delivered to the agent | +| `status` | `CommandStatus DEFAULT 'pending'` | `'pending'` → `'sent'` at heartbeat; `'completed'`/`'failed'` at completion | +| `sent_at` | `timestamp NULL` | Set when delivered via heartbeat response | +| `completed_at` | `timestamp NULL` | | +| `execution_time` | `integer NULL` | Seconds, as reported by the agent | +| `error_message` | `text NULL` | | +| `results_path` | `text NULL` | | + +Unique constraint: `(hostname, service_name)` — one active command per host/service pair. + +### `ProfilingExecutions` + +Audit table. One row is inserted (with `status='assigned'`) when the command is delivered to the agent at heartbeat time — not at completion. + +| Column | Type | Notes | +|---|---|---| +| `id` | `bigserial PK` | | +| `command_id` | `uuid NOT NULL` | | +| `hostname` | `text NOT NULL` | | +| `profiling_request_id` | `uuid FK → ProfilingRequests` | | +| `status` | `ProfilingRequestStatus DEFAULT 'pending'` | | +| `started_at` / `completed_at` | `timestamp NULL` | | +| `execution_time` | `integer NULL` | | +| `error_message` | `text NULL` | | +| `results_path` | `text NULL` | | ## API Endpoints +| Method | Path | Purpose | +|---|---|---| +| `POST` | `/api/metrics/profile_request` | Create a profiling request for one or more hosts | +| `POST` | `/api/metrics/profile_request/bulk` | Create multiple profiling requests atomically | +| `POST` | `/api/metrics/heartbeat` | Agent heartbeat — returns pending commands | +| `POST` | `/api/metrics/command_completion` | Agent reports command execution result | +| `GET` | `/api/metrics/profiling/host_status` | Dashboard: per-host profiling status | + ### 1. Create Profiling Request + ```http POST /api/metrics/profile_request Content-Type: application/json +``` + +**Request:** +```json { "service_name": "my-service", "request_type": "start", + "continuous": false, "duration": 60, "frequency": 11, "profiling_mode": "cpu", @@ -62,101 +167,368 @@ Content-Type: application/json "host2": null }, "stop_level": "process", - "additional_args": {} + "additional_args": {}, + "dry_run": false } ``` +| Field | Required | Default | Notes | +|---|---|---|---| +| `service_name` | yes | — | | +| `request_type` | yes | — | `"start"` or `"stop"` | +| `target_hosts` | yes | — | Dict of `hostname → [pids]` or `hostname → null` | +| `continuous` | no | `false` | Keep profiling running until an explicit stop | +| `duration` | no | `60` | Seconds; must be > 0 | +| `frequency` | no | `11` | Hz; must be > 0 | +| `profiling_mode` | no | `"cpu"` | `"cpu"`, `"allocation"`, or `"none"` | +| `stop_level` | no | `"process"` | `"process"` requires at least one host to have PIDs; `"host"` forbids PIDs | +| `additional_args` | no | `{}` | Merged flat into `combined_config` | +| `dry_run` | no | `false` | Validates and returns a response without writing to the database | + +#### `additional_args.profiler_configs` + +The `profiler_configs` key inside `additional_args` controls which profilers are enabled and how they run. All keys are optional; omitting a key uses the profiler's default. + +**Async Profiler (Java)** + +```json +"profiler_configs": { + "async_profiler": { + "enabled": true, + "time": "cpu", + "alloc_interval": "2MB" + } +} +``` + +| Field | Default | Values | Notes | +|---|---|---|---| +| `enabled` | `true` | `true` / `false` | Set `false` to disable Java profiling entirely | +| `time` | `"cpu"` | `"cpu"`, `"itimer"`, `"wall"`, `"auto"`, `"alloc"` | Profiling mode for async-profiler (see table below) | +| `alloc_interval` | `"2MB"` | valid size string using bitmath notation e.g. `"2MB"`, `"512KiB"`, `"1GiB"` | Allocation interval; **required** when `time` is `"alloc"` | + +| `time` value | Description | +|---|---| +| `"cpu"` | CPU time — samples only while the thread is on-CPU | +| `"itimer"` | Interval timer — uses OS `SIGPROF`; lower overhead than `cpu` | +| `"wall"` | Wall-clock time — includes threads blocked on I/O or locks | +| `"auto"` | Auto-select between `cpu` and `itimer` at runtime based on host capabilities | +| `"alloc"` | Allocation profiling — samples on heap allocations instead of time; `alloc_interval` controls the sampling granularity | + +> **Validation:** The server rejects any `time` value outside the five listed above with HTTP 422. For `"alloc"` mode, `alloc_interval` must be a non-empty, parseable size string in bitmath notation (e.g. `"2MB"`, `"512KiB"`, `"1GiB"`); absent, empty, or malformed values (e.g. `"2 bananas"`, `"2mb"`) are rejected with HTTP 422. + +**Other profilers** (`perf`, `pyperf`, `pyspy`, `phpspy`, `rbspy`, `dotnet_trace`, `nodejs_perf`) are unchanged and described in the Agent Integration section. + **Response:** + ```json { "success": true, - "message": "Start profiling request submitted successfully", - "request_id": "uuid", - "command_id": "uuid", - "estimated_completion_time": "2025-01-15T10:30:00Z" + "message": "Start profiling request submitted successfully for service 'my-service' across 2 hosts", + "request_id": "req-uuid", + "command_ids": ["cmd-uuid-host1", "cmd-uuid-host2"], + "estimated_completion_time": "2026-03-04T10:30:00Z" } ``` -### 2. Agent Heartbeat +> **Note:** `command_ids` is a list — one UUID per target host. For `"stop"` requests `estimated_completion_time` is `null`. For `dry_run=true`, `request_id` is `null` and `command_ids` is empty. + +### 2. Bulk Create Profiling Requests + +Atomically submit multiple profiling requests. Capacity validation (`MAX_PROFILING_REQUEST_HOSTS`, `MAX_SIMULTANEOUS_PROFILING_HOSTS`) runs once across the entire batch before any individual request is processed. + +```http +POST /api/metrics/profile_request/bulk +Content-Type: application/json +``` + +**Request:** + +```json +{ + "requests": [ + { "service_name": "svc-a", "request_type": "start", "target_hosts": {"host1": null} }, + { "service_name": "svc-b", "request_type": "start", "target_hosts": {"host2": [1234]} } + ], + "dry_run": false +} +``` + +`dry_run` at the bulk level overrides every individual request's `dry_run`. + +**Response:** + +```json +{ + "total_submitted": 2, + "successful_count": 2, + "failed_count": 0, + "results": [ + { + "index": 0, + "service_name": "svc-a", + "success": true, + "response": { "success": true, "request_id": "...", "command_ids": ["..."] }, + "error": null + } + ] +} +``` + +### 3. Agent Heartbeat + ```http POST /api/metrics/heartbeat Content-Type: application/json +``` + +**Request:** +```json { "hostname": "host1", "ip_address": "10.0.1.100", "service_name": "my-service", "last_command_id": "previous-command-uuid", - "status": "active" + "received_command_ids": ["uuid-a", "uuid-b"], + "executed_command_ids": ["uuid-a"], + "status": "active", + "timestamp": "2026-03-04T10:00:00Z", + "perf_supported_events": ["cpu-cycles", "cache-misses", "instructions"] } ``` -**Response (with command):** +| Field | Required | Notes | +|---|---|---| +| `hostname` | yes | | +| `ip_address` | yes | | +| `service_name` | yes | | +| `last_command_id` | no | Last command ID the agent acknowledged | +| `received_command_ids` | no | All command IDs the agent has received (for fine-grained tracking) | +| `executed_command_ids` | no | All command IDs the agent has started executing | +| `status` | no | `"active"` (default), `"idle"`, or `"error"` | +| `timestamp` | no | ISO 8601; set by the server if absent | +| `perf_supported_events` | no | PMU events available on this host; used for bulk request validation | + +**Response — no pending command:** + +```json +{ + "success": true, + "message": "Heartbeat received. No profiling commands.", + "profiling_command": null, + "command_id": null +} +``` + +**Response — command available:** + ```json { "success": true, "message": "Heartbeat received. New profiling command available.", - "command_id": "new-command-uuid", + "command_id": "cmd-uuid", "profiling_command": { "command_type": "start", "combined_config": { + "continuous": false, "duration": 60, "frequency": 11, "profiling_mode": "cpu", - "pids": "1234,5678" + "pids": [1234, 5678], + "stop_level": "process" } } } ``` -### 3. Command Completion +> **Note:** `combined_config.pids` is a JSON integer array `[1234, 5678]`, not a comma-delimited string. + +The heartbeat endpoint always returns HTTP 200. If an internal error occurs while looking up commands, the response body will contain `success: true` with an error message — it never returns 5xx from this path. + +### 4. Command Completion + ```http POST /api/metrics/command_completion Content-Type: application/json +``` + +**Request:** +```json { - "command_id": "command-uuid", + "command_id": "cmd-uuid", "hostname": "host1", "status": "completed", "execution_time": 65, + "error_message": null, "results_path": "/path/to/results" } ``` +| Field | Required | Notes | +|---|---|---| +| `command_id` | yes | | +| `hostname` | yes | | +| `status` | yes | `"completed"` or `"failed"` | +| `execution_time` | no | Seconds | +| `error_message` | no | Populated on `"failed"` status | +| `results_path` | no | | + +**Response:** + +```json +{ + "success": true, + "message": "Command completion recorded for cmd-uuid" +} +``` + +Or, if the command is not found / not in `'assigned'` state for this host: + +```json +{ + "success": false, + "message": "Command cmd-uuid not found for host host1" +} +``` + +> **Note:** If the agent reports a `command_id` that no longer matches the host's current active command (i.e., a stale completion), `ProfilingRequests` status is not updated — only completions for the current active command trigger request status propagation. + +### 5. Host Status Dashboard + +```http +GET /api/metrics/profiling/host_status +``` + +Query parameters (all optional, repeatable): `service_name[]`, `hostname[]`, `ip_address[]`, `profiling_status[]`, `command_type[]`, `pids[]`, `exact_match`. + +**Response:** + +```json +{ + "hosts": [ + { + "id": 1, + "service_name": "my-service", + "hostname": "host1", + "ip_address": "10.0.1.100", + "pids": [1234], + "command_type": "start", + "profiling_status": "sent", + "heartbeat_timestamp": "2026-03-04T10:00:00Z" + } + ], + "active_count": 1, + "total_count": 5 +} +``` + +`profiling_status` reflects the current `ProfilingCommands.status`. If no command exists for a host, it reports `"stopped"`. + ## Agent Integration -### Heartbeat Configuration +### Two-Slot Architecture + +The agent uses two independent execution slots so that non-overlapping profiler types can run in parallel: + +``` +DynamicGProfilerManager +├── continuous: ContinuousProfilerSlot ← main continuous / single-run profiler +├── adhoc: AdhocProfilerSlot ← parallel ad-hoc profiler +└── command_manager: CommandManager ← priority queue (stop > adhoc > continuous) +``` + +#### ContinuousProfilerSlot + +Handles commands where `combined_config.continuous=true` or single-run commands sent to the continuous slot. Can be paused (preempted) when a new higher-priority command arrives, and will resume after the ad-hoc command finishes. + +#### AdhocProfilerSlot + +Handles ad-hoc (non-continuous) commands. Can run **in parallel** with `ContinuousProfilerSlot` if the two commands profile different runtime types (no overlapping profiler types). If there is overlap, the continuous command is paused (time-sliced) until the ad-hoc command completes. + +Profiler type overlap is detected from `profiler_configs` keys using this mapping: + +| Config Key | Canonical Type | +|---|---| +| `perf` | `perf` | +| `async_profiler` | `java` | +| `pyperf` / `pyspy` | `python` | +| `phpspy` | `php` | +| `rbspy` | `ruby` | +| `dotnet_trace` | `dotnet` | +| `nodejs_perf` | `nodejs` | + +#### CommandManager Priority Queue + +Three queues with fixed capacities. `get_next_command()` peeks (non-destructively) in priority order: + +1. **Stop queue** (max 1) — processed immediately; clears all other queues +2. **Ad-hoc queue** (max 10) +3. **Continuous queue** (max 1) — if a second continuous command arrives, earlier pending continuous commands are silently discarded + +Commands are dequeued by the profiler thread's `finally` block after the run completes. Paused continuous commands are deliberately kept in the queue so they can be resumed by the next heartbeat tick. + +### Heartbeat Loop + +Every `--heartbeat-interval` seconds the agent: + +1. Sends `POST /api/metrics/heartbeat` with current state +2. If a command is returned, enqueues it (idempotency check against `received_command_ids`) +3. Cleans up any completed ad-hoc profiler threads +4. Peeks at the next command and dispatches it if conditions are met +5. Sleeps until next tick (interruptible) + +Idempotency is enforced by two sets (`received_command_ids`, `executed_command_ids`) capped at 1000 entries each, trimmed by keeping the most recent entries. + +### `send_command_completion` Timing + +`POST /api/metrics/command_completion` is called when a profiler is **started** (not when it finishes). Consequently, `execution_time` is always sent as `0` from the agent. The actual duration is not reported back to the backend. + +### Startup Command + ```bash python3 gprofiler/main.py \ --enable-heartbeat-server \ + --upload-results \ --api-server "https://perf-studio.example.com" \ - --heartbeat-interval 30 \ --service-name "my-service" \ - --token "api-token" + --token "api-token" \ + --heartbeat-interval 30 ``` -### Heartbeat Flow -1. **Agent sends heartbeat** every 30 seconds (configurable) -2. **Backend checks for pending commands** for this hostname/service -3. **If command available**, backend responds with command details -4. **Agent executes command** and reports completion -5. **Idempotency ensured** by tracking `last_command_id` +### CLI Reference + +#### Heartbeat Flags (the `--enable-heartbeat-server` group requires all three starred flags) -## Command Types +| Flag | Default | Notes | +|---|---|---| +| `--enable-heartbeat-server` | off | Activates heartbeat mode; mutually exclusive with normal profiling | +| `--upload-results` / `-u` | off | ★ Required with `--enable-heartbeat-server` | +| `--token TOKEN` | — | ★ Required; sent as `Authorization: Bearer` on profile uploads | +| `--service-name NAME` | — | ★ Required | +| `--api-server URL` | (default address) | Backend base URL; `--server` is a deprecated alias | +| `--heartbeat-interval SECONDS` | `30` | Sleep between heartbeat POST requests | +| `--no-verify` | — | Skip TLS server certificate verification | -### START Commands -- Create new profiling sessions -- Merge multiple requests for same host -- Include combined configuration (duration, frequency, PIDs) +> **Note:** `--heartbeat-file PATH` is an **unrelated** feature — it touches a filesystem timestamp inside the normal snapshot loop as a liveness probe. It does not interact with the heartbeat protocol described here. -### STOP Commands -- **Process-level**: Stop specific PIDs -- **Host-level**: Stop entire profiling session -- Automatic conversion when only one PID remains +#### mTLS Flags + +| Flag | Default | Notes | +|---|---|---| +| `--tls-client-cert PATH` | — | PEM client certificate | +| `--tls-client-key PATH` | — | PEM client private key; both cert and key must be provided for mTLS to activate | +| `--tls-ca-bundle PATH` | — | PEM CA bundle; overrides system CA store | +| `--tls-cert-refresh-enabled` | off | Enables background cert rotation thread | +| `--tls-cert-refresh-interval SECONDS` | `21600` | Cert refresh interval (default: 6 hours) | ## Data Flow Example ### 1. Create Profiling Request + +Basic request (defaults): + ```bash curl -X POST http://localhost:8000/api/metrics/profile_request \ -H "Content-Type: application/json" \ @@ -164,25 +536,65 @@ curl -X POST http://localhost:8000/api/metrics/profile_request \ "service_name": "web-service", "request_type": "start", "duration": 120, - "target_hostnames": ["web-01", "web-02"], + "target_hosts": {"web-01": null, "web-02": null}, "profiling_mode": "cpu" }' ``` -### 2. Agent Heartbeat +With explicit async-profiler mode (wall-clock): + +```bash +curl -X POST http://localhost:8000/api/metrics/profile_request \ + -H "Content-Type: application/json" \ + -d '{ + "service_name": "web-service", + "request_type": "start", + "duration": 120, + "target_hosts": {"web-01": null}, + "additional_args": { + "profiler_configs": { + "async_profiler": { "enabled": true, "time": "wall" } + } + } + }' +``` + +Allocation profiling: + +```bash +curl -X POST http://localhost:8000/api/metrics/profile_request \ + -H "Content-Type: application/json" \ + -d '{ + "service_name": "web-service", + "request_type": "start", + "duration": 60, + "target_hosts": {"web-01": null}, + "additional_args": { + "profiler_configs": { + "async_profiler": { "enabled": true, "time": "alloc", "alloc_interval": "2MB" } + } + } + }' +``` + +### 2. Agent Heartbeat (sent automatically by the agent) + ```bash -# Agent automatically sends: curl -X POST http://localhost:8000/api/metrics/heartbeat \ -H "Content-Type: application/json" \ -d '{ "hostname": "web-01", "ip_address": "10.0.1.10", "service_name": "web-service", - "status": "active" + "status": "active", + "received_command_ids": [], + "executed_command_ids": [], + "perf_supported_events": ["cpu-cycles", "instructions"] }' ``` ### 3. Agent Receives Command + ```json { "success": true, @@ -192,13 +604,15 @@ curl -X POST http://localhost:8000/api/metrics/heartbeat \ "combined_config": { "duration": 120, "frequency": 11, - "profiling_mode": "cpu" + "profiling_mode": "cpu", + "continuous": false } } } ``` ### 4. Agent Reports Completion + ```bash curl -X POST http://localhost:8000/api/metrics/command_completion \ -H "Content-Type: application/json" \ @@ -206,140 +620,143 @@ curl -X POST http://localhost:8000/api/metrics/command_completion \ "command_id": "cmd-12345", "hostname": "web-01", "status": "completed", - "execution_time": 122 + "execution_time": 0 }' ``` -## Testing - -### 1. Test Heartbeat System -```bash -cd gprofiler-performance-studio -python3 test_heartbeat_system.py -``` - -This script: -- Simulates agent heartbeat behavior -- Creates test profiling requests -- Verifies command delivery and idempotency -- Tests both start and stop commands +## Configuration -### 2. Run Test Agent -```bash -python3 run_heartbeat_agent.py -``` +### Backend Environment Variables -This script: -- Starts a real gProfiler agent in heartbeat mode -- Connects to the Performance Studio backend -- Receives and executes actual profiling commands +| Variable | Default | Purpose | +|---|---|---| +| `MAX_PROFILING_REQUEST_HOSTS` | `20` | Max number of target hosts per single profiling request | +| `MAX_SIMULTANEOUS_PROFILING_HOSTS` | `10` (%) | Max percentage of the total fleet that can be profiled simultaneously; enforced at bulk-request level | +| `ACTIVE_HOST_HEARTBEAT_MAX_DELTA_HOURS` | `24` | Hours of inactivity before a host is excluded from capacity calculations | +| `SLACK_BOT_TOKEN` | — | If set, Slack notifications are sent on every profiling request | +| `SLACK_CHANNELS` | `#gprofiler-notifications` | Comma-separated Slack channels for notifications | +| `METRICS_ENABLED` | `false` | Enable internal SLI metric publishing via ZMQ | +| `METRICS_AGENT_URL` | `tcp://localhost:18126` | ZMQ metrics agent address | +| `METRICS_SERVICE_NAME` | `gprofiler-webapp` | Service label for SLI metrics | -## Configuration +### Agent Configuration Summary -### Backend Configuration -```yaml -# Backend settings -database: - host: localhost - port: 5432 - database: gprofiler - -heartbeat: - max_age_minutes: 10 # Consider hosts offline after 10 minutes - cleanup_interval: 300 # Clean up old records every 5 minutes -``` +See the [CLI Reference](#cli-reference) above. The minimum viable invocation for heartbeat mode: -### Agent Configuration ```bash -# Required parameters ---enable-heartbeat-server # Enable heartbeat mode ---api-server URL # Performance Studio backend URL ---service-name NAME # Service identifier ---heartbeat-interval SECONDS # Heartbeat frequency (default: 30) - -# Optional parameters ---token TOKEN # Authentication token ---server-host URL # Profile upload server (can be same as api-server) ---no-verify # Skip SSL verification (testing only) +python3 gprofiler/main.py \ + --enable-heartbeat-server \ + --upload-results \ + --token "api-token" \ + --service-name "my-service" \ + --api-server "https://perf-studio.example.com" ``` ## Monitoring and Debugging ### Database Queries + ```sql --- Check active hosts -SELECT hostname, service_name, status, heartbeat_timestamp -FROM HostHeartbeats -WHERE status = 'active' AND heartbeat_timestamp > NOW() - INTERVAL '10 minutes'; +-- Check active hosts (default: hosts seen within last 24 hours) +SELECT hostname, service_name, status, heartbeat_timestamp +FROM HostHeartbeats +WHERE heartbeat_timestamp > NOW() - INTERVAL '24 hours' +ORDER BY heartbeat_timestamp DESC; -- Check pending commands -SELECT hostname, service_name, command_type, status, created_at -FROM ProfilingCommands +SELECT hostname, service_name, command_type, status, created_at +FROM ProfilingCommands WHERE status = 'pending'; -- Check command execution history -SELECT pe.hostname, pr.request_type, pe.status, pe.execution_time +SELECT pe.hostname, pr.request_type, pe.status, pe.execution_time, pe.error_message FROM ProfilingExecutions pe -JOIN ProfilingRequests pr ON pe.profiling_request_id = pr.ID +JOIN ProfilingRequests pr ON pe.profiling_request_id = pr.request_id ORDER BY pe.created_at DESC; ``` ### Log Monitoring + ```bash # Backend logs tail -f /var/log/gprofiler-studio/backend.log | grep -E "(heartbeat|command)" -# Agent logs +# Agent logs tail -f /tmp/gprofiler-heartbeat.log | grep -E "(heartbeat|command)" ``` +## Testing + +### Heartbeat System Tests + +Test scripts are located in `heartbeat_doc/`: + +```bash +cd gprofiler-performance-studio/heartbeat_doc +python3 test_heartbeat_system.py +``` + +This script: +- Simulates agent heartbeat behavior +- Creates test profiling requests +- Verifies command delivery and idempotency +- Tests both start and stop commands + +### Run Test Agent + +```bash +cd gprofiler-performance-studio/heartbeat_doc +python3 run_heartbeat_agent.py +``` + ## Troubleshooting ### Common Issues 1. **Agents not receiving commands** - - Check heartbeat connectivity to backend - - Verify service_name matches between request and agent - - Check agent authentication (token) + - Verify `service_name` matches exactly between the profiling request and the agent's `--service-name` flag + - Confirm the agent's hostname appears in `target_hosts` in the profiling request + - Check that the agent's heartbeat is reaching the backend (use the debug `curl` below) 2. **Commands executing multiple times** - - Verify agent is tracking `last_command_id` correctly - - Check for agent restarts that reset command tracking + - The agent tracks `received_command_ids` and `executed_command_ids` across heartbeats; a process restart clears these in-memory sets, which can allow re-execution of previously seen commands + - Check for rapid agent restarts + +3. **Commands not appearing for the agent** + - The command `status` may already be `'sent'` or `'completed'` from a previous heartbeat; only `'pending'` commands are dispatched + - Ensure no stale `ProfilingCommands` row with `(hostname, service_name)` unique constraint is blocking new commands -3. **Commands not being created** - - Verify `target_hostnames` includes the agent's hostname - - Check database constraints and foreign key relationships +4. **Bulk request rejected** + - Check `MAX_PROFILING_REQUEST_HOSTS` (default 20) and `MAX_SIMULTANEOUS_PROFILING_HOSTS` (default 10%) limits + - The bulk endpoint validates capacity across all requests in the batch before processing any of them ### Debug Commands + ```bash # Test backend connectivity -curl -v http://localhost:8000/api/metrics/heartbeat \ +curl -s -X POST http://localhost:8000/api/metrics/heartbeat \ -H "Content-Type: application/json" \ - -d '{"hostname":"test","ip_address":"127.0.0.1","service_name":"test","status":"active"}' + -d '{"hostname":"test","ip_address":"127.0.0.1","service_name":"test","status":"active"}' | python3 -m json.tool # Check database state -psql -d gprofiler -c "SELECT * FROM HostHeartbeats ORDER BY heartbeat_timestamp DESC LIMIT 5;" +psql -d gprofiler -c "SELECT hostname, service_name, status, heartbeat_timestamp FROM HostHeartbeats ORDER BY heartbeat_timestamp DESC LIMIT 10;" + +# Check pending and sent commands +psql -d gprofiler -c "SELECT hostname, service_name, command_type, status, created_at, sent_at FROM ProfilingCommands WHERE status IN ('pending','sent') ORDER BY created_at DESC;" ``` ## Security Considerations -1. **Authentication**: Use API tokens for agent authentication -2. **Network**: Secure communication with HTTPS/TLS -3. **Authorization**: Validate service permissions before creating commands -4. **Rate Limiting**: Implement rate limits on heartbeat endpoints -5. **Input Validation**: Sanitize all input parameters +1. **Authentication on heartbeat endpoints**: The `/api/metrics/heartbeat`, `/api/metrics/profile_request`, and `/api/metrics/command_completion` endpoints do **not** currently enforce authentication. The agent's `--token` flag is used for profile-upload API calls, not for the heartbeat protocol endpoints. +2. **Network**: Secure all communication with HTTPS. Use `--tls-client-cert` / `--tls-client-key` for mTLS where required. +3. **mTLS cert rotation**: Enable `--tls-cert-refresh-enabled` with an appropriate `--tls-cert-refresh-interval` to periodically rotate the client certificate without restarting the agent. +4. **TLS verification**: Never use `--no-verify` in production. +5. **Input Validation**: All request payloads are validated via Pydantic models; invalid inputs return HTTP 422. ## Performance Considerations -1. **Database Indexes**: Essential indexes are created for all lookup patterns -2. **Heartbeat Frequency**: Balance between responsiveness and load (default: 30s) -3. **Command Cleanup**: Implement periodic cleanup of old commands/executions -4. **Connection Pooling**: Use connection pooling for database access - -## Future Enhancements - -1. **Agent Discovery**: Automatic service registration -2. **Command Queuing**: Support for command queues per host -3. **Conditional Commands**: Commands based on host metrics or state -4. **Command Templates**: Predefined command templates for common scenarios -5. **Real-time Dashboard**: Web UI for monitoring active agents and commands +1. **Database Indexes**: All high-frequency lookup patterns (`hostname`, `service_name`, `status`, `heartbeat_timestamp`) are indexed. +2. **Heartbeat Frequency**: Default 30 s balances responsiveness against backend load. Reduce `--heartbeat-interval` for faster command pickup. +3. **Command Merging**: Multiple profiling requests for the same host are merged into a single `ProfilingCommands` row (max duration, max frequency, union of PIDs). This avoids parallel command dispatch to the same host. +4. **Connection Pooling**: Use connection pooling (e.g., PgBouncer) for database access at scale. +5. **Capacity Limits**: `MAX_SIMULTANEOUS_PROFILING_HOSTS` prevents fleet-wide profiling storms; tune this percentage for your environment. diff --git a/heartbeat_doc/WORKLOAD_LEVEL_PROFILING_SPEC.md b/heartbeat_doc/WORKLOAD_LEVEL_PROFILING_SPEC.md new file mode 100644 index 00000000..99f2739f --- /dev/null +++ b/heartbeat_doc/WORKLOAD_LEVEL_PROFILING_SPEC.md @@ -0,0 +1,365 @@ +# Workload-Level Profiling Spec for Performance Studio + +## Purpose + +This spec defines the backend and UI design for workload-level profiling in +Performance Studio. It also serves as the source-of-truth document for +spec-driven development of future workload-selection and heartbeat-inventory +changes in this repo. + +The design extends the existing heartbeat control plane rather than replacing +it: the backend stores workload inventory from agent heartbeats, exposes +workload-aware status views, and resolves workload selections into host/PID +commands before dispatch. + +## Problem Statement + +The existing dynamic profiling flow is host-centric: + +- the UI shows one row per host +- requests target `target_hosts` +- the backend persists host heartbeats and host commands +- the agent receives commands by host/service + +This is insufficient for Kubernetes-heavy deployments where users want to start +profiling from the level they reason about operationally: + +- namespace +- workload +- pod +- container +- process + +## Motivation + +Before this work, Performance Studio only supported **host-level** profiling: +the user picked individual hosts and profiling commands were issued per host. +Workload-level profiling exists to address two concrete operational pain points. + +### 1. Cluster churn breaks host-pinned profiling + +Hosts are constantly removed from and added to a cluster (autoscaling, spot +reclamation, rolling replacements). With host-pinned selection, every time the +fleet changes the user has to return to the UI and re-select hosts, and any +host added after the original selection is simply **not profiled**. + +Workload-level profiling fixes this by letting the user select an entire +**service** (and, in future, broader scopes). When a service is selected for +continuous profiling, the selection is treated as a durable **subscription**: +as new hosts for that service register via heartbeat, they are **immediately +and automatically enrolled** in profiling — no manual re-selection. See +[Continuous Service Subscriptions & Auto-Enrollment](#continuous-service-subscriptions--auto-enrollment). + +### 2. Users often want a specific process/container/pod, not whole hosts + +A host can run many workloads, but the user frequently cares about one +container, pod, or process (e.g. a single Java service in a shared node). Whole- +host profiling is both noisier and more expensive than necessary. + +Workload-level profiling lets the user target the precise scope they reason +about (`namespace`, `workload`, `pod`, `container`, `process`) and the backend +resolves that selection down to the exact `hostname -> [pid, ...]` mapping the +agent executes. See [Resolution Model](#resolution-model). + +## Goals + +1. Add workload-aware inventory without breaking the current heartbeat protocol. +2. Keep command dispatch backward compatible with host-based agent execution. +3. Let the UI present workload tabs and workload-aware confirmation summaries. +4. Create a spec that future work can evolve first, before code changes. + +## Non-Goals + +- redesigning profiling commands around pod-native execution +- adding a brand-new command queue model +- guaranteeing globally stable Kubernetes workload IDs in v1 +- solving historical inventory retention beyond current heartbeat freshness + +## Design Principles + +- **Additive schema changes only** for heartbeat inventory fields. +- **Backend resolution, agent execution**: workload targeting resolves to + host/PID mappings in the backend. +- **Best-effort metadata**: missing namespace/pod/container fields must not + break host-level behavior. +- **Freshness over history**: UI tabs represent active inventory from recent + heartbeats. + +## Data Model Changes + +`HostHeartbeats` is extended with: + +- `agent_version` +- `run_mode` +- `namespace` +- `pod_name` + +The flattened workload inventory reported by the agent is stored in the +normalized `HeartbeatContainers` and `HeartbeatProcesses` tables (an earlier +transitional `containers jsonb` column on `HostHeartbeats` has been dropped). +Each container entry may include: + +- container identity +- namespace/pod/workload metadata +- process list + +Only containerized workloads are stored: the agent reports an empty list for +hosts with no container runtime, and processes not mapped to a container are +covered by host-scope profiling instead. On each heartbeat the inventory is +diffed against the stored rows (upsert keyed on `(host_id, container_id)` and +`(container_row_id, pid)`), so an unchanged inventory performs no row writes. + +`ProfilingRequests` remains API-level intent storage. Workload selectors are +stored in `additional_args` as part of the request contract so the existing +table does not need a full relational redesign in v1. + +## API Contract + +### Heartbeat Ingress + +The existing `POST /api/metrics/heartbeat` endpoint now accepts optional +workload inventory fields: + +```json +{ + "hostname": "node-a", + "service_name": "checkout", + "namespace": "observability", + "pod_name": "gprofiler-abcde", + "agent_version": "1.2.3", + "run_mode": "k8s", + "containers": [ + { + "container_name": "checkout", + "namespace": "shop", + "pod_name": "checkout-7f8d9", + "workload_name": "checkout", + "workload_kind": "k8s", + "processes": [ + { "pid": 1234, "process_name": "java" } + ] + } + ] +} +``` + +### Profiling Request Ingress + +The request contract is extended with: + +- `target_scope` +- `target_entities` +- optional `target_hosts` for pure host targeting + +Supported `target_scope` values: + +- `host` +- `service` +- `namespace` +- `workload` +- `pod` +- `container` +- `process` + +Each entry in `target_entities` may include service/namespace/host/pod/container +and process selectors. + +## Resolution Model + +Before creating commands, the backend resolves workload selectors against the +fresh heartbeat inventory: + +1. fetch recent host heartbeats for the service +2. flatten `containers -> processes` into inventory records +3. filter records by the requested scope and selectors +4. convert the selection into: + - `hostname -> null` for host-wide execution + - `hostname -> [pid, ...]` for process-scoped execution + +The command queue remains unchanged after this resolution step. + +## Continuous Service Subscriptions & Auto-Enrollment + +This realizes [Motivation #1](#1-cluster-churn-breaks-host-pinned-profiling): +a service-wide continuous profiling request behaves as a standing subscription +so that hosts which register *after* the request still get profiled. + +### Subscription definition + +A service is **actively subscribed** when its most recent service-scoped +(`additional_args.target_scope == "service"`) continuous (`continuous == true`) +`start` request in `ProfilingRequests` is newer than any service-scoped `stop` +request for that service, and the start request was not cancelled. +Implemented by `DBManager.get_active_service_subscription(service_name)`. + +### Auto-enrollment on heartbeat + +On every `POST /api/metrics/heartbeat`, after the host row is upserted, the +backend calls `DBManager.auto_subscribe_host_to_service(hostname, service_name)` +(see `receive_heartbeat`). The logic is: + +1. Look up the active service subscription. If none, do nothing. +2. If the reporting host already has a current command, do nothing (so explicit + per-host actions — including stops — are preserved). +3. Otherwise create a `start` command for the host, rebuilt from the + subscription request's stored configuration (frequency, duration, mode, + profiler configs). The command is created *before* the command lookup in the + same heartbeat, so the new host receives it on the very next response. + +### Behavior summary + +| Situation | Result | +|-----------|--------| +| New host heartbeats for a service with an active subscription | Auto-enrolled (start command created immediately) | +| New host heartbeats for a service with no subscription | No command | +| New host heartbeats after a service-wide stop | No command (stop is newer than start) | +| Existing host already has a command | Left untouched | + +### Edge cases / limitations (v1) + +- Auto-enrollment only fires when a host has **no** current command. A host + whose previous command reached a terminal state (`completed`/`failed`) is not + re-enrolled in v1; continuous commands remain in `sent` state, so this is rare. +- Subscriptions are scoped to `service` only. Namespace/pod/container/process + subscriptions are intentionally deferred (see Future Extensions). +- A host-level stop issued while a service subscription is active will keep that + host stopped only until its command state is cleared; durable per-host opt-out + within a subscription is future work. + +## UI Design + +The profiling console exposes tabs for: + +- Services +- Namespaces +- Hosts +- Pods +- Containers +- Processes + +The same page also supports: + +- scope-aware filters +- tab counts derived from active inventory +- scope-aware selection summaries in the confirmation dialog +- reuse of the existing bulk start/stop workflow + +## Freshness Rules + +Workload inventory is based on recent heartbeats only. The initial design uses +the same active-host time window already used by the status page, so stale pods +and containers naturally disappear when agents stop reporting them. + +## Failure Handling + +If workload resolution finds no active targets: + +- the API should reject the request with a clear validation error +- no commands should be created + +If workload inventory is incomplete: + +- host-level targeting must continue to work +- workload tabs may show partial data +- the UI should not invent missing workload relationships + +## Backward Compatibility + +This design preserves compatibility because: + +- old heartbeats can omit new fields +- old host-level requests still work +- commands sent to agents remain host/PID based +- the UI can still represent host-only rows + +## Acceptance Tests + +These acceptance criteria define "done" for the studio side of workload-level +profiling. They are written as Given/When/Then so they can drive spec-first +development and be implemented as automated API/integration tests. The freshness +window referenced below is the active-host heartbeat window (currently 2 +minutes). + +### Inventory & status views + +- **AT-S1 — Heartbeat populates inventory.** *Given* an agent posts a heartbeat + with `hostname`, `service_name`, optional `namespace`/`pod_name`, and + `containers[]` with processes, *When* it is stored, *Then* + `GET /api/metrics/profiling/workload_status?scope=host` returns a row for that + host while the heartbeat is within the freshness window. +- **AT-S2 — Tab counts per scope.** *Given* a set of fresh heartbeats, *When* + `workload_status` is queried, *Then* `tabCounts` reports the correct number of + distinct groups for each of `service`, `namespace`, `host`, `pod`, + `container`, and `process`, and `activeHosts` equals the distinct fresh + hostnames. +- **AT-S3 — Service tab is grouped by service.** *Given* multiple hosts of one + service, *When* `scope=service`, *Then* exactly **one** aggregated row is + returned per service (with host/pod/container/process counts), not one row per + host. +- **AT-S4 — Freshness filtering.** *Given* a host whose latest heartbeat is older + than the freshness window, *When* `workload_status` is queried, *Then* that + host (and its pods/containers/processes) is excluded from all tabs. + +### Resolution & command creation + +- **AT-S5 — Host-level start.** *Given* `scope=host` targeting host `H`, *When* a + start request is submitted, *Then* a `start` command is created for `H` and is + returned to `H` on its next heartbeat with the requested config. +- **AT-S6 — Service-level start fans out.** *Given* service `S` with hosts + `{H1, H2}`, *When* a `scope=service` start is submitted, *Then* a `start` + command is created for every current host of `S`. +- **AT-S7 — Workload scope resolves to PIDs.** *Given* a `process`, `container`, + or `pod` selection, *When* a start request is submitted, *Then* it resolves to + `hostname -> [pid, ...]` and the resulting commands carry exactly those PIDs. +- **AT-S8 — Empty resolution is rejected.** *Given* a selection that resolves to + zero active targets, *When* submitted, *Then* the API responds `422` and no + command is created. +- **AT-S9 — PMU validation.** *Given* requested perf events that a target host + does not report in `supported_perf_events`, *When* a start is submitted with + perf enabled, *Then* the API rejects it with a clear per-host validation error. + +### Continuous service subscriptions & auto-enrollment + +- **AT-S10 — New host auto-enrolls.** *Given* service `S` has an active + service-wide continuous `start` subscription, *When* a host that was **not** + part of the original selection heartbeats for `S` and has no current command, + *Then* a `start` command (built from the subscription config) is created and + returned on that same heartbeat. +- **AT-S11 — No subscription, no enrollment.** *Given* `S` has no active + subscription, *When* a new host heartbeats, *Then* no command is created. +- **AT-S12 — Stop deactivates the subscription.** *Given* a service-wide `stop` + newer than the latest service-wide `start`, *When* a new host heartbeats for + `S`, *Then* it is **not** enrolled. +- **AT-S13 — Existing command preserved.** *Given* a host already has a current + command, *When* it heartbeats under an active subscription, *Then* + auto-enrollment does **not** overwrite that command (explicit per-host actions + win). + +### Compatibility & failure handling + +- **AT-S14 — Legacy heartbeat.** *Given* a heartbeat without any workload fields, + *When* stored, *Then* host-level status and host/service commands still work, + and the host simply contributes no namespace/pod/container/process rows. +- **AT-S15 — Partial inventory.** *Given* heartbeats missing some workload fields + (e.g. no `pod_name`), *When* tabs are computed, *Then* unaffected scopes still + return rows and the backend does not invent missing relationships. + +## Spec-Driven Development Workflow + +All future workload-level backend or UI changes should follow: + +1. update this spec first +2. describe contract/schema changes explicitly +3. describe rollback and compatibility behavior +4. implement code afterward +5. keep the implementation aligned with the repo’s spec-driven guidance + +## Future Extensions + +Likely follow-up specs include: + +- durable workload identifiers and richer workload kinds +- workload-level stop semantics that survive pod churn +- historical inventory snapshots +- workload-level flamegraph pivots and deep links +- stronger validation for mixed host and workload selections diff --git a/scripts/dev/seed_heartbeats.py b/scripts/dev/seed_heartbeats.py new file mode 100644 index 00000000..e6c69b80 --- /dev/null +++ b/scripts/dev/seed_heartbeats.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Local dev helper: send synthetic agent heartbeats to the Performance Studio. + +This populates HostHeartbeats so the Adhoc Profile Configuration page +(/profiling) shows data across the Services / Namespaces / Hosts / Pods / +Containers / Processes tabs. The workload_status API only returns hosts whose +heartbeat_timestamp is within the last 2 minutes, so this script loops and +re-sends heartbeats on an interval to keep the fleet "live". + +Usage: + python3 seed_heartbeats.py # loop forever (default 45s) + python3 seed_heartbeats.py --once # send a single round and exit + python3 seed_heartbeats.py --interval 30 # custom loop interval (seconds) + +Env / flags: + --base-url default https://localhost:4433 + --user/--password basic-auth creds (default user/admin) +""" +import argparse +import json +import ssl +import sys +import time +import urllib.request +from datetime import datetime, timezone + +# Synthetic fleet: service -> list of host specs. +# Each container carries a namespace/pod/workload plus a few processes so that +# every scope tab (service/namespace/host/pod/container/process) has rows. +RUNTIMES = { + "java": "java -Xmx4g -jar app.jar", + "python": "python3 /srv/app/server.py", + "node": "node dist/server.js", + "nginx": "nginx: master process", + "envoy": "/usr/local/bin/envoy -c /etc/envoy/envoy.yaml", + "go": "/srv/bin/service", +} + +FLEET = [ + { + "service": "ingress-webapp-canary-use1", + "namespace": "canary", + "agent_version": "1.53.1", + "hosts": [ + {"host": "ip-10-0-1-245", "ip": "10.0.1.245", + "containers": [ + {"name": "webapp-nginx", "pod": "ingress-webapp-canary-a4b1", "kind": "Deployment", + "procs": [("nginx", 7891), ("node", 9156)]}, + ]}, + {"host": "ip-10-0-2-156", "ip": "10.0.2.156", + "containers": [ + {"name": "webapp-api", "pod": "ingress-webapp-canary-77cd", "kind": "Deployment", + "procs": [("java", 8234), ("python", 8412)]}, + ]}, + {"host": "ip-10-0-3-089", "ip": "10.0.3.089", + "containers": [ + {"name": "webapp-worker", "pod": "ingress-webapp-canary-2f0a", "kind": "Deployment", + "procs": [("python", 9001)]}, + ]}, + ], + }, + { + "service": "homefeed", + "namespace": "production", + "agent_version": "1.53.1", + "hosts": [ + {"host": "ip-10-1-4-234", "ip": "10.1.4.234", + "containers": [ + {"name": "homefeed-app", "pod": "unity-homefeed-docker-prod-7d8f", "kind": "StatefulSet", + "procs": [("node", 9156), ("go", 9210)]}, + ]}, + {"host": "ip-10-1-5-101", "ip": "10.1.5.101", + "containers": [ + {"name": "homefeed-ranker", "pod": "unity-homefeed-docker-prod-9911", "kind": "StatefulSet", + "procs": [("java", 8801)]}, + ]}, + ], + }, + { + "service": "unity-p2p", + "namespace": "production", + "agent_version": "1.53.1", + "hosts": [ + {"host": "ip-10-1-6-12", "ip": "10.1.6.12", + "containers": [ + {"name": "unity-p2p-main", "pod": "unity-p2p-docker-prod-7d8f", "kind": "Deployment", + "procs": [("java", 8234)]}, + {"name": "unity-p2p-sidecar", "pod": "unity-p2p-docker-prod-7d8f", "kind": "Deployment", + "procs": [("envoy", 8421)]}, + ]}, + ], + }, + { + "service": "ingress-trk-canary-use1", + "namespace": "staging", + "agent_version": "1.53.1", + "hosts": [ + {"host": "ip-10-2-7-55", "ip": "10.2.7.55", + "containers": [ + {"name": "trk-collector", "pod": "ingress-trk-staging-1a2b", "kind": "DaemonSet", + "procs": [("go", 7001)]}, + ]}, + ], + }, + { + "service": "ingress-widgets-canary-use1", + "namespace": "development", + "agent_version": "1.53.1", + "hosts": [ + {"host": "ip-10-3-8-77", "ip": "10.3.8.77", + "containers": [ + {"name": "widgets-web", "pod": "ingress-widgets-dev-9f2c", "kind": "Deployment", + "procs": [("python", 6001), ("nginx", 6010)]}, + ]}, + ], + }, +] + + +def build_heartbeats(): + rounds = [] + for svc in FLEET: + for host in svc["hosts"]: + containers = [] + for c in host["containers"]: + containers.append({ + "containerId": f"{c['name']}-{host['host']}", + "containerName": c["name"], + "runtime": "containerd", + "namespace": svc["namespace"], + "podName": c["pod"], + "workloadName": c["pod"].rsplit("-", 1)[0], + "workloadKind": c["kind"], + "processes": [ + {"pid": pid, "processName": RUNTIMES.get(rt, rt)} + for (rt, pid) in c["procs"] + ], + }) + rounds.append({ + "ip_address": host["ip"], + "hostname": host["host"], + "service_name": svc["service"], + "agent_version": svc["agent_version"], + "run_mode": "container", + "namespace": svc["namespace"], + "pod_name": host["containers"][0]["pod"], + "containers": containers, + "status": "active", + # Agent-normalized PMU event names (UI 'cpu-cycles' -> 'cycles') + "perf_supported_events": [ + "cycles", "instructions", "cache-misses", "cache-references", + "branch-instructions", "branch-misses", + ], + }) + return rounds + + +def send(base_url, user, password, payloads): + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() + mgr.add_password(None, base_url, user, password) + opener = urllib.request.build_opener( + urllib.request.HTTPBasicAuthHandler(mgr), + urllib.request.HTTPSHandler(context=ctx), + ) + ok = 0 + for hb in payloads: + hb["timestamp"] = datetime.now(timezone.utc).isoformat() + data = json.dumps(hb).encode("utf-8") + req = urllib.request.Request( + f"{base_url}/api/metrics/heartbeat", data=data, + headers={"Content-Type": "application/json"}, method="POST", + ) + try: + with opener.open(req, timeout=10) as resp: + if resp.status == 200: + ok += 1 + except Exception as e: # noqa: BLE001 + print(f" heartbeat failed for {hb['hostname']}: {e}", file=sys.stderr) + return ok + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--base-url", default="https://localhost:4433") + ap.add_argument("--user", default="user") + ap.add_argument("--password", default="admin") + ap.add_argument("--interval", type=int, default=45) + ap.add_argument("--once", action="store_true") + args = ap.parse_args() + + payloads = build_heartbeats() + print(f"Seeding {len(payloads)} hosts across {len(FLEET)} services to {args.base_url}") + while True: + ok = send(args.base_url, args.user, args.password, payloads) + print(f"[{datetime.now().strftime('%H:%M:%S')}] sent {ok}/{len(payloads)} heartbeats") + if args.once: + break + time.sleep(args.interval) + + +if __name__ == "__main__": + main() diff --git a/scripts/setup/postgres/gprofiler_recreate.sql b/scripts/setup/postgres/gprofiler_recreate.sql index 7f507576..46d81b78 100644 --- a/scripts/setup/postgres/gprofiler_recreate.sql +++ b/scripts/setup/postgres/gprofiler_recreate.sql @@ -233,6 +233,27 @@ CREATE TABLE ProfilerSnapshots ( filter_content jsonb NULL ); +-- AdhocFlamegraphMetadata table for storing PMU events and other adhoc profiling metadata +CREATE TABLE AdhocFlamegraphMetadata ( + ID bigserial PRIMARY KEY, + service_id bigint NOT NULL, + hostname text NOT NULL, + s3_key text NOT NULL UNIQUE, + perf_events text[], + start_time timestamp NOT NULL, + end_time timestamp NOT NULL, + file_size bigint, + created_at timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT fk_adhoc_flamegraph_service + FOREIGN KEY (service_id) + REFERENCES Services(ID) + ON DELETE CASCADE +); + +CREATE INDEX idx_adhoc_metadata_service_time ON AdhocFlamegraphMetadata(service_id, start_time DESC); +CREATE INDEX idx_adhoc_metadata_s3_key ON AdhocFlamegraphMetadata(s3_key); +CREATE INDEX idx_adhoc_metadata_hostname ON AdhocFlamegraphMetadata(hostname); + CREATE TABLE MinesweeperFrames ( ID bigserial PRIMARY KEY, snapshot bigint NOT NULL CONSTRAINT "minesweeper_frame must belong to a valid snapshot" REFERENCES ProfilerSnapshots, @@ -255,11 +276,16 @@ CREATE TABLE HostHeartbeats ( hostname text NOT NULL, ip_address inet NOT NULL, service_name text NOT NULL, + agent_version text NULL, + run_mode text NULL, + namespace text NULL, + pod_name text NULL, last_command_id uuid NULL, received_command_ids uuid[] NULL, executed_command_ids uuid[] NULL, status HostStatus NOT NULL DEFAULT 'active', heartbeat_timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + supported_perf_events text[] NULL, created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, CONSTRAINT "unique_host_heartbeat" UNIQUE (hostname, service_name) @@ -270,6 +296,50 @@ CREATE INDEX idx_hostheartbeats_hostname ON HostHeartbeats (hostname); CREATE INDEX idx_hostheartbeats_service_name ON HostHeartbeats (service_name); CREATE INDEX idx_hostheartbeats_status ON HostHeartbeats (status); CREATE INDEX idx_hostheartbeats_heartbeat_timestamp ON HostHeartbeats (heartbeat_timestamp); +CREATE INDEX idx_hostheartbeats_namespace ON HostHeartbeats (namespace); +CREATE INDEX idx_hostheartbeats_pod_name ON HostHeartbeats (pod_name); + +-- Structured workload inventory (normalized form of the container/process data +-- reported in each heartbeat). Every process is scoped to a container, so this +-- currently models containerized workloads only; non-containerized (e.g. +-- systemd/bare-metal) processes are not represented here and are covered by +-- host-scope profiling instead. +CREATE TABLE HeartbeatContainers ( + id bigserial PRIMARY KEY, + host_id bigint NOT NULL REFERENCES HostHeartbeats (ID) ON DELETE CASCADE, + container_id text NULL, + container_name text NULL, + runtime text NULL, + namespace text NULL, + pod_name text NULL, + workload_name text NULL, + workload_kind text NULL, + updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT unique_heartbeat_container UNIQUE (host_id, container_id) +); + +CREATE INDEX idx_hb_containers_host_id ON HeartbeatContainers (host_id); +CREATE INDEX idx_hb_containers_namespace ON HeartbeatContainers (namespace); +CREATE INDEX idx_hb_containers_pod_name ON HeartbeatContainers (pod_name); +CREATE INDEX idx_hb_containers_workload_name ON HeartbeatContainers (workload_name); + +CREATE TABLE HeartbeatProcesses ( + id bigserial PRIMARY KEY, + container_row_id bigint NOT NULL REFERENCES HeartbeatContainers (id) ON DELETE CASCADE, + pid integer NOT NULL, + process_name text NULL, + updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT unique_heartbeat_process UNIQUE (container_row_id, pid) +); + +CREATE INDEX idx_hb_processes_container_row_id ON HeartbeatProcesses (container_row_id); +CREATE INDEX idx_hb_processes_process_name ON HeartbeatProcesses (process_name); + +-- Cache a block of ids per backend so high-frequency heartbeat inserts don't +-- contend on the sequence buffer lock (see migrations/increase_heartbeat_sequence_cache.sql). +ALTER SEQUENCE hostheartbeats_id_seq CACHE 500; +ALTER SEQUENCE heartbeatcontainers_id_seq CACHE 500; +ALTER SEQUENCE heartbeatprocesses_id_seq CACHE 500; -- Profiling Requests Table (simplified) CREATE TABLE ProfilingRequests ( @@ -354,27 +424,6 @@ CREATE INDEX idx_profilingexecutions_hostname ON ProfilingExecutions (hostname); CREATE INDEX idx_profilingexecutions_profiling_request_id ON ProfilingExecutions (profiling_request_id); CREATE INDEX idx_profilingexecutions_status ON ProfilingExecutions (status); --- Adhoc Flamegraph Metadata Table -CREATE TABLE AdhocFlamegraphMetadata ( - ID bigserial PRIMARY KEY, - service_id bigint NOT NULL, - hostname text NOT NULL, - s3_key text NOT NULL UNIQUE, - perf_events text[], - start_time timestamp NOT NULL, - end_time timestamp NOT NULL, - file_size bigint, - created_at timestamp DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT fk_adhoc_flamegraph_service - FOREIGN KEY (service_id) - REFERENCES Services(ID) - ON DELETE CASCADE -); - -CREATE INDEX idx_adhoc_metadata_service_time ON AdhocFlamegraphMetadata(service_id, start_time DESC); -CREATE INDEX idx_adhoc_metadata_s3_key ON AdhocFlamegraphMetadata(s3_key); -CREATE INDEX idx_adhoc_metadata_hostname ON AdhocFlamegraphMetadata(hostname); - -- FUNCTIONS CREATE OR REPLACE FUNCTION calc_profiler_usage_history(start_date timestamp without time zone, end_date timestamp without time zone, interval_s bigint, max_iterations bigint DEFAULT 3) @@ -806,3 +855,337 @@ create aggregate zz_hashagg(text) ( stype = text, initcond = ''); + + +-- ============================================================================ +-- Precomputed workload_status store (see migrations/add_workload_precompute_store.sql) +-- ============================================================================ +-- ---------------------------------------------------------------- generation meta +CREATE TABLE IF NOT EXISTS workload_snapshot_meta ( + id smallint PRIMARY KEY DEFAULT 1, + active_generation smallint NOT NULL DEFAULT 0, + built_at timestamp NULL, + build_duration_ms integer NULL, + snapshot_rows bigint NULL, + CONSTRAINT workload_snapshot_meta_singleton CHECK (id = 1) +); + +INSERT INTO workload_snapshot_meta (id, active_generation) +VALUES (1, 0) +ON CONFLICT (id) DO NOTHING; + +-- ------------------------------------------------------------- Layer 1: snapshot +-- One physical table per generation; identical schema. The row grain is one row +-- per (fresh host, container, process); hosts with no containers / containers +-- with no processes contribute a row with NULL child columns (LEFT JOIN grain), +-- matching the previous live flatten exactly. +DO $$ +DECLARE + gen text; +BEGIN + FOREACH gen IN ARRAY ARRAY['0', '1'] LOOP + EXECUTE format($f$ + CREATE TABLE IF NOT EXISTS workload_snapshot_%1$s ( + host_id bigint NOT NULL, + hostname text NOT NULL, + ip_address text NULL, + service_name text NOT NULL, + agent_version text NULL, + run_mode text NULL, + heartbeat_timestamp timestamp NOT NULL, + namespace text NULL, + pod_name text NULL, + container_name text NULL, + workload_name text NULL, + workload_kind text NULL, + pid integer NULL, + process_name text NULL, + command_type text NULL, + command_status text NULL, + combined_config jsonb NULL, + profiling_status text NULL + ) + $f$, gen); + -- Filter-column indexes so filtered reads touch only the matching subset. + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_service ON workload_snapshot_%1$s (service_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_hostname ON workload_snapshot_%1$s (hostname)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_namespace ON workload_snapshot_%1$s (namespace)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_pod ON workload_snapshot_%1$s (pod_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_container ON workload_snapshot_%1$s (container_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_process ON workload_snapshot_%1$s (process_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_pid ON workload_snapshot_%1$s (pid)', gen); + END LOOP; +END $$; + +-- ------------------------------------------------- Layer 2: per-scope grouped rows +-- Precomputed grouped rows for the coarse, few/large-entity scopes that are slow +-- to aggregate live (service / namespace / pod / host). Fine scopes +-- (container / process) are served from the Layer 1 snapshot directly. Column +-- shape mirrors the dicts get_workload_inventory_status returns, so the reader +-- reuses the existing row-building code. +DO $$ +DECLARE + gen text; +BEGIN + FOREACH gen IN ARRAY ARRAY['0', '1'] LOOP + EXECUTE format($f$ + CREATE TABLE IF NOT EXISTS workload_scope_summary_%1$s ( + scope text NOT NULL, + sort_seq integer NOT NULL, -- default (key) ordering position + row_id text NOT NULL, + service_name text NULL, + hostname text NULL, + namespace text NULL, + pod_name text NULL, + container_name text NULL, + host_count integer NULL, + namespace_count integer NULL, + pod_count integer NULL, + container_count integer NULL, + process_count integer NULL, + pids integer[] NULL, + l_hostname text NULL, + l_ip_address text NULL, + l_namespace text NULL, + l_pod_name text NULL, + l_container_name text NULL, + l_workload_name text NULL, + l_workload_kind text NULL, + l_process_name text NULL, + l_pid integer NULL, + l_command_type text NULL, + l_command_status text NULL, + l_combined_config jsonb NULL, + any_active boolean NULL, + l_profiling_status text NULL, + l_agent_version text NULL, + l_run_mode text NULL, + l_heartbeat_timestamp timestamp NULL + ) + $f$, gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_summary_%1$s_scope_seq ON workload_scope_summary_%1$s (scope, sort_seq)', gen); + END LOOP; +END $$; + +-- ------------------------------------------------------- Layer 2: tab counts +CREATE TABLE IF NOT EXISTS workload_tab_counts_0 ( + scope text PRIMARY KEY, + count bigint NOT NULL +); +CREATE TABLE IF NOT EXISTS workload_tab_counts_1 ( + scope text PRIMARY KEY, + count bigint NOT NULL +); + +-- --------------------------------------------------------------- read-side views +-- Point at generation 0 initially; the refresh worker re-points these on swap. +CREATE OR REPLACE VIEW workload_snapshot AS SELECT * FROM workload_snapshot_0; +CREATE OR REPLACE VIEW workload_scope_summary AS SELECT * FROM workload_scope_summary_0; +CREATE OR REPLACE VIEW workload_tab_counts AS SELECT * FROM workload_tab_counts_0; + +-- Optimized workload_status refresh. +-- +-- The first version built Layer 2 by aggregating the flat, process-grain +-- workload_snapshot (~1.9M rows) with naive 7x COUNT(DISTINCT) tab counts and +-- array_agg(DISTINCT pid) coarse GROUP BYs -- the exact patterns the live +-- endpoint was optimized away from -- which took ~18 min at prod scale and, +-- under the ~30s cron, piled up on the meta-row lock. +-- +-- This version: +-- * Computes Layer 2 (tab counts + service/namespace/pod summaries) from the +-- SOURCE tables at the appropriate grain, reusing the tiered / two-grain +-- tricks (COUNT(*) FROM (SELECT DISTINCT ...) counts; container-grain +-- aggregates; distinct-subquery process_count; host-grain any_active). Prod +-- read-only prototypes: tab_counts 7.7s, service summary 5.3s. +-- * Guards against overlap with a non-blocking advisory lock, so a slow build +-- can never queue behind another (no pile-up), regardless of cron cadence. +-- * No longer populates the Layer 1 workload_snapshot table (nothing reads it +-- yet -- filtered reads still use the live path). Re-introduce an optimized +-- snapshot build when the filtered-read-from-snapshot path is implemented. +-- +-- any_active for the coarse scopes is computed at the host grain (a host with an +-- active whole-host command marks its groups active). For service scope this is +-- provably identical to the PID-aware value; for namespace/pod it can over-mark +-- only under PID-targeted commands, which are effectively unused in practice. + +CREATE OR REPLACE PROCEDURE refresh_workload_snapshot(IN fresh_interval interval DEFAULT '2 minutes') + LANGUAGE plpgsql +AS $procedure$ +DECLARE + cur_gen smallint; + new_gen smallint; + sum_tbl text; + cnt_tbl text; + t0 timestamptz := clock_timestamp(); + src_cte text; + agg_tail text; + sum_cols text := 'scope, sort_seq, row_id, service_name, hostname, namespace, pod_name, container_name,' + || 'host_count, namespace_count, pod_count, container_count, process_count, pids,' + || 'l_hostname, l_ip_address, l_namespace, l_pod_name, l_container_name, l_workload_name, l_workload_kind,' + || 'l_process_name, l_pid, l_command_type, l_command_status, l_combined_config, any_active, l_profiling_status,' + || 'l_agent_version, l_run_mode, l_heartbeat_timestamp'; +BEGIN + -- Non-blocking guard: if a refresh is already running, skip instead of queuing. + IF NOT pg_try_advisory_lock(3126073) THEN + RAISE NOTICE 'refresh_workload_snapshot: another refresh is running, skipping'; + RETURN; + END IF; + + SELECT active_generation INTO cur_gen FROM workload_snapshot_meta WHERE id = 1; + new_gen := 1 - cur_gen; + sum_tbl := 'workload_scope_summary_' || new_gen; + cnt_tbl := 'workload_tab_counts_' || new_gen; + + EXECUTE format('TRUNCATE %I', sum_tbl); + EXECUTE format('TRUNCATE %I', cnt_tbl); + + -- Tab counts: tiered, each counted at the shallowest grain that exposes it, + -- via COUNT(*) FROM (SELECT DISTINCT ...) (hash-distinct, not sort-per-agg). + EXECUTE format($f$ + INSERT INTO %1$I (scope, count) + WITH fresh AS MATERIALIZED ( + SELECT id, hostname, service_name FROM HostHeartbeats + WHERE heartbeat_timestamp > NOW() - %2$L::interval + ) + SELECT k, v FROM ( + SELECT + (SELECT COUNT(DISTINCT service_name) FROM fresh) AS c_service, + (SELECT COUNT(DISTINCT hostname) FROM fresh) AS active_hosts, + (SELECT COUNT(*) FROM (SELECT DISTINCT service_name, hostname FROM fresh) d) AS c_host, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.service_name, hc.namespace + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + WHERE hc.namespace IS NOT NULL) d) AS c_namespace, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.service_name, hc.namespace, hc.pod_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + WHERE hc.pod_name IS NOT NULL) d) AS c_pod, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.service_name, f.hostname, hc.namespace, hc.pod_name, hc.container_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + WHERE hc.container_name IS NOT NULL) d) AS c_container, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.id, hp.pid + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + WHERE hp.pid IS NOT NULL) d) AS c_process + ) t, + LATERAL (VALUES ('service', c_service), ('host', c_host), ('namespace', c_namespace), + ('pod', c_pod), ('container', c_container), ('process', c_process), + ('active_hosts', active_hosts)) x(k, v) + $f$, cnt_tbl, fresh_interval); + + -- Shared container-grain source for the coarse summaries. + src_cte := format($c$ + fresh AS MATERIALIZED ( + SELECT id, hostname, host(ip_address) AS ip_address, service_name, agent_version, run_mode, heartbeat_timestamp + FROM HostHeartbeats WHERE heartbeat_timestamp > NOW() - %L::interval + ), + cc AS (SELECT hostname, service_name, command_type, status, combined_config FROM ProfilingCommands), + cont AS ( + SELECT f.id, f.hostname, f.ip_address, f.service_name, f.agent_version, f.run_mode, f.heartbeat_timestamp, + hc.namespace, hc.pod_name, hc.container_name, hc.workload_name, hc.workload_kind, + COALESCE(c.command_type, 'N/A') AS command_type, c.status AS command_status, c.combined_config, + (c.command_type = 'start' AND c.status IN ('pending','sent','completed')) AS host_active + FROM fresh f + LEFT JOIN cc c ON c.hostname = f.hostname AND c.service_name = f.service_name + LEFT JOIN HeartbeatContainers hc ON hc.host_id = f.id + ) + $c$, fresh_interval); + + -- Per-group aggregate tail over the container-grain `cont` relation (unqualified cols). + agg_tail := $a$ + COUNT(DISTINCT hostname) AS host_count, + COUNT(DISTINCT namespace) FILTER (WHERE namespace IS NOT NULL) AS namespace_count, + COUNT(DISTINCT pod_name) FILTER (WHERE pod_name IS NOT NULL) AS pod_count, + COUNT(DISTINCT container_name) FILTER (WHERE container_name IS NOT NULL) AS container_count, + bool_or(host_active) AS any_active, + (array_agg(hostname ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_hostname, + (array_agg(ip_address ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_ip_address, + (array_agg(namespace ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_namespace, + (array_agg(pod_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_pod_name, + (array_agg(container_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_container_name, + (array_agg(workload_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_name, + (array_agg(workload_kind ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_kind, + (array_agg(command_type ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_type, + (array_agg(command_status ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_status, + (array_agg(combined_config ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_combined_config, + (array_agg(agent_version ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_agent_version, + (array_agg(run_mode ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_run_mode, + MAX(heartbeat_timestamp) AS l_heartbeat_timestamp + $a$; + + -- service + EXECUTE format($f$ + INSERT INTO %1$I (%2$s) + WITH %3$s, + pc AS (SELECT service_name, COUNT(*) AS process_count FROM ( + SELECT DISTINCT f.service_name, hp.pid, hp.process_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id WHERE hp.pid IS NOT NULL + ) d GROUP BY service_name) + SELECT 'service', row_number() OVER (ORDER BY ca.service_name ASC NULLS LAST), ca.service_name, + ca.service_name, NULL, NULL, NULL, NULL, + ca.host_count, ca.namespace_count, ca.pod_count, ca.container_count, COALESCE(pc.process_count, 0), NULL::integer[], + ca.l_hostname, ca.l_ip_address, ca.l_namespace, ca.l_pod_name, ca.l_container_name, ca.l_workload_name, ca.l_workload_kind, + NULL::text, NULL::integer, ca.l_command_type, ca.l_command_status, ca.l_combined_config, ca.any_active, NULL::text, + ca.l_agent_version, ca.l_run_mode, ca.l_heartbeat_timestamp + FROM (SELECT service_name, %4$s FROM cont GROUP BY service_name) ca + LEFT JOIN pc ON pc.service_name = ca.service_name + $f$, sum_tbl, sum_cols, src_cte, agg_tail); + + -- namespace + EXECUTE format($f$ + INSERT INTO %1$I (%2$s) + WITH %3$s, + pc AS (SELECT service_name, namespace, COUNT(*) AS process_count FROM ( + SELECT DISTINCT f.service_name, hc.namespace, hp.pid, hp.process_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + WHERE hp.pid IS NOT NULL AND hc.namespace IS NOT NULL + ) d GROUP BY service_name, namespace) + SELECT 'namespace', row_number() OVER (ORDER BY ca.service_name ASC NULLS LAST, ca.namespace ASC NULLS LAST), + ca.service_name || '|' || COALESCE(ca.namespace, ''), + ca.service_name, NULL, ca.namespace, NULL, NULL, + ca.host_count, ca.namespace_count, ca.pod_count, ca.container_count, COALESCE(pc.process_count, 0), NULL::integer[], + ca.l_hostname, ca.l_ip_address, ca.l_namespace, ca.l_pod_name, ca.l_container_name, ca.l_workload_name, ca.l_workload_kind, + NULL::text, NULL::integer, ca.l_command_type, ca.l_command_status, ca.l_combined_config, ca.any_active, NULL::text, + ca.l_agent_version, ca.l_run_mode, ca.l_heartbeat_timestamp + FROM (SELECT service_name, namespace, %4$s FROM cont WHERE namespace IS NOT NULL GROUP BY service_name, namespace) ca + LEFT JOIN pc ON pc.service_name = ca.service_name AND COALESCE(pc.namespace, '') = COALESCE(ca.namespace, '') + $f$, sum_tbl, sum_cols, src_cte, agg_tail); + + -- pod + EXECUTE format($f$ + INSERT INTO %1$I (%2$s) + WITH %3$s, + pc AS (SELECT service_name, namespace, pod_name, COUNT(*) AS process_count FROM ( + SELECT DISTINCT f.service_name, hc.namespace, hc.pod_name, hp.pid, hp.process_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + WHERE hp.pid IS NOT NULL AND hc.pod_name IS NOT NULL + ) d GROUP BY service_name, namespace, pod_name) + SELECT 'pod', row_number() OVER (ORDER BY ca.service_name ASC NULLS LAST, ca.namespace ASC NULLS LAST, ca.pod_name ASC NULLS LAST), + ca.service_name || '|' || COALESCE(ca.namespace, '') || '|' || COALESCE(ca.pod_name, ''), + ca.service_name, NULL, ca.namespace, ca.pod_name, NULL, + ca.host_count, ca.namespace_count, ca.pod_count, ca.container_count, COALESCE(pc.process_count, 0), NULL::integer[], + ca.l_hostname, ca.l_ip_address, ca.l_namespace, ca.l_pod_name, ca.l_container_name, ca.l_workload_name, ca.l_workload_kind, + NULL::text, NULL::integer, ca.l_command_type, ca.l_command_status, ca.l_combined_config, ca.any_active, NULL::text, + ca.l_agent_version, ca.l_run_mode, ca.l_heartbeat_timestamp + FROM (SELECT service_name, namespace, pod_name, %4$s FROM cont WHERE pod_name IS NOT NULL GROUP BY service_name, namespace, pod_name) ca + LEFT JOIN pc ON pc.service_name = ca.service_name + AND COALESCE(pc.namespace, '') = COALESCE(ca.namespace, '') + AND pc.pod_name = ca.pod_name + $f$, sum_tbl, sum_cols, src_cte, agg_tail); + + EXECUTE format('ANALYZE %I', sum_tbl); + + -- Atomic swap of the read-side views + flip the pointer, in this transaction. + EXECUTE format('CREATE OR REPLACE VIEW workload_scope_summary AS SELECT * FROM %I', sum_tbl); + EXECUTE format('CREATE OR REPLACE VIEW workload_tab_counts AS SELECT * FROM %I', cnt_tbl); + + UPDATE workload_snapshot_meta + SET active_generation = new_gen, + built_at = clock_timestamp(), + build_duration_ms = (EXTRACT(EPOCH FROM (clock_timestamp() - t0)) * 1000)::integer + WHERE id = 1; + + PERFORM pg_advisory_unlock(3126073); +END; +$procedure$; diff --git a/scripts/setup/postgres/migrations/add_structured_workload_inventory_tables.sql b/scripts/setup/postgres/migrations/add_structured_workload_inventory_tables.sql new file mode 100644 index 00000000..1d057fbe --- /dev/null +++ b/scripts/setup/postgres/migrations/add_structured_workload_inventory_tables.sql @@ -0,0 +1,54 @@ +-- +-- Copyright (C) 2023 Intel Corporation +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +-- Normalized workload inventory tables. +-- +-- These replace the per-host HostHeartbeats.containers JSONB snapshot with a +-- structured, queryable model. The heartbeat path writes the inventory into +-- these tables, and inventory reads (target resolution and workload status) +-- join/aggregate over them directly in SQL. The legacy JSONB column is removed +-- by drop_containers_jsonb_from_hostheartbeats.sql. + +CREATE TABLE IF NOT EXISTS HeartbeatContainers ( + id bigserial PRIMARY KEY, + host_id bigint NOT NULL REFERENCES HostHeartbeats (ID) ON DELETE CASCADE, + container_id text NULL, + container_name text NULL, + runtime text NULL, + namespace text NULL, + pod_name text NULL, + workload_name text NULL, + workload_kind text NULL, + updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT unique_heartbeat_container UNIQUE (host_id, container_id) +); + +CREATE INDEX IF NOT EXISTS idx_hb_containers_host_id ON HeartbeatContainers (host_id); +CREATE INDEX IF NOT EXISTS idx_hb_containers_namespace ON HeartbeatContainers (namespace); +CREATE INDEX IF NOT EXISTS idx_hb_containers_pod_name ON HeartbeatContainers (pod_name); +CREATE INDEX IF NOT EXISTS idx_hb_containers_workload_name ON HeartbeatContainers (workload_name); + +CREATE TABLE IF NOT EXISTS HeartbeatProcesses ( + id bigserial PRIMARY KEY, + container_row_id bigint NOT NULL REFERENCES HeartbeatContainers (id) ON DELETE CASCADE, + pid integer NOT NULL, + process_name text NULL, + updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT unique_heartbeat_process UNIQUE (container_row_id, pid) +); + +CREATE INDEX IF NOT EXISTS idx_hb_processes_container_row_id ON HeartbeatProcesses (container_row_id); +CREATE INDEX IF NOT EXISTS idx_hb_processes_process_name ON HeartbeatProcesses (process_name); diff --git a/scripts/setup/postgres/migrations/add_workload_inventory_to_hostheartbeats.sql b/scripts/setup/postgres/migrations/add_workload_inventory_to_hostheartbeats.sql new file mode 100644 index 00000000..9cfc804d --- /dev/null +++ b/scripts/setup/postgres/migrations/add_workload_inventory_to_hostheartbeats.sql @@ -0,0 +1,25 @@ +-- +-- Copyright (C) 2023 Intel Corporation +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +ALTER TABLE HostHeartbeats +ADD COLUMN IF NOT EXISTS agent_version text NULL, +ADD COLUMN IF NOT EXISTS run_mode text NULL, +ADD COLUMN IF NOT EXISTS namespace text NULL, +ADD COLUMN IF NOT EXISTS pod_name text NULL, +ADD COLUMN IF NOT EXISTS containers jsonb NOT NULL DEFAULT '[]'::jsonb; + +CREATE INDEX IF NOT EXISTS idx_hostheartbeats_namespace ON HostHeartbeats (namespace); +CREATE INDEX IF NOT EXISTS idx_hostheartbeats_pod_name ON HostHeartbeats (pod_name); diff --git a/scripts/setup/postgres/migrations/add_workload_precompute_store.sql b/scripts/setup/postgres/migrations/add_workload_precompute_store.sql new file mode 100644 index 00000000..6ef414f8 --- /dev/null +++ b/scripts/setup/postgres/migrations/add_workload_precompute_store.sql @@ -0,0 +1,312 @@ +-- +-- Copyright (C) 2023 Intel Corporation +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +-- Precomputed workload-status store. +-- +-- The workload_status endpoint used to aggregate the live +-- HostHeartbeats -> HeartbeatContainers -> HeartbeatProcesses flatten on every +-- request, which is O(fresh rows) and does not scale as fleets onboard. This +-- introduces a precomputed store, rebuilt every ~30s by the periodic-tasks +-- worker and read by the endpoint: +-- +-- * Layer 1 -- workload_snapshot: the fresh (host x container x process) +-- flatten, denormalized and indexed. Serves FILTERED reads (filters use the +-- indexes, so only a small subset is scanned; no live 3-way join). +-- * Layer 2 -- workload_scope_summary + workload_tab_counts: precomputed +-- per-scope grouped rows and the six tab counts for the UNFILTERED view. +-- Serves the default landing view instantly. +-- +-- Atomic swap: each object has two physical tables (_0 / _1); the reader always +-- queries a VIEW, and the refresh worker rebuilds the inactive generation then +-- re-points the views + flips workload_snapshot_meta in one transaction. + +-- ---------------------------------------------------------------- generation meta +CREATE TABLE IF NOT EXISTS workload_snapshot_meta ( + id smallint PRIMARY KEY DEFAULT 1, + active_generation smallint NOT NULL DEFAULT 0, + built_at timestamp NULL, + build_duration_ms integer NULL, + snapshot_rows bigint NULL, + CONSTRAINT workload_snapshot_meta_singleton CHECK (id = 1) +); + +INSERT INTO workload_snapshot_meta (id, active_generation) +VALUES (1, 0) +ON CONFLICT (id) DO NOTHING; + +-- ------------------------------------------------------------- Layer 1: snapshot +-- One physical table per generation; identical schema. The row grain is one row +-- per (fresh host, container, process); hosts with no containers / containers +-- with no processes contribute a row with NULL child columns (LEFT JOIN grain), +-- matching the previous live flatten exactly. +DO $$ +DECLARE + gen text; +BEGIN + FOREACH gen IN ARRAY ARRAY['0', '1'] LOOP + EXECUTE format($f$ + CREATE TABLE IF NOT EXISTS workload_snapshot_%1$s ( + host_id bigint NOT NULL, + hostname text NOT NULL, + ip_address text NULL, + service_name text NOT NULL, + agent_version text NULL, + run_mode text NULL, + heartbeat_timestamp timestamp NOT NULL, + namespace text NULL, + pod_name text NULL, + container_name text NULL, + workload_name text NULL, + workload_kind text NULL, + pid integer NULL, + process_name text NULL, + command_type text NULL, + command_status text NULL, + combined_config jsonb NULL, + profiling_status text NULL + ) + $f$, gen); + -- Filter-column indexes so filtered reads touch only the matching subset. + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_service ON workload_snapshot_%1$s (service_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_hostname ON workload_snapshot_%1$s (hostname)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_namespace ON workload_snapshot_%1$s (namespace)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_pod ON workload_snapshot_%1$s (pod_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_container ON workload_snapshot_%1$s (container_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_process ON workload_snapshot_%1$s (process_name)', gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_snap_%1$s_pid ON workload_snapshot_%1$s (pid)', gen); + END LOOP; +END $$; + +-- ------------------------------------------------- Layer 2: per-scope grouped rows +-- Precomputed grouped rows for the coarse, few/large-entity scopes that are slow +-- to aggregate live (service / namespace / pod / host). Fine scopes +-- (container / process) are served from the Layer 1 snapshot directly. Column +-- shape mirrors the dicts get_workload_inventory_status returns, so the reader +-- reuses the existing row-building code. +DO $$ +DECLARE + gen text; +BEGIN + FOREACH gen IN ARRAY ARRAY['0', '1'] LOOP + EXECUTE format($f$ + CREATE TABLE IF NOT EXISTS workload_scope_summary_%1$s ( + scope text NOT NULL, + sort_seq integer NOT NULL, -- default (key) ordering position + row_id text NOT NULL, + service_name text NULL, + hostname text NULL, + namespace text NULL, + pod_name text NULL, + container_name text NULL, + host_count integer NULL, + namespace_count integer NULL, + pod_count integer NULL, + container_count integer NULL, + process_count integer NULL, + pids integer[] NULL, + l_hostname text NULL, + l_ip_address text NULL, + l_namespace text NULL, + l_pod_name text NULL, + l_container_name text NULL, + l_workload_name text NULL, + l_workload_kind text NULL, + l_process_name text NULL, + l_pid integer NULL, + l_command_type text NULL, + l_command_status text NULL, + l_combined_config jsonb NULL, + any_active boolean NULL, + l_profiling_status text NULL, + l_agent_version text NULL, + l_run_mode text NULL, + l_heartbeat_timestamp timestamp NULL + ) + $f$, gen); + EXECUTE format('CREATE INDEX IF NOT EXISTS idx_wl_summary_%1$s_scope_seq ON workload_scope_summary_%1$s (scope, sort_seq)', gen); + END LOOP; +END $$; + +-- ------------------------------------------------------- Layer 2: tab counts +CREATE TABLE IF NOT EXISTS workload_tab_counts_0 ( + scope text PRIMARY KEY, + count bigint NOT NULL +); +CREATE TABLE IF NOT EXISTS workload_tab_counts_1 ( + scope text PRIMARY KEY, + count bigint NOT NULL +); + +-- --------------------------------------------------------------- read-side views +-- Point at generation 0 initially; the refresh worker re-points these on swap. +CREATE OR REPLACE VIEW workload_snapshot AS SELECT * FROM workload_snapshot_0; +CREATE OR REPLACE VIEW workload_scope_summary AS SELECT * FROM workload_scope_summary_0; +CREATE OR REPLACE VIEW workload_tab_counts AS SELECT * FROM workload_tab_counts_0; + +-- ---------------------------------------------------------- refresh procedure +-- Rebuilds the inactive generation (Layer 1 snapshot + Layer 2 summaries/counts) +-- from the live heartbeat tables, then atomically re-points the read-side views +-- and flips workload_snapshot_meta. Readers only ever touch the active +-- generation's tables (via the views), so the rebuild never blocks reads. +CREATE OR REPLACE PROCEDURE refresh_workload_snapshot(IN fresh_interval interval DEFAULT '2 minutes') + LANGUAGE plpgsql +AS $procedure$ +DECLARE + cur_gen smallint; + new_gen smallint; + snap_tbl text; + sum_tbl text; + cnt_tbl text; + t0 timestamptz := clock_timestamp(); + n_rows bigint; + -- Shared per-group aggregate tail (identical for every coarse scope). + agg_tail text := $a$ + COUNT(DISTINCT hostname), + COUNT(DISTINCT namespace) FILTER (WHERE namespace IS NOT NULL), + COUNT(DISTINCT pod_name) FILTER (WHERE pod_name IS NOT NULL), + COUNT(DISTINCT container_name) FILTER (WHERE container_name IS NOT NULL), + COUNT(DISTINCT (pid, process_name)) FILTER (WHERE pid IS NOT NULL), + array_agg(DISTINCT pid) FILTER (WHERE pid IS NOT NULL), + (array_agg(hostname ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(ip_address ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(namespace ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(pod_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(container_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(workload_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(workload_kind ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(process_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(pid ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(command_type ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(command_status ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(combined_config ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + bool_or(profiling_status = 'active'), + (array_agg(profiling_status ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(agent_version ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + (array_agg(run_mode ORDER BY heartbeat_timestamp DESC NULLS LAST))[1], + MAX(heartbeat_timestamp) + $a$; + sum_cols text := $c$scope, sort_seq, row_id, service_name, hostname, namespace, pod_name, container_name, + host_count, namespace_count, pod_count, container_count, process_count, pids, + l_hostname, l_ip_address, l_namespace, l_pod_name, l_container_name, l_workload_name, l_workload_kind, + l_process_name, l_pid, l_command_type, l_command_status, l_combined_config, any_active, l_profiling_status, + l_agent_version, l_run_mode, l_heartbeat_timestamp$c$; +BEGIN + SELECT active_generation INTO cur_gen FROM workload_snapshot_meta WHERE id = 1 FOR UPDATE; + new_gen := 1 - cur_gen; + snap_tbl := 'workload_snapshot_' || new_gen; + sum_tbl := 'workload_scope_summary_' || new_gen; + cnt_tbl := 'workload_tab_counts_' || new_gen; + + EXECUTE format('TRUNCATE %I', snap_tbl); + EXECUTE format('TRUNCATE %I', sum_tbl); + EXECUTE format('TRUNCATE %I', cnt_tbl); + + -- Layer 1: the fresh flatten (host x container x process), PID-aware status. + EXECUTE format($f$ + INSERT INTO %1$I (host_id, hostname, ip_address, service_name, agent_version, run_mode, + heartbeat_timestamp, namespace, pod_name, container_name, workload_name, workload_kind, + pid, process_name, command_type, command_status, combined_config, profiling_status) + WITH fresh_hosts AS MATERIALIZED ( + SELECT fh.id, fh.hostname, host(fh.ip_address) AS ip_address, fh.service_name, + fh.agent_version, fh.run_mode, fh.heartbeat_timestamp + FROM HostHeartbeats fh + WHERE fh.heartbeat_timestamp > NOW() - %2$L::interval + ), + current_commands AS ( + SELECT hostname, service_name, command_type, status, combined_config FROM ProfilingCommands + ) + SELECT fh.id, fh.hostname, fh.ip_address, fh.service_name, fh.agent_version, fh.run_mode, + fh.heartbeat_timestamp, hc.namespace, hc.pod_name, hc.container_name, hc.workload_name, + hc.workload_kind, hp.pid, hp.process_name, + COALESCE(c.command_type, 'N/A'), c.status::text, c.combined_config, + CASE + WHEN c.status IS NULL THEN 'stopped' + WHEN c.command_type = 'start' AND c.status IN ('pending','sent','completed') THEN + CASE + WHEN c.combined_config IS NULL OR (c.combined_config -> 'pids') IS NULL + OR jsonb_typeof(c.combined_config -> 'pids') <> 'array' + OR jsonb_array_length(c.combined_config -> 'pids') = 0 + THEN 'active' + WHEN hp.pid IS NOT NULL AND EXISTS ( + SELECT 1 FROM jsonb_array_elements_text(c.combined_config -> 'pids') AS t(pid) + WHERE t.pid = hp.pid::text + ) THEN 'active' + ELSE 'stopped' + END + ELSE c.status::text + END + FROM fresh_hosts fh + LEFT JOIN current_commands c ON fh.hostname = c.hostname AND fh.service_name = c.service_name + LEFT JOIN HeartbeatContainers hc ON hc.host_id = fh.id + LEFT JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + $f$, snap_tbl, fresh_interval); + GET DIAGNOSTICS n_rows = ROW_COUNT; + + -- Layer 2: coarse-scope grouped rows (service / namespace / pod). host, + -- container and process scopes are served live from the snapshot (fast). + EXECUTE format($f$ + INSERT INTO %1$I (%3$s) + SELECT 'service', row_number() OVER (ORDER BY service_name ASC NULLS LAST), service_name, + service_name, NULL, NULL, NULL, NULL, %4$s + FROM %2$I GROUP BY service_name + $f$, sum_tbl, snap_tbl, sum_cols, agg_tail); + + EXECUTE format($f$ + INSERT INTO %1$I (%3$s) + SELECT 'namespace', row_number() OVER (ORDER BY service_name ASC NULLS LAST, namespace ASC NULLS LAST), + service_name || '|' || COALESCE(namespace, ''), + service_name, NULL, namespace, NULL, NULL, %4$s + FROM %2$I WHERE namespace IS NOT NULL GROUP BY service_name, namespace + $f$, sum_tbl, snap_tbl, sum_cols, agg_tail); + + EXECUTE format($f$ + INSERT INTO %1$I (%3$s) + SELECT 'pod', row_number() OVER (ORDER BY service_name ASC NULLS LAST, namespace ASC NULLS LAST, pod_name ASC NULLS LAST), + service_name || '|' || COALESCE(namespace, '') || '|' || COALESCE(pod_name, ''), + service_name, NULL, namespace, pod_name, NULL, %4$s + FROM %2$I WHERE pod_name IS NOT NULL GROUP BY service_name, namespace, pod_name + $f$, sum_tbl, snap_tbl, sum_cols, agg_tail); + + -- Layer 2: the six tab counts + active_hosts, one snapshot scan. + EXECUTE format($f$ + INSERT INTO %1$I (scope, count) + SELECT k, v FROM ( + SELECT COUNT(DISTINCT service_name) AS c_service, + COUNT(DISTINCT (service_name, hostname)) AS c_host, + COUNT(DISTINCT (service_name, namespace)) FILTER (WHERE namespace IS NOT NULL) AS c_namespace, + COUNT(DISTINCT (service_name, namespace, pod_name)) FILTER (WHERE pod_name IS NOT NULL) AS c_pod, + COUNT(DISTINCT (service_name, hostname, namespace, pod_name, container_name)) FILTER (WHERE container_name IS NOT NULL) AS c_container, + COUNT(DISTINCT (service_name, hostname, pid)) FILTER (WHERE pid IS NOT NULL) AS c_process, + COUNT(DISTINCT hostname) AS active_hosts + FROM %2$I + ) t, LATERAL (VALUES ('service', t.c_service), ('host', t.c_host), ('namespace', t.c_namespace), + ('pod', t.c_pod), ('container', t.c_container), ('process', t.c_process), + ('active_hosts', t.active_hosts)) AS x(k, v) + $f$, cnt_tbl, snap_tbl); + + -- Atomic swap: re-point views + flip the pointer, all in this transaction. + EXECUTE format('CREATE OR REPLACE VIEW workload_snapshot AS SELECT * FROM %I', snap_tbl); + EXECUTE format('CREATE OR REPLACE VIEW workload_scope_summary AS SELECT * FROM %I', sum_tbl); + EXECUTE format('CREATE OR REPLACE VIEW workload_tab_counts AS SELECT * FROM %I', cnt_tbl); + + UPDATE workload_snapshot_meta + SET active_generation = new_gen, + built_at = clock_timestamp(), + build_duration_ms = (EXTRACT(EPOCH FROM (clock_timestamp() - t0)) * 1000)::integer, + snapshot_rows = n_rows + WHERE id = 1; +END; +$procedure$; diff --git a/scripts/setup/postgres/migrations/drop_containers_jsonb_from_hostheartbeats.sql b/scripts/setup/postgres/migrations/drop_containers_jsonb_from_hostheartbeats.sql new file mode 100644 index 00000000..4dd23a01 --- /dev/null +++ b/scripts/setup/postgres/migrations/drop_containers_jsonb_from_hostheartbeats.sql @@ -0,0 +1,28 @@ +-- +-- Copyright (C) 2023 Intel Corporation +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +-- Drop the transitional HostHeartbeats.containers JSONB snapshot. +-- +-- Workload inventory now lives entirely in the normalized HeartbeatContainers / +-- HeartbeatProcesses tables (see add_structured_workload_inventory_tables.sql). +-- +-- DEPLOY ORDERING: apply this migration only AFTER the application code that no +-- longer reads or writes the containers column is live. The current heartbeat +-- write path stops populating this column and the read path no longer selects it, +-- so once that build is deployed the column is dead and safe to drop. + +ALTER TABLE HostHeartbeats +DROP COLUMN IF EXISTS containers; diff --git a/scripts/setup/postgres/migrations/increase_heartbeat_sequence_cache.sql b/scripts/setup/postgres/migrations/increase_heartbeat_sequence_cache.sql new file mode 100644 index 00000000..74c19b21 --- /dev/null +++ b/scripts/setup/postgres/migrations/increase_heartbeat_sequence_cache.sql @@ -0,0 +1,20 @@ +-- Increase the id-sequence cache on the heartbeat inventory tables. +-- +-- Every heartbeat re-proposes its full container/process inventory. The previous +-- INSERT ... ON CONFLICT evaluated the id DEFAULT (nextval) for every proposed row +-- BEFORE resolving the conflict, so ~all rows (unchanged, re-reported each beat) +-- still burned a sequence value. At fleet scale this was ~58k nextval/s on cache=1 +-- sequences, making the sequence buffer lock (LWLock:SerialBuffer) a top contention +-- point once heartbeat writes ran in parallel. +-- +-- The application fix (only inserting genuinely-new rows) removes most of that burn; +-- this larger cache is a complementary safety margin for the remaining real inserts +-- and the one-per-heartbeat HostHeartbeats upsert. Each backend now reserves a block +-- of ids per sequence-lock acquisition (~cache-factor fewer acquisitions). Gaps and +-- non-monotonic ids across sessions are harmless for these surrogate keys. +-- +-- Safe to run online and idempotent (re-running just re-sets the same cache). + +ALTER SEQUENCE heartbeatprocesses_id_seq CACHE 500; +ALTER SEQUENCE heartbeatcontainers_id_seq CACHE 500; +ALTER SEQUENCE hostheartbeats_id_seq CACHE 500; diff --git a/scripts/setup/postgres/migrations/optimize_workload_precompute_build.sql b/scripts/setup/postgres/migrations/optimize_workload_precompute_build.sql new file mode 100644 index 00000000..4a2a3a4e --- /dev/null +++ b/scripts/setup/postgres/migrations/optimize_workload_precompute_build.sql @@ -0,0 +1,225 @@ +-- +-- Copyright (C) 2023 Intel Corporation +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +-- Optimized workload_status refresh. +-- +-- The first version built Layer 2 by aggregating the flat, process-grain +-- workload_snapshot (~1.9M rows) with naive 7x COUNT(DISTINCT) tab counts and +-- array_agg(DISTINCT pid) coarse GROUP BYs -- the exact patterns the live +-- endpoint was optimized away from -- which took ~18 min at prod scale and, +-- under the ~30s cron, piled up on the meta-row lock. +-- +-- This version: +-- * Computes Layer 2 (tab counts + service/namespace/pod summaries) from the +-- SOURCE tables at the appropriate grain, reusing the tiered / two-grain +-- tricks (COUNT(*) FROM (SELECT DISTINCT ...) counts; container-grain +-- aggregates; distinct-subquery process_count; host-grain any_active). Prod +-- read-only prototypes: tab_counts 7.7s, service summary 5.3s. +-- * Guards against overlap with a non-blocking advisory lock, so a slow build +-- can never queue behind another (no pile-up), regardless of cron cadence. +-- * No longer populates the Layer 1 workload_snapshot table (nothing reads it +-- yet -- filtered reads still use the live path). Re-introduce an optimized +-- snapshot build when the filtered-read-from-snapshot path is implemented. +-- +-- any_active for the coarse scopes is computed at the host grain (a host with an +-- active whole-host command marks its groups active). For service scope this is +-- provably identical to the PID-aware value; for namespace/pod it can over-mark +-- only under PID-targeted commands, which are effectively unused in practice. + +CREATE OR REPLACE PROCEDURE refresh_workload_snapshot(IN fresh_interval interval DEFAULT '2 minutes') + LANGUAGE plpgsql +AS $procedure$ +DECLARE + cur_gen smallint; + new_gen smallint; + sum_tbl text; + cnt_tbl text; + t0 timestamptz := clock_timestamp(); + src_cte text; + agg_tail text; + sum_cols text := 'scope, sort_seq, row_id, service_name, hostname, namespace, pod_name, container_name,' + || 'host_count, namespace_count, pod_count, container_count, process_count, pids,' + || 'l_hostname, l_ip_address, l_namespace, l_pod_name, l_container_name, l_workload_name, l_workload_kind,' + || 'l_process_name, l_pid, l_command_type, l_command_status, l_combined_config, any_active, l_profiling_status,' + || 'l_agent_version, l_run_mode, l_heartbeat_timestamp'; +BEGIN + -- Non-blocking guard: if a refresh is already running, skip instead of queuing. + IF NOT pg_try_advisory_lock(3126073) THEN + RAISE NOTICE 'refresh_workload_snapshot: another refresh is running, skipping'; + RETURN; + END IF; + + SELECT active_generation INTO cur_gen FROM workload_snapshot_meta WHERE id = 1; + new_gen := 1 - cur_gen; + sum_tbl := 'workload_scope_summary_' || new_gen; + cnt_tbl := 'workload_tab_counts_' || new_gen; + + EXECUTE format('TRUNCATE %I', sum_tbl); + EXECUTE format('TRUNCATE %I', cnt_tbl); + + -- Tab counts: tiered, each counted at the shallowest grain that exposes it, + -- via COUNT(*) FROM (SELECT DISTINCT ...) (hash-distinct, not sort-per-agg). + EXECUTE format($f$ + INSERT INTO %1$I (scope, count) + WITH fresh AS MATERIALIZED ( + SELECT id, hostname, service_name FROM HostHeartbeats + WHERE heartbeat_timestamp > NOW() - %2$L::interval + ) + SELECT k, v FROM ( + SELECT + (SELECT COUNT(DISTINCT service_name) FROM fresh) AS c_service, + (SELECT COUNT(DISTINCT hostname) FROM fresh) AS active_hosts, + (SELECT COUNT(*) FROM (SELECT DISTINCT service_name, hostname FROM fresh) d) AS c_host, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.service_name, hc.namespace + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + WHERE hc.namespace IS NOT NULL) d) AS c_namespace, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.service_name, hc.namespace, hc.pod_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + WHERE hc.pod_name IS NOT NULL) d) AS c_pod, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.service_name, f.hostname, hc.namespace, hc.pod_name, hc.container_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + WHERE hc.container_name IS NOT NULL) d) AS c_container, + (SELECT COUNT(*) FROM (SELECT DISTINCT f.id, hp.pid + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + WHERE hp.pid IS NOT NULL) d) AS c_process + ) t, + LATERAL (VALUES ('service', c_service), ('host', c_host), ('namespace', c_namespace), + ('pod', c_pod), ('container', c_container), ('process', c_process), + ('active_hosts', active_hosts)) x(k, v) + $f$, cnt_tbl, fresh_interval); + + -- Shared container-grain source for the coarse summaries. + src_cte := format($c$ + fresh AS MATERIALIZED ( + SELECT id, hostname, host(ip_address) AS ip_address, service_name, agent_version, run_mode, heartbeat_timestamp + FROM HostHeartbeats WHERE heartbeat_timestamp > NOW() - %L::interval + ), + cc AS (SELECT hostname, service_name, command_type, status, combined_config FROM ProfilingCommands), + cont AS ( + SELECT f.id, f.hostname, f.ip_address, f.service_name, f.agent_version, f.run_mode, f.heartbeat_timestamp, + hc.namespace, hc.pod_name, hc.container_name, hc.workload_name, hc.workload_kind, + COALESCE(c.command_type, 'N/A') AS command_type, c.status AS command_status, c.combined_config, + (c.command_type = 'start' AND c.status IN ('pending','sent','completed')) AS host_active + FROM fresh f + LEFT JOIN cc c ON c.hostname = f.hostname AND c.service_name = f.service_name + LEFT JOIN HeartbeatContainers hc ON hc.host_id = f.id + ) + $c$, fresh_interval); + + -- Per-group aggregate tail over the container-grain `cont` relation (unqualified cols). + agg_tail := $a$ + COUNT(DISTINCT hostname) AS host_count, + COUNT(DISTINCT namespace) FILTER (WHERE namespace IS NOT NULL) AS namespace_count, + COUNT(DISTINCT pod_name) FILTER (WHERE pod_name IS NOT NULL) AS pod_count, + COUNT(DISTINCT container_name) FILTER (WHERE container_name IS NOT NULL) AS container_count, + bool_or(host_active) AS any_active, + (array_agg(hostname ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_hostname, + (array_agg(ip_address ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_ip_address, + (array_agg(namespace ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_namespace, + (array_agg(pod_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_pod_name, + (array_agg(container_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_container_name, + (array_agg(workload_name ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_name, + (array_agg(workload_kind ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_kind, + (array_agg(command_type ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_type, + (array_agg(command_status ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_status, + (array_agg(combined_config ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_combined_config, + (array_agg(agent_version ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_agent_version, + (array_agg(run_mode ORDER BY heartbeat_timestamp DESC NULLS LAST))[1] AS l_run_mode, + MAX(heartbeat_timestamp) AS l_heartbeat_timestamp + $a$; + + -- service + EXECUTE format($f$ + INSERT INTO %1$I (%2$s) + WITH %3$s, + pc AS (SELECT service_name, COUNT(*) AS process_count FROM ( + SELECT DISTINCT f.service_name, hp.pid, hp.process_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id WHERE hp.pid IS NOT NULL + ) d GROUP BY service_name) + SELECT 'service', row_number() OVER (ORDER BY ca.service_name ASC NULLS LAST), ca.service_name, + ca.service_name, NULL, NULL, NULL, NULL, + ca.host_count, ca.namespace_count, ca.pod_count, ca.container_count, COALESCE(pc.process_count, 0), NULL::integer[], + ca.l_hostname, ca.l_ip_address, ca.l_namespace, ca.l_pod_name, ca.l_container_name, ca.l_workload_name, ca.l_workload_kind, + NULL::text, NULL::integer, ca.l_command_type, ca.l_command_status, ca.l_combined_config, ca.any_active, NULL::text, + ca.l_agent_version, ca.l_run_mode, ca.l_heartbeat_timestamp + FROM (SELECT service_name, %4$s FROM cont GROUP BY service_name) ca + LEFT JOIN pc ON pc.service_name = ca.service_name + $f$, sum_tbl, sum_cols, src_cte, agg_tail); + + -- namespace + -- pc joins use COALESCE(...)= equality, NOT `IS NOT DISTINCT FROM`: the latter + -- is not hashable, so the planner merges on service_name alone and filters the + -- ns/pod match post-join -> quadratic per service (10min+ at pod grain). + EXECUTE format($f$ + INSERT INTO %1$I (%2$s) + WITH %3$s, + pc AS (SELECT service_name, namespace, COUNT(*) AS process_count FROM ( + SELECT DISTINCT f.service_name, hc.namespace, hp.pid, hp.process_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + WHERE hp.pid IS NOT NULL AND hc.namespace IS NOT NULL + ) d GROUP BY service_name, namespace) + SELECT 'namespace', row_number() OVER (ORDER BY ca.service_name ASC NULLS LAST, ca.namespace ASC NULLS LAST), + ca.service_name || '|' || COALESCE(ca.namespace, ''), + ca.service_name, NULL, ca.namespace, NULL, NULL, + ca.host_count, ca.namespace_count, ca.pod_count, ca.container_count, COALESCE(pc.process_count, 0), NULL::integer[], + ca.l_hostname, ca.l_ip_address, ca.l_namespace, ca.l_pod_name, ca.l_container_name, ca.l_workload_name, ca.l_workload_kind, + NULL::text, NULL::integer, ca.l_command_type, ca.l_command_status, ca.l_combined_config, ca.any_active, NULL::text, + ca.l_agent_version, ca.l_run_mode, ca.l_heartbeat_timestamp + FROM (SELECT service_name, namespace, %4$s FROM cont WHERE namespace IS NOT NULL GROUP BY service_name, namespace) ca + LEFT JOIN pc ON pc.service_name = ca.service_name AND COALESCE(pc.namespace, '') = COALESCE(ca.namespace, '') + $f$, sum_tbl, sum_cols, src_cte, agg_tail); + + -- pod + EXECUTE format($f$ + INSERT INTO %1$I (%2$s) + WITH %3$s, + pc AS (SELECT service_name, namespace, pod_name, COUNT(*) AS process_count FROM ( + SELECT DISTINCT f.service_name, hc.namespace, hc.pod_name, hp.pid, hp.process_name + FROM fresh f JOIN HeartbeatContainers hc ON hc.host_id = f.id + JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id + WHERE hp.pid IS NOT NULL AND hc.pod_name IS NOT NULL + ) d GROUP BY service_name, namespace, pod_name) + SELECT 'pod', row_number() OVER (ORDER BY ca.service_name ASC NULLS LAST, ca.namespace ASC NULLS LAST, ca.pod_name ASC NULLS LAST), + ca.service_name || '|' || COALESCE(ca.namespace, '') || '|' || COALESCE(ca.pod_name, ''), + ca.service_name, NULL, ca.namespace, ca.pod_name, NULL, + ca.host_count, ca.namespace_count, ca.pod_count, ca.container_count, COALESCE(pc.process_count, 0), NULL::integer[], + ca.l_hostname, ca.l_ip_address, ca.l_namespace, ca.l_pod_name, ca.l_container_name, ca.l_workload_name, ca.l_workload_kind, + NULL::text, NULL::integer, ca.l_command_type, ca.l_command_status, ca.l_combined_config, ca.any_active, NULL::text, + ca.l_agent_version, ca.l_run_mode, ca.l_heartbeat_timestamp + FROM (SELECT service_name, namespace, pod_name, %4$s FROM cont WHERE pod_name IS NOT NULL GROUP BY service_name, namespace, pod_name) ca + LEFT JOIN pc ON pc.service_name = ca.service_name + AND COALESCE(pc.namespace, '') = COALESCE(ca.namespace, '') + AND pc.pod_name = ca.pod_name + $f$, sum_tbl, sum_cols, src_cte, agg_tail); + + EXECUTE format('ANALYZE %I', sum_tbl); + + -- Atomic swap of the read-side views + flip the pointer, in this transaction. + EXECUTE format('CREATE OR REPLACE VIEW workload_scope_summary AS SELECT * FROM %I', sum_tbl); + EXECUTE format('CREATE OR REPLACE VIEW workload_tab_counts AS SELECT * FROM %I', cnt_tbl); + + UPDATE workload_snapshot_meta + SET active_generation = new_gen, + built_at = clock_timestamp(), + build_duration_ms = (EXTRACT(EPOCH FROM (clock_timestamp() - t0)) * 1000)::integer + WHERE id = 1; + + PERFORM pg_advisory_unlock(3126073); +END; +$procedure$; diff --git a/src/gprofiler-dev/gprofiler_dev/config.py b/src/gprofiler-dev/gprofiler_dev/config.py index e84eb8f7..cada4004 100644 --- a/src/gprofiler-dev/gprofiler_dev/config.py +++ b/src/gprofiler-dev/gprofiler_dev/config.py @@ -38,6 +38,10 @@ BUCKET_NAME = os.getenv("BUCKET_NAME", "gprofiler") BASE_DIRECTORY = "products" +# Optional prefix prepended to every S3 key; empty = no prefix (default) +S3_PATH_PREFIX = os.getenv("S3_PATH_PREFIX", "").strip("/") # Optional: Custom S3 endpoint for local testing (e.g., LocalStack) or S3-compatible services # In production, leave unset to use default AWS S3 endpoints S3_ENDPOINT_URL = os.getenv("S3_ENDPOINT_URL") + +ACTIVE_HOST_HEARTBEAT_MAX_DELTA_HOURS = int(os.getenv("ACTIVE_HOST_HEARTBEAT_MAX_DELTA_HOURS", 24)) diff --git a/src/gprofiler-dev/gprofiler_dev/perf_utils.py b/src/gprofiler-dev/gprofiler_dev/perf_utils.py new file mode 100644 index 00000000..bbda39ea --- /dev/null +++ b/src/gprofiler-dev/gprofiler_dev/perf_utils.py @@ -0,0 +1,60 @@ +# +# Copyright (C) 2023 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +""" +Utility functions for PMU (Performance Monitoring Unit) event handling. +""" + +# Perf event name normalization mapping +# Maps UI event names (e.g., 'cpu-cycles') to agent-normalized names (e.g., 'cycles') +PERF_EVENT_NORMALIZATION_MAP = { + "cpu-cycles": "cycles", + "cpu-instructions": "instructions", + "cpu-cache-misses": "cache-misses", + "cpu-cache-references": "cache-references", + "cpu-branch-instructions": "branch-instructions", + "cpu-branch-misses": "branch-misses", + "cpu-stalled-cycles-frontend": "stalled-cycles-frontend", + "cpu-stalled-cycles-backend": "stalled-cycles-backend", +} + + +def normalize_perf_event_name(event: str) -> str: + """ + Normalize perf event names to match what the agent stores. + + The agent normalizes events like 'cpu-cycles' -> 'cycles', + 'cpu/cache-misses/' -> 'cache-misses' + + Args: + event: Event name from UI or user input (e.g., 'cpu-cycles') + + Returns: + Normalized event name (e.g., 'cycles') + + Example: + >>> normalize_perf_event_name('cpu-cycles') + 'cycles' + >>> normalize_perf_event_name('cpu/cache-misses/') + 'cache-misses' + >>> normalize_perf_event_name('cycles') + 'cycles' + """ + # Remove cpu/ prefix and trailing slash if present + event = event.replace("cpu/", "").replace("/", "") + + # Apply normalization mapping + return PERF_EVENT_NORMALIZATION_MAP.get(event, event) diff --git a/src/gprofiler-dev/gprofiler_dev/postgres/db_manager.py b/src/gprofiler-dev/gprofiler_dev/postgres/db_manager.py index 2ecc466b..89b39f37 100644 --- a/src/gprofiler-dev/gprofiler_dev/postgres/db_manager.py +++ b/src/gprofiler-dev/gprofiler_dev/postgres/db_manager.py @@ -18,17 +18,24 @@ import json import threading import time +import uuid +import psycopg2.extras from collections import defaultdict from datetime import datetime, timedelta from secrets import token_urlsafe from typing import Any, Dict, List, Optional, Set, Tuple, Union -from gprofiler_dev.config import INSTANCE_RUNS_LRU_CACHE_LIMIT, PROFILER_PROCESSES_LRU_CACHE_LIMIT +from gprofiler_dev.config import ( + ACTIVE_HOST_HEARTBEAT_MAX_DELTA_HOURS, + INSTANCE_RUNS_LRU_CACHE_LIMIT, + PROFILER_PROCESSES_LRU_CACHE_LIMIT, +) from gprofiler_dev.lru_cache_impl import LRUCache from gprofiler_dev.postgres import get_postgres_db from gprofiler_dev.postgres.postgresdb import DBConflict from gprofiler_dev.postgres.queries import AggregationSQLQueries, SQLQueries from gprofiler_dev.postgres.schemas import AgentMetadata, CloudProvider, GetServiceResponse +from gprofiler_dev.perf_utils import normalize_perf_event_name AGENT_RETENTION_HOURS = 24 LAST_SEEN_UPDATES_INTERVAL_MINUTES = 5 @@ -88,7 +95,6 @@ def __call__(cls, *args, **kwargs): class DBManager(metaclass=Singleton): def __init__(self): - self.db = get_postgres_db() self.machine_types: Dict[Tuple[str, str], int] = {} self.machine_types_ids: Dict[int, int] = {} self.profiler_versions: Dict[Tuple[int, int, int], int] = {} @@ -110,6 +116,14 @@ def __init__(self): lambda: time.time() - (LAST_SEEN_UPDATES_INTERVAL_MINUTES + 1) * 60 ) + @property + def db(self): + # Resolve per access: this Singleton is process-wide, so caching one + # connection would re-serialize every thread on its lock. get_postgres_db() + # hands each thread its own connection when GPROFILER_POSTGRES_CONN_PER_THREAD + # is set (and the shared instance otherwise). + return get_postgres_db() + def get_libc(self, libc_type, libc_version): key = (libc_type, libc_version) if key not in self.libcs: @@ -603,11 +617,16 @@ def save_profiling_request( target_hostnames: Optional[List[str]] = None, pids: Optional[List[int]] = None, host_pid_mapping: Optional[Dict[str, List[int]]] = None, + target_scope: str = "host", + target_entities: Optional[List[Dict[str, Any]]] = None, additional_args: Optional[Dict] = None, ) -> bool: """Save a profiling request with support for host-to-PID mapping""" # Store additional_args WITHOUT host_pid_mapping (keep that separate) clean_additional_args = additional_args.copy() if additional_args else {} + clean_additional_args["target_scope"] = target_scope + if target_entities: + clean_additional_args["target_entities"] = target_entities # Store host_pid_mapping separately in a dedicated field if we add one, # for now, we'll handle it during command creation to avoid polluting additional_args @@ -850,11 +869,17 @@ def upsert_host_heartbeat( hostname: str, ip_address: str, service_name: str, + agent_version: Optional[str] = None, + run_mode: Optional[str] = None, + namespace: Optional[str] = None, + pod_name: Optional[str] = None, + containers: Optional[List[Dict[str, Any]]] = None, last_command_id: Optional[str] = None, received_command_ids: Optional[List[str]] = None, executed_command_ids: Optional[List[str]] = None, status: str = "active", heartbeat_timestamp: Optional[datetime] = None, + supported_perf_events: Optional[List[str]] = None, ) -> bool: """ Update or insert host heartbeat information using pure SQL. @@ -864,48 +889,255 @@ def upsert_host_heartbeat( if heartbeat_timestamp is None: heartbeat_timestamp = datetime.now() + # Container/process inventory lives entirely in the normalized + # HeartbeatContainers/HeartbeatProcesses tables (synced below). ``RETURNING ID`` + # gives us the stable host row id (the upsert key is (hostname, service_name)) + # used to attach those child rows. query = """ INSERT INTO HostHeartbeats ( - hostname, ip_address, service_name, last_command_id, + hostname, ip_address, service_name, agent_version, run_mode, namespace, pod_name, last_command_id, received_command_ids, executed_command_ids, - status, heartbeat_timestamp, created_at, updated_at + status, heartbeat_timestamp, supported_perf_events, created_at, updated_at ) VALUES ( - %(hostname)s, %(ip_address)s::inet, %(service_name)s, + %(hostname)s, %(ip_address)s::inet, %(service_name)s, %(agent_version)s, %(run_mode)s, + %(namespace)s, %(pod_name)s, %(last_command_id)s::uuid, %(received_command_ids)s::uuid[], %(executed_command_ids)s::uuid[], %(status)s::HostStatus, - %(heartbeat_timestamp)s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP + %(heartbeat_timestamp)s, %(supported_perf_events)s::text[], CURRENT_TIMESTAMP, CURRENT_TIMESTAMP ) ON CONFLICT (hostname, service_name) DO UPDATE SET ip_address = EXCLUDED.ip_address, + agent_version = EXCLUDED.agent_version, + run_mode = EXCLUDED.run_mode, + namespace = EXCLUDED.namespace, + pod_name = EXCLUDED.pod_name, last_command_id = EXCLUDED.last_command_id, received_command_ids = EXCLUDED.received_command_ids, executed_command_ids = EXCLUDED.executed_command_ids, status = EXCLUDED.status, heartbeat_timestamp = EXCLUDED.heartbeat_timestamp, + supported_perf_events = EXCLUDED.supported_perf_events, updated_at = CURRENT_TIMESTAMP + RETURNING ID """ values = { "hostname": hostname, "ip_address": ip_address, "service_name": service_name, + "agent_version": agent_version, + "run_mode": run_mode, + "namespace": namespace, + "pod_name": pod_name, "last_command_id": last_command_id, "received_command_ids": received_command_ids, "executed_command_ids": executed_command_ids, "status": status, "heartbeat_timestamp": heartbeat_timestamp, + "supported_perf_events": supported_perf_events, } - self.db.execute(query, values, has_value=False) + # Host upsert and the normalized inventory sync share one transaction so a + # reader never observes a host whose structured inventory is half-written. + with self.db.transaction() as cursor: + cursor.execute(query, values) + row = cursor.fetchone() + host_id = row[0] if row else None + if host_id is not None: + self._sync_host_inventory(cursor, host_id, containers or []) return True + def _sync_host_inventory(self, cursor, host_id: int, containers: List[Dict[str, Any]]) -> None: + """Diff the normalized inventory for a host against the latest heartbeat snapshot. + + Runs inside the caller's transaction. Rather than deleting and re-inserting every + row on each heartbeat, it upserts containers/processes on their natural identity + and rewrites a row only when a value actually changed. At fleet scale most beats + report an unchanged inventory, so the guarded upserts produce zero row writes + (no dead tuples, WAL, or index churn) in the steady state. + + Only containerized workloads are stored: the agent sends an empty list for + non-containerized hosts and never emits a usable NULL container_id, so entries + without a container_id are skipped — they also can't be diffed via the + UNIQUE (host_id, container_id) key. Any legacy NULL-id rows are cleaned on the + next heartbeat by the prune below. + """ + seen_container_ids: Set[str] = set() + normalized: List[Dict[str, Any]] = [] + for container in containers: + if not isinstance(container, dict): + continue + container_id = container.get("container_id") + if container_id is None or container_id in seen_container_ids: + continue + seen_container_ids.add(container_id) + normalized.append(container) + + incoming_ids = [c["container_id"] for c in normalized] + + # Drop containers the host no longer reports (cascades to their processes). The + # IS NULL clause also reclaims any legacy NULL-id rows left by the old writer. + cursor.execute( + "DELETE FROM HeartbeatContainers " + "WHERE host_id = %s AND (container_id IS NULL OR container_id <> ALL(%s::text[]))", + (host_id, incoming_ids), + ) + + if not normalized: + return + + # Container tuples, reused by the insert-new and update-changed passes below. + container_rows = [ + ( + host_id, + container["container_id"], + container.get("container_name"), + container.get("runtime"), + container.get("namespace"), + container.get("pod_name"), + container.get("workload_name"), + container.get("workload_kind"), + ) + for container in normalized + ] + + # Insert only genuinely-new containers. A blanket INSERT ... ON CONFLICT would + # evaluate the id DEFAULT (nextval) for every proposed row before resolving the + # conflict, burning a sequence value per unchanged container re-reported each + # beat. Filtering to non-existing rows means nextval fires only for real inserts; + # ON CONFLICT DO NOTHING still guards the rare concurrent-insert race. + psycopg2.extras.execute_values( + cursor, + """ + INSERT INTO HeartbeatContainers ( + host_id, container_id, container_name, runtime, namespace, + pod_name, workload_name, workload_kind, updated_at + ) + SELECT v.host_id, v.container_id, v.container_name, v.runtime, v.namespace, + v.pod_name, v.workload_name, v.workload_kind, CURRENT_TIMESTAMP + FROM (VALUES %s) AS v(host_id, container_id, container_name, runtime, + namespace, pod_name, workload_name, workload_kind) + WHERE NOT EXISTS ( + SELECT 1 FROM HeartbeatContainers hc + WHERE hc.host_id = v.host_id AND hc.container_id = v.container_id + ) + ON CONFLICT (host_id, container_id) DO NOTHING + """, + container_rows, + template="(%s::bigint, %s::text, %s::text, %s::text, %s::text, %s::text, %s::text, %s::text)", + ) + + # Rewrite metadata only for containers that actually changed. + psycopg2.extras.execute_values( + cursor, + """ + UPDATE HeartbeatContainers hc SET + container_name = v.container_name, + runtime = v.runtime, + namespace = v.namespace, + pod_name = v.pod_name, + workload_name = v.workload_name, + workload_kind = v.workload_kind, + updated_at = CURRENT_TIMESTAMP + FROM (VALUES %s) AS v(host_id, container_id, container_name, runtime, + namespace, pod_name, workload_name, workload_kind) + WHERE hc.host_id = v.host_id AND hc.container_id = v.container_id + AND ( + hc.container_name, hc.runtime, hc.namespace, hc.pod_name, + hc.workload_name, hc.workload_kind + ) IS DISTINCT FROM ( + v.container_name, v.runtime, v.namespace, v.pod_name, + v.workload_name, v.workload_kind + ) + """, + container_rows, + template="(%s::bigint, %s::text, %s::text, %s::text, %s::text, %s::text, %s::text, %s::text)", + ) + + # Map every reported container_id to its stable row id (changed or not) so + # processes can be attached — the guarded upsert above returns nothing for + # unchanged rows, so we can't rely on RETURNING here. + cursor.execute( + "SELECT id, container_id FROM HeartbeatContainers " + "WHERE host_id = %s AND container_id = ANY(%s::text[])", + (host_id, incoming_ids), + ) + row_id_by_container_id = {container_id: row_id for row_id, container_id in cursor.fetchall()} + + for container in normalized: + container_row_id = row_id_by_container_id.get(container["container_id"]) + if container_row_id is None: + continue + self._sync_container_processes(cursor, container_row_id, container.get("processes") or []) + + def _sync_container_processes(self, cursor, container_row_id: int, processes: List[Dict[str, Any]]) -> None: + """Diff a single container's processes, mirroring the container-level upsert.""" + process_rows = [] + seen_pids: Set[int] = set() + for process in processes: + if not isinstance(process, dict): + continue + pid = process.get("pid") + if pid is None or not str(pid).isdigit(): + continue + pid = int(pid) + if pid in seen_pids: # satisfy UNIQUE (container_row_id, pid) + continue + seen_pids.add(pid) + process_rows.append((container_row_id, pid, process.get("process_name"))) + + # Drop processes the container no longer reports. + cursor.execute( + "DELETE FROM HeartbeatProcesses WHERE container_row_id = %s AND pid <> ALL(%s::int[])", + (container_row_id, [pid for _, pid, _ in process_rows]), + ) + + if not process_rows: + return + + # Insert only genuinely-new (container_row_id, pid) rows. A blanket + # INSERT ... ON CONFLICT evaluates the id DEFAULT (nextval) for every proposed + # row before resolving the conflict, so the ~all-unchanged processes re-reported + # each beat burned a sequence value each (~58k nextval/s fleet-wide -> sequence + # buffer lock contention). Filtering to non-existing rows means nextval fires + # only for real inserts; ON CONFLICT DO NOTHING guards the concurrent-insert race. + psycopg2.extras.execute_values( + cursor, + """ + INSERT INTO HeartbeatProcesses (container_row_id, pid, process_name) + SELECT v.container_row_id, v.pid, v.process_name + FROM (VALUES %s) AS v(container_row_id, pid, process_name) + WHERE NOT EXISTS ( + SELECT 1 FROM HeartbeatProcesses hp + WHERE hp.container_row_id = v.container_row_id AND hp.pid = v.pid + ) + ON CONFLICT (container_row_id, pid) DO NOTHING + """, + process_rows, + template="(%s::bigint, %s::int, %s::text)", + ) + + # Rewrite names only for rows whose process_name actually changed. + psycopg2.extras.execute_values( + cursor, + """ + UPDATE HeartbeatProcesses hp SET process_name = v.process_name + FROM (VALUES %s) AS v(container_row_id, pid, process_name) + WHERE hp.container_row_id = v.container_row_id AND hp.pid = v.pid + AND hp.process_name IS DISTINCT FROM v.process_name + """, + process_rows, + template="(%s::bigint, %s::int, %s::text)", + ) + def get_host_heartbeat(self, hostname: str) -> Optional[Dict]: """Get the latest heartbeat information for a host""" query = """ SELECT hostname, ip_address, service_name, last_command_id, + received_command_ids, executed_command_ids, status, heartbeat_timestamp, created_at, updated_at FROM HostHeartbeats WHERE hostname = %(hostname)s @@ -920,6 +1152,7 @@ def get_active_hosts(self, service_name: Optional[str] = None) -> List[Dict]: query = """ SELECT hostname, ip_address, service_name, last_command_id, + received_command_ids, executed_command_ids, status, heartbeat_timestamp FROM HostHeartbeats WHERE status = 'active' @@ -935,11 +1168,187 @@ def get_active_hosts(self, service_name: Optional[str] = None) -> List[Dict]: return self.db.execute(query, values, one_value=False, return_dict=True, fetch_all=True) + def get_active_hosts_count(self, service_name: Optional[str] = None, max_delta_hours: int = ACTIVE_HOST_HEARTBEAT_MAX_DELTA_HOURS) -> int: + """ + Get the count of active hosts based on the provided time window. + A host is considered active if its last heartbeat is within the specified time window. + + Args: + service_name: Optional service name to filter hosts by + max_delta_hours: Maximum hours since last heartbeat to consider a host active (default: ACTIVE_HOST_HEARTBEAT_MAX_DELTA_HOURS) + + Returns: + int: Count of active hosts + """ + query = """ + SELECT COUNT(*) as active_hosts_count + FROM HostHeartbeats + WHERE heartbeat_timestamp >= NOW() - INTERVAL '%(max_delta_hours)s hours' + """ + + values = {"max_delta_hours": max_delta_hours} + + if service_name: + query += " AND service_name = %(service_name)s" + values["service_name"] = service_name + + result = self.db.execute(query, values, one_value=True, return_dict=True) + return result.get("active_hosts_count", 0) if result else 0 + + def validate_perf_events_support( + self, + service_name: str, + requested_events: List[str], + target_hostnames: Optional[List[str]] = None + ) -> Dict[str, Any]: + """ + Validate if the requested perf events are supported by target hosts. + + Args: + service_name: Service name to check + requested_events: List of requested perf events (e.g., ['cpu-cycles', 'cache-misses']) + target_hostnames: Optional list of specific hostnames to check (None = all hosts for service) + + Returns: + Dict with validation results: + { + "valid": bool, + "unsupported_hosts": List[Dict], # Hosts that don't support some events + "error_message": Optional[str] + } + """ + # Normalize requested events to match agent's format + normalized_requested_events = [normalize_perf_event_name(event) for event in requested_events] + + # Build query to get hosts and their supported events + query = """ + SELECT + hostname, + supported_perf_events + FROM HostHeartbeats + WHERE service_name = %(service_name)s + AND heartbeat_timestamp >= NOW() - INTERVAL '2 minutes' + """ + + values = {"service_name": service_name} + + # Filter by specific hostnames if provided + if target_hostnames: + query += " AND hostname = ANY(%(target_hostnames)s)" + values["target_hostnames"] = target_hostnames + + hosts = self.db.execute(query, values, one_value=False, return_dict=True, fetch_all=True) + + if not hosts: + return { + "valid": False, + "unsupported_hosts": [], + "error_message": f"No active hosts found for service '{service_name}'" + } + + unsupported_hosts = [] + + for host in hosts: + hostname = host.get("hostname") + supported_events = host.get("supported_perf_events") or [] + + # If host hasn't sent supported events yet, assume compatibility issue + if not supported_events: + unsupported_hosts.append({ + "hostname": hostname, + "missing_events": requested_events, # Use original names in error message + "reason": "Host has not reported supported PMU events yet" + }) + continue + + # Check which requested events are NOT supported (using normalized names) + missing_events = [] + for i, normalized_event in enumerate(normalized_requested_events): + if normalized_event not in supported_events: + missing_events.append(requested_events[i]) # Use original name in error message + + if missing_events: + unsupported_hosts.append({ + "hostname": hostname, + "missing_events": missing_events, + "supported_events": supported_events + }) + + # Build error message if there are unsupported hosts + if unsupported_hosts: + total_unsupported = len(unsupported_hosts) + host_details = [] + + # Show up to 10 hosts so users know exactly which hosts have issues + max_hosts_to_show = 10 + for host_info in unsupported_hosts[:max_hosts_to_show]: + hostname = host_info["hostname"] + missing = ", ".join(host_info["missing_events"]) + host_details.append(f" - {hostname}: missing {missing}") + + more_hosts = total_unsupported - max_hosts_to_show + if more_hosts > 0: + host_details.append(f" ...and {more_hosts} more host(s)") + + # Include summary for better context + summary = f"{total_unsupported} host(s) don't support the selected events:" + error_message = summary + "\n" + "\n".join(host_details) + + return { + "valid": False, + "unsupported_hosts": unsupported_hosts, + "error_message": error_message + } + + return { + "valid": True, + "unsupported_hosts": [], + "error_message": None + } + + def get_actively_profiling_hosts_count(self, service_name: Optional[str] = None, host_exclusion_list: Optional[List[str]] = None, host_inclusion_list: Optional[List[str]] = None) -> int: + """ + Get the count of hosts that are actively profiling. + A host is considered actively profiling if it has a completed start command. + + Args: + service_name: Optional service name to filter hosts by + host_exclusion_list: Optional list of hostnames to exclude from the count + host_inclusion_list: Optional list of hostnames to include in the count (only these hosts will be counted) + + Returns: + int: Count of actively profiling hosts + """ + query = """ + SELECT COUNT(DISTINCT hostname) as profiling_hosts_count + FROM ProfilingCommands + WHERE command_type = 'start' + AND status = 'completed' + """ + + values = {} + + if service_name: + query += " AND service_name = %(service_name)s" + values["service_name"] = service_name + + if host_inclusion_list: + query += " AND hostname IN %(host_inclusion_list)s" + values["host_inclusion_list"] = tuple(host_inclusion_list) + + if host_exclusion_list: + query += " AND hostname NOT IN %(host_exclusion_list)s" + values["host_exclusion_list"] = tuple(host_exclusion_list) + + result = self.db.execute(query, values, one_value=True, return_dict=True) + return result.get("profiling_hosts_count", 0) if result else 0 + def get_all_host_heartbeats(self, limit: Optional[int] = None, offset: Optional[int] = None) -> List[Dict]: """Get all host heartbeat records with optional pagination""" query = """ SELECT ID, hostname, ip_address, service_name, last_command_id, + received_command_ids, executed_command_ids, status, heartbeat_timestamp, created_at, updated_at FROM HostHeartbeats ORDER BY heartbeat_timestamp DESC @@ -955,9 +1364,7 @@ def get_all_host_heartbeats(self, limit: Optional[int] = None, offset: Optional[ return self.db.execute(query, values, one_value=False, return_dict=True, fetch_all=True) - def get_host_heartbeats_by_service( - self, service_name: str, limit: Optional[int] = None, exact_match: bool = False - ) -> List[Dict]: + def get_host_heartbeats_by_service(self, service_name: str, limit: Optional[int] = None, exact_match: bool = False) -> List[Dict]: """Get all host heartbeat records for a specific service with optional partial matching""" if exact_match: # Use exact match for backward compatibility @@ -971,6 +1378,7 @@ def get_host_heartbeats_by_service( query = f""" SELECT ID, hostname, ip_address, service_name, last_command_id, + received_command_ids, executed_command_ids, status, heartbeat_timestamp, created_at, updated_at FROM HostHeartbeats {where_clause} @@ -989,6 +1397,7 @@ def get_host_heartbeats_by_status(self, status: str, limit: Optional[int] = None query = """ SELECT ID, hostname, ip_address, service_name, last_command_id, + received_command_ids, executed_command_ids, status, heartbeat_timestamp, created_at, updated_at FROM HostHeartbeats WHERE status = %(status)s @@ -1076,7 +1485,7 @@ def create_or_update_profiling_command( "frequency": request_result["frequency"], "profiling_mode": request_result["profiling_mode"], } - + # Merge additional_args directly into new_config if request_result["additional_args"]: additional_args = request_result["additional_args"] @@ -1169,6 +1578,78 @@ def create_or_update_profiling_command( self.db.execute(upsert_query, values, has_value=False) return True + def get_active_service_subscription(self, service_name: str) -> Optional[str]: + """Return the request_id of the active service-wide profiling subscription + for a service, or None. + + A service is "actively subscribed" when its most recent service-scoped + continuous "start" request is newer than any service-scoped "stop" request + (and was not cancelled). This is what lets hosts that register *after* a + service-wide profiling request auto-join the in-progress profile. + """ + start_query = """ + SELECT request_id, created_at + FROM ProfilingRequests + WHERE service_name = %(service_name)s + AND request_type = 'start' + AND continuous = TRUE + AND COALESCE(additional_args->>'target_scope', 'host') = 'service' + AND status != 'cancelled' + ORDER BY created_at DESC + LIMIT 1 + """ + start_row = self.db.execute( + start_query, {"service_name": service_name}, one_value=True, return_dict=True + ) + if not start_row: + return None + + stop_query = """ + SELECT created_at + FROM ProfilingRequests + WHERE service_name = %(service_name)s + AND request_type = 'stop' + AND COALESCE(additional_args->>'target_scope', 'host') = 'service' + ORDER BY created_at DESC + LIMIT 1 + """ + stop_row = self.db.execute( + stop_query, {"service_name": service_name}, one_value=True, return_dict=True + ) + if stop_row and stop_row["created_at"] >= start_row["created_at"]: + return None + + return str(start_row["request_id"]) + + def auto_subscribe_host_to_service(self, hostname: str, service_name: str) -> bool: + """Enroll a host into its service's active service-wide profiling. + + Invoked on every heartbeat. When a service has an active service-wide + profiling subscription and the reporting host has no current command + (e.g. a node that was just added to the cluster by autoscaling), a start + command is created for it from the subscription's configuration. Hosts + that already have command state are left untouched so explicit per-host + actions (including stops) are preserved. + + Returns True if a new subscription command was created for the host. + """ + subscription_request_id = self.get_active_service_subscription(service_name) + if not subscription_request_id: + return False + + current_command = self.get_current_profiling_command(hostname, service_name) + if current_command is not None: + return False + + command_id = str(uuid.uuid4()) + return self.create_or_update_profiling_command( + command_id=command_id, + hostname=hostname, + service_name=service_name, + command_type="start", + new_request_id=subscription_request_id, + ) + def _merge_profiling_configs(self, existing_config: Dict, new_config: Dict) -> Dict: """Merge two profiling configurations, combining parameters appropriately""" # Handle case where existing_config might be None or empty @@ -1354,15 +1835,1005 @@ def get_current_profiling_command(self, hostname: str, service_name: str) -> Opt result = self.db.execute(query, values, one_value=True, return_dict=True) return result if result else None - def get_profiling_host_status_optimized( + @staticmethod + def _parse_json_field(value: Any, default: Any) -> Any: + if value is None: + return default + if isinstance(value, str): + try: + return json.loads(value) + except json.JSONDecodeError: + return default + return value + + @staticmethod + def _normalize_profiling_status(command_type: Optional[str], command_status: Optional[str]) -> str: + if command_status is None: + return "stopped" + if command_type == "start" and command_status in ["pending", "sent", "completed"]: + return "active" + return command_status + + @staticmethod + def _summarize_enabled_profilers(combined_config: Dict[str, Any]) -> str: + profiler_configs = combined_config.get("profiler_configs", {}) or {} + if not profiler_configs: + return "Default" + + profiler_labels = { + "perf": "Perf", + "async_profiler": "Java", + "pyperf": "Pyperf", + "pyspy": "Pyspy", + "rbspy": "Rbspy", + "phpspy": "PHPspy", + "dotnet_trace": ".NET", + "nodejs_perf": "NodeJS", + } + enabled = [] + for key, label in profiler_labels.items(): + value = profiler_configs.get(key) + if value is None: + continue + if isinstance(value, dict): + if value.get("enabled") is False or value.get("mode") == "disabled": + continue + elif value == "disabled": + continue + enabled.append(label) + return ", ".join(enabled) if enabled else "Disabled" + + def _extract_command_metadata(self, combined_config: Any, command_type: Optional[str], command_status: Optional[str]) -> Dict[str, Any]: + config = self._parse_json_field(combined_config, {}) or {} + current_pids = [] + for pid in config.get("pids", []) or []: + if str(pid).isdigit(): + current_pids.append(int(pid)) + + return { + "combined_config": config, + "pids": current_pids, + "frequency": config.get("frequency"), + "profiling_mode": "Continuous" if config.get("continuous") else "Ad Hoc", + "profiler_summary": self._summarize_enabled_profilers(config), + "command_type": command_type or "N/A", + "profiling_status": self._normalize_profiling_status(command_type, command_status), + } + + # Group keys per scope; the first element is always the grouping granularity and is + # also used to build the stable row "id". Scopes that key on an optional column skip + # records where that column is NULL (matching the previous Python behavior). + _WORKLOAD_SCOPE_KEYS: Dict[str, List[str]] = { + "service": ["service_name"], + "namespace": ["service_name", "namespace"], + "host": ["service_name", "hostname"], + "pod": ["service_name", "namespace", "pod_name"], + "container": ["service_name", "hostname", "namespace", "pod_name", "container_name"], + "process": ["service_name", "hostname", "pid"], + } + _WORKLOAD_SCOPE_NULL_GUARD: Dict[str, str] = { + "namespace": "namespace IS NOT NULL", + "pod": "pod_name IS NOT NULL", + "container": "container_name IS NOT NULL", + "process": "pid IS NOT NULL", + } + # Join depth at which each logical column becomes available in the flatten base: + # 0 = HostHeartbeats (+ its latest command), 1 = HeartbeatContainers, 2 = HeartbeatProcesses. + _WORKLOAD_COLUMN_DEPTH: Dict[str, int] = { + "hostname": 0, + "ip_address": 0, + "service_name": 0, + "agent_version": 0, + "run_mode": 0, + "heartbeat_timestamp": 0, + "command_type": 0, + "namespace": 1, + "pod_name": 1, + "container_name": 1, + "workload_name": 1, + "workload_kind": 1, + "process_name": 2, + "pid": 2, + "profiling_status": 2, + } + # Minimum base join depth needed to enumerate the distinct entities for each scope. + _WORKLOAD_SCOPE_DEPTH: Dict[str, int] = { + "service": 0, + "host": 0, + "namespace": 1, + "pod": 1, + "container": 1, + "process": 2, + } + # Key column -> base-table qualified expression, used to push the per-entity + # correlation down onto the indexed base tables in the hydrate LATERAL. + _WORKLOAD_KEY_ALIAS: Dict[str, str] = { + "service_name": "fh.service_name", + "hostname": "fh.hostname", + "namespace": "hc.namespace", + "pod_name": "hc.pod_name", + "container_name": "hc.container_name", + "pid": "hp.pid", + } + # Whitelist of sortable columns that are NOT scope key columns -> (aggregate expression + # over the entity's rows on the wrapped alias ``b``, required base depth). Scope key + # columns are sorted directly on the key set and are handled separately. Client input is + # matched against these keys only, so no caller string reaches the SQL. + _WORKLOAD_SORT_AGG: Dict[str, Tuple[str, int]] = { + "hostname": ("(array_agg(b.hostname ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 0), + "ip_address": ("(array_agg(b.ip_address ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 0), + "namespace": ("(array_agg(b.namespace ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 1), + "pod_name": ("(array_agg(b.pod_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 1), + "container_name": ("(array_agg(b.container_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 1), + "process_name": ("(array_agg(b.process_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 2), + "pid": ("(array_agg(b.pid ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 2), + "heartbeat_timestamp": ("MAX(b.heartbeat_timestamp)", 0), + "profiling_status": ("(array_agg(b.profiling_status ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 2), + "agent_version": ("(array_agg(b.agent_version ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1]", 0), + "host_count": ("COUNT(DISTINCT b.hostname)", 0), + "namespace_count": ("COUNT(DISTINCT b.namespace) FILTER (WHERE b.namespace IS NOT NULL)", 1), + "pod_count": ("COUNT(DISTINCT b.pod_name) FILTER (WHERE b.pod_name IS NOT NULL)", 1), + "container_count": ("COUNT(DISTINCT b.container_name) FILTER (WHERE b.container_name IS NOT NULL)", 1), + "process_count": ("COUNT(DISTINCT (b.pid, b.process_name)) FILTER (WHERE b.pid IS NOT NULL)", 2), + } + # Sortable column -> output column/alias produced by the grouped aggregation, used by + # the single-pass (whole-scope GROUP BY) query path for its ORDER BY. + _WORKLOAD_SORT_ALIAS: Dict[str, str] = { + "hostname": "l_hostname", + "ip_address": "l_ip_address", + "namespace": "l_namespace", + "pod_name": "l_pod_name", + "container_name": "l_container_name", + "process_name": "l_process_name", + "pid": "l_pid", + "heartbeat_timestamp": "l_heartbeat_timestamp", + "profiling_status": "l_profiling_status", + "agent_version": "l_agent_version", + "host_count": "host_count", + "namespace_count": "namespace_count", + "pod_count": "pod_count", + "container_count": "container_count", + "process_count": "process_count", + } + + # Row-level PID-aware profiling status. Only valid at depth >= 2 (needs ``hp.pid``); + # references the latest command columns (``c.*``) exposed by ``current_commands``. + _WORKLOAD_PROFILING_STATUS_CASE = """CASE + WHEN c.status IS NULL THEN 'stopped' + WHEN c.command_type = 'start' AND c.status IN ('pending', 'sent', 'completed') THEN + -- PID-aware: an active "start" command may target only a subset of + -- PIDs on the host (e.g. a single container/pod). A row is only + -- "active" when the command targets all PIDs on the host (no/empty + -- "pids" list == whole host) OR this row's PID is in the target set. + CASE + WHEN c.combined_config IS NULL + OR (c.combined_config -> 'pids') IS NULL + OR jsonb_typeof(c.combined_config -> 'pids') <> 'array' + OR jsonb_array_length(c.combined_config -> 'pids') = 0 + THEN 'active' + WHEN hp.pid IS NOT NULL AND EXISTS ( + SELECT 1 + FROM jsonb_array_elements_text(c.combined_config -> 'pids') AS target(pid) + WHERE target.pid = hp.pid::text + ) THEN 'active' + ELSE 'stopped' + END + ELSE c.status::text + END AS profiling_status""" + + def _workload_filter_spec( + self, + service_names: Optional[List[str]], + exact_match: bool, + hostnames: Optional[List[str]], + ip_addresses: Optional[List[str]], + namespaces: Optional[List[str]], + pod_names: Optional[List[str]], + container_names: Optional[List[str]], + workload_names: Optional[List[str]], + process_names: Optional[List[str]], + profiling_statuses: Optional[List[str]], + command_types: Optional[List[str]], + pids: Optional[List[int]], + ) -> Dict[str, Any]: + """Translate the caller-supplied filters into SQL fragments. + + Returns a spec with: + * ``service_filter`` – predicate on the raw ``HostHeartbeats h`` alias, pushed + into the materialized ``fresh_hosts`` CTE. + * ``conditions`` – list of ``(sql, depth)`` predicates on the wrapped base alias + ``b`` (standard column names). ``depth`` is the minimum join depth at which the + referenced column becomes available. + * ``filter_depth`` – deepest ``depth`` among the active predicates. + * ``params`` – bound query parameters. + + Every filter is applied identically wherever the base is used (tab counts, the + entity key set, and the per-entity aggregation), preserving the previous semantics + where all filters lived in a single ``filtered`` CTE. + """ + params: Dict[str, Any] = {} + conditions: List[Tuple[str, int]] = [] + + def add_partial(column: str, values: Optional[List[Any]], prefix: str) -> None: + if not values: + return + depth = self._WORKLOAD_COLUMN_DEPTH[column] + ors = [] + for idx, value in enumerate(values): + key = f"{prefix}_{idx}" + ors.append(f"b.{column}::text ILIKE %({key})s") + params[key] = f"%{value}%" + conditions.append(("(" + " OR ".join(ors) + ")", depth)) + + def add_exact(column: str, values: Optional[List[Any]], prefix: str) -> None: + if not values: + return + depth = self._WORKLOAD_COLUMN_DEPTH[column] + ors = [] + for idx, value in enumerate(values): + key = f"{prefix}_{idx}" + ors.append(f"LOWER(b.{column}::text) = LOWER(%({key})s)") + params[key] = str(value) + conditions.append(("(" + " OR ".join(ors) + ")", depth)) + + service_filter = "TRUE" + if service_names: + if exact_match: + service_filter = "fh.service_name = ANY(%(service_names)s)" + params["service_names"] = service_names + else: + ors = [] + for idx, service_name in enumerate(service_names): + key = f"svc_{idx}" + ors.append(f"fh.service_name ILIKE %({key})s") + params[key] = f"%{service_name}%" + service_filter = "(" + " OR ".join(ors) + ")" + + add_partial("hostname", hostnames, "host") + add_partial("ip_address", ip_addresses, "ip") + add_partial("namespace", namespaces, "ns") + add_partial("pod_name", pod_names, "pod") + add_partial("container_name", container_names, "cont") + add_partial("workload_name", workload_names, "wl") + add_partial("process_name", process_names, "proc") + add_exact("profiling_status", profiling_statuses, "pstat") + add_exact("command_type", command_types, "ctype") + if pids: + conditions.append(("b.pid = ANY(%(pids)s)", self._WORKLOAD_COLUMN_DEPTH["pid"])) + params["pids"] = pids + + filter_depth = max((depth for _, depth in conditions), default=0) + return { + "params": params, + "conditions": conditions, + "service_filter": service_filter, + "filter_depth": filter_depth, + } + + def _workload_cte_prefix(self, service_filter: str) -> str: + """Shared ``WITH`` prefix: the materialized active fleet and its latest commands.""" + return f""" + WITH fresh_hosts AS MATERIALIZED ( + -- Restrict to the active fleet FIRST and materialize it, so grouping/sorting + -- downstream can never drive a full-index scan over the (heavily stale) + -- HostHeartbeats table. This is the difference between ~1s and a timeout. + SELECT + fh.id, + fh.hostname, + host(fh.ip_address) AS ip_address, + fh.service_name, + fh.agent_version, + fh.run_mode, + fh.heartbeat_timestamp + FROM HostHeartbeats fh + WHERE fh.heartbeat_timestamp > NOW() - INTERVAL '2 minutes' + AND {service_filter} + ), + current_commands AS ( + -- ProfilingCommands is UNIQUE (hostname, service_name), so it is already + -- one row per host/service; no window/dedup needed. + SELECT hostname, service_name, command_type, status, combined_config + FROM ProfilingCommands + )""" + + def _workload_base_sql( + self, + depth: int, + where_extra: Optional[List[str]] = None, + driving: str = "cte", + service_filter: str = "TRUE", + ) -> str: + """Build the flatten sub-SELECT down to ``depth`` (0=host, 1=+container, 2=+process). + + Columns are exposed under stable, unqualified names so a single set of filter and + correlation predicates works at any depth. The container/process joins are only + added when the depth requires them, so shallow scopes never pay for the full + cross-product fan-out. + + ``driving`` selects the host source: + * ``"cte"`` – scan the materialized ``fresh_hosts`` CTE once. Used for the + (single-pass) entity key set and tab counts. + * ``"direct"`` – read ``HostHeartbeats`` directly with the freshness + service + predicate inline. Used inside the hydrate LATERAL so a correlated + ``fh.service_name = p.service_name`` / ``fh.hostname = p.hostname`` predicate + in ``where_extra`` uses the HostHeartbeats indexes instead of re-scanning the + whole materialized CTE for every page entity. + """ + ip_expr = "host(fh.ip_address) AS ip_address" if driving == "direct" else "fh.ip_address" + cols = [ + "fh.id AS host_id", + "fh.hostname", + ip_expr, + "fh.service_name", + "fh.agent_version", + "fh.run_mode", + "fh.heartbeat_timestamp", + "COALESCE(c.command_type, 'N/A') AS command_type", + "c.status AS command_status", + "c.combined_config AS combined_config", + ] + joins = [ + "LEFT JOIN current_commands c ON fh.hostname = c.hostname AND fh.service_name = c.service_name", + ] + if depth >= 1: + cols += ["hc.container_name", "hc.namespace", "hc.pod_name", "hc.workload_name", "hc.workload_kind"] + joins.append("LEFT JOIN HeartbeatContainers hc ON hc.host_id = fh.id") + if depth >= 2: + cols += ["hp.pid", "hp.process_name"] + joins.append("LEFT JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id") + cols.append(self._WORKLOAD_PROFILING_STATUS_CASE) + select_list = ",\n ".join(cols) + join_list = "\n ".join(joins) + + where_terms: List[str] = [] + if driving == "direct": + from_clause = "HostHeartbeats fh" + where_terms.append("fh.heartbeat_timestamp > NOW() - INTERVAL '2 minutes'") + where_terms.append(service_filter) + else: + from_clause = "fresh_hosts fh" + if where_extra: + where_terms.extend(where_extra) + where_sql = ("\n WHERE " + " AND ".join(where_terms)) if where_terms else "" + return f"""SELECT + {select_list} + FROM {from_clause} + {join_list}{where_sql}""" + + @staticmethod + def _workload_where(conditions: List[Tuple[str, int]], extra: Optional[List[str]] = None) -> str: + terms = [sql for sql, _ in conditions] + if extra: + terms.extend(extra) + return (" WHERE " + " AND ".join(terms)) if terms else "" + + @staticmethod + def _workload_agg_columns() -> str: + """The per-group aggregate SELECT list (references the wrapped base alias ``b``). + + Shared by both the entity-first LATERAL hydrate and the single-pass GROUP BY, so + the two code paths return byte-identical row shapes. ``l_*`` columns are the latest + (by heartbeat) representative for the group; the counts are per-scope cardinalities. + """ + return """ + COUNT(DISTINCT b.hostname) AS host_count, + COUNT(DISTINCT b.namespace) FILTER (WHERE b.namespace IS NOT NULL) AS namespace_count, + COUNT(DISTINCT b.pod_name) FILTER (WHERE b.pod_name IS NOT NULL) AS pod_count, + COUNT(DISTINCT b.container_name) FILTER (WHERE b.container_name IS NOT NULL) AS container_count, + COUNT(DISTINCT (b.pid, b.process_name)) FILTER (WHERE b.pid IS NOT NULL) AS process_count, + array_agg(DISTINCT b.pid) FILTER (WHERE b.pid IS NOT NULL) AS pids, + (array_agg(b.hostname ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_hostname, + (array_agg(b.ip_address ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_ip_address, + (array_agg(b.namespace ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_namespace, + (array_agg(b.pod_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_pod_name, + (array_agg(b.container_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_container_name, + (array_agg(b.workload_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_name, + (array_agg(b.workload_kind ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_kind, + (array_agg(b.process_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_process_name, + (array_agg(b.pid ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_pid, + (array_agg(b.command_type ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_type, + (array_agg(b.command_status ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_status, + (array_agg(b.combined_config ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_combined_config, + bool_or(b.profiling_status = 'active') AS any_active, + (array_agg(b.profiling_status ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_profiling_status, + (array_agg(b.agent_version ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_agent_version, + (array_agg(b.run_mode ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_run_mode, + MAX(b.heartbeat_timestamp) AS l_heartbeat_timestamp""" + + def _workload_tab_counts(self, spec: Dict[str, Any]) -> Tuple[Dict[str, int], int]: + """Compute the six scope tab counts + active host total using tiered queries. + + Instead of one pass of seven ``COUNT(DISTINCT tuple)`` over the full + host x container x process flatten (which times out once the child tables are + populated), each count is computed from the shallowest base that exposes its + columns, using ``COUNT(*) FROM (SELECT DISTINCT ...)`` (a hash-distinct that is far + cheaper than ``COUNT(DISTINCT tuple)``). All filters are still applied at every + tier, so the counts match the previous single-pass semantics. + """ + prefix = self._workload_cte_prefix(spec["service_filter"]) + params = spec["params"] + conditions = spec["conditions"] + filter_depth = spec["filter_depth"] + + # Tier A (host/service/active): base at host depth (or deeper if filters require it). + base_a = self._workload_base_sql(max(0, filter_depth)) + where_a = self._workload_where(conditions) + query_a = prefix + f""", + fa AS MATERIALIZED ( + SELECT b.service_name, b.hostname FROM ({base_a}) b{where_a} + ) + SELECT + (SELECT COUNT(DISTINCT service_name) FROM fa) AS c_service, + (SELECT COUNT(*) FROM (SELECT DISTINCT service_name, hostname FROM fa) d) AS c_host, + (SELECT COUNT(DISTINCT hostname) FROM fa) AS active_hosts + """ + row_a = (self.db.execute(query_a, params, one_value=False, return_dict=True, fetch_all=True) or [{}])[0] or {} + + # Tier B (namespace/pod/container): base at container depth. + base_b = self._workload_base_sql(max(1, filter_depth)) + where_b = self._workload_where(conditions) + query_b = prefix + f""", + fb AS MATERIALIZED ( + SELECT b.service_name, b.hostname, b.namespace, b.pod_name, b.container_name + FROM ({base_b}) b{where_b} + ) + SELECT + (SELECT COUNT(*) FROM (SELECT DISTINCT service_name, namespace FROM fb WHERE namespace IS NOT NULL) d) AS c_namespace, + (SELECT COUNT(*) FROM (SELECT DISTINCT service_name, namespace, pod_name FROM fb WHERE pod_name IS NOT NULL) d) AS c_pod, + (SELECT COUNT(*) FROM (SELECT DISTINCT service_name, hostname, namespace, pod_name, container_name FROM fb WHERE container_name IS NOT NULL) d) AS c_container + """ + row_b = (self.db.execute(query_b, params, one_value=False, return_dict=True, fetch_all=True) or [{}])[0] or {} + + # Tier C (process): base at process depth. ``(host_id, pid)`` is 1:1 with + # ``(service_name, hostname, pid)`` (unique_host_heartbeat) but distinct on the two + # integer columns is dramatically cheaper than on the wide text tuple. + base_c = self._workload_base_sql(max(2, filter_depth)) + where_c = self._workload_where(conditions, extra=["b.pid IS NOT NULL"]) + query_c = prefix + f""", + fc AS MATERIALIZED ( + SELECT b.host_id, b.pid FROM ({base_c}) b{where_c} + ) + SELECT (SELECT COUNT(*) FROM (SELECT DISTINCT host_id, pid FROM fc) d) AS c_process + """ + row_c = (self.db.execute(query_c, params, one_value=False, return_dict=True, fetch_all=True) or [{}])[0] or {} + + tab_counts = { + "service": row_a.get("c_service") or 0, + "namespace": row_b.get("c_namespace") or 0, + "host": row_a.get("c_host") or 0, + "pod": row_b.get("c_pod") or 0, + "container": row_b.get("c_container") or 0, + "process": row_c.get("c_process") or 0, + } + active_hosts = row_a.get("active_hosts") or 0 + return tab_counts, active_hosts + + def _build_workload_row( + self, db_row: Dict[str, Any], scope: str, key_cols: List[str] + ) -> Dict[str, Any]: + """Build one API row dict from a grouped/summary DB row. + + Shared by the live grouped query and the precomputed-summary reader; both + expose the same column names (key cols + ``l_*``/count aggregates). + """ + key_values = [db_row.get(col) for col in key_cols] + row_id = "|".join("" if value is None else str(value) for value in key_values) + command_metadata = self._extract_command_metadata( + db_row.get("l_combined_config"), + db_row.get("l_command_type"), + db_row.get("l_command_status"), + ) + # Prefer the PID-aware, row-level status computed in SQL over the host-level + # command status. A group is "active" when any of its rows are actively + # targeted (whole-host command or a matching PID); otherwise fall back to the + # latest row status so entities on a host that is only partially profiled + # (e.g. one container) are not incorrectly shown as active. + profiling_status = "active" if db_row.get("any_active") else db_row.get("l_profiling_status") + if not profiling_status: + profiling_status = command_metadata.get("profiling_status") + host_count = db_row.get("host_count") or 0 + return { + "id": row_id, + "scope": scope, + "service_name": db_row.get("service_name"), + "namespace": db_row.get("l_namespace"), + "hostname": db_row.get("l_hostname"), + "ip_address": db_row.get("l_ip_address"), + "pod_name": db_row.get("l_pod_name"), + "container_name": db_row.get("l_container_name"), + "workload_name": db_row.get("l_workload_name"), + "workload_kind": db_row.get("l_workload_kind"), + "process_name": db_row.get("l_process_name"), + "pid": db_row.get("l_pid"), + "pids": sorted(db_row.get("pids") or []), + "active_hosts": host_count, + "host_count": host_count, + "namespace_count": db_row.get("namespace_count") or 0, + "pod_count": db_row.get("pod_count") or 0, + "container_count": db_row.get("container_count") or 0, + "process_count": db_row.get("process_count") or 0, + "command_type": command_metadata.get("command_type"), + "profiling_status": profiling_status, + "profiling_mode": command_metadata.get("profiling_mode"), + "frequency": command_metadata.get("frequency"), + "profiler_summary": command_metadata.get("profiler_summary"), + "heartbeat_timestamp": db_row.get("l_heartbeat_timestamp"), + "agent_version": db_row.get("l_agent_version"), + "run_mode": db_row.get("l_run_mode"), + } + + # Scopes whose grouped rows are precomputed into workload_scope_summary. + _WORKLOAD_SUMMARY_SCOPES = frozenset({"service", "namespace", "pod"}) + + def _precomputed_tab_counts(self) -> Optional[Tuple[Dict[str, int], int]]: + """Read the six tab counts + active_hosts from the precomputed store. + + Returns ``None`` when the store is unavailable -- not built yet (fresh + deploy before the first refresh) or the migration has not been applied -- + so callers transparently fall back to the live computation. + """ + try: + rows = self.db.execute( + "SELECT scope, count FROM workload_tab_counts", + {}, one_value=False, return_dict=True, fetch_all=True, + ) + except Exception: + # Store missing/unavailable -> fall back to the live path. + return None + if not rows: + return None + by_scope = {r["scope"]: r["count"] for r in rows} + tab_counts = { + "service": by_scope.get("service") or 0, + "namespace": by_scope.get("namespace") or 0, + "host": by_scope.get("host") or 0, + "pod": by_scope.get("pod") or 0, + "container": by_scope.get("container") or 0, + "process": by_scope.get("process") or 0, + } + return tab_counts, (by_scope.get("active_hosts") or 0) + + def _precomputed_scope_rows( + self, scope: str, page: int, page_size: int, sort_by: Optional[str], sort_order: str + ) -> Tuple[List[Dict[str, Any]], int]: + """Read a page of grouped rows for a coarse scope from workload_scope_summary.""" + key_cols = self._WORKLOAD_SCOPE_KEYS.get(scope, self._WORKLOAD_SCOPE_KEYS["service"]) + direction = "DESC" if str(sort_order).lower() == "desc" else "ASC" + sort_col = None + if sort_by and sort_by in key_cols: + sort_col = sort_by + elif sort_by and sort_by in self._WORKLOAD_SORT_ALIAS: + sort_col = self._WORKLOAD_SORT_ALIAS[sort_by] + # Deterministic order: requested sort (if any) then the precomputed key order. + order_by = (f"{sort_col} {direction} NULLS LAST, sort_seq ASC" if sort_col else "sort_seq ASC") + query = f""" + SELECT *, COUNT(*) OVER () AS total_groups + FROM workload_scope_summary + WHERE scope = %(scope)s + ORDER BY {order_by} + LIMIT %(page_size)s OFFSET %(offset)s + """ + params = {"scope": scope, "page_size": page_size, "offset": page * page_size} + db_rows = self.db.execute(query, params, one_value=False, return_dict=True, fetch_all=True) + total_count = db_rows[0]["total_groups"] if db_rows else 0 + rows = [self._build_workload_row(db_row, scope, key_cols) for db_row in db_rows or []] + return rows, total_count + + def _query_workload_groups( self, + scope: str, + spec: Dict[str, Any], + page: int = 0, + page_size: int = 50, + sort_by: Optional[str] = None, + sort_order: str = "asc", + ) -> Tuple[List[Dict[str, Any]], int]: + key_cols = self._WORKLOAD_SCOPE_KEYS.get(scope, self._WORKLOAD_SCOPE_KEYS["process"]) + scope_depth = self._WORKLOAD_SCOPE_DEPTH.get(scope, 2) + conditions = spec["conditions"] + filter_depth = spec["filter_depth"] + params = dict(spec["params"]) + + # ------------------------------------------------------------------ strategy + # Two aggregation strategies, chosen by whether the scope keys on a specific host: + # * hostname IN key (host/container/process): "entity-first". Enumerate the page + # of entities from the shallow key set, then hydrate ONLY that page via a + # LATERAL that is bounded to a single host (indexed) per row. Great for the many + # small entities of these scopes (the default host view is ~0.1s of query time). + # * hostname NOT IN key (service/namespace/pod): "single-pass". These scopes have + # comparatively few, large entities that each span many hosts, so a per-entity + # LATERAL would re-scan whole services repeatedly. Instead compute every group + # in one GROUP BY over the flatten (a couple of seconds) and paginate the result. + direction = "DESC" if str(sort_order).lower() == "desc" else "ASC" + agg_columns = self._workload_agg_columns() + guard_col = key_cols[-1] if scope in self._WORKLOAD_SCOPE_NULL_GUARD else None + use_entity_first = "hostname" in key_cols + # `service` has no host key and few/large groups, but (unlike namespace/pod) its + # groups span whole hosts, so its per-group counts can be computed at the cheap + # container grain and its `any_active` at the host grain -- for service scope that + # is provably identical to the PID-aware value (a command's target PIDs always + # belong to the service's own hosts). Only when no process-tier filter is present + # (which would need the process grain to stay correct). + use_two_grain = scope == "service" and filter_depth < 2 + + if use_entity_first: + # ---------- ordering: page the key set, sorting on a key or (materialized) aggregate + sort_agg_expr: Optional[str] = None + sort_depth = 0 + order_pairs: List[Tuple[str, str]] = [] + if sort_by and sort_by in key_cols: + order_pairs.append((sort_by, direction)) + order_pairs.extend((col, "ASC") for col in key_cols if col != sort_by) + elif sort_by and sort_by in self._WORKLOAD_SORT_AGG: + sort_agg_expr, sort_depth = self._WORKLOAD_SORT_AGG[sort_by] + order_pairs.append(("__sort", direction)) + order_pairs.extend((col, "ASC") for col in key_cols) + else: + order_pairs.extend((col, "ASC") for col in key_cols) + + # ---------- entity key/summary set (shallowest base that exposes key + filters + sort) + page_depth = max(scope_depth, filter_depth, sort_depth) + guard_extra = [f"b.{guard_col} IS NOT NULL"] if guard_col else [] + base_page = self._workload_base_sql(page_depth) + summary_where = self._workload_where(conditions, extra=guard_extra) + summary_select = ", ".join(f"b.{col}" for col in key_cols) + if sort_agg_expr: + summary_select += f", {sort_agg_expr} AS __sort" + group_by = ", ".join(f"b.{col}" for col in key_cols) + page_order = ", ".join(f"{col} {dir_} NULLS LAST" for col, dir_ in order_pairs) + + # ---------- per-entity hydrate. Only host-level keys are correlated (with ``=``) + # inside the base sub-SELECT so the plan drives from the HostHeartbeats + # service_name/hostname indexes; finer keys are NULL-safe residual filters on the + # host-bounded set (never pushed onto child tables, which would let the planner + # drive from a global index scan and explode on common namespaces). + host_key_corr: List[str] = [] + residual_corr: List[str] = [] + for col in key_cols: + if self._WORKLOAD_COLUMN_DEPTH[col] == 0: + host_key_corr.append(f"{self._WORKLOAD_KEY_ALIAS[col]} = p.{col}") + else: + residual_corr.append(f"b.{col} IS NOT DISTINCT FROM p.{col}") + base_full = self._workload_base_sql( + 2, where_extra=host_key_corr, driving="direct", service_filter=spec["service_filter"] + ) + lateral_terms = residual_corr + [sql for sql, _ in conditions] + lateral_where = ("\n WHERE " + " AND ".join(lateral_terms)) if lateral_terms else "" + final_order = ", ".join(f"p.{col} {dir_} NULLS LAST" for col, dir_ in order_pairs) + page_key_select = ", ".join(f"p.{col}" for col in key_cols) + + query = self._workload_cte_prefix(spec["service_filter"]) + f""", + entity_summary AS MATERIALIZED ( + SELECT {summary_select} + FROM ({base_page}) b{summary_where} + GROUP BY {group_by} + ), + page AS ( + SELECT * FROM entity_summary + ORDER BY {page_order} + LIMIT %(page_size)s OFFSET %(offset)s + ) + SELECT + {page_key_select}, + agg.*, + (SELECT COUNT(*) FROM entity_summary) AS total_groups + FROM page p + LEFT JOIN LATERAL ( + SELECT{agg_columns} + FROM ({base_full}) b + {lateral_where} + ) agg ON true + ORDER BY {final_order} + """ + elif use_two_grain: + # ---------- two-grain single-pass (service scope). Counts + latest metadata at + # the container grain (~316K rows), process_count via the cheap + # COUNT(*) FROM (SELECT DISTINCT ...) trick, and any_active at the host grain + # (identical to PID-aware for service scope). Avoids aggregating the full + # ~1.08M process flatten -> ~16s down to ~4s on prod. Process-grain + # representatives (l_process_name/l_pid/pids) are not shown at service scope and + # are returned NULL/empty. + sort_out_col = None + if sort_by and sort_by in key_cols: + sort_out_col = sort_by + elif sort_by and sort_by in self._WORKLOAD_SORT_ALIAS: + sort_out_col = self._WORKLOAD_SORT_ALIAS[sort_by] + order_terms = [f"{sort_out_col} {direction} NULLS LAST"] if sort_out_col else [] + order_terms.extend(f"{col} ASC NULLS LAST" for col in key_cols if col != sort_out_col) + two_order = ", ".join(order_terms) + + cbase = self._workload_base_sql(1) + cwhere = self._workload_where(conditions) + pbase = self._workload_base_sql(2) + pwhere = self._workload_where(conditions, extra=["b.pid IS NOT NULL"]) + + query = self._workload_cte_prefix(spec["service_filter"]) + f""", + cagg AS ( + SELECT + b.service_name, + COUNT(DISTINCT b.hostname) AS host_count, + COUNT(DISTINCT b.namespace) FILTER (WHERE b.namespace IS NOT NULL) AS namespace_count, + COUNT(DISTINCT b.pod_name) FILTER (WHERE b.pod_name IS NOT NULL) AS pod_count, + COUNT(DISTINCT b.container_name) FILTER (WHERE b.container_name IS NOT NULL) AS container_count, + NULL::integer[] AS pids, + (array_agg(b.hostname ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_hostname, + (array_agg(b.ip_address ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_ip_address, + (array_agg(b.namespace ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_namespace, + (array_agg(b.pod_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_pod_name, + (array_agg(b.container_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_container_name, + (array_agg(b.workload_name ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_name, + (array_agg(b.workload_kind ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_workload_kind, + NULL::text AS l_process_name, + NULL::integer AS l_pid, + (array_agg(b.command_type ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_type, + (array_agg(b.command_status ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_command_status, + (array_agg(b.combined_config ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_combined_config, + bool_or(b.command_type = 'start' AND b.command_status IN ('pending', 'sent', 'completed')) AS any_active, + NULL::text AS l_profiling_status, + (array_agg(b.agent_version ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_agent_version, + (array_agg(b.run_mode ORDER BY b.heartbeat_timestamp DESC NULLS LAST))[1] AS l_run_mode, + MAX(b.heartbeat_timestamp) AS l_heartbeat_timestamp, + COUNT(*) OVER () AS total_groups + FROM ({cbase}) b{cwhere} + GROUP BY b.service_name + ), + pcount AS ( + SELECT service_name, COUNT(*) AS process_count + FROM (SELECT DISTINCT b.service_name, b.pid, b.process_name FROM ({pbase}) b{pwhere}) d + GROUP BY service_name + ) + SELECT + c.service_name, + c.host_count, c.namespace_count, c.pod_count, c.container_count, + COALESCE(pc.process_count, 0) AS process_count, + c.pids, c.l_hostname, c.l_ip_address, c.l_namespace, c.l_pod_name, c.l_container_name, + c.l_workload_name, c.l_workload_kind, c.l_process_name, c.l_pid, + c.l_command_type, c.l_command_status, c.l_combined_config, c.any_active, + c.l_profiling_status, c.l_agent_version, c.l_run_mode, c.l_heartbeat_timestamp, + c.total_groups + FROM cagg c + LEFT JOIN pcount pc ON pc.service_name = c.service_name + ORDER BY {two_order} + LIMIT %(page_size)s OFFSET %(offset)s + """ + else: + # ---------- single-pass: one GROUP BY over the flatten, paginated. ORDER BY + # references the grouped output columns (key columns or ``l_*``/count aliases). + sort_out_col = None + if sort_by and sort_by in key_cols: + sort_out_col = sort_by + elif sort_by and sort_by in self._WORKLOAD_SORT_ALIAS: + sort_out_col = self._WORKLOAD_SORT_ALIAS[sort_by] + order_terms = [f"{sort_out_col} {direction} NULLS LAST"] if sort_out_col else [] + order_terms.extend(f"{col} ASC NULLS LAST" for col in key_cols if col != sort_out_col) + single_order = ", ".join(order_terms) + + guard_extra = [f"b.{guard_col} IS NOT NULL"] if guard_col else [] + base_single = self._workload_base_sql(2) + single_where = self._workload_where(conditions, extra=guard_extra) + key_select = ", ".join(f"b.{col}" for col in key_cols) + group_by = ", ".join(f"b.{col}" for col in key_cols) + + query = self._workload_cte_prefix(spec["service_filter"]) + f""" + SELECT + {key_select},{agg_columns}, + COUNT(*) OVER () AS total_groups + FROM ({base_single}) b{single_where} + GROUP BY {group_by} + ORDER BY {single_order} + LIMIT %(page_size)s OFFSET %(offset)s + """ + + group_params = {**params, "page_size": page_size, "offset": page * page_size} + db_rows = self.db.execute(query, group_params, one_value=False, return_dict=True, fetch_all=True) + total_count = db_rows[0]["total_groups"] if db_rows else 0 + rows = [self._build_workload_row(db_row, scope, key_cols) for db_row in db_rows or []] + # Ordering and pagination are done in SQL (see order_by/LIMIT above). + return rows, total_count + + def get_workload_inventory_status( + self, + scope: str, service_names: Optional[List[str]] = None, hostnames: Optional[List[str]] = None, ip_addresses: Optional[List[str]] = None, + namespaces: Optional[List[str]] = None, + pod_names: Optional[List[str]] = None, + container_names: Optional[List[str]] = None, + workload_names: Optional[List[str]] = None, + process_names: Optional[List[str]] = None, profiling_statuses: Optional[List[str]] = None, command_types: Optional[List[str]] = None, pids: Optional[List[int]] = None, exact_match: bool = False, + page: int = 0, + page_size: int = 50, + sort_by: Optional[str] = None, + sort_order: str = "asc", + ) -> Dict[str, Any]: + spec = self._workload_filter_spec( + service_names=service_names, + exact_match=exact_match, + hostnames=hostnames, + ip_addresses=ip_addresses, + namespaces=namespaces, + pod_names=pod_names, + container_names=container_names, + workload_names=workload_names, + process_names=process_names, + profiling_statuses=profiling_statuses, + command_types=command_types, + pids=pids, + ) + + # Fast path: with no filters, serve from the precomputed store (rebuilt every + # ~30s by the periodic worker). Counts + coarse-scope rows come from the Layer 2 + # summaries; host/container/process rows still use the (fast) live grouped query. + # Falls back to the fully live path when the store has not been built yet. + no_filters = not spec["conditions"] and spec["service_filter"] == "TRUE" + if no_filters: + precomputed = self._precomputed_tab_counts() + if precomputed is not None: + tab_counts, active_hosts = precomputed + if scope in self._WORKLOAD_SUMMARY_SCOPES: + rows, total_count = self._precomputed_scope_rows( + scope, page, page_size, sort_by, sort_order + ) + else: + rows, total_count = self._query_workload_groups( + scope, spec, page=page, page_size=page_size, sort_by=sort_by, sort_order=sort_order + ) + return { + "scope": scope, + "rows": rows, + "tab_counts": tab_counts, + "active_hosts": active_hosts, + "total_count": total_count, + "page": page, + "page_size": page_size, + } + + # Filtered (or store-not-built) path: tiered counts + live grouped query. + tab_counts, active_hosts = self._workload_tab_counts(spec) + + rows, total_count = self._query_workload_groups( + scope, spec, page=page, page_size=page_size, sort_by=sort_by, sort_order=sort_order + ) + + return { + "scope": scope, + "rows": rows, + "tab_counts": tab_counts, + "active_hosts": active_hosts, + "total_count": total_count, + "page": page, + "page_size": page_size, + } + + def resolve_workload_targets( + self, + service_name: str, + target_scope: str, + target_entities: Optional[List[Dict[str, Any]]] = None, + ) -> Dict[str, Optional[List[int]]]: + """Resolve workload selectors into concrete host/PID profiling targets. + + Matching is pushed down to SQL joins over HostHeartbeats / HeartbeatContainers / + HeartbeatProcesses so we never materialize the full inventory in Python. + Service- and host-scope resolve to host-level targets (PID list = None); the + finer scopes resolve to per-host PID sets. + """ + entities = target_entities or [{"service_name": service_name}] + recency = "h.heartbeat_timestamp > NOW() - INTERVAL '2 minutes'" + + if target_scope in ("service", "host"): + query = ( + f"SELECT DISTINCT h.hostname FROM HostHeartbeats h " + f"WHERE {recency} AND h.service_name = %(service_name)s" + ) + params: Dict[str, Any] = {"service_name": service_name} + if target_scope == "host": + wanted_hosts = sorted( + { + entity.get("hostname") + for entity in entities + if entity.get("hostname") + and (not entity.get("service_name") or entity.get("service_name") == service_name) + } + ) + if not wanted_hosts: + return {} + query += " AND h.hostname = ANY(%(wanted_hosts)s)" + params["wanted_hosts"] = wanted_hosts + + rows = self.db.execute(query, params, one_value=False, return_dict=True, fetch_all=True) + hostnames = sorted({row["hostname"] for row in (rows or []) if row.get("hostname")}) + return {hostname: None for hostname in hostnames} + + # Finer scopes require a concrete process, so inner-join through to processes. + base_from = ( + "FROM HostHeartbeats h " + "JOIN HeartbeatContainers hc ON hc.host_id = h.id " + "JOIN HeartbeatProcesses hp ON hp.container_row_id = hc.id " + f"WHERE {recency} AND h.service_name = %(service_name)s" + ) + + host_pid_mapping: Dict[str, Set[int]] = defaultdict(set) + for entity in entities: + if entity.get("service_name") and entity.get("service_name") != service_name: + continue + + conditions: List[str] = [] + params = {"service_name": service_name} + + if target_scope == "namespace": + conditions.append("hc.namespace IS NOT DISTINCT FROM %(e_namespace)s") + params["e_namespace"] = entity.get("namespace") + elif target_scope == "workload": + conditions.append("hc.workload_name IS NOT DISTINCT FROM %(e_workload)s") + params["e_workload"] = entity.get("workload_name") + if entity.get("namespace") is not None: + conditions.append("hc.namespace = %(e_namespace)s") + params["e_namespace"] = entity.get("namespace") + elif target_scope == "pod": + conditions.append("hc.pod_name IS NOT DISTINCT FROM %(e_pod)s") + params["e_pod"] = entity.get("pod_name") + if entity.get("namespace") is not None: + conditions.append("hc.namespace = %(e_namespace)s") + params["e_namespace"] = entity.get("namespace") + elif target_scope == "container": + conditions.append("hc.container_name IS NOT DISTINCT FROM %(e_container)s") + params["e_container"] = entity.get("container_name") + if entity.get("pod_name") is not None: + conditions.append("hc.pod_name = %(e_pod)s") + params["e_pod"] = entity.get("pod_name") + if entity.get("namespace") is not None: + conditions.append("hc.namespace = %(e_namespace)s") + params["e_namespace"] = entity.get("namespace") + elif target_scope == "process": + # (pid match) OR (process_name match + optional container/pod/namespace), + # mirroring the original disjunction. + or_parts: List[str] = [] + if entity.get("pid") is not None: + or_parts.append("hp.pid = %(e_pid)s") + params["e_pid"] = entity.get("pid") + if entity.get("process_name") is not None: + name_conditions = ["hp.process_name = %(e_process)s"] + params["e_process"] = entity.get("process_name") + if entity.get("container_name") is not None: + name_conditions.append("hc.container_name = %(e_container)s") + params["e_container"] = entity.get("container_name") + if entity.get("pod_name") is not None: + name_conditions.append("hc.pod_name = %(e_pod)s") + params["e_pod"] = entity.get("pod_name") + if entity.get("namespace") is not None: + name_conditions.append("hc.namespace = %(e_namespace)s") + params["e_namespace"] = entity.get("namespace") + or_parts.append("(" + " AND ".join(name_conditions) + ")") + if not or_parts: + continue + conditions.append("(" + " OR ".join(or_parts) + ")") + else: + continue + + where_extra = (" AND " + " AND ".join(conditions)) if conditions else "" + query = f"SELECT h.hostname, hp.pid {base_from}{where_extra}" + rows = self.db.execute(query, params, one_value=False, return_dict=True, fetch_all=True) + for row in rows or []: + if row.get("hostname") is not None and row.get("pid") is not None: + host_pid_mapping[row["hostname"]].add(row["pid"]) + + return {hostname: sorted(pid_set) for hostname, pid_set in host_pid_mapping.items()} + + def get_profiling_host_status_optimized( + self, + service_names: Optional[List[str]] = None, + hostnames: Optional[List[str]] = None, + ip_addresses: Optional[List[str]] = None, + profiling_statuses: Optional[List[str]] = None, + command_types: Optional[List[str]] = None, + pids: Optional[List[int]] = None, + exact_match: bool = False ) -> List[Dict]: """ Get profiling host status with all filters applied in a single optimized query. @@ -1422,6 +2893,9 @@ def get_profiling_host_status_optimized( LEFT JOIN current_commands c ON h.hostname = c.hostname AND h.service_name = c.service_name WHERE 1=1 + -- Only show hosts that sent heartbeat in last 2 minutes (recently active) + -- This improves page load performance by filtering out stale/inactive hosts + AND h.heartbeat_timestamp > NOW() - INTERVAL '2 minutes' """ values: Dict[str, Any] = {} @@ -1466,7 +2940,7 @@ def get_profiling_host_status_optimized( status_conditions = [] has_stopped = False for idx, status in enumerate(profiling_statuses): - if status.lower() == "stopped": + if status.lower() == 'stopped': has_stopped = True else: param_name = f"status_{idx}" @@ -1484,7 +2958,7 @@ def get_profiling_host_status_optimized( command_type_conditions = [] has_na = False for idx, cmd_type in enumerate(command_types): - if cmd_type.lower() == "n/a": + if cmd_type.lower() == 'n/a': has_na = True else: param_name = f"command_type_{idx}" @@ -1540,6 +3014,48 @@ def get_profiling_host_status_optimized( return results + def get_total_host_count( + self, + service_names: Optional[List[str]] = None, + exact_match: bool = False, + ) -> int: + """ + Get total host count for the selected service(s). + This count IS filtered by service_name - shows total hosts for the selected service. + Fast query using COUNT with indexed columns. + + Args: + service_names: Optional list of service names to filter by + exact_match: If True, use exact match for service names; if False, use partial match (ILIKE) + + Returns: + Total count of distinct hosts for the selected service(s) + """ + query = """ + SELECT COUNT(DISTINCT hostname) as total_count + FROM HostHeartbeats + WHERE 1=1 + """ + + values: Dict[str, Any] = {} + + # Apply service_name filter if provided + if service_names: + if exact_match: + query += " AND service_name = ANY(%(service_names)s)" + values["service_names"] = service_names + else: + # Use ILIKE with OR for partial matching across multiple service names + service_conditions = [] + for idx, service_name in enumerate(service_names): + param_name = f"service_name_{idx}" + service_conditions.append(f"service_name ILIKE %({param_name})s") + values[param_name] = f"%{service_name}%" + query += f" AND ({' OR '.join(service_conditions)})" + + result = self.db.execute(query, values, one_value=False, return_dict=True, fetch_all=True) + return result[0]["total_count"] if result and len(result) > 0 else 0 + def get_pending_profiling_command( self, hostname: str, service_name: str, exclude_command_id: Optional[str] = None ) -> Optional[Dict]: @@ -1807,36 +3323,36 @@ def get_adhoc_flamegraphs_metadata( ) -> List[Dict[str, Any]]: """ Retrieve adhoc flamegraph metadata with optional filters. - + Args: service_id: ID of the service start_time: Optional filter for profiles after this time end_time: Optional filter for profiles before this time hostname_filters: Optional list of hostnames to filter by - + Returns: List of metadata dictionaries containing s3_key, hostname, perf_events, and start_time """ conditions = ["service_id = %s"] params: List[Any] = [service_id] - + if start_time: conditions.append("start_time >= %s") params.append(start_time) - + if end_time: conditions.append("end_time <= %s") params.append(end_time) - + if hostname_filters: placeholders = ", ".join(["%s"] * len(hostname_filters)) conditions.append(f"hostname IN ({placeholders})") params.extend(hostname_filters) - + where_clause = " AND ".join(conditions) - + query = f""" - SELECT + SELECT s3_key, hostname, perf_events, @@ -1846,12 +3362,12 @@ def get_adhoc_flamegraphs_metadata( WHERE {where_clause} ORDER BY start_time DESC """ - + results = self.db.execute(query, tuple(params), one_value=False, fetch_all=True) - + if not results: return [] - + return [ { "s3_key": row[0], diff --git a/src/gprofiler-dev/gprofiler_dev/postgres/postgresdb.py b/src/gprofiler-dev/gprofiler_dev/postgres/postgresdb.py index 8bebf634..855682f8 100644 --- a/src/gprofiler-dev/gprofiler_dev/postgres/postgresdb.py +++ b/src/gprofiler-dev/gprofiler_dev/postgres/postgresdb.py @@ -60,6 +60,43 @@ def get_locked_conn(self): with self._conn_lock: yield self._thread_unsafe_conn + @contextmanager + def transaction(self, return_dict: bool = False): + """Run multiple statements on a single connection within one transaction. + + Yields a cursor; commits on success, rolls back on any exception. Use this + when a logical write spans several statements that must be atomic (e.g. a + parent upsert followed by dependent child writes). + + Unlike ``execute`` this does not retry mid-transaction (a context manager + can only yield once), so it probes the connection up front and reconnects + once if it is dead before handing out the cursor. + """ + with self._conn_lock: + with self.get_locked_conn() as current_conn: + try: + if current_conn.closed: + self._reconnect() + current_conn = self._thread_unsafe_conn + except Exception: + self._reconnect() + current_conn = self._thread_unsafe_conn + + cursor = current_conn.cursor( + cursor_factory=psycopg2.extras.RealDictCursor if return_dict else None + ) + try: + yield cursor + current_conn.commit() + except Exception: + try: + current_conn.rollback() + except Exception: + pass + raise + finally: + cursor.close() + @staticmethod def _execute( cursor, diff --git a/src/gprofiler-dev/gprofiler_dev/s3_profile_dal.py b/src/gprofiler-dev/gprofiler_dev/s3_profile_dal.py index 70162c6c..46064f09 100644 --- a/src/gprofiler-dev/gprofiler_dev/s3_profile_dal.py +++ b/src/gprofiler-dev/gprofiler_dev/s3_profile_dal.py @@ -17,7 +17,7 @@ from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime from io import BytesIO -from typing import Callable, Dict, List, Optional, Set +from typing import Callable, Optional, List, Dict, Set import boto3 from botocore.config import Config @@ -39,7 +39,8 @@ def __init__( ): self.logger = logger self.bucket_name = config.BUCKET_NAME - self.base_directory = config.BASE_DIRECTORY + _prefix = config.S3_PATH_PREFIX + self.base_directory = f"{_prefix}/{config.BASE_DIRECTORY}" if _prefix else config.BASE_DIRECTORY self.input_folder_name = input_folder_name if session is None: with boto3_lock: @@ -50,9 +51,7 @@ def __init__( ) # endpoint_url allows connecting to LocalStack or S3-compatible services for testing # When None (default), uses standard AWS S3 endpoints - self._s3_client = session.client( - "s3", config=Config(max_pool_connections=50), endpoint_url=config.S3_ENDPOINT_URL - ) + self._s3_client = session.client("s3", config=Config(max_pool_connections=50), endpoint_url=config.S3_ENDPOINT_URL) self._s3_resource = session.resource("s3", endpoint_url=config.S3_ENDPOINT_URL) @staticmethod @@ -124,15 +123,22 @@ def _head(key: str) -> Optional[str]: return existing def list_files_with_prefix(self, prefix: str) -> List[Dict]: - """List files in S3 with the given prefix.""" + """List files in S3 with the given prefix""" try: - response = self._s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=prefix) - + response = self._s3_client.list_objects_v2( + Bucket=self.bucket_name, + Prefix=prefix + ) + files = [] - if "Contents" in response: - for obj in response["Contents"]: - files.append({"Key": obj["Key"], "Size": obj["Size"], "LastModified": obj["LastModified"]}) - + if 'Contents' in response: + for obj in response['Contents']: + files.append({ + 'Key': obj['Key'], + 'Size': obj['Size'], + 'LastModified': obj['LastModified'] + }) + return files except Exception as e: self.logger.error(f"Error listing files with prefix {prefix}: {e}") diff --git a/src/gprofiler/backend/config.py b/src/gprofiler/backend/config.py index 8510ad10..1956adc9 100644 --- a/src/gprofiler/backend/config.py +++ b/src/gprofiler/backend/config.py @@ -56,3 +56,11 @@ METRICS_SLI_UUID = os.getenv("METRICS_SLI_UUID", None) BACKEND_ROOT = os.path.dirname(os.path.realpath(__file__)) + +MAX_PROFILING_REQUEST_HOSTS = int(os.getenv("MAX_PROFILING_REQUEST_HOSTS", 20)) + +# Per-worker threadpool size for sync route handlers (Starlette/anyio default: 40). +# With GPROFILER_POSTGRES_CONN_PER_THREAD=TRUE each thread holds its own DB connection, +# so total DB connections ~= gunicorn workers * this value * replicas. 0 keeps the default. +WEBAPP_THREAD_POOL_SIZE = int(os.getenv("GPROFILER_WEBAPP_THREAD_POOL_SIZE", 0)) +MAX_SIMULTANEOUS_PROFILING_HOSTS_PERCENT = int(os.getenv("MAX_SIMULTANEOUS_PROFILING_HOSTS", 10)) diff --git a/src/gprofiler/backend/main.py b/src/gprofiler/backend/main.py index beef14d8..0ef641ba 100644 --- a/src/gprofiler/backend/main.py +++ b/src/gprofiler/backend/main.py @@ -15,12 +15,17 @@ # from datetime import datetime +import logging -from backend import routers +import anyio +from backend import config, routers +from backend.utils.metrics_publisher import MetricsPublisher from fastapi import FastAPI from fastapi.responses import JSONResponse, Response from starlette.exceptions import HTTPException as StarletteHTTPException +logger = logging.getLogger(__name__) + def format_time(dt: datetime): iso_format = dt.isoformat() @@ -32,6 +37,44 @@ def format_time(dt: datetime): app = FastAPI(openapi_url="/api/v1/openapi.json", docs_url="/api/v1/docs") +@app.on_event("startup") +async def startup_event(): + """Initialize services on application startup.""" + # Size the sync-route threadpool per worker; also caps DB connections/worker + # when GPROFILER_POSTGRES_CONN_PER_THREAD=TRUE. + if config.WEBAPP_THREAD_POOL_SIZE > 0: + anyio.to_thread.current_default_thread_limiter().total_tokens = config.WEBAPP_THREAD_POOL_SIZE + logger.info("Webapp threadpool size set to %s", config.WEBAPP_THREAD_POOL_SIZE) + + # Initialize MetricsPublisher + metrics_publisher = MetricsPublisher( + server_url=config.METRICS_AGENT_URL, + service_name=config.METRICS_SERVICE_NAME, + sli_metric_uuid=config.METRICS_SLI_UUID, + enabled=config.METRICS_ENABLED + ) + + # Log initialization status + publisher = MetricsPublisher.get_instance() + if publisher: + logger.info( + f"MetricsPublisher initialized: service={config.METRICS_SERVICE_NAME}, " + f"server={config.METRICS_AGENT_URL}" + ) + else: + logger.info("MetricsPublisher disabled") + + +@app.on_event("shutdown") +async def shutdown_event(): + """Cleanup services on application shutdown.""" + # Cleanup MetricsPublisher + publisher = MetricsPublisher.get_instance() + if publisher: + publisher.flush_and_close() + logger.info("MetricsPublisher closed") + + @app.exception_handler(StarletteHTTPException) async def http_exception_handler(_, exc): if exc.status_code == 204: diff --git a/src/gprofiler/backend/models/filters_models.py b/src/gprofiler/backend/models/filters_models.py index 3f18a06b..d9d64949 100644 --- a/src/gprofiler/backend/models/filters_models.py +++ b/src/gprofiler/backend/models/filters_models.py @@ -14,6 +14,7 @@ # limitations under the License. # +import json from enum import Enum from typing import Any, Dict, List, Optional @@ -35,6 +36,7 @@ class FilterTypes(str, Enum): class RQLCompareOperators(str, Enum): eq_op = "$eq" neq_op = "$neq" + like_op = "$like" class RQLLogicOperators(str, Enum): @@ -60,6 +62,31 @@ def get_formatted_filter(self) -> str: res.append(logic_op) return "__".join(res[:-1]).replace("$", "") + def flamedb_filter_json(self) -> bytes: + """Serialize the filter for the query service, wrapping each ``$like`` value in + ``%…%`` so it performs substring matching. The stored/displayed value stays raw; + only the copy sent to flamedb gets wildcards. + """ + + def _key(item: Any) -> str: + return item.value if isinstance(item, Enum) else str(item) + + out_filter: Dict[str, Any] = {} + for logic_op, expressions in self.filter.items(): + new_expressions = [] + for expression in expressions: + new_expression: Dict[str, Any] = {} + for key, cmp_op_value in expression.items(): + new_cmp: Dict[str, Any] = {} + for cmp_op, value in cmp_op_value.items(): + if _key(cmp_op) == RQLCompareOperators.like_op.value and value and "%" not in value: + value = f"%{value}%" + new_cmp[_key(cmp_op)] = value + new_expression[_key(key)] = new_cmp + new_expressions.append(new_expression) + out_filter[_key(logic_op)] = new_expressions + return json.dumps({"filter": out_filter}).encode() + class Config: @staticmethod def schema_extra(schema: Dict[str, Any]) -> None: diff --git a/src/gprofiler/backend/models/metrics_models.py b/src/gprofiler/backend/models/metrics_models.py index 32fcd0b8..f0d75cee 100644 --- a/src/gprofiler/backend/models/metrics_models.py +++ b/src/gprofiler/backend/models/metrics_models.py @@ -18,7 +18,7 @@ from typing import Any, Dict, List, Optional from backend.models import CamelModel -from pydantic import BaseModel, root_validator, validator +from pydantic import BaseModel, Field, root_validator, validator class SampleCount(BaseModel): @@ -89,6 +89,36 @@ class HTMLMetadata(CamelModel): content: str +class WorkloadTargetEntity(CamelModel): + id: Optional[str] = None + service_name: Optional[str] = None + namespace: Optional[str] = None + hostname: Optional[str] = None + ip_address: Optional[str] = None + pod_name: Optional[str] = None + container_name: Optional[str] = None + workload_name: Optional[str] = None + workload_kind: Optional[str] = None + pid: Optional[int] = None + process_name: Optional[str] = None + + +class HeartbeatProcessInfo(CamelModel): + pid: int + process_name: str + + +class HeartbeatContainerInfo(CamelModel): + container_id: Optional[str] = None + container_name: str + runtime: Optional[str] = None + namespace: Optional[str] = None + pod_name: Optional[str] = None + workload_name: Optional[str] = None + workload_kind: Optional[str] = None + processes: List[HeartbeatProcessInfo] = Field(default_factory=list) + + class ProfilingRequest(BaseModel): """Model for profiling request parameters""" @@ -99,8 +129,11 @@ class ProfilingRequest(BaseModel): frequency: Optional[int] = 11 profiling_mode: Optional[str] = "cpu" # "cpu", "allocation", "none" target_hosts: Optional[Dict[str, Optional[List[int]]]] = None + target_scope: Optional[str] = "host" + target_entities: Optional[List[WorkloadTargetEntity]] = None stop_level: Optional[str] = "process" # "process" or "host" additional_args: Optional[Dict[str, Any]] = None + dry_run: Optional[bool] = False @validator("request_type") def validate_request_type(cls, v): @@ -114,6 +147,19 @@ def validate_profiling_mode(cls, v): raise ValueError('profiling_mode must be "cpu", "allocation", or "none"') return v + @validator("target_scope") + def validate_target_scope(cls, v): + if v not in ["host", "service", "namespace", "workload", "pod", "container", "process"]: + raise ValueError('target_scope must be one of "host", "service", "namespace", "workload", "pod", "container", or "process"') + return v + + @validator("target_hosts") + def validate_target_hosts(cls, v): + """Validate that target_hosts is not empty when provided.""" + if v is not None and len(v) == 0: + raise ValueError("target_hosts cannot be empty") + return v + @validator("stop_level") def validate_stop_level(cls, v): if v not in ["process", "host"]: @@ -134,23 +180,29 @@ def validate_frequency(cls, v): @root_validator def validate_profile_request(cls, values): - """Validate that PIDs are provided when request_type is stop and stop_level is process""" + """Validate target requirements for host- and workload-level requests.""" request_type = values.get("request_type") stop_level = values.get("stop_level") target_hosts = values.get("target_hosts") + target_entities = values.get("target_entities") or [] + + if not target_hosts and not target_entities: + raise ValueError("At least one target_hosts entry or target_entities selector must be provided") if request_type == "stop" and stop_level == "process": # Check if PIDs are provided in target_hosts mapping has_pids = target_hosts and any(pids for pids in target_hosts.values() if pids) - if not has_pids: + has_entity_targets = len(target_entities) > 0 + if not has_pids and not has_entity_targets: raise ValueError( - 'At least one PID must be provided when request_type is "stop" and stop_level is "process"' + 'At least one PID or workload selector must be provided when request_type is "stop" and stop_level is "process"' ) # Validate if a process id is provided when request_type is stop and stop_level is host, if so raises if request_type == "stop" and stop_level == "host": has_pids = target_hosts and any(pids for pids in target_hosts.values() if pids is not None) - if has_pids: + has_process_entities = any(entity.pid is not None for entity in target_entities) + if has_pids or has_process_entities: raise ValueError('No PIDs should be provided when request_type is "stop" and stop_level is "host"') return values @@ -166,17 +218,67 @@ class ProfilingResponse(BaseModel): estimated_completion_time: Optional[datetime] = None +class BulkProfilingRequest(BaseModel): + """Model for bulk profiling request parameters""" + + requests: List[ProfilingRequest] + dry_run: Optional[bool] = False + + @validator("requests") + def validate_requests_not_empty(cls, v): + if len(v) == 0: + raise ValueError("requests list cannot be empty") + return v + + @root_validator + def apply_bulk_dry_run(cls, values): + """Apply bulk-level dry_run to all individual requests, overwriting their dry_run values""" + bulk_dry_run = values.get("dry_run", False) + requests = values.get("requests", []) + + # Overwrite dry_run for each individual request with the bulk-level dry_run + for request in requests: + request.dry_run = bulk_dry_run + + return values + + +class BulkProfilingRequestResult(BaseModel): + """Individual result for a bulk profiling request item""" + + index: int + service_name: str + success: bool + response: Optional[ProfilingResponse] = None + error: Optional[str] = None + + +class BulkProfilingResponse(BaseModel): + """Response model for bulk profiling requests""" + + total_submitted: int + successful_count: int + failed_count: int + results: List[BulkProfilingRequestResult] + + class HeartbeatRequest(BaseModel): """Model for host heartbeat request""" ip_address: str hostname: str service_name: str + agent_version: Optional[str] = None + run_mode: Optional[str] = None + namespace: Optional[str] = None + pod_name: Optional[str] = None + containers: Optional[List[HeartbeatContainerInfo]] = None last_command_id: Optional[str] = None received_command_ids: Optional[List[str]] = None executed_command_ids: Optional[List[str]] = None status: str = "active" # active, idle, error timestamp: Optional[datetime] = None + perf_supported_events: Optional[List[str]] = None # Changed to match agent format class HeartbeatResponse(BaseModel): @@ -207,7 +309,7 @@ def validate_status(cls, v): class ProfilingHostStatusRequest(BaseModel): """Model for profiling host status request parameters""" - + service_name: Optional[List[str]] = None exact_match: bool = False hostname: Optional[List[str]] = None @@ -226,3 +328,95 @@ class ProfilingHostStatus(BaseModel): command_type: str profiling_status: str heartbeat_timestamp: datetime + + +class ProfilingHostStatusResponse(BaseModel): + """Response model for profiling host status with counts""" + hosts: List[ProfilingHostStatus] + active_count: int # Hosts with heartbeat in last 2 minutes + total_count: int # total number of hosts for the selected service + + +class ProfilingInventoryStatusRequest(BaseModel): + """Model for workload inventory status request parameters.""" + + scope: str = "host" + service_name: Optional[List[str]] = None + exact_match: bool = False + hostname: Optional[List[str]] = None + ip_address: Optional[List[str]] = None + namespace: Optional[List[str]] = None + pod_name: Optional[List[str]] = None + container_name: Optional[List[str]] = None + workload_name: Optional[List[str]] = None + process_name: Optional[List[str]] = None + profiling_status: Optional[List[str]] = None + command_type: Optional[List[str]] = None + pids: Optional[List[int]] = None + page: int = 0 + page_size: int = 50 + sort_by: Optional[str] = None + sort_order: str = "asc" + + @validator("scope") + def validate_scope(cls, v): + if v not in ["service", "namespace", "host", "pod", "container", "process"]: + raise ValueError('scope must be one of "service", "namespace", "host", "pod", "container", or "process"') + return v + + @validator("page") + def validate_page(cls, v): + return v if v and v > 0 else 0 + + @validator("page_size") + def validate_page_size(cls, v): + if not v or v <= 0: + return 50 + return min(v, 200) + + @validator("sort_order") + def validate_sort_order(cls, v): + v = (v or "asc").lower() + if v not in ("asc", "desc"): + raise ValueError('sort_order must be "asc" or "desc"') + return v + + +class ProfilingInventoryStatus(CamelModel): + id: str + scope: str + service_name: str + namespace: Optional[str] = None + hostname: Optional[str] = None + ip_address: Optional[str] = None + pod_name: Optional[str] = None + container_name: Optional[str] = None + workload_name: Optional[str] = None + workload_kind: Optional[str] = None + process_name: Optional[str] = None + pid: Optional[int] = None + pids: Optional[List[int]] = None + active_hosts: Optional[int] = None + host_count: Optional[int] = None + namespace_count: Optional[int] = None + pod_count: Optional[int] = None + container_count: Optional[int] = None + process_count: Optional[int] = None + command_type: Optional[str] = None + profiling_status: Optional[str] = None + profiling_mode: Optional[str] = None + frequency: Optional[int] = None + profiler_summary: Optional[str] = None + heartbeat_timestamp: Optional[datetime] = None + agent_version: Optional[str] = None + run_mode: Optional[str] = None + + +class ProfilingInventoryStatusResponse(CamelModel): + scope: str + rows: List[ProfilingInventoryStatus] + tab_counts: Dict[str, int] + active_hosts: int + total_count: int # total groups matching the filter (not just the current page) + page: int + page_size: int diff --git a/src/gprofiler/backend/routers/__init__.py b/src/gprofiler/backend/routers/__init__.py index 9bddeebe..34775018 100644 --- a/src/gprofiler/backend/routers/__init__.py +++ b/src/gprofiler/backend/routers/__init__.py @@ -23,6 +23,7 @@ metrics_routes, minesweeper_routes, overview_routes, + perfspect_routes, profiles_routes, services_routes, ) @@ -40,3 +41,4 @@ router.include_router(filters_routes.router, prefix="/v1/filters", tags=["filters"]) router.include_router(overview_routes.router, prefix="/overview", tags=["overview"]) router.include_router(minesweeper_routes.router, prefix="/snapshots", tags=["snapshots"]) +router.include_router(perfspect_routes.router, prefix="/perfspect", tags=["perfspect"]) diff --git a/src/gprofiler/backend/routers/metrics_routes.py b/src/gprofiler/backend/routers/metrics_routes.py index b9650681..06faa79a 100644 --- a/src/gprofiler/backend/routers/metrics_routes.py +++ b/src/gprofiler/backend/routers/metrics_routes.py @@ -17,13 +17,16 @@ import json import math import uuid -from datetime import datetime, timedelta +from datetime import datetime, timedelta, timezone from logging import getLogger from typing import List, Optional from backend.models.filters_models import FilterTypes from backend.models.flamegraph_models import FGParamsBaseModel from backend.models.metrics_models import ( + BulkProfilingRequest, + BulkProfilingRequestResult, + BulkProfilingResponse, CommandCompletionRequest, CpuMetric, CpuTrend, @@ -35,20 +38,27 @@ MetricNodesAndCores, MetricNodesCoresSummary, MetricSummary, + ProfilingInventoryStatusRequest, + ProfilingInventoryStatusResponse, ProfilingHostStatus, ProfilingHostStatusRequest, + ProfilingHostStatusResponse, ProfilingRequest, ProfilingResponse, SampleCount, ) -from backend.utils.filters_utils import get_rql_all_eq_values, get_rql_first_eq_key, get_rql_only_for_one_key +from backend.utils.dynamic_profiling_utils import validate_profiling_capacity, validate_pmu_events, validate_async_profiler_config +from backend.utils.filters_utils import get_rql_first_eq_key, get_rql_only_for_one_key, get_rql_all_eq_values from backend.utils.notifications import SlackNotifier from backend.utils.request_utils import flamegraph_base_request_params, get_metrics_response, get_query_response from botocore.exceptions import ClientError from fastapi import APIRouter, Depends, HTTPException, Query -from fastapi.responses import Response +from fastapi.responses import JSONResponse, Response from gprofiler_dev import S3ProfileDal +from gprofiler_dev import config as _dev_config from gprofiler_dev.postgres.db_manager import DBManager + +# Adhoc profiling models from pydantic import BaseModel logger = getLogger(__name__) @@ -101,14 +111,10 @@ def get_time_interval_value(start_time: datetime, end_time: datetime, interval: def profiling_host_status_params( service_name: Optional[List[str]] = Query(None, description="Filter by service name(s)"), - exact_match: bool = Query( - False, description="Use exact match for service name (default: false for partial matching)" - ), + exact_match: bool = Query(False, description="Use exact match for service name (default: false for partial matching)"), hostname: Optional[List[str]] = Query(None, description="Filter by hostname(s)"), ip_address: Optional[List[str]] = Query(None, description="Filter by IP address(es)"), - profiling_status: Optional[List[str]] = Query( - None, description="Filter by profiling status(es) (e.g., pending, completed, stopped)" - ), + profiling_status: Optional[List[str]] = Query(None, description="Filter by profiling status(es) (e.g., pending, completed, stopped)"), command_type: Optional[List[str]] = Query(None, description="Filter by command type(s) (e.g., start, stop)"), pids: Optional[List[int]] = Query(None, description="Filter by PIDs"), ) -> ProfilingHostStatusRequest: @@ -123,6 +129,46 @@ def profiling_host_status_params( ) +def profiling_inventory_status_params( + scope: str = Query("host", description="Inventory scope: service, namespace, host, pod, container, process"), + service_name: Optional[List[str]] = Query(None, description="Filter by service name(s)"), + exact_match: bool = Query(False, description="Use exact match for service name (default: false for partial matching)"), + hostname: Optional[List[str]] = Query(None, description="Filter by hostname(s)"), + ip_address: Optional[List[str]] = Query(None, description="Filter by IP address(es)"), + namespace: Optional[List[str]] = Query(None, description="Filter by namespace(s)"), + pod_name: Optional[List[str]] = Query(None, description="Filter by pod name(s)"), + container_name: Optional[List[str]] = Query(None, description="Filter by container name(s)"), + workload_name: Optional[List[str]] = Query(None, description="Filter by workload name(s)"), + process_name: Optional[List[str]] = Query(None, description="Filter by process name(s)"), + profiling_status: Optional[List[str]] = Query(None, description="Filter by profiling status(es)"), + command_type: Optional[List[str]] = Query(None, description="Filter by command type(s)"), + pids: Optional[List[int]] = Query(None, description="Filter by PIDs"), + page: int = Query(0, ge=0, description="Zero-based page index"), + page_size: int = Query(50, ge=1, le=200, description="Rows per page (max 200)"), + sort_by: Optional[str] = Query(None, description="Column to sort by (defaults to scope key order)"), + sort_order: str = Query("asc", description="Sort direction: asc or desc"), +) -> ProfilingInventoryStatusRequest: + return ProfilingInventoryStatusRequest( + scope=scope, + service_name=service_name, + exact_match=exact_match, + hostname=hostname, + ip_address=ip_address, + namespace=namespace, + pod_name=pod_name, + container_name=container_name, + workload_name=workload_name, + process_name=process_name, + profiling_status=profiling_status, + command_type=command_type, + pids=pids, + page=page, + page_size=page_size, + sort_by=sort_by, + sort_order=sort_order, + ) + + @router.get("/instance_type_count", response_model=List[InstanceTypeCount]) def get_instance_type_count(fg_params: FGParamsBaseModel = Depends(flamegraph_base_request_params)): response = get_query_response(fg_params, lookup_for="instance_type_count") @@ -319,19 +365,46 @@ def create_profiling_request(profiling_request: ProfilingRequest) -> ProfilingRe "mode": profiling_request.profiling_mode, "target_hosts": profiling_request.target_hosts, "stop_level": profiling_request.stop_level, + "dry_run": profiling_request.dry_run, }, ) - db_manager = DBManager() + + target_entities = [entity.dict() for entity in (profiling_request.target_entities or [])] + target_scope = profiling_request.target_scope or "host" + resolved_target_hosts = profiling_request.target_hosts or {} + if target_entities or target_scope != "host": + resolved_target_hosts = db_manager.resolve_workload_targets( + service_name=profiling_request.service_name, + target_scope=target_scope, + target_entities=target_entities, + ) + if not resolved_target_hosts: + raise HTTPException( + status_code=422, + detail=f"No active targets were resolved for scope '{target_scope}' in service '{profiling_request.service_name}'", + ) + + # Handle dry run requests. + # No DB changes, just validate and return success. + if profiling_request.dry_run: + return ProfilingResponse( + success=True, + message="Dry run: Profiling request validated successfully.", + request_id=None, + command_ids=[], + estimated_completion_time=None, + ) + request_id = str(uuid.uuid4()) command_ids = [] # Track all command IDs created try: # Convert target_hosts to legacy format for database compatibility - target_hostnames = list(profiling_request.target_hosts.keys()) if profiling_request.target_hosts else None + target_hostnames = list(resolved_target_hosts.keys()) if resolved_target_hosts else None host_pid_mapping = ( - {hostname: pids for hostname, pids in profiling_request.target_hosts.items() if pids} - if profiling_request.target_hosts + {hostname: pids for hostname, pids in resolved_target_hosts.items() if pids} + if resolved_target_hosts else None ) @@ -347,6 +420,8 @@ def create_profiling_request(profiling_request: ProfilingRequest) -> ProfilingRe target_hostnames=target_hostnames, pids=None, # Deprecated field, always None host_pid_mapping=host_pid_mapping, + target_scope=target_scope, + target_entities=target_entities, additional_args=profiling_request.additional_args, ) @@ -359,8 +434,8 @@ def create_profiling_request(profiling_request: ProfilingRequest) -> ProfilingRe target_hosts = [] # Determine target hosts from target_hosts mapping - if profiling_request.target_hosts: - target_hosts = list(profiling_request.target_hosts.keys()) + if resolved_target_hosts: + target_hosts = list(resolved_target_hosts.keys()) if target_hosts: # Create commands for specific hosts @@ -391,8 +466,8 @@ def create_profiling_request(profiling_request: ProfilingRequest) -> ProfilingRe target_hosts = [] # Determine target hosts for stop commands - if profiling_request.target_hosts: - target_hosts = list(profiling_request.target_hosts.keys()) + if resolved_target_hosts: + target_hosts = list(resolved_target_hosts.keys()) if target_hosts: for hostname in target_hosts: @@ -410,8 +485,8 @@ def create_profiling_request(profiling_request: ProfilingRequest) -> ProfilingRe else: # process level stop # Get PIDs for this specific host from target_hosts mapping host_pids = None - if profiling_request.target_hosts and hostname in profiling_request.target_hosts: - host_pids = profiling_request.target_hosts[hostname] + if resolved_target_hosts and hostname in resolved_target_hosts: + host_pids = resolved_target_hosts[hostname] # Stop specific processes for this host db_manager.handle_process_level_stop( @@ -472,6 +547,161 @@ def create_profiling_request(profiling_request: ProfilingRequest) -> ProfilingRe raise HTTPException(status_code=500, detail="Internal server error while processing profiling request") +@router.post("/profile_request/bulk", response_model=BulkProfilingResponse) +def create_bulk_profiling_requests(bulk_request: BulkProfilingRequest) -> BulkProfilingResponse: + """ + Create multiple profiling requests in a single API call. + + This endpoint accepts a list of profiling requests and processes them in bulk. + It provides better efficiency for operations that need to start/stop profiling + across multiple services simultaneously. + + Benefits over multiple single requests: + - Single API call reduces network overhead + - Atomic capacity validation across all requests + - Better rate limiting control + - Partial success/failure reporting per request + + Each request in the bulk operation is validated independently, and the response + includes detailed results for each request, including successes and failures. + + Returns: + BulkProfilingResponse with individual results for each request + """ + try: + logger.info( + f"Received bulk profiling request with {len(bulk_request.requests)} requests", + extra={"total_requests": len(bulk_request.requests)}, + ) + + # Initialize database manager + db_manager = DBManager() + + # Validate profiling capacity across all requests in the bulk operation + is_valid, error_message, target_hostnames = validate_profiling_capacity( + bulk_profiling_request=bulk_request, + db_manager=db_manager, + service_name=None # Validate globally across all services + ) + + # Validate PMU events support across all requests + is_valid, error_message = validate_pmu_events( + bulk_profiling_request=bulk_request, + db_manager=db_manager + ) + + if not is_valid: + logger.warning( + f"Bulk profiling capacity validation failed: {error_message}" + ) + # Return structured error response + return JSONResponse( + status_code=422, + content={ + "detail": [ + { + "loc": ["body", "requests"], + "msg": error_message, + "type": "value_error" + } + ] + } + ) + + # Validate async profiler config across all requests + is_valid, error_message = validate_async_profiler_config( + bulk_profiling_request=bulk_request + ) + + if not is_valid: + logger.warning( + f"Async profiler config validation failed: {error_message}" + ) + return JSONResponse( + status_code=422, + content={ + "detail": [ + { + "loc": ["body", "requests"], + "msg": error_message, + "type": "value_error" + } + ] + } + ) + + logger.info( + f"Bulk profiling capacity validated successfully. Total target hosts: {len(target_hostnames)}" + ) + + results: List[BulkProfilingRequestResult] = [] + successful_count = 0 + failed_count = 0 + + # Process each request individually by calling create_profiling_request + for index, profiling_request in enumerate(bulk_request.requests): + try: + # Call the existing single request endpoint logic + response = create_profiling_request(profiling_request) + + # Record successful result + result = BulkProfilingRequestResult( + index=index, + service_name=profiling_request.service_name, + success=True, + response=response, + error=None + ) + results.append(result) + successful_count += 1 + + except HTTPException as http_exc: + # Handle HTTP exceptions from create_profiling_request + logger.warning( + f"Failed to process bulk request at index {index} for service {profiling_request.service_name}: {http_exc.detail}" + ) + result = BulkProfilingRequestResult( + index=index, + service_name=profiling_request.service_name, + success=False, + response=None, + error=str(http_exc.detail) + ) + results.append(result) + failed_count += 1 + + except Exception as e: + # Handle unexpected exceptions + logger.error( + f"Unexpected error processing bulk request at index {index} for service {profiling_request.service_name}: {str(e)}", + exc_info=True + ) + result = BulkProfilingRequestResult( + index=index, + service_name=profiling_request.service_name, + success=False, + response=None, + error=f"Unexpected error: {str(e)}" + ) + results.append(result) + failed_count += 1 + + logger.info( + f"Bulk profiling request completed: {successful_count} successful, {failed_count} failed out of {len(bulk_request.requests)} total" + ) + + return BulkProfilingResponse( + total_submitted=len(bulk_request.requests), + successful_count=successful_count, + failed_count=failed_count, + results=results + ) + + except Exception as e: + logger.error(f"Failed to process bulk profiling request: {str(e)}", exc_info=True) + raise HTTPException(status_code=500, detail="Internal server error while processing bulk profiling request") + + def _create_slack_blocks(profiling_request: ProfilingRequest, request_id: str) -> list: """ Create Slack message blocks for profiling request notifications. @@ -495,8 +725,8 @@ def _create_slack_blocks(profiling_request: ProfilingRequest, request_id: str) - "type": "section", "text": { "type": "mrkdwn", - "text": f"A new request was made to {profiling_request.request_type} a profile and the details are shown below:", - }, + "text": f"A new request was made to {profiling_request.request_type} a profile and the details are shown below:" + } }, { "type": "section", @@ -561,13 +791,37 @@ def receive_heartbeat(heartbeat: HeartbeatRequest): hostname=heartbeat.hostname, ip_address=heartbeat.ip_address, service_name=heartbeat.service_name, + agent_version=heartbeat.agent_version, + run_mode=heartbeat.run_mode, + namespace=heartbeat.namespace, + pod_name=heartbeat.pod_name, + containers=[container.dict() for container in (heartbeat.containers or [])], last_command_id=heartbeat.last_command_id, received_command_ids=heartbeat.received_command_ids, executed_command_ids=heartbeat.executed_command_ids, status=heartbeat.status, heartbeat_timestamp=heartbeat.timestamp, + supported_perf_events=heartbeat.perf_supported_events, # Use agent field name ) + # 1b. Auto-subscribe newly-registered hosts to an active service-wide + # profiling session. Hosts that join a service after a service-scoped + # request was issued (e.g. cluster autoscaling) are enrolled here so + # users do not have to re-select the service. + try: + if db_manager.auto_subscribe_host_to_service( + hostname=heartbeat.hostname, + service_name=heartbeat.service_name, + ): + logger.info( + f"Auto-subscribed host {heartbeat.hostname} to active service-wide " + f"profiling for service {heartbeat.service_name}" + ) + except Exception as e: + logger.warning( + f"Auto-subscribe check failed for {heartbeat.hostname}/{heartbeat.service_name}: {e}" + ) + # 2. Check for current profiling command for this host/service current_command = db_manager.get_current_profiling_command( hostname=heartbeat.hostname, @@ -744,7 +998,7 @@ def report_command_completion(completion: CommandCompletionRequest): raise HTTPException(status_code=500, detail="Internal server error while processing command completion") -@router.get("/profiling/host_status", response_model=List[ProfilingHostStatus]) +@router.get("/profiling/host_status", response_model=ProfilingHostStatusResponse) def get_profiling_host_status( profiling_params: ProfilingHostStatusRequest = Depends(profiling_host_status_params), ): @@ -762,7 +1016,7 @@ def get_profiling_host_status( profiling_params: ProfilingHostStatusRequest object containing all filter parameters Returns: - List of host statuses filtered by the specified criteria + ProfilingHostStatusResponse with hosts list, active count, and total count """ db_manager = DBManager() @@ -774,7 +1028,14 @@ def get_profiling_host_status( profiling_statuses=profiling_params.profiling_status, command_types=profiling_params.command_type, pids=profiling_params.pids, - exact_match=profiling_params.exact_match, + exact_match=profiling_params.exact_match + ) + + # Get total host count (all hosts for the selected service) + # This shows the total fleet size for the service + total_count = db_manager.get_total_host_count( + service_names=profiling_params.service_name, + exact_match=profiling_params.exact_match ) # Convert database results to response model @@ -813,7 +1074,37 @@ def get_profiling_host_status( ) ) - return results + return ProfilingHostStatusResponse( + hosts=results, + active_count=len(results), + total_count=total_count + ) + + +@router.get("/profiling/workload_status", response_model=ProfilingInventoryStatusResponse) +def get_profiling_workload_status( + profiling_params: ProfilingInventoryStatusRequest = Depends(profiling_inventory_status_params), +): + db_manager = DBManager() + return db_manager.get_workload_inventory_status( + scope=profiling_params.scope, + service_names=profiling_params.service_name, + hostnames=profiling_params.hostname, + ip_addresses=profiling_params.ip_address, + namespaces=profiling_params.namespace, + pod_names=profiling_params.pod_name, + container_names=profiling_params.container_name, + workload_names=profiling_params.workload_name, + process_names=profiling_params.process_name, + profiling_statuses=profiling_params.profiling_status, + command_types=profiling_params.command_type, + pids=profiling_params.pids, + exact_match=profiling_params.exact_match, + page=profiling_params.page, + page_size=profiling_params.page_size, + sort_by=profiling_params.sort_by, + sort_order=profiling_params.sort_order, + ) @router.get("/adhoc_flamegraphs", response_model=List[FlamegraphFile]) @@ -828,13 +1119,13 @@ def get_adhoc_flamegraphs( try: db_manager = DBManager() service_name = fg_params.service_name - + # Get service_id for metadata query service_id = db_manager.get_service(service_name) - + # Extract hostname filters from fg_params if present hostname_filters = get_rql_all_eq_values(fg_params.filter, FilterTypes.HOSTNAME_KEY) - + # Get metadata from database (already filtered by service, time, and hostname) metadata_list = db_manager.get_adhoc_flamegraphs_metadata( service_id=service_id, @@ -842,23 +1133,21 @@ def get_adhoc_flamegraphs( end_time=fg_params.end_time, hostname_filters=hostname_filters, ) - + # Convert metadata to FlamegraphFile objects flamegraph_files = [] for metadata in metadata_list: s3_key = metadata["s3_key"] - filename = s3_key.split("/")[-1] - - flamegraph_files.append( - FlamegraphFile( - filename=filename, - timestamp=datetime.fromisoformat(metadata["start_time"]), - hostname=metadata["hostname"], - size=metadata.get("file_size"), - s3_path=s3_key, - perf_events=metadata.get("perf_events"), - ) - ) + filename = s3_key.split('/')[-1] + + flamegraph_files.append(FlamegraphFile( + filename=filename, + timestamp=datetime.fromisoformat(metadata["start_time"]).replace(tzinfo=timezone.utc), + hostname=metadata["hostname"], + size=metadata.get("file_size"), + s3_path=s3_key, + perf_events=metadata.get("perf_events") + )) # Mark entries whose S3 file no longer exists. # All head_object calls are issued in parallel (one thread per key) @@ -871,7 +1160,7 @@ def get_adhoc_flamegraphs( f.removed = f.s3_path not in existing_keys return flamegraph_files - + except Exception as e: logger.error(f"Error fetching adhoc flamegraphs: {e}") raise HTTPException(status_code=500, detail="Failed to fetch adhoc flamegraph files") @@ -888,21 +1177,26 @@ def get_adhoc_flamegraph_content( """ try: s3_dal = S3ProfileDal(logger) - + # Build full S3 path for flamegraph HTML files - s3_path = f"products/{service_name}/stacks/flamegraph/{filename}" - + _prefix = _dev_config.S3_PATH_PREFIX + _root = f"{_prefix}/products" if _prefix else "products" + s3_path = f"{_root}/{service_name}/stacks/flamegraph/{filename}" + # Fetch file content from S3 (flamegraph HTML files are not gzipped) try: html_content = s3_dal.get_object(s3_path, is_gzip=False) except ClientError as e: - if e.response["Error"]["Code"] == "NoSuchKey": + if e.response['Error']['Code'] == 'NoSuchKey': raise HTTPException(status_code=404, detail="Flamegraph file not found") else: raise HTTPException(status_code=500, detail="Failed to fetch flamegraph content from S3") - - return FlamegraphContent(content=html_content, filename=filename) - + + return FlamegraphContent( + content=html_content, + filename=filename + ) + except HTTPException: raise except Exception as e: diff --git a/src/gprofiler/backend/utils/dynamic_profiling_utils.py b/src/gprofiler/backend/utils/dynamic_profiling_utils.py new file mode 100644 index 00000000..c5635f0e --- /dev/null +++ b/src/gprofiler/backend/utils/dynamic_profiling_utils.py @@ -0,0 +1,248 @@ +# +# Copyright (C) 2023 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from typing import List, Optional + +import bitmath + +from backend.config import MAX_SIMULTANEOUS_PROFILING_HOSTS_PERCENT, MAX_PROFILING_REQUEST_HOSTS +from backend.models.metrics_models import BulkProfilingRequest + + +def validate_profiling_capacity( + bulk_profiling_request: BulkProfilingRequest, + db_manager, + service_name: Optional[str] = None +) -> tuple[bool, Optional[str], List[str]]: + """ + Validate that the bulk profiling request doesn't exceed the maximum simultaneous profiling capacity. + + This function aggregates all target hosts across all requests in the bulk operation and validates + the total capacity requirements. + + Args: + bulk_profiling_request: The BulkProfilingRequest object containing multiple requests + db_manager: Instance of DBManager to query active profiling hosts + service_name: Optional service name to filter by (if None, checks globally) + + Returns: + tuple: (is_valid: bool, error_message: Optional[str], target_hostnames: List[str]) + - is_valid: True if capacity check passes + - error_message: Error description if validation fails + - target_hostnames: List of all unique hostnames from all requests + + Example: + >>> from gprofiler_dev.postgres.db_manager import DBManager + >>> db_manager = DBManager() + >>> is_valid, error, hostnames = validate_profiling_capacity(bulk_request, db_manager) + >>> if not is_valid: + ... raise ValueError(error) + """ + # Collect all target hostnames from all requests + all_target_hostnames: List[str] = [] + has_start_requests = False + + for profiling_request in bulk_profiling_request.requests: + # Track if there are any "start" requests + if profiling_request.request_type == "start": + has_start_requests = True + + # Collect all hostnames from target_hosts + if profiling_request.target_hosts: + all_target_hostnames.extend(profiling_request.target_hosts.keys()) + + # Only validate capacity for bulk requests that contain "start" operations + if not has_start_requests: + return True, None, all_target_hostnames + + # Calculate total request size from all collected hostnames + request_size = len(all_target_hostnames) + + # Validate that request size doesn't exceed MAX_PROFILING_REQUEST_HOSTS + if request_size > MAX_PROFILING_REQUEST_HOSTS: + error_msg = ( + f"Request size exceeded.\n" + f"Request size: {request_size} hosts\n" + f"Maximum allowed per request: {MAX_PROFILING_REQUEST_HOSTS} hosts\n" + f"Please reduce the number of hosts in your request by {request_size - MAX_PROFILING_REQUEST_HOSTS} hosts." + ) + return False, error_msg, all_target_hostnames + + # Get counts, excluding hosts from the current bulk request + active_hosts_count = db_manager.get_active_hosts_count(service_name) + currently_profiling_host_count = db_manager.get_actively_profiling_hosts_count( + service_name=service_name, + ) + currently_profiling_host_count_inside_selection = db_manager.get_actively_profiling_hosts_count( + service_name=service_name, + host_inclusion_list=all_target_hostnames + ) + currently_profiling_host_count_outside_selection = db_manager.get_actively_profiling_hosts_count( + service_name=service_name, + host_exclusion_list=all_target_hostnames + ) + + # Calculate maximum allowed profiling hosts + max_profiling_hosts = int((active_hosts_count * MAX_SIMULTANEOUS_PROFILING_HOSTS_PERCENT) / 100) + + # Calculate new total if this request is approved + new_profiling_total = currently_profiling_host_count_outside_selection + request_size + + # Check if it would exceed the limit + if new_profiling_total > max_profiling_hosts: + error_msg = ( + f"Profiling capacity exceeded.\n" + f"Currently profiling: {currently_profiling_host_count} hosts\n" + f"Currently profiling inside selection: {currently_profiling_host_count_inside_selection} hosts\n" + f"Currently profiling outside selection: {currently_profiling_host_count_outside_selection} hosts\n" + f"Request size: {request_size} hosts\n" + f"Active hosts: {active_hosts_count}\n" + f"Maximum allowed ({MAX_SIMULTANEOUS_PROFILING_HOSTS_PERCENT}%): {max_profiling_hosts} hosts\n" + f"This request would result in {new_profiling_total} profiling hosts, " + f"which exceeds the limit by {new_profiling_total - max_profiling_hosts} hosts." + ) + return False, error_msg, all_target_hostnames + + return True, None, all_target_hostnames + + +def validate_pmu_events( + bulk_profiling_request: BulkProfilingRequest, + db_manager +) -> tuple[bool, Optional[str]]: + """ + Validate that all hosts in the bulk profiling request support the requested PMU events. + + This function validates PMU event support for all "start" requests with perf enabled. + It checks each service's requested events against host capabilities stored in the database. + + Args: + bulk_profiling_request: The BulkProfilingRequest object containing multiple requests + db_manager: Instance of DBManager to query host PMU capabilities + + Returns: + tuple: (is_valid: bool, error_message: Optional[str]) + - is_valid: True if all hosts support requested events + - error_message: Combined error messages if validation fails for any service + + Example: + >>> from gprofiler_dev.postgres.db_manager import DBManager + >>> db_manager = DBManager() + >>> is_valid, error = validate_pmu_events(bulk_request, db_manager) + >>> if not is_valid: + ... raise ValueError(error) + """ + pmu_validation_errors = [] + + for profiling_request in bulk_profiling_request.requests: + # Only validate "start" requests with additional_args + if profiling_request.request_type == "start" and profiling_request.additional_args: + profiler_configs = profiling_request.additional_args.get("profiler_configs", {}) + perf_config = profiler_configs.get("perf", {}) + perf_mode = perf_config.get("mode", "disabled") + perf_events = perf_config.get("events", []) + + # Only validate if perf is enabled and events are specified + if perf_mode != "disabled" and perf_events: + # target_hosts is only populated for host-scope requests. For + # workload scopes (service/namespace/pod/container/process) the + # UI sends no target_hosts because the concrete hosts are + # resolved later; fall back to None so events are validated + # against all of the service's active hosts instead of crashing + # on None.keys(). + target_hostnames_for_service = ( + list(profiling_request.target_hosts.keys()) + if profiling_request.target_hosts + else None + ) + + validation_result = db_manager.validate_perf_events_support( + service_name=profiling_request.service_name, + requested_events=perf_events, + target_hostnames=target_hostnames_for_service + ) + + if not validation_result["valid"]: + error_msg = validation_result["error_message"] + pmu_validation_errors.append(f"Service '{profiling_request.service_name}': {error_msg}") + + # If PMU validation failed for any service, return combined error + if pmu_validation_errors: + combined_error = "\n\n".join(pmu_validation_errors) + return False, combined_error + + return True, None + + +_VALID_AP_TIME_MODES = frozenset({"cpu", "itimer", "wall", "auto", "alloc"}) + + +def validate_async_profiler_config(bulk_profiling_request: BulkProfilingRequest) -> tuple[bool, Optional[str]]: + """ + Validate the async_profiler config in each profiling request's additional_args. + + Checks that: + - async_profiler.time is one of the supported modes + - alloc_interval is a non-empty string when time == 'alloc' + + Returns: + tuple: (is_valid: bool, error_message: Optional[str]) + """ + errors = [] + + for profiling_request in bulk_profiling_request.requests: + if profiling_request.request_type != "start" or not profiling_request.additional_args: + continue + + profiler_configs = profiling_request.additional_args.get("profiler_configs", {}) + async_profiler_config = profiler_configs.get("async_profiler") + if not isinstance(async_profiler_config, dict): + continue + + if not async_profiler_config.get("enabled", True): + continue + + time_mode = async_profiler_config.get("time", "cpu") + if time_mode not in _VALID_AP_TIME_MODES: + errors.append( + f"Service '{profiling_request.service_name}': " + f"Invalid async_profiler time mode {time_mode!r}. " + f"Valid modes: {sorted(_VALID_AP_TIME_MODES)}" + ) + continue + + if time_mode == "alloc": + alloc_interval = async_profiler_config.get("alloc_interval", "") + if not isinstance(alloc_interval, str) or not alloc_interval: + errors.append( + f"Service '{profiling_request.service_name}': " + f"Invalid alloc_interval value {alloc_interval!r}: " + "must be a non-empty string (e.g. '2MB', '512KiB')" + ) + continue + try: + bitmath.parse_string(alloc_interval) + except ValueError: + errors.append( + f"Service '{profiling_request.service_name}': " + f"Could not parse alloc_interval {alloc_interval!r}: " + "must be a number followed by a size unit (e.g. '2MB', '512KiB')" + ) + + if errors: + return False, "\n\n".join(errors) + + return True, None diff --git a/src/gprofiler/backend/utils/filters_utils.py b/src/gprofiler/backend/utils/filters_utils.py index 3ab95fe9..9baed4aa 100644 --- a/src/gprofiler/backend/utils/filters_utils.py +++ b/src/gprofiler/backend/utils/filters_utils.py @@ -59,22 +59,22 @@ def get_rql_only_for_one_key(rql_filter: Optional[RQLFilter], key: FilterTypes) def get_rql_all_eq_values(rql_filter: Optional[RQLFilter], key: FilterTypes) -> List[str]: """ Extract all values for a given key with equality comparison operator from RQL filter. - + Args: rql_filter: The RQL filter object key: The filter type key to extract values for - + Returns: List of all values found for the key with eq operator """ if rql_filter is None: return [] - + values = [] for log_op in rql_filter.filter.values(): for filter_type in log_op: if key.value in filter_type: if RQLCompareOperators.eq_op in filter_type[key]: values.append(filter_type[key][RQLCompareOperators.eq_op]) - + return values diff --git a/src/gprofiler/backend/utils/metrics_publisher.py b/src/gprofiler/backend/utils/metrics_publisher.py new file mode 100644 index 00000000..1d3c86fe --- /dev/null +++ b/src/gprofiler/backend/utils/metrics_publisher.py @@ -0,0 +1,342 @@ +""" +Metrics Publisher - Singleton class for publishing metrics to metrics agent. + +This module provides a thread-safe singleton class for publishing SLI (Service Level +Indicator) metrics to a metrics agent for monitoring and alerting. + +Metric Types: + - SLI Metrics (Primary): Track success/failure rates for SLO monitoring + - Error Metrics: Available for operational error tracking (not currently used) + +Usage: + # Initialize once (typically in main application startup) + metrics_publisher = MetricsPublisher( + server_url="tcp://localhost:18126", + service_name="gprofiler-backend", + sli_metric_uuid="your-uuid-here", + enabled=True + ) + + # Use anywhere in your code - get_instance() always returns valid object + MetricsPublisher.get_instance().send_sli_metric( + "success", + "profile_upload", + {"service": "devapp"} + ) +""" + +import logging +import socket +import threading +import time +from typing import Dict, Optional, Any +from enum import Enum + +logger = logging.getLogger(__name__) + + +class MetricCategory(str, Enum): + """Categories for error metrics - Backend specific""" + API = "api" # API endpoint errors + DATABASE = "database" # Database connection/query errors + STORAGE = "storage" # S3/SQS storage errors + AUTHENTICATION = "authentication" # Auth/API key validation errors + EXTERNAL_SERVICE = "external_service" # Calls to external services (if any) + + +# Constants for SLI response types +RESPONSE_TYPE_SUCCESS = "success" +RESPONSE_TYPE_FAILURE = "failure" +RESPONSE_TYPE_IGNORED_FAILURE = "ignored_failure" + + +class NoopMetricsPublisher: + """ + No-op metrics publisher for graceful degradation. + Used when MetricsPublisher is not initialized. All methods do nothing. + Matches agent's pattern for handling uninitialized state. + """ + def send_error_metric(self, *args, **kwargs) -> bool: + return False + + def send_sli_metric(self, *args, **kwargs) -> bool: + return False + + def flush_and_close(self): + pass + + +class MetricsPublisher: + """ + Thread-safe singleton class for publishing metrics to a metrics agent. + + Supports two types of metrics: + 1. SLI Metrics (Primary): Service Level Indicator tracking for SLO monitoring + Pattern: error-budget.counters.{uuid}{response_type=*, method_name=*} + + 2. Error Metrics: Operational error tracking (available but not currently used) + Pattern: gprofiler.{category}.{error_type}.error + """ + + _instance: Optional['MetricsPublisher'] = None + _lock = threading.Lock() + + def __new__(cls, *args, **kwargs): + """Thread-safe singleton implementation""" + if cls._instance is None: + with cls._lock: + if cls._instance is None: + cls._instance = super(MetricsPublisher, cls).__new__(cls) + return cls._instance + + def __init__( + self, + server_url: str = "tcp://localhost:18126", + service_name: str = "gprofiler-backend", + sli_metric_uuid: Optional[str] = None, + enabled: bool = True + ): + """ + Initialize MetricsPublisher with configuration. + + Args: + server_url: TCP endpoint URL for metrics agent (default: tcp://localhost:18126) + service_name: Service name for metric tagging + sli_metric_uuid: UUID for SLI metrics (required for SLI tracking) + enabled: Whether metrics publishing is enabled + """ + # Only initialize once (singleton pattern) + if hasattr(self, '_initialized'): + return + + self._initialized = True + self._enabled = enabled + self._service_name = service_name + self._server_url = server_url + self._sli_metric_uuid = sli_metric_uuid + self._connection_failed = False + self._last_error_log_time = 0 + self._error_log_interval = 300 # Log connection errors at most once per 5 minutes + + # Parse server URL (only matters if enabled) + if server_url.startswith('tcp://'): + url_parts = server_url[6:].split(':') + self.host = url_parts[0] + self.port = int(url_parts[1]) if len(url_parts) > 1 else 18126 + else: + # If disabled, don't raise error for invalid URL + if enabled: + raise ValueError(f"Unsupported server URL format: {server_url}") + else: + self.host = "localhost" + self.port = 18126 + + if enabled: + logger.info( + f"MetricsPublisher initialized: service={service_name}, " + f"server={self.host}:{self.port}, sli_enabled={sli_metric_uuid is not None}" + ) + else: + logger.info("MetricsPublisher disabled") + + @classmethod + def get_instance(cls) -> 'MetricsPublisher': + """ + Get the MetricsPublisher singleton instance. + + Returns the instance even if disabled - methods check _enabled internally. + get_instance() is always safe to call. + + Returns: + MetricsPublisher instance (or NoopMetricsPublisher if not initialized) + """ + if cls._instance is None: + # Return noop instance if not initialized (graceful degradation) + return NoopMetricsPublisher() + + return cls._instance + + def _send_metric(self, metric_line: str) -> bool: + """ + Send a metric line to the metrics agent using TCP socket. + + Args: + metric_line: Formatted metric string + + Returns: + True if sent successfully, False otherwise + """ + if not self._enabled or not self.host or not self.port: + return False + + try: + # Create TCP connection and send metric (agent pattern) + with socket.create_connection((self.host, self.port), timeout=1.0) as sock: + # Ensure message ends with newline + message = metric_line if metric_line.endswith('\n') else metric_line + '\n' + sock.sendall(message.encode('utf-8')) + + # Reset connection failed flag on success + if self._connection_failed: + self._connection_failed = False + logger.info("Metrics agent connection restored") + + return True + + except Exception as e: + # Rate-limit error logging to avoid log spam + current_time = time.time() + if not self._connection_failed or (current_time - self._last_error_log_time) > self._error_log_interval: + logger.warning(f"Failed to send metric: {e}") + self._last_error_log_time = current_time + self._connection_failed = True + + return False + + def send_error_metric( + self, + category: str, + error_type: str, + tags: Optional[Dict[str, Any]] = None + ) -> bool: + """ + Send an operational error metric. + + Pattern: gprofiler.{category}.{error_type}.error + + Args: + category: Error category (e.g., "api", "database", "storage", "authentication") + error_type: Specific error type (e.g., "connection_failed", "timeout") + tags: Optional tags to include with the metric (e.g., {"endpoint": "/api/profiles"}) + + Returns: + True if sent successfully, False otherwise + + Example: + # Backend-specific error metrics + publisher.send_error_metric("database", "connection_timeout", {"host": "db-primary"}) + publisher.send_error_metric("storage", "s3_upload_failed", {"bucket": "profiles"}) + publisher.send_error_metric("api", "endpoint_timeout", {"endpoint": "/api/v2/profiles"}) + """ + if not self._enabled: + return False + + # Build metric name + metric_name = f"gprofiler.{category}.{error_type}.error" + + # Get current epoch timestamp + timestamp = int(time.time()) + + # Build tag string (Graphite plaintext protocol format) + tag_parts = [f"service={self._service_name}"] + if tags: + for key, value in tags.items(): + tag_parts.append(f"{key}={value}") + tag_string = " ".join(tag_parts) + + # Format: put metric_name timestamp value tag1=value1 tag2=value2 ... + metric_line = f"put {metric_name} {timestamp} 1 {tag_string}" + + logger.debug(f"Sending error metric: {metric_line}") + return self._send_metric(metric_line) + + def send_sli_metric( + self, + response_type: str, + method_name: str, + extra_tags: Optional[Dict[str, Any]] = None + ) -> bool: + """ + Send an SLI (Service Level Indicator) metric for SLO tracking. + + Pattern: error-budget.counters.{uuid}{response_type=*, method_name=*} + + Args: + response_type: Type of response ("success", "failure", "ignored_failure") + method_name: Name of the method/operation (e.g., "profile_upload", "send_heartbeat") + extra_tags: Optional additional tags (matches agent parameter name) + + Returns: + True if sent successfully, False otherwise + + Example: + # Backend SLI metrics + publisher.send_sli_metric("success", "profile_upload", {"service": "devapp"}) + publisher.send_sli_metric("failure", "send_heartbeat", {"status_code": 500}) + publisher.send_sli_metric("ignored_failure", "profile_upload", {"reason": "authentication_failed"}) + """ + if not self._enabled or not self._sli_metric_uuid: + return False + + # Build metric name using configured SLI UUID + # Format: error-budget.counters.{uuid} + # Example: error-budget.counters.test-sli-uuid-12345 + metric_name = f"error-budget.counters.{self._sli_metric_uuid}" + + # Get current epoch timestamp + timestamp = int(time.time()) + + # Build tag string with required SLI tags (Graphite plaintext protocol format) + tag_parts = [ + f"service={self._service_name}", + f"response_type={response_type}", + f"method_name={method_name}" + ] + + if extra_tags: + for key, value in extra_tags.items(): + tag_parts.append(f"{key}={value}") + + tag_string = " ".join(tag_parts) + + # Format: put metric_name timestamp value tag1=value1 tag2=value2 ... + metric_line = f"put {metric_name} {timestamp} 1 {tag_string}" + + # Log at INFO level for verification (shows actual metric being sent) + logger.info(f"📊 Sending SLI metric: {metric_line}") + return self._send_metric(metric_line) + + def flush_and_close(self): + """ + Flush any pending metrics and close the publisher. + + Note: Backend sends metrics synchronously over TCP (no buffering), + so there's nothing to flush, but we keep the method name consistent + with the agent's interface for easier code review. + + This should be called during application shutdown. + """ + with self._lock: + logger.info("MetricsPublisher closed") + self._enabled = False + + def __del__(self): + """Cleanup on deletion""" + try: + self.flush_and_close() + except Exception: + pass # Ignore errors during cleanup + + +# Convenience functions for common use cases +def send_error_metric(category: str, error_type: str, tags: Optional[Dict[str, Any]] = None) -> bool: + """ + Convenience function to send an error metric. + + Usage: + from backend.utils.metrics_publisher import send_error_metric + send_error_metric("database", "connection_failed", {"host": "db-1"}) + """ + return MetricsPublisher.get_instance().send_error_metric(category, error_type, tags) + + +def send_sli_metric(response_type: str, method_name: str, extra_tags: Optional[Dict[str, Any]] = None) -> bool: + """ + Convenience function to send an SLI metric. + + Usage: + from backend.utils.metrics_publisher import send_sli_metric + send_sli_metric("success", "profile_upload", {"service": "devapp"}) + """ + return MetricsPublisher.get_instance().send_sli_metric(response_type, method_name, extra_tags) + diff --git a/src/gprofiler/backend/utils/notifications.py b/src/gprofiler/backend/utils/notifications.py index 97ee6210..d9a4f3e9 100644 --- a/src/gprofiler/backend/utils/notifications.py +++ b/src/gprofiler/backend/utils/notifications.py @@ -3,11 +3,12 @@ """ import logging -from typing import Any, Dict, List, Optional, Union +from typing import Dict, List, Optional, Any, Union +from slack_sdk import WebClient +from slack_sdk.errors import SlackApiError + +from ..config import SLACK_BOT_TOKEN, SLACK_CHANNELS, DEFAULT_SLACK_CHANNELS -from backend.config import DEFAULT_SLACK_CHANNELS, SLACK_BOT_TOKEN, SLACK_CHANNELS -from slack_sdk import WebClient # type: ignore -from slack_sdk.errors import SlackApiError # type: ignore logger = logging.getLogger(__name__) @@ -15,15 +16,15 @@ class SlackNotifier: """ A utility class for sending notifications to Slack channels using the Slack SDK. - + This class provides methods to send various types of messages to Slack channels, including basic text messages and rich messages with blocks and attachments. """ - + def __init__(self, token: Optional[str] = None, default_channel: Optional[str] = None): """ Initialize the SlackNotifier with a bot token. - + Args: token: Slack bot token (starts with 'xoxb-'). If not provided, reads from SLACK_BOT_TOKEN environment variable. default_channel: Default channel to send messages to (e.g., '#general', '@user', 'C1234567890'). @@ -31,14 +32,12 @@ def __init__(self, token: Optional[str] = None, default_channel: Optional[str] = """ # Use provided token or fall back to config bot_token = token or SLACK_BOT_TOKEN - + if not bot_token: - raise ValueError( - "Slack bot token must be provided either as parameter or via SLACK_BOT_TOKEN environment variable" - ) - + raise ValueError("Slack bot token must be provided either as parameter or via SLACK_BOT_TOKEN environment variable") + self.client = WebClient(token=bot_token) - + # Set default channel - use provided, or first from config, or fallback to hardcoded default if default_channel: self.default_channel = default_channel @@ -46,7 +45,7 @@ def __init__(self, token: Optional[str] = None, default_channel: Optional[str] = self.default_channel = SLACK_CHANNELS[0].strip() else: self.default_channel = DEFAULT_SLACK_CHANNELS[0] - + # Test the connection try: response = self.client.auth_test() @@ -54,22 +53,26 @@ def __init__(self, token: Optional[str] = None, default_channel: Optional[str] = except SlackApiError as e: logger.error(f"Failed to authenticate with Slack: {e.response['error']}") raise - + def send_message( - self, text: str, channel: Optional[str] = None, thread_ts: Optional[str] = None, **kwargs + self, + text: str, + channel: Optional[str] = None, + thread_ts: Optional[str] = None, + **kwargs ) -> Dict[str, Any]: """ Send a basic text message to a Slack channel. - + Args: text: The message text to send channel: Target channel (uses default_channel if not provided) thread_ts: Timestamp of parent message to reply in thread **kwargs: Additional arguments to pass to the Slack API - + Returns: Dict containing the Slack API response - + Raises: SlackApiError: If the Slack API returns an error ValueError: If no channel is specified and no default channel is set @@ -77,15 +80,20 @@ def send_message( target_channel = channel or self.default_channel if not target_channel: raise ValueError("No channel specified and no default channel set") - + try: - response = self.client.chat_postMessage(channel=target_channel, text=text, thread_ts=thread_ts, **kwargs) + response = self.client.chat_postMessage( + channel=target_channel, + text=text, + thread_ts=thread_ts, + **kwargs + ) logger.info(f"Message sent successfully to {target_channel}") return response except SlackApiError as e: logger.error(f"Failed to send message to {target_channel}: {e.response['error']}") raise - + def send_rich_message( self, blocks: List[Dict[str, Any]] = None, @@ -93,11 +101,11 @@ def send_rich_message( text: str = "", channel: Optional[str] = None, thread_ts: Optional[str] = None, - **kwargs, + **kwargs ) -> Dict[str, Any]: """ Send a rich message with blocks and/or attachments to a Slack channel. - + Args: blocks: List of block elements for rich formatting attachments: List of legacy attachments @@ -105,10 +113,10 @@ def send_rich_message( channel: Target channel (uses default_channel if not provided) thread_ts: Timestamp of parent message to reply in thread **kwargs: Additional arguments to pass to the Slack API - + Returns: Dict containing the Slack API response - + Raises: SlackApiError: If the Slack API returns an error ValueError: If no channel is specified and no default channel is set @@ -116,66 +124,76 @@ def send_rich_message( target_channel = channel or self.default_channel if not target_channel: raise ValueError("No channel specified and no default channel set") - + try: response = self.client.chat_postMessage( - channel=target_channel, text=text, blocks=blocks, attachments=attachments, thread_ts=thread_ts, **kwargs + channel=target_channel, + text=text, + blocks=blocks, + attachments=attachments, + thread_ts=thread_ts, + **kwargs ) logger.info(f"Rich message sent successfully to {target_channel}") return response except SlackApiError as e: logger.error(f"Failed to send rich message to {target_channel}: {e.response['error']}") raise - + def send_alert( self, title: str, message: str, severity: str = "info", channel: Optional[str] = None, - additional_fields: Optional[Dict[str, str]] = None, + additional_fields: Optional[Dict[str, str]] = None ) -> Dict[str, Any]: """ Send an alert message with consistent formatting. - + Args: title: Alert title message: Alert message body severity: Alert severity ('info', 'warning', 'error', 'success') channel: Target channel (uses default_channel if not provided) additional_fields: Additional fields to include in the alert - + Returns: Dict containing the Slack API response """ # Color mapping for different severities color_map = { - "info": "#36a64f", # Green - "warning": "#ff9900", # Orange - "error": "#ff0000", # Red - "success": "#36a64f", # Green + "info": "#36a64f", # Green + "warning": "#ff9900", # Orange + "error": "#ff0000", # Red + "success": "#36a64f" # Green } - + color = color_map.get(severity, "#36a64f") - + # Build attachment fields fields = [] if additional_fields: - fields = [{"title": key, "value": value, "short": True} for key, value in additional_fields.items()] - - attachments = [ - { - "color": color, - "title": title, - "text": message, - "fields": fields, - "footer": "gProfiler Performance Studio", - "ts": int(__import__("time").time()), - } - ] - - return self.send_rich_message(attachments=attachments, text=f"{title}: {message}", channel=channel) - + fields = [ + {"title": key, "value": value, "short": True} + for key, value in additional_fields.items() + ] + + attachments = [{ + "color": color, + "title": title, + "text": message, + "fields": fields, + "footer": "gProfiler Performance Studio", + "ts": int(__import__('time').time()) + }] + + return self.send_rich_message( + attachments=attachments, + text=f"{title}: {message}", + channel=channel + ) + def send_performance_alert( self, service_name: str, @@ -183,11 +201,11 @@ def send_performance_alert( current_value: Union[str, float], threshold: Union[str, float], severity: str = "warning", - channel: Optional[str] = None, + channel: Optional[str] = None ) -> Dict[str, Any]: """ Send a performance-related alert with structured data. - + Args: service_name: Name of the service experiencing issues metric_name: Name of the performance metric @@ -195,64 +213,72 @@ def send_performance_alert( threshold: Threshold that was exceeded severity: Alert severity level channel: Target channel (uses default_channel if not provided) - + Returns: Dict containing the Slack API response """ title = f"Performance Alert: {service_name}" message = f"Metric `{metric_name}` has exceeded threshold" - + additional_fields = { "Service": service_name, "Metric": metric_name, "Current Value": str(current_value), "Threshold": str(threshold), - "Severity": severity.upper(), + "Severity": severity.upper() } - + return self.send_alert( - title=title, message=message, severity=severity, channel=channel, additional_fields=additional_fields + title=title, + message=message, + severity=severity, + channel=channel, + additional_fields=additional_fields ) - + def update_message( self, ts: str, channel: str, text: Optional[str] = None, blocks: Optional[List[Dict[str, Any]]] = None, - attachments: Optional[List[Dict[str, Any]]] = None, + attachments: Optional[List[Dict[str, Any]]] = None ) -> Dict[str, Any]: """ Update an existing message. - + Args: ts: Timestamp of the message to update channel: Channel containing the message text: New text content blocks: New block elements attachments: New attachments - + Returns: Dict containing the Slack API response """ try: response = self.client.chat_update( - ts=ts, channel=channel, text=text, blocks=blocks, attachments=attachments + ts=ts, + channel=channel, + text=text, + blocks=blocks, + attachments=attachments ) logger.info(f"Message updated successfully in {channel}") return response except SlackApiError as e: logger.error(f"Failed to update message in {channel}: {e.response['error']}") raise - + def delete_message(self, ts: str, channel: str) -> Dict[str, Any]: """ Delete a message. - + Args: ts: Timestamp of the message to delete channel: Channel containing the message - + Returns: Dict containing the Slack API response """ @@ -263,95 +289,110 @@ def delete_message(self, ts: str, channel: str) -> Dict[str, Any]: except SlackApiError as e: logger.error(f"Failed to delete message from {channel}: {e.response['error']}") raise - + def get_available_channels(self) -> List[str]: """ Get the list of available channels from configuration. - + Returns: List of channel names configured via SLACK_CHANNELS environment variable or the default channels if not configured. """ return [channel.strip() for channel in SLACK_CHANNELS if channel.strip()] - + def is_valid_channel(self, channel: str) -> bool: """ Check if a channel is in the list of available channels. - + Args: channel: Channel name to validate - + Returns: True if channel is in the available channels list, False otherwise """ available_channels = self.get_available_channels() return channel in available_channels - - def send_to_all_channels(self, text: str, thread_ts: Optional[str] = None, **kwargs) -> List[Dict[str, Any]]: + + def send_to_all_channels( + self, + text: str, + thread_ts: Optional[str] = None, + **kwargs + ) -> List[Dict[str, Any]]: """ Send a message to all configured channels. - + Args: text: The message text to send thread_ts: Timestamp of parent message to reply in thread **kwargs: Additional arguments to pass to the Slack API - + Returns: List of responses from each channel - + Raises: SlackApiError: If the Slack API returns an error for any channel """ responses = [] available_channels = self.get_available_channels() - + for channel in available_channels: try: - response = self.send_message(text=text, channel=channel, thread_ts=thread_ts, **kwargs) + response = self.send_message( + text=text, + channel=channel, + thread_ts=thread_ts, + **kwargs + ) responses.append(response) except SlackApiError as e: logger.error(f"Failed to send message to {channel}: {e.response['error']}") # Continue with other channels even if one fails responses.append({"error": str(e), "channel": channel}) - + return responses - + def send_rich_message_to_all_channels( self, blocks: List[Dict[str, Any]] = None, attachments: List[Dict[str, Any]] = None, text: str = "", thread_ts: Optional[str] = None, - **kwargs, + **kwargs ) -> List[Dict[str, Any]]: """ Send a rich message with blocks and/or attachments to all configured channels. - + Args: blocks: List of block elements for rich formatting attachments: List of legacy attachments text: Fallback text for notifications thread_ts: Timestamp of parent message to reply in thread **kwargs: Additional arguments to pass to the Slack API - + Returns: List of responses from each channel - + Raises: SlackApiError: If the Slack API returns an error for any channel """ responses = [] available_channels = self.get_available_channels() - + for channel in available_channels: try: response = self.send_rich_message( - blocks=blocks, attachments=attachments, text=text, channel=channel, thread_ts=thread_ts, **kwargs + blocks=blocks, + attachments=attachments, + text=text, + channel=channel, + thread_ts=thread_ts, + **kwargs ) responses.append(response) except SlackApiError as e: logger.error(f"Failed to send rich message to {channel}: {e.response['error']}") # Continue with other channels even if one fails responses.append({"error": str(e), "channel": channel}) - + return responses diff --git a/src/gprofiler/backend/utils/request_utils.py b/src/gprofiler/backend/utils/request_utils.py index 583187f5..08b66c6f 100644 --- a/src/gprofiler/backend/utils/request_utils.py +++ b/src/gprofiler/backend/utils/request_utils.py @@ -79,7 +79,7 @@ def get_flamegraph_response( metadata: str = None, stream=False, ): - fg_filter = fg_params.filter.json().encode() if fg_params.filter else None + fg_filter = fg_params.filter.flamedb_filter_json() if fg_params.filter else None db_api_params = get_api_params( fg_params.service_name, fg_params.start_time, @@ -139,7 +139,7 @@ def _common_fg_rest_response(response: Response, db_api_params: Dict) -> Union[L def get_query_response( fg_params: FGParamsBaseModel, lookup_for: str = "time", resolution=None, interval=None ) -> Union[List, Dict, str]: - fg_filter = fg_params.filter.json().encode() if fg_params.filter else None + fg_filter = fg_params.filter.flamedb_filter_json() if fg_params.filter else None db_api_params = get_api_params( fg_params.service_name, fg_params.start_time, @@ -170,7 +170,7 @@ def get_metrics_response( compared_start_datetime=None, compared_end_datetime=None, ) -> Union[List, Dict, str]: - fg_filter = fg_params.filter.json().encode() if fg_params.filter else None + fg_filter = fg_params.filter.flamedb_filter_json() if fg_params.filter else None db_api_params = get_api_params( fg_params.service_name, fg_params.start_time, diff --git a/src/gprofiler/frontend/package.json b/src/gprofiler/frontend/package.json index 5aaed13b..45a35d29 100644 --- a/src/gprofiler/frontend/package.json +++ b/src/gprofiler/frontend/package.json @@ -15,11 +15,8 @@ "picomatch": "^4.0.4", "esbuild": "^0.28.1", "@babel/core": "^7.29.6", - "postcss": "^8.5.18", - "browserslist": "^4.28.7", - "nanoid": "^3.3.18", - "decode-uri-component": "^0.2.2", - "@humanfs/node": "^0.16.8" + "js-cookie": "^3.0.7", + "postcss": "^8.5.18" }, "dependencies": { "@date-io/date-fns": "^2.13.1", @@ -57,7 +54,8 @@ "format": "prettier --write \"src/**/*.{js,jsx,scss}\"", "format-check": "prettier --check \"src/**/*.{js,jsx,scss}\"", "eslint": "eslint \"./src/**/*.{js,jsx}\" --max-warnings=0", - "eslint-fix": "eslint --fix \"./src/**/*.{js,jsx}\"" + "eslint-fix": "eslint --fix \"./src/**/*.{js,jsx}\"", + "test": "node --test src" }, "browserslist": [ ">0.2%", diff --git a/src/gprofiler/frontend/src/api/urls.js b/src/gprofiler/frontend/src/api/urls.js index 30de1528..39c183f1 100644 --- a/src/gprofiler/frontend/src/api/urls.js +++ b/src/gprofiler/frontend/src/api/urls.js @@ -33,6 +33,9 @@ export const DATA_URLS = { GET_METRICS_CPU_AND_MEMORY_TREND: `${API_PREFIX}/metrics/cpu_trend`, GET_NODES_AND_CORES: `${API_PREFIX}/metrics/nodes_cores/summary`, GET_LAST_HTML: `${API_PREFIX}/metrics/html_metadata`, + GET_ADHOC_FLAMEGRAPHS: `${API_PREFIX}/metrics/adhoc_flamegraphs`, + GET_ADHOC_FLAMEGRAPH_CONTENT: `${API_PREFIX}/metrics/adhoc_flamegraph_content`, + GET_PERFSPECT_REPORT_DOWNLOAD: `${API_PREFIX}/perfspect/download_report`, GET_INSTANCE_TYPE: `${API_PREFIX}/metrics/instance_type_count`, GET_GRAPH_METRICS: `${API_PREFIX}/metrics/graph`, GET_FUCNTION_CPU_GRAPH: `${API_PREFIX}/metrics/function_cpu`, @@ -41,12 +44,11 @@ export const DATA_URLS = { GET_API_KEY: `${API_PREFIX}/api_key`, // Profiling endpoints GET_PROFILING_HOST_STATUS: `${API_PREFIX}/metrics/profiling/host_status`, + GET_PROFILING_WORKLOAD_STATUS: `${API_PREFIX}/metrics/profiling/workload_status`, POST_PROFILING_REQUEST: `${API_PREFIX}/metrics/profile_request`, + POST_PROFILING_REQUEST_BULK: `${API_PREFIX}/metrics/profile_request/bulk`, POST_HEARTBEAT: `${API_PREFIX}/metrics/heartbeat`, POST_COMMAND_COMPLETION: `${API_PREFIX}/metrics/command_completion`, - // Adhoc flamegraph endpoints - GET_ADHOC_FLAMEGRAPHS: `${API_PREFIX}/metrics/adhoc_flamegraphs`, - GET_ADHOC_FLAMEGRAPH_CONTENT: `${API_PREFIX}/metrics/adhoc_flamegraph_content`, // Filter endpoints FILTERS: `${API_PREFIX}${FILETERS_PREFIX}`, GET_FILTER_OPTIONS_VALUE: (filterType, params) => diff --git a/src/gprofiler/frontend/src/components/common/dataDisplay/table/MuiTable.jsx b/src/gprofiler/frontend/src/components/common/dataDisplay/table/MuiTable.jsx index e2ad9e92..07f1c2d1 100644 --- a/src/gprofiler/frontend/src/components/common/dataDisplay/table/MuiTable.jsx +++ b/src/gprofiler/frontend/src/components/common/dataDisplay/table/MuiTable.jsx @@ -50,6 +50,15 @@ const MuiTable = ({ checkboxSelection = false, onSelectionModelChange = undefined, selectionModel = [], + paginationMode = 'client', + rowCount = undefined, + page = undefined, + onPageChange = undefined, + onPageSizeChange = undefined, + rowsPerPageOptions = [15, 25, 50, 100], + sortingMode = 'client', + sortModel = undefined, + onSortModelChange = undefined, }) => { const isDarkMode = variant !== 'light'; const isSmallTableMode = size === 'small'; @@ -83,6 +92,15 @@ const MuiTable = ({ loading={isLoading} autoHeight pagination + paginationMode={paginationMode} + rowCount={rowCount} + page={page} + onPageChange={onPageChange} + onPageSizeChange={onPageSizeChange} + rowsPerPageOptions={rowsPerPageOptions} + sortingMode={sortingMode} + sortModel={sortModel} + onSortModelChange={onSortModelChange} hideFooter={hideFooter} pageSize={pageSize} rowHeight={rowHeight} diff --git a/src/gprofiler/frontend/src/components/common/icon/iconsData.js b/src/gprofiler/frontend/src/components/common/icon/iconsData.js index 9558d99c..eb8d4f0f 100644 --- a/src/gprofiler/frontend/src/components/common/icon/iconsData.js +++ b/src/gprofiler/frontend/src/components/common/icon/iconsData.js @@ -101,8 +101,7 @@ export const ICONS = { 'M19,3H5C3.89,3 3,3.89 3,5V19A2,2 0 0,0 5,21H19A2,2 0 0,0 21,19V5C21,3.89 20.1,3 19,3M19,5V19H5V5H19Z', Kibana: 'M28.1,32H5.6l13.2-15.8C24.4,19.9,28.1,25.6,28.1,32z M28.1,0.1H4.1v28.7L28.1,0.1z', Eraser: 'M16.24,3.56L21.19,8.5C21.97,9.29 21.97,10.55 21.19,11.34L12,20.53C10.44,22.09 7.91,22.09 6.34,20.53L2.81,17C2.03,16.21 2.03,14.95 2.81,14.16L13.41,3.56C14.2,2.78 15.46,2.78 16.24,3.56M4.22,15.58L7.76,19.11C8.54,19.9 9.8,19.9 10.59,19.11L14.12,15.58L9.17,10.63L4.22,15.58Z', - Crosshairs: - 'M12,2A10,10 0 0,0 2,12A10,10 0 0,0 12,22A10,10 0 0,0 22,12A10,10 0 0,0 12,2M12,4A8,8 0 0,1 20,12A8,8 0 0,1 12,20A8,8 0 0,1 4,12A8,8 0 0,1 12,4M11,6V9.07C9.61,9.41 8.5,10.57 8.15,12H5V13H8.15C8.5,14.43 9.61,15.59 11,15.93V19H13V15.93C14.39,15.59 15.5,14.43 15.85,13H19V12H15.85C15.5,10.57 14.39,9.41 13,9.07V6M12,10A2,2 0 0,1 14,12A2,2 0 0,1 12,14A2,2 0 0,1 10,12A2,2 0 0,1 12,10Z', + Crosshairs: 'M12,2A10,10 0 0,0 2,12A10,10 0 0,0 12,22A10,10 0 0,0 22,12A10,10 0 0,0 12,2M12,4A8,8 0 0,1 20,12A8,8 0 0,1 12,20A8,8 0 0,1 4,12A8,8 0 0,1 12,4M11,6V9.07C9.61,9.41 8.5,10.57 8.15,12H5V13H8.15C8.5,14.43 9.61,15.59 11,15.93V19H13V15.93C14.39,15.59 15.5,14.43 15.85,13H19V12H15.85C15.5,10.57 14.39,9.41 13,9.07V6M12,10A2,2 0 0,1 14,12A2,2 0 0,1 12,14A2,2 0 0,1 10,12A2,2 0 0,1 12,10Z', }; export const ICONS_NAMES = { diff --git a/src/gprofiler/frontend/src/components/console/ProfilingStatusPage.jsx b/src/gprofiler/frontend/src/components/console/ProfilingStatusPage.jsx index d99fed8b..8ede6b9f 100644 --- a/src/gprofiler/frontend/src/components/console/ProfilingStatusPage.jsx +++ b/src/gprofiler/frontend/src/components/console/ProfilingStatusPage.jsx @@ -1,4 +1,5 @@ import { + Alert, Box, Button, Chip, @@ -7,430 +8,718 @@ import { DialogContent, DialogTitle, Divider, + Snackbar, + Tab, + Tabs, Typography, } from '@mui/material'; import queryString from 'query-string'; -import React, { useCallback, useEffect, useState } from 'react'; +import React, { useCallback, useEffect, useMemo, useRef, useState } from 'react'; import { useHistory, useLocation } from 'react-router-dom'; import { DATA_URLS } from '../../api/urls'; import { PAGES } from '../../utils/consts'; +import Icon from '../common/icon/Icon'; +import { ICONS_NAMES } from '../common/icon/iconsData'; import MuiTable from '../common/dataDisplay/table/MuiTable'; -import PageHeader from '../common/layout/PageHeader'; import ProfilingHeader from './header/ProfilingHeader'; import ProfilingTopPanel from './header/ProfilingTopPanel'; +import { buildProfilingRequests } from './profilingRequestBuilder.mjs'; + +const DEFAULT_PROFILING_FREQUENCY = 11; +const DEFAULT_MAX_PROCESSES = 10; +const DEFAULT_DURATION = 60; +const CONFIG_STORAGE_KEY = 'gprofiler.adhocProfilingConfig'; + +const SCOPES = [ + { id: 'service', label: 'Services' }, + { id: 'namespace', label: 'Namespaces' }, + { id: 'host', label: 'Hosts' }, + { id: 'pod', label: 'Pods' }, + { id: 'container', label: 'Containers' }, + { id: 'process', label: 'Processes' }, +]; + +const EMPTY_FILTERS = { + service: '', + hostname: '', + pids: '', + ip: '', + namespace: '', + podName: '', + containerName: '', + processName: '', + commandType: '', + status: '', +}; + +const DEFAULT_PAGE_SIZE = 50; +const AUTO_REFRESH_INTERVAL_MS = 30000; + +// DataGrid column field (camelCase) -> backend sort_by key (snake_case). Only +// mapped fields are sent; anything else is ignored (backend also whitelists). +const SORT_FIELD_MAP = { + service: 'service_name', + hostname: 'hostname', + namespace: 'namespace', + podName: 'pod_name', + containerName: 'container_name', + workloadName: 'workload_name', + processName: 'process_name', + pid: 'pid', + heartbeatTimestamp: 'heartbeat_timestamp', + profilingStatus: 'profiling_status', + agentVersion: 'agent_version', + hostCount: 'host_count', + namespaceCount: 'namespace_count', + podCount: 'pod_count', + containerCount: 'container_count', + processCount: 'process_count', +}; + +const readField = (row, camelKey, snakeKey = camelKey) => row[camelKey] ?? row[snakeKey]; + +const formatHeartbeat = (value) => { + if (!value) { + return 'N/A'; + } + + try { + let utcTimestamp = value; + if (!utcTimestamp.endsWith('Z') && !utcTimestamp.includes('+') && !utcTimestamp.includes('-', 10)) { + utcTimestamp += 'Z'; + } + return new Date(utcTimestamp).toLocaleString(navigator.language, { + day: '2-digit', + month: '2-digit', + year: 'numeric', + hour: '2-digit', + minute: '2-digit', + second: '2-digit', + hour12: true, + }); + } catch (error) { + return 'Invalid date'; + } +}; -// Helper function to build profile URL -const buildProfileUrl = (host, service, view) => { +// Build a profiles-view deep link for a status row. Each scope maps to the +// filters the profiles view supports: the exact `service` param, exact Host name +// (hn,is), a "Contains" Container name match (cn,has) that joins the available +// container/deployment/namespace parts, and an exact process for the process scope. +const buildScopeProfileUrl = (row, scope, view) => { const baseUrl = `${window.location.protocol}//${window.location.host}`; - const params = new URLSearchParams({ - filter: `hn,is,${host}`, - gtab: '1', - pm: '1', - rtms: '1', - service: service, - time: '1h', - view: view, - wp: '100', - }); - return `${baseUrl}${PAGES.profiles.to}?${params.toString()}`; + const params = { gtab: '1', pm: '1', rtms: '1', time: '1h', wp: '100', service: row.service, view }; + const rules = []; + const namespace = row.namespace || ''; + const deployment = row.workloadName || ''; + const container = row.containerName || ''; + // Join only the available parts with "_" (skip empties to avoid "__"). + const containsValue = (parts) => parts.filter(Boolean).join('_'); + + if (scope === 'namespace') { + const value = containsValue([namespace]); + if (value) rules.push(`cn,has,${value}`); + } else if (scope === 'host') { + if (row.host) rules.push(`hn,is,${row.host}`); + } else if (scope === 'pod') { + if (row.host) rules.push(`hn,is,${row.host}`); + const value = containsValue([deployment, namespace]); + if (value) rules.push(`cn,has,${value}`); + } else if (scope === 'container') { + if (row.host) rules.push(`hn,is,${row.host}`); + const value = containsValue([container, deployment, namespace]); + if (value) rules.push(`cn,has,${value}`); + } else if (scope === 'process') { + if (row.host) rules.push(`hn,is,${row.host}`); + // Flamegraph process nodes use the 15-char kernel comm, so match that prefix. + if (row.processName) params.p = row.processName.slice(0, 15); + } + // service scope: exact service via the `service` param only, no RQL rule. + if (rules.length) { + params.filter = rules.join(',a,'); // ",a," is the profiles-view AND separator + } + return `${baseUrl}${PAGES.profiles.to}?${new URLSearchParams(params).toString()}`; }; -const columns = [ - { field: 'service', headerName: 'service name', flex: 1, sortable: true }, - { field: 'host', headerName: 'host name', flex: 1, sortable: true }, - { field: 'pids', headerName: 'pids (if profiled)', flex: 1, sortable: true }, - { field: 'ip', headerName: 'IP', flex: 1, sortable: true }, - { field: 'commandType', headerName: 'command type', flex: 1, sortable: true }, - { field: 'status', headerName: 'profiling status', flex: 1, sortable: true }, - { - field: 'heartbeat_timestamp', +const makeProfileColumn = (scope) => ({ + field: 'profile', + headerName: 'profile', + flex: 1.2, + sortable: false, + renderCell: (params) => { + const { service, profilingStatus } = params.row; + if (!service || profilingStatus !== 'active') { + return ''; + } + + return ( + + + View Continuous Profile + + + View Adhoc Profile + + + ); + }, +}); + +const getScopeColumns = (scope) => { + const sharedTimestampColumn = { + field: 'heartbeatTimestamp', headerName: 'last heartbeat', flex: 1, sortable: true, - renderCell: (params) => { - if (!params.value) return 'N/A'; - try { - // The backend sends UTC timestamp without 'Z' suffix, so we need to explicitly treat it as UTC - let utcTimestamp = params.value; - if (!utcTimestamp.endsWith('Z') && !utcTimestamp.includes('+') && !utcTimestamp.includes('-', 10)) { - utcTimestamp += 'Z'; - } + renderCell: (params) => formatHeartbeat(params.value), + }; + const sharedStatusColumns = [ + { field: 'profilingStatus', headerName: 'profiling', flex: 1, sortable: true }, + { field: 'profilingMode', headerName: 'mode', flex: 1, sortable: true }, + { field: 'profilerSummary', headerName: 'profilers', flex: 1.3, sortable: false }, + { field: 'frequency', headerName: 'frequency', flex: 0.8, sortable: true }, + ]; + + if (scope === 'service') { + return [ + { field: 'service', headerName: 'service name', flex: 1.4, sortable: true }, + { field: 'namespaceCount', headerName: 'namespaces', flex: 0.8, sortable: true }, + { field: 'hostCount', headerName: 'hosts', flex: 0.7, sortable: true }, + { field: 'podCount', headerName: 'pods', flex: 0.7, sortable: true }, + { field: 'containerCount', headerName: 'containers', flex: 0.8, sortable: true }, + { field: 'processCount', headerName: 'processes', flex: 0.8, sortable: true }, + ...sharedStatusColumns, + sharedTimestampColumn, + { field: 'agentVersion', headerName: 'version', flex: 0.8, sortable: true }, + makeProfileColumn('service'), + ]; + } - const utcDate = new Date(utcTimestamp); - // Convert to user's local timezone - const localDateTimeString = utcDate.toLocaleString(navigator.language, { - day: '2-digit', - month: '2-digit', - year: 'numeric', - hour: '2-digit', - minute: '2-digit', - second: '2-digit', - hour12: true, - }); - return localDateTimeString; - } catch (error) { - return 'Invalid date'; - } - }, - }, - { - field: 'profile', - headerName: 'profile', - flex: 1, - renderCell: (params) => { - const { host, service, commandType, status } = params.row; + if (scope === 'namespace') { + return [ + { field: 'namespace', headerName: 'namespace', flex: 1, sortable: true }, + { field: 'service', headerName: 'service name', flex: 1.2, sortable: true }, + { field: 'hostCount', headerName: 'hosts', flex: 0.7, sortable: true }, + { field: 'podCount', headerName: 'pods', flex: 0.7, sortable: true }, + { field: 'containerCount', headerName: 'containers', flex: 0.8, sortable: true }, + { field: 'processCount', headerName: 'processes', flex: 0.8, sortable: true }, + ...sharedStatusColumns, + sharedTimestampColumn, + makeProfileColumn('namespace'), + ]; + } - // Only show profile link for rows with commandType="start" and status="completed" - if (commandType !== 'start' || status !== 'completed') { - return ''; - } + if (scope === 'host') { + return [ + { field: 'host', headerName: 'host name', flex: 1.1, sortable: true }, + { field: 'service', headerName: 'service name', flex: 1, sortable: true }, + { field: 'namespace', headerName: 'namespace', flex: 0.9, sortable: true }, + { field: 'pids', headerName: 'pids (if profiled)', flex: 1, sortable: false }, + { field: 'ip', headerName: 'IP', flex: 0.9, sortable: true }, + { field: 'commandType', headerName: 'command type', flex: 0.8, sortable: true }, + { field: 'profilingStatus', headerName: 'profiling status', flex: 0.9, sortable: true }, + sharedTimestampColumn, + makeProfileColumn('host'), + ]; + } - if (!host || !service) return ''; - - const continuousProfileUrl = buildProfileUrl(host, service, 'flamegraph'); - const adhocProfileUrl = buildProfileUrl(host, service, 'adhoc'); - - return ( - - (e.target.style.textDecoration = 'underline')} - onMouseOut={(e) => (e.target.style.textDecoration = 'none')}> - View Continuous Profile - - (e.target.style.textDecoration = 'underline')} - onMouseOut={(e) => (e.target.style.textDecoration = 'none')}> - View Adhoc Profile - - - ); - }, - }, -]; + if (scope === 'pod') { + return [ + { field: 'podName', headerName: 'pod', flex: 1.2, sortable: true }, + { field: 'namespace', headerName: 'namespace', flex: 0.9, sortable: true }, + { field: 'service', headerName: 'service name', flex: 1, sortable: true }, + { field: 'hostCount', headerName: 'hosts', flex: 0.7, sortable: true }, + { field: 'containerCount', headerName: 'containers', flex: 0.8, sortable: true }, + { field: 'processCount', headerName: 'processes', flex: 0.8, sortable: true }, + ...sharedStatusColumns, + sharedTimestampColumn, + makeProfileColumn('pod'), + ]; + } -const ProfilingStatusPage = () => { - const [rows, setRows] = useState([]); - const [loading, setLoading] = useState(false); - const [selectionModel, setSelectionModel] = useState([]); - const [filters, setFilters] = useState({ - service: '', - hostname: '', - pids: '', - ip: '', - commandType: '', - status: '', - }); - const [appliedFilters, setAppliedFilters] = useState({ - service: '', - hostname: '', - pids: '', - ip: '', - commandType: '', - status: '', - }); + if (scope === 'container') { + return [ + { field: 'containerName', headerName: 'container', flex: 1.1, sortable: true }, + { field: 'podName', headerName: 'pod', flex: 1, sortable: true }, + { field: 'namespace', headerName: 'namespace', flex: 0.9, sortable: true }, + { field: 'host', headerName: 'host', flex: 1, sortable: true }, + { field: 'processCount', headerName: 'processes', flex: 0.8, sortable: true }, + ...sharedStatusColumns, + sharedTimestampColumn, + makeProfileColumn('container'), + ]; + } - // PerfSpect state - const [enablePerfSpect, setEnablePerfSpect] = useState(false); + return [ + { field: 'processName', headerName: 'process', flex: 1.1, sortable: true }, + { field: 'pid', headerName: 'pid', flex: 0.6, sortable: true }, + { field: 'containerName', headerName: 'container', flex: 1, sortable: true }, + { field: 'podName', headerName: 'pod', flex: 1, sortable: true }, + { field: 'namespace', headerName: 'namespace', flex: 0.9, sortable: true }, + { field: 'host', headerName: 'host', flex: 1, sortable: true }, + ...sharedStatusColumns, + sharedTimestampColumn, + makeProfileColumn('process'), + ]; +}; - // Profiling frequency state - const [profilingFrequency, setProfilingFrequency] = useState(11); +const formatRowForScope = (row, scope) => ({ + id: readField(row, 'id'), + scope, + service: readField(row, 'serviceName', 'service_name'), + namespace: readField(row, 'namespace'), + host: readField(row, 'hostname'), + ip: readField(row, 'ipAddress', 'ip_address'), + podName: readField(row, 'podName', 'pod_name'), + containerName: readField(row, 'containerName', 'container_name'), + workloadName: readField(row, 'workloadName', 'workload_name'), + workloadKind: readField(row, 'workloadKind', 'workload_kind'), + processName: readField(row, 'processName', 'process_name'), + pid: readField(row, 'pid'), + pids: readField(row, 'pids', 'pids') || [], + activeHosts: readField(row, 'activeHosts', 'active_hosts'), + hostCount: readField(row, 'hostCount', 'host_count'), + namespaceCount: readField(row, 'namespaceCount', 'namespace_count'), + podCount: readField(row, 'podCount', 'pod_count'), + containerCount: readField(row, 'containerCount', 'container_count'), + processCount: readField(row, 'processCount', 'process_count'), + commandType: readField(row, 'commandType', 'command_type') || 'N/A', + profilingStatus: readField(row, 'profilingStatus', 'profiling_status') || 'stopped', + profilingMode: readField(row, 'profilingMode', 'profiling_mode') || 'N/A', + frequency: readField(row, 'frequency'), + profilerSummary: readField(row, 'profilerSummary', 'profiler_summary') || 'N/A', + heartbeatTimestamp: readField(row, 'heartbeatTimestamp', 'heartbeat_timestamp'), + agentVersion: readField(row, 'agentVersion', 'agent_version'), + runMode: readField(row, 'runMode', 'run_mode'), +}); + +const scopeEntityLabel = (scope) => { + const lookup = { + service: 'services', + namespace: 'namespaces', + host: 'hosts', + pod: 'pods', + container: 'containers', + process: 'processes', + }; + return lookup[scope] || 'entities'; +}; - // Max processes state - const [maxProcesses, setMaxProcesses] = useState(10); +const scopeActiveLabel = (scope) => { + const lookup = { + service: 'Active Services', + namespace: 'Active Namespaces', + host: 'Active Hosts', + pod: 'Active Pods', + container: 'Active Containers', + process: 'Active Processes', + }; + return lookup[scope] || 'Active Entities'; +}; - // Profiling mode state (Ad Hoc vs Continuous) - const [profilingMode, setProfilingMode] = useState('continuous'); // 'adhoc' or 'continuous' +const rowDisplayName = (row, scope) => { + if (scope === 'service') return row.service; + if (scope === 'namespace') return `${row.namespace}`; + if (scope === 'host') return row.host; + if (scope === 'pod') return `${row.namespace}/${row.podName}`; + if (scope === 'container') return `${row.namespace}/${row.podName}/${row.containerName}`; + return `${row.processName} (${row.pid})`; +}; - // Duration state - const [duration, setDuration] = useState(60); +const ProfilingStatusPage = () => { + const history = useHistory(); + const location = useLocation(); - // Profiler configurations state + const [activeScope, setActiveScope] = useState('service'); + const [rows, setRows] = useState([]); + const [scopeCounts, setScopeCounts] = useState({}); + const [loading, setLoading] = useState(false); + const [selectionModel, setSelectionModel] = useState([]); + const [totalCount, setTotalCount] = useState(0); + const [page, setPage] = useState(0); + const [pageSize, setPageSize] = useState(DEFAULT_PAGE_SIZE); + const [sortModel, setSortModel] = useState([]); + // Refs mirror the paging/sort state so the stable fetch callback and the + // 30s refresh can read current values without being re-created on change. + const pageRef = useRef(0); + const pageSizeRef = useRef(DEFAULT_PAGE_SIZE); + const sortRef = useRef([]); + // Monotonic request id so a slow response for a stale page/scope is dropped. + const requestSeq = useRef(0); + // Timer for the periodic background refresh. It is always cleared and + // rescheduled by fetchProfilingStatus itself, so it can never fire concurrently + // with (or duplicate) a manual fetch. + const autoRefreshTimeoutRef = useRef(null); + // The query string of the last history.replace we triggered ourselves, so the + // location-sync effect below can tell that apart from a real external navigation + // (deep link, browser back/forward) and avoid double-fetching for our own updates. + const lastSelfUpdatedSearchRef = useRef(null); + const [filters, setFilters] = useState(EMPTY_FILTERS); + const [appliedFilters, setAppliedFilters] = useState(EMPTY_FILTERS); + const [enablePerfSpect, setEnablePerfSpect] = useState(false); + const [profilingFrequency, setProfilingFrequency] = useState(DEFAULT_PROFILING_FREQUENCY); + const [maxProcesses, setMaxProcesses] = useState(DEFAULT_MAX_PROCESSES); + const [profilingMode, setProfilingMode] = useState('continuous'); + const [duration, setDuration] = useState(DEFAULT_DURATION); const [profilerConfigs, setProfilerConfigs] = useState({ - perf: 'enabled_restricted', // 'enabled_restricted', 'enabled_aggressive', 'disabled' - async_profiler: 'enabled', // 'enabled', 'disabled' + perf: { + mode: 'enabled_restricted', // 'enabled_restricted', 'enabled_aggressive', 'disabled' + events: ['cpu-cycles'] // Array of events: 'cpu-cycles', 'instructions', 'cache-misses', etc. + }, + async_profiler: { + enabled: true, + time: 'cpu', // 'cpu', 'itimer', 'wall', 'auto', 'alloc' + alloc_interval: '2MB' // used only when time === 'alloc' + }, pyperf: 'enabled', // 'enabled', 'disabled' pyspy: 'enabled_fallback', // 'enabled_fallback', 'enabled', 'disabled' - rbspy: 'enabled', // 'enabled', 'disabled' - phpspy: 'enabled', // 'enabled', 'disabled' - dotnet_trace: 'enabled', // 'enabled', 'disabled' + rbspy: 'disabled', // 'enabled', 'disabled' + phpspy: 'disabled', // 'enabled', 'disabled' + dotnet_trace: 'disabled', // 'enabled', 'disabled' nodejs_perf: 'enabled', // 'enabled', 'disabled' }); - - // Confirmation dialog state const [confirmationDialog, setConfirmationDialog] = useState({ open: false, action: null, selectedRows: [], serviceGroups: {}, }); + const [dryRunValidation, setDryRunValidation] = useState({ + isValidating: false, + isValid: false, + errors: [], + }); + const [snackbar, setSnackbar] = useState({ open: false, message: '' }); - const history = useHistory(); - const location = useLocation(); - - const fetchProfilingStatus = useCallback((filterParams) => { - setLoading(true); - - // Build query parameters - const params = new URLSearchParams(); + const columns = useMemo(() => getScopeColumns(activeScope), [activeScope]); - if (filterParams.service) { - params.append('service_name', filterParams.service); - } - if (filterParams.hostname) { - params.append('hostname', filterParams.hostname); - } - if (filterParams.pids) { - params.append('pids', filterParams.pids); - } - if (filterParams.ip) { - params.append('ip_address', filterParams.ip); + useEffect(() => { + try { + const saved = JSON.parse(localStorage.getItem(CONFIG_STORAGE_KEY)); + if (saved) { + if (typeof saved.enablePerfSpect === 'boolean') setEnablePerfSpect(saved.enablePerfSpect); + if (saved.profilingFrequency) setProfilingFrequency(saved.profilingFrequency); + if (saved.maxProcesses != null) setMaxProcesses(saved.maxProcesses); + if (saved.profilingMode) setProfilingMode(saved.profilingMode); + if (saved.duration) setDuration(saved.duration); + if (saved.profilerConfigs) setProfilerConfigs(saved.profilerConfigs); + } + } catch (error) { + // ignore malformed persisted config } - if (filterParams.commandType) { - params.append('command_type', filterParams.commandType); + }, []); + + const handleSaveConfiguration = useCallback(() => { + const config = { + enablePerfSpect, + profilingFrequency, + maxProcesses, + profilingMode, + duration, + profilerConfigs, + }; + try { + localStorage.setItem(CONFIG_STORAGE_KEY, JSON.stringify(config)); + setSnackbar({ open: true, message: 'Configuration saved' }); + } catch (error) { + setSnackbar({ open: true, message: 'Failed to save configuration' }); } - if (filterParams.status) { - params.append('profiling_status', filterParams.status); + }, [duration, enablePerfSpect, maxProcesses, profilerConfigs, profilingFrequency, profilingMode]); + + const fetchProfilingStatus = useCallback((filterParams, scope, opts = {}) => { + const pageArg = opts.page ?? pageRef.current; + const pageSizeArg = opts.pageSize ?? pageSizeRef.current; + const sortArg = opts.sortModel ?? sortRef.current; + setLoading(true); + const params = new URLSearchParams(); + params.append('scope', scope); + + if (filterParams.service) params.append('service_name', filterParams.service); + if (filterParams.hostname) params.append('hostname', filterParams.hostname); + if (filterParams.pids) params.append('pids', filterParams.pids); + if (filterParams.ip) params.append('ip_address', filterParams.ip); + if (filterParams.namespace) params.append('namespace', filterParams.namespace); + if (filterParams.podName) params.append('pod_name', filterParams.podName); + if (filterParams.containerName) params.append('container_name', filterParams.containerName); + if (filterParams.processName) params.append('process_name', filterParams.processName); + if (filterParams.commandType) params.append('command_type', filterParams.commandType); + if (filterParams.status) params.append('profiling_status', filterParams.status); + + params.append('page', pageArg); + params.append('page_size', pageSizeArg); + if (sortArg && sortArg.length && SORT_FIELD_MAP[sortArg[0].field]) { + params.append('sort_by', SORT_FIELD_MAP[sortArg[0].field]); + params.append('sort_order', sortArg[0].sort || 'asc'); } - const url = params.toString() - ? `${DATA_URLS.GET_PROFILING_HOST_STATUS}?${params.toString()}` - : DATA_URLS.GET_PROFILING_HOST_STATUS; + // Cancel any pending background refresh so it can never fire concurrently with + // this fetch (manual or auto); the next refresh is (re)scheduled once this settles. + if (autoRefreshTimeoutRef.current) { + clearTimeout(autoRefreshTimeoutRef.current); + } - fetch(url) + const seq = ++requestSeq.current; + fetch(`${DATA_URLS.GET_PROFILING_WORKLOAD_STATUS}?${params.toString()}`) .then((res) => res.json()) .then((data) => { - setRows( - data.map((row) => ({ - id: row.id, - service: row.service_name, - host: row.hostname, - pids: row.pids, - ip: row.ip_address, - commandType: row.command_type || 'N/A', - status: row.profiling_status, - heartbeat_timestamp: row.heartbeat_timestamp, - })) - ); + if (seq !== requestSeq.current) return; // a newer request superseded this one + const normalizedRows = (data.rows || []).map((row) => formatRowForScope(row, scope)); + setRows(normalizedRows); + setScopeCounts(data.tabCounts || data.tab_counts || {}); + setTotalCount(data.totalCount || data.total_count || normalizedRows.length); setLoading(false); }) - .catch(() => setLoading(false)); - }, []); // No dependencies needed since it takes filterParams as argument - - // Initialize filters from URL parameters for direct URL visits (shareable links) - useEffect(() => { - const searchParams = queryString.parse(location.search); - const hasFilterParams = ['service', 'hostname', 'pids', 'ip', 'commandType', 'status'].some( - (param) => searchParams[param] - ); - - // Only initialize from URL if there are actual filter parameters - if (hasFilterParams) { - const urlFilters = { - service: searchParams.service || '', - hostname: searchParams.hostname || '', - pids: searchParams.pids || '', - ip: searchParams.ip || '', - commandType: searchParams.commandType || '', - status: searchParams.status || '', - }; - setFilters(urlFilters); - setAppliedFilters(urlFilters); - // Automatically fetch data with URL filters on page load - fetchProfilingStatus(urlFilters); - } else { - // No URL params, fetch all data - const emptyFilters = { - service: '', - hostname: '', - pids: '', - ip: '', - commandType: '', - status: '', - }; - fetchProfilingStatus(emptyFilters); - } - - // Clean up profile-specific parameters if they exist (mixed URLs) - const profileParams = [ - 'gtab', - 'view', - 'time', - 'startTime', - 'endTime', - 'filter', - 'rt', - 'rtms', - 'p', - 'pm', - 'wt', - 'wp', - 'search', - 'fullscreen', - ]; - const hasProfileParams = profileParams.some((param) => searchParams[param]); - - if (hasProfileParams) { - // Remove only profile params, keep filter params - const cleanedParams = { ...searchParams }; - profileParams.forEach((param) => { - delete cleanedParams[param]; - }); - history.replace({ search: queryString.stringify(cleanedParams) }); - } - }, [fetchProfilingStatus, history, location.search]); // Add dependencies - - // Auto-refresh every 30 seconds for dynamic profiling - useEffect(() => { - const refreshInterval = setInterval(() => { - // Refresh with current applied filters - fetchProfilingStatus(appliedFilters); - }, 30000); // 30 seconds - - // Cleanup interval on component unmount - return () => clearInterval(refreshInterval); - }, [appliedFilters, fetchProfilingStatus]); // Re-create interval when filters change - - // Update URL when filters change (with focus preservation) - const updateURL = useCallback( - (newFilters) => { - // Use replace instead of push to avoid navigation history buildup - // and reduce re-render impact on focus - const searchParams = {}; - - // Add new filter parameters - Object.keys(newFilters).forEach((key) => { - if (newFilters[key]) { - searchParams[key] = newFilters[key]; + .catch(() => { + if (seq === requestSeq.current) setLoading(false); + }) + .finally(() => { + // Only the request that is still current gets to schedule the next + // background refresh, using the filters/scope it just displayed. + if (seq === requestSeq.current) { + autoRefreshTimeoutRef.current = setTimeout(() => { + fetchProfilingStatus(filterParams, scope); + }, AUTO_REFRESH_INTERVAL_MS); } }); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); - const newSearch = queryString.stringify(searchParams); - // Use replace instead of push to minimize focus disruption - if (newSearch === '') { - history.replace('/profiling'); - } else { - history.replace({ pathname: '/profiling', search: newSearch }); + const updateURL = useCallback((scope, nextFilters) => { + const searchParams = { scope }; + Object.keys(nextFilters).forEach((key) => { + if (nextFilters[key]) { + searchParams[key] = nextFilters[key]; } - }, - [history] - ); - - // Function to update individual filter (optimized for focus preservation) - const updateFilter = useCallback((field, value) => { - setFilters((prev) => ({ ...prev, [field]: value })); - }, []); // Stable function reference + }); + const search = queryString.stringify(searchParams); + // Remember that we caused this URL change so the location-sync effect below + // (deep links / browser back-forward) doesn't also re-fetch for it. + lastSelfUpdatedSearchRef.current = search; + history.replace({ pathname: '/profiling', search }); + }, [history]); - // Apply filters function const applyFilters = useCallback(() => { setAppliedFilters(filters); - fetchProfilingStatus(filters); - updateURL(filters); - }, [filters, fetchProfilingStatus, updateURL]); + setSelectionModel([]); + pageRef.current = 0; + setPage(0); + fetchProfilingStatus(filters, activeScope, { page: 0 }); + updateURL(activeScope, filters); + }, [activeScope, fetchProfilingStatus, filters, updateURL]); - // Clear all filters function const clearAllFilters = useCallback(() => { - const emptyFilters = { - service: '', - hostname: '', - pids: '', - ip: '', - commandType: '', - status: '', + setFilters(EMPTY_FILTERS); + setAppliedFilters(EMPTY_FILTERS); + setSelectionModel([]); + pageRef.current = 0; + setPage(0); + fetchProfilingStatus(EMPTY_FILTERS, activeScope, { page: 0 }); + updateURL(activeScope, EMPTY_FILTERS); + }, [activeScope, fetchProfilingStatus, updateURL]); + + const updateFilter = useCallback((field, value) => { + setFilters((prev) => ({ ...prev, [field]: value })); + }, []); + + useEffect(() => { + // Skip URL changes we triggered ourselves (applyFilters/clearAllFilters/ + // handleScopeChange already fetched with the new state); only react here to + // real external navigation, such as a deep link or the browser back/forward + // buttons, so we don't fire a second, duplicate request for our own updates. + const currentSearch = location.search.replace(/^\?/, ''); + if (lastSelfUpdatedSearchRef.current !== null && currentSearch === lastSelfUpdatedSearchRef.current) { + lastSelfUpdatedSearchRef.current = null; + return; + } + + const searchParams = queryString.parse(location.search); + const scope = searchParams.scope || 'service'; + const urlFilters = { + service: searchParams.service || '', + hostname: searchParams.hostname || '', + pids: searchParams.pids || '', + ip: searchParams.ip || '', + namespace: searchParams.namespace || '', + podName: searchParams.podName || '', + containerName: searchParams.containerName || '', + processName: searchParams.processName || '', + commandType: searchParams.commandType || '', + status: searchParams.status || '', }; - setFilters(emptyFilters); - setAppliedFilters(emptyFilters); - fetchProfilingStatus(emptyFilters); - updateURL(emptyFilters); - }, [fetchProfilingStatus, updateURL]); - - // Bulk Start/Stop handlers - function handleBulkAction(action) { - const selectedRows = rows.filter((row) => selectionModel.includes(row.id)); + setActiveScope(scope); + setFilters(urlFilters); + setAppliedFilters(urlFilters); + pageRef.current = 0; + setPage(0); + sortRef.current = []; + setSortModel([]); + fetchProfilingStatus(urlFilters, scope, { page: 0, sortModel: [] }); + }, [fetchProfilingStatus, location.search]); - // Group selected rows by service name - const serviceGroups = selectedRows.reduce((groups, row) => { - if (!groups[row.service]) { - groups[row.service] = []; + useEffect(() => { + // fetchProfilingStatus reschedules itself after every call (see above); this + // only needs to cancel a pending timer if the page unmounts mid-cycle. + return () => { + if (autoRefreshTimeoutRef.current) { + clearTimeout(autoRefreshTimeoutRef.current); } - groups[row.service].push(row.host); - return groups; - }, {}); + }; + }, []); + + const handleScopeChange = (_, nextScope) => { + setActiveScope(nextScope); + setSelectionModel([]); + pageRef.current = 0; + setPage(0); + sortRef.current = []; + setSortModel([]); + fetchProfilingStatus(appliedFilters, nextScope, { page: 0, sortModel: [] }); + updateURL(nextScope, appliedFilters); + }; + + const handlePageChange = useCallback((newPage) => { + pageRef.current = newPage; + setPage(newPage); + setSelectionModel([]); // selection is per-page (Gmail-style) + fetchProfilingStatus(appliedFilters, activeScope, { page: newPage }); + }, [activeScope, appliedFilters, fetchProfilingStatus]); + + const handlePageSizeChange = useCallback((newSize) => { + pageSizeRef.current = newSize; + setPageSize(newSize); + pageRef.current = 0; + setPage(0); + setSelectionModel([]); + fetchProfilingStatus(appliedFilters, activeScope, { page: 0, pageSize: newSize }); + }, [activeScope, appliedFilters, fetchProfilingStatus]); + + const handleSortModelChange = useCallback((model) => { + sortRef.current = model; + setSortModel(model); + pageRef.current = 0; + setPage(0); + fetchProfilingStatus(appliedFilters, activeScope, { page: 0, sortModel: model }); + }, [activeScope, appliedFilters, fetchProfilingStatus]); + + const buildRequests = useCallback((action, selectedRows) => buildProfilingRequests(action, selectedRows, { + scope: activeScope, + profilingMode, + duration, + profilingFrequency, + enablePerfSpect, + profilerConfigs, + maxProcesses, + }), [activeScope, duration, enablePerfSpect, maxProcesses, profilerConfigs, profilingFrequency, profilingMode]); + + const executeDryRun = useCallback((action, selectedRows) => { + const { requests } = buildRequests(action, selectedRows); + setDryRunValidation({ isValidating: true, isValid: false, errors: [] }); + + fetch(DATA_URLS.POST_PROFILING_REQUEST_BULK, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ requests, dry_run: true }), + }) + .then((res) => { + if (!res.ok) { + return res.json().then((errData) => { + if (errData.detail && Array.isArray(errData.detail)) { + throw new Error(errData.detail.map((err) => err.msg || JSON.stringify(err)).join('; ')); + } + throw new Error(errData.detail || `Validation failed with status ${res.status}`); + }); + } + return res.json(); + }) + .then((bulkResponse) => { + const errors = (bulkResponse.results || []) + .filter((result) => !result.success) + .map((result) => `${result.service_name}: ${result.error}`); + setDryRunValidation({ + isValidating: false, + isValid: (bulkResponse.failed_count || 0) === 0, + errors, + }); + }) + .catch((error) => { + setDryRunValidation({ + isValidating: false, + isValid: false, + errors: [error.message], + }); + }); + }, [buildRequests]); - // Show confirmation dialog + const handleBulkAction = (action) => { + const selectedRows = rows.filter((row) => selectionModel.includes(row.id)); + const { grouped } = buildRequests(action, selectedRows); setConfirmationDialog({ open: true, action, selectedRows, - serviceGroups, + serviceGroups: grouped, }); - } + executeDryRun(action, selectedRows); + }; - // Execute the actual profiling action after confirmation - function executeProfilingAction() { - const { action, serviceGroups } = confirmationDialog; - - // Create one request per service with all hosts for that service - const requests = Object.entries(serviceGroups).map(([serviceName, hosts]) => { - const target_host = hosts.reduce((hostObj, host) => { - hostObj[host] = null; - return hostObj; - }, {}); - - const submitData = { - service_name: serviceName, - request_type: action, - continuous: profilingMode === 'continuous', - duration: profilingMode === 'continuous' ? 60 : duration, - frequency: profilingFrequency, // Use frequency from UI - profiling_mode: 'cpu', // Default profiling mode, can't be adjusted yet - target_hosts: target_host, - additional_args: { - enable_perfspect: enablePerfSpect, // Include PerfSpect setting - profiler_configs: profilerConfigs, // Include all profiler configurations - max_processes: maxProcesses, // Include max processes setting - }, - }; - - // append 'stop_level: host' when action is 'stop' - if (action === 'stop') { - submitData.stop_level = 'host'; - } - - return fetch(DATA_URLS.POST_PROFILING_REQUEST, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(submitData), - }); - }); + const executeProfilingAction = () => { + const { action, selectedRows } = confirmationDialog; + const { requests } = buildRequests(action, selectedRows); - // Close dialog and wait for all requests to finish before refreshing setConfirmationDialog({ open: false, action: null, selectedRows: [], serviceGroups: {} }); - Promise.all(requests).then(() => { - // Maintain current filter state when refreshing - fetchProfilingStatus(appliedFilters); - setSelectionModel([]); // Clear all checkboxes after API requests complete - setEnablePerfSpect(false); // Reset PerfSpect checkbox after action completes - // Note: Keep profiling frequency as is - user may want to reuse the same frequency + setDryRunValidation({ isValidating: false, isValid: false, errors: [] }); + + fetch(DATA_URLS.POST_PROFILING_REQUEST_BULK, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ requests, dry_run: false }), + }).then(() => { + fetchProfilingStatus(appliedFilters, activeScope); + setSelectionModel([]); + setEnablePerfSpect(false); }); - } + }; - // Close confirmation dialog without action - function handleDialogClose() { + const handleDialogClose = () => { setConfirmationDialog({ open: false, action: null, selectedRows: [], serviceGroups: {} }); - } + setDryRunValidation({ isValidating: false, isValid: false, errors: [] }); + }; return ( + + + + + + Adhoc Profile Configuration + + + Select workloads and configure profiling parameters + + + + + + {scopeActiveLabel(activeScope)} + + + {(scopeCounts[activeScope] ?? totalCount ?? 0).toLocaleString()} + + + + { onClearFilters={clearAllFilters} /> - + + + {SCOPES.map((scope) => ( + + ))} + + + + + + + + Showing {rows.length} of {totalCount} {scopeEntityLabel(activeScope)} + {selectionModel.length ? ` \u00b7 ${selectionModel.length} selected on this page` : ''} + + + fetchProfilingStatus(scopeFilters, activeScope)} filters={appliedFilters} loading={loading} - rowsCount={rows.length} - clearAllFilters={clearAllFilters} enablePerfSpect={enablePerfSpect} onPerfSpectChange={setEnablePerfSpect} profilingFrequency={profilingFrequency} @@ -464,148 +784,154 @@ const ProfilingStatusPage = () => { onDurationChange={setDuration} profilerConfigs={profilerConfigs} onProfilerConfigsChange={setProfilerConfigs} + onSaveConfiguration={handleSaveConfiguration} /> - - {/* Data Table */} - - - - {/* Confirmation Dialog */} - + - + Confirm {confirmationDialog.action === 'start' ? 'Start' : 'Stop'} Profiling - - Are you sure you want to {confirmationDialog.action} profiling for the - following hosts? + {dryRunValidation.isValidating && ( + + + Validating request... + + + )} + {!dryRunValidation.isValidating && dryRunValidation.isValid && ( + + + ✓ Validation successful + + + )} + {!dryRunValidation.isValidating && !dryRunValidation.isValid && dryRunValidation.errors.length > 0 && ( + + + ✗ Validation failed + + {dryRunValidation.errors.map((error, idx) => ( + + • {error} + + ))} + + )} + + + Are you sure you want to {confirmationDialog.action} profiling for the selected {scopeEntityLabel(activeScope)}? - {/* Selected Hosts Summary */} - - Selected Hosts ({confirmationDialog.selectedRows.length}): + + Selected {scopeEntityLabel(activeScope)} ({confirmationDialog.selectedRows.length}): - {Object.entries(confirmationDialog.serviceGroups).map(([serviceName, hosts]) => ( - - + {Object.entries(confirmationDialog.serviceGroups).map(([serviceName, serviceRows]) => ( + + {serviceName}: - - {hosts.map((host) => ( - + + {serviceRows.map((row) => ( + ))} ))} - {/* Configuration Summary (only for start action) */} {confirmationDialog.action === 'start' && ( <> - + Profiling Configuration: - - {/* Basic Settings */} - + Basic Settings: - - • Mode: {profilingMode === 'continuous' ? 'Continuous' : 'Ad Hoc'} - - - • Duration: {profilingMode === 'continuous' ? 60 : duration} seconds - - • Frequency: {profilingFrequency} Hz - • Max Processes: {maxProcesses} - - • PerfSpect HW Metrics: {enablePerfSpect ? 'Enabled' : 'Disabled'} - + • Frequency: {profilingFrequency} Hz + • Max Processes: {maxProcesses} + • PerfSpect HW Metrics: {enablePerfSpect ? 'Enabled' : 'Disabled'} + • Profiling Mode: {profilingMode === 'adhoc' ? 'Ad Hoc' : 'Continuous'} + • Duration: {profilingMode === 'continuous' ? 60 : duration} seconds + • Mode: CPU profiling - - {/* Profiler Settings */} - + Profiler Settings: - - • Perf (C/C++/Go):{' '} - {profilerConfigs.perf === 'enabled_restricted' - ? 'Enabled (Restricted)' - : profilerConfigs.perf === 'enabled_aggressive' - ? 'Enabled (Aggressive)' - : 'Disabled'} - - - • Java Async Profiler:{' '} - {profilerConfigs.async_profiler === 'enabled' ? 'Enabled' : 'Disabled'} - - - • Pyperf (Python):{' '} - {profilerConfigs.pyperf === 'enabled' ? 'Enabled' : 'Disabled'} - - - • Pyspy (Python):{' '} - {profilerConfigs.pyspy === 'enabled_fallback' - ? 'Enabled (Fallback)' - : profilerConfigs.pyspy === 'enabled' - ? 'Enabled' - : 'Disabled'} - - - • Rbspy (Ruby): {profilerConfigs.rbspy === 'enabled' ? 'Enabled' : 'Disabled'} - - - • PHPspy (PHP): {profilerConfigs.phpspy === 'enabled' ? 'Enabled' : 'Disabled'} - - - • .NET Trace:{' '} - {profilerConfigs.dotnet_trace === 'enabled' ? 'Enabled' : 'Disabled'} - - - • NodeJS Perf:{' '} - {profilerConfigs.nodejs_perf === 'enabled' ? 'Enabled' : 'Disabled'} - + • Perf (C/C++/Go): { + profilerConfigs.perf?.mode === 'enabled_restricted' ? 'Enabled (Restricted)' : + profilerConfigs.perf?.mode === 'enabled_aggressive' ? 'Enabled (Aggressive)' : 'Disabled' + } + • Java Async Profiler: { + profilerConfigs.async_profiler?.enabled + ? `Enabled (${ + profilerConfigs.async_profiler.time === 'wall' ? 'Wall Time' : + profilerConfigs.async_profiler.time === 'itimer' ? 'ITimer' : + profilerConfigs.async_profiler.time === 'auto' ? 'Auto' : + profilerConfigs.async_profiler.time === 'alloc' ? `Allocation (${profilerConfigs.async_profiler.alloc_interval || '2MB'})` : + 'CPU Time' + })` + : 'Disabled' + } + • Pyperf (Python): {profilerConfigs.pyperf === 'enabled' ? 'Enabled' : 'Disabled'} + • Pyspy (Python): { + profilerConfigs.pyspy === 'enabled_fallback' ? 'Enabled (Fallback)' : + profilerConfigs.pyspy === 'enabled' ? 'Enabled' : 'Disabled' + } + • Rbspy (Ruby): {profilerConfigs.rbspy === 'enabled' ? 'Enabled' : 'Disabled'} + • PHPspy (PHP): {profilerConfigs.phpspy === 'enabled' ? 'Enabled' : 'Disabled'} + • .NET Trace: {profilerConfigs.dotnet_trace === 'enabled' ? 'Enabled' : 'Disabled'} + • NodeJS Perf: {profilerConfigs.nodejs_perf === 'enabled' ? 'Enabled' : 'Disabled'} )} - + + + setSnackbar({ open: false, message: '' })} + anchorOrigin={{ vertical: 'bottom', horizontal: 'center' }} + > + setSnackbar({ open: false, message: '' })} + severity="success" + variant="filled" + sx={{ width: '100%' }} + > + {snackbar.message} + + ); }; diff --git a/src/gprofiler/frontend/src/components/console/header/ProfilingHeader.jsx b/src/gprofiler/frontend/src/components/console/header/ProfilingHeader.jsx index 58fcfa2e..826b0a7d 100644 --- a/src/gprofiler/frontend/src/components/console/header/ProfilingHeader.jsx +++ b/src/gprofiler/frontend/src/components/console/header/ProfilingHeader.jsx @@ -21,7 +21,11 @@ const ProfilingHeader = ({ filters, updateFilter, isLoading = false, onApplyFilt size='large' onClick={() => setFiltersExpanded(!filtersExpanded)} startIcon={} - endIcon={} + endIcon={ + + } sx={{ textTransform: 'none', color: COLORS.PRIMARY_BLUE, @@ -42,207 +46,329 @@ const ProfilingHeader = ({ filters, updateFilter, isLoading = false, onApplyFilt {/* Collapsible Filters Section */} - + - {/* Service Name Filter */} - updateFilter('service', e.target.value)} - placeholder='Filter by service...' - disabled={isLoading} - fullWidth - sx={{ - backgroundColor: 'white !important', - borderRadius: '4px', - boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', - '& .MuiOutlinedInput-root': { - backgroundColor: 'white !important', - '& fieldset': { - borderColor: 'rgba(0, 0, 0, 0.23)', - }, - '&:hover fieldset': { - borderColor: 'rgba(0, 0, 0, 0.87)', - }, - }, - '& .MuiInputBase-input': { - backgroundColor: 'white !important', - }, - }} - /> + {/* Service Name Filter */} + updateFilter('service', e.target.value)} + placeholder='Filter by service...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> - {/* Hostname Filter */} - updateFilter('hostname', e.target.value)} - placeholder='Filter by hostname...' - disabled={isLoading} - fullWidth - sx={{ - backgroundColor: 'white !important', - borderRadius: '4px', - boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', - '& .MuiOutlinedInput-root': { - backgroundColor: 'white !important', - '& fieldset': { - borderColor: 'rgba(0, 0, 0, 0.23)', - }, - '&:hover fieldset': { - borderColor: 'rgba(0, 0, 0, 0.87)', - }, - }, - '& .MuiInputBase-input': { - backgroundColor: 'white !important', - }, - }} - /> + {/* Hostname Filter */} + updateFilter('hostname', e.target.value)} + placeholder='Filter by hostname...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> - {/* PIDs Filter */} - updateFilter('pids', e.target.value)} - placeholder='Filter by PIDs...' - disabled={isLoading} - fullWidth - sx={{ - backgroundColor: 'white !important', - borderRadius: '4px', - boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', - '& .MuiOutlinedInput-root': { - backgroundColor: 'white !important', - '& fieldset': { - borderColor: 'rgba(0, 0, 0, 0.23)', - }, - '&:hover fieldset': { - borderColor: 'rgba(0, 0, 0, 0.87)', - }, - }, - '& .MuiInputBase-input': { - backgroundColor: 'white !important', - }, - }} - /> + {/* PIDs Filter */} + updateFilter('pids', e.target.value)} + placeholder='Filter by PIDs...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> - {/* IP Address Filter */} - updateFilter('ip', e.target.value)} - placeholder='Filter by IP...' - disabled={isLoading} - fullWidth - sx={{ - backgroundColor: 'white !important', - borderRadius: '4px', - boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', - '& .MuiOutlinedInput-root': { - backgroundColor: 'white !important', - '& fieldset': { - borderColor: 'rgba(0, 0, 0, 0.23)', - }, - '&:hover fieldset': { - borderColor: 'rgba(0, 0, 0, 0.87)', - }, - }, - '& .MuiInputBase-input': { - backgroundColor: 'white !important', - }, - }} - /> + {/* IP Address Filter */} + updateFilter('ip', e.target.value)} + placeholder='Filter by IP...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> - {/* Command Type Filter */} - - Command Type - - + {/* Namespace Filter */} + updateFilter('namespace', e.target.value)} + placeholder='Filter by namespace...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> - {/* Profiling Status Filter */} - - Profiling Status - - - + {/* Pod Filter */} + updateFilter('podName', e.target.value)} + placeholder='Filter by pod...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> + + {/* Container Filter */} + updateFilter('containerName', e.target.value)} + placeholder='Filter by container...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> + + {/* Process Filter */} + updateFilter('processName', e.target.value)} + placeholder='Filter by process...' + disabled={isLoading} + fullWidth + sx={{ + backgroundColor: 'white !important', + borderRadius: '4px', + boxShadow: '0px 2px 4px rgba(0, 0, 0, 0.1)', + '& .MuiOutlinedInput-root': { + backgroundColor: 'white !important', + '& fieldset': { + borderColor: 'rgba(0, 0, 0, 0.23)', + }, + '&:hover fieldset': { + borderColor: 'rgba(0, 0, 0, 0.87)', + }, + }, + '& .MuiInputBase-input': { + backgroundColor: 'white !important', + }, + }} + /> + {/* Command Type Filter */} + + Command Type + + + + {/* Profiling Status Filter */} + + Profiling Status + + + + {/* Action Buttons */} - + - - - {/* PerfSpect Hardware Metrics Checkbox */} + + {/* Basic Configuration */} + + + Basic Configuration + + + {/* Left column */} + onPerfSpectChange(e.target.checked)} size='small' color='primary' - disabled /> } label={ @@ -197,240 +200,370 @@ const ProfilingTopPanel = ({ PerfSpect HW Metrics } - sx={{ ml: 1 }} - disabled + sx={{ m: 0 }} /> - {/* Profiling Frequency Field */} - - - - Profiling Frequency: - + + + { - const value = parseInt(e.target.value, 10); - if (!isNaN(value) && value > 0 && value <= 1000) { - onProfilingFrequencyChange(value); - } + const value = clampInt(e.target.value, 1, 1000); + if (value !== null) onProfilingFrequencyChange(value); }} type='number' size='small' - inputProps={{ - min: 1, - max: 1000, - style: { textAlign: 'center' }, - }} - sx={{ - width: '70px', - '& .MuiOutlinedInput-root': { - height: '32px', - fontSize: '0.875rem', - }, - }} + inputProps={{ min: 1, max: 1000 }} + sx={fieldSx} /> - + Hz - + Sampling rate (11, 49, 99, 199 Hz) + - {/* Max Processes Field */} - - - - Max Processes: - + + + { - const value = parseInt(e.target.value, 10); - if (!isNaN(value) && value >= 0 && value <= 1000) { - onMaxProcessesChange(value); - } + const value = clampInt(e.target.value, 0, 1000); + if (value !== null) onMaxProcessesChange(value); }} type='number' size='small' - inputProps={{ - min: 0, - max: 1000, - style: { textAlign: 'center' }, - }} - sx={{ - width: '70px', - '& .MuiOutlinedInput-root': { - height: '32px', - fontSize: '0.875rem', - }, - }} + inputProps={{ min: 0, max: 1000 }} + sx={fieldSx} /> - - procs + + processes - - + + - {/* Profiling Mode and Duration Section */} - - {/* Profiling Mode Radio Buttons */} - - - - Mode: - - onProfilingModeChange(e.target.value)} - sx={{ gap: 1, ml: 1 }}> - } - label={ - - Continuous - - } - /> - } - label={ - - Ad Hoc - - } - /> - - - + {/* Right column */} + + + + onProfilingModeChange(e.target.value)}> + } + label={Ad Hoc} + /> + } + label={Continuous} + /> + + + {isAdhoc + ? 'Single profiling session with specified duration' + : 'Ongoing profiling (duration fixed at 60 seconds)'} + + - {/* Duration Field */} - + + - - Duration: - { - const value = parseInt(e.target.value, 10); - if (!isNaN(value) && value > 0 && value <= 3600) { - onDurationChange(value); - } + const value = clampInt(e.target.value, 1, 3600); + if (value !== null) onDurationChange(value); }} type='number' size='small' - disabled={profilingMode === 'continuous'} - inputProps={{ - min: 1, - max: 3600, - style: { textAlign: 'center' }, - }} + disabled={!isAdhoc} + inputProps={{ min: 1, max: 3600 }} sx={{ - width: '70px', + ...fieldSx, '& .MuiOutlinedInput-root': { - height: '32px', - fontSize: '0.875rem', + ...fieldSx['& .MuiOutlinedInput-root'], + backgroundColor: isAdhoc ? 'white' : '#f5f5f5', }, }} /> - - sec + + seconds - - + + + + - {/* Right side - Info and Clear Filters */} - }> - {hasActiveFilters && ( - - )} - - {rowsCount} hosts found - - - - - {/* Profiler Configuration Section */} - - - ▼} - sx={{ - backgroundColor: '#f5f5f5', - '& .MuiAccordionSummary-content': { - alignItems: 'center', - }, - }}> - - Click for Advanced Profiler Configuration - - - + {/* Advanced Profiler Configuration */} + + } + sx={{ '& .MuiAccordionSummary-content': { alignItems: 'center', gap: 1 } }}> + + Advanced Profiler Configuration + + - {profilerDefinitions.map((profiler) => ( - - - {profiler.name} - - - {profiler.description} + i + + + + + + {/* Perf Profiler */} + + + Perf Profiler + + + C, C++, Go, Kernel + + handlePerfProfilerConfigChange('mode', e.target.value)}> + + } + label={Enabled Restricted} + sx={{ mb: 0.5 }} + /> + + + } + label={Enabled Aggressive} + sx={{ mb: 0.5 }} + /> + + } + label={Disabled} + sx={{ mb: 0.5 }} + /> + + + {profilerConfigs.perf?.mode !== 'disabled' && ( + + + Event Types (select one or more): - handleProfilerConfigChange(profiler.key, e.target.value)}> - {profiler.options.map((option) => ( - + + {perfEvents.map((event) => ( + } - label={ - - {option.label} - + control={ + handlePerfEventToggle(event.value, e.target.checked)} + size='small' + /> } + label={{event.label}} sx={{ mb: 0.5 }} /> ))} + + + )} + + + {/* Async Profiler */} + + + Async Profiler + + + Java + + handleAsyncProfilerConfigChange('enabled', e.target.checked)} + size='small' + /> + } + label={Enabled} + sx={{ mb: 1 }} + /> + {profilerConfigs.async_profiler?.enabled && ( + + + Time Mode: + + handleAsyncProfilerConfigChange('time', e.target.value)}> + + } + label={CPU Time} + sx={{ mb: 0.5 }} + /> + + + } + label={ITimer} + sx={{ mb: 0.5 }} + /> + + + } + label={Wall Time} + sx={{ mb: 0.5 }} + /> + + + } + label={Auto} + sx={{ mb: 0.5 }} + /> + + + } + label={Allocation} + sx={{ mb: 0.5 }} + /> + + {profilerConfigs.async_profiler?.time === 'alloc' && ( + + + Allocation Interval: + + + handleAsyncProfilerConfigChange('alloc_interval', e.target.value)} + placeholder='e.g. 2MB, 512KiB' + style={{ + fontSize: '0.75rem', + padding: '4px 8px', + border: '1px solid #ccc', + borderRadius: '4px', + width: '120px', + }} + /> + + + )} - ))} + )} - - + + {/* Remaining language profilers */} + {profilerDefinitions.map((profiler) => ( + + + {profiler.name} + + + {profiler.description} + + handleProfilerConfigChange(profiler.key, e.target.value)}> + {profiler.options.map((option) => ( + + } + label={{option.label}} + sx={{ mb: 0.5 }} + /> + + ))} + + + ))} + + + + + {/* Action bar */} + + + + + + - + ); }; diff --git a/src/gprofiler/frontend/src/components/console/profilingRequestBuilder.mjs b/src/gprofiler/frontend/src/components/console/profilingRequestBuilder.mjs new file mode 100644 index 00000000..7af52746 --- /dev/null +++ b/src/gprofiler/frontend/src/components/console/profilingRequestBuilder.mjs @@ -0,0 +1,142 @@ +/* + * Pure (React-free) helpers that translate selected inventory rows into the + * payload accepted by POST /api/metrics/profile_request[/bulk]. + * + * This module is intentionally dependency-free so the request-shaping "spec" + * can be exercised by fast unit/acceptance tests (see + * profilingRequestBuilder.test.mjs) without spinning up React or a browser. + * + * Contract (kept in sync with the backend ProfilingRequest validator): + * - stop_level is "host" for the coarse scopes (service, host) and "process" + * for the finer scopes; it is only set for stop requests. + * - A host-/service-level stop must NOT carry any PID (neither in + * target_hosts nor in target_entities), otherwise the backend rejects it. + * - Process-level identifiers (pid, process_name) are therefore only attached + * to target entities when the active scope is "process". + */ + +// Scopes that resolve to whole-host targets (no per-process PIDs). +export const HOST_LEVEL_SCOPES = ['service', 'host']; + +const CONTINUOUS_DURATION_SECONDS = 60; + +/** + * Build a single target entity descriptor for a selected row. + * + * Process-level identifiers (pid, process_name) are only included for the + * "process" scope. Attaching them to coarser scopes is unnecessary for target + * resolution and, for stop requests, trips the backend validation that forbids + * PIDs when stop_level is "host". + * + * @param {object} row - Normalized inventory row. + * @param {string} scope - Active scope (service|namespace|host|pod|container|process). + * @returns {object} Target entity payload. + */ +export const buildTargetEntity = (row, scope) => { + const entity = { + id: row.id, + service_name: row.service, + namespace: row.namespace || undefined, + hostname: row.host || undefined, + ip_address: row.ip || undefined, + pod_name: row.podName || undefined, + container_name: row.containerName || undefined, + workload_name: row.workloadName || undefined, + workload_kind: row.workloadKind || undefined, + }; + + if (scope === 'process') { + // Use a null check (not truthiness) so a legitimate PID of 0 is preserved. + entity.pid = row.pid == null ? undefined : row.pid; + entity.process_name = row.processName || undefined; + } + + return entity; +}; + +/** + * Resolve the stop_level for a request given the action and scope. + * @returns {string|undefined} "host" | "process" for stops, undefined otherwise. + */ +export const resolveStopLevel = (action, scope) => { + if (action !== 'stop') { + return undefined; + } + return HOST_LEVEL_SCOPES.includes(scope) ? 'host' : 'process'; +}; + +/** + * Group selected rows by their service name (preserving selection order). + * @param {Array} selectedRows + * @returns {Record>} + */ +export const groupRowsByService = (selectedRows) => + selectedRows.reduce((groups, row) => { + if (!groups[row.service]) { + groups[row.service] = []; + } + groups[row.service].push(row); + return groups; + }, {}); + +/** + * Build the per-service profiling requests for a bulk start/stop action. + * + * @param {'start'|'stop'} action + * @param {Array} selectedRows - Normalized inventory rows. + * @param {object} config + * @param {string} config.scope - Active scope. + * @param {'continuous'|'adhoc'|string} config.profilingMode + * @param {number} config.duration + * @param {number} config.profilingFrequency + * @param {boolean} config.enablePerfSpect + * @param {object} config.profilerConfigs + * @param {number} config.maxProcesses + * @returns {{grouped: Record>, requests: Array}} + */ +export const buildProfilingRequests = (action, selectedRows, config) => { + const { + scope, + profilingMode, + duration, + profilingFrequency, + enablePerfSpect, + profilerConfigs, + maxProcesses, + } = config; + + const grouped = groupRowsByService(selectedRows); + const isContinuous = profilingMode === 'continuous'; + const stopLevel = resolveStopLevel(action, scope); + + const requests = Object.entries(grouped).map(([serviceName, serviceRows]) => { + const targetHosts = + scope === 'host' + ? serviceRows.reduce((hostMap, row) => { + // null => whole-host target (no PIDs) per the backend contract. + hostMap[row.host] = null; + return hostMap; + }, {}) + : undefined; + + return { + service_name: serviceName, + request_type: action, + continuous: isContinuous, + duration: isContinuous ? CONTINUOUS_DURATION_SECONDS : duration, + frequency: profilingFrequency, + profiling_mode: 'cpu', + target_scope: scope, + target_hosts: targetHosts, + target_entities: serviceRows.map((row) => buildTargetEntity(row, scope)), + stop_level: stopLevel, + additional_args: { + enable_perfspect: enablePerfSpect, + profiler_configs: profilerConfigs, + max_processes: maxProcesses, + }, + }; + }); + + return { grouped, requests }; +}; diff --git a/src/gprofiler/frontend/src/components/console/profilingRequestBuilder.test.mjs b/src/gprofiler/frontend/src/components/console/profilingRequestBuilder.test.mjs new file mode 100644 index 00000000..c3ee24d6 --- /dev/null +++ b/src/gprofiler/frontend/src/components/console/profilingRequestBuilder.test.mjs @@ -0,0 +1,296 @@ +/* + * Acceptance / spec tests for the profiling request builder. + * + * These encode the executable specification for how the Profiling Status page + * turns selected inventory rows into POST /api/metrics/profile_request payloads. + * They run with Node's built-in test runner (no extra dependencies): + * + * node --test src/gprofiler/frontend/src/components/console/profilingRequestBuilder.test.mjs + * + * The central invariant (which the original host/service stop bug violated): + * A host- or service-level STOP must never carry a PID, or the backend + * ProfilingRequest validator rejects it with + * 'No PIDs should be provided when request_type is "stop" and stop_level is "host"'. + */ + +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; + +import { + HOST_LEVEL_SCOPES, + buildProfilingRequests, + buildTargetEntity, + groupRowsByService, + resolveStopLevel, +} from './profilingRequestBuilder.mjs'; + +const baseConfig = { + scope: 'host', + profilingMode: 'adhoc', + duration: 120, + profilingFrequency: 11, + enablePerfSpect: false, + profilerConfigs: {}, + maxProcesses: 10, +}; + +const makeRow = (overrides = {}) => ({ + id: 'row-1', + service: 'svc-a', + namespace: 'ns-a', + host: 'host-a', + ip: '10.0.0.1', + podName: 'pod-a', + containerName: 'cont-a', + workloadName: 'wl-a', + workloadKind: 'Deployment', + processName: 'python', + pid: 4242, + ...overrides, +}); + +// Every target entity produced for a request, flattened. +const allEntities = (requests) => requests.flatMap((r) => r.target_entities); +// True when any produced entity carries a PID (what the backend forbids for host stops). +const anyEntityHasPid = (requests) => allEntities(requests).some((e) => e.pid !== undefined); + +describe('buildTargetEntity', () => { + it('omits process-level identifiers (pid/process_name) for coarse scopes', () => { + for (const scope of ['service', 'host', 'namespace', 'workload', 'pod', 'container']) { + const entity = buildTargetEntity(makeRow(), scope); + assert.equal(entity.pid, undefined, `pid must be undefined for scope=${scope}`); + assert.equal( + entity.process_name, + undefined, + `process_name must be undefined for scope=${scope}` + ); + } + }); + + it('includes pid and process_name for the process scope', () => { + const entity = buildTargetEntity(makeRow({ pid: 4242, processName: 'python' }), 'process'); + assert.equal(entity.pid, 4242); + assert.equal(entity.process_name, 'python'); + }); + + it('preserves a legitimate PID of 0 (not dropped as falsy)', () => { + const entity = buildTargetEntity(makeRow({ pid: 0 }), 'process'); + assert.equal(entity.pid, 0); + }); + + it('treats a missing PID as undefined for the process scope', () => { + const entity = buildTargetEntity(makeRow({ pid: null }), 'process'); + assert.equal(entity.pid, undefined); + }); + + it('always carries the identifying fields needed for host-scope resolution', () => { + const entity = buildTargetEntity(makeRow(), 'host'); + assert.equal(entity.service_name, 'svc-a'); + assert.equal(entity.hostname, 'host-a'); + }); +}); + +describe('resolveStopLevel', () => { + it('is undefined for start requests regardless of scope', () => { + for (const scope of ['service', 'host', 'process']) { + assert.equal(resolveStopLevel('start', scope), undefined); + } + }); + + it('is "host" for coarse (service/host) stop scopes', () => { + for (const scope of HOST_LEVEL_SCOPES) { + assert.equal(resolveStopLevel('stop', scope), 'host'); + } + }); + + it('is "process" for finer stop scopes', () => { + for (const scope of ['namespace', 'workload', 'pod', 'container', 'process']) { + assert.equal(resolveStopLevel('stop', scope), 'process'); + } + }); +}); + +describe('groupRowsByService', () => { + it('groups rows by service preserving order', () => { + const grouped = groupRowsByService([ + makeRow({ id: '1', service: 'svc-a' }), + makeRow({ id: '2', service: 'svc-b' }), + makeRow({ id: '3', service: 'svc-a' }), + ]); + assert.deepEqual(Object.keys(grouped), ['svc-a', 'svc-b']); + assert.equal(grouped['svc-a'].length, 2); + assert.equal(grouped['svc-b'].length, 1); + }); +}); + +describe('buildProfilingRequests — host-level stop (the regression)', () => { + it('produces a host-level stop with no PIDs anywhere', () => { + const { requests } = buildProfilingRequests('stop', [makeRow()], { + ...baseConfig, + scope: 'host', + }); + + assert.equal(requests.length, 1); + const [req] = requests; + assert.equal(req.request_type, 'stop'); + assert.equal(req.stop_level, 'host'); + // target_hosts maps host -> null (whole-host target, no PIDs). + assert.deepEqual(req.target_hosts, { 'host-a': null }); + // No target entity may carry a PID. + assert.equal(anyEntityHasPid(requests), false); + }); + + it('produces a service-level stop with no PIDs and no target_hosts map', () => { + const { requests } = buildProfilingRequests('stop', [makeRow()], { + ...baseConfig, + scope: 'service', + }); + + const [req] = requests; + assert.equal(req.stop_level, 'host'); + assert.equal(req.target_scope, 'service'); + assert.equal(req.target_hosts, undefined); + assert.equal(anyEntityHasPid(requests), false); + }); +}); + +describe('buildProfilingRequests — process-level stop', () => { + it('is a process-level stop that includes the PID', () => { + const { requests } = buildProfilingRequests('stop', [makeRow({ pid: 4242 })], { + ...baseConfig, + scope: 'process', + }); + + const [req] = requests; + assert.equal(req.stop_level, 'process'); + assert.equal(req.target_hosts, undefined); + assert.equal(req.target_entities[0].pid, 4242); + assert.equal(anyEntityHasPid(requests), true); + }); +}); + +describe('buildProfilingRequests — start requests', () => { + it('never sets stop_level for a start', () => { + for (const scope of ['service', 'host', 'process']) { + const { requests } = buildProfilingRequests('start', [makeRow()], { + ...baseConfig, + scope, + }); + assert.equal(requests[0].stop_level, undefined); + } + }); + + it('forces continuous mode to a 60s duration', () => { + const { requests } = buildProfilingRequests('start', [makeRow()], { + ...baseConfig, + profilingMode: 'continuous', + duration: 999, + }); + assert.equal(requests[0].continuous, true); + assert.equal(requests[0].duration, 60); + }); + + it('uses the configured duration for ad-hoc mode', () => { + const { requests } = buildProfilingRequests('start', [makeRow()], { + ...baseConfig, + profilingMode: 'adhoc', + duration: 120, + }); + assert.equal(requests[0].continuous, false); + assert.equal(requests[0].duration, 120); + }); +}); + +describe('buildProfilingRequests — multi-service', () => { + it('emits one request per service', () => { + const { requests } = buildProfilingRequests( + 'stop', + [ + makeRow({ id: '1', service: 'svc-a', host: 'h1' }), + makeRow({ id: '2', service: 'svc-b', host: 'h2' }), + ], + { ...baseConfig, scope: 'host' } + ); + + assert.equal(requests.length, 2); + assert.deepEqual( + requests.map((r) => r.service_name).sort(), + ['svc-a', 'svc-b'] + ); + assert.equal(anyEntityHasPid(requests), false); + }); +}); + +/* + * The client half of the WORKLOAD_LEVEL_PROFILING_SPEC.md resolution contract. + * The backend does the actual host/PID resolution, but the request the UI emits + * must carry exactly the selectors each scope needs — and nothing that would be + * rejected. These map to AT-S5 (host start), AT-S6 (service fan-out), and AT-S7 + * (workload scopes resolve to PIDs). + */ +describe('buildProfilingRequests — scope matrix (start)', () => { + // AT-S5: a host-scope start targets the concrete host via target_hosts. + it('host scope emits a target_hosts map and no PIDs', () => { + const { requests } = buildProfilingRequests('start', [makeRow({ host: 'host-a' })], { + ...baseConfig, + scope: 'host', + }); + assert.deepEqual(requests[0].target_hosts, { 'host-a': null }); + assert.equal(anyEntityHasPid(requests), false); + }); + + // AT-S6: a service-scope start carries no target_hosts (backend fans out to + // every current host of the service) but still names the service. + it('service scope carries the service selector and no target_hosts', () => { + const { requests } = buildProfilingRequests( + 'start', + [makeRow({ service: 'svc-a', host: 'h1' }), makeRow({ id: '2', service: 'svc-a', host: 'h2' })], + { ...baseConfig, scope: 'service' } + ); + assert.equal(requests.length, 1); + assert.equal(requests[0].service_name, 'svc-a'); + assert.equal(requests[0].target_hosts, undefined); + assert.equal(requests[0].target_scope, 'service'); + assert.equal(anyEntityHasPid(requests), false); + }); + + // AT-S7: namespace/workload/pod/container select via entity metadata (no PID + // on the client — the backend resolves these to PIDs), while `process` + // carries the concrete PID. + it('sub-host scopes attach the right selectors; only process carries a PID', () => { + const cases = { + namespace: (e) => assert.equal(e.namespace, 'ns-a'), + workload: (e) => assert.equal(e.workload_name, 'wl-a'), + pod: (e) => assert.equal(e.pod_name, 'pod-a'), + container: (e) => assert.equal(e.container_name, 'cont-a'), + }; + for (const [scope, assertSelector] of Object.entries(cases)) { + const { requests } = buildProfilingRequests('start', [makeRow()], { ...baseConfig, scope }); + const [entity] = requests[0].target_entities; + assertSelector(entity); + assert.equal(entity.pid, undefined, `pid must be undefined for scope=${scope}`); + } + + const { requests: procReqs } = buildProfilingRequests('start', [makeRow({ pid: 4242 })], { + ...baseConfig, + scope: 'process', + }); + assert.equal(procReqs[0].target_entities[0].pid, 4242); + }); + + it('threads perf/perfspect config through additional_args unchanged', () => { + const profilerConfigs = { perf: { mode: 'enabled_restricted', events: ['cycles'] } }; + const { requests } = buildProfilingRequests('start', [makeRow()], { + ...baseConfig, + scope: 'service', + enablePerfSpect: true, + profilerConfigs, + maxProcesses: 25, + }); + assert.deepEqual(requests[0].additional_args, { + enable_perfspect: true, + profiler_configs: profilerConfigs, + max_processes: 25, + }); + }); +}); diff --git a/src/gprofiler/frontend/src/components/filters/form/FilterForm.jsx b/src/gprofiler/frontend/src/components/filters/form/FilterForm.jsx index 83fd4ab9..07424a00 100644 --- a/src/gprofiler/frontend/src/components/filters/form/FilterForm.jsx +++ b/src/gprofiler/frontend/src/components/filters/form/FilterForm.jsx @@ -136,7 +136,11 @@ const FilterForm = ({ postCreateCallback, onClose, valueOptions, valueOptionsLoa diff --git a/src/gprofiler/frontend/src/components/filters/select/FilterOptionsSelect.jsx b/src/gprofiler/frontend/src/components/filters/select/FilterOptionsSelect.jsx index dd94837c..f2cb8785 100644 --- a/src/gprofiler/frontend/src/components/filters/select/FilterOptionsSelect.jsx +++ b/src/gprofiler/frontend/src/components/filters/select/FilterOptionsSelect.jsx @@ -151,26 +151,41 @@ const StyledPopper = styled(Popper)({ }, }); -const FilterOptionsSelect = ({ disabled = false, loading, value, onChange, options }) => { +const FilterOptionsSelect = ({ disabled = false, loading, value, onChange, options, freeSolo = false }) => { const [inputValue, setInputValue] = useState(value); useEffect(() => { setInputValue(value); }, [value]); + const commitValue = (newValue) => { + const name = typeof newValue === 'string' ? newValue : newValue?.name; + if (!name) { + return; + } + setInputValue(name); + onChange(name); + }; + return ( option?.name || ''} + getOptionLabel={(option) => (typeof option === 'string' ? option : option?.name || '')} isOptionEqualToValue={(option, value) => option.name === value} filterOptions={(options, { inputValue }) => options.filter((item) => item.name.includes(inputValue))} onInputChange={(event, value, reason) => { if (event && event.type === 'blur') { - setInputValue(''); + // In contains (free-text) mode keep the typed substring instead of clearing it. + if (freeSolo && value) { + commitValue(value); + } else { + setInputValue(''); + } } else if (reason !== 'reset') { setInputValue(value); } @@ -179,8 +194,7 @@ const FilterOptionsSelect = ({ disabled = false, loading, value, onChange, optio if (!newValue || (event.type === 'keydown' && event.key === 'Backspace')) { return; } - setInputValue(newValue.name); - onChange(newValue.name); + commitValue(newValue); }} renderTags={() => null} noOptionsText={'no options'} diff --git a/src/gprofiler/frontend/src/hooks/useFiltersQueryParams.js b/src/gprofiler/frontend/src/hooks/useFiltersQueryParams.js index 1a48bb63..9d513c70 100644 --- a/src/gprofiler/frontend/src/hooks/useFiltersQueryParams.js +++ b/src/gprofiler/frontend/src/hooks/useFiltersQueryParams.js @@ -91,6 +91,7 @@ const useFiltersQueryParams = ({ wp: NumberParam, }); const [isMounted, setMounted] = useState(false); + const [scalarFiltersApplied, setScalarFiltersApplied] = useState(false); const { rt: runtimeFilters, rtms: runTimeMixedStacks, @@ -124,14 +125,34 @@ const useFiltersQueryParams = ({ } }, [filters, handleFilterChange, location.pathname]); + // Runtime, mixed-stacks and weight filters do not depend on the flamegraph + // data, so restore them from the URL on mount even when the selected time + // window is empty (otherwise a deep link into an empty window drops them). + useEffect(() => { + if (scalarFiltersApplied || location.pathname !== PAGES.profiles.to) { + return; + } + if (runtimeFilters) { + setRuntimeFilters(parseQueryParamsToFilters(runtimeFilters)); + } + if (runTimeMixedStacks && !filters.runtime.isMixedRuntimeStacksModeEnabled) { + setIsMixedRuntimeStacksModeEnabled(); + } + if (weightThreshold) { + setWeightThreshold(weightThreshold); + } + if (weightPercentile) { + setWeightPercentile(weightPercentile); + } + setScalarFiltersApplied(true); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [location.pathname]); + + // The process filter is stored as an inclusion/exclusion list and can only + // be resolved against the flamegraph's process list, so it must wait for + // data to arrive. useEffect(() => { if (!isMounted && filters.processes.processesList?.length > 0) { - if (runtimeFilters) { - setRuntimeFilters(parseQueryParamsToFilters(runtimeFilters)); - } - if (runTimeMixedStacks && !filters.runtime.isMixedRuntimeStacksModeEnabled) { - setIsMixedRuntimeStacksModeEnabled(); - } if (processes?.length > 0) { let proccessListValues = filters.processes.processesList.map((proccess) => proccess.value); let approvedFilters = []; @@ -147,12 +168,6 @@ const useFiltersQueryParams = ({ } setProcessesFilters(approvedFilters); } - if (weightThreshold) { - setWeightThreshold(weightThreshold); - } - if (weightPercentile) { - setWeightPercentile(weightPercentile); - } setMounted(true); } diff --git a/src/gprofiler/frontend/src/hooks/useMainFiltersQueryParams.js b/src/gprofiler/frontend/src/hooks/useMainFiltersQueryParams.js index f3072b8f..e80b8111 100644 --- a/src/gprofiler/frontend/src/hooks/useMainFiltersQueryParams.js +++ b/src/gprofiler/frontend/src/hooks/useMainFiltersQueryParams.js @@ -43,10 +43,12 @@ const mapOperatorToQueryParam = { const mapEqualToQueryParam = { [FILTER_EQUALNESS.$eq.value]: 'is', [FILTER_EQUALNESS.$neq.value]: 'not', + [FILTER_EQUALNESS.$like.value]: 'has', }; const mapQueryParamToEqual = { is: [FILTER_EQUALNESS.$eq.value], not: [FILTER_EQUALNESS.$neq.value], + has: [FILTER_EQUALNESS.$like.value], }; const delimiter = ','; @@ -115,9 +117,10 @@ const useMainFiltersQueryParams = ({ activeFilterTag, setActiveFilterTag }) => { if (filter) { const parsedFilter = parseQueryParamsToFilter(filter); - setTimeout(() => { - setActiveFilterTag({ id: '', filter: parsedFilter }); - }, 3000); + // Apply the URL filter synchronously on mount. A previous 3s + // setTimeout here caused the view to load once unfiltered and then + // reload with the filter a beat later. + setActiveFilterTag({ id: '', filter: parsedFilter }); } // eslint-disable-next-line react-hooks/exhaustive-deps }, []); diff --git a/src/gprofiler/frontend/src/states/filters/FiltersTagsContext.jsx b/src/gprofiler/frontend/src/states/filters/FiltersTagsContext.jsx index b9246c60..92498d30 100644 --- a/src/gprofiler/frontend/src/states/filters/FiltersTagsContext.jsx +++ b/src/gprofiler/frontend/src/states/filters/FiltersTagsContext.jsx @@ -17,7 +17,7 @@ } import _ from 'lodash'; -import { createContext, useCallback, useContext, useEffect, useReducer, useState } from 'react'; +import { createContext, useCallback, useContext, useEffect, useReducer, useRef, useState } from 'react'; import useGetServiceFilters from '../../api/filters/useGetServiceFilters'; import useMainFiltersQueryParams from '../../hooks/useMainFiltersQueryParams'; @@ -87,11 +87,20 @@ export const FilterTagsContextProvider = ({ children }) => { } }, [dispatchFilterTags, activeFilterTag]); + // Clear filters only when the user switches to a *different* service, not on + // the initial selection — otherwise a deep link's URL filters get wiped before + // they are applied. + const previousServiceRef = useRef(undefined); useEffect(() => { - if (selectedService) { + if ( + selectedService && + previousServiceRef.current !== undefined && + previousServiceRef.current !== selectedService + ) { setActiveFilterTag(undefined); dispatchFilterTags({ type: FILTER_TAGS_ACTIONS.CLEAR }); } + previousServiceRef.current = selectedService; }, [selectedService]); return ( @@ -149,7 +158,6 @@ const reducer = (state, action) => { [operation]: rules2.map((rule, ruleIndex) => { if (ruleIndex === index) { const [, subItem] = Object.entries(rule)[0]; - const [equal, value] = Object.entries(subItem)[0]; return { [payload]: { [equal]: '' } }; } diff --git a/src/gprofiler/frontend/src/utils/consts.js b/src/gprofiler/frontend/src/utils/consts.js index b44111df..0edd41e8 100644 --- a/src/gprofiler/frontend/src/utils/consts.js +++ b/src/gprofiler/frontend/src/utils/consts.js @@ -49,7 +49,7 @@ export const PAGES = { }, profiling: { key: 'profiling', - label: 'Dynamic Profiling', + label: 'Profile Configurations', to: '/profiling', }, }; @@ -58,7 +58,7 @@ export const EXTERNAL_URLS = { documentation: { key: 'documentation', label: 'documentation', - to: 'https://docs.gprofiler.io/', + to: import.meta.env.VITE_DOCUMENTATION_URL || 'https://docs.gprofiler.io/', }, github: { key: 'github', diff --git a/src/gprofiler/frontend/src/utils/datetimesUtils.js b/src/gprofiler/frontend/src/utils/datetimesUtils.js index f4953638..f0d35c5e 100644 --- a/src/gprofiler/frontend/src/utils/datetimesUtils.js +++ b/src/gprofiler/frontend/src/utils/datetimesUtils.js @@ -33,6 +33,8 @@ import { sub, } from 'date-fns'; +import { getDateLocale } from './localeUtils'; + export const TIME_UNITS = { minutes: 'minutes', seconds: 'seconds', @@ -42,15 +44,15 @@ export const TIME_UNITS = { export const TIME_FORMATS = { DATETIME_BASIC: `yyyy-MM-dd'T'HH:mm:00`, - DATETIME_PRINTED: 'dd/MM/yyyy HH:mm', - DATETIME_WITH_SECONDS: 'dd/MM/yyyy HH:mm:ss', - DATE_BASIC: 'dd/MM/yyyy', + DATETIME_PRINTED: 'Pp', + DATETIME_WITH_SECONDS: 'PPpp', + DATE_BASIC: 'P', + TIME_LOCALE: 'p', TIME_24H: 'HH:mm', }; -// Datetime formatting export const formatDate = (date, toFormat = TIME_FORMATS.DATETIME_BASIC) => { - return format(date, toFormat); + return format(date, toFormat, { locale: getDateLocale() }); }; export const localDatetimeToUtc = (datetime) => { @@ -130,5 +132,5 @@ export const isDateYesterday = (date) => { // Datetimes creation export const getCurrentTime = () => { - return formatDate(new Date(), TIME_FORMATS.TIME_24H); + return formatDate(new Date(), TIME_FORMATS.TIME_LOCALE); }; diff --git a/src/gprofiler/frontend/src/utils/filtersUtils.jsx b/src/gprofiler/frontend/src/utils/filtersUtils.jsx index a34f44bb..4c388548 100644 --- a/src/gprofiler/frontend/src/utils/filtersUtils.jsx +++ b/src/gprofiler/frontend/src/utils/filtersUtils.jsx @@ -148,4 +148,5 @@ export const FILTER_OPERATIONS = { export const FILTER_EQUALNESS = { $eq: { value: '$eq', display: 'Is' }, $neq: { value: '$neq', display: 'Is not' }, + $like: { value: '$like', display: 'Contains' }, }; diff --git a/src/gprofiler/frontend/src/utils/localeUtils.js b/src/gprofiler/frontend/src/utils/localeUtils.js new file mode 100644 index 00000000..95598323 --- /dev/null +++ b/src/gprofiler/frontend/src/utils/localeUtils.js @@ -0,0 +1,98 @@ +{ + /* + * Copyright (C) 2023 Intel Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +} + +import { enUS } from 'date-fns/locale'; + +let cachedLocale = enUS; +let initialized = false; + +const LOCALE_MAP = { + 'en-US': () => import('date-fns/locale/en-US'), + 'en-GB': () => import('date-fns/locale/en-GB'), + 'en-AU': () => import('date-fns/locale/en-AU'), + 'en-CA': () => import('date-fns/locale/en-CA'), + 'en-IN': () => import('date-fns/locale/en-IN'), + 'en-NZ': () => import('date-fns/locale/en-NZ'), + 'en-IE': () => import('date-fns/locale/en-IE'), + 'en-ZA': () => import('date-fns/locale/en-ZA'), + 'pt-BR': () => import('date-fns/locale/pt-BR'), + pt: () => import('date-fns/locale/pt'), + es: () => import('date-fns/locale/es'), + fr: () => import('date-fns/locale/fr'), + de: () => import('date-fns/locale/de'), + it: () => import('date-fns/locale/it'), + ja: () => import('date-fns/locale/ja'), + ko: () => import('date-fns/locale/ko'), + 'zh-CN': () => import('date-fns/locale/zh-CN'), + 'zh-TW': () => import('date-fns/locale/zh-TW'), + ru: () => import('date-fns/locale/ru'), + ar: () => import('date-fns/locale/ar'), + he: () => import('date-fns/locale/he'), + hi: () => import('date-fns/locale/hi'), + nl: () => import('date-fns/locale/nl'), + pl: () => import('date-fns/locale/pl'), + sv: () => import('date-fns/locale/sv'), + da: () => import('date-fns/locale/da'), + fi: () => import('date-fns/locale/fi'), + nb: () => import('date-fns/locale/nb'), + tr: () => import('date-fns/locale/tr'), + th: () => import('date-fns/locale/th'), + vi: () => import('date-fns/locale/vi'), + uk: () => import('date-fns/locale/uk'), + cs: () => import('date-fns/locale/cs'), + ro: () => import('date-fns/locale/ro'), + hu: () => import('date-fns/locale/hu'), + el: () => import('date-fns/locale/el'), + id: () => import('date-fns/locale/id'), + ms: () => import('date-fns/locale/ms'), +}; + +function findLocaleImporter(browserLocale) { + if (LOCALE_MAP[browserLocale]) { + return LOCALE_MAP[browserLocale]; + } + + const langOnly = browserLocale.split('-')[0]; + if (LOCALE_MAP[langOnly]) { + return LOCALE_MAP[langOnly]; + } + + return null; +} + +export async function initDateLocale() { + if (initialized) return; + + const browserLocale = navigator.language; + const importer = findLocaleImporter(browserLocale); + + if (importer) { + try { + const mod = await importer(); + cachedLocale = mod.default || mod; + } catch { + // Keep enUS fallback + } + } + + initialized = true; +} + +export function getDateLocale() { + return cachedLocale; +} diff --git a/src/gprofiler/requirements.txt b/src/gprofiler/requirements.txt index 49748ad6..861ecf90 100644 --- a/src/gprofiler/requirements.txt +++ b/src/gprofiler/requirements.txt @@ -25,3 +25,4 @@ fastapi==0.109.0 uvicorn==0.27.0 gunicorn==23.0.0 slack_sdk==3.36.0 +bitmath==2.1.1 diff --git a/src/tests/e2e/Dockerfile b/src/tests/e2e/Dockerfile new file mode 100644 index 00000000..9d8d2b87 --- /dev/null +++ b/src/tests/e2e/Dockerfile @@ -0,0 +1,20 @@ +# In-network acceptance/UI test runner for the e2e harness. +# Runs pytest (API acceptance) from inside the compose network, so it can reach +# Postgres/ClickHouse/webapp by service name without exposing host ports. +FROM python:3.11-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 + +WORKDIR /work + +# Kept minimal on purpose; extended with playwright in the UI phase. +RUN pip install --no-cache-dir \ + pytest==8.2.0 \ + requests==2.32.3 \ + psycopg2-binary==2.9.9 + +# Test sources are mounted or copied at run time. +COPY tests/e2e /work/tests/e2e + +CMD ["pytest", "-q", "tests/e2e"] diff --git a/src/tests/e2e/README.md b/src/tests/e2e/README.md new file mode 100644 index 00000000..246253b3 --- /dev/null +++ b/src/tests/e2e/README.md @@ -0,0 +1,16 @@ +# API acceptance tests (AT-S1 .. AT-S15) + +In-network pytest suite that drives the **live** studio stack over HTTP and +codifies the workload-level profiling acceptance criteria from +[`heartbeat_doc/WORKLOAD_LEVEL_PROFILING_SPEC.md`](../../../heartbeat_doc/WORKLOAD_LEVEL_PROFILING_SPEC.md). + +Do not run these directly — they need the full stack up. Use the e2e harness: + +```bash +cd ../../../deploy +make -f Makefile.e2e e2e-up # or e2e-up-src (source-built agent) +make -f Makefile.e2e e2e-test # builds this runner and executes it in-network +``` + +Full harness docs (topology, LocalStack S3/SQS, portability): +[`deploy/E2E_HARNESS.md`](../../../deploy/E2E_HARNESS.md). diff --git a/src/tests/e2e/conftest.py b/src/tests/e2e/conftest.py new file mode 100644 index 00000000..b05c18f1 --- /dev/null +++ b/src/tests/e2e/conftest.py @@ -0,0 +1,14 @@ +"""Fixtures for the in-network e2e acceptance suite. + +Runs from a container on the compose network (see docker-compose.e2e.yml) so +services are reachable by their compose names (webapp, db_postgres, ...). Also +works from the host against nginx with basic auth via E2E_* env vars. +""" +import pytest + +from harness import Client + + +@pytest.fixture(scope="session") +def client() -> Client: + return Client() diff --git a/src/tests/e2e/harness.py b/src/tests/e2e/harness.py new file mode 100644 index 00000000..2590b3ff --- /dev/null +++ b/src/tests/e2e/harness.py @@ -0,0 +1,225 @@ +"""HTTP harness helpers for the in-network e2e acceptance suite. + +The suite drives the *live* studio stack over HTTP (heartbeat ingress, profile +requests, workload status). It is designed to run from a container on the +compose network (E2E_BASE_URL=http://webapp, no auth) but also works from the +host against nginx (E2E_BASE_URL=https://localhost:4433 with basic auth). + +Everything is keyed on unique service/host names per test, so runs are isolated +from each other and from any real agent reporting into the same stack. +""" +from __future__ import annotations + +import os +import uuid +from datetime import datetime, timedelta +from typing import Any, Dict, List, Optional + +import requests +import urllib3 + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +HEARTBEAT = "/api/metrics/heartbeat" +PROFILE_REQUEST = "/api/metrics/profile_request" +PROFILE_REQUEST_BULK = "/api/metrics/profile_request/bulk" +WORKLOAD_STATUS = "/api/metrics/profiling/workload_status" + +# Freshness window used by workload_status (see db_manager: INTERVAL '2 minutes'). +FRESHNESS_SECONDS = 120 + + +class Client: + """Thin requests wrapper that targets the studio API.""" + + def __init__(self) -> None: + self.base = os.environ.get("E2E_BASE_URL", "http://webapp").rstrip("/") + user = os.environ.get("E2E_BASIC_AUTH_USER") + password = os.environ.get("E2E_BASIC_AUTH_PASSWORD") + self.auth = (user, password) if user else None + self.session = requests.Session() + + def post(self, path: str, payload: Dict[str, Any]) -> requests.Response: + return self.session.post( + self.base + path, json=payload, auth=self.auth, verify=False, timeout=30 + ) + + def get(self, path: str, params: Optional[Dict[str, Any]] = None) -> requests.Response: + return self.session.get( + self.base + path, params=params, auth=self.auth, verify=False, timeout=30 + ) + + +# --------------------------------------------------------------------------- # +# Payload builders +# --------------------------------------------------------------------------- # + +def unique(prefix: str) -> str: + return f"{prefix}-{uuid.uuid4().hex[:8]}" + + +def process(pid: int, name: str = "java") -> Dict[str, Any]: + return {"pid": pid, "process_name": name} + + +def container( + container_name: str, + processes: Optional[List[Dict[str, Any]]] = None, + *, + namespace: Optional[str] = None, + pod_name: Optional[str] = None, + workload_name: Optional[str] = None, + workload_kind: Optional[str] = None, + container_id: Optional[str] = None, +) -> Dict[str, Any]: + return { + "container_id": container_id or unique("cid"), + "container_name": container_name, + "namespace": namespace, + "pod_name": pod_name, + "workload_name": workload_name, + "workload_kind": workload_kind, + "processes": processes or [], + } + + +def heartbeat_payload( + hostname: str, + service_name: str, + *, + namespace: Optional[str] = None, + pod_name: Optional[str] = None, + containers: Optional[List[Dict[str, Any]]] = None, + perf_supported_events: Optional[List[str]] = None, + age_seconds: float = 0.0, + agent_version: str = "e2e-1.0.0", + run_mode: str = "container", + status: str = "idle", + last_command_id: Optional[str] = None, +) -> Dict[str, Any]: + ts = datetime.now() - timedelta(seconds=age_seconds) + return { + "hostname": hostname, + "ip_address": "10.0.0.1", + "service_name": service_name, + "agent_version": agent_version, + "run_mode": run_mode, + "namespace": namespace, + "pod_name": pod_name, + "containers": containers or [], + "last_command_id": last_command_id, + "status": status, + "timestamp": ts.isoformat(), + "perf_supported_events": perf_supported_events, + } + + +# --------------------------------------------------------------------------- # +# API calls +# --------------------------------------------------------------------------- # + +def send_heartbeat(client: Client, **kwargs: Any) -> Dict[str, Any]: + resp = client.post(HEARTBEAT, heartbeat_payload(**kwargs)) + resp.raise_for_status() + return resp.json() + + +def get_workload_status( + client: Client, + scope: str = "host", + service_name: Optional[str] = None, + page: Optional[int] = None, + page_size: Optional[int] = None, + sort_by: Optional[str] = None, + sort_order: Optional[str] = None, +) -> Dict[str, Any]: + params: Dict[str, Any] = {"scope": scope} + if service_name: + params["service_name"] = service_name + params["exact_match"] = "true" + if page is not None: + params["page"] = page + if page_size is not None: + params["page_size"] = page_size + if sort_by is not None: + params["sort_by"] = sort_by + if sort_order is not None: + params["sort_order"] = sort_order + resp = client.get(WORKLOAD_STATUS, params=params) + resp.raise_for_status() + return resp.json() + + +def rows_for(status: Dict[str, Any], service_name: str) -> List[Dict[str, Any]]: + """Rows belonging to our service (robust even if server-side filter is loose).""" + return [r for r in status.get("rows", []) if r.get("serviceName") == service_name] + + +def _default_entities(service_name, target_scope, target_hosts, target_entities): + """The request validator requires at least one target_hosts entry or + target_entities selector. For non-host scopes with nothing else provided, + mirror the UI by attaching a service selector (resolution uses the request's + service_name; the entity just satisfies the contract).""" + if target_entities: + return target_entities + if target_scope != "host" and not target_hosts: + return [{"service_name": service_name}] + return target_entities + + +def start_request( + service_name: str, + *, + target_scope: str = "host", + target_hosts: Optional[Dict[str, List[int]]] = None, + target_entities: Optional[List[Dict[str, Any]]] = None, + continuous: bool = False, + duration: int = 30, + frequency: int = 11, + additional_args: Optional[Dict[str, Any]] = None, +) -> Dict[str, Any]: + target_entities = _default_entities(service_name, target_scope, target_hosts, target_entities) + return { + "service_name": service_name, + "request_type": "start", + "continuous": continuous, + "duration": duration, + "frequency": frequency, + "profiling_mode": "cpu", + "target_scope": target_scope, + "target_hosts": target_hosts, + "target_entities": target_entities, + "additional_args": additional_args or {}, + } + + +def stop_request( + service_name: str, + *, + target_scope: str = "host", + stop_level: str = "host", + target_hosts: Optional[Dict[str, List[int]]] = None, + target_entities: Optional[List[Dict[str, Any]]] = None, +) -> Dict[str, Any]: + target_entities = _default_entities(service_name, target_scope, target_hosts, target_entities) + return { + "service_name": service_name, + "request_type": "stop", + "continuous": False, + "duration": 30, + "frequency": 11, + "profiling_mode": "cpu", + "target_scope": target_scope, + "stop_level": stop_level, + "target_hosts": target_hosts, + "target_entities": target_entities, + "additional_args": {}, + } + + +def submit(client: Client, payload: Dict[str, Any]) -> requests.Response: + return client.post(PROFILE_REQUEST, payload) + + +def submit_bulk(client: Client, requests_list: List[Dict[str, Any]]) -> requests.Response: + return client.post(PROFILE_REQUEST_BULK, {"requests": requests_list}) diff --git a/src/tests/e2e/test_workload_acceptance.py b/src/tests/e2e/test_workload_acceptance.py new file mode 100644 index 00000000..94a82bfe --- /dev/null +++ b/src/tests/e2e/test_workload_acceptance.py @@ -0,0 +1,381 @@ +"""End-to-end acceptance tests for workload-level profiling (AT-S1 .. AT-S15). + +These drive the *live* studio stack over HTTP and codify the acceptance criteria +from `heartbeat_doc/WORKLOAD_LEVEL_PROFILING_SPEC.md`. Each test uses a unique +service/host name so runs are isolated from each other and from any real agent. + +Bring the stack up first (see deploy/Makefile.e2e), then: + + make -f Makefile.e2e e2e-test # in-network + # or from host: + E2E_BASE_URL=https://localhost:4433 E2E_BASIC_AUTH_USER=admin \ + E2E_BASIC_AUTH_PASSWORD=admin pytest -q src/tests/e2e +""" +import json + +import harness as h +import pytest + + +# --------------------------------------------------------------------------- # +# Inventory & status views (AT-S1 .. AT-S4) +# --------------------------------------------------------------------------- # + +def test_at_s1_heartbeat_populates_inventory(client): + """AT-S1: a fresh heartbeat shows up as a host row in workload_status.""" + service = h.unique("s1") + host = h.unique("host") + h.send_heartbeat( + client, + hostname=host, + service_name=service, + namespace="obs", + pod_name="pod-a", + containers=[h.container("app", [h.process(1234)], namespace="obs", pod_name="pod-a")], + ) + + status = h.get_workload_status(client, scope="host", service_name=service) + hosts = {r["hostname"] for r in h.rows_for(status, service)} + assert host in hosts + + +def test_at_s2_tab_counts_per_scope(client): + """AT-S2: tabCounts reflect distinct groups per scope; activeHosts == hosts.""" + service = h.unique("s2") + host = h.unique("host") + h.send_heartbeat( + client, + hostname=host, + service_name=service, + namespace="team-a", + pod_name="pod-1", + containers=[ + h.container( + "checkout", + [h.process(11, "java"), h.process(22, "python")], + namespace="team-a", + pod_name="pod-1", + workload_name="checkout", + workload_kind="deployment", + ) + ], + ) + + status = h.get_workload_status(client, scope="host", service_name=service) + tabs = status["tabCounts"] + assert tabs["service"] == 1 + assert tabs["host"] == 1 + assert tabs["namespace"] == 1 + assert tabs["pod"] == 1 + assert tabs["container"] == 1 + assert tabs["process"] == 2 + assert status["activeHosts"] == 1 + + +def test_at_s3_service_tab_grouped_by_service(client): + """AT-S3: scope=service returns exactly one aggregated row per service.""" + service = h.unique("s3") + for host in (h.unique("host"), h.unique("host")): + h.send_heartbeat( + client, + hostname=host, + service_name=service, + containers=[h.container("app", [h.process(1)])], + ) + + status = h.get_workload_status(client, scope="service", service_name=service) + rows = h.rows_for(status, service) + assert len(rows) == 1 + assert rows[0]["hostCount"] == 2 + + +def test_at_s4_freshness_filtering(client): + """AT-S4: a host whose latest heartbeat is stale is excluded from all tabs.""" + service = h.unique("s4") + host = h.unique("host") + # Backdate the heartbeat well beyond the 2-minute freshness window. + h.send_heartbeat( + client, + hostname=host, + service_name=service, + age_seconds=h.FRESHNESS_SECONDS + 180, + containers=[h.container("app", [h.process(1)])], + ) + + status = h.get_workload_status(client, scope="host", service_name=service) + hosts = {r["hostname"] for r in h.rows_for(status, service)} + assert host not in hosts + + +# --------------------------------------------------------------------------- # +# Resolution & command creation (AT-S5 .. AT-S9) +# --------------------------------------------------------------------------- # + +def test_at_s5_host_level_start(client): + """AT-S5: a host-scope start yields a start command returned to that host.""" + service = h.unique("s5") + host = h.unique("host") + h.send_heartbeat(client, hostname=host, service_name=service) + + resp = h.submit(client, h.start_request(service, target_scope="host", target_hosts={host: []})) + assert resp.status_code == 200, resp.text + + beat = h.send_heartbeat(client, hostname=host, service_name=service) + assert beat["profiling_command"] is not None + assert beat["profiling_command"]["command_type"] == "start" + + +def test_at_s6_service_level_start_fans_out(client): + """AT-S6: a service-scope start creates a command for every current host.""" + service = h.unique("s6") + h1, h2 = h.unique("host"), h.unique("host") + h.send_heartbeat(client, hostname=h1, service_name=service) + h.send_heartbeat(client, hostname=h2, service_name=service) + + resp = h.submit(client, h.start_request(service, target_scope="service")) + assert resp.status_code == 200, resp.text + assert len(resp.json()["command_ids"]) == 2 + + for host in (h1, h2): + beat = h.send_heartbeat(client, hostname=host, service_name=service) + assert beat["profiling_command"] is not None, f"no command for {host}" + assert beat["profiling_command"]["command_type"] == "start" + + +def test_at_s7_process_scope_resolves_to_pids(client): + """AT-S7: a process selection resolves to hostname->[pid] and the command carries the PID.""" + service = h.unique("s7") + host = h.unique("host") + pid = 4242 + h.send_heartbeat( + client, + hostname=host, + service_name=service, + containers=[h.container("app", [h.process(pid, "java")])], + ) + + resp = h.submit( + client, + h.start_request( + service, + target_scope="process", + target_entities=[{"service_name": service, "hostname": host, "pid": pid}], + ), + ) + assert resp.status_code == 200, resp.text + + beat = h.send_heartbeat(client, hostname=host, service_name=service) + assert beat["profiling_command"] is not None + assert str(pid) in json.dumps(beat["profiling_command"]["combined_config"]) + + +def test_at_s8_empty_resolution_is_rejected(client): + """AT-S8: a selection resolving to zero targets returns 422 and creates no command.""" + service = h.unique("s8") + host = h.unique("host") + h.send_heartbeat( + client, + hostname=host, + service_name=service, + containers=[h.container("app", [h.process(1234)])], + ) + + resp = h.submit( + client, + h.start_request( + service, + target_scope="process", + target_entities=[{"service_name": service, "hostname": host, "pid": 999999}], + ), + ) + assert resp.status_code == 422, resp.text + + beat = h.send_heartbeat(client, hostname=host, service_name=service) + assert beat["profiling_command"] is None + + +def test_at_s9_pmu_validation_rejects_unsupported_events(client): + """AT-S9: a start requesting perf events a host doesn't support is rejected (bulk).""" + service = h.unique("s9") + host = h.unique("host") + h.send_heartbeat( + client, + hostname=host, + service_name=service, + perf_supported_events=["cycles"], + ) + + req = h.start_request( + service, + target_scope="host", + target_hosts={host: []}, + additional_args={"profiler_configs": {"perf": {"mode": "smart", "events": ["instructions"]}}}, + ) + resp = h.submit_bulk(client, [req]) + assert resp.status_code == 422, resp.text + assert "perf" in resp.text.lower() or "event" in resp.text.lower() + + +# --------------------------------------------------------------------------- # +# Continuous service subscriptions & auto-enrollment (AT-S10 .. AT-S13) +# --------------------------------------------------------------------------- # + +def _subscribe_service(client, service, seed_host): + """Create an active service-wide continuous subscription.""" + h.send_heartbeat(client, hostname=seed_host, service_name=service) + resp = h.submit( + client, h.start_request(service, target_scope="service", continuous=True) + ) + assert resp.status_code == 200, resp.text + + +def test_at_s10_new_host_auto_enrolls(client): + """AT-S10: a new host under an active subscription gets a start on its first heartbeat.""" + service = h.unique("s10") + seed, newcomer = h.unique("host"), h.unique("host") + _subscribe_service(client, service, seed) + + beat = h.send_heartbeat(client, hostname=newcomer, service_name=service) + assert beat["profiling_command"] is not None + assert beat["profiling_command"]["command_type"] == "start" + + +def test_at_s11_no_subscription_no_enrollment(client): + """AT-S11: with no active subscription, a new host gets no command.""" + service = h.unique("s11") + host = h.unique("host") + beat = h.send_heartbeat(client, hostname=host, service_name=service) + assert beat["profiling_command"] is None + + +def test_at_s12_stop_deactivates_subscription(client): + """AT-S12: after a service-wide stop, a new host is not enrolled.""" + service = h.unique("s12") + seed, newcomer = h.unique("host"), h.unique("host") + _subscribe_service(client, service, seed) + + resp = h.submit( + client, h.stop_request(service, target_scope="service", stop_level="host") + ) + assert resp.status_code == 200, resp.text + + beat = h.send_heartbeat(client, hostname=newcomer, service_name=service) + assert beat["profiling_command"] is None or beat["profiling_command"]["command_type"] != "start" + + +def test_at_s13_existing_command_preserved(client): + """AT-S13: auto-enroll does not overwrite a host's existing command.""" + service = h.unique("s13") + seed = h.unique("host") + _subscribe_service(client, service, seed) + + first = h.send_heartbeat(client, hostname=seed, service_name=service) + assert first["profiling_command"] is not None + command_id = first["command_id"] + + # Subsequent heartbeats under the active subscription must not replace it. + second = h.send_heartbeat(client, hostname=seed, service_name=service) + assert second["command_id"] == command_id + + +# --------------------------------------------------------------------------- # +# Compatibility & failure handling (AT-S14 .. AT-S15) +# --------------------------------------------------------------------------- # + +def test_at_s14_legacy_heartbeat(client): + """AT-S14: a heartbeat with no workload fields still yields host status + host commands.""" + service = h.unique("s14") + host = h.unique("host") + # No namespace/pod/containers at all. + h.send_heartbeat(client, hostname=host, service_name=service) + + status = h.get_workload_status(client, scope="host", service_name=service) + rows = h.rows_for(status, service) + assert host in {r["hostname"] for r in rows} + assert status["tabCounts"]["namespace"] == 0 + assert status["tabCounts"]["pod"] == 0 + assert status["tabCounts"]["container"] == 0 + + # Host-level command path still works. + resp = h.submit(client, h.start_request(service, target_scope="host", target_hosts={host: []})) + assert resp.status_code == 200, resp.text + beat = h.send_heartbeat(client, hostname=host, service_name=service) + assert beat["profiling_command"] is not None + + +def test_at_s15_partial_inventory(client): + """AT-S15: heartbeats missing some fields (no pod_name) don't break other scopes.""" + service = h.unique("s15") + host = h.unique("host") + # Container has a namespace + processes but no pod_name. + h.send_heartbeat( + client, + hostname=host, + service_name=service, + namespace="team-x", + containers=[h.container("app", [h.process(7)], namespace="team-x")], + ) + + status = h.get_workload_status(client, scope="host", service_name=service) + tabs = status["tabCounts"] + assert host in {r["hostname"] for r in h.rows_for(status, service)} + assert tabs["namespace"] == 1 + assert tabs["container"] == 1 + assert tabs["process"] == 1 + # No invented pod relationships. + assert tabs["pod"] == 0 + + +# --------------------------------------------------------------------------- # +# Pagination & server-side sorting (AT-S16 .. AT-S17) +# --------------------------------------------------------------------------- # + +def test_at_s16_host_scope_pagination(client): + """AT-S16: host scope is paginated; pages are disjoint and total_count is the full match.""" + service = h.unique("s16") + hosts = sorted(h.unique("host") for _ in range(5)) + for host in hosts: + h.send_heartbeat( + client, + hostname=host, + service_name=service, + containers=[h.container("app", [h.process(1234)])], + ) + + first = h.get_workload_status( + client, scope="host", service_name=service, page=0, page_size=2, sort_by="hostname", sort_order="asc" + ) + assert first["totalCount"] == 5 + assert len(h.rows_for(first, service)) == 2 + + seen = [] + for page in range(3): + status = h.get_workload_status( + client, scope="host", service_name=service, page=page, page_size=2, sort_by="hostname", sort_order="asc" + ) + seen.extend(r["hostname"] for r in h.rows_for(status, service)) + + # All five hosts appear exactly once across the three pages (no overlap, no drops). + assert sorted(seen) == hosts + + +def test_at_s17_host_scope_sorting(client): + """AT-S17: sort_order flips the first row (server-side sort over the full set).""" + service = h.unique("s17") + hosts = sorted(h.unique("host") for _ in range(3)) + for host in hosts: + h.send_heartbeat( + client, + hostname=host, + service_name=service, + containers=[h.container("app", [h.process(1234)])], + ) + + asc = h.get_workload_status( + client, scope="host", service_name=service, page=0, page_size=1, sort_by="hostname", sort_order="asc" + ) + desc = h.get_workload_status( + client, scope="host", service_name=service, page=0, page_size=1, sort_by="hostname", sort_order="desc" + ) + assert h.rows_for(asc, service)[0]["hostname"] == hosts[0] + assert h.rows_for(desc, service)[0]["hostname"] == hosts[-1] diff --git a/src/tests/integration/backend_db/test_heartbeat_inventory_sync.py b/src/tests/integration/backend_db/test_heartbeat_inventory_sync.py new file mode 100644 index 00000000..027250fd --- /dev/null +++ b/src/tests/integration/backend_db/test_heartbeat_inventory_sync.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +""" +Integration tests for the normalized heartbeat workload inventory sync. + +These validate the diff/upsert behavior of ``DBManager._sync_host_inventory`` end +to end: a heartbeat is POSTed to the live backend and the resulting rows in +``HeartbeatContainers`` / ``HeartbeatProcesses`` are inspected directly in +PostgreSQL. + +The central regression this guards is write amplification at fleet scale: an +unchanged inventory must NOT rewrite rows on every heartbeat. We prove that by +asserting both the primary key ``id`` and the physical row version ``ctid`` are +unchanged across identical beats (a delete-then-insert or an unconditional +``DO UPDATE`` would change both), while a real change to a single process rewrites +only that row and leaves its siblings physically untouched. + +Run just this file (needs the live stack + Postgres, see deploy/Makefile.e2e): + + cd src && python -m pytest tests/integration/backend_db/test_heartbeat_inventory_sync.py -v +""" + +import uuid +from typing import Any, Dict, List + +import psycopg2 +import psycopg2.extras +import pytest +import requests + + +@pytest.fixture(scope="session") +def postgres_connection(pytestconfig): + """Create a PostgreSQL connection for direct row inspection.""" + conn = psycopg2.connect( + host=pytestconfig.getoption("--postgres-host", default="localhost"), + port=pytestconfig.getoption("--postgres-port", default=5432), + user=pytestconfig.getoption("--postgres-user", default="performance_studio"), + password=pytestconfig.getoption("--postgres-password", default="performance_studio_password"), + database=pytestconfig.getoption("--postgres-db", default="performance_studio_db"), + ) + conn.autocommit = True + yield conn + conn.close() + + +@pytest.fixture(scope="session") +def heartbeat_url(backend_base_url) -> str: + return f"{backend_base_url}/api/metrics/heartbeat" + + +def _container(container_id: str, name: str, processes: List[Dict[str, Any]], **overrides: Any) -> Dict[str, Any]: + payload = { + "container_id": container_id, + "container_name": name, + "runtime": "containerd", + "namespace": "obs", + "pod_name": "pod-a", + "workload_name": "checkout", + "workload_kind": "k8s", + "processes": processes, + } + payload.update(overrides) + return payload + + +def _send(heartbeat_url: str, credentials: Dict[str, str], hostname: str, service: str, + containers: List[Dict[str, Any]]) -> None: + body = { + "ip_address": "127.0.0.1", + "hostname": hostname, + "service_name": service, + "status": "active", + "containers": containers, + } + response = requests.post(heartbeat_url, headers=credentials, json=body, timeout=10, verify=False) + assert response.status_code == 200, f"Heartbeat failed: {response.status_code}: {response.text}" + + +def _containers(conn, hostname: str, service: str) -> List[Dict[str, Any]]: + with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cursor: + cursor.execute( + """ + SELECT hc.id, hc.ctid::text AS ctid, hc.container_id, hc.container_name + FROM HeartbeatContainers hc + JOIN HostHeartbeats h ON h.id = hc.host_id + WHERE h.hostname = %s AND h.service_name = %s + ORDER BY hc.container_id + """, + (hostname, service), + ) + return [dict(row) for row in cursor.fetchall()] + + +def _processes(conn, hostname: str, service: str) -> List[Dict[str, Any]]: + with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cursor: + cursor.execute( + """ + SELECT hp.id, hp.ctid::text AS ctid, hp.pid, hp.process_name, hc.container_id + FROM HeartbeatProcesses hp + JOIN HeartbeatContainers hc ON hc.id = hp.container_row_id + JOIN HostHeartbeats h ON h.id = hc.host_id + WHERE h.hostname = %s AND h.service_name = %s + ORDER BY hc.container_id, hp.pid + """, + (hostname, service), + ) + return [dict(row) for row in cursor.fetchall()] + + +def _cleanup(conn, hostname: str) -> None: + with conn.cursor() as cursor: + # HeartbeatContainers/Processes cascade from HostHeartbeats. + cursor.execute("DELETE FROM HostHeartbeats WHERE hostname = %s", (hostname,)) + + +@pytest.fixture +def host_service(postgres_connection): + hostname = f"inv-host-{uuid.uuid4().hex[:8]}" + service = f"inv-svc-{uuid.uuid4().hex[:8]}" + _cleanup(postgres_connection, hostname) + yield hostname, service + _cleanup(postgres_connection, hostname) + + +def test_non_containerized_host_stores_no_rows(heartbeat_url, credentials, postgres_connection, host_service): + """A host with no containerized workload (empty list) stores zero inventory rows.""" + hostname, service = host_service + _send(heartbeat_url, credentials, hostname, service, containers=[]) + + assert _containers(postgres_connection, hostname, service) == [] + assert _processes(postgres_connection, hostname, service) == [] + + +def test_stable_inventory_is_not_rewritten(heartbeat_url, credentials, postgres_connection, host_service): + """Repeated identical heartbeats must not rewrite rows (stable id AND ctid).""" + hostname, service = host_service + containers = [ + _container("cid-a", "app", [{"pid": 11, "process_name": "java"}, {"pid": 22, "process_name": "python"}]), + _container("cid-b", "sidecar", [{"pid": 33, "process_name": "envoy"}]), + ] + + _send(heartbeat_url, credentials, hostname, service, containers) + containers_first = _containers(postgres_connection, hostname, service) + processes_first = _processes(postgres_connection, hostname, service) + assert [c["container_id"] for c in containers_first] == ["cid-a", "cid-b"] + assert [(p["container_id"], p["pid"]) for p in processes_first] == [ + ("cid-a", 11), ("cid-a", 22), ("cid-b", 33), + ] + + # Second identical beat: nothing should be physically rewritten. + _send(heartbeat_url, credentials, hostname, service, containers) + containers_second = _containers(postgres_connection, hostname, service) + processes_second = _processes(postgres_connection, hostname, service) + + assert {(c["id"], c["ctid"]) for c in containers_second} == {(c["id"], c["ctid"]) for c in containers_first} + assert {(p["id"], p["ctid"]) for p in processes_second} == {(p["id"], p["ctid"]) for p in processes_first} + + +def test_process_change_is_isolated(heartbeat_url, credentials, postgres_connection, host_service): + """Changing one process rewrites only that row; unchanged siblings keep their ctid.""" + hostname, service = host_service + containers = [ + _container("cid-a", "app", [{"pid": 11, "process_name": "java"}, {"pid": 22, "process_name": "python"}]), + ] + _send(heartbeat_url, credentials, hostname, service, containers) + before = {(p["pid"]): p for p in _processes(postgres_connection, hostname, service)} + + # Rename only pid 22. + containers[0]["processes"] = [ + {"pid": 11, "process_name": "java"}, + {"pid": 22, "process_name": "python3"}, + ] + _send(heartbeat_url, credentials, hostname, service, containers) + after = {(p["pid"]): p for p in _processes(postgres_connection, hostname, service)} + + # Unchanged sibling: same physical row. + assert after[11]["ctid"] == before[11]["ctid"] + # Changed row: same logical row (id) but rewritten (new ctid) and new name. + assert after[22]["id"] == before[22]["id"] + assert after[22]["ctid"] != before[22]["ctid"] + assert after[22]["process_name"] == "python3" + + +def test_removed_container_is_pruned(heartbeat_url, credentials, postgres_connection, host_service): + """A container that stops being reported is deleted along with its processes.""" + hostname, service = host_service + _send( + heartbeat_url, credentials, hostname, service, + containers=[ + _container("cid-a", "app", [{"pid": 11, "process_name": "java"}]), + _container("cid-b", "sidecar", [{"pid": 33, "process_name": "envoy"}]), + ], + ) + assert {c["container_id"] for c in _containers(postgres_connection, hostname, service)} == {"cid-a", "cid-b"} + + # Drop cid-b. + _send( + heartbeat_url, credentials, hostname, service, + containers=[_container("cid-a", "app", [{"pid": 11, "process_name": "java"}])], + ) + assert {c["container_id"] for c in _containers(postgres_connection, hostname, service)} == {"cid-a"} + assert {p["pid"] for p in _processes(postgres_connection, hostname, service)} == {11} + + +def test_null_container_id_is_skipped(heartbeat_url, credentials, postgres_connection, host_service): + """Entries without a container_id are not stored (containerized-only model).""" + hostname, service = host_service + _send( + heartbeat_url, credentials, hostname, service, + containers=[ + _container("cid-a", "app", [{"pid": 11, "process_name": "java"}]), + _container(None, "bare", [{"pid": 99, "process_name": "systemd"}]), + ], + ) + + stored = _containers(postgres_connection, hostname, service) + assert [c["container_id"] for c in stored] == ["cid-a"] + assert {p["pid"] for p in _processes(postgres_connection, hostname, service)} == {11} diff --git a/src/tests/playwright/.gitignore b/src/tests/playwright/.gitignore new file mode 100644 index 00000000..0fb21413 --- /dev/null +++ b/src/tests/playwright/.gitignore @@ -0,0 +1,6 @@ +node_modules/ +package-lock.json +test-results/ +playwright-report/ +blob-report/ +.cache/ diff --git a/src/tests/playwright/Dockerfile b/src/tests/playwright/Dockerfile new file mode 100644 index 00000000..c184c305 --- /dev/null +++ b/src/tests/playwright/Dockerfile @@ -0,0 +1,13 @@ +# Playwright UI test runner. Uses the official image (browsers + system deps +# preinstalled) so it works regardless of host libraries, and runs on the +# compose network to reach the console via the internal nginx service. +# +# Image tag MUST match the @playwright/test version in package.json. +FROM mcr.microsoft.com/playwright:v1.61.1-noble + +WORKDIR /work +COPY package.json package-lock.json* ./ +RUN npm install +COPY . . + +CMD ["npx", "playwright", "test"] diff --git a/src/tests/playwright/README.md b/src/tests/playwright/README.md new file mode 100644 index 00000000..a6c4994e --- /dev/null +++ b/src/tests/playwright/README.md @@ -0,0 +1,18 @@ +# Playwright UI acceptance tests + +Drives the profiling console through the internal nginx (basic auth + +self-signed TLS) and asserts the start/stop confirmation flow (dry-run +validation + submit). The STOP case is the UI regression test for the +host-level-stop bug. + +Runs in the official Playwright browser image on the compose network — don't run +it against the host (host browser libs may be missing). Use the e2e harness: + +```bash +cd ../../../deploy +make -f Makefile.e2e e2e-up # or e2e-up-src (source-built agent) +make -f Makefile.e2e e2e-ui-test # builds this runner and executes it in-network +``` + +Full harness docs (topology, LocalStack S3/SQS, portability): +[`deploy/E2E_HARNESS.md`](../../../deploy/E2E_HARNESS.md). diff --git a/src/tests/playwright/package.json b/src/tests/playwright/package.json new file mode 100644 index 00000000..f23708ef --- /dev/null +++ b/src/tests/playwright/package.json @@ -0,0 +1,13 @@ +{ + "name": "gprofiler-studio-e2e-ui", + "version": "0.0.0", + "private": true, + "description": "Playwright UI acceptance tests for the Performance Studio profiling console", + "scripts": { + "test": "playwright test", + "install-browser": "playwright install chromium" + }, + "devDependencies": { + "@playwright/test": "1.61.1" + } +} diff --git a/src/tests/playwright/playwright.config.ts b/src/tests/playwright/playwright.config.ts new file mode 100644 index 00000000..63a62e57 --- /dev/null +++ b/src/tests/playwright/playwright.config.ts @@ -0,0 +1,26 @@ +import { defineConfig, devices } from '@playwright/test'; + +// Drives the running console behind nginx (self-signed TLS + basic auth). +// From host: E2E_UI_URL=https://localhost:4433 (default) +// From network: E2E_UI_URL=https://nginx-load-balancer +export default defineConfig({ + testDir: './tests', + timeout: 60_000, + expect: { timeout: 15_000 }, + fullyParallel: false, + retries: process.env.CI ? 1 : 0, + reporter: 'list', + use: { + baseURL: process.env.E2E_UI_URL || 'https://localhost:4433', + ignoreHTTPSErrors: true, + httpCredentials: { + username: process.env.E2E_BASIC_AUTH_USER || 'admin', + password: process.env.E2E_BASIC_AUTH_PASSWORD || 'admin', + }, + trace: 'on-first-retry', + screenshot: 'only-on-failure', + }, + projects: [ + { name: 'chromium', use: { ...devices['Desktop Chrome'] } }, + ], +}); diff --git a/src/tests/playwright/tests/profiling.spec.ts b/src/tests/playwright/tests/profiling.spec.ts new file mode 100644 index 00000000..99af01fc --- /dev/null +++ b/src/tests/playwright/tests/profiling.spec.ts @@ -0,0 +1,62 @@ +import { test, expect, Page } from '@playwright/test'; + +// The service the harness agent reports as (see docker-compose.e2e.yml). +const SERVICE = process.env.E2E_UI_SERVICE || 'e2e-sample-app'; +const BULK_URL = '/api/metrics/profile_request/bulk'; + +async function openConsoleAndSelectService(page: Page) { + await page.goto('/profiling'); + + // The DataGrid renders one row per workload; wait for our service to appear. + const row = page.getByRole('row', { name: new RegExp(SERVICE) }).first(); + await expect(row).toBeVisible({ timeout: 30_000 }); + + // Select the row via its checkbox (MUI DataGrid checkboxSelection). + await row.getByRole('checkbox').check(); + return row; +} + +async function confirmAndAwaitSubmit(page: Page, action: 'Start' | 'Stop') { + // Open the confirmation dialog from the top-panel action button. + await page.getByRole('button', { name: new RegExp(`${action} Profiling \\(\\d+\\)`) }).click(); + + const dialog = page.getByRole('dialog'); + await expect(dialog).toBeVisible(); + + // The confirm button is gated on a server-side dry-run validation: + // disabled={dryRunValidation.isValidating || !dryRunValidation.isValid} + // So it only enables when the request validates. For a host/service-level + // STOP this is exactly the regression that previously failed with + // "No PIDs should be provided when request_type is 'stop' ...". + const confirm = dialog.getByRole('button', { name: new RegExp(`^${action} Profiling$`) }); + await expect(confirm).toBeEnabled({ timeout: 15_000 }); + + // Submit and assert the real (non dry-run) bulk request succeeds. + const [resp] = await Promise.all([ + page.waitForResponse((r) => { + if (!r.url().includes(BULK_URL) || r.request().method() !== 'POST') return false; + try { + return JSON.parse(r.request().postData() || '{}').dry_run === false; + } catch { + return false; + } + }), + confirm.click(), + ]); + expect(resp.status()).toBe(200); + return resp; +} + +test.describe('profiling console', () => { + test('service/host-level STOP validates and submits (regression AT for the stop bug)', async ({ page }) => { + await openConsoleAndSelectService(page); + await confirmAndAwaitSubmit(page, 'Stop'); + // No "No PIDs should be provided" validation error should ever surface. + await expect(page.getByText(/No PIDs should be provided/i)).toHaveCount(0); + }); + + test('service/host-level START validates and submits', async ({ page }) => { + await openConsoleAndSelectService(page); + await confirmAndAwaitSubmit(page, 'Start'); + }); +}); diff --git a/src/tests/spec/backend/test_pmu_and_capacity_spec.py b/src/tests/spec/backend/test_pmu_and_capacity_spec.py new file mode 100644 index 00000000..67166041 --- /dev/null +++ b/src/tests/spec/backend/test_pmu_and_capacity_spec.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +""" +Fast acceptance tests for pre-dispatch profiling guardrails: + +* PMU event validation -> WORKLOAD_LEVEL_PROFILING_SPEC.md AT-S9 +* Profiling capacity -> capacity guardrails referenced by AT-S6 fan-out + +These exercise the pure decision logic in +``backend.utils.dynamic_profiling_utils`` with an in-memory fake DBManager, so +they run in-process with no database or HTTP server (fast). + +Run: + cd src && python -m pytest tests/spec/backend/test_pmu_and_capacity_spec.py -v +""" + +import pytest + +pytest.importorskip("pydantic", reason="pydantic is required for these spec tests") +pytest.importorskip("humps", reason="pyhumps is required for these spec tests") + +try: + from backend.models.metrics_models import BulkProfilingRequest, ProfilingRequest + from backend.utils.dynamic_profiling_utils import validate_pmu_events, validate_profiling_capacity +except Exception as exc: # pragma: no cover - environment guard + pytest.skip(f"backend modules not importable: {exc}", allow_module_level=True) + + +class FakeDBManager: + """Minimal stand-in for DBManager covering only the methods the guardrails call.""" + + def __init__( + self, + *, + active_hosts=0, + profiling_total=0, + profiling_inside=0, + profiling_outside=0, + unsupported_events=None, + ): + self._active_hosts = active_hosts + self._profiling_total = profiling_total + self._profiling_inside = profiling_inside + self._profiling_outside = profiling_outside + self._unsupported_events = set(unsupported_events or []) + + # -- capacity -- + def get_active_hosts_count(self, service_name=None): + return self._active_hosts + + def get_actively_profiling_hosts_count( + self, service_name=None, host_inclusion_list=None, host_exclusion_list=None + ): + if host_inclusion_list is not None: + return self._profiling_inside + if host_exclusion_list is not None: + return self._profiling_outside + return self._profiling_total + + # -- PMU -- + def validate_perf_events_support(self, service_name, requested_events, target_hostnames=None): + bad = sorted(self._unsupported_events.intersection(requested_events)) + if bad: + return {"valid": False, "error_message": f"Unsupported perf events: {', '.join(bad)}"} + return {"valid": True, "error_message": None} + + +def _start_request(**overrides) -> ProfilingRequest: + data = { + "service_name": "svc-a", + "request_type": "start", + "target_scope": "host", + "target_hosts": {"host-a": None}, + } + data.update(overrides) + return ProfilingRequest(**data) + + +def _bulk(*requests) -> BulkProfilingRequest: + return BulkProfilingRequest(requests=list(requests)) + + +def _perf_args(events, mode="enabled_restricted"): + return {"profiler_configs": {"perf": {"mode": mode, "events": events}}} + + +# --------------------------------------------------------------------------- +# AT-S9 — PMU validation +# --------------------------------------------------------------------------- + + +class TestPmuValidationSpec: + def test_supported_events_pass(self): + db = FakeDBManager(unsupported_events=[]) + req = _start_request(additional_args=_perf_args(["cycles", "instructions"])) + ok, err = validate_pmu_events(_bulk(req), db) + assert ok is True + assert err is None + + def test_unsupported_event_is_rejected_with_clear_error(self): + db = FakeDBManager(unsupported_events=["cache-misses"]) + req = _start_request(additional_args=_perf_args(["cycles", "cache-misses"])) + ok, err = validate_pmu_events(_bulk(req), db) + assert ok is False + assert "svc-a" in err + assert "cache-misses" in err + + def test_perf_disabled_skips_pmu_validation(self): + db = FakeDBManager(unsupported_events=["cache-misses"]) + req = _start_request(additional_args=_perf_args(["cache-misses"], mode="disabled")) + ok, err = validate_pmu_events(_bulk(req), db) + assert ok is True and err is None + + def test_stop_requests_are_not_pmu_validated(self): + db = FakeDBManager(unsupported_events=["cache-misses"]) + req = ProfilingRequest( + service_name="svc-a", + request_type="stop", + target_scope="host", + stop_level="host", + target_hosts={"host-a": None}, + additional_args=_perf_args(["cache-misses"]), + ) + ok, err = validate_pmu_events(_bulk(req), db) + assert ok is True and err is None + + def test_workload_scope_without_target_hosts_does_not_crash(self): + # AT-S9 for non-host scopes: target_hosts is None; events are validated + # against the whole service instead of blowing up on None.keys(). + db = FakeDBManager(unsupported_events=[]) + req = _start_request( + target_scope="service", + target_hosts=None, + target_entities=[{"service_name": "svc-a"}], + additional_args=_perf_args(["cycles"]), + ) + ok, err = validate_pmu_events(_bulk(req), db) + assert ok is True and err is None + + +# --------------------------------------------------------------------------- +# Capacity guardrails +# --------------------------------------------------------------------------- + + +class TestCapacitySpec: + def test_within_capacity_passes(self): + # 100 active hosts, 10% cap => 10 allowed; request of 1 with 0 already profiling. + db = FakeDBManager(active_hosts=100, profiling_total=0, profiling_outside=0) + req = _start_request(target_hosts={"host-a": None}) + ok, err, hosts = validate_profiling_capacity(_bulk(req), db) + assert ok is True + assert err is None + assert hosts == ["host-a"] + + def test_exceeding_percentage_capacity_is_rejected(self): + # 100 active hosts, 10% cap => 10 allowed; 10 already profiling outside selection + # + 1 new = 11 > 10 => rejected. + db = FakeDBManager(active_hosts=100, profiling_total=10, profiling_outside=10) + req = _start_request(target_hosts={"host-new": None}) + ok, err, _ = validate_profiling_capacity(_bulk(req), db) + assert ok is False + assert "capacity exceeded" in err.lower() + + def test_request_size_limit_is_rejected(self): + db = FakeDBManager(active_hosts=10_000) + many_hosts = {f"host-{i}": None for i in range(21)} # MAX_PROFILING_REQUEST_HOSTS default = 20 + req = _start_request(target_hosts=many_hosts) + ok, err, _ = validate_profiling_capacity(_bulk(req), db) + assert ok is False + assert "request size exceeded" in err.lower() + + def test_stop_only_bulk_skips_capacity(self): + db = FakeDBManager(active_hosts=0) + stop = ProfilingRequest( + service_name="svc-a", + request_type="stop", + target_scope="host", + stop_level="host", + target_hosts={"host-a": None}, + ) + ok, err, _ = validate_profiling_capacity(_bulk(stop), db) + assert ok is True and err is None diff --git a/src/tests/spec/backend/test_profiling_request_spec.py b/src/tests/spec/backend/test_profiling_request_spec.py new file mode 100644 index 00000000..569ababd --- /dev/null +++ b/src/tests/spec/backend/test_profiling_request_spec.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +""" +Acceptance / spec tests for the ProfilingRequest validation contract. + +Unlike the sibling ``test_metrics_profile_request.py`` (which drives a *live* +backend over HTTP), these tests import the ``ProfilingRequest`` pydantic model +directly and exercise its validation rules in-process. They are fast, need no +running server or database, and serve as the executable specification for the +start/stop targeting rules. + +The central invariant (which the host/service stop regression violated): + + A host-/service-level STOP must not carry any PID — neither in + ``target_hosts`` nor via a ``target_entities`` selector. The frontend + request builder is expected to produce payloads that satisfy this spec + (see ``profilingRequestBuilder.mjs`` / ``profilingRequestBuilder.test.mjs``). + +Run just this file: + + cd src && python -m pytest tests/unit/backend/test_profiling_request_spec.py -v +""" + +import sys +from pathlib import Path + +import pytest + +# Make the backend + gprofiler_dev packages importable without a full install: +# .../src/gprofiler contains the ``backend`` package +# .../src/gprofiler-dev contains the ``gprofiler_dev`` package +_SRC_ROOT = Path(__file__).resolve().parents[3] +for _pkg_root in (_SRC_ROOT / "gprofiler", _SRC_ROOT / "gprofiler-dev"): + if str(_pkg_root) not in sys.path: + sys.path.insert(0, str(_pkg_root)) + +# Skip cleanly if backend deps (pydantic, humps, ...) are not installed here. +pytest.importorskip("pydantic", reason="pydantic is required for the ProfilingRequest spec tests") +pytest.importorskip("humps", reason="pyhumps is required for the ProfilingRequest spec tests") + +try: + from pydantic import ValidationError + from backend.models.metrics_models import ProfilingRequest +except Exception as exc: # pragma: no cover - environment guard + pytest.skip(f"backend.models.metrics_models not importable: {exc}", allow_module_level=True) + + +def _base(**overrides): + """A minimal valid request skeleton; override per-case.""" + data = { + "service_name": "svc-a", + "request_type": "start", + "target_scope": "host", + "target_hosts": {"host-a": None}, + } + data.update(overrides) + return data + + +def _make(**overrides) -> ProfilingRequest: + return ProfilingRequest(**_base(**overrides)) + + +def _entity(**fields): + """Build a target_entities selector dict (snake_case field names).""" + return {"service_name": "svc-a", **fields} + + +# --------------------------------------------------------------------------- +# Host-/service-level STOP — the regression surface +# --------------------------------------------------------------------------- + + +class TestHostLevelStopSpec: + def test_host_stop_without_pids_is_valid(self): + req = _make( + request_type="stop", + target_scope="host", + stop_level="host", + target_hosts={"host-a": None}, + target_entities=[_entity(hostname="host-a")], + ) + assert req.request_type == "stop" + assert req.stop_level == "host" + + def test_service_stop_without_pids_is_valid(self): + req = _make( + request_type="stop", + target_scope="service", + stop_level="host", + target_hosts=None, + target_entities=[_entity(hostname="host-a")], + ) + assert req.stop_level == "host" + + def test_host_stop_with_pids_in_target_hosts_is_rejected(self): + with pytest.raises(ValidationError, match="No PIDs should be provided"): + _make( + request_type="stop", + target_scope="host", + stop_level="host", + target_hosts={"host-a": [4242]}, + ) + + def test_host_stop_with_pid_in_target_entity_is_rejected(self): + # This is exactly what the buggy frontend used to send. + with pytest.raises(ValidationError, match="No PIDs should be provided"): + _make( + request_type="stop", + target_scope="host", + stop_level="host", + target_hosts={"host-a": None}, + target_entities=[_entity(hostname="host-a", pid=4242)], + ) + + def test_host_stop_with_pid_zero_in_entity_is_rejected(self): + # A PID of 0 still counts as "a PID was provided" for a host-level stop. + with pytest.raises(ValidationError, match="No PIDs should be provided"): + _make( + request_type="stop", + target_scope="host", + stop_level="host", + target_hosts={"host-a": None}, + target_entities=[_entity(hostname="host-a", pid=0)], + ) + + +# --------------------------------------------------------------------------- +# Process-level STOP +# --------------------------------------------------------------------------- + + +class TestProcessLevelStopSpec: + def test_process_stop_with_pids_in_target_hosts_is_valid(self): + req = _make( + request_type="stop", + target_scope="process", + stop_level="process", + target_hosts={"host-a": [4242]}, + ) + assert req.stop_level == "process" + + def test_process_stop_with_pid_in_entity_is_valid(self): + req = _make( + request_type="stop", + target_scope="process", + stop_level="process", + target_hosts=None, + target_entities=[_entity(hostname="host-a", pid=4242, process_name="python")], + ) + assert req.stop_level == "process" + + def test_process_stop_without_any_pid_is_rejected(self): + with pytest.raises(ValidationError, match="At least one PID or workload selector"): + _make( + request_type="stop", + target_scope="process", + stop_level="process", + target_hosts={"host-a": None}, + ) + + +# --------------------------------------------------------------------------- +# START requests are unaffected by the PID rules +# --------------------------------------------------------------------------- + + +class TestStartSpec: + def test_start_with_target_hosts_is_valid(self): + req = _make(request_type="start", target_scope="host", target_hosts={"host-a": None}) + assert req.request_type == "start" + + def test_start_with_entities_carrying_pid_is_valid(self): + # PID rules only apply to stop requests, so a start is fine either way. + req = _make( + request_type="start", + target_scope="process", + target_hosts=None, + target_entities=[_entity(hostname="host-a", pid=4242)], + ) + assert req.request_type == "start" + + +# --------------------------------------------------------------------------- +# Shared targeting requirements +# --------------------------------------------------------------------------- + + +class TestTargetingRequirementsSpec: + def test_request_without_any_target_is_rejected(self): + with pytest.raises(ValidationError, match="At least one target_hosts entry or target_entities"): + ProfilingRequest( + service_name="svc-a", + request_type="start", + target_scope="host", + target_hosts=None, + target_entities=None, + ) + + @pytest.mark.parametrize("bad_scope", ["cluster", "", "HOST"]) + def test_invalid_target_scope_is_rejected(self, bad_scope): + with pytest.raises(ValidationError): + _make(target_scope=bad_scope) + + @pytest.mark.parametrize("bad_stop_level", ["node", "pod", ""]) + def test_invalid_stop_level_is_rejected(self, bad_stop_level): + with pytest.raises(ValidationError): + _make(request_type="stop", stop_level=bad_stop_level, target_hosts={"host-a": None}) diff --git a/src/tests/spec/conftest.py b/src/tests/spec/conftest.py new file mode 100644 index 00000000..6b4481d5 --- /dev/null +++ b/src/tests/spec/conftest.py @@ -0,0 +1,34 @@ +# +# Copyright (C) 2023 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +""" +Shared setup for the *fast* backend unit/acceptance tests. + +These tests import backend modules in-process (no running server or database), +so they need the ``backend`` and ``gprofiler_dev`` packages importable. This +conftest makes both source roots available on sys.path before collection. +""" + +import sys +from pathlib import Path + +# src/tests/unit/ -> parents[2] == src/ +_SRC_ROOT = Path(__file__).resolve().parents[2] + +for _pkg_root in (_SRC_ROOT / "gprofiler", _SRC_ROOT / "gprofiler-dev"): + _path = str(_pkg_root) + if _pkg_root.is_dir() and _path not in sys.path: + sys.path.insert(0, _path)