Cloudbreak

Cloudbreak is a Postgres-backed Solana RPC implementation focused on serving getProgramAccounts-style endpoints. It maintains a filtered segment of Solana account state by program owner, sourced from Yellowstone gRPC streams and optional full/incremental snapshots.

Supported RPC Methods

The API server exposes the following JSON-RPC methods:

Method	Description
getProgramAccounts	Returns all accounts owned by a program. Supports memcmp, dataSize, and dataSlice filters.
getTokenAccountsByOwner	Returns SPL token accounts owned by a specific wallet.
getTokenAccountsByDelegate	Returns SPL token accounts delegated to a specific address.
getTokenAccountsByMint	Returns all token accounts for a given mint. Generic implementation backed by getProgramAccounts + a memcmp(offset=0, mint) filter; the SPL Token program is used by default, override with programId in the config (e.g. for Token-2022). Only SPL Token and Token-2022 are accepted; any other programId returns -32602 Invalid param: unrecognized Token program id. Streamed response shape, same as getProgramAccounts.
getAccountInfo	Returns the latest version of a single account. Supports base58, base64, base64+zstd, and jsonParsed encodings, dataSlice, and minContextSlot.
getMultipleAccounts	Batched getAccountInfo for up to [server].max-multiple-accounts pubkeys per request (default 100). Returns null per position for missing or indexer-filter-excluded accounts.
getBalance	Returns the lamport balance of an account. Returns 0 for missing or closed accounts (Agave-compatible).
getTokenAccountBalance	Returns the UiTokenAmount of an SPL Token / Token-2022 account, including mint-aware decimals and UI amounts. WSOL native mint is recognised explicitly.
getTokenSupply	Returns the total supply of an SPL Token / Token-2022 mint as a UiTokenAmount, with mint-aware decimals and UI amounts.
getTokenLargestAccounts	Returns the 20 largest token accounts of a mint. Optional; requires [token-largest-accounts] on both indexer and API. Only mints in the indexer's tracked-mints are served, from a maintained top-N record; other mints get a fast "mint not tracked" error. See Largest Accounts.
getSlot	Returns the current slot at the requested commitment level.
getHealth	Returns the health status of the service.
getVersion	Returns the cluster version, Agave-compatible ({"solana-core": "<string>"}). See note below for the composite-string format Cloudbreak uses.
getGenesisHash	Returns the cluster genesis hash as a base58 string.
getVoteAccounts	Returns the cluster's current and delinquent vote accounts with per-voter activated stake, commission, last vote, and recent epoch credits. Optional; only available when the Vote and Stake programs are indexed. See Vote Accounts.
simulateTransaction	Simulates a transaction against indexed account state at the requested slot and returns logs, compute units, return data, requested account state, and balance changes. Supports sigVerify, replaceRecentBlockhash, accounts, innerInstructions, and minContextSlot. Optional; only available on a full unfiltered index. See Simulate Transaction.
getLargestAccounts	Returns the 20 largest accounts by lamport balance, with filter: circulating|nonCirculating support. Optional; requires [largest-accounts] on both indexer and API. See Largest Accounts.
getSupply	Returns the total and circulating supply in lamports plus the non-circulating account list. Optional; requires the [supply] section on the indexer and the [supply] section on the API. See Supply.

Confirmed and finalized commitment levels are supported for every method. With the optional [processed-accounts] section, getAccountInfo, getMultipleAccounts, getBalance, getTokenAccountBalance, getTokenSupply and getSlot also serve processed commitment from an in-memory block feed. Every other method handles processed through the processed-commitment option, which rejects it by default (see API Configuration).

Note on getVersion. The solana-core field returned by Cloudbreak is a composite string of the form "<upstream-solana-core>-cloudbreak<cloudbreak-version>" (e.g. "2.0.21-cloudbreak0.1.0"). The upstream half is the solana-core version reported by the gRPC source the indexer is subscribed to (persisted to the environment_info table on indexer startup); the suffix is Cloudbreak's own crate version. This lets clients see both what cluster they're effectively talking to and which Cloudbreak build is serving them. If the indexer has never written an upstream version, the prefix falls back to "unknown". The response is cached in-process for 10 minutes.

Roadmap

Paginated Responses

Support for paginated responses to queries is planned, enabling clients to efficiently iterate over large result sets without loading all accounts into a single response.

Architecture Overview

                                ┌────────────────┐
                                │  Solana Node   │
                                │  (Yellowstone  │
                                │   gRPC plugin) │
                                └───────┬────────┘
                                        │
                                        ▼
┌──────────────┐            ┌───────────────────┐           ┌───────────────────┐
│   Cluster    │───────────▶│     Indexer       │──────────▶│    PostgreSQL     │
│   Tracker    │            │  (gRPC consumer)  │           │  (accounts/slots) │
└──────┬───────┘            └───────────────────┘           └────────┬──────────┘
       │ scrapes                                                     │
       ▼                                                             ▼
┌──────────────┐                                            ┌───────────────────┐
│  Snapshot    │                                            │   Query Tracker   │◀──┐
│  sources     │                                            │   (auto-indexing) │   │
│  (RPC nodes  │                                            └───────────────────┘   │
│   / sidecars)│                                                                    │
└──────────────┘                                            ┌───────────────────┐   │
                                                            │    API Server     │───┘
                                                            │    (JSON-RPC)     │
                                                            └───────────────────┘

The cluster tracker is a Blockdaemon solcluster tracker instance that scrapes one or more snapshot sources — either Solana RPC nodes that serve snapshot archives or solcluster sidecar agents running next to validators — and exposes a unified /v1/snapshots API listing the available full and incremental snapshot pairs. The indexer asks the tracker which snapshots cover a given slot and downloads the archives directly from the source the tracker reports. A tracker is provided in docker-compose.yaml for local development; see Cluster Tracker.

Project Structure

Crates

Crate	Package	Description
crates/command/	cloudbreak	Main binary. Dispatches to api, index, snapshot, and query-tracker subcommands. Initializes OpenTelemetry tracing. Uses jemalloc as the global allocator on non-MSVC targets.
crates/api/	cloudbreak-api	HTTP JSON-RPC server (Hyper). Serves getProgramAccounts, getTokenAccountsByOwner, getTokenAccountsByDelegate, getTokenAccountsByMint, getAccountInfo, getMultipleAccounts, getBalance, getTokenAccountBalance, getSlot, getHealth, getVersion, getGenesisHash, and (when the Vote and Stake programs are indexed) getVoteAccounts. Streams getProgramAccounts, getTokenAccountsByOwner, getTokenAccountsByDelegate and getTokenAccountsByMint responses as chunked JSON-RPC body frames, applies per-request statement_timeout and a total request timeout, supports batch requests with bounded concurrency, optionally caches results in memory ([gpa-cache]), and reports query usage to the query tracker.
crates/index/	cloudbreak-index	Live indexer. Subscribes to Yellowstone gRPC, applies program filters, writes account/slot state to Postgres. Includes finalize-slot pipeline, self-healing for slot gaps. Uses the snapshot crate to optionally load a full snapshot on startup for fast bootstrapping.
crates/snapshot/	cloudbreak-snapshot	Batch loads Solana full and incremental snapshots. Queries a cluster tracker (/v1/snapshots) to find a covering snapshot pair, downloads the archives from the source reported by the tracker, unpacks them, reads Solana AccountsFile entries, bulk-upserts into Postgres, handles deduplication, optional partition clustering, and index creation.
crates/query-tracker/	cloudbreak-query-tracker	HTTP sidecar service (Hyper). Records getProgramAccounts-shaped demand in a durable index_patterns table, ranks candidates by a configurable priority mode, and optionally drives automatic CREATE INDEX/eviction on Postgres, reconciling API demand against Postgres idx_scan supply. Fully restartable; exposes /track, /debug/*, /metrics, /health on a single port.
crates/core/	cloudbreak-core	Shared library. Contains all configuration struct definitions (TOML deserialization), database connection helpers, the AccountOwnerMap abstraction, the AccountSelectorConfig (include/exclude program filter logic synced to DB), the shared service_health writer/reader, cross-crate metrics, and the self-contained getLargestAccounts and getSupply feature modules (largest_accounts top-N tracker with persistence and pruning, non_circulating stake map and real-time membership, and supply running total). The indexer drives these through a few thin calls.
crates/entity/	cloudbreak-entity	SeaORM entity definitions for Postgres tables: accounts, snapshot_accounts, slots, service_health. Also defines the CommitmentLevel enum and account conversion types. (The environment_info table is accessed via raw SQL in cloudbreak-core, not as a SeaORM entity.)
crates/migration/	cloudbreak-migration	Schema evolution via sea-orm-migration. Defines accounts/snapshot_accounts DDL helpers with TOML-configurable owner partitioning (none / HASH / LIST / LIST+HASH) and per-index toggles, plus bs58 encode/decode PL/pgSQL functions, the service_health table, and the environment_info table (program filter + Solana version). See crates/migration/README.md.
crates/dbtools/	cloudbreak-dbtools	Standalone operator CLI for running analytics queries and toggling Postgres configuration across one or more servers. Does not depend on other cloudbreak crates. See crates/dbtools/README.md.
crates/integration_tests/	integration_tests	Black-box benchmarking and correctness testing tool. Constant-rate load generation, dual-endpoint comparison with slot compensation, VictoriaLogs integration, and optional DB forensics. See crates/integration_tests/README.md.

Getting Started

What You Need

Everything for local development is included in this repository except the following external data sources that you must provide:

1. A Yellowstone gRPC endpoint — for streaming live account and slot updates from a Solana validator. This is the primary data source for the indexer.
2. (Optional) Snapshot sources for the cluster tracker — by default, the tracker shipped in docker-compose.yaml is pre-configured (via tracker-config.yml) to scrape the public Solana mainnet endpoint https://api.mainnet.solana.com, so snapshots work out of the box with no extra setup. The only caveat is that the public endpoint can be rate-limited and may slow down or refuse downloads under load. For production or heavy local use, point tracker-config.yml at your own solcluster sidecar instances (or other Solana RPC nodes that expose snapshot archives) or either use your own valid tracker endpoint — see Cluster Tracker.

Everything else (PostgreSQL, Prometheus, Grafana, Tempo, the cluster tracker — including a working default snapshot source) is provided via Docker Compose, and the example config files are pre-configured for local development.

Quick Start

Heads up: the very first cargo build (or first cargo run) of this workspace can take some time depending on your machine — there are a lot of crates to compile from scratch. The terminal will look idle for long stretches; that's normal. Subsequent builds are incremental and finish in seconds.

# 1. Start all infrastructure (Postgres, Prometheus, Grafana, Tempo, cluster tracker)
docker compose up -d

# 2. Run database migrations (point the binary at the example migration config)
export DATABASE_URL="postgres://cloudbreak:cloudbreak@localhost:5432/cloudbreak"
cp example.cloudbreak.migration.toml cloudbreak.migration.toml
export CLOUDBREAK_MIGRATION_CONFIG=./cloudbreak.migration.toml
cargo run -p cloudbreak-migration

# 3. Copy example configs and set your gRPC + (optionally) tracker endpoint
cp example.cloudbreak.index.toml cloudbreak.index.toml
cp example.cloudbreak.api.toml cloudbreak.api.toml
cp example.cloudbreak.query-tracker.toml cloudbreak.query-tracker.toml
# Edit cloudbreak.index.toml → set [grpc] endpoint and x-token
#   (the default [snapshot.tracker_endpoint] already points at the local Compose tracker,
#    which by default scrapes https://api.mainnet.solana.com — works out of the box, rate-limited)
# (Optional) Edit tracker-config.yml → swap the public endpoint for your own sidecars/RPC nodes

# 4. Start the services (in separate terminals)
cargo run -p cloudbreak -- -c ./cloudbreak.index.toml index
cargo run -p cloudbreak -- -c ./cloudbreak.api.toml api
cargo run -p cloudbreak -- -c ./cloudbreak.query-tracker.toml query-tracker

The API will be available at http://localhost:4000, Grafana at http://localhost:3000 (admin/admin), and Prometheus at http://localhost:9090.

Step-by-Step Setup

1. Start Infrastructure

The docker-compose.yaml starts everything you need for local development:

docker compose up -d

First-run note: The solana-tracker service is built from a Git context (rpcpool/solana-cluster#triton), so the first docker compose up will clone the repo and pull the Go toolchain to build the image. This can add several minutes on top of the normal Compose start. Subsequent runs reuse the cached image. See Troubleshooting: solana-tracker build fails if you ever need to force a rebuild.

This creates:

Service	Port	Details
PostgreSQL	5432	User/password/database: cloudbreak / cloudbreak / cloudbreak
Prometheus	9090	Pre-configured to scrape cloudbreak services
Pushgateway	9091	Push-based metrics
Tempo	4317 (OTLP gRPC), 4318 (OTLP HTTP), 3200	Distributed tracing backend
Grafana	3000	admin/admin, Tempo + Prometheus datasources pre-configured
Cluster Tracker	8458 (public API), 8457 (metrics)	solcluster tracker from rpcpool/solana-cluster#triton. Configured via ./tracker-config.yml.

The database connection URL for all configs is: postgres://cloudbreak:cloudbreak@localhost:5432/cloudbreak

2. Run Database Migrations

The migration CLI requires a TOML config (partitioning shape + per-index toggles) pointed at by CLOUDBREAK_MIGRATION_CONFIG, plus a database URL via DATABASE_URL or the -u <url> flag:

cp example.cloudbreak.migration.toml cloudbreak.migration.toml
export CLOUDBREAK_MIGRATION_CONFIG=./cloudbreak.migration.toml
export DATABASE_URL="postgres://cloudbreak:cloudbreak@localhost:5432/cloudbreak"
cargo run -p cloudbreak-migration

First build can take a while. This is the first cargo run in the workspace, so Cargo will compile every dependency from scratch. The same applies to the first cargo run of the indexer / API / query-tracker in Step 4 (each binary triggers another compile of its own dependency graph the first time). All subsequent runs are incremental.

Migrations create the accounts and snapshot_accounts tables (with the partitioning shape and indexes defined in cloudbreak.migration.toml), PL/pgSQL helper functions, and the service_health and environment_info tables.

Note: The migrations attempt to load the pg_tracing Postgres extension. This is optional and will be skipped with a warning if the extension is not installed (the stock Docker Postgres image does not include it). Everything works without it. The compose file uses PostgreSQL 16, which is the version compatible with pg_tracing if you want to install it later.

Other migration commands:

cargo run -p cloudbreak-migration -- status    # Check migration status
cargo run -p cloudbreak-migration -- up        # Apply all pending
cargo run -p cloudbreak-migration -- up -n 1   # Apply next N
cargo run -p cloudbreak-migration -- down      # Rollback last
cargo run -p cloudbreak-migration -- fresh     # Drop all, reapply
cargo run -p cloudbreak-migration -- refresh   # Rollback all, reapply

⚠️ The database is ephemeral, and each run expects a clean, empty schema. The main data tables (accounts, snapshot_accounts, and their partitions) are created as Postgres UNLOGGED tables — they trade crash-durability for write throughput, so a Postgres crash or restart truncates them. Cloudbreak is designed around this: the indexer rebuilds state from the snapshot + live gRPC stream, it does not treat the database as durable storage. Because of that, do not reuse a dirty database between runs — start each run against a freshly-created, empty schema. Reset with cargo run -p cloudbreak-migration -- fresh (drop all + reapply) or refresh (rollback all + reapply), or drop and recreate the database, rather than pointing a new run at leftover tables from a previous one.

See crates/migration/README.md for the full config reference (partitioning shapes, per-index toggles, env vars, and CLI flags).

Table Partitioning & Indexes

The accounts and snapshot_accounts tables can be partitioned by owner (HASH, LIST, both, or not at all), and each migration-time index on accounts can be individually toggled. Both are controlled by the TOML file pointed at by CLOUDBREAK_MIGRATION_CONFIG. The default example.cloudbreak.migration.toml enables HASH partitioning with 10 partitions plus the standard composite + token indexes — sensible defaults for most setups.

See crates/migration/README.md for the four supported partitioning shapes, the full set of index toggles, and worked examples.

3. Configure the Services

Copy the example configs:

cp example.cloudbreak.index.toml cloudbreak.index.toml
cp example.cloudbreak.api.toml cloudbreak.api.toml
cp example.cloudbreak.query-tracker.toml cloudbreak.query-tracker.toml

The example configs are pre-configured for local development — database URL, metrics ports, and all defaults are already set. The only values you need to fill in are in cloudbreak.index.toml:

Setting	Section	What to set
endpoint	[grpc]	Your Yellowstone gRPC endpoint URL
x-token	[grpc]	Your authentication token (if required)
endpoint	[snapshot.tracker_endpoint]	Cluster tracker URL. Defaults to http://localhost:8458 (the Compose tracker). Remove the [snapshot] section entirely if you don't want snapshot bootstrap/heal.
include	[programs]	Base58 pubkeys of programs you want to index

4. Start the Services

Startup order matters — start the services in this order, each in its own terminal:

1. Indexer first. It populates the environment_info table that the API reads on startup. The API panics on launch if this row does not exist (see Troubleshooting: API returns "environment_info row not found"). Wait until the indexer has finished its own startup (you'll see it begin consuming gRPC blocks in the logs) before moving on.
2. API server. Once the indexer is up and the environment_info row is populated, start the API.
3. Query tracker (optional). Only needed if you want automatic DB index creation; safe to skip for a basic dev loop.

Run each service in a separate terminal:

# 1. Start the indexer (streams account updates from gRPC into Postgres)
cargo run -p cloudbreak -- --config ./cloudbreak.index.toml index

# 2. Start the API server (serves JSON-RPC queries from Postgres)
cargo run -p cloudbreak -- --config ./cloudbreak.api.toml api

# 3. (Optional) Start the query tracker — enables automatic DB index creation
cargo run -p cloudbreak -- --config ./cloudbreak.query-tracker.toml query-tracker

First startup notes:

• If [snapshot] is configured, the indexer will download a full Solana snapshot on first start. This can be very large (100+ GB for mainnet) and take significant time. With the default tracker-config.yml (which uses the public, rate-limited https://api.mainnet.solana.com endpoint), the download can also be throttled, so allow extra time or point tracker-config.yml at your own snapshot source — see Cluster Tracker. For a lighter local setup, either remove the [snapshot] section to skip snapshot loading entirely (the indexer will begin from live gRPC data only), or index a small program like Stake11111111111111111111111111111111111111.
• The correct, healthy steady state requires the snapshot to be loaded. When [snapshot] is configured, the indexer stays unhealthy until snapshot processing finishes — the database holds only the partial data streamed in live from gRPC until then, and the service_health row (and getHealth) stays unhealthy. This is expected: getSlot works immediately as data flows in, but health is intentionally the last thing to clear, and while it is unhealthy the slot-gated account methods (getAccountInfo, getMultipleAccounts, getProgramAccounts, the token methods, simulateTransaction) return JSON-RPC error -32005 (Node is unhealthy). Running without [snapshot] is only meant for a quick smoke test of the full setup, or for iterating on a code change that doesn't need a complete dataset — in that mode there is no startup snapshot to wait on, so the node reports healthy as soon as it begins processing blocks and serves those account methods against the partial live dataset (do not treat a no-snapshot node as a source of complete state). See Troubleshooting: getHealth returns -32005 Node is unhealthy for details.
• The API example config has [tracing] enabled = true, which sends traces to the Tempo instance from Docker Compose. If you're not running the compose stack, set enabled = false or remove the [tracing] section to avoid connection errors in logs.

Manual PostgreSQL Setup (Alternative)

If you prefer not to use Docker, set up Postgres manually:

CREATE USER cloudbreak WITH PASSWORD 'cloudbreak';
CREATE DATABASE cloudbreak OWNER cloudbreak;

Then follow steps 2-4 above. The pg_tracing extension is optional and will be skipped if not installed.

Cluster Tracker

For snapshot bootstrap and self-healing, the indexer queries a cluster tracker (solcluster tracker from Blockdaemon's solana-cluster, built from the rpcpool/solana-cluster triton branch) on its /v1/snapshots endpoint. The tracker periodically scrapes one or more snapshot sources (Solana RPC nodes or solcluster sidecar instances running next to validators), aggregates the available snapshot archives, and tells the indexer which source to download each archive from.

For local development you have two options:

1. Use the tracker provided by Docker Compose (the default). The solana-tracker service in docker-compose.yaml builds a container directly from the rpcpool/solana-cluster#triton Git context, runs it locally, and exposes:

   • :8458 — public API (/v1/snapshots, /v1/snapshot/<filename>). The indexer talks to this port.
   • :8457 — internal/metrics endpoint. The Compose healthcheck uses this.
   • Config mounted from ./tracker-config.yml.

   This is what the example configs point at out of the box and requires no extra setup.

2. Point the indexer at any tracker endpoint you already have access to. Simply set [snapshot.tracker_endpoint].endpoint in cloudbreak.index.toml (and [tracker_endpoint].endpoint in cloudbreak.snapshot.toml if you use the standalone snapshot subcommand) to the URL of an existing solcluster tracker instance, and remove or ignore the solana-tracker Compose service.

tracker-config.yml

The config file controls how often the tracker scrapes its targets and which targets it scrapes. The default file shipped in the repo is:

scrape_interval: 30s

target_groups:
  - group: mainnet
    http_targets:
      targets:
        - https://api.mainnet.solana.com

Key	Description
scrape_interval	How often the tracker polls each target for snapshot metadata.
target_groups[].group	A label for the group (free-form).
target_groups[].http_targets.targets	List of HTTP base URLs to scrape. Each target can be either a solcluster sidecar instance or a regular Solana RPC node that serves snapshot archives.

Default works out of the box. The triton tracker fork can scrape regular Solana RPC nodes that expose snapshot archives at the standard paths (/snapshot.tar.bz2, /incremental-snapshot-<base>-<slot>-<hash>.tar.zst), not just dedicated solcluster sidecar agents. The default target https://api.mainnet.solana.com is a real, working snapshot source — the indexer will be able to bootstrap from it without any further configuration. The trade-off is that it's a public, rate-limited endpoint: downloads can be slow, throttled, or temporarily refused under load. For production or heavy local use, replace the targets: list with your own solcluster sidecar instances (typically http://<validator-host>:13080) or your own RPC nodes that serve snapshots.

After changing tracker-config.yml, restart the service:

docker compose restart solana-tracker

The compose file also publishes the internal port, so you can hot-reload via POST http://localhost:8457/reload instead of restarting.

Pointing the Indexer/Snapshot Crate at the Tracker

In cloudbreak.index.toml:

[snapshot]
accounts-file-concurency = 32

[snapshot.tracker_endpoint]
endpoint = "http://localhost:8458"

In cloudbreak.snapshot.toml (only when running the standalone snapshot subcommand):

[tracker_endpoint]
endpoint = "http://localhost:8458"

If you want to run a tracker outside Compose, point endpoint at it.

Configuration Reference

Indexer (cloudbreak.index.toml)

Top-level

Key	Type	Default	Description
finalize-slot-buffer-size	usize	1000	Buffer size for queuing finalize-slot events.
accounts-owner-map-enabled	bool	false	Enables in-memory account-to-owner map for closed-account handling and owner-change tracking. Increases memory usage.
cleanup-interval-slots	u64	1	Finalized slots between cleanup drains. 1 drains every slot. Higher values coalesce repeat touches of an account into one delete, at the cost of extra account versions every latest-version read scans past.

[supply] (optional)

Maintains the running total-supply figure served by getSupply from the non-circulating stake map and a bounded hot-accounts cache. Requires an empty [programs] filter, accounts-owner-map-enabled = false, owner partitioning off, and the [snapshot] section. from_config validates all of it against the config and the database catalog and panics on any unmet requirement. Run on a dedicated supply-only instance. See Supply.

Key	Type	Default	Description
enabled	bool	false	Enables the supply tracker.
hot-accounts	usize	1000000	Accounts kept resident in the hot-accounts cache (~65 bytes each). Stake accounts live in the non-circulating stake map instead.

[mint-cache] (optional)

Fills and maintains the mints table that the API [mint-cache] loads. See Mint Cache. Requires the [snapshot] section and both token programs in [programs]; the indexer panics at startup otherwise.

Key	Type	Default	Description
enabled	bool	false	Enable the indexer half of the mint cache.
db-write-interval-slots	u64	1000	Slots between two writes of the changed mints to mints. A larger value keeps the API reading mutable mints from Postgres longer after a gap.

[snapshot] (optional)

When this section is present, the indexer downloads and processes snapshots on startup for fast bootstrapping.

Key	Type	Default	Description
accounts-file-concurency	usize	(none)	Max number of AccountsFile entries to process simultaneously.
gap-fill-max-snapshot-retries	u32	10	Max consecutive gap-filling iterations (~30 s each) allowed to fail fetching a covering snapshot pair from the tracker before the self-healing task gives up and fails the indexer. Resets on the first successful fetch.

[snapshot.tracker_endpoint] (optional)

Key	Type	Description
endpoint	String	URL of the cluster tracker HTTP API (e.g. http://localhost:8458). The tracker is provided by Docker Compose for local dev — see Cluster Tracker for how to point it at your own snapshot sources.

If the [snapshot] section is omitted entirely, the indexer skips snapshot processing.

[database]

Shared across all services. Controls the SeaORM/SQLx connection pool and query timeouts.

Key	Type	Default	Description
url	String	required	Postgres connection string.
max-connections	u32	(SQLx default)	Maximum pool connections.
min-connections	u32	(SQLx default)	Minimum pool connections.
connect-timeout	Duration	(SQLx default)	Connection establishment timeout (e.g. "30s").
idle-timeout	Duration	(SQLx default)	How long idle connections are kept alive (e.g. "10m").
acquire-timeout	Duration	(SQLx default)	Max wait time to acquire a connection from the pool (e.g. "60s").
max-lifetime	Duration	(SQLx default)	Max total lifetime of a connection (e.g. "30m").
sqlx-logging	bool	(SQLx default)	Enable SQLx statement logging.
schema-search-path	String	(none)	Postgres search_path (e.g. "public").
test-before-acquire	bool	(none)	Ping connection before returning from pool.
connect-lazy	bool	(none)	Defer connection until first use.
partition-clustering-threshold	u64	(none)	Partition size in bytes above which CLUSTER is skipped during snapshot processing.
save-block-queries-timeout	u64	30	Timeout in seconds for save-block queries.
finalize-slot-queries-timeout	u64	300	Timeout in seconds for finalize-slot queries.
api-queries-timeout	u64	10	Timeout in seconds for API-facing queries.
server-side-timeout	u64	300000	Server-side statement timeout in milliseconds.
max-db-errors-threshold	f64	100	Number of accumulated DB errors before the process exits.

[grpc]

Key	Type	Default	Description
endpoint	String	required	Yellowstone gRPC server URL.
x-token	String	(none)	Authentication token for the gRPC connection.
timeout	u64	required	Connection timeout in seconds.
worker-count	usize	(none)	Number of workers handling subscription events simultaneously.
channel-size	usize	1000	Buffer size for queuing subscription events from gRPC.
chunk-size	usize	1000	Chunk size for subscription events.
max-chunk-bytes-data	usize	2097152 (2 MiB)	Max bytes per data chunk.
max-grpc-errors	usize	required	Max gRPC errors before attempting reconnection (always reconnects on stream None).
reconnect-give-up	Duration	"600s"	How long to keep retrying (re)connection/subscription before giving up and panicking.
reconnect-backoff	Duration	"5s"	Delay between (re)connection/subscription attempts.
reconnect-from-slot-retain	Duration	"300s"	How long a reconnection keeps replaying from the last received slot before dropping from_slot (the server may no longer have it buffered).

[programs]

Controls which program-owned accounts are indexed. The filter is synced to the environment_info table in Postgres so the API can read it.

Key	Type	Default	Description
include	Vec<String>	[]	Base58-encoded program pubkeys to index. If non-empty, only these programs are indexed.
exclude	Vec<String>	[]	Base58-encoded program pubkeys to exclude. Only used when include is empty.

Use either include or exclude, not both. If include is non-empty, exclude is ignored.

[metrics]

Key	Type	Default	Description
host	String	"0.0.0.0"	Prometheus metrics server bind address.
port	u16	8875	Prometheus metrics server port.
subscription-id-key	String	"x-subscription-id"	HTTP header name for subscription ID. Labels the per-subscription metrics and is recorded on the json_encoding span.
request-id-key	String	"x-request-id"	HTTP header name for the caller's request ID. Recorded on the json_encoding span so traces can be correlated with upstream request logs; never used as a metric label.
client-ip-bandwidth-enabled	bool	false	Enable the per-client-IP egress bandwidth metrics (peak gauge + throughput histogram).
client-ip-key	Option<String>	(unset)	HTTP header name carrying the client IP. Groups the bandwidth metrics and is recorded on the json_encoding span. When unset, or absent on a request, both report unconfigured instead of reading any client-settable/PII header.

The last three keys only take effect in the API server.

API Server (cloudbreak.api.toml)

[server]

Key	Type	Default	Description
host	String	"0.0.0.0"	Bind address for the JSON-RPC server.
port	u16	4000	Listen port.
max-connections	u32	100	Maximum concurrent HTTP connections.
batch-handling-max-concurrency	usize	5	Maximum concurrent requests within a single batch JSON-RPC call.
gpa-stream-batch-size	usize (non-zero)	(unset)	Unset (default): getProgramAccounts, getTokenAccountsByOwner, getTokenAccountsByDelegate and getTokenAccountsByMint fetch and encode every account as one batch before the response starts, so every failure is a JSON-RPC error. Set it to opt in to streaming in batches of this size, which lowers peak memory and time to first byte but aborts the connection on a mid-stream failure. See Streamed Responses.
request-timeout	Duration	"60s"	Total per-request wall-clock budget (handler + body transport). When exceeded, the response stream is truncated and the request is counted under the timeout status. See Streamed Responses.
max-multiple-accounts	usize	100	Maximum number of pubkeys accepted per getMultipleAccounts request. Requests exceeding this limit are rejected with an InvalidParams error.

Streamed Responses

getProgramAccounts, getTokenAccountsByOwner, getTokenAccountsByDelegate and getTokenAccountsByMint responses are written as a Transfer-Encoding: chunked JSON-RPC body. The three token methods are served as a getProgramAccounts on the token program, see Token account methods.

Pipeline:

1. DB fetch — Postgres rows are read by sqlx. By default all rows form a single batch. With gpa-stream-batch-size set, rows are grouped into batches of that size.
2. Unbounded channel — completed batches are pushed onto an unbounded tokio::mpsc channel so the database connection can close as fast as possible (no head-of-line blocking from slow encoding/clients).
3. Encoding — each batch is decoded into UiAccounts (per-batch spawn_blocking to keep the runtime responsive).
4. JSON serialization — encoded accounts are serialized into a 64 KB pre-allocated BytesMut. Whenever the buffer reaches 32 KB, the filled portion is frozen into a Bytes chunk and yielded as a single HTTP body frame. Accounts larger than the chunk threshold cause the buffer to grow naturally and are flushed as a single oversized frame.

The first batch is fetched, encoded, and serialized before the response status line is committed. Any failure up to that point is a proper JSON-RPC error response.

Default (single batch). The first batch is the whole result, so SQL errors, query timeouts, and encoding errors (for example base58 data over 128 bytes) are always JSON-RPC errors. This matches Agave, which builds the full result before it responds. The cost is that peak memory per request holds every encoded account, and the first byte waits for the full query and encoding.

Opt-in streaming (gpa-stream-batch-size set). Accounts are fetched and encoded in batches, which lowers peak memory and time to first byte for large responses. The trade-off: a failure after the first batch cannot be reported as a JSON-RPC error, because HTTP 200 and part of the body are already sent. The server aborts the connection instead (HTTP/1: no final chunk; HTTP/2: stream reset). Clients see a transport error, such as an incomplete body, and must treat the request as failed and retry. A proxy in front of the API must pass the abort through, not buffer the body and end it normally. JSON-RPC batch requests are not affected: they buffer each entry, and a mid-stream failure becomes a -32603 Internal error entry.

Token account methods

getTokenAccountsByOwner and getTokenAccountsByDelegate build Agave's filters and run them as a getProgramAccounts on the token program. They share its streaming, its cache and its SQL.

Method	Filters
getTokenAccountsByOwner	memcmp(32, owner), tokenAccountState, plus memcmp(0, mint) with { "mint" }
getTokenAccountsByDelegate	memcmp(72, [1,0,0,0]) (delegate is set), memcmp(76, delegate), tokenAccountState, plus memcmp(0, mint) with { "mint" }

• { "mint" } reads the mint first. Its token program becomes the program. A missing or closed mint returns Invalid param: could not find mint, and a mint that does not unpack returns Invalid param: Token mint could not be unpacked. { "programId" } must be a token program.
• The response always has a context.
• For a token program, the SQL uses the token indexes: memcmp(32) becomes token_owner, memcmp(0) becomes token_mint, and memcmp(72, [1,0,0,0]) is written as the delegate index predicate. An owner and a mint filter together use the (token_owner, token_mint) index on snapshot_accounts. tokenAccountState matches Agave's Account::valid_account_data: initialized, 165 bytes, or longer with account type 2 and not a 355-byte multisig.
• A token-program getProgramAccounts must filter on owner, mint or delegate (memcmp of 32 bytes at 32, 0 or 76). Otherwise it returns Invalid params. Agave accepts it as a full scan.
• With jsonParsed, each account is encoded with its own mint, as in Agave. Each batch of fresh rows looks up its distinct mints in one query by pubkey. A token account whose mint is missing or does not unpack has no decimals, does not parse, and is left out, as in Agave. This applies to getProgramAccounts on a token program too.

Known difference: a token-program getProgramAccounts with memcmp(76, delegate) and no filter at offset 72 gets memcmp(72, [1,0,0,0]) added, so the delegate partial index serves it. Agave would also return accounts whose delegation was revoked: revoke clears the delegate tag and leaves the old delegate bytes in place.

request-timeout is enforced by a TrackedBody wrapping the response body. On expiry the body is truncated, the cloudbreak_api_request_duration_ms http_with_transport observation is not recorded, and a cloudbreak_api_requests_total{method="http",status="timeout"} counter is incremented instead.

[gpa-cache] (optional)

Optional in-memory cache for getProgramAccounts responses. Omitting this section disables the module entirely.

Key	Type	Default	Description
max-total-bytes	usize	required	Maximum cache size in bytes. When inserting a new query would exceed this, the oldest cached queries (by slot) are evicted until enough room exists.
min-bytes-per-query	usize	required	Minimum serialized size of a query for it to be cached at all. Smaller queries are skipped so they don't dilute room available for heavier ones.
max-bytes-query-cleanup	usize	(unset)	Optional upper bound on the size of a query that is eligible for eviction. Queries larger than this are pinned: they are normally skipped by cleanup and stay cached (until replaced by a newer version of the same query). The pin is soft — see max-pinned-bytes-ratio. Omit to make every cached query evictable. Note: when the oldest slots are mostly pinned (and under the pinned cap), a new insert may fail to free enough room and be skipped.
max-pinned-bytes-ratio	f64	(unset)	Cap on how much space pinned queries may collectively occupy, expressed as a fraction of max-total-bytes (e.g. 0.5 = at most half the cache). When pinned usage exceeds max-total-bytes × ratio, the oldest pinned queries are evicted via the normal cleanup process until usage drops back under the cap — so the pin set can never grow unbounded. Required when max-bytes-query-cleanup is set (config load fails otherwise); ignored when it is unset (nothing is pinned). Must be in the range (0.0, 1.0].

The cache is keyed by (program, sorted filters, encoding, data_slice, commitment) and holds the pre-serialized JSON bytes of each account, ref-counted via bytes::Bytes. Concretely:

• Postgres only sends what's missing. When a cached entry exists for a query, the SQL returns full row data only for accounts whose slot is newer than the cached slot; for everything else it returns just (pubkey, slot) to confirm the cached version is still valid. Less data over the wire, less server-side decoding.
• Cache hits are zero-copy and zero-allocation. Cached account bytes are appended into the outgoing response buffer by ref-counting an existing Bytes slice — no clone, no re-encode, no extra allocation.
• No JSON re-serialization. Cached entries store the exact JSON fragment that already left the encoder once; on subsequent serves the same bytes are emitted verbatim.

A jsonParsed token-program response embeds data from each account's mint, and a cache entry is only invalidated when an account changes. So such a response is cached only when every mint it uses is immutable: a Tokenkeg mint, the native mint, or a Token-2022 mint without MintCloseAuthority, InterestBearingConfig and ScaledUiAmount. A response with a mutable mint, a missing mint or a left-out account is not cached. cloudbreak_gpa_cache_skipped_total counts these by method, size and reason.

Eviction is age-ordered by slot (oldest cached query goes first), skipping any query pinned by max-bytes-query-cleanup. Pinning is bounded: once pinned queries collectively exceed max-total-bytes × max-pinned-bytes-ratio, the oldest pinned queries are evicted in the same age order until usage is back under the cap. Current pinned usage, the configured ratio, and the resulting threshold are reported by /debug/modules/gpa_cache (stats.pinned_size_bytes, config.max_pinned_bytes_ratio, stats.pinned_threshold_bytes).

[database]

Same as the indexer [database] section. For the API, api-queries-timeout and server-side-timeout are the most relevant timeout settings.

[metrics]

Same as indexer [metrics].

[query-tracker-client]

Connects the API to the query tracker service for reporting GPA query patterns.

Key	Type	Default	Description
endpoint	String	required	Query tracker RPC endpoint (e.g. http://localhost:4001).
timeout	Duration	(none)	Request timeout (e.g. "5s").
flush-interval	Duration	(none)	How often batched query counts are flushed to the tracker (e.g. "1m").

[slot-syncronizer] (optional)

Keeps the confirmed and finalized slots and the service health in memory. The indexer's slots_notify trigger sends every real change of a slots row on the Postgres channel cloudbreak_slots, and the API applies it as it arrives. A lost listener connection, a bad payload, or 60 s without a notification panics. Until the first notifications arrive after a start, the slots are 0 and the node reads as unhealthy. Enabled by default; omit this section or set enabled = false to disable.

Key	Type	Default	Description
enabled	bool	true	Enable/disable slot synchronization.

processed-commitment (top-level, optional)

Controls how the API handles requests that specify the processed commitment level. This is a top-level key (not inside any section).

Value	Description
"reject"	(default) Return JSON-RPC -32602 Processed commitment level is not supported when a client requests processed commitment.
"use-confirmed"	Silently respond with confirmed data instead of rejecting the request.

Example:

processed-commitment = "use-confirmed"

[grpc] (optional)

The API gRPC feed: one Yellowstone subscription to blocks with accounts at processed commitment. It runs when [processed-accounts] or [mint-cache] is enabled, and feeds both. It uses the indexer's subscription code (cloudbreak_core::modules::grpc): the same reconnection, from_slot replay, and panic after reconnect-give-up. The connection fields default to the values the indexer runs with.

Field	Type	Default	Description
enabled	bool	false	Enable the feed.
endpoint	string	""	Yellowstone gRPC endpoint that allows processed commitment.
x-token	string	none	Yellowstone gRPC access token.
timeout	u64	60	Connection timeout in seconds.
max-grpc-errors	usize	1	Max gRPC errors before attempting reconnection (always reconnects on stream None).
reconnect-give-up	Duration	"600s"	How long to keep retrying (re)connection/subscription before the API panics.
reconnect-backoff	Duration	"5s"	Delay between (re)connection/subscription attempts.
reconnect-from-slot-retain	Duration	"300s"	How long a reconnection keeps replaying from the last received slot.

[mint-cache] (optional)

Keeps the mint data jsonParsed needs in memory. See Mint Cache. Requires [grpc] and [slot-syncronizer], both with enabled = true, and the indexer's [mint-cache]. Startup fails without [grpc] and [slot-syncronizer].

Field	Type	Default	Description
enabled	bool	false	Enable the mint cache.

[processed-accounts] (optional)

Serves processed commitment for getAccountInfo, getMultipleAccounts, getBalance, getTokenAccountBalance, getTokenSupply and getSlot. The API subscribes to Yellowstone blocks at processed commitment and keeps the blocks around the Postgres confirmed slot in memory. A key written in those blocks is answered from memory. Any other key reads Postgres at the confirmed slot. When the blocks cannot be linked to the confirmed slot, a processed request answers exactly as a confirmed request would, even with processed-commitment = "reject". Other methods keep following processed-commitment.

Requires [grpc] and [slot-syncronizer], both with enabled = true. Startup fails without them. Each API instance carries its own block feed.

Field	Type	Default	Description
enabled	bool	false	Enable processed commitment for the methods above.

Example:

[grpc]
enabled = true
endpoint = "https://grpc.example:443"
x-token = "..."

[processed-accounts]
enabled = true

unhealthy-response (top-level, optional)

Controls how the API responds to requests while the node is unhealthy (the slots.health flag is unset / the slot syncronizer reports unhealthy). This is a top-level key (not inside any section).

Value	Description
"json-rpc-error"	(default) Return a JSON-RPC error -32005 (Node is unhealthy) with HTTP 200 OK.
"http-unavailable"	Return an HTTP 503 Service Unavailable response instead.

Note: http-unavailable only applies to single requests. Batch requests always return HTTP 200 OK with per-item JSON-RPC errors, since an HTTP status cannot be expressed per batch item.

Example:

unhealthy-response = "http-unavailable"

[supply] (optional)

Serves getSupply from the supply ring the indexer maintains. The section carries only enabled (default false). Set it together with the indexer's [supply] section; a node with the section absent or disabled answers getSupply with method-not-found. See Supply.

[tracing] (optional)

OpenTelemetry tracing configuration. If this section is omitted, OTel is disabled and only tracing-subscriber is used.

Key	Type	Default	Description
enabled	bool	(none)	Enable OTel export.
endpoint	String	"http://localhost:4317"	OTLP gRPC collector endpoint.
sample-ratio	f64	(none)	Trace sampling ratio (0.0 to 1.0).
export-interval	u64	(none)	Export interval in seconds.
max-batch-size	u64	(none)	Max spans per export batch.
max-queue-size	u64	(none)	Max queued spans before dropping.
track-idle-time	bool	(none)	Include idle time in spans.
span-filter	Vec<String>	(none)	Span names to export. See example.cloudbreak.api.toml for the current full list (covers gPA, gTABO/gTABD, JSON encoding, HTTP transport, mint lookups, the cache finalize span, and the processed and account reads).

Query Tracker Service (cloudbreak.query-tracker.toml)

[server]

Key	Type	Default	Description
host	String	"0.0.0.0"	Bind address.
port	u16	4001	Listen port for the single HTTP server, which serves both the functional endpoints (/track, /debug/*) and the operational endpoints (/metrics, /health).
max-connections	u32	100	Max concurrent connections.

[database]

Same as other services.

[query-tracker]

Controls demand recording, automatic index creation, usage-based eviction and discrepancy detection. All state lives in the index_patterns table, so the service is fully restartable; these keys only tune behavior. Defaults reproduce the previous behavior.

Key	Type	Default	Description
create-database-indexes	bool	false	Enable automatic CREATE INDEX from recorded demand. When off, demand is still recorded.
index-generation-threshold	u32	10	Minimum demand count before a pattern becomes a creation candidate.
index-creation-delay	Duration	"10s"	Poll interval of the creation loop.
priority-mode	String | table	"frequency"	Ranking criterion shared by creation (highest first) and eviction (lowest first). Bare strings rank on lifetime totals: frequency, cost, cost-per-hit. weighted is a table that ranks on recent windowed activity and carries its own weights + measurement window: priority-mode = { weighted = { demand-weight = 1.0, supply-weight = 0.0, failure-weight = 0.0, latency-weight = 0.0, rate-window = "1h" } }. Score = (avg_cost · gain) · (1 + demand-weight·demand + supply-weight·(supply/2) + failure-weight·failed), each count being the value in the last rate-window (default 1h); supply is halved (a GPA scans both tables). gain = 1 + latency-weight·ln((without-index-compensation-factor·without)/with) scales avg_cost by the measured latency win from the index (neutral until the pattern has served requests both with and without the index). All weights default 0.0 = plain average cost-per-hit (no rate roll).
cost-eligibility-threshold-us	u64?	(none)	Only patterns whose average cost-per-hit reaches this many µs are eligible.
included-programs	Vec<String>	[]	Only create indexes for these program pubkeys. Empty means all programs are eligible.
excluded-programs	Vec<String>	[]	Never create indexes for these program pubkeys.
indexer-metrics	String	required	host:port of the indexer's Prometheus metrics endpoint. Used to defer DDL (create/drop) when the indexer is behind.
indexer-metrics-threshold	u64	5	cloudbreak_finalize_slot_handler_queue_size threshold above which DDL is deferred.
indexer-cleanup-lag-threshold	u64	32	cloudbreak_cleanup_lag_slots threshold above which DDL is deferred. An indexer that does not publish the gauge reads as no pressure.
max-auto-indexes	usize?	(none)	Optional cap on total indexes on the snapshot_accounts table.
index-eviction-enabled	bool	false	Enable usage-based eviction of idle auto-indexes.
use-supply-for-eviction	bool	false	When true, also require supply-idle (last_seen_used / no idx_scan growth) for eviction eligibility. Off = demand-idle + age-grace only. Enabling is more conservative but the eviction-pass supply refresh can collapse the candidates list for up to index-min-idle afterward on high-rotation DBs (mitigate with more frequent eviction passes).
index-eviction-interval	Duration	"1h"	How often the eviction pass runs.
index-min-age-grace	Duration	"1h"	Minimum age before an index is eligible for eviction.
index-min-idle	Duration	"24h"	How long a pattern must be idle to be droppable. Demand-idle is always required; supply-idle only when use-supply-for-eviction is on.
eviction-fill-threshold	f64	0.9	Fill target as a fraction of max-auto-indexes: creation is unguarded below it, value-guarded in the buffer band above it, and eviction trims back down to it (needs max-auto-indexes).
value-guard-creation-bias	f64	1.0	Multiplier on a creation candidate's score in the creation-time value guard, tuning stickiness toward existing indexes (which carry gain/idx_scan signal new candidates lack). > 1.0 builds new indexes more readily (less sticky), < 1.0 favors incumbents (stickier), 1.0 compares as-is. Only consulted in the buffer band.
drop-lock-timeout	Duration	"5s"	lock_timeout for each DROP INDEX; on timeout the drop is skipped and retried next pass.
drop-retries	u32	1	Extra DROP INDEX attempts on lock timeout within a pass.
index-regression-guard	String	"off"	What to do when an index measures slower than the pattern was without it. off ignores it; warn logs + reports the count via query_tracker_regressed_indexes; evict drops the pair and marks the pattern rejected (not rebuilt until proven slower without the index). Runs in the eviction pass (requires index-eviction-enabled) but ignores the fill threshold.
index-regression-ratio	f64	1.2	How many times slower with the index than without before it counts as a regression (1.2 = ≥20% slower). Also the hysteresis a rejected pattern must clear before it is retried.
index-regression-retry-delay	Duration	"6h"	How long a pattern rejected for a regression must stay rejected — accumulating fresh without-index samples — before it may be retried (and only if its without-index average has climbed past index-regression-ratio × the with-index average recorded at rejection). Guards against rebuild churn. Detection needs no separate window: an index is judged once it is older than index-min-age-grace.
without-index-compensation-factor	f64	1.0	Multiplier on the without-index average cost wherever it is compared to the with-index average — the weighted latency gain and the regression guard's detection/recovery. Compensates for pg settings that keep the two sides from being strictly comparable (e.g. a without-index GPA fans out across parallel workers, so its wall-clock looks cheap while it burns far more CPU/IO than a single-worker index scan). > 1 inflates the measured without-index cost so the index earns proportional credit and is less likely to be judged a regression; 1.0 compares raw wall-clock as-is.
discrepancy-enabled	bool	true	Flag/metric when demand (API) and supply (idx_scan) diverge (e.g. Postgres ignoring an index).
discrepancy-delta	f64	0.10	Relative gap between demand and supply beyond which a pattern is flagged.
explain-enabled	bool	false	Optional third signal: periodic EXPLAIN (plan only) sampling of each index, on both tables, using the stored example_request values.
explain-interval	Duration	"6h"	How often the EXPLAIN sampling pass runs when enabled.

Snapshot Processor (cloudbreak.snapshot.toml)

Note: Snapshot processing is already built into the normal indexer operation (when [snapshot] is present in the indexer config). The standalone snapshot subcommand exists for special cases where you need to run snapshot ingestion independently from the indexer — for example, to bulk-load account state into an existing database without starting a gRPC subscription. In most setups you do not need to run this separately.

Top-level

Key	Type	Description
accounts-file-concurency	usize	Max number of AccountsFile entries to process simultaneously.

[tracker_endpoint]

Key	Type	Description
endpoint	String	Cluster tracker HTTP API URL (e.g. http://localhost:8458). See Cluster Tracker.

[database]

Same as other services. The partition-clustering-threshold is particularly relevant here (skips CLUSTER on partitions larger than this size in bytes).

[programs]

Same include/exclude program filter as the indexer.

[metrics]

Same as other services.

Environment Variables

Variable	Where	Description
RUST_LOG	All binaries	Tracing filter directive. Default: info,sqlx=error. Supports per-target filtering (e.g. debug,hyper=warn).
DATABASE_URL	Migration CLI	Standard SeaORM/SQLx convention for the migration tool. Can also be supplied via the -u <url> CLI flag (which takes precedence).
CLOUDBREAK_MIGRATION_CONFIG	Migration CLI	Required. Filesystem path to the migration TOML config (partitioning shape + per-index toggles). See crates/migration/README.md for the schema.

Automatic Database Index Creation

The query tracker enables automatic creation of database indexes based on observed query patterns. All decision state is persisted in the durable index_patterns table, so the service is fully restartable and never loses recorded demand. This is how it works:

1. The API server reports getProgramAccounts-shaped query demand (request counts, cost, failures) to the query tracker at the /track endpoint, which folds it into each pattern's row in index_patterns.
2. When a pattern's cumulative demand reaches index-generation-threshold, it becomes a creation candidate. Demand is cumulative and persisted — there is no reset interval.
3. The creation loop wakes every index-creation-delay, ranks the candidates on read according to priority-mode (frequency, cost, cost-per-hit, or weighted), and builds the highest-priority one. Eviction uses the same ranking in reverse (drops the lowest), so "most worth building" and "least worth keeping" are two ends of one order. Indexes are always created as a pair on the accounts and snapshot_accounts tables.
4. Only one index pair is created per tick, to avoid overloading the database.
5. Before creating (or dropping) an index, the query tracker checks two indexer gauges: cloudbreak_finalize_slot_handler_queue_size against indexer-metrics-threshold, and cloudbreak_cleanup_lag_slots against indexer-cleanup-lag-threshold. Either one over its threshold defers DDL until the indexer catches up.
6. If max-auto-indexes is set, creation is bounded by two sizes: the fill target floor(eviction-fill-threshold × max-auto-indexes) and the hard cap max-auto-indexes. Below the target the top candidate is built freely; in the buffer band (target, cap] a creation-time value guard (see below) only builds a candidate that out-values the index it would displace — unless nothing is reclaimable to displace, in which case it builds anyway up to the cap and lets eviction reclaim later; at the cap creation is paused until eviction reclaims the buffer. Candidates are kept queued in index_patterns with no loss of data.
7. Use included-programs and excluded-programs to control which programs are eligible for automatic indexing.

Eviction and observability

When index-eviction-enabled is set, a periodic pass (index-eviction-interval) reclaims index budget, but conservatively. An index pair is only ever dropped when all of these hold (any one failing defers the drop, never losing the pattern):

• track_counts is on in Postgres — otherwise idx_scan is frozen and the whole pass is skipped, so stale stats can never drive a drop;
• max-auto-indexes is set and the snapshot_accounts table is above the fill target (eviction-fill-threshold) — at or below the target nothing is dropped;
• the pair is demand-idle for index-min-idle (and supply-idle too when use-supply-for-eviction is on) and older than index-min-age-grace — demand-idle alone stops a still-wanted index from being dropped and avoids the drop→rebuild churn loop;
• the indexer is not under backpressure.

When above the fill target (eviction-fill-threshold), eligible pairs are dropped in ascending priority-mode order (least worth keeping first) just until the table is back at the target. The trim is unconditional — the value trade-off already happened at creation (below), so eviction simply reclaims the buffer.

Value guard (at creation time). The value trade-off lives at creation, not eviction: the fill target is the operating size, and the buffer band up to max-auto-indexes is entered only by indexes worth more than what they displace. When a new index would land in the buffer it is scored by priority-mode and compared against the index it would push out — which is not the single worst index. over = current − target indexes are already destined for the next trim, so the candidate competes with the first index that would survive that trim: the one at position over in eviction (ascending-score) order. Building the candidate raises the overflow to over + 1, the next trim drops positions 0..=over, and the candidate keeps a lasting slot only if it out-scores that boundary index. Example (max-auto-indexes = 200, eviction-fill-threshold = 0.7 ⇒ target 140): at current = 140 (over = 0) the candidate must beat the single worst eligible index (a straight swap-the-worst); at current = 176 (over = 36) the 36 worst are leaving anyway, so it must beat the 37th-worst to earn a durable slot. If it cannot beat that boundary, creation is deferred and the candidate stays queued. If nothing is currently reclaimable to displace (fewer eligible indexes than the overflow), the candidate is built anyway up to the hard cap — the cap is the real ceiling, and eviction reclaims later once indexes go idle, rather than stalling creation behind indexes that are not yet droppable. When eviction is disabled or no cap is set, the guard is inactive and creation is cap-only.

Stickiness (value-guard-creation-bias). Both sides of the guard use the same priority-mode score, but a created index carries realized signal a fresh candidate cannot — its with-index latency gain and its idx_scan supply — which structurally favors incumbents. That is usually desirable (proven indexes should not yield to unproven ones), but value-guard-creation-bias tunes it: the factor multiplies the candidate's score in the comparison, so > 1.0 lets new indexes win more easily (less sticky, more churn) and < 1.0 favors incumbents (stickier); 1.0 (default) compares as-is. The guard inherits the mode's notion of value — lifetime-total modes let a once-hot idle index resist displacement, while weighted weighs recent windowed activity so decisions track current throughput.

Latency regression guard. The tracker records each pattern's average query latency with the index versus without it (cost is routed into a with- or without-index bucket by the pattern's status at record time). When index-eviction-enabled is on and index-regression-guard is not off, the eviction pass — before the supply refresh, and regardless of the fill threshold, since a harmful index should go even when there is room — flags any created index that is slower with the index than without by more than index-regression-ratio, once the index is older than index-min-age-grace (long enough to have gathered with-index latency to compare against its frozen without-index baseline). In warn mode it logs and reports the current count via the query_tracker_regressed_indexes gauge; in evict mode it also drops the pair and marks the pattern rejected (so that gauge stays near zero and the count shows up under query_tracker_patterns{status="rejected"} instead). A rejected pattern is not rebuilt by demand alone — only after it has been rejected for at least index-regression-retry-delay (time spent gathering fresh without-index samples) and its without-index average since rejection has climbed past the with-index average captured at rejection (times index-regression-ratio) does the creation loop return it to candidate. Both the detection and the recovery comparison first scale the without-index average by without-index-compensation-factor, so a without-index scan whose wall-clock is deflated by parallel workers (fast in elapsed time, but far heavier in CPU/IO) is not mistaken for a cheaper alternative and does not push a genuinely useful index into a false regression. The same latency signal optionally feeds the weighted score via latency-weight, and the per-index averages are visible on /debug/created.

Discrepancy detection (discrepancy-enabled) and EXPLAIN sampling (explain-enabled) are observational only: they flag/log when API demand and Postgres idx_scan diverge, or when the planner would not use an index, surfacing problems on /debug/* and /metrics. To keep the logs quiet, each pass emits one summary line instead of a message per index: the discrepancy pass reports N checked — X ok, Y starved, Z over_scanned (naming the worst offenders, warn when anything is starved), and the EXPLAIN pass reports the per-state counts plus the fully unused indexes with their compensated_idx_scan (warn when any index is unused or a probe errored). The EXPLAIN probe is built from the row's stored example_request — a real request captured on first sight — so its memcmp constants are realistic values the planner has statistics for (it falls back to zeros per column when no example is stored yet), and it runs on both physical tables (accounts and snapshot_accounts). Because those tables are partitioned, the probe matches the plan against the index's partition-tree members (the auto-named per-partition child indexes that EXPLAIN actually prints), not the partitioned parent's name — which is never scanned directly and would otherwise make every index look unused. Each pass folds the two per-table verdicts into one explain_state (none / accounts_table / snapshot_accounts_table / both), persists it to the pattern (surfaced on /debug/created, filterable via filter=explain_incomplete/explain_none/explain_partial for the detailed per-index view), and refreshes the query_tracker_explain_state{state} gauge with the per-state counts of the latest run. That example request is also surfaced as example_request on the /debug/created and /debug/candidates views (with ?example=true) for documentation/sampling. They never create or drop anything and do not gate eviction. Note a served getProgramAccounts is a UNION ALL over accounts + snapshot_accounts, so it scans both indexes of a pair — raw idx_scan runs ~2× the request count. The tracker divides supply by that factor (SCANS_PER_REQUEST = 2) before comparing it to demand, so a healthy index sits near a 1:1 ratio.

Self-Healing (Slot Gap Detection and Repair)

The indexer includes a self-healing mechanism that automatically detects and repairs gaps in the slot stream. This is critical for maintaining data consistency when the gRPC stream drops slots due to network issues, reconnections, or transient failures.

How It Works

1. Gap detection (no RPC): As each block arrives from the gRPC stream, the indexer checks that it builds directly on the last block it received, using the block's own parent_slot / parent_blockhash against the in-memory blocks map. If it does, any slots in between were simply empty/skipped by the validator and are ignored. If it does not, at least one real block was missed: the whole intervening range is queued for repair in gaps_list. This replaces the previous getBlocksWithLimit RPC confirmation — gaps are confirmed purely from the chain.

2. Pause + mark unhealthy: The instant a gap is confirmed, finalization is paused and the service is flagged unhealthy via the service_health table. Live finalized notifications keep buffering (bounded by finalize-slot-buffer-size, then back-pressuring the stream) but are not applied until the gap is repaired, preserving in-order finalization.

3. Gap filling via incremental snapshots: Every 30 s a background task processes confirmed gaps by asking the cluster tracker for an incremental snapshot pair covering the newest missing slot, downloading it (into a timestamped directory), and processing only the gap slots. Repaired accounts are written to the database and enqueued for finalization directly (snapshot data is already finalized). Gap slots that have no accounts in the snapshot are empty/skipped slots: they are logged (target self_healing_empty_slots) and dropped from the list. If the tracker has no covering pair yet, the task retries on the next tick, up to gap-fill-max-snapshot-retries (default 10) consecutive attempts — past that the indexer gives up and exits with an error rather than staying unhealthy forever. The counter resets as soon as a snapshot pair is fetched successfully.

4. Missed finalized notifications: A reconnect can also drop finalized notifications for slots just below a large gap. When finalizing a live slot the finalizer walks its ancestor chain (hash-checked) to finalize any ancestors whose notification was missed; additionally the slot just before each repaired gap (gap_start - 1) is seeded so its ancestors are caught even though repaired slots carry no chain data to bridge the walk.

5. Recovery: Once gaps_list drains, finalization is resumed, which restores the service to healthy. Out-of-order slots that arrive late (e.g. gRPC buffering) are removed from the gaps list when received.

A confirmed gap is only expected after startup has finished. During startup the (now-paused) finalizer worker is what completes startup, so a gap there can never be repaired; the fill task fails fast (panics) rather than stalling forever.

Monitoring

Use the indexer's operational debug endpoints to inspect gap and finalizer state:

• /debug/modules/self_healing — chain tips, still-missing slots grouped into gaps (with boundary, length, and distance behind confirmed), and summary counts.
• /debug/modules/finalizer — the confirmed-but-not-finalized blocks map, the ordered pending queue, and pause state.
• /debug/modules/supply — the supply tracker status, total, slot, and hot-accounts cache counters, plus a single cache entry lookup.
• /debug/modules/non_circulating — the non-circulating tracker status, clock, member count, stake map size, running non-circulating lamports, the heads of the two lockup-expiry sets, plus a single stake map entry lookup.

The [snapshot] section (with [snapshot.tracker_endpoint]) must be present in the indexer config for gap filling to work, since it relies on the cluster tracker to discover and download incremental snapshots.

Program Filter Configuration

Programs are configured in two different places, each serving a different purpose. Understanding how they relate helps avoid confusion:

Config	Location	Purpose
[programs] section	Indexer config (cloudbreak.index.toml)	Controls which accounts are indexed. Only accounts owned by included programs (or all except excluded programs) are written to the database. This filter is synced to the environment_info table so the API knows which programs are available.
included-programs / excluded-programs	Query tracker config (cloudbreak.query-tracker.toml)	Controls which programs are eligible for automatic index creation. This is a subset concern — it only makes sense to auto-index programs that are already being indexed. Use excluded-programs to skip programs like SPL Token that have too many accounts for useful indexes.

These two are independent but related: the indexer [programs] determines what data exists, and the query tracker programs control which of that data gets automatic database indexes.

Note (storage layout): If you want certain programs to get dedicated LIST partitions in the accounts / snapshot_accounts tables (a schema-level storage optimization, separate from the indexing filter above), configure [pg-owner-partitions].programs-for-list-partition in the migration TOML. This must be set before running migrations; changing it later requires re-running migrations (e.g. fresh). See crates/migration/README.md.

Optional Modules

Vote Accounts (getVoteAccounts)

getVoteAccounts is optional and only served when the indexer's [programs] filter includes both the Vote (Vote111111111111111111111111111111111111111) and Stake (Stake11111111111111111111111111111111111111) programs. Cloudbreak checks this at startup; if either is missing, the method returns JSON-RPC -32601 Method not found and the supporting background tasks are not started.

commitment selects the slot the Vote accounts are read at, and the same slot is the reference for delinquentSlotDistance. Activated stake comes from the epoch cache below, which is per epoch, so commitment does not change it. processed follows the processed-commitment option; this method has no native processed support.

The response combines two sources. The per-account fields (commission, last vote, root slot, recent epoch credits, node pubkey) are read from the indexed Vote accounts. The per-voter activated stake and epoch-set membership (epochVoteAccount) come from a separate epoch_stakes table, since effective stake cannot be derived from a single account.

The epoch_stakes table is seeded from snapshot metadata. When the indexer loads a snapshot, it extracts each voter's activated stake and epoch-set membership from the snapshot's bank stakes and versioned epoch-stakes data and upserts them into the table. The API loads the latest epoch's rows into an in-memory cache on startup and polls the table every 30 s, refreshing when the epoch advances or the in-epoch stake total changes. Before the indexer has completed a snapshot pass with stake data the cache is empty, and getVoteAccounts returns a node-unhealthy error rather than partial results.

A snapshot captures stake at a single point in time, but activated stake drifts within an epoch as stake warms up or cools down and as rewards are distributed at the epoch boundary. To track this, the indexer runs a background recomputer that recalculates each voter's effective stake for the current epoch from the indexed Stake accounts (using the same warmup/cooldown rules as the runtime), upserts the result into epoch_stakes, and keeps only the two most recent epochs.

The recomputer detects drift by comparing the total activated stake across runs. It recomputes every 60 s and treats an epoch as converged once the total is unchanged for three consecutive polls and the indexed EpochRewards sysvar reports that reward distribution has finished. If that sysvar is not indexed, it converges on stability alone after roughly 30 minutes. After converging it keeps recomputing on a slower 600 s heartbeat, returning to the 60 s cadence if a late reward write or a healed ingestion gap moves the total again.

Simulate Transaction (simulateTransaction)

simulateTransaction is optional and only served on a full, unfiltered index (empty [programs] include and exclude lists). Simulation must be able to load any account a transaction touches — including program, lookup-table, sysvar, and feature-gate accounts — so a filtered index cannot serve it. Cloudbreak checks this at startup; if the index is filtered, the method returns JSON-RPC -32601 Method not found.

The transaction is executed read-only against the indexed account state at the requested slot; nothing is committed. Cloudbreak reconstructs the cluster's actually-activated feature set at that slot from the on-chain feature accounts (rather than enabling all features), so compute-unit accounting and execution behaviour match mainnet. replaceRecentBlockhash substitutes the latest recorded blockhash before execution and reports it with its lastValidBlockHeight; sigVerify verifies signatures; accounts returns post-simulation state for the requested addresses; innerInstructions includes decoded inner instructions.

Cloudbreak accepts legacy, v0, and v1 transactions. A v1 transaction can be larger than the legacy 1232-byte packet, so the decoder applies the version's own size bound and rejects anything larger with an invalid-params error.

Supply (getSupply)

getSupply is optional. It needs a full, unfiltered index (empty [programs] include and exclude lists), accounts-owner-map-enabled = false, owner partitioning off (single accounts and snapshot_accounts tables), and the [snapshot] section. Enable the [supply] section on the indexer and the [supply] section on the API. The two must be set consistently; a node with the API section absent or disabled answers getSupply with method-not-found. Run it on a dedicated supply-only instance: de-partitioning removes the owner pruning that getProgramAccounts relies on.

The indexer seeds the total supply from the snapshot bank's capitalization on every (re)start. The previous balance for a block's delta comes from two in-memory sources, and each pubkey belongs to exactly one of them. Stake accounts belong to the non-circulating stake map, which holds every stake account and reports the summed stake delta per block, so the once-per-epoch reward burst is all memory reads. Every other account belongs to a bounded hot-accounts cache keyed by pubkey, holding the last lamports and the slot of the last write: a cache hit subtracts the stored lamports from the new balance in memory, and a miss falls to one batched by-pubkey read against the de-partitioned tables, with no owner routing, which a live block runs alongside its own row inserts. The cache is bounded: entries are capped (hot-accounts, default 1,000,000) and swept by a slot-window off the block path. The cache is about 136 MB and the stake map about 153 MB (both estimated from the record sizes), against the ~185 GiB of the owner-map delta. While the snapshot is still ingesting, accounts touched by live blocks are recorded in a separate unbounded touched-account set; once the snapshot pass completes, those touches are reconciled against the startup-slot balances read back by pubkey from snapshot_accounts and the tracker goes live. One supply row carrying the total and the running non-circulating lamports is persisted per confirmed block (pruned to the last 128 slots).

The non-circulating set is maintained in real time by the stake map. It is Agave's pinned non-circulating account list plus every stake account whose lockup is in force or whose withdrawer is a listed authority, evaluated against the Clock sysvar carried by each block. Members whose lockup can expire sit in two ordered sets keyed by their expiry epoch and expiry timestamp, and the head of each set is popped as the clock passes it, so a lockup expiry moves the account out of the set in the block whose clock reached it. The member list is persisted on the bootstrap flip and again on every block that changed it. Membership is evaluated against the confirmed block's clock while Agave evaluates against the bank clock at the requested commitment, so an account whose lockup expires inside the confirmed-to-finalized window is classified differently by the two for up to about 32 slots.

The API serves from a polled cache. Requests return a node-unhealthy error until the bootstrap pass has completed and the non-circulating set has been computed. On a detected slot gap the tracker keeps applying block deltas but stops publishing totals until every gap slot has been repaired (requests fail as node-unhealthy in the meantime). A failed account write while live pins the block's touched accounts so the cache stays authoritative and the total stays exact; if the failure-pin budget is exceeded the tracker marks stale and stays fail-closed until the indexer restarts. For finalized commitment the response picks the newest persisted total at or below the finalized slot; confirmed returns the newest total. context.slot is the slot of the served row; the non-circulating account list may lag it by one block.

The response matches Agave's getSupply exactly: the RpcSupply shape (total, circulating, nonCirculating, nonCirculatingAccounts), total is the bank capitalization, nonCirculating is the summed pinned-list-plus-locked-stake lamports, circulating is total - nonCirculating, and excludeNonCirculatingAccountsList = true returns the same numbers with an empty account list.

Largest Accounts (getLargestAccounts, getTokenLargestAccounts)

These methods are served from maintained top-N holder sets rather than live table scans, so they stay cheap at any index size and work even where upstream RPC nodes disable them. Each method is enabled independently by its own config section: [largest-accounts] for getLargestAccounts, and [token-largest-accounts] for getTokenLargestAccounts. The section must be present with enabled = true on both the indexer and the API config of a co-deployed node. An absent section, or enabled = false, turns the method off. There is no shared enablement state in the database. A request for a method that is not enabled on the API gets a JSON-RPC -32601 Method not found.

The two methods have different prerequisites, which is why they are enabled separately. getLargestAccounts needs a full unfiltered index and the [snapshot] section. getTokenLargestAccounts needs the token programs (Tokenkeg / Token-2022) indexed, because the mint-metadata lookup and the reservoir read token accounts. That means an unfiltered [programs], or an include-list containing both. It also needs the [snapshot] section. It does not need the owner map, the non-circulating tracker, or the GLA sentinels, so a token-filtered node can serve token tops alone.

When enabled, the indexer keeps an in-memory top-N per tracked entity: the native-SOL top and the circulating/non-circulating class tops under [largest-accounts], and one token top per mint listed in tracked-mints. tracked-mints is the set of mints that get served, so pick the ones whose getTokenLargestAccounts traffic matters. When a top-N changes, its top 20 rows are persisted as a single packed record per (mint, slot). Reads pick the newest record at or below the requested commitment's slot. Old generations are pruned every prune-interval-slots finalized slots, off the finalize critical path.

The API routes getTokenLargestAccounts purely on record presence. The indexer persists a record for exactly the mints it tracks, so a persisted record is served and an absent record returns a fast "mint not tracked" error. This keeps the API in lockstep with the indexer without any mint-list handshake, and there is no fallback table scan. During the indexer's bootstrap window a tracked mint has no record yet, so it gets the same "mint not tracked" error until its first record persists. Clients cannot distinguish "warming up" from "never tracked" for getTokenLargestAccounts. getLargestAccounts with no record yet (the indexer is still bootstrapping) returns a retriable "not available yet" error instead of an internal error.

The tracker is seeded from the startup snapshot pass and then advanced per confirmed block. It keeps accounts-per-mint holders in memory (default 100) so that evictions from the top 20 can be backfilled without rescanning. If churn ever exhausts that reservoir, the affected mint is marked stale and its persisted rows are dropped rather than serving a possibly-wrong answer. The non-circulating membership set behind the filter option comes from the same real-time stake map that serves getSupply (see Supply): each block's accounts are classified by the membership after that block, and a member whose lockup expired without a write is moved to the circulating top in the block whose clock passed the expiry.

Config keys: [largest-accounts] (indexer) takes enabled, accounts-per-mint (default 100), and prune-interval-slots (default 20). [token-largest-accounts] (indexer) takes enabled, tracked-mints (an array of mint addresses), accounts-per-mint, and prune-interval-slots. The API sections carry only enabled. See example.cloudbreak.index.toml and example.cloudbreak.api.toml.

Snapshot on Indexer Startup

When the [snapshot] section (with [snapshot.tracker_endpoint]) is present in the indexer config, the indexer queries the cluster tracker, downloads the latest covering snapshot pair from the source the tracker reports, and processes the archives before beginning gRPC streaming. This provides fast bootstrapping of account state. Omit the entire [snapshot] section to skip this.

Account Owner Map

Set accounts-owner-map-enabled = true in the indexer config to maintain an in-memory map of account pubkey to owner + slot. This is used for tracking owner changes and closed-account handling. It increases memory usage but improves correctness for owner-change scenarios.

The map also speeds up the finalize-slot cleanup. The accounts and snapshot_accounts tables are hash-partitioned by owner, so a cleanup DELETE that only knows the pubkey has to probe all 64 partitions. When the map is enabled, the indexer captures each account's owner at save time and the cleanup carries (pubkey, owner) pairs, letting Postgres prune each delete to a single partition. For an account whose owner changed, the old-owner pair recorded at the change routes the deletion of its old rows and closed-account mask to the old owner's partition. When the map is disabled the cleanup runs the no-owner SQL, with no owner routing.

Slot Synchronizer

The API server's [slot-syncronizer] section keeps the latest slots in memory from Postgres LISTEN/NOTIFY on cloudbreak_slots. Enabled by default. Disable with enabled = false if not needed.

Query Tracker Integration

The API can operate with or without the query tracker. To disable it, remove the [query-tracker-client] section from the API config. Without it, no query patterns are tracked and automatic index creation is unavailable.

GPA Cache

Set [gpa-cache] in the API config to enable an in-memory cache for getProgramAccounts responses. The cache stores pre-serialized JSON bytes (ref-counted, zero-copy on hits) and changes the SQL to skip re-sending the body of already-cached accounts. See [gpa-cache] (optional) for the config keys and a benefits breakdown. Omit the section to disable.

Mint Cache

jsonParsed token responses need each account's mint decimals and the mint's interest-bearing and scaled-UI configs. The mint cache keeps that data in memory. getTokenAccountsByOwner, getTokenAccountsByDelegate, getTokenAccountsByMint and token-program getProgramAccounts then read it without a Postgres lookup. Enable [mint-cache] on the indexer and on the API.

Indexer

• The indexer keeps every mint in memory, immutable ones included (without them, every mint or burn of one would look like a new mint and rewrite its row). While the startup snapshot loads, each snapshot mint and close goes into the in-memory copy, and every 50,000 changed keys are written to mints in the background. Confirmed blocks that arrive meanwhile go into the same copy. Per mint, the higher slot wins, so files and blocks can arrive in any order.
• A close can arrive before the mint it closes (in a snapshot file or block read earlier). Until the bootstrap ends, such a close waits in closed_pending and is applied at the end.
• When the snapshot is loaded, the indexer writes every change not yet in mints, then environment_info.mints_saved_slot (last_db_slot): the slot the table is complete through. The node turns healthy only after this.
• For each confirmed block, the indexer compares every mint write and every zero-lamport write with its copy, and keeps the real changes for the next write: a new mint, a decimals or config change, a close, a re-init. A supply-only change is skipped.
• Every db-write-interval-slots slots, a background task writes the changed mints, then last_db_slot. last_db_slot never passes a slot that self-healing still has to repair: an API that loads or reloads during an indexer gap then reads the repaired rows once the gap is filled. One write runs at a time. A failed write is retried with the next block. An upsert never overwrites a row with a newer slot.

The mints table

One row per mint: pubkey, slot (the slot of the last change), decimals, mutable, closed, interest_bearing_config, scaled_ui_amount_config, and an index on slot.

A Tokenkeg mint, or a Token-2022 mint without MintCloseAuthority, InterestBearingConfig and ScaledUiAmount, is immutable: nothing jsonParsed reads can change. The cache keeps only its decimals, with no slot. Every other mint is mutable and keeps the slot of its last change.

Tombstones

A close is stored as a row with closed = true and the close slot. The row is never deleted, for four reasons:

1. The API reload WHERE slot > applied_through must see a close the API missed. A deleted row returns nothing. The API would then keep the closed mint and return decimals where Agave drops the account.
2. At API startup, a close the API applied from its buffer must beat an older live row of the table. The tombstone keeps its slot, so the higher slot wins.
3. A re-init at the same address updates the tombstone row with a higher slot. It needs no special case.
4. A loaded tombstone answers "closed" from memory, without a Postgres lookup.

Tombstones are not pruned. If pruning is added later, the API must know the pruning horizon and do a full reload when its applied_through is below it.

API

• The [grpc] feed buffers the mint changes of every processed block, by slot and blockhash.
• On each confirmed slots notification, the slot syncronizer first applies the buffered changes of that block and its unapplied ancestors, then publishes the slot. The cache is current through every slot a request can pick.
• When a block is missing from the buffer, the slot syncronizer asks a background task for a reload and goes on. The task reads last_db_slot every 10 s. Once it covers the slot, the task reads WHERE slot > applied_through from mints, and the cache is current through last_db_slot. Until then, requests past applied_through read mutable mints from Postgres.
• At startup, the background task waits until last_db_slot is set (the indexer bootstrap wrote the table), reads it, and only then loads the whole table in pages of 100,000 rows (each page one short query, so api-queries-timeout applies). The loaded rows hold at least every change through that slot, and the buffer holds the blocks after it. Until the load ends, every lookup reads Postgres.
• When the indexer turns healthy again after a gap repair, the API reloads the rows changed since it turned unhealthy, in the same background task, once last_db_slot covers the confirmed slot.

Lookups

Each request picks its slot once and uses it for its accounts and its mints.

Cache entry	Answer
Immutable	Memory.
Mutable or tombstone, slot at or below request slot	Memory. A tombstone answers "closed": the account gets no decimals, as in Agave.
Mutable or tombstone, slot above the request slot	Postgres at the request slot. The mint changed after the request picked its slot.
Missing	Postgres, with a warning. While live this means a bug or an indexer gap.
Cache behind the request slot, or still loading	Postgres for mutable and missing mints.

OpenTelemetry Tracing

Add a [tracing] section to any service config to enable OTLP trace export. When omitted, only standard tracing-subscriber logging is used.

On the API server the json_encoding span additionally carries request_id, subscription_id and client_ip, read from the headers configured in [metrics] (request-id-key, subscription-id-key, client-ip-key). request_id is there to join a span to the corresponding upstream request log entry.

HTTP Endpoints

Each service exposes HTTP endpoints on its metrics port (or server port for the API and query tracker). Below is a breakdown per service.

API Server (default :4000)

Endpoint	Method	Description
/	POST	JSON-RPC endpoint for all RPC methods (getProgramAccounts, getTokenAccountsByOwner, etc.). Supports batch requests.
/metrics	GET	Prometheus metrics.
/debug/log_filter	GET	Returns the current tracing log filter.
/debug/log_filter?filter=<value>	GET	Dynamically updates the log filter at runtime without restarting the service. Example: curl "http://localhost:4000/debug/log_filter?filter=debug,hyper=warn"
/debug/modules/gpa_cache	GET	Inspects the live GPA cache. Supports ?detail=summary|queries|full, ?program=<pubkey>, ?with_pubkeys=true, ?min_size_bytes, ?max_size_bytes, ?limit. Returns {"enabled": false} when [gpa-cache] is not configured. See the rustdoc on gpa_cache_handler in crates/api/src/http/operational_endpoints.rs for the full query-parameter and response-body reference.
/debug/modules/mint_cache	GET	Inspects the API mint cache: status, applied_through, unhealthy_since, the reload waiting for last_db_slot (pending_reload), entry counts and the buffer. Load time and reload counts are in the cloudbreak_mint_cache_* metrics. ?pubkey=<base58> adds one mint's map (immutable, mutable, tombstone, missing), slot (null for an immutable entry), closed state and decimals. Returns {"enabled": false} without [mint-cache]. See the rustdoc on mint_cache_handler in crates/api/src/http/operational_endpoints.rs.

Indexer (default metrics port :8875)

Endpoint	Method	Description
/metrics	GET	Prometheus metrics (block processing, chunk processing, slot finalization, gRPC stats, etc.).
/debug/modules/finalizer	GET	Inspects the live slot finalizer (confirmed-but-not-finalized blocks map, ordered pending queue, pause state). Returns JSON with the DB chain tips (confirmed/finalized + lag). Supports ?detail=summary|slots|full, ?kind=all|live|repaired, ?min_slot, ?max_slot, ?limit, ?with_pubkeys=true. See the rustdoc on handle in crates/index/src/operational_endpoints/finalizer.rs for the full reference.
/debug/modules/self_healing	GET	Inspects self-healing gap state. Returns JSON with the DB chain tips and the still-missing slots grouped into gaps (each with boundary_slot, start/end, len, and slots_behind_confirmed). Supports ?detail=summary|slots|full (summary returns only the stats counts, omitting the gap list), plus ?min_slot, ?max_slot, ?limit to filter the gap view. See the rustdoc on handle in crates/index/src/operational_endpoints/self_healing.rs.
/debug/modules/supply	GET	Inspects the live supply tracker: status, total, slot, startup slot, gap-close count, and the hot-accounts cache counters (hot, pinned, capacity, cap, last sweep slot). Supports ?detail=full&pubkey=<base58> for one cache entry. Returns {"enabled": false} when [supply] is not configured. See the rustdoc on handle in crates/index/src/operational_endpoints/supply.rs.
/debug/modules/mint_cache	GET	Inspects the indexer half of the mint cache: last_db_slot (null until the bootstrap wrote mints), last_slot, entry counts, closes waiting for their mint (closed_pending, null after the bootstrap), keys waiting for the next write (pending_to_db), whether a write runs, and the last block's changes by kind. The bootstrap write time is the cloudbreak_mint_cache_startup_seconds{side="indexer"} metric. ?pubkey=<base58> adds one mint's map, slot, closed state and decimals. Returns {"enabled": false} without [mint-cache]. See the rustdoc on handle in crates/index/src/operational_endpoints/mint_cache.rs.
/debug/modules/non_circulating	GET	Inspects the live non-circulating tracker: status, slot, clock, member count, running non-circulating lamports, stake map size, the sizes and heads of the two lockup-expiry sets (?limit= heads each, default 10). Supports ?detail=full&pubkey=<base58> for one stake map entry. Returns {"enabled": false} when neither [largest-accounts] nor [supply] is configured. See the rustdoc on handle in crates/index/src/operational_endpoints/non_circulating.rs.
/debug/accounts_owner_map	GET	Returns debug info about the in-memory account-to-owner map. Only populated when accounts-owner-map-enabled = true in the indexer config.

Query Tracker (default :4001)

All endpoints — functional and operational — are served on the single server port.

Endpoint	Method	Description
/track	POST	Ingest a batch of observed getProgramAccounts-shaped demand (body is a TrackBatch).
/debug/candidates	GET	The ranked creation queue, each candidate annotated with its priority score and the with/without-index latency behind it.
/debug/created	GET	Currently active auto-created indexes with demand (API) vs. supply (compensated idx_scan), size, latency gain and explain_state.
/debug/discrepancies	GET	Created indexes where demand and supply disagree (e.g. planner ignoring an index).
/metrics	GET	Prometheus metrics.
/health	GET	Returns 200 OK if the service is running.

The three /debug/* endpoints share query-string conventions. Latencies are rendered in ms (2 dp) (*_ms), index size in MiB (2 dp) (index_mb), and supply as compensated_idx_scan (raw idx_scan ÷ 2, since a served GPA scans both tables of the pair). Each row's score (the shared priority-mode ranking) is shown in thousands, 1 dp (e.g. "37.2K") — the raw score derives from microsecond costs, so ÷ 1000 makes it proportional to the *_ms figures. Under the weighted mode the field also carries, in parentheses, the three factors whose product is the score — avg (mean cost/request in ms) × gain (latency multiplier, 1 = neutral) × counts (windowed demand/supply/failure volume multiplier, in K), e.g. "37.2K (avg 12.34ms × gain 1.05 × counts 13.0K)". Every response repeats these notes in a top-level docs field. Every response is { "total", "count", "limit", "docs", "<items>": [...] }. Common parameters:

• limit=N — cap returned rows (default: all); order=<key> + dir=asc|desc — sort.
• example=true / pattern_id=true (or verbose=true for both) — include the heavier fields, omitted by default.
• Order keys: /candidates → score (default), demand_count, avg_cost_ms, variety_estimate; /created → created_at (default, newest first), index_mb, demand_count, idx_scan, avg_cost_with_index_ms, with_without_idx_ratio, variety_estimate; /discrepancies → discrepancy_ratio (default, dir=asc = most-starved first) with min_ratio / max_ratio bounds.
• /created shows each index's score — the same shared priority-mode ranking value as /debug/candidates (creation builds highest, eviction drops lowest) — plus a human-readable created_at, and can be ordered by score in any view. It also accepts min_ratio / max_ratio bounds on with_without_idx_ratio (rows without both latency buckets are dropped when a bound is set); since > 1 means the index helps, max_ratio=1 surfaces indexes that are not helping. filter=eviction_candidates shows only the indexes the eviction pass would consider (past the idle + age-grace gates), ordered least-useful-first by default (order=score, ascending). The real drop still depends on the table being above the fill target at runtime — eviction only trims the buffer band back to the target.
• /created also accepts filter=explain_incomplete (planner uses the index on neither or only one table), filter=explain_none (neither table), and filter=explain_partial (exactly one table) — the detailed per-index source of truth behind the query_tracker_explain_state gauge and the EXPLAIN summary log. Rows with no verdict yet (explain-enabled off) are excluded.
• Unknown keys or malformed values return 400. Example: GET /debug/created?order=index_mb&limit=10&verbose=true.

Metrics Reference

All metrics are emitted in the Prometheus text exposition format on each service's /metrics endpoint (see HTTP Endpoints). Metric names are stable and intended to be referenced from Grafana panels and alert rules. Labels are described per-metric below; when label values are enumerated, they are the complete set currently emitted by the code.

API Server (cloudbreak-api)

Metric	Type	Labels	Description
cloudbreak_api_requests_total	Counter	method, status	Count of RPC method invocations grouped by outcome. method ∈ {gPA, gTABO, gTABD, gTABM, gAI, getBalance, getMultipleAccounts, getTokenAccountBalance, http}: gPA = getProgramAccounts, gTABO = getTokenAccountsByOwner, gTABD = getTokenAccountsByDelegate, gTABM = getTokenAccountsByMint, gAI = getAccountInfo, http is connection-level. status ∈ {success, error, timeout}: error is incremented on RPC-level failures (bad params, DB failure, stream-mid-error); timeout is incremented on http when the total request-timeout fires. The point-lookup methods (gAI, getBalance, getMultipleAccounts, getTokenAccountBalance) emit both success and error. The streaming methods (gPA, gTABO, gTABD, gTABM) emit error only — for their total throughput use cloudbreak_api_request_duration_ms instead. Note: getSlot, getHealth, getVersion, and getGenesisHash are not currently surfaced under this counter.
cloudbreak_api_request_duration_ms	Histogram	method, bytes	Per-stage request latency in milliseconds. bytes is the response-size bucket (0-1KB, 1-10KB, 10-100KB, 100KB-1MB, 1MB-10MB, 10MB-50MB, 50MB-100MB, 100MB-200MB, 200MB-500MB, 500MB+). method values: gpa (total in-handler time for getProgramAccounts; on a token program it carries the indexed key it filters on: gpa_mint, gpa_delegate or gpa_token), gpa_db (Postgres query time), gpa_db_first_row_time (time-to-first-row), gpa_encode (account-encoding time), gpa_json (JSON serialization time); analogous gtabo* / gtabd* / gtabm* for the token-account methods; gAI / getBalance / getMultipleAccounts / getTokenAccountBalance / getTokenSupply (single observation per request — total handler + serialization time for the point-lookup methods, in fractional milliseconds); http_with_transport (end-to-end including body transport, label bytes reflects response size); http_connection (per-TCP-connection lifetime, label bytes="0").
cloudbreak_api_requests_by_subscription_id	Counter	subscription_id_key	Per-client request counter, attributed via the HTTP header configured by [metrics].subscription-id-key (default x-subscription-id). Only the four heavy methods report here: getProgramAccounts, getTokenAccountsByMint, getTokenAccountsByOwner, getTokenAccountsByDelegate — the point-lookup methods (getAccountInfo, getBalance, …) are not counted. Requests with the header absent fold into unknown-subscription-id. Batch requests count once per batch entry.
cloudbreak_api_data_fetched_by_subscription_id	Counter	subscription_id_key	Per-client cumulative response size in bytes (JSON-encoded payload). Same method scope as cloudbreak_api_requests_by_subscription_id.
cloudbreak_api_duration_us_by_subscription_id	Counter	subscription_id_key	Per-client cumulative request handling time in microseconds — divide by 1000 for milliseconds. Measures handler entry through the last encoded JSON chunk (the same duration the json_encoding span reports as total_wall_time), excluding HTTP body transport. Same method scope as cloudbreak_api_requests_by_subscription_id, so dividing the two yields a mean latency over those methods only.
cloudbreak_api_inflight_requests	Gauge	method	Currently in-flight requests per stage. Emitted values: http_connection (live TCP connections), http (active HTTP requests), gpa, gtabo, gtabd, gai, gma, getBalance, getTokenAccountBalance. The special label max is set once at startup to [server].max-connections; plot against the live gauges to visualize saturation.
cloudbreak_api_batch_requests_total	Counter	batch_size	Counter incremented once per batch JSON-RPC request, bucketed by batch size: 1-5, 6-10, 11-20, 21-50, 51-100, 100+.
cloudbreak_api_cache_hit_percent	Histogram	method, bytes	Per-request GPA cache hit percentage (0-100), bucketed for both ends so fully-cached (100) and fully-fresh (0) requests stand out. method matches the getProgramAccounts/token-account method labels used by cloudbreak_api_request_duration_ms and bytes is the same response-size bucket. Reports 0 when the cache is inactive or the response has no accounts.
cloudbreak_api_response_bytes_total	Counter	method, kind	Response bytes served. kind ∈ {total (all response bytes), cached (subset served from the GPA cache; 0 on paths that don't use the cache)}. Effective cache utilization is kind="cached" / kind="total" in Grafana. method matches the GPA/token-account method labels.
cloudbreak_api_db_pool_connections	Gauge	origin	SQLx database connection-pool state, sampled at scrape time. origin ∈ {max (configured [database].max-connections), idle (connections idle and immediately available), total (established connections, i.e. idle + in-use)}. In-use is total - idle; headroom before queries start waiting is max - (total - idle), both computable in Grafana.
cloudbreak_gpa_cache_size_bytes	IntGauge	—	Current size of the GPA cache in bytes. Pair with cloudbreak_gpa_cache_max_bytes to compute utilization (size / max).
cloudbreak_gpa_cache_max_bytes	IntGauge	—	Configured maximum size of the GPA cache in bytes ([gpa-cache].max-total-bytes).
cloudbreak_gpa_cache_evictions_total	Counter	used	GPA cache entries evicted by cleanup to make room for a different query. used ∈ {used, unused} indicating whether the evicted entry had ever served a cache hit. A high rate of unused evictions indicates cache churn (e.g. min-bytes-per-query set too low).
cloudbreak_gpa_cache_evicted_bytes_total	Counter	used	Total bytes evicted from the GPA cache by cleanup, with the same used labelling as cloudbreak_gpa_cache_evictions_total.
cloudbreak_gpa_cache_skipped_total	Counter	method, bytes, reason	Responses large enough to cache (min-bytes-per-query) that were not cached. method is the request label (gpa, gpa_mint, gpa_delegate, gpa_token, gtabo, gtabd, gtabm), bytes the response-size bucket. reason is mutable_mint (a jsonParsed token account uses a mint with close authority, interest-bearing or scaled-UI config, or a missing mint) or dropped_account (a jsonParsed token account did not parse and was left out).
cloudbreak_api_processed_requests_total	Counter	method, route, reason	Processed commitment requests for the methods [processed-accounts] serves. route is view when the processed blocks answer, degraded when the request reads as confirmed. reason is none, no_blocks, unhealthy, head_behind or finalized_above_anchor. Registered only when [processed-accounts] is enabled.
cloudbreak_api_processed_confirm_latency_ms	Histogram	—	Time from receiving a processed block to the Postgres confirmed slot reaching it, in milliseconds. Registered only when [processed-accounts] is enabled.
cloudbreak_mint_cache_entries	IntGaugeVec	side, map	Mint cache entries: immutable, mutable, tombstone. side is api here and indexer on the indexer. Registered only with [mint-cache].
cloudbreak_mint_cache_changes_per_block	HistogramVec	side, kind	Mint changes applied per confirmed block: create, update, close, reinit.
cloudbreak_mint_cache_block_microseconds	HistogramVec	side, stage	Mint cache time in microseconds. API stages: ingest (feed thread), apply (confirmed notification). Reloads run in a background task.
cloudbreak_mint_cache_startup_seconds	GaugeVec	side	API: duration of the full mints load at startup.
cloudbreak_mint_cache_lookups_total	IntCounterVec	source, reason	Mint lookups. source="memory" with reason hit or closed; source="postgres" with loading, newer_than_request, behind or missing. missing while live also logs a warning.
cloudbreak_mint_cache_reloads_total	IntCounterVec	reason, result	Reloads of WHERE slot > applied_through, run by a background task once last_db_slot covers the slot. reason: block_missing (the feed lagged or had a gap), health_restored (after an indexer gap). result: ok, error.
cloudbreak_mint_cache_buffer_blocks	IntGauge	—	Blocks waiting in the mint buffer for their confirmed notification.
cloudbreak_mint_cache_applied_through	IntGauge	—	Last confirmed slot applied to the mint cache.
cloudbreak_grpc_errors, cloudbreak_grpc_timeout_errors, cloudbreak_grpc_total_updates_received, cloudbreak_grpc_buffer_channel_size_sender	Counter / IntGauge	—	The [grpc] feed, with the same meaning as the indexer metrics of the same name. Registered only when [grpc] is enabled.

Indexer (cloudbreak-index)

Metric	Type	Labels	Description
cloudbreak_block_processing	Histogram	origin	Total time in seconds to process a full gRPC block (parse + per-chunk insert). origin = block. Buckets: 10 ms – 10 s.
cloudbreak_chunk_processing	Histogram	origin	Per-chunk DB insert latency in seconds. origin = block. Buckets: 10 ms – 10 s.
cloudbreak_block_size	Histogram	origin	Block size in bytes. origin = block. Buckets: 100 KB – 30 MB.
cloudbreak_chunk_size	Histogram	origin	Chunk size in bytes. origin = chunk. Buckets: 50 KB – 2 MB.
cloudbreak_finalize_slot	Histogram	origin	Finalize-slot stage latency in seconds. origin values: total (one drain of a cleanup slot window, every chunk on both tables), cleanup_accounts, cleanup_snapshot_accounts (every chunk of one drain on that table, wall clock), cleanup_accounts_batch, cleanup_snapshot_accounts_batch (one drainer statement of at most cleanup-batch-size keys), cleanup_startup_snapshot_accounts_batch (one statement of the startup one-shot). Buckets: 5 ms – 60 s.
cloudbreak_finalize_slot_deleted_accounts	Histogram	—	Number of older account versions deleted per cleanup_accounts call. Useful to spot slots that churn an unusually large number of rows.
cloudbreak_new_accounts_in_slot	Histogram	origin	Accounts seen in a slot. origin = new_accounts_in_slot (accounts net-new to the database after finalize-time deduplication) or block_accounts_total (raw count of account updates in the block, including overwrites). The ratio gives a sense of write churn per slot.
cloudbreak_grpc_buffer_channel_size	Histogram	origin	gRPC ingestion buffer depth sampled per push. origin = grpc_buffer_channel_size. Buckets: 1 – 10 000. Rising values indicate the indexer is falling behind the stream.
cloudbreak_grpc_buffer_channel_size_sender	IntGauge	—	Latest gRPC buffer channel depth (last sample, gauge form for easy alerting).
cloudbreak_grpc_timeout_errors	Counter	—	gRPC stream timeout errors.
cloudbreak_grpc_errors	Counter	—	gRPC errors other than timeouts (transport, decoding, etc.).
cloudbreak_grpc_total_updates_received	Counter	—	Total messages received from the gRPC subscription. Use the rate as the input-side throughput.
cloudbreak_grpc_gap_errors	Counter	—	Slot gaps detected on gRPC reconnection. Each increment corresponds to a missing-slot range observed at reconnect.
cloudbreak_db_errors	Counter	—	All database errors (insert / cleanup / read) hit by the indexer. When the cumulative count exceeds database.max-db-errors-threshold (default 100), the indexer process exits.
cloudbreak_closed_accounts_per_slot	Histogram	—	Number of accounts marked closed per slot. Buckets: 1 – 10 000.
cloudbreak_insert_closed_accounts_per_slot_ms	Histogram	—	Latency in milliseconds of inserting closed-account markers for one slot. Buckets: 0.1 ms – 10 s.
cloudbreak_current_tokio_tasks	IntGauge	task_type	Current count of spawned Tokio tasks per category. task_type values: grpc, slot_finalizer, self_healing_fill_gaps, snapshot_processing, insert_accounts_chunk, insert_closed_accounts, startup_snapshot_accounts_cleanup, finalize_cleanup_drainer, metrics_server, epoch_stakes_recomputer, largest_accounts_pruner, supply_cache_sweeper. Counter is guarded by TokioTaskCounterGuard so panicking tasks still decrement.
cloudbreak_finalize_slot_handler_queue_size	IntGauge	—	Pending items in the finalize-slot handler channel. The query tracker reads this via the configured indexer-metrics endpoint to pause CREATE INDEX when the indexer is behind (indexer-metrics-threshold).
cloudbreak_cleanup_lag_slots	IntGauge	—	Finalized slots since the oldest cleanup key still waiting. Cleanup runs off the finalize worker, so this is the signal that it is falling behind, and the query tracker reads it as the second DDL gate (indexer-cleanup-lag-threshold).
cloudbreak_largest_accounts_db_errors	Counter	—	Largest-accounts DB write/prune errors (record persist, stale/cleared-mint deletes, and the pruner). Only emitted when a largest-accounts section ([largest-accounts] / [token-largest-accounts]) is enabled.
cloudbreak_largest_accounts_stale_mints	IntGauge	—	Number of tracked mints currently marked stale in the largest-accounts tracker (reservoir exhausted or a persist failure dropped the record). A non-zero value means those mints have no persisted record and getTokenLargestAccounts fails fast for them. Only emitted when a largest-accounts section ([largest-accounts] / [token-largest-accounts]) is enabled.
cloudbreak_supply_total_lamports	IntGauge	—	Running total supply in lamports, updated on every committed block while the supply tracker is live. Only emitted when the [supply] section is enabled.
cloudbreak_supply_slot	IntGauge	—	Slot of the last committed supply total.
cloudbreak_supply_status	IntGauge	—	Supply tracker status: 0 bootstrapping, 1 live, 2 gap filling, 3 stale, 4 bootstrap failed. Set at every transition.
cloudbreak_supply_cache_entries	IntGaugeVec	kind	Hot-accounts cache population, labelled pinned (by a live write failure) and hot (unpinned).
cloudbreak_supply_cache_hits_total	Counter	—	Per-block cache hits (previous balance served from memory).
cloudbreak_supply_cache_misses_total	Counter	—	Per-block cache misses (previous balance resolved by a DB read).
cloudbreak_supply_write_failures_total	Counter	—	Blocks whose account writes failed while live (pinned on failure, or marked stale beyond the pin budget).
cloudbreak_supply_block_microseconds	Histogram	—	Per-block supply delta time in microseconds, from apply_block entry (block-writes lock wait included) to the last miss write-back.
cloudbreak_supply_probe_microseconds	Histogram	—	Per-block cache probe time in microseconds (phase one, under the state mutex).
cloudbreak_supply_db_microseconds	HistogramVec	query	Supply DB query time in microseconds: prev_balances (the batched miss read), persist_row (the supply row upsert), startup_balances (the bootstrap resolve read).
cloudbreak_supply_sweep_microseconds	Histogram	—	One hot-accounts cache sweep pass in microseconds.
cloudbreak_supply_query_errors	Counter	—	Supply persistence queries that failed or timed out (the per-block supply-row upsert and the non-circulating list upsert).
cloudbreak_non_circulating_block_microseconds	Histogram	—	Per-block non-circulating time in microseconds: the stake map apply plus the lockup-expiry pop. Only emitted when [largest-accounts] or [supply] is enabled.
cloudbreak_non_circulating_members	IntGauge	—	Current non-circulating member count, set on the bootstrap flip and on every block that changed membership.
cloudbreak_non_circulating_changes_total	IntCounterVec	kind	Non-circulating membership changes, labelled join and leave.
cloudbreak_mint_cache_entries	IntGaugeVec	side, map	Entries of the indexer's mint store (side="indexer"): immutable, mutable, tombstone. Only with [mint-cache].
cloudbreak_mint_cache_changes_per_block	HistogramVec	side, kind	Mint changes per confirmed block, kept for the next write to mints: create, update, close, reinit.
cloudbreak_mint_cache_block_microseconds	HistogramVec	side, stage	Mint cache indexer time in microseconds: classify and apply per block, write per write to mints.
cloudbreak_mint_cache_startup_seconds	GaugeVec	side	Indexer: duration of the final mints write of the bootstrap.
cloudbreak_mint_cache_unchanged_writes_total	IntCounter	—	Mint writes that changed nothing jsonParsed reads (supply only).
cloudbreak_mint_cache_db_write_errors_total	IntCounter	—	Failed writes to mints. The rows are retried with the next write.

Snapshot (cloudbreak-snapshot)

Snapshot metrics are emitted whenever snapshot processing runs — either embedded in the indexer at startup (when [snapshot] is configured) or via the standalone snapshot subcommand. They piggyback on the metrics endpoint of the calling service.

Metric	Type	Labels	Description
cloudbreak_db_snapshot_errors	Counter	—	Database error counter for snapshot-time inserts and dedup queries.
cloudbreak_snapshot_batch_insert_time	Histogram	—	Latency in seconds of a single batched AccountsFile insert. Uses Prometheus default buckets.
cloudbreak_processed_snapshot_items	Gauge	type	Snapshot ingestion progress. type ∈ {accounts_files_total (count of completed AccountsFile entries — incremented as a counter despite the gauge type), accounts_files_percentage (0 – 100), accounts_total (cumulative account count written from the snapshot)}.
cloudbreak_snapshot_accounts_buffer_size	Gauge	type	Snapshot-time intermediate buffer depths. type = cleanup_duplicated (size of the dedup-stage receive channel).
cloudbreak_snapshot_clean_up_duplicated_accounts_batch_time	Histogram	—	Latency in seconds of de-duplicating a batch of overlapping snapshot accounts against the live accounts table. Uses Prometheus default buckets.
cloudbreak_snapshot_clean_up_duplicated_accounts_batch_rows_affected	Counter	type	Registered but not currently observed in code; reserved for surfacing the row count touched by each dedup batch.

Query Tracker (cloudbreak-query-tracker)

The query tracker exposes its metrics on /metrics (same port as /track and /debug/*). Aggregate gauges and the lifecycle counters are refreshed by the eviction pass; the per-index gauges are labelled with the human-readable index name so a dashboard can show demand (API) against supply (idx_scan) per active index. Use /health for liveness probes.

Metric	Type	Labels	Description
query_tracker_snapshot_accounts_indexes_total	IntGauge	—	Current number of indexes present on the capped snapshot_accounts table (the value the max-auto-indexes cap and eviction-fill-threshold are measured against).
query_tracker_patterns	IntGaugeVec	status	Number of distinct index patterns tracked, partitioned by lifecycle status ∈ {created, candidate, evicted, rejected}. The total across all statuses is sum(query_tracker_patterns) (no total label, to avoid double-counting on aggregation).
query_tracker_discrepant_indexes	IntGauge	—	Number of created indexes where demand (API) and supply (idx_scan) currently diverge (see discrepancy-enabled).
query_tracker_index_created_total	IntCounter	—	Cumulative number of auto-index pairs created.
query_tracker_index_evicted_total	IntCounter	—	Cumulative number of auto-index pairs evicted (idle eviction and regression drops).
query_tracker_index_create_failures_total	IntCounter	—	Cumulative number of failed auto-index creation attempts.
query_tracker_regressed_indexes	IntGauge	—	Created indexes currently slower with the index than without it (see index-regression-guard). Refreshed each eviction pass; 0 when the guard is off. Near zero in evict mode (drops move them to status="rejected"); reflects tolerated regressions in warn mode.
query_tracker_observations_total	IntCounter	—	Cumulative number of accepted demand observations ingested via /track.
query_tracker_index_demand	IntGaugeVec	index	Cumulative API demand (request count) per created index.
query_tracker_index_compensated_idx_scan	IntGaugeVec	index	Compensated Postgres idx_scan (supply, raw idx_scan ÷ 2 since a served GPA scans both tables of the pair) per created index. Lines up ~1:1 with query_tracker_index_demand for a healthy index; the raw counter is never exported.
query_tracker_index_variety	IntGaugeVec	index	Estimated distinct filter values served per created index (HyperLogLog variety estimate).
query_tracker_explain_state	IntGaugeVec	state	Result of the last EXPLAIN sampling pass (see explain-enabled): created indexes in each planner-usage verdict state ∈ {none, accounts_table, snapshot_accounts_table, both}. A snapshot of the latest run, not cumulative; stays at zero when explain-enabled is off. Alert on none.

dbtools CLI

The dbtools binary is a standalone operator CLI for running analytics and configuration commands against one or more Postgres servers.

Config file (cloudbreak.dbtools.toml):

[[servers]]
name = "production"
database_url = "postgres://user:pass@host/cloudbreak"

[[servers]]
name = "staging"
database_url = "postgres://user:pass@host2/cloudbreak"

Analytics Commands

cargo run -p cloudbreak-dbtools -- analytics <command>

Command	Description
get-biggest-programs	Top program owners by account count across partitions.
table-size	Database and table sizes (heap, indexes, total) for accounts and snapshot_accounts.
indexes-sizes	Aggregated index sizes across the partition tree.
partition-sizes	Recursive partition hierarchy with sizes.
distinct-owners-count	Per-partition COUNT(DISTINCT owner) with totals.
mint-accounts-count	SPL Token and Token-2022 mint account counts.
indexes-count	Total number of indexes on accounts and snapshot_accounts.
slow-queries	Currently running queries exceeding 1 minute (from pg_stat_activity).
get-delegates	Sample delegated token accounts; outputs benchmark request bodies.
accounts-count	Total row counts for accounts and snapshot_accounts.

See crates/dbtools/README.md for more details.

Integration Tests

The integration_tests crate is a standalone binary for benchmarking and correctness testing of any Solana RPC endpoint (not just Cloudbreak). It supports constant-rate load generation, dual-endpoint comparison with slot compensation, multiple request sources (static files, VictoriaLogs, mismatch replays), and optional database forensics.

Quick Start

# Copy and edit the example config
cp example.cloudbreak.integration_tests.toml cloudbreak.integration_tests.toml

# Run a benchmark
cargo run --bin integration_tests -- benchmark gpa
cargo run --bin integration_tests -- benchmark gtabo
cargo run --bin integration_tests -- benchmark gtabd
cargo run --bin integration_tests -- benchmark simulate-transaction

Commands

Command	Description
benchmark <type>	Main command. Load test with optional dual-endpoint comparison. Types: gpa, gtabo, gtabd, gpa-token-owner, gpa-token-mint, simulate-transaction.
compare	(Legacy) Full pubkey set comparison between two endpoints with transaction history checks.
get-slot	(Legacy) Polls getSlot on rpc1 every 100ms.

See crates/integration_tests/README.md for the full documentation including configuration reference, comparison semantics, slot compensation, logging targets, and output format.

Benchmarks & Load Testing

Published benchmark and load-testing results live in docs/benchmarks/. All runs replay real production traffic captured from VictoriaLogs archives, measured end-to-end within the same datacenter. Current cases:

• Case 1 — low-RPS correctness + latency comparison against a reference Agave endpoint.
• Case 2 — sustained single-node load on an AMD EPYC Gen3 node (1 API task).
• Case 3 — scale-out load on an AMD EPYC Gen4 node (3 API tasks → 1 shared Postgres).

See docs/benchmarks/README.md for the full write-up, latency tables, and throughput charts.

Development

Building

cargo build
cargo build --release

Optional modules

New optional modules follow two rules. The mint cache is the reference: MintCacheConfig in crates/core/src/config.rs, MintCache in crates/core/src/modules/mint_cache/mod.rs, and its halves MintCacheApi in crates/core/src/modules/mint_cache/api.rs and MintCacheIndexer in crates/core/src/modules/mint_cache/indexer.rs.

Config: an Option section with an enabled flag

• The binary's config has a field x: Option<XConfig>, and XConfig has enabled: bool and #[serde(deny_unknown_fields)].
• An x_enabled() -> bool accessor on the binary's config gives the switch, for example ApiConfig::mint_cache_enabled().
• When the module needs other sections, XConfig checks them in check_required_modules(&self, config). The check does nothing while the module is disabled. A section that two binaries share has one check per binary, for example check_required_api_modules and check_required_indexer_modules.
• The binary runs these checks at startup, before it builds any module. A failed check stops startup with an error.

Module: a null-object handle

• The handle is pub struct X(Option<Arc<Inner>>) and derives Clone and Default. Default is the disabled handle.
• X::from_config(Option<&XConfig>) returns the disabled handle when the section is absent or enabled = false. It is the only place where config becomes the module.
• Each method starts with let Some(inner) = &self.0 else { return … }; and does nothing on the disabled handle. Callers do not check whether the module is enabled.
• is_enabled() is for the few callers that must act differently when the module is off, for example to reject a method.
• A module that runs in two binaries is one handle with a half per binary, for example MintCache with api() and indexer(). Each half is a null-object handle. Each binary builds the handle with its own constructor (MintCache::from_api_config, MintCache::from_indexer_config), which enables only its half, and calls its half by name: mint_cache.api().on_block(..).
• A handle with a single owner holds Option<Inner> without Arc and does not derive Clone. ProcessedWriter in crates/core/src/modules/processed/writer.rs is the example: it owns the processed block store without a lock, and ProcessedAccounts::writer() returns the disabled writer when [processed-accounts] is off.

Some older modules do not follow these rules yet. For example, [gpa-cache] and [query-tracker-client] have no enabled flag, and [slot-syncronizer] is not an Option.

Logging

All services use tracing with RUST_LOG environment variable control:

# Default level
RUST_LOG=info cargo run -p cloudbreak -- --config ./cloudbreak.index.toml index

# Debug with SQLx silenced
RUST_LOG=debug,sqlx=error cargo run -p cloudbreak -- --config ./cloudbreak.api.toml api

# Target-specific filtering
RUST_LOG=cloudbreak_api=debug,hyper=warn cargo run -p cloudbreak -- --config ./cloudbreak.api.toml api

Database Access

All crates use SeaORM for database access with the sqlx-postgres + runtime-tokio-rustls backend.

Observability Stack

See the infrastructure table in Getting Started for all ports and services provided by docker compose up. Prometheus scrape targets are configured in prometheus-config.yaml.

Troubleshooting

Migration shows pg_tracing warning

The accounts table migration attempts to load the pg_tracing Postgres extension. If the extension is not installed (e.g. stock Docker Postgres image), a warning is printed but the migration continues successfully. The extension is optional and only needed for detailed query plan tracing via the dbtools pg-tracing-high-detail command.

Postgres container crashes with "database files are incompatible"

This happens when the Docker volume contains data from a different Postgres major version (e.g. data initialized by PG17 but the compose file uses PG16). Fix by removing the volume and starting fresh:

docker compose down -v
docker compose up -d

Indexer shows unhealthy status

The service_health table is set to unhealthy when confirmed slot gaps are detected. Check the indexer's /debug/modules/self_healing endpoint to see which slots are missing:

curl http://localhost:8875/debug/modules/self_healing

If the [snapshot] section is configured, the self-healing mechanism will automatically attempt to fill gaps via incremental snapshots fetched through the cluster tracker. If it's not configured (or the tracker has no source producing usable snapshots), gaps cannot be repaired automatically and require manual intervention (e.g. re-running a snapshot or restarting the indexer).

getHealth returns -32005 Node is unhealthy

getHealth reads the service_health row that the indexer maintains. When the node is unhealthy it returns JSON-RPC error -32005 with message Node is unhealthy and data: {"numSlotsBehind": null}. With unhealthy-response = "http-unavailable", a single request gets HTTP 503 instead of HTTP 200. numSlotsBehind is always null, because the indexer flag does not measure a distance from the cluster tip.

The indexer keeps the node unhealthy while any of these reasons is active:

• Startup snapshot processing. With [snapshot] configured, the node stays unhealthy until the snapshot is loaded and cleaned up. This can take hours on mainnet and clears on its own. Without [snapshot], this reason clears as soon as the indexer starts processing blocks, so a no-snapshot node reports healthy on a partial dataset (see Start the Services).
• Gap filling. A confirmed slot gap pauses finalization and marks the node unhealthy until the gap is repaired. See Indexer shows unhealthy status.

While the node is unhealthy, getAccountInfo, getMultipleAccounts, getProgramAccounts, getTokenAccountsByOwner, getTokenAccountsByDelegate, getTokenAccountsByMint, getTokenAccountBalance, getTokenSupply, getTokenLargestAccounts, getLargestAccounts, and simulateTransaction return the same -32005 error. getSupply does too when the slot syncronizer is enabled. getSupply, getLargestAccounts, and getVoteAccounts also return -32005 while their caches or records are not ready, whatever the health flag says. getSlot, getBalance, getVoteAccounts, getVersion, and getGenesisHash do not check the health flag.

If [snapshot] is configured and getHealth does not clear after the snapshot finished downloading, check the indexer logs for snapshot-processing errors and the service_health row directly:

psql "$DATABASE_URL" -c 'SELECT * FROM service_health;'

Indexer falling behind (growing gRPC buffer)

If the indexer can't keep up with the gRPC stream, the buffer channel fills up. Monitor the cloudbreak_grpc_buffer_channel_size Prometheus metric. Common causes:

• Database writes are too slow — check slow-queries via dbtools or increase max-connections.
• Snapshot processing is running (expected during startup) — wait for it to finish.
• Too many programs selected — narrow the [programs].include list.

Query tracker pauses index creation

The query tracker checks the indexer's cloudbreak_finalize_slot_handler_queue_size metric before creating indexes. If the queue exceeds indexer-metrics-threshold (default 5), index creation is paused to avoid overloading the database while the indexer is catching up. This is expected behavior — index creation resumes automatically once the queue drains.

API returns "environment_info row not found"

The API loads the program filter from the environment_info table, which is populated by the indexer on startup. This error means the indexer has never run against this database. Start the indexer at least once before starting the API.

Changing indexed programs

If you change the [programs] filter in the indexer config, the environment_info table is updated on the next indexer startup. However, existing account data for previously indexed (now excluded) programs remains in the database. To clean it up, you can either:

• Run cargo run -p cloudbreak-migration -- fresh to drop and recreate all tables (destructive).
• Manually delete rows for the unwanted programs.

High memory usage with account owner map

When accounts-owner-map-enabled = true, the indexer maintains an in-memory map of every tracked account's pubkey to its owner and slot. For large program sets, this can consume significant memory. If memory is a concern and you don't need owner-change tracking, set accounts-owner-map-enabled = false.

Indexer logs No incremental snapshot available / Snapshot is not available for gap filling yet

The cluster tracker hasn't reported a snapshot pair that covers the slot the indexer asked for. Common causes:

• The targets in tracker-config.yml are unreachable, rate-limited (common with the default https://api.mainnet.solana.com), or haven't produced an incremental snapshot covering that slot yet.
• The slot you asked for is newer than any snapshot the tracker currently knows about. The indexer will retry on the next iteration; this is informational, not fatal.
• You replaced the default with custom sidecars but the tracker hasn't completed its first scrape yet (controlled by scrape_interval in tracker-config.yml).

Inspect what the tracker is exposing with:

curl http://localhost:8458/v1/snapshots | jq .

If that response is empty or missing files entries, the issue is upstream of cloudbreak — fix tracker-config.yml and reload.

solana-tracker build fails

The solana-tracker Compose service uses a Git build context (https://github.com/rpcpool/solana-cluster.git#triton). The first docker compose up needs network access to clone the repo and pull the Go toolchain. After upstream commits to the triton branch, Compose will reuse the cached image — to pick up new commits run:

docker compose build --no-cache --pull solana-tracker
docker compose up -d solana-tracker

Correctness Testing

How we validate correctness. Cloudbreak intentionally does not ship synthetic fixtures or a seed subcommand for testing query results. The chosen approach is to validate against real Solana state, because the only ground truth we trust is what a Solana validator actually sees. There are two supported, end-to-end ways to do this locally:

1. API-vs-Agave comparison (response correctness). Start the indexer with [snapshot] configured, start the API, and run the integration_tests benchmark in dual-endpoint mode against a reference Agave RPC node pointed at the same cluster. The harness compensates for slot skew between the two endpoints and reports per-request mismatches, so any divergence between Cloudbreak's responses and a stock Agave node is surfaced directly.
2. Hash-checker (database-vs-snapshot correctness). Start only the indexer (with [snapshot] configured) and enable the [hash-checker] section described below. The indexer streams gRPC until a target slot, then re-fetches a covering snapshot pair and lattice-hashes its own DB against the snapshot. A match is a cryptographic guarantee that every indexed account (address, data, lamports, owner, executable) is identical to what the validator wrote.

Both paths require a real upstream — a Yellowstone gRPC endpoint and a working cluster-tracker source — and both require [snapshot] to be configured on the indexer. Running without [snapshot] is for smoke-testing the setup only and cannot be used to assert correctness.

The indexer can run an automated correctness test by adding a [hash-checker] section to cloudbreak.index.toml. When enabled, the indexer runs normally until a target slot is reached, then asks the cluster tracker for the next covering snapshot pair, downloads the full + incremental archives from the source reported by the tracker, replays buffered updates up to that snapshot slot, computes a lattice hash over both the database and the snapshot, and exits with 0 on match or 1 on mismatch.

A lattice hash is calculated with all accounts addresses, data, lamports, owner and executable. If the two hashes match, it's sure that the indexed data is correct.

Configuration

Requires the [snapshot] section (with [snapshot.tracker_endpoint]) to be present. Set one of time-limit or slot-limit:

[hash-checker]
# Either:
time-limit = "6h"
# Or:
slot-limit = 405775018

• time-limit: target slot is computed as first_grpc_slot + time_limit / 400ms.
• slot-limit: target slot is used directly.

Running

Just run the indexer normally:

cargo run -p cloudbreak -- --config ./cloudbreak.index.toml index

The process stays up until the target slot is finalized and the next covering snapshot pair is available from the tracker, then prints the comparison result and exits.

Debugging a mismatch with snapshot-diff

When the hash-check fails, snapshot-diff shows which pubkeys differ between the indexer DB and the snapshot, instead of just a hash mismatch. It scans the snapshot into per-prefix temp files, then compares them against the DB filtered by slot <= target_slot and the configured program selector.

cargo run -p cloudbreak -- --config ./cloudbreak.index.toml snapshot-diff \
    --target-slot 416520592 \
    --full-snapshot /path/to/snapshot-...tar.zst \
    --incremental-snapshot /path/to/incremental-...tar.zst \
    --dump-mismatches ./diff.dump

Optional flags: --prefix 0a1b to scan only a single 16-bit prefix, --keep-tmp to retain ./snapshot_diff_tmp/ after exit.

Output ends with four counters:

match: <pubkeys identical in both>
mismatch: <pubkeys present in both but with different state>
only_in_snapshot: <pubkeys in snapshot but missing/older in DB>
only_in_db: <pubkeys in DB but missing in snapshot — typical leak symptom>

When --dump-mismatches is set, every diverging pubkey is appended to that file (uncapped); stdout shows only the first 50 of each category as a sample.

License

Copyright 2025-2026 Triton One Limited. All rights reserved.

Licensed under the GNU Affero General Public License v3.0 only. See LICENSE.