Repository navigation
feat(gcp): query Google Cloud metrics live with PromQL - #1475
JeremyFunk wants to merge 35 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Maple review🟢 Confidence 8/10 · likely safe to merge Adds live Google Cloud PromQL: four session-authed internal endpoints, one Google call per target project with per-connection reader tokens, edge-cached results, and the poller's cause texts moved into a shared
Before merge
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 5 of 5 changes observable
Files not reviewed (1)The review ended before it read these diffs, so nothing above vouches for them.
|
Maple review🟢 Confidence 8/10 · likely safe to merge Adds
Before merge
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 4 of 4 changes observable
|
Maple review🟢 Confidence 7/10 · likely safe to merge Adds live Google Cloud PromQL querying (range, instant, metric search/describe) to the internal integrations API, one Cloud Monitoring call per metrics scope with per-connection read-only tokens, results cached and failures reported beside answers. One misleading error path; otherwise safe to merge.
Before merge
Findings🟠 Warning · F1 ·
|
| Change | Kind | Observable | Evidence |
|---|---|---|---|
| POST /internal/integrations/gcp/promql/query_range and /query | Server | yes | handlers on the traced IntegrationsApiGroup (integrations.http.ts:478), service methods wrapped in Effect.fn("GcpPromQlService.queryRange") |
| GET gcp/metrics/search and gcp/metrics/describe | Server | yes | integrations.http.ts:488; Effect.fn("GcpPromQlService.searchMetrics") / describeMetric |
| Cloud Monitoring, Cloud Asset and IAM Credentials calls | Client | yes | call annotates peer.service=google-cloud (api.ts:198) on every request |
GcpConnectors read in targetsOf |
Client | yes | dbExecute/effect-orm over MAPLE_DB (GcpPromQlService.ts:233) |
7be72a2 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.
![maple-review-bot[bot]](https://avatars.githubusercontent.com/in/4108169?s=60&v=4){kind=small}
Maple review🟢 Confidence 8/10 · likely safe to merge Adds
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
Observability coverage: 2 of 2 changes observable
|
Maple review🟢 Confidence 7/10 · likely safe to merge Adds live PromQL over an org's connected GCP projects: per-connection readers, metrics-scope fan-out with caching, failure lists, and metric search/describe over four internal routes. Not safe to merge as is — the one open defect from the earlier review is unchanged at this head.
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 2 of 2 changes observable
|
Maple review🟢 Confidence 9/10 · safe to merge Adds a live PromQL read path for Google Cloud connections: four internal endpoints, one Google call per metrics scope fanned out 4 at a time, per-project failures returned beside the answers, and an 8 MB/500-series cap. The new logic is tested and tenant-scoped; safe to merge.
Before merge
Fixed since the last review
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 2 of 2 changes observable
|
Maple review🟢 Confidence 9/10 · safe to merge Raises the web startup bundle budget to 701.5 KB for the PromQL query contract, on top of the GCP PromQL backend work reviewed earlier. The delta is a single constant with the required justification comment; safe to merge.
Before merge
Production impactTelemetry this change adds and removes (5)
What was checked
|
Maple review🟢 Confidence 8/10 · likely safe to merge Adds a live Google Cloud PromQL path — range, instant, metric search, descriptor — behind the internal integrations routes, with per-metrics-scope fan-out, 1 h caching and typed failures. The fan-out, dedup and failure accounting read correctly; one error-reporting conflation in
Before merge
Findings🟠 Warning · F2 ·
|
| Change | Kind | Observable | Evidence |
|---|---|---|---|
| POST /gcp/promql/query_range, POST /gcp/promql/query, GET /gcp/metrics/search | server | yes | handler spans from Effect.fn on queryRange/queryInstant/searchMetrics (GcpPromQlService.ts:495,510,542,574) |
Google Monitoring / Cloud Asset calls from call |
client | yes | traced httpClient with annotateCurrentSpan({ "peer.service": ... }) (gcp/api.ts:198) |
| GcpConnectors read per request | client | yes | dbExecute inside the request's Effect.fn span (packages/backend/src/platform/db-execute.ts:67) |
0977b2b · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.
# Conflicts: # apps/web/perf/check-bundle-budget.ts
Maple review🟢 Confidence 7/10 · likely safe to merge The change since the last review only raises the web startup gzip budget to 706.5 KB for the new PromQL query contract; the constant and its comment agree with the CI measurements. The earlier
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
|
Maple review🟢 Confidence 8/10 · likely safe to merge Backend for live PromQL over connected Google Cloud projects: a fan-out per metrics scope with each connection's reader token, four new internal endpoints, and failure texts shared with the poller. Safe to merge.
Before merge
Fixed since the last review
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 3 of 3 changes observable
|
# Conflicts: # apps/web/perf/check-bundle-budget.ts
Maple review🟢 Confidence 8/10 · likely safe to merge Adds live PromQL over a customer's Google Cloud metrics: a per-org fan-out service, four internal endpoints, and the poller's failure texts moved to
Before merge
Findings🔵 Note · F3 · Poller shows a connection problem for Maple-side
|
| Change | Kind | Observable | Evidence |
|---|---|---|---|
| Per-project PromQL / metrics-scope / metric-name / descriptor calls to Google | outbound HTTP | yes | call annotates peer.service=google-cloud (gcp/api.ts:198) and the HTTP client records one span per request |
gcpPromQlQueryRange, gcpPromQlQuery, gcpMetricSearch, gcpMetricDescribe |
inbound HTTP endpoint | yes | every sibling handler in HttpIntegrationsLive has no route-local span; the API self-traces through ingest |
e9d3883 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.
Maple review🟢 Confidence 8/10 · likely safe to merge Adds live Google Cloud PromQL:
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 4 of 4 changes observable
|
Maple review🟢 Confidence 8/10 · likely safe to merge The only delta since the previous review is the base branch merged in: the six web files at this head are byte-identical to Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
|
Maple review🟢 Confidence 8/10 · likely safe to merge Adds the backend live PromQL path: connection and project discovery, a 4-way Google fan-out with per-project failures returned beside the series, and edge-cached results. Safe to merge; the only outstanding item is the already-filed poller message.
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 4 of 4 changes observable
|
Maple review🟢 Confidence 7/10 · likely safe to merge This PR adds the backend for live Google Cloud PromQL queries: an org-scoped fan-out service, the GCP API client, shared error explanations and the internal route. Nothing changed in it since the last review except base-branch merges, and the one open finding still stands.
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 1 of 1 changes observable
|
Maple review🟢 Confidence 8/10 · likely safe to merge Adds live Google Cloud PromQL querying to the API: org-scoped targets built from GCP connections, one Google call per metrics scope with cached tokens, and typed errors for invalid, oversized or unreachable queries. Backend-only and safe to merge; the poller still records Maple-side sign-in failures against the connection.
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 3 of 3 changes observable
|
Maple review🟢 Confidence 7/10 · likely safe to merge Live PromQL over a customer's connected Google Cloud projects:
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 2 of 2 changes observable
|
Maple review🟢 Confidence 8/10 · likely safe to merge Adds live Google Cloud PromQL: tenant-scoped internal endpoints, a service that fans out one Google call per metrics scope with each connection's read-only account, and the request/result contract. Nothing new is filed; F3 stays open at this head.
Before merge
Still open from earlier reviews
Production impactTelemetry this change adds and removes (5)
What was checked
Observability coverage: 2 of 2 changes observable
|
|
Note ⏳ Maple is reviewing this pull request at |
This branch has not been deployed
1 participant
Query a customer's Google Cloud Monitoring metrics live, through Google's PromQL-compatible API. Nothing is ingested on this path; the 5-minute poller is unchanged. First of a 3-PR stack: this PR is the backend; the dashboard widget (web) and the MCP tools (apps/ai) build on the contract below.
What it does
applied_metrics_enabled = true): true with the metrics switch on today, and true for every set-up connection after the sibling access PR. Other connections are left out silently while at least one qualifies. When none does,GcpPromQlFailedErrorlists each connection with what to do (run the setup script; the guide's section on domain-restricted sharing if a policy blocks it). A project-scope connection is its project. A folder or organization connection is the active projects a live Cloud Asset search finds under it (Project asset type only, up to 2,000 projects, cached 1 h per connection in the edge cache).gcp_resourcesis not read: only the poller fills it. Projects are deduplicated across connections; overlapping connections read a project through the oldest one.x-goog-user-project: <host project>. Google evaluates a project URL over that project's whole metrics scope. So each target's scope is looked up first (v1locations/global/metricsScopes/{project}, cached 1 h per project; needsresourcemanager.projects.get, which Monitoring Viewer holds). A target that another target's scope already holds is not read again; it answers through that scope (widest scopes first). Withoutprojects, more than 20 projects is refused withGcpPromQlTooManyProjectsError, which lists them so the caller can choose.projectsmust be a subset of the targets. If a connection's project listing failed, a named project that may sit under it fails withGcpPromQlFailedErrorand the listing failure, not a 400. Project ids are validated (GcpProjectId) before they reach a URL path.project_idgets the project it was read from, so identical label sets from two projects never merge. When the project's metrics scope holds several projects, it getsmetrics_scope=<project>instead: an aggregation over that scope is not one project's. Google's ownproject_idis kept.resultTypeis Google's: an instant query of a range selector answers a matrix. NaN and ±Inf points are dropped. A scalar answer (1 + 1) comes back as a vector of one series. Astringresult is refused as an invalid query.failureslist: one entry per project whose query failed, or per connection that could not sign in or list its projects. They use the poller's cause-and-action texts, moved togcp/explain.ts(the poller's output is byte-identical; its tests pass unchanged). When nothing answers,GcpPromQlFailedErrorcarries the list. Google refusing the query itself (400bad_data) fails the request withGcpPromQlInvalidQueryErrorcarrying Google's message. Some failures count as that project's alone, with Google's text and "Narrow the query or widen the step": a 422executionerror (too many samples), and a 503timeout. A 429 says the answer may have passed Google's size limit and to narrow the query. If Maple's own sign-in fails, the request fails withIntegrationsUpstreamErrorand the cause goes to the log, not to the customer. That covers the key, Google OAuth, and IAM Credentials refusing Maple's project withSERVICE_DISABLED/BILLING_DISABLED/USER_PROJECT_DENIED.importServiceAccountKey,fetchMapleAccessTokenandimpersonateReader. Maple's token and each connection's reader token are kept per isolate and reused for 7 of their 10 minutes, which leaves room for the slowest fan-out. Only settled tokens are kept, never a pending sign-in shared across requests. Readers are acquired one connection at a time before the fan-out, so a cold isolate signs in as Maple once.orgIdin every key):Contract for the web widget and MCP tools
Endpoints in
IntegrationsApiGroup(/internal/integrations, session auth, any org member):gcpPromQlQueryRangePOST /gcp/promql/query_rangeGcpPromQlRangeRequest{ query, startTime, endTime, stepSeconds, projects? }GcpPromQlResultgcpPromQlQueryPOST /gcp/promql/queryGcpPromQlInstantRequest{ query, time, projects? }GcpPromQlResultgcpMetricSearchGET /gcp/metrics/search?search=&limit=&project=GcpMetricSearchResponse{ projectId, metrics }gcpMetricDescribeGET /gcp/metrics/describe?type=&project=GcpMetricDescriptor@maple/domain/http(packages/domain/src/http/gcp-promql.ts).TinybirdDateTime(UTCYYYY-MM-DD HH:mm:ss), like the query engine.stepSecondsis a whole number from 60 to 86,400, with at most 11,000 points per series on the step-aligned range Google is sent (Google's limit, refused as a 400).queryis trimmed, 1 to 4,000 characters.projectsholds 1 to 20GcpProjectId.limitis 1 to 200 (default 50).GcpPromQlResult:{ resultType: "matrix" | "vector", series: [{ labels: Record<string,string>, points: [[epochSeconds, number]] }], projects: GcpProjectId[] /* the ones that answered, directly or through another's metrics scope */, failures: [{ scope, message }] }.scopeisprojects/<id>,folders/<n>ororganizations/<n>.labelscarriesproject_id, ormetrics_scopefor an aggregation over a multi-project scope.GcpMetricSearchResponse.metrics: PromQL UTF-8 names (each legacya_b_c:dtwin dropped; Prometheus-native names, colons included, are kept), case-insensitive substring match, name order. Distributions appear as_bucket/_count/_sum.GcpMetricDescriptor:{ type, metricKind, valueType, unit, description, monitoredResourceTypes, labels: [{ key, description }] }. A_bucket/_count/_sumname resolves to its distribution's descriptor. More than one resource type means a starter query needs amonitored_resourcematcher.projectif given (it must be one of the targets), else the first target.Service, for apps/ai:
GcpPromQlService(@maple/backend/services/integrations/GcpPromQlService,static readonly layer; needsDatabase,Env,EdgeCacheServicefrom the app root):Errors (
GcpPromQlError):@maple/http/errors/GcpPromQlUnavailableErrorMAPLE_GCP_SERVICE_ACCOUNT_KEYis unset@maple/http/errors/IntegrationsNotConnectedError@maple/http/errors/IntegrationsValidationError@maple/http/errors/GcpPromQlInvalidQueryErrormessageis Google's text, plusprojectId@maple/http/errors/GcpPromQlTooManyProjectsErrorprojectslists them@maple/http/errors/GcpPromQlTooManySeriesError{ message }; the message says to aggregate (sum by/avg by), narrow the selector, or widen the step. Never truncated.@maple/http/errors/GcpPromQlFailedErrorfailures@maple/http/errors/IntegrationsUpstreamError@maple/http/errors/IntegrationsPersistenceErrorThe five new tags are in the regenerated anticipated-errors list.
Decisions and deviations
failuresentries are{ scope, message }, not{ projectId, message }(design decision 7). A connection that cannot sign in or list its folder's projects has no single project to name.project, so a caller can move off a first project that fails (no billing, say).timeoutparameter is18s, so Google answers before the client's existing 20 s request timeout gives up.gcp/api.tscallgained an optionalrefusehook, which turns a failed answer into the caller's own typed error. The poller's error behaviour is unchanged.stringresults. Only an explicit string literal query produces one, and it fails as an unexpected body.Known limits, left for later
GCP_PROMQL_MAX_SERIES), checked as each project answers. Each live-query answer (PromQL, metric names, descriptor, metrics scope, live project search) is read up to 8 MB (MAX_BODY_BYTES). A PromQL answer past it is refused withGcpPromQlTooManySeriesErrorand the rest is left unread; on the other live calls it is a per-project failure. The poller and resource sync read uncapped, as before. With 4 projects in flight that bounds raw bodies at 32 MB per request.GcpPromQlFailedErrorisretry: never. Its failures mix setup problems, which a retry doesn't fix, with rate limits, which it does. Each failure's message says which.Env / infra
MAPLE_GCP_SERVICE_ACCOUNT_KEYis now bound on the api Worker (optionalSecret, the same way as on alerting), for prd and previews. Infisicaldevalready has it. apps/ai is not touched; the MCP PR binds it there.Not verified without a live system
bad_data400,label/__name__/values, v3metricDescriptors). The tests use anonymised versions of those shapes.POST /internal/integrations/gcp/promql/query_rangewithsum by (service) (increase({"serviceruntime.googleapis.com/api/request_count", monitored_resource="consumed_api"}[10m]))and step 600 over 2 h: series carryproject_id.monitored_resourcematcher: 400GcpPromQlInvalidQueryErrorwith Google's text.GET /internal/integrations/gcp/metrics/search?search=run.googleapisandGET .../gcp/metrics/describe?type=run.googleapis.com/request_latencies_bucket.