diff --git a/apps/landing/src/content/docs/integrations/gcp-opentelemetry.md b/apps/landing/src/content/docs/integrations/gcp-opentelemetry.md new file mode 100644 index 0000000000..85d0e0d903 --- /dev/null +++ b/apps/landing/src/content/docs/integrations/gcp-opentelemetry.md @@ -0,0 +1,178 @@ +--- +title: "Google Cloud with OpenTelemetry" +description: "Which telemetry to send from your workloads over OpenTelemetry and which to collect through the Google Cloud connection, for GKE, Cloud Run and Compute Engine." +group: "Integrations" +order: 8 +navLabel: "Google Cloud + OpenTelemetry" +--- + +Your workloads send traces, logs and metrics straight to Maple over OpenTelemetry. The [Google Cloud connection](/docs/integrations/gcp) forwards Cloud Logging entries and reads Cloud Monitoring metrics for everything your code never sees. Set up both, and keep container logs on the OpenTelemetry side. + +| Signal | Over OpenTelemetry | Google Cloud connection | +| --------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| Traces | Every span your services create | None | +| Logs | Application logs, linked to their traces | Cloud Run and load balancer request logs, audit logs, managed-service logs such as Cloud SQL and Pub/Sub, GKE node and cluster logs | +| Metrics | Application and runtime metrics | Platform metrics from Cloud Monitoring, named `gcp.*` | +| Resources | None | The resource list: projects, services, clusters, instances and databases | + +## Recommended setup + +### GKE + +1. Instrument each service with an OpenTelemetry SDK. Export to Maple, or to the [Maple Kubernetes collector](/docs/infrastructure/kubernetes) in the cluster. +2. [Connect Google Cloud](/docs/integrations/gcp#connect) and keep the log filter on **Recommended: no GKE container logs**. + +The connection then adds node and cluster logs, audit logs and the `gcp.gke.*` metrics. + +### Cloud Run + +1. Instrument the service with an OpenTelemetry SDK that exports to Maple, with the settings under [Instrument a Cloud Run service](#instrument-a-cloud-run-service). +2. [Connect Google Cloud](/docs/integrations/gcp#connect). +3. If the service exports logs over OpenTelemetry, [leave out its container output](#change-the-filter) too: the recommended filter forwards what a Cloud Run container writes to stdout and stderr. + +The connection adds Cloud Run's request logs and the `gcp.cloud_run.*` metrics. Cloud Run sends a `traceparent` header with every request. An SDK that continues it gives the server span the trace ID of Google's request log for that request, so the request log, your spans and your log lines join on one trace. + +### Compute Engine + +1. Instrument the application with an OpenTelemetry SDK that exports to Maple. +2. Run the OpenTelemetry Collector on the VM for [host metrics](/docs/infrastructure/hosts). +3. [Connect Google Cloud](/docs/integrations/gcp#connect). + +The connection adds audit logs and the `gcp.gce.*` metrics. + +## GKE container logs + +The recommended filter leaves out `resource.type="k8s_container"`: what the containers of a GKE cluster write to stdout and stderr. + +| Reason | Detail | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Duplicates | A workload that sends its logs over OpenTelemetry, or a collector that reads pod logs, already delivers each line to Maple. Forwarding it from Google Cloud stores it twice. | +| No trace link | The OpenTelemetry copy carries the trace and span ID of the request that wrote it. The copy from Google Cloud has neither. | +| Volume | Container output grows with traffic. Every forwarded line counts toward your Maple plan and toward the Pub/Sub usage Google bills to your account. | + +Include GKE container logs when the workloads don't send their logs over OpenTelemetry and no collector reads them. Cloud Logging is then the only place those lines exist. + +## Change the filter + +On the connection's card, click **Configure** and choose a filter under **Log filter**. Click **Continue to the script**, then copy the script and run it in Cloud Shell again. + +| Log filter | The sink forwards | +| -------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| **Recommended: no GKE container logs** | Platform logs. Leaves out GKE container logs and the [other exclusions](/docs/integrations/gcp#logs). | +| **Include GKE container logs** | The same, plus GKE container logs. Maple asks you to acknowledge the duplicates before you continue. | +| **Keep the sink's current filter** | What it forwards now. Offered, and preselected, once a run has set log forwarding up. | + +For any other filter, paste the script into an editor, edit `LOG_FILTER` near its top, set `LOG_FILTER_MODE` to `set` and run it. To also leave out Cloud Run container output, for services that send their logs over OpenTelemetry, add this at the end of `LOG_FILTER`, inside the quotes: + +```text +AND NOT log_id("run.googleapis.com/stdout") AND NOT log_id("run.googleapis.com/stderr") +``` + +Cloud Run's request logs are in `run.googleapis.com/requests` and still pass. The filter uses the [Logging query language](https://cloud.google.com/logging/docs/view/logging-query-language). + +## Instrument a Cloud Run service + +These settings decide whether every trace arrives and joins Google's request log. They apply in any language. The example is Node.js, and the [Node.js guide](/docs/guides/instrumentation-nodejs) covers the SDK basics. + +| Setting | On Cloud Run | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Service name | Set `service.name` to the Cloud Run service name, `K_SERVICE`. Google's request logs arrive in Maple under that name. With another one, your spans and logs land in a different service than the request logs. | +| Sampling | Sample every trace: `AlwaysOnSampler`, or `OTEL_TRACES_SAMPLER=always_on`. Cloud Run sets the sampled flag of `traceparent` on about one request per ten seconds per instance. The SDK's default sampler follows that flag and drops the rest, while the log records of those requests still arrive. | +| Flushing | Flush the span and log processors when the server span ends. Call `sdk.shutdown()` on `SIGTERM` and keep `node` as PID 1. Cloud Run throttles the container's CPU once the response is sent: with the default batch timers, telemetry arrives 5 to 6 seconds after the request on a warm instance and about 30 seconds after a cold start. | +| Logs | Emit application logs over OpenTelemetry, not stdout. Each record carries the trace and span ID of its request, and nothing is stored twice. | + +For Node.js 22 with `@opentelemetry/sdk-node`: + +```js +// tracing.mjs +import { register } from "node:module" +import { SpanKind } from "@opentelemetry/api" +import { NodeSDK } from "@opentelemetry/sdk-node" +import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node" +import { envDetector, processDetector, resourceFromAttributes } from "@opentelemetry/resources" +import { gcpDetector } from "@opentelemetry/resource-detector-gcp" +import { AlwaysOnSampler, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base" +import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs" +import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics" +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto" +import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-proto" +import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto" + +// Lets the instrumentations patch ES modules. +register("@opentelemetry/instrumentation/hook.mjs", import.meta.url) + +// The exporters read OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. +const spans = new BatchSpanProcessor(new OTLPTraceExporter()) +const logRecords = new BatchLogRecordProcessor({ exporter: new OTLPLogExporter() }) +const metricExporter = new OTLPMetricExporter() + +// Export as each request finishes, before Cloud Run throttles the CPU. +const flushOnRequestEnd = { + onStart() {}, + onEnd(span) { + if (span.kind !== SpanKind.SERVER) return + void spans.forceFlush() + void logRecords.forceFlush() + }, + forceFlush: () => Promise.resolve(), + shutdown: () => Promise.resolve(), +} + +const sdk = new NodeSDK({ + // The name Google's request logs carry. + resource: resourceFromAttributes({ "service.name": process.env.K_SERVICE }), + // gcpDetector adds cloud.platform, cloud.region and the faas.* attributes. + resourceDetectors: [envDetector, processDetector, gcpDetector], + sampler: new AlwaysOnSampler(), + spanProcessors: [spans, flushOnRequestEnd], + logRecordProcessors: [logRecords], + metricReaders: [new PeriodicExportingMetricReader({ exporter: metricExporter })], + instrumentations: [getNodeAutoInstrumentations()], +}) +sdk.start() + +// Cloud Run sends SIGTERM before it stops an instance. +process.on("SIGTERM", () => { + sdk.shutdown().finally(() => process.exit(0)) +}) +``` + +Start the container with `node` as PID 1, so that it receives `SIGTERM`: + +```dockerfile +CMD ["node", "--import", "./tracing.mjs", "server.mjs"] +``` + +Set these on the service: + +- `OTEL_EXPORTER_OTLP_ENDPOINT`: `https://ingest.maple.dev`, or `https://ingest.eu.maple.dev` for an EU organization. +- `OTEL_EXPORTER_OTLP_HEADERS`: `Authorization=Bearer YOUR_INGEST_KEY`. + +Emit a log record inside a request and it carries the active trace and span ID: + +```js +import { logs, SeverityNumber } from "@opentelemetry/api-logs" + +const logger = logs.getLogger("checkout") + +logger.emit({ + severityText: "INFO", + severityNumber: SeverityNumber.INFO, + body: "checkout completed", + attributes: { "order.id": orderId }, +}) +``` + +A service that has to log to stdout keeps the link by writing JSON lines with `logging.googleapis.com/trace` (`projects/PROJECT_ID/traces/TRACE_ID`) and `logging.googleapis.com/spanId`. Send each line one way only: over OpenTelemetry or through stdout. + +Limits of this setup: + +- Your service's top span is the child of a span inside Google's infrastructure that Maple does not receive, so the trace has no root span. +- Flushing per request makes one export per request. Check the overhead before you use it on a service with a high request rate. + +## Instrumentation guides + +- Languages: [Node.js](/docs/guides/instrumentation-nodejs), [Next.js](/docs/guides/instrumentation-nextjs), [Python](/docs/guides/instrumentation-python), [Go](/docs/guides/instrumentation-go), [Rust](/docs/guides/instrumentation-rust), [Java](/docs/guides/instrumentation-java), [Kotlin](/docs/guides/instrumentation-kotlin), [C# / .NET](/docs/guides/instrumentation-csharp) and [Laravel](/docs/guides/instrumentation-laravel). +- [Kubernetes infrastructure](/docs/infrastructure/kubernetes): the collector that receives OTLP from in-cluster services. +- [Hosts](/docs/infrastructure/hosts): host metrics from a VM. +- [Google Cloud](/docs/integrations/gcp): connect, verify and troubleshoot the connection. diff --git a/apps/landing/src/content/docs/integrations/gcp.md b/apps/landing/src/content/docs/integrations/gcp.md new file mode 100644 index 0000000000..0a39beb6b7 --- /dev/null +++ b/apps/landing/src/content/docs/integrations/gcp.md @@ -0,0 +1,506 @@ +--- +title: "Google Cloud" +description: "Connect a Google Cloud organization, folder or project by running one script in Cloud Shell. Maple receives Cloud Logging entries through Pub/Sub and reads Cloud Monitoring metrics every 5 minutes." +group: "Integrations" +order: 7 +--- + +The Google Cloud integration connects an organization, a folder or a single project to Maple. You choose what each connection collects: + +| What to collect | Collects | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Log forwarding** | A Log Router sink sends Cloud Logging entries through Pub/Sub to Maple. They appear in [Logs](/docs/explore/logs) under the service that wrote them. | +| **Metrics and resources** | Maple reads Cloud Monitoring metrics for Cloud Run, Cloud Functions, GKE, Compute Engine, Cloud SQL, Pub/Sub and HTTP(S) load balancers every 5 minutes, and lists the resources behind them every hour. The metrics are named `gcp.*` and work in dashboards and alert rules. | + +There is no OAuth step and no service account key. You run a generated `gcloud` script in Cloud Shell, and Maple gets no write access to Google Cloud. + +If your workloads also send telemetry over OpenTelemetry, read [Google Cloud with OpenTelemetry](/docs/integrations/gcp-opentelemetry): it covers which telemetry to send over OpenTelemetry and which to collect here, and why GKE container logs are left out by default. + +## Prerequisites + +- You are an admin of the Maple organization. +- You can sign in to Cloud Shell with the roles for what you connect: + +| You connect | Roles you need | +| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| A project | Owner on the project. | +| A folder | Owner on the host project. On the folder: Logs Configuration Writer for the log sink, and Folder IAM Admin for the read-only roles. | +| An organization | Owner on the host project. On the organization: Logs Configuration Writer for the log sink, and Organization Administrator for the read-only roles. | + +If you can't get Owner, see [Roles for running the script](#roles-for-running-the-script). + +The host project holds Maple's Pub/Sub topic, subscription and read-only service account. A project is its own host project. For a folder or organization, name a shared operations project inside it that won't be deleted. Google bills the Pub/Sub and Cloud Monitoring usage to the host project. + +**Metrics and resources** needs billing enabled on the host project: Cloud Monitoring only answers for projects that have a billing account. The script checks this before it changes anything. + +### Find your IDs + +| ID | Where to find it | +| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Project ID | The console's project picker lists it next to each project name, or run `gcloud projects list`. It is not the project name or number. | +| Organization ID | **IAM & Admin → [Manage resources](https://console.cloud.google.com/cloud-resource-manager)** in the console, or run `gcloud organizations list`. Digits only. | +| Folder ID | The same console page, or run `gcloud resource-manager folders list --organization=ORGANIZATION_ID` with your organization's ID in place of `ORGANIZATION_ID`. Digits only. | + +## Connect + +1. Open **Integrations → Google Cloud** in Maple and click **Connect Google Cloud**. When the page already has a connection, click **Add connection** above the list. The dialog **Connect Google Cloud** opens on its first stage, **Connect**. +2. Under **What to connect**, choose **Organization**, **Folder** or **Project** and enter its ID. An organization or folder covers every project in it, including new ones. +3. For an organization or folder, enter the **Host project ID**. +4. Under **What to collect**, leave **Log forwarding** and **Metrics and resources** ticked, or untick one. To change which logs are forwarded, choose a [log filter](#log-filter) under **Log filter**. +5. Click **Create and continue**. Maple creates the connection and the dialog moves to its second stage, **Apply in Google Cloud**. Nothing changes in Google Cloud until you run the script. +6. In step 1, click **Open Cloud Shell** and sign in with the roles above. Click **Authorize** if Cloud Shell asks. +7. In step 2, click **Copy script**, paste it into Cloud Shell and press Enter. The script takes about a minute. **Read the script** shows it in the dialog. +8. Return to Maple. Step 3, **Maple is waiting for the run**, updates on its own. Within a minute of the script ending, usually in seconds, the dialog shows **Google Cloud matches this configuration**. Click **Done**. + +If you close the dialog before the script has run, the connection's card shows **Setup pending**. Click **Finish setup** to open the dialog at the script. After five minutes without a report from the script, step 3 reads **Maple has no confirmation yet** and lists what to check. + +A first run for a project prints: + +```text +Maple setup for project acme-prod + Log forwarding on + Metrics and resources on + +Checking access + ✓ Signed in as jane@acme.com + ✓ Project acme-prod found (Acme Production) + ✓ Billing is enabled + ✓ jane@acme.com has the permissions this script needs + ✓ Log filter accepted + +Log forwarding + ✓ APIs enabled (Pub/Sub, Cloud Logging, Cloud Resource Manager) + ✓ Topic created + ✓ Push subscription created + ✓ Log sink created + ✓ Sink allowed to write to the project + ✓ Sink allowed to publish to the topic + ✓ Check message sent to Maple through the topic + +Metrics and resources + ✓ APIs enabled (Cloud Monitoring, Cloud Asset, IAM, IAM Credentials, Cloud Resource Manager) + ✓ Read-only service account created + ✓ Read-only roles granted (Monitoring Viewer, Cloud Asset Viewer, Service Usage Consumer) + ✓ Maple allowed to read as that account + ✓ Maple notified + +Done. Google Cloud is set up for Maple. + Maple confirms it within a minute: https://app.maple.dev/integrations?integration=gcp + Logs: a new sink can take about 10 minutes to start forwarding. What is logged before + it does is not forwarded later. + Metrics: the first read lands within about 10 minutes. +``` + +If a step fails, the script stops and prints what to do. Fix it and paste the script again: it continues where it stopped. Running the script again is safe, and its last line says what the run did: + +| The script ends with | The run | +| -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| "Done. Google Cloud is set up for Maple." | Created the log sink or the read-only service account. | +| "Done. Everything is in place." | Created and removed nothing. It makes every grant again, so it also restores one that was removed. | +| "Done. The log sink has the filter of this script. Everything else is in place." | Replaced the sink's filter and created nothing. | +| "Done. Google Cloud matches your Maple switches." | Removed what **Log forwarding** or **Metrics and resources**, now off, had set up. | + +After Maple confirms the run, **Log forwarding** and **Metrics and resources** each wait for their first data: + +| What to collect | First data | +| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| Log forwarding | A new sink can take about 10 minutes to start forwarding. Entries logged before that are not forwarded. | +| Metrics and resources | The first read lands within about 10 minutes. Maple reads every 5 minutes, about 5 minutes behind, so that each minute is complete. | + +The script contains a secret that lets anyone send logs to your Maple organization. Don't share it or commit it, and run it in Cloud Shell: there neither the paste nor the secret ends up in shell history. The pasted text keeps itself out of the history files of bash 5 and zsh, but it still shows in your terminal's scrollback. + +The push endpoint with the secret is also visible in the subscription's configuration and in the host project's Admin Activity audit log, to anyone who can read those. If it leaks, disconnect and connect again: the new connection gets a new secret. + +The first two lines and the last two lines of the pasted text run the script in a bash process of its own, so a failed step can't close your Cloud Shell session. + +A project, folder or organization can be connected once per Maple organization. A project that sits inside a connected organization or folder is collected twice if you also connect it on its own. + +### Change what a connection collects + +1. On the connection's card, click **Configure**. The dialog opens on its first stage, **Configure**. +2. Under **What to collect**, tick or untick **Log forwarding** or **Metrics and resources**. The line under each checkbox says what saving that choice does. +3. Click **Save and continue**. For a change that takes effect in Maple at once, Maple asks first, in a dialog titled for example "Turn log forwarding off for Project acme-prod?". The dialog says what happens **In Maple** and what goes on **In Google Cloud**. Click **Turn off and continue**, or **Turn on** when you turn something back on that still exists in Google Cloud. Then the button of the first stage reads **Save**, and there is nothing to run. +4. The dialog moves to **Apply in Google Cloud**. Its recap says what is saved and what this run does. Copy the script and run it in Cloud Shell again. +5. When the run reports, the dialog shows **Google Cloud matches this configuration**. Click **Done**. + +A change takes effect in Maple and in Google Cloud at different times: + +| Change in **Configure** | In Maple | In Google Cloud | +| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| **Log forwarding** off | Maple stops storing the connection's logs within about a minute of saving. | Google Cloud keeps publishing logs to Pub/Sub, billed by Google, until the script has removed the log sink. | +| **Metrics and resources** off | Maple stops reading from the next 5-minute read. | The read-only service account and its roles stay until the script has removed them. | +| **Log forwarding** or **Metrics and resources** on | Starts once the script has run. | The script creates the log sink, or the read-only service account. | +| Back on while Google Cloud still has the log sink or the service account | At once: Maple stores the logs again as soon as you save, and reads metrics again from the next 5-minute read. | Nothing changes. | + +Until the script has run, the connection's card shows **Changes not applied in Google Cloud** and says what goes on until then, for example "Google Cloud keeps forwarding logs, billed by Google, until the script has removed the log sink." Click **Apply changes** to open the dialog at the script. + +A run that removes the log sink takes one to two minutes: after it deletes the sink it waits a minute for Google to stop routing to the topic, so your project logs no sink error. + +When nothing is changed, the button in the first stage reads **Continue to the script**. The setup script exists only in the second stage of this dialog, so this is how you get it again, for example to change the [log filter](#log-filter) or to repair something. **Back** returns to the first stage. + +A connection collects at least one of the two: the last one that is on can't be unticked. To stop collecting, [disconnect](#disconnect). + +### Log filter + +**Log filter** decides which filter the script writes onto the sink. It is in the first stage of **Connect Google Cloud** and of **Configure**, below **What to collect**, while **Log forwarding** is ticked. To change it on a connection, click **Configure**, choose a filter, click **Continue to the script** and run the script again. + +| Log filter | The script | +| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Recommended: no GKE container logs** | Sets the [recommended filter](#logs). Preselected for a connection that has no sink yet. | +| **Include GKE container logs** | Sets the recommended filter without its GKE container clause. Maple asks you to acknowledge the duplicates before you continue. | +| **Keep the sink's current filter** | Leaves an existing sink's filter as it is. Offered, and preselected, once a run has set log forwarding up. Maple can't see the sink's filter: read it in the console under **Logging → [Log Router](https://console.cloud.google.com/logs/router)**. | + +Include GKE container logs only for workloads that don't send their logs to Maple over OpenTelemetry. Otherwise each line is stored twice: see [GKE container logs](/docs/integrations/gcp-opentelemetry#gke-container-logs). + +Choosing **Include GKE container logs** shows the notice **Not recommended when your GKE workloads send traces to Maple**. The button that continues stays disabled until you tick **I understand that GKE container logs can be stored twice**. To back out, click **Use recommended filter**, or **Keep current filter** on a connection that has a sink. + +For any other filter, paste the script into an editor first, edit `LOG_FILTER` near its top and set `LOG_FILTER_MODE` to `set`. The filter uses the [Logging query language](https://cloud.google.com/logging/docs/view/logging-query-language). The script checks the filter with Google before it creates anything. Later runs with **Keep the sink's current filter** leave your filter in place, while the other two options replace it. + +## Verify + +Under **Connections**, each connection has a card, the worst state first. A card shows a status for **Log forwarding** and one for **Metrics and resources**. The page updates on its own. + +| Log forwarding | Meaning | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Not set up** | The setup script has not set log forwarding up yet. Run it. | +| **Setup running** | A run reported on metrics and resources in the last two minutes and has not reported on log forwarding yet. | +| **Waiting for first logs** | The script ran and no entry has arrived. After 20 minutes the row shows the hint **No log entry in 20 minutes** with a `gcloud logging write` command that writes a test entry. | +| **Receiving logs** | Entries arrive. The row shows when the last one did, and **View logs** opens Logs filtered to the connection. | +| **No logs in 24 hours** | Nothing passed the filter for a day. Not an error. | +| **Rejecting logs** | Maple refused the last push. The row says why and what to do. | +| **Off** | Turned off in **Configure**. While Google Cloud still has the log sink, the row reads "Maple discards what Google Cloud still forwards". | + +| Metrics and resources | Meaning | +| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Not set up** | The setup script has not granted Maple read access yet. Run it. | +| **Setup running** | A run reported on log forwarding in the last two minutes and has not reported on metrics and resources yet. | +| **Waiting for first metrics** | The script ran and the first read has not landed. After 15 minutes the row says that no metrics have arrived yet. | +| **Receiving metrics** | Reads succeed. The row shows when the last metrics arrived, and **Open Infrastructure** opens **Infrastructure → Google Cloud**. A folder or organization also shows how many projects Maple found. The label reads **Receiving metrics, resource list incomplete** when Maple could not list every resource. | +| **Receiving metrics, incomplete** | The last read is under 10 minutes old, but some metric queries failed or were cut short. The row says which. | +| **Metrics stalled** | No read for 30 minutes and no error. Maple retries every 5 minutes. | +| **Can't read metrics** | Reads fail, and none has succeeded in the last 10 minutes. The row says why and what to do. | +| **Off** | Turned off in **Configure**. While the read-only service account still exists, the row reads "Maple no longer uses its service account". | + +The page header and the Integrations list show the connection's worst status: **Needs attention**, **Setup pending**, **Changes pending**, **Waiting for data** or **Healthy**. + +A card whose connection needs the setup script shows a band with one button. The button opens **Configure** at its second stage, **Apply in Google Cloud**. + +| Band | Button | Shown when | +| --------------------------------------- | ------------------------ | ---------------------------------------------------------------------------------------------------------------------- | +| **Setup pending** | **Finish setup** | The script of a new connection has not reported, or its last run stopped part of the way. | +| **Changes not applied in Google Cloud** | **Apply changes** | The saved configuration and what the script last reported differ. The band says what goes on until the script has run. | +| **The setup script needs to run again** | **Run the script again** | The text of a failure asks for another run of the setup script. | + +Members who are not admins of the Maple organization see the statuses and the numbers, without **Configure**, the **⋮** menu or the band's button. The band asks them to have an admin run the setup script, and a line under the list says "Only Maple organization admins can add, configure or disconnect connections." + +Then check the data: + +1. Open [Logs](/docs/explore/logs) and look for a service named after one of your Cloud Run services, functions or instances. +2. In the [metrics explorer](/docs/explore/metrics), search for the prefix `gcp`. +3. Under **Dashboards → Browse templates**, select **Google Cloud** and click **Create dashboard**. It has one tab per service. Use the **Project**, **Region** and **Workload** filters in the toolbar to narrow every chart. To open it on one project, first fill in **Project ID** under **Parameters**, below the preview. + +## Permissions + +The script enables these APIs in the host project: + +- `pubsub.googleapis.com` and `logging.googleapis.com`, for log forwarding: carrying the log entries and routing them through the sink. +- `monitoring.googleapis.com`, `cloudasset.googleapis.com`, `iam.googleapis.com` and `iamcredentials.googleapis.com`, for metrics and resources: reading metrics, listing resources, creating the read-only service account and minting its short-lived tokens. No key is ever created. +- `cloudresourcemanager.googleapis.com`, for both: granting the roles below. + +For log forwarding it creates a Pub/Sub topic and push subscription in the host project and a log sink on the project, folder or organization. For metrics and resources it creates a read-only service account in the host project. Each is named `maple-` followed by 24 hexadecimal characters unique to the connection. + +It grants these roles: + +- **Pub/Sub Publisher** (`roles/pubsub.publisher`) to the sink's Google-managed writer identity, on Maple's topic: publishing log entries to the topic. +- **Logs Writer** (`roles/logging.logWriter`) to the same writer identity, on the host project: Google requires it on the project that holds a sink's destination. +- **Monitoring Viewer** (`roles/monitoring.viewer`) to the read-only service account, on the project, folder or organization: reading metrics (`monitoring.timeSeries.list`). +- **Cloud Asset Viewer** (`roles/cloudasset.viewer`) to the read-only service account, on the project, folder or organization: listing resources (`cloudasset.assets.searchAllResources`). The role can also read resource metadata and IAM policies. +- **Service Usage Consumer** (`roles/serviceusage.serviceUsageConsumer`) to the read-only service account, on the host project: calling both read APIs through the host project. +- **Service Account Token Creator** (`roles/iam.serviceAccountTokenCreator`) to Maple's service account, on the read-only service account: minting short-lived tokens for that one account. + +`roles/monitoring.viewer` and `roles/cloudasset.viewer` are the narrowest predefined roles for these calls, and both are read-only. On a folder or organization, every project under it inherits them. + +### Roles for running the script + +Owner on the host project is the simplest. Without it, these predefined roles together cover what the script does: + +| On | Role | For | +| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- | +| The host project | Service Usage Admin (`roles/serviceusage.serviceUsageAdmin`) | Switching on the APIs above. | +| The host project | Project IAM Admin (`roles/resourcemanager.projectIamAdmin`) | The role grants on the host project. | +| The host project | Pub/Sub Admin (`roles/pubsub.admin`) | Log forwarding: the topic and the subscription. | +| The host project | Service Account Admin (`roles/iam.serviceAccountAdmin`) | Metrics and resources: the read-only service account. | +| The project, folder or organization | Logs Configuration Writer (`roles/logging.configWriter`) | Log forwarding: the sink. | +| The project, folder or organization | Project IAM Admin (`roles/resourcemanager.projectIamAdmin`), Folder IAM Admin (`roles/resourcemanager.folderIamAdmin`) or Organization Administrator (`roles/resourcemanager.organizationAdmin`) | Metrics and resources: the read-only roles. | + +Before it changes anything, the script asks Google which permissions the signed-in account holds and stops with the missing ones. + +## Collected data + +### Logs + +Maple receives every entry the sink's filter lets through. The recommended filter excludes: + +| Excluded | Reason | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Data Access audit logs | They record every API read. | +| Load balancer health checks | Probes hit each backend every few seconds. | +| Kubernetes lease renewals | A GKE cluster renews its leader-election leases all day: hundreds of audit entries a minute for an idle node. | +| VM serial console output | A VM writes thousands of lines of raw terminal output at boot. | +| GKE container logs | Workloads that send logs over OpenTelemetry already deliver them to Maple, so each line would be stored twice. See [GKE container logs](/docs/integrations/gcp-opentelemetry#gke-container-logs). | + +The filter, one clause per line in the same order: + +```text +NOT log_id("cloudaudit.googleapis.com/data_access") +AND NOT httpRequest.userAgent:"GoogleHC" +AND NOT protoPayload.methodName="io.k8s.coordination.v1.leases.update" +AND NOT logName:"serialconsole.googleapis.com" +AND NOT resource.type="k8s_container" +``` + +**Include GKE container logs** sets the filter without the last clause. To forward different logs, see [Log filter](#log-filter). + +The script reports to Maple through a log named `maple-setup`. Maple reads that entry as the report and does not store it, so don't write your own logs under that name. + +Each entry gets its `service.name` from the resource that wrote it, so workload logs join the traced service of the same name: + +| Resource | `service.name` | +| ----------------------- | ----------------------------------------------------------------- | +| Cloud Run service | The service name. | +| Cloud Run job | The job name. | +| Cloud Function | The function name. | +| GKE container | The container name. | +| Compute Engine instance | The instance name, or `gcp/gce_instance` when the entry has none. | +| App Engine | The module ID. | +| Everything else | `gcp/`, for example `gcp/cloudsql_database`. | + +Every entry also carries `cloud.provider` (`gcp`), `gcp.resource.type` and the resource's labels as `gcp.resource.labels.*`, plus `cloud.account.id` (the project ID) and `cloud.region` when the resource has them. GKE container entries add `k8s.cluster.name`, `k8s.namespace.name`, `k8s.pod.name` and `k8s.container.name`. Cloud Run service entries add `faas.name`, `faas.version` (the revision) and `faas.instance`, Cloud Run job entries `gcp.cloud_run.job.execution` and `gcp.cloud_run.job.task_index`, and Compute Engine entries `host.id`, `host.name` and `gcp.gce.instance.name`. + +The log body is `textPayload`, or the `message` or `msg` field of `jsonPayload`, or the whole `jsonPayload` as JSON when it has neither. The other `jsonPayload` fields become log attributes. Audit logs use the method name as the body, and request logs the method, URL and status. An entry's `trace` and `spanId` become the log's trace and span IDs. + +Entry fields that have an OpenTelemetry semantic convention take its name and type: + +| Cloud Logging field | Attribute | +| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `httpRequest` | `http.request.method`, `url.full`, `http.response.status_code`, `http.request.size`, `http.response.size`, `user_agent.original`, `client.address`, `server.address`, `http.request.header.referer` | +| `sourceLocation` | `code.file.path`, `code.line.number`, `code.function.name` | +| `insertId` | `log.record.uid` | +| Audit `authenticationInfo.principalEmail`, `status.code` | `user.email`, `rpc.response.status_code` (the code name, for example `PERMISSION_DENIED`) | + +Other fields keep their Cloud Logging names under `gcp.`: `gcp.log_name`, `gcp.labels.*`, `gcp.http_request.*` (for example `latency`), `gcp.source_location.*` and `gcp.audit.*`. + +| Cloud Logging severity | Maple severity | +| -------------------------------- | -------------- | +| `DEBUG` | `DEBUG` | +| `INFO`, `NOTICE` | `INFO` | +| `WARNING` | `WARN` | +| `ERROR` | `ERROR` | +| `CRITICAL`, `ALERT`, `EMERGENCY` | `FATAL` | +| `DEFAULT` | Not set | + +### Metrics + +Maple stores these Cloud Monitoring metrics at one-minute resolution, under the `gcp.` namespace and named the way OpenTelemetry semantic conventions name metrics. Two Cloud Monitoring types that measure the same thing share a name: the byte and operation counters are split by a direction attribute, shown in brackets, and Cloud SQL reports connections from whichever of the two types its engine has. + +| Service | Metric | Cloud Monitoring type | Kind | Unit | +| ------------------------------------ | ---------------------------------------------------- | --------------------------------------------------------------------------------- | ------------ | -------------- | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.requests` | `run.googleapis.com/request_count` | Counter | `{request}` | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.request.duration` | `run.googleapis.com/request_latencies` | Distribution | `s` | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.container.instance.count` | `run.googleapis.com/container/instance_count` | Gauge | `{instance}` | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.container.cpu.utilization` | `run.googleapis.com/container/cpu/utilizations` | Distribution | `1` | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.container.memory.utilization` | `run.googleapis.com/container/memory/utilizations` | Distribution | `1` | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.container.request.max_concurrency` | `run.googleapis.com/container/max_request_concurrencies` | Distribution | `{request}` | +| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.container.billable_time` | `run.googleapis.com/container/billable_instance_time` | Counter | `s` | +| Cloud Functions (1st gen) | `gcp.cloud_functions.executions` | `cloudfunctions.googleapis.com/function/execution_count` | Counter | `{execution}` | +| Cloud Functions (1st gen) | `gcp.cloud_functions.execution.duration` | `cloudfunctions.googleapis.com/function/execution_times` | Distribution | `s` | +| Cloud Functions (1st gen) | `gcp.cloud_functions.instance.count` | `cloudfunctions.googleapis.com/function/instance_count` | Gauge | `{instance}` | +| Cloud Functions (1st gen) | `gcp.cloud_functions.memory.usage` | `cloudfunctions.googleapis.com/function/user_memory_bytes` | Distribution | `By` | +| Cloud Functions (1st gen) | `gcp.cloud_functions.network.io` (`transmit`) | `cloudfunctions.googleapis.com/function/network_egress` | Counter | `By` | +| GKE containers | `gcp.gke.container.cpu.time` | `kubernetes.io/container/cpu/core_usage_time` | Counter | `s` | +| GKE containers | `gcp.gke.container.cpu.limit.utilization` | `kubernetes.io/container/cpu/limit_utilization` | Gauge | `1` | +| GKE containers | `gcp.gke.container.memory.usage` | `kubernetes.io/container/memory/used_bytes` | Gauge | `By` | +| GKE containers | `gcp.gke.container.memory.limit.utilization` | `kubernetes.io/container/memory/limit_utilization` | Gauge | `1` | +| GKE containers | `gcp.gke.container.restarts` | `kubernetes.io/container/restart_count` | Counter | `{restart}` | +| GKE nodes | `gcp.gke.node.cpu.allocatable.utilization` | `kubernetes.io/node/cpu/allocatable_utilization` | Gauge | `1` | +| GKE nodes | `gcp.gke.node.memory.allocatable.utilization` | `kubernetes.io/node/memory/allocatable_utilization` | Gauge | `1` | +| Compute Engine | `gcp.gce.cpu.utilization` | `compute.googleapis.com/instance/cpu/utilization` | Gauge | `1` | +| Compute Engine | `gcp.gce.memory.usage` | `compute.googleapis.com/instance/memory/balloon/ram_used` (E2 machine types only) | Gauge | `By` | +| Compute Engine | `gcp.gce.network.io` (`receive`) | `compute.googleapis.com/instance/network/received_bytes_count` | Counter | `By` | +| Compute Engine | `gcp.gce.network.io` (`transmit`) | `compute.googleapis.com/instance/network/sent_bytes_count` | Counter | `By` | +| Compute Engine | `gcp.gce.disk.io` (`read`) | `compute.googleapis.com/instance/disk/read_bytes_count` | Counter | `By` | +| Compute Engine | `gcp.gce.disk.io` (`write`) | `compute.googleapis.com/instance/disk/write_bytes_count` | Counter | `By` | +| Cloud SQL | `gcp.cloud_sql.cpu.utilization` | `cloudsql.googleapis.com/database/cpu/utilization` | Gauge | `1` | +| Cloud SQL | `gcp.cloud_sql.memory.utilization` | `cloudsql.googleapis.com/database/memory/utilization` | Gauge | `1` | +| Cloud SQL | `gcp.cloud_sql.disk.utilization` | `cloudsql.googleapis.com/database/disk/utilization` | Gauge | `1` | +| Cloud SQL | `gcp.cloud_sql.connection.count` | `cloudsql.googleapis.com/database/network/connections` (MySQL, SQL Server) | Gauge | `{connection}` | +| Cloud SQL | `gcp.cloud_sql.connection.count` | `cloudsql.googleapis.com/database/postgresql/num_backends` (PostgreSQL) | Gauge | `{connection}` | +| Cloud SQL | `gcp.cloud_sql.disk.operations` (`read`) | `cloudsql.googleapis.com/database/disk/read_ops_count` | Counter | `{operation}` | +| Cloud SQL | `gcp.cloud_sql.disk.operations` (`write`) | `cloudsql.googleapis.com/database/disk/write_ops_count` | Counter | `{operation}` | +| Cloud SQL | `gcp.cloud_sql.replication.lag` | `cloudsql.googleapis.com/database/replication/replica_lag` | Gauge | `s` | +| Pub/Sub subscriptions | `gcp.pubsub.subscription.undelivered_messages` | `pubsub.googleapis.com/subscription/num_undelivered_messages` | Gauge | `{message}` | +| Pub/Sub subscriptions | `gcp.pubsub.subscription.oldest_unacked_message.age` | `pubsub.googleapis.com/subscription/oldest_unacked_message_age` | Gauge | `s` | +| Pub/Sub subscriptions | `gcp.pubsub.subscription.sent_messages` | `pubsub.googleapis.com/subscription/sent_message_count` | Counter | `{message}` | +| Pub/Sub subscriptions | `gcp.pubsub.subscription.acked_messages` | `pubsub.googleapis.com/subscription/ack_message_count` | Counter | `{message}` | +| Pub/Sub subscriptions | `gcp.pubsub.subscription.push_requests` | `pubsub.googleapis.com/subscription/push_request_count` | Counter | `{request}` | +| Pub/Sub subscriptions | `gcp.pubsub.subscription.dead_lettered_messages` | `pubsub.googleapis.com/subscription/dead_letter_message_count` | Counter | `{message}` | +| Pub/Sub topics | `gcp.pubsub.topic.publish_requests` | `pubsub.googleapis.com/topic/send_request_count` | Counter | `{request}` | +| Load balancers | `gcp.load_balancing.requests` | `loadbalancing.googleapis.com/https/request_count` | Counter | `{request}` | +| Load balancers | `gcp.load_balancing.request.duration` | `loadbalancing.googleapis.com/https/total_latencies` | Distribution | `s` | +| Load balancers | `gcp.load_balancing.backend.request.duration` | `loadbalancing.googleapis.com/https/backend_latencies` | Distribution | `s` | +| Load balancers | `gcp.load_balancing.backend.requests` | `loadbalancing.googleapis.com/https/backend_request_count` | Counter | `{request}` | +| Load balancers | `gcp.load_balancing.network.io` (`receive`) | `loadbalancing.googleapis.com/https/request_bytes_count` | Counter | `By` | +| Load balancers | `gcp.load_balancing.network.io` (`transmit`) | `loadbalancing.googleapis.com/https/response_bytes_count` | Counter | `By` | + +Value conventions: + +- Counters are delta sums per minute. Chart them with `sum`, not `rate`. +- Distributions are gauges with a `quantile` attribute (`0.5`, `0.95`, `0.99`). +- Utilization is a fraction from 0 to 1. +- Durations, such as latency, execution time, message age and replica lag, are in seconds. + +Service names follow the same rule as logs: Cloud Run services, functions, GKE containers and Compute Engine instances use the workload name, and everything else is `gcp/`. Narrow further by resource attribute: + +| Service | Resource attributes | +| -------------- | ----------------------------------------------------------------------------- | +| GKE containers | `k8s.cluster.name`, `k8s.namespace.name`, `k8s.container.name` | +| GKE nodes | `k8s.cluster.name` | +| Compute Engine | `gcp.gce.instance.name`, `host.name`, `host.id` | +| Cloud SQL | `gcp.resource.labels.database_id` | +| Pub/Sub | `gcp.resource.labels.subscription_id`, `gcp.resource.labels.topic_id` | +| Load balancers | `gcp.resource.labels.url_map_name`, `gcp.resource.labels.backend_target_name` | + +Every series also carries `cloud.account.id` (the project ID) and, for resources with a location, `cloud.region`. Google aggregates away every other label, such as revision, pod, response code and device, before the data reaches Maple. These attributes are kept, Cloud Monitoring metric labels under `gcp.metric.labels.`: + +| Attribute | On | +| --------------------------------------- | ---------------------------------------------------------------------------------------------- | +| `gcp.metric.labels.response_code_class` | `gcp.cloud_run.requests`, `gcp.load_balancing.requests`, `gcp.load_balancing.backend.requests` | +| `gcp.metric.labels.state` | `gcp.cloud_run.container.instance.count`, `gcp.cloud_functions.instance.count` | +| `gcp.metric.labels.status` | `gcp.cloud_functions.executions` | +| `gcp.metric.labels.memory_type` | The GKE memory metrics | +| `gcp.metric.labels.response_class` | `gcp.pubsub.subscription.push_requests`, `gcp.pubsub.topic.publish_requests` | +| `network.io.direction` | The `network.io` metrics: `receive` or `transmit` | +| `disk.io.direction` | `gcp.gce.disk.io`, `gcp.cloud_sql.disk.operations`: `read` or `write` | + +### Resources + +Every hour Maple lists these resources from Cloud Asset Inventory, with their project, location, state and labels: projects, Cloud Run services and jobs, Cloud Functions, GKE clusters, Compute Engine instances, Cloud SQL instances, Pub/Sub topics and subscriptions, and load balancer URL maps, backend services and forwarding rules. + +## Google Cloud costs + +Google bills Pub/Sub and Cloud Monitoring usage to your account, separately from your Maple plan. + +Google bills Pub/Sub for the log entries that pass through the topic and the subscription. A narrower `LOG_FILTER` lowers it. See [Pub/Sub pricing](https://cloud.google.com/pubsub/pricing). + +Google bills Cloud Monitoring API reads to the host project. Each connection runs 46 `timeSeries.list` queries every 5 minutes, and the cost grows with the number of time series in the project, folder or organization. See [Google Cloud Observability pricing](https://cloud.google.com/stackdriver/pricing). + +Cloud Asset Inventory searches are free of charge. See [Cloud Asset Inventory pricing](https://cloud.google.com/asset-inventory/pricing). + +## Limits + +- Pub/Sub delivers one log entry per HTTP request, at least once. A redelivered entry is stored again. Its Cloud Logging `insertId` is in the `log.record.uid` attribute. +- The subscription keeps undelivered entries for one day. Entries Maple could not accept within that day are lost. +- Connecting an organization and a folder or project inside it collects that part twice. Each connection has its own sink and its own metric reads. +- Metrics arrive every 5 minutes, 5 minutes behind. The first read, and the first read after a pause, covers at most the last hour. +- One read takes up to 20,000 data points per metric, about 4,000 series, and up to 2 minutes in total. Past that, the connection says how many metric queries were cut short. For a folder or organization, connect its folders or projects separately to collect all of it. +- Maple reads up to 10 connections of one Maple organization every 5 minutes. With more, they take turns. +- The resource list holds up to 10,000 resources per connection. +- Load balancer metrics cover global external Application Load Balancers only. + +## Troubleshooting + +To run the setup script again, click **Configure** on the connection's card, then **Continue to the script**. When a failure asks for another run, the card also shows **The setup script needs to run again** with the button **Run the script again**. + +### In Cloud Shell + +A step that fails stops the script. It prints what went wrong, Google's answer, and a line that starts with `What to do:`. Nothing needs undoing: fix it and paste the script again. A connection's ID can't be changed, so a wrong ID is fixed by removing the connection in Maple and connecting the right ID. In the lines below, _account_ stands for the account you are signed in as. + +| The script prints | Cause and fix | +| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| "Can't open project acme-prod as _account_." | The ID is wrong, or the account has no access to the project. Check the ID with `gcloud projects list`. It is the project ID, not the name or number. If it is wrong, remove the connection and connect the right ID. | +| "_account_ has none of the permissions this script needs on project acme-prod." | Check the ID in Maple first. If it is wrong, remove the connection and connect the right ID. If it is right, the account has no rights there: ask for the roles under [Prerequisites](#prerequisites), or have an administrator run the script. | +| "_account_ is missing permissions on project acme-prod:" and a list | The account lacks a role. The line names the permissions and the role to ask for. If the only one listed is the permission to enable services, an API the script needs is off: ask for Service Usage Admin on the host project, or have an administrator switch the named APIs on. Nothing was created yet. | +| "Project acme-prod has no billing account." | Link a billing account to the host project, then paste the script again. | +| "gcloud _version_ is too old for this script." | Run the script in Cloud Shell, or update gcloud (`gcloud components update`, or the package manager that installed it). Versions before 470 can't update a folder or organization sink. | +| "_account_ can't see folder 123456789012." (or organization) | The ID is wrong, or the account has no rights there. Check it with `gcloud resource-manager folders list --organization=ORGANIZATION_ID` or `gcloud organizations list`. | +| "Couldn't look for _a resource_: its API is off in project acme-prod." | Cleanup only. The API was switched off after a setup run, so the resource may still be there. Switch the API on and paste the cleanup script again. | +| "Google does not accept the log filter." | `LOG_FILTER` was edited into something Google can't parse. Correct it near the top of the script and paste it again. Nothing was created yet. | +| "An organization policy (domain restricted sharing) blocks granting Maple's service account" | See [Domain restricted sharing](#domain-restricted-sharing). | +| "Google is still switching an API on, or the API is off." | Wait a minute and paste the script again. | +| "Google has not published the new service account yet." | Wait a minute and paste the script again. | +| "Couldn't reach Maple to confirm." | Cloud Shell could not reach Maple. Google Cloud is set up, and Maple shows **Setup pending** or **Changes pending** until a later run reaches it. Paste the script again. | + +When you create a connection in Maple, "The Google Cloud project acme-prod is already connected." means this Maple organization already has a connection for it. Click **Configure** on that connection instead. + +### Log forwarding + +| The connection shows | Cause and fix | +| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Setup pending** after the script ran | The script stopped before it finished, or could not reach Maple. Read its last lines in Cloud Shell and paste it again. | +| **Waiting for first logs** for more than 20 minutes | Either nothing was logged that passes the filter, or the sink can't publish. Run the `gcloud logging write` command the row shows. If the entry does not arrive within a minute, open **Configure** and run the setup script again. | +| **Rejecting logs**: "The Pub/Sub subscription wraps each entry in an envelope Maple can't read." | The subscription was changed to deliver wrapped messages. Run the setup script again: it resets the subscription. Entries sent meanwhile are lost. | +| **Rejecting logs**: "Maple could not store an entry just now." | Nothing to do. Pub/Sub retries the entry for up to a day, and the status returns to **Receiving logs** with the next accepted entry. | +| **Rejecting logs**: "This Maple organization is over its plan limit, so Maple refuses new logs." | Raise the plan limit under **Settings → Billing**. Pub/Sub retries refused entries for up to a day. | +| GKE container logs appear twice | The sink forwards GKE container logs and the workloads also send them over OpenTelemetry. In **Configure**, choose **Recommended: no GKE container logs** as the [log filter](#log-filter) and run the script again. | +| GKE container logs are missing | The recommended filter leaves them out. If the workloads don't send their logs over OpenTelemetry, choose **Include GKE container logs** as the [log filter](#log-filter) in **Configure** and run the script again. | + +### Metrics and resources + +**Can't read metrics** and **Receiving metrics, incomplete** come with one of the messages below. The first means no read has succeeded in the last 10 minutes. The second means the last read is under 10 minutes old and something is missing. The messages about the resource list show under a green **Receiving metrics**. Where a project's message says "the project", a folder's or organization's says "the host project". + +| The connection shows | Cause and fix | +| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Metrics stalled** | No read for 30 minutes and no error. Maple retries on its own. If it stays for more than an hour, write to support. | +| "Maple can't sign in as this connection's read-only service account yet." | Shown from 10 minutes after a setup run, on a connection Maple has never read. The account does not exist, or the grant to Maple is missing or blocked by an organization policy. Run the setup script again and read its last lines. See [Domain restricted sharing](#domain-restricted-sharing). | +| "Maple can no longer sign in as this connection's read-only service account." | Maple has read this connection before. The account was deleted, or the grant to Maple was removed or is blocked by an organization policy. Run the setup script again: it restores both, or says what blocks it. | +| "The project acme-prod has no active billing account" | Link a billing account to the host project. The message carries the link. Maple retries every 5 minutes. | +| "The Cloud Monitoring API is switched off in the project acme-prod." | Run the setup script again: it switches the API on. | +| "Google denied Maple's read of project acme-prod: the read-only roles are missing." | Run the setup script again: it grants them. A new grant can take a few minutes to work. | +| "Google rate-limited the Cloud Monitoring API for the project acme-prod." | Maple keeps what it read and retries in 5 minutes. If it repeats, raise that API's quota on the host project. | +| "3 of 46 metric queries failed, first ..." | The rest were stored. The failed metrics miss those minutes, and Maple retries in 5 minutes. | +| "3 of 46 metric queries were not read in full" | The project, folder or organization holds more series than one read takes. Connect the folders or projects of a folder or organization as separate connections. For a single project, write to support. | +| "Reading the metrics took longer than two minutes." | Maple reads the same minutes again, which can store part of them twice. If it repeats, connect the folders or projects of a folder or organization separately. For a single project, write to support. | +| "Metrics are paused: this Maple organization is over its plan limit." | Raise the plan limit under **Settings → Billing**. Maple tries again in an hour. | +| "Maple could not store the metrics it read just now." | Nothing to do. Maple reads the same minutes again. | +| "Google denied the resource listing for project acme-prod." | Run the setup script again: it grants Cloud Asset Viewer. Metrics are unaffected. | +| "The project holds more than 10,000 resources, so the resource list is incomplete." | Metrics are unaffected. For a full list, connect the folders or projects of a folder or organization separately. For a single project, write to support. | +| "The resource listing ran out of time and is incomplete." | Nothing to do. Maple retries within the hour. | + +A message that ends in parentheses, such as "(Cloud Monitoring returned 403, BILLING_DISABLED)", quotes Google's answer. Include it when you write to support. + +On a self-hosted Maple, the **Metrics and resources** checkbox reads **Not available on this Maple deployment** until `MAPLE_GCP_SERVICE_ACCOUNT_EMAIL` names a Google service account the deployment owns. Reading metrics also needs `MAPLE_GCP_SERVICE_ACCOUNT_KEY`, that account's key file, base64-encoded. + +### Domain restricted sharing + +An organization policy can restrict which identities may hold IAM roles in your organization (`constraints/iam.allowedPolicyMemberDomains`, or a custom constraint on member domains). Maple's service account lives outside your organization, so such a policy can reject a grant the script makes, most often the one that lets Maple read as your read-only service account. The script then stops with "An organization policy (domain restricted sharing) blocks granting Maple's service account". + +Don't turn the policy off or allow all principals for this. An Organization Policy Administrator can allow Maple's one service account on the host project, and nothing else: + +1. Use the managed constraint **Restrict allowed policy members in IAM allow policies** (`iam.managed.allowedPolicyMembers`). The older `iam.allowedPolicyMemberDomains` can't make an exception for a single principal. +2. On the host project, set a policy for it whose allowed principals are your organization's principal set plus Maple's service account (`serviceAccount:` followed by the address the script's error names). Listing your organization keeps everyone in it allowed on the project. Google's steps: [Restrict identities with domain-restricted sharing](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains). +3. Paste the setup script again. It continues where it stopped. + +Google checks the policy when a grant is made, so leave the exception in place: a later re-run grants again. + +## Disconnect + +1. On **Integrations → Google Cloud**, open the **⋮** menu (**More actions**) on the connection's card and choose **Disconnect**. +2. Click **Copy cleanup script** and run it in Cloud Shell. It deletes the log sink, topic, subscription and read-only service account, and ends with "Done. Everything the setup script created is gone." +3. Wait for the check mark in the dialog, then click **Disconnect**. Maple stops reading the connection's metrics, and stops accepting its logs within about a minute. Data already in Maple is kept. + +The cleanup takes one to two minutes when the connection forwards logs: after it deletes the sink it waits a minute for Google to stop routing to the topic, so your project logs no sink error. Turning **Log forwarding** off in **Configure** and running the setup script takes as long, for the same reason. + +**Disconnect anyway** disconnects without the cleanup. Google Cloud then keeps publishing logs to Pub/Sub, billed by Google, until the cleanup script runs. Maple keeps a panel with **Copy cleanup script** on the page until you click **Dismiss** and confirm. + +The cleanup script leaves the APIs it switched on enabled, and the Logs Writer role of Google's logging service account on the host project, which other sinks share. + +For a connection whose setup script Maple never saw run, **Disconnect** asks once and has no cleanup step. Maple still offers the cleanup script afterwards, in case the setup script ran part of the way. + +## Next steps + +- [Google Cloud with OpenTelemetry](/docs/integrations/gcp-opentelemetry): what to instrument on GKE, Cloud Run and Compute Engine. +- [Logs](/docs/explore/logs): search Google Cloud logs next to your application logs. +- [Dashboards](/docs/dashboards/build-dashboards#templates): start from the **Google Cloud** template. +- **Infrastructure → Google Cloud**: scan every workload of a service in one table. +- [Alert rules](/docs/alerting/alert-rules): alert on any `gcp.*` metric. +- [API reference](/docs/reference/api): manage connections with the `/v2/integrations/gcp` endpoints. diff --git a/apps/web/perf/check-bundle-budget.ts b/apps/web/perf/check-bundle-budget.ts index 78668ea33d..04f58a87cb 100644 --- a/apps/web/perf/check-bundle-budget.ts +++ b/apps/web/perf/check-bundle-budget.ts @@ -105,8 +105,11 @@ const gzipBytes = chunks.reduce((total, chunk) => total + chunk.gzipBytes, 0) // #1502) — the connector group's endpoints and schemas on the v2 contract // every page's client carries. Each Google Cloud step below keeps ~1 KB of // headroom, since CI measures the PR merged with main. -// 699.5 KB with main's 1 KB from #1494, #1500, #1502 merged up the stack. -const maxGzipBytes = 699.5 * 1024 +// 699.5 KB from the Google Cloud integration card (#1334, 2026-10-15): ~0.9 KB +// of startup measured in CI against #1333's 697.3 from the card and its +// contract additions. +// 700.5 KB with main's 1 KB from #1494, #1500, #1502 merged up the stack. +const maxGzipBytes = 700.5 * 1024 const budgetLabel = `${(maxGzipBytes / 1024).toFixed(1)} KB` // Anything lazy-only: chat, replay, and every dev-only lab surface. The diff --git a/apps/web/src/atoms/gcp-removed-connectors-atoms.ts b/apps/web/src/atoms/gcp-removed-connectors-atoms.ts new file mode 100644 index 0000000000..588db99d19 --- /dev/null +++ b/apps/web/src/atoms/gcp-removed-connectors-atoms.ts @@ -0,0 +1,28 @@ +import { Schema } from "effect" + +import { Atom } from "@/lib/effect-atom" +import { localStorageRuntime } from "@/lib/services/common/storage-runtime" + +/** + * A connection that was removed before its cleanup script was seen to run. Kept so the + * script stays reachable across reloads until the admin dismisses it. The script comes from the + * delete response, which carries no secret. + */ +const RemovedGcpConnector = Schema.Struct({ + id: Schema.String, + label: Schema.String, + hostProjectId: Schema.String, + cleanupScript: Schema.String, + /** Whether a script run ever reported for it: decides how the panel words itself. */ + reported: Schema.Boolean, +}) +export type RemovedGcpConnector = Schema.Schema.Type + +export const removedGcpConnectorsAtomFamily = Atom.family((orgId: string) => + Atom.kvs({ + runtime: localStorageRuntime, + key: `maple.gcp.removed-connectors.${orgId}`, + schema: Schema.Array(RemovedGcpConnector), + defaultValue: (): ReadonlyArray => [], + }), +) diff --git a/apps/web/src/components/icons/google-cloud.tsx b/apps/web/src/components/icons/google-cloud.tsx index 879c966dff..0bd319ba62 100644 --- a/apps/web/src/components/icons/google-cloud.tsx +++ b/apps/web/src/components/icons/google-cloud.tsx @@ -1,7 +1,26 @@ import type { IconProps } from "./icon" // Source: devicon (MIT) — https://github.com/devicons/devicon, `googlecloud-original` -function GoogleCloudIcon({ size = 24, className, ...props }: IconProps) { +const PATHS = [ + { + fill: "#ea4535", + d: "M80.6 40.3h.4l-.2-.2 14-14v-.3c-11.8-10.4-28.1-14-43.2-9.5C36.5 20.8 24.9 32.8 20.7 48c.2-.1.5-.2.8-.2 5.2-3.4 11.4-5.4 17.9-5.4 2.2 0 4.3.2 6.4.6.1-.1.2-.1.3-.1 9-9.9 24.2-11.1 34.6-2.6h-.1z", + }, + { + fill: "#557ebf", + d: "M108.1 47.8c-2.3-8.5-7.1-16.2-13.8-22.1L80 39.9c6 4.9 9.5 12.3 9.3 20v2.5c16.9 0 16.9 25.2 0 25.2H63.9v20h-.1l.1.2h25.4c14.6.1 27.5-9.3 31.8-23.1 4.3-13.8-1-28.8-13-36.9z", + }, + { + fill: "#36a852", + d: "M39 107.9h26.3V87.7H39c-1.9 0-3.7-.4-5.4-1.1l-15.2 14.6v.2c6 4.3 13.2 6.6 20.7 6.6z", + }, + { + fill: "#f9bc15", + d: "M40.2 41.9c-14.9.1-28.1 9.3-32.9 22.8-4.8 13.6 0 28.5 11.8 37.3l15.6-14.9c-8.6-3.7-10.6-14.5-4-20.8 6.6-6.4 17.8-4.4 21.7 3.8L68 55.2C61.4 46.9 51.1 42 40.2 42.1z", + }, +] + +function Glyph({ mono, size = 24, className, ...props }: IconProps & { mono: boolean }) { return ( ) } -export { GoogleCloudIcon } +function GoogleCloudIcon(props: IconProps) { + return +} + +/** The mark in `currentColor`, for surfaces that own the color (dim backer plates, buttons). */ +function GoogleCloudMonoIcon(props: IconProps) { + return +} + +export { GoogleCloudIcon, GoogleCloudMonoIcon } diff --git a/apps/web/src/components/icons/index.ts b/apps/web/src/components/icons/index.ts index ad04b8db19..459385827d 100644 --- a/apps/web/src/components/icons/index.ts +++ b/apps/web/src/components/icons/index.ts @@ -137,7 +137,7 @@ export { GearIcon } from "./gear" export { GeminiIcon } from "./gemini" export { GithubIcon } from "./github" export { GoogleIcon } from "./google" -export { GoogleCloudIcon } from "./google-cloud" +export { GoogleCloudIcon, GoogleCloudMonoIcon } from "./google-cloud" export { GrokIcon } from "./grok" export { HaystackIcon } from "./haystack" export { HuggingFaceIcon } from "./huggingface" diff --git a/apps/web/src/components/integrations/gcp-configure.tsx b/apps/web/src/components/integrations/gcp-configure.tsx new file mode 100644 index 0000000000..26c29b798b --- /dev/null +++ b/apps/web/src/components/integrations/gcp-configure.tsx @@ -0,0 +1,1859 @@ +import { Fragment, useId, useState } from "react" +import type React from "react" +import { Exit, Option } from "effect" +import type { V2GcpConnector } from "@maple/domain/http/v2" +import type { GcpLogFilter, GcpScopeType } from "@maple/domain/primitives" +import { Alert, AlertDescription } from "@maple/ui/components/ui/alert" +import { Button } from "@maple/ui/components/ui/button" +import { Checkbox } from "@maple/ui/components/ui/checkbox" +import { Collapsible, CollapsiblePanel, CollapsibleTrigger } from "@maple/ui/components/ui/collapsible" +import { ConfirmDialog } from "@maple/ui/components/ui/confirm-dialog" +import { CopyButton } from "@maple/ui/components/ui/copy-button" +import { + Dialog, + DialogClose, + DialogContent, + DialogFooter, + DialogHeader, + DialogPanel, + DialogTitle, +} from "@maple/ui/components/ui/dialog" +import { Eyebrow } from "@maple/ui/components/ui/eyebrow" +import { Field, FieldDescription, FieldError, FieldLabel } from "@maple/ui/components/ui/field" +import { InlineCode } from "@maple/ui/components/ui/inline-code" +import { Input } from "@maple/ui/components/ui/input" +import { Radio, RadioGroup } from "@maple/ui/components/ui/radio-group" +import { Skeleton } from "@maple/ui/components/ui/skeleton" +import { Spinner } from "@maple/ui/components/ui/spinner" +import { StatusDot } from "@maple/ui/components/ui/status-dot" +import { Switch } from "@maple/ui/components/ui/switch" +import { ToggleGroup, ToggleGroupItem } from "@maple/ui/components/ui/toggle-group" +import { countLabel } from "@maple/ui/lib/format" +import { cn } from "@maple/ui/lib/utils" + +import { RelativeTime } from "@/components/common/relative-time" +import { REPLAY_BLOCK_CLASS } from "@/components/common/replay-privacy" +import { + AlertWarningIcon, + CheckIcon, + ChevronDownIcon, + CircleCheckIcon, + CircleInfoIcon, + ExternalLinkIcon, + GoogleCloudIcon, + LockIcon, +} from "@/components/icons" +import { useIntervalRefresh } from "@/hooks/use-interval-refresh" +import { useAsyncAction } from "@/hooks/use-mutation-action" +import { docsUrl } from "@/lib/docs" +import { Result, useAtomRefresh, useAtomSet, useAtomValue } from "@/lib/effect-atom" +import { errorMessage, showErrorToast } from "@/lib/error-toast" +import { MapleApiV2AtomClient } from "@/lib/services/common/v2-atom-client" +import { + GCP_LOG_STATUS, + GCP_METRICS_STATUS, + GCP_SCOPE_NAMES, + cloudShellUrl, + gcpCollectLock, + gcpCreateRequest, + gcpDraftEffect, + gcpLogFilterChoice, + gcpLogFilters, + gcpLogState, + gcpMetricsState, + gcpOverlapNote, + gcpScopeRoles, + gcpScriptNeeded, + gcpScriptOverdue, + gcpSetupRunning, + isGcpProjectId, + isGcpResourceNumber, + logRouterUrl, + type GcpCapability, + type GcpCollectLock, + type GcpConnectorDraft, + type GcpDraftEffect, + type GcpFlags, +} from "./gcp-connector-state" +import { gcpStatusQuery } from "./integration-catalog" + +export const GCP_REACTIVITY_KEYS = ["gcpIntegration"] +/** Invalidated by a saved configuration: the script depends on what the connector collects. */ +const SCRIPT_REACTIVITY_KEYS = ["gcpSetupScripts"] + +/** Fast enough to see a script run confirmed, the first data land, or an error clear. */ +export const GCP_SETTLING_REFRESH_MS = 10_000 + +const DOCS = docsUrl("gcp") +const OTEL_DOCS = docsUrl("gcpOpenTelemetry") + +const PROJECT_ID_RULE = + "A project ID is 6 to 30 lowercase letters, digits and hyphens. It starts with a letter and can't end with a hyphen. The project name and number don't work." + +/** The scope choices, in display order: most connect one project. */ +const SCOPES = { + project: { + description: "One project.", + idLabel: "Project ID", + idRule: PROJECT_ID_RULE, + placeholder: "acme-prod", + hint: "The ID, not the name or number.", + listCommand: "gcloud projects list", + }, + folder: { + description: "Every project in the folder, including new ones.", + idLabel: "Folder ID", + idRule: "A folder ID is digits only, such as 123456789012.", + placeholder: "123456789012", + hint: "Digits only, with your organization's ID in place of ORGANIZATION_ID.", + listCommand: "gcloud resource-manager folders list --organization=ORGANIZATION_ID", + }, + organization: { + description: "Every project in the organization, including new ones.", + idLabel: "Organization ID", + idRule: "An organization ID is digits only, such as 123456789012.", + placeholder: "123456789012", + hint: "Digits only.", + listCommand: "gcloud organizations list", + }, +} as const +const SCOPE_TYPES: ReadonlyArray = ["project", "folder", "organization"] + +const LOCK_NOTES = { + "last-on": "A connection collects at least one of the two. To stop collecting, disconnect.", + "metrics-unavailable": "Not available on this Maple deployment.", +} as const satisfies { readonly [Lock in GcpCollectLock]: string } + +const CAPABILITIES = { + logs: { + title: "Log forwarding", + flag: "logs_enabled", + description: "Request, audit and managed-service logs, through a Log Router sink and Pub/Sub.", + run: { on: "Log sink and Pub/Sub topic", off: "Removes the log sink" }, + effects: { + "starts-after-script": "Starts once the script has run in Google Cloud.", + "resumes-now": + "Maple stores these logs again as soon as you save. Google Cloud still forwards them.", + "stops-now": + "Maple stops storing these logs within about a minute of saving. Google Cloud keeps forwarding them until the script has run.", + "removed-by-script": "Off in Maple. Google Cloud keeps forwarding until the script has run.", + }, + now: { + "stops-now": "Stops storing this connection's logs within about a minute.", + "resumes-now": "Stores this connection's logs again, at once.", + }, + later: { + "stops-now": "Keeps forwarding them to Pub/Sub, billed by Google, until the script has run.", + "resumes-now": "Nothing to run: the log sink is still there and forwarding.", + }, + }, + metrics: { + title: "Metrics and resources", + flag: "metrics_enabled", + description: + "Cloud Run, GKE, Compute Engine, Cloud SQL, Pub/Sub, Load Balancing and Cloud Functions. Read-only, every 5 minutes.", + run: { on: "Read-only service account", off: "Removes the service account" }, + effects: { + "starts-after-script": "Starts once the script has run in Google Cloud.", + "resumes-now": + "Maple reads metrics again from the next 5-minute read. Its service account is still there.", + "stops-now": + "Maple stops reading from the next 5-minute read. Its service account stays in Google Cloud until the script has run.", + "removed-by-script": + "Off in Maple. Its service account stays in Google Cloud until the script has run.", + }, + now: { + "stops-now": "Stops reading this connection's metrics and resources from the next 5-minute read.", + "resumes-now": "Reads this connection's metrics and resources again from the next 5-minute read.", + }, + later: { + "stops-now": "Keeps the read-only service account and its roles until the script has run.", + "resumes-now": "Nothing to run: the read-only service account is still there.", + }, + }, +} as const satisfies { + readonly [Capability in GcpCapability]: { + readonly title: string + readonly flag: keyof GcpFlags + readonly description: string + /** What the script sets up for it, or removes, in the run step's status. */ + readonly run: { readonly on: string; readonly off: string } + readonly effects: { readonly [Effect in GcpDraftEffect]: string } + /** What a choice that acts at once does when it is saved, and what stays until the script runs. */ + readonly now: { readonly [Effect in "stops-now" | "resumes-now"]: string } + readonly later: { readonly [Effect in "stops-now" | "resumes-now"]: string } + } +} +const CAPABILITY_IDS: ReadonlyArray = ["logs", "metrics"] + +/** The log filters' names, short in the closed disclosure and the run summary, and what each does. */ +const FILTER_TEXT = { + default: { + short: "Recommended", + title: "Recommended", + description: + "Request, audit and managed-service logs. Leaves out GKE container logs and health-check noise.", + }, + include_gke_container_logs: { + short: "With GKE container logs", + title: "Include GKE container logs", + description: "Everything Recommended forwards, plus GKE container logs.", + }, + keep: { + short: "Current filter", + title: "Keep the current filter", + description: "The filter the log sink has now. Maple doesn't store it.", + }, +} as const satisfies { + readonly [Filter in GcpLogFilter]: { + readonly short: string + readonly title: string + readonly description: string + } +} + +/** The page's one link style, the other integrations' (Railway's token link). */ +export const GCP_LINK = "underline underline-offset-2 hover:no-underline" +/** Small print: one size, one colour, short lines. */ +const SMALL = "text-xs/5 text-pretty text-muted-foreground" +/** A section's or a step's name. On a phone it grows with the controls. */ +export const GCP_TITLE = "text-base/6 font-medium sm:text-sm/5" + +/** A named link stays on one line. A bare address may break. */ +export function GcpExternalLink({ href, children }: { href: string; children: React.ReactNode }) { + return ( + + {children} + {/* A word joiner in a no-wrap span: the icon never wraps away from the last word. */} + + {"⁠"} + + + + ) +} + +/** An identifier inside a sentence: mono without a chip, so punctuation sits tight against it. */ +function Mono({ children }: { children: React.ReactNode }) { + return {children} +} + +/** + * A shell command to copy: a code chip with its copy button, on one line where the width allows. + * Too long for its line, it breaks between its words and never inside one: a flag or a name split + * at a hyphen reads as another. + */ +export function GcpCommand({ children }: { children: string }) { + return ( + + + {children.split(" ").map((word, index) => ( + + {index === 0 ? null : " "} + {word} + + ))} + + + + ) +} + +/** Filled where opening Cloud Shell is a step of its own, outlined next to a copy button. */ +export function GcpOpenCloudShellButton({ + projectId, + variant = "outline", +}: { + projectId: string + variant?: "default" | "outline" +}) { + return ( + + + {failure === null ? null : ( +
+

+ {failure.message} +

+ +
+ )} +

+ + Holds this connection's secret. Don't share or commit it. +

+ {!shown || failure !== null ? null : text === null ? ( + + ) : ( + // The frame of the app's code blocks: a label strip over the text. +
+
+ Bash + + {countLabel(text.split("\n").length, "line")} + +
+ {/* The dashboard records itself with rrweb, which serializes plain text verbatim. */} + 
+						{text}
+
+
+ )} + + ) +} + +type StepState = "current" | "pending" | "waiting" | "done" + +/** + * One step of a setup or cleanup: a numbered marker on a line that runs to the next step, the + * title with one line under it, then the controls. Only a step to act on now is bright; one that + * waits on Google Cloud spins, and a finished one is checked. + */ +function Step({ + number, + state, + title, + detail, + children, +}: { + number: number + state: StepState + title: string + detail?: React.ReactNode + children?: React.ReactNode +}) { + return ( +
  • + {state === "done" ? ( + + ) : state === "waiting" ? ( + + ) : ( + + {number} + + )} +
    +
    +

    {title}

    + {detail === undefined ? null :

    {detail}

    } +
    + {children} +
    +
    • + ) +} + +/** + * A connector's scripts. A bare query on purpose: `retainedQueryV2` keeps results past unmount, + * and both scripts embed the connector's secret, so they live only while they are on screen. + */ +function useGcpScripts(connector: V2GcpConnector, logFilter: GcpLogFilter) { + const query = MapleApiV2AtomClient.query("gcpIntegration", "setupScripts", { + params: { id: connector.id }, + payload: { log_filter: logFilter }, + reactivityKeys: SCRIPT_REACTIVITY_KEYS, + }) + const result = useAtomValue(query) + const retry = useAtomRefresh(query) + return { + // `waiting` is the script for the previous filter or configuration, held over while the new + // one loads. Copying that would set up the wrong thing, so only a settled result counts. + scripts: Result.isSuccess(result) && !result.waiting ? result.value : null, + failure: Result.isFailure(result) + ? { + message: errorMessage(result.cause, "Couldn't load the script."), + retry, + retrying: result.waiting, + } + : null, + } +} + +export interface ScriptFailure { + readonly message: string + readonly retry: () => void + readonly retrying: boolean +} + +/** The log filter the script is asked for, and whether including GKE container logs was confirmed. */ +interface FilterState { + readonly chosen: GcpLogFilter | null + readonly acknowledged: boolean +} + +const STEPS = ["Scope", "Data", "Run setup"] as const +/** The step the flow is on. */ +type StepIndex = 0 | 1 | 2 + +/** Where the flow is: done steps are checked, the current one is filled, later ones are outlined. */ +function Stepper({ current }: { current: StepIndex }) { + return ( +
      + {STEPS.map((label, index) => { + const done = index < current + const active = index === current + return ( +
    1. 0 && "flex-1")} + > + {index === 0 ? null : ( + + )} + + + {done ? : index + 1} + + + {label} + {done ? (done) : null} + + +
      2. + ) + })} +
    + ) +} + +/** `step` is undefined once every step is done: the finished flow shows no stepper. */ +function ConfigureHeader({ title, step }: { title: React.ReactNode; step?: StepIndex }) { + return ( + + + + + + {title} + + {step === undefined ? null : } + + ) +} + +/** One short line at the footer's start: why the action waits, or what closing leaves behind. */ +function FooterNote({ children }: { children: React.ReactNode }) { + return

    {children}

    +} + +/** + * What to collect: a switch per capability, with the log filter under Logs. Nothing is saved here. + * `effect` says, for a connection, what saving a choice does and when. + */ +function CollectFields({ + flags, + onChange, + metricsAvailable, + effect, + logFilter, +}: { + flags: GcpFlags + onChange: (flags: GcpFlags) => void + metricsAvailable: boolean + effect?: (capability: GcpCapability) => GcpDraftEffect | null + logFilter: React.ReactNode +}) { + const id = useId() + const locks = CAPABILITY_IDS.map((capability) => gcpCollectLock(flags, capability, metricsAvailable)) + return ( +
    + What should Maple collect? +
    + {CAPABILITY_IDS.map((capability, index) => { + const { title, flag, description, effects } = CAPABILITIES[capability] + const note = effect?.(capability) ?? null + return ( +
    +
    +
    + +

    + {locks[index] === "metrics-unavailable" + ? LOCK_NOTES["metrics-unavailable"] + : description} +

    + {note === null ? null : ( +

    + {note === "stops-now" ? ( + + ) : ( + + )} + {effects[note]} +

    + )} +
    + onChange({ ...flags, [flag]: checked })} + /> +
    + {capability === "logs" && flags.logs_enabled ? logFilter : null} +
    + ) + })} +
    + {/* A rule of the pair, so it stands under both and not under the one it happens to lock. */} + {locks.includes("last-on") ?

    {LOCK_NOTES["last-on"]}

    : null} +
    + ) +} + +/** + * The filter the script gives the log sink, closed on its current choice. Including GKE container + * logs asks to be confirmed, and stays open until it is. + */ +function LogFilter({ + sinkExists, + filter, + onFilter, + logRouter, +}: { + sinkExists: boolean + filter: FilterState + onFilter: (filter: FilterState) => void + /** The console page that shows an existing sink's filter. */ + logRouter?: string +}) { + const checkboxId = useId() + const [open, setOpen] = useState(false) + const filters = gcpLogFilters(sinkExists) + const choice = gcpLogFilterChoice(filter.chosen, filter.acknowledged, sinkExists) + return ( + + + Log filter + {FILTER_TEXT[choice.selected].short} + + + + { + const next = filters.find((option) => option.value === value) + if (next !== undefined) onFilter({ chosen: next.value, acknowledged: false }) + }} + > + {filters.map(({ value }) => ( + + ))} + + {choice.selected !== "include_gke_container_logs" ? null : ( +
    +

    + + + Workloads instrumented with OpenTelemetry already send these logs to Maple, + linked to their traces. Including them stores each line twice.{" "} + + Google Cloud with OpenTelemetry + + +

    + +
    + )} +

    + {sinkExists && choice.selected !== "keep" + ? "Replaces the sink's current filter when the script runs. " + : null} + Write your own filter +

    +     +     + ) +} + +/** The scope step: what kind of thing to connect, and its ID. */ +function ScopeFields({ + draft, + onDraft, + touched, + onTouched, + existing, +}: { + draft: GcpConnectorDraft + onDraft: (patch: Partial) => void + touched: ScopeTouched + onTouched: (touched: ScopeTouched) => void + existing: ReadonlyArray +}) { + const headingId = useId() + const scope = SCOPES[draft.scopeType] + const scopeName = GCP_SCOPE_NAMES[draft.scopeType].toLowerCase() + const aggregated = draft.scopeType !== "project" + const scopeId = draft.scopeId.trim() + const hostProjectId = draft.hostProjectId.trim() + // A rule turns red once its field was left, not while the first characters are being typed. + const scopeIdInvalid = + touched.scopeId && + scopeId.length > 0 && + !(aggregated ? isGcpResourceNumber(scopeId) : isGcpProjectId(scopeId)) + const hostInvalid = + touched.hostProjectId && aggregated && hostProjectId.length > 0 && !isGcpProjectId(hostProjectId) + const overlap = gcpOverlapNote(draft.scopeType, existing) + + return ( + <> +
    +

    + What should Maple watch? +

    + { + const next = SCOPE_TYPES.find( + (scopeType) => scopeType !== draft.scopeType && values.includes(scopeType), + ) + if (next === undefined) return + // The two kinds of ID never carry over, so the field starts over. + onDraft({ scopeType: next, scopeId: "" }) + onTouched({ ...touched, scopeId: false }) + }} + className="w-full" + > + {SCOPE_TYPES.map((scopeType) => ( + + {GCP_SCOPE_NAMES[scopeType]} + + ))} + +

    {scope.description}

    +
    + + {scope.idLabel} + + {scope.hint} To list them, run {scope.listCommand} + + onDraft({ scopeId: event.target.value })} + onBlur={() => onTouched({ ...touched, scopeId: true })} + className="font-mono" + /> + {scopeIdInvalid ? ( + + {scope.idRule} + + ) : null} + + {aggregated ? ( + + Host project ID + + Holds Maple's Pub/Sub topic and read-only service account. A project inside the{" "} + {scopeName} with billing enabled that won't be deleted. + + onDraft({ hostProjectId: event.target.value })} + onBlur={() => onTouched({ ...touched, hostProjectId: true })} + className="font-mono" + /> + {hostInvalid ? ( + + {PROJECT_ID_RULE} + + ) : null} + + ) : null} + {overlap === null ? null : ( + + + {overlap} + + )} + + ) +} + +interface ScopeTouched { + readonly scopeId: boolean + readonly hostProjectId: boolean +} + +/** + * A new connection's first two steps: the scope, then what to collect. Creating it leads into the + * run step, where the script is. + */ +function NewConnection({ + title, + metricsAvailable, + existing, + filter, + onFilter, + onCreated, +}: { + title: React.ReactNode + metricsAvailable: boolean + existing: ReadonlyArray + filter: FilterState + onFilter: (filter: FilterState) => void + onCreated: (connector: V2GcpConnector) => void +}) { + const create = useAtomSet(MapleApiV2AtomClient.mutation("gcpIntegration", "createConnector"), { + mode: "promiseExit", + }) + const [step, setStep] = useState<"scope" | "data">("scope") + const [draft, setDraft] = useState({ + scopeType: "project", + scopeId: "", + hostProjectId: "", + logsEnabled: true, + metricsEnabled: metricsAvailable, + }) + const [error, setError] = useState(null) + const [touched, setTouched] = useState({ scopeId: false, hostProjectId: false }) + const formId = useId() + + // Decoded with the API's own schema, so a rejected ID is caught before the request. + const request = gcpCreateRequest(draft) + const aggregated = draft.scopeType !== "project" + const scopeId = draft.scopeId.trim() + const scopeReady = aggregated + ? isGcpResourceNumber(scopeId) && isGcpProjectId(draft.hostProjectId.trim()) + : isGcpProjectId(scopeId) + const unacknowledged = + draft.logsEnabled && gcpLogFilterChoice(filter.chosen, filter.acknowledged, false).unacknowledged + + const edit = (patch: Partial) => { + setDraft({ ...draft, ...patch }) + setError(null) + } + + const [submit, submitting] = useAsyncAction(async () => { + if (Option.isNone(request)) return + setError(null) + // The client types the request as one object per payload variant, so the union has to be + // narrowed before it is passed; both branches make the same call. + const payload = request.value + const result = await (payload.scope_type === "project" + ? create({ payload, reactivityKeys: GCP_REACTIVITY_KEYS }) + : create({ payload, reactivityKeys: GCP_REACTIVITY_KEYS })) + if (Exit.isSuccess(result)) { + onCreated(result.value) + return + } + // An already-connected scope answers with a message naming it; keep that on screen. + setError(errorMessage(result, "Couldn't connect. Try again.")) + }) + + function handleSubmit(event: React.FormEvent) { + event.preventDefault() + if (step === "scope") { + if (scopeReady) setStep("data") + return + } + if (!submitting && !unacknowledged) void submit() + } + + return ( + <> + + +
    + {step === "scope" ? ( + + ) : ( + + edit({ + logsEnabled: flags.logs_enabled, + metricsEnabled: flags.metrics_enabled, + }) + } + metricsAvailable={metricsAvailable} + logFilter={} + /> + )} + {error !== null ? ( +

    + {error} +

    + ) : null} + +
    + + {step === "data" && unacknowledged ? ( + Confirm the warning to continue. + ) : null} + {step === "scope" ? ( + }>Cancel + ) : ( + + )} + + + + ) +} + +/** A connection's data step: what it collects and its log filter. */ +function ChooseStage({ + title, + connector, + metricsAvailable, + filter, + onFilter, + onApply, +}: { + title: React.ReactNode + connector: V2GcpConnector + metricsAvailable: boolean + filter: FilterState + onFilter: (filter: FilterState) => void + /** Leads into the run step, with what was saved if anything changed. */ + onApply: (saved: GcpFlags | null) => void +}) { + const update = useAtomSet(MapleApiV2AtomClient.mutation("gcpIntegration", "updateConnector"), { + mode: "promiseExit", + }) + const [draft, setDraft] = useState({ + logs_enabled: connector.logs_enabled, + metrics_enabled: connector.metrics_enabled, + }) + const [confirming, setConfirming] = useState(false) + const sinkExists = connector.applied_logs_enabled === true + const unacknowledged = + draft.logs_enabled && + gcpLogFilterChoice(filter.chosen, filter.acknowledged, sinkExists).unacknowledged + const changed = CAPABILITY_IDS.filter( + (capability) => draft[CAPABILITIES[capability].flag] !== connector[CAPABILITIES[capability].flag], + ) + const effect = (capability: GcpCapability) => + gcpDraftEffect(connector, capability, draft[CAPABILITIES[capability].flag]) + // What acts in Maple the moment it is saved: that asks first. + const immediate = CAPABILITY_IDS.flatMap((capability) => { + const acts = effect(capability) + return acts === "stops-now" || acts === "resumes-now" ? [{ capability, acts }] : [] + }) + const stopping = immediate.filter(({ acts }) => acts === "stops-now") + const resuming = immediate.filter(({ acts }) => acts === "resumes-now") + const named = (list: typeof immediate) => + list.map(({ capability }) => CAPABILITIES[capability].title.toLowerCase()).join(" and ") + // One save can turn one capability off and the other back on: the question names both. + const turned = [ + stopping.length > 0 ? `${named(stopping)} off` : null, + resuming.length > 0 ? `${named(resuming)} back on` : null, + ] + .filter((part) => part !== null) + .join(" and ") + // Whether the script has anything to do once this is saved. Turning back on what still exists leaves nothing. + const scripted = CAPABILITY_IDS.some((capability) => { + const acts = effect(capability) + return acts !== null && acts !== "resumes-now" + }) + + // Only what changed is sent: the API leaves an omitted capability as it is, so a stale view of + // the other cannot overwrite it. Resolving to `false` keeps the confirmation open for a retry. + const [save, saving] = useAsyncAction(async () => { + const result = await update({ + params: { id: connector.id }, + payload: + changed.length === 2 + ? draft + : changed[0] === "logs" + ? { logs_enabled: draft.logs_enabled } + : { metrics_enabled: draft.metrics_enabled }, + reactivityKeys: [...GCP_REACTIVITY_KEYS, ...SCRIPT_REACTIVITY_KEYS], + }) + if (Exit.isFailure(result)) { + showErrorToast(result, { title: "Failed to save the change" }) + return false + } + onApply(draft) + return true + }) + + return ( + <> + + + + } + /> + + + {unacknowledged ? ( + Confirm the warning to continue. + ) : changed.length > 0 && !scripted ? ( + Nothing to run after this: Google Cloud still has what it needs. + ) : null} + }>Cancel + + + 0 ? "destructive" : "default"} + icon={stopping.length > 0 ? undefined : null} + title={ + <> + {`Turn ${turned}`} for {GCP_SCOPE_NAMES[connector.scope_type]}{" "} + {connector.scope_id}? + + } + // Turning off leaves something for the script to remove; turning back on leaves nothing to run. + confirmLabel={ + stopping.length === 0 + ? "Turn on" + : resuming.length === 0 + ? "Turn off and continue" + : "Save and continue" + } + onConfirm={save} + > +
    +
    +
    In Maple
    + {immediate.map(({ capability, acts }) => ( +
    + {CAPABILITIES[capability].now[acts]} +
    + ))} +
    +
    +
    In Google Cloud
    + {immediate.map(({ capability, acts }) => ( +
    + {CAPABILITIES[capability].later[acts]} +
    + ))} +
    +
    +     + + ) +} + +/** + * When a connection's run step was entered. `rerun`: nobody has said a run is under way, so + * the step stays open until a newer run reports and its status does not time one. + */ +export interface ApplyOpened { + readonly at: number + readonly rerun: boolean + /** The report on record at that moment: a different one is a newer run. */ + readonly seen: string | null +} + +const Chip = ({ children }: { children: React.ReactNode }) => ( + + {children} + +) + +/** What the script applies, so it is never a black box, and the way back to change it. */ +function RunSummary({ + connector, + logFilter, + onEdit, +}: { + connector: V2GcpConnector + logFilter: GcpLogFilter + onEdit: () => void +}) { + const capabilities = [ + connector.logs_enabled + ? `Logs · ${FILTER_TEXT[logFilter].short}` + : connector.applied_logs_enabled === true + ? "Logs · removed by this run" + : null, + connector.metrics_enabled + ? "Metrics" + : connector.applied_metrics_enabled === true + ? "Metrics · removed by this run" + : null, + ].filter((label) => label !== null) + return ( +
    + + {GCP_SCOPE_NAMES[connector.scope_type]} + {connector.scope_id} + + {connector.scope_type === "project" ? null : ( + + Host + {connector.project_id} + + )} + {capabilities.map((label) => ( + {label} + ))} + +
    + ) +} + +/** + * The setup script: copied, or run straight from Cloud Shell. Both scripts are a few hundred lines, + * so the card shows the start and opens the rest on request. + */ +function ScriptCard({ + script, + failure, + projectId, +}: { + script: string | null + failure: ScriptFailure | null + projectId: string +}) { + const [expanded, setExpanded] = useState(false) + const text = script?.trimEnd() ?? null + return ( +
    +
    + + setup.sh{text === null ? null : ` · ${countLabel(text.split("\n").length, "line")}`} + + + + + +
    + {failure !== null ? ( +
    +

    + {failure.message} +

    + +
    + ) : text === null ? ( +
    + +
    + ) : ( +
    + {/* The dashboard records itself with rrweb, which serializes plain text verbatim. */} + 
    +						{text}
+
    + {expanded ? null : ( +
    + )} +
    + )} +
    + + + Holds this connection's secret. Don't share or commit it. + + {text === null || failure !== null ? null : ( + + )} +
    +
    + ) +} + +type RunRowState = "pending" | "running" | "done" + +function RunRowMarker({ state }: { state: RunRowState }) { + return state === "done" ? ( + + ) : state === "running" ? ( + + ) : ( + + ) +} + +/** + * What Maple has heard from the script, kept in view under the script: a row per capability the + * run sets up or removes, ticked as its section reports. Past five minutes, what to check. + */ +function RunStatus({ + connector, + opened, + overdue, + nowMs, +}: { + connector: V2GcpConnector + opened: ApplyOpened + overdue: boolean + nowMs: number +}) { + // A row ticks on a report made since the step opened, not on what an earlier run left in place. + const reported = connector.setup_reported_at !== opened.seen + const running = gcpSetupRunning(connector, nowMs) + const rows = CAPABILITY_IDS.flatMap((capability) => { + const { title, flag, run } = CAPABILITIES[capability] + const wanted = connector[flag] + const applied = + capability === "logs" ? connector.applied_logs_enabled : connector.applied_metrics_enabled + if (!wanted && applied !== true) return [] + const state: RunRowState = applied === wanted && reported ? "done" : running ? "running" : "pending" + return [{ capability, title, detail: wanted ? run.on : run.off, state }] + }) + const done = rows.filter((row) => row.state === "done").length + const reporting = done > 0 || running + + return ( +
    +
    + {overdue ? ( + + ) : ( + + )} +

    + {overdue + ? "No report from the script yet" + : reporting + ? "The script is reporting back" + : "Waiting for the script to report back"} +

    + + {reporting ? `${done} of ${rows.length} done` : "updates on its own"} + +
    + {overdue ? ( +
    +
      +
    • + Stopped with an error? Fix + what it names and paste it again. It continues where it stopped. +
      • +
    • + Wrong ID? Disconnect this + connection and connect the right one. An ID can't be changed. +
      • +
    +

    + Troubleshooting +

    +
    + ) : ( +
      + {rows.map((row) => ( +
    • + + {row.title} + {row.detail} +
      • + ))} +
    + )} +
    + ) +} + +/** The run confirmed: what each capability is doing now, live, so the first data is seen landing. */ +function Applied({ + connector, + created, + nowMs, +}: { + connector: V2GcpConnector + created: boolean + nowMs: number +}) { + const log = gcpLogState(connector, nowMs) + const metrics = gcpMetricsState(connector, nowMs) + const rows = [ + log.kind === "off" + ? null + : { + title: "Logs", + status: GCP_LOG_STATUS[log.kind], + detail: log.kind === "waiting" ? "First entries within about 10 minutes" : null, + }, + metrics.kind === "off" + ? null + : { + title: "Metrics", + status: GCP_METRICS_STATUS[metrics.kind], + detail: metrics.kind === "waiting" ? "First read within about 10 minutes" : null, + }, + ].filter((row) => row !== null) + + return ( +
    + + + +
    +

    + {created ? ( + <> + {connector.scope_id} is connected + + ) : ( + "Google Cloud matches this configuration" + )} +

    +

    The script finished. Data shows up in Maple on its own.

    +
    +
      + {rows.map((row) => ( +
    • + {row.title} + {row.detail} + + + {row.status.label} + +
      • + ))} +
    +
    + ) +} + +/** A run's sections report one after the other: confirmed once the last one has. */ +const runConfirmed = (connector: V2GcpConnector, opened: ApplyOpened, nowMs: number) => + gcpScriptNeeded(connector, nowMs) === null && + !gcpSetupRunning(connector, nowMs) && + (!opened.rerun || connector.setup_reported_at !== opened.seen) + +/** The run step: loads the script for the chosen filter and polls until Maple confirms the run. */ +function ApplyStage({ + filter, + ...view +}: Omit & { filter: FilterState }) { + const { connector, opened, nowMs } = view + const choice = gcpLogFilterChoice( + filter.chosen, + filter.acknowledged, + connector.applied_logs_enabled === true, + ) + const { scripts, failure } = useGcpScripts(connector, choice.scriptFilter) + const refresh = useAtomRefresh(gcpStatusQuery()) + useIntervalRefresh(refresh, { + intervalMs: GCP_SETTLING_REFRESH_MS, + enabled: !runConfirmed(connector, opened, nowMs), + }) + return ( + + ) +} + +export interface GcpApplyViewProps { + title: React.ReactNode + connector: V2GcpConnector + /** Created in this surface: the confirmed run reads as the connection being made. */ + created: boolean + logFilter: GcpLogFilter + script: string | null + failure: ScriptFailure | null + nowMs: number + opened: ApplyOpened + onBack: () => void + onClose: () => void +} + +/** The run step as it looks: the script and the run's status, or the confirmed run. Exported for the lab. */ +export function GcpApplyView({ + title, + connector, + created, + logFilter, + script, + failure, + nowMs, + opened, + onBack, + onClose, +}: GcpApplyViewProps) { + const sinkExists = connector.applied_logs_enabled === true + const reportedAt = connector.setup_reported_at + const confirmed = runConfirmed(connector, opened, nowMs) + const overdue = !opened.rerun && gcpScriptOverdue(connector, opened.at, nowMs) + + const scopeRoles = gcpScopeRoles(connector.scope_type, connector) + const scopeName = GCP_SCOPE_NAMES[connector.scope_type].toLowerCase() + const role = (name: string) => {name} + + if (confirmed) { + return ( + <> + + + + + + {reportedAt === null ? null : ( + + + + )} + + + + ) + } + + return ( + <> + + + +
    +
    +

    Run this script in Cloud Shell

    +

    Nothing changes in Google Cloud until it runs.

    +
    + +

    + Run it as an {role("Owner")} of{" "} + {connector.scope_type === "project" ? null : "the host project "} + {connector.project_id} + {scopeRoles.map((name, index) => ( + + {index === 0 ? " who also has " : " and "} + {role(name)} + + ))} + {scopeRoles.length > 0 ? ` on the ${scopeName}` : null}.{" "} + {!connector.logs_enabled && sinkExists + ? "Takes one to two minutes. " + : "Takes about a minute and is safe to run again. "} + + Roles if you aren't an Owner + +

    +
    +     + + + Saved. Close this any time and come back to it. + + + + ) +} + +/** The dialog every step renders in. */ +export function GcpConfigureFrame({ onClose, children }: { onClose: () => void; children: React.ReactNode }) { + return ( + (open ? undefined : onClose())}> + {children} + + ) +} + +/** What the configuration surface is open on. */ +export type GcpConfigureTarget = + | { readonly kind: "new" } + | { readonly kind: "connection"; readonly id: string; readonly stage: "choose" | "apply" } + +/** A connection's two steps. Admin-only: fetching the script is refused for everyone else. */ +function ConnectionStages({ + title, + connector: saved, + created, + initialStage, + metricsAvailable, + filter, + onFilter, + nowMs, + onClose, +}: { + title: React.ReactNode + connector: V2GcpConnector + /** Created in this surface a moment ago: the script has not run and a run is expected. */ + created: boolean + initialStage: "choose" | "apply" + metricsAvailable: boolean + filter: FilterState + onFilter: (filter: FilterState) => void + nowMs: number + onClose: () => void +}) { + // What was just saved, shown until the status read delivers a newer connector. Without it the + // run step would read the old configuration for the length of the refetch. + const [asked, setAsked] = useState<{ readonly of: V2GcpConnector; readonly flags: GcpFlags } | null>(null) + const connector = asked !== null && asked.of === saved ? { ...saved, ...asked.flags } : saved + const open = (rerun: boolean): ApplyOpened => ({ + at: Date.now(), + rerun, + seen: saved.setup_reported_at, + }) + // Found waiting on a first run, the status times it from the connection's creation. Found + // any other way, nobody has said a run is under way. + const [opened, setOpened] = useState(() => + initialStage === "apply" + ? open(!created && gcpScriptNeeded(saved, Date.now()) !== "setup-pending") + : null, + ) + + return opened === null ? ( + { + if (flags !== null) setAsked({ of: saved, flags }) + setOpened(open(flags === null)) + }} + /> + ) : ( + setOpened(null)} + onClose={onClose} + /> + ) +} + +/** + * The one place a connection is configured, as three steps: the scope, what to collect with its log + * filter, then the script that applies it in Google Cloud. A connection reopens on the last two. + */ +export function GcpConfigure({ + target, + connectors, + metricsAvailable, + nowMs, + onCreated, + onClose, +}: { + target: GcpConfigureTarget + connectors: ReadonlyArray + metricsAvailable: boolean + nowMs: number + onCreated: (connector: V2GcpConnector) => void + onClose: () => void +}) { + // Not saved: the script is asked for with it. Reopening the surface starts over. + const [filter, setFilter] = useState({ chosen: null, acknowledged: false }) + // The connector the surface created, until the status read lists it. + const [created, setCreated] = useState(null) + const connector = + target.kind === "new" + ? undefined + : (connectors.find((candidate) => candidate.id === target.id) ?? + (created?.id === target.id ? created : undefined)) + const createdHere = connector !== undefined && created?.id === connector.id + // A connection made here keeps the flow's name to its end. + const [newTitle] = useState(connectors.length === 0 ? "Connect Google Cloud" : "Add connection") + const title = + connector === undefined || createdHere ? ( + newTitle + ) : ( + <> + {GCP_SCOPE_NAMES[connector.scope_type]}{" "} + {connector.scope_id} + + ) + + return ( + + {/* Each step brings its own header, panel and footer. */} + {target.kind !== "new" && connector === undefined ? ( + <> + + + + + + ) : connector === undefined ? ( + { + setCreated(connector) + onCreated(connector) + }} + /> + ) : ( + + )} + + ) +} + +/** + * Disconnecting a connection whose script has run: clean Google Cloud up first, then remove the + * connection. The cleanup script reports to Maple, so the dialog sees it finish. + */ +export function GcpDisconnectDialog({ + connector, + disconnecting, + onDisconnect, + onClose, +}: { + connector: V2GcpConnector + disconnecting: boolean + onDisconnect: () => void + onClose: () => void +}) { + const { scripts, failure } = useGcpScripts(connector, "keep") + useIntervalRefresh(useAtomRefresh(gcpStatusQuery()), { + intervalMs: GCP_SETTLING_REFRESH_MS, + enabled: true, + }) + const cleaned = connector.applied_logs_enabled === false && connector.applied_metrics_enabled === false + + return ( + (open || disconnecting ? undefined : onClose())}> + + + + Disconnect {GCP_SCOPE_NAMES[connector.scope_type]}{" "} + {connector.scope_id} + + + +
      + + + . + + ) : ( + `Run the cleanup script in Cloud Shell. Takes ${connector.applied_logs_enabled === true ? "one to two minutes" : "about a minute"}.` + ) + } + > + {cleaned ? null : ( + <> +
      +
      Deletes
      +
      + The log sink, topic, subscription and read-only service account. +
      +
      Keeps
      +
      The APIs it switched on.
      +
      + +

      + Waiting for the cleanup script. This updates on its own. +

      + + )} +       + +
    + {cleaned ? null : ( +

    + + Disconnecting before the cleanup leaves Google Cloud publishing logs to Pub/Sub, + billed by Google. +

    + )} +     + + }> + Cancel + + + +     +     + ) +} diff --git a/apps/web/src/components/integrations/gcp-connector-state.test.ts b/apps/web/src/components/integrations/gcp-connector-state.test.ts new file mode 100644 index 0000000000..fa6f66d9ee --- /dev/null +++ b/apps/web/src/components/integrations/gcp-connector-state.test.ts @@ -0,0 +1,739 @@ +import { describe, expect, it } from "vitest" + +import { Option } from "effect" +import { + cloudShellUrl, + gcpApplyLine, + gcpCollectLock, + gcpConnectionState, + gcpCreateRequest, + gcpDraftEffect, + gcpLogFilterChoice, + gcpLogFilters, + gcpLogState, + gcpMessageParts, + gcpMetricsState, + gcpOverlapNote, + gcpPendingChanges, + gcpRunAsked, + gcpScopeLabel, + gcpScopeRoles, + gcpScriptNeeded, + gcpScriptOverdue, + gcpSetupRunning, + gcpWorstState, + logRouterUrl, + type GcpConnectorDraft, +} from "./gcp-connector-state" + +const NOW = Date.parse("2026-10-08T12:00:00.000Z") +const ago = (minutes: number) => new Date(NOW - minutes * 60_000).toISOString() + +/** A project connection whose script ran an hour ago and that delivers both. */ +const connector = (over: Partial[0]> = {}) => ({ + scope_type: "project" as const, + logs_enabled: true, + metrics_enabled: true, + applied_logs_enabled: true as boolean | null, + applied_metrics_enabled: true as boolean | null, + setup_reported_at: ago(60) as string | null, + last_log_received_at: ago(1) as string | null, + last_log_error: null as string | null, + last_metrics_received_at: ago(4) as string | null, + last_metrics_error: null as string | null, + discovered_project_count: 1, + last_resources_error: null as string | null, + ...over, +}) +/** As created: nothing reported, nothing arrived. */ +const fresh = { applied_logs_enabled: null, applied_metrics_enabled: null, setup_reported_at: null } +const NEVER = { last_log_received_at: null, last_metrics_received_at: null } + +describe("gcpLogState", () => { + const log = (over: Parameters[0]) => gcpLogState(connector(over), NOW) + + it("is off, and says whether the sink is still set up", () => { + expect(log({ logs_enabled: false, applied_logs_enabled: false, last_log_error: "x" })).toEqual({ + kind: "off", + stillSetUp: false, + }) + expect(log({ logs_enabled: false, applied_logs_enabled: null })).toEqual({ + kind: "off", + stillSetUp: false, + }) + expect(log({ logs_enabled: false })).toEqual({ kind: "off", stillSetUp: true }) + }) + + it("is failing while the last push was rejected, with the last accepted log", () => { + expect(log({ last_log_error: "wrapped", last_log_received_at: ago(5) })).toEqual({ + kind: "failing", + error: "wrapped", + lastLogReceivedAt: ago(5), + }) + expect(log({ ...fresh, ...NEVER, last_log_error: "wrapped" })).toMatchObject({ + kind: "failing", + lastLogReceivedAt: null, + }) + }) + + it("is setup pending until a run reports log forwarding", () => { + expect(log({ ...fresh, ...NEVER })).toEqual({ kind: "setup-pending" }) + }) + + it("is setup pending after logs were removed and switched on again, whatever arrived before", () => { + expect(log({ applied_logs_enabled: false, last_log_received_at: ago(22) })).toEqual({ + kind: "setup-pending", + }) + }) + + it("is setup running for two minutes after a run reported on metrics alone", () => { + const midRun = { applied_logs_enabled: null, last_log_received_at: null } + expect(log({ ...midRun, setup_reported_at: ago(1) })).toEqual({ kind: "setup-running" }) + expect(log({ ...midRun, setup_reported_at: ago(3) })).toEqual({ kind: "setup-pending" }) + // A report of removal is a finished section, not one that is still to come. + expect(log({ applied_logs_enabled: false, setup_reported_at: ago(1) })).toEqual({ + kind: "setup-pending", + }) + }) + + it("trusts a log from the last 15 minutes over a missing report, not over a report of removal", () => { + expect(log({ ...fresh, last_log_received_at: ago(14) })).toMatchObject({ kind: "receiving" }) + // The cleanup script just ran: the last entries it forwarded prove nothing. + expect(log({ applied_logs_enabled: false, last_log_received_at: ago(1) })).toEqual({ + kind: "setup-pending", + }) + }) + + it("waits for the first log once a run reported, and is overdue after 20 minutes", () => { + expect(log({ ...NEVER, setup_reported_at: ago(1) })).toEqual({ + kind: "waiting", + reportedAt: ago(1), + overdue: false, + }) + expect(log({ ...NEVER, setup_reported_at: ago(21) })).toMatchObject({ + kind: "waiting", + overdue: true, + }) + }) + + it("is idle after 24 hours without a log", () => { + expect(log({ last_log_received_at: ago(3 * 24 * 60) })).toEqual({ + kind: "idle", + lastLogReceivedAt: ago(3 * 24 * 60), + }) + }) + + it("is receiving otherwise", () => { + expect(log({})).toEqual({ kind: "receiving", lastLogReceivedAt: ago(1) }) + }) +}) + +describe("gcpMetricsState", () => { + const metrics = (over: Parameters[0]) => gcpMetricsState(connector(over), NOW) + const SIGN_IN = "Maple can't sign in as this connection's read-only service account yet." + + it("is off, and says whether the service account is still set up", () => { + expect(metrics({ metrics_enabled: false, applied_metrics_enabled: null })).toEqual({ + kind: "off", + stillSetUp: false, + }) + expect(metrics({ metrics_enabled: false, last_metrics_error: "x" })).toEqual({ + kind: "off", + stillSetUp: true, + }) + }) + + it("is setup pending until a run reports, and hides the poller's sign-in error", () => { + expect(metrics({ ...fresh, ...NEVER, last_metrics_error: SIGN_IN })).toEqual({ + kind: "setup-pending", + }) + }) + + it("is setup running for two minutes after a run reported on logs alone", () => { + const midRun = { applied_metrics_enabled: null, ...NEVER, last_metrics_error: SIGN_IN } + expect(metrics({ ...midRun, setup_reported_at: ago(1) })).toEqual({ kind: "setup-running" }) + expect(metrics({ ...midRun, setup_reported_at: ago(3) })).toEqual({ kind: "setup-pending" }) + }) + + it("trusts a read from the last 15 minutes over a missing report, not over a report of removal", () => { + expect(metrics({ ...fresh })).toMatchObject({ kind: "receiving" }) + expect(metrics({ applied_metrics_enabled: false })).toEqual({ kind: "setup-pending" }) + }) + + it("waits for the first read, and for 10 minutes after a run whatever the poller said before", () => { + expect(metrics({ ...NEVER, setup_reported_at: ago(14) })).toEqual({ + kind: "waiting", + reportedAt: ago(14), + overdue: false, + }) + // Past 15 minutes the row stops promising the first read. + expect(metrics({ ...NEVER, setup_reported_at: ago(20) })).toEqual({ + kind: "waiting", + reportedAt: ago(20), + overdue: true, + }) + expect(metrics({ ...NEVER, setup_reported_at: ago(2), last_metrics_error: SIGN_IN })).toEqual({ + kind: "waiting", + reportedAt: ago(2), + overdue: false, + }) + // Switched off, cleaned up and set up again: the old read and the error in between don't count. + expect( + metrics({ + last_metrics_received_at: ago(120), + last_metrics_error: SIGN_IN, + setup_reported_at: ago(2), + }), + ).toMatchObject({ kind: "waiting" }) + // A run that repairs a connection whose reads fail. + expect( + metrics({ + last_metrics_received_at: ago(12), + last_metrics_error: "denied", + setup_reported_at: ago(2), + }), + ).toMatchObject({ kind: "waiting" }) + }) + + it("is failing once the grace is over and nothing was read in the last two polls", () => { + expect(metrics({ ...NEVER, setup_reported_at: ago(11), last_metrics_error: SIGN_IN })).toEqual({ + kind: "failing", + error: SIGN_IN, + lastMetricsReceivedAt: null, + }) + expect(metrics({ last_metrics_received_at: ago(11), last_metrics_error: "denied" })).toEqual({ + kind: "failing", + error: "denied", + lastMetricsReceivedAt: ago(11), + }) + }) + + it("is incomplete while a read from the last 10 minutes came with an error", () => { + expect(metrics({ last_metrics_error: "3 of 46 metric queries failed." })).toEqual({ + kind: "incomplete", + error: "3 of 46 metric queries failed.", + lastMetricsReceivedAt: ago(4), + }) + expect(metrics({ last_metrics_received_at: ago(9), last_metrics_error: "denied" })).toMatchObject({ + kind: "incomplete", + }) + }) + + it("keeps receiving when only the resource list is incomplete", () => { + expect(metrics({ last_resources_error: "The resource list is incomplete." })).toMatchObject({ + kind: "receiving", + resourcesError: "The resource list is incomplete.", + }) + }) + + it("is stalled after 30 minutes without a read and without an error", () => { + expect(metrics({ last_metrics_received_at: ago(42) })).toEqual({ + kind: "stalled", + lastMetricsReceivedAt: ago(42), + }) + }) + + it("is receiving otherwise, with the project count of a folder or organization", () => { + expect(metrics({})).toEqual({ + kind: "receiving", + lastMetricsReceivedAt: ago(4), + projectCount: null, + resourcesError: null, + }) + expect(metrics({ scope_type: "organization", discovered_project_count: 14 })).toMatchObject({ + projectCount: 14, + }) + expect(metrics({ scope_type: "folder", discovered_project_count: 0 })).toMatchObject({ + projectCount: null, + }) + }) +}) + +describe("gcpConnectionState", () => { + const state = (over: Parameters[0]) => gcpConnectionState(connector(over), NOW) + + it("needs attention when a capability is failing, incomplete or stalled", () => { + expect(state({ last_log_error: "wrapped" })).toBe("attention") + expect(state({ last_metrics_error: "3 of 46 metric queries failed." })).toBe("attention") + expect(state({ last_metrics_received_at: ago(42) })).toBe("attention") + // Worse than the other capability still waiting on its script. + expect(state({ last_log_error: "wrapped", applied_metrics_enabled: false, ...NEVER })).toBe( + "attention", + ) + }) + + it("is setup pending before any run reported", () => { + expect(state({ ...fresh, ...NEVER })).toBe("setup-pending") + }) + + it("has changes pending when the configuration and the last report disagree", () => { + expect(state({ logs_enabled: false })).toBe("changes-pending") + expect(state({ applied_metrics_enabled: false, last_metrics_received_at: ago(90) })).toBe( + "changes-pending", + ) + }) + + it("waits between the sections of a first run, and is setup pending if the second never reports", () => { + const midRun = { applied_metrics_enabled: null, setup_reported_at: ago(0), ...NEVER } + expect(state(midRun)).toBe("waiting") + expect(gcpSetupRunning(connector(midRun), NOW)).toBe(true) + expect(gcpScriptNeeded(connector(midRun), NOW)).toBeNull() + expect(gcpPendingChanges(connector(midRun), NOW)).toEqual([]) + + const stopped = { ...midRun, setup_reported_at: ago(3) } + expect(state(stopped)).toBe("setup-pending") + expect(gcpSetupRunning(connector(stopped), NOW)).toBe(false) + }) + + it("waits for data once the script reported", () => { + expect(state({ ...NEVER, setup_reported_at: ago(1) })).toBe("waiting") + }) + + it("is healthy while it receives, idles, or has a capability switched off and removed", () => { + expect(state({})).toBe("healthy") + expect(state({ last_log_received_at: ago(3 * 24 * 60) })).toBe("healthy") + expect(state({ last_resources_error: "The resource list is incomplete." })).toBe("healthy") + expect(state({ metrics_enabled: false, applied_metrics_enabled: false })).toBe("healthy") + expect(state({ logs_enabled: false, applied_logs_enabled: null })).toBe("healthy") + }) +}) + +describe("gcpPendingChanges", () => { + const changes = (over: Parameters[0]) => gcpPendingChanges(connector(over), NOW) + + it("lists nothing while the configuration matches the last report, or before the first run", () => { + expect(changes({})).toEqual([]) + expect(changes({ ...fresh, ...NEVER })).toEqual([]) + }) + + const LOGS_START = "Log forwarding starts once the script has created the log sink." + const LOGS_STOP = + "Google Cloud keeps forwarding logs, billed by Google, until the script has removed the log sink." + const METRICS_START = + "Metrics and resources start once the script has created the read-only service account." + const METRICS_STOP = + "The read-only service account stays in Google Cloud until the script has removed it." + + it("says what goes on until a re-run removes and creates", () => { + expect(changes({ logs_enabled: false })).toEqual([LOGS_STOP]) + expect( + changes({ + logs_enabled: false, + applied_metrics_enabled: false, + last_metrics_received_at: ago(90), + }), + ).toEqual([LOGS_STOP, METRICS_START]) + expect( + changes({ metrics_enabled: false, applied_logs_enabled: false, last_log_received_at: null }), + ).toEqual([LOGS_START, METRICS_STOP]) + }) +}) + +describe("gcpScriptNeeded", () => { + const needed = (over: Parameters[0]) => gcpScriptNeeded(connector(over), NOW) + + it("is null while the configuration matches what the runs reported", () => { + expect(needed({})).toBeNull() + expect(needed({ ...NEVER, setup_reported_at: ago(1) })).toBeNull() + }) + + it("says so whatever else is wrong: a failing capability does not hide a pending change", () => { + expect(needed({ logs_enabled: false, last_metrics_error: "denied" })).toBe("changes-pending") + expect(needed({ ...fresh, ...NEVER, last_log_error: "wrapped" })).toBe("setup-pending") + }) +}) + +describe("gcpScriptOverdue", () => { + const OPENED = NOW - 60_000 + + it("counts from the creation of a connection no run has reported on, not from the panel's opening", () => { + const unreported = { setup_reported_at: null } + expect(gcpScriptOverdue({ ...unreported, created_at: ago(4) }, OPENED, NOW)).toBe(false) + expect(gcpScriptOverdue({ ...unreported, created_at: ago(6) }, OPENED, NOW)).toBe(true) + }) + + it("counts from the panel's opening once a run has reported", () => { + const reported = { setup_reported_at: ago(30), created_at: ago(600) } + expect(gcpScriptOverdue(reported, OPENED, NOW)).toBe(false) + expect(gcpScriptOverdue(reported, NOW - 6 * 60_000, NOW)).toBe(true) + }) +}) + +describe("gcpWorstState", () => { + it("takes the worst of several connections", () => { + expect(gcpWorstState(["healthy", "waiting", "changes-pending"])).toBe("changes-pending") + expect(gcpWorstState(["setup-pending", "attention"])).toBe("attention") + expect(gcpWorstState(["healthy", "healthy"])).toBe("healthy") + }) +}) + +describe("gcpScopeRoles", () => { + const both = { logs_enabled: true, metrics_enabled: true } + + it("asks for nothing beyond Owner on a project", () => { + expect(gcpScopeRoles("project", both)).toEqual([]) + }) + + it("asks for the sink role with logs and the IAM role with metrics", () => { + expect(gcpScopeRoles("organization", both)).toEqual([ + "Logs Configuration Writer", + "Organization Administrator", + ]) + expect(gcpScopeRoles("folder", { logs_enabled: false, metrics_enabled: true })).toEqual([ + "Folder IAM Admin", + ]) + expect(gcpScopeRoles("folder", { logs_enabled: true, metrics_enabled: false })).toEqual([ + "Logs Configuration Writer", + ]) + }) +}) + +describe("gcpOverlapNote", () => { + const project = { scope_type: "project" as const, scope_id: "acme-prod" } + const staging = { scope_type: "project" as const, scope_id: "acme-staging" } + const organization = { scope_type: "organization" as const, scope_id: "123456789012" } + + it("says nothing when the new scope cannot overlap an existing one", () => { + expect(gcpOverlapNote("project", [])).toBeNull() + expect(gcpOverlapNote("project", [project])).toBeNull() + expect(gcpOverlapNote("folder", [organization])).toBeNull() + }) + + it("names the connected projects an organization or folder may contain", () => { + expect(gcpOverlapNote("organization", [project, staging, organization])).toBe( + "Projects you already connected (acme-prod, acme-staging) are collected twice if they sit inside this organization. Disconnect them once this connection receives data.", + ) + }) + + it("warns a project that it may sit inside a connected organization or folder", () => { + expect(gcpOverlapNote("project", [organization])).toBe( + "If this project sits inside an organization or folder you already connected, it is collected twice.", + ) + }) +}) + +describe("gcpCollectLock", () => { + const both = { logs_enabled: true, metrics_enabled: true } + const logsOnly = { logs_enabled: true, metrics_enabled: false } + const metricsOnly = { logs_enabled: false, metrics_enabled: true } + + it("leaves both free while both are on", () => { + expect(gcpCollectLock(both, "logs", true)).toBeNull() + expect(gcpCollectLock(both, "metrics", true)).toBeNull() + }) + + it("locks the last one that is on", () => { + expect(gcpCollectLock(logsOnly, "logs", true)).toBe("last-on") + expect(gcpCollectLock(metricsOnly, "metrics", true)).toBe("last-on") + }) + + it("lets the other be turned on", () => { + expect(gcpCollectLock(logsOnly, "metrics", true)).toBeNull() + expect(gcpCollectLock(metricsOnly, "logs", true)).toBeNull() + }) + + it("locks metrics off when the deployment cannot read them", () => { + expect(gcpCollectLock(logsOnly, "metrics", false)).toBe("metrics-unavailable") + }) + + it("still lets metrics be turned off on such a deployment", () => { + expect(gcpCollectLock(both, "metrics", false)).toBeNull() + }) +}) + +describe("gcpDraftEffect", () => { + it("starts a capability only once the script has run", () => { + expect(gcpDraftEffect(connector({ ...fresh, ...NEVER }), "logs", true)).toBe("starts-after-script") + expect( + gcpDraftEffect( + connector({ metrics_enabled: false, applied_metrics_enabled: false }), + "metrics", + true, + ), + ).toBe("starts-after-script") + }) + + it("changes nothing for a capability that stays as it is", () => { + expect(gcpDraftEffect(connector(), "logs", true)).toBeNull() + expect( + gcpDraftEffect( + connector({ metrics_enabled: false, applied_metrics_enabled: false }), + "metrics", + false, + ), + ).toBeNull() + }) + + it("stops in Maple at once when a capability that is set up is turned off", () => { + expect(gcpDraftEffect(connector(), "logs", false)).toBe("stops-now") + expect(gcpDraftEffect(connector(), "metrics", false)).toBe("stops-now") + }) + + it("counts arrived data as set up when no run reported", () => { + expect(gcpDraftEffect(connector(fresh), "logs", false)).toBe("stops-now") + }) + + it("stops nothing when what is turned off was never set up", () => { + expect(gcpDraftEffect(connector({ ...fresh, ...NEVER }), "logs", false)).toBeNull() + }) + + it("leaves the removal to the script for a capability already off in Maple", () => { + expect(gcpDraftEffect(connector({ logs_enabled: false }), "logs", false)).toBe("removed-by-script") + }) + + it("resumes at once when a capability is turned back on before the script removed it", () => { + expect(gcpDraftEffect(connector({ logs_enabled: false }), "logs", true)).toBe("resumes-now") + }) +}) + +describe("gcpRunAsked", () => { + const RESET = "The subscription wraps each entry. Run the setup script again: it resets the subscription." + + it("names the capabilities whose failure asks for the setup script", () => { + expect(gcpRunAsked(connector({ last_log_error: RESET }), NOW)).toEqual(["logs"]) + expect( + gcpRunAsked( + connector({ + last_log_error: RESET, + last_metrics_error: "Maple can't sign in. Run the setup script and read its last lines.", + last_metrics_received_at: ago(50), + }), + NOW, + ), + ).toEqual(["logs", "metrics"]) + }) + + it("asks for nothing when a failure has another remedy, or nothing fails", () => { + expect( + gcpRunAsked(connector({ last_log_error: "Over the plan limit. See Settings, Billing." }), NOW), + ).toEqual([]) + expect(gcpRunAsked(connector(), NOW)).toEqual([]) + }) + + it("does not ask for a capability that is off, whatever its last error says", () => { + expect(gcpRunAsked(connector({ logs_enabled: false, last_log_error: RESET }), NOW)).toEqual([]) + }) +}) + +describe("gcpApplyLine", () => { + it("says what the run does about a capability", () => { + expect(gcpApplyLine(true, true)).toBe("On") + expect(gcpApplyLine(true, null)).toBe("On after this run") + expect(gcpApplyLine(true, false)).toBe("On after this run") + expect(gcpApplyLine(false, true)).toBe("Off in Maple. This run removes it from Google Cloud.") + expect(gcpApplyLine(false, false)).toBe("Off") + expect(gcpApplyLine(false, null)).toBe("Off") + }) +}) + +describe("gcpMessageParts", () => { + it("cuts the first sentence, the remedy and what Google answered, without rewording", () => { + expect( + gcpMessageParts( + "Maple can't sign in as this connection's read-only service account yet. After a setup run Google needs a few minutes to accept the new grant, and Maple retries every 5 minutes. If this stays, run the setup script and read its last lines. (IAM Credentials returned 403)", + ), + ).toEqual({ + headline: "Maple can't sign in as this connection's read-only service account yet", + body: [ + "After a setup run Google needs a few minutes to accept the new grant, and Maple retries every 5 minutes.", + "If this stays, run the setup script and read its last lines.", + ], + answer: "IAM Credentials returned 403", + }) + }) + + it("keeps an address whole and starts a new line after it", () => { + expect( + gcpMessageParts( + "The host project acme has no active billing account. Link one: https://console.cloud.google.com/billing/linkedaccount?project=acme Maple retries every 5 minutes. (Cloud Monitoring returned 403, BILLING_DISABLED)", + ), + ).toEqual({ + headline: "The host project acme has no active billing account", + body: [ + "Link one: https://console.cloud.google.com/billing/linkedaccount?project=acme", + "Maple retries every 5 minutes.", + ], + answer: "Cloud Monitoring returned 403, BILLING_DISABLED", + }) + }) + + it("does not cut inside a metric name", () => { + expect( + gcpMessageParts( + "3 of 46 metric queries failed, first run.googleapis.com/request_latencies. The rest were stored.", + ), + ).toEqual({ + headline: "3 of 46 metric queries failed, first run.googleapis.com/request_latencies", + body: ["The rest were stored."], + answer: null, + }) + }) + + it("is a headline alone for a single sentence", () => { + expect(gcpMessageParts("Maple retries on its own.")).toEqual({ + headline: "Maple retries on its own", + body: [], + answer: null, + }) + }) +}) + +describe("gcpScopeLabel", () => { + it("names the scope kind and its id", () => { + expect(gcpScopeLabel({ scope_type: "organization", scope_id: "123456789012" })).toBe( + "Organization 123456789012", + ) + expect(gcpScopeLabel({ scope_type: "project", scope_id: "acme-prod" })).toBe("Project acme-prod") + }) +}) + +describe("gcpCreateRequest", () => { + const draft = (over: Partial): GcpConnectorDraft => ({ + scopeType: "project", + scopeId: "", + hostProjectId: "", + logsEnabled: true, + metricsEnabled: false, + ...over, + }) + const optIns = { logs_enabled: true, metrics_enabled: false } + const request = (over: Partial) => Option.getOrNull(gcpCreateRequest(draft(over))) + + it("sends a project as its own host project, trimmed", () => { + expect(request({ scopeId: " acme-prod ", hostProjectId: "ignored-host" })).toEqual({ + scope_type: "project", + scope_id: "acme-prod", + ...optIns, + }) + }) + + it("rejects a project name or number as a project ID", () => { + expect(request({ scopeId: "Acme Prod" })).toBeNull() + expect(request({ scopeId: "123456789012" })).toBeNull() + expect(request({ scopeId: "acme-prod-" })).toBeNull() + }) + + it("sends a folder or organization with its host project", () => { + expect( + request({ + scopeType: "organization", + scopeId: "123456789012", + hostProjectId: "acme-observability", + }), + ).toEqual({ + scope_type: "organization", + scope_id: "123456789012", + project_id: "acme-observability", + ...optIns, + }) + expect(request({ scopeType: "folder", scopeId: "42", hostProjectId: "acme-observability" })).toEqual({ + scope_type: "folder", + scope_id: "42", + project_id: "acme-observability", + ...optIns, + }) + }) + + it("needs a numeric ID and a host project for a folder or organization", () => { + expect(request({ scopeType: "folder", scopeId: "my-folder", hostProjectId: "acme-prod" })).toBeNull() + expect(request({ scopeType: "organization", scopeId: "123456789012" })).toBeNull() + expect( + request({ scopeType: "organization", scopeId: "123456789012", hostProjectId: "Acme" }), + ).toBeNull() + }) +}) + +describe("gcpCreateRequest opt-ins", () => { + const draft = (logsEnabled: boolean, metricsEnabled: boolean): GcpConnectorDraft => ({ + scopeType: "project", + scopeId: "acme-prod", + hostProjectId: "", + logsEnabled, + metricsEnabled, + }) + + it("sends a metrics-only connection in one request", () => { + expect(Option.getOrNull(gcpCreateRequest(draft(false, true)))).toEqual({ + scope_type: "project", + scope_id: "acme-prod", + logs_enabled: false, + metrics_enabled: true, + }) + }) + + it("needs at least one opt-in", () => { + expect(Option.isNone(gcpCreateRequest(draft(false, false)))).toBe(true) + }) +}) + +describe("gcpLogFilters", () => { + it("lists the recommended filter first, and keeping a filter only when a sink exists", () => { + expect(gcpLogFilters(false).map((filter) => filter.value)).toEqual([ + "default", + "include_gke_container_logs", + ]) + expect(gcpLogFilters(true).map((filter) => filter.value)).toEqual([ + "default", + "include_gke_container_logs", + "keep", + ]) + expect(gcpLogFilters(false)[0].label).toBe("Recommended: no GKE container logs") + }) +}) + +describe("gcpLogFilterChoice", () => { + it("starts a new sink on the recommended filter and leaves an existing sink's filter alone", () => { + expect(gcpLogFilterChoice(null, false, false)).toEqual({ + selected: "default", + unacknowledged: false, + scriptFilter: "default", + }) + expect(gcpLogFilterChoice(null, false, true)).toEqual({ + selected: "keep", + unacknowledged: false, + scriptFilter: "keep", + }) + }) + + it("includes GKE container logs only once acknowledged", () => { + expect(gcpLogFilterChoice("include_gke_container_logs", false, false)).toEqual({ + selected: "include_gke_container_logs", + unacknowledged: true, + scriptFilter: "default", + }) + // An existing sink's filter is not replaced by a choice that was never confirmed. + expect(gcpLogFilterChoice("include_gke_container_logs", false, true).scriptFilter).toBe("keep") + expect(gcpLogFilterChoice("include_gke_container_logs", true, true)).toEqual({ + selected: "include_gke_container_logs", + unacknowledged: false, + scriptFilter: "include_gke_container_logs", + }) + }) + + it("applies the other filters as chosen", () => { + expect(gcpLogFilterChoice("default", false, true).scriptFilter).toBe("default") + expect(gcpLogFilterChoice("keep", false, true).scriptFilter).toBe("keep") + }) +}) + +describe("logRouterUrl", () => { + it("opens the Log Router of the scope that holds the sink", () => { + expect(logRouterUrl({ scope_type: "project", scope_id: "acme-prod" })).toBe( + "https://console.cloud.google.com/logs/router?project=acme-prod", + ) + expect(logRouterUrl({ scope_type: "folder", scope_id: "123456789012" })).toBe( + "https://console.cloud.google.com/logs/router?folder=123456789012", + ) + expect(logRouterUrl({ scope_type: "organization", scope_id: "123456789012" })).toBe( + "https://console.cloud.google.com/logs/router?organizationId=123456789012", + ) + }) +}) + +describe("cloudShellUrl", () => { + it("opens the console on the host project with Cloud Shell attached", () => { + expect(cloudShellUrl("acme-prod")).toBe( + "https://console.cloud.google.com/?cloudshell=true&project=acme-prod", + ) + }) +}) diff --git a/apps/web/src/components/integrations/gcp-connector-state.ts b/apps/web/src/components/integrations/gcp-connector-state.ts new file mode 100644 index 0000000000..c96ab45a1f --- /dev/null +++ b/apps/web/src/components/integrations/gcp-connector-state.ts @@ -0,0 +1,478 @@ +// Google Cloud connector rules shared by the card, the hub and the Infrastructure page: what +// each capability is doing, what a connection asks of its owner, what a change to its configuration +// does and when, and the add form's validation. Pure (no React, no atoms), like planetscale-setup-steps.ts. + +import { Option, Schema } from "effect" +import { V2GcpCreateConnectorRequest, type V2GcpConnector } from "@maple/domain/http/v2" +import { + GcpProjectId, + GcpResourceNumber, + type GcpLogFilter, + type GcpScopeType, +} from "@maple/domain/primitives" +import type { Tone } from "@maple/ui/lib/tone" + +const MINUTE_MS = 60_000 +/** A log or read this recent proves the setup is in place, whatever the script reported. */ +const FRESH_MS = 15 * MINUTE_MS +/** Past this without a log, the detail turns to "no logs in 24 hours". */ +const IDLE_LOG_MS = 24 * 60 * MINUTE_MS +/** A confirmed sink that stays silent this long gets a test command. */ +const OVERDUE_LOG_MS = 20 * MINUTE_MS +/** Confirmed access that stays unread this long stops promising the first read. */ +const OVERDUE_READ_MS = 15 * MINUTE_MS +/** Two missed reads: an error with nothing read since means the reads fail. */ +const FAILED_READ_MS = 10 * MINUTE_MS +/** Six missed reads: the numbers on screen are no longer current. */ +const STALE_READ_MS = 30 * MINUTE_MS +/** A new grant needs up to two minutes to work and the first read follows about five minutes later. */ +const GRANT_GRACE_MS = 10 * MINUTE_MS +/** A run's sections report seconds apart. A report this recent is a run that may still be going. */ +const RUNNING_MS = 2 * MINUTE_MS +/** A script run takes about a minute. Past this the setup panel says what to check. */ +const SCRIPT_OVERDUE_MS = 5 * MINUTE_MS + +const ageMs = (iso: string | null, nowMs: number) => + iso === null ? Number.POSITIVE_INFINITY : nowMs - Date.parse(iso) + +/** + * Whether the script still has to set a switched-on capability up: a run reported it removed, or + * no run reported and nothing recent arrived. Recent data stands in for a report that never + * reached Maple, but not against one that did. + */ +const needsSetup = (applied: boolean | null, lastDataAgeMs: number) => + applied === false || (applied === null && lastDataAgeMs > FRESH_MS) + +/** + * What a capability that needs setup says. A run's sections report one after the other, so no + * report on this one right after a report on the other means the script is still working. + */ +const setupKind = ( + applied: boolean | null, + reportedAt: string | null, + nowMs: number, +): "setup-pending" | "setup-running" => + applied === null && ageMs(reportedAt, nowMs) < RUNNING_MS ? "setup-running" : "setup-pending" + +type GcpLogState = + /** `stillSetUp`: the sink still forwards, because the script has not run since the switch. */ + | { readonly kind: "off"; readonly stillSetUp: boolean } + /** Maple rejected the most recent push. Earlier pushes may have been accepted. */ + | { readonly kind: "failing"; readonly error: string; readonly lastLogReceivedAt: string | null } + /** The setup script has not set log forwarding up, as far as Maple knows, or is doing so now. */ + | { readonly kind: "setup-pending" | "setup-running" } + /** The script reported, and no entry has arrived yet. `overdue` after 20 minutes. */ + | { readonly kind: "waiting"; readonly reportedAt: string | null; readonly overdue: boolean } + | { readonly kind: "idle"; readonly lastLogReceivedAt: string } + | { readonly kind: "receiving"; readonly lastLogReceivedAt: string } + +type LogFields = Pick< + V2GcpConnector, + "logs_enabled" | "last_log_received_at" | "last_log_error" | "applied_logs_enabled" | "setup_reported_at" +> + +export function gcpLogState(connector: LogFields, nowMs: number): GcpLogState { + const applied = connector.applied_logs_enabled === true + if (!connector.logs_enabled) return { kind: "off", stillSetUp: applied } + const lastLogReceivedAt = connector.last_log_received_at + if (connector.last_log_error !== null) { + return { kind: "failing", error: connector.last_log_error, lastLogReceivedAt } + } + const age = ageMs(lastLogReceivedAt, nowMs) + if (needsSetup(connector.applied_logs_enabled, age)) { + return { kind: setupKind(connector.applied_logs_enabled, connector.setup_reported_at, nowMs) } + } + if (lastLogReceivedAt === null) { + const reportedAt = connector.setup_reported_at + return { kind: "waiting", reportedAt, overdue: ageMs(reportedAt, nowMs) > OVERDUE_LOG_MS } + } + return { kind: age > IDLE_LOG_MS ? "idle" : "receiving", lastLogReceivedAt } +} + +type GcpMetricsState = + /** `stillSetUp`: the read-only service account still exists in Google Cloud. */ + | { readonly kind: "off"; readonly stillSetUp: boolean } + | { readonly kind: "setup-pending" | "setup-running" } + /** The script reported, and the first read after it has not landed. `overdue` after 15 minutes. */ + | { readonly kind: "waiting"; readonly reportedAt: string | null; readonly overdue: boolean } + /** Reads fail: nothing was read in the last two polls. */ + | { readonly kind: "failing"; readonly error: string; readonly lastMetricsReceivedAt: string | null } + /** A read arrived within the last two polls, but part of the metrics is missing. */ + | { readonly kind: "incomplete"; readonly error: string; readonly lastMetricsReceivedAt: string } + /** No error, and no read for half an hour. */ + | { readonly kind: "stalled"; readonly lastMetricsReceivedAt: string } + | { + readonly kind: "receiving" + readonly lastMetricsReceivedAt: string + /** Projects found under a folder or organization; null for a project or before the first sync. */ + readonly projectCount: number | null + /** Why the resource list is incomplete. Metrics are unaffected, so this is not a failure. */ + readonly resourcesError: string | null + } + +type MetricsFields = Pick< + V2GcpConnector, + | "scope_type" + | "metrics_enabled" + | "last_metrics_received_at" + | "last_metrics_error" + | "discovered_project_count" + | "last_resources_error" + | "applied_metrics_enabled" + | "setup_reported_at" +> + +export function gcpMetricsState(connector: MetricsFields, nowMs: number): GcpMetricsState { + const applied = connector.applied_metrics_enabled === true + if (!connector.metrics_enabled) return { kind: "off", stillSetUp: applied } + const lastMetricsReceivedAt = connector.last_metrics_received_at + const error = connector.last_metrics_error + const age = ageMs(lastMetricsReceivedAt, nowMs) + // Before the script has run the poller says it can't sign in. That is the expected start. + const reportedAt = connector.setup_reported_at + if (needsSetup(connector.applied_metrics_enabled, age)) { + return { kind: setupKind(connector.applied_metrics_enabled, reportedAt, nowMs) } + } + const reportedAge = ageMs(reportedAt, nowMs) + const waiting = { kind: "waiting", reportedAt, overdue: reportedAge > OVERDUE_READ_MS } as const + // A run just reported: what the poller said before its grant works counts for nothing yet. + if (age > FAILED_READ_MS && reportedAge < GRANT_GRACE_MS) return waiting + if (lastMetricsReceivedAt === null) { + return error === null ? waiting : { kind: "failing", error, lastMetricsReceivedAt } + } + if (error !== null) { + return { kind: age > FAILED_READ_MS ? "failing" : "incomplete", error, lastMetricsReceivedAt } + } + if (age > STALE_READ_MS) return { kind: "stalled", lastMetricsReceivedAt } + return { + kind: "receiving", + lastMetricsReceivedAt, + projectCount: + connector.scope_type !== "project" && connector.discovered_project_count > 0 + ? connector.discovered_project_count + : null, + resourcesError: connector.last_resources_error, + } +} + +/** The card's dot and label for each state a capability that is on can be in. */ +export const GCP_LOG_STATUS = { + failing: { tone: "crit", label: "Rejecting logs" }, + "setup-pending": { tone: "neutral", label: "Not set up" }, + "setup-running": { tone: "neutral", label: "Setup running" }, + waiting: { tone: "neutral", label: "Waiting for first logs" }, + idle: { tone: "neutral", label: "No logs in 24 hours" }, + receiving: { tone: "ok", label: "Receiving logs" }, +} as const satisfies Record, { tone: Tone; label: string }> + +export const GCP_METRICS_STATUS = { + "setup-pending": { tone: "neutral", label: "Not set up" }, + "setup-running": { tone: "neutral", label: "Setup running" }, + waiting: { tone: "neutral", label: "Waiting for first metrics" }, + failing: { tone: "crit", label: "Can't read metrics" }, + incomplete: { tone: "warn", label: "Receiving metrics, incomplete" }, + stalled: { tone: "warn", label: "Metrics stalled" }, + receiving: { tone: "ok", label: "Receiving metrics" }, +} as const satisfies Record, { tone: Tone; label: string }> + +type ConnectorFields = LogFields & MetricsFields + +/** + * What still waits on a re-run of the setup script, for a connection whose configuration disagrees + * with what a run reported: a sentence per capability, saying what goes on until then. A capability + * no run has reported on is not in the list: that is setup, and while a first run is under way its + * sections report one after the other. + */ +export const gcpPendingChanges = (connector: ConnectorFields, nowMs: number): ReadonlyArray => { + const log = gcpLogState(connector, nowMs) + const metrics = gcpMetricsState(connector, nowMs) + return [ + log.kind === "setup-pending" && connector.applied_logs_enabled === false + ? "Log forwarding starts once the script has created the log sink." + : null, + log.kind === "off" && log.stillSetUp + ? "Google Cloud keeps forwarding logs, billed by Google, until the script has removed the log sink." + : null, + metrics.kind === "setup-pending" && connector.applied_metrics_enabled === false + ? "Metrics and resources start once the script has created the read-only service account." + : null, + metrics.kind === "off" && metrics.stillSetUp + ? "The read-only service account stays in Google Cloud until the script has removed it." + : null, + ].filter((line) => line !== null) +} + +/** A connection's worst capability, worst first. */ +type GcpConnectionState = "attention" | "changes-pending" | "setup-pending" | "waiting" | "healthy" + +/** + * Whether Google Cloud waits on the setup script, and for what: a change to what a run set up, or + * the setup itself. Null when the configuration matches what the runs reported. + */ +export function gcpScriptNeeded( + connector: ConnectorFields, + nowMs: number, +): "changes-pending" | "setup-pending" | null { + if (gcpPendingChanges(connector, nowMs).length > 0) return "changes-pending" + const pending = + gcpLogState(connector, nowMs).kind === "setup-pending" || + gcpMetricsState(connector, nowMs).kind === "setup-pending" + return pending ? "setup-pending" : null +} + +/** A run reported on one capability and not yet on the other: the script is still working. */ +export const gcpSetupRunning = (connector: ConnectorFields, nowMs: number): boolean => + gcpLogState(connector, nowMs).kind === "setup-running" || + gcpMetricsState(connector, nowMs).kind === "setup-running" + +export function gcpConnectionState(connector: ConnectorFields, nowMs: number): GcpConnectionState { + const log = gcpLogState(connector, nowMs).kind + const metrics = gcpMetricsState(connector, nowMs).kind + if (log === "failing" || metrics === "failing" || metrics === "incomplete" || metrics === "stalled") { + return "attention" + } + return ( + gcpScriptNeeded(connector, nowMs) ?? + (log === "waiting" || metrics === "waiting" || gcpSetupRunning(connector, nowMs) + ? "waiting" + : "healthy") + ) +} + +/** + * Whether the setup panel's confirm step has waited long enough to say what to check. A connection + * no run has reported on counts from its creation, so a reload does not start the wait over. After + * a report the API has no time for the change, and the opening of the apply step stands in. + */ +export const gcpScriptOverdue = ( + connector: Pick, + openedAtMs: number, + nowMs: number, +): boolean => + nowMs - (connector.setup_reported_at === null ? Date.parse(connector.created_at) : openedAtMs) > + SCRIPT_OVERDUE_MS + +/** The page header and the hub row name a connection's state in these words. */ +export const GCP_CONNECTION_LABEL = { + attention: "Needs attention", + "changes-pending": "Changes pending", + "setup-pending": "Setup pending", + waiting: "Waiting for data", + healthy: "Healthy", +} as const satisfies Record + +/** The states worst first: the order of a list of connections, and of what the page header names. */ +export const GCP_STATE_ORDER = [ + "attention", + "setup-pending", + "changes-pending", + "waiting", + "healthy", +] as const satisfies ReadonlyArray + +/** The worst state among several connections; healthy when there are none. */ +export const gcpWorstState = (states: ReadonlyArray): GcpConnectionState => + GCP_STATE_ORDER.find((state) => states.includes(state)) ?? "healthy" + +/** + * The capabilities whose failure asks, in the API's words, for another run of the setup script. + * The connection then offers that run once, however many failures ask for it. + */ +export function gcpRunAsked(connector: ConnectorFields, nowMs: number): ReadonlyArray { + const log = gcpLogState(connector, nowMs) + const metrics = gcpMetricsState(connector, nowMs) + const asked: Array = [] + if (log.kind === "failing" && log.error.includes("setup script")) asked.push("logs") + if ( + (metrics.kind === "failing" || metrics.kind === "incomplete") && + metrics.error.includes("setup script") + ) { + asked.push("metrics") + } + return asked +} + +export type GcpFlags = Pick + +export type GcpCapability = "logs" | "metrics" + +/** Why a capability cannot be changed: it is the last one on, or the deployment cannot read metrics. */ +export type GcpCollectLock = "last-on" | "metrics-unavailable" + +/** The API refuses both of these, so the configuration disables the choice and says why. */ +export function gcpCollectLock( + flags: GcpFlags, + capability: GcpCapability, + metricsAvailable: boolean, +): GcpCollectLock | null { + const on = capability === "logs" ? flags.logs_enabled : flags.metrics_enabled + const otherOn = capability === "logs" ? flags.metrics_enabled : flags.logs_enabled + if (on) return otherOn ? null : "last-on" + return capability === "metrics" && !metricsAvailable ? "metrics-unavailable" : null +} + +/** + * What saving a choice does, and when. Turning a capability off acts in Maple at once and in + * Google Cloud when the script runs; turning it on does nothing until the script has run, unless + * what it needs is still there from before. + */ +export type GcpDraftEffect = "starts-after-script" | "resumes-now" | "stops-now" | "removed-by-script" + +export function gcpDraftEffect( + connector: ConnectorFields, + capability: GcpCapability, + draft: boolean, +): GcpDraftEffect | null { + const logs = capability === "logs" + const wanted = logs ? connector.logs_enabled : connector.metrics_enabled + const applied = logs ? connector.applied_logs_enabled : connector.applied_metrics_enabled + const lastAt = logs ? connector.last_log_received_at : connector.last_metrics_received_at + // Data that arrived stands in for a report that never reached Maple. + const setUp = applied === true || (applied === null && lastAt !== null) + if (draft) return setUp ? (wanted ? null : "resumes-now") : "starts-after-script" + return setUp ? (wanted ? "stops-now" : "removed-by-script") : null +} + +/** One line of the apply step's recap: what the script does about a capability. */ +export const gcpApplyLine = (wanted: boolean, applied: boolean | null): string => + wanted + ? applied === true + ? "On" + : "On after this run" + : applied === true + ? "Off in Maple. This run removes it from Google Cloud." + : "Off" + +/** + * A failure in the API's words, cut for display and not reworded: the first sentence, the + * sentences that say what to do, and what Google answered, which closes the message in brackets. + */ +export function gcpMessageParts(text: string): { + readonly headline: string + readonly body: ReadonlyArray + readonly answer: string | null +} { + const [, message = text, answer] = /^(.*?)\s*(\([^()]*\))?$/s.exec(text.trim()) ?? [] + // A sentence ends at a full stop before a space. An address ends before one too, without a stop. + const [headline = "", ...body] = message.split(/(?<=\.)\s+|(?<=https:\/\/\S+)\s+(?=[A-Z])/) + return { headline: headline.replace(/\.$/, ""), body, answer: answer?.slice(1, -1) ?? null } +} + +export const GCP_SCOPE_NAMES = { + project: "Project", + folder: "Folder", + organization: "Organization", +} as const + +/** "Organization 123456789012", "Project acme-prod". */ +export const gcpScopeLabel = (connector: Pick): string => + `${GCP_SCOPE_NAMES[connector.scope_type]} ${connector.scope_id}` + +/** + * The roles the person running the script needs on a folder or organization, next to Owner on + * the host project. A project needs Owner alone. Mirrors the script's own permission check. + */ +export const gcpScopeRoles = (scopeType: GcpScopeType, flags: GcpFlags): ReadonlyArray => + scopeType === "project" + ? [] + : [ + flags.logs_enabled ? "Logs Configuration Writer" : null, + flags.metrics_enabled + ? scopeType === "folder" + ? "Folder IAM Admin" + : "Organization Administrator" + : null, + ].filter((role) => role !== null) + +/** Why adding this scope next to the existing connections may collect a project twice, or null. */ +export function gcpOverlapNote( + scopeType: GcpScopeType, + existing: ReadonlyArray>, +): string | null { + if (scopeType === "project") { + return existing.some((connector) => connector.scope_type !== "project") + ? "If this project sits inside an organization or folder you already connected, it is collected twice." + : null + } + const projects = existing.filter((connector) => connector.scope_type === "project") + return projects.length === 0 + ? null + : `Projects you already connected (${projects.map((project) => project.scope_id).join(", ")}) are collected twice if they sit inside this ${scopeType}. Disconnect them once this connection receives data.` +} + +/** What the add form holds. */ +export interface GcpConnectorDraft { + readonly scopeType: GcpScopeType + /** A project ID, or the numeric ID of a folder or organization. */ + readonly scopeId: string + /** Read only for a folder or organization; a project is its own host project. */ + readonly hostProjectId: string + readonly logsEnabled: boolean + readonly metricsEnabled: boolean +} + +const decodeCreateRequest = Schema.decodeUnknownOption(V2GcpCreateConnectorRequest) + +/** + * The create request for a draft, decoded with the API's own schema; none while a field is wrong. + * The schema allows both opt-ins off and the API then refuses, so that case is none here too. + */ +export const gcpCreateRequest = (draft: GcpConnectorDraft): Option.Option => { + if (!draft.logsEnabled && !draft.metricsEnabled) return Option.none() + const optIns = { logs_enabled: draft.logsEnabled, metrics_enabled: draft.metricsEnabled } + return decodeCreateRequest( + draft.scopeType === "project" + ? { scope_type: "project", scope_id: draft.scopeId.trim(), ...optIns } + : { + scope_type: draft.scopeType, + scope_id: draft.scopeId.trim(), + project_id: draft.hostProjectId.trim(), + ...optIns, + }, + ) +} + +export const isGcpProjectId = Schema.is(GcpProjectId) +export const isGcpResourceNumber = Schema.is(GcpResourceNumber) + +const LOG_FILTERS = [ + { value: "default", label: "Recommended: no GKE container logs" }, + { value: "include_gke_container_logs", label: "Include GKE container logs" }, + { value: "keep", label: "Keep the sink's current filter" }, +] as const satisfies ReadonlyArray<{ value: GcpLogFilter; label: string }> + +/** The configuration's log filters in display order. Keeping a filter needs a sink that has one. */ +export const gcpLogFilters = (sinkExists: boolean) => + LOG_FILTERS.filter((filter) => filter.value !== "keep" || sinkExists) + +/** + * What the configuration's log filter stands on. Nothing chosen: an existing sink keeps its filter, + * so running the script again for another change never resets it, and a new sink gets the + * recommended one. Including GKE container logs counts once it is acknowledged: until then the + * script stays on the filter the configuration started with, which is also where backing out + * returns to. + */ +export const gcpLogFilterChoice = ( + chosen: GcpLogFilter | null, + acknowledged: boolean, + sinkExists: boolean, +) => { + const initial: GcpLogFilter = sinkExists ? "keep" : "default" + const selected = chosen ?? initial + const unacknowledged = selected === "include_gke_container_logs" && !acknowledged + return { selected, unacknowledged, scriptFilter: unacknowledged ? initial : selected } +} + +const LOG_ROUTER_SCOPE = { project: "project", folder: "folder", organization: "organizationId" } as const + +/** The console's Log Router for the scope that holds the connection's sink. */ +export const logRouterUrl = (connector: Pick): string => + `https://console.cloud.google.com/logs/router?${LOG_ROUTER_SCOPE[connector.scope_type]}=${encodeURIComponent(connector.scope_id)}` + +/** + * Opens the Google Cloud console on the host project with a Cloud Shell terminal attached. Cloud + * Shell starts with the console's active project, and the scripts name the project on every + * command, so a console that ignores the hint still runs them against the right one. + */ +export const cloudShellUrl = (projectId: string): string => + `https://console.cloud.google.com/?cloudshell=true&project=${encodeURIComponent(projectId)}` diff --git a/apps/web/src/components/integrations/gcp-integration-card.tsx b/apps/web/src/components/integrations/gcp-integration-card.tsx new file mode 100644 index 0000000000..175f5ca858 --- /dev/null +++ b/apps/web/src/components/integrations/gcp-integration-card.tsx @@ -0,0 +1,841 @@ +import { Fragment, useState } from "react" +import type React from "react" +import { Link } from "@tanstack/react-router" +import { Exit, Option } from "effect" +import * as AsyncResult from "effect/reactivity/AsyncResult" +import type { V2GcpConnector } from "@maple/domain/http/v2" +import { Badge } from "@maple/ui/components/ui/badge" +import { Button } from "@maple/ui/components/ui/button" +import { ConfirmDialog } from "@maple/ui/components/ui/confirm-dialog" +import { DropdownMenuItem, DropdownMenuSeparator } from "@maple/ui/components/ui/dropdown-menu" +import { InlineCode } from "@maple/ui/components/ui/inline-code" +import { Panel } from "@maple/ui/components/ui/panel" +import { RowActionsMenu } from "@maple/ui/components/ui/row-actions-menu" +import { Skeleton } from "@maple/ui/components/ui/skeleton" +import { StatusDot } from "@maple/ui/components/ui/status-dot" +import { countLabel } from "@maple/ui/lib/format" +import { TONE_TEXT, type Tone } from "@maple/ui/lib/tone" +import { cn } from "@maple/ui/lib/utils" + +import { + removedGcpConnectorsAtomFamily, + type RemovedGcpConnector, +} from "@/atoms/gcp-removed-connectors-atoms" +import { ErrorState } from "@/components/common/error-state" +import { RelativeTime } from "@/components/common/relative-time" +import { SectionHeading } from "@/components/common/section-heading" +import { ArrowRightIcon, GoogleCloudIcon, GoogleCloudMonoIcon } from "@/components/icons" +import { useIntervalRefresh } from "@/hooks/use-interval-refresh" +import { useIsOrgAdmin } from "@/hooks/use-is-org-admin" +import { useLiveClock } from "@/hooks/use-live-clock" +import { useAsyncAction } from "@/hooks/use-mutation-action" +import { Result, useAtom, useAtomRefresh, useAtomSet, useAtomValue } from "@/lib/effect-atom" +import { showErrorToast } from "@/lib/error-toast" +import { encodeLogAttributeFilter } from "@/lib/logs/log-attribute-filters" +import { retainedQuery } from "@/lib/services/common/atom-client" +import { getActiveOrgId } from "@/lib/services/common/auth-headers" +import { MapleApiV2AtomClient } from "@/lib/services/common/v2-atom-client" +import { + GCP_LINK, + GCP_REACTIVITY_KEYS, + GCP_SETTLING_REFRESH_MS, + GCP_TITLE, + GcpCommand, + GcpConfigure, + GcpCopyScriptButton, + GcpDisconnectDialog, + GcpExternalLink, + GcpOpenCloudShellButton, + type GcpConfigureTarget, +} from "./gcp-configure" +import { + GCP_CONNECTION_LABEL, + GCP_LOG_STATUS, + GCP_METRICS_STATUS, + GCP_SCOPE_NAMES, + GCP_STATE_ORDER, + gcpConnectionState, + gcpLogState, + gcpMessageParts, + gcpMetricsState, + gcpPendingChanges, + gcpRunAsked, + gcpScopeLabel, + gcpScriptNeeded, + gcpSetupRunning, + gcpWorstState, + type GcpCapability, +} from "./gcp-connector-state" +import { GCP_ACCENT, HEALTH_TONE, gcpStatusQuery } from "./integration-catalog" +import { + IntegrationEmpty, + IntegrationEmptyCard, + IntegrationEmptyFeature, + IntegrationEmptyFeatures, + IntegrationEmptyHint, + IntegrationEmptyMedia, +} from "./integration-empty-state" + +/** Keeps the "last log" and "last read" times and a new error current on a page left open. */ +const STEADY_REFRESH_MS = 60_000 + +/** How the plan-limit texts name the billing page. */ +const BILLING_PAGE = "Settings, Billing" +// An address ends before the punctuation or the bracket that follows it. +const LINK_PATTERN = new RegExp(`(https://[^\\s)]*[^\\s).,]|${BILLING_PAGE})`) + +/** A sentence of a failure: an address in it becomes a named link, the billing page a link to it. */ +function Linked({ text }: { text: string }) { + return text.split(LINK_PATTERN).map((part, index) => + index % 2 === 0 ? ( + part + ) : part === BILLING_PAGE ? ( + + {part} + + ) : ( + + {part.startsWith("https://console.cloud.google.com/") ? "Google Cloud console" : part} + + ), + ) +} + +/** + * A failure in the API's own words, laid out to be scanned: its first sentence as the headline, + * what to do under it, and what Google answered as code. + */ +export function GcpMessage({ text }: { text: string }) { + const { headline, body, answer } = gcpMessageParts(text) + return ( +
    +

    + +

    + {body.length === 0 ? null : ( +

    + {body.map((sentence, index) => ( + + {index === 0 ? null : " "} + + + ))} +

    + )} + {answer === null ? null : ( + {answer} + )} +
    + ) +} + +const ASK_ADMIN = "Ask a Maple organization admin to run the setup script." + +/** The card's one filled button, which runs the script; for everyone but admins, the sentence asking for it. */ +function RunScript({ label, onRun }: { label: string; onRun: (() => void) | null }) { + return onRun === null ? ( +

    {ASK_ADMIN}

    + ) : ( + + ) +} + +const RETRIES = "Maple retries every 5 minutes. If this lasts an hour, write to support@maple.dev." + +/** Something to know or do about a connection: a sentence for a headline, what it means, the one thing to do. */ +interface Issue { + readonly key: string + readonly tone: "crit" | "warn" | "live" + readonly title: string + readonly body?: React.ReactNode + readonly action?: React.ReactNode +} + +function IssueNote({ issue }: { issue: Issue }) { + return ( +
    + {/* As wide as the row's dot lane, so the headline lines up with the connection's name. */} + + + +
    +

    {issue.title}

    + {issue.body === undefined ? null : ( +
    {issue.body}
    + )} + {issue.action === undefined ? null :
    {issue.action}
    } +
    +
    + ) +} + +type LogState = ReturnType +type MetricsState = ReturnType + +/** + * What a connection needs said about it, worst first; empty when it is healthy. The card has one + * filled button: the script being due takes it, else the first failure another run repairs. + */ +function connectionIssues({ + connector, + log, + metrics, + nowMs, + onRunScript, +}: { + connector: V2GcpConnector + log: LogState + metrics: MetricsState + nowMs: number + onRunScript: (() => void) | null +}): ReadonlyArray { + const needed = gcpScriptNeeded(connector, nowMs) + const repairs = needed === null ? gcpRunAsked(connector, nowMs)[0] : undefined + const rerun = (capability: GcpCapability) => + repairs === capability ? : undefined + const cleanedUp = connector.applied_logs_enabled === false && connector.applied_metrics_enabled === false + const issues: Array = [] + + if (log.kind === "failing") { + issues.push({ + key: "logs-failing", + tone: "crit", + title: "Log forwarding is failing", + body: , + action: rerun("logs"), + }) + } + if (metrics.kind === "failing") { + issues.push({ + key: "metrics-failing", + tone: "crit", + title: "Key metrics can't be read", + body: , + action: rerun("metrics"), + }) + } + if (needed !== null) { + const lines = + needed === "setup-pending" + ? [ + connector.setup_reported_at === null + ? "Google Cloud has nothing of Maple's yet. Running the setup script takes about a minute." + : "The last run stopped part of the way. Run the script again: it continues where it stopped.", + ] + : gcpPendingChanges(connector, nowMs) + issues.push({ + key: "script", + tone: "warn", + title: needed === "setup-pending" ? "Setup pending" : "Changes not applied in Google Cloud", + body: ( + <> + {lines.length === 1 ? ( +

    {lines[0]}

    + ) : ( +
      + {lines.map((line) => ( +
    • {line}
      • + ))} +
    + )} + {onRunScript !== null && needed === "changes-pending" && cleanedUp ? ( +

    To remove the connection instead, choose Disconnect in its menu.

    + ) : null} + + ), + action: ( + + ), + }) + } + if (metrics.kind === "incomplete") { + issues.push({ + key: "metrics-incomplete", + tone: "warn", + title: "Key metrics are incomplete", + body: , + action: rerun("metrics"), + }) + } + if (metrics.kind === "stalled") { + issues.push({ key: "metrics-stalled", tone: "warn", title: "No metrics in 30 minutes", body: RETRIES }) + } + if (log.kind === "waiting" && log.overdue) { + issues.push({ + key: "logs-overdue", + tone: "warn", + title: "No log entry in 20 minutes", + body: "Nothing that passes the filter was logged, or the sink can't publish. Write a test entry. If it isn't here within a minute, run the setup script again.", + action: ( + + {`gcloud logging write maple-test "hello from Maple" --project=${connector.project_id}`} + + ), + }) + } + if (metrics.kind === "waiting" && metrics.overdue) { + issues.push({ key: "metrics-overdue", tone: "warn", title: "No metrics have arrived yet", body: RETRIES }) + } + if (metrics.kind === "receiving" && metrics.resourcesError !== null) { + issues.push({ + key: "resources", + tone: "warn", + title: "Resource list incomplete", + body: , + }) + } + if (issues.length > 0) return issues + + // Nothing wrong yet: what the connection is waiting for. + if (gcpSetupRunning(connector, nowMs)) { + return [ + { + key: "running", + tone: "live", + title: "The setup script is still running", + body: "Logs and metrics start once it has finished.", + }, + ] + } + if (log.kind === "waiting" || metrics.kind === "waiting") { + return [ + { + key: "waiting", + tone: "live", + title: "Waiting for the first data", + body: + log.kind === "waiting" + ? "Google Cloud usually starts sending within 10 minutes. Logs written before then aren't forwarded." + : "The first metrics arrive within about 10 minutes.", + }, + ] + } + return [] +} + +/** A capability's state in a word or two, with when it last delivered. */ +function signalText( + capability: GcpCapability, + log: LogState, + metrics: MetricsState, +): { readonly tone: Tone; readonly text: React.ReactNode } { + if (capability === "logs") { + switch (log.kind) { + case "off": + return { tone: "neutral", text: "Off" } + case "failing": + return { + tone: "crit", + text: + log.lastLogReceivedAt === null ? ( + "Rejected" + ) : ( + + ), + } + case "setup-pending": + case "setup-running": + return { tone: "neutral", text: GCP_LOG_STATUS[log.kind].label } + case "waiting": + return log.overdue + ? { tone: "warn", text: "No entries in 20 minutes" } + : { tone: "neutral", text: "No entries yet" } + case "idle": + return { + tone: "neutral", + text: , + } + case "receiving": + return { tone: "ok", text: } + } + } + switch (metrics.kind) { + case "off": + return { tone: "neutral", text: "Off" } + case "setup-pending": + case "setup-running": + return { tone: "neutral", text: GCP_METRICS_STATUS[metrics.kind].label } + case "waiting": + return { tone: metrics.overdue ? "warn" : "neutral", text: "No reads yet" } + case "failing": + return { + tone: "crit", + text: + metrics.lastMetricsReceivedAt === null ? ( + "Can't be read" + ) : ( + + ), + } + case "incomplete": + case "stalled": + return { + tone: "warn", + text: ( + + ), + } + case "receiving": + return { tone: "ok", text: } + } +} + +const SIGNAL_LINK = "inline-flex items-center gap-1 text-xs text-muted-foreground hover:text-foreground" + +/** The logs this connection forwarded: ingest stamps each with the connection's id. */ +const logsSearch = (connector: V2GcpConnector) => ({ + attrs: [ + encodeLogAttributeFilter({ + source: "resource", + key: "maple_gcp_connector_id", + value: connector.id, + negated: false, + }), + ], +}) + +/** One capability in an expanded card: its name, its state, and where its data shows up once it does. */ +function Signal({ + name, + signal, + link, +}: { + name: string + signal: { readonly tone: Tone; readonly text: React.ReactNode } + link?: React.ReactNode +}) { + return ( +
    + {name} + {signal.text} + {link === undefined ? null :
    {link}
    } +
    + ) +} + +/** A column of the compact row: a label over a time. */ +function Fact({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
    + {label} + {children} +
    + ) +} + +export function GcpConnection({ + connector, + isAdmin, + nowMs, + onConfigure, + onRemoved, +}: { + connector: V2GcpConnector + isAdmin: boolean + nowMs: number + onConfigure: (stage: "choose" | "apply") => void + onRemoved: (removed: RemovedGcpConnector) => void +}) { + const remove = useAtomSet(MapleApiV2AtomClient.mutation("gcpIntegration", "deleteConnector"), { + mode: "promiseExit", + }) + const label = gcpScopeLabel(connector) + // No run has reported: as far as Maple knows, nothing exists in Google Cloud. + const neverReported = + connector.applied_logs_enabled === null && connector.applied_metrics_enabled === null + const cleanedUp = connector.applied_logs_enabled === false && connector.applied_metrics_enabled === false + const [disconnectOpen, setDisconnectOpen] = useState(false) + + // Resolving to `false` keeps the dialog open for a retry. + const [disconnect, disconnecting] = useAsyncAction(async () => { + const result = await remove({ params: { id: connector.id }, reactivityKeys: GCP_REACTIVITY_KEYS }) + if (Exit.isFailure(result)) { + showErrorToast(result, { title: "Failed to disconnect" }) + return false + } + // Kept unless the cleanup script was seen to run. A script that stopped part of the way, or + // could not reach Maple, leaves resources behind without a report. + if (!cleanedUp) { + onRemoved({ + id: connector.id, + label, + hostProjectId: connector.project_id, + cleanupScript: result.value.cleanup_script, + reported: !neverReported, + }) + } + return true + }) + + const log = gcpLogState(connector, nowMs) + const metrics = gcpMetricsState(connector, nowMs) + const issues = connectionIssues({ + connector, + log, + metrics, + nowMs, + onRunScript: isAdmin ? () => onConfigure("apply") : null, + }) + const lastLog = log.kind === "receiving" || log.kind === "idle" ? log.lastLogReceivedAt : null + const lastRead = "lastMetricsReceivedAt" in metrics ? metrics.lastMetricsReceivedAt : null + const projects = + metrics.kind === "receiving" && metrics.projectCount !== null + ? countLabel(metrics.projectCount, "project") + : null + const collects = [connector.logs_enabled ? "Logs" : null, connector.metrics_enabled ? "key metrics" : null] + .filter((part) => part !== null) + .join(" and ") + + const name = ( +
    +

    + {connector.scope_id} +

    + + {GCP_SCOPE_NAMES[connector.scope_type]} + +
    + ) + const subtitle = [ + collects === "" ? "Collects nothing" : collects.charAt(0).toUpperCase() + collects.slice(1), + projects, + connector.scope_type === "project" ? null : `host ${connector.project_id}`, + ] + .filter((part) => part !== null) + .join(" · ") + const menu = ( + + {isAdmin ? onConfigure("choose")}>Configure : null} + }>View logs + {isAdmin ? ( + <> + + setDisconnectOpen(true)}> + Disconnect + + + ) : null} + + ) + + return ( + + {issues.length === 0 ? ( + // Healthy: one row, the name and the times its data last arrived. +
    +
    + + + +
    + {name} +

    {subtitle}

    +
    +
    +
    + + {lastLog === null ? "Off" : } + + + {lastRead === null ? "Off" : } + +
    + {menu} +
    + ) : ( + <> +
    +
    +
    + {name} + {connector.scope_type === "project" ? null : ( +

    {subtitle}

    + )} +
    +
    {menu}
    +
    + {issues.map((issue) => ( + + ))} +
    +
    + + View logs + + + ) + } + /> + +
    + + )} + + {neverReported ? ( + +
    +
    +
    In Maple
    +
    Removes the connection.
    +
    +
    +
    In Google Cloud
    +
    + Nothing should exist: Maple hasn't seen the setup script run. If it ran + part of the way, Maple offers the cleanup script afterwards. +
    +
    +
    +     + ) : disconnectOpen ? ( + void disconnect().then((done) => done && setDisconnectOpen(false))} + onClose={() => setDisconnectOpen(false)} + /> + ) : null} +     + ) +} + +/** Every connection, and the clock their states turn on. */ +function useGcpConnectors() { + const status = Option.getOrNull(AsyncResult.value(useAtomValue(gcpStatusQuery()))) + return { connectors: status?.connectors ?? [], nowMs: useLiveClock() } +} + +/** The page header's state: the worst connection's, how many of several share it, and how many projects send data. */ +export function GcpHeaderStatus() { + const { connectors, nowMs } = useGcpConnectors() + const states = connectors.map((connector) => gcpConnectionState(connector, nowMs)) + const worst = gcpWorstState(states) + const inWorst = states.filter((state) => state === worst).length + const projects = connectors.reduce( + (sum, connector) => + sum + (connector.scope_type === "project" ? 1 : connector.discovered_project_count), + 0, + ) + return ( +
    + + {GCP_CONNECTION_LABEL[worst]} + {inWorst < connectors.length ? ` · ${inWorst} of ${connectors.length} connections` : null} + {projects > 1 ? ` · ${countLabel(projects, "project")}` : null} +
    + ) +} + +/** + * The Google Cloud integration page. No OAuth: an admin registers a connection for a project, + * folder or organization and runs the generated script in Cloud Shell. The page shows what each + * connection collects as status and numbers. Everything that changes a connection, and the script + * that applies the change, is behind Configure. Everyone but admins sees the status only. + */ +export function GcpIntegrationCard() { + const statusQuery = gcpStatusQuery() + const statusResult = useAtomValue(statusQuery) + const refreshStatus = useAtomRefresh(statusQuery) + + // The gate the API applies. `useIsOrgAdmin` is false until the session lands, so the + // admin-only note waits for a settled session; the controls stay hidden meanwhile. + const isAdmin = useIsOrgAdmin() + const sessionResult = useAtomValue(retainedQuery("auth", "session", {})) + const showNotAdmin = !isAdmin && !Result.isInitial(sessionResult) + + const [target, setTarget] = useState(null) + // The removed connection whose cleanup script is about to be dismissed. + const [dismissing, setDismissing] = useState(null) + const [removed, setRemoved] = useAtom(removedGcpConnectorsAtomFamily(getActiveOrgId() ?? "default")) + // The states turn on the clock: a grant's grace ends, a silent sink becomes overdue. + const nowMs = useLiveClock() + + // Keep the last loaded status if a poll fails. + const status = Option.getOrNull(AsyncResult.value(statusResult)) + const connectors = status?.connectors ?? [] + const metricsAvailable = status?.metrics_available === true + + const settling = connectors.some((connector) => gcpConnectionState(connector, nowMs) !== "healthy") + useIntervalRefresh(refreshStatus, { + intervalMs: settling ? GCP_SETTLING_REFRESH_MS : STEADY_REFRESH_MS, + enabled: connectors.length > 0, + }) + + if (Result.isInitial(statusResult)) { + return + } + if (Result.isFailure(statusResult) && status === null) { + return ( + + ) + } + + const cleanup = removed.map((entry) => ( + +
    +

    + {entry.label} {entry.reported ? "disconnected, not cleaned up" : "disconnected"} +

    +

    + {entry.reported + ? "Google Cloud still holds what the setup script created, and bills for it. Dismiss this once the cleanup script has run." + : "If the setup script ran part of the way, the cleanup script removes what it created."} +

    +
    +
    + + + +
    +     + )) + const dismiss = ( + (open ? undefined : setDismissing(null))} + title="Dismiss the cleanup script?" + description="Dismiss it once the cleanup has run. Maple keeps the script only here, so it can't be shown again." + tone="default" + icon={null} + confirmLabel="Dismiss" + onConfirm={() => { + setRemoved(removed.filter((entry) => entry.id !== dismissing)) + setDismissing(null) + }} + /> + ) + const configure = + target === null ? null : ( + setTarget({ kind: "connection", id: connector.id, stage: "apply" })} + onClose={() => setTarget(null)} + /> + ) + + if (connectors.length === 0) { + return ( +
    + {cleanup} + + + + + + + + + {isAdmin ? ( + <> + + Connect a project, folder or organization. Its logs, workloads and + resources appear here. + + + + ) : ( + + A Maple organization admin connects Google Cloud by running a setup script in + Cloud Shell. + + )} + + + {configure} + {dismiss} +
    + ) + } + + // Worst first: what needs its admin is at the top, however many connections there are. + const ordered = connectors.toSorted( + (a, b) => + GCP_STATE_ORDER.indexOf(gcpConnectionState(a, nowMs)) - + GCP_STATE_ORDER.indexOf(gcpConnectionState(b, nowMs)), + ) + + // Sans for words, mono for ids, times and commands. + return ( +
    + {cleanup} +
    + 1 ? connectors.length : undefined} + actions={ + isAdmin ? ( + + ) : undefined + } + /> + {ordered.map((connector) => ( + setTarget({ kind: "connection", id: connector.id, stage })} + onRemoved={(entry) => setRemoved([...removed, entry])} + /> + ))} + {showNotAdmin ? ( +

    + Only Maple organization admins can add, configure or disconnect connections. +

    + ) : null} +
    + {configure} + {dismiss} +
    + ) +} diff --git a/apps/web/src/components/integrations/integration-catalog.tsx b/apps/web/src/components/integrations/integration-catalog.tsx index dd24562cfa..81dd642a0f 100644 --- a/apps/web/src/components/integrations/integration-catalog.tsx +++ b/apps/web/src/components/integrations/integration-catalog.tsx @@ -7,6 +7,8 @@ import { ChevronRightIcon, CloudflareIcon, GithubIcon, + GoogleCloudIcon, + GoogleCloudMonoIcon, HazelIcon, PlanetScaleIcon, PrometheusIcon, @@ -27,6 +29,7 @@ import { Result, useAtomValue } from "@/lib/effect-atom" import { retainedInternalQuery } from "@/lib/services/common/internal-atom-client" import { retainedQueryV2 } from "@/lib/services/common/v2-atom-client" import { scrapeTargetsListAtom } from "@/lib/services/atoms/scrape-target-atoms" +import { GCP_CONNECTION_LABEL, gcpConnectionState, gcpScopeLabel, gcpWorstState } from "./gcp-connector-state" /** * A chat connector's catalog id. The connector half is data from @@ -39,6 +42,7 @@ export type IntegrationId = | "cloudflare" | "prometheus" | "planetscale" + | "gcp" | "railway" | "warpstream" | "hazel" @@ -94,6 +98,7 @@ export const GITHUB_ACCENT = "#181717" export const HAZEL_ACCENT = "#F46F0F" export const CLOUDFLARE_ACCENT = "#F38020" export const RAILWAY_ACCENT = "#0B0D0E" +export const GCP_ACCENT = "#4285F4" export interface CatalogEntry { readonly id: IntegrationId @@ -172,6 +177,16 @@ const CATALOG: ReadonlyArray = [ accent: PLANETSCALE_COLOR, docsUrl: docsUrl("planetscale"), }, + { + id: "gcp", + category: "infrastructure", + isNew: true, + name: "Google Cloud", + description: "Logs, metrics and resources from a project, folder or organization.", + icon: GoogleCloudIcon, + monoIcon: GoogleCloudMonoIcon, + accent: GCP_ACCENT, + }, { id: "railway", category: "infrastructure", @@ -218,6 +233,13 @@ const CATALOG: ReadonlyArray = [ ...CHAT_ENTRIES, ] +/** + * The Google Cloud status read, shared by the catalog and the card. Called in a render body, like + * every `retainedQueryV2`: the atom it returns is memoized per organization. + */ +export const gcpStatusQuery = () => + retainedQueryV2("gcpIntegration", "status", { reactivityKeys: ["gcpIntegration"] }) + export const catalogEntry = (id: IntegrationId): CatalogEntry => CATALOG.find((entry) => entry.id === id)! /** The catalog this org is shown, in catalog order. */ @@ -293,6 +315,15 @@ export function useIntegrationStatuses(): Partial { + if (status.connectors.length === 0) return NOT_CONNECTED + return { label: countLabel(status.connectors.length, "connection"), variant: "ok" } + }) + .onInitial(() => null) + .orElse(() => STATUS_UNAVAILABLE) const railway: CardStatus | null = Result.builder(railwayResult) .onSuccess((status): CardStatus => { @@ -394,6 +425,7 @@ export function useIntegrationStatuses(): Partial { + const connectors = status.connectors + if (connectors.length === 0) return SET_UP + const now = Date.now() + const states = connectors.map((connector) => gcpConnectionState(connector, now)) + const count = (state: (typeof states)[number]) => states.filter((entry) => entry === state).length + const worst = gcpWorstState(states) + return { + kind: "connected", + health: worst === "attention" || worst === "healthy" ? worst : "pending", + stateLabel: GCP_CONNECTION_LABEL[worst], + context: + connectors.length === 1 + ? gcpScopeLabel(connectors[0]) + : countLabel(connectors.length, "connection"), + // What the connections collect: true in every state, unlike a count of healthy ones. + stat: [ + connectors.some((connector) => connector.logs_enabled) ? "Logs" : null, + connectors.some((connector) => connector.metrics_enabled) + ? "Metrics and resources" + : null, + ] + .filter((part) => part !== null) + .join(" · "), + // Not next to "Waiting for data": the two would contradict each other. + lastSyncLabel: + worst === "waiting" + ? null + : syncedLabel( + maxMs( + connectors + .flatMap((connector) => [ + connector.logs_enabled ? connector.last_log_received_at : null, + connector.metrics_enabled + ? connector.last_metrics_received_at + : null, + ]) + .map((iso) => (iso ? Date.parse(iso) : null)), + ), + "last data", + ), + issue: + worst === "attention" + ? `${countLabel(count("attention"), "connection")} failing` + : worst === "setup-pending" || worst === "changes-pending" + ? "Run setup script" + : null, + } + }) + .onInitial(() => null) + .orElse(() => UNAVAILABLE) const railway: IntegrationOverview = Result.builder(railwayResult) .onSuccess((status): IntegrationOverview => { @@ -754,6 +841,7 @@ export function useIntegrationOverviews(): Record 0 && ( - {attention} need attention + {attention} {attention === 1 ? "needs" : "need"} attention )}
    ) } -function HealthDot({ health }: { health: "healthy" | "attention" | "unavailable" }) { - if (health === "unavailable") return - return -} +/** Dot tone per health. The integration's page header uses it too. */ +export const HEALTH_TONE = { + healthy: "ok", + attention: "warn", + pending: "neutral", + unavailable: "neutral", +} as const function ConnectedRow({ entry, @@ -825,8 +916,8 @@ function ConnectedRow({ {connected.context} )} - - + + {connected?.stateLabel ?? "Status unavailable"} {connected?.stat && ( @@ -841,7 +932,11 @@ function ConnectedRow({ )} {connected?.issue && ( - + {connected.issue} )} diff --git a/apps/web/src/components/integrations/integration-empty-state.tsx b/apps/web/src/components/integrations/integration-empty-state.tsx index 7173c6ebdf..1e85bbb533 100644 --- a/apps/web/src/components/integrations/integration-empty-state.tsx +++ b/apps/web/src/components/integrations/integration-empty-state.tsx @@ -93,7 +93,8 @@ export interface IntegrationFeatureCopy { */ export function IntegrationEmptyFeatures({ children }: { children: React.ReactNode }) { return ( -
      + // On the page's width, not the viewport's: beside two sidebars a tablet-wide page is phone-narrow. +
        {Children.toArray(children).map((child, index) => (
      • new Date(NOW - minutes * 60_000).toISOString() + +const decode = Schema.decodeUnknownSync(V2GcpConnector) +const connector = (over: Record = {}) => + decode({ + id: "gcpc_YofPTrK9782DWwcnXhpcCw", + object: "gcp_connector", + scope_type: "project", + scope_id: "acme-prod", + project_id: "acme-prod", + logs_enabled: true, + metrics_enabled: true, + created_at: ago(1), + last_log_received_at: null, + last_log_error: null, + applied_logs_enabled: null, + applied_metrics_enabled: null, + setup_reported_at: null, + last_metrics_received_at: null, + last_metrics_error: null, + discovered_project_count: 0, + last_resources_error: null, + ...over, + }) + +const ORGANIZATION = { + scope_type: "organization", + scope_id: "123456789012", + project_id: "acme-observability", + created_at: ago(60 * 24 * 7), +} + +const SCRIPT = `#!/usr/bin/env bash +# Maple setup for Google Cloud project acme-prod +# Log forwarding: on +# Metrics and resources: on +# Safe to run again: it sets up what is on and removes what is off. +# Keep it private: PUSH_ENDPOINT contains this connection's secret. +set -euo pipefail +# gcloud would log every argument, the secret too, and the access token to ~/.config/gcloud/logs. +export CLOUDSDK_CORE_DISABLE_PROMPTS=1 CLOUDSDK_CORE_DISABLE_FILE_LOGGING=1 + +# ---- Which logs are forwarded ---- +LOG_FILTER_MODE='set' +LOG_FILTER='severity>=DEFAULT' +PROJECT='acme-prod' +PUSH_ENDPOINT='https://ingest.maple.dev/v1/logpush/gcp/gcpc_YofPTrK9782DWwcnXhpcCw?secret=lab-fixture' +SINK_CREATED=0 +ACCOUNT_CREATED=0 + +gcloud services enable pubsub.googleapis.com logging.googleapis.com monitoring.googleapis.com \\ + cloudasset.googleapis.com --project="$PROJECT" +gcloud pubsub topics create maple-logs-7f3k --project="$PROJECT" +gcloud logging sinks create maple-7f3k "pubsub.googleapis.com/projects/$PROJECT/topics/maple-logs-7f3k" \\ + --log-filter="$LOG_FILTER" --project="$PROJECT" +gcloud iam service-accounts create maple-7f3k --display-name="Maple (read-only)" --project="$PROJECT" +` + +const FIRST_RUN: ApplyOpened = { at: NOW - 30_000, rerun: false, seen: null } +const noop = () => {} + +type ApplyFixture = Omit + +const STATES = { + new: { + label: "New connection", + note: "Interactive. Continue on the Data step calls the API and fails here.", + }, + configure: { label: "Configure", note: "An existing connection's Data step." }, + waiting: { label: "Run · waiting", note: "Just created, nothing reported yet." }, + reporting: { label: "Run · reporting", note: "Log forwarding reported, metrics still running." }, + overdue: { label: "Run · overdue", note: "Six minutes without a report." }, + removing: { label: "Run · org, removing logs", note: "An organization turning log forwarding off." }, + failed: { label: "Run · script failed", note: "The script couldn't be loaded." }, + connected: { label: "Connected", note: "First run confirmed, data not in yet." }, + applied: { label: "Applied", note: "A reconfigured organization, receiving." }, +} as const +type LabState = keyof typeof STATES + +const APPLY = { + waiting: { + title: "Connect Google Cloud", + connector: connector(), + created: true, + logFilter: "default", + script: SCRIPT, + failure: null, + opened: FIRST_RUN, + }, + reporting: { + title: "Connect Google Cloud", + connector: connector({ applied_logs_enabled: true, setup_reported_at: ago(0.3) }), + created: true, + logFilter: "default", + script: SCRIPT, + failure: null, + opened: FIRST_RUN, + }, + overdue: { + title: "Connect Google Cloud", + connector: connector({ created_at: ago(6) }), + created: true, + logFilter: "include_gke_container_logs", + script: SCRIPT, + failure: null, + opened: FIRST_RUN, + }, + removing: { + title: "Organization 123456789012", + connector: connector({ + ...ORGANIZATION, + logs_enabled: false, + applied_logs_enabled: true, + applied_metrics_enabled: true, + setup_reported_at: ago(60 * 24), + last_log_received_at: ago(1), + last_metrics_received_at: ago(3), + discovered_project_count: 14, + }), + created: false, + logFilter: "keep", + script: SCRIPT, + failure: null, + opened: { at: NOW - 30_000, rerun: false, seen: ago(60 * 24) }, + }, + failed: { + title: "Connect Google Cloud", + connector: connector(), + created: true, + logFilter: "default", + script: null, + failure: { message: "Couldn't load the script.", retry: noop, retrying: false }, + opened: FIRST_RUN, + }, + connected: { + title: "Connect Google Cloud", + connector: connector({ + applied_logs_enabled: true, + applied_metrics_enabled: true, + setup_reported_at: ago(3), + }), + created: true, + logFilter: "default", + script: SCRIPT, + failure: null, + opened: FIRST_RUN, + }, + applied: { + title: "Organization 123456789012", + connector: connector({ + ...ORGANIZATION, + applied_logs_enabled: true, + applied_metrics_enabled: true, + setup_reported_at: ago(4), + last_log_received_at: ago(1), + last_metrics_received_at: ago(3), + discovered_project_count: 14, + }), + created: false, + logFilter: "keep", + script: SCRIPT, + failure: null, + opened: { at: NOW - 8 * 60_000, rerun: true, seen: ago(60 * 24) }, + }, +} satisfies { readonly [State in Exclude]: ApplyFixture } + +const CONFIGURED = connector({ + ...ORGANIZATION, + applied_logs_enabled: true, + applied_metrics_enabled: true, + setup_reported_at: ago(60 * 24), + last_log_received_at: ago(1), + last_metrics_received_at: ago(3), + discovered_project_count: 14, +}) + +/** Every step and run state of the Google Cloud connect dialog, over fixture connectors. */ +export function GcpConnectLab() { + const [state, setState] = useState("new") + // Bumped to reopen the dialog after it is closed, from its first step. + const [opening, setOpening] = useState(0) + const [open, setOpen] = useState(true) + const close = () => setOpen(false) + const show = (next: LabState) => { + setState(next) + setOpening(opening + 1) + setOpen(true) + } + + return ( +
        +
        +

        Google Cloud connect flow

        +

        {STATES[state].note}

        +
        +
        + { + const next = (Object.keys(STATES) as LabState[]).find( + (key) => key !== state && values.includes(key), + ) + if (next !== undefined) show(next) + }} + > + {(Object.keys(STATES) as LabState[]).map((key) => ( + + {STATES[key].label} + + ))} + + {open ? null : ( + + )} +
        + {!open ? null : state === "new" ? ( + + ) : state === "configure" ? ( + + ) : ( + + + + )} +
        + ) +} diff --git a/apps/web/src/lab/gcp-connections-lab.tsx b/apps/web/src/lab/gcp-connections-lab.tsx new file mode 100644 index 0000000000..7496ac73f2 --- /dev/null +++ b/apps/web/src/lab/gcp-connections-lab.tsx @@ -0,0 +1,110 @@ +import { Schema } from "effect" +import { V2GcpConnector } from "@maple/domain/http/v2" + +import { GcpConnection } from "@/components/integrations/gcp-integration-card" +import { DashboardLayout } from "@/components/layout/dashboard-layout" + +const NOW = Date.now() +const ago = (minutes: number) => new Date(NOW - minutes * 60_000).toISOString() + +/** A project connection whose script ran an hour ago and that delivers both. */ +const connector = (over: Partial = {}): V2GcpConnector => + Schema.decodeUnknownSync(V2GcpConnector)({ + id: "gcpc_YofPTrK9782DWwcnXhpcCw", + object: "gcp_connector", + scope_type: "project", + scope_id: "maple-development-project", + project_id: "maple-development-project", + logs_enabled: true, + metrics_enabled: true, + created_at: ago(90), + last_log_received_at: ago(1), + last_log_error: null, + applied_logs_enabled: true, + applied_metrics_enabled: true, + setup_reported_at: ago(60), + last_metrics_received_at: ago(4), + last_metrics_error: null, + discovered_project_count: 1, + last_resources_error: null, + ...over, + }) + +const SCENARIOS: ReadonlyArray<{ title: string; connector: V2GcpConnector }> = [ + { title: "Healthy project", connector: connector() }, + { + title: "Healthy folder", + connector: connector({ + scope_type: "folder", + scope_id: "482910374615", + project_id: "maple-shared-logging", + discovered_project_count: 13, + last_log_received_at: ago(0.2), + last_metrics_received_at: ago(2), + }), + }, + { + title: "Waiting for the first data", + connector: connector({ + setup_reported_at: ago(3), + last_log_received_at: null, + last_metrics_received_at: null, + }), + }, + { + title: "Overdue", + connector: connector({ + setup_reported_at: ago(24), + last_log_received_at: null, + last_metrics_received_at: null, + }), + }, + { + title: "Log forwarding failing", + connector: connector({ + last_log_received_at: ago(120), + last_log_error: + "The log sink can't publish to Maple's topic. Run the setup script again: it restores the permission. Google answered: PERMISSION_DENIED: pubsub.topics.publish.", + last_metrics_received_at: ago(2), + }), + }, + { + title: "Setup pending", + connector: connector({ + applied_logs_enabled: null, + applied_metrics_enabled: null, + setup_reported_at: null, + last_log_received_at: null, + last_metrics_received_at: null, + }), + }, +] + +/** Each state of a Google Cloud connection over fixture data, as an admin sees it. */ +export function GcpConnectionsLab() { + return ( + + + + + +
        + {SCENARIOS.map((scenario) => ( +
        +

        {scenario.title}

        + undefined} + onRemoved={() => undefined} + /> +
        + ))} +
        +         +         +         +         + ) +} diff --git a/apps/web/src/lab/registry.ts b/apps/web/src/lab/registry.ts index 62593a6cb3..2907f69373 100644 --- a/apps/web/src/lab/registry.ts +++ b/apps/web/src/lab/registry.ts @@ -52,6 +52,22 @@ export const LAB_ENTRIES: ReadonlyArray = [ kind: "lab", session: "none", }, + { + path: "/lab/gcp-connect", + title: "Google Cloud connect", + description: + "Every step and run state of the Google Cloud connect dialog over fixture connectors, no API needed.", + kind: "lab", + session: "none", + }, + { + path: "/lab/gcp-connections", + title: "Google Cloud connections", + description: + "A Google Cloud connection in each state, healthy to failing, over fixture connectors, no API needed.", + kind: "lab", + session: "none", + }, { path: "/lab/onboarding", title: "Onboarding", diff --git a/apps/web/src/lib/docs.ts b/apps/web/src/lib/docs.ts index fbe9afb089..775f47af6d 100644 --- a/apps/web/src/lib/docs.ts +++ b/apps/web/src/lib/docs.ts @@ -32,6 +32,8 @@ export const DOCS = { docker: "/docs/infrastructure/docker", kubernetes: "/docs/infrastructure/kubernetes", cloudflare: "/docs/integrations/cloudflare", + gcp: "/docs/integrations/gcp", + gcpOpenTelemetry: "/docs/integrations/gcp-opentelemetry", planetscale: "/docs/integrations/planetscale", prometheus: "/docs/integrations/prometheus", warpstream: "/docs/integrations/warpstream", diff --git a/apps/web/src/routeTree.gen.ts b/apps/web/src/routeTree.gen.ts index a2acd187dc..f4529645a6 100644 --- a/apps/web/src/routeTree.gen.ts +++ b/apps/web/src/routeTree.gen.ts @@ -59,6 +59,8 @@ import { Route as LabCloudflareRouteImport } from './routes/lab/cloudflare' import { Route as LabCodeReviewReposRouteImport } from './routes/lab/code-review-repos' import { Route as LabErrorsRouteImport } from './routes/lab/errors' import { Route as LabFlowRouteImport } from './routes/lab/flow' +import { Route as LabGcpConnectRouteImport } from './routes/lab/gcp-connect' +import { Route as LabGcpConnectionsRouteImport } from './routes/lab/gcp-connections' import { Route as LabInfraRouteImport } from './routes/lab/infra' import { Route as LabLoadersRouteImport } from './routes/lab/loaders' import { Route as LabLogsRouteImport } from './routes/lab/logs' @@ -371,6 +373,16 @@ const LabFlowRoute = LabFlowRouteImport.update({ path: '/flow', getParentRoute: () => LabRouteRoute, } as any) +const LabGcpConnectRoute = LabGcpConnectRouteImport.update({ + id: '/gcp-connect', + path: '/gcp-connect', + getParentRoute: () => LabRouteRoute, +} as any) +const LabGcpConnectionsRoute = LabGcpConnectionsRouteImport.update({ + id: '/gcp-connections', + path: '/gcp-connections', + getParentRoute: () => LabRouteRoute, +} as any) const LabInfraRoute = LabInfraRouteImport.update({ id: '/infra', path: '/infra', @@ -732,6 +744,8 @@ export interface FileRoutesByFullPath { '/lab/code-review-repos': typeof LabCodeReviewReposRoute '/lab/errors': typeof LabErrorsRoute '/lab/flow': typeof LabFlowRoute + '/lab/gcp-connect': typeof LabGcpConnectRoute + '/lab/gcp-connections': typeof LabGcpConnectionsRoute '/lab/infra': typeof LabInfraRoute '/lab/loaders': typeof LabLoadersRoute '/lab/logs': typeof LabLogsRoute @@ -844,6 +858,8 @@ export interface FileRoutesByTo { '/lab/code-review-repos': typeof LabCodeReviewReposRoute '/lab/errors': typeof LabErrorsRoute '/lab/flow': typeof LabFlowRoute + '/lab/gcp-connect': typeof LabGcpConnectRoute + '/lab/gcp-connections': typeof LabGcpConnectionsRoute '/lab/infra': typeof LabInfraRoute '/lab/loaders': typeof LabLoadersRoute '/lab/logs': typeof LabLogsRoute @@ -958,6 +974,8 @@ export interface FileRoutesById { '/lab/code-review-repos': typeof LabCodeReviewReposRoute '/lab/errors': typeof LabErrorsRoute '/lab/flow': typeof LabFlowRoute + '/lab/gcp-connect': typeof LabGcpConnectRoute + '/lab/gcp-connections': typeof LabGcpConnectionsRoute '/lab/infra': typeof LabInfraRoute '/lab/loaders': typeof LabLoadersRoute '/lab/logs': typeof LabLogsRoute @@ -1073,6 +1091,8 @@ export interface FileRouteTypes { | '/lab/code-review-repos' | '/lab/errors' | '/lab/flow' + | '/lab/gcp-connect' + | '/lab/gcp-connections' | '/lab/infra' | '/lab/loaders' | '/lab/logs' @@ -1185,6 +1205,8 @@ export interface FileRouteTypes { | '/lab/code-review-repos' | '/lab/errors' | '/lab/flow' + | '/lab/gcp-connect' + | '/lab/gcp-connections' | '/lab/infra' | '/lab/loaders' | '/lab/logs' @@ -1298,6 +1320,8 @@ export interface FileRouteTypes { | '/lab/code-review-repos' | '/lab/errors' | '/lab/flow' + | '/lab/gcp-connect' + | '/lab/gcp-connections' | '/lab/infra' | '/lab/loaders' | '/lab/logs' @@ -1805,6 +1829,20 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof LabFlowRouteImport parentRoute: typeof LabRouteRoute } + '/lab/gcp-connect': { + id: '/lab/gcp-connect' + path: '/gcp-connect' + fullPath: '/lab/gcp-connect' + preLoaderRoute: typeof LabGcpConnectRouteImport + parentRoute: typeof LabRouteRoute + } + '/lab/gcp-connections': { + id: '/lab/gcp-connections' + path: '/gcp-connections' + fullPath: '/lab/gcp-connections' + preLoaderRoute: typeof LabGcpConnectionsRouteImport + parentRoute: typeof LabRouteRoute + } '/lab/infra': { id: '/lab/infra' path: '/infra' @@ -2246,6 +2284,8 @@ interface LabRouteRouteChildren { LabCodeReviewReposRoute: typeof LabCodeReviewReposRoute LabErrorsRoute: typeof LabErrorsRoute LabFlowRoute: typeof LabFlowRoute + LabGcpConnectRoute: typeof LabGcpConnectRoute + LabGcpConnectionsRoute: typeof LabGcpConnectionsRoute LabInfraRoute: typeof LabInfraRoute LabLoadersRoute: typeof LabLoadersRoute LabLogsRoute: typeof LabLogsRoute @@ -2281,6 +2321,8 @@ const LabRouteRouteChildren: LabRouteRouteChildren = { LabCodeReviewReposRoute: LabCodeReviewReposRoute, LabErrorsRoute: LabErrorsRoute, LabFlowRoute: LabFlowRoute, + LabGcpConnectRoute: LabGcpConnectRoute, + LabGcpConnectionsRoute: LabGcpConnectionsRoute, LabInfraRoute: LabInfraRoute, LabLoadersRoute: LabLoadersRoute, LabLogsRoute: LabLogsRoute, diff --git a/apps/web/src/routes/integrations.tsx b/apps/web/src/routes/integrations.tsx index 317cbf106c..36f2602dab 100644 --- a/apps/web/src/routes/integrations.tsx +++ b/apps/web/src/routes/integrations.tsx @@ -9,12 +9,17 @@ import { CloudflareAccountCard, CloudflareHeaderActions, } from "@/components/integrations/cloudflare-account-card" +import { + GcpHeaderStatus, + GcpIntegrationCard, +} from "@/components/integrations/gcp-integration-card" import { GithubIntegrationCard } from "@/components/integrations/github-integration-card" import { HazelIntegrationCard } from "@/components/integrations/hazel-integration-card" import { PlanetScaleIntegrationCard } from "@/components/integrations/planetscale-integration-card" import { RailwayIntegrationCard } from "@/components/integrations/railway-integration-card" import { ChatIntegrationCard } from "@/components/integrations/chat-integration-card" import { + HEALTH_TONE, IntegrationCatalog, IntegrationIconPlate, IntegrationsSummary, @@ -230,6 +235,8 @@ function IntegrationsPage() { ) : integration === "planetscale" ? ( + ) : integration === "gcp" ? ( + ) : integration === "railway" ? ( ) : ( @@ -275,13 +282,15 @@ function IntegrationHeader({ integration }: { integration: IntegrationId }) { />
        -

        {entry.name}

        +

        {entry.name}

        {/* The badge covers the non-connected states; connected reads as the status line. */} {!connected && status ? {status.label} : null}
        - {connected && statusLine ? ( + {connected && integration === "gcp" ? ( + + ) : connected && statusLine ? (
        - + {statusLine} diff --git a/apps/web/src/routes/lab/gcp-connect.tsx b/apps/web/src/routes/lab/gcp-connect.tsx new file mode 100644 index 0000000000..4e002d7308 --- /dev/null +++ b/apps/web/src/routes/lab/gcp-connect.tsx @@ -0,0 +1,5 @@ +import { createFileRoute } from "@tanstack/react-router" + +import { GcpConnectLab } from "@/lab/gcp-connect-lab" + +export const Route = createFileRoute("/lab/gcp-connect")({ component: GcpConnectLab }) diff --git a/apps/web/src/routes/lab/gcp-connections.tsx b/apps/web/src/routes/lab/gcp-connections.tsx new file mode 100644 index 0000000000..dc796111c5 --- /dev/null +++ b/apps/web/src/routes/lab/gcp-connections.tsx @@ -0,0 +1,5 @@ +import { createFileRoute } from "@tanstack/react-router" + +import { GcpConnectionsLab } from "@/lab/gcp-connections-lab" + +export const Route = createFileRoute("/lab/gcp-connections")({ component: GcpConnectionsLab })