Skip to content

Refresh the sumologic provider and microsite (provider-utils 0.7.9, Makefile pipeline, snake_case surface, GitHub Pages docs) - #2

Merged
jeffreyaven merged 3 commits into
mainfrom
feature/provider-dev
Sep 15, 2026
Merged

jeffreyaven merged 3 commits into
mainfrom
feature/provider-dev

Conversation

@jeffreyaven

Copy link
Copy Markdown
Contributor

Summary

Rebuilds the sumologic provider from the current vendor OpenAPI document plus a hand-authored Collector Management API spec, on the same repository pattern as the clickhouse and github provider repos: a deterministic make pipeline, a durable operation-mapping CSV, offline / mock-integration / meta-route test layers, a live smoke suite, a CLAUDE.md, and a Docusaurus 3.10 microsite deployed with GitHub Pages.

Provider

  • Spec pinned by hash (provider-dev/config/spec_pin.json): https://api.sumologic.com/docs/sumologic-api.yaml, OpenAPI 3.0.0, 305 paths, 443 operations, 58 tags. The old sumologic-openapi.yaml URL is a 404.
  • The Collector Management API (collectors, sources, collector upgrades - the surface Terraform's sumologic_collector / sumologic_*_source use) is not in the vendor document; it is maintained in provider-dev/downloaded/sumologic-collector-management-api.yaml (18 operations) and merged at split time.
  • 50 services (58 tags -> 49 via provider-dev/config/service_names.json, plus collectors), 163 resources, 451 methods: 199 SELECT, 50 INSERT, 53 UPDATE, 1 REPLACE, 49 DELETE, 99 EXEC, 10 reason-coded skips (deprecated v1 source templates, multipart lookup upload, PDF report result).
  • Auth: HTTP basic on SUMOLOGIC_ACCESSID / SUMOLOGIC_ACCESSKEY (Terraform parity).
  • Deployment: region server variable on https://api.{region}.sumologic.com/api, enum of the 11 deployments, x-stackQL-envVar: SUMOLOGIC_ENVIRONMENT (Terraform's variable), default us2 kept from the previous release; a WHERE region value wins.
  • snake_case surface: snake_case_aliases: true on the provider config plus request.nativeCasing: camel per method (post-process). Body-less EXEC methods carry no request block (stackql's EXEC analyzer fails on a request block without a body schema).
  • Request bodies: --naive-req-body-translate. The Collector API wraps bodies in collector / source objects, so those INSERTs take one JSON-valued column.
  • Pagination: service-level token (query) / next (body) config, with post-process overrides for nextToken (data archiving, logs data forwarding), token (saved log searches) and the POST-driven OpenTelemetry collectors list whose cursor travels in the request body.
  • Pushdown: every declared query / header parameter is a WHERE parameter sent on the wire (email, name, filter, query, ids, If-Match, ...).
  • Lifecycle operations are EXEC methods on their parent resource (users.unlock, partitions.decommission, scheduled_views.pause, roles.assign_user); async triads are <x>_jobs (EXEC start + SELECT status) and <x>_results. Seven EXEC-only resources remain, all genuinely action-only.
  • v1 / v2 pairs both exist with a _v2 suffix on the newer surface (roles_v2, apps_v2, metrics_searches_v2), matching the Terraform resource names.
  • GET /v1/account/accountOwner returns a bare JSON string; post-process wraps it with a response transform so it projects a row.

Mapping contract

provider-dev/config/all_services.csv is committed as the durable record of every operation's resource, method, SQL verb and object key. map_operations.mjs is fill-only by default (new operations derived, retired rows pruned, committed rows untouched); --rebuild re-derives everything for deliberate renames and --check reports rule drift without writing. Verified stable across repeated make mappings runs. compare_predecessor.mjs dispositions every method of the previous release against this build and writes the README "Breaking changes" section: 151 carried, 105 moved, 11 retired, 195 new; 92 resource renames listed.

Tooling

  • @stackql/provider-utils 0.7.9, @stackql/pgwire-lite 1.0.2, Node >= 20.
  • Makefile: make all (deps, spec refresh, split, pre-normalize, normalize, mappings, generate + post-process, compare, offline + integration + meta-route tests, docs, site build), make spec-check (fail on drift), make smoke-test / make smoke-test-live / make smoke-cleanup.
  • bin/split.mjs fails on any tag missing from the service map; pre_normalize.mjs unwraps the spec's Redoc-relative anchors; post_process.mjs validates and fails without writing.
  • .github/workflows/build-and-test.yml: build from the pinned spec, generation-drift check, Breaking Changes check, all non-live tests, docs generation; secret-gated live smoke on pushes; weekly spec-drift job that opens an issue.
  • CLAUDE.md, .env.example, refreshed .gitignore (provider-dev/source and website/docs are committed, generated artefacts are reviewable diffs).

Tests

Suite Result
tests/offline_validation.mjs (SHOW / DESCRIBE against the file registry) 30/30
tests/integration/ (mock Sumo Logic API, row-level: auth, region resolution, pagination across pages, pushdown, snake -> camel on the wire, user / role / collector / source lifecycles, EXECs, SCIM, scalar, 404 envelope) 54/54
bin/test-meta-routes.cjs (SHOW / DESCRIBE over every resource via stackql srv) 50 services, 163 resources, 451 methods, 0 failures
website build success, no broken links or anchors

Run under WSL with stackql v0.11.669. The live smoke suite (tests/smoke_test.mjs) has not been run in this PR - the maintainer's account is locked out; it needs .env with SUMOLOGIC_ACCESSID, SUMOLOGIC_ACCESSKEY, SUMOLOGIC_ENVIRONMENT:

make smoke-test                            # local provider
make smoke-test SMOKE_ARGS="--read-only"   # no writes
make smoke-test-live                       # published provider, after registry publish

It creates only free objects (a hosted collector with an HTTP source, a role, a personal folder, a monitor folder), all named stackql-smoke-* and swept before and after the run.

Docs microsite

  • Docusaurus 3.10.2 on the shared stackql/docusaurus-config (vendored at build time); showLastUpdateTime enabled so every page carries a "Last updated on ..." stamp.
  • Landing page mastered in provider-dev/docgen/provider-data/headerContent1.txt / headerContent2.txt: installation, authentication, deployment, casing, pagination and pushdown, and getting-started queries (users and roles, collectors and sources, content and monitors, partitions and budgets, provision / mutate / tear down, lifecycle EXECs, async jobs).
  • Docs generated with --snake-case-aliases; website/scripts/sanitize-docs.mjs escapes MDX-hostile description text and annotates region as "required unless SUMOLOGIC_ENVIRONMENT is set".
  • Deployed with GitHub Pages through prod-web-deploy.yml (pushes to main touching website/**); website/static/CNAME now pins sumologic-provider.stackql.io (the previous file carried the snowflake hostname). No Netlify, no gh-pages branch. One-time repo setup: Pages source = GitHub Actions, custom domain, enforce HTTPS; DNS CNAME sumologic-provider.stackql.io -> stackql.github.io..

Behaviour notes for reviewers

  • any-sdk lists a server variable as required whenever its env var is unset, regardless of default. With SUMOLOGIC_ENVIRONMENT unset a SELECT still falls back to us2, but INSERT / UPDATE / DELETE / EXEC must supply region. Documented on the landing page and in the README.
  • Booleans in SET must be quoted (is_active = 'false'); the stackql parser rejects a bare true / false. Documented with an example.
  • any-sdk infers SELECT from method names select / list / aggregatedList / get on resources with no SQL verbs; the mapping validator now rejects those names on EXEC methods.
  • The docs landing page reports 213 resources because docgen counts service index files; the provider has 163.

Follow-ups

  1. Run make smoke-test with a working account and fix anything it surfaces.
  2. Publish provider-dev/openapi/src/sumologic to providers/src in stackql-provider-registry (dev branch), then make smoke-test-live.
  3. Confirm the GitHub Pages settings on the repository.

🤖 Generated with Claude Code

@jeffreyaven jeffreyaven self-assigned this Sep 14, 2026
jeffreyaven and others added 2 commits September 15, 2026 09:42
The generator writes providerServices in filesystem readdir order, which is
sorted on Windows (NTFS) but not on Linux, so the tree generated locally and
the one CI regenerates differed in the order of log_searches and
logs_data_forwarding and the generation-drift gate failed. post_process.mjs
now rewrites providerServices in byte order; provider.yaml regenerated under
Linux (WSL) so it matches CI.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The website lockfile resolves mermaid 12 (via @docusaurus/theme-mermaid),
which requires Node >= 22.12, so `yarn install --frozen-lockfile` failed on
the Node 20 runners of the web workflows. Both web workflows now use Node 22
(current LTS); website/package.json declares the engine and the README /
CLAUDE.md note that the provider pipeline itself still runs on Node >= 20.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jeffreyaven
jeffreyaven merged commit 1823c1f into main Sep 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant