feat: standalone pelican tests across accounts with shared sending

.downstream/native-domains.md

@@ -19,17 +19,18 @@ nothing. While no Codex runtime row is stored,
 
 ## Codex gateway borrowing
 
-The Extensions sidebar has one Codex Borrowing entry. Its in-page navigation
-switches between three independently addressable administrator pages: settings at
+The Extensions sidebar has separate Codex Borrowing and Pelican Tests entries.
+Borrowing's in-page navigation switches between settings at
 `/admin/codex-gateway-borrow`, route status at
-`/admin/codex-gateway-borrow/status`, and manual comparison/history at
-`/admin/codex-pelican-comparison`. Opening or navigating these pages only reads
+`/admin/codex-gateway-borrow/status`. Pelican generation/history lives at
+`/admin/pelican-tests`; `/admin/codex-pelican-comparison` redirects there.
+Opening or navigating these pages only reads
 local data. Settings drafts contain configuration IDs/options only, persist in
 the browser session, and never affect the saved configuration used by status or
 generation. Only explicit actions save, prepare, verify or generate. The status
 page polls read-only status every five seconds while preparation/verification is
 active; local countdowns expire displayed routes without renewing them. Leaving
-the comparison page cancels its active generation.
+the Pelican page cancels its queued and active generation.
 
 `GET/PUT /admin/codex-gateway-borrow/config` owns one JSON setting,
 `codex_gateway_borrow_config`: `enabled`, `source_account_ids`,
@@ -54,17 +55,44 @@ one finite preparation; business cache misses prepare synchronously. There is
 no renewal timer. `/prepare` and `/verify` explicitly prepare or revalidate.
 All probes only record observations, including failures before business dispatch.
 
-`POST /admin/codex-gateway-borrow/tests` accepts a timestamped UUIDv7 and explicit
-account/model pairs. It sends the fixed pelican prompt once per pair, only with
-matching unexpired qualification, using at most three shared generation slots
-and serializing each account. It does not prepare missing routes. An expired
-or future UUID is rejected; replaying a saved UUID cannot dispatch again.
-Migration 275 adds only this feature's task/result tables. Records expire 24
-hours after task creation, disappear from reads immediately, and are removed
-by scoped minute cleanup. Output/errors retain their original bytes.
+The historical `/admin/codex-gateway-borrow/tests` endpoints retain their
+cache-qualified, 90-second request contract. They share execution admission with
+the independent tests described below.
+
+## Independent Pelican tests
+
+`/admin/pelican-tests` provides local account/model options, explicit generation
+with SSE progress, and stored history/detail. Candidates include all platforms,
+account types and account states, using local mappings, saved catalogs and
+platform defaults. Option reads never refresh credentials or request an upstream
+model directory. Each selected account/model receives the fixed Pelican prompt
+once. The server chooses its actual existing account sending path, model mapping,
+credentials, identity headers, proxy, TLS, protocol and borrowing configuration;
+it creates no API Key, group policy or client handshake, and never switches to a
+different account on failure. Unsupported text capabilities retain a skip reason.
+
+All batches share ten model execution slots, with one request per account.
+Nested borrowing probes participate in the same coordination and normal business
+account capacity. Generation starts after preparation and admission; its default
+budget is 600 seconds, configurable from 60 to 1800 seconds. Queue, preparation
+and generation durations are recorded separately. Observed business account slots
+renew every 30 seconds; lost capacity cancels the operation. Existing credential
+refresh remains available, while account health, quota, scheduling, affinity,
+usage and billing writes are suppressed, including background callbacks.
+
+Task IDs are timestamped UUIDv7 values. Expired/future IDs are rejected and replay
+cannot dispatch again. Migration 275 owns the task/result tables; migration 276
+adds budgets, execution phases and actual invocation metadata in place, preserving
+old records and raw BYTEA output. Existing records retain their 90-second budget.
+Records expire 24 hours after task creation, immediately disappear from reads,
+and are removed by scoped minute cleanup. Output/errors retain original bytes;
+binary upstream envelopes use lossless Base64. Truncated or incomplete output is
+recorded without another generation or automatic repair.
 
 Result previews use short random capability URLs under
-`/codex-gateway-borrow/preview/`, an opaque script-enabled sandbox and their own
+`/pelican-tests/preview/` (the historical borrowing path remains an alias),
+expire within 30 minutes and the task's remaining lifetime, and use an opaque
+script-enabled sandbox and their own
 response CSP. Inline animation and external scripts/styles/fonts/images are
 allowed. Preview documents receive no administrator token or UI bridge.
 Adaptation sources and retained notices are in

.github/workflows/downstream-verify.yml

@@ -44,7 +44,7 @@ jobs:
       - name: Native gateway borrowing contracts
         working-directory: backend
         run: |
-          go test -p=1 -tags unit ./internal/service ./internal/repository ./internal/handler ./internal/handler/admin -run '^TestCodexGatewayBorrow' -count=1
+          go test -p=1 -tags unit ./internal/service ./internal/repository ./internal/handler ./internal/handler/admin -run '^Test(CodexGatewayBorrow|Pelican|AccountObservation|ObservedAccountLease)' -count=1
           go test -p=1 -tags integration ./internal/repository -run '^TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle$' -count=1
       - name: Upstream automation classifier tests
         run: |
@@ -145,6 +145,7 @@ jobs:
             src/views/admin/__tests__/CodexGatewayBorrowView.spec.ts \
             src/views/admin/__tests__/CodexGatewayBorrowStatusView.spec.ts \
             src/views/admin/__tests__/CodexPelicanComparisonView.spec.ts \
+            src/api/admin/__tests__/pelicanTests.spec.ts \
             src/components/admin/__tests__/CodexBorrowNav.spec.ts \
             src/router/__tests__/codex-borrow-routes.spec.ts \
             src/components/account/__tests__/EditAccountModal.spec.ts \

backend/internal/handler/admin/codex_gateway_borrow_handler_test.go

@@ -84,6 +84,7 @@ func TestCodexGatewayBorrowHandlerPreviewUsesOnlyCapabilityAndOverridesCSP(t *te
 	r := gin.New()
 	r.Use(middleware.SecurityHeaders(config.CSPConfig{Enabled: true}, nil))
 	r.GET("/api/v1/codex-gateway-borrow/preview/:cap/index.html", h.ServePreview)
+	r.GET("/api/v1/pelican-tests/preview/:cap/index.html", h.ServePreview)
 	r.GET("/", func(c *gin.Context) { c.String(http.StatusOK, "main app") })
 	w := httptest.NewRecorder()
 	request := httptest.NewRequest(http.MethodGet, decorated.PreviewURL, nil)

backend/internal/handler/admin/pelican_test_handler.go

@@ -0,0 +1,218 @@
+package admin
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/Wei-Shaw/sub2api/internal/pkg/response"
+	"github.com/Wei-Shaw/sub2api/internal/server/middleware"
+	"github.com/Wei-Shaw/sub2api/internal/service"
+	"github.com/gin-gonic/gin"
+	"github.com/google/uuid"
+)
+
+// PelicanTestHandler owns the independent administrator entry point. Borrowing
+// configuration and preparation remain in CodexGatewayBorrowHandler; this entry
+// selects only an account, a model, an effort, and a generation budget.
+type PelicanTestHandler struct {
+	catalog *service.PelicanTestCatalog
+	runner  *service.CodexGatewayBorrowTestRunner
+	preview *service.CodexGatewayBorrowPreviewService
+}
+
+func NewPelicanTestHandler(catalog *service.PelicanTestCatalog, runner *service.CodexGatewayBorrowTestRunner, preview *service.CodexGatewayBorrowPreviewService) *PelicanTestHandler {
+	return &PelicanTestHandler{catalog: catalog, runner: runner, preview: preview}
+}
+
+func (h *PelicanTestHandler) GetOptions(c *gin.Context) {
+	h.options(c, nil)
+}
+
+func (h *PelicanTestHandler) GetAccountOptions(c *gin.Context) {
+	codexGatewayBorrowPrivateHeaders(c)
+	var request struct {
+		AccountIDs []int64 `json:"account_ids"`
+	}
+	if err := c.ShouldBindJSON(&request); err != nil {
+		response.BadRequest(c, err.Error())
+		return
+	}
+	h.options(c, request.AccountIDs)
+}
+
+func (h *PelicanTestHandler) options(c *gin.Context, accountIDs []int64) {
+	codexGatewayBorrowPrivateHeaders(c)
+	options, err := h.catalog.Options(c.Request.Context(), accountIDs)
+	if err != nil {
+		if errors.Is(err, service.ErrPelicanTestOptionsInvalidRequest) {
+			response.BadRequest(c, err.Error())
+		} else {
+			response.Error(c, http.StatusInternalServerError, err.Error())
+		}
+		return
+	}
+	response.Success(c, options)
+}
+
+func (h *PelicanTestHandler) StartTests(c *gin.Context) {
+	codexGatewayBorrowPrivateHeaders(c)
+	var request service.CodexGatewayBorrowTestRequest
+	if err := c.ShouldBindJSON(&request); err != nil {
+		response.BadRequest(c, err.Error())
+		return
+	}
+	if _, err := uuid.Parse(strings.TrimSpace(request.ClientTaskID)); err != nil {
+		response.BadRequest(c, "client_task_id must be a UUID")
+		return
+	}
+	if len(request.Targets) == 0 {
+		response.BadRequest(c, "targets is required")
+		return
+	}
+	// Only this server-side entry point can select account execution. A JSON
+	// field cannot change the legacy borrowing handler's existing contract.
+	request.Standalone = true
+	var createdBy int64
+	if subject, ok := middleware.GetAuthSubjectFromContext(c); ok {
+		createdBy = subject.UserID
+	}
+	task, replayed, err := h.runner.Create(c.Request.Context(), createdBy, request)
+	if err != nil {
+		codexGatewayBorrowTestError(c, err)
+		return
+	}
+	middleware.SetAuditExtra(c, map[string]any{"result": "pelican_manual_test", "task_id": task.ID, "replayed": replayed, "targets": task.Total})
+	ctx, cancel := context.WithCancel(c.Request.Context())
+	defer cancel()
+	c.Header("Content-Type", "text/event-stream")
+	c.Header("Connection", "keep-alive")
+	c.Header("X-Accel-Buffering", "no")
+	c.Status(http.StatusOK)
+	var writeMu sync.Mutex
+	emit := func(event service.CodexGatewayBorrowTestEvent) {
+		if ctx.Err() != nil {
+			return
+		}
+		if event.Task != nil {
+			event.Task = h.preview.DecorateTask(event.Task)
+		}
+		if event.Result != nil {
+			event.Result = h.preview.DecorateResult(event.Result)
+		}
+		payload, marshalErr := json.Marshal(event)
+		if marshalErr != nil {
+			cancel()
+			return
+		}
+		writeMu.Lock()
+		defer writeMu.Unlock()
+		if ctx.Err() != nil {
+			return
+		}
+		if _, writeErr := fmt.Fprintf(c.Writer, "data: %s\n\n", payload); writeErr != nil {
+			cancel()
+			return
+		}
+		c.Writer.Flush()
+	}
+	emit(service.CodexGatewayBorrowTestEvent{Type: "task_start", Task: task})
+	if replayed {
+		emit(service.CodexGatewayBorrowTestEvent{Type: "task_complete", Task: task})
+		return
+	}
+	keepAliveDone, keepAliveStopped := make(chan struct{}), make(chan struct{})
+	go func() {
+		defer close(keepAliveStopped)
+		ticker := time.NewTicker(15 * time.Second)
+		defer ticker.Stop()
+		for {
+			select {
+			case <-keepAliveDone:
+				return
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+				writeMu.Lock()
+				if ctx.Err() == nil {
+					if _, writeErr := fmt.Fprint(c.Writer, ": keep-alive\n\n"); writeErr != nil {
+						cancel()
+					} else {
+						c.Writer.Flush()
+					}
+				}
+				writeMu.Unlock()
+			}
+		}
+	}()
+	runErr := h.runner.Run(ctx, task, emit)
+	close(keepAliveDone)
+	<-keepAliveStopped
+	if runErr != nil && ctx.Err() == nil {
+		task.Status, task.Error = "incomplete", runErr.Error()
+		emit(service.CodexGatewayBorrowTestEvent{Type: "task_complete", Task: task})
+	}
+}
+
+func (h *PelicanTestHandler) ListTests(c *gin.Context) {
+	codexGatewayBorrowPrivateHeaders(c)
+	page, size := response.ParsePagination(c)
+	if rawSize := c.Query("size"); rawSize != "" {
+		if parsed, err := strconv.Atoi(rawSize); err == nil && parsed > 0 {
+			size = parsed
+		}
+	}
+	list, err := h.runner.List(c.Request.Context(), page, size)
+	if err != nil {
+		codexGatewayBorrowTestError(c, err)
+		return
+	}
+	response.Success(c, list)
+}
+
+func (h *PelicanTestHandler) GetTest(c *gin.Context) {
+	codexGatewayBorrowPrivateHeaders(c)
+	id := strings.TrimSpace(c.Param("id"))
+	if id == "last" {
+		list, err := h.runner.List(c.Request.Context(), 1, 1)
+		if err != nil {
+			codexGatewayBorrowTestError(c, err)
+			return
+		}
+		if len(list.Items) == 0 {
+			response.NotFound(c, "no unexpired pelican test")
+			return
+		}
+		id = list.Items[0].ID
+	}
+	parsed, err := uuid.Parse(id)
+	if err != nil {
+		response.BadRequest(c, "test ID must be a UUID or last")
+		return
+	}
+	task, err := h.runner.Get(c.Request.Context(), parsed.String())
+	if err != nil {
+		codexGatewayBorrowTestError(c, err)
+		return
+	}
+	response.Success(c, h.preview.DecorateTask(task))
+}
+
+func (h *PelicanTestHandler) ServePreview(c *gin.Context) {
+	codexGatewayBorrowPrivateHeaders(c)
+	c.Header("Content-Security-Policy", service.CodexGatewayBorrowPreviewCSP)
+	c.Header("X-Frame-Options", "SAMEORIGIN")
+	c.Header("Cross-Origin-Resource-Policy", "cross-origin")
+	html, err := h.preview.Resolve(c.Request.Context(), strings.TrimSpace(c.Param("cap")))
+	if err != nil {
+		c.Status(http.StatusGone)
+		return
+	}
+	c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
+}