Move Feature Flagging evaluation into the provider runtime - #12097
Move Feature Flagging evaluation into the provider runtime#12097leoromanovsky wants to merge 12 commits into
Conversation
leoromanovsky
added
type: refactoring
comp: remote config
|
🎯 Code Coverage (details) 🔗 Commit SHA: e646951 | Docs | Datadog PR Page | Give us feedback! |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8bfa6f072a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| valueKind(target), | ||
| key, | ||
| unwrapDefaultValue(defaultValue), | ||
| toCoreContext(context)); |
There was a problem hiding this comment.
Every evaluation with a non-null context calls toCoreContext before the core checks whether configuration or the requested flag exists; that helper walks every attribute and recursively copies structured values, and the core EvaluationContext then copies the attribute map again. This adds escaping allocations to every request-hot evaluation, including missing, disabled, and static flags that do not inspect attributes, so pass a lazy context view or defer conversion until targeting rules require it and verify with an allocation profile.
AGENTS.md reference: AGENTS.md:L77-L77
Useful? React with 👍 / 👎.
| final CompletableFuture<java.net.http.HttpResponse<byte[]>> future = | ||
| client.sendAsync(request.build(), BodyHandlers.ofByteArray()); |
There was a problem hiding this comment.
When a CDN, proxy, or custom endpoint returns Content-Encoding: gzip, Java's HttpClient passes the compressed bytes through BodyHandlers.ofByteArray() rather than decompressing them, so the UFC parser rejects every such response as malformed and the provider never becomes ready or retains stale configuration. The previous OkHttp transport transparently decoded gzip; restore equivalent decoding based on the response encoding.
Useful? React with 👍 / 👎.
| @SuppressForbidden | ||
| private static String setting(final String property, final String environment) { | ||
| final String propertyValue = System.getProperty(property); | ||
| return propertyValue != null ? propertyValue : System.getenv(environment); |
There was a problem hiding this comment.
When an attached deployment sets Feature Flagging through a supported non-system/environment source such as Fleet or local stable configuration or DD_TRACE_CONFIG, this helper cannot see the setting and defaults the application provider to CDN while the agent may resolve and start remote_config. The agent then publishes raw RC bytes that the provider never subscribes to, and provider initialization instead polls CDN with potentially missing credentials; obtain the agent-resolved source/options through the bridge when available while retaining the direct property/environment fallback for provider-only deployments.
AGENTS.md reference: AGENTS.md:L37-L44
Useful? React with 👍 / 👎.
| snapshot, | ||
| valueKind(target), | ||
| key, | ||
| unwrapDefaultValue(defaultValue), |
There was a problem hiding this comment.
When an object flag is disabled or no allocation matches, the core returns the default passed here, but unwrapping and rebuilding an OpenFeature Value changes representational types—for example, an Instant becomes a string and an integral Double can become an Integer. Unlike error paths, successful DISABLED and DEFAULT results therefore do not return the caller's original default value; retain the original Value for fallback results and unwrap only values that the core must actually inspect.
Useful? React with 👍 / 👎.
1 participant
Motivation
Before this change, Java Feature Flags required the application to start with the Datadog Java agent:
The provider delegated UFC parsing and evaluation to classes owned by the Java agent. Without
-javaagent, the provider could not load configuration or evaluate flags.Despite its name,
feature-flagging-libwas never a reusable Feature Flagging library. Since its introduction, it owned agent-side Remote Configuration and telemetry delivery. It depended on agent communication, bootstrap, configuration, and tracer classes. It was not published indd-openfeatureand could not run as a provider-only dependency.Reusing that module would retain the agent coupling. This change creates a dependency-light provider core. It renames the historical module to describe its agent-runtime role.
Managed serverless platforms make the startup requirement costly:
JAVA_TOOL_OPTIONS=-javaagent:/opt/java/lib/dd-java-agent.jar.JAVA_TOOL_OPTIONS. First-generation functions require two agents.languageWorkers__java__argumentsorJAVA_OPTS, depending on the hosting plan.These steps couple flag evaluation to tracing packages, platform-specific runtime configuration, and agent compatibility. The default CDN path must work with
dd-openfeaturealone.The project must also preserve Remote Configuration. Existing customers need the agent-backed source. Released providers must continue to work with a new agent.
Note to reviewers
Main net-new code here is the "bridge" between the java agent and openfeature client. Please pay special attention to the architecture and implementation.
Changes
The default OpenFeature startup is now provider-only:
The application supplies
DD_API_KEY,DD_ENV, andDD_SITE. It does not set a configuration-source selector or attach-javaagent. The provider starts and owns the internal CDN poller.ffe-dogfooding #102 uses this startup path. The script runs
java -jar app.jarunlessremote_configis explicit.HTTP cancellation now handles both direct and
ExecutionException-wrappedCancellationExceptionresults. Both paths returnInterruptedIOException.The published coordinate remains
com.datadoghq:dd-openfeature. The provider JAR embedsfeature-flagging-coreandfeature-flagging-http.flowchart TD APP["Application and OpenFeature SDK"] --> API["feature-flagging-api<br/>published as dd-openfeature"] subgraph PROVIDER["Provider-owned modules embedded in dd-openfeature"] API --> CORE["feature-flagging-core<br/>UFC model, parser, evaluator,<br/>snapshot, last-known-good state"] API --> HTTP["feature-flagging-http<br/>Java 11 HttpClient, ETag,<br/>retry, scheduling, lifecycle"] HTTP --> CORE end API -. "reflective JDK-only calls" .-> BRIDGE["feature-flagging-bootstrap<br/>raw compatibility bridge"] subgraph AGENT["Java agent modules"] ENTRY["feature-flagging-agent<br/>agent lifecycle"] --> RUNTIME["feature-flagging-agent-runtime<br/>Remote Configuration,<br/>EVP transport, trace adapter"] RUNTIME --> TELEMETRY["feature-flagging-telemetry<br/>exposure deduplication,<br/>span-tag state and encoding"] end RUNTIME <--> BRIDGEBoth delivery sources feed the same provider-owned evaluator:
flowchart TD CDN["Default: Datadog CDN"] --> HTTP["HTTP source"] RC["Explicit: Remote Configuration"] --> AGENT["Java agent"] AGENT --> BRIDGE["Raw byte[] bridge"] HTTP --> CORE["Core snapshot and evaluator"] BRIDGE --> CORE CORE --> RESULT["OpenFeature result"]feature-flagging-libis nowfeature-flagging-agent-runtime.feature-flagging-telemetrymodule owns exposure deduplication and span-tag state and encoding.feature-flagging-agent-runtime.feature-flagging-telemetryis not embedded indd-openfeatureyet. The provider does not emit telemetry through it in this change.Decisions
feature-flagging-core. Future CDN, Remote Configuration, offline, tracer API, and Lambda entry points can reuse one implementation.feature-flagging-http. Offline and Remote Configuration sources do not inherit HTTP dependencies or polling behavior.core. UFC evaluation remains independent from telemetry state and transport.feature-flagging-telemetry. The module depends only on the JDK.feature-flagging-libtofeature-flagging-agent-runtime. The old module was never a reusable library.dd-openfeature.This organization is the last required configuration and evaluator ownership refactor. It is not the final telemetry architecture.
The bootstrap bridge abstracts provider-to-agent communication across classloaders. It does not abstract local relay or direct intake transport. It also cannot create or enrich a trace when no tracer exists.
Next steps
The provider must own the agentless telemetry runtime because it is the only component present in provider-only deployments. The agent runtime remains an optional adapter.
Before agentless telemetry delivery, the follow-up should:
feature-flagging-telemetryindd-openfeature.flowchart TB EVAL["Provider evaluation"] --> EVENT["Immutable evaluation event"] EVENT --> PIPE["Provider-owned telemetry runtime"] PIPE --> EVP["Exposure and aggregate EVP"] PIPE --> METRIC["Evaluation OTLP metric"] PIPE --> TRACE["Trace enrichment"] EVP --> EVP_ROUTE{"Agent accepts EVP?"} EVP_ROUTE -- Yes --> EVP_AGENT["Raw bridge to agent writer"] EVP_ROUTE -- No --> EVP_DIRECT["Local relay or direct EVP"] METRIC --> METRIC_ROUTE{"Local OTLP configured?"} METRIC_ROUTE -- Yes --> OTLP_LOCAL["OTLP HTTP on 4318"] METRIC_ROUTE -- No --> OTLP_DIRECT["Direct OTLP metrics intake"] TRACE --> TRACE_ROUTE{"Trace producer?"} TRACE_ROUTE -- "Datadog Java agent" --> TRACE_AGENT["Raw bridge to local-root span"] TRACE_ROUTE -- "OpenTelemetry SDK" --> TRACE_OTEL["Future OpenTelemetry span adapter"] TRACE_ROUTE -- None --> TRACE_NONE["No trace enrichment"]The required follow-up depends on the deployment:
serverless-initserverless-initor the Java agent4318only when explicitly enabledDirect OTLP trace intake makes an agentless trace path possible, but
dd-openfeaturemust not become a tracer. A future OpenTelemetry adapter can enrich an active OpenTelemetry span. The application must still install a trace SDK and exporter.PR risk
The customer-facing goal is narrow. CDN evaluation can run without the Java agent, while Remote Configuration remains available. The implementation is substantial because the previous design made the agent the configuration and lifecycle owner.
Net-new or changed behavior:
dd-openfeaturecan fetch, parse, store, and evaluate CDN configuration without-javaagent.Moved or ported behavior:
DDEvaluatorintofeature-flagging-core. The expected OpenFeature results remain unchanged, but the implementation and model changed.HttpClientimplementation.304, retry, timeout, overlap prevention, scheduling, and shutdown behavior were ported rather than added for the first time.feature-flagging-librename and telemetry extraction are primarily structural. They do not add telemetry delivery.The main residual risks are UFC semantic drift, provider lifecycle leaks, source-selection compatibility, HTTP behavior differences, and cross-classloader compatibility. The dogfooding proof, system tests, child-JVM tests, last-known-good tests, and compatibility matrix reduce these risks. They do not make the change risk-free.
This assessment comes from an LLM review of the static diff and recorded validation. It is not a formal safety proof or a substitute for human review.
Validation
8bfa6f072a97ac8a7d4977c42e04fe8d63652e25.1.65.0-SNAPSHOTran beside published Node.jsdd-trace5.118.0.-javaagentand reportedjavaAgentAttached=false.PROVIDER_READYin 1,094 ms. Node.js reachedPROVIDER_READYin 706 ms.truefor the same configured flag.providerShutdown=true.system-tests #7300 used
java@1.65.0-SNAPSHOT+8bfa6f072a.FFL-2446,FFL-1729, andFFL-2184.javaAgentAttached=false.on-value.The compatibility matrix passed:
1.64.2agent + CDN.1.64.2provider + new agent + Remote Configuration through the legacy bridge.1.64.2agent + Remote Configuration returned the expected unsupported-bridge error.