Add smoke test JUnit framework - #12111
Conversation
PerfectSlayer
added
type: feature
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
4a50f04 to
11530c9
Compare
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit 4abdba0 · Any feedback? Reach out in #deveng-pr-agent |
11530c9 to
4abdba0
Compare
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4abdba0567
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| new PollingConditions(timeoutSeconds) | ||
| .eventually(() -> SmokeTraceAssertions.assertTraces(getTraces(), options, matchers)); |
There was a problem hiding this comment.
For the default count-exact assertion, PollingConditions returns immediately when one snapshot matches. Because both backends receive traces asynchronously, the expected traces can match first and an additional trace from the same test action can arrive just afterward, causing the test to pass even though the final collection violates the exact-count contract; this also makes IGNORE_ADDITIONAL_TRACES ineffective as the distinction between strict and permissive behavior in that scenario. The strict path needs a stabilization/final-count check rather than accepting the first transient match.
Useful? React with 👍 / 👎.
b169c19 to
9c11764
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c1176468d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @Override | ||
| public void beforeEach(ExtensionContext context) { | ||
| if (clearsBetweenTests()) { | ||
| clear(); | ||
| } |
There was a problem hiding this comment.
When a SmokeCliApp uses a backend that is also registered as a shared extension, the CLI process can finish and emit all of its traces during beforeAll, after which this beforeEach callback clears them before the test body runs. The CLI app's deliberate lack of an owned-backend reset cannot prevent this; TestAgentBackend only works if callers discover and enable retainAcrossTests(), while MockAgentBackend offers no retention mode, so shared CLI trace assertions are otherwise unusable.
Useful? React with 👍 / 👎.
| if (!this.process.waitFor(5, SECONDS)) { | ||
| this.process.destroyForcibly(); | ||
| this.process.waitFor(10, SECONDS); |
There was a problem hiding this comment.
If a child remains alive for ten seconds after destroyForcibly()—for example because process termination is delayed on the host—this ignores the false result from waitFor and lets teardown report success. The orphan can retain its server port and files and interfere with later test classes or Gradle tasks; check the result and surface a cleanup failure, as the existing ProcessManager teardown does.
Useful? React with 👍 / 👎.
9c11764 to
b6b0809
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6b0809cde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @Override | ||
| public final void beforeEach(ExtensionContext context) { | ||
| onBeforeEach(); | ||
| this.outputThreads.clearMessages(); |
There was a problem hiding this comment.
When a fast SmokeCliApp emits its output during beforeAll, this unconditional clear can erase those messages before the first test method calls awaitLogLine; whether the output survives depends on the race between the child/output thread and beforeEach. CLI suites that assert diagnostics produced during their one-shot execution therefore become flaky or always time out, so startup messages need the same retention treatment as CLI startup traces.
Useful? React with 👍 / 👎.
| public void assertTelemetryReceived() { | ||
| this.backend.telemetry().waitForCount(1, this.startupTimeoutSeconds); | ||
| } |
There was a problem hiding this comment.
When multiple instrumented apps share one registered backend, every app's afterEach calls this against the same aggregated telemetry session, so a message or repeated heartbeat from app A makes app B's automatic check pass even if B never initialized telemetry. This creates a false negative in the supported shared multi-app composition; the check needs an app-specific identifier such as its runtime ID, or must otherwise avoid treating another app's telemetry as evidence for this app.
Useful? React with 👍 / 👎.
| private static final String INFO_BODY = | ||
| "{\"version\":\"7.77.0\",\"endpoints\":[\"/v0.4/traces\",\"/v0.5/traces\",\"/v1.0/traces\"]}"; |
There was a problem hiding this comment.
For sampled traces with SAMPLER_DROP or USER_DROP, this /info response omits client_drop_p0s, so DDAgentFeaturesDiscovery leaves its dropping policy inactive and the tracer serializes those traces to the mock backend. A real/test agent advertises this capability, and the existing smoke-test mock does too, so exact trace assertions can pass against one backend and receive additional dropped traces against the other; include the capability in the advertised agent information.
Useful? React with 👍 / 👎.
1 participant
What Does This Do
DRAFT. Being refined. Opened for bot review.
This PR adds a JUnit 5 / Java smoke test framework that allows to write smoke tests using Java/JUnit 5 instead of Groovy/Spock.
The framework has four parts:
App runners:
AbstractSmokeAppwithSmokeServerApp(long-running/HTTP) andSmokeCliApp(batch) concrete types, registered as a@RegisterExtension.Owns the launched app's lifecycle and wires in the agent jar + test agent session token.
Backends: a common
TraceBackendfacade with two interchangeable implementations:MockAgentBackend(in-processJavaTestHttpServer, no Docker) andTestAgentBackend(a Testcontainers-managed, or external/CI, dd-apm-test-agent, scoped per test via
X-Datadog-Test-Session-Token).A test written against the facade runs unchanged on either backend but the goal is moving toward
TestAgentBackendby default.Capture surfaces:
Traces(msgpack/JSON decoded through the shared decoder),Telemetry(with message-batch flattening), andRemoteConfig(push a config thetracer polls for, and read back its
/v0.7/configpolls, with products/capabilitiesdecoders).
Traces, telemetry, and remote config are all isolated per test.
Motivation
WIP (extracting from one-pager, JIRA cards, and the other PRs)
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: APMLP-1247