Skip to content

Dump forked-process threads on log-injection smoke-test timeout#11930

Draft
bm1549 wants to merge 1 commit into
masterfrom
brian.marks/loginjection-openj9-threaddump
Draft

Dump forked-process threads on log-injection smoke-test timeout#11930
bm1549 wants to merge 1 commit into
masterfrom
brian.marks/loginjection-openj9-threaddump

Conversation

@bm1549

@bm1549 bm1549 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

On the "check raw file injection" smoke-test 30s timeout, jstack the forked process and attach the dump to the failure — so the OpenJ9/Semeru hang can be diagnosed instead of a bare "Condition not satisfied". Also distinguishes an early process exit from a genuine hang. Reinstates (a much smaller version of) the diagnostic from #11400, removed in #11548.

Motivation

The test times out with traceCount=0 and the forked process never exiting — a startup hang, only on IBM Semeru / OpenJ9. #11509 fixed the earlier DDTraceId clinit deadlock; this resurfaced after Semeru jobs began running on master/MQ (#11788, #11832). It couldn't be reproduced by brute force (~12k startups), so capture a thread dump when it next hangs on master/MQ.

Additional Notes

Runs only on the failure path. Follow-up (separate PR): @Flaky's isIbm8() reads java.vm.vendor="Eclipse OpenJ9", so it never matches Semeru — should be isJ9().

Jira ticket: N/A

🤖 Generated with Claude Code

@bm1549 bm1549 added type: bug fix Bug fix comp: testing Testing tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM labels Jul 13, 2026
bm1549 added a commit that referenced this pull request Jul 13, 2026
DO NOT MERGE — temporary. Adds test_loginjection_openj9_repro (amd64
semeru8/11/17) and _arm64 (semeru11/17) jobs that build once then loop the
log-injection smoke test (whose only feature method is "check raw file
injection") 20x per JVM to force the traceCount=0 startup hang and capture the
reinstated forked-process thread dump into the loop-output/ artifact. Delete
before marking PR #11930 ready.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@datadog-official

datadog-official Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 57.03% (-0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5f34d40 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.76 s 14.72 s [-0.5%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 13.53 s 13.73 s [-2.4%; -0.5%] (maybe better)
startup:petclinic:appsec:Agent 16.79 s 16.63 s [-0.0%; +1.9%] (no difference)
startup:petclinic:iast:Agent 16.84 s 16.93 s [-1.2%; +0.2%] (no difference)
startup:petclinic:profiling:Agent 16.75 s 16.85 s [-1.9%; +0.7%] (no difference)
startup:petclinic:sca:Agent 16.95 s 16.80 s [+0.0%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 16.06 s 15.74 s [-2.4%; +6.5%] (no difference)

Commit: 5f34d403 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@bm1549 bm1549 force-pushed the brian.marks/loginjection-openj9-threaddump branch from 0c59c6a to ad11c19 Compare July 13, 2026 18:48
On the "check raw file injection" 30s timeout, jstack the forked process so the
OpenJ9/Semeru startup hang (traceCount=0, process never exits) can be diagnosed
instead of a bare "Condition not satisfied". Reinstates the diagnostic from
#11400 (removed in #11548), now much smaller, and distinguishes an early process
exit from a genuine hang.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bm1549 bm1549 force-pushed the brian.marks/loginjection-openj9-threaddump branch from ad11c19 to 5f34d40 Compare July 13, 2026 19:26
@bm1549 bm1549 changed the title Reinstate log-injection forked-process thread dump on timeout Dump forked-process threads on log-injection smoke-test timeout Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: testing Testing tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant