Skip to content

Commit a587cb2

Browse files
hmottestadhmottestad
authored
Add HotSpot JIT forensics skill for codex (#5661)
GH-0000 Add HotSpot JIT forensics skill
1 parent 31a0d4f commit a587cb2

4 files changed

Lines changed: 366 additions & 0 deletions

File tree

‎.codex/skills/hotspot-jit-forensics/SKILL.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
---
2+
name: hotspot-jit-forensics
3+
description: Diagnose Java performance issues by inspecting HotSpot bytecode, tiered compilation state, inlining decisions, C2 assembly, compilation limits, and object layout. Produces reproducible artifacts (jit.xml, directives, JFR, disassembly) and links them to actionable code changes.
4+
---
5+
6+
# HotSpot C2 / JIT Forensics & Optimization
7+
8+
A practical playbook for diagnosing Java performance issues by looking at what HotSpot *actually* does.
9+
10+
Use this skill when you suspect:
11+
- Lost inlining, megamorphic dispatch, or deoptimization.
12+
- A hot method is slow because it never reaches C2.
13+
- High CPU in tiny methods (often a no-inline or missed intrinsic).
14+
- High allocation rate or cache-unfriendly object layout.
15+
- Compilation failures/bailouts (node limits, method too big, deep inlining).
16+
- Code cache pressure, safepoint stalls, or lock contention.
17+
18+
---
19+
20+
## Quick start (repeatable artifacts)
21+
22+
1) **Collect JVM facts**
23+
```bash
24+
.codex/skills/hotspot-jit-forensics/scripts/jit-facts.sh --out jit-facts.txt
25+
```
26+
For a running process:
27+
```bash
28+
.codex/skills/hotspot-jit-forensics/scripts/jit-facts.sh --pid <pid> --out jit-facts.txt
29+
```
30+
31+
2) **Generate a C2 directives file**
32+
```bash
33+
.codex/skills/hotspot-jit-forensics/scripts/jit-directives.sh \
34+
--method "com/foo/MyClass.myMethod()" \
35+
--out c2-directives.json5
36+
```
37+
38+
3) **Run the target command with compiler logging**
39+
```bash
40+
.codex/skills/hotspot-jit-forensics/scripts/jit-run-log.sh \
41+
--directives c2-directives.json5 \
42+
--logfile jit.xml \
43+
-- java -XX:+UnlockDiagnosticVMOptions -jar app.jar
44+
```
45+
46+
Artifacts produced:
47+
- `jit-facts.txt` (version, flags, OS/arch, optional jcmd output)
48+
- `c2-directives.json5` (method-scoped compiler diagnostics)
49+
- `jit.xml` (HotSpot compilation log)
50+
- Console output with inlining and assembly (if `hsdis` is available)
51+
52+
---
53+
54+
## Core workflow
55+
56+
### 1) Profile first (do not guess)
57+
Use **JFR** or **async-profiler** to identify the actual hot method(s).
58+
59+
### 2) Confirm compilation tier
60+
Determine if the method is interpreted, C1, or C2.
61+
62+
Runtime:
63+
```bash
64+
jcmd <pid> Compiler.codelist | head
65+
jcmd <pid> Compiler.queue
66+
jcmd <pid> Compiler.codecache
67+
```
68+
Start-up (noisy):
69+
```bash
70+
java -XX:+PrintCompilation -jar app.jar
71+
```
72+
73+
### 3) Capture inlining + compilation decisions for ONE method
74+
Prefer **Compiler Directives** with a focused match.
75+
76+
### 4) If assembly doesn’t print, fix hsdis
77+
HotSpot only prints assembly with the **hsdis** plugin installed.
78+
79+
### 5) Read “why not inlined?” and “why not compiled?”
80+
Check `jit.xml` (or JITWatch) for inline failures and compilation bailouts.
81+
82+
### 6) Inspect object layout
83+
Use **JOL** (CLI or code) and `jcmd` class histograms.
84+
85+
### 7) Cross-check system effects
86+
GC, safepoints, locks, and code cache can dominate CPU.
87+
88+
---
89+
90+
## Key flags (diagnosis only)
91+
92+
- `-XX:+UnlockDiagnosticVMOptions`
93+
- `-XX:+LogCompilation -XX:LogFile=jit.xml`
94+
- `-XX:+CompilerDirectivesPrint -XX:CompilerDirectivesFile=c2-directives.json5`
95+
- `-XX:+PrintCompilation`
96+
- `-Xlog:safepoint=info` (JDK 9+)
97+
- `-Xlog:gc*` (JDK 9+)
98+
99+
---
100+
101+
## Script reference
102+
103+
### `jit-facts.sh`
104+
Collect JVM + OS facts and optional `jcmd` diagnostics into one file.
105+
106+
```
107+
Usage: jit-facts.sh [--pid <pid>] [--out <file>]
108+
```
109+
110+
### `jit-directives.sh`
111+
Generate a compiler directives file for a target method.
112+
113+
```
114+
Usage: jit-directives.sh --method <pattern> [--out <file>]
115+
```
116+
117+
### `jit-run-log.sh`
118+
Run a `java` command with directive-based C2 logging.
119+
120+
```
121+
Usage: jit-run-log.sh --directives <file> --logfile <file> -- java <args...>
122+
```
123+
124+
---
125+
126+
## Deliverables checklist
127+
128+
- Repro command line (exact).
129+
- JVM version + flags (`jit-facts.txt`).
130+
- Profile (`recording.jfr` or `cpu.svg`/`alloc.svg`).
131+
- `jit.xml` (LogCompilation).
132+
- directives file (`c2-directives.json5`).
133+
- Assembly snippet for target method(s).
134+
- Object layout output (JOL internals + footprint).
135+
- Summary: “hypothesis → evidence → change → result”.
136+
137+
---
138+
139+
## Reference URLs
140+
141+
- https://docs.oracle.com/en/java/javase/21/docs/specs/man/java.html
142+
- https://docs.oracle.com/en/java/javase/12/vm/compiler-control1.html
143+
- https://docs.oracle.com/en/java/javase/12/vm/writing-directives.html
144+
- https://docs.oracle.com/en/java/javase/11/troubleshoot/diagnostic-tools.html
145+
- https://openjdk.org/projects/code-tools/jol/
146+
- https://openjdk.org/jeps/520

‎.codex/skills/hotspot-jit-forensics/scripts/jit-directives.sh‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
usage() {
5+
cat <<'USAGE'
6+
Usage: jit-directives.sh --method <pattern> [--out <file>]
7+
8+
Example:
9+
jit-directives.sh --method "com/foo/MyClass.myMethod()" --out c2-directives.json5
10+
USAGE
11+
}
12+
13+
method=""
14+
out_file="c2-directives.json5"
15+
16+
while [[ $# -gt 0 ]]; do
17+
case "$1" in
18+
--method)
19+
method="$2"
20+
shift 2
21+
;;
22+
--out)
23+
out_file="$2"
24+
shift 2
25+
;;
26+
-h|--help)
27+
usage
28+
exit 0
29+
;;
30+
*)
31+
echo "Unknown argument: $1" >&2
32+
usage
33+
exit 1
34+
;;
35+
esac
36+
done
37+
38+
if [[ -z "$method" ]]; then
39+
echo "--method is required" >&2
40+
usage
41+
exit 1
42+
fi
43+
44+
python3 - "$out_file" "$method" <<'PY'
45+
import sys
46+
47+
out_file = sys.argv[1]
48+
method = sys.argv[2]
49+
50+
content = f"""[
51+
{{
52+
// Match the method(s) you care about. Wildcards are allowed.
53+
match: [\"{method}\"],
54+
55+
c2: {{
56+
PrintInlining: true,
57+
PrintAssembly: true,
58+
PrintIntrinsics: true,
59+
IGVPrintLevel: 3,
60+
// MaxNodeLimit: 80000,
61+
}},
62+
63+
c1: {{
64+
PrintInlining: false,
65+
PrintAssembly: false,
66+
}},
67+
}}
68+
]
69+
"""
70+
71+
with open(out_file, "w", encoding="utf-8") as handle:
72+
handle.write(content)
73+
PY
74+
75+
echo "Wrote $out_file"

‎.codex/skills/hotspot-jit-forensics/scripts/jit-facts.sh‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
usage() {
5+
cat <<'USAGE'
6+
Usage: jit-facts.sh [--pid <pid>] [--out <file>]
7+
8+
Collect JVM and OS facts plus optional jcmd diagnostics for a running process.
9+
USAGE
10+
}
11+
12+
pid=""
13+
out_file="jit-facts.txt"
14+
15+
while [[ $# -gt 0 ]]; do
16+
case "$1" in
17+
--pid)
18+
pid="$2"
19+
shift 2
20+
;;
21+
--out)
22+
out_file="$2"
23+
shift 2
24+
;;
25+
-h|--help)
26+
usage
27+
exit 0
28+
;;
29+
*)
30+
echo "Unknown argument: $1" >&2
31+
usage
32+
exit 1
33+
;;
34+
esac
35+
done
36+
37+
{
38+
echo "=== Timestamp ==="
39+
date -u
40+
echo
41+
42+
echo "=== OS/CPU ==="
43+
uname -a || true
44+
if command -v lscpu >/dev/null 2>&1; then
45+
lscpu
46+
fi
47+
echo
48+
49+
echo "=== java -version ==="
50+
java -version 2>&1
51+
echo
52+
53+
echo "=== PrintFlagsFinal (head) ==="
54+
java -XX:+PrintFlagsFinal -version 2>&1 | head -n 30
55+
echo
56+
57+
if [[ -n "$pid" ]]; then
58+
if ! command -v jcmd >/dev/null 2>&1; then
59+
echo "jcmd not found; skipping VM diagnostics for pid=$pid" >&2
60+
else
61+
echo "=== jcmd VM.info ==="
62+
jcmd "$pid" VM.info
63+
echo
64+
65+
echo "=== jcmd VM.flags ==="
66+
jcmd "$pid" VM.flags
67+
echo
68+
69+
echo "=== jcmd VM.command_line ==="
70+
jcmd "$pid" VM.command_line
71+
echo
72+
73+
echo "=== jcmd Compiler.codecache ==="
74+
jcmd "$pid" Compiler.codecache
75+
echo
76+
fi
77+
fi
78+
} > "$out_file"
79+
80+
echo "Wrote $out_file"

‎.codex/skills/hotspot-jit-forensics/scripts/jit-run-log.sh‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
usage() {
5+
cat <<'USAGE'
6+
Usage: jit-run-log.sh --directives <file> --logfile <file> -- java <args...>
7+
8+
Example:
9+
jit-run-log.sh --directives c2-directives.json5 --logfile jit.xml -- \
10+
java -XX:+UnlockDiagnosticVMOptions -jar app.jar
11+
USAGE
12+
}
13+
14+
directives=""
15+
logfile="jit.xml"
16+
17+
while [[ $# -gt 0 ]]; do
18+
case "$1" in
19+
--directives)
20+
directives="$2"
21+
shift 2
22+
;;
23+
--logfile)
24+
logfile="$2"
25+
shift 2
26+
;;
27+
--)
28+
shift
29+
break
30+
;;
31+
-h|--help)
32+
usage
33+
exit 0
34+
;;
35+
*)
36+
echo "Unknown argument: $1" >&2
37+
usage
38+
exit 1
39+
;;
40+
esac
41+
done
42+
43+
if [[ -z "$directives" ]]; then
44+
echo "--directives is required" >&2
45+
usage
46+
exit 1
47+
fi
48+
49+
if [[ $# -eq 0 ]]; then
50+
echo "Missing java command after --" >&2
51+
usage
52+
exit 1
53+
fi
54+
55+
if [[ "$1" != "java" ]]; then
56+
echo "First command after -- must be 'java'" >&2
57+
usage
58+
exit 1
59+
fi
60+
61+
exec "$@" \
62+
-XX:+CompilerDirectivesPrint \
63+
-XX:CompilerDirectivesFile="$directives" \
64+
-XX:+LogCompilation \
65+
-XX:LogFile="$logfile"

0 commit comments

Comments
 (0)