Background#
I was setting up a fine-tuning pipeline for Foundation-Sec-8B on a Windows 11 machine with an RX 9070 XT using ROCm 7.14. The stack: PyTorch 2.12.0+rocm7.14.0, PEFT, standard HuggingFace Transformers. GPU correctly detected, VRAM visible, torch.cuda.is_available() returned True.
The first sign was the crash output in the terminal itself — the process printed a memory address alongside 0xC0000005 before dying. No Python exception was raised, no traceback. Just a raw AV and an exit.
This is the analysis of what caused it, what the wrong fix looked like, and the correct patch.
Environment#
|
|
| GPU |
AMD Radeon RX 9070 XT (gfx1201, RDNA 4) |
| OS |
Windows 11 (build 26100) |
| ROCm |
7.14.0 (torch.version.hip = 7.14.60850) |
| PyTorch |
2.12.0+rocm7.14.0 |
| Driver |
Adrenalin 32.0.31041.1004 |
| Issue |
ROCm/TheRock #7732 |
Reproducing the Crash#
Any operation dispatching a HIP kernel crashes immediately:
1
2
|
import torch
t = torch.tensor([1.0, 2.0, 3.0]).cuda() # process dies here
|
No exception is raised. The Python process exits with 0xC0000005.
WinDbg Analysis#
Standard crash dump analysis shows the process is already dead at exit. To catch the access violation at its origin, attach WinDbg with child process tracking and break on first-chance AV:
windbg -o -G rocm_env\Scripts\python.exe finetune.py
At the initial break:
sxe av
g
The process breaks mid-execution with:
(1820.1f1c): Access violation - code c0000005 (first chance)
amdhip64_7!hipProfilerRegisterChunkCallbackExt+0x5c257:
00007ffd`4bf08bb7 mov rax, qword ptr [rcx+8]
r gives:
rcx = 7b2c450fc892ac84 ← not null — garbage pointer
rip = 00007ffd4bf08bb7
~* kn on thread 9:
# 00 amdhip64_7!hipProfilerRegisterChunkCallbackExt+0x5c257
# 01 amdhip64_7!hipRegisterTracerCallback+0x14a0e8
# 02 amdhip64_7!hipRegisterTracerCallback+0x135c16
# 03 amdhip64_7!hipRegisterTracerCallback+0x120ece
# 04 amdhip64_7!hipRegisterTracerCallback+0x10c296
# 05 amdhip64_7!hipRegisterTracerCallback+0xf93ce
# 06 amdhip64_7!hipLaunchKernel+0x88
# 07 torch_hip!at::native::...
The call chain: hipLaunchKernel+0x85 dispatches into the ROCprofiler callback registry. The registry walks a linked list. One node contains rcx = 7b2c450fc892ac84 — a garbage, non-null pointer. Dereferencing [rcx+8] produces the AV.
Root Cause#
hipLaunchKernel calls into the ROCprofiler callback notification chain on every kernel dispatch via an indirect call at +0x85:
1
2
3
4
|
; hipLaunchKernel+0x82:
48 8b ce mov rcx, rsi
41 ff d2 call r10 ; ← dispatch to profiler chain
eb 04 jmp +4
|
r10 holds a function pointer to the callback notification wrapper. Inside that chain, hipProfilerRegisterChunkCallbackExt walks a linked list of registered profiler callbacks. On Windows gfx1201, this list contains an uninitialized or corrupted node — likely a race condition during runtime initialization on first kernel dispatch.
The linked list walk:
1
2
3
4
5
6
|
; hipProfilerRegisterChunkCallbackExt+0x5c257:
48 8b 41 08 mov rax, [rcx+8] ; rcx = 7b2c450fc892ac84 → CRASH
49 3b c0 cmp rax, r8
74 21 je (not found)
48 8b 09 mov rcx, [rcx] ; next node
...
|
First Patch — Wrong#
The obvious fix: NOP out the call r10 at hipLaunchKernel+0x85.
File offset 0x4549B5:
Before: 41 FF D2 (call r10)
After: 90 90 90 (NOP)
The crash stopped. Training appeared to run. GPU showed 100% compute in Task Manager.
But something was wrong. Checking model weights after loading:
1
2
3
|
emb = model.model.embed_tokens.weight
print(emb.float().abs().max().item())
# 0.0
|
Every tensor transferred to GPU was zero:
1
2
3
|
t = torch.tensor([0.265625], dtype=torch.bfloat16)
print(t.cuda())
# tensor([0.], device='cuda:0')
|
The NOP stopped the crash but also prevented actual kernel execution. Looking at the context after the patched call:
1
2
3
4
|
0x4549b5: 90 90 90 NOP NOP NOP ← patched
0x4549b8: eb 04 jmp +4 ← jumps to epilogue
0x4549ba: 8b 44 24 40 mov eax, [rsp+40]
0x4549be: 48 8b 8c 24 ... ; ← function epilogue
|
Without the call r10, execution falls through to jmp +4 which jumps directly to the function epilogue. hipLaunchKernel returns without ever dispatching the kernel. GPU memory allocations succeed (DMA path), but all compute kernels are silently skipped — leaving tensors at their initial zero state.
Loss during training: 0.000, nan, -8.5e+36. The model was “training” on all-zero tensors.
Correct Patch#
Instead of patching the call site in hipLaunchKernel, patch the crashing function itself. hipProfilerRegisterChunkCallbackExt is a callback lookup — it should return 0 (not found) when the list is invalid. Making it immediately return 0 is safe: the profiler system simply finds no registered callbacks and skips notification. Kernel dispatch proceeds normally.
Locate the export via the PE export table:
hipProfilerRegisterChunkCallbackExt RVA=0x4BC960 FileOffset=0x4BBD60
Patch the function prologue:
File offset 0x4BBD60:
Before: 48 89 5c ... (push rbx; function prologue)
After: 33 c0 c3 (xor eax, eax; ret → return 0)
Verification:
1
2
3
4
5
6
7
8
|
t = torch.tensor([0.265625, -0.1, 0.5], dtype=torch.bfloat16)
print(t.cuda())
# tensor([ 0.2656, -0.1001, 0.5000], device='cuda:0')
model = AutoModelForCausalLM.from_pretrained('./foundation-sec-8b',
torch_dtype=torch.bfloat16, device_map={'': 0})
print(model.model.embed_tokens.weight.float().abs().max().item())
# 0.265625
|
Patch Script#
GitHub: JM00NJ/amdhip64-hipLaunchKernel-crash-fix
The script locates hipProfilerRegisterChunkCallbackExt dynamically via the PE export table (no hardcoded offset dependency), creates a backup, and reverts the old wrong patch automatically if present.
1
|
python patch_amdhip64.py "rocm_env\Lib\site-packages\_rocm_sdk_core\bin\amdhip64_7.dll"
|
Fine-Tuning Script#
With the patch applied, standard BF16 training works. The script below fine-tunes Foundation-Sec-8B with LoRA. Modify the variables at the top for your setup — everything else can stay as-is.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
"""
Foundation-Sec-8B LoRA v8
Clean version — DLL correctly patched, standard BF16 training
No FP32LossTrainer hack, no manual label shift, no FP32 LoRA cast
"""
import os, json, math
import torch
import torch.nn.functional as F
from datasets import Dataset
from transformers import (
AutoModelForCausalLM, AutoTokenizer,
Trainer, TrainingArguments,
DataCollatorForLanguageModeling,
)
from peft import LoraConfig, get_peft_model, TaskType
MODEL_PATH = r".\foundation-sec-8b"
DATASET_PATH = r".\final_dataset.jsonl"
OUTPUT_DIR = r".\finetuned-model"
LORA_R = 8
LORA_ALPHA = 16
LORA_DROPOUT = 0.0
TARGET_MODULES = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"]
BATCH_SIZE = 1
GRAD_ACCUM = 8
LEARNING_RATE = 2e-5
NUM_EPOCHS = 3
MAX_SEQ_LEN = 1024
MAX_GRAD_NORM = 1.0
SAVE_STEPS = 50
LOGGING_STEPS = 5
def load_dataset_from_jsonl(path):
records = []
with open(path, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if line:
records.append(json.loads(line))
print(f"[+] {len(records)} entries loaded")
return Dataset.from_list(records)
def format_alpaca(example):
msgs = example["messages"]
parts = []
system_msg = next((m["content"] for m in msgs if m["role"] == "system"), None)
if system_msg:
parts.append(f"### System:\n{system_msg}")
for msg in msgs:
if msg["role"] == "user":
parts.append(f"### User:\n{msg['content']}")
elif msg["role"] == "assistant":
parts.append(f"### Assistant:\n{msg['content']}")
return {"text": "\n\n".join(parts) + "\n"}
def main():
print("=" * 60)
print(" Foundation-Sec-8B LoRA v8 — JM00NJ")
print(" Standard BF16 training (DLL patched correctly)")
print("=" * 60)
if not torch.cuda.is_available():
print("[!] No GPU found!"); return
print(f"[+] GPU : {torch.cuda.get_device_name(0)}")
print(f"[+] VRAM : {torch.cuda.get_device_properties(0).total_memory/1e9:.1f} GB")
# Dataset
print(f"\n[1/4] Dataset: {DATASET_PATH}")
raw = load_dataset_from_jsonl(DATASET_PATH)
text_dataset = raw.map(format_alpaca, remove_columns=raw.column_names)
# Tokenizer
print(f"[2/4] Tokenizer: {MODEL_PATH}")
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
def tokenize(example):
return tokenizer(example["text"], truncation=True,
max_length=MAX_SEQ_LEN, padding=False)
print("[+] Tokenizing...")
tokenized = text_dataset.map(tokenize, remove_columns=["text"], desc="Tokenizing")
tokenized.set_format("torch")
print(f"[+] {len(tokenized)} examples ready")
# Model — BF16, DLL now correctly patched
print(f"\n[3/4] Loading model (BF16)...")
model = AutoModelForCausalLM.from_pretrained(
MODEL_PATH, dtype=torch.bfloat16, device_map={"": 0},
trust_remote_code=True, low_cpu_mem_usage=True,
attn_implementation="eager",
)
print(f"[+] VRAM: {torch.cuda.memory_allocated(0)/1e9:.1f} GB")
# LoRA — stay in BF16, no FP32 cast
model = get_peft_model(model, LoraConfig(
r=LORA_R, lora_alpha=LORA_ALPHA, target_modules=TARGET_MODULES,
lora_dropout=LORA_DROPOUT, bias="none",
task_type=TaskType.CAUSAL_LM, init_lora_weights=True,
))
model.print_trainable_parameters()
model.gradient_checkpointing_enable(
gradient_checkpointing_kwargs={"use_reentrant": False})
model.enable_input_require_grads()
# Sanity check
print("\n[*] Sanity check...")
model.eval()
inp = tokenized[0]["input_ids"].unsqueeze(0).to("cuda")
with torch.no_grad():
out = model(input_ids=inp, labels=inp)
print(f" Loss (model internal): {out.loss.item():.4f} (expected 1.5-4.0)")
if out.loss.item() < 0.1:
print(" [!] WARNING: near zero")
model.train()
# Training — standard Trainer, bf16=True
total = (len(tokenized) // (BATCH_SIZE * GRAD_ACCUM)) * NUM_EPOCHS
print(f"\n[4/4] Training... steps:~{total}\n")
trainer = Trainer(
model=model,
args=TrainingArguments(
output_dir=OUTPUT_DIR,
num_train_epochs=NUM_EPOCHS,
per_device_train_batch_size=BATCH_SIZE,
gradient_accumulation_steps=GRAD_ACCUM,
learning_rate=LEARNING_RATE,
fp16=False,
bf16=True, # standard BF16, DLL is now fixed
max_grad_norm=MAX_GRAD_NORM,
warmup_steps=10,
logging_steps=LOGGING_STEPS,
save_steps=SAVE_STEPS, save_total_limit=3,
report_to="none", dataloader_num_workers=0,
optim="adamw_torch", remove_unused_columns=False,
lr_scheduler_type="cosine",
),
train_dataset=tokenized,
data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
)
try:
trainer.train()
print("\n[+] Training complete!")
except torch.cuda.OutOfMemoryError:
print("\n[!] OOM"); return
except Exception as ex:
print(f"\n[!] Error: {ex}")
import traceback; traceback.print_exc(); return
adapter_path = os.path.join(OUTPUT_DIR, "final-adapter")
model.save_pretrained(adapter_path)
tokenizer.save_pretrained(adapter_path)
print(f"[+] Adapter saved: {adapter_path}")
print("\n[*] Test...")
model.eval()
prompt = ("### User:\nI found a NULL pointer dereference in a Windows kernel "
"driver at DISPATCH_LEVEL. What should I investigate next?\n\n"
"### Assistant:\n")
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=200, temperature=0.7,
do_sample=True, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
print("\nDone!")
if __name__ == "__main__":
main()
|
Notes#
- Re-run the patcher after ROCm updates — the DLL will be overwritten
- The patch disables ROCprofiler tracing.
rocprof will not intercept HIP API calls on the patched binary. For profiling, wait for the upstream fix
- Reported to AMD: ROCm/TheRock#7732, ROCm/rocm-systems#10924
