Background

I was setting up a fine-tuning pipeline for Foundation-Sec-8B on a Windows 11 machine with an RX 9070 XT using ROCm 7.14. The stack: PyTorch 2.12.0+rocm7.14.0, PEFT, standard HuggingFace Transformers. GPU correctly detected, VRAM visible, torch.cuda.is_available() returned True.

The first sign was the crash output in the terminal itself — the process printed a memory address alongside 0xC0000005 before dying. No Python exception was raised, no traceback. Just a raw AV and an exit.

This is the analysis of what caused it, what the wrong fix looked like, and the correct patch.

Environment
GPU AMD Radeon RX 9070 XT (gfx1201, RDNA 4)
OS Windows 11 (build 26100)
ROCm 7.14.0 (torch.version.hip = 7.14.60850)
PyTorch 2.12.0+rocm7.14.0
Driver Adrenalin 32.0.31041.1004
Issue ROCm/TheRock #7732

Reproducing the Crash

Any operation dispatching a HIP kernel crashes immediately:

1
2

import torch
t = torch.tensor([1.0, 2.0, 3.0]).cuda()  # process dies here

No exception is raised. The Python process exits with 0xC0000005.

WinDbg Analysis

Standard crash dump analysis shows the process is already dead at exit. To catch the access violation at its origin, attach WinDbg with child process tracking and break on first-chance AV:

windbg -o -G rocm_env\Scripts\python.exe finetune.py

At the initial break:

sxe av
g

The process breaks mid-execution with:

(1820.1f1c): Access violation - code c0000005 (first chance)
amdhip64_7!hipProfilerRegisterChunkCallbackExt+0x5c257:
00007ffd`4bf08bb7  mov rax, qword ptr [rcx+8]

r gives:

rcx = 7b2c450fc892ac84   ← not null — garbage pointer
rip = 00007ffd4bf08bb7

~* kn on thread 9:

# 00  amdhip64_7!hipProfilerRegisterChunkCallbackExt+0x5c257
# 01  amdhip64_7!hipRegisterTracerCallback+0x14a0e8
# 02  amdhip64_7!hipRegisterTracerCallback+0x135c16
# 03  amdhip64_7!hipRegisterTracerCallback+0x120ece
# 04  amdhip64_7!hipRegisterTracerCallback+0x10c296
# 05  amdhip64_7!hipRegisterTracerCallback+0xf93ce
# 06  amdhip64_7!hipLaunchKernel+0x88
# 07  torch_hip!at::native::...

The call chain: hipLaunchKernel+0x85 dispatches into the ROCprofiler callback registry. The registry walks a linked list. One node contains rcx = 7b2c450fc892ac84 — a garbage, non-null pointer. Dereferencing [rcx+8] produces the AV.

Root Cause

hipLaunchKernel calls into the ROCprofiler callback notification chain on every kernel dispatch via an indirect call at +0x85:

1
2
3
4

; hipLaunchKernel+0x82:
48 8b ce        mov    rcx, rsi
41 ff d2        call   r10        ; ← dispatch to profiler chain
eb 04           jmp    +4

r10 holds a function pointer to the callback notification wrapper. Inside that chain, hipProfilerRegisterChunkCallbackExt walks a linked list of registered profiler callbacks. On Windows gfx1201, this list contains an uninitialized or corrupted node — likely a race condition during runtime initialization on first kernel dispatch.

The linked list walk:

1
2
3
4
5
6

; hipProfilerRegisterChunkCallbackExt+0x5c257:
48 8b 41 08     mov rax, [rcx+8]    ; rcx = 7b2c450fc892ac84 → CRASH
49 3b c0        cmp rax, r8
74 21           je  (not found)
48 8b 09        mov rcx, [rcx]      ; next node
...

First Patch — Wrong

The obvious fix: NOP out the call r10 at hipLaunchKernel+0x85.

File offset 0x4549B5:
  Before: 41 FF D2   (call r10)
  After:  90 90 90   (NOP)

The crash stopped. Training appeared to run. GPU showed 100% compute in Task Manager.

But something was wrong. Checking model weights after loading:

1
2
3

emb = model.model.embed_tokens.weight
print(emb.float().abs().max().item())
# 0.0

Every tensor transferred to GPU was zero:

1
2
3

t = torch.tensor([0.265625], dtype=torch.bfloat16)
print(t.cuda())
# tensor([0.], device='cuda:0')

The NOP stopped the crash but also prevented actual kernel execution. Looking at the context after the patched call:

1
2
3
4

0x4549b5: 90 90 90        NOP NOP NOP       ← patched
0x4549b8: eb 04           jmp +4            ← jumps to epilogue
0x4549ba: 8b 44 24 40     mov eax, [rsp+40]
0x4549be: 48 8b 8c 24 ... ; ← function epilogue

Without the call r10, execution falls through to jmp +4 which jumps directly to the function epilogue. hipLaunchKernel returns without ever dispatching the kernel. GPU memory allocations succeed (DMA path), but all compute kernels are silently skipped — leaving tensors at their initial zero state.

Loss during training: 0.000, nan, -8.5e+36. The model was “training” on all-zero tensors.

Correct Patch

Instead of patching the call site in hipLaunchKernel, patch the crashing function itself. hipProfilerRegisterChunkCallbackExt is a callback lookup — it should return 0 (not found) when the list is invalid. Making it immediately return 0 is safe: the profiler system simply finds no registered callbacks and skips notification. Kernel dispatch proceeds normally.

Locate the export via the PE export table:

hipProfilerRegisterChunkCallbackExt  RVA=0x4BC960  FileOffset=0x4BBD60

Patch the function prologue:

File offset 0x4BBD60:
  Before: 48 89 5c ...   (push rbx; function prologue)
  After:  33 c0 c3       (xor eax, eax; ret  →  return 0)

Verification:

1
2
3
4
5
6
7
8

t = torch.tensor([0.265625, -0.1, 0.5], dtype=torch.bfloat16)
print(t.cuda())
# tensor([ 0.2656, -0.1001,  0.5000], device='cuda:0')

model = AutoModelForCausalLM.from_pretrained('./foundation-sec-8b',
    torch_dtype=torch.bfloat16, device_map={'': 0})
print(model.model.embed_tokens.weight.float().abs().max().item())
# 0.265625

Patch Script

GitHub: JM00NJ/amdhip64-hipLaunchKernel-crash-fix

The script locates hipProfilerRegisterChunkCallbackExt dynamically via the PE export table (no hardcoded offset dependency), creates a backup, and reverts the old wrong patch automatically if present.

1

python patch_amdhip64.py "rocm_env\Lib\site-packages\_rocm_sdk_core\bin\amdhip64_7.dll"

Fine-Tuning Script

With the patch applied, standard BF16 training works. The script below fine-tunes Foundation-Sec-8B with LoRA. Modify the variables at the top for your setup — everything else can stay as-is.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181

"""
Foundation-Sec-8B LoRA v8
Clean version — DLL correctly patched, standard BF16 training
No FP32LossTrainer hack, no manual label shift, no FP32 LoRA cast
"""

import os, json, math
import torch
import torch.nn.functional as F
from datasets import Dataset
from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    Trainer, TrainingArguments,
    DataCollatorForLanguageModeling,
)
from peft import LoraConfig, get_peft_model, TaskType

MODEL_PATH   = r".\foundation-sec-8b"
DATASET_PATH = r".\final_dataset.jsonl"
OUTPUT_DIR   = r".\finetuned-model"

LORA_R         = 8
LORA_ALPHA     = 16
LORA_DROPOUT   = 0.0
TARGET_MODULES = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]

BATCH_SIZE    = 1
GRAD_ACCUM    = 8
LEARNING_RATE = 2e-5
NUM_EPOCHS    = 3
MAX_SEQ_LEN   = 1024
MAX_GRAD_NORM = 1.0
SAVE_STEPS    = 50
LOGGING_STEPS = 5


def load_dataset_from_jsonl(path):
    records = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                records.append(json.loads(line))
    print(f"[+] {len(records)} entries loaded")
    return Dataset.from_list(records)


def format_alpaca(example):
    msgs = example["messages"]
    parts = []
    system_msg = next((m["content"] for m in msgs if m["role"] == "system"), None)
    if system_msg:
        parts.append(f"### System:\n{system_msg}")
    for msg in msgs:
        if msg["role"] == "user":
            parts.append(f"### User:\n{msg['content']}")
        elif msg["role"] == "assistant":
            parts.append(f"### Assistant:\n{msg['content']}")
    return {"text": "\n\n".join(parts) + "\n"}


def main():
    print("=" * 60)
    print("  Foundation-Sec-8B LoRA v8 — JM00NJ")
    print("  Standard BF16 training (DLL patched correctly)")
    print("=" * 60)

    if not torch.cuda.is_available():
        print("[!] No GPU found!"); return

    print(f"[+] GPU  : {torch.cuda.get_device_name(0)}")
    print(f"[+] VRAM : {torch.cuda.get_device_properties(0).total_memory/1e9:.1f} GB")

    # Dataset
    print(f"\n[1/4] Dataset: {DATASET_PATH}")
    raw = load_dataset_from_jsonl(DATASET_PATH)
    text_dataset = raw.map(format_alpaca, remove_columns=raw.column_names)

    # Tokenizer
    print(f"[2/4] Tokenizer: {MODEL_PATH}")
    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
    tokenizer.padding_side = "right"

    def tokenize(example):
        return tokenizer(example["text"], truncation=True,
                         max_length=MAX_SEQ_LEN, padding=False)

    print("[+] Tokenizing...")
    tokenized = text_dataset.map(tokenize, remove_columns=["text"], desc="Tokenizing")
    tokenized.set_format("torch")
    print(f"[+] {len(tokenized)} examples ready")

    # Model — BF16, DLL now correctly patched
    print(f"\n[3/4] Loading model (BF16)...")
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_PATH, dtype=torch.bfloat16, device_map={"": 0},
        trust_remote_code=True, low_cpu_mem_usage=True,
        attn_implementation="eager",
    )
    print(f"[+] VRAM: {torch.cuda.memory_allocated(0)/1e9:.1f} GB")

    # LoRA — stay in BF16, no FP32 cast
    model = get_peft_model(model, LoraConfig(
        r=LORA_R, lora_alpha=LORA_ALPHA, target_modules=TARGET_MODULES,
        lora_dropout=LORA_DROPOUT, bias="none",
        task_type=TaskType.CAUSAL_LM, init_lora_weights=True,
    ))
    model.print_trainable_parameters()
    model.gradient_checkpointing_enable(
        gradient_checkpointing_kwargs={"use_reentrant": False})
    model.enable_input_require_grads()

    # Sanity check
    print("\n[*] Sanity check...")
    model.eval()
    inp = tokenized[0]["input_ids"].unsqueeze(0).to("cuda")
    with torch.no_grad():
        out = model(input_ids=inp, labels=inp)
    print(f"    Loss (model internal): {out.loss.item():.4f}  (expected 1.5-4.0)")
    if out.loss.item() < 0.1:
        print("    [!] WARNING: near zero")
    model.train()

    # Training — standard Trainer, bf16=True
    total = (len(tokenized) // (BATCH_SIZE * GRAD_ACCUM)) * NUM_EPOCHS
    print(f"\n[4/4] Training...  steps:~{total}\n")

    trainer = Trainer(
        model=model,
        args=TrainingArguments(
            output_dir=OUTPUT_DIR,
            num_train_epochs=NUM_EPOCHS,
            per_device_train_batch_size=BATCH_SIZE,
            gradient_accumulation_steps=GRAD_ACCUM,
            learning_rate=LEARNING_RATE,
            fp16=False,
            bf16=True,         # standard BF16, DLL is now fixed
            max_grad_norm=MAX_GRAD_NORM,
            warmup_steps=10,
            logging_steps=LOGGING_STEPS,
            save_steps=SAVE_STEPS, save_total_limit=3,
            report_to="none", dataloader_num_workers=0,
            optim="adamw_torch", remove_unused_columns=False,
            lr_scheduler_type="cosine",
        ),
        train_dataset=tokenized,
        data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
    )

    try:
        trainer.train()
        print("\n[+] Training complete!")
    except torch.cuda.OutOfMemoryError:
        print("\n[!] OOM"); return
    except Exception as ex:
        print(f"\n[!] Error: {ex}")
        import traceback; traceback.print_exc(); return

    adapter_path = os.path.join(OUTPUT_DIR, "final-adapter")
    model.save_pretrained(adapter_path)
    tokenizer.save_pretrained(adapter_path)
    print(f"[+] Adapter saved: {adapter_path}")

    print("\n[*] Test...")
    model.eval()
    prompt = ("### User:\nI found a NULL pointer dereference in a Windows kernel "
              "driver at DISPATCH_LEVEL. What should I investigate next?\n\n"
              "### Assistant:\n")
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=200, temperature=0.7,
                             do_sample=True, pad_token_id=tokenizer.eos_token_id)
    print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
    print("\nDone!")


if __name__ == "__main__":
    main()

Notes

  • Re-run the patcher after ROCm updates — the DLL will be overwritten
  • The patch disables ROCprofiler tracing. rocprof will not intercept HIP API calls on the patched binary. For profiling, wait for the upstream fix
  • Reported to AMD: ROCm/TheRock#7732, ROCm/rocm-systems#10924