Critical supervisor bug: completion check + --resume both used a glob that
only matched '{stage}_step*.pt', so a finished '{stage}_final.pt' (whose
step lives inside the file, not the filename) was invisible. Symptom: after
stage1 hit 31250 and saved final.pt, the supervisor re-launched --resume,
which picked step8000 (oldest sort-order quirk) and started OVERWRITING the
just-finished run from step 8000. Caught mid-overwrite; final.pt survived.
Fix: _latest_step (both stage1 & stage2) now globs step*.pt AND final.pt,
reads each one's 'step' field, returns the true highest. Supervisor's
run_until uses the same helper + reads step from the chosen ckpt to decide
completion. Verified: stage1(31250) now correctly reports 'done' and the
supervisor skips straight to stage2.
Also corrected supervisor stage1 config: bs=4 ga=8 (not bs=8 ga=4) — the
latter OOMs on long-sequence batches (logits 4.6GB alloc). expandable_segments
set to reduce fragmentation.
Full-scale retrain completed: stage1 31250 steps EMA 1.26, stage2 9375
steps EMA 0.75 (beats small-scale avg 0.81).
Addresses both points from review:
1) Resume (so a killed run converges across multiple restarts, like gen_supervisor):
- checkpoints now carry optimizer state + EMA value in a 'train_state' field
- --resume auto-picks the latest step ckpt and continues; --resume_from for explicit
- stage2 resume rebuilds the LoRA structure via enable_lora (same path as fresh)
then overlays saved adapter weights via set_peft_model_state_dict. Using
PeftModel.from_pretrained instead reset requires_grad and shrank the trainable
set 129->33 tensors, corrupting the optimizer state — verified and fixed.
- Smoke-verified: stage1 step4->10, stage2 step4->8, both restore optimizer +
EMA + LoRA with no state mismatch.
2) EMA loss logging (honest answer to 'is the loss swing a real problem?'):
- the old log printed the raw per-micro-batch loss every N steps; on
heterogeneous 6-task data that swings 2-3x purely from sampling (anomaly
JSON is low-loss, forecast numbers high-loss), so it looked 'unstable'
while the model may well have been converging.
- EMA(alpha=0.05) now logged alongside raw + as a tensorboard scalar
(total_loss_ema / sft_loss_ema). Read the EMA, not the raw, to judge
convergence. lr left unchanged (1e-4) pending EMA evidence.
Adds scripts/train_full_supervisor.sh: stage1->stage2 resume loop, same
crash-convergence guarantee as gen_full_supervisor.
The original gen_synthetic.py held all N samples in a [None]*n list and
only wrote at the end — three full-scale runs were killed mid-way (env
reaps long sessions) and lost everything (0 bytes on disk each time).
Fixes (proven on the 500k+100k run that just completed):
- run() now streams: each completed future is written to disk the instant
it's produced, via an O(workers) reorder buffer that keeps output in
index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers).
- --resume flag: counts existing lines and appends only the missing
suffix (same seed => deterministic prefix, so resume is content-safe).
- scripts/gen_full_supervisor.sh: loops 'run --resume -> check count'
until target reached. Each surviving iteration adds >=1 line, so the
run converges even across multiple kills (took 2 kills + 2 resumes to
finish the 500k align). On this host: 2 partial runs (50k + 437k) then
a final resume filled the last 62k.
- Verified: order-deterministic (same-seed runs byte-identical), resume
appends without rewriting the prefix, target-already-met => no-op.
Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) +
eval_synth 2k held-out (untouched).
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
张宗平