fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
This commit is contained in:
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# M4 + M5 exit-validation eval (T4.2 / T4.3 / T5.6).
|
||||
#
|
||||
# Runs (1) the instruction-following check (T4.2) and (2) the full eval report
|
||||
# (T5.5/5.6) including the trained model + trivial baselines, on the stage-2
|
||||
# checkpoint. Requires a free GPU (so run AFTER training finishes).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/run_m4_m5_eval.sh [stage2_ckpt]
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
export PYTHONPATH=src
|
||||
|
||||
CKPT="${1:-checkpoints/stage2/stage2_final.pt}"
|
||||
if [ ! -f "$CKPT" ]; then
|
||||
echo "[run_m4_m5_eval] checkpoint $CKPT not found"; exit 1
|
||||
fi
|
||||
|
||||
echo "=== T4.2 instruction-following check (6 cats × 10) ==="
|
||||
.venv/bin/python -m tsmm.eval.instruct_check \
|
||||
--data data/eval_synth.jsonl \
|
||||
--per_cat 10 \
|
||||
--stage2_ckpt "$CKPT" \
|
||||
--judge_backend stub \
|
||||
--out reports/instruct_check.json
|
||||
|
||||
echo
|
||||
echo "=== T5.5/T5.6 full eval report (model + trivial baselines) ==="
|
||||
.venv/bin/python -m tsmm.eval.report \
|
||||
--synth data/eval_synth.jsonl \
|
||||
--model \
|
||||
--stage2_ckpt "$CKPT" \
|
||||
--judge_backend stub \
|
||||
--report_dir reports
|
||||
|
||||
echo
|
||||
echo "[run_m4_m5_eval] done. See reports/."
|
||||
Reference in New Issue
Block a user