fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline

Critical eval-pipeline bugs found while running the stage-2 model:

1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
   data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
   The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
   both schemas + a balanced-brace JSON scanner (the flat regex could not
   match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
   to 192 in report + instruct_check.

Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
  'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
  one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.

Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
This commit is contained in:
张宗平
2026-06-30 10:02:59 +00:00
parent 8aeddebce9
commit 40ec960cf8
8 changed files with 430 additions and 55 deletions
+33
View File
@@ -0,0 +1,33 @@
# Eval Report — 2026-06-30
Generated by `scripts/run_eval.sh``tsmm.eval.report`.
## Discipline notes
- **Main metric: VUS-PR** (threshold-free).
- **Point-F1 reported WITHOUT point-adjust.** PA-F1 shown separately as a CONTROL only.
- Trivial baselines (random / constant / all-report) are included.
- LLM-judge backend: `stub`.
## Dataset: synth
| predictor | n | VUS-PR | Aff-F1 | AUC-PR | Point-F1 | PA-F1(control) | parse_rate |
|---|---|---|---|---|---|---|---|
| model | 40 | 0.1414 | 0.1674 | 0.2343 | 0.1414 | 0.4296 | 1.00 |
| pure_llm | 40 | 0.2195 | 0.0000 | 0.2357 | 0.0000 | 0.0000 | 0.00 |
| random | 40 | 0.2859 | 0.3484 | 0.2241 | 0.2859 | 0.9746 | 1.00 |
| constant | 40 | 0.2195 | 0.3771 | 0.2357 | 0.3395 | 1.0000 | 1.00 |
| all_report | 40 | 0.2195 | 0.3771 | 0.2357 | 0.3395 | 1.0000 | 1.00 |
_PA = point-adjust; shown as control only, not headline._
### QA judge (stub 0-5) — synth
| predictor | anomaly | compare | describe | event | forecast | root_cause | mean |
|---|---|---|---|---|---|---|---|
| model | 2.60 | 2.64 | 1.65 | 4.08 | 0.12 | 3.60 | 2.45 |
| pure_llm | 0.04 | 1.21 | 0.17 | 1.57 | 0.13 | 2.18 | 0.88 |
| random | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| constant | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| all_report | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
_QA judge = offline stub token-overlap heuristic; swap `--judge_backend openai` for GPT-4o scoring._