From cd6209119a92dd1365eb2230d76f6c6e6f3b5ea3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=BC=A0=E5=AE=97=E5=B9=B3?= <zhangzp@qq.com>
Date: Wed, 1 Jul 2026 13:44:31 +0000
Subject: [PATCH] =?UTF-8?q?eval(m5):=20full-scale=20retrain=20report=20?=
 =?UTF-8?q?=E2=80=94=20VUS-PR=200.14=E2=86=920.22,=20model=20beats=20pure?=
 =?UTF-8?q?=5Fllm?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Full-scale retrain (stage1 31250 steps on 500k align, stage2 9375 steps on
100k sft) vs the prior small-scale run (5万/5k steps):

  metric              small-scale    full-scale
  stage2 EMA          0.81           0.75
  model VUS-PR        0.14           0.22  (+57%)
  model Aff-F1        0.21           0.25  (vs pure_llm 0.003 — TS modality helps)
  model QA mean       2.45           2.41
  QA vs pure_llm      2.8×           2.7×  (stable lead, 6 categories)

Confirms the hypothesis that weak anomaly localization was mostly a
data-volume problem, not an algorithmic one: full-scale training lifted
VUS-PR 57% and the model now clearly beats the pure-LLM baseline on
affiliation-F1 (0.25 vs 0.003). VUS-PR still trails trivial baselines
(0.22 vs random 0.33) — precise localization remains the open frontier,
but the TS modality is demonstrably contributing (was unclear before).
---
 reports/eval-2026-07-01.md | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 reports/eval-2026-07-01.md

diff --git a/reports/eval-2026-07-01.md b/reports/eval-2026-07-01.md
new file mode 100644
index 0000000..ae79673
--- /dev/null
+++ b/reports/eval-2026-07-01.md
@@ -0,0 +1,33 @@
+# Eval Report — 2026-07-01
+
+Generated by `scripts/run_eval.sh` → `tsmm.eval.report`.
+
+## Discipline notes
+- **Main metric: VUS-PR** (threshold-free).
+- **Point-F1 reported WITHOUT point-adjust.** PA-F1 shown separately as a CONTROL only.
+- Trivial baselines (random / constant / all-report) are included.
+- LLM-judge backend: `stub`.
+
+## Dataset: synth
+
+| predictor | n | VUS-PR | Aff-F1 | AUC-PR | Point-F1 | PA-F1(control) | parse_rate |
+|---|---|---|---|---|---|---|---|
+| model | 361 | 0.2241 | 0.2474 | 0.3050 | 0.2186 | 0.5018 | 0.98 |
+| pure_llm | 361 | 0.2391 | 0.0029 | 0.2940 | 0.0013 | 0.0071 | 0.00 |
+| random | 361 | 0.3266 | 0.3970 | 0.2689 | 0.3266 | 0.9771 | 1.00 |
+| constant | 361 | 0.2601 | 0.4302 | 0.2951 | 0.3914 | 1.0000 | 1.00 |
+| all_report | 361 | 0.2601 | 0.4302 | 0.2951 | 0.3914 | 1.0000 | 1.00 |
+
+_PA = point-adjust; shown as control only, not headline._
+
+### QA judge (stub 0-5) — synth
+
+| predictor | anomaly | compare | describe | event | forecast | root_cause | mean |
+|---|---|---|---|---|---|---|---|
+| model | 2.45 | 2.75 | 1.34 | 3.89 | 0.03 | 3.99 | 2.41 |
+| pure_llm | 0.03 | 1.25 | 0.19 | 1.64 | 0.12 | 2.17 | 0.90 |
+| random | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
+| constant | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
+| all_report | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
+
+_QA judge = offline stub token-overlap heuristic; swap `--judge_backend openai` for GPT-4o scoring._