fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline

Critical eval-pipeline bugs found while running the stage-2 model: 1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}. The model learned the correct format but scored 0 → VUS-PR=0. Now accepts both schemas + a balanced-brace JSON scanner (the flat regex could not match nested objects the model emits). 2. qa_judge._extract_json had the same nesting bug — reuse the scanner. 3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped to 192 in report + instruct_check. Report enhancements: - wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4 'model vs pure LLM' is actually computed. Heavy predictors built/evaluated one at a time with GPU cleanup (single 12GB card). - add QA-judge table across all 6 categories (design §6.5 问答均分). - add --pure_llm / --max_per_cat flags. Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm 0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
2026-06-30 10:02:59 +00:00
parent 8aeddebce9
commit 40ec960cf8
8 changed files with 430 additions and 55 deletions
@@ -0,0 +1,33 @@
+# Eval Report — 2026-06-30
+
+Generated by `scripts/run_eval.sh` → `tsmm.eval.report`.
+
+## Discipline notes
+- **Main metric: VUS-PR** (threshold-free).
+- **Point-F1 reported WITHOUT point-adjust.** PA-F1 shown separately as a CONTROL only.
+- Trivial baselines (random / constant / all-report) are included.
+- LLM-judge backend: `stub`.
+
+## Dataset: synth
+
+| predictor | n | VUS-PR | Aff-F1 | AUC-PR | Point-F1 | PA-F1(control) | parse_rate |
+|---|---|---|---|---|---|---|---|
+| model | 40 | 0.1414 | 0.1674 | 0.2343 | 0.1414 | 0.4296 | 1.00 |
+| pure_llm | 40 | 0.2195 | 0.0000 | 0.2357 | 0.0000 | 0.0000 | 0.00 |
+| random | 40 | 0.2859 | 0.3484 | 0.2241 | 0.2859 | 0.9746 | 1.00 |
+| constant | 40 | 0.2195 | 0.3771 | 0.2357 | 0.3395 | 1.0000 | 1.00 |
+| all_report | 40 | 0.2195 | 0.3771 | 0.2357 | 0.3395 | 1.0000 | 1.00 |
+
+_PA = point-adjust; shown as control only, not headline._
+
+### QA judge (stub 0-5) — synth
+
+| predictor | anomaly | compare | describe | event | forecast | root_cause | mean |
+|---|---|---|---|---|---|---|---|
+| model | 2.60 | 2.64 | 1.65 | 4.08 | 0.12 | 3.60 | 2.45 |
+| pure_llm | 0.04 | 1.21 | 0.17 | 1.57 | 0.13 | 2.18 | 0.88 |
+| random | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
+| constant | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
+| all_report | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
+
+_QA judge = offline stub token-overlap heuristic; swap `--judge_backend openai` for GPT-4o scoring._
@@ -0,0 +1,130 @@
+{
+  "summary": {
+    "anomaly_json_parse_rate": 1.0,
+    "anomaly_json_pass": true,
+    "overall_usability": 0.9166666666666666,
+    "overall_usability_pass": true
+  },
+  "per_category": {
+    "anomaly": {
+      "n": 10,
+      "parse_rate": 1.0,
+      "nonempty_rate": 1.0,
+      "rule_acc": 0.02923274723648774,
+      "judge_mean": 2.4099999999999997,
+      "usability": 1.0,
+      "examples": [
+        {
+          "pred": "{\"anomaly_regions\": [{\"start\": 10, \"end\": 93}, {\"start\": 245, \"end\": 286}], \"n_regions\": 2}",
+          "ref": "{\"anomaly_regions\": [{\"start\": 100, \"end\": 145}, {\"start\": 146, \"end\": 200}, {\"start\": 371, \"end\": 373}, {\"start\": 382, \"end\": 445}], \"n_regions\": 4}",
+          "parsed": true
+        },
+        {
+          "pred": "{\"anomaly_regions\": [{\"start\": 10, \"end\": 29}, {\"start\": 43, \"end\": 56}], \"n_regions\": 2}",
+          "ref": "{\"anomaly_regions\": [{\"start\": 130, \"end\": 202}, {\"start\": 274, \"end\": 328}, {\"start\": 344, \"end\": 346}], \"n_regions\": 3}",
+          "parsed": true
+        }
+      ]
+    },
+    "describe": {
+      "n": 10,
+      "parse_rate": 1.0,
+      "nonempty_rate": 1.0,
+      "rule_acc": 0.0,
+      "judge_mean": 1.72,
+      "usability": 1.0,
+      "examples": [
+        {
+          "pred": "{\"stats\": [{\"name\": \"Active connections\", \"unit\": \"conn\", \"mean\": 43.0162578900111, \"std\": 10.0440401101011, \"min\": 25.0000091902004, \"max\": 61.204120208029}, {",
+          "ref": "{\"stats\": [{\"name\": \"Active connections\", \"unit\": \"conn\", \"mean\": 70.08136858780824, \"std\": 5.166230550292378, \"min\": 60.15172120024713, \"max\": 84.5416429780042",
+          "parsed": true
+        },
+        {
+          "pred": "{\"stats\": [{\"name\": \"Queue length\", \"unit\": \"tasks\", \"mean\": 42.0167398510011, \"std\": 5.04020002110111, \"min\": 31.0000000000001, \"max\": 55.0604040420208}, {\"sta",
+          "ref": "{\"stats\": [{\"name\": \"Queue length\", \"unit\": \"tasks\", \"mean\": 71.13851266805797, \"std\": 5.420230450566408, \"min\": 51.888456943983975, \"max\": 85.02043191625414}, ",
+          "parsed": true
+        }
+      ]
+    },
+    "forecast": {
+      "n": 10,
+      "parse_rate": 0.5,
+      "nonempty_rate": 0.5,
+      "rule_acc": 0.0,
+      "judge_mean": 0.14,
+      "usability": 0.5,
+      "examples": [
+        {
+          "pred": "{\"stats\": [{\"name\": \"Disk I/O\", \"unit\": \"MB/s\", \"mean\": 43.0125678911111, \"std\": 10.0404029110121, \"min\": 27.0000001200001, \"max\": 61.912222028021}, {\"name\": \"M",
+          "ref": "{\"forecast\": [[61.36301953352816, 61.372689003454326, 61.382358473380485, 61.392027943306644, 61.40169741323281, 61.41136688315897, 61.42103635308513, 61.430705",
+          "parsed": true
+        },
+        {
+          "pred": "{\"stats\": [{\"name\": \"Disk I/O\", \"unit\": \"MB/s\", \"mean\": 43.0125678912111, \"std\": 10.0400219204111, \"min\": 25.0000000200004, \"max\": 61.821123021212}, {\"name\": \"B",
+          "ref": "{\"forecast\": [[35.87848293955361, 35.69287951372365, 35.5072760878937, 35.321672662063754, 35.1360692362338, 34.95046581040385, 34.764862384573895, 34.579258958",
+          "parsed": true
+        }
+      ]
+    },
+    "root_cause": {
+      "n": 10,
+      "parse_rate": 1.0,
+      "nonempty_rate": 1.0,
+      "rule_acc": null,
+      "judge_mean": 3.75,
+      "usability": 1.0,
+      "examples": [
+        {
+          "pred": "The deviation in Active connections is most consistent with a sudden level shift. A plausible root cause is a deployment or configuration change that altered th",
+          "ref": "The deviation in Active connections is most consistent with an anomaly. A plausible root cause is a deployment or configuration change that altered the operatin",
+          "parsed": true
+        },
+        {
+          "pred": "The deviation in CPU usage is most consistent with a variance expansion. A plausible root cause is a deployment or configuration change that altered the operati",
+          "ref": "The deviation in CPU usage is most consistent with a gradual drift. A plausible root cause is a deployment or configuration change that altered the operating po",
+          "parsed": true
+        }
+      ]
+    },
+    "event": {
+      "n": 10,
+      "parse_rate": 1.0,
+      "nonempty_rate": 1.0,
+      "rule_acc": null,
+      "judge_mean": 4.5,
+      "usability": 1.0,
+      "examples": [
+        {
+          "pred": "No known event corresponds to the observed change.",
+          "ref": "No known event corresponds to the observed change.",
+          "parsed": true
+        },
+        {
+          "pred": "No known event corresponds to the observed change.",
+          "ref": "No known event corresponds to the observed change.",
+          "parsed": true
+        }
+      ]
+    },
+    "compare": {
+      "n": 10,
+      "parse_rate": 1.0,
+      "nonempty_rate": 1.0,
+      "rule_acc": null,
+      "judge_mean": 2.3899999999999997,
+      "usability": 1.0,
+      "examples": [
+        {
+          "pred": "Per-channel summary:\nActive connections: mean 30.26conn, range [15.47, 48.90];\nGPU utilization: mean 32.01%, range [19.00, 48.91];\nBattery level: mean 55.01%, r",
+          "ref": "Per-channel summary:\nActive connections: mean 57.19conn, range [28.39, 84.93]\nGPU utilization: mean 52.88%, range [17.81, 92.55]\nBattery level: mean 49.57%, ran",
+          "parsed": true
+        },
+        {
+          "pred": "Per-channel summary:\nCPU usage: mean 36.04%, range [12.57, 58.97]; Request latency p50: mean 32.04ms; Queue length: mean 31.05tasks; Disk I/O: mean 40.07MB/s; R",
+          "ref": "Per-channel summary:\nCPU usage: mean 33.10%, range [1.57, 54.50]\nRequest latency p50: mean 35.22ms, range [9.60, 59.28]\nQueue length: mean 26.72tasks, range [12",
+          "parsed": true
+        }
+      ]
+    }
+  }
+}