diff --git a/reports/eval-2026-07-01.md b/reports/eval-2026-07-01.md new file mode 100644 index 0000000..ae79673 --- /dev/null +++ b/reports/eval-2026-07-01.md @@ -0,0 +1,33 @@ +# Eval Report — 2026-07-01 + +Generated by `scripts/run_eval.sh` → `tsmm.eval.report`. + +## Discipline notes +- **Main metric: VUS-PR** (threshold-free). +- **Point-F1 reported WITHOUT point-adjust.** PA-F1 shown separately as a CONTROL only. +- Trivial baselines (random / constant / all-report) are included. +- LLM-judge backend: `stub`. + +## Dataset: synth + +| predictor | n | VUS-PR | Aff-F1 | AUC-PR | Point-F1 | PA-F1(control) | parse_rate | +|---|---|---|---|---|---|---|---| +| model | 361 | 0.2241 | 0.2474 | 0.3050 | 0.2186 | 0.5018 | 0.98 | +| pure_llm | 361 | 0.2391 | 0.0029 | 0.2940 | 0.0013 | 0.0071 | 0.00 | +| random | 361 | 0.3266 | 0.3970 | 0.2689 | 0.3266 | 0.9771 | 1.00 | +| constant | 361 | 0.2601 | 0.4302 | 0.2951 | 0.3914 | 1.0000 | 1.00 | +| all_report | 361 | 0.2601 | 0.4302 | 0.2951 | 0.3914 | 1.0000 | 1.00 | + +_PA = point-adjust; shown as control only, not headline._ + +### QA judge (stub 0-5) — synth + +| predictor | anomaly | compare | describe | event | forecast | root_cause | mean | +|---|---|---|---|---|---|---|---| +| model | 2.45 | 2.75 | 1.34 | 3.89 | 0.03 | 3.99 | 2.41 | +| pure_llm | 0.03 | 1.25 | 0.19 | 1.64 | 0.12 | 2.17 | 0.90 | +| random | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | +| constant | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | +| all_report | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | + +_QA judge = offline stub token-overlap heuristic; swap `--judge_backend openai` for GPT-4o scoring._