 张宗平
|
4c4a8f6435
|
feat(m5): full eval pipeline — parse/metrics/judge/baselines/report (T5.1-T5.5)
- src/tsmm/eval/parse_answer.py: text→JSON segments→point scores (JSON-first,
regex fallback, all-zero on fail).
- src/tsmm/eval/ts_metrics.py: VUS-PR (main, threshold-free; constant-score →
prevalence, NOT inflated), AUC-PR, Point-F1 (no PA), PA-F1 (control only),
self-contained Affiliation-F1.
- src/tsmm/eval/qa_judge.py: RuleJudge (anomaly IoU / describe+forecast numeric
tolerance) + LLMJudge (stub backend = deterministic offline heuristic for the
no-network host; openai backend = GPT-4o 0-5; pluggable callable).
- src/tsmm/eval/baselines.py: trivial (Random/Constant/AllReport) + ts_to_text
+ pure_llm_answer glue; Time-LLM/ChatTS marked not-reproduced (honest).
- src/tsmm/eval/report.py + scripts/run_eval.sh: trained model + all baselines
through the SAME pipeline → reports/eval-YYYYMMDD.md (VUS-PR main, PA-F1
explicitly labelled control, trivial baselines included).
- tests: 45 new (parse 13, metrics 12, judge 12, baselines 8). 145 passing.
- Smoke: run_eval on 40 samples produces valid markdown; PA-F1≈1.0 vs
VUS-PR≈0.35 for all_report demonstrates why PA must not be the headline.
|
2026-06-30 03:18:14 +00:00 |
|