feat(m5): full eval pipeline — parse/metrics/judge/baselines/report (T5.1-T5.5)
- src/tsmm/eval/parse_answer.py: text→JSON segments→point scores (JSON-first, regex fallback, all-zero on fail). - src/tsmm/eval/ts_metrics.py: VUS-PR (main, threshold-free; constant-score → prevalence, NOT inflated), AUC-PR, Point-F1 (no PA), PA-F1 (control only), self-contained Affiliation-F1. - src/tsmm/eval/qa_judge.py: RuleJudge (anomaly IoU / describe+forecast numeric tolerance) + LLMJudge (stub backend = deterministic offline heuristic for the no-network host; openai backend = GPT-4o 0-5; pluggable callable). - src/tsmm/eval/baselines.py: trivial (Random/Constant/AllReport) + ts_to_text + pure_llm_answer glue; Time-LLM/ChatTS marked not-reproduced (honest). - src/tsmm/eval/report.py + scripts/run_eval.sh: trained model + all baselines through the SAME pipeline → reports/eval-YYYYMMDD.md (VUS-PR main, PA-F1 explicitly labelled control, trivial baselines included). - tests: 45 new (parse 13, metrics 12, judge 12, baselines 8). 145 passing. - Smoke: run_eval on 40 samples produces valid markdown; PA-F1≈1.0 vs VUS-PR≈0.35 for all_report demonstrates why PA must not be the headline.
This commit is contained in:
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# Full evaluation (T5.5): trained model + all baselines through the same
|
||||
# parse_answer + ts_metrics pipeline → reports/eval-YYYYMMDD.md
|
||||
#
|
||||
# Usage:
|
||||
# scripts/run_eval.sh # trivial baselines only (CPU, fast)
|
||||
# scripts/run_eval.sh --model # include the trained model (GPU)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
export PYTHONPATH=src
|
||||
.venv/bin/python -m tsmm.eval.report \
|
||||
--synth data/eval_synth.jsonl \
|
||||
--real data/eval_real.jsonl \
|
||||
--report_dir reports \
|
||||
--judge_backend "${JUDGE_BACKEND:-stub}" \
|
||||
"$@"
|
||||
Reference in New Issue
Block a user