40ec960cf8
fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline
...
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
2026-06-30 10:02:59 +00:00
8aeddebce9
test(m4): instruct_check unit tests + unbuffered training scripts
...
- tests/test_instruct_check.py: cover load_by_category (bucketing/cap/seed
determinism/short-pool/blank-line) and _forecast_list (nested/flat/
dict/garbage). 8 tests pass.
- train_stage{1,2}.sh: export PYTHONPATH=src + PYTHONUNBUFFERED=1 so future
background runs stream logs live (current stage2 run is block-buffered).
2026-06-30 07:27:03 +00:00
4c4a8f6435
feat(m5): full eval pipeline — parse/metrics/judge/baselines/report (T5.1-T5.5)
...
- src/tsmm/eval/parse_answer.py: text→JSON segments→point scores (JSON-first,
regex fallback, all-zero on fail).
- src/tsmm/eval/ts_metrics.py: VUS-PR (main, threshold-free; constant-score →
prevalence, NOT inflated), AUC-PR, Point-F1 (no PA), PA-F1 (control only),
self-contained Affiliation-F1.
- src/tsmm/eval/qa_judge.py: RuleJudge (anomaly IoU / describe+forecast numeric
tolerance) + LLMJudge (stub backend = deterministic offline heuristic for the
no-network host; openai backend = GPT-4o 0-5; pluggable callable).
- src/tsmm/eval/baselines.py: trivial (Random/Constant/AllReport) + ts_to_text
+ pure_llm_answer glue; Time-LLM/ChatTS marked not-reproduced (honest).
- src/tsmm/eval/report.py + scripts/run_eval.sh: trained model + all baselines
through the SAME pipeline → reports/eval-YYYYMMDD.md (VUS-PR main, PA-F1
explicitly labelled control, trivial baselines included).
- tests: 45 new (parse 13, metrics 12, judge 12, baselines 8). 145 passing.
- Smoke: run_eval on 40 samples produces valid markdown; PA-F1≈1.0 vs
VUS-PR≈0.35 for all_report demonstrates why PA must not be the headline.
2026-06-30 03:18:14 +00:00
357e2e60bd
feat(m4): stage 2 SFT training script (LoRA r=16) + smoke (T4.1)
...
- src/tsmm/train/stage2.py: loads stage1 ckpt (optional), freeze_llm +
enable_lora(r=16) on q/v_proj, masked LM loss on sft.jsonl, bs=4/grad_accum=8/
ctx=1024/BF16/gradient checkpointing, ckpt every 2000 steps (encoder+projector
state + peft LoRA adapter via save_pretrained).
- scripts/train_stage2.sh: wrapper with design defaults.
- Smoke verified: --max_steps 4, LoRA injected, finite loss, peak 3.29 GB
(≤11GB budget), ckpt+adapter saved.
2026-06-30 03:11:00 +00:00
bf95ac18e4
feat(m3): stage 1 alignment training script + smoke (T3.2)
...
- src/tsmm/train/stage1.py: streams align.jsonl, freeze_llm, trains
Encoder+Projector (AdamW lr=1e-4), bs=8/grad_accum=4/ctx=512/BF16,
gradient checkpointing + enable_input_require_grads, loss = lm + lambda*InfoNCE
(original vs perturbed TS representation), ckpt + tensorboard every 2000 steps.
- scripts/train_stage1.sh: wrapper with design defaults.
- src/tsmm/data/collator.py: handle JSON null (missing markers) -> NaN -> fill.
- .gitignore: /checkpoints/ and data/*.jsonl.
- Smoke verified: --max_steps 5 on 64 samples, finite loss, peak 5.52 GB (≤6GB).
- 100 tests passing.
2026-06-30 02:22:32 +00:00
9cfda37275
feat(m1): offline generation pipeline + CLI (T1.5)
...
- tsmm.data.gen_pipeline.gen_one_sample: composes attrs/series/anomaly/event/missing/instruct
into one JSON-serializable sample (full schema)
- write_jsonl helper
- scripts/gen_synthetic.py: multiprocessing + tqdm CLI (--n/--out/--seed/--workers)
- regression: build_instruction('event') with empty events no longer raises (TDD RED first)
- tests/test_gen_pipeline.py (6 tests, RED->GREEN)
Exit criteria: python scripts/gen_synthetic.py --n 1000 -> 1000 samples, schema-OK,
all 6 categories balanced (~130-190 each).
2026-06-29 23:29:04 +08:00
748bf41c53
feat(m1): scaffold tsmm package (T1.1)
...
- pyproject.toml (src-layout, deps per plan)
- src/tsmm/{data,model,train,eval} skeletons
- tests/test_smoke.py (RED->GREEN: importable + subpackages)
- configs/, scripts/, data/, checkpoints/ + .gitignore
- project venv (.venv) for reproducible env (M1 deps only; CUDA torch @ M2)
Verifications: import tsmm OK; pytest tests/ 3 passed.
2026-06-29 23:21:47 +08:00
张宗平