1a096ff989
feat(gen): streaming + crash-resilient resume for full-scale data gen
...
The original gen_synthetic.py held all N samples in a [None]*n list and
only wrote at the end — three full-scale runs were killed mid-way (env
reaps long sessions) and lost everything (0 bytes on disk each time).
Fixes (proven on the 500k+100k run that just completed):
- run() now streams: each completed future is written to disk the instant
it's produced, via an O(workers) reorder buffer that keeps output in
index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers).
- --resume flag: counts existing lines and appends only the missing
suffix (same seed => deterministic prefix, so resume is content-safe).
- scripts/gen_full_supervisor.sh: loops 'run --resume -> check count'
until target reached. Each surviving iteration adds >=1 line, so the
run converges even across multiple kills (took 2 kills + 2 resumes to
finish the 500k align). On this host: 2 partial runs (50k + 437k) then
a final resume filled the last 62k.
- Verified: order-deterministic (same-seed runs byte-identical), resume
appends without rewriting the prefix, target-already-met => no-op.
Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) +
eval_synth 2k held-out (untouched).
2026-06-30 23:48:41 +00:00
9b9aa35f13
docs(ts-as-modality): mark M4 (T4.2/T4.3) + M5 (T5.6) done with results
...
- T4.2/T4.3 (M4): anomaly JSON parse 1.00, usability 0.92; stage2 ckpt 0.81 loss, 10.03GB.
- T5.6 (M5): ablation change_rate 0.98 ✅ , QA mean 2.45 vs pure_llm 0.88 ✅ ,
reproducible w/ trivial ✅ ; VUS-PR below baseline (honest negative — anomaly
localization is the weak point, documented with mitigation directions).
- All M1-M5 tasks now closed; real-benchmark download deferred.
2026-06-30 10:03:35 +00:00
40ec960cf8
fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline
...
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
2026-06-30 10:02:59 +00:00
8aeddebce9
test(m4): instruct_check unit tests + unbuffered training scripts
...
- tests/test_instruct_check.py: cover load_by_category (bucketing/cap/seed
determinism/short-pool/blank-line) and _forecast_list (nested/flat/
dict/garbage). 8 tests pass.
- train_stage{1,2}.sh: export PYTHONPATH=src + PYTHONUNBUFFERED=1 so future
background runs stream logs live (current stage2 run is block-buffered).
2026-06-30 07:27:03 +00:00
76fd636fae
feat(m4/m5): T4.2 instruct-check + QA judge table in eval report
...
- eval/instruct_check.py (T4.2): per-category (6 types) parse_rate +
rule_acc + stub-judge mean + usability; pass bars anomaly-JSON>80%,
overall usability>70%.
- eval/report.py: wire eval_qa_task across all categories so the report
now covers design §6.5 问答均分 (QA judge 0-5 per category per predictor),
not just anomaly-detection metrics. Fix latent bug: describe rule judge
expects a dict ref, so parse the JSON ref_answer before passing.
- Verified trivial-only smoke renders both tables; all_report VUS-PR=0.26
(not inflated) vs PA-F1~1.0 confirms PA is control-only.
2026-06-30 07:24:55 +00:00
aecccfab1d
feat(m3): TS-token effectiveness check (T3.3) — change_rate=0.98/50 ≫ 0.5 target
...
- eval/ts_effectiveness.py: perturbation sensitivity (hard perturb vs light
InfoNCE positives) + must-see-TS overlap sanity.
- Measured on stage1_final.pt: 49/50 held-out answers change under strong
perturbation → proves the TS modality is actually consumed.
- Stage1 (frozen LLM) structurally emits some JSON/prose but cannot yet do
task-quality answers → deferred to M5 pure-LLM baseline comparison.
- Mark T3.3/T3.4 (M3 exit) done.
2026-06-30 07:14:22 +00:00
0723b257fd
fix(m1): _random_window fallback could return end < start
...
Root cause (found during 50k sample generation at seed 6975, surfacing as
'ValueError: Number of samples, -187, must be non-negative' from linspace in
_inject_drift): _random_window's best-effort fallback after 50 failed
non-overlap attempts drew start and end from TWO INDEPENDENT random integers,
so end could be < start. The primary loop was consistent (start, start+length).
Fix: reuse the same start in the fallback so end >= start always holds
(overlap is still allowed in best-effort, just no inversion).
- tests/test_synthesis.py: regression sweeping 300 seeds × 4 type combos,
asserting every segment's end >= start. 146 tests passing.
2026-06-30 03:22:18 +00:00
4c4a8f6435
feat(m5): full eval pipeline — parse/metrics/judge/baselines/report (T5.1-T5.5)
...
- src/tsmm/eval/parse_answer.py: text→JSON segments→point scores (JSON-first,
regex fallback, all-zero on fail).
- src/tsmm/eval/ts_metrics.py: VUS-PR (main, threshold-free; constant-score →
prevalence, NOT inflated), AUC-PR, Point-F1 (no PA), PA-F1 (control only),
self-contained Affiliation-F1.
- src/tsmm/eval/qa_judge.py: RuleJudge (anomaly IoU / describe+forecast numeric
tolerance) + LLMJudge (stub backend = deterministic offline heuristic for the
no-network host; openai backend = GPT-4o 0-5; pluggable callable).
- src/tsmm/eval/baselines.py: trivial (Random/Constant/AllReport) + ts_to_text
+ pure_llm_answer glue; Time-LLM/ChatTS marked not-reproduced (honest).
- src/tsmm/eval/report.py + scripts/run_eval.sh: trained model + all baselines
through the SAME pipeline → reports/eval-YYYYMMDD.md (VUS-PR main, PA-F1
explicitly labelled control, trivial baselines included).
- tests: 45 new (parse 13, metrics 12, judge 12, baselines 8). 145 passing.
- Smoke: run_eval on 40 samples produces valid markdown; PA-F1≈1.0 vs
VUS-PR≈0.35 for all_report demonstrates why PA must not be the headline.
2026-06-30 03:18:14 +00:00
357e2e60bd
feat(m4): stage 2 SFT training script (LoRA r=16) + smoke (T4.1)
...
- src/tsmm/train/stage2.py: loads stage1 ckpt (optional), freeze_llm +
enable_lora(r=16) on q/v_proj, masked LM loss on sft.jsonl, bs=4/grad_accum=8/
ctx=1024/BF16/gradient checkpointing, ckpt every 2000 steps (encoder+projector
state + peft LoRA adapter via save_pretrained).
- scripts/train_stage2.sh: wrapper with design defaults.
- Smoke verified: --max_steps 4, LoRA injected, finite loss, peak 3.29 GB
(≤11GB budget), ckpt+adapter saved.
2026-06-30 03:11:00 +00:00
8286b8276c
chore(ts-as-modality): build checkpoint — M2 done + M3 code done, training decision pending
2026-06-30 02:23:27 +00:00
bf95ac18e4
feat(m3): stage 1 alignment training script + smoke (T3.2)
...
- src/tsmm/train/stage1.py: streams align.jsonl, freeze_llm, trains
Encoder+Projector (AdamW lr=1e-4), bs=8/grad_accum=4/ctx=512/BF16,
gradient checkpointing + enable_input_require_grads, loss = lm + lambda*InfoNCE
(original vs perturbed TS representation), ckpt + tensorboard every 2000 steps.
- scripts/train_stage1.sh: wrapper with design defaults.
- src/tsmm/data/collator.py: handle JSON null (missing markers) -> NaN -> fill.
- .gitignore: /checkpoints/ and data/*.jsonl.
- Smoke verified: --max_steps 5 on 64 samples, finite loss, peak 5.52 GB (≤6GB).
- 100 tests passing.
2026-06-30 02:22:32 +00:00
204d5237ba
feat(m3): losses — masked LM CE + symmetric InfoNCE (T3.1)
...
- src/tsmm/train/losses.py:
* lm_loss(logits, labels): shifted masked CE (answer span only); clean 0.0
when all positions masked.
* infonce_contrastive_loss(z1, z2): symmetric CLIP-style InfoNCE in TS-
embedding space; positive = original vs light-augmentation view.
- tests/test_losses.py: 9 tests. 100 tests passing.
Spec clarification (small tier): InfoNCE operates in TS-embedding space
(encoder/projector output) rather than answer-embedding space; light
augmentations are positives. T3.3 strong-perturbation sensitivity is a
separate downstream check.
2026-06-30 02:19:02 +00:00
133bbac316
chore(ts-as-modality): M2-complete build checkpoint (T2.6 verified)
...
M2 exit verification:
- 8-step fwd+bwd on RTX 3060, loss 9.82→5.54 (descending), finite.
- Stage ① peak 2.34 GB (<5GB design budget).
- Updated build-checkpoint.json: M2-complete, 91 tests, env notes
(CUDA torch, ~/models/Qwen2.5-0.5B-Instruct, proxy).
2026-06-30 02:15:57 +00:00
7e1df76e3e
feat(m2): Collator — JSONL → wrapper raw batch (T2.5)
...
- src/tsmm/data/collator.py: Collator maps JSONL sample dicts to the wrapper
batch contract {series [B,max_T,max_C], attributes, timestamps, events,
question, answer}. Variable-T pad/truncate, variable-C pad, NaN→fill.
Variable-text padding/attention_mask deferred to the wrapper's per-sample
splice + max_seq padding.
- tests/test_collator.py: 11 tests. End-to-end Collator→Wrapper fwd+bwd+generate
verified (loss finite, peak 2.32 GB, generate emits text). 91 tests passing.
2026-06-30 02:14:47 +00:00
aec8fd5ef1
feat(m2): MultimodalTSModel wrapper — end-to-end fwd/generate + LoRA (T2.4)
...
- src/tsmm/model/wrapper.py: MultimodalTSModel combining TS Encoder + Projector
+ Qwen2.5-0.5B LLM + LoRA. Two modes: freeze_llm() (stage ①) and
enable_lora(r=16) (stage ②, peft on q/v_proj). forward() takes a raw batch
{series, attributes, timestamps, events, question, answer} end-to-end
(encoder→projector→splice→LLM) OR pre-spliced inputs_embeds.
- src/tsmm/model/multimodal.py: device-aware splice (move ids/ts_embeds to
the bound embedding layer's device).
- tests/test_wrapper.py: 6 tests — finite loss, grad flows into
encoder/projector under freeze_llm, generate returns text, freeze_llm and
enable_lora mode invariants, GPU memory budget. 80 tests passing.
- Measured: stage ① fwd+bwd peak 1.87 GB (design budget ~5 GB).
Spec clarifications (small tier, comet-build Step 4):
- Wrapper batch contract = {series, attributes, timestamps, events, question,
answer} (lists of str + series tensor). Collator (T2.5) will produce this.
- Encoder/Projector kept fp32; their output cast to LLM dtype (bf16) before
splice, so inputs_embeds matches LLM weights.
2026-06-30 02:11:36 +00:00
f509d47878
feat(m2): multimodal splice [属性][时间戳][TS tok][事件][问题][回答] (T2.3)
...
- src/tsmm/model/multimodal.py: MultimodalSplicer — binds LLM input
embedding layer, assembles text embeds (via tokenizer) + TS token embeds
(from Projector) into inputs_embeds/attention_mask/labels; labels = -100
everywhere except answer+EOS (training); left-truncation to max_len=1024.
- tests/test_multimodal.py: 8 tests against real Qwen2.5-0.5B tokenizer +
stub embedding (shapes, all-ones mask, answer-only labels, TS token count
in sequence, no-NaN, truncation). 74 tests passing.
- configs/llm.yaml: offline model path (downloaded via proxy to
~/models/Qwen2.5-0.5B-Instruct; HF hub unreachable on host).
Spec clarifications (small tier, comet-build Step 4):
- Training appends answer + EOS to the prompt layout.
- Build embedding layer with len(tokenizer) (incl. added special tokens),
not tokenizer.vocab_size which excludes them (eos=151645).
2026-06-30 02:05:15 +00:00
f20c2db803
feat(m2): Projector Linear(256→896)+LayerNorm (T2.2)
...
- src/tsmm/model/projector.py: Projector (Linear + LayerNorm).
- tests/test_projector.py: 5 tests (shape, Qwen hidden alignment,
LayerNorm present, gradient flow, no-NaN).
- 66 tests passing.
2026-06-30 00:17:47 +00:00
545802d62c
feat(m2): TS Encoder — Patchify + 2-layer Transformer (T2.1)
...
- src/tsmm/model/ts_encoder.py: Patchify (channel-independent unfold) +
TSEncoder (shared per-channel patch linear, mean-pool over channels,
learnable positional embedding, 2-layer pre-norm Transformer, LayerNorm).
- configs/ts_encoder.yaml: encoder/projector hyperparameters.
- tests/test_ts_encoder.py: 10 tests (shape, n_patches formula, window
values, determinism, gradient flow, param sanity).
- 61 tests passing; GPU fwd/bwd verified on RTX 3060 (39 MB peak).
Spec clarifications (small tier, comet-build Step 4):
- n_patches = (T-P)//S+1 ⇒ 127 for T=512 (design said 128; arithmetic slip).
- 'Channel-independent' = shared per-channel patch embed + mean-pool →
C-free output [B,n_patches,d]; channel identity carried by [属性] tokens.
- Actual params ≈2.1M (design's '≈4M' was a rough estimate).
2026-06-30 00:16:54 +00:00
98fb1bfc53
chore(ts-as-modality): M1-complete build checkpoint + resume note
...
Record resume state for build phase:
- M1 (T1.1-T1.7) complete; 51 tests pass; branch feature/20260629/ts-as-modality
- next: T2.1 (TS Encoder), blocked on M2 env decision (CPU vs CUDA torch)
- handoff/build-checkpoint.json captures blocker + resume instructions
2026-06-30 07:57:43 +08:00
bbc60f2cdd
test(m1): M1 exit verification (T1.7)
...
Gate results:
- pytest tests/ -> 51 passed
- load_windows: >=2 datasets load (toy + registry of 5 real; real raise cleanly when absent)
- 5 synthetic samples spot-checked: components vary, labels cover every segment,
events cause persistent per-channel mean shifts, ~5% NaN missing
M1 (data pipeline) complete.
2026-06-29 23:31:06 +08:00
722102b335
feat(m1): real benchmark loader (T1.6)
...
- SUPPORTED_DATASETS registry: smd/msl/smap/swat/psm (+toy)
- load_windows(name,T,stride): sliding windows + per-channel z-score norm
-> shapes (N,T,C) and (N,T)
- windows_to_qa: anomaly/describe QA pairs from windows (non-empty)
- real loaders read data/real/<name>/test.npy (+test_label.npy) or TSMM_DATA_DIR;
raise clear FileNotFoundError when absent (testable offline via 'toy')
- write_eval_jsonl helper for eval_real.jsonl
- tests/test_real_bench.py (7 tests, RED->GREEN)
Exit criteria: >=2 datasets load (toy + registry of 5 real); window/label shapes;
QA non-empty; missing real data raises clearly.
2026-06-29 23:30:30 +08:00
9cfda37275
feat(m1): offline generation pipeline + CLI (T1.5)
...
- tsmm.data.gen_pipeline.gen_one_sample: composes attrs/series/anomaly/event/missing/instruct
into one JSON-serializable sample (full schema)
- write_jsonl helper
- scripts/gen_synthetic.py: multiprocessing + tqdm CLI (--n/--out/--seed/--workers)
- regression: build_instruction('event') with empty events no longer raises (TDD RED first)
- tests/test_gen_pipeline.py (6 tests, RED->GREEN)
Exit criteria: python scripts/gen_synthetic.py --n 1000 -> 1000 samples, schema-OK,
all 6 categories balanced (~130-190 each).
2026-06-29 23:29:04 +08:00
1b4c6ddf4c
feat(m1): instruction & answer generation, 6 categories (T1.4)
...
- describe / anomaly / root_cause / forecast / compare / event
- JSON answers (stats, anomaly_regions aligned w/ labels, forecast), text for cause/event
- evolve_instruction: deterministic Evol-Instruct-style rephrasing
- tests/test_instruct.py (12 tests, RED->GREEN)
Exit criteria: anomaly JSON json.loads-able & regions reconstruct labels; full suite 37 passed.
2026-06-29 23:26:41 +08:00
e2c0756a78
feat(m1): synthesis — components + anomaly/event/missing (T1.3)
...
- generate_series: baseline + trend + seasonality + noise (opt heavy-tail)
- inject_anomaly: spike / level_shift / variance_change / missing_segment / drift
* non-overlapping segments (TDD caught an overlap bug) -> labels exactly
match contiguous [start,end) runs
* returns (series, point labels, segment metadata)
- couple_event: persistent per-channel step at t + event text
- add_missing: NaN drop at given rate
- tests/test_synthesis.py (14 tests, RED->GREEN; fixed overlap regression)
Exit criteria: seed-deterministic; labels cover segments; event step persistent.
2026-06-29 23:25:08 +08:00
6d10454484
feat(m1): attribute vocab + sampling (T1.2)
...
- ATTRIBUTE_VOCAB (16 AIOps channels w/ units)
- FREQ_BANDS (6 sampling resolutions)
- sample_attributes(n_channels, seed) -> unique-name, reproducible
- tests/test_attributes.py (8 tests, RED->GREEN)
Exit criteria: fields complete, count correct, seed-reproducible.
2026-06-29 23:22:40 +08:00
748bf41c53
feat(m1): scaffold tsmm package (T1.1)
...
- pyproject.toml (src-layout, deps per plan)
- src/tsmm/{data,model,train,eval} skeletons
- tests/test_smoke.py (RED->GREEN: importable + subpackages)
- configs/, scripts/, data/, checkpoints/ + .gitignore
- project venv (.venv) for reproducible env (M1 deps only; CUDA torch @ M2)
Verifications: import tsmm OK; pytest tests/ 3 passed.
2026-06-29 23:21:47 +08:00
c160718934
chore(ts-as-modality): scaffold OpenSpec change & adopt reviewed design
...
- OpenSpec change: proposal/design/tasks + 3 delta specs
(ts-synthesis, ts-multimodal-model, ts-evaluation; 21 requirements)
- comet state initialized (phase=build), plan recorded
- adopt existing reviewed design doc as Design Doc (+ comet frontmatter)
- existing implementation plan (+ comet frontmatter)
- workspace .gitignore
2026-06-29 22:29:31 +08:00
7a5a1d033f
init version
2026-06-29 21:16:52 +08:00
张宗平