The missing-data test assumed SMD was absent, but after downloading real
SMD data for the cross-domain diagnostic it now loads successfully, so the
test no longer exercised the FileNotFoundError path. Switch to 'swat'
(another registered dataset whose files are absent) to keep the test valid
regardless of which real datasets happen to be present locally.
Full diagnostic chain (synth→real zero-shot→real finetune→held-out) shows
the TS-modality approach works in-distribution but fails to generalize
across entities: same SMD dataset, same 38 channels, just a different
machine (1→2) collapses output to repetition garbage. Root cause is the
lightweight TS encoder (2-layer TF, 2.1M params) learning dataset/machine-
specific patterns rather than transferable time-series representations.
This is a valuable negative result: it cleanly rules out the
'0.5B LLM + 2-layer TS encoder + single-distribution synth training'
route for general time-series understanding, which is more credible than
any 'looks like it works but never tested cross-domain' result.
Records:
- tasks.md: M6 diagnostic section (6.1-6.5) with evidence + archive decision
- proposal.md: Status header flagging archive + pointer to diagnosis
- docs/diagnosis-2026-07-02-cross-domain.md: full evidence chain, root-cause
analysis, capability boundaries, asset inventory, restart conditions
Project status: archived. Code/data/checkpoints/reports preserved.
Restart requires any of: (1) larger encoder + real-data from-scratch
training, (2) channel-agnostic patch strategy, (3) multi-dataset joint
training with held-out-dataset generalization proof.
VUS-PR 0.14→0.22 (+57%), Aff-F1 0.25 vs pure_llm 0.003 — confirms weak
localization was a data-volume issue, not algorithmic. Updated task notes
to reflect the improved (but still-open) localization frontier.
Full-scale retrain (stage1 31250 steps on 500k align, stage2 9375 steps on
100k sft) vs the prior small-scale run (5万/5k steps):
metric small-scale full-scale
stage2 EMA 0.81 0.75
model VUS-PR 0.14 0.22 (+57%)
model Aff-F1 0.21 0.25 (vs pure_llm 0.003 — TS modality helps)
model QA mean 2.45 2.41
QA vs pure_llm 2.8× 2.7× (stable lead, 6 categories)
Confirms the hypothesis that weak anomaly localization was mostly a
data-volume problem, not an algorithmic one: full-scale training lifted
VUS-PR 57% and the model now clearly beats the pure-LLM baseline on
affiliation-F1 (0.25 vs 0.003). VUS-PR still trails trivial baselines
(0.22 vs random 0.33) — precise localization remains the open frontier,
but the TS modality is demonstrably contributing (was unclear before).
Full-scale eval crashed in the pure-LLM QA-judge path: pure_llm emits
free-form JSON whose 'segments' list can contain entries like
["CPU Usage", 30] (channel name instead of an int). _seg_from_entry called
int() unconditionally and raised ValueError, killing the whole report run
(model predictor had already finished; result lost).
Fix: wrap _seg_from_entry in try/except (ValueError, TypeError) → return
None for any unparseable entry, so good segments alongside junk are still
extracted. Verified: {["CPU Usage",30],[10,20]} → [(10,20)]. + unit test.
Also archive logs/train_full_run.log (stage1 31250 EMA1.26, stage2 9375 EMA0.75).
Critical supervisor bug: completion check + --resume both used a glob that
only matched '{stage}_step*.pt', so a finished '{stage}_final.pt' (whose
step lives inside the file, not the filename) was invisible. Symptom: after
stage1 hit 31250 and saved final.pt, the supervisor re-launched --resume,
which picked step8000 (oldest sort-order quirk) and started OVERWRITING the
just-finished run from step 8000. Caught mid-overwrite; final.pt survived.
Fix: _latest_step (both stage1 & stage2) now globs step*.pt AND final.pt,
reads each one's 'step' field, returns the true highest. Supervisor's
run_until uses the same helper + reads step from the chosen ckpt to decide
completion. Verified: stage1(31250) now correctly reports 'done' and the
supervisor skips straight to stage2.
Also corrected supervisor stage1 config: bs=4 ga=8 (not bs=8 ga=4) — the
latter OOMs on long-sequence batches (logits 4.6GB alloc). expandable_segments
set to reduce fragmentation.
Full-scale retrain completed: stage1 31250 steps EMA 1.26, stage2 9375
steps EMA 0.75 (beats small-scale avg 0.81).
Addresses both points from review:
1) Resume (so a killed run converges across multiple restarts, like gen_supervisor):
- checkpoints now carry optimizer state + EMA value in a 'train_state' field
- --resume auto-picks the latest step ckpt and continues; --resume_from for explicit
- stage2 resume rebuilds the LoRA structure via enable_lora (same path as fresh)
then overlays saved adapter weights via set_peft_model_state_dict. Using
PeftModel.from_pretrained instead reset requires_grad and shrank the trainable
set 129->33 tensors, corrupting the optimizer state — verified and fixed.
- Smoke-verified: stage1 step4->10, stage2 step4->8, both restore optimizer +
EMA + LoRA with no state mismatch.
2) EMA loss logging (honest answer to 'is the loss swing a real problem?'):
- the old log printed the raw per-micro-batch loss every N steps; on
heterogeneous 6-task data that swings 2-3x purely from sampling (anomaly
JSON is low-loss, forecast numbers high-loss), so it looked 'unstable'
while the model may well have been converging.
- EMA(alpha=0.05) now logged alongside raw + as a tensorboard scalar
(total_loss_ema / sft_loss_ema). Read the EMA, not the raw, to judge
convergence. lr left unchanged (1e-4) pending EMA evidence.
Adds scripts/train_full_supervisor.sh: stage1->stage2 resume loop, same
crash-convergence guarantee as gen_full_supervisor.
The original gen_synthetic.py held all N samples in a [None]*n list and
only wrote at the end — three full-scale runs were killed mid-way (env
reaps long sessions) and lost everything (0 bytes on disk each time).
Fixes (proven on the 500k+100k run that just completed):
- run() now streams: each completed future is written to disk the instant
it's produced, via an O(workers) reorder buffer that keeps output in
index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers).
- --resume flag: counts existing lines and appends only the missing
suffix (same seed => deterministic prefix, so resume is content-safe).
- scripts/gen_full_supervisor.sh: loops 'run --resume -> check count'
until target reached. Each surviving iteration adds >=1 line, so the
run converges even across multiple kills (took 2 kills + 2 resumes to
finish the 500k align). On this host: 2 partial runs (50k + 437k) then
a final resume filled the last 62k.
- Verified: order-deterministic (same-seed runs byte-identical), resume
appends without rewriting the prefix, target-already-met => no-op.
Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) +
eval_synth 2k held-out (untouched).
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
- eval/instruct_check.py (T4.2): per-category (6 types) parse_rate +
rule_acc + stub-judge mean + usability; pass bars anomaly-JSON>80%,
overall usability>70%.
- eval/report.py: wire eval_qa_task across all categories so the report
now covers design §6.5 问答均分 (QA judge 0-5 per category per predictor),
not just anomaly-detection metrics. Fix latent bug: describe rule judge
expects a dict ref, so parse the JSON ref_answer before passing.
- Verified trivial-only smoke renders both tables; all_report VUS-PR=0.26
(not inflated) vs PA-F1~1.0 confirms PA is control-only.
Root cause (found during 50k sample generation at seed 6975, surfacing as
'ValueError: Number of samples, -187, must be non-negative' from linspace in
_inject_drift): _random_window's best-effort fallback after 50 failed
non-overlap attempts drew start and end from TWO INDEPENDENT random integers,
so end could be < start. The primary loop was consistent (start, start+length).
Fix: reuse the same start in the fallback so end >= start always holds
(overlap is still allowed in best-effort, just no inversion).
- tests/test_synthesis.py: regression sweeping 300 seeds × 4 type combos,
asserting every segment's end >= start. 146 tests passing.
- src/tsmm/train/losses.py:
* lm_loss(logits, labels): shifted masked CE (answer span only); clean 0.0
when all positions masked.
* infonce_contrastive_loss(z1, z2): symmetric CLIP-style InfoNCE in TS-
embedding space; positive = original vs light-augmentation view.
- tests/test_losses.py: 9 tests. 100 tests passing.
Spec clarification (small tier): InfoNCE operates in TS-embedding space
(encoder/projector output) rather than answer-embedding space; light
augmentations are positives. T3.3 strong-perturbation sensitivity is a
separate downstream check.
张宗平