张宗平
1a096ff989
The original gen_synthetic.py held all N samples in a [None]*n list and only wrote at the end — three full-scale runs were killed mid-way (env reaps long sessions) and lost everything (0 bytes on disk each time). Fixes (proven on the 500k+100k run that just completed): - run() now streams: each completed future is written to disk the instant it's produced, via an O(workers) reorder buffer that keeps output in index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers). - --resume flag: counts existing lines and appends only the missing suffix (same seed => deterministic prefix, so resume is content-safe). - scripts/gen_full_supervisor.sh: loops 'run --resume -> check count' until target reached. Each surviving iteration adds >=1 line, so the run converges even across multiple kills (took 2 kills + 2 resumes to finish the 500k align). On this host: 2 partial runs (50k + 437k) then a final resume filled the last 62k. - Verified: order-deterministic (same-seed runs byte-identical), resume appends without rewriting the prefix, target-already-met => no-op. Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) + eval_synth 2k held-out (untouched).
tsmm
Time-series as a Modality — an experimental, single-GPU multimodal model that treats multivariate time series as an independent modality, jointly injected into a small LLM (Qwen2.5-0.5B) for time-series question answering and reasoning (ChatTS route).
Status
M1 (data pipeline) under construction. See
docs/superpowers/specs/2026-06-29-ts-as-modality-design.md for the design and
docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md for the task plan.
Layout
src/tsmm/ package (data, model, train, eval)
configs/ training / data configs
scripts/ offline generation & training entrypoints
tests/ pytest suite
data/ generated datasets (gitignored)
checkpoints/ model checkpoints (gitignored)
Install (dev)
python3 -m venv .venv
. .venv/bin/activate
pip install -e . # full deps (incl. torch/transformers/peft/...)
# M1-only lightweight install:
pip install numpy tqdm pyyaml pytest && pip install -e . --no-deps