张宗平
aec8fd5ef1
- src/tsmm/model/wrapper.py: MultimodalTSModel combining TS Encoder + Projector
+ Qwen2.5-0.5B LLM + LoRA. Two modes: freeze_llm() (stage ①) and
enable_lora(r=16) (stage ②, peft on q/v_proj). forward() takes a raw batch
{series, attributes, timestamps, events, question, answer} end-to-end
(encoder→projector→splice→LLM) OR pre-spliced inputs_embeds.
- src/tsmm/model/multimodal.py: device-aware splice (move ids/ts_embeds to
the bound embedding layer's device).
- tests/test_wrapper.py: 6 tests — finite loss, grad flows into
encoder/projector under freeze_llm, generate returns text, freeze_llm and
enable_lora mode invariants, GPU memory budget. 80 tests passing.
- Measured: stage ① fwd+bwd peak 1.87 GB (design budget ~5 GB).
Spec clarifications (small tier, comet-build Step 4):
- Wrapper batch contract = {series, attributes, timestamps, events, question,
answer} (lists of str + series tensor). Collator (T2.5) will produce this.
- Encoder/Projector kept fp32; their output cast to LLM dtype (bf16) before
splice, so inputs_embeds matches LLM weights.
tsmm
Time-series as a Modality — an experimental, single-GPU multimodal model that treats multivariate time series as an independent modality, jointly injected into a small LLM (Qwen2.5-0.5B) for time-series question answering and reasoning (ChatTS route).
Status
M1 (data pipeline) under construction. See
docs/superpowers/specs/2026-06-29-ts-as-modality-design.md for the design and
docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md for the task plan.
Layout
src/tsmm/ package (data, model, train, eval)
configs/ training / data configs
scripts/ offline generation & training entrypoints
tests/ pytest suite
data/ generated datasets (gitignored)
checkpoints/ model checkpoints (gitignored)
Install (dev)
python3 -m venv .venv
. .venv/bin/activate
pip install -e . # full deps (incl. torch/transformers/peft/...)
# M1-only lightweight install:
pip install numpy tqdm pyyaml pytest && pip install -e . --no-deps