张宗平
f509d47878
- src/tsmm/model/multimodal.py: MultimodalSplicer — binds LLM input embedding layer, assembles text embeds (via tokenizer) + TS token embeds (from Projector) into inputs_embeds/attention_mask/labels; labels = -100 everywhere except answer+EOS (training); left-truncation to max_len=1024. - tests/test_multimodal.py: 8 tests against real Qwen2.5-0.5B tokenizer + stub embedding (shapes, all-ones mask, answer-only labels, TS token count in sequence, no-NaN, truncation). 74 tests passing. - configs/llm.yaml: offline model path (downloaded via proxy to ~/models/Qwen2.5-0.5B-Instruct; HF hub unreachable on host). Spec clarifications (small tier, comet-build Step 4): - Training appends answer + EOS to the prompt layout. - Build embedding layer with len(tokenizer) (incl. added special tokens), not tokenizer.vocab_size which excludes them (eos=151645).
tsmm
Time-series as a Modality — an experimental, single-GPU multimodal model that treats multivariate time series as an independent modality, jointly injected into a small LLM (Qwen2.5-0.5B) for time-series question answering and reasoning (ChatTS route).
Status
M1 (data pipeline) under construction. See
docs/superpowers/specs/2026-06-29-ts-as-modality-design.md for the design and
docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md for the task plan.
Layout
src/tsmm/ package (data, model, train, eval)
configs/ training / data configs
scripts/ offline generation & training entrypoints
tests/ pytest suite
data/ generated datasets (gitignored)
checkpoints/ model checkpoints (gitignored)
Install (dev)
python3 -m venv .venv
. .venv/bin/activate
pip install -e . # full deps (incl. torch/transformers/peft/...)
# M1-only lightweight install:
pip install numpy tqdm pyyaml pytest && pip install -e . --no-deps