张宗平
f509d47878
- src/tsmm/model/multimodal.py: MultimodalSplicer — binds LLM input embedding layer, assembles text embeds (via tokenizer) + TS token embeds (from Projector) into inputs_embeds/attention_mask/labels; labels = -100 everywhere except answer+EOS (training); left-truncation to max_len=1024. - tests/test_multimodal.py: 8 tests against real Qwen2.5-0.5B tokenizer + stub embedding (shapes, all-ones mask, answer-only labels, TS token count in sequence, no-NaN, truncation). 74 tests passing. - configs/llm.yaml: offline model path (downloaded via proxy to ~/models/Qwen2.5-0.5B-Instruct; HF hub unreachable on host). Spec clarifications (small tier, comet-build Step 4): - Training appends answer + EOS to the prompt layout. - Build embedding layer with len(tokenizer) (incl. added special tokens), not tokenizer.vocab_size which excludes them (eos=151645).
11 lines
442 B
YAML
11 lines
442 B
YAML
# LLM backbone config (T2.3 / T2.4)
|
|
# Design ref: docs/superpowers/specs/2026-06-29-ts-as-modality-design.md §2.2
|
|
# Qwen2.5-0.5B-Instruct: hidden=896, 24 layers, vocab=151936, BF16 ~1GB
|
|
|
|
# Local offline path (downloaded via proxy; HF hub unreachable on this host)
|
|
model_path: /home/zhangzp/models/Qwen2.5-0.5B-Instruct
|
|
hf_model_id: Qwen/Qwen2.5-0.5B-Instruct # for reference; offline use model_path
|
|
|
|
hidden_size: 896
|
|
torch_dtype: bfloat16
|