张宗平
1a096ff989
The original gen_synthetic.py held all N samples in a [None]*n list and only wrote at the end — three full-scale runs were killed mid-way (env reaps long sessions) and lost everything (0 bytes on disk each time). Fixes (proven on the 500k+100k run that just completed): - run() now streams: each completed future is written to disk the instant it's produced, via an O(workers) reorder buffer that keeps output in index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers). - --resume flag: counts existing lines and appends only the missing suffix (same seed => deterministic prefix, so resume is content-safe). - scripts/gen_full_supervisor.sh: loops 'run --resume -> check count' until target reached. Each surviving iteration adds >=1 line, so the run converges even across multiple kills (took 2 kills + 2 resumes to finish the 500k align). On this host: 2 partial runs (50k + 437k) then a final resume filled the last 62k. - Verified: order-deterministic (same-seed runs byte-identical), resume appends without rewriting the prefix, target-already-met => no-op. Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) + eval_synth 2k held-out (untouched).
Scripts directory
Offline entrypoints: gen_synthetic.py, train_stage1.sh, train_stage2.sh, run_eval.sh.