The original gen_synthetic.py held all N samples in a [None]*n list and
only wrote at the end — three full-scale runs were killed mid-way (env
reaps long sessions) and lost everything (0 bytes on disk each time).
Fixes (proven on the 500k+100k run that just completed):
- run() now streams: each completed future is written to disk the instant
it's produced, via an O(workers) reorder buffer that keeps output in
index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers).
- --resume flag: counts existing lines and appends only the missing
suffix (same seed => deterministic prefix, so resume is content-safe).
- scripts/gen_full_supervisor.sh: loops 'run --resume -> check count'
until target reached. Each surviving iteration adds >=1 line, so the
run converges even across multiple kills (took 2 kills + 2 resumes to
finish the 500k align). On this host: 2 partial runs (50k + 437k) then
a final resume filled the last 62k.
- Verified: order-deterministic (same-seed runs byte-identical), resume
appends without rewriting the prefix, target-already-met => no-op.
Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) +
eval_synth 2k held-out (untouched).
张宗平