张宗平
1a096ff989
The original gen_synthetic.py held all N samples in a [None]*n list and only wrote at the end — three full-scale runs were killed mid-way (env reaps long sessions) and lost everything (0 bytes on disk each time). Fixes (proven on the 500k+100k run that just completed): - run() now streams: each completed future is written to disk the instant it's produced, via an O(workers) reorder buffer that keeps output in index order. Peak RAM drops from O(n) (~2.7GB for 50k) to O(workers). - --resume flag: counts existing lines and appends only the missing suffix (same seed => deterministic prefix, so resume is content-safe). - scripts/gen_full_supervisor.sh: loops 'run --resume -> check count' until target reached. Each surviving iteration adds >=1 line, so the run converges even across multiple kills (took 2 kills + 2 resumes to finish the 500k align). On this host: 2 partial runs (50k + 437k) then a final resume filled the last 62k. - Verified: order-deterministic (same-seed runs byte-identical), resume appends without rewriting the prefix, target-already-met => no-op. Full-scale data now on disk: align 500k (27G) + sft 100k (5.4G) + eval_synth 2k held-out (untouched).
40 lines
1.7 KiB
Bash
Executable File
40 lines
1.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Crash-resilient full-scale generation supervisor.
|
|
#
|
|
# gen_synthetic.py now (a) streams each sample to disk the instant it's produced
|
|
# (O(workers) memory, not O(n)) and (b) supports --resume: it counts existing
|
|
# lines and only generates+appends the missing suffix. So a kill mid-run keeps
|
|
# all progress on disk; re-running --resume continues from where it stopped.
|
|
#
|
|
# This supervisor loops "run --resume -> check count" until the target line
|
|
# count is reached. If the supervisor itself is killed, the data already on
|
|
# disk survives; relaunch this script and it picks up. Convergence is guaranteed:
|
|
# every iteration that survives adds >= 1 line.
|
|
#
|
|
# Usage: scripts/gen_full_supervisor.sh
|
|
set -uo pipefail # NB: no -e — a killed gen sub-call must NOT abort the loop
|
|
cd "$(dirname "$0")/.."
|
|
export PYTHONPATH=src
|
|
W=$(nproc)
|
|
|
|
gen() { # target_n out seed [extra flags...]
|
|
local target=$1 out=$2 seed=$3; shift 3
|
|
while :; do
|
|
local cur=0
|
|
if [ -f "$out" ]; then cur=$(wc -l < "$out"); fi
|
|
if [ "$cur" -ge "$target" ]; then
|
|
echo "[sup] $out done: $cur/$target lines"; return 0
|
|
fi
|
|
echo "[sup] $out at $cur/$target — launching --resume $(date '+%H:%M:%S')"
|
|
.venv/bin/python scripts/gen_synthetic.py \
|
|
--n "$target" --out "$out" --seed "$seed" --workers "$W" --resume "$@"
|
|
# loop: re-check count (gen may have been killed or completed)
|
|
done
|
|
}
|
|
|
|
echo "[sup] === START $(date '+%H:%M:%S') ==="
|
|
gen 500000 data/align.jsonl 0
|
|
gen 100000 data/sft.jsonl 0 --evolve
|
|
echo "[sup] === ALL DONE $(date '+%H:%M:%S') ==="
|
|
echo "[sup] align=$(wc -l < data/align.jsonl) sft=$(wc -l < data/sft.jsonl) eval=$(wc -l < data/eval_synth.jsonl)"
|