set up resuming for failed runs

[iejMac]

something like rom suggested where you generate input shard first then you just test if the output shard for that input shard already exists

[iejMac]

• some additional json for if the shard is complete or not

[iejMac]

make it like in video2dataset using input_sharder and all

[iejMac]

or we can hack it for now: when you're done writing a shard, write a key_stat.json file as a signal for that shard being done

[iejMac]

based on found .json files in dataset folder update this

clip-video-encode/clip_video_encode/clip_video_encode.py

Line 113 in 44f2214

starting_shard_id += math.ceil(work_size / shard_sample_count) * global_rank