Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virgo HDF5 file format #240

Draft
wants to merge 24 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3f7282f
update virgo generated dataset to use hdf5 format
jarlsondre Nov 6, 2024
9a39414
add functionality for selecting output location
jarlsondre Nov 6, 2024
865eda9
set new data format as standard
jarlsondre Nov 6, 2024
01ae8c0
make virgo work with new data loader and add progress bar
jarlsondre Nov 7, 2024
31fdbd2
fix merge conflict
jarlsondre Nov 7, 2024
3e674e6
remove old generation files and add script for concatenating hdf5 files
jarlsondre Nov 7, 2024
4bc8cd6
remove old generation files and add script for concatenating hdf5 files
jarlsondre Nov 7, 2024
ec2e589
Merge branch 'virgo-hdf5' of github.com:interTwin-eu/itwinai into vir…
jarlsondre Nov 7, 2024
35d86ab
rename folder using hyphens
jarlsondre Nov 7, 2024
4dc1dea
remove multiprocessing
jarlsondre Nov 7, 2024
812696f
add multiprocessing at correct place
jarlsondre Nov 7, 2024
c039052
update handling of seed and num processes
jarlsondre Nov 8, 2024
7687be9
Gpu monitoring (#237)
jarlsondre Nov 7, 2024
322acbc
Scalability test wall clock (#239)
jarlsondre Nov 7, 2024
1d1343f
make virgo work with new data loader and add progress bar
jarlsondre Nov 7, 2024
060e572
add contributors
jarlsondre Nov 8, 2024
8c18316
update ruff settings in pyproject
jarlsondre Nov 8, 2024
bed5249
update virgo dataset concatenation
jarlsondre Nov 8, 2024
b7e72a7
fix merge
jarlsondre Nov 8, 2024
46b8c96
add isort option to ruff
jarlsondre Nov 8, 2024
933fce0
break imports on purpose
jarlsondre Nov 8, 2024
c094bb6
break more imports to test
jarlsondre Nov 8, 2024
0017e0d
remove ruff config file
jarlsondre Nov 8, 2024
85d855d
😀
jarlsondre Nov 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion use-cases/virgo/config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
root_folder: /p/scratch/intertwin/datasets/virgo
hdf5_file_location: /p/project1/intertwin/saether1/itwinai/use-cases/virgo/data/virgo_data.hdf5
hdf5_dataset_name: virgo_dataset
chunk_size: 5
jarlsondre marked this conversation as resolved.
Show resolved Hide resolved
data_root: ./data
epochs: 2
epochs: 10
batch_size: 3
learning_rate: 0.0001
strategy: ddp
Expand All @@ -17,6 +20,9 @@ training_pipeline:
validation_proportion: 0.1
rnd_seed: 42
root_folder: ${root_folder}
hdf5_file_location: ${hdf5_file_location}
chunk_size: ${chunk_size}
jarlsondre marked this conversation as resolved.
Show resolved Hide resolved
hdf5_dataset_name: ${hdf5_dataset_name}
- class_path: trainer.NoiseGeneratorTrainer
init_args:
config:
jarlsondre marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
Loading