Fine-Tuning "2023-10-29-mace-16M-pbenner-mptrj-no-conditional-loss.model" with a Subset of Elements

[AkarisDimitry]

Hello,

I am currently working on a project where I need to fine-tune the "2023-10-29-mace-16M-pbenner-mptrj-no-conditional-loss.model" with my own dataset. My dataset is a subset of the original dataset used for this model, containing only a specific group of elements out of the 89 included in the original dataset.

Right now, I am not able to read the previously trained model and continue training with my data. How can this be done?

I am seeking advice or guidance on the best practices to approach this fine-tuning process. I have the following questions:

Data Preparation: Are there any specific preprocessing steps recommended for a dataset that only includes a subset of the original model's elements?

Model Adjustments: Given that my dataset includes fewer elements than the original model was trained on, are there any necessary modifications or considerations I should take into account before starting the fine-tuning?

Training Process: Could you provide any tips or recommendations on the fine-tuning procedure itself, such as learning rate adjustments, batch sizes, number of epochs, etc.?

Potential Challenges: Are there any common challenges or pitfalls I should be aware of when fine-tuning a model on a dataset that is different in scope from the original training data?

Any insights, resources, or examples you could provide would be immensely helpful.

Thank you in advance for your time and assistance.

Best regards,

[ilyes319]

If you go to the foundations branch in MACE, I made a way to fine tune a foundation model to your own dataset.
There is a new arg parser input called --foundation_model where you can put the path of the foundation model you want to fine tune. If you just do --foundation_model="use_mp" you will fine tune the material project model (no need to download anything). For now you will need to use the same settings as the foundation model for your model (probably there are ways to go beyond that). Here are the setting for the mp model. There might checkpointing problems so tell me.

python mace/cli/run_train.py ^
   --name="MACE_model" ^
   --model="ScaleShiftMACE" ^
   --hidden_irreps="64x0e + 64x1o + 64x2e" ^
   --foundation_model="use_mp" ^
   --foundation_model_readout ^
   --MLP_irreps="64x0e" ^
   --num_radial_basis=10 ^
   --num_cutoff_basis=10 ^
   --r_max=6.0 ^

The model will select the species that are contained in your dataset.
This functionality is has very little testing, so I don't have an answer to most of your questions. Please tell us if you figure out.

[AkarisDimitry]

I noticed that the mace/cli/run_train.py script in its current form does not support the arguments --foundation_model="use_mp" and --foundation_model_readout. These features might not be integrated yet or could be in a different version.

[davkovacs]

They are on the foundations branch

[jcwang587]

Just have a quick concern or suggestion regarding the plan for the fine-tune function. It seems the fine-tuned model is currently available only in the foundation branch. Will it be merged into the main branch in the near future? This feature could be quite helpful for usage.

[naoki-titech]

I agree with the comment from @jcwang587. Universal and fine-tuning branch are quite useful for users to train using multi-gpu and the pretrained model. It would be helpful if you could merge these two branch.

[turbosonics]

Hello, sorry for the unrelated reply, but do you have the development branch or fine-tuning branch? If yes, could you share me how to get and compile that version?