Train/Val MAE for MACE on MPTraj

[DeNeutoy]

Hello! Awesome model. We've been evaluating it and are finding it to be quite robust. We'd love to be able to compare some metrics for the training runs you did for the released models, specifically:

• raw energy MAE eV/atom for whatever train/validation splits you were using for MPTraj
• raw forces MAE eV/angs for the train/validation splits
• The MP ids for the validation split you used, so we could compare exactly (I understand if this is not available; the general trend throughout the papers is evaluating on OOD domains, for which I applaud you!)

I looked at:

• https://arxiv.org/abs/2401.00096
• https://arxiv.org/abs/2308.14920

which I think are the two papers which have MACE models trained on MPTraj. Please let me know if I should look elsewhere!

We're also finding the training process on MPTraj fairly unstable - was this also your experience? I am guessing this is the motivation for the Huber Loss + conditional huber delta parameter for the forces, basically due to this graph (otherwise the gigantic forces wreck training at random points):

Would that be a correct characterization?

Thanks again for the excellent paper and code!

[ilyes319]

Hey @DeNeutoy,

Happy that you liked our work!

We have put the training input scripts for the small, medium, and large models here: https://github.com/ACEsuit/mace-mp/tree/main/mace_mp_0. Hopefully, that will help you with your runs. To use these scripts, you will need to run with the universal branch.

You can find the validation Energy and Forces RMSE here: https://docs.google.com/spreadsheets/d/1NzftwWFR8n1AK9ypDW35H3PtzFabq72e2FbUv0bkSg8/edit#gid=0.

The extract training and validation can be downloaded at these links in hdf5 format for our distributed dataloader (see the in the universal branch).

In the input script, you will notice the "universal" loss function that was used to stabilize the training. We indeed see that the training process with a normal MSE loss is quite noisy. This is partially due to the presence of very high forces. The loss was designed to enable the training to focus on accuracy in the most important part of phase space, while still learning good high forces.

[ilyes319]

@DeNeutoy I made it a discussion,.

[Yasir-Ghunaim]

Hi @ilyes319,
As a follow up to this discussion, I'm also trying to run the training code of the foundation model. The scripts in mace_mp_0 include this argument:
--statistics_file="../../dataset/mptrj-gga-ggapu-statistics.json"
However, I cannot find the statistics file anywhere in the repo. Can you please share the statistics? The code in the "universal" branch doesn't seem to run without this statistics file.

Thanks a lot.

[ilyes319]

Hey, it is here: https://drive.google.com/file/d/1F-izb02huo19mByJ0u8yujdvEKDK4lFT/view?usp=drive_link

[Yasir-Ghunaim]

Amazing, thanks a lot

[chouyoudou]

Hi @ilyes319 ,
I found that the latest releases can also use the 'universal' tag, but I'm curious what the purpose of this statistics file is?