Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High tolerance needed for perf model IN calibration tests #459

Open
amylu00 opened this issue Sep 18, 2024 · 1 comment
Open

High tolerance needed for perf model IN calibration tests #459

amylu00 opened this issue Sep 18, 2024 · 1 comment
Assignees

Comments

@amylu00
Copy link
Member

amylu00 commented Sep 18, 2024

Within the perfect model calibration tests in the ice_nucleation_calibration.jl file under test, each test is being run with different, arbitrary tolerances. Calibration might not be working as we want it because some values wouldn't pass the test if we set a reasonable relative tolerance.

For example:
ABDINM calibrated c coefficient is tested with an arbitrary absolute tolerance of 3. This is because the true value is around ~2, but because of the noise and standard deviation given to the priors, the calibrated value could be as high as 4. This is double the true value and requires around rtol = 0.9 which seems wrong. However, if we run the perfect_calib.jl file, we see that the resulting parcel output for ICNC matches fairly well with parcel outputs using the true coefficients.

Need to check with Ollie to see if this is okay. If not, find a way for these unit tests to pass with a relative tolerance of ~0.3.

@amylu00 amylu00 self-assigned this Sep 18, 2024
@trontrytel
Copy link
Member

All ice calibration tests are now running with 0.3 relative tolerance on the final number concentration. This results in similar tolerance for all free parameters but for the second parameter in ABDINM experiment. Is that accuracy acceptable, or should we expect better from the algorithm?

    if IN_mode == "ABDINM"
        TT.@test calibrated_parameters[1] ≈ coeff_true[1] rtol = FT(0.3)
        TT.@test calibrated_parameters[2] ≈ coeff_true[2] rtol = FT(1.5)
        TT.@test calibrated_soln[9, end] ≈ true_soln[9, end] rtol = FT(0.3)
    elseif IN_mode == "ABIFM"
        TT.@test calibrated_parameters[1] ≈ coeff_true[1] rtol = FT(0.3)
        TT.@test calibrated_parameters[2] ≈ coeff_true[2] rtol = FT(0.3)
        TT.@test calibrated_soln[9, end] ≈ true_soln[9, end] rtol = FT(0.3)
    elseif IN_mode == "ABHOM"
        TT.@test calibrated_parameters[1] ≈ coeff_true[1] rtol = FT(0.3)
        TT.@test calibrated_parameters[2] ≈ coeff_true[2] rtol = FT(0.3)
        TT.@test calibrated_soln[9, end] ≈ true_soln[9, end] rtol = FT(0.3)
    end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants