Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More robust parameter file handling #346

Draft
wants to merge 35 commits into
base: master
Choose a base branch
from

Conversation

ozmorph
Copy link
Contributor

@ozmorph ozmorph commented May 28, 2020

Part 2 of #74. This pull is related to my other PR #228 which deals with the CLI parameter parsing aspects of #74. You may note that the raw_extract() number extraction code from ParamFile.hpp looks similar to the parse_number() function in PR #228. Eventually these two implementations could be combined into a separate file, but I think they can exist independently for now.

The main goals of this Pull-Request are:

  • A more clean and concise way of extracting parameter values (fewer 160+ character lines)
  • A type-safe approach towards extracting values
    • Logging when a value overflows/underflows the storage size of the output variable
    • Replacing the use of fscanf() and sprintf() with std::ifstream and std::istringstream.
  • Logging when the actual number of parameters extracted doesn't match the expected
  • Logging when a parameter value isn't specified and a default value is instead used
  • Exiting when a required parameter value isn't found

Things that are missing from this draft PR right now (I will update as I make more commits):

  • Wildcard CLI parameters (e.g. specifying #1 in a param file and using /CLP1 value from the CLI)
  • Extracting string values
  • Extracting matrix values (values that span more than 1 line)
  • Extracting values into a proportion array where the default value is 1 at index 0 and 0 in every other place.
  • Extracting values into an array with a default array specified (e.g. output[4] can have default{ 1, 2, 3, 4})
  • Extracting ICDF values. I'm currently watching Converted Param.h icdf arrays from double array to InverseCdf class #316
  • Outputting a parameter file with all merged and de-duplicated values

I will continue to work on this PR, but please let me know if there are any design decisions you would like to discuss or changes you would like to be made!

Cheers!

@ozmorph
Copy link
Contributor Author

ozmorph commented May 28, 2020

It appears that old ABI of the libstdc++ used in the Docker CentOS 7 container doesn't allow for the C++11 function std::string::erase(const_iterator) to be used, which is why that CI check is failing.

Either -DGLIBCXX_USE_CXX11_ABI=1 needs to be passed to the command-line for that build, or an alternative should be found and used.

References:

GCC Bug
StackOverflow

@ozmorph ozmorph force-pushed the robust_parameter_file_handling branch 2 times, most recently from 9fdbebb to 3922765 Compare June 4, 2020 16:11
@ozmorph ozmorph force-pushed the robust_parameter_file_handling branch 2 times, most recently from 77cc98c to 1ad9897 Compare June 15, 2020 15:47
@ozmorph ozmorph force-pushed the robust_parameter_file_handling branch from 1ad9897 to df5466d Compare June 18, 2020 19:46
@ozmorph ozmorph force-pushed the robust_parameter_file_handling branch 2 times, most recently from a399414 to d68eebd Compare June 22, 2020 17:34
// https://stackoverflow.com/a/217605

// trim from start (in place)
static inline void ltrim(std::string& s) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for inline here - its only a hint to the compiler - which will ignore it.

@matt-gretton-dann
Copy link
Collaborator

Things I would like to see in this PR:

  • Detection and warning of when we do have multiple values for a field
  • Detection and warning of when a parameter is specified in a config file but not read.
  • A new command-line option that enables output of a parameter file that when used as the input will result in the same config.

@ozmorph ozmorph force-pushed the robust_parameter_file_handling branch 3 times, most recently from 76dba63 to c8f7142 Compare July 1, 2020 15:26
…eam reader for parameter files; this class will eventually be used to phase out the GetInputParameter*() functions
…r2() with ParamReader's extract_or_exit() or extract() functions respectively; currently doesn't pass regression tests
…' argument name with 'default_value' to not use reserved keyword
…om an earlier file; this commit resolves the regression checksum mismatch error
…mFile.hpp; changed all simple invocations of GetInputParameter and GetInputParameter2 that were reading in multiple values
…_exit() functions to ParamFile.(h/c)pp; converted several locations in CovidSim.cpp to use these new functions which required changing fields in Model.h to use std::string instead of char arrays
…ser to conditionally extract values from the param files or choose the default value
… allow its functionality to be shared with ParamFile.cpp; updated build files accordingly
…xit() functions for integers and doubles in Parsers; replaced template code in ParamFile with more explicit function names for extracting types from the param files; updated CovidSim.cpp to use these new functions; updated CLI.cpp to use the new functions
…th an iostream call and return false for functions that aren't supposed to exit
…rom CovidSim.cpp to InverseCdf.h; removed GetInverseCdf() from CovidSim.cpp
…ed so that fields don't need to be individually initialized in ReadParams()
…boolean values from parameter files (leading to the potential of one day specifying "true" or "false" instead of "1" and "0" respectively)
…at act as flags into bool so that the intent is more clear; updated CovidSim.cpp to use the new extract_bool() function
@ozmorph ozmorph force-pushed the robust_parameter_file_handling branch from ca92b50 to 4cd99a8 Compare July 13, 2020 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants