pythonizing regression

Pythonizing: Regression

There exist two workflows for regression testing, interactive mode and global mode.

Intent

The intent is that Chapter Committees run this automated regression tool on their chapters and commit the resulting approvals.json file to the pull request branch. When we have a full/complete approvals.json (for every single binding), we'll know that the process is complete.

Interactive Workflow

• Get two git clones: one of the main repo, and one for the Pythonization PR.
  • For example:

$ git clone [email protected]:mpi-forum/mpi-standard.git mpi4
...git clone output...
$ git clone [email protected]:martinruefenacht/mpi-standard.git mpi-pythonization
...git clone output...

• Git checkout the mpi-4.x branch in the main repo clone.
• Git checkout the feat-modern-lang-bindings in the Pythonization PR repo clone.
  • For example:

$ cd mpi4
$ git checkout mpi-4.x
...some git output...
$ cd ../mpi-pythonization
$ git checkout feat-modern-lang-bindings
...some git output...

• In the Pythonization repo:
  1. Run make cleandoc
  2. Run ./binding-tool/regress.py --interactive <PATH_TO_MPI4_REPO> <PATH_TO_PYTHONIZATION_REPO>.
     • For example: ./binding-tool/regress.py --interactive ../mpi4 .

The interactive mode will iterate through all common bindings between the two repositories and show them to the user for approval.

When a non-approved binding is found, a diff is shown to the user with a likeness ratio and locations in files of both repositories with line of file, respectively.

This is an example of a "big" difference:

Name: comm_copy_attr_function
## BIG CHANGE #####################################################################################################################################

Language: Language.F90
likeness ratio 88.8%
- ../ms2/chap-context/context.tex
+ chap-context/context-rendered.tex

Probable reasons: unknown

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
- \mpifsubbind{COMM\_COPY\_ATTR\_FUNCTION(OLDCOMM, COMM\_KEYVAL, EXTRA\_STATE, ATTRIBUTE\_VAL\_IN, ATTRIBUTE\_VAL\_OUT, FLAG, IERROR)\fargs INTEGER
?                                                                                                                                                  
+ \mpifsubbind{COMM\_COPY\_ATTR\_FUNCTION(OLDCOMM, COMM\_KEYVAL, EXTRA\_STATE, ATTRIBUTE\_VAL\_IN, ATTRIBUTE\_VAL\_OUT, FLAG, IERROR) \fargs INTEGE
?                                                                                                                                    +             
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
 OLDCOMM, COMM\_KEYVAL, IERROR\\INTEGER(KIND=MPI\_ADDRESS\_KIND) EXTRA\_STATE, ATTRIBUTE\_VAL\_IN,\bindindent/ATTRIBUTE\_VAL\_OUT\\LOGICAL FLAG}
                              ^^                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

R OLDCOMM, COMM\_KEYVAL, IERROR \\ INTEGER(KIND=MPI\_ADDRESS\_KIND) EXTRA\_STATE, ATTRIBUTE\_VAL\_IN, ATTRIBUTE\_VAL\_OUT \\ LOGICAL FLAG}
                               ^^^^                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


approve(a), skip(s), exit(x): 

A ###### marker is used for difference below 95% matching ratio. A <><><> is shown for less severe mismatches.

The prompt asks for approval for this binding. The user can exit or skip the binding instead of approving. When approved a binding, when encountered is automatically skipped.

Approvals serve as a mechanism to allow for changes between MPI-4 version bindings and the Pythonized version bindings. For example spacing changes, reordering of tildes (non-linebreaking spacing in LaTeX), reordered bindings, etc.

An example of a less severe mismatch is:

-- small change -----------------------------------------------------------------------------------------------------------------------------------

Language: Language.F90
likeness ratio 98.3%
- ../ms2/chap-context/context.tex
+ chap-context/context-rendered.tex

Probable reasons: spacing

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
- \mpifsubbind{COMM\_DELETE\_ATTR\_FUNCTION(COMM, COMM\_KEYVAL, ATTRIBUTE\_VAL, EXTRA\_STATE, IERROR)\fargs INTEGER COMM, COMM\_KEYVAL, IERROR\\INT
?                                                                                                                                             ^^

+ \mpifsubbind{COMM\_DELETE\_ATTR\_FUNCTION(COMM, COMM\_KEYVAL, ATTRIBUTE\_VAL, EXTRA\_STATE, IERROR) \fargs INTEGER COMM, COMM\_KEYVAL, IERROR \\ 
?                                                                                                    +                                         ^^^^
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
EGER(KIND=MPI\_ADDRESS\_KIND) ATTRIBUTE\_VAL, EXTRA\_STATE}

INTEGER(KIND=MPI\_ADDRESS\_KIND) ATTRIBUTE\_VAL, EXTRA\_STATE}



approve(a), skip(s), exit(x):x

Automatic approving is done when identical bindings are discovered between the two repositories.

Automatic skipping is performed for approved bindings, which are saved in the approvals.json file, stored in the working directory.