pythonizing regression

Pythonizing: Regression

There exist two workflows for regression testing, interactive mode and global mode.

Interactive Workflow

1. git clone two repositories
2. Go to tag mpi-3.1 on one of them, go to the one you are trying to verify in the other repository
3. Run "./binding_tool/regress.py --interactive ../mpi-repo-reference ../mpi-repo-new-shiny

The interactive mode will iterate through all common bindings between the two repositories and show them to the user for approval.

When a non-approved binding is found a diff is shown to the user with a likeness ratio and locations in files of both repositories with line of file, respectively.

This is an example of a difference:

################################################################################
################################################################################

likeness ratio 0.9048843187660668
original ../mpis/mpi3/chap-coll/coll.tex, 1154
modified chap-coll/coll-rendered.tex, 1103


- \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, re
?                                                                               
+ \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, re
?                                                                               

cvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
                                                                                
cvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
                                                                                

 \\ TYPE(*), DIMENSION(..) :: recvbuf \\ INTEGER, INTENT(IN) :: sendcount, recvc
   -------------------------------------

 \\ INTEGER, INTENT(IN) :: sendcount, recvcount, root \\ TYPE(MPI\_Datatype), IN
                                                                                

ount, root \\ TYPE(MPI\_Datatype), INTENT(IN) :: sendtype, recvtype \\ TYPE(MPI\

TENT(IN) :: sendtype, recvtype \\ TYPE(*), DIMENSION(..) :: recvbuf \\ TYPE(MPI\
                                 +++++++++++++++++++++++++++++++++++++


_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}

_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}

approve(a), skip(s), exit(x): 

A "######" marker is used for difference below 95% matching ratio. A "<><><>" is shown for less severe mismatches.

The prompt asks for approval for this binding. The user can exit or skip the binding instead of approving. When approved a binding, when encountered is automatically skipped.

Approvals serve as a mechanism to allow for changes between MPI 3.1 version bindings and MPI 4.x version bindings. For example spacing changes, reordering of tildas (non-linebreaking spacing in latex), reordered bindings, etc.

An example of a less severe mismatch is:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

likeness ratio 0.9841269841269841
original ../mpis/mpi3/chap-io/io-2.tex, 3342
modified chap-io/io-2-rendered.tex, 3221


- \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR)\fargs INTEGER FH, IERROR
+ \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR) \fargs INTEGER FH, IERRO
?                                                      +                        

\\LOGICAL FLAG}
R \\ LOGICAL FLAG}
 +  +

approve(a), skip(s), exit(x):

Automatic approving is done when identical bindings are discovered between the two repositories. Automatic skipping is performed for approved bindings, which are saved in the approvals.db json file which is stored in the working directory. Therefore it is advisable to run the regression test from the "modified" repository.

Global Workflow

1. git clone two repositories
2. Go to tag mpi-3.1 on one of them, go to the one you are trying to verify in the other repository
3. Run "./binding_tool/regress.py ../mpi-repo-reference ../mpi-repo-new-shiny

TODOs

• Document all new types and such...
• Ingest LIS and compare...
• Heuristic comparison for reordering, same length, split and set difference.
• Convert parameter f90_suppress_paren into suppress parameter format.
• Allow for reversion of previous approved, mistakes happen. "Return to previous"
• Potentially rewrite ordered of descriptors for F90 and F08? Ordering is inconsistent, but similar types are grouped spatially. Not order of arguments.
• Handle callback functions.
• Handle weird macros that are being skipped. mpifoverloadOnlyInAnnex
• Clean up regression tool!