pythonizing regression

Pythonizing: Regression

There exist two workflows for regression testing, interactive mode and global mode.

Interactive Workflow

• Git clone the main repo and the Pythonization PR. For example:

$ git clone [email protected]:mpi-forum/mpi-standard.git mpi-3.x
...git clone output...
$ git clone [email protected]:martinruefenacht/mpi-standard.git mpi-pythonization
...git clone output...

• Git checkout the mpi-3.1 tag in the 3.1 repo
• Git checkout the feat-modern-lang-bindings in the Pythonization repo
• For example:

$ cd mpi-3.x
$ git checkout mpi-3.1
...some git output...
$ cd ../mpi-pythonization
$ git checkout feat-modern-lang-bindings
...some git output...

• In the Pythonization repo:
  1. Run make cleandoc
  2. Run ./binding0tool/regress.py --interactive <PATH_TO_MPI31_REPO> <PATH_TO_MPI4_REPO>

The interactive mode will iterate through all common bindings between the two repositories and show them to the user for approval.

When a non-approved binding is found a diff is shown to the user with a likeness ratio and locations in files of both repositories with line of file, respectively.

This is an example of a difference:

################################################################################
################################################################################

likeness ratio 0.9048843187660668
original ../mpis/mpi3/chap-coll/coll.tex, 1154
modified chap-coll/coll-rendered.tex, 1103


- \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
?                                                                               
+ \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
?                                                                               

 \\ TYPE(*), DIMENSION(..) :: recvbuf \\ INTEGER, INTENT(IN) :: sendcount, recvcount, root \\ TYPE(MPI\_Datatype), INTENT(IN) :: sendtype, recvtype \\ TYPE(MPI\_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}
   -------------------------------------

 \\ INTEGER, INTENT(IN) :: sendcount, recvcount, root \\ TYPE(MPI\_Datatype), INTENT(IN) :: sendtype, recvtype \\ TYPE(*), DIMENSION(..) :: recvbuf \\ TYPE(MPI\_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}

                                 +++++++++++++++++++++++++++++++++++++

approve(a), skip(s), exit(x): 

A "######" marker is used for difference below 95% matching ratio. A "<><><>" is shown for less severe mismatches.

The prompt asks for approval for this binding. The user can exit or skip the binding instead of approving. When approved a binding, when encountered is automatically skipped.

Approvals serve as a mechanism to allow for changes between MPI 3.1 version bindings and MPI 4.x version bindings. For example spacing changes, reordering of tildas (non-linebreaking spacing in latex), reordered bindings, etc.

An example of a less severe mismatch is:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

likeness ratio 0.9841269841269841
original ../mpis/mpi3/chap-io/io-2.tex, 3342
modified chap-io/io-2-rendered.tex, 3221


- \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR)\fargs INTEGER FH, IERROR
+ \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR) \fargs INTEGER FH, IERRO
?                                                      +                        

\\LOGICAL FLAG}
R \\ LOGICAL FLAG}
 +  +

approve(a), skip(s), exit(x):

Automatic approving is done when identical bindings are discovered between the two repositories. Automatic skipping is performed for approved bindings, which are saved in the approvals.db json file which is stored in the working directory. Therefore it is advisable to run the regression test from the "modified" repository.

Global Workflow

1. git clone two repositories
2. Go to tag mpi-3.1 on one of them, go to the one you are trying to verify in the other repository
3. Run "./binding_tool/regress.py ../mpi-repo-reference ../mpi-repo-new-shiny