-
Notifications
You must be signed in to change notification settings - Fork 0
pythonizing regression
There exist two workflows for regression testing, interactive mode and global mode.
- git clone two repositories
- Go to tag mpi-3.1 on one of them, go to the one you are trying to verify in the other repository
- Run "./binding_tool/regress.py --interactive ../mpi-repo-reference ../mpi-repo-new-shiny
The interactive mode will iterate through all common bindings between the two repositories and show them to the user for approval.
When a non-approved binding is found a diff is shown to the user with a likeness ratio and locations in files of both repositories with line of file, respectively.
This is an example of a difference:
################################################################################
################################################################################
likeness ratio 0.9048843187660668
original ../mpis/mpi3/chap-coll/coll.tex, 1154
modified chap-coll/coll-rendered.tex, 1103
- \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, re
?
+ \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, re
?
cvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
cvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
\\ TYPE(*), DIMENSION(..) :: recvbuf \\ INTEGER, INTENT(IN) :: sendcount, recvc
-------------------------------------
\\ INTEGER, INTENT(IN) :: sendcount, recvcount, root \\ TYPE(MPI\_Datatype), IN
ount, root \\ TYPE(MPI\_Datatype), INTENT(IN) :: sendtype, recvtype \\ TYPE(MPI\
TENT(IN) :: sendtype, recvtype \\ TYPE(*), DIMENSION(..) :: recvbuf \\ TYPE(MPI\
+++++++++++++++++++++++++++++++++++++
_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}
_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}
approve(a), skip(s), exit(x):
A "######" marker is used for difference below 95% matching ratio. A "<><><>" is shown for less severe mismatches.
The prompt asks for approval for this binding. The user can exit or skip the binding instead of approving. When approved a binding, when encountered is automatically skipped.
Approvals serve as a mechanism to allow for changes between MPI 3.1 version bindings and MPI 4.x version bindings. For example spacing changes, reordering of tildas (non-linebreaking spacing in latex), reordered bindings, etc.
An example of a less severe mismatch is:
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
likeness ratio 0.9841269841269841
original ../mpis/mpi3/chap-io/io-2.tex, 3342
modified chap-io/io-2-rendered.tex, 3221
- \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR)\fargs INTEGER FH, IERROR
+ \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR) \fargs INTEGER FH, IERRO
? +
\\LOGICAL FLAG}
R \\ LOGICAL FLAG}
+ +
approve(a), skip(s), exit(x):
Automatic approving is done when identical bindings are discovered between the two repositories. Automatic skipping is performed for approved bindings, which are saved in the approvals.db json file which is stored in the working directory. Therefore it is advisable to run the regression test from the "modified" repository.
- git clone two repositories
- Go to tag mpi-3.1 on one of them, go to the one you are trying to verify in the other repository
- Run "./binding_tool/regress.py ../mpi-repo-reference ../mpi-repo-new-shiny
- Document all new types and such...
- Ingest LIS and compare...
- Heuristic comparison for reordering, same length, split and set difference.
Convert parameter f90_suppress_paren into suppress parameter format.- Allow for reversion of previous approved, mistakes happen.
- Potentially rewrite ordered of descriptors for F90 and F08? Ordering is inconsistent, but similar types are grouped spatially. Not order of arguments.