Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Labels for Pass/Fail Notifications are Inconsistent #73

Open
d215steinberg opened this issue Aug 17, 2015 · 1 comment
Open

Labels for Pass/Fail Notifications are Inconsistent #73

d215steinberg opened this issue Aug 17, 2015 · 1 comment
Labels

Comments

@d215steinberg
Copy link

  1. Pass/Fail Notifications for EXPECT assertions are labeled with test name (duplicating test name when multiple EXPECTs under test)
  2. Pass/Fail Notifications for VERIFY assertions are labeled with generated phrases
  3. There is no indication of the hierarchical relationship of test cases and assertions

EXAMPLE (based on FILEDEMT):

TESTCASE 'IT WORKS FOR THE SIMPLE STUFF'
       EXPECT '1' TO BE '1'
       EXPECT '2' TO BE '2' 

 TESTCASE 'IT MOCKS ERROR ON FILE OPEN INPUT' 
       MOCK
           FILE INPUT-FILE 
           ON OPEN STATUS '87'
       END-MOCK    
       PERFORM 0100-OPEN-INPUT
       EXPECT WS-INPUT-FILE-STATUS TO BE '87'
       EXPECT '1' TO BE '2'
       VERIFY FILE INPUT-FILE READ NEVER HAPPENED
       VERIFY FILE INPUT-FILE OPEN HAPPENED 3 TIMES

Output:

PASS:   4. IT WORKS FOR THE SIMPLE STUFF                                                   
PASS:   5. IT WORKS FOR THE SIMPLE STUFF                                                   
PASS:   6. IT MOCKS ERROR ON FILE OPEN INPUT                                               
**** FAIL:   7. IT MOCKS ERROR ON FILE OPEN INPUT                                               
    EXPECTED <2               >, WAS <1               > 
PASS:   8. VERIFY     0 ACCESSES
**** FAIL:   9. VERIFY ACCESSES TO OPEN ON INPUT-FILE | EXPECTED  3 ACCESSES, WAS 1                           

Expect to see something more like the following:

PASS:  4. IT WORKS FOR THE SIMPLE STUFF
**** FAIL:   5. IT MOCKS ERROR ON FILE OPEN INPUT
    EXPECTED <2               >, WAS <1               > 
    VERIFY ACCESSES TO OPEN ON INPUT-FILE | EXPECTED  3 ACCESSES, WAS 1        

When all assertions of a test case pass, we should report only that the test case has passed. When any assertions of a test case fail, we should report the test case failure along with the details of each failing assertion.

@d215steinberg d215steinberg changed the title Labels by Pass/Fail Notifications are Inconsistent Labels for Pass/Fail Notifications are Inconsistent Aug 17, 2015
@neopragma neopragma added the bug label Aug 18, 2015
@neopragma
Copy link
Owner

DN. I'm calling this a bug because of the inaccurate verify count shown in the example above ("output").

It's partly an an enhancement to improve the messaging (testcase text is displayed for every assertion under the given testcase). Something similar to rspec 'describe' and 'it' might be preferable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants