Use 0 seconds as execution time when a test file was not executed by test runner

[ArturT]

This should help with edge case when test file cannot be executed by test runner and because of that we won't record timing of that test file.

Edge case could happen when:

• test file is empty
• test file has only pending tests
• test file has syntax error

In such case, we could assume the test file has some default timing like 0.1s of 0 seconds.

Solution idea

We should improve this part of code:

// src/knapsack-pro-core.ts
        onSuccess(queueTestFiles).then(({ recordedTestFiles, isTestSuiteGreen }) => {
          // here we should take all tests from queueTestFiles and assume 0s of execution time for them
          // and merge them with recordedTestFiles.
          // In case recordedTestFiles has less test files than queueTestFiles then 
          // we would have 0s execution time for not executed test files by test runner
          // recordedTestFiles could have less tests when the test runner like Cypress/Jest won't be able to run it and it won't report the test file back to us
          this.recordedTestFiles = this.recordedTestFiles.concat(recordedTestFiles);
          this.isTestSuiteGreen = this.isTestSuiteGreen && isTestSuiteGreen;

          this.fetchTestsFromQueue(false, onSuccess, onFailure);
        });

Related

A similar issue was solved here: KnapsackPro/knapsack_pro-ruby#71 (this is old PR, we should use 0s as default time for not executed tests).

[ArturT]

Just note: It's possible now to set tests timing to 0 seconds. Knapsack Pro API will accept such time.
When tests will be split in the queue then tests with 0 seconds will be served one by one similar as test file with not recorded time.

[ArturT]

Use 0 seconds as execution time when a test file was not executed by a test runner (Cypress/Jest)

This issue solves 2 problems:

• When a test runner (Cypress/Jest) did not execute a test file, then it might not be reported to the API so it could be skipped when the CI node is retried. If we would save to the API the test file with 0 seconds of execution time then this way when you retry the CI node you would attempt to run the test file again (you won't skip it. We solve the problem with accidentally skipping tests)
  • this bug happen for one of Cypress users, when Cypress crashes it won't return the test file that supposed to be executed. Story: https://trello.com/c/tml2tHW3 Related line in code:

    knapsack-pro-js/packages/cypress/src/knapsack-pro-cypress.ts

    Line 69 in 09783ed

    recordedTestFiles: [],

• If the test file was not executed by a test runner you would not see it in the user dashboard. If we solve this issue then you will see in the user dashboard all test files that were assigned to the CI nodes. You will see the test file has 0s in the dashboard so you would know that test cases were not executed for it.

solution

Should be done on @knapsack-pro/core level as described in the PR description.

story

https://trello.com/c/voKluDGW