Report on missing tests

[chaws]

Last week @mrchapp asked me if SQUAD had any way to tell if a test went missing, because sometimes testruns get stuck and him and @nareshkamboju need to manually "detect" missing tests. Quick answer for that: no, there's currently no way to detect missing tests automatically.

A long time ago Milosz and I discussed about the idea of a test plan, where each project would keep a list of all tests for that project, so when a build is finished, we could compare build's tests against project's test plan. Although this seemed like a good approach, it would be a hustle to keep up with such test plan (LKFT usually runs over 70k tests, and Android is over 1M).

My idea for short-term implementation is to keep instead a list of important tests each build is required to contain, then when builds are finished, there could also be a "missing_tests" attribute with a list of missing tests.

cc: @mrchapp @nareshkamboju

[nareshkamboju]

@chaws This is very important.
Because, There are four cases and may be more,

1. LAVA Jobs yet to run and did not have results and missing test case names (wait for the LAVA jobs to complete).
2. Device got kernel panic / got a boot problem and LAVA jobs timeout and In this case missing test case names.
3. mkfs ext4 failed / mounting SSD drive failed and exit the LAVA job and then we do not see test case names.
4. Manually cancelled LAVA jobs and did not have test results and no test names.

Risk:
Different branches have different set of test results to compare with.
How to find the last good build results set.