Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage reporting not 100% accurate #471

Open
overlookmotel opened this issue Dec 29, 2022 · 3 comments
Open

Coverage reporting not 100% accurate #471

overlookmotel opened this issue Dec 29, 2022 · 3 comments
Labels
dev Issue related to dev or tests

Comments

@overlookmotel
Copy link
Owner

Since switched to jest-light-runner (#470), code coverage reporting is no longer working.

jest-light-runner's docs say it only supports Istanbul, but this would probably be a bad idea as that heavily instruments the code, which Livepack is also doing. Need to find a way to get V8 coverage working again.

overlookmotel added a commit that referenced this issue Dec 29, 2022
Code coverage is not working, so it's misleading. See #471.
overlookmotel added a commit that referenced this issue Dec 29, 2022
@overlookmotel overlookmotel reopened this Dec 29, 2022
@overlookmotel
Copy link
Owner Author

Actually, coverage information is not accurate.

Coveralls is showing 82% coverage on lib/serialize/arrays.js whereas previously it was 100%. I think 100% is correct.

Old coverage report
New coverage report

@overlookmotel
Copy link
Owner Author

Replicated on local machine. Seems to be related to multiple test files running on same thread.

Reports 100% coverage on lib/serialize/arrays.js:

npx jest --coverage test/arrays.test.js test/functions.test.js

Reports 85% coverage:

npx jest --maxWorkers 1 --coverage test/arrays.test.js test/functions.test.js

Reports 85% coverage:

npx jest --coverage test/functions.test.js

So it looks like the coverage from test/arrays.test.js is ignored when another test file runs on same thread. I think test/functions.test.js run first in these scenarios.

Maybe related to virtual fixture files. test/functions.test.js uses these, but test/symbols.test.js doesn't. When test/symbols.test.js and test/arrays.test.js are run together, get 100% coverage reported on both lib/serialize/arrays.js and lib/serialize/symbols.js - which neither test achieves on its own.

@overlookmotel
Copy link
Owner Author

Now each test file runs in own worker thread. This has largely fixed the problem. However, it's still not quite right. Coverage report is non-sensical in some places e.g.:

https://coveralls.io/builds/55534172/source?filename=lib%2Fshared%2Ffunctions.js#L157

This shows some lines in a function as executed, but the following lines not, even though there's no conditional or return. If the lines before execute, these lines must too.

@overlookmotel overlookmotel reopened this Dec 30, 2022
@overlookmotel overlookmotel changed the title Coverage no longer working Coverage reporting not 100% accurate Dec 30, 2022
@overlookmotel overlookmotel added the dev Issue related to dev or tests label Dec 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev Issue related to dev or tests
Projects
None yet
Development

No branches or pull requests

1 participant