[RFC] define a testing strategy

[openscript]

With the ide extension we had a few regressions recently. We intend to improve the testing. I suggest to write down a TESTING.md strategy. I think to write down some goals and putting it into a markdown file is enough to begin with. A few suggestions:

• Must use e2e testing for platform dependent tests only, where unit tests with mocks aren't sufficient.
• Must write tests for bugs which were reported.
• Can write tests for new features.
• Should reach a 66% testing coverage.

After a week I would close the discussions and put the findings into a markdown file. Of course it can evolve later on. What do you think?

[samuelstroschein]

Should reach a 66% testing coverage.

Do this as a lint rule as part of npm run ci. If we don't have a rule, the testing coverage will be ignored.

Regarding the testing strategy document, I fear that nobody will read it/the guidelines be ignored. Excpet ofc, if a rule like coverage must be greater than 66% is enforced. The coverage rule might be good for the entire monorepo to increase quality.

[felixhaeberle]

Related issue #1428

Can write tests for new features.

@openscript why optional if we enforce 66% coverage?

[openscript]

why optional if we enforce 66% coverage?

I made the experience that in the early stages of a project, where things change quickly and moving fast is necessary, enforcing test coverage can scare contributors and can slow down the project during this phase. Later on we can make it must.

[samuelstroschein]

where things change quickly and moving fast is necessary, enforcing test coverage [...] can slow down the project during this phase

That's why I was hesitant with enforcing test coverage till now. It seems like we can lower the required test coverage for the early stage project though and increase it over time? E.g. start with 50% then 55% then 60% and so on?

a lower test coverage requirement gives more room for quick experimentation.