Check if CorrectAnswer passes the tests in AnswerTests

[wush978]

Dear maintainers,

Thanks for your work of swirlify. I am using swirlify to develop our swirl course and this is great. However, I guess the swirlify::testit does not automatically check if CorrectAnswer passes the tests in AnswerTests.

ChihChengLiang and I develop a test script which scans the lesson.yaml and tests if all CorrectAnswer passes the AnswerTests. Based on this test, I build up a
CI system (Based on Appveyor and Travis-CI) to check the syntax of the lesson.yaml and validate if the answers are correct or not. I think this feature is very useful to develop a swirl course.

Could we send a PR to let swirlify has a function to automatically test all answers?

Best,
Wush

[seankross]

Sounds very interesting! Send us a PR and I'll take a look. I always appreciate your work Wush.

[wush978]

Dear @seankross

ChihChengLiang will give you a PR because this feature is proposed by him and the credit should be his.

[seankross]

I'm eagerly awaiting it!

[wush978]

@seankross

I aware that you implementing test_course and test_lesson under branch 3.1

I am also improving the testing script for my own swirl course. Is it possible for us to work together? I think the current process makes it hard to directly merge the changes into the branch 3.1 and makes it hard for me to understand your latest thinking about the testing script.

[seankross]

Hi @wush978,

I absolutely want to continue working together! What repository and branch have you been working on recently? I'll take a look at that and then I'll explain how I see your ideas fitting into what I've done.

[wush978]

Dear @seankross,

In fact my latest test scripts were written in my own swirl course project. I am glad to contribute them into swirlify.

Both I and @ChihChengLiang are familiar with github, so you can give us a branch and we could keep sending pull request for you. According to previous experience, I think manually merge these codes will make it harder to work together because we might miss your opinions of our PR. For example, I was confused about the latest change of test_lesson and the test script in travis-ci. It seems that they do not test anything, do they?

To make test_lesson reliable, by the way, I want to make all courses and lessons in https://github.com/swirldev/swirl_courses pass the test_lesson. Also I want to write some invalid lesson.yaml to make sure the test_lesson does capture these errors. However, I cannot do this without your help, so that's why I want to make sure how we work together.

[jonmcalder]

I had a similar thing in mind (improving the scope of testing while creating course content and potentially extending this to CI/automated testing). This would be a great addition to swirlify!