test:Change test order until swarm + openshift issue is fixed.

[jcvenegas]

Debug why cc-runtime is now woking after swarm test we need to keep checking our PRs move the order of the tests.

Because sometimes an issue can only be reproduced in CI add a function to show cc-runtime logs on error.

Fixes: #835

[clearcontainersbot]

kubernetes qa-passed 👍

[clearcontainersbot]

kubernetes qa-passed 👍

[jodh-intel]

Is that "Fixes" issue number correct?

[grahamwhaley]

@jodh-intel - I think this is just a 'test run' PR - just to extract some info from the CI system, so the fixes number is quite possibly fake'd up, but probably had to have something to get past the checkcommits is my guess :-)

[grahamwhaley]

as such, I've marked as DNM - @jcvenegas - feel free to change that to reflect reality if you need.

[jcvenegas]

@grahamwhaley @jodh-intel , yes - @chavafg could not have a reproducible way to know what is the makes fail fedora 26. I want some logs from here.

[clearcontainersbot]

kubernetes qa-passed 👍

[clearcontainersbot]

kubernetes qa-passed 👍

[jcvenegas]

@jodh-intel @sameo @chavafg removing swarm test from the ci confirms that when we use swarm is cc-runtime list breaks. I would like to remove it while we can confirm what is why what is the case that we miss to not cleanup swarm containers correctly.

This is happneing with docker 17.09 - 17.12. And we dont have support netwoking support for this versions. So were not doing a big testing here.

[jcvenegas]

And also I like to print cc-runtime logs in case of errors.
https://github.com/clearcontainers/tests/pull/833/files#diff-669214c54933afe193e42915ac9980f4R23

[jcvenegas]

@mcastelino any issue to skip swarm test ? Until we can have it more robust ?

@GabyCT also we probably need to add extra testing to know the runtime works correctly after run tests like swarm or even just do quick tests after run all our tests.

[chavafg]

@jcvenegas also by moving the swarm test to be executed first, everything works correctly, which is very strange

[jcvenegas]

@chavafg So could be a combination of running swarm and tests ?

[chavafg]

maybe, but I don't understand why.
Also, another strange behavior is that virtcontainers jobs have been passing:
http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-fedora-26-master/

[GabyCT]

@jcvenegas which is the failing that you are obtaining after running swarm? did you try it with #831 which is cleaning the environment

[GabyCT]

ok @jcvenegas lgtm

[GabyCT]

lgtm

[jcvenegas]

@chavafg yes, we need a reducible test to know what how to fix it. Looking at logs dont see something obvious. But a lot of warns and issues that we should try to reduce.

@GabyCT well not clear, but the swarm test are the last that are executed before openshift and after run them cc-rutime list is not working because try to check if a directory from exits /var/lib/docker/... so because after we run make check all the contaienrs must be removed, seems that something is happening and not totally removed at least there is virtcontaienrs metadata where the runtime is getting information about the directory is trying to accesses. So I think we porbably can add some final test after all tests from ./run.sh to check if we can still do cc-runtime list is working or /var/lib/vircontainers/pods is empty or qemu shim instances are running to check all was cleaned up correctly.

@jodh-intel @grahamwhaley sorry if this PR is more like a workarround but please check if you want to merge it until the root cause if found.

[grahamwhaley]

I edited the PR message to set the fixes to the correct Issue (it was self-referencing the PR number)
It might have been nice to have the cleanup addition as a separate commit (nice btw :-) ).

Given we really need the CI working, and we have the Issue open to track the core problem, this is fine with me.

[grahamwhaley]

lgtm

[jodh-intel]

lgtm