Travis: avoid redundancy between parallel tests cases

[commial]

Continue the work of #932.

To ensure all tests are present, here is a way:

# testset has been built by test_all.py
t = list(testset.tests)
travis = ""
s = set()
for e, i in [
        # Basic tests
        [["long", "python", "llvm", "gcc", "z3", "qemu", "cparser"], ["regression",]],
        [["long", "python", "llvm", "gcc", "z3", "qemu", "cparser"], ["example",]],
        # Jitter tests
        ## Special case for 'long' tests
        [[], ["long",]],
        ## Special case for 'qemu' tests (splitted by jitter)
        [["llvm", "gcc"], ["qemu",]],
        [["python", "gcc"], ["qemu",]],
        [["python", "llvm"], ["qemu",]],
        ## Other jitter tests (splitted by jitter)
        [["qemu", "long"], ["llvm",]],
        [["qemu", "long"], ["gcc",]],
        [["qemu", "long"], ["python",]],
        # Extensions tests
        [[], ["z3",]],
        [[], ["cparser",]],
]:
    e_tags = [TAGS[x] for x in e]
    i_tags = [TAGS[x] for x in i]
    testset.tests = list(t)
    testset.filter_tags(exclude_tags=e_tags, include_tags=i_tags)
    print "Ex: %s, Inc: %s -> %d tests" % ("," .join(e), ",".join(i), len(testset.tests))
    # Build travis matrix
    out = "-o " + ",".join(i)
    if e:
        out += " -t " + ",".join(e)
    travis += '- MIASM_TEST_EXTRA_ARG="%s"\n' % out
    s.update(testset.tests)

# Ensure accumulated filtered tests are the same as all-in-one
assert s == set(t)

Which outs:

Ex: long,python,llvm,gcc,z3,qemu,cparser, Inc: regression -> 92 tests
Ex: long,python,llvm,gcc,z3,qemu,cparser, Inc: example -> 71 tests
Ex: , Inc: long -> 3 tests
Ex: llvm,gcc, Inc: qemu -> 66 tests
Ex: python,gcc, Inc: qemu -> 67 tests
Ex: python,llvm, Inc: qemu -> 67 tests
Ex: qemu,long, Inc: llvm -> 70 tests
Ex: qemu,long, Inc: gcc -> 82 tests
Ex: qemu,long, Inc: python -> 68 tests
Ex: , Inc: z3 -> 37 tests
Ex: , Inc: cparser -> 4 tests

Indeed:

$ python test_all.py -o regression -t long,python,llvm,gcc,z3,qemu,cparser
...
Result: 92/92 pass
$ python test_all.py -o cparser
...
Result: 4/4 pass

Actually, the number of accumulated test ran is more than the number of test ran by the all-in-one testset. This is completely normal, because some tests have dependency, such as shellcode assembling. As a result, they will be redundant between matrix tests (but that's alright).

[commial]

As a result, "Ran for 17 min 14 sec" \o/
Instead of "Ran for 47 min 31 sec", so thanks @stephengroat for the idea

Now, it's AppVeyor which is taking all the time. Not sure we can apply the same trick.

[stephengroat]

@commial definitely possible, let me see what i can setup

since appveyor doesn't allow concurrent jobs on open source projects, it'll be most important to put the least technically expensive jobs first

[commial]

As it is not parallel, shouldn't we keep one all-in-one test? The matrix has redundancy by design (see above), so in the end it will take longer to finish :/

[stephengroat]

sounds good. was thinking you might want fast feedback (order tests by time taken, fail on first failure)

i'll copy the complete test structure into #930