Insufficient combination when filter function is specified?

[kmaehashi]

Hello, thank you for providing a great library!
I tried to use filter function to cut invalid combination, but it seems generated combinations are insufficient.

# Generate combinations of every supported Python + NumPy version combination.

from allpairspy import AllPairs

numpy = ['1.9', '1.10', '1.11', '1.12', '1.13', '1.14']
python = ['2.7', '3.4', '3.5', '3.6']

def _validate(numpy, python):
    valid = []
    if numpy == '1.9':
        valid = ['2.7', '3.4']
    elif numpy == '1.10':
        valid = ['2.7', '3.4']
    elif numpy == '1.11':
        valid = ['2.7', '3.4', '3.5']
    elif numpy == '1.12':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.13':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.14':
        valid = ['2.7', '3.4', '3.5', '3.6']
    return python in valid


def _filter_func(row):
    if len(row) == 2:
        stat = _validate(*row)
        #print('Valid? ', row, ' -> ', stat)
        return stat
    return True


if __name__ == '__main__':
    for i, pairs in enumerate(AllPairs([numpy, python], filter_func=_filter_func)):
        print(i, ':', pairs)

I got:

0 : ['1.9', '2.7']
1 : ['1.10', '2.7']
2 : ['1.11', '2.7']
3 : ['1.12', '2.7']
4 : ['1.13', '2.7']
5 : ['1.14', '2.7']
6 : ['1.14', '3.4']
7 : ['1.13', '3.4']
8 : ['1.12', '3.4']
9 : ['1.11', '3.4']
10 : ['1.10', '3.4']
11 : ['1.9', '3.4']

It seems Python 3.5/3.6 variations are totally ignored. Am I missing something?
Any suggestions are welcome. Thanks in advance!

[thombashi]

Hello, @kmaehashi

AllPairs may not properly work when you try to generate combinations from two parameters.
filter_func will properly work if you use three or more parameters as following (a little bit modified your sample code):

from allpairspy import AllPairs

numpy = ['1.9', '1.10', '1.11', '1.12', '1.13', '1.14']
python = ['2.7', '3.4', '3.5', '3.6']
os = ["Linux", "Windows", "macOS"]

def _validate(row):
    numpy, python = row[0], row[1]
    valid = []
    if numpy == '1.9':
        valid = ['2.7', '3.4']
    elif numpy == '1.10':
        valid = ['2.7', '3.4']
    elif numpy == '1.11':
        valid = ['2.7', '3.4', '3.5']
    elif numpy == '1.12':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.13':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.14':
        valid = ['2.7', '3.4', '3.5', '3.6']

    return python in valid


def _filter_func(row):
    if len(row) > 2:
        stat = _validate(row)
        #print('Valid? ', row, ' -> ', stat)
        return stat

    return True


if __name__ == '__main__':
    for i, pairs in enumerate(AllPairs([numpy, python, os], filter_func=_filter_func)):
        print(i, ':', pairs)

0 : ['1.9', '2.7', 'Linux']
1 : ['1.10', '3.4', 'Linux']
2 : ['1.11', '3.5', 'Linux']
3 : ['1.12', '3.6', 'Linux']
4 : ['1.13', '3.6', 'Windows']
5 : ['1.14', '3.5', 'Windows']
6 : ['1.14', '3.4', 'macOS']
7 : ['1.13', '2.7', 'macOS']
8 : ['1.12', '2.7', 'Windows']
9 : ['1.11', '3.4', 'Windows']
10 : ['1.10', '2.7', 'Windows']
11 : ['1.9', '3.4', 'Windows']
12 : ['1.9', '3.4', 'macOS']
13 : ['1.10', '3.4', 'macOS']
14 : ['1.11', '2.7', 'macOS']
15 : ['1.12', '3.5', 'macOS']
16 : ['1.13', '3.4', 'Linux']
17 : ['1.14', '3.6', 'macOS']
18 : ['1.14', '2.7', 'Linux']

product with filter functions might be more suitable for your sample data:

from allpairspy import AllPairs
from itertools import product

numpy = ['1.9', '1.10', '1.11', '1.12', '1.13', '1.14']
python = ['2.7', '3.4', '3.5', '3.6']


def _validate(numpy, python):
    valid = []
    if numpy == '1.9':
        valid = ['2.7', '3.4']
    elif numpy == '1.10':
        valid = ['2.7', '3.4']
    elif numpy == '1.11':
        valid = ['2.7', '3.4', '3.5']
    elif numpy == '1.12':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.13':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.14':
        valid = ['2.7', '3.4', '3.5', '3.6']

    return python in valid


def _filter_func(row):
    return _validate(*row)


if __name__ == '__main__':
    for i, pairs in enumerate(filter(_filter_func, product(numpy, python))):
        print(i, ':', pairs)

0 : ('1.9', '2.7')
1 : ('1.9', '3.4')
2 : ('1.10', '2.7')
3 : ('1.10', '3.4')
4 : ('1.11', '2.7')
5 : ('1.11', '3.4')
6 : ('1.11', '3.5')
7 : ('1.12', '2.7')
8 : ('1.12', '3.4')
9 : ('1.12', '3.5')
10 : ('1.12', '3.6')
11 : ('1.13', '2.7')
12 : ('1.13', '3.4')
13 : ('1.13', '3.5')
14 : ('1.13', '3.6')
15 : ('1.14', '2.7')
16 : ('1.14', '3.4')
17 : ('1.14', '3.5')
18 : ('1.14', '3.6')

[kmaehashi]

@thombashi Thank you for your help.
According to the output from your code, when focusing on Python = 3.5 combination,

2 : ['1.11', '3.5', 'Linux']
5 : ['1.14', '3.5', 'Windows']
15 : ['1.12', '3.5', 'macOS']

NumPy = 1.13 & Python = 3.5 combination, which is not expected to be filtered, seems missing. Is this an expected behavior?

[thombashi]

@kmaehashi
Thank you for your comment.
That is unexpected behavior.
I will check into it.

[kmaehashi]

@thombashi Thanks for the quick response!
Actually I was trying to generate combinations of 8 axes with filtering (chainer/chainer-test#403) and found some pairs missing from generated patterns. (The first example (NumPy/Python) I wrote was the minimum code to reproduce the issue, not the actual usecase.)

[thombashi]

@kmaehashi
Thank you for additional information.
This issue might take some time (I had little time to solve the issue for now).

[kmaehashi]

Hello, any progress on this issue?

[thombashi]

Sorry to keep you waiting.
Still work in progress.

[pavelicii]

@thombashi
Also, the example generation provided in the README lacks following pairs:

98 - Hourly
98 - Contr.
98 - 10
98 - 30
98 - 60
NT - 15
2000 - 6
XP - 10
Salaried - 30
Hourly - 30
Hourly - 60
Part-Time - 60

Are there any plans to fix it?

[pavelicii]

Fixed in this PR.

For @kmaehashi's original input the result is:

{numpy=1.9, python=2.7}
{numpy=1.1, python=2.7}
{numpy=1.11, python=2.7}
{numpy=1.12, python=2.7}
{numpy=1.13, python=2.7}
{numpy=1.14, python=2.7}
{numpy=1.14, python=3.4}
{numpy=1.13, python=3.4}
{numpy=1.12, python=3.4}
{numpy=1.11, python=3.4}
{numpy=1.1, python=3.4}
{numpy=1.9, python=3.4}
{numpy=1.11, python=3.5}
{numpy=1.12, python=3.5}
{numpy=1.13, python=3.5}
{numpy=1.14, python=3.5}
{numpy=1.14, python=3.6}
{numpy=1.13, python=3.6}
{numpy=1.12, python=3.6}

[ashfaq92]

Code:

parameters = OrderedDict(
    {"x1": [True, False],
     "x2": [True, False]})


def constraints(row):
    n = len(row)
    if n > 1:
        if row[0] is False and row[1] is False:
            return False
    return True


print("PAIRWISE:")
res = AllPairs(parameters, filter_func=constraints, n=2)

Desired Output:

0 Pairs(x1=True, x2=True)
1 Pairs(x1=False, x2=True)
2 Pairs(x1=True, x2=False)

Actual Output:

0 Pairs(x1=True, x2=True)
1 Pairs(x1=False, x2=True)