Can't define a custom tokenizer

[gcjbr]

Hi, thanks for the package.

I've been struggling for a while and the docs are not helping.

I'm trying to use indexSettings to define a custom ngram tokenizer, but it will just ignore it. I'm making sure I delete the index before rebuilding it.

Here are some of my attemps

public function indexSettings(): array
{
    return [
        'analysis' => [
            'analyzer' => [
                'ngram_lowercase' => [
                    'type' => 'standard',
                    'tokenizer' => 'ngram',
                    'filter' => ['lowercase'],
                ],
            ],
        ],
    ];
}

   public function indexSettings(): array
    {
        return [
            'analysis' => [
                'tokenizer' => [
                    'nGram_tokenizer' => [
                        'type' => 'nGram',
                        'min_gram' => 2,
                        'max_gram' => 20,
                    ],
                ],
                'analyzer' => [
                    'nGram_analyzer' => [
                        'type' => 'custom',
                        'tokenizer' => 'nGram_tokenizer',
                        'filter' => [
                            'lowercase',
                        ],
                    ],
                ],
            ],

        ];
    }

public function indexSettings(): array
    {
        return [
            'number_of_shards' => 2,
            'number_of_replicas' => 1,
            'max_ngram_diff' => 20,
            'analysis' => [
                'tokenizer' => [
                    'nGram_tokenizer' => [
                        'type' => 'nGram',
                        'min_gram' => 2,
                        'max_gram' => 20,
                    ],
                ],
                'analyzer' => [
                    'nGram_analyzer' => [
                        'type' => 'custom',
                        'tokenizer' => 'nGram_tokenizer',
                        'filter' => [
                            'lowercase',
                        ],
                    ],
                ],
            ],
        ];
    }

None produced the expected result.

Any help?

[Jeroen-G]

Does this from the demo help a bit?

https://github.com/Jeroen-G/explorer-demo/blob/825dd78f89912e121fb4b88736e76ac5b344c6f1/app/Models/Cartographer.php#L60

[gcjbr]

Does this from the demo help a bit?

https://github.com/Jeroen-G/explorer-demo/blob/825dd78f89912e121fb4b88736e76ac5b344c6f1/app/Models/Cartographer.php#L60

Not really :(

I see it implements a OO approach, but it gives me no hint as to how to change the tokenizer and analyser to a custom one.

Here's what I have so far:

    public function mappableAs(): array
    {
        return [

            'sku' => [
                'type' => 'text',
            ],
            'category' => [
                'type' => 'text',
                'analyzer' => 'nGram_analyzer',
                'search_analyzer' => 'standard',
            ],
            'category_id' => [
                'type' => 'keyword',
            ],
            'subcategory' => [
                'type' => 'text',
                'analyzer' => 'nGram_analyzer',
                'search_analyzer' => 'standard',
            ],
            'subcategory_id' => [
                'type' => 'keyword',
            ],
            'brand' => [
                'type' => 'text',
                'analyzer' => 'nGram_analyzer',
                'search_analyzer' => 'standard',
            ],
            'brand_id' => [
                'type' => 'keyword',
            ],
            'affiliate' => [
                'type' => 'keyword',
            ],
            'affiliate_id' => [
                'type' => 'keyword',
            ],
            'status' => [
                'type' => 'keyword',
            ],
            'status_id' => [
                'type' => 'keyword',
            ],
            'title' => [
                'type' => 'text',
                'analyzer' => 'nGram_analyzer',
                'search_analyzer' => 'standard',
                'fields' => [
                    'keyword' => [
                        'type' => 'keyword',
                    ],
                ],
            ],
            'description' => [
                'type' => 'text',
                'analyzer' => 'nGram_analyzer',
                'search_analyzer' => 'standard',
            ],
            'occasions' => [
                'properties' => [
                    'value' => [
                        'type' => 'keyword',
                    ],
                ],
            ],
            'styles' => [
                'properties' => [
                    'value' => [
                        'type' => 'keyword',
                    ],
                ],
            ],
            'seasons' => [
                'properties' => [
                    'value' => [
                        'type' => 'keyword',
                    ],
                ],
            ],
            'gender' => [
                'type' => 'keyword',
            ],
            'channel' => [
                'type' => 'keyword',
            ],
            'fashion_seasons' => [
                'properties' => [
                    'value' => [
                        'type' => 'text',
                    ],
                ],
            ],
            'ages' => [
                'properties' => [
                    'value' => [
                        'type' => 'keyword',
                    ],
                ],
            ],
            'price' => [
                'type' => 'float',
            ],
            'is_in_stock' => [
                'type' => 'keyword',
            ],
            'shopify_id' => [
                'type' => 'keyword',
            ],
            'user_id' => [
                'type' => 'keyword',
            ],
            'date' => [
                'type' => 'date',
            ],
            'images' => [
                'properties' => [
                    'value' => [
                        'type' => 'text',
                    ],
                    'color' => [
                        'analyzer' => 'nGram_analyzer',
                        'search_analyzer' => 'standard',
                        'type' => 'text',
                    ],
                    'is_in_stock' => [
                        'type' => 'boolean',
                    ],
                ],
            ],

        ];
    }

    public function indexSettings(): array
    {
        return [
            'number_of_shards' => 2,
            'number_of_replicas' => 1,
            'max_ngram_diff' => 20,
            'analysis' => [
                'tokenizer' => [
                    'nGram_tokenizer' => [
                        'type' => 'nGram',
                        'min_gram' => 2,
                        'max_gram' => 20,
                    ],
                ],
                'analyzer' => [
                    'nGram_analyzer' => [
                        'type' => 'custom',
                        'tokenizer' => 'nGram_tokenizer',
                        'filter' => [
                            'lowercase',
                        ],
                    ],
                ],
            ],
        ];
    }

[Jeroen-G]

If you use arrays for index settings instead of objects you have to make sure they return the correct format, which you can find out by diving into the objects that the demo uses, i.e. this part:

return (new Analysis())
            ->addAnalyzer($synonymAnalyzer)
            ->addFilter($synonymFilter)
            ->build();

[Jeroen-G]

I am also pretty sure that this won't do anything by the way:

            'number_of_shards' => 2,
            'number_of_replicas' => 1,
            'max_ngram_diff' => 20,

[moeen-basra]

@souljacker I also faced same problem, if you implement the Aliased Interface hopefully that will solve the problem

[gcjbr]

Thanks moeen

[Telexx12]

Can you explain more your solution, because I face this problem, and I don't find the solution for it. This is what I have so far

class Product extends Model implements Explored, Aliased, IndexSettings
{
    use HasFactory,Searchable;

    protected $fillable = ['name', 'sku', 'description'];

    public function mappableAs(): array
    {
        return [
            'id' => 'keyword',
            'name'=> [
                'type' => 'text',
                'analyzer' => 'nGram_analyzer',
                'search_analyzer' => 'standard',
            ],
            'sku' => 'text',
            'description' => 'text',
        ];
    }

    public function toSearchableArray()
    {
        return[
            'id' => $this->id,
            'name' => $this->name,
            'description' => strip_tags($this->description),
            'sku' => $this->sku
        ];
    }

    public function indexSettings(): array
    {
        return [
            'settings' => [
                'analysis' => [
                    'tokenizer' => [
                        'nGram_tokenizer' => [
                            'type' => 'nGram',
                        ],
                    ],
                    'analyzer' => [
                        'nGram_analyzer' => [
                            'type' => 'custom',
                            'tokenizer' => 'nGram_tokenizer',
                            'filter' => [
                                'lowercase',
                            ],
                        ],
                    ],
                ],
            ],
        ];
    }
}

After rebuilding the index, it always returns 0 results, altough I have a bunch of records indexed. If it were working optimally, it would bring several results.

I want to define a tokenizer because, for example, I have many products whose names start with "product." If I type just "pro" it should return the products with the name "product," but by default, it doesn't. Do you think defining an edge_ngram tokenizer will solve my problem?