Define field to search on at search-time

[aaqibjavith]

Is your feature request related to a problem? Please describe.

We have 10M users database where we would like to search only by location field. Currently using facetFilters, I can search for an exact match. In our case, if we type "a", we would like to get all available locations starting with "a".

By the way, We don't want to search in all fields and also don't want to set attributesForFaceting at the index level.

Describe the solution you'd like

curl 'http://localhost:7700/indexes/users/search' --data '{ "q": "a", "searchAttributes": [ "location"] }' 

We like to limit the search attribute during search calls.
Is it possible now?

[qdequele]

Hello @aaqibjavith,

Thanks for this question. What you want is totally possible with MeiliSearch by changing your searchableAttributes on the setting. You can find the way to do it on this page of the documentation. If you only want to search on your location attribute, you can leave only this attribute on the searchableAttributes.

[aaqibjavith]

@qdequele The documentation says we need to update index settings through /indexes/:index_uid/settings/searchable-attributes. But we don't want to limit the searchable attribute in the index level.

We have fields such as location, role, plan, and many more attributes. When our customer types "a" in the location field, I want to list all the locations starts with "a", not from role and plan. Similarly, we need an autocomplete feature for other fields such as role, plan, etc.

Also, note that we have 1000+ customers who use our autocomplete features on different fields. That's why I said, index level setting won't work for us. We need these settings per-query basis.

[CaroFG]

Could filtering be an option?

[aaqibjavith]

@CaroFG Thanks for the doc. This is interesting.

Here I see that filters are applied along with query string. In our case, we want autocomplete on certain fields.

Is it possible to give regex like string "*uni*" in the filter field?
Is there any performance impact of using filter instead of directly using query (q=au)?

[CaroFG]

I'm sorry @aaqibjavith, I'm afraid filtering is not the right solution for you. I thought you could use placeholder search combined with filtering but filtering will only return exact matches, it doesn't support regex either.

I'm not quite sure why the searchable attribute option doesn't work for you, if you only want to search for the location, you can still display all the attributes you want while only searching on location. (searchableAttributes vs displayedAttributes)

In any case, you can always add your ideas to the roadmap if they aren't already there :)

[hiroshi]

I think I have same question.

e.g. I want to search "fight" only in the title field (not in the description field) while search "happy" in only description (not in the title field) on a same index.

I think any of searchableAttributes, filters or facetFilters is not an option for this situation.

[hiroshi]

Just an idea,
Using distinct feature on movie_id
create separate documents for an object

[
  { "id": "001-title", "movie_id": "001", "text": "Fight club'", "field": "title" },
  { "id": "001-desc", "movie_id": "001", "text": "description of the movie...'", "field": "desc" },
  ....
]

I think I can search in only title by filter: "field = title".
Yes, I cannot get wole object from search results, but if only needed movie_id.

[Keoghan]

@qdequele, what a great tool, with easy setup, thanks.

This is one of the first things I was trying to do - essentially search a specific attribute or set of attributes in a single index.

Say I have a few attributes - id, name, keywords and author.

If I want to search for 'Novel' in name, I could use a filter, but this would only return an exact match as I understand it.

If I want to take advantage of non-exact matches, I could create a new index that has searchableAttributes set to name, and use that, but then would need several indexes to handle the different permutations ([name, keywords], [name, author], [author, keywords]) and so on -which could be rather a lot of indexes.

Wildcards in the filters may work for this, or perhaps sending the searchableAttribute(s) that you want to use at query time? I can see a card in the roadmap for allowing filters to have wildcards, but I am not sure due to the architecture if querytime searchableAttributes would be possible?

@hiroshi idea could also be a good workaround at the moment, but could mean many multiples of records.

[curquiza]

but I am not sure due to the architecture if querytime searchableAttributes would be possible?

hello @Keoghan! For the moment it is indeed not possible to define the searchable attributes during the query time sorry.

[Keoghan]

but I am not sure due to the architecture if querytime searchableAttributes would be possible?

hello @Keoghan! For the moment it is indeed not possible to define the searchable attributes during the query time sorry.

Thank you @curquiza, for now I have used an alternative search solution 👍

[Sembiance]

Being able to choose which fields to query by at search time would be awesome. It doesn't have to be per term.

Algolia supports this with the restrictSearchableAttributes property: https://www.algolia.com/doc/api-reference/api-parameters/restrictSearchableAttributes/

Typesense supports this with the query_by property: https://typesense.org/docs/0.21.0/api/documents.html#arguments

[gmourier]

Hello everyone 👋

We've moved the product discussions to a dedicated repository. I'd like to ask you some questions to better understand the needs behind these requests.

Why do you want to do this at the time of the search?
What's not enough about restricting these fields in the settings?

Reading some of the posts here, I get the impression that based on a term typed by the end-user you would like to be able to make the searchableAttributes dynamic at search time. I imagine that this particular need arises because depending on the user journey within your products you would like to emphasize certain fields rather than others but I may be wrong.

I'm really curious to know what brings you to this particular need.

Thank you for your answers and help

[Sembiance]

I have 4 fields that are searchable with a query: content filename extension song title
I want the user, when they perform the search, to be able to CHOOSE which of these fields they want to search:

Say they want to search for a filename containing hello.
They would uncheck all the fields except filename then type hello into the query field.

This is how I would accomplish this in competing products:
typesense : {"q" : "hello", "query_by" : ["filename"]}
algolia : {"query" : "hello", "searchParameters" : {"searchableAttributes" : ["filename"]}}

Since meilisearch doesn't allow changing this at search time, the user has no choice but to search all 4 fields. This results in finding any file that contains the word hello, when the user just wants filenames that have this word in it.

So what I propose is adding a searchableAttributes = ["field_name", ...] variable to the body of the search, which would allow one to further restrict which fields are searched by the q at search time. Again, this would allow one to restrict to a subset of the fields that are set in the searchableAttributes of the index settings.

[gmourier]

Thank you @Sembiance

[flevi29]

I + this.

I was very confused that this wasn't already a thing when looking at the docs, considering we can already search by all the searchable attributes. It's probably way more complicated than it seems to me, but for the time being wouldn't Meili be able to simply ignore results that don't have a match in some specified desired attributes? Would it add too much delay, and/or/because it's not as simple because of how limit and offset works?

I could make this work by doing manual filtration (I think), but I don't know if it's worth the time, both in work and delay it adds to search (because I'd have to possibly make more than several calls to the API, as opposed to it being an internal thing).

[salv0]

Hi there!
+1 on this!!!
do we have any news as of 0.25?

What is described by @Sembiance above, is very similar to our use case.
Our dataset contains various hotels with fields like country, city, etc.

For example, we want to give the ability to our users to search for hotels in a specific country, and thus, be able to specify dynamically the searchableAttributes (to country only, in this case).

[gmourier]

Thanks @salv0 for the precision!

[kitty-eu-org]

@gmourier First of all, thanks to the developer of meilisearch, he is really a good product, I also have such a requirement, but it cannot be realized, so I have to build two indexes and set different searchableAttributes for the two indexes, which will increase the follow-up The burden of maintaining two indexes, I think this is a very common and urgent need, I don't know when this need will be supported?

[gmourier]

Hi @hezhaozhao-git 👋

Thanks for the details and for explaining your current workaround. I agree that it's not practical at the moment. I can't give a precise ETA right now since it's not a priority compared to other work going on. I hope we can reconsider and prioritize this before the end of the year.

[francoism90]

@hezhaozhao-git How did you solved this with Laravel? It seems you're using that framework.

@gmourier It would be a great feature, as you sometimes want to restrict the column results. Any ETA yet?

[kitty-eu-org]

I built two indexes to solve this problem

[Packoverflow]

+1

[cv-chameleon]

+1 for this, this is also the first thing I tried achieving with Meilisearch and it's too bad this is not supported.

My usecase:

Ecommerce platform with 3 supported languages. My products have translated attributes, where "title" is 1 of them.
I store following attributes in my product meilisearch document: "title_en", "title_nl", "title_fr".

There is a search bar in my navigation that let's the user search products.

Currently, when a user would search a product on the FR website, Meilisearch is returning documents that match the given searchquery on all "title_*" fields.

A very simple ( on our end of course :) ) would be to able to restrict the attributes where the search is performed upon. There, I should be able to only search on "title_fr".

I can't imagine this not being a high priority. For perfomance reasons, it makes sense to store attributes in Meilisearch on which you don't even wan't to search, because afterwards you don't have to refetch them from your database.

Like for building a product list, I could use my info from my Meilisearch index directly, and not have to load all my models from the database after my search.

[vpanyushenko]

I think this is a great use case for it. The workaround for this I guess is to make all of the fields searchable, in your case the three titles, then have Meili do the initial fetch from the database. When the result batch is small enough that it would not have to paginate, you can run your own search fuzzy search locally. It's not ideal, and would be a game changer if this was supported natively.

[dutchkillscreative]

Agree this would be really useful for me. I have a platform with posts containing metadata on each image for a "camera", "lens", and "film" in one object.

While a user might have a Leica camera they might have a Minolta lens that they took the image with so being able to filter by a specific attribute at the time of search would really help. So that a search for Camera: 'Minolta' does not include facet results from a post that has a Minolta lens. Currently the facetDistribution would return 'Leica' for that query.

At present I have to break out each of the 3 pieces of metadata into their own shallow index of what could just be a single post index in order to maintain specific search rules for each query.

[Fronix]

Agree, we have a usecase where we have nested objects in our documents, sometimes the users need to search only for names in an array of objects that exists under all documents.

Simple example

We have two construction firms, each firm has several construction sites, our users have the need to be able to search trough all construction sites on a specific field.

This could be solved by having two indexes but the problem is the users want the results of all construction firms that have sites with a certain text, having to search first in the sites index and then try to link this by getting all construction firms is not ideal. Handling the results locally after the results are returned is not ideal either. In the searchableAttributes list we have to rank what fields are most important and well somtimes the ranks of fields change depending on what the user wants to look for.

[
    {
        "name": "Foo's construction",
        "sites": [
            {
                "name": "Foo's construction site 1",
                "address": "Site address 12"
            },
            {
                "name": "Foo's construction site 2",
                "address": "Site address 1"
            }
        ]
    },
    {
        "name": "Bar's construction",
        "sites": [
            {
                "name": "Bar's construction site 1",
                "address": "Site address 1"
            },
            {
                "name": "Bar's construction site 2",
                "address": "Site address 55"
            }
        ]
    }
]

Having this feature implemented would allow users to choose if they want to do very specific searches for data. And it would provide much more flexibility.

Extra idea

This could also be a feature in the tenant token's searchRules, it would make it possible to also restrict the user from doing searches or getting results in certain fields. Paired together with displayableAttributes as well i think this could be really powerful

[Sembiance]

Could anyone from the Meilisearch team chime in on this?
What sort of difficulty level is it to implement this into Meiliesearch?

Without this feature (which both your competitors have) several folks (including myself) cannot use Meilisearch.
Out of the 225 discussions, this is the #4th highest ranked item.

Seems like it should get bumped up the priority list :)

[gmourier]

Hey everyone 👋

Thank you for reminding us; we know it's important for you. This pain point is not forgotten; it's always there in our heads.

Right now, our priorities are elsewhere. We are working on resolving workflow blockages with Meilisearch to release a v1.0 in early 2023.

If we have to do it all by ourselves on the core-team side, it won't happen before Q2 2023 because of our current priorities.

We'll talk about it early next week with @meilisearch/core-team and see if we can do something to open it up as an external contribution by then. From what I remember being told about a previous technical exploration, it's a complex task requiring going through many layers of Meilisearch.

I will keep you up-to-date next week here, and I wish you a great weekend until then.

[gmourier]

Hello everyone 👋

Once again, time flies. I apologize for the late response after promising you a quick answer.

We were able to discuss it within the core team last week. The changes are significant. Most of them involve deep changes in the parts of milli, both for search and indexing. It requires generating DBs by fields which is not the current case in the codebase.

Unfortunately, this is something we can't open to contribution today; it would require too many round trips and would make the contribution tedious. We need to do some work beforehand to make this possible.

2023 will bring important additions/enhancements to the search; this one is on the list.

Thanks.

[Sembiance]

Just want to say THANK YOU very much for taking the time to discuss this further. Also THANK YOU for updating this issue. Based on the level of re-work required, I'd imagine 2023 is pretty optimistic, but hey, you never know :)

Thanks again for the update.

[gmourier]

Hey everyone!

Still, no ETA to give about it yet, but @loiclec has done exploration work, and it may solve the need under certain circumstances.

We would love to get your initial feedback.

Thank you!

[briavers]

I'm glad I'm not the only one with this need. In our application depending on your user role, (API vs admin vs user) you can search/see different fields.
for example:

Movies have a title, a director and a cast (json/array field of there names)
free users can search for the title and director but cannot see the cast
pro users can also search for the cast.

[baoanhng]

This is an actual use case; I am also stuck here.
I have an advanced search that offers looking up multiple fields simultaneously.
As other people have described, Meilisearch currently does not support the feature.

[hoc-ht]

+1
I have the same problem. I loved Meilisearch at first sight, but after reading the document I couldn't find this feature. So I'm looking for an alternative search engine.

[francoism90]

@hoc-ht Elasticsearch supports this, however it has a difficult license.

You could create multiple indexes, however I don't know how to do in Laravel yet.

[hoc-ht]

@francoism90 Thank you. I know that. I'm using Elastichsearch. It works well, but it eats my RAM. So I'm looking for a lightweight seach engine. I hope the Meilisearch development teams can develop this feature asap <3

[derekperkins]

While this would be useful, I think the searching for facet values feature helps a lot. It makes for a two stage search, where we autocomplete the value, then submit the search with the exact match filter, which isn't too bad. You could even fake this feature by taking the facet search response, then submitting all variations to the filter, which simulates the full-text search people are looking for. Not sure how well Meilisearch would handle potentially hundreds or thousands of filter values

[macraig]

Hello everyone! 👋

Quick update: we've started working on a solution to define searchable fields at search-time.

Stay tuned for updates, and feel free to keep the feedback coming!

[Packoverflow]

Great to hear! Thanks and good luck!

[dimmou]

Hey there! Thank you for the meilisearch :) Do you have any estimated time of this release?

[macraig]

Hi @dimmou , it will be released in v1.3 (next release) or v1.4. It's hard to say which one exactly at this point in the implementation phase, but I'll keep you posted!

[macraig]

Hello everyone 👋

We just released a 🧪 prototype that allows Meilisearch to define searchable fields at runtime.

You can now use a new search parameter restrictSearchableAttributes to force Meilisearch to search in a smaller scope than the one defined in the settings.

How to get the prototype?

Using docker, use the following command:

docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-define-searchable-fields-at-search-time-0

From source, compile Meilisearch on the prototype-define-searchable-fields-at-search-time-0 tag

How to use the prototype?

You can find an example below, or look at the original PR for more info.

This feature allows the end-user to search in one or multiple attributes using the search parameter restrictSearchableAttributes:

{
  "q": "Captain Marvel",
  "restrictSearchableAttributes": ["title"]
}

This feature act like a filter, forcing Meilisearch only to return the documents containing the requested words in the restricted Attributes. Note that, with the matching strategy last, Meilisearch will only ensure that the first word is in the restricted Attributes, but the retrieved documents will be ordered considering the word contained in the restricted attributes.

Questions we have for you

• Does this prototype cover your use case?
• Are the retrieved documents the ones you expected?
• Do you have any feedback on the API's format and naming?

⚠️ We do NOT recommend using this prototype in production. This is for test purposes only.

Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️

[dimmou]

Thank a lot for this update. Is super needed.

Is there any way to test it without having to run it in Docker?

[ManyTheFish]

Hello @dimmou, we don't build binaries for prototypes.
However, if you install RustLang on your machine you can build it by yourself by:

1. downloading the sources or on pulling the tag prototype-define-searchable-fields-at-search-time-0
2. using cargo to build and run Meilisearch
   a) cargo run --release to build and run
   b) cargo build --release to build and ./target/release/meilisearch to run the binary

Hope this helps! 😄

[dimmou]

Thanx a lot for your response and your time. We will give it a try :)

[thePanz]

Unfortunately this does not address the case for searching in multiple fields with different terms, like available from ElasticSearch.
An example: title:"Captain Marvel" AND director:"Mike"

[gmourier]

Hey @thePanz 👋

Would it help #544 (comment) ?

[macraig]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing this new feature!
You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.3.0-rc.0

You are welcome to leave your feedback in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 The official and stable release containing this change will be available on July 31st, 2023

⚠️ RC (release candidates) are not recommended for production

[macraig]

Hey folks 👋

v1.3 has been released! 🦁 You can now define fields to search on at search time ✨

Note: Attributes passed to attributesToSearchOn must be in the searchable attributes list.

📚 https://www.meilisearch.com/docs/reference/api/search#customize-attributes-to-search-on-at-search-time