Meilisearch
Multi-Index Search (Aggregated search results in a single list) #489
Replies: 14 comments 38 replies
-
|
We have a global search field where one can type anything, and it will search both the
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @sandstrom! Is the boosting by index in the multi-index search payload something super important for you? I wonder how it would translate with indexes that don't share the same ranking rules or even the same scheme. |
Beta Was this translation helpful? Give feedback.
-
|
I think it depends a bit on the use-case. It probably wouldn't be a blocker for us. (per-index boost in multi-index search) |
Beta Was this translation helpful? Give feedback.
-
|
The multi-index search with Elasticsearch/Searchkick can be super lightweight. Here is an example: I'm not personally doing index boosting but I can see how it could be useful. My preference between aggregated and non-aggregated is aggregated. Our resource preview can be large and dense enough that it's best to display results in a single column. And yes we do have individual resource searching with custom filtering and sorting. You can for example search for documents by going to the documents index and using that search, or just use the "workspace search" which is in the navbar. |
Beta Was this translation helpful? Give feedback.
-
|
Let's say a query like that is generated by Searchkick (same searched field for both indexes, no boosting per index rule) query_body = {
:query => {
:bool => {
:should => [
# Users
{
:bool => {
:should => [{
:multi_match => {
:query => query,
:type => 'cross_fields',
:fields => ['name'],
:operator => 'and',
},
}],
:filter => {
:term => { :_index => 'users' }, # index restriction
},
},
},
# Companies
{
:bool => {
:should => [{
:multi_match => {
:query => query,
:type => 'cross_fields',
:fields => ['name'],
:operator => 'and',
},
}],
:filter => {
:term => { :_index => 'companies' }, # index restriction
},
},
},
],
},
},
}How will it rank two documents having exactly the same name value for an aggregated results list use-case? 🤔 This may not be a real problem. In most cases, several fields will be searched by indexes and thus documents can be ranked quite "easily" by terms frequency, proximity, etc... (I'm thinking out loud at that point) |
Beta Was this translation helpful? Give feedback.
-
|
I just created 2 resources (document and comment) with the same name (meilisearch) and then did a federated search for "meilisearch". The comment is first and the document is second. I'm unsure why the comment is first but it's not really a problem as the interface clearly distinguishes between the two resource types. |
Beta Was this translation helpful? Give feedback.
-
We don't have instant search but we have a "workspace" search which searches indexes for documents, comments and users.
We're currently using Elasticsearch + Searchkick which supports multi-index searching. We have a Meilisearch PR ready to merge except for the lack of this feature.
Scribehub considers this a must-have, not a nice-to-have. Users sometimes forget where they've seen content and only have some keyword(s) to go by. We also want just about any page in the application to be 1 action away.
Yes. Users sometimes forget where specific content is. It could be in a discussion (comment/reply), a note (associated with a document), or in a document. If they remember keywords in the content that should be all they need to find it again. We also have tags which can be used for both documents and users and being able to do global searches for documents pertaining to a certain expertise or search for the experts themselves is useful. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @archonic Do the documents between your different indices share searchable fields having the same name and also have common ranking rules? |
Beta Was this translation helpful? Give feedback.
-
|
Chiming in here (I'm not in the same company as archonic). In our case (which is similar) the searchable fields doesn't have the same name, and usually not the same ranking rules either. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @sandstrom! |
Beta Was this translation helpful? Give feedback.
-
|
@gmourier The fields are different between the indexes although there is some overlap in attribute names. The ranking rules are also different, mostly because I want a resource-aware default sort. Documents for example are default sorted by most recently updated and users are by default sorted by reputation descending. |
Beta Was this translation helpful? Give feedback.
-
|
Let's say that you are searching into the User index plus the Document index and you need to sort results by search relevance into a single result list. How are you doing that on Elasticsearch today? Is it similar to the solution given by @sandstrom #489 (reply in thread)? |
Beta Was this translation helpful? Give feedback.
-
|
If you want to know more about our product management process and follow our progress, you can consult the public space dedicated to this need. |
Beta Was this translation helpful? Give feedback.
-
|
I had a look there and it looks like you guys are following a good process but I didn't see anywhere to engage with that process. I assume the answers here will be included somewhere in that Notion.io workspace? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @archonic. This is the very first time we are doing that. Since we are an open-source company we want to be as transparent as possible about our way of working as a product team. This whole part of product discovery was completely in the shadows before today. We are trying a discovery process in several steps and this discussion, as well as user interviews sessions, helped us to define the use case we want to focus on. (This is the second step of a process where there is a total of seven steps, we are currently in the Observe step and next week we will work on other steps until we reach the Decide step). What we imagine for the Decide step is to make users test a prototype. After that, we will decide if we go to the delivery or if we still have to iterate to improve it.
Answers are not necessarily quoted or copied and pasted right away for now, but this could happen in the future if the need arises to highlight a frustration or a user citation. ✨ I don't know if this answers your question, but we can discuss it more precisely and go into more detail if you wish. You said "engaging with the process", what do you mean by that? I'm super curious! Note: We are only testing it for the federated-search / multi-index search need for now. 🤓 |
Beta Was this translation helpful? Give feedback.
-
|
I'm happy to test things out! When you mentioned the public space I figured there would be some way to encourage shipping a federated search feature, but it's fair to say I've already done that. I guess the other feature I was looking for was a way to know when there'd be something to test out by subscribing to updates. I can just watch releases on GitHub through meilisearch-rails though. |
Beta Was this translation helpful? Give feedback.
-
|
It makes sense. Yeah, you've already encouraged us to ship a federated search feature at that point! Thanks for helping us by taking the time to answer all our questions. It's possible that the prototype will be shared as a dedicated docker image and will at least be testable with Curl/Postman (we'll see how we can do that better, e.g. providing an A to Z prototype with a doc page and a testable branch on meilisearch-js for example). Nothing has been decided yet on this particular subject. |
Beta Was this translation helpful? Give feedback.
-
|
Hey everyone 👋 This problem-solving has been delayed as we will focus on critical needs to move towards a v1.0 as our top-most priority. Sorry for that; I don't think we should expect a solution until early 2023. I hope to come back here with great news! |
Beta Was this translation helpful? Give feedback.
-
|
Hello 👋 , I'm currently working on an abstraction around Search Engines for PHP and really would like to support Meilisearch in it: schranz-search/schranz-search#27
I plan to move Different Indexes are here (sulu.rocks, People, Organizations, ...)
On the website also a MultiIndex search exist but that frontend is project specific. A filter by Index is there mostly not used and just search field over a multiple indexes with optional some filters.
Currently the Abstraction will throw an Exception when the query tries to search in multiple indexes:
Have Global Search where you can easiliy search for Pages, Products, Contacts, ... in such data management systems make sense.
In Sulu CMS there is an additional Website search where an aggregated result make sense. Products, Articles and Content Pages mixed and sorted by score. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone 👋 This a small update that could be of interest to people who need a multi-index search that returns non-aggregated results. (Each index has its own list of results). We have released this beta capability in instant-meilisearch, and we would like to have your feedback here. Thank you! ✨ |
Beta Was this translation helpful? Give feedback.
-
|
Hey folks ✨ v1.1 has been released and features the Note: it's not possible to aggregate results coming from different indexes on a single list natively yet. 📚 https://docs.meilisearch.com/learn/what_is_meilisearch/features.html#multi-index-search |
Beta Was this translation helpful? Give feedback.
-
|
about the aggregation, can it be done by adding |
Beta Was this translation helpful? Give feedback.
-
|
Hey @gmourier and team 👋 I've recently started exploring use cases for the multi-search API, and I wondered what people thought about adding (optional) support for keyed queries in multi-search? Where currently a request looks like this: and the response returns an array in the same order as the one provided in the request. I wonder whether we could enable clients to provide keys (names or IDs, however you'd like to refer to them) for each of the queries. This would look like: The response for such a query would return an object instead of an array utilising the same keys provided in the request. An error would be raised if the same key is used more than once. The reason such a feature appeals to me is for compiled languages such as Swift (which I regularly use). Keyed queries would allow us to use type-safe structures to get compiler safety and confidence with our queries. Specifically, it would turn code from This could (and should) be done as a non-breaking contribution, allowing clients to choose which method to use to best suit their neds. What are people's thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
For what it's worth, I definitely see a need for aggregate results in a single list. It's currently not possible to sort all of the results by a single, common filter (at least if I'm understanding it right). It's easy to take the results and flatten them into a list, but nothing inside of that flattened list will be sorted correctly by the search query's filter (as each index was filtered independently). So, for example, if you're sorting by a numerical ranking, you'd be filtering per-index, rather than globally for all of the returned hits. After flattening the list, you'd end up with numerical rankings out of order depending on how the indexes were sorted before flattening. In my case, it makes the multi-search kind of useless. It solves the problem of not needing to fire off multiple queries, but most of the utility (in my opinion) is in the aggregate, single-list results. Is the single list results still a possibility? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I have contacts and societies pages. I need to score up my societies results when i'm on societies pages. Typically, need to score up a specific index like elasticSearch. With ElasticSeach we do that with a score function : Sorry for my english i'm french |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone 👋 We just released a 🧪 prototype that allows returning ranking details at search, and we'd love your feedback as it could also allow you to aggregate multi-index search results in a single list. How to get the prototype?Using docker, use the following command:
Alternatively, you can also build the prototype from source by checkout'ing to the How to use the prototype?Getting the ranking details for the results of multiple indexes to be able to re-rank documents coming from distinct indexes:
Questions we have for you
Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️ |
Beta Was this translation helpful? Give feedback.
-
yes, your understanding is correct 👍
Exactly, the current attribute/field ranking rule is working with an inferred mapping of attributes to their importance (or relevancy). It worked pretty well as long as we were no scores, but if we want to produce scores that are comparable between indexes, then we need more information.
Yes that's a very good solution (additionally/alternatively, this mapping could also be specified at search time). Meanwhile, something that should work right now as a workaround is setting the same I'll see what can be done about this in the v1.3 timeframe but it might be a bit too short for us to add this extension to the attribute/field ranking rule.
While the details only expose the Then the aggregate, normalized
For example in the diagram above, there are two ranking rules in the settings:
For each of these “outer” buckets, The aggregate score of a document is computed by looking into which buckets they fall into. For instance in the diagram, the first documents fall into the first You can find that computation in the
I can point you to the code, but I think an explanation is probably easier. Now, this maximum number of typos in a query is dependent on the settings. If a word appears in the list of typo intolerant words, then its max number of typos will be set to 0. Imagine index A that has "Batman" as typo intolerant and index B that doesn't have it. Now, should really the score 667 be considered greater than 500 in this instance? Both have one typo, the difference is that one index allows for more typos than the other. I think both scores are not comparable in this instance, in the sense that comparing them does not compare their relevancy to this query in a significant way. Other settings with a similar influence on typo are if typo tolerance is enabled and the number of letters required to allow for one or two typos, respectively. I apologize for the long-winded explanation, I hope it makes things clearer! |
Beta Was this translation helpful? Give feedback.
-
Oh thats a very interesting idea! Thanks.
Thanks for the detailed explanation! I am not sure I understand every detail yet, but I got the gist. So the aggregate score is "lossy" in that it might not be able to represent the individual scores exactly due to rounding. Right? So each ranking rule has a digit basically, right? What happens if there are more than 4 rules? Then the last rule can never have an influence on the final aggregate score, as it only influences the fractional part of the intermediate float, right? In other words: sorting by the aggregate score might result in a slightly different order than sorting be the scoreDetails, correct? That's not really criticism btw, just making sure I understand everything correctly.
Thanks for this explanation as well! I think I understand most of this now! If typo tolerance for an index is completely disabled then the relevance rule would always return score 1000, correct? And in that case, the index should probably not have the "typo" rule to begin with. However, I still don't understand why the list of attributes for which typo tolerance is allowed matters. You said: the specific score only depends on the maximum number of typos and the actual number of typos. Specifically |
Beta Was this translation helpful? Give feedback.
-
that's correct
Not necessarily, but each ranking rule is refining the score of the previous ranking rules, so the first ranking rules is overwhelmingly important to determine the score. We're discussing internally whether it should be better to keep the scale of 1000 or if we should go to a float unit scale, as it works better for instance to combine with dot products in vector search.
hmmm, you're correct 😄, it doesn't matter. I was being overly conservative. |
Beta Was this translation helpful? Give feedback.
-
Oh right, I assumed because you mentioned
Neat! That makes things easier for us ^_^ Thanks again for your prompt replies, the explanations and for asking for community feedback. This is a great example of open source project community interaction! Looking forward to the release. In the meantime, I'm of course always here to give more feedback! |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome and thanks for the kind words! We'll very probably release another prototype with some changes (I added the missing score to |
Beta Was this translation helpful? Give feedback.
-
|
Hello again 👋 We just released a new version of this prototype with some improvements thanks to your feedback ❤️ How to get the prototype?Using docker, use the following command:
Alternatively, you can also build the prototype from source by checkout'ing to the What has changed?
|
Beta Was this translation helpful? Give feedback.
-
|
Update: Scoring details will be available as an experimental feature in Meilisearch You can find the details in #674 |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Just getting started on Meilisearch and this use case is quite important for us. We want to provide the user the ability to just type something and get the most relevant result, over several types of objects. (Even though we’ll probably provide additional buttons/links to narrow down the search to a specific type/index.) I think the benefit of the aggregated search is that the user doesn’t have to think and understand what are the differences between different kinds of pages, which might not be obvious depending on the application. Also, that’s one less action for the user. My workaround currently is to search over several indexes, merge the results, and sort using the ranking score. This seems to work okay, but now I have an issue with pagination:
Unless I’m getting something wrong, it seems to me that I cannot use native pagination with this kind of search, and that I have to load a ton of results and do the pagination by myself, but I’m actually not sure how to go about this. |
Beta Was this translation helpful? Give feedback.
-
|
I’m thinking of two ways I can solve the pagination issue (but obviously the more practical way would be to have a native multi-index search with aggregated search results, I think). Let’s say I have two indexes A and B, and I want 20 results per page. The first page is easy enough, I just have to query 20 results from A and 20 results from B, merge, sort by descending score, take the first 20 results. Now I want to load the second page of search results. I’m thinking of these two techniques (haven’t implemented them yet):
I’m afraid the first method will not be stable due to the fact that I’m merging and sorting and that many documents can have equal scores. Also I’ll need to pass the various offsets as GET parameters I guess, not pretty. And I’m afraid the second method might be underperforming the more the user advances through the result pages, because I’m getting more and more results from Meilisearch. |
Beta Was this translation helpful? Give feedback.
-
|
Has anybody thought about this pagination issue and found a better workaround? |
Beta Was this translation helpful? Give feedback.
-
|
Now using the second workaround I described, and it seems to work ok (in my local dev environment with a single user, at least). I’m thinking I might limit all global searches to something like 5 pages anyway (100 results; if what I’m looking for isn’t in those, either it doesn’t exist, or my search query will never find it). |
Beta Was this translation helpful? Give feedback.
-
|
Hi @sylbru , thanks for sharing your use case and workaround strategy, I'm glad to hear it's working ok! To your point, native pagination indeed doesn't work in this case; we've gotten similar feedback in the dedicated discussion in #674, we will definitely take this into consideration for future iterations. |
Beta Was this translation helpful? Give feedback.
-
|
Any updates? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @nitrag , we don't have any updates for the time being. However, we're currently discussing our roadmap for the next few months and this feature is listed as one of the potential candidates. We'll make sure to update this discussion if we prioritize it. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone 👋 We have just released the first RC (release candidate) of Meilisearch containing federated search: a multi-search in which the results of the various queries are combined. You can test it by using You are welcome to leave your feedback in this discussion. If you encounter any bugs, please report them here. 🎉 Official and stable release containing this change will be available on |
Beta Was this translation helpful? Give feedback.
-
|
Hey folks 👋 v1.10 has been released! 🦩 You can now aggregate multi-index search results in a single list ✨ Check out the documentation for a detailed description of the feature. |
Beta Was this translation helpful? Give feedback.
-
|
Great, looking forward to try it! |
Beta Was this translation helpful? Give feedback.
-
Hello 👋
This discussion is created to centralize and discuss your needs about being able to search within multiple indexes.
Some suggested questions that may help move the conversation forward.
Thanks ✨
If you want to know more about our product management process and follow our progress, you can consult the public space dedicated to this need.
Beta Was this translation helpful? Give feedback.
All reactions