Get Documents by Batch

[derekperkins]

Today, there is a Get One Document endpoint, and a Delete Documents by Batch. We have cases where we want to retrieve a large batch of documents by id (~2k), and today we have to loop through and do a call per id, which often causes availability errors. It would be very useful to do a bulk retrieval. I would expect the input to be the same as the batch delete, and the response to be the same as Get Documents, so there's nothing really new to architect.

[gmourier]

Hello, @derekperkins 👋

What do you think of retrieving them with a filter on the id field? We are currently working on a solution to delete documents by filter, and we have the idea to propose a Get Document by filter method.

If id is defined as a filterableAttributes, you will be able to get documents matching the list of ids using the IN operator

Could it work for you?

cc @macraig @irevoire

[derekperkins]

If the filter can take a list of 10k IDs, I don't particularly care about the implementation details

[derekperkins]

Closing the loop here, it looks like it just released in v1.2.0. Thanks for the quick turnaround!

meilisearch/meilisearch#3570
https://github.com/meilisearch/meilisearch/releases/tag/v1.2.0
https://www.meilisearch.com/docs/reference/api/documents#get-documents-with-post

[derekperkins]

I'll watch for your suggestion. I get your point that a facet may not be ideal performance / duplication wise, but it's infinitely better than before where we had to fetch one at a time

[tacman]

I don't think this issue should be closed.

[tacman]

I've added it as a suggestion, but it's not available to vote on yet.

[derekperkins]

Ok, reopened

[tacman]

I added an item to vote on, but it never showed up in the list, so I assume it needed to be approved. I can try again.

[derekperkins]

I'm reopening this discussion per @tacman wanting to make it available without requiring the id to be a filterable attribute.

[irevoire]

Hey, I think we can actually close this discussion as we effectively gave the possibility to get documents by batch via the filters.
And @tacman, I can’t see your suggestion, but the best to help us track it would be to post it here where someone was explicitly asking to get many documents by IDs; #99

[tacman]

Again, using filters is a hack for smaller collections, but isn't the right solution. It duplicates the data, and filters aren't set up to handle large quantities of data that way.

For example, when you do a facet distribution query, you're going to get back the entire database with a quanity of "1". What a waste, I don't want to have the id duplicated in the data or hacking away and "give me the facets except id".

[irevoire]

Again, using filters is a hack for smaller collections, but isn't the right solution. It duplicates the data, and filters aren't set up to handle large quantities of data that way.

That’s only true for the primary key, which wasn’t the main point of this discussion: « Get Documents by Batch ».
I’m not saying your point is not valid; I’m simply saying that if you want a solution to be implemented, the best thing is to help us track and find your comments easily => by posting your message in the right discussion instead of here.

[tacman]

I see, so close this not because it's fixed, but because it should be in another thread (and indeed, I've posted there as well and am tracking it.) This is a big issue for me, the ability to get a batch of documents by identifier. I'll move to the discussion thread.

[irevoire]

Perfect, thanks a lot; it’ll be easier to differentiate and track both issues!