Need Efficient Solution for Slow Collection Restoration in Qdrant

[ruben-quilez]

Hi!
I am currently using Qdrant for vector storage and have encountered a critical issue.

When I take a snapshot of a collection, the process is quick and the download is fast. However, when I attempt to upload and restore the collection, the process becomes extremely slow. For example, I tested this with a collection that is several gigabytes in size, and it took hours to complete the upload.

If for any reason (due to any problem), I need to recreate a collection, it takes an excessive amount of time, leaving my service unavailable for an unacceptably long period.

I have been following the documentation provided here: https://qdrant.tech/documentation/concepts/snapshots/. Is there any faster method available?

This is a critical issue for us, and without a resolution, it significantly undermines the reliability of Qdrant for our operations. I would greatly appreciate our solution to this problem.

Thank you!

Rubén

[generall]

hey @ruben-quilez, how do you upload a snapshot? Do you upload it from your local machine or some server?

[ruben-quilez]

From my local machine to some server. With this method:

def recover_snapshot_from_file(self, file_path: str) -> dict:
        """
        Recovers a collection from a snapshot file.
        
        Args:
            file_path (str): The local path of the snapshot file.
        
        Returns:
            dict: The response from the server.
        """
        url = f"{self.qdrant_url}/collections/{self.collection_name}/snapshots/upload?priority=snapshot"
        headers = {
            'api-key': self.api_key,
            'Content-Type': 'multipart/form-data'
        }
        files = {'snapshot': open(file_path, 'rb')}
        response = requests.post(url, headers=headers, files=files, verify=False)
        response.raise_for_status()
        return response.json()

[ruben-quilez]

One of the main reasons for using snapshots is to be able to restore a backup quickly. In the specific case I mentioned, I set up another Qdrant instance on a different machine and server. My goal is to "migrate" the collections from one instance to another.

[generall]

From my local machine to some server.

Might it be, that your internet connection is just asymmetric for download and upload? It is usually the case

[ruben-quilez]

The same thing happens when I upload them to a local machine (from local also)

[ruben-quilez]

In any case, the current collection size is 1.2 GB and contains 199,628 points. This shouldn't be a problematic size. In the future, we plan to scale up significantly, handling about 100 times this amount.

[adantart]

I confirm the same issue.
Restoring a 2Gb collection (with about 300.000 points) even into local is a nightmare.
I'm really worried the day I had to restore

[adantart]

I confirm this:

enabling api_key and tls, 2Gb of restoration takes about 20 minutes
disabling it, it takes 15 seconds.
Definetely, there must be a bug in the TLS handshake uploading loop ...

Please, Qdrant, take a look !

[generall]

thanks for reporting, we will try to reproduce on our side

[tellet-q]

@ruben-quilez Thank you for the code snippet. I've noticed that you explicitly pass 'Content-Type': 'multipart/form-data' header into the request. Could you please avoid doing that and see if the issue still reproduces? So, the final code will look like this:

        headers = {
            'api-key': self.api_key
        }
        files = {'snapshot': open(file_path, 'rb')}
        response = requests.post(url, headers=headers, files=files, verify=False)

I tried to reproduce your issue on our side. In my case everything works fast with or without TLS if I don't provide the header, but when I do the upload takes ages. It's not Qdrant's issue. The requests library provides multipart/form-data header and calculates boundary value on its own when files arg is provided. If user provides header explicitly then the library does not calculate the boundary value and that may lead to a longer upload time.

[DavidOsparks]

That's funny the snapshot upload function on Qdrant Cloud also pass 'Content-Type': 'multipart/form-data' header...

[Faisalee50]

hi @ruben-quilez , I am getting issue when i vector count goes above 300k+, it comes automatically to zero, without having any log on qdrant server, can you please tell me what is the size of your data ? and did you get any kind of this behaviour? and what is your docker compose configuration , if you can share will be very helpful

[DavidOsparks]

I use Qdrant Cloud GUI to upload snapshots, and it still very slow.
Currently I try to upload an only 1.13GB snapshot. At the beginning, the upload speed is fast. The first 20% can be uploaded in 1 mins. However, the speed becomes slow and slow obviously.
When the upload progress is 30%, the speed becomes 1%/10 secs.
When the upload progress is 40%, the speed becomes 1%/15 secs.
And now I stuck at 58% for several minutes, I don't even know if it still continues.

Maybe provide a function that create split snapshot files (for example, 100MB a file)? Seems small files can be uploaded fast.

[djptek]

Hi @DavidOsparks yes, this is normal the Snapshot upload can appear to pause and it may take some time - provided there is sufficient memory it will upload in time

[DavidOsparks]

So what can I do? Waiting? Or I have to upgrade the size of RAM of my Qdrant Cloud Node? Only a 1GB snapshot uploading takes me more than 1 hour and still stucks at 58%, and I even have no clue that it can be finished.
My node has 4GB RAM. Can you clearly tell me how much RAM I actually need to complete the upload of a < 2GB snapshot within a normal upload time?
I mean, if this is 50GB, I will go to sleep two days and back to see if it is finished. But...this is 1GB.
In fact, during the uploading, I don't see the usage of node RAM increasing.

[DavidOsparks]

And I found the upload is dead on 59% (this time. I try four times and stuck at 58%, 57%, 59%, 59%). It seems impossible to be done. It is timeout at 10.1 mins.

[DavidOsparks]

This is a 502 Bad Gateway. I try different computer to do this, all were stopped at 10 mins.