Killed odoorpc process on restore large dump

[DenisShalaevSetronica]

Hi,
I restoring dump by the following command:
with open(dump_file_name) as dump_zip: odoo.db.restore(odoo_pass, db_name, dump_zip)

With small dumps the restore function runs good.

But if I try to restore a bigger dump (zip file 4.6 GB) Free memory ends and I get the message:
Killed

# dmesg|tail -n2
[3846365.574816] Out of memory: Kill process 26435 (python3) score 1567 or sacrifice child
[3846365.576304] Killed process 26435 (python3) total-vm:11203904kB, anon-rss:4604408kB, file-rss:4kB

# uname -a
Linux odoo-backup-6666f7df88-vt8jr 4.4.121-k8s #1 SMP Sun Mar 11 19:39:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.5G        2.7G        4.0G         54M        741M        4.5G
Swap:            0B          0B          0B

# python3 --version
Python 3.6.7

# pip3 show odoorpc
Name: OdooRPC
Version: 0.7.0
Summary: OdooRPC is a Python package providing an easy way to pilot your Odoo servers through RPC.
Home-page: https://github.com/OCA/odoorpc
Author: Sebastien Alix
Author-email: [email protected]
License: LGPL v3
Location: /usr/local/lib/python3.6/dist-packages
Requires:
Required-by:

[pedrobaeza]

I think this is not something that can be handled at OdooRPC level, as it's how Odoo works. You should have more memory for that operation.

@sebalix am I right?

[DenisShalaevSetronica]

But the recovery of the same 4.5 Gb dump via Odoo UI is successful. The problem occurs only through OdooRPC.

[pedrobaeza]

The culprit might be RPC, not OdooRPC 😉 but let's wait for the confirmation.

[sebalix]

We need to investigate, there are two ways (two different RPC calls) to restore a dump if I remember well, but one requires a CSRF token and not the other, maybe the first is used by Odoo UI and is able to handle big dump (sent as a binary content through a HTTP request), and the other (used by OdooRPC, sending the dump as base64 in a JSON parameter) is crashing.
Is the base64 module of Python able to encode such big file?

As a general rule, Odoo UI uses /web controllers (it often requires a token) and can handle such situations, while the JSON-RPC endpoint /jsonrpc is a little less efficient.

EDIT: maybe not about the CSRF token: https://github.com/odoo/odoo/blob/12.0/addons/web/controllers/main.py#L766 so it could be possible to dump/restore big databases with these HTTP controllers.

[sebalix]

@Man-1980 can you try this piece of code with your big file please?

import base64

with open(dump_file_name) as dump_zip:
    b64_data = base64.standard_b64encode(dump_zip.read()).decode()

Are you getting the same result?

[DenisShalaevSetronica]

Are you getting the same result?
yes

[sebalix]

@Man-1980 Thank you, so this is the b64 encoding of such large files which consumes too much memory. Two possibilities:

1. Keep the current RPC (on /jsonrpc) call but encode the file in b64 by chunks, but I don't know if Odoo can decode it without consuming a lot of memory too.
2. Switch on the /web/database/X HTTP methods to manage databases.

The 2) is the best option, quicker to implement.

I let this issue opened, this needs some work, thanks for the report.
Note: if you think you are able to implement this, go ahead, I'll be glad to review your PR :)

[azawawi]

I think this is related to #4.

[pedrobaeza]

What do we finally do with this?

[sebalix]

@pedrobaeza didn't have time to work on this unfortunately. You can let it open

[GSLabIt]

@sebalix any news on this?

[sebalix]

@GSLabIt still not sadly, no time to work on that ATM :(

[Thulium-Drake]

For those that still have issues, I have replace the following code I used to make backups:

    # TODO this seems bad if backup files are larger
    with open(conf['backup_path'] + '/' + database + '-' + date_stamp
              + '.zip', 'wb') as dump_zip:
        dump = odoo.db.dump(conf['password'], database)
        dump_zip.write(dump.read())

with this:

    # using HTTP connection here as RPC connection has issues with memory management
    url = 'http://localhost:8069/web/database/backup'
    args = {'master_pwd': conf['password'], 'backup_format': 'zip', 'name': database}

    dump = requests.post(url, data=args, stream=True)
    dump_zip = open(conf['backup_path'] + '/' + database + '-' + date_stamp
                    + '.zip', 'wb')
    for chunk in dump.iter_content(chunk_size=8192):
        dump_zip.write(chunk)
    dump_zip.close()

The rest of my script still uses RPC to collect all data I need (lists of databases mostly)