Machine translation crash: memory issue

[afarajian]

Hi,

I am trying to run the machine translation example, but I face the memory allocation error:
ImportError: ('The following error happened while compiling the node, Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1}, Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), \n, /hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so: failed to map segment from shared object: Cannot allocate memory\n\nOriginal exception:\n\tImportError: The following error happened while compiling the node, Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1}, Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), \n, /hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so: failed to map segment from shared object: Cannot allocate memory', Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1}, Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), '\n', '/hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so: failed to map segment from shared object: Cannot allocate memory')

I tried with different settings, but no success. I also checked the maximum memory which was used in running the script and realized that it was less than 1GB (Max vmem = 830.188M).
Any idea about this problem and how it can be solved?

[abergeron]

Do you have a limit on the process? Because that is probably the only
thing that could cause a mmap to fail like that.

You can check your current limits with "ulimit -a"

2016-04-11 12:17 GMT-04:00 PersianNLPer [email protected]:

Hi,

I am trying to run the machine translation example, but I face the memory
allocation error:
ImportError: ('The following error happened while compiling the node,
Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4,
i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0,
TensorConstant{0}, TensorConstant{-1},
Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1,
i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1,
i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1,
Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 +
i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4,
maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8),
i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1,
i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1,
i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1,
Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 +
i2), i0)}(Composite{Switch(i0, i1, Switch(A ND(LT(i2 , i1), GT(i3, i1)),
i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9),
i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0,
Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), \n,
/hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so:
failed to map segment from shared object: Cannot allocate
memory\n\nOriginal exception:\n\tImportError: The following error happened
while compiling the node, Elemwise{Composite{Switch(i0, i1,
Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3),
i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1},
Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1,
i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1,
i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1,
Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 +
i2), i 0)}(Comp osite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)),
i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9),
i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1,
i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1,
i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1,
Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 +
i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4,
maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8),
i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0,
Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), \n,
/hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so:
failed to map segment from shared object: Cannot allocate memory',
Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4,
i1)), i5, minimum(( i2 + i3) , i6)))}}(Elemwise{le,no_inplace}.0,
TensorConstant{0}, TensorConstant{-1},
Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1,
i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1,
i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1,
Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 +
i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4,
maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8),
i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1,
i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1,
i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1,
Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 +
i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4,
maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8),
i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{s
ub,no_in place}.0, Elemwise{switch,no_inplace}.0), '\n',
'/hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so:
failed to map segment from shared object: Cannot allocate memory')

I tried with different settings, but no success. I also checked the
maximum memory which was used in running the script and realized that it
was less than 1GB (Max vmem = 830.188M).
Any idea about this problem and how it can be solved?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#93

[afarajian]

I am running my experiments on our cluster, and to submit the job I need to define the amount of memory that my process would need. So, I have used 10GB, 20GB, and 30GB so far, and nothing has changed.
About the ulimit -a: the virtual memory size is exactly the same as the amount of memory that I asked for running the experiment. So, basically it is 10, 20, and 30GB in different settings. Here you can see the results of ulimit -a in one of the experiments:
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 514573
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 514573
virtual memory (kbytes, -v) 20971520
file locks (-x) unlimited

[abergeron]

I can't really help anymore then. It works locally and there is no memory
leak so the problem comes from the cluster configuration somehow. I will
admit that I have no other ideas.

2016-04-11 12:29 GMT-04:00 PersianNLPer [email protected]:

I am running my experiments on our cluster, and to submit the job I need
to define the amount of memory that my process would need. So, I have used
10GB, 20GB, and 30GB so far, and nothing has changed.
About the ulimit -a: the virtual memory size is exactly the same as the
amount of memory that I asked for running the experiment. So, basically it
is 10, 20, and 30GB in different settings. Here you can see the results of
ulimit -a in one of the experiments:

core file size (blocks, -c) unlimited data seg size (kbytes, -d)
unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited
pending signals (-i) 514573 max locked memory (kbytes, -l) unlimited max
memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512
bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority
(-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited
max user processes (-u) 514573 virtual memory (kbytes, -v) 20971520 file
locks (-x) unlimited

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#93 (comment)

[afarajian]

Thank you very much.
Actually, the problem was related to the incompatibility of some of the libraries. And is solved now after I removed all the libraries and installed updated versions.

But, now I have another issue which I think is related to the memory again.
Using vocabularies of size 80,000 and higher, the process crashes. I believe it is related to the fact that by increasing the vocabulary size the model size will increase as well, so, it is not able to keep all the model in the memory.
But, is there any way around to be able to work with larger vocabularies?

Thanks.

[nouiz]

Can you give the error message?

In Theano doc, there is a section about speed memory thread off.
Le 13 avr. 2016 05:35, "PersianNLPer" [email protected] a écrit :

Thank you very much.
Actually, the problem was related to the incompatibility of some of the
libraries. And is solved now after I removed all the libraries and
installed updated versions.

But, now I have another issue which I think is related to the memory again.
Using vocabularies of size 80,000 and higher, the process crashes. I
believe it is related to the fact that by increasing the vocabulary size
the model size will increase as well, so, it is not able to keep all the
model in the memory.
But, is there any way around to be able to work with larger vocabularies?

Thank you very much,

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#93 (comment)

[rizar]

Using large vocabularies in NMT is an open problem, there are some papers
on that, see e.g. http://arxiv.org/abs/1412.2007

On 13 April 2016 at 08:37, Frédéric Bastien [email protected]
wrote:

Can you give the error message?

In Theano doc, there is a section about speed memory thread off.
Le 13 avr. 2016 05:35, "PersianNLPer" [email protected] a écrit :

Thank you very much.
Actually, the problem was related to the incompatibility of some of the
libraries. And is solved now after I removed all the libraries and
installed updated versions.

But, now I have another issue which I think is related to the memory
again.
Using vocabularies of size 80,000 and higher, the process crashes. I
believe it is related to the fact that by increasing the vocabulary size
the model size will increase as well, so, it is not able to keep all the
model in the memory.
But, is there any way around to be able to work with larger vocabularies?

Thank you very much,

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
<
#93 (comment)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#93 (comment)

[afarajian]

@rizar: Thank you for the paper.
@nouiz: sorry for my late reply. actually I am now busy with something else, so for now I am running the experiments using the 30K vocabulary. But, I will try larger vocabularies next week and will send you the errors then.