machine translation example crashes

[thompsonb]

I'm getting the following errors when I try to run the blocks machine translation example (repost from blocks-users, maybe this is a real bug?)

python -m machine_translation
< ... >
INFO:machine_translation:Total number of parameters: 37
INFO:machine_translation:Building model
INFO:machine_translation:Initializing extensions
INFO:machine_translation:Building sampling model
INFO:machine_translation:Building sampler
INFO:machine_translation:Building bleu validator
INFO:machine_translation:Initializing training algorithm
INFO:blocks.algorithms:Taking the cost gradient
INFO:blocks.algorithms:The cost gradient computation graph is built
INFO:machine_translation:Initializing main loop
INFO:blocks.main_loop:Entered the main loop
INFO:blocks.algorithms:Initializing the training algorithm
<<!! BUG IN FGRAPH.REPLACE OR A LISTENER !!>> <type 'exceptions.Exception'> Constant{-1} is already owned by another fgraph local_shape_to_shape_i
ERROR (theano.gof.opt): Optimization failure due to: local_shape_to_shape_i
ERROR:theano.gof.opt:Optimization failure due to: local_shape_to_shape_i
ERROR (theano.gof.opt): node: Shape(for{cpu,grad_of_att_trans_do_apply_scan}.20)
ERROR:theano.gof.opt:node: Shape(for{cpu,grad_of_att_trans_do_apply_scan}.20)
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1822, in process_node
fgraph.replace_all_validate(repl_pairs, reason=lopt)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/toolbox.py", line 306, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 592, in replace
self.change_input(node, i, new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 530, in change_input
self.import_r(new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 363, in import_r
self.import(variable.owner, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 394, in import
raise Exception("%s is already owned by another fgraph" % r)
Exception: Constant{-1} is already owned by another fgraph

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1822, in process_node
fgraph.replace_all_validate(repl_pairs, reason=lopt)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/toolbox.py", line 306, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 592, in replace
self.change_input(node, i, new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 530, in change_input
self.import_r(new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 363, in import_r
self.import(variable.owner, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 394, in import
raise Exception("%s is already owned by another fgraph" % r)
Exception: Constant{-1} is already owned by another fgraph

ERROR (theano.gof.opt): Optimization failure due to: local_subtensor_lift
ERROR:theano.gof.opt:Optimization failure due to: local_subtensor_lift
ERROR (theano.gof.opt): node: Subtensor{int64}(MakeVector{dtype='int64'}.0, Constant{2})
ERROR:theano.gof.opt:node: Subtensor{int64}(MakeVector{dtype='int64'}.0, Constant{2})
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/tensor/opt.py", line 2559, in local_subtensor_lift
if not u.owner or len(u.clients) > 1:
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/tensor/opt.py", line 2559, in local_subtensor_lift
if not u.owner or len(u.clients) > 1:
AttributeError: 'TensorVariable' object has no attribute 'clients'

< ... many more AttributeErrors for local_add_canonizer, local_mul_canonizer, etc >

ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run on_error extensions, potentially saving data, before exiting and reraising the error. Note that the usual after_training extensions will not be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.
Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/bjt/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/state/partition1/localdata/bjt/blocks-examples/machine_translation/main.py", line 41, in
get_dev_stream(*_configuration), args.bokeh)
File "machine_translation/init.py", line 175, in main
main_loop.run()
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/main_loop.py", line 197, in run
reraise_as(e)
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/utils/init.py", line 225, in reraise_as
six.reraise(type(new_exc), new_exc, orig_exc_traceback)
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/main_loop.py", line 172, in run
self.algorithm.initialize()
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/algorithms/init.py", line 253, in initialize
self.inputs, [], updates=all_updates, *_self.theano_func_kwargs)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/pfunc.py", line 526, in pfunc
output_keys=output_keys)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1777, in orig_function
output_keys=output_keys).create(
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1457, in init
optimizer_profile = optimizer(fgraph)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 101, in call
return self.optimize(fgraph)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 230, in apply
sub_prof = optimizer.optimize(fgraph)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 2211, in apply
lopt_change = self.process_node(fgraph, node, lopt)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1830, in process_node
self.failure_callback(e, self, repl_pairs, lopt, node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1688, in warn_inplace
return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1674, in warn
raise exc
AssertionError:

Original exception:
AssertionError

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Here are the versions of things that seems relevant:

python --version
Python 2.7.10 :: Anaconda 1.9.1 (64-bit)

python -c "import theano; print theano.version"
0.7.0.dev-80197cf574b62fa7d869f4de800700d2f75b7f4e

pip install git+git://github.com/mila-udem/blocks.git -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt
Collecting git+git://github.com/mila-udem/blocks.git
Cloning git://github.com/mila-udem/blocks.git to /tmp/pip-VSApNQ-build
Requirement already satisfied (use --upgrade to upgrade): blocks==0.1.1 from git+git://github.com/mila-udem/blocks.git in /home/bjt/anaconda/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): numpy==1.9.3 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): picklable-itertools==0.1.1 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): progressbar2==2.7.3 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): pyyaml==3.11 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): toolz==0.7.2 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): theano from git+https://github.com/Theano/Theano.git#egg=theano in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): fuel from git+https://github.com/mila-udem/fuel#egg=fuel in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.11 in /home/bjt/anaconda/lib/python2.7/site-packages (from theano->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): h5py in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): tables in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pyzmq in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pillow in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): requests in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2014 NVIDIA Corporation
Built on Wed_Aug_27_10:36:36_CDT_2014
Cuda compilation tools, release 6.5, V6.5.16

[thompsonb]

It looks like a recent Theano change broke things in blocks... I reverted to the Oct 20 version of Theano (9ee4918c0b418ae8863dc0eaab0e220564d6d038) and everything works.

[shuboma]

I also encounter this problem, can you fix it？

[sjtufs]

@thompsonb This solved my problem. Thanks!

[shuboma]

where and how did you install the Oct 20 version of Theano? @sjtufs @thompsonb

[sjtufs]

@shuboma Hi. You just have to clone the Theano repo, and
git checkout 9ee4918c0b418ae8863dc0eaab0e220564d6d038 then use python setup.py install to install.

[shuboma]

thanks! it works! @sjtufs @thompsonb

[guxd]

I encountered the same problem! I reverted theano to Oct 20. The first time that the theano was re-installed, it runs OK. but when I re-run the program, it crashes again! I think the problem should be fixed according to the latest theano.

[rizar]

I just ran with the latest Blocks and Theano: it works perfectly! Can you guys please doublecheck?