Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decoding error 'Fsa' object doesn't support assignment. #133

Open
videodanchik opened this issue Nov 27, 2021 · 12 comments
Open

Decoding error 'Fsa' object doesn't support assignment. #133

videodanchik opened this issue Nov 27, 2021 · 12 comments

Comments

@videodanchik
Copy link
Contributor

videodanchik commented Nov 27, 2021 •
edited
Loading

Hi, I'm experiencing this error while decoding Librispeech with Conformer model.

./conformer_ctc/decode.py --exp-dir conformer_ctc/exp_500_att0.8 \
                          --lang-dir data/lang_bpe_500 \
                          --max-duration 30 \
                          --concatenate-cuts 0 \
                          --bucketing-sampler true \
                          --num-paths 1000 \
                          --epoch 5 \
                          --avg 1 \
                          --method attention-decoder \
                          --nbest-scale 0.5
2021-11-26 19:08:00,025 INFO [decode.py:549] Decoding started
2021-11-26 19:08:00,026 INFO [decode.py:550] {
    'subsampling_factor': 4, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'feature_dim': 80, 'nhead': 8, 
    'attention_dim': 512, 'num_decoder_layers': 6, 'search_beam': 20, 'output_beam': 8, 'min_active_states': 30, 
    'max_active_states': 10000, 'use_double_scores': True, 
    'env_info': {'k2-version': '1.10', 'k2-build-type': 'Release', 'k2-with-cuda': True,
                 'k2-git-sha1': 'fd5565d32ffa8274ff9700453b1e543f34343ed1', 'k2-git-date': 'Wed Nov 10 08:31:12 2021',
                 'lhotse-version': '0.12.0.dev+git.d5e7815.dirty', 'torch-cuda-available': True, 'torch-cuda-version': '11.2',
                 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': 'b945223-dirty',
                 'icefall-git-date': 'Thu Nov 11 03:26:02 2021', 'icefall-path': '/home/audiodan/asr/icefall',
                 'k2-path': '/home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/k2/__init__.py',
                 'lhotse-path': '/home/audiodan/asr/lhotse/lhotse/init.py'},
    'epoch': 5, 'avg': 1, 'method': 'attention-decoder', 'num_paths': 1000, 'nbest_scale': 0.5, 'export': False,
    'exp_dir': PosixPath('conformer_ctc/exp_500_att0.8'), 'lang_dir': PosixPath('data/lang_bpe_500'), 
    'lm_dir': PosixPath('data/lm'), 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 30, 
    'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0,
    'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 10
}
2021-11-26 19:08:00,218 INFO [lexicon.py:176] Loading pre-compiled data/lang_bpe_500/Linv.pt
2021-11-26 19:08:00,253 INFO [decode.py:560] device: cpu
2021-11-26 19:08:01,984 INFO [decode.py:597] Loading G_4_gram.fst.txt
2021-11-26 19:08:01,984 WARNING [decode.py:598] It may take 8 minutes.
Traceback (most recent call last):
  File "./conformer_ctc/decode.py", line 704, in <module>
    main()
  File "/home/audiodan/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "./conformer_ctc/decode.py", line 615, in main
    G["dummy"] = 1
TypeError: 'Fsa' object does not support item assignment
@csukuangfj
Copy link
Collaborator

Oh, please change

G["dummy"] = 1

G.dummy = 1. Fixed in #134

@videodanchik
Copy link
Contributor Author

Thanks!

@videodanchik
Copy link
Contributor Author

For some reason I got Segmentation fault (core dumped) that happens on next line
torch.save(G.as_dict(), params.lm_dir / "G_4_gram.pt")

@videodanchik videodanchik reopened this Nov 27, 2021
@csukuangfj
Copy link
Collaborator

For some reason I got Segmentation fault (core dumped) that happens on next line

Are there additional error logs?

@videodanchik
Copy link
Contributor Author

videodanchik commented Nov 27, 2021 •
edited
Loading

No just simply

(base) audiodan@audiodan-MS-7A94:~/asr/icefall/egs/librispeech/ASR$ ./conformer_ctc/decode.py --exp-dir conformer_ctc/exp_500_att0.8 --lang-dir data/lang_bpe_500 --max-duration 30 --concatenate-cuts 0 --bucketing-sampler true --num-paths 1000 --epoch 5 --avg 1 --method attention-decoder --nbest-scale 0.5
2021-11-26 22:23:36,049 INFO [decode.py:549] Decoding started
2021-11-26 22:23:36,049 INFO [decode.py:550] {'subsampling_factor': 4, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'feature_dim': 80, 'nhead': 8, 'attention_dim': 512, 'num_decoder_layers': 6, 'search_beam': 20, 'output_beam': 8, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'env_info': {'k2-version': '1.10', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': 'fd5565d32ffa8274ff9700453b1e543f34343ed1', 'k2-git-date': 'Wed Nov 10 08:31:12 2021', 'lhotse-version': '0.12.0.dev+git.d5e7815.dirty', 'torch-cuda-available': True, 'torch-cuda-version': '11.2', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': 'b945223-dirty', 'icefall-git-date': 'Thu Nov 11 03:26:02 2021', 'icefall-path': '/home/audiodan/asr/icefall', 'k2-path': '/home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/home/audiodan/asr/lhotse/lhotse/init.py'}, 'epoch': 5, 'avg': 1, 'method': 'attention-decoder', 'num_paths': 1000, 'nbest_scale': 0.5, 'export': False, 'exp_dir': PosixPath('conformer_ctc/exp_500_att0.8'), 'lang_dir': PosixPath('data/lang_bpe_500'), 'lm_dir': PosixPath('data/lm'), 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 30, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 10}
2021-11-26 22:23:36,241 INFO [lexicon.py:176] Loading pre-compiled data/lang_bpe_500/Linv.pt
2021-11-26 22:23:36,273 INFO [decode.py:560] device: cpu
2021-11-26 22:23:37,856 INFO [decode.py:597] Loading G_4_gram.fst.txt
2021-11-26 22:23:37,856 WARNING [decode.py:598] It may take 8 minutes.
Segmentation fault (core dumped)

I've checked It's successfully executed G.dummy = 1 and it can cast G to a dictionary with G.as_dict() the problem happens with torch.save() for sure but I didn't investigate with gdb. I'm curious if it's reproducible or not.

@csukuangfj
Copy link
Collaborator

Screen Shot 2021-11-27 at 11 53 36

I just tested the fix and it works. Not sure why it does not work in your case.

@csukuangfj
Copy link
Collaborator

I don't understand why torch.save(G.as_dict(), params.lm_dir / "G_4_gram.pt") fails.

@danpovey
Copy link
Collaborator

It should be possible to run it in gdb and get a stack trace from the segfault.

@videodanchik
Copy link
Contributor Author

Thanks guys, I got this error in gdb

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:384
384  ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.

BTW I'm decoding with CPU.

@csukuangfj
Copy link
Collaborator

After starting gdb,

(gdb) catch throw
(gdb) run

When segfault occurs, I think you are able to do

(gdb) bt

to show the backtrace.

@videodanchik
Copy link
Contributor Author

videodanchik commented Nov 27, 2021 •
edited
Loading 

(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:384
#1  0x00007fff712fb356 in k2::Array1<int> k2::Cat<int>(std::shared_ptr<k2::Context>, int, k2::Array1<int> const**) ()
   from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/libk2context.so
#2  0x00007fff712e3b0c in k2::FsaVecToTensor(k2::Ragged<k2::Arc> const&) ()
   from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/libk2context.so
#3  0x00007fff726e725b in ?? ()
   from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so
#4  0x00007fff726cc7f7 in ?? ()
   from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so
#5  0x00005555556a8348 in cfunction_call_varargs (kwargs=<optimized out>, args=<optimized out>, func=0x7fff7337a720)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:743
#6  PyCFunction_Call (func=0x7fff7337a720, args=<optimized out>, kwargs=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:773
#7  0x0000555555697dbc in _PyObject_MakeTpCall (callable=0x7fff7337a720, args=<optimized out>, nargs=<optimized out>, keywords=0x0)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:159
#8  0x0000555555723666 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fff6fc89fd8, callable=0x7fff7337a720)
    at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:125
#9  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5cf0)
    at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4963
#10 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3469
#11 0x00005555556ee36b in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:284
#12 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x5555592410b0, func=0x7fff70cdb940)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:411
#13 _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=0x5555592410b0, callable=0x7fff70cdb940)
    at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:127
#14 method_vectorcall (method=<optimized out>, args=0x5555592410b8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/classobject.c:60
#15 0x0000555555657a61 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x5555592410b8, callable=0x7fff6fd86380)
    at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:127
#16 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5cf0)
    at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4963
#17 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3469
#18 0x00005555556edfcb in function_code_fastcall (globals=<optimized out>, nargs=0, args=<optimized out>, co=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--<RET>
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:284
#19 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7ffff6e0f058, nargsf=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:411
#20 0x0000555555698754 in PyVectorcall_Call (kwargs=0x7fff73388d80, tuple=0x7ffff6e0f040, callable=0x7fff6fc72c10)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:200
#21 PyObject_Call (callable=0x7fff6fc72c10, args=0x7ffff6e0f040, kwargs=0x7fff73388d80) at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:228
#22 0x00005555557210b6 in do_call_core (kwdict=0x7fff73388d80, callargs=0x7ffff6e0f040, func=0x7fff6fc72c10, tstate=<optimized out>)
    at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:5010
#23 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3559
#24 0x00005555556ed821 in PyEval_EvalFrameEx (throwflag=0, f=0x7fff7020b800) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:741
#25 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, 
    kwnames=<optimized out>, kwargs=0x55555594b790, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7fff73422b40, name=0x7ffff6d8c0f0, 
    qualname=0x7ffff6d8c0f0) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4298
#26 0x00005555556ee0a3 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x55555594b790, nargsf=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:436
#27 0x00005555556575db in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55555594b790, callable=0x7fff6fc72ca0)
    at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:127
#28 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5cf0)
    at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4963
#29 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3500
#30 0x00005555556ed270 in PyEval_EvalFrameEx (throwflag=0, f=0x55555594b620) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:741
#31 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, 
    kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4298
#32 0x0000555555782543 in PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4327
#33 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:718
#34 0x00005555557825e4 in run_eval_code_obj (co=0x7ffff6c6d500, globals=0x7ffff6dbf1c0, locals=0x7ffff6dbf1c0)
    at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:1165
#35 0x00005555557a8854 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff6dbf1c0, locals=0x7ffff6dbf1c0, flags=<optimized out>, 
    arena=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:1187
#36 0x0000555555669390 in pyrun_file (fp=0x5555558f4130, filename=0x7ffff6d84210, start=<optimized out>, globals=0x7ffff6dbf1c0, locals=0x7ffff6dbf1c0, closeit=1, 
    flags=0x7fffffffdad8) at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:1084
#37 0x000055555566c0d2 in pyrun_simple_file (flags=0x7fffffffdad8, closeit=1, filename=0x7ffff6d84210, fp=0x5555558f4130)
--Type <RET> for more, q to quit, c to continue without paging--<RET>
    at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:439
#38 PyRun_SimpleFileExFlags (fp=0x5555558f4130, filename=<optimized out>, closeit=1, flags=0x7fffffffdad8)
    at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:472
#39 0x000055555566cbf0 in pymain_run_file (cf=0x7fffffffdad8, config=0x5555558f4ec0) at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:391
#40 pymain_run_python (exitcode=0x7fffffffdad0) at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:616
#41 Py_RunMain () at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:695
#42 0x00005555557aba09 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:1141
#43 0x00007ffff7dbe0b3 in __libc_start_main (main=0x55555566d460 <main>, argc=22, argv=0x7fffffffdcd8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffdcc8) at ../csu/libc-start.c:308
#44 0x000055555573afe5 in _start () at ../sysdeps/x86_64/elf/start.S:103

@danpovey
Copy link
Collaborator

danpovey commented Nov 27, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@danpovey @csukuangfj @videodanchik