nginx crash with many requests #25

jalonsoa · 2015-01-15T14:42:32Z

Hi
the nginx crash when many clients request audio files. I think that this is a problem about thread safe.

Are av libraries (avformat, avcodec and avutil) thread safe ?

the bt on nginx crash:

(gdb) bt
#0  0x00007fdd8379cad0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fdd8379d839 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fdd854bb25c in av_freep () from /usr/lib/x86_64-linux-gnu/libavutil.so.51
#3  0x00007fdd8659f606 in avformat_free_context () from /usr/lib/x86_64-linux-gnu/libavformat.so.53
#4  0x00007fdd8659f811 in avformat_close_input () from /usr/lib/x86_64-linux-gnu/libavformat.so.53
#5  0x00000000004dd128 in ngx_http_aac_extract_audio (destination=0x1f53da8, outputfmt=..., log=0x1e2c820, pool=<optimized out>, source=...)
    at src/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:236
#6  ngx_http_aac_handler (r=0x1f52e20) at src/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31
#7  0x000000000044db52 in ngx_http_core_content_phase (r=0x1f52e20, ph=0x1e5b010) at src/http/ngx_http_core_module.c:1407
#8  0x0000000000448325 in ngx_http_core_run_phases (r=0x1f52e20) at src/http/ngx_http_core_module.c:888
#9  0x0000000000454451 in ngx_http_process_request (r=0x1f52e20) at src/http/ngx_http_request.c:1902
#10 0x00000000004553a6 in ngx_http_process_request_line (rev=0x1e86250) at src/http/ngx_http_request.c:1012
#11 0x000000000043d731 in ngx_epoll_process_events (cycle=0x1e1e280, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:691
#12 0x00000000004340d7 in ngx_process_events_and_timers (cycle=0x1e1e280) at src/event/ngx_event.c:248
#13 0x000000000043b5c9 in ngx_worker_process_cycle (cycle=0x1e1e280, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:822
#14 0x0000000000439f35 in ngx_spawn_process (cycle=0x1e1e280, proc=0x43b510 <ngx_worker_process_cycle>, data=0x0, name=0x516bde "worker process", respawn=0) at src/os/unix/ngx_process.c:198
#15 0x000000000043c9c2 in ngx_reap_children (cycle=0x1e1e280) at src/os/unix/ngx_process_cycle.c:631
#16 ngx_master_process_cycle (cycle=0x1e1e280) at src/os/unix/ngx_process_cycle.c:184
#17 0x0000000000419c0d in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:407

The text was updated successfully, but these errors were encountered:

flavioribeiro · 2015-01-15T16:16:31Z

did you saw these errors before the pull request?

flavioribeiro · 2015-01-15T16:19:13Z

other question, are you caching the results of this location? what do you mean by many requests?

jalonsoa · 2015-01-15T16:56:15Z

Hi Flavio,
yes, I'm using the module with the new features.

I have this nginx server behind varnish. But we have 5 varnish farms (diferent cdn providers). When we have more of 3 simultaneous request to the same audio file, the nginx crash.

I think that my installation of av libs don't have thread support (seems Ubuntu forgot add the --threads flag when compiled).

I'm trying with mutex before and after free context ... I'll tell you how works.

If this not work, I will try with av_lockmgr_register (http://ffmpeg.org/doxygen/trunk/group__lavc__misc.html#gae120ec60cdf49cf0f793eff3ec731bcc) on module init

j

jalonsoa · 2015-01-16T09:08:42Z

Hi Flavio,
all work fine after put mutex around avformat_free_context and avformat_close_input.

But, I've been thinking about this, and I think that the best solution is create one init function for the module and move the av_register_all and av_log_set_level. And, in this module init function add the av_lockmgr_register.

What think about this ?

thanks
j

flavioribeiro · 2015-01-16T12:26:31Z

hi @jalonsoa, sounds good to me. I'm still surprised that I hadn't face this problem here. Have you tried compiling your libav with threads?

Just curious, can you tell me how many RPM nginx can serve with this mutex?

jalonsoa · 2015-01-16T13:08:44Z

Hi Flavio,
sounds like my libav have threads ... (at least, is linked with the
library). Maybe the mpegts output format don't work fine, I don't known.

On this server, all request = 1751 RPM, only audio.ts related, 115 RPM

j

On Fri, Jan 16, 2015 at 1:26 PM, Flávio Ribeiro [email protected]
wrote:

hi @jalonsoa https://github.com/jalonsoa, sounds good to me. I'm still
surprised that I hadn't face this problem here. Have you tried compiling
your libav with threads?

Just curious, can you tell me how many RPM nginx can serve with this mutex?

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

flavioribeiro · 2015-01-16T15:59:25Z

It's ok for me @jalonsoa, can you make another pull request?

flavioribeiro · 2015-01-16T16:02:02Z

add yourself on AUTHORS file also 🍻

jalonsoa · 2015-01-19T12:07:53Z

Hi Flavio,
I'm trying the changes that I tell you, but don't work. I created the init function put the initialitations function into it, but the nginx crash into libav function (different functions - av_write_trailer, avformat_close_input, av_interleaved_write_frame).

I'm trying to update my libav libs to ensure support of threads or solve bugs in libav libs. I will tell you about progress on this way.

j

flavioribeiro · 2015-01-19T12:11:44Z

great, thank you for making me updated.

flavioribeiro · 2015-01-27T16:50:08Z

hey @jalonsoa any updates on this?

jalonsoa · 2015-01-28T12:44:38Z

Hi Flavio,
sorry. I'm bussy this week .. ;-( .. yesterday compile new libav for my
machine (this time I'm sure that have threads support), and recompile nginx
against the new lib. Bad news ... nginx crash again.

Any idea ?

(gdb) bt
#0 0x00007ffc914b83c4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffc9379d650 in av_parser_close () from
/usr/lib/x86_64-linux-gnu/libavcodec.so.56
#2 0x00007ffc944a0d24 in ?? () from
/usr/lib/x86_64-linux-gnu/libavformat.so.56
#3 0x00007ffc944a1137 in ?? () from
/usr/lib/x86_64-linux-gnu/libavformat.so.56
#4 0x00000000004dd0c4 in ngx_http_aac_extract_audio
(destination=0x1916b48, outputfmt=..., log=0x18fdc80, pool=,
source=...)
at
/html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:245
#5 ngx_http_aac_handler (r=0x1915bd0) at
/html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31
#6 0x000000000044db52 in ngx_http_core_content_phase (r=0x1915bd0,
ph=0x189bdb0) at src/http/ngx_http_core_module.c:1407
#7 0x0000000000448325 in ngx_http_core_run_phases (r=0x1915bd0) at
src/http/ngx_http_core_module.c:888
#8 0x0000000000454451 in ngx_http_process_request (r=0x1915bd0) at
src/http/ngx_http_request.c:1902
#9 0x00000000004553a6 in ngx_http_process_request_line (rev=0x18c8d18) at
src/http/ngx_http_request.c:1012
#10 0x000000000043d731 in ngx_epoll_process_events (cycle=0x185f060,
timer=, flags=) at
src/event/modules/ngx_epoll_module.c:691
#11 0x00000000004340d7 in ngx_process_events_and_timers (cycle=0x185f060)
at src/event/ngx_event.c:248
#12 0x000000000043b5c9 in ngx_worker_process_cycle (cycle=0x185f060,
data=) at src/os/unix/ngx_process_cycle.c:822
#13 0x0000000000439f35 in ngx_spawn_process (cycle=0x185f060, proc=0x43b510
<ngx_worker_process_cycle>, data=0x0, name=0x516b1e "worker process",
respawn=-3)
at src/os/unix/ngx_process.c:198
#14 0x000000000043b804 in ngx_start_worker_processes (cycle=0x185f060, n=1,
type=-3) at src/os/unix/ngx_process_cycle.c:368
#15 0x000000000043c486 in ngx_master_process_cycle (cycle=0x185f060) at
src/os/unix/ngx_process_cycle.c:140
#16 0x0000000000419c0d in main (argc=, argv=)
at src/core/nginx.c:407

On Tue, Jan 27, 2015 at 5:50 PM, Flávio Ribeiro [email protected]
wrote:

hey @jalonsoa https://github.com/jalonsoa any updates on this?

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

jalonsoa · 2015-01-28T12:45:11Z

I try this week without the patchs about lock manager

j

On Wed, Jan 28, 2015 at 1:44 PM, Justo Alonso [email protected]
wrote:

Hi Flavio,
sorry. I'm bussy this week .. ;-( .. yesterday compile new libav for my
machine (this time I'm sure that have threads support), and recompile nginx
against the new lib. Bad news ... nginx crash again.

Any idea ?

(gdb) bt
#0 0x00007ffc914b83c4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffc9379d650 in av_parser_close () from
/usr/lib/x86_64-linux-gnu/libavcodec.so.56
#2 0x00007ffc944a0d24 in ?? () from
/usr/lib/x86_64-linux-gnu/libavformat.so.56
#3 0x00007ffc944a1137 in ?? () from
/usr/lib/x86_64-linux-gnu/libavformat.so.56
#4 0x00000000004dd0c4 in ngx_http_aac_extract_audio
(destination=0x1916b48, outputfmt=..., log=0x18fdc80, pool=,
source=...)
at
/html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:245
#5 ngx_http_aac_handler (r=0x1915bd0) at
/html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31
#6 0x000000000044db52 in ngx_http_core_content_phase (r=0x1915bd0,
ph=0x189bdb0) at src/http/ngx_http_core_module.c:1407
#7 0x0000000000448325 in ngx_http_core_run_phases (r=0x1915bd0) at
src/http/ngx_http_core_module.c:888
#8 0x0000000000454451 in ngx_http_process_request (r=0x1915bd0) at
src/http/ngx_http_request.c:1902
#9 0x00000000004553a6 in ngx_http_process_request_line (rev=0x18c8d18) at
src/http/ngx_http_request.c:1012
#10 0x000000000043d731 in ngx_epoll_process_events (cycle=0x185f060,
timer=, flags=) at
src/event/modules/ngx_epoll_module.c:691
#11 0x00000000004340d7 in ngx_process_events_and_timers (cycle=0x185f060)
at src/event/ngx_event.c:248
#12 0x000000000043b5c9 in ngx_worker_process_cycle (cycle=0x185f060,
data=) at src/os/unix/ngx_process_cycle.c:822
#13 0x0000000000439f35 in ngx_spawn_process (cycle=0x185f060,
proc=0x43b510 <ngx_worker_process_cycle>, data=0x0, name=0x516b1e "worker
process", respawn=-3)
at src/os/unix/ngx_process.c:198
#14 0x000000000043b804 in ngx_start_worker_processes (cycle=0x185f060,
n=1, type=-3) at src/os/unix/ngx_process_cycle.c:368
#15 0x000000000043c486 in ngx_master_process_cycle (cycle=0x185f060) at
src/os/unix/ngx_process_cycle.c:140
#16 0x0000000000419c0d in main (argc=, argv=) at src/core/nginx.c:407

On Tue, Jan 27, 2015 at 5:50 PM, Flávio Ribeiro [email protected]
wrote:

hey @jalonsoa https://github.com/jalonsoa any updates on this?

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

flavioribeiro · 2015-01-28T15:57:47Z

I think we can go ahead with lock manager.

jalonsoa · 2015-01-28T16:10:43Z

Hi Flavio,
I try without lock manager, and all work fine now (from this morning
without crashes).

I'm preparing a patch to send you a patch ...

thanks !
j

On Wed, Jan 28, 2015 at 4:57 PM, Flávio Ribeiro [email protected]
wrote:

I think we can go ahead with lock manager.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

flavioribeiro · 2015-01-28T16:24:14Z

Awesome, thanks 🍻

jalonsoa · 2015-01-30T12:30:08Z

Hi Flavio,
sorry, but the nginx crash again. I install debug symbols for libc and the new libav libs ... and this is the backtrace:

(gdb) bt
#0  0x00007f3321ebead0 in malloc_consolidate (av=0x7f33221f8720) at malloc.c:4286
#1  0x00007f3321ebf839 in malloc_consolidate (av=0x7f33221f8720) at malloc.c:4255
#2  _int_free (av=0x7f33221f8720, p=<optimized out>, have_lock=0) at malloc.c:4186
#3  0x00007f3323be335c in av_freep (arg=0x372e438) at /usr/src/libav/libav-11/libavutil/mem.c:201
#4  0x00007f3324eaa8ad in avformat_free_context (s=0x372e420) at /usr/src/libav/libav-11/libavformat/utils.c:2484
#5  0x00007f3324eaaa7d in avformat_close_input (ps=0x7fffac2509f8) at /usr/src/libav/libav-11/libavformat/utils.c:2511
#6  0x0000000000543c62 in ngx_http_aac_extract_audio (pool=0x37362c0, log=0x360f9c0, source=..., outputfmt=..., destination=0x3737288)
    at /html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:241
#7  0x000000000054333d in ngx_http_aac_handler (r=0x3736310) at /html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31
#8  0x0000000000463d0f in ngx_http_core_content_phase (r=0x3736310, ph=0x363e4c8) at src/http/ngx_http_core_module.c:1407
#9  0x0000000000462845 in ngx_http_core_run_phases (r=0x3736310) at src/http/ngx_http_core_module.c:888
#10 0x00000000004627bc in ngx_http_handler (r=0x3736310) at src/http/ngx_http_core_module.c:871
#11 0x00000000004724fc in ngx_http_process_request (r=0x3736310) at src/http/ngx_http_request.c:1902
#12 0x0000000000470e15 in ngx_http_process_request_headers (rev=0x366cc20) at src/http/ngx_http_request.c:1333
#13 0x000000000047018e in ngx_http_process_request_line (rev=0x366cc20) at src/http/ngx_http_request.c:1012
#14 0x0000000000475040 in ngx_http_keepalive_handler (rev=0x366cc20) at src/http/ngx_http_request.c:3181
#15 0x000000000045284f in ngx_epoll_process_events (cycle=0x3601ff0, timer=10070, flags=1) at src/event/modules/ngx_epoll_module.c:691
#16 0x000000000044296c in ngx_process_events_and_timers (cycle=0x3601ff0) at src/event/ngx_event.c:248
#17 0x000000000045071a in ngx_worker_process_cycle (cycle=0x3601ff0, data=0x0) at src/os/unix/ngx_process_cycle.c:822
#18 0x000000000044ce54 in ngx_spawn_process (cycle=0x3601ff0, proc=0x450519 <ngx_worker_process_cycle>, data=0x0, name=0x592773 "worker process", respawn=-3)
    at src/os/unix/ngx_process.c:198
#19 0x000000000044f432 in ngx_start_worker_processes (cycle=0x3601ff0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:368
#20 0x000000000044ea6c in ngx_master_process_cycle (cycle=0x3601ff0) at src/os/unix/ngx_process_cycle.c:140
#21 0x000000000041968b in main (argc=1, argv=0x7fffac251338) at src/core/nginx.c:407

It crash freeing pointer from s->priv_data (s is the input_format_context). Reading the docs from libav, the priv_data holds information about options, only if codecs need it. I don't known about my input ts files (generates by nginx-rtmp module) need thes structs and your files don't need (or don't crash here).

This is the info from AVFormatContext when crash

(gdb) print *(AVFormatContext *)s
$13 = {av_class = 0x7f33250ee120, iformat = 0x7f33250f8d20, oformat = 0x0, priv_data = 0x3749360, pb = 0x3729ae0, ctx_flags = 1, nb_streams = 2, streams = 0x3651990, 
  filename = "/html/nginx/hls/0_zkamo5t3_1/1200756780.ts", '\000' <repeats 981 times>, start_time = 13342281333, duration = 9985000, bit_rate = 83748, packet_size = 0, 
  max_delay = -1, flags = 512, probesize = 5000000, max_analyze_duration = 5000000, key = 0x0, keylen = 0, nb_programs = 1, programs = 0x0, video_codec_id = AV_CODEC_ID_NONE, 
  audio_codec_id = AV_CODEC_ID_NONE, subtitle_codec_id = AV_CODEC_ID_NONE, max_index_size = 1048576, max_picture_buffer = 3041280, nb_chapters = 0, chapters = 0x0, 
  metadata = 0x0, start_time_realtime = 0, fps_probe_size = -1, error_recognition = 1, interrupt_callback = {callback = 0, opaque = 0x0}, debug = 0, 
  max_interleave_delta = 10000000, strict_std_compliance = 0, event_flags = 0, packet_buffer = 0x0, packet_buffer_end = 0x0, data_offset = 0, raw_packet_buffer = 0x0, 
  raw_packet_buffer_end = 0x0, parse_queue = 0x0, parse_queue_end = 0x0, raw_packet_buffer_remaining_size = 2500000, offset = 0, offset_timebase = {num = 0, den = 0}, 
  internal = 0x360af80}
(gdb)

and the priv_data before free it

(gdb) print *(PESContext *)ptr
$14 = {pid = 57971552, pcr_pid = 0, stream_type = 57842400, ts = 0x200000001, stream = 0x3651990, st = 0x676e2f6c6d74682f, sub_st = 0x2f736c682f786e69, state = 1803181872, 
  data_index = 896494945, flags = 828322676, total_size = 808595759, pes_header_size = 909457200, extended_stream_id = 774912055, pts = 29556, dts = 0, ts_packet_pos = 0, 
  header = '\000' <repeats 263 times>, buffer = 0x0, sl = {use_au_start = 0, use_au_end = 0, use_rand_acc_pt = 0, use_padding = 0, use_timestamps = 0, use_idle = 0, 
    timestamp_res = 0, timestamp_len = 0, ocr_len = 0, au_len = 0, inst_bitrate_len = 0, degr_prior_len = 0, au_seq_num_len = 0, packet_seq_num_len = 0}}
(gdb)

What do you think about this ?

flavioribeiro added the bug label Jan 16, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx crash with many requests #25

nginx crash with many requests #25

jalonsoa commented Jan 15, 2015

flavioribeiro commented Jan 15, 2015

flavioribeiro commented Jan 15, 2015

jalonsoa commented Jan 15, 2015

jalonsoa commented Jan 16, 2015

flavioribeiro commented Jan 16, 2015

jalonsoa commented Jan 16, 2015

flavioribeiro commented Jan 16, 2015

flavioribeiro commented Jan 16, 2015

jalonsoa commented Jan 19, 2015

flavioribeiro commented Jan 19, 2015

flavioribeiro commented Jan 27, 2015

jalonsoa commented Jan 28, 2015

jalonsoa commented Jan 28, 2015

flavioribeiro commented Jan 28, 2015

jalonsoa commented Jan 28, 2015

flavioribeiro commented Jan 28, 2015

jalonsoa commented Jan 30, 2015

nginx crash with many requests #25

nginx crash with many requests #25

Comments

jalonsoa commented Jan 15, 2015

flavioribeiro commented Jan 15, 2015

flavioribeiro commented Jan 15, 2015

jalonsoa commented Jan 15, 2015

jalonsoa commented Jan 16, 2015

flavioribeiro commented Jan 16, 2015

jalonsoa commented Jan 16, 2015

flavioribeiro commented Jan 16, 2015

flavioribeiro commented Jan 16, 2015

jalonsoa commented Jan 19, 2015

flavioribeiro commented Jan 19, 2015

flavioribeiro commented Jan 27, 2015

jalonsoa commented Jan 28, 2015

jalonsoa commented Jan 28, 2015

flavioribeiro commented Jan 28, 2015

jalonsoa commented Jan 28, 2015

flavioribeiro commented Jan 28, 2015

jalonsoa commented Jan 30, 2015