Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi Process Workers doesn't work #25

Open
vkakhnych opened this issue Jul 9, 2020 · 2 comments
Open

Multi Process Workers doesn't work #25

vkakhnych opened this issue Jul 9, 2020 · 2 comments

Comments

@vkakhnych
Copy link

We have a simple config:

<system>
  workers 4
</system>

<source>
  @type beats
  bind 127.0.0.1
</source>

But running top I see load on the only one worker. It hits 100% CPU consuming and we're loosing logs.

top - 11:17:00 up 324 days, 16:45,  1 user,  load average: 13.00, 11.62, 10.53
Tasks: 408 total,   1 running, 315 sleeping,   0 stopped,   0 zombie
Cpu(s): 50.9%us,  2.9%sy,  0.0%ni, 45.4%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:  64463604k total, 64052784k used,   410820k free,  2680436k buffers
Swap:        0k total,        0k used,        0k free, 32854600k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                        
11142 td-agent  20   0  608m  92m 8560 S 99.6  0.1  52:54.93 ruby                                                                                                                                           
11144 td-agent  20   0  255m  31m 8220 S  0.3  0.1   0:02.03 ruby                                                                                                                                           
11137 td-agent  20   0  232m  24m 4588 S  0.0  0.0   0:00.57 ruby                                                                                                                                           
11143 td-agent  20   0  255m  31m 8220 S  0.0  0.1   0:02.04 ruby                                                                                                                                           
11145 td-agent  20   0  255m  31m 8244 S  0.0  0.0   0:02.03 ruby

Amazon Linux ver. 1,

fluent-plugin-beats (1.1.0)
fluentd (1.3.3)
@vkakhnych
Copy link
Author

UPD: No any additional listener in the system:

[root@batch01 ~]# netstat -natp | grep 504[4-9]
tcp        0      0 127.0.0.1:5044              0.0.0.0:*                   LISTEN      17190/ruby          
tcp        0      0 127.0.0.1:43746             127.0.0.1:5044              ESTABLISHED 17773/filebeat      
tcp   1452155      0 127.0.0.1:5044              127.0.0.1:43746             ESTABLISHED 17190/ruby          

@vkakhnych
Copy link
Author

Ok, I might misunderstood multi workers approach. Get it worked after adjusting config:

<system>
  workers 4
</system>

<worker 0>
  <source>
    @type beats
    bind 127.0.0.1
  </source>
</worker>

<worker 1>
  <source>
    @type beats
    bind 127.0.0.1
  </source>
</worker>

<worker 2>
  <source>
    @type beats
    bind 127.0.0.1
  </source>
</worker>

<worker 3>
  <source>
    @type beats
    bind 127.0.0.1
  </source>
</worker>

And added these hosts/ports to filebeat config accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant