[QUESTION]:采集过程中插件报错error:loggroup queue is full #775

xjdszp · 2023-04-07T12:39:15Z

xjdszp
Apr 7, 2023

在压测过程中，采集插件报错，怎样优化loggroup queue
2023-04-07 09:18:55 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/json_log.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/json_log.yaml

Answered by shalousun

May 8, 2023

loggroup queue is full有三个可能的原因：

ilogtail采集了日志，日志聚合后flusher往下游写入数据时候性能根不上，logprocess queue是一个有容量的队列，类似生产者消费模型，flusher处理不过来，队列出现阻塞。
日志聚合后flusher往下游写入时候网络不通，或者是网络不佳，然后导致数据堆积后logprocess queue队列无法再放入数据。
网络畅通，ilogtail的flusher和下游数据写入服务器（如：kafka、pulsar）之间存在sdk版本或者配置的不兼容，ilogtail flusher直接报错后无法处理采集的数据导致堆积。

View full answer

messixukejia · 2023-04-09T05:41:48Z

messixukejia
Apr 9, 2023
Maintainer

问题描述建议完整些，例如采集配置、流量等都需要给下。

2 replies

xjdszp Apr 10, 2023
Author

服务在做压测时，日志量比较大。ilogtial的logtail_plugin.LOG日志中会报logtail_plugin.LOG，
配置文件如下：
enable: true
inputs:

Type: file_log
LogPath: /home/logs
FilePattern: "*.log"
MaxDepth: 100
ContainerInfo:
K8sNamespaceRegex: "^(zxc-sit|zop-uat)$"
processors:
Type: processor_split_log_regex
SplitRegex: [\d+-\d+-\d+.|^[\d+-\d+-\d+|^\d+-\d+-\d+|[\d+-\d+-\d+\s\d+:\d+:\d+.\d+]\s.|\d+-\d+-\d+.|{"datetime".
SplitKey: content
PreserveOthers: true
Type: processor_add_fields
Fields:
cluster: sit
IgnoreIfExist: false
flushers:
Type: flusher_kafka
Brokers:
- 10.240.140.16:9092
- 10.240.140.17:9092
- 10.240.140.18:9092

messixukejia Apr 10, 2023
Maintainer

服务在做压测时，日志量比较大。这个需要定量给下每s流量，日志也给下样例吧。

wkai886 · 2023-04-27T09:14:51Z

wkai886
Apr 27, 2023

非压测场景下，监听了机器上的 container 去动态生成配置，每个 container 一个收集配置。每秒约2~3k 条日志，每台机器上约100 个左右配置。
采集一段时间之后大量报错 error:loggroup queue is full

单个配置如下：

enable: true
flushers:
- Type: flusher_pulsar
  URL: pulsar://10.1.1.1:6650
  Topic: persistent://public/xxxx-log/xxx
  Authentication:
    Token:
      Token: xxx
processors:
- Type: processor_add_fields
  Fields:
    logFormat: "xxxxxxx
  IgnoreIfExist: false
inputs:
- Type: file_log
  LogPath: /data/logs/xxxxxx/
  FilePattern: "xxxx_warn.log"
  ContainerInfo:
    IncludeK8sLabel:
      xxxx.com/app: xxxx-job-dubbo
    ExternalK8sLabelTag:
      xxxxx.com/app: app
      xxxxx.com/group: group
      xxxxx.com/system: system
    K8sPodRegex: xxxx-xxx-dubbo.qa-vgwls
    K8sContainerRegex: main

0 replies

wkai886 · 2023-04-27T09:32:20Z

wkai886
Apr 27, 2023

补充错误日志:
2023-04-27 09:31:33 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/5e5375ffdfa8797a5d60f863dc666d7d5112f1551279f35ac22ab8346062d488-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/5e5375ffdfa8797a5d60f863dc666d7d5112f1551279f35ac22ab8346062d488-wruqgsh.yaml
2023-04-27 09:31:33 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/49c2e293a5ffe51c89750f24d35acfa0c80d1e1c971edb5e13fe45f901b30c93-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/49c2e293a5ffe51c89750f24d35acfa0c80d1e1c971edb5e13fe45f901b30c93-wruqgsh.yaml
2023-04-27 09:31:34 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/f99f2e122444ed627355586379aa89de010028e5ca0ebd01726779ef4d321bda-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/f99f2e122444ed627355586379aa89de010028e5ca0ebd01726779ef4d321bda-wruqgsh.yaml
2023-04-27 09:31:34 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/1e76570a9bfb95007bcd2a4cae6d671a11fe8c805e9e4ee9a4b483ec2294e9d5-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/1e76570a9bfb95007bcd2a4cae6d671a11fe8c805e9e4ee9a4b483ec2294e9d5-wruqgsh.yaml
2023-04-27 09:31:34 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/77b2e3284f8ee9c436cbe8a9b1fd84787a449108fe866ef54a89298b479193aa-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/77b2e3284f8ee9c436cbe8a9b1fd84787a449108fe866ef54a89298b479193aa-wruqgsh.yaml
2023-04-27 09:31:34 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/a843bfaedd769074c0150ca1bac18c8576c42607c444e3c69ec5f8c887a5e824-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/a843bfaedd769074c0150ca1bac18c8576c42607c444e3c69ec5f8c887a5e824-wruqgsh.yaml
2023-04-27 09:31:34 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/5e5375ffdfa8797a5d60f863dc666d7d5112f1551279f35ac22ab8346062d488-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/5e5375ffdfa8797a5d60f863dc666d7d5112f1551279f35ac22ab8346062d488-wruqgsh.yaml
2023-04-27 09:31:35 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/49c2e293a5ffe51c89750f24d35acfa0c80d1e1c971edb5e13fe45f901b30c93-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/49c2e293a5ffe51c89750f24d35acfa0c80d1e1c971edb5e13fe45f901b30c93-wruqgsh.yaml
2023-04-27 09:31:35 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/f99f2e122444ed627355586379aa89de010028e5ca0ebd01726779ef4d321bda-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/f99f2e122444ed627355586379aa89de010028e5ca0ebd01726779ef4d321bda-wruqgsh.yaml
2023-04-27 09:31:35 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/1e76570a9bfb95007bcd2a4cae6d671a11fe8c805e9e4ee9a4b483ec2294e9d5-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/1e76570a9bfb95007bcd2a4cae6d671a11fe8c805e9e4ee9a4b483ec2294e9d5-wruqgsh.yaml
2023-04-27 09:31:35 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/77b2e3284f8ee9c436cbe8a9b1fd84787a449108fe866ef54a89298b479193aa-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/77b2e3284f8ee9c436cbe8a9b1fd84787a449108fe866ef54a89298b479193aa-wruqgsh.yaml
2023-04-27 09:31:35 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/a843bfaedd769074c0150ca1bac18c8576c42607c444e3c69ec5f8c887a5e824-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/a843bfaedd769074c0150ca1bac18c8576c42607c444e3c69ec5f8c887a5e824-wruqgsh.yaml
2023-04-27 09:31:35 [WRN] [plugin_runner_v1.go:251] [runProcessorInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/5e5375ffdfa8797a5d60f863dc666d7d5112f1551279f35ac22ab8346062d488-wruqgsh.yaml,] AlarmType:AGGREGATOR_ADD_ALARM error:loggroup queue is full logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/5e5375ffdfa8797a5d60f863dc666d7d5112f1551279f35ac22ab8346062d488-wruqgsh.yaml

1 reply

messixukejia Apr 27, 2023
Maintainer

完整的文件发下

jingliming · 2023-05-08T08:34:15Z

jingliming
May 8, 2023

我也遇到了类似的错误，不是压测，是生产环境，相关配置如下

集群环境

阿里云ACK

ilogtail config配置

enable: true
inputs:
  - Type: service_docker_stdout
    Stderr: true
    Stdout: true
    IncludeEnv:
      BW_LOGS_CATALINA: stdout
    ExternalEnvTag:
      BW_LOGS_CATALINA_TAGS: BW_LOGS_CATALINA_TAGS
processors:
  - Type: processor_add_fields
    Fields:
      logCollector: ilogtail
    IgnoreIfExist: false
  - Type: processor_regex
    SourceKey: BW_LOGS_CATALINA_TAGS
    Regex: topic=(\S+)\,tenantId=(\S+)\,clusterCode=(\S+)\,envType=(\S+)\,envCode=(\S+)\,appCode=(\S+)\,projectCode=(\S+)
    Keys:
      - topic
      - tenantId
      - clusterCode
      - envType
      - envCode
      - appCode
      - projectCode
flushers:
  - Type: flusher_stdout
    OnlyStdout: true
  - Type: flusher_kafka_v2
    Brokers:
      - 172.x.x.x:9092
      - 172.x.x.x:9092
      - 172.x.x.x.:9092
    Topic: bizworks-logging-trace

错误信息

2023-05-08 08:26:14 [ERR] [flusher_kafka_v2.go:254] [func1] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]      AlarmType:FLUSHER_FLUSH_ALARM   flush kafka write data fail, error:kafka: Failed to produce message to topic bizworks-logging-trace: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)     logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:14 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:15 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:16 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:17 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:18 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:19 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml
2023-05-08 08:26:20 [WRN] [logstore_config.go:319] [processInternal] [config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml,]     AlarmType:AGGREGATOR_ADD_ALARM  error:loggroup queue is full    logstore:       config:config#/usr/local/ilogtail/./user_yaml_config.d/stdout.yaml

0 replies

shalousun · 2023-05-08T09:52:19Z

shalousun
May 8, 2023
Collaborator

loggroup queue is full有三个可能的原因：

ilogtail采集了日志，日志聚合后flusher往下游写入数据时候性能根不上，logprocess queue是一个有容量的队列，类似生产者消费模型，flusher处理不过来，队列出现阻塞。
日志聚合后flusher往下游写入时候网络不通，或者是网络不佳，然后导致数据堆积后logprocess queue队列无法再放入数据。
网络畅通，ilogtail的flusher和下游数据写入服务器（如：kafka、pulsar）之间存在sdk版本或者配置的不兼容，ilogtail flusher直接报错后无法处理采集的数据导致堆积。

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QUESTION]:采集过程中插件报错error:loggroup queue is full #775

{{title}}

Replies: 5 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[QUESTION]:采集过程中插件报错error:loggroup queue is full #775

xjdszp Apr 7, 2023

Replies: 5 comments · 3 replies

messixukejia Apr 9, 2023 Maintainer

xjdszp Apr 10, 2023 Author

messixukejia Apr 10, 2023 Maintainer

wkai886 Apr 27, 2023

wkai886 Apr 27, 2023

messixukejia Apr 27, 2023 Maintainer

jingliming May 8, 2023

集群环境

ilogtail config配置

错误信息

shalousun May 8, 2023 Collaborator

xjdszp
Apr 7, 2023

Replies: 5 comments 3 replies

messixukejia
Apr 9, 2023
Maintainer

xjdszp Apr 10, 2023
Author

messixukejia Apr 10, 2023
Maintainer

wkai886
Apr 27, 2023

wkai886
Apr 27, 2023

messixukejia Apr 27, 2023
Maintainer

jingliming
May 8, 2023

shalousun
May 8, 2023
Collaborator