feat: reload pipeline config independently #1713

Abingcbc · 2024-08-22T07:53:22Z

背景

目前加载配置都是全量停止，全量重新加载和全量启动。这会造成一些没有变化的配置的重复加载。使得转发类和抓取类Input的流水线出现采集中断

方案

C++

重新实现了加载逻辑，不会在加载配置前停止所有流水线，按照如下的顺序，独立加载

新流水线创建
旧流水线停止
新流水线启动

Go

修改了CGO接口，提供了新的Go流水线加载接口，命名与C++侧统一。

StopAll：用于进程退出
Stop：停止指定配置
Start：启动指定配置

henryzhx8

配置变更的时候是先init新的，然后stop老的，再start新的，目前看代码里是不支持这种模式的

henryzhx8 · 2024-09-19T02:48:05Z

plugin_main/plugin_export.go

以下接口已经废弃，直接移除：ProcessRawLog、ProcessRawLogV2、ProcessLog（C++ observer代码有依赖，但是observer也会整体废除，所以可以不用管）

plugin_main/plugin_export.go

henryzhx8 · 2024-09-19T03:00:30Z

plugin_main/plugin_export.go

@@ -134,10 +133,6 @@ func LoadGlobalConfig(jsonStr string) int {
 //export LoadConfig
 func LoadConfig(project string, logstore string, configName string, logstoreKey int64, jsonStr string) int {


这个接口需要从长计议，logstorekey这个肯定不能要，但是Go流水线需要知道是不是要往C++发送，所以这个信息也是需要放到Go的GlobalConfig里的。flusher的IsReady函数需要进行修改，理论上不应该传logstorekey，而应该传configname

plugin_main/plugin_export.go

core/pipeline/Pipeline.cpp

henryzhx8 · 2024-09-19T07:56:55Z

core/pipeline/plugin/interface/Flusher.cpp

+    if (HasContext()) {
+        unique_ptr<SenderQueueItem> tombStone = make_unique<SenderQueueItem>("", 0, this, mQueueKey);
+        tombStone->mPipeline = PipelineManager::GetInstance()->FindConfigByName(mContext->GetConfigName());
+        if (!tombStone->mPipeline) {
+            LOG_ERROR(sLogger, ("failed to find pipeline when stop flusher", mContext->GetConfigName()));
+        }
+        SenderQueueManager::GetInstance()->PushQueue(mQueueKey, std::move(tombStone));
+    }


这里的实现肯定不对，要是这么简单当时就这么弄了，去看一下senderqueue的实现，想一想这么写能保证tombstone在所有item之后再pop吗

core/pipeline/Pipeline.h

core/pipeline/queue/ProcessQueueItem.h

core/pipeline/PipelineManager.cpp

plugin_main/plugin_export.go

pluginmanager/logstore_config.go

pluginmanager/plugin_manager.go

core/go_pipeline/LogtailPlugin.cpp

core/pipeline/plugin/interface/Flusher.cpp

core/pipeline/queue/ProcessQueueItem.h

core/pipeline/Pipeline.cpp

pluginmanager/plugin_manager.go

messixukejia · 2024-09-23T13:47:20Z

core/unittest/pipeline/PipelineUnittest.cpp

@@ -2841,6 +2853,29 @@ void PipelineUnittest::TestFlushBatch() const {
    }
 }

+void PipelineUnittest::TestInProcessingCount() const {


UT很弱，没有什么针对性的用例

plugin_main/plugin_export.go

core/pipeline/queue/BoundedProcessQueue.h

core/pipeline/queue/ExactlyOnceQueueManager.cpp

core/pipeline/queue/ProcessQueueManager.cpp

core/pipeline/queue/BoundedSenderQueueInterface.h

core/pipeline/queue/SenderQueue.cpp

core/pipeline/queue/ExactlyOnceSenderQueue.cpp

core/pipeline/queue/SenderQueueManager.cpp

core/pipeline/plugin/interface/Flusher.cpp

core/pipeline/PipelineManager.cpp

core/pipeline/queue/ExactlyOnceSenderQueue.cpp

core/pipeline/queue/SenderQueue.cpp

core/runner/ProcessorRunner.h

core/pipeline/PipelineManager.h

henryzhx8 · 2024-09-26T03:55:53Z

core/pipeline/PipelineManager.cpp

@@ -37,37 +36,36 @@ using namespace std;

 namespace logtail {

+PipelineManager::PipelineManager()
+    : mInputRunners({
+          PrometheusInputRunner::GetInstance(),


逗号的位置是不是应该在宏里面？

最后一个元素有逗号应该不影响

core/pipeline/PipelineManager.cpp

plugin_main/plugin_export.go

core/pipeline/PipelineManager.cpp

Abingcbc force-pushed the go-config branch from 075bdd7 to 529332f Compare August 22, 2024 08:11

Abingcbc changed the title ~~feat: reload Go pipeline config independently~~ feat: reload pipeline config independently Aug 26, 2024

henryzhx8 requested changes Aug 26, 2024

View reviewed changes

Abingcbc added 6 commits September 12, 2024 16:19

feat: reload Go pipeline config independently

9f501d7

fix unittest

f8d7a0a

fix unittest

750f732

fix

8f73fa7

fix

3496d95

fix

c5206af

Abingcbc force-pushed the go-config branch from 00ed039 to c5206af Compare September 12, 2024 08:20

fix

a6556f3

Abingcbc marked this pull request as draft September 12, 2024 17:13

Abingcbc force-pushed the go-config branch from a323173 to 1c5d08c Compare September 13, 2024 07:36

fix

5a3c10f

Abingcbc force-pushed the go-config branch from 1c5d08c to 5a3c10f Compare September 13, 2024 13:26

Abingcbc added 3 commits September 15, 2024 22:12

unittest

d8d4c5b

Merge remote-tracking branch 'alibaba/main' into go-config

96955d9

fix

62f3a35

Abingcbc marked this pull request as ready for review September 16, 2024 08:51

fix

d602845

Abingcbc marked this pull request as draft September 16, 2024 09:15

Abingcbc added 4 commits September 16, 2024 18:27

fix

8c1ad10

fix

fd478b7

fix

9ddb08d

self telemetry

409f575

Abingcbc marked this pull request as ready for review September 18, 2024 03:06

fix

9e76b8a

henryzhx8 requested changes Sep 19, 2024

View reviewed changes

Abingcbc added 2 commits September 19, 2024 18:08

Merge remote-tracking branch 'alibaba/main' into go-config

d307d95

fix

82baa18

Abingcbc added 8 commits September 20, 2024 15:58

Merge remote-tracking branch 'alibaba/main' into go-config

d8c28cd

fix

6442438

fix

fad1bfe

fix

0d9a2a7

fix

9f0424e

fix

4e022c5

fix

5155278

fix

0a05f02

henryzhx8 requested changes Sep 23, 2024

View reviewed changes

pluginmanager/plugin_manager.go Show resolved Hide resolved

messixukejia reviewed Sep 23, 2024

View reviewed changes

Abingcbc added 2 commits September 24, 2024 13:46

fix

b29b57f

fix

e770c59

henryzhx8 requested changes Sep 24, 2024

View reviewed changes

henryzhx8 reviewed Sep 24, 2024

View reviewed changes

core/pipeline/PipelineManager.cpp Show resolved Hide resolved

Abingcbc added 2 commits September 24, 2024 20:04

fix

5d46f61

fix

7714655

Abingcbc force-pushed the go-config branch from 17d47d0 to 7714655 Compare September 24, 2024 16:21

Merge remote-tracking branch 'alibaba/main' into go-config

f5c4bf4

Abingcbc force-pushed the go-config branch from 2c7a2a9 to f5c4bf4 Compare September 24, 2024 16:32

Abingcbc added 3 commits September 25, 2024 11:45

test: calculate incremental unittest coverage

bea03ba

fix

2bf161e

Merge remote-tracking branch 'origin/coverage' into go-config

1cda480

Abingcbc force-pushed the go-config branch from 1cda480 to 2bf161e Compare September 25, 2024 08:29

Merge remote-tracking branch 'alibaba/main' into go-config

98e14ec

henryzhx8 requested changes Sep 26, 2024

View reviewed changes

fix

7634196

henryzhx8 reviewed Sep 27, 2024

View reviewed changes

plugin_main/plugin_export.go Outdated Show resolved Hide resolved

plugin_main/plugin_export.go Outdated Show resolved Hide resolved

core/pipeline/PipelineManager.cpp Outdated Show resolved Hide resolved

fix

25b10d2

henryzhx8 approved these changes Sep 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: reload pipeline config independently #1713

feat: reload pipeline config independently #1713

Abingcbc commented Aug 22, 2024 •

edited

Loading

henryzhx8 left a comment

henryzhx8 Sep 19, 2024

henryzhx8 Sep 19, 2024

henryzhx8 Sep 19, 2024

messixukejia Sep 23, 2024

henryzhx8 Sep 26, 2024

Abingcbc Sep 26, 2024

		@@ -134,10 +133,6 @@ func LoadGlobalConfig(jsonStr string) int {
		//export LoadConfig
		func LoadConfig(project string, logstore string, configName string, logstoreKey int64, jsonStr string) int {

feat: reload pipeline config independently #1713

Are you sure you want to change the base?

feat: reload pipeline config independently #1713

Conversation

Abingcbc commented Aug 22, 2024 • edited Loading

背景

方案

C++

Go

henryzhx8 left a comment

Choose a reason for hiding this comment

henryzhx8 Sep 19, 2024

Choose a reason for hiding this comment

henryzhx8 Sep 19, 2024

Choose a reason for hiding this comment

henryzhx8 Sep 19, 2024

Choose a reason for hiding this comment

messixukejia Sep 23, 2024

Choose a reason for hiding this comment

henryzhx8 Sep 26, 2024

Choose a reason for hiding this comment

Abingcbc Sep 26, 2024

Choose a reason for hiding this comment

Abingcbc commented Aug 22, 2024 •

edited

Loading