Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Github runner add benchmark #1599

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Github runner add benchmark #1599

wants to merge 16 commits into from
+1,132 −44

Conversation

Assassin718
Copy link

  • add ilogtail file-to-blackhole performance test

@CLAassistant
Copy link

CLAassistant commented Jul 10, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@Assassin718 Assassin718 reopened this Jul 10, 2024
@Abingcbc Abingcbc marked this pull request as draft July 10, 2024 09:16
@Abingcbc Abingcbc closed this Jul 10, 2024
@Abingcbc Abingcbc reopened this Jul 10, 2024
@Assassin718 Assassin718 marked this pull request as ready for review July 11, 2024 02:16
@Assassin718 Assassin718 force-pushed the e2e-performance-test branch 10 times, most recently from 09dc837 to 73af851 Compare July 17, 2024 12:47
@Assassin718 Assassin718 force-pushed the e2e-performance-test branch 2 times, most recently from f269180 to cb5545d Compare July 21, 2024 08:37
@Abingcbc Abingcbc changed the title WIP: e2e performance test add benchmark compared with other agent Jul 23, 2024
@Abingcbc Abingcbc changed the title add benchmark compared with other agent add benchmark with other agents Jul 23, 2024
@Assassin718
Copy link
Author

新增

  • 为test/engine新增了benchmark docker-compose启动流程：基于e2e docker-compose启动流程做出修改，去除掉默认启动ilogtail、goc-server容器，新增默认启动cadvisor容器（可以在benchmark时监控容器资源占用）
  • 为test/engine新增了setup/monitor：通过访问cadvisor容器，记录benchmark过程中对应容器的资源占用（目前主要记录了CPU、内存占用的最大值和平均值），统计数据，在benchmark结束后将数据输出到文件中
  • 新增benchmark testcases：添加了ilogtail、filebeat、fluentbit、vector在速率为10MB/s的文件输入、json解析的benchmark testcase，为避免下游接收程序对吞吐量的影响，在flusher侧设置了抛弃所有事件
  • benmark CI workflow：在pull request时触发，执行所有benchmark testcase后记录数据，在action中输出benmark结果（待完成benmark结果图形可视化）

@Assassin718 Assassin718 changed the title add benchmark with other agents [WIP] Github runner add benchmark Jul 24, 2024
Abingcbc
Abingcbc requested changes Jul 25, 2024
View reviewed changes
.github/workflows/benchmark.yaml Outdated Show resolved Hide resolved
test/benchmark/benchmark_docker_test.go Show resolved Hide resolved
test/benchmark/test_cases/performance_file_to_blackhole_filebeat/case.feature Outdated
Given {docker-compose} environment
Given docker-compose type {benchmark}
When start docker-compose {performance_file_to_blackhole_filebeat}
When start monitor {e2e-filebeat-1}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这块可以不填具体的容器名吗？比如填个正则，匹配一个容器

@Abingcbc Abingcbc changed the title [WIP] Github runner add benchmark Github runner add benchmark Jul 25, 2024
messixukejia
messixukejia requested changes Jul 26, 2024
View reviewed changes
test/benchmark/test_cases/performance_file_to_blackhole_filebeat/filebeat.yml Show resolved Hide resolved
test/engine/trigger/file.go Outdated

import (
"context"
"fmt"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Abingcbc 这个生成测试文件的方法没有通用的？还需要新建一个？

scripts/benchmark_collect_result.sh
@@ -0,0 +1,24 @@
#!/bin/bash

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

增加下文档说明吧
1、如何使用
2、效果怎么看

@Assassin718 Assassin718 force-pushed the e2e-performance-test branch 2 times, most recently from 4828481 to 72180b5 Compare July 29, 2024 03:41
yyuuttaaoo
yyuuttaaoo requested changes Aug 19, 2024
View reviewed changes
docs/cn/developer-guide/test/benchmark.md Outdated
- `@e2e-performance @docker-compose`: 表示测试场景为e2e-performance，测试场景由本地docker-compose运行
- `Given {docker-compose} environment`: 配置启动测试环境，以docker-compose环境启动测试
- `Given docker-compose type {benchmark}`: 配置docker-compose启动模式，以benchmark模式启动docker-compose，`{}`中参数有两种选项，`e2e`/`benchmark`。以`e2e`模式启动会默认启动ilogtail、goc-server容器，用作e2e测试；以`benchmark`模式启动会默认启动cadvisor容器，用于监控容器运行过程中的资源占用；若在配置文件中不配置该参数，则默认以上一个scenario的启动模式启动。
- `When start docker-compose {scenario directory}`: `{}`中参数为当前scenario的文件夹名，该行动作会读取`scenario director`文件夹下的docker-compose.yaml文件，通过docker-compose命令启动所有容器
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

directory

test/benchmark/test_cases/performance_file_to_blackhole_vector/vector.yaml Outdated
- transform_json
condition:
type: "vrl"
source: ."usr-agent" == "no-agent"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

user-agent?

test/engine/setup/dockercompose/common.go Outdated
mu.Lock()
defer mu.Unlock()
instance = NewComposeBooter()
switch loadType {
case "e2e":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

抽取为常量

test/engine/setup/dockercompose/compose benchmark.go Outdated
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

文件名用下划线吧

yyuuttaaoo
yyuuttaaoo requested changes Aug 19, 2024
View reviewed changes
scripts/benchmark_collect_result.sh
# Combine the statistic and records files from each benchmark run into a single file

# Define the input files and the output file
input_statistic_files=($(find test/benchmark/report -type f -name '*ilogtail_statistic.json'))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*_statistic.json?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里主要是因为其他agent也会生成文件名以{agentname}_statistic.json为后缀的测试统计结果，而benchmark action只需要关注文件名以ilogtail_statistic.json为后缀的测试统计结果

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yyuuttaaoo yyuuttaaoo yyuuttaaoo requested changes

@messixukejia messixukejia Awaiting requested review from messixukejia

@Abingcbc Abingcbc Awaiting requested review from Abingcbc

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Assassin718 @CLAassistant @yyuuttaaoo @messixukejia @Abingcbc