Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Add parallel writer support #738

Closed
wants to merge 1 commit into from
Closed

Conversation

suxb201
Copy link
Member

@suxb201 suxb201 commented Dec 22, 2023

shake 运行环境:

  • ecs.c8i.4xlarge(Intel(R) Xeon(R) Platinum 8475B)
  • 云盘、网络均不构成瓶颈

测试 Redis 均为阿里云云数据库 Redis 云原生版。


源端为 Redis 7.0 集群,64 分片

  • 目的端为 Redis 7.0 单实例,ops 为 280k
  • 目的端为 Redis 7.0 集群,64 分片,ops 为 300k

从上述可以看出,理论上的 Redis 7.0 集群的写性能应该可以做到 280*64k,但受限于代码实例没有做到。

应用此 PR 后:

  • 目的端为 Redis 7.0 集群,64 分片,ops 为 600k

提升不明显,且此修改较丑不准备合入。


在高 ops 场景下代码中有多处均构成瓶颈,无法通过简单修改实现。

@Zheaoli
Copy link
Collaborator

Zheaoli commented Dec 22, 2023

我准备先把 Parse 还有 goroutine 之间数据传输的瓶颈解决了再来处理下并行 writer

@suxb201 suxb201 marked this pull request as draft January 3, 2024 10:59
@suxb201 suxb201 mentioned this pull request May 20, 2024
@suxb201 suxb201 closed this Oct 17, 2024
@suxb201 suxb201 deleted the feature-parallel-writer branch October 24, 2024 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants