-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add degradation tests into CI #1331
Open
AKorpusenko
wants to merge
35
commits into
stage
Choose a base branch
from
feat/add-degradation-tests-into-ci
base: stage
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 29 commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
1569672
init commit
AKorpusenko 919fca8
added degradation tests and benching scripts
AKorpusenko 9dc6356
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko a0841c3
added config for deltas secifying
AKorpusenko f951faa
minor fixes
AKorpusenko 3b6366f
added workflow
AKorpusenko 13cf9bc
workflow fix
AKorpusenko 39b9793
added make to ci
AKorpusenko 1a447fd
fixed makefile degradation-test
AKorpusenko 6175cec
added logging the benchstat to workflow
AKorpusenko 2ae158d
added benchstat
AKorpusenko b82fae5
added default behavior
AKorpusenko e527a78
added readme, minor changes, update benchmarks script
AKorpusenko c3c7aa8
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko 2a8bf6a
minor pr review comments fixes
AKorpusenko 49d6528
Merge branch 'feat/add-degradation-tests-into-ci' of github.com:bloxa…
AKorpusenko 3081814
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko 77f5b70
added -format csv to benchstat. added basic test
AKorpusenko 60d5757
added manual button update the benchmarks
AKorpusenko 22ec9e8
update workflows
AKorpusenko c81ae4e
added manual benchmarks update
AKorpusenko 7a8c12d
added benchstat script some tests. Removed outdated code and improved…
AKorpusenko f31259b
updated README.md for degradation-tester
AKorpusenko 03ed165
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko f9794d9
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko 888a7f8
updated to support current op keys incapsulation
AKorpusenko 1515b4d
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko 2b5ec08
changed trigger of manual benchmarks updating
AKorpusenko b114c82
skip b/op section. changed default alloc/op thereshold
AKorpusenko 895ab32
Merge branch 'stage' of github.com:bloxapp/ssv into feat/add-degradat…
AKorpusenko 11cef51
added new benchmarking tests
AKorpusenko f352802
updated message validation benchmark
AKorpusenko 2f6b37f
updated benchmarks
AKorpusenko b822d9c
fixed degradation test script
AKorpusenko 14b6064
added compact mode
AKorpusenko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
name: Benchmarks Degradation Test | ||
|
||
on: | ||
push: | ||
branches: | ||
- "**" | ||
|
||
jobs: | ||
performance-degradation-check: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Check out code | ||
uses: actions/checkout@v3 | ||
|
||
- name: Setup make | ||
run: sudo apt-get update && sudo apt-get install make | ||
|
||
- name: Set up Go | ||
uses: actions/setup-go@v4 | ||
with: | ||
go-version: "1.20.x" | ||
|
||
- name: Tidy up dependencies | ||
run: go mod tidy | ||
|
||
- name: Install yq | ||
run: sudo snap install yq | ||
|
||
- name: Install benchstat | ||
run: go install golang.org/x/perf/cmd/benchstat@latest | ||
|
||
- name: Build degradation tester tool | ||
run: | | ||
cd ./scripts/degradation-tester | ||
go install . | ||
|
||
- name: Run degradation-test | ||
run: | | ||
make degradation-test | ||
env: | ||
GO111MODULE: on |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
name: Manual Benchmarks Update | ||
|
||
on: workflow_dispatch | ||
|
||
jobs: | ||
optional-benchmark-update: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Check out code | ||
uses: actions/checkout@v4 | ||
with: | ||
ref: ${{ github.event.pull_request.head.sha }} # Ensures the PR's head is checked out | ||
|
||
- name: Setup make | ||
run: sudo apt-get update && sudo apt-get install make | ||
|
||
- name: Set up Go | ||
uses: actions/setup-go@v4 | ||
with: | ||
go-version: "1.20.x" | ||
|
||
- name: Tidy up dependencies | ||
run: go mod tidy | ||
|
||
- name: Update benchmarks | ||
if: github.event_name == 'workflow_dispatch' | ||
run: make update-benchmarks | ||
|
||
- name: Update repo | ||
uses: EndBug/[email protected] | ||
with: | ||
# limit the scope of the commit with the benchmarks dir path only | ||
add: "scripts/degradation-tester/benchmarks" | ||
default_author: github_actor | ||
pathspec_error_handling: exitImmediately | ||
message: "benchmarks updated" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
package validation | ||
|
||
import ( | ||
"crypto" | ||
nkryuchkov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
crand "crypto/rand" | ||
"crypto/rsa" | ||
"crypto/sha256" | ||
"testing" | ||
|
||
eth2apiv1 "github.com/attestantio/go-eth2-client/api/v1" | ||
specqbft "github.com/bloxapp/ssv-spec/qbft" | ||
spectypes "github.com/bloxapp/ssv-spec/types" | ||
spectestingutils "github.com/bloxapp/ssv-spec/types/testingutils" | ||
"github.com/bloxapp/ssv/network/commons" | ||
"github.com/bloxapp/ssv/networkconfig" | ||
"github.com/bloxapp/ssv/operator/storage" | ||
beaconprotocol "github.com/bloxapp/ssv/protocol/v2/blockchain/beacon" | ||
ssvtypes "github.com/bloxapp/ssv/protocol/v2/types" | ||
registrystorage "github.com/bloxapp/ssv/registry/storage" | ||
"github.com/bloxapp/ssv/storage/basedb" | ||
"github.com/bloxapp/ssv/storage/kv" | ||
"github.com/bloxapp/ssv/utils/rsaencryption" | ||
"github.com/ethereum/go-ethereum/common" | ||
pubsub "github.com/libp2p/go-libp2p-pubsub" | ||
pspb "github.com/libp2p/go-libp2p-pubsub/pb" | ||
"github.com/stretchr/testify/require" | ||
"go.uber.org/zap/zaptest" | ||
) | ||
|
||
func BenchmarkVerifyRSASignature(b *testing.B) { | ||
logger := zaptest.NewLogger(b) | ||
db, err := kv.NewInMemory(logger, basedb.Options{}) | ||
require.NoError(b, err) | ||
|
||
ns, err := storage.NewNodeStorage(logger, db) | ||
require.NoError(b, err) | ||
|
||
const validatorIndex = 123 | ||
const operatorID = spectypes.OperatorID(1) | ||
|
||
ks := spectestingutils.Testing4SharesSet() | ||
share := &ssvtypes.SSVShare{ | ||
Share: *spectestingutils.TestingShare(ks), | ||
Metadata: ssvtypes.Metadata{ | ||
BeaconMetadata: &beaconprotocol.ValidatorMetadata{ | ||
Status: eth2apiv1.ValidatorStateActiveOngoing, | ||
Index: validatorIndex, | ||
}, | ||
Liquidated: false, | ||
}, | ||
} | ||
err = ns.Shares().Save(nil, share) | ||
require.NoError(b, err) | ||
|
||
netCfg := networkconfig.TestNetwork | ||
|
||
roleAttester := spectypes.BNRoleAttester | ||
|
||
mv := NewMessageValidator(netCfg, WithNodeStorage(ns)).(*messageValidator) | ||
|
||
slot := netCfg.Beacon.FirstSlotAtEpoch(123456789) | ||
|
||
validSignedMessage := spectestingutils.TestingProposalMessageWithHeight(ks.Shares[1], 1, specqbft.Height(slot)) | ||
|
||
encoded, err := validSignedMessage.Encode() | ||
require.NoError(b, err) | ||
|
||
message := &spectypes.SSVMessage{ | ||
MsgType: spectypes.SSVConsensusMsgType, | ||
MsgID: spectypes.NewMsgID(netCfg.Domain, share.ValidatorPubKey, roleAttester), | ||
Data: encoded, | ||
} | ||
|
||
encodedMsg, err := commons.EncodeNetworkMsg(message) | ||
require.NoError(b, err) | ||
|
||
hash := sha256.Sum256(encodedMsg) | ||
privateKey, err := rsa.GenerateKey(crand.Reader, 2048) | ||
require.NoError(b, err) | ||
pubkey := privateKey.Public().(*rsa.PublicKey) | ||
|
||
pubKey, err := rsaencryption.ExtractPublicKey(pubkey) | ||
require.NoError(b, err) | ||
|
||
od := ®istrystorage.OperatorData{ | ||
ID: operatorID, | ||
PublicKey: []byte(pubKey), | ||
OwnerAddress: common.Address{}, | ||
} | ||
|
||
found, err := ns.SaveOperatorData(nil, od) | ||
require.NoError(b, err) | ||
|
||
require.False(b, found) | ||
|
||
signature, err := rsa.SignPKCS1v15(crand.Reader, privateKey, crypto.SHA256, hash[:]) | ||
require.NoError(b, err) | ||
|
||
encodedMsg = commons.EncodeSignedSSVMessage(encodedMsg, operatorID, signature) | ||
|
||
topicID := commons.ValidatorTopicID(message.GetID().GetPubKey()) | ||
pMsg := &pubsub.Message{ | ||
Message: &pspb.Message{ | ||
Topic: &topicID[0], | ||
Data: encodedMsg, | ||
}, | ||
} | ||
|
||
messageData := pMsg.GetData() | ||
decMessageData, operatorIDX, signature, err := commons.DecodeSignedSSVMessage(messageData) | ||
require.NoError(b, err) | ||
|
||
messageData = decMessageData | ||
|
||
b.ResetTimer() | ||
for i := 0; i < b.N; i++ { | ||
err := mv.verifySignature(messageData, operatorIDX, signature) | ||
require.NoError(b, err) | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
# Degradation Tester | ||
|
||
### Continuous Benchmarking | ||
|
||
Degradation Tester is a tool for detecting performance degradation. It allows to compare current benchmark results against previous ones to identify potential performance issues. | ||
It's important to use the same environment and CPU for testing. We use the Github Actions workflow for the benchmarks, which currently uses the `AMD EPYC 7763 64-Core Processor`. | ||
|
||
### Approving Performance Changes | ||
|
||
If you've made changes that affect performance, whether they are improvements or degradations, it's essential to regenerate the benchmarks and replace the existing files in the `benchmarks` directory. This ensures that future comparisons accurately reflect the impact of your changes and maintains the integrity of the performance testing process. | ||
|
||
To regenerate and update the benchmarks, use the `update-benchmarks` command in the Makefile. To update the benchmarks in the repo using Github Actions, you should use the manually triggered workflow for your Pull Request named **Manual Benchmarks Update**. | ||
|
||
To run the workflow: | ||
- Open the PR | ||
- Go to Actions | ||
- Choose `Manual Benchmarks Update` from the list of available actions | ||
- Click `Run workflow` and choose your branch from the dropdown menu | ||
|
||
The workflow is located in `.github/workflows/manual-benchmarks-update-.yml`. | ||
|
||
## Configuration | ||
|
||
To use the tool, you need to set up a YAML configuration file. Here is an example configuration: | ||
|
||
```yaml | ||
DefaultOpDelta: 4.0 | ||
DefaultAllocDelta: 0 | ||
Packages: | ||
- Path: "./message/validation" | ||
Tests: | ||
- Name: "VerifyRSASignature" | ||
OpDelta: 3.0 | ||
- Path: "./protocol/v2/types" | ||
Tests: | ||
- Name: "VerifyBLS" | ||
OpDelta: 6.0 | ||
- Name: "VerifyPKCS1v15" | ||
OpDelta: 4.0 | ||
- Name: "VerifyPKCS1v15FastHash" | ||
OpDelta: 6.0 | ||
- Name: "VerifyPSS" | ||
OpDelta: 5.0 | ||
``` | ||
|
||
- `DefaultOpDelta` and `DefaultAllocDelta` specify the default performance change thresholds (in percentages) that are allowed for all tests unless specific values are provided. | ||
- `Path` refers to the directory containing the tests. | ||
- `Tests` is a list of tests with names and permissible performance change thresholds. | ||
|
||
## Local Usage | ||
|
||
### Dependecies | ||
|
||
- `yq` | ||
- `benchstat` | ||
|
||
### Installation | ||
|
||
Install the benchmarks degradation checker tool: | ||
|
||
`cd scripts/degradation-tester/ && go install .` | ||
|
||
### Running Degradation Tests | ||
|
||
To run the degradation tests, execute: | ||
|
||
```bash | ||
make degradation-test | ||
``` | ||
|
||
### Updating Benchmarks | ||
|
||
To replace the benchmarks with your local ones, execute: | ||
|
||
```bash | ||
make update-benchmarks | ||
``` | ||
|
||
## Usage with GitHub Actions | ||
|
||
`.github/workflows/degradation-test.yml` containing a workflow for automatically running degradation tests as part of the CI/CD process | ||
|
||
`.github/workflows/manual-benchmarks-update-.yml`. containing a workflow for updating the benchmarks with manual triggering | ||
|
||
## Scripts | ||
|
||
The `./scripts/degradation-tester` directory contains the following scripts: | ||
|
||
- `degradation-check.sh` - running degradation tests. Runs benchmarks, saves the new results, compares benchmarking results per all listed packages in `config.yaml` | ||
|
||
- `update-benchmarks.sh` - A script for updating benchmarks |
16 changes: 16 additions & 0 deletions
16
scripts/degradation-tester/benchmarks/message_validation_benchmarks_old.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
goos: linux | ||
goarch: amd64 | ||
pkg: github.com/bloxapp/ssv/message/validation | ||
cpu: AMD EPYC 7763 64-Core Processor | ||
BenchmarkVerifyRSASignature-4 4729 256788 ns/op 1364 B/op 7 allocs/op | ||
BenchmarkVerifyRSASignature-4 4712 254533 ns/op 1469 B/op 8 allocs/op | ||
BenchmarkVerifyRSASignature-4 4722 254782 ns/op 1577 B/op 8 allocs/op | ||
BenchmarkVerifyRSASignature-4 4735 255405 ns/op 1685 B/op 9 allocs/op | ||
BenchmarkVerifyRSASignature-4 4573 258438 ns/op 1804 B/op 9 allocs/op | ||
BenchmarkVerifyRSASignature-4 4694 265556 ns/op 1940 B/op 10 allocs/op | ||
BenchmarkVerifyRSASignature-4 4491 259423 ns/op 2034 B/op 10 allocs/op | ||
BenchmarkVerifyRSASignature-4 4677 255195 ns/op 2129 B/op 10 allocs/op | ||
BenchmarkVerifyRSASignature-4 4364 256322 ns/op 2246 B/op 11 allocs/op | ||
BenchmarkVerifyRSASignature-4 4694 255884 ns/op 2350 B/op 11 allocs/op | ||
PASS | ||
ok github.com/bloxapp/ssv/message/validation 35.074s |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test/benchmark needs to be updated after #1274 and #1358
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated