Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanity Test for nodes running in k8s (BFT-394) #63

Merged
merged 151 commits into from
Mar 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
6bd6170
Add dockerfile for executable node
IAvecilla Jan 10, 2024
b2fb9ab
Add compose file for testing purposes
IAvecilla Jan 10, 2024
e768bc6
Add entrypoint for node consensus dockerfile
IAvecilla Jan 12, 2024
e6b6487
Add some comments and improve compose test file
IAvecilla Jan 12, 2024
4ca7fdc
Add new makefile commands to run dockerized consensus node
IAvecilla Jan 12, 2024
28df58b
Update readme with docs to run consensus node in docker
IAvecilla Jan 12, 2024
3e4b881
Delete unnecesary building dependency in compose file
IAvecilla Jan 15, 2024
febcafc
Rename docker image
IAvecilla Jan 15, 2024
b2e7e9c
Set container names manually in compose file
IAvecilla Jan 15, 2024
5e27e6e
Separate config directory for nodes running in docker
IAvecilla Jan 15, 2024
6e2236a
Fix node configuration for docker consensus example
IAvecilla Jan 15, 2024
810a4b5
Improve command to run a node in a container
IAvecilla Jan 15, 2024
4088c0a
Generate the node configs in release mode
IAvecilla Jan 15, 2024
4475489
Fix docker cleanup to force deletion
IAvecilla Jan 15, 2024
689b6d5
Remove unnecesary copies to container
IAvecilla Jan 15, 2024
99dc691
Add target dir to docker ignore
IAvecilla Jan 16, 2024
a3b270d
Move every docker related config file to the project root
IAvecilla Jan 16, 2024
f93890f
Fix typo in README
IAvecilla Jan 17, 2024
f3893cf
Fix typo in makefile command
IAvecilla Jan 17, 2024
1c721c8
Make the path to makefile be the same for local and docker
IAvecilla Jan 17, 2024
8cac863
disabled clipply lint
pompon0 Jan 17, 2024
d24c8bc
fixed lint, updated deps
pompon0 Jan 17, 2024
9520c4c
Change file name for the docker entrypoint and add comment to the script
IAvecilla Jan 18, 2024
54c72ce
Move version to latest for rust image
IAvecilla Jan 18, 2024
13e61e1
Change name of command and dir generation for node config
IAvecilla Jan 18, 2024
3b8d299
Add command to stop dockerized nodes
IAvecilla Jan 18, 2024
2d09c1d
Add example file with local address for node configuration
IAvecilla Jan 18, 2024
c287c56
Update README with new updates
IAvecilla Jan 18, 2024
e66cec9
Write a general overview on the README
IAvecilla Jan 18, 2024
eb7077e
Add tonic as dependency
IAvecilla Jan 19, 2024
058418e
Create test proto file with helathcheck endpoint
IAvecilla Jan 19, 2024
f323873
Add basic rpc server implementation
IAvecilla Jan 19, 2024
d0d9b13
Add basic rpc client implementation to use with CLI
IAvecilla Jan 19, 2024
a332327
Run RPC server for the standalone node
IAvecilla Jan 19, 2024
8e2d83c
Update docker config files to expose rpc server
IAvecilla Jan 19, 2024
96e18b7
Replace tonic crate for jsonrpsee
IAvecilla Jan 23, 2024
6b57f2a
Delete protobuff related files
IAvecilla Jan 23, 2024
c0aaa9b
Delete unnecesary RPC client
IAvecilla Jan 23, 2024
88a5b23
Add new jsonrpsee server and health endpoint
IAvecilla Jan 23, 2024
890bdee
Add inner docs for new rpc module
IAvecilla Jan 23, 2024
8f806bc
quick and dirty k8s deployment scripts for test framework
ElFantasma Jan 23, 2024
7592041
Merge branch 'main' into rpc_test_layer
IAvecilla Jan 23, 2024
23b8ea4
Merge branch 'rpc_test_layer' into k8s_deploy_script
IAvecilla Jan 23, 2024
438a2b8
Change name for generated config folders
IAvecilla Jan 24, 2024
19d7fcc
Update localnet config bin
IAvecilla Jan 24, 2024
b49af96
Delete old docker entrypoint file
IAvecilla Jan 24, 2024
2fd8061
Update k8s deployment manifest adding node env var
IAvecilla Jan 24, 2024
758ff9e
Add new stage in dockerfile to generate executor binary
IAvecilla Jan 25, 2024
addacc8
Build correct target in compose file
IAvecilla Jan 25, 2024
e263eb6
Update Makefile with all new targets
IAvecilla Jan 25, 2024
9503f05
Add addresses example file for docker config
IAvecilla Jan 25, 2024
8bcdb88
Remove docker config dir from clean command
IAvecilla Jan 25, 2024
2323211
Added some utility targets to Makefile
ElFantasma Jan 25, 2024
2720e74
Make rpc port a CLI argument for the executor
IAvecilla Jan 25, 2024
af1caaa
Change param type to run the server
IAvecilla Jan 25, 2024
77591f3
Merge branch 'rpc_test_layer' into k8s_deploy_script
IAvecilla Jan 25, 2024
24cddea
Add config generation for k8s node deployer
IAvecilla Jan 25, 2024
2cfaff4
Add makefile target to clean k8s config
IAvecilla Jan 25, 2024
4ed83cc
Improve config generation struct
IAvecilla Jan 26, 2024
00a008a
Use new config app functions to build localent and k8s config
IAvecilla Jan 26, 2024
4ce804b
Add kube as dependency for tools crate
IAvecilla Jan 26, 2024
b723715
Fix node config generation
IAvecilla Jan 29, 2024
83589a1
Add k8s-openapi as dependency
IAvecilla Jan 29, 2024
6248277
Add subcommands to generate config and deploy pods
IAvecilla Jan 29, 2024
87d9def
Override entrypoint to run node in k8s pod
IAvecilla Jan 29, 2024
251052f
Add makefile target to run k8s pods
IAvecilla Jan 29, 2024
09484fd
Merge branch 'main' into rpc_test_layer
IAvecilla Jan 30, 2024
ac145ee
Merge branch 'rpc_test_layer' into k8s_deploy_script
IAvecilla Jan 30, 2024
10ba6d6
Update nodes argument type to generate nodes config
IAvecilla Jan 30, 2024
991a8ac
Fix deployer config generation after merge
IAvecilla Jan 30, 2024
926b387
Fix clippy and format warnings
IAvecilla Jan 30, 2024
5fca6b8
Improve RPC server API to scale it easier
IAvecilla Jan 30, 2024
0726aff
Fix concurrency management for rpc server
IAvecilla Jan 30, 2024
21fb1c6
Add missing docs to solve clippy warnings
IAvecilla Jan 30, 2024
41ec845
Revert change for executor new flag
IAvecilla Jan 30, 2024
7ba8cb3
shutting down server when context is cancelled
pompon0 Jan 31, 2024
d859f39
Removed k8s yml files and modularized kube.rs access
ElFantasma Feb 1, 2024
95ec2b7
Update consensus docker example with new rpc ports
IAvecilla Feb 1, 2024
d697cc1
Add new endpoint to get peers of the node
IAvecilla Feb 1, 2024
7fe6ed9
Fix format and linter
IAvecilla Feb 1, 2024
57f600a
Update cargo.lock
IAvecilla Feb 1, 2024
8ffa0c0
Delete unnecesary features in added crates
IAvecilla Feb 1, 2024
44c979e
Added k8s module
ElFantasma Feb 1, 2024
5257592
Add old crate versions to deny.toml file
IAvecilla Feb 2, 2024
75cd2cf
Use new app config api for test in tools crate
IAvecilla Feb 2, 2024
958954f
Merge branch 'rpc_test_layer' into k8s_deploy_script
IAvecilla Feb 2, 2024
b0803dc
Deploying seed peers in a different phase
ElFantasma Feb 5, 2024
3a249dc
Retrieving seed peers IPs and using it in the rest of the peers
ElFantasma Feb 5, 2024
eb7512a
Add necessary features in kube and jsonrpsee dependencies
IAvecilla Feb 6, 2024
b587174
Deploy service to communicate with k8s pods
IAvecilla Feb 6, 2024
0f1e075
Add sanity test to check node is running in a pod
IAvecilla Feb 6, 2024
3ba75b1
Add service deployment to deployer binary
IAvecilla Feb 6, 2024
8e49847
Add bash script for setup and test run
IAvecilla Feb 6, 2024
ea6c3b9
Make executor argument optional
IAvecilla Feb 6, 2024
d612889
Busy waiting for pods to start to obtain their IPs
ElFantasma Feb 7, 2024
75b21ed
Add makefile targets to run tests inside kubernetes
IAvecilla Feb 8, 2024
5d599a8
Add new binary to run test scenarios
IAvecilla Feb 8, 2024
cf48dff
Add Dockerfile and kubernetes config files
IAvecilla Feb 8, 2024
7b5d264
Fix format
IAvecilla Feb 8, 2024
6bd5394
Revert change to have node ips as directory names for local config
IAvecilla Feb 8, 2024
1d58431
Making cli argument optional and forwarding arguments in entrypoint …
ElFantasma Feb 9, 2024
960ff13
Merge branch 'main' into k8s_deploy_script
ElFantasma Feb 9, 2024
c864f51
Fixed compiler warnings
ElFantasma Feb 9, 2024
b5b5ee6
Consensus node now can obtain it's public address from an ENV VAR. Al…
ElFantasma Feb 14, 2024
702a587
Added config RPC endpoint
ElFantasma Feb 15, 2024
025203f
Merge branch 'k8s_deploy_script' into k8s_node_communication_test
IAvecilla Feb 15, 2024
1ef6877
Corrected k8s_entrypoint.sh script to forward cli arguments
ElFantasma Feb 15, 2024
8b9d149
Update config files to run tester in k8s pod
IAvecilla Feb 15, 2024
0d1511d
Make tester a CLI tool to generate config and run the tests
IAvecilla Feb 15, 2024
91632d0
Update Makefile with new tester commands
IAvecilla Feb 15, 2024
b97e025
Add clap and tracing dependencies to the tester crate
IAvecilla Feb 15, 2024
1a2dc84
Remove service creation
IAvecilla Feb 15, 2024
aeae5fb
Ignore binaries for bin tools generated in docker
IAvecilla Feb 15, 2024
61fa5b7
Move makefile targets of tester to new makefile inside that crate
IAvecilla Feb 15, 2024
6480c23
Fix tester entrypoint to run the tests
IAvecilla Feb 15, 2024
636ab28
Add function to deploy pod for tests inside rust
IAvecilla Feb 15, 2024
a15b7bf
Update Dockerfile with new commands
IAvecilla Feb 15, 2024
1ae8e6b
Add readme to run tests pod
IAvecilla Feb 15, 2024
85b66d0
Delete outdated kubernetes yaml and dockerfile
IAvecilla Feb 15, 2024
5e8b011
Several corrections on naming and coding style
ElFantasma Feb 15, 2024
6ef448e
Fix missing documentation
IAvecilla Feb 16, 2024
4cf876f
Fix deny toml file
IAvecilla Feb 16, 2024
c62f0b4
Fixed deny.toml for kube.rs
ElFantasma Feb 16, 2024
2e89af8
Fixed clippy suggestions
ElFantasma Feb 19, 2024
8c2f79b
Merge branch 'k8s_deploy_script' into k8s_node_communication_test
IAvecilla Feb 19, 2024
fb0c345
Added amount of seed nodes as cli argument
ElFantasma Feb 19, 2024
3b1256f
Merge branch 'main' into k8s_deploy_script
ElFantasma Feb 19, 2024
23a7761
Added brief description for k8s deployment in README.md
ElFantasma Feb 20, 2024
7dbf270
Unified node_name and node_id values
ElFantasma Feb 20, 2024
7f489fe
Merge branch 'k8s_deploy_script' into k8s_node_communication_test
IAvecilla Feb 21, 2024
9b309c4
Remove unwraps and improve error handling
IAvecilla Feb 21, 2024
f15d2bf
Change hardcoded deployment json to deployment strcut
IAvecilla Feb 21, 2024
04f1a2f
Remove unwraps in k8s module
IAvecilla Feb 21, 2024
93824c0
Add anyhow as dependency for tests crate
IAvecilla Feb 21, 2024
be03154
Get pods from consensus docker image
IAvecilla Feb 22, 2024
7e4c33d
Improve error logging for the tester
IAvecilla Feb 22, 2024
573d238
Do general cleanup of k8s new functions
IAvecilla Feb 22, 2024
abab39a
Use of filter_map and add context to errors
IAvecilla Feb 22, 2024
9bf46f5
Clean test-suite image in the clean target
IAvecilla Feb 22, 2024
37c74eb
Add doc comment explaining the unwraps usage
IAvecilla Feb 22, 2024
8b6f54d
Remove unnecesary clones
IAvecilla Feb 22, 2024
65e5295
Fix config generation for tester
IAvecilla Feb 23, 2024
5178b7d
Fix some minor suggestions
IAvecilla Feb 26, 2024
3803b8a
Change back the filter and map methods to use Results
IAvecilla Feb 26, 2024
fb791b7
Use a unique for loop to get nodes address and delete map
IAvecilla Feb 27, 2024
5b29f13
Merge branch 'main' into k8s_node_communication_test
IAvecilla Feb 27, 2024
f49f306
Fix image name duplicated after merge
IAvecilla Feb 27, 2024
6b3e15c
Merge branch 'main' into k8s_node_communication_test
IAvecilla Feb 29, 2024
02e5898
Merge branch 'main' into k8s_node_communication_test
IAvecilla Mar 7, 2024
eac0682
Avoid cloning and use refs instead for all kube structures
IAvecilla Mar 7, 2024
fae944f
Update mio crate
IAvecilla Mar 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@ logs/
.terraform
.ssh
**/*terraform.tfstate*

# Binaries generated in Docker
node/tools/docker_binaries
27 changes: 21 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
# Build Stage
FROM rust:latest as builder
COPY /node/ /node/
WORKDIR /node
COPY /node/ /app/
WORKDIR /app
RUN apt-get update && apt-get install -y libclang-dev
RUN cargo build --release

# Binary copy stage
FROM scratch as binary
COPY --from=builder /node/target/release/executor .
FROM scratch as executor-binary
COPY --from=builder /app/target/release/executor .

# Binary copy stage
FROM scratch as tester-binary
COPY --from=builder /app/target/release/tester .

# Runtime Stage
FROM debian:stable-slim as runtime
# Executor runtime Stage
FROM debian:stable-slim as executor-runtime

COPY /node/tools/docker_binaries/executor /node/
COPY /node/tools/k8s_configs/ /node/k8s_config
Expand All @@ -26,3 +30,14 @@ ENTRYPOINT ["./docker-entrypoint.sh"]

EXPOSE 3054
EXPOSE 3051

# Tester runtime Stage
FROM debian:stable-slim as tester-runtime
COPY node/tools/docker_binaries/tester /test/
COPY node/tests/tester_entrypoint.sh /test/
COPY node/tests/config.txt /test/

WORKDIR /test

RUN chmod +x tester_entrypoint.sh

9 changes: 5 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ nodes_config:
# Docker commands

docker_build_executor:
docker build --output=node/tools/docker_binaries --target=binary .
docker build --output=node/tools/docker_binaries --target=executor-binary .

docker_node_image:
docker build -t consensus-node --target=runtime .
docker build -t consensus-node --target=executor-runtime .

docker_nodes_config:
cd ${EXECUTABLE_NODE_DIR} && cargo run --release --bin localnet_config -- --input-addrs docker-config/addresses.txt --output-dir docker-config
cd ${EXECUTABLE_NODE_DIR} && cargo run --bin localnet_config -- --input-addrs docker-config/addresses.txt --output-dir docker-config

docker_node:
$(MAKE) docker_node_image
Expand All @@ -39,7 +39,7 @@ stop_docker_nodes:
docker stop consensus-node-1 consensus-node-2

start_k8s_nodes:
cd ${EXECUTABLE_NODE_DIR} && cargo run --release --bin deployer generate-config --nodes ${NODES}
cd ${EXECUTABLE_NODE_DIR} && cargo run --bin deployer generate-config --nodes ${NODES}
$(MAKE) docker_node_image
minikube image load consensus-node:latest
cd ${EXECUTABLE_NODE_DIR} && cargo run --release --bin deployer deploy --nodes ${NODES} --seed-nodes ${SEED_NODES}
Expand All @@ -60,6 +60,7 @@ clean_docker:
docker rm -f consensus-node-2
docker network rm -f node-net
docker image rm -f consensus-node
docker image rm -f test-suite

addresses_file:
mkdir -p ${EXECUTABLE_NODE_DIR}/docker-config
Expand Down
Loading
Loading