Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 391: Update Pravega library versions and deployment script #392

Merged
merged 3 commits into from
Sep 26, 2023
Merged

Issue 391: Update Pravega library versions and deployment script #392

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0b1ba0c
Updates, fixes, and improvements in Pravega driver.
RaulGracia Sep 6, 2023
d298309
Revert Zookeeper version upgrade.
RaulGracia Sep 6, 2023
139e3cb
Restore Bookkeeper version.
RaulGracia Sep 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions benchmark-framework/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,11 +71,6 @@
<artifactId>driver-nats</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>driver-nats-streaming</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>driver-nsq</artifactId>
Expand Down Expand Up @@ -145,6 +140,12 @@
<artifactId>HdrHistogram</artifactId>
<version>2.1.12</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>driver-nats-streaming</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this change intended?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @merlimat, yes this change is intended. I tried to explain the justification in the PR description:
Set scope provided to driver-nats-streaming. The reason for doing that is that driver-nats-streaming is transitively importing a very old version of Google Protobuf (3.9.1 - Aug 06, 2019). Pravega also uses Google Protobuf. When upgrading Pravega, we found a NoSuchMethodError related to Google Protobuf and the reason was that the newer Pravega version are compiled with a more recent Google Protobuf version with incompatible changes (since version 3.17.2). Having both Google Protobuf versions in the classpath leads to runtime errors, so I decided to prevent the oldest Google Protobuf version to be transitively imported by driver-nats-streaming. Given that NATS Streaming has been deprecated, I feel that it is a safe action to do (which can precede the removal of NATS Streaming until NATS JetStream is available).

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
Expand Down
24 changes: 13 additions & 11 deletions driver-pravega/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,15 @@ check [how to build Pravega](doc/build_pravega.md).

# DEPLOY A PRAVEGA CLUSTER ON AMAZON WEB SERVICES

You can deploy a Pravega cluster on AWS (for benchmarking purposes) using [Terraform 0.12.20](https://www.terraform.io/) and [Ansible 2.8.5](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html).
You’ll need to have both of those tools installed as well as the `terraform-inventory` [plugin](https://github.com/adammck/terraform-inventory) for Terraform.
You can deploy a Pravega cluster on AWS (for benchmarking purposes) using [Terraform 0.12.20](https://www.terraform.io/)
and [Ansible 2.8.5](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) (with a
a version of `Jinja2=<3.0.3`).
You’ll need to have both tools installed, as well as the `terraform-inventory` [plugin](https://github.com/adammck/terraform-inventory) for Terraform.

You also need to install an Ansible modules to support metrics.
You also need to install an Ansible modules to support metrics:

```
ansible-galaxy install cloudalchemy.node-exporter
ansible-galaxy install cloudalchemy.node_exporter
```

In addition, you’ll need to:
Expand Down Expand Up @@ -74,7 +76,7 @@ This will install the following [EC2](https://aws.amazon.com/ec2) instances (plu
| Resource | Description | Count |
|----------------------|-------------------------------------------------------------|-------|
| Controller instances | The VMs on which a Pravega controller will run | 1 |
| Bookkeeper instances | The VMs on which a Bookkeeper and Segmentstore will run | 3 |
| Bookkeeper instances | The VMs on which a Bookkeeper and Segment Store will run | 3 |
| ZooKeeper instances | The VMs on which a ZooKeeper node will run | 3 |
| Client instance | The VM from which the benchmarking suite itself will be run | 2 |

Expand All @@ -84,12 +86,12 @@ When you run `terraform apply`, you will be prompted to type `yes`. Type `yes` t

There’s a handful of configurable parameters related to the Terraform deployment that you can alter by modifying the defaults in the `terraform.tfvars` file.

| Variable | Description | Default |
|-------------------|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `region` | The AWS region in which the Pravega cluster will be deployed | `us-west-2` |
| `public_key_path` | The path to the SSH public key that you’ve generated | `~/.ssh/pravega_aws.pub` |
| `ami` | The [Amazon Machine Image (AWI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) to be used by the cluster’s machines | `ami-9fa343e7` |
| `instance_types` | The EC2 instance types used by the various components | `i3.4xlarge` (BookKeeper bookies), `m5.large`(Controller), `t3.small` (ZooKeeper), `c5.4xlarge` (benchmarking client) |
| Variable | Description | Default |
|-------------------|--------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| `region` | The AWS region in which the Pravega cluster will be deployed | `us-east-2` |
| `public_key_path` | The path to the SSH public key that you’ve generated | `~/.ssh/pravega_aws.pub` |
| `ami` | The [Amazon Machine Image (AWI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) to be used by the cluster’s machines | `ami-0bb2449c2217cb9b0` |
| `instance_types` | The EC2 instance types used by the various components | `i3en.2xlarge` (Segment Store + Bookkeeper), `m5n.xlarge`(Controller), `t2.small` (ZooKeeper), `m5n.xlarge` (benchmarking client), `t2.large` (metrics) |

If you modify the `public_key_path`, make sure that you point to the appropriate SSH key path when running the [Ansible playbook](#_RUNNING_THE_ANSIBLE_PLAYBOOK).

Expand Down
61 changes: 36 additions & 25 deletions driver-pravega/deploy/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,20 +98,6 @@
echo 'LANG=en_US.utf-8
LC_ALL=en_US.utf-8' > /etc/environment

#- name: Install nmon
# hosts: ["!tier2"]
# tags: ["nmon"]
# connection: ssh
# become: true
# tasks:
# - name: Download nmon
# unarchive:
# src: "http://sourceforge.net/projects/nmon/files/nmon16j.tar.gz"
# remote_src: yes
# dest: /tmp
# - command: cp /tmp/nmon_AMD64_ubuntu1804 /usr/local/bin/nmon
# - command: chmod a+x /usr/local/bin/nmon

- name: Metrics installation
hosts: ["metrics"]
tags: ["metrics"]
Expand All @@ -120,13 +106,38 @@
tasks:
- name: Add Extras Repo
shell: yum-config-manager --enable rhui-REGION-rhel-server-extras
- name: Install RPM packages
yum: pkg={{ item }} state=latest
with_items:
- docker
- systemd:
when:
- ansible_facts['distribution'] == 'RedHat'
- ansible_facts['distribution_major_version'] | int <= 7
- name: Docker repo
yum_repository:
name: docker
description: repo for docker
baseurl: "https://download.docker.com/linux/centos/{{ ansible_facts['distribution_major_version'] }}/x86_64/stable/"
gpgcheck: no
when: ansible_facts['distribution'] == 'RedHat'
- name: Create Docker repo dir
file:
path: "/etc/yum.repos.d/"
state: directory
- name: Add Docker Centos extras
copy:
dest: /etc/yum.repos.d/docker-ce.repo
content: |
[centos-extras]
name=Centos extras - $basearch
baseurl=http://mirror.centos.org/centos/7/extras/x86_64
enabled=1
gpgcheck=1
gpgkey=http://centos.org/keys/RPM-GPG-KEY-CentOS-7
- name: Installing docker
yum:
state: latest
pkg: ['slirp4netns', 'fuse-overlayfs', 'container-selinux', 'docker-ce']
- name: Start docker
service:
name: docker
state: started
name: "docker"
enabled: yes

- name: Prometheus installation
Expand Down Expand Up @@ -192,7 +203,7 @@
hosts: ["!tier2"]
tags: ["node-exporter"]
roles:
- cloudalchemy.node-exporter
- cloudalchemy.node_exporter

- name: ZooKeeper setup
hosts: zookeeper
Expand Down Expand Up @@ -304,15 +315,15 @@
path: "{{ item.path }}"
state: unmounted
with_items:
- { path: "/mnt/journal", src: "/dev/nvme0n1" }
- { path: "/mnt/journal", src: "/dev/nvme2n1" }
- { path: "/mnt/storage", src: "/dev/nvme1n1" }
- name: Format disks
filesystem:
fstype: xfs
dev: '{{ item }}'
force: yes
with_items:
- '/dev/nvme0n1'
- '/dev/nvme2n1'
- '/dev/nvme1n1'
- name: Mount disks
mount:
Expand All @@ -322,7 +333,7 @@
opts: defaults,noatime,nodiscard
state: mounted
with_items:
- { path: "/mnt/journal", src: "/dev/nvme0n1" }
- { path: "/mnt/journal", src: "/dev/nvme2n1" }
- { path: "/mnt/storage", src: "/dev/nvme1n1" }

- name: BookKeeper setup
Expand Down Expand Up @@ -350,7 +361,7 @@
dest: "/opt/bookkeeper/bin/common.sh"
- name: Format BookKeeper metadata in Zookeeper
command: >
bin/bookkeeper shell metaformat -nonInteractive --force
bin/bookkeeper shell initnewcluster
args:
chdir: /opt/bookkeeper
when: groups['bookkeeper'][0] == inventory_hostname
Expand Down
2 changes: 1 addition & 1 deletion driver-pravega/deploy/provision-pravega-aws.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ resource "aws_instance" "metrics" {
# Change the EFS provisioned TP here
resource "aws_efs_file_system" "tier2" {
throughput_mode = "provisioned"
provisioned_throughput_in_mibps = 1000
provisioned_throughput_in_mibps = 100
tags = {
Name = "pravega-tier2"
}
Expand Down
10 changes: 5 additions & 5 deletions driver-pravega/deploy/terraform.tfvars
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
public_key_path = "~/.ssh/pravega_aws.pub"
region = "us-west-2"
ami = "ami-9fa343e7" // RHEL-7.4 us-west-2
region = "us-east-2"
ami = "ami-0bb2449c2217cb9b0" // RHEL-7.9 us-east-2

instance_types = {
"controller" = "m5.large"
"bookkeeper" = "i3en.6xlarge"
"bookkeeper" = "i3en.2xlarge"
"zookeeper" = "t2.small"
"client" = "m5n.8xlarge"
"client" = "m5n.xlarge"
"metrics" = "t2.large"
}

num_instances = {
"controller" = 1
"bookkeeper" = 3
"zookeeper" = 3
"client" = 2
"client" = 1
"metrics" = 1
}
2 changes: 1 addition & 1 deletion driver-pravega/deploy/vars.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
#

---
pravegaVersion: "0.10.1"
pravegaVersion: "0.12.0"
zookeeperVersion: "3.5.5"
bookkeeperVersion: "4.14.2"
prometheusVersion: "2.2.1"
Expand Down
4 changes: 2 additions & 2 deletions driver-pravega/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
<properties>
<!--Specify pravegaVersion in properties or use default: 0.10.0-->
<!--Provide Pravega version for build: mvn clean install "-DpravegaVersion=0.9.0-2766.2515791-SNAPSHOT" -DskipTests=true -Dlicense.skip=true-->
<pravegaVersion>0.10.2</pravegaVersion>
<pravegaVersion>0.12.0</pravegaVersion>
</properties>

<dependencies>
Expand Down Expand Up @@ -64,7 +64,7 @@
<dependency>
<groupId>io.pravega</groupId>
<artifactId>pravega-keycloak-client</artifactId>
<version>0.11.0</version>
<version>${pravegaVersion}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
Expand Down