Skip to content

Commit

Permalink
Add tooling to enable quick release testing [skip ci]
Browse files Browse the repository at this point in the history 
Signed-off-by: Alexander Patrikalakis <[email protected]>
  • Loading branch information
Alexander Patrikalakis committed Jun 27, 2017
1 parent dcd03ff commit 4f6d207
Show file tree
Hide file tree
Showing 15 changed files with 731 additions and 92 deletions.
18 changes: 18 additions & 0 deletions buildspec.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
version: 0.1

phases:
build:
commands:
- echo Build started on `date`
- mvn install --update-snapshots --threads 2.0C --batch-mode -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.compiler.showWarnings=false
- chmod -R a+x ./janusgraph-hbase-parent/janusgraph-hbase-098/bin && chmod -R a+x ./janusgraph-hbase-parent/janusgraph-hbase-10/bin
- mvn verify --projects $MODULE $ARGS
- mvn surefire-report:report --projects $MODULE
post_build:
commands:
- echo Build completed on `date`
- ln -s ./$MODULE ./module
artifacts:
base-directory: module
files:
- target/**/*
130 changes: 130 additions & 0 deletions janusgraph-codepipelines-ci/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# JanusGraph CodePipelines CI
CodePipelines CI is a mechanism JanusGraph can use to do release testing in massively
parallel fashion (to the extent of AWS CodePipelines and CodeBuild service limits).

## Prerequisites
This procedure requires you to have an AWS account and a GitHub account.
It also requires you to create two service roles in IAM: one for CodePipeline and
one for CodeBuild. Finally, you need to have the [AWS CLI](https://aws.amazon.com/cli/) installed and on your path.

1. Get a personal access token from [GitHub](https://github.com/settings/tokens) with `repo` and `admin:repo_hook` scopes.
The `repo` scope is used to push the latest updates on the branch selected below. The `admin:repo_hook` scope is for
setting up the post-commit hook on GitHub programmatically.
2. Navigate to the [AWS Console](https://console.aws.amazon.com) and
[create an IAM User](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) with the following managed
policies: AmazonS3FullAccess, AWSCodePipelineFullAccess, AWSCodeBuildAdminAccess. This user should be created for
__Programmatic access__.
3. For this user, create security credentials and then register them in the `code-pipelines` profile on your computer
with `aws configure --profile code-pipelines`. Create this profile with a default region that
[supports CodePipeline and CodeBuild](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/)
and a default [output format](http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html#controlling-output-format)
of your choice (`json`, `text`, and `table` are available).
4. Create an IAM policy for CodeBuild and associate it to a new service role as described
[here](http://docs.aws.amazon.com/codebuild/latest/userguide/setting-up.html#setting-up-service-role). Select the
__Amazon EC2__ role type as CodeBuild is not yet available on this page. Edit the new role's trust relationship
and replace it with the text in step 16 of the CodeBuild user guide.
5. Create an IAM policy for CodePipelines and associate it to a new service role as described
[here](http://docs.aws.amazon.com/codepipeline/latest/userguide/iam-identity-based-access-control.html#view-default-service-role-policy).
Select the __Amazon EC2__ role type as CodePipeline is not yet available on this page. Edit the new role's trust
relationship and replace the principal `ec2.amazonaws.com` with `codepipeline.amazonaws.com`.

## Setup environment
Follow the steps below to prepare to create a test pipeline and run tests. First, clean up and set some constant
environment variables.

```bash
mvn clean package
#constants
export AWS_PROFILE_NAME='code-pipelines'
export AWS_ACCOUNT_NUMBER=`aws sts get-caller-identity --profile ${AWS_PROFILE_NAME} --output text | cut -f1`
```

Second, configure the parameters below to customize your test stack.

```bash
#GitHub personal access token you created in step 1
export GITHUB_TOKEN=''
#GitHub organization or user name of the repository to build
export GITHUB_USERNAME=''
#GitHub repository name to use for builds
export GITHUB_REPOSITORY=''
#GitHub branch in the above repository to use for builds
export GITHUB_BRANCH=''
#Name of the AWS CodeBuild service role you created in step 4
export AWS_CODEBUILD_ROLE=''
#Name of the AWS CodePipeline role you created in step 6
export AWS_CODEPIPELINE_ROLE=''
#Region you want to set up your test stack in
export AWS_REGION_NAME=''
#Name of bucket you want to use or create and use for storing your build artifacts.
export AWS_S3_BUCKET_NAME=''
#Name of pipeline JSON configuration file to use
export PIPELINE_CONFIGURATION=''
```

## Create test pipeline
This tool uses templates defined in JSON files to configure parallel builds in AWS CodePipeline. By default,
you can have up to [five parallel actions per stage](http://docs.aws.amazon.com/codepipeline/latest/userguide/limits.html)
in a pipeline and up to [twenty parallel builds](http://docs.aws.amazon.com/codebuild/latest/userguide/limits.html#limits-builds)
per region per account. If you need more, you can
[request to be whitelisted](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for more.

Here is an example of a file that defines two pipelines with parallel build actions.

```json
{
"j1" : {
"bdb" : { "MODULE" : "janusgraph-berkeleyje" },
"h-l-s" : { "MODULE" : "janusgraph-hadoop-parent/janusgraph-hadoop-2,janusgraph-lucene,janusgraph-solr" },
"cassandra" : { "MODULE" : "janusgraph-cassandra" },
"test" : { "MODULE" : "janusgraph-test" }
},
"j2" : {
"hbase098" : { "MODULE" : "janusgraph-hbase-parent/janusgraph-hbase-098" },
"hbase10" : { "MODULE" : "janusgraph-hbase-parent/janusgraph-hbase-10" },
"es" : { "MODULE" : "janusgraph-es" },
"cql" : { "MODULE" : "janusgraph-cql" }
}
}
```

The first pipeline is called `j1` and the second pipeline is called `j2`. Each of these pipelines have four
parallel build actions defined. Each build action is keyed at the action name and includes the environment
variables to be passed CodeBuild and used in the buildspec.yml file. Define the `PIPELINE_CONFIGURATION` environment
variable to point to your custom pipeline file or use the existing `pipe.json` or `tp-pipe.json` to run all default or
TinkerPop tests respectively. For example,

```bash
export PIPELINE_CONFIGURATION=pipe.json
```

## Run Test Pipelines
Run the following command with all the requirements, environment variables, and JSON
files you prepared to set up your pipelines and run tests.

```bash
java -jar target/janusgraph-codepipelines-ci-0.2.0-SNAPSHOT.jar \
--region ${AWS_REGION_NAME} \
--bucket ${AWS_S3_BUCKET_NAME} \
--github-owner ${GITHUB_USERNAME} \
--github-repo ${GITHUB_REPOSITORY} \
--github-branch ${GITHUB_BRANCH} \
--profile ${AWS_PROFILE_NAME} \
--codebuild-role-arn arn:aws:iam::${AWS_ACCOUNT_NUMBER}:role/${AWS_CODEBUILD_ROLE} \
--codepipeline-role-arn arn:aws:iam::${AWS_ACCOUNT_NUMBER}:role/${AWS_CODEPIPELINE_ROLE} \
--github-token ${GITHUB_TOKEN} \
--pipelines ${PIPELINE_CONFIGURATION}
```

Navigate to the [AWS Console](https://console.aws.amazon.com/codepipeline) and check on the status of your pipeline.

## Cleaning up
After you follow the steps above, you will end up with resources in the following services:
1. Build artifacts and source code zips in S3
2. CloudWatch logs
3. CodeBuild builds
4. CodePipeline pipelines, stages, and actions
5. IAM policies, roles and an IAM user
6. A GitHub personal access token

You can delete these resources after you are finished running your tests.
14 changes: 14 additions & 0 deletions janusgraph-codepipelines-ci/pipe.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"j1" : {
"bdb" : { "MODULE" : "janusgraph-berkeleyje" },
"h-l-s" : { "MODULE" : "janusgraph-hadoop-parent/janusgraph-hadoop-2,janusgraph-lucene,janusgraph-solr" },
"cassandra" : { "MODULE" : "janusgraph-cassandra" },
"test" : { "MODULE" : "janusgraph-test" }
},
"j2" : {
"hbase098" : { "MODULE" : "janusgraph-hbase-parent/janusgraph-hbase-098" },
"hbase10" : { "MODULE" : "janusgraph-hbase-parent/janusgraph-hbase-10" },
"es" : { "MODULE" : "janusgraph-es" },
"cql" : { "MODULE" : "janusgraph-cql" }
}
}
126 changes: 126 additions & 0 deletions janusgraph-codepipelines-ci/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.janusgraph</groupId>
<artifactId>janusgraph</artifactId>
<version>0.2.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>janusgraph-codepipelines-ci</artifactId>
<name>JanusGraph CodePipelines CI: Distributed release testing.</name>
<url>http://janusgraph.org</url>
<properties>
<top.level.basedir>${basedir}/..</top.level.basedir>
<aws.sdk.version>1.11.143</aws.sdk.version>
<lombok.version>1.16.16</lombok.version>
</properties>

<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-codepipeline</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-codebuild</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-iam</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.12</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.janusgraph</groupId>
<artifactId>janusgraph-test</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.janusgraph.codepipelines.AwsCodePipelinesCi</mainClass>
</transformer>
</transformers>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.6</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.6.6</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.6.6</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.8.1</version>
</dependency>
</dependencies>
</dependencyManagement>
</project>
Loading

0 comments on commit 4f6d207

Please sign in to comment.