Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide aws command to create CloudFormation template of resources instead of actual resources #624

Open
golharam opened this issue Dec 13, 2019 · 4 comments

Comments

@golharam
Copy link

golharam commented Dec 13, 2019

We operate in a controlled environment and all resources need to be created/managed via CloudFormation templates. Also, debugging issues will be easier:

failed to create ComputeEnvironment: error attaching policies to ecsInstanceRole: NoSuchEntity: Policy arn:aws:iam::aws:policy/AmazonEC2ContainerServiceforEC2Role does not exist or is not attachable.

@adamstruck
Copy link
Contributor

I will look into this. Realistically I will likely not have time until the new year.

I am posting an example for my reference:
https://github.com/dejonghe/aws-batch-example/blob/master/cloudformation/batch/batch-example.yml

The main custom components that Funnel will need is the JobDef and a custom AMI depending on your use case.

@adamstruck
Copy link
Contributor

adamstruck commented Mar 3, 2020

Current WIP:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Sample Template Managed Single Batch Job Queue: This
  template demonstrates the usage of simple Job Queue and EC2 style Compute Environment.  **WARNING**
  You will be billed for the AWS resources used if you create a stack from this template.'
Resources:
  ##-------------
  ## Network
  ##-------------
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
  InternetGateway:
    Type: AWS::EC2::InternetGateway
  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId:
        Ref: VPC
      InternetGatewayId:
        Ref: InternetGateway
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
      VpcId:
        Ref: VPC
  Subnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 10.0.0.0/24
      VpcId:
        Ref: VPC
      MapPublicIpOnLaunch: 'True'
  Route:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId:
        Ref: RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway
  SubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: RouteTable
      SubnetId:
        Ref: Subnet

  ##-----------
  ## Roles
  ##-----------
  BatchServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: batch.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole

  EcsInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

  EcsTaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FunnelS3
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Effect: Allow
                Action:
                  - 's3:GetBucketLocation'
                  - 's3:GetObject'
                  - 's3:ListObjects'
                  - 's3:ListBucket'
                  - 's3:CreateBucket'
                  - 's3:PutObject'
                Resource: '*'
        - PolicyName: FunnelDynamodb
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Effect: Allow
                Action:
                  - 'dynamodb:GetItem'
                  - 'dynamodb:PutItem'
                  - 'dynamodb:UpdateItem'
                  - 'dynamodb:Query'
                Resource: '*'

  ##-----------------
  ## Batch Resources
  ##-----------------
  JobDefinition:
    Type: AWS::Batch::JobDefinition
    Properties:
      Type: container
      ContainerProperties:
        Image: docker.io/ohsu-comp-bio/funnel:latest
        Vcpus: 1
        Memory: 512
        Command:
          - 'worker'
          - 'run'
          - '--EventWriters'
          - 'log'
          - '--EventWriters'
          - 'dynamodb'
          - '--Database'
          - 'dynamodb'
          - '--DynamoDB.Region'
          - Ref: AWS::Region
          - '--DynamoDB.TableBasename'
          - 'funnel'
          - '--Worker.WorkDir'
          - '/opt/funnel-work-dir'
          - '--taskID'
          - 'Ref::TASKID'
        Volumes:
          - Host:
              SourcePath: /var/run/docker.sock
            Name: docker-sock
          - Host:
              SourcePath: /opt/funnel-work-dir
            Name: funnel-work-dir
        MountPoints:
          - ContainerPath: /var/run/docker.sock
            ReadOnly: false
            SourceVolume: docker-sock
          - ContainerPath: /opt/funnel-work-dir
            ReadOnly: false
            SourceVolume: funnel-work-dir
        Environment: []
        Ulimits: []
        ResourceRequirements: []
        User: ""
        ReadonlyRootFilesystem: true
        Privileged: true
        JobRoleArn:
          Ref: EcsTaskRole
      RetryStrategy:
        Attempts: 1

  JobQueue:
    Type: AWS::Batch::JobQueue
    Properties:
      Priority: 1
      ComputeEnvironmentOrder:
        - Order: 1
          ComputeEnvironment:
            Ref: ComputeEnvironment
 
  ComputeEnvironment:
    Type: AWS::Batch::ComputeEnvironment
    Properties:
      Type: MANAGED
      ComputeResources:
        Type: EC2
        MinvCpus: 0
        DesiredvCpus: 0
        MaxvCpus: 256
        InstanceTypes:
          - optimal
        AllocationStrategy: BEST_FIT
        Subnets:
          - Ref: Subnet
        SecurityGroupIds:
          - Ref: SecurityGroup
        InstanceRole:
          Ref: EcsInstanceRole
      ServiceRole:
        Ref: BatchServiceRole

Outputs:
  ComputeEnvironmentArn:
    Value:
      Ref: ComputeEnvironment
  JobQueueArn:
    Value:
      Ref: JobQueue
  JobDefinitionArn:
    Value:
      Ref: JobDefinition

@adamstruck
Copy link
Contributor

@golharam the above seems to produce all of the necessary AWS Batch resources.

Note: the template does not handle setting up the Funnel server.

Let me know if this works for you.

@golharam
Copy link
Author

This works. I only needed parts of it, but still good that all is there. I recommend you make a command to output the CloudFormation template and let the user create the resources, rather than having funnel create the resources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants