| path | title |
|---|---|
/learnings/aws |
Learnings: AWS |
- >
- >
- >
- >
- >
- >
- >
- >
- >
- >
- >
- >
- >
- >
- Creating a No Code Backend with DynamoDB, Cognito, DynamoDB and AWS Gateway
From http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-swap-volumes.html:
The c1.medium and m1.small instance types have a limited amount of physical memory to work with, and they are given a 900 MiB swap volume at launch time to act as virtual memory for Linux AMIs.
... but only those - EC2 instances not those seem to have no swap (!!??)
- Learning_Ops_Unix_Controlling_Swappiness
- Learning_AWS_Good_DNS_Names_For_Instances
By default given DNS names will be ip-private-ipv4-address.region.compute.internal
(for non default VPC ???)
For better names:
- Chef: setting /etc/hosts everywhere to not require a DNS server
- Chef: use Route53 (assumes OpsWorks, but is portable)
- Use a userdata script
... or run your own bind server and create a Chef recipe to update that...
1024 packets per second per network card
AZs are presented to you called "Subnets". Note:
- CIDR block for subnet (see Learning_Ops_Networking_CIDR_Blocks )
- listed Availability zone
Upload a .zip of application, < 512MB.
Can use eb deploy to deploy current folder. Will always deploy latest commit, even if you have pending changes.
Extend ElasticSearch configuration by .ebextensions folder.
Can do super customized platform using AMIs + Packer
Supported Types:
- All at once
- Rolling
- Rolling with additional batch <-- rolls a new N instances so you don't lose N capacity as you deploy the new version to those N instances
- Immutable
Can use menu in AWS console to Save Configuration (or use eb config)
Name this env.yaml in your top level folder name.
Called: "Environment Links"
KEY: (environment name)
value of key will be the cname to the environment with that name.
environment names and links let you create environment groups.
Undr Ruby this is /var/log/(nginx|http)/(error_log|access_log); /var/log/puma/puma.log (or /var/app/support/logs/passenger.log)
S3 upload (can) happen on logrotate.
- tail : gets last 100 lines of all files, packages up. S3 dest will delete after 15 minutes
- Bundled : full log to S3
- rotated : full log to S3 when a log rotation would happen
Remember that AWS Log agent works like so:
config files for what to monitor are stored in /etc/awslogs/config/ <-- many configuration files, one file per file to monitor. (Normal awslogs operation)
Info about this:
Drawbacks / things need to worry about before using ECR:
-
Before pushing to ECR you need an Image Repository created already to store your docker image. One Repository = One Docker Image TERMINOLOGY:
- Docker Registry: Docker Hub
- Docker Repository: A collection of images with the same name but different tags (aka https://hub.docker.com/r/library/python/tags/ is a Repository)
- Thus ECR Image Repository = Docker Repository
... the documentation seems to imply differently, but experience seems to be one repository = one docker image ???
-
IAM roles apply on the repository level (only that granular)
Neat commands:
aws ecr describe-repositories
AWS Code Pipeline: Build Pipeline creation
Pipelines contain steps like AWS CodeBuild, and others
CodePipeline: a lot of power hidden behind first interface!!! At first blush looks limiting (SOURCE, BUILD, DEPLOY) but can hook up many more stages: ie multiple CodeBuild commands, direct push to OpsWorks or ECS, etc
Limits:
- AWS CodePipeline does not currently support Bitbucket.
- does not easily support LightSail(??)
Inputs:
- Github
- AWS S3
- AWS CodeCommit
CodePipeline can trigger AWS CodeBuild as part of its steps.
NOTE: CodePipeline does not support Bitbucket
Types of stages:
- Build <-- trigger CodeBuild or Jenkins
- Approval <-- Humans!
- Deploy <-- to CodeDeploy, ECS, OpsWorks
- Invoke <-- trigger a AWS Lambda function!!
Example Deploy pipeline -> ECS:
- (Code trigger)
- Code Build -> ECR
- Deploy via Cloudformation via new task definition revision
Because the CFN regenerated will pull the Docker image from ECR.
Example Deploy pipeline -> React:
- [TODO]: Think about this use of CodePipeline!!!!
Deploy and run code arctifacts on a VM
Inputs:
- Github
- Bitbucket
- S3
(Thus can do Heroku style deploys just with CodeDeploy)
Two deployment types:
- in place
- blue / green — requires EC2 instances with auto scale groups / and configured IAM (dynamic???)
Requirements:
* must have CodeDeploy agent installed
During deploy CodeDeploy agent on box reads AppSpec.yml and executes instructions
Allows many hooks to mutate state of machine before you launch your software on it.
-
Can use CodeDeploy to let your AWS Lambda functions have pre and post deploy hooks!!!!
-
By using / abusing post / pre hooks, can use this as a poor man's Chef: ie installing Wordpress or a third party application on Ec2 instances.
Takes files in (github, bitbucket, s3) and (optionally) copies the file result of some build to S3.
Thus if your build process produces artifacts, you could use this to build/produce them. (Or if your pipeline tests and then produces Docker artifacts???)
Tracking Code Build step results: http://docs.aws.amazon.com/codebuild/latest/userguide/view-build-details.html
- "serverless"
- per minute billing
- can listen to Github, Bitbucket, S3 repos
- Docker features (can build your app in the service's custom Docker image)
- 5 minute build costs ~ 3cents
if CodeBuild doesn't have version of language etc you need, can use own Docker image.
Can also use docker-in-docker - have used this to install docker compose and bring up a suite of required services (ie databases), each in their own seperate container.
- AWS CodeBuild projects: 1,000
- Max number of concurrent builds: 20 (varies based on instance size, but most are 20)
Behaviors:
- it should update commit / PR status when the build completes
- it should link to the AWS CodeBuild result in the build status
CodeBuild auto does this if you set it up to trigger from Github (and make sure the CodeBuild app has permissions). Will update commit status with a pointer to the CodeBuild result!!!
Behaviors:
- it should update commit / PR status when the build completes
- it should link to the CodeBuild result in the build status
CodeBuild does this, even for Bitbucket or Bitbucket Cloud. AKA: Bitbucket fully supported citizen!!!
- Early early stage startups (5 cents a build, no $10-60/month min)
Default ( < 10-30ish????? minutes ) of requests seem to be rate limited at 6K rps / 800-ishMB/second
Can call up account rep and ask to pre-warm.
Should be able to scale *LBs up to Web Scale traffic: aka all of ESPN.com is served off one ELB configuration in AWS.
Allows SSL termination on chosen instance, not just at load balancer level
$ sudo yum install java-1.8.0-openjdk-develsudo alternatives --config javasudo yum remove java-1.7.0-openjdk-devel
For graphql APIs can auto generate your code for you to talk back to AWS AppSync. Start with the graphql schema and it will generate :
- CRUD queries, mutations and subscriptions
- provision databases with amplify push
Can generate in javascript, typescript or Flow Can also use Cognito to create application specific users for your app. Including, for react, widgets for sign up and sign in.
Slows users to create sign ins with your application. provides a console to view all the users created, serarch through them, set the attributes of the username and password rules, customize the UI. can also use a lambda function to trigger when users are created etc etc
Can also use Google Auth time based one time passwords from the users auth app of choice.
Also supports Sign In with Apple
Also lets people sign in with their Facebook or Google account
If your data model fits in a document database, you can
Advantages:
- free
- good for startups where you're optimizing either cost or code
- API gateway can abstract away when you move to (a real full code solution) from having API gateway do all the transformation logic
- Ooops Just Mapping Code
Concepts:
Resource -> Stage has many resources -> custom domain names map resources + stages to a URL / path exposed to the Internet.
API Gateway -> Custom Domain Mappings: where you can configure what path the API is under. Want all your APIs to be under /rest/ or /api ? Set it in your custom domain name's API mapping, but this is optional (although likely a best practice - you don't know what non API related things your API gateway could serve)
Resources - how to define your routes
Can nest resources - ie have a resource that is /documents/ and a sub resource in that which is v1, so your resulting routes will be example.com/documents/v1/
DynamoDB actions (what you put in the Action section of the integration request)
Remember regardless of what ACTION the request comes in as, you need to send POST requests into Dynamo.
Resolver mapping template utility reference looks like it's about AppSync, but these functions also seem to be there for API Gateway mapping documents.
What's in the $context object anyway?
Assumes that the DynamoDB document has a UserId field, which is populated with the Cognito username. You want to, on the "server side" ask for the documents (the logged in user) owns, for security reasons. Which involves some processing logic in the mapping document in the Integration Request:
Content-Type: application/json
{
"TableName": PUT_YOUR_TABLE_NAME_HERE_AS_A_STRING,
"FilterExpression": "UserId = :val",
"ExpressionAttributeValues": { ":val": {"S": "$context.authorizer.claims['cognito:username']"} }
}(Yes, you need tho {"S": ..} stuff!)
You need to specify action. AWS DynamoDB Actions API reference
I had to explicitly set my request's Content-Type to application/json to get the mapper to pick up what I was trying to do.
You'll need to process DyamoDB's results to a better format, as DynamoDB will return the attribute type where sterotypically REST apis would just return the value.
So "foobar": {"S": "baz"}. Which is very odd: not very RESTful, not very expected. You'll need to map the DynamoDB results back to something useful
In the Integration Response, the mapping template:
"foobar": "$elem.SomeField.S"
NOT returning the attribute type implementation detail will give us a more stereotypical API for our users (and if we rip up this endpoint with something less "no code").
This does mean you'll need to transform every field from your data source. Setting up a Model for your object type (on the left sidebar) will at least give you hints about what fields you need to process.
Since you'll need to loop over multiple documents coming back from DynamoDB (it's an index page after all) you need to use a VTL loop. Putting these facts together...
#set($inputRoot = $input.path('$'))
{
"YOUR_OBJECT_TYPE_HERE": [
#foreach($elem in $inputRoot.Items) {
"user_id": "$elem.UserId.S",
"title": "$elem.title.S"
"type": "object"
}#if($foreach.hasNext),#end
#end
]
}
See that hasNext block here? Trailing commas are invalid JSON, so we must not include the for the last element. Use this hasNext trick to only include commas if there's more elements comming.
Getting values from the post: "somefieldInDynamo": {"S": $input.json('$.someField')}
For documentId I cheated and just used "DocumentId": { "S": "$context.requestTimeEpoch" }. The Dynamodb design documents say to not do this for partition keys but I don't have that much of a choice with regard to auto-generating keys.
Work through Routing Traffic to an Amazon API Gateway API by using your domain name
In API Gateway under the custom domains, you can set up API mappings - mapping a URL with an API Gateway Resource you created.
if you have an API gateway route you want to run locally, you'll likely just need to set the Access-Control-Allow-Origin header to * (not multiple domains, that doesn't seem to work, or I didn't know how to set it up correctly)
Under Actions there's an Enable CORS choice, which propegates your CORS settings to your routes. It doesn't seem to be stateful, but that's how you do it (I did not have luck just creating an OPTIONS route myself).
This Enable CORS seems to not be once and done thing, and does not apply to subresources. Keep a watch over your network opertions tab where you're calling API Gateway to see if you need to do it again, or on different or new subset.
Logging: the logging stuff, through to CloudWatch logs, is really good at seeing what your mapping documents are doing.
You may need to set up the IAM permissions correctly, In the global API gateway Settings.
Q: What's the UX around this, or around this making it easy? Plus, there's a debug flag, right?
Figure the answers to these questions out then integrate them into the article
Q: how do I hook up a route that passes everything through: ie I've stood up a lambda with POST/PUT/GET verbs and want /api/v1/foobar to always go to that? Is that easy? AND what about /api/v2 routing to ie a replacement Rails app?
A:
Q: Q: what does canary deploys do / how to use that? ie want to be able to test this thing on prod before sending all the traffic to that point. Can do that with canary, somehow although the interface is not good.
A: turn on canary, then when you deploy a change and it lets you pick a stage. It will default to the canary version of that stage. When you are happy use promote in the canary's settings page.
Canary deployments in API gateway blog entry from Amazon
Q: how can I monitor that a deployment has really happened?
Q: can I use API Gateway to control traffic to my api (aka maintaince window)?
TODO: make one of those AWS arch diagrams: see mah, no step 3!
A less in depth version of this information. But it's a great summary!