PyTapir

Python flavored tapir output of protoc.

A general API documentation can be found here.

(Disclaimer: this is my very first python module ever)

Usage

pytapir is usually distributed via python packages that are hosted on GitHub. We're trying to have the latest version available all the time; but since python is more of a side project we cannot keep that promise.

In case this library diverges from the tAPIr-API, please give me (@thisismana) a bump to fix this.

pipenv

pipenv install -e git+https://github.com:stroeer/[email protected]#egg=requests

Pipfile

This should go into the Pipfile

[packages]
pytapir = { git = "[email protected]:stroeer/pytapir.git", ref = "v0.45.0"}

Lambda layers

Permissions should allow every account within our AWS organization to access the pytapir lambda layers.

The lambda layer contains everything you'll need to invoke our gRPC services. See ./examples/ folder for sample code.

• pytapir
• grpcio

tapir version	Python Version	Arch	ARN
0.45.0	3.11	arm64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-311-Arm64:3
0.45.0	3.11	x86_64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-311-Amd64:3
0.45.0	3.12	arm64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-312-Arm64:3
0.45.0	3.12	x86_64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-312-Amd64:3
0.48.0	3.11	arm64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-311-Arm64:4
0.48.0	3.11	x86_64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-311-Amd64:4
0.48.0	3.12	arm64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-312-Arm64:4
0.48.0	3.12	x86_64	arn:aws:lambda:eu-west-1:053041861227:layer:PyTapir-312-Amd64:4

Build tapir

dependencies

We're using a virtual env, so do the following within the project root after cloning this repo:

python3 -m venv venv
source venv/bin/activate
pip install --requirement requirements.txt

# you can work now, e.g. compile .proto to .py
python3 setup.py protoc

# once your done, deactivate the venv
deactivate

bump tapir submodule

tapir is mounted as a git submodule

# bump tapir to latest
git submodule update --init --recursive
git submodule foreach git fetch --all --tags --prune
git submodule foreach git pull origin main

# commit changes
git add tapir

# if there are changes, commit and push them:
git commit -m "bumped tapir to latest (origin/main)"
git push

Release

You have to set an environment variable to create a proper GitHub release.

make release GITHUB_TOKEN=xxxx

AWS Lambda Layer

Useful reads:

• how to build a python lambda layer
• Add library dependencies to layer

NEXT_TAG=$(echo "import stroeer; print(stroeer.__version__)" | python3)
ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
for PYTHON_VERSION in 3.11 3.12 ; do
  for ARCH in arm64 x86_64 ; do
    docker buildx build --load --platform linux/${ARCH} --build-arg="PYTHON_VERSION=${PYTHON_VERSION}" --tag "${NEXT_TAG}:${ARCH}" .
  
    docker run --platform linux/${ARCH} --rm --entrypoint cat "${NEXT_TAG}:${ARCH}" layer.zip > ${ARCH}_layer.zip
    
    aws s3 cp ${ARCH}_layer.zip s3://ci-${ACCOUNT_ID}-eu-west-1/pytapir/layer_${NEXT_TAG}_${ARCH}.zip
    
    if [ "$ARCH" = "x86_64" ] ; then
      ARCH_Pretty='Amd64'
    else
      ARCH_Pretty='Arm64'
    fi
    
    # create lambda layer
    layer_version=$(aws lambda publish-layer-version \
      --layer-name PyTapir-${ARCH_Pretty} \
      --description "PyTapir ${NEXT_TAG}" \
      --license-info "MIT" \
      --content S3Bucket=ci-${ACCOUNT_ID}-eu-west-1,S3Key=pytapir/layer_${NEXT_TAG}_${ARCH}.zip \
      --compatible-runtimes python${PYTHON_VERSION} \
      --compatible-architectures ${ARCH} \
      --query "Version")
    
    # allow aws org to access layer
    org_id=$(aws organizations describe-organization --query 'Organization.Id' --output text)
    
    aws lambda add-layer-version-permission \
      --layer-name PyTapir-${ARCH_Pretty} \
      --statement-id xaccount \
      --action lambda:GetLayerVersion \
      --organization-id "${org_id}" \
      --principal '*' \
      --version-number "${layer_version}" 
  done  
done  

Release 2024 version

# update submodule
make bump_tapir
# create python github release
make release
# build lambda layer for python 3.11/3.12 for amd64/arm64
make lambda_layer

Local testing lambda and lambda layer

To test, we need:

• an environment that replicates the AWS environment (e.g. don't use native MacOS binaries).
• our lambda layer mounted
• our sample code mounted

export GRPC_ENDPOINT="$(aws ssm get-parameter --name /internal/api/endpoint --with-decryption --query 'Parameter.Value' --output text)"
export GRPC_AUTHORIZATION="Bearer $(aws ssm get-parameter --name /internal/api/auth.token --with-decryption --query 'Parameter.Value' --output text)"
docker run --rm \
  --volume "${PWD}/examples":/var/task:ro \
  --volume "${PWD}/python":/opt/python:ro \
  --env GRPC_ENDPOINT \
  --env GRPC_AUTHORIZATION \
  lambci/lambda:python3.8 lambda_function.lambda_handler