In order to compile MXNet for the Lambda function, you would need an environment running the AMI that the Lambda service is using. The current version is:
amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2
double-checking that the current version is still the one above by visiting https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html.
You will have to launch an Amazon EC2 instance with that AMI and then access it by using an SSH client.
First, install development tools and the libraries required to compile MXNet. Apache MXNet requires several libraries as prerequisites. In this tutorial we will be using:
- Atlas (Linear Algebra Library) http://math-atlas.sourceforge.net/
- OpenBlas (Basic Linear Algebra Subprograms) https://www.openblas.net/
- Lapack (Linear Algebra Package) http://www.netlib.org/lapack/
We are NOT going to use OpenCV, required by MXNet for image processing functions, and NVIDIA CUDA / cuDNN since we do not need to run Apache MXNet with GPU support.
If you want to know more on the dependencies and options to build Apache MXNet from sources, please visit https://mxnet.incubator.apache.org/install/build_from_source.html.
Execute the following commands in your home folder:
sudo yum groupinstall -y "Development Tools" && sudo yum install -y git
sudo yum install atlas-devel
sudo yum install openblas openblas-devel.x86_64 lapack-devel.x86_64
We are now ready to build Apache MXNet from sources. We are going to get the sources from the GitHub repository where MXNet is maintained and then build with the proper settings.
git clone --recursive https://github.com/apache/incubator-mxnet mxnet
cd mxnet
make -j $(nproc) USE_OPENCV=0 USE_CUDNN=0 USE_CUDA=0 USE_BLAS=openblas USE_LAPACK=1
When the build process is completed, we will need to install the Python bindings as we want to use Python as programming language. All language bindings supported by MXNet are configured after building the framework, except C++ that needs to be included during the compilation.
To install language bindings, execute the following commands:
cd python
sudo python setup.py install
Once the bindings are installed, we are ready to build a package that will contain all the relevant libraries to run MXNet in Lambda.
Please note that these instructions assume using the latest version of Apache MXNet and Numpoy, which, at the time being, are 1.3.1 and 1.15.2. This means that, if a new version is released, you might have to update the paths below.
cd
mkdir mxnetpackage
cp -r /usr/local/lib/python2.7/site-packages/mxnet-1.3.1-py2.7.egg/mxnet mxnetpackage/
cd mxnetpackage
sudo pip install numpy
cp -r /usr/local/lib/python2.7/site-packages/numpy-1.15.2-py2.7-linux-x86_64.egg/numpy/ .
mkdir lib
cp /usr/lib64/atlas/libatlas.so.3 lib/
cp /usr/lib64/atlas/libcblas.so.3 lib/
cp /usr/lib64/atlas/libclapack.so.3 lib/
cp /usr/lib64/atlas/libf77blas.so.3 lib/
cp /usr/lib64/libgfortran.so.3 lib/
cp /usr/lib64/libgfortran.so.3.0.0 lib/
cp /usr/lib64/libgomp.so.1 lib/
cp /usr/lib64/libgomp.so.1.0.0 lib/
cp /usr/lib64/atlas/liblapack.so.3 lib/
cp /usr/lib64/libopenblas.so.0 lib/
cp /usr/lib64/atlas/libptcblas.so.3 lib/
cp /usr/lib64/atlas/libptf77blas.so.3 lib/
cp /usr/lib64/libquadmath.so.0 lib/
We are now going to compress and save the package to Amazon S3 as the final step.
tar cfz mxnet.tar.gz *
aws s3 cp mxnet.tar.gz s3://{your_bucket}/
Now you can run a test by creating an AWS Lambda function starting from the package content and write a simple Python lambda handler that executes some basic MXNet code.
Finally, after ensuring the package works as expected, you can terminate the EC2 instance to avoid incurring in unexpected charges.