Deploy custom Python package on AWS Lambda

We will deploy a python package with 3rd Party modules and schedule the lambda to run every day.

So what I am trying to do is to call the open weather API every 6 hours and store the data in a MongoDB database. This article is mainly focused on how to create the python package.

Note: If your lambda is configured with VPC you need to make sure to follow the post and this post to configure it correctly.

There are a few pre-requisites

1. Make sure Gitbash is installed if using windows. It gives Unix style command-line environment.
2. Make sure 7zip is installed and added to the PATH. By this, if you type 7z and hit enter you see —

Now let’s come to the packaging part, make sure you are using a virtual environment. You can easily create one using (make sure using Gitbash if windows)

$ python -m venv venv
$ source venv/Scripts/activate

So when you will see a prompt, you can find (venv) written before the prompt like this —

Suppose the folder structure is like that

temperature_monitor/
    |--venv/
    |--lambda_function.py

My lambda_function.py is a simple python script that basically fetches data from openweathermap.org and stores it inside MongoDB.

when we do pip list we can see the 3rd party packages installed in the virtual environment.

Now you can run your script and it executes successfully.

Now comes the packaging part. I have seen most of the people just putting the lambda_function.py inside Lib/site-packages and making the zip, but there are many other packages inside the site-packages folder that we don’t need. here is an example of the zip sizes —

As you can see, the site-packages_large.zip is created using the method mentioned just now and it is much larger (5MB compared to 2MB of the method that I will discuss now).

I have created one script to automate this, given the folder structure —

temperature_monitor/
    |--venv/
    |--lambda_function.py

create a file called package_maker.sh and paste the code inside it

# just checking if the filename is not null
if [ -z "$1" ]
  then
    echo "You did not put the file name"
    echo "it should be like ./package_maker.sh <file-name>"
    exit 1
fi
# creating the requirements.txt file
pip freeze > requirements.txt
# removing any pre existing lambda_package folder
rm -rf ./lambda_package
# making new lambda_package folder
mkdir ./lambda_package
# copy the main python file into the directory
cp $1 ./lambda_package/$1
# installing the packages into that directory
pip install --target=./lambda_package -r requirements.txt
# creating the zip
echo "make sure 7z.exe is added in your path"
echo "it normally resides in C:\Program Files\7-Zip"
7z a -r -tzip lambda_package.zip ./lambda_package/*
# removing the lambda_package directory
rm -rf ./lambda_package
# completed
echo "completed, find the zip named lambda_package.zip"

It is very simple and self-explanatory. This script will basically create a temporary folder named lambda_package/ and put only the specific packages and create the zip named lambda_package.zip and delete the temporary folder. To use this script type ./package_maker.sh lambda_function.py inside in your git bash. ./ before package_maker.sh is important, because without this it will search inside PATHs.

(venv)probhakarsarkar@xxxx~/Desktop/works/temperature_monitor $ ./package_maker.sh lambda_function.py
.
.
.
.
Everything is Ok
completed, find the zip named lambda_package.zip

You can change the function name from lambda_function.py and the callable inside it from lambda_handler to anything else but make sure to change it accordingly inside runtime settings —

Now that you have created the lambda_package.zip t is just a matter of uploading it into lambda —

After the lambda is deployed and tested successfully we can configure the CloudWatch event to schedule it —

Then just configure details > create rule . It will call the lambda every 6 hours.