Deploy custom Python package on AWS Lambda
We will deploy a python package with 3rd Party modules and schedule the lambda to run every day.
So what I am trying to do is to call the open weather API every 6 hours and store the data in a MongoDB database. This article is mainly focused on how to create the python package.
There are a few pre-requisites
Now let’s come to the packaging part, make sure you are using a virtual environment. You can easily create one using (make sure using Gitbash if windows)
$ python -m venv venv
$ source venv/Scripts/activate
So when you will see a prompt, you can find
(venv) written before the prompt like this —
Suppose the folder structure is like that
lambda_function.py is a simple python script that basically fetches data from openweathermap.org and stores it inside MongoDB.
when we do
pip list we can see the 3rd party packages installed in the virtual environment.
Now you can run your script and it executes successfully.
Now comes the packaging part. I have seen most of the people just putting the
Lib/site-packages and making the zip, but there are many other packages inside the
site-packages folder that we don’t need. here is an example of the zip sizes —
As you can see, the
site-packages_large.zip is created using the method mentioned just now and it is much larger (5MB compared to 2MB of the method that I will discuss now).
I have created one script to automate this, given the folder structure —
create a file called
package_maker.sh and paste the code inside it
# just checking if the filename is not null
if [ -z "$1" ]
echo "You did not put the file name"
echo "it should be like ./package_maker.sh <file-name>"
fi# creating the requirements.txt file
pip freeze > requirements.txt# removing any pre existing lambda_package folder
rm -rf ./lambda_package# making new lambda_package folder
mkdir ./lambda_package# copy the main python file into the directory
cp $1 ./lambda_package/$1# installing the packages into that directory
pip install --target=./lambda_package -r requirements.txt# creating the zip
echo "make sure 7z.exe is added in your path"
echo "it normally resides in C:\Program Files\7-Zip"7z a -r -tzip lambda_package.zip ./lambda_package/*# removing the lambda_package directory
rm -rf ./lambda_package# completed
echo "completed, find the zip named lambda_package.zip"
It is very simple and self-explanatory. This script will basically create a temporary folder named
lambda_package/ and put only the specific packages and create the zip named
lambda_package.zip and delete the temporary folder. To use this script type ./package_maker.sh lambda_function.py inside in your git bash.
package_maker.sh is important, because without this it will search inside PATHs.
(venv)probhakarsarkar@xxxx~/Desktop/works/temperature_monitor $ ./package_maker.sh lambda_function.py.
Everything is Okcompleted, find the zip named lambda_package.zip
You can change the function name from
lambda_function.py and the callable inside it from
lambda_handler to anything else but make sure to change it accordingly inside runtime settings —
Now that you have created the
lambda_package.zip t is just a matter of uploading it into lambda —
After the lambda is deployed and tested successfully we can configure the CloudWatch event to schedule it —
configure details >
create rule . It will call the lambda every 6 hours.