Amazon MWAA tips

This note was created on 2023-08-21
This note was last edited 2023-08-21

The biggest problem with Amazon MWAA: there's no way to provide Internet access to its web server. Therefore, the required plugins could not be downloaded from the Internet. But there's a workaround... We need to build our own plugins.zip with the necessary dependencies.

=== How to install dependencies from requirements.txt ===
1. Download dependencies from requirements.txt:
   $ pip3 download -r /path/to/requirements.txt -d /tmp/mwaa-plugins
   
2. Set execute permission:
   $ chmod +x /tmp/mwaa-plugins/*

3. Compress all plugins into .zip file:
   $ zip -7 plugins.zip /tmp/mwaa-plugins/*

Or, here's a one-liner for you:
$ mkdir /tmp/mwaa-plugins && pip3 download -r requirements.txt -d /tmp/mwaa-plugins && chmod +x /tmp/mwaa-plugins/* && zip -7 ./plugins2.zip /tmp/mwaa-plugins/* && rm -rf /tmp/mwaa-plugins
Note: do not run this as root and be careful about rm -rf

4. Update requirements.txt. 
Specify the plugins directory at the top of your requirements.txt using --find-links and instruct pip not to install from other sources using --no-index:
~~~
--find-links /usr/local/airflow/plugins
--no-index

numpy
~~~
In this example, Amazon MWAA fetches the numpy-1.20.1-cp37-cp37m-manylinux1_x86_64.whl wheel from the plugins folder and installs it on your environment.

=== Possible solution: plugins are not installed, Airflow connections are empty ===
1. Turn on webserver logging with INFO level.
2. Search for the following lines in log:
   ~~~
   ERROR: Could not find a version that satisfies the requirement pymssql (from versions: none)
   ERROR: No matching distribution found for pymssql
   ~~~
   Instead of pymssql, there could be any other package.
   
3. If you have the same error, then you need a different type of package. The best way would be to download *ALL* versions of the package from pypi.org and then add all of them to our zip.
4. If you'll find in logs later, which exact version on package is needed, you can remove all other versions.