Managing your Python dependencies with Poetry

It’s been quite the time since pip was included to python by default and the tool came a long way since it’s first version. It has all you need to start your basic python project with primitive dependency management. Additionally, you would probably use built-in Python’s venv for your virtual environment management. Now you may be enjoying your basic setup thinking you got everything taken care of, but let me tell you, there are better ways to begin your next Python project. And that’s where Poetry comes into the rescue.

Classic approach

So, what exactly is this tool and how it can improve your experience developing your next spectacular application? Well, before I answer this question let’s look at some pain points of using venv – pip combination. First thing is that in some big projects you may want to have different dependencies for both development and production environments. Using pip you may settle for splitting your requirements.txt into different files all in the same requirements directory with each file name corresponding to stage at which the file would be used. Using this approach your folder hierarchy may look like that:

|-- django_project_root
|-- requirements
|  |-- common.txt
|  |-- dev.txt
|  |-- prod.txt
|-- requirements.txt

You would have your basic requirements in common.txt file and then in dev.txt or prod.txt you would first call common.txt file followed by specifying the rest of the dependencies needed for each environment.

dev.txt:

-r common.txt
dev_req==1.0
...

Prod.txt:

-r common.txt
prod_req==1.0
...

This way, if you’d like to deploy your code to production you would specify pip install -r requirements/prod.txt in your dockerfile while running project locally would call pip install -r requirements/dev.txt. This approach, while may look straightforward, becomes more complex as number of dependencies in your project grows. Because it is you who needs to make sure that each file is up to date with what your project requires. And as the time goes by, the chances for a mess in those files grows. Another thing is when you use venv to create your environments. You may have multiple different projects you work on, each using different python version and dependencies. Some people like to keep their venv folder inside project directory (and that’s what pycharm would do by default), but in my opinion, it’s just adding another folder in already possibly beefy project’s directory hierarchy. Not to mention someone may accidentally push their own venv to VCS making a mess in other developer’s environments. It’s usually better to keep all your virtual environments in separate, dedicated space on your machine, but then you’d need something like virtualenvwrapper to easily organize them.

Using Poetry

Those were just two examples of possible pain-points of your project, but there are of course more ways in which poetry might be of use to you and make your development process at least a bit easier. “Poetry is a tool for dependency management and packaging in Python”, as the docs write in the introduction. The installation is straightforward, you just call:

|curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -`

in osx/linux or:

(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python -

on windows.

You can of course install poetry via pip, but that would make Poetry always use Python version for which it has been installed to create virtual environments. So, you will need to install Poetry for each Python version you want to use and switch between them.

After successful installation let’s see how to use this tool. If you are just starting a new project, you can use poetry new testproject which will create testproject directory with all the content like this:

|-- testproject
|-- pyproject.toml
|-- readme.rst
|-- testproject
|  |-- __init__.py
| -- tests
  | -- __init__.py
  | -- test_testproject.py

Or if you want to have a src folder you can alternatively call poetry new –src testproject which would result in the following folder structure:

|-- testproject
|-- pyproject.toml
|-- readme.rst
|-- src
|  |-- testproject
|       |-- __init__.py
| -- tests
  | -- __init__.py
  | -- test_testproject.py

You can also have your project named differently from your root folder by passing --name argument like: poetry new testfolder –name testproject. What if you already have your project and you want it to use poetry? Then you use poetry init in your project’s root directory, which would help you interactively create pyproject.toml and ask you for both main and development dependencies along the way. Poetry uses this file to keep track of your packages for both production and development environments.

Now, when you have your folder structure and pyproject.toml in place it’s time to create the virtual environment. All you need to do is call poetry install which would create your environment in default location and install all the dependencies. This default location for the environment is {cache-dir}/virtualenvs, where {cache-dir} is specific to whatever os you are using:

• Macos: ~/Library/Caches/pypoetry
• Windows: C:\\Users\\<username>\\AppData\\Local\\pypoetry\\Cache
• Unix: ~/.cache/pypoetry

You can modify this behaviour if you really want your virtual environment to be installed in your project’s directory using poetry config virtualenvs.in-project true –local. The --local flag would set this setting for specific project only. If you want it to be default for all your projects, just omit this flag. For all the configuration settings available use poetry config –list or see https://python-poetry.org/docs/configuration.

When you run poetry install for the first time, poetry.lock will be created in your project's directory. This file specifies which version of the packages were installed and locks your environment to those versions. You want both pyproject.toml and poetry.lock to be committed to your VCS, so the next time poetry install is called it would use package’s versions specified in poetry.lock. If you want to update versions later, you use poetry update command to update them all or poetry update django requests black to update specific dependencies. As for adding new dependencies to your project you use poetry add followed by the name of the package. This command will also install the package in your environment. With --dev option the package would be added as dev-dependency. What about getting into and out of virtual environment? poetry shell will activate it for you, and then you can use deactivate to get out of it just like with venv. There is much more you can do with poetry besides basic usage presented here, including built-in support for building and publishing your packages to the remote package repository (like pypi), but you can read it in a documentation on the official website: https://python-poetry.org/docs/

Poetry in dockerfile

When installing poetry inside docker container you don’t actually need the virtual environment, as those are only useful in local development. So, your command in dockerfile to install Poetry could look like this:

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | POETRY_HOME=/opt/poetry python && \\
  cd /usr/local/bin && \\
  ln -s /opt/poetry/bin/poetry && \\
  poetry config virtualenvs.create false

Then let’s copy pyproject.toml and also poetry.lock:

COPY ./app/pyproject.toml ./app/poetry.lock* /src/

After that we install all our dependencies and we can set additional argument if we want to run our app in development mode for tests:

ARG INSTALL_DEV=false
RUN bash -c \"if [ $INSTALL_DEV == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi\"

And finally, we can copy the content of our src folder:

COPY ./src /src

Remember, that it is good practice to copy and install your dependencies in dockerfile first, and then copy the content of your src directory instead of using COPY . .. The second approach would cause the docker to unnecessarily reinstall all our dependencies on every change in code itself even if we didn’t add any new libraries! By splitting it and installing dependencies first, and later copying the code itself we leverage docker’s cached layers, so on changing the app’s code, only the COPY ./src /src layer would be actually rebuilt.

Conclusion

Poetry is the tool that combines the power of both pip and venv, making development process easier to handle. It has all the commands allowing you to manage your dependencies and virtual environments and is highly customisable on top of that. I dare to say, that if you are familiar with all the pip and venv functionality and concepts, then poetry is both easy to learn and straightforward to use. I encourage you to give it a try in your next project.