July 14, 2024

Use Poetry for Python Package and Dependency Management

python

I see more Python projects are using Poetry to manage packages and dependencies and want to learn what Poetry is and why it’s better than (pip + venv).

I found this article very useful, which has a nice comparison between poetry vs. pip + venv.

A quick summary based on my learning:

for simple projects, pip + venv works just fine - it comes with Python, no additional learning or setup needed.
for complex projects, poetry has features that make the package and dependency management easier, such as no manual maintenance of various requirements.txt files.

The following are some of my notes.

Installation

Install pipx:

brew install pipx
pipx ensurepath
sudo pipx ensurepath --global

Install/upgrade/uninstall Poetry:

pipx install poetry
pipx upgrade poetry
pipx uninstall poetry

Setup a Repo with Poetry

Create a repo as usual, clone the repo, and initialize with Poetry:

git clone https://github.com/harrywang/self-contained-python-repo-poetry.git
cd self-contained-python-repo-poetry/
poetry init

poetry init will guide you through creating your pyproject.toml config.

Now add/remove packages using poetry add/poetry remove (.venv is created in the project folder):

poetry add openai tqdm python-dotenv

which outputs:

% poetry add openai tqdm torch torchvision python-dotenv
Creating virtualenv self-contained-python-repo-poetry in /Users/harrywang/sandbox/self-contained-python-repo-poetry/.venv
Using version ^1.35.13 for openai

Updating dependencies
Resolving dependencies... (5.2s)
...

If you need to use the shell with virtual environment activated:

poetry shell

If you need the requirements.txt file for frameworks or systems that rely on that file (see docs):

poetry export --output requirements.txt
poetry export --without-hashes --output requirements.txt

If you clone a repo with poetry setup, you can install the packages: poetry install, which will install dependencies for all non-optional groups. By default, main group is created for dependencies.

You can create a test group: poetry add pytest --group test

Only install main: poetry install --only main

Exclude group: poetry install --without test,docs

Include group” poetry install --with docs

Install from a specific source: https://github.com/python-poetry/poetry/issues/7685 has following example for pytorch:

poetry source add --priority=explicit pytorch-gpu-src https://download.pytorch.org/whl/cu118
poetry add --source pytorch-gpu-src torch torchvision torchaudio