July 14, 2024

Use Poetry for Python Package and Dependency Management

I see more Python projects are using Poetry to manage packages and dependencies and want to learn what Poetry is and why it’s better than (pip + venv).

I found this article very useful, which has a nice comparison between poetry vs. pip + venv.

A quick summary based on my learning:

The following are some of my notes.

Installation

Install pipx:

brew install pipx
pipx ensurepath
sudo pipx ensurepath --global

Install/upgrade/uninstall Poetry:

pipx install poetry
pipx upgrade poetry
pipx uninstall poetry

Setup a Repo with Poetry

Create a repo as usual, clone the repo, and initialize with Poetry:

git clone https://github.com/harrywang/self-contained-python-repo-poetry.git
cd self-contained-python-repo-poetry/
poetry init

poetry init will guide you through creating your pyproject.toml config.

Now add/remove packages using poetry add/poetry remove (.venv is created in the project folder):

poetry add openai tqdm python-dotenv

which outputs:

% poetry add openai tqdm torch torchvision python-dotenv
Creating virtualenv self-contained-python-repo-poetry in /Users/harrywang/sandbox/self-contained-python-repo-poetry/.venv
Using version ^1.35.13 for openai

Updating dependencies
Resolving dependencies... (5.2s)
...

If you need to use the shell with virtual environment activated:

poetry shell

If you need the requirements.txt file for frameworks or systems that rely on that file (see docs):

poetry export --output requirements.txt
poetry export --without-hashes --output requirements.txt

If you clone a repo with poetry setup, you can install the packages: poetry install, which will install dependencies for all non-optional groups. By default, main group is created for dependencies.

You can create a test group: poetry add pytest --group test

Only install main: poetry install --only main

Exclude group: poetry install --without test,docs

Include group” poetry install --with docs

Install from a specific source: https://github.com/python-poetry/poetry/issues/7685 has following example for pytorch:

poetry source add --priority=explicit pytorch-gpu-src https://download.pytorch.org/whl/cu118
poetry add --source pytorch-gpu-src torch torchvision torchaudio