How to Create and Upload Your First Python Package to PyPI

David Amos

Creating and sharing a Python package can be a major milestone in your growth as a developer. It‘s an opportunity to contribute to the vibrant ecosystem of open-source software, and to create something that solves a real problem for yourself and others.

Python‘s package management system, centered around the Python Package Index (PyPI), makes it incredibly easy to share your code with the world. But packaging your code properly and navigating the distribution process can still be daunting, especially the first time.

In this comprehensive guide, we‘ll walk through the entire process of creating and uploading your first Python package to PyPI. We‘ll cover everything from structuring your package and writing tests to choosing a license and distributing your code. Along the way, we‘ll dive deep into tools and best practices used by professional Python developers.

Whether you‘re an aspiring full-stack developer looking to level up your skills, or a seasoned Pythonista ready to give back to the community, this guide will equip you with the knowledge and confidence to create and share high-quality Python packages. Let‘s get started!

The Power of Python Packages

Before we dive into the technical details, let‘s take a moment to appreciate the significance of Python packages.

At the time of writing, PyPI hosts over 400,000 packages, covering every domain imaginable – from web development and data science to machine learning and beyond. These packages represent the collective efforts of countless developers worldwide, each contributing a small piece to the larger puzzle of open-source software.

Figure 1: The number of packages on PyPI has grown exponentially over the past decade. (Source: PePy)

This abundance of packages is one of Python‘s greatest strengths as a language. With a well-written package, you can encapsulate complex functionality into a reusable, shareable unit of code. This not only saves you time and effort, but also allows others to benefit from your work.

Creating a package also forces you to design and structure your code in a modular, maintainable way. This is an essential skill for any full-stack developer working on large, complex codebases.

Of course, creating a successful package is not without its challenges. You need to follow best practices for packaging and distribution, write clear documentation and tests, and be prepared to maintain and update your code over time. But with the right approach and tools, these challenges are more than manageable.

Structuring Your Package

The first step in creating a Python package is deciding how to structure your code. While there‘s no one "right" way to do this, following established conventions will make your life easier and your package more user-friendly.

Here‘s a typical structure for a simple Python package:

my_package/
    src/
        my_package/
            __init__.py
            module1.py
            module2.py
    tests/
        test_module1.py 
        test_module2.py
    LICENSE
    pyproject.toml
    README.md
    setup.cfg

Let‘s break this down:

• my_package/ is the root directory, named after your package.
• src/ contains all source files. This separates your code from other files in the root directory.
• my_package/ inside src/ is the actual package directory, where your modules and __init__.py live.
• tests/ contains your test files (more on testing later).
• LICENSE specifies the terms under which your package can be used and distributed.
• pyproject.toml defines build system requirements and other package metadata.
• README.md provides an overview of your package, its features, and how to use it.
• setup.cfg contains configuration details for building and distributing your package.

This structure follows the "src layout" convention, which is recommended by the Python Packaging Authority for new projects. The key advantage of this layout is that it isolates your source code from other project files, reducing clutter and potential conflicts.

Naming Your Package and Modules

Choosing a good name for your package and its modules is important for clarity and discoverability. Here are some guidelines:

• Use a short, descriptive name that conveys the purpose of your package.
• Avoid names that are too generic or that clash with existing packages on PyPI.
• Follow PEP 8 conventions for naming: use lowercase, separate words with underscores, and prefer nouns for modules.
• Keep module names concise but descriptive, and organize them logically within your package.

For example, if you were creating a package for working with DNA sequences, you might name it dnatools, with modules like dnatools.sequence, dnatools.align, etc.

Creating Your Package Metadata

Once you have your package structure in place, the next step is to define your package metadata. This includes information like your package‘s name, version, dependencies, and supported Python versions.

The two main files for configuring your package metadata are setup.cfg and pyproject.toml.

setup.cfg

setup.cfg is a configuration file used by the setuptools package, which is the most common tool for building and distributing Python packages. Here‘s an example setup.cfg for our hypothetical dnatools package:

[metadata]
name = dnatools
version = 0.1.0
author = Your Name
author_email = [email protected]
description = A Python package for working with DNA sequences
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/yourusername/dnatools
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6
install_requires =
    biopython>=1.78

[options.packages.find]
where = src

This file includes metadata about your package (name, version, author, etc.), as well as options for building and installing it. Key points:

• package_dir and packages tell setuptools where to find your package‘s source code (src/) and modules (find:).
• python_requires specifies the Python versions your package supports.
• install_requires lists your package‘s dependencies.

pyproject.toml

pyproject.toml is a newer, standardized file for configuring Python projects. It‘s intended to eventually replace setup.cfg, but for now, it‘s often used in conjunction with it.

Here‘s a minimal pyproject.toml file:

[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"

This file specifies that your project uses setuptools for building and packaging. As pyproject.toml evolves, it will likely subsume more of the functionality currently handled by setup.cfg.

Writing Tests for Your Package

No package is complete without tests! Writing tests for your code helps ensure its correctness, prevents regressions, and makes it easier for others (including future you) to modify and extend your package with confidence.

For our dnatools package, we might write some tests for a dnatools.sequence module:

# tests/test_sequence.py

from dnatools.sequence import Sequence

def test_sequence_creation():
    seq = Sequence("ATGC")
    assert str(seq) == "ATGC"

def test_sequence_complement():
    seq = Sequence("ATGC")
    assert str(seq.complement()) == "TACG"

def test_sequence_reverse_complement():
    seq = Sequence("ATGC")
    assert str(seq.reverse_complement()) == "GCAT"

These tests cover basic functionality of the Sequence class, like creation, complementation, and reverse complementation. We use the assert statement to verify that the actual output matches the expected output.

To run these tests, you can use any test runner that‘s compatible with pytest-style tests, such as unittest or pytest. For example:

python -m unittest discover tests

This will automatically discover and run all tests in the tests/ directory.

As your package grows, you‘ll want to expand your test suite to cover more edge cases, error conditions, and integration scenarios. Aim for high test coverage, but remember that quality is more important than quantity. Focus on testing the most critical and complex parts of your code.

Packaging and Distributing Your Code

With your code and tests in place, you‘re ready to package and distribute your code. This process involves generating distribution packages and uploading them to PyPI (or TestPyPI for testing).

Creating Distribution Packages

The first step is to create distribution packages for your code. These are archive files that contain your package‘s source code, along with any necessary metadata and dependencies.

The two main types of distribution packages are:

• Source distribution (sdist): A compressed archive (usually .tar.gz) containing your package‘s source code.
• Wheel distribution (wheel): A built package that can be installed directly, without needing to be built locally.

To create these packages, run the following command from your package‘s root directory:

python setup.py sdist bdist_wheel

This will create an sdist and a wheel in a new dist/ directory.

Uploading to TestPyPI

Before uploading to the real PyPI, it‘s a good idea to test your package by uploading it to TestPyPI, a separate instance of PyPI for testing purposes.

First, create an account on TestPyPI and generate an API token for your account. Then, use the twine tool to upload your packages:

twine upload --repository testpypi dist/*

You‘ll be prompted for your TestPyPI username and password (API token). After uploading, your package will be available on TestPyPI.

Installing and Testing Your Package

To test your package, create a new virtual environment and install your package from TestPyPI:

python -m venv env
source env/bin/activate
pip install --index-url https://test.pypi.org/simple/ dnatools

Then, try importing and using your package in a Python shell or script. If everything works as expected, you‘re ready to upload to the real PyPI!

Uploading to PyPI

The process for uploading to PyPI is the same as for TestPyPI, except you‘ll use your regular PyPI account and omit the --repository testpypi flag:

twine upload dist/*

After uploading, your package will be publicly available on PyPI for anyone to install with pip:

pip install dnatools

Congratulations, you‘ve just published your first Python package!

Conclusion and Next Steps

Creating and sharing a Python package can be a rewarding experience, both personally and professionally. By encapsulating your code into a reusable, distributable format, you make it easier for others to benefit from your work and contribute back to the community.

As you continue to develop your package, here are some next steps to consider:

• Expand your package with new modules and features
• Refactor and optimize your code for performance and maintainability
• Engage with your users and gather feedback for improvements
• Contribute to other open-source projects and learn from their code and practices
• Explore advanced packaging techniques like using tox or nox for testing on multiple Python versions and platforms
• Set up continuous integration and delivery (CI/CD) for your package to automate testing and deployment

Remember, creating a successful package is an iterative process. Don‘t get discouraged if your first attempt isn‘t perfect – the important thing is to keep learning and improving.

I hope this guide has given you the knowledge and confidence to create and share your own Python packages. Happy packaging!