Packing Python Packages

This tutorial walks you through the publication process of PyPI packages in your Python project’s Package Registry such as GitHub or GitLab. We will look at my first PyPI project as an example.

Creating a file structure

I create the following file structure for a toy package named SSH-Colab. This is the package that we will pack into a Pypi accepted package to distribute on the Internet. Now, save the accompanied main program code.py in a folder named SSHColab. Later, I will edit LICENSE, setup.py and README.md, all of which are mandatory files to be included in a Pypi package.

Note: it doesn’t matter how you name the package and its supporting Python files in the file structure. Neither does it matter which directory the package is located in your computer. The pip install command can be used with a package name that is totally different from what we specify at the moment. We will reach more details when editing setup.py.

libin@localmachine:~/PackageArchive/SSH_Colab_pypi$ tree
.
├── SSHColab         # name of package
│   ├── __init__.py
│   └── code.py      # main program
├── setup.py         # setup文件
├── LICENSE          # 
└── README.md        # 

Suppose the code.py Python file contains three functions, connect, kill and info.

import os
def connect():
    ...
def kill():
    ...
def info():
    ...

Then open the __init__.py file and add:

from .code import connect, kill, info

Editing the metadata file

Open the setup.py file and add into it basic information in a similar format to those listed below:

import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="SSH-Colab",
    version="0.1.2",
    author="Li-Pin Juan",
    author_email="lipin.juan02@gmail.com",
    description="Google Colab connection helper in setting up Ngrok tunnels for SSH, TPU and TensorBoard.",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/libinruan/SSHColab",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)

Check this document PEP 314 – Metadata for Python Software Packages v1.1 if you like to understand the definition of each field.

It’s noteworthy that the field name is the package name that is used in the pip install command when we install the package. It can be different from the package name we see/define in the file system.

Besides, every time we submit a new version, the value of the field version should be different from those in previous releases unless you had deleted the previous release in your pypi console (which is not a recommended practice). Otherwise, pypi or test.pypi would reject your submission.

Creating a .pypirc file

Create a .pypirc file (check PyPA doc for details) in home directory C:\Users\libin\.pypirc. Add:

[distutils]
index-servers =
  pypi
  testpypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: libinruan
password: abcdefghijklmnopq

[pypitest]
repository: https://test.pypi.org/legacy/
username: libinruan
password: abcdefghijklmnopq

Username and passwords should be consistent with your Pypi and test.Pypi accounts.

Editing the LICENSE file

Open the LICENSE file. Copy and paste the MIT license from this webpage.

Packing the project

Ensure pip, setuptools and wheel are up to date.

$ cd ~/PackageArchive/SSH_Colab_pypi
$ pip install --user --upgrade pip setuptools wheel

Check whether there exists a package name conflicts:

$ python setup.py check

Packing the whole package:

$ python setup.py sdist bdist_wheel

The Distutils sdist command will extract the metadata fields from the arguments and write them to a file in the generated zipfile or tarball. This file will be named PKG-INFO and will be placed in the top directory of the source distribution (where the README, INSTALL, and other files usually go).

In the generated dist folder, you would find something similar to this:

$ cd ./dist
$ tree
.
├── SSH-Colab-0.1.2.tar.gz
└── SSH_Colab-0.1.2-py3-none-any.whl

Uploading the package

In the test phase, run:

$ cd ~/PackageArchive/SSH_Colab_pypi
$ python -m twine upload --repository pypitest dist/*

In the production phase, run:

$ cd ~/PackageArchive/SSH_Colab_pypi
$ python -m twine upload --repository pypi dist/*

Submitting new releases

Firstly, check if there are any conflicts:

$ cd /your/package/root/directory
$ python setup.py check # Be mindful that the package version is updated
$ python setup.py sdist bdist_wheel
$ python -m twine upload --repository pypitest dist/*
$ python -m twine upload --repository pypi dist/* 

8. Conclusion

I followed the same procedure to distribute my first Python package on the Internet. The package I created can be downloaded from Pypi using the command:

$ pip install SSH-Colab

For readers who are interested to fork the project, the GitHub homepage of the package can be found via this link.

Troubleshooting

If you encounter the error message saying bash: twine: command not found and you are working in a Conda virtual environment, you may solve this problem by simply running:

conda install twine

If your upload to test.pypi fails, check if the index server’s name specified in .pypirc file is consistent with your bash command. For example, if in the .pypirc the name of test.pypi is testpypi (“pypi” has a prefix “test”), and you change the order of the strings, the upload would surely fail.

If the upload of a updated project were rejected, check if the increment of package version is omitted.

References

[1]	Packaging Python Projects, Python.org
[2]	PyPI packages in the Package Registry, GitLab
[3]	零基礎編程——Python模塊項目打包發布PyPI，pip可安裝
[4]	python核心 - 打包与发布
[5]	Using hyphen/dash in python repository name and package name, StackOverflow