{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Python Package: Distribution\n", "\n", "To illustrate the process we continue using the `company_package` that we developed in \n", "[part 4](../part4/notebook.html#Working-example).\n", "\n", "\n", "In principle, the code of your package should be developed with `git` and uploaded on GitHub \n", "(or GitLab, or Bitbucket, etc.).\n", "\n", "If you want to keep your code private you can of course do so, while developing it and sharing it with your chosen colleagues by inviting them to your private repository.\n", "\n", "\n", "Here, we will assume your are ready to share your code with the world and \n", "that you want it to be usable by users on Mac OS, Linux, and Windows machines. \n", "\n", "## Source and wheels\n", "\n", "\n", "When we used \n", "\n", "```bash\n", "pip install -e .\n", "```\n", "inside our `company_package` folder, we created a sub-folder called `company.egg-info`, which contains metadata about the package.\n", "\n", "
\n", "**Exercise:** Look inside the `company.egg-info` folder and check-out the content of the files there. \n", "
\n", "\n", "More importantly, we also automatically created a new folder called `dist-info` inside our environment site-packages folder, which contains the same sort of information.\n", "\n", "This folder will typically be at \n", "\n", "```bash\n", "//lib/python3.X/site-packages/company-.dist-info\n", "```\n", "\n", "
\n", "**Exercise:** Look inside the `dist-info` folder of your site-packages folder and check-out the content of the files there. \n", "
\n", "\n", "One of the files is `direct_url.json`. In our case it shows:\n", "\n", "```json\n", "{\"dir_info\": {\"editable\": true}, \"url\": \"file:///path/to/company_package\"}\n", "```\n", "\n", "which means we have installed the package in editable mode and the location of the **source code** is given by the `url` field.\n", "\n", "\n", "However, we have not **built** the package yet, i.e., we have not created a binary distribution of the code.\n", "\n", "Let's do it. To do so, in a Terminal we run:\n", "\n", "```bash\n", "python -m build\n", "```\n", "\n", "from inside the package folder. Executing the command may throw out an error like:\n", "\n", "```\n", "No module named build.__main__; 'build' is a package and cannot be directly executed\n", "```\n", "\n", "If so, you simply need to degrade your build version by running the following command:\n", "\n", "```bash\n", "pip install 'build<0.10.0'\n", "```\n", "\n", "At the end of the process, a `dist` folder should be created and we should see something like:\n", "\n", "```\n", "Successfully built company-0.0.0b1.dev6+gfa37a84.d20241029.tar.gz and company-0.0.0b1.dev6+gfa37a84.d20241029-py3-none-any.whl\n", "```\n", "\n", "These two files, stored in the `dist` folder, are the source distribution and the wheel distribution of the package. \n", "\n", "The wheel distribution is a binary distribution of the package. (In fact, for a pure Python package, it amounts to an archive of the package efficiently organized. For a package with compiled extensions, it also contains the compiled files.)\n", "\n", "Installing from the wheel can be much faster than from the source. To do so, we can run:\n", "\n", "```bash\n", "pip install ---.whl\n", "```\n", "\n", "(However, note that this is not allowing you to install the package in editable mode)\n", "\n", "To see what's inside the wheel, we can extract it using:\n", "\n", "```bash\n", "unzip ---.whl -d \n", "```\n", "\n", "
\n", "**Exercise:** Create, install and extract the wheel distribution of the `company` package and inspect its content.\n", "
\n", "\n", "## Docker Images and Containers\n", "\n", "Here we notice that our wheel file says `py3-none-any`. This means that it is compatible with Python 3 (any version), \n", "and will work on any platform (macOS, Linux, Windows, etc.) and any architecture (x86, arm, etc.).\n", "\n", "This is generally the case for pure Python packages.\n", "\n", "For more complex packages that involve compiled extensions, we will see how to build wheels for multiple platforms and architectures using \n", "[**cibuildwheel**](https://cibuildwheel.readthedocs.io/en/stable/).\n", "\n", "[**cibuildwheel**](https://cibuildwheel.readthedocs.io/en/stable/) is a tool that relies on [**Docker**](https://en.wikipedia.org/wiki/Docker_(software)).\n", "\n", "With [**Docker**](https://en.wikipedia.org/wiki/Docker_(software)) we can test how our package installs and behaves on different platforms.\n", "\n", "We will cover [**cibuildwheel**](https://cibuildwheel.readthedocs.io/en/stable/) in more details later in the course and focus on Docker for now.\n", "\n", "Docker can be used in CLI but also through a graphical interface known as [Docker Desktop](https://www.docker.com/products/docker-desktop/). \n", "\n", "You are encouraged to install it.\n", "\n", "To put things simply, with Docker we create a sort of virtual Linux machine on \n", "our machine. This virtual machine has its own operating system and can be seen as completely isolated from the rest of our local machine.\n", "\n", "The key step to set-up Docker is to create a so-called **Dockerfile**. It is a script detailing the setup of the environment, including dependencies, for the package.\n", "\n", "For our `company_package`, a valid Dockerfile could be: \n", "\n", "```dockerfile\n", "# Use an official Python image as the base image\n", "FROM python:3.12-slim\n", "\n", "# Install Git\n", "RUN apt-get update && apt-get install -y git\n", "\n", "\n", "# Set the working directory in the container\n", "WORKDIR /app\n", "\n", "# Copy the project files to the working directory\n", "COPY . /app\n", "\n", "# Install required dependencies for building the package\n", "RUN pip install --upgrade pip setuptools wheel setuptools_scm build\n", "\n", "# Install runtime dependencies listed in pyproject.toml\n", "RUN pip install .\n", "\n", "# Build the package\n", "RUN python -m build\n", "```\n", "\n", "(See [here](https://github.com/borisbolliet/company_package/blob/main/Dockerfile).)\n", "\n", "\n", "The next step is to **build the Docker image**.\n", "\n", "First, we check that Docker is installed and active on our machine. To do so, we can run:\n", "\n", "```bash\n", "docker info\n", "```\n", "\n", "If Docker is not active. The easiest way to start it is to open Docker Desktop. Then the command above should work (and print info about the Docker version on your machine).\n", "\n", "\n", "
\n", "**Exercise:** Install Docker on your machine and try the `docker info` command.\n", "
\n", "\n", "\n", "With docker active, we **build** the Docker image by running:\n", "\n", "```bash\n", "docker build -t .\n", "```\n", "\n", "In Docker desktop we should be able to see the image being built.\n", "\n", "To list the images on our machine we can run:\n", "\n", "```bash\n", "docker images\n", "```\n", "\n", "Finally, we can **run** the Docker image in a container by running:\n", "\n", "```bash\n", "docker run -it \n", "```\n", "\n", "For our `company_package`, this generates a container that has its own Python environment and can be used to test the package in this isolated environment. The `it` option means **interactive**: the command will open a Python shell in the container (and the container will stop when you exit the Python shell). The output looks like:\n", "\n", "```bash\n", "docker run -it company-image\n", "Python 3.11.10 (main, Oct 19 2024, 03:39:30) [GCC 12.2.0] on linux\n", "Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n", ">>> import company as cp\n", "Company package version: 0.0.0b1.dev7+g74c5191.d20241029\n", ">>> \n", "```\n", "\n", "Now, you understand that you can use Docker to distribute your package in a very robust way. Indeed, by providing a Dockerfile, other users can readily test and use your package on their own machines by running:\n", "\n", "```bash\n", "docker build -t .\n", "docker run -it \n", "```\n", "\n", "With this method, users do not need to worry at all about dependencies of your package, python version, etc. Everything is specified in the Dockerfile.\n", "\n", "This covers the essential aspects of Docker and hopefully conveys the idea that it is a very powerful tool for software development. It is worth noting that the Docker (and Remote - Containers) extensions of VSCode allow you to develop and debug inside the container.\n", "\n", "
\n", "**Exercise:** Use the VSCode extensions to set-up and run a container based on the `company_package` Docker image.\n", "
\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "c1_base_env", "language": "python", "name": "c1_base_env" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" } }, "nbformat": 4, "nbformat_minor": 4 }