{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Environments\n", "\n", "\n", "There are two main types of environments that you need to consider when developing code in Python.\n", "The `bash` environment which knows about the shell and your OS, and the `python` environment which knows about the python packages you are using.\n", "\n", "\n", "## Bash environment\n", "\n", "The bash environment is defined by a set of environment variables. \n", "\n", "### Environment variables\n", "\n", "It is a key-value pair, stored by the OS, used by programs and shell scripts to configure the system behavior. \n", "\n", "The key is the variable name, and the value is the variable value.\n", "\n", "To set an environment variable, you can use the `export` command. For example:\n", "\n", "```bash\n", "export MY_VARIABLE=\"my_value\"\n", "```\n", "\n", "To print the value of an environment variable, you can use the `echo` command. For example:\n", "\n", "```bash\n", "echo $MY_VARIABLE\n", "```\n", "\n", "with the dollar sign `$` to get the value of the variable (think of dollar/value).\n", "\n", "You can list the environment variables currently set in your bash session by typing:\n", "\n", "```bash\n", "printenv\n", "```\n", "\n", "This is a bash command that is **very important**, you should not forget about it.\n", "\n", "\n", "
\n", "**Exercise:** Connect to CSD3 and list the environment variables.\n", "
\n", "\n", "\n", "The main environment variables you need to know are:\n", "\n", "- `PATH`: Defines the directories where the system searches for executable programs. It's essential for adding research software tools like compilers, or compiled programs.\n", "\n", "
\n", "**Exercise:** On your local machine list the executables found in some of the directories in your `PATH`.\n", "
\n", "\n", "\n", "- `LD_LIBRARY_PATH` (Linux) / `DYLD_LIBRARY_PATH` (macOS): Used to specify additional directories where the system looks for shared libraries (.so or .dylib files). For example, such librairies can be math libraries (e.g., [LAPACK](https://en.wikipedia.org/wiki/LAPACK), [BLAS](https://en.wikipedia.org/wiki/BLAS), [FFTW](https://en.wikipedia.org/wiki/FFTW), [GSL](https://en.wikipedia.org/wiki/GNU_Scientific_Library), etc.).\n", "\n", "Note: In `LD_LIBRARY_PATH`, the `LD` stands for \"Linker/Loader\". It refers to the **dynamic linker or dynamic loader**, which is the part of the operating system responsible for loading shared libraries (also known as dynamic libraries) when an executable program is run.\n", "\n", "\n", "- `HOME`: Your home directory.\n", "\n", "\n", "- `USER`: Holds your username.\n", "\n", "- `PYTHONPATH`: Used to specify additional directories where the system looks for Python packages. Typically, where the systems look when you run `import ...` in Python.\n", "\n", "\n", "
\n", "**Exercise:** Open a Colab notebook and print the values of each of these environment variables.\n", "\n", "Remember to use the `!` to run the bash commands in Colab.\n", "
\n", "\n", "### Adding to your `PATH`\n", "\n", "You can add to your `PATH` by appending the directory to the `PATH` variable. For example:\n", "\n", "```bash\n", "export PATH=$PATH:/path/to/your/program\n", "```\n", "\n", "You can also prepend to your `PATH` by adding the directory at the beginning of the `PATH` variable. For example:\n", "\n", "```bash\n", "export PATH=/path/to/your/program:$PATH\n", "```\n", "\n", "**Important**: The order in which you add directories to your `PATH` is important. The first directory in the `PATH` is the first one that is searched.\n", "\n", "
\n", "**Exercise:** Add a directory to your `PATH` and check that it works. If you have a directory that contains an executable, add this one and try running the program from anywhere in the system.\n", "
\n", "\n", "\n", "### Adding to your path variables\n", "\n", "You can add to any path variables in exactly the same way as we have seen for `PATH` above.\n", "\n", "For example, to add a directory to your `PYTHONPATH`, you can use:\n", "\n", "```bash\n", "export PYTHONPATH=/path/to/your/program:$PYTHONPATH\n", "```\n", "\n", "### Making the changes persistent\n", "\n", "You can make the changes persistent by adding them to your `~/.bashrc` (or `~/.bash_profile` on macOS) file. For example, to add a directory to your `PATH`, you can type in your terminal:\n", "\n", "```bash\n", "echo \"export PATH=/path/to/your/program:$PATH\" >> ~/.bashrc\n", "```\n", "\n", "You can then reload the `~/.bashrc` file by typing:\n", "\n", "```bash\n", "source ~/.bashrc\n", "```\n", "\n", "so the changes are applied in your current shell as well. \n", "\n", "\n", "
\n", "**Exercise:** Do the following:\n", "\n", "1. Add a fake directory to your `PATH`. For instance `/path/to/nowhere`, using the `export` command.\n", "2. Use echo to check the `PATH` has been updated.\n", "3. Close your current shell (i.e., Terminal) and re-open it (or open a new one).\n", "4. Check the `PATH` and note that the fake path is not there.\n", "5. Redo 1-4 but this time modify the `~/.bashrc` file to make the change persistent before step 3.\n", "6. Check that the change is persistent.\n", "\n", "To undo the change, remove the new line added to the `~/.bashrc` file. Use vim or VS code to do so. \n", "
\n", "\n", "\n", "\n", "## Python environment\n", "\n", "Python was invented in December 1989 by [Guido van Rossum](https://gvanrossum.github.io) at the Centrum Wiskunde & Informatica (abbr. CWI; English: \"National Research Institute for Mathematics and Computer Science\") in the Netherlands.\n", "\n", "\n", "Python is one of the most popular programming languages, for its widespread use in machine learning and data science. \n", "\n", "The name Python comes from the British comedy group Monty Python. You will occasionnaly find some further references, such as the use of the terms \"spam\" and \"eggs\". Python is fun. \n", "\n", "As researcher data scientists, your Python environment consists mainly of **three components**:\n", "\n", "1. Python interpreter/version,\n", "2. Virtual environment(s),\n", "3. Jupyter notebook and kernel.\n", "\n", "### Python interpreter\n", "\n", "\n", "The **Python interpreter** is the program that **runs your Python code**. It is the interface between your code and the Python language.\n", "\n", "Python (i.e., a Python interpreter) may not be natively installed on your system. \n", "\n", "If you type `python` in your terminal, and get an error such as:\n", "\n", "```bash\n", "bash: python: command not found\n", "```\n", "\n", "then you need to install Python.\n", "\n", "**On MacOS** you can install Python (i.e., a Python interpreter) using Homebrew, e.g.,\n", "\n", "```bash\n", "brew install python\n", "```\n", "\n", "**On Linux** you can install Python using your package manager, e.g.,\n", "\n", "```bash\n", "sudo apt-get install python\n", "```\n", "\n", "**On CSD3** several Python interpreters are available. You can load the one you need using the `module` command. For example:\n", "\n", "```bash\n", "module load miniconda/3\n", "```\n", "\n", "You can then check the default python version you get from the loaded module by typing:\n", "\n", "```bash\n", "python --version\n", "```\n", "\n", "If you then start a Python session by typing `python` in your terminal, you should see something like this:\n", "\n", "```bash\n", "Python 3.7.4 (default, Aug 13 2019, 20:35:49) \n", "[GCC 7.3.0] :: Anaconda, Inc. on linux\n", "Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n", ">>> \n", "```\n", "The `>>>` is the **Python prompt**. You can type Python code between the prompt and **press Enter to execute the code**.\n", "\n", "To exit the Python interpreter, you can type `exit()` and press Enter. \n", "\n", "Note for CSD3: if you don't use this command, other Pythons are available, but you need to explicitly type the python version you want to use. They are stored in `/usr/bin`. You can list them using `ls /usr/bin/python*`.\n", "\n", "\n", "\n", "\n", "\n", "Now, in a Python session, type `import platform; platform.python_implementation()` and press Enter. You should see something like this:\n", "\n", "```bash\n", "'CPython'\n", "```\n", "\n", "Your Python interpreter is **CPython**. This is the reference implementation of the Python programming language. Written in C and Python, CPython is the default and most widely used implementation of the Python language ([Wikipedia](https://en.wikipedia.org/wiki/CPython#:~:text=CPython%20is%20the%20reference%20implementation,implementation%20of%20the%20Python%20language.)).\n", "\n", "To check the version of Python you are using from within Python, type:\n", "\n", "```python\n", "import sys\n", "print(sys.version)\n", "```\n", "\n", "
\n", "**Exercise:** Check the version of Python and interpreter you are using, in Colab, your laptop and CSD3.\n", "
\n", "\n", "\n", "There are other implementations of Python, such as [PyPy](https://en.wikipedia.org/wiki/PyPy), [Jython](https://en.wikipedia.org/wiki/Jython) and [IronPython](https://en.wikipedia.org/wiki/IronPython). You can read about them online. \n", "\n", "\n", "### Virtual environments\n", "\n", "\n", "If you type `pip list` in a Terminal you will see the packages currently installed in your Python environment.\n", "\n", "\n", "
\n", "**Exercise:** Try pip list in Colab (it runs the shell command from within the notebook), your laptop and CSD3.\n", "
\n", "\n", "\n", "Packages can have conflicts, and some software may require a specific version of a package. \n", "\n", "A Python environment is a directory that contains a Python interpreter and a curated set of installed packages readily available. \n", "\n", "Good practice is to create a new virtual environment for each project. \n", "\n", "There are multiple tools to create and manage virtual environments. Here we use `venv`. See [here](https://docs.python.org/3/library/venv.html) for the documentation. \n", "\n", "To create a new virtual environment, you can use the following command:\n", "\n", "First, create a directory to store the virtual environments, e.g., `venvs`.\n", "\n", "```bash\n", "mkdir venvs\n", "```\n", "\n", "Then, create the virtual environment in this directory, do:\n", "\n", "```bash\n", "python -m venv \n", "```\n", "\n", "Here, `-m venv` specifies that we use the `venv` module to create a virtual environment. The `-m` option tells Python to run the `venv` module.\n", "\n", "`` is the path to the directory where the virtual environment will be created, and `nameofenv` is the name of the virtual environment. Just above, we have created the directory `venvs`. We can use any name for the virtual environment, e.g., `venv_lecture2` or anything else.\n", "\n", "To activate the virtual environment, you can use the following command:\n", "\n", "```bash \n", "source \n", "```\n", "\n", "In the Terminal, you should see something like this:\n", "\n", "```bash\n", "(nameofenv) \n", "```\n", "\n", "Now if you run the command `pip list`, you will see the packages currently installed in your virtual environment. This should be different from the packages installed in your *global* Python environment.\n", "\n", "To deactivate the virtual environment, you can use the following command in the Terminal:\n", "\n", "```bash\n", "deactivate\n", "```\n", "\n", "This will return you to your global Python environment.\n", "\n", "\n", "
\n", "**Exercise:** Working with Virtual Environments\n", "\n", "1. Create a new virtual environment named 'venv_lecture2' using Python 3.11\n", "\n", " Hint: Use `python3.11 -m venv venvs/venv_lecture2`\n", "\n", "2. Activate the virtual environment\n", "\n", " Hint: Use `source venvs/venv_lecture2/bin/activate`\n", "\n", "3. Use the `tree` command to see the structure of your virtual environment\n", "\n", " Hint: Use `tree venvs/venv_lecture2`\n", "\n", "4. Check the Python version in your virtual environment\n", "\n", " Hint: Use `python --version`\n", "\n", "5. List all packages installed in your virtual environment\n", "\n", " Hint: Use `pip list`\n", "\n", "6. Install a new package (e.g., `numpy`) in your virtual environment\n", "\n", " Hint: Use `pip install numpy`\n", "\n", "7. Use `tree` again to see how the structure has changed after installing numpy\n", "\n", " Hint: Use `tree venvs/venv_lecture2`\n", "\n", "8. List the packages again to confirm the installation\n", "\n", " Hint: Use `pip list`\n", "\n", "9. Deactivate the virtual environment\n", "\n", " Hint: Use `deactivate`\n", "
\n", "\n", "\n", "**Important**: To create an environment with a specific Python version, just use the Python version number, e.g., `python3.11 -m venv /` when creating the environment.\n", "\n", "\n", "\n", "### Fresh and non-fresh environments\n", "\n", "You can create a virtual environment while allowing it to access packages installed in your global environment. To do so use the `--system-site-packages` option:\n", "\n", "```bash\n", "python -m venv --system-site-packages \n", "```\n", "\n", "
\n", "**Exercise:** Create an environment with access to your global packages.\n", "and use pip list to check that you can see the packages in your global environment.\n", "
\n", "\n", "\n", "This can get confusing though because some packages installed in your global environment may be incompatible with the new packages you will want to install in your virtual environment.\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "### Managing Python versions\n", "\n", "The Python language evolves fast. Currently, a new version is released every year.\n", "\n", "You can consult the status of Python versions [here](https://devguide.python.org/versions/) and read about Python development on the same website.\n", "\n", "You can install new Python versions on your machine using tools like Anaconda, Miniconda (i.e., `conda`) or Homebrew (on macOS, `brew`).\n", "\n", "
\n", "**Exercise:** Checking Python Versions on CSD3 with Miniconda\n", "\n", "1. Log in to CSD3 if you haven't already\n", "\n", "2. Load the Miniconda module\n", " Hint: Use `module load miniconda/3`\n", "\n", "3. List the available Python versions\n", " Hint: Type `which python` and then list the files in the directory returned by this command.\n", "\n", "4. Now list the versions available by default: `ls /usr/bin/python*` and try to start a Python session with one of them.\n", " Hint: Just type `python3.11` (or another version number) and press Enter.\n", "\n", "What is the difference between the versions available by default and the one loaded by the Miniconda module?\n", "
\n", "\n", "\n", "### Jupyter and IPython kernels\n", "\n", "Jupyter provides an interactive environment to run Python code.\n", "It typically opens in a web browser. The most common feature is the notebook, which corresponds to \n", "what you get in Colab.\n", "\n", "Jupyter is an evolution of IPython, \"interactive Python\". IPython is an interactive shell that supports more features than the standard Python shell. Type `ipython` in your terminal and you will understand why.\n", "\n", "We don't generally use IPython, but instead use Jupyter which is based on it. \n", "\n", "To start a Jupyter notebook, having access to your Python environment, you need to create the \n", "relevant kernels. One kernel is created for each environment.\n", "\n", "Assume your virtual environment is called `venv_lecture2` and you have activated it.\n", "\n", "To create the kernel for this environment, you can use the following command:\n", "\n", "```bash\n", "python -m ipykernel install --user --name venv_lecture2 --display-name \"Python (venv_lecture2)\"\n", "```\n", "\n", "This creates a file (kernel) that contains info about the environment and can be propagated to Jupyter. Typically this file is in:\n", "\n", "```bash\n", "/kernels/kernels//kernel.json\n", "```\n", "\n", "
\n", "**Exercise:** Create a new kernel from a new environment. Find the `kernel.json` file and look at it using `vim`.\n", "
\n", "\n", "\n", "You can then start a Jupyter notebook by typing `jupyter-lab` in your terminal.\n", "\n", "\n", "\n", "## Symbolic links and aliases\n", "\n", "When you load or source an environment or install new codes, aliases and symbolic links may be created.\n", "\n", "\n", "\n", "### Symbolic links\n", "\n", "A symbolic link is a file that is **pointer** to another file or directory. To create a symbolic link, you can use the `ln -s` command. For example:\n", "\n", "```bash\n", "ln -s /path/to/target /path/to/link\n", "```\n", "\n", "To see which file a symbolic link points to, you can use the `ls -l` command. For example:\n", "\n", "```bash\n", "ls -l /path/to/link\n", "```\n", "\n", "For instance, on CSD3, there are symbolic links to useful Job submission files. (A job means a heavy computation that is submitted to the batch system.)\n", "\n", "### Aliases\n", "\n", "An alias is a **shortcut for a command**. To list the aliases currently set in your bash session, you can use the `alias` command.\n", "\n", "```bash\n", "alias\n", "```\n", "\n", "For instance on CSD3, by default, we have\n", "\n", "```bash\n", "alias vi='vim'\n", "```\n", "\n", "So that typing `vi` is equivalent to typing `vim` and opens the vim editor.\n", "\n", "\n", "\n", "## Workflow with environments \n", "\n", "You create a Python environment for each project.\n", "You need to store the information about your projects and relevant environment in a well organised manner. \n", "\n", "A good way to stay organised and keep track of what you are doing is certainly to create a github repository for each project, with a README explaining what to setup and how to do it, and storing your codes in it.\n", "\n", "\n", "### Bash scripts to load environments\n", "\n", "To make your life easier, you can create bash scripts to load your environments.\n", "One bash script per environment. \n", "\n", "The script is used to update environment variables and also activate relevant Python virtual environment. It would look something like this:\n", "\n", "```bash\n", "#!/bin/bash\n", "\n", "# Step 1: Define environment variables\n", "export MY_VAR1=\"value1\"\n", "export MY_VAR2=\"value2\"\n", "export PATH=\"/my/custom/path:$PATH\"\n", "export LD_LIBRARY_PATH=\"/my/custom/path:$LD_LIBRARY_PATH\"\n", "\n", "# Step 2: Source the Python virtual environment\n", "# Replace the path below with the actual path to your virtual environment's 'activate' script\n", "source /path/to/your/venv/bin/activate\n", "\n", "# Optional: Print a message to confirm the environment is set\n", "echo \"Environment variables are set, and the virtual environment is activated.\"\n", "```\n", "\n", "and saved as `activate_my_env.sh`.\n", "\n", "You can then source the script to load the environment, including bash variables and the Python virtual environment:\n", "\n", "```bash\n", "source activate_my_env.sh\n", "```\n", "\n", "
\n", "**Exercise:** Create a bash script to load your Python environment and check that it works.\n", "
\n", "\n", "\n", "### Note on File Permissions\n", "\n", "You may have noticed that when you list with the `ls -alh` command, the first thing to appear is a string of letters and numbers, such as `-rwxr-xr-x`.\n", "\n", "This represents the **permissions** of the file.\n", "\n", "\n", "When you create a bash script, it is generally **not executable**. To make it executable, you can use the `chmod` command. For example:\n", "\n", "```bash\n", "chmod +x activate_my_env.sh\n", "```\n", "\n", "
\n", "**Exercise:** Show the permission of the bash script from above before and after making it executable with `chmod`.\n", "
\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 2 }