{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "VVYeubPuDFyi" }, "source": [ "# Getting started\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Operating Systems \n", "\n", "Operating systems are referred to as OS. \n", "\n", " \"Operating\n", " \n", "The three main OSes are:\n", "\n", "- Windows (Microsoft)\n", "- MacOS (Apple) -- in fact a Unix-based OS\n", "- Linux (Open Source)\n", "\n", "Google GCP, Amazon AWS and Microsoft Azure are not traditional provide virtualized OS instances \"on the cloud\".\n", "\n", "They are the interface between you and the hardware.\n", "\n", "**Note**: MacOS and Linux platforms are sometimes referred to as POSIX (\"Portable Operating System Interface for UNIX\").\n", "\n", "\n", "## Graphical User Interface (GUI)\n", "\n", "The GUI is a visual interface to the OS.\n", "\n", "It allows you to interact with the OS using graphical icons and a mouse.\n", "\n", "Common examples of GUIs are:\n", "\n", "- Windows Explorer\n", "- MacOS Finder\n", "\n", "\n", "## Command Line Interface (CLI)\n", "\n", "The CLI is a text-based interface to the OS.\n", "\n", "It allows you to pass commands to the OS. \n", "\n", "It displays a plain text-based screen with a prompt (e.g., `>` or `$`) where you can type commands.\n", "\n", "Common examples of CLIs are:\n", "\n", "- Unix/Linux Terminal \n", "- macOS Terminal \n", "- Windows Terminal\n", "\n", "Note concerning Windows: In this course we will not cover Windows or MacOS in details, but rather Linux. \n", "Nonetheless, MacOS and Windows have Unix-like kernels. In particular, Windows Users should refer to the Windows Subsystem for Linux (WSL), which allows them to access a Linux subsystem. \n", "See this [LinkedIn Learning Tutorial](https://www.linkedin.com/learning/learning-windows-subsystem-for-linux-16134127/windows-subsystem-for-linux) recommended by Hadi (MPhil DiS 1).\n", "See also the Microsoft docs: [Install WSL on Windows](https://learn.microsoft.com/en-us/windows/wsl/install). Ask ChatGPT. \n", "\n", "## Shell\n", "\n", "The shell interprets and executes the commands typed in the CLI.\n", "\n", "Common shells are:\n", "\n", "- Unix/Linux: Bash, Zsh\n", "- Windows: PowerShell\n", "\n", "Open a terminal on your OS and type: \n", "\n", "```bash\n", "echo $SHELL\n", "```\n", "\n", "This will print the shell you are using. \n", "\n", "On CSD3 it prints: \n", "\n", "```bash\n", "/bin/bash\n", "```\n", "\n", "On a Mac it prints: \n", "\n", "```bash\n", "/bin/zsh\n", "```\n", "\n", "BASH stands for [Bourne Again SHell](https://en.wikipedia.org/wiki/Bash_(Unix_shell)), and ZSH stands for [Z Shell](https://en.wikipedia.org/wiki/Z_shell) (an enhanced version of the BASH shell). We will not make the difference between the two in this course and refer to them simply as \"the shell\".\n", "\n", "\n", "A shell script is a program that can be directly executed by the shell. It consists of a sequence of commands that you could write down in your terminal. \n", "\n", "\n", "## Linux\n", "\n", "Linux is the most widely used OS in scientific computing. In fact it refers to a family of OS which are Unix-like. (MacOS is also Unix-like, but not Windows!)\n", "\n", "Popular Linux distributions (or *distros*) include **Debian**, **Fedora**, **Arch**, and **Ubuntu**. \n", "Commercial distributions include **Red Hat Enterprise** Linux and **SUSE Linux Enterprise**. \n", "Desktop Linux distributions include a windowing system such as X11 or Wayland and a desktop environment such as GNOME, KDE Plasma or Xfce ([Wikipedia](https://en.wikipedia.org/wiki/Linux)).\n", "\n", "The Ice Lake nodes on CSD3 (that you are probably going to use) are running Rocky Linux 8, which is a rebuild of Red Hat Enterprise Linux 8 (RHEL8).\n", "\n", "Google Colab also runs on a Linux environment. The virtual machines (VMs) behind Google Colab are based on Ubuntu. Today, Colab's default runtime is Ubuntu 20.04 LTS.\n", "\n", "You can type:\n", "```bash\n", "uname -a\n", "```\n", "to get know what your kernel/runtime is. This command in Unix/Linux and macOS systems displays various details about the system, such as:\n", "\n", "- Kernel name\n", "- Node name (hostname)\n", "- Kernel version and release\n", "- Machine hardware name\n", "- Processor architecture\n", "- OS\n", "\n", "\n", "Note that in when you install Windows Subsystem for Linux (WSL), you will be prompted to choose a Linux distribution. By default, it is Ubuntu.\n", "\n", "\n", "## Useful terminal commands\n", "\n", "We refer to the following commands throughout the course, and call them \"bash commands\".\n", "\n", "**echo**\n", "\n", "```bash\n", "echo \n", "```\n", "prints the text.\n", "\n", "We use `echo` to display the value of a variable.\n", "\n", "```bash\n", "NAME=\"Boris\"\n", "echo \"Hello, $NAME\"\n", "```\n", "\n", "will print:\n", "\n", "```bash\n", "Hello, Boris\n", "```\n", "\n", "**pwd**\n", "\n", "```bash\n", "pwd\n", "```\n", "tells you where you are in the file system.\n", "\n", "**ls**\n", "\n", "```bash\n", "ls \n", "```\n", "lists the files and directories in the specified directory.\n", "\n", "We often use the following options with `ls`:\n", "\n", "- `-a`: all files (including hidden files, see [section on hidden files](#Hidden-files) below)\n", "- `-l`: long format (with permissions, number of links, owner, group, size, and timestamp)\n", "- `-h`: human readable (e.g., 1493934 bytes -> 1.4 MB)\n", "- `-S`: sort by size from largest to smallest\n", "So you would type:\n", "\n", "```bash\n", "ls -alhS\n", "```\n", "\n", "**du**\n", "\n", "```\n", "du -sh . \n", "```\n", "\n", "shows the disk usage of the current directory (\".\") as one total.\n", "\n", "The `-s` option is for summary, and `-h` for human readable. \n", "\n", "**cd**\n", "\n", "```bash\n", "cd \n", "```\n", "changes the current directory to the specified directory.\n", "\n", "**mkdir**\n", "\n", "```bash\n", "mkdir \n", "```\n", "creates a new directory.\n", "\n", "**cp**\n", "\n", "```bash\n", "cp \n", "```\n", "copies the source file to the destination directory.\n", "\n", "**mv**\n", "\n", "```bash\n", "mv \n", "```\n", "moves the source file to the destination directory.\n", "\n", "**rm**\n", "\n", "```bash\n", "rm \n", "```\n", "removes the file.\n", "\n", "**recursive option**\n", "\n", "```bash\n", "rm -r \n", "```\n", "removes the directory and all its contents. \n", "\n", "```bash\n", "rm -rf \n", "```\n", "removes the directory and all its contents without asking for confirmation.\n", "\n", "```bash\n", "rm -rf /*\n", "```\n", "removes all the files in the directory without removing the directory itself.\n", "\n", "**Important:** `rm -rf` and `rm -r` are very dangerous commands that will delete files and directories without asking for confirmation. Use them with great caution.\n", "\n", "\n", "```bash\n", "cp -r \n", "```\n", "copies the source directory folder and all its contents to the destination directory.\n", "\n", "Note: `mv` moves files and directories by default, including all the contents within directories. It does not need the `-r` option.\n", "\n", "\n", "**git**\n", "\n", "Git is distributed version control system (VCS) that tracks versions of files ([Wikipedia](https://en.wikipedia.org/wiki/Git)). \n", "We use it to maintain collaborative research projects.\n", "It will be covered in details in the course.\n", "\n", "Some useful git bash commands that you will often use are:\n", "\n", "- `git clone `: to clone a remote repository\n", "- `git status`: to check the status of your repository\n", "- `git add `: to add a file to the staging area\n", "- `git commit -m \"\"`: to commit the changes you added to the staging area\n", "- `git push`: to push your changes to the remote repository\n", "\n", "Note that git is not a native bash command. It is a separate utility that can be installed on most Unix-based systems (like Linux and macOS) through package managers such as `apt` (on Ubuntu/Debian), `yum` (on CentOS/RHEL) or `brew` (on macOS).\n", "\n", "It is often installed by default on HPC systems, and is installed on Colab.\n", "\n", "\n", "
\n", "**Exercise:** Clone our gitlab course repository with git.\n", "\n", "Click on the badge to open the example notebook in Colab: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisbolliet/ResearchComputing/blob/main/docs/material/part1/gitlab.ipynb)\n", "
\n", "\n", "In this Colab notebook you can also experiment with most of the bash commands we have seen above.\n", "\n", "\n", "\n", "**tree**\n", "\n", "```bash\n", "tree -L \n", "```\n", "lists the files and directories in the specified directory in a tree-like structure.\n", "`` is a number indicating the depth of the tree to be displayed.\n", "\n", "Note that `tree` is not a native bash command. It is a separate utility that can be installed on most Unix-based systems (like Linux and macOS) through package managers such as `apt` (on Ubuntu/Debian), `yum` (on CentOS/RHEL) or `brew` (on macOS).\n", "\n", "To install `tree`, you can use the following commands depending on your system:\n", "\n", "- On Ubuntu/Debian:\n", " ```bash\n", " sudo apt-get install tree\n", " ```\n", "\n", "- On CentOS/RHEL:\n", " ```bash\n", " sudo yum install tree\n", " ```\n", "\n", "- On macOS (with Homebrew):\n", " ```bash\n", " brew install tree\n", " ```\n", "\n", "After installing, you can use `tree` to display directory structures in a tree-like format.\n", "\n", "## File system hierarchy\n", "\n", "\n", "Most Linux distributions follow the following [Filesystem Hierarchy Standard (FHS)](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard):\n", "\n", "- **/bin**: Essential binaries.\n", "- **/etc**: System configuration files.\n", "- **/usr**: User binaries, libraries, and documentation.\n", "- **/var**: Variable data like logs and spools.\n", "- **/dev**: Device files.\n", "- **/home**: User home directories.\n", "- **/tmp**: Temporary files.\n", "- **/proc**: Virtual files for system processes.\n", "\n", "\n", "The *root* directory is `/`. If you type: `tree -L 1 /` in CSD3 you will see something like this:\n", "\n", "```bash\n", "/\n", "├── bin -> usr/bin\n", "├── boot\n", "├── cgroup-sl\n", "├── datasets\n", "├── dev\n", "├── etc\n", "├── home\n", "├── IMAGE\n", "├── lib -> usr/lib\n", "├── lib64 -> usr/lib64\n", "├── local\n", "├── lost+found\n", "├── media\n", "├── mfa-data\n", "├── misc\n", "├── mnt\n", "├── net\n", "├── opt\n", "├── private\n", "├── proc\n", "├── ramdisks\n", "├── rcs\n", "├── rcs1\n", "├── rcs2\n", "├── rcs3\n", "├── rds\n", "├── rds-d2\n", "├── rds-d3\n", "├── rds-d4\n", "├── rds-d5\n", "├── rds-d6\n", "├── rds-d7\n", "├── rds-d8\n", "├── rfs\n", "├── root\n", "├── run\n", "├── sbin -> usr/sbin\n", "├── scratch\n", "├── slurm\n", "├── srv\n", "├── sys\n", "├── tmp\n", "├── usr\n", "└── var\n", "```\n", "\n", "If you do the same in Colab, you will see that the structure is similar:\n", "\n", "```bash\n", "/\n", "├── bin -> usr/bin\n", "├── boot\n", "├── content\n", "├── cuda-keyring_1.0-1_all.deb\n", "├── datalab\n", "├── dev\n", "├── etc\n", "├── home\n", "├── lib -> usr/lib\n", "├── lib32 -> usr/lib32\n", "├── lib64 -> usr/lib64\n", "├── libx32 -> usr/libx32\n", "├── media\n", "├── mnt\n", "├── NGC-DL-CONTAINER-LICENSE\n", "├── opt\n", "├── proc\n", "├── python-apt\n", "├── python-apt.tar.xz\n", "├── root\n", "├── run\n", "├── sbin -> usr/sbin\n", "├── srv\n", "├── sys\n", "├── tmp\n", "├── tools\n", "├── usr\n", "└── var\n", "```\n", "\n", "\n", "## Hidden files\n", "\n", "Files or directories that are not displayed by default when you list the contents of a directory. \n", "\n", "Used to **store configuration settings or metadata** that users do not usually need to see or modify directly.\n", "\n", "You can list hidden files in current directory by typing `ls -a`.\n", "\n", "Some of the most important hidden files for you, as researchers, are:\n", "\n", "- `.bashrc`: It contains the configuration for your bash shell, i.e., your bash environment (see next lectures).\n", "- `.bash_profile`: It is usually linked to `.bashrc`. Generally used on MacOS instead of `.bashrc`.\n", "- `.gitignore`: It specifies intentionally untracked files that Git should ignore.\n", "- `.ssh/`: It contains your SSH keys for connecting to remote servers (for instance CSD3).\n", "\n", "\n", "## Permissions\n", "\n", "Every file and directory has permissions that determine who can read, write, or execute it.\n", "\n", "You can view permissions by typing:\n", "\n", "```bash\n", "ls -l\n", "```\n", "\n", "Example output:\n", "\n", "```bash\n", "-rw-r--r-- 1 bb667 users 1234 Oct 9 10:30 notes.txt\n", "```\n", "\n", "Meaning:\n", "\n", "| Symbol | Meaning | Who it applies to |\n", "| :------------ | :------------------------------------ | :-------------------------- |\n", "| `r` | Read (view file contents) | |\n", "| `w` | Write (modify or delete file) | |\n", "| `x` | Execute (run file or enter directory) | |\n", "| `-` | No permission | |\n", "| Positions 1–3 | Owner permissions | The user who owns the file |\n", "| Positions 4–6 | Group permissions | Members of the same group |\n", "| Positions 7–9 | Other users | Everyone else on the system |\n", "\n", "\n", "(3 categories × 3 permissions = 9 positions)\n", "\n", "\n", "You can change permissions with `chmod`, change owner with `chown`, etc. \n", "\n", "Sometimes, you will see the notation:\n", "\n", "```\n", "chmod XYZ file\n", "```\n", "\n", "where `XYZ` is a number like `700` or `755` for example.\n", "Each of the letters `r`, `w`, `x` has a value (4,2,1) respectively. The digits in `XYZ` are just the sum of the letters values in each category (owner, group, user).\n", "\n", "\n", "## Administrator rights\n", "\n", "To get administrator rights on file system use: `sudo` (superuser do). You will only be able to do that if you are a user in the sudoer list.\n", "\n", "\n", "\n", "## Home directory\n", "\n", "The path to your home directory is stored in the **environment variable** `$HOME`.\n", "\n", "You can print it by typing:\n", "\n", "```bash\n", "echo $HOME\n", "```\n", "\n", "On CSD3 it prints:\n", "\n", "```bash\n", "/home/\n", "```\n", "where `` is your username (e.g., `bb667`).\n", "\n", "A shortcut to your home directory is `~`. You can change directory to your home directory by typing:\n", "\n", "```bash\n", "cd ~\n", "```\n", "\n", "which is equivalent to:\n", "\n", "```bash\n", "cd $HOME\n", "```\n", "\n", "On Colab if you type `echo $HOME` it prints:\n", "\n", "```bash\n", "/root\n", "```\n", "\n", "\n", "\n", "\n", "\n" ] } ], "metadata": { "colab": { "authorship_tag": "ABX9TyMxA3MikQu6HqzjYlVwK9lQ", "mount_file_id": "1ReIH5du5mFG-773sn9dNEA04qoevEq5O", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python (benv)", "language": "python", "name": "benv" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" } }, "nbformat": 4, "nbformat_minor": 4 }