Research Computing and Software Development

Author: Boris Bolliet

Note

Teaching Schedule and Material for the MPhil in Data Intensive Science.

Note

The course is shared with other MPhil’s and programmes at the University of Cambridge. Any information about assessments, courseworks, etc. provided here and in class by Boris Bolliet are relevant to the MPhil in Data Intensive Science only. All assessments for this course for Economics and Data Science students is arranged separately by the Faculty of Economics. Please, contact the director of the MPhil in Economics and Data Science program Alexei Onatski with specific questions about the form and the arrangements of these assessments.

Goal

Equip you with the computing and software development skills that you need in order to work as researchers and data scientists in the age of artificial intelligence (i.e. not be replaced by AI).

Schedule

Time:

• Tuesdays: 12-1pm

• Thursdays: 12-1pm

• Fridays: 9-10am

Location:

Ray Dolby Centre, Cavendish Laboratory, University of Cambridge.

#	Date	Content
1	Thu 9, October	Motivations (see recording).
2	Fri 10, October	Getting started
3	Tue 14, October	Bash Environment
4	Thu 16, October	Python Environment
5	Fri 17, October	Git
6	Tue 21, October	Python Package: Initialisation
7	Thu 23, October	Python Package: Structure
8	Fri 24, October	Python Package: Distribution
9	Tue 28, October	Basic Computing in Python
10	Thu 30, October	More Advanced Computing in Python
11	Fri 31, October	Basic Signal Processing Tools
12	Tue 4, November	Test and Docs
13	Thu 6, November	Full Stack: FastAPI Backend
14	Fri 7, November	Full Stack: Next.js Frontend
15	Tue 11, November	Full Stack: Deployment
16	Thu 13, November	More Advanced Python Concepts
17	Fri 14, November	Multi-language Programming
18	Tue 18, November	Complexity: Introduction
19	Thu 20, November	Complexity: Big O Notation
20	Fri 21, November	Complexity: Fibonacci Sequence
21	Tue 25, November	Complexity: Insertion Sort
22	Thu 27, November	Complexity: Merge Sort
23	Fri 28, November	Coursework Demonstration
24	Tue 2, December	Complexity: Divide and Conquer

The full notes from the previous year are available here.

This course builds on material from James Fergusson, made availble to UoC students under the course archive.

Also for UoC students, the course moodle is here with links to our slides and discord server. Our lecture recordings are available online.

Lecture Notes

• 1. Getting started
  • 1.1. Operating Systems
  • 1.2. Graphical User Interface (GUI)
  • 1.3. Command Line Interface (CLI)
  • 1.4. Shell
  • 1.5. Linux
  • 1.6. Useful terminal commands
  • 1.7. File system hierarchy
  • 1.8. Hidden files
  • 1.9. Permissions
  • 1.10. Administrator rights
  • 1.11. Home directory
• 2. Environments
  • 2.1. Bash environment
  • 2.2. Python environment
  • 2.3. Symbolic links and aliases
  • 2.4. Workflow with environments
• 3. Conda
  • 3.1. Conda vs Python virtual environments
  • 3.2. Conda virtual environment
  • 3.3. Activate and deactivate the environment
  • 3.4. Installing more packages
  • 3.5. List all conda environments
  • 3.6. Remove a conda environment
  • 3.7. Access your conda environment from Jupyter
• 4. Git
  • 4.1. What is Git?
  • 4.2. Git platforms
  • 4.3. Forks
  • 4.4. Clones
  • 4.5. Branches
  • 4.6. Pulls
  • 4.7. Commits
  • 4.8. Pull requests
  • 4.9. Updating your fork
  • 4.10. Merge changes between branches
• 5. Python Package: Initialization and Structure
  • 5.1. What is a Python package?
  • 5.2. The folder structure of a Python package
  • 5.3. Working example
  • 5.4. Naming conventions
• 6. Python Package: Distribution
  • 6.1. Source and wheels
  • 6.2. Docker Images and Containers
• 7. Python Package Index
  • 7.1. PyPi
  • 7.2. Installing a package from PyPi
  • 7.3. Installing multiple packages
  • 7.4. Uploading your package to PyPi
  • 7.5. Versioning
• 8. Automatic documentation
  • 8.1. Docstrings
  • 8.2. What is automatic documentation?
  • 8.3. Sphinx
  • 8.4. Building your documentation
  • 8.5. Publishing your documentation
  • 8.6. Alternatives
• 9. Errors
  • 9.1. Types of errors in Python
  • 9.2. Exception handling
• 10. Tests
  • 10.1. Test suite with pytest
  • 10.2. Additional features
  • 10.3. Running the test suite
• 11. Computing in Python
  • 11.1. Basic computations
  • 11.2. Advanced computations
  • 11.3. Other tools
• 12. More Advanced Python
  • 12.1. Decorators
  • 12.2. Numpy Universal Functions (ufuncs)
  • 12.3. Logging
  • 12.4. Global and Local Variables
  • 12.5. Deep and Shallow Copies
  • 12.6. Lists and tuples
  • 12.7. Class polymorphism
  • 12.8. Just-in-time compilation
• 13. Multi-Language Programming
  • 13.1. Cython
  • 13.2. Differentiable Programming
• 14. Backend: FastAPI
  • 14.1. Declaration
  • 14.2. Shape of data
  • 14.3. Endpoint
  • 14.4. Port
  • 14.5. Handy extras
  • 14.6. Endpoint types
• 15. Frontend: Next js
• 16. Deploying an app
• 17. Continuous Integration
  • 17.1. Webhooks
  • 17.2. GitHub Actions

Miscellaneous

• VSCode and Remote SSH
  • Jupyter kernel
• Jupyter and IPython on CSD3

Example Classes

• Brownian motion simulations
• Curse of Dimensionality I: Position of the Problem
• Curse of Dimensionality II: Solution to the Problem

Useful Resources and further reading

• Duke’s Computational Statistics and Statistical Computing

• Phillip Lippe’s Machine Learning and Computing material

• Bruno Régaldo-Saint Blancard’s Numerical Physics exercises