Research Computing and Software Development

Author: Boris Bolliet

Note

Teaching Schedule and Material for the MPhil in Data Intensive Science and the MPhil in Economics and Data Science, Michaelmas Term 2024 at the University of Cambridge.

Goal

This course aims to equip you with the computing and software development skills that you will need to work as researchers and data scientists in the age of artificial intelligence.

Schedule

Time: Tuesdays, Wednesdays, Fridays between 11:00 and 12:00. 24 lectures.

Location: Small Lecture Theatre, Cavendish Laboratory, University of Cambridge.

Date	Content
1 Fri 11, October	Part 1: Getting started
2 Tue 15, October	Part 2: Terminal
3 Wed 16, October	Part 2: Bash
4 Fri 18, October	Part 2: Python
5 Tue 22, October	Part 2: Jupyter
6 Wed 23, October	Part 3: Git
7 Fri 25, October	Part 4: Building your Python package
8 Tue 29, October	Part 5: Distributing your package
9 Wed 30, October	Part 5: Docker Images and Containers
10 Fri 1, November	Part 6 and 7: Docs and Tests
11 Tue 5, November	Part 8 and 9: PyPi and CI
12 Wed 6, November	Part 10 and 11: Conda and Essential tools
13 Fri 8, November	Part 11: Simple operations
14 Tue 12, November	Part 11: Optimisation
15 Wed 13, November	Part 11: Linear Algebra
16 Fri 15, November	Part 11: FFT
17 Tue 19, November	Part 12: Wavelet Scattering Transform
18 Wed 20, November	Part 12: Wavelet Scattering Transform [continued]
19 Fri 22, November	Part 13: Advanced Python
20 Tue 26, November	Part 14: Optimization and Algorithm Complexity
21 Fri 29, November	Part 15: Multi-language programming
22 Mon 2, December	Part 16: Random Matrices
23 Tue 3, December	Part 17: Multi-Agent Systems for Data Science
24 Wed 4, December	Recap and feedback

This course builds on material from James Fergusson’s made availble to UoC students under the course archive.

Also for UoC students, the course moodle is here with links to our slides and discord server. Our lecture recordings are here.

Lectures

• 1. Getting started
  • 1.1. Operating Systems
  • 1.2. Graphical User Interface (GUI)
  • 1.3. Command Line Interface (CLI)
  • 1.4. Shell
  • 1.5. Linux
  • 1.6. Useful terminal commands
  • 1.7. File system hierarchy
  • 1.8. Hidden files
  • 1.9. Home directory
• 2. Knowing your environment
  • 2.1. Bash environment
  • 2.2. Python environment
  • 2.3. Symbolic links and aliases
  • 2.4. Workflow with environments
• 3. Git
  • 3.1. What is Git?
  • 3.2. Git platforms
  • 3.3. Forks
  • 3.4. Clones
  • 3.5. Branches
  • 3.6. Pulls
  • 3.7. Commits
  • 3.8. Pull requests
  • 3.9. Updating your fork
  • 3.10. Merge changes between branches
• 4. Building your Python package
  • 4.1. What is a Python package?
  • 4.2. The folder structure of a Python package
  • 4.3. Working example
  • 4.4. Naming conventions
• 5. Distributing your package
  • 5.1. Source and wheels
  • 5.2. Docker Images and Containers
• 6. Automatic documentation
  • 6.1. Docstrings
  • 6.2. What is automatic documentation?
  • 6.3. Sphinx
  • 6.4. Building your documentation
  • 6.5. Publishing your documentation
• 7. Tests and Errors
  • 7.1. Types of errors in Python
  • 7.2. Test suite
• 8. Python Package Index
  • 8.1. PyPi
  • 8.2. Installing a package from PyPi
  • 8.3. Installing multiple packages
  • 8.4. Uploading your package to PyPi
  • 8.5. Versioning
• 9. Continuous Integration
  • 9.1. Webhooks
  • 9.2. GitHub Actions
• 10. Conda
  • 10.1. Conda vs Python virtual environments
  • 10.2. Conda virtual environment
  • 10.3. Activate and deactivate the environment
  • 10.4. Installing more packages
  • 10.5. List all conda environments
  • 10.6. Remove a conda environment
  • 10.7. Access your conda environment from Jupyter
• 11. Computing in Python
  • 11.1. Basic computations
  • 11.2. Advanced computations
  • 11.3. Other tools
• 12. Wavelet scattering transform
  • 12.1. Limitation of non-localized kernels
  • 12.2. Wavelets
  • 12.3. Wavelet transform
  • 12.4. Building translation-invariant features
  • 12.5. Convolution cascade
  • 12.6. Scattering transform coefficients
  • 12.7. Applications
• 13. Advanced Python
  • 13.1. Decorators
  • 13.2. Universal Functions (ufuncs)
  • 13.3. Logging
  • 13.4. Global and Local Variables
  • 13.5. Deep and Shallow Copies
  • 13.6. list and tuples
  • 13.7. Class polymorphism
  • 13.8. Just-in-time compilation
• 14. Optimization and Algorithm Complexity
• 15. Multi-language programming
  • 15.1. Cython
  • 15.2. Differentiable programming
• 16. Random Matrices
  • 16.1. Wigner Semicircle Law
  • 16.2. Wishart Matrices
  • 16.3. Marchenko Pastur Distribution
  • 16.4. Application to Finance
  • 16.5. Application to Medical Imaging
  • 16.6. Application to Learning Algorithms
• 17. Multi-Agent Systems for Data Science
  • 17.1. Agency
  • 17.2. Reinforcement Learning Basics
  • 17.3. Deep Reinforcement Learning

Miscellaneous

• VSCode and Remote SSH
  • Jupyter kernel
• Jupyter and IPython on CSD3

Example Classes

• Brownian motion simulations
  • Python warm-up
    • Exercise 1.1 and 1.2
    • Exercise 1.3
    • Exercise 1.4
    • Exercise 1.5
    • Exercise 1.6
    • Exercise 1.7
  • Geometric Brownian motion simulations
• Curse of dimensionality
  • Question 1
    • Solution
  • Question 2
    • Solution
  • Question 3
    • Solution
  • Question 4
    • Solution
  • Question 5
    • Solution
  • Question 6
    • Solution
  • Question 7
    • Solution
  • Question 8
    • Solution
  • Question 9
    • Solution
  • Question 10
    • Solution
  • Question 11
    • Solution
  • Question 12
    • Solution
  • Question 13
    • Solution
  • Question 14
    • Solution
  • Question 15
    • Solution
    • Note on Gaussian Process vs Neural Net
  • Note on Latin Hyper Cube sampling
• Audio scattering classification
  • Question 1
    • Solution
  • Question 2
    • Solution
  • Question 3
    • Solution
  • Question 4
    • Solution
  • Question 5
    • Solution
  • Question 6
    • Solution
  • Question 7
    • Solution
  • Question 8
    • Solution
  • Question 9
    • Solution
  • Question 10
    • Solution
  • Question 11
    • Solution
  • Question 12
    • Solution
  • Question 13
    • Solution

Useful Resources and further reading

• Duke’s Computational Statistics and Statistical Computing

• Phillip Lippe’s Machine Learning and Computing material

• Bruno Régaldo-Saint Blancard’s Numerical Physics exercises