12. More Advanced Python

There are several advanced Python notions that you need to be familiar with.

12.1. Decorators

12.1.1. What is a Decorator?

A decorator is a function that takes another function as input, adds some functionality to it, and returns it. You can apply decorators using the @decorator_name syntax.

12.1.2. Basic Example of a Decorator

# Define a simple decorator
def my_decorator(func):
    def wrapper():
        print("Something is happening before the function is called.")
        func()
        print("Something is happening after the function is called.")
    return wrapper

# Define a function and decorate it
@my_decorator
def say_hello():
    print("Hello!")

# Call the decorated function
say_hello()

Something is happening before the function is called.
Hello!
Something is happening after the function is called.

@my_decorator is syntactic sugar for: python   say_hello = my_decorator(say_hello)

12.1.3. Decorator with Arguments

def repeat_decorator(times):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(times):
                func(*args, **kwargs)
        return wrapper
    return decorator

@repeat_decorator(times=3)
def greet(name):
    print(f"Hello, {name}!")

greet("Boris")

Hello, Boris!
Hello, Boris!
Hello, Boris!

Here:

• The repeat_decorator takes an argument (times) and returns the actual decorator.

• The decorator wraps the greet function to execute it multiple times.

12.1.4. Using functools.wraps to Preserve Metadata

When you wrap a function using a decorator, the original function’s metadata (like its name and docstring) can be lost.

Use functools.wraps to preserve it.

import functools

def my_decorator(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        """Wrapper function."""
        print("Calling decorated function...")
        return func(*args, **kwargs)
    return wrapper

@my_decorator
def example_function():
    """This is an example function."""
    print("Example function is running!")

example_function()
print(example_function.__name__)  # Output: example_function
print(example_function.__doc__)  # Output: This is an example function.

Calling decorated function...
Example function is running!
example_function
This is an example function.

In this example, we see that the metadata of the decorated function is preserved, namely the name (__name__) and the docstring (__doc__).

12.1.5. Class-Based Decorators

Decorators can also be implemented as classes.

class MyDecorator:
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        print("Before the function call.")
        result = self.func(*args, **kwargs)
        print("After the function call.")
        return result

@MyDecorator
def say_hello():
    print("Hello!")

say_hello()

Before the function call.
Hello!
After the function call.

Here, the __call__ method makes the class instance callable, so it acts like a decorator. In this case, you don’t need to define a wrapper function inside the decorator.

12.1.6. Chaining Multiple Decorators

You can apply multiple decorators to a function.

def uppercase(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return result.upper()
    return wrapper

def exclaim(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return result + "!"
    return wrapper

@exclaim
@uppercase
def greet(name):
    return f"hello, {name}"

print(greet("Boris"))

HELLO, BORIS!

• @uppercase is applied first, transforming the string to uppercase.

• @exclaim is applied next, adding an exclamation mark.

12.1.7. Real-World Example:

12.1.7.1. Timing a Function

import time

def timing_decorator(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        print(f"Execution time: {end_time - start_time:.4f} seconds")
        return result
    return wrapper

@timing_decorator
def slow_function():
    time.sleep(2)
    print("Finished!")



@timing_decorator
def fast_function():
    time.sleep(0.1)
    print("Finished!")

slow_function()
fast_function()

Finished!
Execution time: 2.0050 seconds
Finished!
Execution time: 0.1015 seconds

In this example we don’t need to re-implement the timing logic, thanks to the decorator.

12.1.7.2. Vectorizing a Function

If you have a regular function and want to apply np.vectorize, you can explicitly wrap the function with np.vectorize.

import numpy as np

# Define a regular function
def my_function(x):
    return x ** 2 if x > 0 else 0

# Vectorize it
vectorized_function = np.vectorize(my_function)

# Apply it to a NumPy array
data = np.array([-2, -1, 0, 1, 2])
result = vectorized_function(data)
print(result)  # Output: [0 0 0 1 4]

[0 0 0 1 4]

# my_function(data)

Using np.vectorize as a decorator simplifies the process and automatically transforms the function for vectorized operations.

import numpy as np

# Use np.vectorize as a decorator
@np.vectorize
def my_function(x):
    return x ** 2 if x > 0 else 0

# Apply it to a NumPy array
data = np.array([-2, -1, 0, 1, 2])
result = my_function(data)
print(result)  # Output: [0 0 0 1 4]

[0 0 0 1 4]

Here, the @np.vectorize decorator is applied directly to my_function, so you don’t need to explicitly wrap it later.

To recap:

1. Without Decorator:

   • You have to manually apply np.vectorize to the function.

   • More verbose.

2. With Decorator:

   • Cleaner and more readable.

   • Automatically makes the function vectorized when defined.

np.vectorize is useful for extending scalar functions to arrays when you don’t want to manually iterate over elements.

However, it doesn’t provide true performance benefits like NumPy’s ufuncs (universal functions), as it’s just a convenience for element-wise operations. If possible, prefer writing functions that natively work with arrays for better performance.

np.sqrt(np.abs(data))

array([1.41421356, 1.        , 0.        , 1.        , 1.41421356])

np.abs(data)**0.5

array([1.41421356, 1.        , 0.        , 1.        , 1.41421356])

12.2. Numpy Universal Functions (ufuncs)

A universal function (or ufunc for short) is a function that operates on ndarrays in an element-by-element fashion, supporting array broadcasting, type casting, and several other standard features. That is, a ufunc is a “vectorized” wrapper for a function that takes a fixed number of specific inputs and produces a fixed number of specific outputs (see here).

For example:

import numpy as np

# Element-wise addition
arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])

result = arr1 + arr2  # Uses ufunc np.add
print(result)  # Output: [5 7 9]

[5 7 9]

Square-root, and trigonometric functions in numpy are also ufuncs:

arr = np.array([1, 4, 9, 16])

# Square root
sqrt_result = np.sqrt(arr)
print(sqrt_result)  # Output: [1. 2. 3. 4.]

# Trigonometric functions
angles = np.array([0, np.pi / 2, np.pi])
sin_result = np.sin(angles)
print(sin_result)  # Output: [0. 1. 0.]

[1. 2. 3. 4.]
[0.0000000e+00 1.0000000e+00 1.2246468e-16]

You can create a custom ufunc from a Python function as well.

def custom_add(x, y):
    return x + y

# Create ufunc
custom_add_ufunc = np.frompyfunc(custom_add, 2, 1)

result = custom_add_ufunc([1, 2, 3], [4, 5, 6])
print(result)  # Output: [5 7 9]

[5 7 9]

Here is an example of the huge performance benefits of ufuncs:

import numpy as np
import time

# Data
arr = np.arange(1e6)

# Using ufunc
start = time.time()
result_ufunc = np.sqrt(arr)
end = time.time()
print(f"ufunc time: {end - start:.6f} seconds")

# Using np.vectorize
def scalar_sqrt(x):
    return x ** 0.5

vectorized_sqrt = np.vectorize(scalar_sqrt)
start = time.time()
result_vectorized = vectorized_sqrt(arr)
end = time.time()
print(f"np.vectorize time: {end - start:.6f} seconds")

ufunc time: 0.007639 seconds
np.vectorize time: 0.117718 seconds

In this example, the ufunc version is 100 times faster than the vectorized version. The vectorized version actually iterates over the elements of the array, while the ufunc version is implemented in C under the hood and is highly optimized.

12.3. Logging

Logging is used to track events in a program and helps debug or monitor it without relying on print statements.

It’s better because it categorizes messages by severity levels (e.g., INFO, WARNING, ERROR) and can output logs to different destinations (e.g., files).

12.3.1. Basic Setup

import logging

# Configure basic logging
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

# Example: A simple computation
def compute_square(x):
    if x < 0:
        logging.warning("Received a negative number. Returning zero.")
        return 0
    result = x ** 2
    logging.info(f"Computed square of {x}: {result}")
    return result

# Call the function
compute_square(5)
compute_square(-3)

INFO: Computed square of 5: 25
WARNING: Received a negative number. Returning zero.

0

1. Log Levels Used:

   • INFO: Logs useful computation results.

   • WARNING: Logs when unexpected input is received.

2. Why Use Logging?

   • It separates debugging messages from your main program output.

   • It’s easy to expand (e.g., write to files, add timestamps).

12.3.2. Activating and Deactivating Logging

Sometimes, you may want to turn logging on or off depending on the situation (e.g., during debugging or production).

How to Deactivate Logging?

To deactivate logging, set the logging level to logging.CRITICAL. Since this is the highest level, only critical errors will be logged, effectively “silencing” other logs.

import logging

# Configure basic logging
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

# Example function with logging
def compute_square(x):
    if x < 0:
        logging.warning("Received a negative number. Returning zero.")
        return 0
    result = x ** 2
    logging.info(f"Computed square of {x}: {result}")
    return result

# Activate logging (default level INFO)
compute_square(5)  # Logs: INFO: Computed square of 5: 25
compute_square(-3)  # Logs: WARNING: Received a negative number. Returning zero.

INFO: Computed square of 5: 25
WARNING: Received a negative number. Returning zero.

0

# Deactivate logging
logging.getLogger().setLevel(logging.CRITICAL)

compute_square(5)  # No logs
compute_square(-3)  # No logs

0

How to Reactivate Logging?

logging.getLogger().setLevel(logging.INFO)

compute_square(5)  # Logs: INFO: Computed square of 5: 25
compute_square(-3)  # Logs: WARNING: Received a negative number. Returning zero.

INFO: Computed square of 5: 25
WARNING: Received a negative number. Returning zero.

0

This allows you to easily toggle logging without removing or modifying your logging statements.

12.4. Global and Local Variables

Variables in python can have global or local scope.

Global variables are accessible everywhere.

Local variables are only accessible in the function they are defined in.

Variables created in for, while and if statements are global, so available outside the loop (as are the iterators, e.g. i in for i in range(10)).

for i in range(3):
    j=100
print(j)
print(i)
i=0

while i<2:
    j=101
    i=5
print(j)

if (True):
    j=102
print(j)

100
2
101
102

Exercise: Explain the output of the code above.

Variables created in functions are local, so not available outside the function.

def f():
    j=103
f()
print(j)

102

Exercise: Explain the output of the code above.

When global and local variables have the same name python creates two instances.

The local variable takes precedence in the function and dies at the end of it, the global variable takes precedence outside the function.

k=104
def function2():
    k=105
    return 0
function2()
print(k)

k=104
def function3():
    k+=105
    return 0
function3()
# print(k)

104

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Cell In [33], line 12
     10     k+=105
     11     return 0
---> 12 function3()
     13 # print(k)

Cell In [33], line 10, in function3()
      9 def function3():
---> 10     k+=105
     11     return 0

UnboundLocalError: local variable 'k' referenced before assignment

Exercise: Explain the output of the code above.

12.5. Deep and Shallow Copies

Here is an example of a shallow copy:

# Original list
original = [[1, 2, 3], [4, 5, 6]]

# Shallow copy
shallow = original

# Modify the nested object
shallow[0][0] = 99

print("Original:", original)  # Output: [[99, 2, 3], [4, 5, 6]]
print("Shallow:", shallow)    # Output: [[99, 2, 3], [4, 5, 6]]


original[0][0] = 87

print("Original:", original)  # Output: [[87, 2, 3], [4, 5, 6]]
print("Shallow:", shallow)    # Output: [[87, 2, 3], [4, 5, 6]]

Original: [[99, 2, 3], [4, 5, 6]]
Shallow: [[99, 2, 3], [4, 5, 6]]
Original: [[87, 2, 3], [4, 5, 6]]
Shallow: [[87, 2, 3], [4, 5, 6]]

Exercise: Explain the output of the code above.

Here is an example of a deep copy:

import copy
# Original list
original = [[1, 2, 3], [4, 5, 6]]

# Deep copy
deep = copy.deepcopy(original)

# Modify the nested object
deep[0][0] = 99

print("Original:", original)  # Output: [[1, 2, 3], [4, 5, 6]]
print("Deep:", deep)    # Output: [[99, 2, 3], [4, 5, 6]]


original[0][0] = 87

print("Original:", original)  # Output: [[87, 2, 3], [4, 5, 6]]
print("Deep:", deep)    # Output: [[99, 2, 3], [4, 5, 6]]

Original: [[1, 2, 3], [4, 5, 6]]
Deep: [[99, 2, 3], [4, 5, 6]]
Original: [[87, 2, 3], [4, 5, 6]]
Deep: [[99, 2, 3], [4, 5, 6]]

Exercise: Explain the output of the code above.

You can also create a deep copy using list comprehension:

deep = [inner[:] for inner in original]

The same applies to dictionaries.

Exercise: Write the examples above for dictionaries.

12.6. Lists and tuples

Lists and tuples are two fundamental data structures in Python, with lists being mutable ordered collections and tuples being immutable ordered collections.

12.6.1. Lists

• Mutable: You can change, add, or remove elements.

• Defined with square brackets (``[]``).

# Create a list
my_list = [1, 2, 3]

# Access elements
print(my_list[0])  # Output: 1

# Modify elements
my_list[0] = 10
print(my_list)  # Output: [10, 2, 3]

# Add elements
my_list.append(4)
print(my_list)  # Output: [10, 2, 3, 4]

# Remove elements
my_list.pop()
print(my_list)  # Output: [10, 2, 3]

1
[10, 2, 3]
[10, 2, 3, 4]
[10, 2, 3]

12.6.2. Tuples

• Immutable: Cannot change elements after creation.

• Defined with parentheses (``()``).

# Create a tuple
my_tuple = (1, 2, 3)

# Access elements
print(my_tuple[0])  # Output: 1

# Cannot modify elements
# my_tuple[0] = 10  # Error: 'tuple' object does not support item assignment

# Tuples support slicing
print(my_tuple[1:])  # Output: (2, 3)

1
(2, 3)

12.6.3. Key Differences

Feature	List	Tuple
Mutability	Mutable	Immutable
Syntax	[1, 2, 3]	(1, 2, 3)
Performance	Slower	Faster
Use Case	Dynamic data	Fixed data

12.6.4. Quick Tip

• Use lists when data changes frequently.

• Use tuples when data is constant (e.g., coordinates, configuration settings).

12.7. Class polymorphism

Here is an explicit example of polymorphism in Python.

12.7.1. Define a Base Class

A base class provides a common interface with a method that can be overridden by subclasses.

class Shape:
    def area(self):
        raise NotImplementedError("Subclasses must override this method")

    def perimeter(self):
        raise NotImplementedError("Subclasses must override this method")

12.7.2. Create Subclasses

Each subclass implements the area and perimeter methods differently, depending on the specific shape.

class Rectangle(Shape):
    def __init__(self, width, height):
        self.width = width
        self.height = height

    def area(self):
        return self.width * self.height

    def perimeter(self):
        return 2 * (self.width + self.height)

class Circle(Shape):
    def __init__(self, radius):
        self.radius = radius

    def area(self):
        return 3.14159 * self.radius ** 2

    def perimeter(self):
        return 2 * 3.14159 * self.radius

12.7.3. Use Polymorphism

You can use objects of Rectangle and Circle interchangeably when working with the Shape interface.

# List of shapes
shapes = [Rectangle(3, 4), Circle(5)]

# Polymorphic behavior
for shape in shapes:
    print(f"Area: {shape.area()}")
    print(f"Perimeter: {shape.perimeter()}")

Area: 12
Perimeter: 14
Area: 78.53975
Perimeter: 31.4159

12.7.4. Explanation

1. Both Rectangle and Circle override the area and perimeter methods from the Shape base class.

2. The loop processes each shape polymorphically, calling the appropriate method implementation depending on the object type.

3. This makes the code flexible and extensible for new shapes without modifying existing logic.

12.8. Just-in-time compilation

Just-In-Time (JIT) compilation is a technique to improve the performance of Python code by compiling parts of the code to machine code at runtime. (Python code is not normally converted to machine code at runtime. Instead, it is interpreted or compiled into bytecode stored in .pyc files, which is then executed by the Python interpreter.)

This can make code run significantly faster. A common tool for JIT in Python is Numba.

Let us see an example.

# Import necessary modules
from numba import jit
import timeit

# Define functions
def sum_of_squares(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

@jit
def sum_of_squares_jit(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

# Input size
n = 10**6

# Time the functions
# Without JIT
%timeit -n 10 -r 3 sum_of_squares(n)

# With JIT
# ensures the function is compiled before timing, so only runtime performance is measured
sum_of_squares_jit(n)

# Time JIT version
%timeit -n 10 -r 3 sum_of_squares_jit(n)

41.6 ms ± 597 µs per loop (mean ± std. dev. of 3 runs, 10 loops each)
147 ns ± 60.9 ns per loop (mean ± std. dev. of 3 runs, 10 loops each)