Extension Modules

Create high-performance Python extension modules with Shed Skin

Shed Skin can compile Python code into extension modules that can be imported and used in regular Python programs. This allows you to accelerate performance-critical parts of your application while keeping the rest in Python.

Understanding Extension Modules

An extension module is a compiled shared library (.so on Linux/macOS, .pyd on Windows) that Python can import like any other module. Shed Skin-generated extension modules provide:

• Massive speedups (typically 10-100x) for computation-intensive code
• Seamless integration with existing Python codebases
• No changes to calling code - import and use normally

Use extension modules when you have performance bottlenecks in pure computation code that doesn’t heavily rely on external libraries.

Creating Your First Extension Module

Real-World Example

Let’s accelerate a prime sieve computation:

# primes.py

def sieve_of_eratosthenes(n):
    """Return list of primes < n using Sieve of Eratosthenes."""
    if n <= 2:
        return []
    
    sieve = list(range(3, n, 2))
    top = len(sieve)
    
    for si in sieve:
        if si:
            bottom = (si*si - 3) // 2
            if bottom >= top:
                break
            sieve[bottom::si] = [0] * -((bottom - top) // si)
    
    return [2] + [el for el in sieve if el]

def count_primes(n):
    """Count how many primes are less than n."""
    return len(sieve_of_eratosthenes(n))

if __name__ == '__main__':
    # These calls enable type inference
    print(count_primes(100))
    primes = sieve_of_eratosthenes(1000)
    print(f"Found {len(primes)} primes")

Compile it:

shedskin build -e primes
cp build/primes.so .  # Or build/Debug/primes.pyd on Windows

Use it in your main program:

# main.py
import time
import primes  # The compiled extension module

def benchmark():
    start = time.time()
    result = primes.count_primes(10_000_000)
    elapsed = time.time() - start
    print(f"Found {result} primes in {elapsed:.2f}s")

if __name__ == '__main__':
    benchmark()

Extension Module Limitations

Extension modules have important restrictions compared to pure Python.

Supported Types

Only these types can be passed across the Python/C++ boundary:

Builtin types:

• Scalars: int, float, complex, bool, str, bytes, bytearray
• Collections: list, tuple, dict, set
• Special: None
• User-defined class instances

Not supported:

• Functions/lambdas
• Generators/iterators
• File objects
• Module objects

# Good: Supported types
def process_data(numbers, threshold):
    return [x for x in numbers if x > threshold]

def create_config():
    class Config:
        pass
    c = Config()
    c.value = 42
    return c

# Bad: Unsupported types
def get_processor():
    return lambda x: x * 2  # Cannot return function

def get_iterator():
    return iter([1, 2, 3])  # Cannot return iterator

Data Conversion Overhead

Builtin types are completely converted on each call/return. This means:

• Large data structures have conversion overhead
• Changes to builtin objects don’t persist across the boundary
• User-defined class instances pass by reference (no conversion)

# extension.py
def modify_list(items):
    items.append(999)  # This append happens in C++
    return items

def modify_object(obj):
    obj.value = 999  # This modifies the Python object

class MyClass:
    pass

if __name__ == '__main__':
    modify_list([1, 2, 3])
    m = MyClass()
    m.value = 0
    modify_object(m)

# main.py using the extension
import extension

# Builtin types: converted each time
original = [1, 2, 3]
result = extension.modify_list(original)
print(original)  # [1, 2, 3] - unchanged!
print(result)    # [1, 2, 3, 999] - new list

# User classes: passed by reference
obj = extension.MyClass()
obj.value = 0
extension.modify_object(obj)
print(obj.value)  # 999 - modified!

Global Variable Behavior

Global variables are converted once at initialization. Changes in Python don’t affect the extension module and vice versa.

# module.py
COUNTER = 0

def increment():
    global COUNTER
    COUNTER += 1
    return COUNTER

if __name__ == '__main__':
    print(increment())

# After compilation, this happens:
import module

print(module.COUNTER)    # 0
print(module.increment()) # 1
print(module.COUNTER)    # 0 - unchanged!

# Solution: Use getter/setter functions
def get_counter():
    return COUNTER

Using with NumPy

While Shed Skin doesn’t support NumPy directly, you can pass arrays as lists:

# matrix_ops.py
def matrix_sum(matrix):
    """Sum all elements in a 2D matrix (list of lists)."""
    total = 0.0
    for row in matrix:
        for val in row:
            total += val
    return total

def matrix_multiply_scalar(matrix, scalar):
    """Multiply each element by a scalar."""
    result = []
    for row in matrix:
        new_row = []
        for val in row:
            new_row.append(val * scalar)
        result.append(new_row)
    return result

if __name__ == '__main__':
    test = [[1.0, 2.0], [3.0, 4.0]]
    print(matrix_sum(test))
    print(matrix_multiply_scalar(test, 2.0))

Compile and use with NumPy:

shedskin build -e matrix_ops
cp build/matrix_ops.so .

import numpy as np
import matrix_ops

# Create NumPy array
array = np.array([[1.0, 2.0, 3.0],
                  [4.0, 5.0, 6.0],
                  [7.0, 8.0, 9.0]])

# Convert to list, process, convert back
total = matrix_ops.matrix_sum(array.tolist())
print(f"Sum: {total}")

result_list = matrix_ops.matrix_multiply_scalar(array.tolist(), 2.0)
result_array = np.array(result_list)
print(result_array)

Conversion with .tolist() is expensive. Only worthwhile if significant computation happens inside the extension module.

Multiprocessing Integration

Extension modules work great with Python’s multiprocessing:

# compute.py
def partial_sum(start, end):
    """Calculate partial sum of a series."""
    total = 0.0
    for x in range(start, end):
        if x % 2 == 0:
            total -= 1.0 / x
        else:
            total += 1.0 / x
    return total

if __name__ == '__main__':
    print(partial_sum(1, 1000))

Compile:

shedskin build -e compute
cp build/compute.so .

Use with multiprocessing:

# parallel.py
from multiprocessing import Pool
import compute

def worker(args):
    start, end = args
    return compute.partial_sum(start, end)

if __name__ == '__main__':
    # Split work across processes
    ranges = [
        (1, 5_000_000),
        (5_000_001, 10_000_000),
        (10_000_001, 15_000_000),
        (15_000_001, 20_000_000)
    ]
    
    with Pool(processes=4) as pool:
        results = pool.map(worker, ranges)
    
    total = sum(results)
    print(f"Result: {total}")

Distribution and Deployment

Best Practices

Keep the interface simple: Pass basic types, return basic types. Use user classes for complex data.

Profile first: Ensure the code is actually a bottleneck before creating an extension module.

Minimize data transfer: Design functions that do significant work per call to amortize conversion overhead.

Test both versions: Keep the Python version and test that both produce identical results.

Troubleshooting

Import fails:

# Check library dependencies
ldd build/module.so  # Linux
otool -L build/module.so  # macOS

Type inference fails:

# Ensure all functions are called in __main__
if __name__ == '__main__':
    # Call every exported function
    func1(5)
    func2(10, 'test')
    # If func3 calls func4, you only need to call func3

Segmentation fault:

• Likely a type mismatch between Python and C++
• Review None handling (only with non-scalars)
• Check array/list bounds

Performance Tips

See the Optimization guide for detailed performance tuning of extension modules.