Performance Benchmarks

Benchmark results comparing Shed Skin with CPython, PyPy, and other Python implementations

Overview

Shed Skin delivers significant performance improvements over standard Python implementations. Measurements across 80+ example programs show typical speedups of 1-100x over CPython, with an average of 20x and median of 12x.

Sieve Benchmark

The classic sieve of Eratosthenes prime number algorithm provides a clear performance comparison across different Python implementations and optimizers.

Results (n=100,000,000)

Implementation          Time (seconds)    Speedup vs CPython 3.10
─────────────────────────────────────────────────────────────────
CPython 3.10.6              13.4 s              1.0x (baseline)
CPython 3.11.0              11.4 s              1.2x
Nuitka 0.6.16               11.4 s              1.2x
PyPy 3.9.12                  5.8 s              2.3x
Numba 0.56.4                 2.5 s              5.4x
Shed Skin 0.9.9              1.9 s              7.1x
Shed Skin (optimized)        1.8 s              7.4x

Code Example

Here’s the sieve implementation used in benchmarking:

def sieveOfEratostenes(n):
    """Return the list of primes < n."""
    if n <= 2:
        return []
    sieve = list(range(3, n, 2))
    top = len(sieve)
    for si in sieve:
        if si:
            bottom = (si*si - 3) // 2
            if bottom >= top:
                break
            sieve[bottom::si] = [0] * -((bottom - top) // si)
    return [2] + [el for el in sieve if el]

Optimization Flags

Shed Skin offers additional optimization flags:

• --nowrap: Disable bounds checking on list/string operations
• --nobounds: Alias for --nowrap
• --int64: Use 64-bit integers (default is 32-bit)

For the sieve benchmark:

shedskin build --nowrap --nobounds sieve

This achieves 1.8 seconds (vs 1.9s without flags), a 7.4x speedup over CPython 3.10.

Comprehensive Performance Comparison

The following chart compares Shed Skin and PyPy speedups versus CPython 3.10 across most Shed Skin examples:

Note: These measurements were performed for the git tag ‘performance_comparison’. PyPy was allowed to stabilize (warm up JIT) before measuring.

Key Observations

1. Consistent speedups: Shed Skin provides reliable performance improvements across diverse workloads
2. Predictable performance: Unlike JIT compilers, Shed Skin delivers consistent performance from the first run
3. Best-case scenarios: Some examples achieve 50-100x speedups, particularly for numeric and algorithmic code
4. Median performance: Half of all examples achieve at least 12x speedup
5. PyPy comparison: Shed Skin often outperforms PyPy, especially on compute-intensive tasks

Performance by Category

Numeric Computing

Typical speedup: 20-50x

Examples: mandelbrot, nbody, bh (Barnes-Hut), neural networks

• Heavy floating-point computation
• Array/list operations
• Minimal Python object overhead

Algorithms

Typical speedup: 10-30x

Examples: dijkstra, astar, sorting algorithms, graph algorithms

• Integer arithmetic
• Data structure operations
• Tight loops

Ray Tracing & Graphics

Typical speedup: 15-40x

Examples: mao, minilight, pylot, path_tracing

• 3D vector operations
• Recursive algorithms
• Intensive floating-point math

Game Engines & Emulators

Typical speedup: 5-20x

Examples: chess, doom, c64, pygasus

• Complex state management
• Mixed integer/object operations
• Real-time performance requirements

String Processing & Compression

Typical speedup: 8-25x

Examples: lz2, ac_encode, block, sha

• Byte/string manipulation
• Bit operations
• Sequential processing

Methodology

Measurement Protocol

1. Warm-up runs: PyPy requires several iterations to warm up its JIT compiler. We run the code 5 times before measuring (as seen in example code):

for x in range(10):
    if x == 5:
        t0 = time.time()  # pypy has stabilized
    mandelbrot()
print('TIME %.2f' % (time.time()-t0))

2. Multiple iterations: Each benchmark runs multiple times to get stable measurements
3. Same hardware: All comparisons run on identical hardware
4. Same Python version: CPython 3.10 is used as the baseline

Compilation Settings

Default Shed Skin settings unless otherwise specified:

• 32-bit integers (int32)
• Bounds checking enabled
• Standard optimizations (-O2 in C++ compiler)

Integer Type Performance

Shed Skin defaults to 32-bit integers while Numba defaults to 64-bit integers. This affects performance comparisons:

# Default (int32)
shedskin build sieve       # 1.9 seconds

# 64-bit integers
shedskin build --int64 sieve   # ~2.5 seconds

When using --int64, Shed Skin and Numba performance is practically equal on the sieve benchmark.

Real-World Performance Examples

Chess Engine

# Simplified chess move evaluation
def alphaBeta(board, alpha, beta, n):
    if n == 0:
        return evaluate(board)
    for mv in legalMoves(board):
        newboard = copy(board)
        move(newboard, mv)
        value = alphaBeta(newboard, -beta, -alpha, n - 1)
        if value >= beta:
            return beta
        if value > alpha:
            alpha = value
    return alpha

Speedup: ~15x over CPython

Mandelbrot Fractal

def mandelbrot(max_iterations=1000):
    bailout = 16
    for y in range(-39, 39):
        for x in range(-39, 39):
            cr = y/40 - 0.5
            ci = x/40
            zi, zr = 0, 0
            for i in range(max_iterations):
                temp = zr * zi
                zr2 = zr * zr
                zi2 = zi * zi
                zr = zr2 - zi2 + cr
                zi = temp + temp + ci
                if zi2 + zr2 > bailout:
                    break

Speedup: ~30x over CPython

Richards Benchmark

The standard Richards benchmark (task scheduling simulation):

Speedup: ~18x over CPython

When to Expect Best Performance

Shed Skin delivers the highest speedups when your code:

1. Is compute-intensive: Heavy loops and calculations
2. Uses basic types: Integers, floats, lists, tuples, strings
3. Has clear type flow: Consistent variable usage
4. Minimizes dynamic features: No eval, exec, or heavy introspection
5. Works with supported modules: Uses the 25+ built-in modules

Performance Tuning Tips

Use Type-Specific Operations

# Good: Clear integer operations
for i in range(1000000):
    result = i * i + 2 * i + 1

# Less optimal: Mixed types
for i in range(1000000):
    result = i * 1.0  # Forces float conversion

Minimize Object Creation

# Good: Reuse objects
result = []
for i in range(n):
    result.append(compute(i))

# Less optimal: Create many intermediate objects
result = [compute(i) for i in range(n)]

Use Local Variables

# Good: Local variable access is fast
def process():
    local_data = self.data
    for i in range(len(local_data)):
        compute(local_data[i])

# Less optimal: Repeated attribute access
def process():
    for i in range(len(self.data)):
        compute(self.data[i])

Benchmark Reproducibility

To reproduce these benchmarks:

# Clone repository
git clone https://github.com/shedskin/shedskin.git
cd shedskin/examples

# Run all benchmarks
shedskin test

# Or run specific benchmark
cd sieve
shedskin build sieve
time build/sieve

# Compare with Python
time python sieve.py

External Benchmarks

For continuous performance monitoring, see the Airspeed Velocity (asv) benchmarks:

http://shedskin.github.io/benchmarks