Skip to content

ThotDjehuty/optimiz-rs

BranchesTags

Repository files navigation

Optimiz-rs 🚀

Optimiz-rs Logo

High-performance optimization algorithms in Rust with Python bindings

Version License Rust Python

Optimiz-rs provides blazingly fast, production-ready implementations of advanced optimization and statistical inference algorithms. Built with Rust for maximum performance and exposed to Python through PyO3, it delivers 50-100× speedup over pure Python implementations.

✨ What's New in v1.0.0

🎉 Production Ready - First stable release with comprehensive documentation
📚 ReadTheDocs - Full documentation at https://optimiz-r.readthedocs.io
🏗️ Published to crates.io - Install with cargo add optimiz-rs
🐍 Published to PyPI - Install with pip install optimiz-rs
🔒 Stable API - Semantic versioning from v1.0.0 forward

Features

✨ Algorithms Included:

  • Mean Field Games: 1D MFG solver, HJB-Fokker-Planck coupling, agent population dynamics
  • Differential Evolution: 5 strategies (rand/1, best/1, current-to-best/1, rand/2, best/2), adaptive jDE, convergence tracking
  • Optimal Control: HJB solvers, regime switching, jump diffusion, MRSJD framework
  • Hidden Markov Models: Baum-Welch training, Viterbi decoding, Gaussian emissions
  • MCMC Sampling: Metropolis-Hastings, adaptive proposals, Bayesian inference
  • Sparse Optimization: Sparse PCA, Box-Tao decomposition, Elastic Net, ADMM
  • Risk Metrics: Hurst exponent, half-life estimation, time series analysis
  • Information Theory: Mutual information, Shannon entropy, feature selection
  • Mathematical Toolkit: Gradient, Hessian, Jacobian, statistics, linear algebra

🚀 Performance:

  • 50-100× faster than pure Python implementations
  • 95% memory reduction vs NumPy/SciPy
  • Parallel-ready with Rayon infrastructure
  • Production-tested on multi-dimensional problems

🐍 Python-First API:

  • Clean, intuitive NumPy-based interface
  • Rich result objects with convergence diagnostics
  • Type hints and comprehensive documentation
  • Jupyter notebook integration

Installation

From PyPI (Python)

pip install optimiz-rs

From crates.io (Rust)

cargo add optimiz-rs

From Source

# Clone the repository
git clone https://github.com/ThotDjehuty/optimiz-r.git
cd optimiz-r

# Install with maturin
pip install maturin
maturin develop --release

# Or install in editable mode
pip install -e .

Using Docker

# Start Jupyter notebook server with examples
docker-compose up dev
# Access at http://localhost:8888

# Run all tests
docker-compose run test

# Build distribution wheels
docker-compose run build

Quick Start

Differential Evolution (Enhanced in v0.2.0)

import numpy as np
from optimizr import differential_evolution

# Rosenbrock function (challenging non-convex problem)
def rosenbrock(x):
    return sum(100.0 * (x[1:] - x[:-1]**2)**2 + (1 - x[:-1])**2)

# Optimize with adaptive jDE (self-tuning parameters)
result = differential_evolution(
    objective_fn=rosenbrock,
    bounds=[(-5, 5)] * 10,
    maxiter=1000,
    strategy='best1',  # 5 strategies: rand1, best1, currenttobest1, rand2, best2
    adaptive=True,     # Adaptive F and CR parameters (jDE algorithm)
    atol=1e-6
)

print(f"Optimum: {result.x}")
print(f"Value: {result.fun} (expected: 0.0)")
print(f"Converged: {result.converged}, Iterations: {result.nit}")
# Typical speedup: 74-88× faster than SciPy

Mathematical Toolkit (New in v0.2.0)

from optimizr import maths_toolkit as mt
import numpy as np

# Numerical differentiation
f = lambda x: x[0]**2 + 2*x[1]**2 + x[0]*x[1]
x = np.array([1.0, 2.0])

gradient = mt.gradient(f, x)        # ∇f(x)
hessian = mt.hessian(f, x)          # H(f)(x)
jacobian = mt.jacobian(f, x)        # J(f)(x)

# Statistics
data = np.random.randn(1000)
stats = {
    'mean': mt.mean(data),
    'var': mt.variance(data),
    'std': mt.std_dev(data),
    'skew': mt.skewness(data),
    'kurt': mt.kurtosis(data)
}

# Linear algebra
A = np.random.randn(5, 5)
norm_l1 = mt.norm_l1(A)
norm_l2 = mt.norm_l2(A)
A_norm = mt.normalize(A)

Mean Field Games (New in v0.3.0)

from optimizr import MFGConfig, solve_mfg_1d_rust
import numpy as np

# Configure MFG problem for population dynamics
config = MFGConfig(
    nx=100, nt=100,           # 100 spatial × 100 temporal grid points
    x_min=0.0, x_max=1.0,     # Spatial domain [0, 1]
    T=1.0,                     # Time horizon
    nu=0.01,                   # Viscosity (diffusion coefficient)
    max_iter=50,               # Fixed-point iteration limit
    tol=1e-5,                  # Convergence tolerance
    alpha=0.5                  # Relaxation parameter
)

# Initial distribution (agents start at x=0.3)
x = np.linspace(0, 1, 100)
m0 = np.exp(-50 * (x - 0.3)**2)
m0 /= np.sum(m0) * (x[1] - x[0])

# Terminal cost (agents want to reach x=0.7)
u_terminal = 0.5 * (x - 0.7)**2

# Solve coupled HJB-Fokker-Planck system
u, m, iterations = solve_mfg_1d_rust(
    m0, u_terminal, config,
    lambda_congestion=0.5
)

print(f"✓ Converged in {iterations} iterations")
print(f"Solution: u{u.shape}, m{m.shape}")
# Typical time: 0.4s for 10,000 space-time points

Hidden Markov Model

from optimizr import HMM
import numpy as np

# Fit HMM with regime switching
returns = np.random.randn(1000)
hmm = HMM(n_states=3)
hmm.fit(returns, n_iterations=100)

# Decode most likely state sequence
states = hmm.predict(returns)
print(f"Transition Matrix:\n{hmm.transition_matrix_}")
print(f"Detected states: {states}")

MCMC Sampling

from optimizr import mcmc_sample

# Define log-posterior
def log_likelihood(params, data):
    mu, sigma = params
    return -0.5 * np.sum(((data - mu) / sigma) ** 2) - len(data) * np.log(sigma)

# Sample from posterior
data = np.random.randn(100) + 2.0
samples = mcmc_sample(
    log_likelihood_fn=log_likelihood,
    data=data,
    initial_params=[0.0, 1.0],
    param_bounds=[(-10, 10), (0.1, 10)],
    n_samples=10000,
    burn_in=1000,
    proposal_std=0.1
)

print(f"Posterior mean: {np.mean(samples, axis=0)}")

Optimal Control (New in v0.2.0)

from optimizr import optimal_control
import numpy as np

# Hamilton-Jacobi-Bellman equation solver
# For stochastic control problem: dX_t = μ dt + σ dW_t

# Define problem parameters
grid = np.linspace(-5, 5, 100)
dt = 0.01
horizon = 1.0

# Solve HJB equation
value_function = optimal_control.solve_hjb(
    grid=grid,
    drift=lambda x: -0.1 * x,      # Mean reversion
    diffusion=lambda x: 0.2,        # Constant volatility
    cost=lambda x, u: x**2 + u**2,  # Quadratic cost
    dt=dt,
    horizon=horizon
)

# Compute optimal control policy
policy = optimal_control.compute_policy(value_function, grid)
print(f"Value at origin: {value_function[len(grid)//2]:.4f}")

Information Theory

from optimizr import mutual_information, shannon_entropy

# Calculate mutual information between two variables
x = np.random.randn(1000)
y = 2 * x + np.random.randn(1000) * 0.5

mi = mutual_information(x, y, n_bins=10)
print(f"Mutual Information: {mi:.4f}")

# Calculate entropy
entropy = shannon_entropy(x, n_bins=10)
print(f"Shannon Entropy: {entropy:.4f}")

Algorithm Details

Hidden Markov Models

Implementation of the Baum-Welch algorithm (Expectation-Maximization) for learning HMM parameters:

  • Forward-Backward Algorithm: Efficient computation of state probabilities
  • Viterbi Decoding: Find most likely state sequence
  • Gaussian Emissions: Continuous observation models
  • Normalization: Numerical stability for long sequences

Use Cases:

  • Regime detection in time series
  • Speech recognition
  • Biological sequence analysis
  • Financial market state identification

MCMC Sampling

Metropolis-Hastings algorithm for sampling from arbitrary probability distributions:

  • Adaptive Proposals: Gaussian random walk
  • Burn-in Period: Discard initial samples
  • Bounded Parameters: Constraint handling
  • Convergence Diagnostics: Track acceptance rates

Use Cases:

  • Bayesian parameter estimation
  • Posterior inference
  • Integration of complex distributions
  • Uncertainty quantification

Differential Evolution (Enhanced in v0.2.0)

Advanced global optimization for non-convex, multimodal, high-dimensional problems:

5 Mutation Strategies:

  • rand/1/bin: Random base vector (exploration)
  • best/1/bin: Best individual base (exploitation)
  • current-to-best/1/bin: Balanced exploration/exploitation
  • rand/2/bin: Two difference vectors (diversity)
  • best/2/bin: Best with two differences (aggressive)

Adaptive jDE Algorithm:

  • Self-tuning mutation factor (F) and crossover rate (CR)
  • Parameter adaptation per individual
  • τ₁, τ₂ control adaptation speed
  • Eliminates manual parameter tuning

Convergence Features:

  • Early stopping with tolerance detection
  • Convergence history tracking
  • Best fitness evolution monitoring
  • Rich diagnostic information

Performance:

  • 74-88× faster than SciPy (Python)
  • Efficient for 10-1000 dimensional problems
  • Memory-efficient population management
  • Parallel-ready architecture

Use Cases:

  • Hyperparameter optimization (ML/DL)
  • Engineering design problems
  • Inverse problems and calibration
  • Non-smooth, noisy objectives
  • Constrained optimization with penalties

Grid Search

Exhaustive search over parameter space:

  • Complete Coverage: Evaluate all grid points
  • Parallel Ready: Independent evaluations
  • Flexible Bounds: Per-parameter ranges
  • Best Score Tracking: Return optimal parameters

Use Cases:

  • Small parameter spaces
  • Benchmark comparisons
  • Hyperparameter tuning
  • Global optima verification

Information Theory Metrics

Quantify information content and dependencies:

  • Mutual Information: I(X;Y) = H(X) + H(Y) - H(X,Y)
  • Shannon Entropy: H(X) = -∑ p(x) log p(x)
  • Binning Strategy: Histogram-based estimation
  • Normalized Variants: Available through Python API

Use Cases:

  • Feature selection
  • Dependency detection
  • Time series analysis
  • Causality testing

Mathematical Toolkit (New in v0.2.0)

Centralized mathematical utilities for all algorithms:

Numerical Differentiation:

  • gradient(): ∇f(x) with central differences
  • hessian(): H(f)(x) second-order derivatives
  • jacobian(): J(f)(x) for vector functions
  • Configurable step size (h)

Statistics:

  • mean(), variance(), std_dev()
  • skewness(), kurtosis() for distribution shape
  • correlation(), covariance() for dependencies
  • Efficient single-pass algorithms

Linear Algebra:

  • norm_l1(), norm_l2(), norm_frobenius()
  • normalize() for vector/matrix normalization
  • trace(), outer_product()
  • ndarray-linalg integration

Integration:

  • trapz(): Trapezoidal rule
  • simpson(): Simpson's rule

Special Functions:

  • sigmoid(), softmax()
  • soft_threshold() for proximal methods

Use Cases:

  • Algorithm development
  • Sensitivity analysis
  • Statistical inference
  • Custom optimization methods

Optimal Control (New in v0.2.0)

Hamilton-Jacobi-Bellman equation solvers for stochastic control:

Features:

  • HJB PDE solver with finite difference schemes
  • Regime-switching models (Markov chains)
  • Jump diffusion processes (Poisson jumps)
  • MRSJD (Markov Regime Switching Jump Diffusion)

Components:

  • Value function computation
  • Optimal policy extraction
  • Boundary conditions handling
  • Grid-based discretization

Use Cases:

  • Portfolio optimization under uncertainty
  • Resource management with regime changes
  • Risk-sensitive control
  • Dynamic programming problems

Performance Benchmarks

Comparison against pure Python/NumPy/SciPy implementations (v0.2.0):

Algorithm Problem Size Optimiz-rs (Rust) NumPy/SciPy Speedup
DE - rand/1 50D Rosenbrock 285ms 21.2s 74×
DE - best/1 50D Rosenbrock 270ms 23.8s 88×
DE - adaptive jDE 50D Rosenbrock 310ms 24.5s 79×
HMM Fit 10k samples 45ms 3.2s 71×
MCMC Sample 100k iterations 120ms 8.5s 71×
Sparse PCA 1000×100 matrix 180ms 12.5s 69×
Mutual Information 50k points 12ms 380ms 32×
Gradient (numerical) 100D function 8ms 145ms 18×
Hessian (numerical) 50D function 95ms 4.2s 44×

Benchmarks run on Apple M1 Pro, 10 cores, 32GB RAM

Documentation

API Reference

Full API documentation is available in the docs/ directory:

Examples & Tutorials

Complete Jupyter notebook tutorials in examples/notebooks/ (all validated in v0.3.0):

  1. Hidden Markov Models - Regime detection, Baum-Welch, Viterbi ✅
  2. MCMC Sampling - Metropolis-Hastings, Bayesian inference ✅
  3. Differential Evolution - 5 strategies, adaptive jDE, convergence ✅
  4. Optimal Control - HJB, regime switching, jump diffusion (theory) ℹ️
  5. Real-World Applications - Complete workflows ✅
  6. Performance Benchmarks - Rust vs Python comparisons ✅
  7. Mean Field Games - Population dynamics, HJB-FP coupling ✅ NEW in v0.3.0

All notebooks tested and production-ready! See NOTEBOOK_AUDIT_REPORT.md for validation details.

Python script examples:

Mathematical Background

Detailed mathematical descriptions and references:

📚 Documentation & Getting Started

Comprehensive documentation is available on ReadTheDocs:

👉 https://optimiz-r.readthedocs.io/en/latest/

The documentation includes:

  • 🚀 Quick Start Guide - Get up and running in minutes
  • 📖 Installation - Detailed setup instructions for all platforms
  • 🎓 Tutorials - Step-by-step guides for each algorithm
  • 📚 API Reference - Complete function and class documentation
  • 🔬 Theory & Math - Mathematical foundations and references
  • 💡 Examples - Real-world use cases and code samples
  • ⚡ Performance - Benchmarks and optimization tips

New to Optimiz-rs? Start with the Quick Start Guide or try the Mean Field Games Tutorial.

Development

Building from Source

# Setup development environment
git clone https://github.com/ThotDjehuty/optimiz-r.git
cd optimiz-r

# Install development dependencies
pip install -e ".[dev]"

# Build Rust extension
maturin develop

# Run tests
pytest tests/ -v

# Run Rust tests
cargo test

# Run benchmarks
cargo bench

Code Quality

# Format code
black python/
cargo fmt

# Lint
ruff check python/
cargo clippy

# Type checking
mypy python/

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Areas for Contribution

  • Advanced DE variants (JADE, SHADE, L-SHADE)
  • GPU acceleration via CUDA/ROCm (see Roadmap)
  • Additional optimization algorithms (PSO, CMA-ES, NES)
  • More probability distributions for HMM
  • Additional language bindings (R, Julia, JavaScript)
  • Documentation improvements and tutorials
  • Benchmark comparisons and case studies

License

MIT License - see LICENSE file for details.

Citation

If you use Optimiz-rs in your research, please cite:

@software{optimizr2024,
  title = {Optimiz-rs: High-Performance Optimization Algorithms in Rust},
  author = {HFThot Research Lab},
  year = {2024},
  version = {1.0.0},
  url = {https://github.com/ThotDjehuty/optimiz-r}
}

Acknowledgments

Built with:

  • Rust - Systems programming language
  • PyO3 - Rust bindings for Python
  • Maturin - Build and publish Rust crates as Python packages
  • NumPy - Numerical computing in Python

Inspired by:

  • scipy.optimize
  • scikit-learn
  • hmmlearn
  • emcee

Contact

Optimiz-rs - Fast optimization for data science and machine learning 🚀

About

High-performance optimization algorithms in Rust with Python bindings.

optimiz-r.readthedocs.io

Topics

hmm optimization mcmc stochastic-control mean-field

Resources

Readme

License

MIT license

Contributing

Contributing
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 3

Mean Field Games (MFG) algorithms with full Python integration and comprehensive tutorial notebooks Latest
Feb 4, 2026
+ 2 releases

Packages

 
 
 

Contributors

Languages