Overview#

A custom implementation of an Echo State Network (ESN); a type of recurrent neural network—for k-step ahead time series forecasting. Built entirely from scratch using Python and NumPy, demonstrating deep understanding of neural network internals beyond library usage.

Key Concepts#

• Implemented Echo State Network architecture from scratch using only Python and NumPy
• Designed custom reservoir computing layer with configurable spectral radius and sparsity
• Built complete training pipeline including:
  • Ridge regression for output weight optimization
  • Hyperparameter tuning across reservoir size, spectral radius, and input scaling
  • K-fold cross-validation for model selection
• Tested on both 2sin and Lorenz attractor time series

Key Findings#

1. As k increases in K-Step Ahead Forecasting, the correlation between MSE and reservoir size (Nr) decreases, potentially becoming negative
2. K-Step Ahead Forecasting provides better results the lower k is, with best results at k=1
3. The difference between local minima and global minima in hyperparameter space can result in vastly different Mean Squared Errors
4. A gradient descent-like algorithm for hyperparameter optimization would likely improve forecasts for all k values

Technologies#

Python, NumPy, Matplotlib, Echo State Networks, Ridge Regression, Time Series Analysis

Related Skills#

• Machine Learning
• Python
• NumPy
• Mathematics
• Data Science