This document presents the performance of the RVAV optimizer and its momentum variant compared to the standard Adam optimizer on a synthetic linear regression task. The goal is to demonstrate the convergence speed and final loss achieved by each optimizer under the same conditions.
All models were trained for 200 epochs on the same dataset with a learning rate of 0.01.
The following plot illustrates the training loss over 200 epochs. The RVAV + Momentum variant demonstrates the fastest initial convergence, quickly reaching a low loss value within the first 40 epochs.
The final loss values after 200 epochs are summarized in the table below. Both RVAV variants significantly outperform the Adam baseline on this task, achieving a much lower final loss.
| Optimizer | Final Loss (after 200 epochs) |
|---|---|
| RVAV | 4.2912 |
| RVAV + Momentum | 4.2908 |
| Adam (Baseline) | 5.6671 |
The results clearly indicate that both the original RVAV and the RVAV + Momentum optimizers are highly effective for this regression problem. The addition of momentum provides a noticeable benefit in terms of initial convergence speed and achieves the lowest overall loss. Both custom optimizers substantially outperformed the Adam baseline, highlighting their potential as powerful alternatives for specific machine learning tasks.