add multi-class classification function with tutorial examples

Author: Leona-LYT

doc/source/example.rst

@@ -17,7 +17,8 @@ Example Gallery
    examples/RankRegression.ipynb
    examples/Path_solution.ipynb
    examples/Warm_start.ipynb
-   examples/Sklearn_Mixin.ipynb   
+   examples/Sklearn_Mixin.ipynb
+   examples/Multiclass_Classification.ipynb   
    examples/NMF.ipynb
 
 List of Examples
@@ -35,5 +36,6 @@ List of Examples
    examples/RankRegression.ipynb
    examples/Path_solution.ipynb
    examples/Warm_start.ipynb
-   examples/Sklearn_Mixin.ipynb 
+   examples/Sklearn_Mixin.ipynb
+   examples/Multiclass_Classification.ipynb 
    examples/NMF.ipynb

doc/source/examples/Multiclass_Classification.ipynb

@@ -0,0 +1,170 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Multi-class Classification Support\n",
+        "\n",
+        "[![Slides](https://img.shields.io/badge/🦌-ReHLine-blueviolet)](https://rehline-python.readthedocs.io/en/latest/)\n",
+        "\n",
+        "\n",
+        "`PLQ_Ridge_Classifier` extends binary PLQ-ERM to multi-class problems via the `multi_class`\n",
+        "parameter, supporting two standard decomposition strategies.\n",
+        "\n",
+        "**One-vs-Rest (OvR)** fits $K$ binary classifiers, one per class, each trained on the\n",
+        "full dataset with relabelled targets ($+1$ for the class, $-1$ for all others).\n",
+        "Prediction selects the class with the highest decision score:\n",
+        "\n",
+        "$$\\widehat{y} = \\arg\\max_{k \\in \\{1, \\dots, K\\}} f_k(\\mathbf{x})$$\n",
+        "\n",
+        "**One-vs-One (OvO)** fits $\\binom{K}{2}$ binary classifiers, one per class pair $(i, j)$,\n",
+        "each trained only on samples belonging to those two classes.\n",
+        "Prediction uses majority voting across all pairwise classifiers:\n",
+        "\n",
+        "$$\\widehat{y} = \\arg\\max_{k \\in \\{1, \\dots, K\\}} \\sum_{j \\neq k} \\mathbf{1}\\bigl[f_{kj}(\\mathbf{x}) > 0\\bigr]$$\n",
+        "\n",
+        "In both cases, each binary sub-problem is solved by a standard `PLQ_Ridge_Classifier`\n",
+        "with the same loss, regularizer, and solver settings passed to the parent estimator."
+      ],
+      "metadata": {
+        "id": "qEniIUh2yiGb"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import pandas as pd\n",
+        "from sklearn.datasets import make_classification\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "from sklearn.metrics import accuracy_score\n",
+        "from rehline import plq_Ridge_Classifier\n",
+        "\n",
+        "\n",
+        "# generate data\n",
+        "X_mc, y_mc = make_classification(\n",
+        "    n_samples=10000, n_features=20, n_informative=10,\n",
+        "    n_classes=4, n_clusters_per_class=1, random_state=42\n",
+        ")\n",
+        "X_train, X_test, y_train, y_test = train_test_split(X_mc, y_mc, test_size=0.2, random_state=42)"
+      ],
+      "metadata": {
+        "id": "W5JaV_p2Ztua"
+      },
+      "execution_count": 2,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### One-vs-Rest (OvR)\n",
+        "\n",
+        "\n",
+        "In OvR, we train $K$ binary classifiers, one per class. Classifier $k$ learns to distinguish\n",
+        "class $k$ from all other classes. The final prediction selects the class whose classifier\n",
+        "reports the highest decision score:\n",
+        "\n",
+        "$$\\widehat{y} \\;=\\; \\arg\\max_{k \\in \\{1, \\dots, K\\}} f_k(\\mathbf{x})$$\n",
+        "\n",
+        "where $f_k(\\mathbf{x})$ is the signed distance from $\\mathbf{x}$ to the decision boundary of classifier $k$."
+      ],
+      "metadata": {
+        "id": "P_kBHXWG_qxa"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# predict using ovr method\n",
+        "plq_ovr = plq_Ridge_Classifier(\n",
+        "    loss={'name': 'svm'}, C=1.0,\n",
+        "    fit_intercept=True, max_iter=50000, multi_class='ovr',\n",
+        ")\n",
+        "plq_ovr.fit(X_train, y_train)\n",
+        "\n",
+        "y_pred = plq_ovr.predict(X_test)\n",
+        "print(f\"plq OvR accuracy: {accuracy_score(y_test, y_pred):.4f}\")"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "eNgPMJdctWyi",
+        "outputId": "b654fab7-2aff-4fa0-cf56-1b4ef6e82313"
+      },
+      "execution_count": 3,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "plq OvR accuracy: 0.7770\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### One-vs-One (OvO)\n",
+        "\n",
+        "In OvO, we train $\\binom{K}{2}$ binary classifiers, one for each pair of classes $(i, j)$.\n",
+        "Each classifier $f_{ij}$ votes for either class $i$ or class $j$. The final prediction\n",
+        "is the class that receives the most votes:\n",
+        "\n",
+        "$$\\widehat{y} = \\arg\\max_{k \\in \\{1, \\dots, K\\}} \\sum_{j \\neq k} \\mathbf{1}\\bigl[f_{kj}(\\mathbf{x}) > 0\\bigr]$$\n",
+        "\n",
+        "where $\\mathbf{1}[\\cdot]$ is the indicator function, and $f_{kj}(\\mathbf{x}) > 0$ means\n",
+        "classifier $(k, j)$ votes for class $k$."
+      ],
+      "metadata": {
+        "id": "bnl41NChAlUr"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# predict using ovo method\n",
+        "plq_ovo = plq_Ridge_Classifier(\n",
+        "    loss={'name': 'svm'}, C=1.0,\n",
+        "    fit_intercept=True, max_iter=50000, multi_class='ovo',\n",
+        ")\n",
+        "plq_ovo.fit(X_train, y_train)\n",
+        "y_pred = plq_ovo.predict(X_test)\n",
+        "\n",
+        "print(f\"plq OvO accuracy: {accuracy_score(y_test, y_pred):.4f}\")"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "s9nalNTit9Jl",
+        "outputId": "134c9f27-444b-41ee-e606-9985dbbdb9c9"
+      },
+      "execution_count": 4,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "plq OvO accuracy: 0.8035\n"
+          ]
+        }
+      ]
+    }
+  ]
+}