01.1b_Linear Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "19c29d13-b7bb-4d55-9bb6-79938f7af157",
   "metadata": {},
   "source": [
    "## 1.1. Linear Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f3803fe-a3a7-4b9b-b90c-73775986038a",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 1.1.3. Lasso"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c63d08c-54d7-47e2-8611-f44247b292cc",
   "metadata": {},
   "source": [
    "The [Lasso](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso) is a linear model that estimates sparse coefficients. It is useful in some contexts due to its tendency to prefer solutions with fewer non-zero coefficients, effectively reducing the number of features upon which the given solution is dependent. For this reason, Lasso and its variants are fundamental to the field of compressed sensing. Under certain conditions, it can recover the exact set of non-zero coefficients (see [Compressive sensing: tomography reconstruction with L1 prior (Lasso)](https://scikit-learn.org/stable/auto_examples/applications/plot_tomography_l1_reconstruction.html#sphx-glr-auto-examples-applications-plot-tomography-l1-reconstruction-py))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89e7fd0f-154d-4a92-ad71-cc13a5b37a35",
   "metadata": {},
   "source": [
    "Mathematically, it consists of a linear model with an added regularization term. The objective function to minimize is:\n",
    "\n",
    "$$\\min_w\\frac{1}{2n_{samples}}\\lVert Xw - y \\lVert_2^2 + \\alpha\\lVert w \\rVert_1$$\n",
    "\n",
    "The lasso estimate thus solves the minimization of the least-squares penalty with $\\alpha\\lVert w \\rVert_1$ added, where $\\alpha$ is a constant and $\\lVert w \\rVert_1$ is the $\\ell_1$-norm of the coefficient vector.\n",
    "\n",
    "The implementation in the :class:`~sklearn.linear_model.Lasso` uses coordinate descent as the algorithm to fit the coefficients. See [Least Angle Regression](https://scikit-learn.org/stable/modules/linear_model.html#least-angle-regression) for another implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "635df19a-d149-4343-8515-b5510df59fb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.8])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "reg = linear_model.Lasso(alpha=0.1)\n",
    "reg.fit([[0, 0], [1, 1]], [0, 1])\n",
    "reg.predict([[1, 1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99fe8372-bc51-4959-95a9-7dba1f9a7d77",
   "metadata": {},
   "source": [
    "The function :class:`~sklearn.linear_model.lasso_path` is useful for lower-level tasks, as it computes the coefficients along the full path of possible values."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e5b50c6-3f3d-48ee-b889-511efe02aa2c",
   "metadata": {},
   "source": [
    "$\\textbf{Examples:}$\n",
    "- L1-based models for Sparse Signals - [Sci-kit Link](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_and_elasticnet.html#sphx-glr-auto-examples-linear-model-plot-lasso-and-elasticnet-py) | [Python code](https://github.com/baksho/ml-handson/blob/main/Examples/06_example_plot_lasso_and_elasticnet.py) | [Jupyter Notebook](https://github.com/baksho/ml-handson/blob/main/Examples/06_example_plot_lasso_and_elasticnet.ipynb)\n",
    "- Compressive sensing: tomography reconstruction with L1 prior (Lasso) - [Sci-kit Link](https://scikit-learn.org/stable/auto_examples/applications/plot_tomography_l1_reconstruction.html#sphx-glr-auto-examples-applications-plot-tomography-l1-reconstruction-py) | [Python code](https://github.com/baksho/ml-handson/blob/main/Examples/07_example_plot_tomography_l1_reconstruction.py) | [Jupyter Notebook](https://github.com/baksho/ml-handson/blob/main/Examples/07_example_plot_tomography_l1_reconstruction.ipynb)\n",
    "- Common pitfalls in the interpretation of coefficients of linear models - [Sci-kit Link](https://scikit-learn.org/stable/auto_examples/inspection/plot_linear_model_coefficient_interpretation.html#sphx-glr-auto-examples-inspection-plot-linear-model-coefficient-interpretation-py) | [Python code](https://github.com/baksho/ml-handson/blob/main/Examples/05_example_plot_linear_model_coefficient_interpretation.py) | [Jupyter Notebook](https://github.com/baksho/ml-handson/blob/main/Examples/05_example_plot_linear_model_coefficient_interpretation.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af7b890f-ff14-49b7-897c-4eaea8e29010",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><h5>Note</h5><h4>Feature selection with Lasso</h4>\n",
    "    \n",
    "<p>\n",
    "\n",
    "As the Lasso regression yields sparse models, it can thus be used to perform feature selection, as detailed in [L1-based feature selection](https://scikit-learn.org/stable/modules/feature_selection.html#l1-feature-selection).</p></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a86e3855-f007-430f-b219-bf1d962c980d",
   "metadata": {},
   "source": [
    "$\\textbf{References}$\n",
    "\n",
    "The following two references explain the iterations used in the coordinate descent solver of scikit-learn, as well as the duality gap computation used for convergence control.\n",
    "\n",
    "- *“Regularization Path For Generalized linear Models by Coordinate Descent”*, Friedman, Hastie & Tibshirani, J Stat Softw, 2010 ([Paper](https://www.jstatsoft.org/article/view/v033i01/v33i01.pdf)).\n",
    "- *“An Interior-Point Method for Large-Scale L1-Regularized Least Squares”*, S. J. Kim, K. Koh, M. Lustig, S. Boyd and D. Gorinevsky, in IEEE Journal of Selected Topics in Signal Processing, 2007 ([Paper](https://web.stanford.edu/~boyd/papers/pdf/l1_ls.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a238c05-177d-448f-8561-d693d2bc8629",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 1.1.3.1. Setting regularization parameter\n",
    "\n",
    "The `alpha` parameter controls the degree of sparsity of the estimated coefficients."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a640c2-830c-407f-84e5-0c13f70539a3",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 1.1.3.1.1. Using cross-validation\n",
    "\n",
    "scikit-learn exposes objects that set the Lasso `alpha` parameter by cross-validation: [LassoCV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV) and [LassoLarsCV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsCV.html#sklearn.linear_model.LassoLarsCV) is based on the [Least Angle Regression](https://scikit-learn.org/stable/modules/linear_model.html#least-angle-regression) algorithm explained below.\n",
    "\n",
    "For high-dimensional datasets with many collinear features, :class:`~sklearn.linear_model.LassoCV`is most often preferable. However, :class:`~sklearn.linear_model.LassoLarsCV`has the advantage of exploring more relevant values of `alpha` parameter, and if the number of samples is very small compared to the number of features, it is often faster than :class:`~sklearn.linear_model.LassoCV`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "634d6f02-57b8-4fea-a740-f16d29e4814d",
   "metadata": {},
   "source": [
    "<center><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_lasso_model_selection_002.png\" /><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_lasso_model_selection_003.png\" /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043e084b-cef4-4300-8871-b425c6b558bc",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 1.1.3.1.2. Information-criteria based mo\n",
    "\n",
    "Alternatively, the estimator [LassoLarsIC](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsIC.html#sklearn.linear_model.LassoLarsIC) proposes to use the **Akaike Information Criterion (AIC)** and **Bayes Information Criterion (BIC)**. It is a computationally cheaper alternative to find the optimal value of alpha as the regularization path is computed only once instead of k+1 times when using k-fold cross-validation.\n",
    "\n",
    "Indeed, these criteria are computed on the in-sample training set. In short, they penalize the over-optimistic scores of the different Lasso models by their flexibility (cf. to “Mathematical details” section below).\n",
    "\n",
    "However, such criteria need a proper estimation of the degrees of freedom of the solution, are derived for large samples (asymptotic results) and assume the correct model is candidates under investigation. They also tend to break when the problem is badly conditioned (e.g. more features than samples).\n",
    "\n",
    "<center><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_lasso_lars_ic_001.png\" /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e769ce9c-154d-4abd-a75b-4a5764ae2212",
   "metadata": {},
   "source": [
    "$\\textbf{Examples}$\n",
    "- Lasso model selection: AIC-BIC / cross-validation - [Sci-kit Link](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html#sphx-glr-auto-examples-linear-model-plot-lasso-model-selection-py) | [Python code](https://github.com/baksho/ml-handson/blob/main/Examples/08_example_plot_lasso_model_selection.py) | [Jupyter Notebook](https://github.com/baksho/ml-handson/blob/main/Examples/08_example_plot_lasso_model_selection.ipynb)\n",
    "- Lasso model selection via information criteria - [Sci-kit Link](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_lars_ic.html#sphx-glr-auto-examples-linear-model-plot-lasso-lars-ic-py) | [Python code](https://github.com/baksho/ml-handson/blob/main/Examples/09_example_plot_lasso_lars_ic.py) | [Jupyter Notebook](https://github.com/baksho/ml-handson/blob/main/Examples/09_example_plot_lasso_lars_ic.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081556a3-5013-4ee6-b4e9-0341c37f2f1c",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 1.1.3.1.3. AIC and BIC criteria\n",
    "\n",
    "The definition of AIC and BIC might differ in the literature. In this section, we give more information regarding the criterion computed in scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffae6204-349b-474f-b60d-4ab819ab8dcd",
   "metadata": {},
   "source": [
    "$\\textbf{Mathematical Details}$\n",
    "\n",
    "The AIC criterion is defined as:\n",
    "\n",
    "$$AIC = -2log(\\hat{L}) + 2d$$\n",
    "\n",
    "where $\\hat{L}$ is the maximum likelihood of the model and $d$ is the number of parameters (as well referred to as degrees of freedom in the previous section).\n",
    "\n",
    "The definition of BIC replace the constant $2$ by $log(N)$:\n",
    "\n",
    "$$AIC = -2log(\\hat{L}) + log(N)d$$\n",
    "\n",
    "where $N$ is the number of samples.\n",
    "\n",
    "For a linear Gaussian model, the maximum log-likelihood is defined as:\n",
    "\n",
    "$$log(\\hat{L}) = -\\frac{n}{2}log(2\\pi) - \\frac{n}{2}ln(\\sigma^2)-\\frac{\\sum_{i=1}^{n}(y_i - \\hat{y}_i)^2}{2\\sigma^2}$$\n",
    "\n",
    "where $\\sigma^2$ is an estimate of the noise variance, $y_i$ and $\\hat{y}_i$ are respectively the true and predicted targets, and $n$ is the number of samples.\n",
    "\n",
    "Plugging the maximum log-likelihood in the AIC formula yields:\n",
    "\n",
    "$$AIC = nlog(2\\pi\\sigma^2) + \\frac{\\sum_{i=1}^{n}(y_i - \\hat{y}_i)^2}{\\sigma^2} + 2d$$\n",
    "\n",
    "The first time of the above expression is sometimes discarded since it is a constant when $\\sigma^2$ is provided. In addition, it is sometimes stated that the AIC is equivalent to the $C_p$ statistic [[1](#ZHT2007)]. In a strict sense, however it is equivalent only up to some constant and a multiplicative factor.\n",
    "\n",
    "At last, we mentioned above that $\\sigma^2$ is an estimate of the noise variance. In :class:`~sklearn.linear_model.LassoLarsIC` when the parameter `noise_variance` is not provided (default), the noise variance is estimated via the unbiased estimator [[2](#CVY2003)] defined as:\n",
    "\n",
    "$$\\sigma^2 = \\frac{\\sum_{i=1}^{n}(y_i - \\hat{y}_i)^2}{n - p}$$\n",
    "\n",
    "where $p$ is the number of features and $\\hat{y}_i$ is the predicted target using an ordinary least squares regression. Note, that this formula is valid only when `n_samples > n_features`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1325bd9a-15c4-4096-9a22-096f59b0f03b",
   "metadata": {},
   "source": [
    "$\\textbf{References}$\n",
    "\n",
    "1.  <a id='ZHT2007'></a>Zou, Hui, Trevor Hastie, and Robert Tibshirani. *“On the degrees of freedom of the lasso.”* The Annals of Statistics 35.5 (2007): 2173-2192 ([Paper](https://arxiv.org/abs/0712.0881.pdf))\n",
    "2.  <a id='CVY2003'></a>Cherkassky, Vladimir, and Yunqian Ma. *“Comparison of model selection for regression.”* Neural computation 15.7 (2003): 1691-1714 ([Paper](https://doi.org/10.1162/089976603321891864))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9afa536-492c-4096-b73a-9bb5e4e3d130",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 1.1.3.1.4. Comparison with the regularization parameter of SVM\n",
    "\n",
    "The equivalence between `alpha` and the regularization parameter of SVM, `C` is given by `alpha = 1 / C` or `alpha = 1 / (n_samples * C)`, depending on the estimator and the exact objective function optimized by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1b5dd77-3a35-4078-9498-0485c3816fac",
   "metadata": {},
   "source": [
    "#### 1.1.3.2. Difference between Ridge Regression and Lasso Regression\n",
    "\n",
    "Below is a comparison between the Ridge Regression and Lasso Regression:\n",
    "\n",
    "| Characteristics | Ridge Regression (L2 Regularization) | Lasso Regression (L1 Regularization) |\n",
    "| :--- | :--- | :--- |\n",
    "| **Penalty Term** | Adds the sum of squared coefficients as penalty term:<br><br>$\\text{Penalty Term} = \\lambda \\sum_{i=1}^{n}b_i^2$ | Adds the sum of the absolute values of coefficients as a penalty term:<br><br>$\\text{Penalty Term} = \\lambda \\sum_{i=1}^{n} \\lvert{b_i}\\rvert$ |\n",
    "| **Effect on Coeffcients** | Shrinks the coefficents towards zero but does not set them exactly to zero. It is effective in preventing overfitting and handling multicollinearity. | Can shrink some coefficients exactly to zero, effectively performing feature selection. It not only prevents overfitting and handles multicollinearity but also provides a form of automatic feature selection. |\n",
    "| **Use Case** | 1. Suitable when you believe that all features are important and should be included in the model.<br><br>2. Useful for handling multicollinearity (high correlation among independent variables). | 1. Useful when you have a large number of features and believe that some of them are less important or irrelevant.<br><br>2. Effective for feature selection as it sets some coefficients to exactly zero. |\n",
    "| **Mathematical Expression** | Minimizes the cost function with the sum of squared coefficeints.<br><br>$\\text{Cost} = \\text{MSE} + \\lambda \\sum_{i=1}^{n}b_i^2$ | Minimizes the cost function with the sum of absolute values of coefficeints.<br><br>$\\text{Cost} = \\text{MSE} + \\lambda \\sum_{i=1}^{n} \\lvert{b_i}\\rvert$ |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b150dc-0d88-4f85-b1b9-b779955f1ceb",
   "metadata": {},
   "source": [
    "### 1.1.4. Multi-task Lasso"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af9dc820-15d5-480f-b52e-375dc0b0e450",
   "metadata": {},
   "source": [
    "The [MultiTaskLasso](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.MultiTaskLasso.html#sklearn.linear_model.MultiTaskLasso) is a linear model that estimates sparse coefficients for multiple regression problems jointly: `y` is a 2D array, of shape `(n_samples, n_tasks)`. The constraint is that the selected features are the same for all the regression problems, also called tasks.\n",
    "\n",
    "The following figure compares the location of the non-zero entries in the coefficient matrix W obtained with a simple Lasso or a MultiTaskLasso. The Lasso estimates yield scattered non-zeros while the non-zeros of the MultiTaskLasso are full columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bb166d7-090c-4ec5-a5d7-3dd37c88ed98",
   "metadata": {},
   "source": [
    "<center><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_multi_task_lasso_support_001.png\" /><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_multi_task_lasso_support_002.png\" /></center>\n",
    "\n",
    "<center>Fitting a time-series model, imposing that any active feature be active at all times.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4883300-3a45-4f49-89b9-3c1a6bbc7b0d",
   "metadata": {},
   "source": [
    "$\\textbf{Example}$\n",
    "- Joint feature selection with multi-task Lasso - [Sci-kit Link](https://scikit-learn.org/stable/auto_examples/linear_model/plot_multi_task_lasso_support.html#sphx-glr-auto-examples-linear-model-plot-multi-task-lasso-support-py) | [Python code]() | [Jupyter Notebook]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d15709f1-2242-412c-8a5d-5d8dbfa4f2f2",
   "metadata": {},
   "source": [
    "$\\textbf{Mathematical Details}$\n",
    "\n",
    "Mathematically, it consists of a linear model trained with a $\\ell_1 \\ell_2$-norm for reglarization. The objective function to minimize is:\n",
    "\n",
    "$$\\min_w \\frac{1}{2n_{samples}}\\lVert Xw - Y \\rVert_{Fro}^2 + \\alpha\\lVert w \\rVert_{21}$$\n",
    "\n",
    "where $Fro$ indicates the Frobenius norm.\n",
    "\n",
    "$$\\lVert A \\rVert_{Fro} = \\sqrt{\\sum_{ij}a^2_{ij}}$$\n",
    "\n",
    "and $\\ell_1 \\ell_2$ reads\n",
    "\n",
    "$$\\lVert A \\rVert_{21} = \\sum_i \\sqrt{\\sum_{j}a^2_{ij}}$$\n",
    "\n",
    "The implementation in the class :class:`~sklearn.linear_model.MultiTaskLasso` uses [coordinate descent](#coord-descent) as the algorithm to fit the coefficients."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51054283-1636-46b6-a3b9-0910423f54a3",
   "metadata": {},
   "source": [
    "<a id='coord-descent'></a>\n",
    "$\\textbf{Coordinate Descent}$\n",
    "\n",
    "Coordinate descent is an optimization algorithm that successively minimizes along coordinate directions to find the minimum of a function.\n",
    "\n",
    "At each iteration, the algorithm determines a coordinate or coordinate block via a coordinate selection rule, then exactly or inexactly minimizes over the corresponding coordinate hyperplane while fixing all other coordinates or coordinate blocks.\n",
    "\n",
    "A line search along the coordinate direction can be performed at the current iterate to determine the appropriate step size. Coordinate descent is applicable in both differentiable and derivative-free contexts.\n",
    "\n",
    "> **Coordinate Descent** is based on the idea that the minimization of a multivariable function $F(x)$ can be achieved by minimizing it along one direction at a time, i.e., solving univariate (or at least much simpler) optimization problems in a loop."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f357dc29-bccb-4a75-8b64-07d2f37abc53",
   "metadata": {},
   "source": [
    "In the simplest case of cyclic coordinate descent, one cyclically iterates through the directions, one at a time, minimizing the objective function with respect to each coordinate direction at a time. That is, starting with initial variable values $X^0 = (x_1^0, \\dots, x_n^0)$, iteration $k+1$ defines $x^{k+1}$ from $x^k$ by iteratively solving the single variable optimization problems\n",
    "\n",
    "$$ x_i^{k+1} = \\arg \\min_{y \\in \\mathbb{R}} f(x_1^{k+1}, \\dots, x_{i-1}^{k+1}, y, x_{i+1}^k, \\dots, x_n^k)$$\n",
    "\n",
    "for each variable $x_i$ of $X$ for $i$ from $1$ to $n$.\n",
    "\n",
    "Thus, one begins with an initial guess $X^0$ for a local minimum of $F$, and gets a sequence $X^0$, $X^1$, $X^2$, ... iteratively.\n",
    "\n",
    "By doing line search in each iteration, one automatically has\n",
    "\n",
    "$$F(X^0) \\geq F(X^1) \\geq F(X^2) \\geq \\dots $$\n",
    "\n",
    "It can be shown that this sequence has similar convergence properties as steepest descent. No improvement after one cycle of line search along coordinate directions implies a stationary point is reached.\n",
    "\n",
    "This process is illustrated below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9d5595f-436c-450d-a78f-b74be6f73bef",
   "metadata": {},
   "source": [
    "<center><img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/e3/Coordinate_descent.svg/1024px-Coordinate_descent.svg.png\" /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e7d12d2-31cb-49bf-bb52-84dafb919813",
   "metadata": {},
   "source": [
    "$\\textbf{Differentiable case}$\n",
    "\n",
    "In the case of a continuosly differentiable function $F$, a coordinate descent algorithm can be sketched as:\n",
    "\n",
    "> - Choose an initial parameter vector $X$\n",
    "> - Until convergence is reached, or for some fixed number of iterations:\n",
    ">   - Choose an index $i$ from $1$ to $n$.\n",
    ">   - Choose a step size $\\alpha$\n",
    ">   - Update $x_i$ to $x_i - \\alpha \\frac{\\delta F}{\\delta x_i}(X)$\n",
    "\n",
    "The step size can be chosen in various ways, e.g., by solving for the exact minimizer of $f(x_i) = F(X)$ (i.e., $F$ with all variables but $x_i$ fixed), or by traditional line search criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de4e73c9-1662-46a7-83c5-803e1acb1d03",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}