PowerModelsONM-Dev-Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced R&D with PowerModelsONM\n",
    "\n",
    "**Author**:\\\n",
    "David M Fobes\\\n",
    "Deputy Group Leader\\\n",
    "A-1 Information Systems and Modeling Group\\\n",
    "[dfobes@lanl.gov](mailto:dfobes@lanl.gov)\n",
    "\n",
    "LA-UR-25-20156\n",
    "\n",
    "**Los Alamos National Laboratory Grid Science Winter School 2025**\n",
    "\n",
    "This is an advanced introduction to developing an advanced problem formulation with PowerModelsONM and PowerModelsDistribution. The PowerModelsITD and PowerModelsONM tutorials are prerequisities to this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Development Environment Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended that you create a new **project** when developing your own custom workflows, formulations, problems, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m\u001b[1m  Activating\u001b[22m\u001b[39m project at `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp`\n",
      "\u001b[33m\u001b[1m┌ \u001b[22m\u001b[39m\u001b[33m\u001b[1mWarning: \u001b[22m\u001b[39mcould not download https://pkg.julialang.org/registries\n",
      "\u001b[33m\u001b[1m│ \u001b[22m\u001b[39m  exception = RequestError: Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries\n",
      "\u001b[33m\u001b[1m└ \u001b[22m\u001b[39m\u001b[90m@ Pkg.Registry ~/.julia/juliaup/julia-1.11.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Pkg/src/Registry/Registry.jl:77\u001b[39m\n",
      " registry at `~/.julia/registries/General.toml`\n",
      "\u001b[32m\u001b[1m   Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Project.toml`\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Manifest.toml`\n",
      "\u001b[32m\u001b[1m   Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Project.toml`\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Manifest.toml`\n",
      "\u001b[32m\u001b[1m   Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Project.toml`\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Manifest.toml`\n",
      "\u001b[32m\u001b[1m   Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Project.toml`\n",
      "\u001b[32m\u001b[1m  No Changes\u001b[22m\u001b[39m to `~/Library/CloudStorage/OneDrive-LosAlamosNationalLaboratory/local/working/winter-school/tmp/Manifest.toml`\n"
     ]
    }
   ],
   "source": [
    "using Pkg\n",
    "\n",
    "Pkg.activate(\"./tmp\")\n",
    "\n",
    "Pkg.add(\"PowerModelsONM\")\n",
    "Pkg.add(\"PowerModelsDistribution\")\n",
    "Pkg.add(\"InfrastructureModels\")\n",
    "Pkg.add(\"JuMP\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason you might instead clone the PowerModelsONM repository and activate that project in Julia is when you plan to develop the ONM package itself, e.g., to submit a bugfix, new feature, etc. \n",
    "\n",
    "Alternatively, you might do this to debug an issue you are seeing, but we also recommend a slightly different workflow for this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To setup the folder structure, in your terminal:\n",
    "\n",
    "```bash\n",
    "cd working\n",
    "\n",
    "git clone https://github.com/lanl-ansi/PowerModelsONM.jl\n",
    "\n",
    "mkdir myproject && cd myproject\n",
    "\n",
    "julia --project=.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In your Julia REPL:\n",
    "\n",
    "```julia\n",
    "]dev ../PowerModelsONM.jl\n",
    "```\n",
    "\n",
    "or \n",
    "\n",
    "```julia\n",
    "using Pkg\n",
    "Pkg.develop(\"../PowerModelsONM.jl\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we do our standard imports setup paths and create a solver.\n",
    "\n",
    "We recommend using `import ... as ..` or just `import` so that it is clear from your code where different function originate.\n",
    "\n",
    "In this tutorial we will need `JuMP` explicitly, to create new variables, constraints, objectives, etc.\n",
    "\n",
    "Since we will only use the MILP formulation (i.e., `LinDist3Flow`), `HiGHS` will be an appropriate solver for the whole notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MathOptInterface.OptimizerWithAttributes(HiGHS.Optimizer, Pair{MathOptInterface.AbstractOptimizerAttribute, Any}[MathOptInterface.RawOptimizerAttribute(\"presolve\") => \"off\", MathOptInterface.RawOptimizerAttribute(\"primal_feasibility_tolerance\") => 1.0e-6, MathOptInterface.RawOptimizerAttribute(\"dual_feasibility_tolerance\") => 1.0e-6, MathOptInterface.RawOptimizerAttribute(\"small_matrix_value\") => 1.0e-12, MathOptInterface.RawOptimizerAttribute(\"allow_unbounded_or_infeasible\") => true, MathOptInterface.RawOptimizerAttribute(\"output_flag\") => false])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import PowerModelsONM as ONM\n",
    "import PowerModelsDistribution as PMD\n",
    "import InfrastructureModels as IM\n",
    "import JuMP\n",
    "\n",
    "ieee13_network_path = joinpath(dirname(pathof(ONM)), \"../examples/data/network.ieee13mod.dss\")\n",
    "ieee13_settings_path = joinpath(dirname(pathof(ONM)), \"../examples/data/settings.ieee13mod.json\")\n",
    "\n",
    "solver = ONM.optimizer_with_attributes(\n",
    "    ONM.HiGHS.Optimizer,\n",
    "    \"presolve\" => \"off\",\n",
    "    \"primal_feasibility_tolerance\" => 1e-6,\n",
    "    \"dual_feasibility_tolerance\" => 1e-6,\n",
    "    \"small_matrix_value\" => 1e-12,\n",
    "    \"allow_unbounded_or_infeasible\" => true,\n",
    "    \"output_flag\" => false\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Data Models in PowerModelsDistribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Data Model portion of this notebook is the only section that really is unique to `PowerModelsDistribution` derived packages like `PowerModelsONM`. Because PowerModelsDistribution has two data models, the ENGINEERING model and the MATHEMATICAL model, additional work is needed to get data to pass through that transformation.\n",
    "\n",
    "In this notebook we use the convention `eng` for `ENGINEERING` data, and `math` for `MATHEMATICAL` data. Also, `_s` indicates some settings have been applied, and `mn_` indicates a multinetwork data structure.\n",
    "\n",
    "First we import our base network data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mPowerModelsDistribution | Info ] : \u001b[22m\u001b[39mCircuit has been reset with the 'clear' on line 1 in 'network.ieee13mod.dss'\n",
      "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mPowerModelsDistribution | Info ] : \u001b[22m\u001b[39mCommand 'calcv' on line 166 in 'network.ieee13mod.dss' is not supported, skipping.\n",
      "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mPowerModelsDistribution | Info ] : \u001b[22m\u001b[39mCommand 'solve' on line 180 in 'network.ieee13mod.dss' is not supported, skipping.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 21 entries:\n",
       "  \"recloser\"                => Dict{String, Any}(\"671700\"=>Dict{String, Any}(\"l…\n",
       "  \"is_kron_reduced\"         => true\n",
       "  \"conductor_ids\"           => [1, 2, 3]\n",
       "  \"bus\"                     => Dict{String, Any}(\"800aux\"=>Dict{String, Any}(\"r…\n",
       "  \"name\"                    => \"ieee13nodeckt\"\n",
       "  \"settings\"                => Dict{String, Any}(\"sbase_default\"=>1000.0, \"vbas…\n",
       "  \"files\"                   => [\"/Users/dfobes/.julia/packages/PowerModelsONM/T…\n",
       "  \"switch_close_actions_ub\" => Inf\n",
       "  \"storage\"                 => Dict{String, Any}(\"battery_mg1a\"=>Dict{String, A…\n",
       "  \"switch\"                  => Dict{String, Any}(\"cb_101\"=>Dict{String, Any}(\"c…\n",
       "  \"generator\"               => Dict{String, Any}(\"675\"=>Dict{String, Any}(\"vg\"=…\n",
       "  \"dss_options\"             => DssOptions(\"%growth\"=>2.5, \"%mean\"=>65.0, \"%norm…\n",
       "  \"time_series\"             => Dict{String, Any}(\"microgrid1a\"=>Dict{String, An…\n",
       "  \"voltage_source\"          => Dict{String, Any}(\"source\"=>Dict{String, Any}(\"s…\n",
       "  \"line\"                    => Dict{String, Any}(\"632670\"=>Dict{String, Any}(\"c…\n",
       "  \"relay\"                   => Dict{String, Any}(\"801675\"=>Dict{String, Any}(\"l…\n",
       "  \"data_model\"              => ENGINEERING\n",
       "  \"transformer\"             => Dict{String, Any}(\"xfm1\"=>Dict{String, Any}(\"pol…\n",
       "  \"shunt\"                   => Dict{String, Any}(\"cap1\"=>Dict{String, Any}(\"sou…\n",
       "  \"solar\"                   => Dict{String, Any}(\"pv_mg1b\"=>Dict{String, Any}(\"…\n",
       "  \"load\"                    => Dict{String, Any}(\"634a\"=>Dict{String, Any}(\"mod…"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eng = ONM.parse_file(ieee13_network_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we generate some basic settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 5 entries:\n",
       "  \"options\" => Dict{String, Any}(\"outputs\"=>Dict{String, Any}(\"log-level\"=>\"war…\n",
       "  \"line\"    => Dict{String, Any}(\"632670\"=>Dict{String, Any}(\"vad_ub\"=>[10.0, 1…\n",
       "  \"solvers\" => Dict{String, Any}(\"HiGHS\"=>Dict{String, Any}(\"presolve\"=>\"choose…\n",
       "  \"switch\"  => Dict{String, Any}(\"cb_101\"=>Dict{String, Any}(\"cm_ub\"=>[Inf, Inf…\n",
       "  \"bus\"     => Dict{String, Any}(\"646\"=>Dict{String, Any}(\"vm_ub\"=>[2.64195, 2.…"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "settings = ONM.build_settings_new(\n",
    "    eng;\n",
    "    vm_lb_pu=0.9,\n",
    "    vm_ub_pu=1.1,\n",
    "    line_limit_multiplier=Inf,\n",
    "    transformer_limit_multiplier=Inf,\n",
    "    vad_deg=10.0,\n",
    "    timestep_hours=0.10,\n",
    "    switch_close_actions_ub=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "PMD.apply_voltage_bounds!(eng; vm_lb=0.9, vm_ub=1.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and apply them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 24 entries:\n",
       "  \"recloser\"                => Dict{String, Any}(\"671700\"=>Dict{String, Any}(\"l…\n",
       "  \"is_kron_reduced\"         => true\n",
       "  \"conductor_ids\"           => [1, 2, 3]\n",
       "  \"time_elapsed\"            => [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]\n",
       "  \"bus\"                     => Dict{String, Any}(\"800\"=>Dict{String, Any}(\"rg\"=…\n",
       "  \"name\"                    => \"ieee13nodeckt\"\n",
       "  \"solvers\"                 => Dict{String, Any}(\"HiGHS\"=>Dict{String, Any}(\"pr…\n",
       "  \"settings\"                => Dict{String, Any}(\"sbase_default\"=>1000.0, \"vbas…\n",
       "  \"files\"                   => [\"/Users/dfobes/.julia/packages/PowerModelsONM/T…\n",
       "  \"switch_close_actions_ub\" => [1, 1, 1, 1, 1, 1, 1, 1]\n",
       "  \"storage\"                 => Dict{String, Any}(\"battery_mg1a\"=>Dict{String, A…\n",
       "  \"switch\"                  => Dict{String, Any}(\"801675\"=>Dict{String, Any}(\"c…\n",
       "  \"generator\"               => Dict{String, Any}(\"675\"=>Dict{String, Any}(\"pg\"=…\n",
       "  \"dss_options\"             => DssOptions(\"%growth\"=>2.5, \"%mean\"=>65.0, \"%norm…\n",
       "  \"time_series\"             => Dict{String, Any}(\"microgrid1a\"=>Dict{String, An…\n",
       "  \"voltage_source\"          => Dict{String, Any}(\"source\"=>Dict{String, Any}(\"s…\n",
       "  \"line\"                    => Dict{String, Any}(\"632670\"=>Dict{String, Any}(\"c…\n",
       "  \"relay\"                   => Dict{String, Any}(\"801675\"=>Dict{String, Any}(\"l…\n",
       "  \"data_model\"              => ENGINEERING\n",
       "  \"options\"                 => Dict{String, Any}(\"outputs\"=>Dict{String, Any}(\"…\n",
       "  \"transformer\"             => Dict{String, Any}(\"xfm1\"=>Dict{String, Any}(\"pol…\n",
       "  \"shunt\"                   => Dict{String, Any}(\"cap1\"=>Dict{String, Any}(\"sou…\n",
       "  \"solar\"                   => Dict{String, Any}(\"pv_mg1b\"=>Dict{String, Any}(\"…\n",
       "  \"load\"                    => Dict{String, Any}(\"634a\"=>Dict{String, Any}(\"mod…"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eng_s = ONM.apply_settings(eng, settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the base workflow in ONM is centered around multinetwork problems, even when working with the base network there can be an issue with applying settings where `switch_close_actions_ub` and `time_elapsed` are vectors in the non-multinetwork structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fix vector properties\n",
    "eng_s[\"switch_close_actions_ub\"]=eng_s[\"switch_close_actions_ub\"][1]\n",
    "eng_s[\"time_elapsed\"]=eng_s[\"time_elapsed\"][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Fields to Data Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets say that you have some custom data you want to add to a particular generator in the network.\n",
    "\n",
    "First, let's look at the properties we have parsed from OpenDSS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 15 entries:\n",
       "  \"pg\"            => [166.667, 166.667, 166.667]\n",
       "  \"connections\"   => [1, 2, 3]\n",
       "  \"bus\"           => \"675\"\n",
       "  \"configuration\" => WYE\n",
       "  \"qg\"            => [116.667, 116.667, 116.667]\n",
       "  \"status\"        => ENABLED\n",
       "  \"dss\"           => Dict{String, Any}(\"kva\"=>\"500\", \"name\"=>\"675\", \"phases\"=>\"…\n",
       "  \"vg\"            => [1.38564, 1.38564, 1.38564]\n",
       "  \"qg_ub\"         => [233.333, 233.333, 233.333]\n",
       "  \"phases\"        => 3\n",
       "  \"source_id\"     => \"generator.675\"\n",
       "  \"qg_lb\"         => [-233.333, -233.333, -233.333]\n",
       "  \"pg_lb\"         => [0.0, 0.0, 0.0]\n",
       "  \"pg_ub\"         => [166.667, 166.667, 166.667]\n",
       "  \"control_mode\"  => FREQUENCYDROOP"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eng_s[\"generator\"][\"675\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can add a new property `ramp_rate`, which is in units % of rated capacity per minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eng_s[\"generator\"][\"675\"][\"ramp_rate\"] = 3.0 # % of rated capacity / min\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we transform the data model to `MATHEMATICAL`, we notice that this new property is missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "KeyError: key \"ramp_rate\" not found",
     "output_type": "error",
     "traceback": [
      "KeyError: key \"ramp_rate\" not found",
      "",
      "Stacktrace:",
      " [1] getindex(h::Dict{String, Any}, key::String)",
      "   @ Base ./dict.jl:477",
      " [2] top-level scope",
      "   @ In[13]:3"
     ]
    }
   ],
   "source": [
    "math = ONM.transform_data_model(eng_s)\n",
    "\n",
    "math[\"gen\"][\"1\"][\"ramp_rate\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get this new property for when we are working with the JuMP Model, we need to use the passthrough interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Vector{String}} with 1 entry:\n",
       "  \"generator\" => [\"ramp_rate\"]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_eng2math_passthrough = Dict{String,Vector{String}}(\n",
    "    \"generator\"=>String[\"ramp_rate\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 22 entries:\n",
       "  \"pg\"            => [0.166667, 0.166667, 0.166667]\n",
       "  \"model\"         => 2\n",
       "  \"connections\"   => [1, 2, 3]\n",
       "  \"shutdown\"      => 0.0\n",
       "  \"startup\"       => 0.0\n",
       "  \"configuration\" => WYE\n",
       "  \"name\"          => \"675\"\n",
       "  \"qg\"            => [0.116667, 0.116667, 0.116667]\n",
       "  \"gen_bus\"       => 8\n",
       "  \"pmax\"          => [0.166667, 0.166667, 0.166667]\n",
       "  \"dss\"           => Dict{String, Any}(\"kva\"=>\"500\", \"name\"=>\"675\", \"phases\"=>\"…\n",
       "  \"vbase\"         => 2.40178\n",
       "  \"source_id\"     => \"generator.675\"\n",
       "  \"index\"         => 1\n",
       "  \"cost\"          => [1.0, 0.0]\n",
       "  \"gen_status\"    => 1\n",
       "  \"ramp_rate\"     => 3.0\n",
       "  \"qmin\"          => [-0.233333, -0.233333, -0.233333]\n",
       "  \"qmax\"          => [0.233333, 0.233333, 0.233333]\n",
       "  \"control_mode\"  => 0\n",
       "  \"pmin\"          => [0.0, 0.0, 0.0]\n",
       "  \"ncost\"         => 2"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "math = ONM.transform_data_model(eng_s, eng2math_passthrough=my_eng2math_passthrough)\n",
    "\n",
    "math[\"gen\"][\"1\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AbstractUnbalancedPowerModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When building a problem, we are working with a Julia Struct called the AbstractUnbalancedPowerModel. The will be different for other packages, but the structure and helper function all work the same way.\n",
    "\n",
    "First, lets transform our data to multinetwork."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 9 entries:\n",
       "  \"options\"      => Dict{String, Any}(\"outputs\"=>Dict{String, Any}(\"log-level\"=…\n",
       "  \"name\"         => \"ieee13nodeckt\"\n",
       "  \"solvers\"      => Dict{String, Any}(\"HiGHS\"=>Dict{String, Any}(\"presolve\"=>\"c…\n",
       "  \"files\"        => [\"/Users/dfobes/.julia/packages/PowerModelsONM/ThIPC/src/..…\n",
       "  \"multinetwork\" => true\n",
       "  \"dss_options\"  => DssOptions(\"%growth\"=>2.5, \"%mean\"=>65.0, \"%normal\"=>100.0,…\n",
       "  \"nw\"           => Dict{String, Any}(\"3\"=>Dict{String, Any}(\"recloser\"=>Dict{S…\n",
       "  \"mn_lookup\"    => Dict(\"8\"=>0.7, \"4\"=>0.3, \"1\"=>0.0, \"5\"=>0.4, \"2\"=>0.1, \"6\"=…\n",
       "  \"data_model\"   => ENGINEERING"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mn_eng_s = ONM.make_multinetwork(eng_s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's create the AbstractUnbalancedPowerModel. To do so, you need at a minimum\n",
    "\n",
    "- input data\n",
    "- formulation\n",
    "- problem definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pm = ONM.instantiate_onm_model(mn_eng_s, ONM.LPUBFDiagPowerModel, ONM.build_mn_block_mld; eng2math_passthrough=my_eng2math_passthrough, multinetwork=true);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The primary properties of this struct are the following `Symbol`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(:model, :data, :setting, :solution, :ref, :var, :con, :sol, :sol_proc, :ext)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "propertynames(pm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the key properties you will care about, which are consistent across the different packages in their usage.\n",
    "\n",
    "- `:model`: JuMP Model\n",
    "- `:data`: The `MATHEMATICAL` data model used in the creation of the `:model`\n",
    "- `:solution`: The solution, which will not be populated until `solve` is called\n",
    "- `:ref`: a reference dict used to quickly access different aspects of the data\n",
    "- `:var`: a quick reference to the different variables created\n",
    "- `:con`: a quick reference to the different constraints created\n",
    "- `:sol`: a way to specify which variables / expressions should appear in the solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a few useful helper functions used to access different parts of the struct. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access the subnetworks of the multinetwork, use `nws`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Int64, Any} with 8 entries:\n",
       "  5 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  4 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  6 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  7 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  2 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  8 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  3 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…\n",
       "  1 => Dict{Symbol, Any}(:block_line_losses=>Dict(5=>0.0, 4=>0.0651414, 6=>0.05…"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.nws(pm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or if you just want the multinetwork ids, `nw_ids`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KeySet for a Dict{Int64, Any} with 8 entries. Keys:\n",
       "  5\n",
       "  4\n",
       "  6\n",
       "  7\n",
       "  2\n",
       "  8\n",
       "  3\n",
       "  1"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.nw_ids(pm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ref` allows us to access special aspects of the data. These are more than just the data itself, although that is usually contained as well. The things in this dictionary will be different for each package, dependent on the types of problems trying to be solved. It is possible to extend the `ref` with an extension to add custom data, but that is out of scope here. See PowerModelsONM source for an example where we extend the PowerModelsDistribution `ref`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Symbol, Any} with 70 entries:\n",
       "  :block_line_losses       => Dict(5=>0.0, 4=>0.0651414, 6=>0.0579035, 7=>5.760…\n",
       "  :buspairs                => Dict{NTuple{4, Int64}, Dict{String, Real}}((23, 8…\n",
       "  :block_graph             => SimpleGraph{Int64}(6, [[3, 6, 7], [4, 5], [1], [2…\n",
       "  :substation_blocks       => [3]\n",
       "  :bus_conns_gen           => Dict(5=>[], 35=>[], 20=>[], 16=>[], 12=>[(2, [1, …\n",
       "  :shunt_block_map         => Dict(2=>1, 1=>7)\n",
       "  :block_loads             => Dict{Int64, Set}(5=>Set([9]), 4=>Set([7, 14]), 6=…\n",
       "  :options                 => Dict{String, Any}(\"data\"=>Dict{String, Any}(\"time…\n",
       "  :gen                     => Dict{Int64, Any}(4=>Dict{String, Any}(\"pg\"=>[0.0,…\n",
       "  :switch_dispatchable     => Dict{Int64, Any}(5=>Dict{String, Any}(\"f_connecti…\n",
       "  :load                    => Dict{Int64, Any}(5=>Dict{String, Any}(\"model\"=>IM…\n",
       "  :switch_scores           => Dict(5=>1.64499, 4=>2.30142, 6=>1.64557, 7=>3.090…\n",
       "  :arcs_branch_to          => [(5, 8, 23), (16, 26, 27), (20, 32, 33), (12, 4, …\n",
       "  :is_projected            => true\n",
       "  :bus_arcs_transformer    => Dict(5=>[], 35=>[(5, 35, 22)], 20=>[(1, 20, 29)],…\n",
       "  :block_shunts            => Dict{Int64, Set{Int64}}(5=>Set(), 4=>Set(), 6=>Se…\n",
       "  :arcs_transformer_from   => [(5, 22, 35), (4, 14, 30), (6, 10, 33), (7, 10, 3…\n",
       "  :arcs_switch_to          => [(5, 25, 19), (4, 12, 24), (6, 13, 3), (7, 9, 7),…\n",
       "  :switch_close_actions_ub => 1\n",
       "  :bus_gens                => Dict(5=>[], 35=>[], 20=>[], 16=>[], 12=>[2], 24=>…\n",
       "  :arcs_branch_from        => [(5, 23, 8), (16, 27, 26), (20, 33, 32), (12, 2, …\n",
       "  :block_gens              => Dict{Int64, Set{Int64}}(5=>Set(), 4=>Set([3]), 6=…\n",
       "  :bus_conns_load          => Dict(5=>[(5, [1])], 35=>[], 20=>[], 16=>[(12, [2]…\n",
       "  :gen_block_map           => Dict(4=>3, 2=>6, 3=>4, 1=>7)\n",
       "  :shunt_dispatchable      => Dict{Int64, Any}()\n",
       "  ⋮                        => ⋮"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.ref(pm, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 22 entries:\n",
       "  \"pg\"            => [0.166667, 0.166667, 0.166667]\n",
       "  \"model\"         => 2\n",
       "  \"connections\"   => [1, 2, 3]\n",
       "  \"shutdown\"      => 0.0\n",
       "  \"startup\"       => 0.0\n",
       "  \"configuration\" => WYE\n",
       "  \"name\"          => \"675\"\n",
       "  \"qg\"            => [0.116667, 0.116667, 0.116667]\n",
       "  \"gen_bus\"       => 8\n",
       "  \"pmax\"          => [0.166667, 0.166667, 0.166667]\n",
       "  \"dss\"           => Dict{String, Any}(\"kva\"=>\"500\", \"name\"=>\"675\", \"phases\"=>\"…\n",
       "  \"vbase\"         => 2.40178\n",
       "  \"source_id\"     => \"generator.675\"\n",
       "  \"index\"         => 1\n",
       "  \"cost\"          => [1.0, 0.0]\n",
       "  \"gen_status\"    => 1\n",
       "  \"ramp_rate\"     => 3.0\n",
       "  \"qmin\"          => [-0.233333, -0.233333, -0.233333]\n",
       "  \"qmax\"          => [0.233333, 0.233333, 0.233333]\n",
       "  \"control_mode\"  => 0\n",
       "  \"pmin\"          => [0.0, 0.0, 0.0]\n",
       "  \"ncost\"         => 2"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.ref(pm, 1, :gen, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `ids` if we only want the ids (keys) of the ref dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KeySet for a Dict{Int64, Any} with 4 entries. Keys:\n",
       "  4\n",
       "  2\n",
       "  3\n",
       "  1"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.ids(pm, 1, :gen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`var` is used to access the different JuMP variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Symbol, Any} with 42 entries:\n",
       "  :alpha                    => Dict{Tuple{Int64, Int64}, Union{Int64, VariableR…\n",
       "  :p                        => Dict{Tuple{Int64, Int64, Int64}, DenseAxisArray{…\n",
       "  :CCdi                     => Dict{Int64, Matrix{AffExpr}}(20=>[0 -1_CCdi_32[2…\n",
       "  :CCdr                     => Dict{Int64, Symmetric{VariableRef, Matrix{Variab…\n",
       "  :z_inverter               => Dict{Tuple{Symbol, Int64}, Union{Int64, Variable…\n",
       "  :se                       => 1-dimensional DenseAxisArray{JuMP.VariableRef,1,…\n",
       "  :qsw                      => Dict{Any, Any}((1, 14, 11)=>1-dimensional DenseA…\n",
       "  :ps                       => Dict{Int64, DenseAxisArray{VariableRef, 1, Tuple…\n",
       "  :lambda                   => Dict{Tuple{Int64, Int64}, VariableRef}((6, 8)=>1…\n",
       "  :w                        => Dict{Int64, DenseAxisArray{VariableRef, 1, Tuple…\n",
       "  :qs                       => Dict{Int64, DenseAxisArray{VariableRef, 1, Tuple…\n",
       "  :capacitor_state          => Dict{Any, Any}()\n",
       "  :tap                      => Dict{Int64, Vector{VariableRef}}(4=>[1_tm_4[1]],…\n",
       "  :qg_bus                   => Dict{Int64, Any}()\n",
       "  :sc                       => 1-dimensional DenseAxisArray{JuMP.VariableRef,1,…\n",
       "  :Xdi                      => Dict{Int64, Matrix{VariableRef}}(20=>[1_Xdi_22[2…\n",
       "  :sc_on                    => 1-dimensional DenseAxisArray{JuMP.VariableRef,1,…\n",
       "  :capacitor_reactive_power => Dict{Any, Any}()\n",
       "  :qg                       => Dict{Int64, DenseAxisArray{VariableRef, 1, Tuple…\n",
       "  :qsc                      => 1-dimensional DenseAxisArray{JuMP.VariableRef,1,…\n",
       "  :pt                       => Dict{Tuple{Int64, Int64, Int64}, DenseAxisArray{…\n",
       "  :qd                       => Dict{Any, Any}(5=>1-dimensional DenseAxisArray{J…\n",
       "  :z_demand                 => 1-dimensional DenseAxisArray{Any,1,...} with ind…\n",
       "  :phi_gfm                  => Dict{Tuple{Int64, Int64}, VariableRef}((1, 12)=>…\n",
       "  :pg_bus                   => Dict{Int64, Any}()\n",
       "  ⋮                         => ⋮"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.var(pm, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1-dimensional DenseAxisArray{JuMP.VariableRef,1,...} with index sets:\n",
       "    Dimension 1, [1, 2, 3]\n",
       "And data, a 3-element Vector{JuMP.VariableRef}:\n",
       " 1_pg_1[1]\n",
       " 1_pg_1[2]\n",
       " 1_pg_1[3]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ONM.var(pm, 1, :pg, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Standard Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the standard problem that we are modifying in the sections below, for reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 9 entries:\n",
       "  \"solve_time\"         => 5.49763\n",
       "  \"optimizer\"          => \"HiGHS\"\n",
       "  \"termination_status\" => OPTIMAL\n",
       "  \"dual_status\"        => NO_SOLUTION\n",
       "  \"primal_status\"      => FEASIBLE_POINT\n",
       "  \"objective\"          => 5.31939\n",
       "  \"solution\"           => Dict{String, Any}(\"nw\"=>Dict{String, Dict{String, Any…\n",
       "  \"mip_gap\"            => 0.0\n",
       "  \"objective_lb\"       => 5.31939"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = ONM.solve_onm_model(mn_eng_s, ONM.LPUBFDiagPowerModel, solver, ONM.build_mn_block_mld; multinetwork=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a Custom Constraint to Existing Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example we add a custom constraint using the `ramp_rate` data we added previously. Here we are constraining the new power output from the generator for which `ramp_rate` was defined to adhere to that rate.\n",
    "\n",
    "We make the assumption that the generator is off at the initial timestep, and needs to be ramped. In each subsequent timestep, we constrain the power output to only increase by the allowed rate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "my_new_constraint (generic function with 1 method)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function my_new_start_constraint(pm::ONM.AbstractUnbalancedPowerModel, nw::Int, i::Int)\n",
    "    gen = ONM.ref(pm, nw, :gen, i)\n",
    "\n",
    "    if !ismissing(get(gen, \"ramp_rate\", missing))\n",
    "        JuMP.@constraint(pm.model, sum(ONM.var(pm, nw, :pg, i)) <= gen[\"ramp_rate\"]/100.0 * sum(gen[\"pmax\"]) * ONM.ref(pm, nw, :time_elapsed) * 60.0)\n",
    "    end\n",
    "end\n",
    "\n",
    "function my_new_constraint(pm::ONM.AbstractUnbalancedPowerModel, nw_1::Int, nw_2::Int, i::Int)\n",
    "    gen = ONM.ref(pm, nw_2, :gen, i)\n",
    "\n",
    "    if !ismissing(get(gen, \"ramp_rate\", missing))\n",
    "        JuMP.@constraint(pm.model, sum(ONM.var(pm, nw_2, :pg, i)) - sum(ONM.var(pm, nw_1, :pg, i)) <= gen[\"ramp_rate\"]/100.0 * sum(gen[\"pmax\"]) * ONM.ref(pm, nw_2, :time_elapsed) * 60.0)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add this constraint, we can create a new problem function which calls an existing problem and then adds the new constraint to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "build_problem_with_added_constraint (generic function with 1 method)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function build_problem_with_added_constraint(pm::ONM.AbstractUnbalancedPowerModel)\n",
    "    ONM.build_mn_block_mld(pm)\n",
    "\n",
    "    network_ids = sort(collect(ONM.nw_ids(pm)))\n",
    "    n_1 = network_ids[1]\n",
    "\n",
    "    for i in ONM.ids(pm, n_1, :gen)\n",
    "        my_new_start_constraint(pm, n_1, i)\n",
    "    end\n",
    "\n",
    "    for n_2 in network_ids[2:end]\n",
    "        for i in ONM.ids(pm, n_2, :gen)\n",
    "            my_new_constraint(pm, n_1, n_2, i)\n",
    "        end\n",
    "        n_1 = n_2\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To solve the custom problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 9 entries:\n",
       "  \"solve_time\"         => 6.27187\n",
       "  \"optimizer\"          => \"HiGHS\"\n",
       "  \"termination_status\" => OPTIMAL\n",
       "  \"dual_status\"        => NO_SOLUTION\n",
       "  \"primal_status\"      => FEASIBLE_POINT\n",
       "  \"objective\"          => 5.34482\n",
       "  \"solution\"           => Dict{String, Any}(\"nw\"=>Dict{String, Dict{String, Any…\n",
       "  \"mip_gap\"            => 0.0\n",
       "  \"objective_lb\"       => 5.34482"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_constraint_result = ONM.solve_onm_model(mn_eng_s, ONM.LPUBFDiagPowerModel, solver, build_problem_with_added_constraint; eng2math_passthrough=my_eng2math_passthrough, multinetwork=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now see that the power output from the generator of interest ramps incrementally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8-element Vector{Float64}:\n",
       "  90.0\n",
       " 180.0\n",
       " 270.0\n",
       " 360.0\n",
       " 449.99999999999994\n",
       " 500.0\n",
       " 500.0\n",
       " 500.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[sum(new_constraint_result[\"solution\"][\"nw\"][\"$n\"][\"generator\"][\"675\"][\"pg\"]) for n in sort(parse.(Int,collect(keys(new_constraint_result[\"solution\"][\"nw\"]))))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whereas, previously the power output was always at the maximum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8-element Vector{Float64}:\n",
       " 500.0\n",
       " 394.0014720780217\n",
       " 500.0\n",
       " 500.0\n",
       " 500.0\n",
       " 500.0\n",
       " 500.0\n",
       " 500.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[sum(result[\"solution\"][\"nw\"][\"$n\"][\"generator\"][\"675\"][\"pg\"]) for n in sort(parse.(Int,collect(keys(result[\"solution\"][\"nw\"]))))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a Custom Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "variable_slack_bus (generic function with 1 method)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function variable_slack_bus(pm::ONM.AbstractUnbalancedPowerModel; nw::Int=ONM.nw_id_default)\n",
    "    slack_bus = ONM.var(pm, nw)[:slack_bus] = JuMP.@variable(pm.model, [i in ONM.ids(pm, nw, :bus)], base_name=\"$(nw)_slack_bus_$(i)\", binary=true)\n",
    "\n",
    "    IM.sol_component_value(pm, PMD.pmd_it_sym, nw, :bus, :slack_bus, ONM.ids(pm, nw, :bus), slack_bus)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "constraint_slack_bus (generic function with 1 method)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function constraint_slack_bus(pm; nw=ONM.nw_id_default)\n",
    "    for i in ONM.ids(pm, nw, :bus)\n",
    "        JuMP.@constraint(pm.model, sum(ONM.var(pm, nw, :slack_bus)) == 1)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "build_problem_with_new_variable (generic function with 1 method)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function build_problem_with_new_variable(pm)\n",
    "    ONM.build_mn_block_mld(pm)\n",
    "\n",
    "    for n in ONM.nw_ids(pm)\n",
    "        variable_slack_bus(pm; nw=n)\n",
    "        constraint_slack_bus(pm; nw=n)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 9 entries:\n",
       "  \"solve_time\"         => 6.92859\n",
       "  \"optimizer\"          => \"HiGHS\"\n",
       "  \"termination_status\" => OPTIMAL\n",
       "  \"dual_status\"        => NO_SOLUTION\n",
       "  \"primal_status\"      => FEASIBLE_POINT\n",
       "  \"objective\"          => 5.31939\n",
       "  \"solution\"           => Dict{String, Any}(\"nw\"=>Dict{String, Dict{String, Any…\n",
       "  \"mip_gap\"            => 0.0\n",
       "  \"objective_lb\"       => 5.31939"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_variable_result = ONM.solve_onm_model(mn_eng_s, ONM.LPUBFDiagPowerModel, solver, build_problem_with_new_variable; multinetwork=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Float64} with 25 entries:\n",
       "  \"800aux\"    => 0.0\n",
       "  \"671\"       => 0.0\n",
       "  \"800\"       => 0.0\n",
       "  \"680\"       => 0.0\n",
       "  \"634\"       => 0.0\n",
       "  \"652\"       => 0.0\n",
       "  \"701\"       => 0.0\n",
       "  \"675\"       => 0.0\n",
       "  \"702\"       => 0.0\n",
       "  \"650\"       => 0.0\n",
       "  \"600\"       => 0.0\n",
       "  \"700\"       => 0.0\n",
       "  \"rg60\"      => 0.0\n",
       "  \"801\"       => 0.0\n",
       "  \"611\"       => 0.0\n",
       "  \"645\"       => 0.0\n",
       "  \"632\"       => 0.0\n",
       "  \"675aux\"    => 1.0\n",
       "  \"703\"       => 0.0\n",
       "  \"633\"       => 0.0\n",
       "  \"684\"       => 0.0\n",
       "  \"sourcebus\" => 0.0\n",
       "  \"692\"       => 0.0\n",
       "  \"670\"       => 0.0\n",
       "  \"646\"       => 0.0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Dict(i => bus[\"slack_bus\"] for (i, bus) in new_variable_result[\"solution\"][\"nw\"][\"1\"][\"bus\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objective Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to create a new objective function, we merely overwrite the previous one. In this example, let's say that we want to minimize the amount of reactive power at the substation (voltage source)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "my_new_objective (generic function with 1 method)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function my_new_objective(pm::ONM.AbstractUnbalancedPowerModel)\n",
    "    # voltage source is gen 4\n",
    "    # voltage sources don't have power limits, use some arbitrarily large ones\n",
    "    qg_lb, qg_ub = -1e5, 1e5\n",
    "\n",
    "    qg_sqr = Dict()\n",
    "    for (n, nw_ref) in ONM.nws(pm)\n",
    "        qg_sqr[n] = [JuMP.@variable(pm.model, base_name=\"$(n)_qg_sqr_4_$(c)\") for c in ONM.ref(pm, n, :gen, 4, \"connections\")]\n",
    "        for (idx,c) in enumerate(ONM.ref(pm, n, :gen, 4, \"connections\"))\n",
    "            JuMP.set_lower_bound(ONM.var(pm, n, :qg, 4)[idx], qg_lb)\n",
    "            JuMP.set_upper_bound(ONM.var(pm, n, :qg, 4)[idx], qg_ub)\n",
    "\n",
    "            IM.relaxation_product(pm.model, ONM.var(pm, n, :qg, 4)[idx], ONM.var(pm, n, :qg, 4)[idx], qg_sqr[n][idx])\n",
    "        end\n",
    "    end\n",
    "\n",
    "    JuMP.@objective(pm.model, Min, sum( sum(qg_sqr[n])+sum((1-ONM.var(pm, n, :z_block, i)) for (i,block) in nw_ref[:blocks]) for (n, nw_ref) in ONM.nws(pm)))\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We modify the problem function in the same way as before, creating a new function, calling the `build_mn_block_mld` function from ONM, and calling our new function that overwrites the objective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "build_problem_with_new_objective (generic function with 1 method)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function build_problem_with_new_objective(pm::ONM.AbstractUnbalancedPowerModel)\n",
    "    ONM.build_mn_block_mld(pm)\n",
    "\n",
    "    my_new_objective(pm)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 9 entries:\n",
       "  \"solve_time\"         => 8.15234\n",
       "  \"optimizer\"          => \"HiGHS\"\n",
       "  \"termination_status\" => OPTIMAL\n",
       "  \"dual_status\"        => NO_SOLUTION\n",
       "  \"primal_status\"      => FEASIBLE_POINT\n",
       "  \"objective\"          => -2.39998e11\n",
       "  \"solution\"           => Dict{String, Any}(\"nw\"=>Dict{String, Dict{String, Any…\n",
       "  \"mip_gap\"            => 6.74821e-6\n",
       "  \"objective_lb\"       => -2.4e11"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_objective_result = ONM.solve_onm_model(mn_eng_s, ONM.LPUBFDiagPowerModel, solver, build_problem_with_new_objective; multinetwork=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check the reactive power at the substation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8-element Vector{Float64}:\n",
       "  230.41050604169695\n",
       "  226.26228592508318\n",
       " 1157.957316562829\n",
       " 1335.483434025467\n",
       " 1259.8473186910396\n",
       " 1325.2366806906377\n",
       " 1220.6729041517285\n",
       " 1341.8733176391056"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[sum(new_objective_result[\"solution\"][\"nw\"][\"$n\"][\"voltage_source\"][\"source\"][\"qg\"]) for n in sort(parse.(Int,collect(keys(new_objective_result[\"solution\"][\"nw\"]))))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whereas before it was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8-element Vector{Float64}:\n",
       "  987.789854172743\n",
       "  920.2997773939512\n",
       " 1000.0\n",
       "  839.7426189737266\n",
       " 1607.7887902649272\n",
       "  687.320201862733\n",
       " 1030.3336744731614\n",
       " 1370.7545462577225"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[sum(result[\"solution\"][\"nw\"][\"$n\"][\"voltage_source\"][\"source\"][\"qg\"]) for n in sort(parse.(Int,collect(keys(result[\"solution\"][\"nw\"]))))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare real power"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8-element Vector{Float64}:\n",
       " 2321.9390217792125\n",
       " 2361.0888092681753\n",
       " 2322.2720672533314\n",
       " 2402.2720672533314\n",
       " 2324.272067253331\n",
       " 2426.772067253331\n",
       " 2413.2720672533314\n",
       " 2454.772067253331"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[sum(new_objective_result[\"solution\"][\"nw\"][\"$n\"][\"voltage_source\"][\"source\"][\"pg\"]) for n in sort(parse.(Int,collect(keys(new_objective_result[\"solution\"][\"nw\"]))))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8-element Vector{Float64}:\n",
       " 2711.1018742997467\n",
       " 3000.0\n",
       " 2877.54527769662\n",
       " 2999.999999999999\n",
       " 3000.000000000001\n",
       " 3000.000000000001\n",
       " 3000.0000000000005\n",
       " 3000.000000000001"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[sum(result[\"solution\"][\"nw\"][\"$n\"][\"voltage_source\"][\"source\"][\"pg\"]) for n in sort(parse.(Int,collect(keys(result[\"solution\"][\"nw\"]))))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Completely Custom Problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "my_custom_problem (generic function with 1 method)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function my_custom_problem(pm::ONM.AbstractUnbalancedPowerModel)\n",
    "    ONM.variable_block_indicator(pm)\n",
    "\n",
    "    PMD.variable_mc_bus_voltage_on_off(pm)\n",
    "\n",
    "    PMD.variable_mc_branch_current(pm)\n",
    "    PMD.variable_mc_branch_power(pm)\n",
    "\n",
    "    PMD.variable_mc_switch_power(pm)\n",
    "    ONM.variable_switch_state(pm)\n",
    "\n",
    "    PMD.variable_mc_transformer_power(pm)\n",
    "    PMD.variable_mc_oltc_transformer_tap(pm)\n",
    "\n",
    "    PMD.variable_mc_generator_power_on_off(pm)\n",
    "\n",
    "    ONM.variable_mc_storage_power_mi_on_off(pm)\n",
    "\n",
    "    PMD.variable_mc_load_power(pm)\n",
    "\n",
    "    PMD.variable_mc_capcontrol(pm)\n",
    "\n",
    "    PMD.constraint_mc_model_current(pm)\n",
    "\n",
    "    for i in ONM.ids(pm, :ref_buses)\n",
    "        PMD.constraint_mc_theta_ref(pm, i)\n",
    "    end\n",
    "\n",
    "    ONM.constraint_mc_bus_voltage_block_on_off(pm)\n",
    "\n",
    "    for i in ONM.ids(pm, :gen)\n",
    "        ONM.constraint_mc_generator_power_block_on_off(pm, i)\n",
    "    end\n",
    "\n",
    "    for i in ONM.ids(pm, :load)\n",
    "        ONM.constraint_mc_load_power_block_on_off(pm, i)\n",
    "    end\n",
    "\n",
    "    for i in ONM.ids(pm, :bus)\n",
    "        ONM.constraint_mc_power_balance_shed_block(pm, i)\n",
    "    end\n",
    "\n",
    "    for i in ONM.ids(pm, :storage)\n",
    "        PMD.constraint_storage_state(pm, i)\n",
    "        ONM.constraint_storage_complementarity_mi_block_on_off(pm, i)\n",
    "        ONM.constraint_mc_storage_block_on_off(pm, i)\n",
    "        ONM.constraint_mc_storage_losses_block_on_off(pm, i)\n",
    "        PMD.constraint_mc_storage_thermal_limit(pm, i)\n",
    "    end\n",
    "\n",
    "    for i in ONM.ids(pm, :branch)\n",
    "        PMD.constraint_mc_power_losses(pm, i)\n",
    "        PMD.constraint_mc_model_voltage_magnitude_difference(pm, i)\n",
    "        PMD.constraint_mc_voltage_angle_difference(pm, i)\n",
    "\n",
    "        PMD.constraint_mc_thermal_limit_from(pm, i)\n",
    "        PMD.constraint_mc_thermal_limit_to(pm, i)\n",
    "\n",
    "        PMD.constraint_mc_ampacity_from(pm, i)\n",
    "        PMD.constraint_mc_ampacity_to(pm, i)\n",
    "    end\n",
    "\n",
    "    ONM.constraint_isolate_block(pm)\n",
    "    for i in ONM.ids(pm, :switch)\n",
    "        ONM.constraint_mc_switch_state_open_close(pm, i)\n",
    "\n",
    "        PMD.constraint_mc_switch_thermal_limit(pm, i)\n",
    "        PMD.constraint_mc_switch_ampacity(pm, i)\n",
    "    end\n",
    "\n",
    "    for i in ONM.ids(pm, :transformer)\n",
    "        ONM.constraint_mc_transformer_power_block_on_off(pm, i; fix_taps=false)\n",
    "    end\n",
    "\n",
    "    my_new_objective(pm)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 9 entries:\n",
       "  \"solve_time\"         => 0.0259009\n",
       "  \"optimizer\"          => \"HiGHS\"\n",
       "  \"termination_status\" => OPTIMAL\n",
       "  \"dual_status\"        => NO_SOLUTION\n",
       "  \"primal_status\"      => FEASIBLE_POINT\n",
       "  \"objective\"          => -3.0e10\n",
       "  \"solution\"           => Dict{String, Any}(\"voltage_source\"=>Dict{String, Any}…\n",
       "  \"mip_gap\"            => 2.3307e-10\n",
       "  \"objective_lb\"       => -3.0e10"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_problem_result = ONM.solve_onm_model(eng_s, ONM.LPUBFDiagPowerModel, solver, my_custom_problem)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.11.2",
   "language": "julia",
   "name": "julia-1.11"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}