diff --git a/fabric_examples/peering_client/figs/peering-fabric.png b/fabric_examples/peering_client/figs/peering-fabric.png new file mode 100644 index 00000000..9741c876 Binary files /dev/null and b/fabric_examples/peering_client/figs/peering-fabric.png differ diff --git a/fabric_examples/peering_client/peering-client-setup.ipynb b/fabric_examples/peering_client/peering-client-setup.ipynb new file mode 100644 index 00000000..59b3ac7c --- /dev/null +++ b/fabric_examples/peering_client/peering-client-setup.ipynb @@ -0,0 +1,393 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "28aeaa24-8a92-40aa-8c84-354837dbfa74", + "metadata": { + "tags": [] + }, + "source": [ + "# Setting Up a PEERING Client on FABRIC\n", + "\n", + "This document will guide you through setting up a [PEERING](https://peering.ee.columbia.edu) client on FABRIC. PEERING runs on its own slice on FABRIC, and your client will run on a slice you control. Connectivity between your client's slice and PEERING's slice will happen over OpenVPN tunnels traversing FABRIC's internal network. The OpenVPN tunnels will operate like a direct link between your slice and the PEERING routers on FABRIC. We will then configure the\n", + "[BIRD](https://bird.network.cz) to establish BGP sessions and exchange routes with PEERING.\n", + "\n", + "For more in-depth information on PEERING, check out our [CoNEXT Paper](https://homepages.dcc.ufmg.br/~cunha/papers/schlinker19peering.pdf) and the [PEERING client's README](https://github.com/PEERINGTestbed/client/blob/master/README.md) file." + ] + }, + { + "cell_type": "markdown", + "id": "e3749313-3839-4ae0-be7d-34b81c43303c", + "metadata": {}, + "source": [ + "## PEERING Experiment Approval\n", + "\n", + "Experiments on PEERING go through a review process that checks they follow PEERING's accepted-use-policy and will not disrupt the Internet. Head over to [PEERING's website](https://peering.ee.columbia.edu), create an account, and submit an experiment proposal. Once your experiment is approved, it will be allocated IP prefixes from PEERING address space and you will receive certificates that will allow you to establish OpenVPN tunnels (and BGP sessions over the tunnels) with PEERING routers." + ] + }, + { + "cell_type": "markdown", + "id": "b9345a32-8a8e-4be4-bc51-696ad0a51852", + "metadata": {}, + "source": [ + "## Connectivity Overview\n", + "\n", + "The PEERING slice and your slice will communicate over FABRIC's internal network. A slice can connect to FABRIC's internal network by requesting FABnet interfaces. A FABnetv6 interface is allocated a /64 prefix inside `2602:fcfb::/40`. To allow your slice's FABnet to speak with PEERING's FABnet, both slices need to route the /40 superprefix over the FABnet interface.\n", + "\n", + "![Deployment Overview](./figs/peering-fabric.png)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7b0f50f7-621b-43c0-bbcb-6baa56a348f8", + "metadata": {}, + "outputs": [], + "source": [ + "import ipaddress\n", + "import json\n", + "import os\n", + "import pathlib\n", + "import time\n", + "\n", + "from fabrictestbed_extensions.fablib.fablib import FablibManager\n", + "\n", + "manager = FablibManager()\n", + "print(f\"FABRIC's FABNet4 superprefix: {FablibManager.FABNETV4_SUBNET}\")\n", + "print(f\"FABRIC's FABNet6 superprefix: {FablibManager.FABNETV6_SUBNET}\")" + ] + }, + { + "cell_type": "markdown", + "id": "83906ccd-05ea-43f8-9504-4822e39d27bd", + "metadata": {}, + "source": [ + "If the command above fails, you may need to update your version of `fabrictestbed-extensions` (this notebook has last been tested with 1.5.4). Set the version number inside `requirements.txt` and run\n", + "\n", + "```bash\n", + "pip install --upgrade -r requirements.txt\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0961d33f-7ad5-4774-8326-116478b039e8", + "metadata": {}, + "outputs": [], + "source": [ + "manager.show_config();" + ] + }, + { + "cell_type": "markdown", + "id": "d2faa64a-4ba9-4716-853a-83028c2e7c98", + "metadata": { + "tags": [] + }, + "source": [ + "## Setting Up the Slice\n", + "\n", + "We create a FABnetv6 named `fab6` that we will use to connect to PEERING's routers. OpenVPN will be the main application using this interface, and the BGP sessions as well as traffic will go over the OpenVPN tunnels.\n", + "\n", + "> The Ubuntu 22.04 image has been tested to work. Using a different distros or versions should work, but may incur in compabitility issues due to different software versions. In particular, the provided configuration files work for BIRD 1 and Openvpn 2.\n", + "\n", + "> Below we will create one node at the Washington FABRIC site, but you can deploy any number of clients at any site within FABRIC.\n", + "\n", + "The cells below will configure a client running on the `WASH` site. You can run as many clients as you want on any site you want." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11cab9d8", + "metadata": {}, + "outputs": [], + "source": [ + "fabsite = \"WASH\"\n", + "slice_name = f\"slice_{fabsite.lower()}\"\n", + "node_name = f\"client_{fabsite.lower()}\"\n", + "fabnet_name = f\"client_{fabsite.lower()}_fab6\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2451977e-8ffb-4031-8276-7a0c9fe03ebe", + "metadata": {}, + "outputs": [], + "source": [ + "slice = manager.new_slice(name=slice_name)\n", + "node = slice.add_node(name=node_name,\n", + " site=fabsite,\n", + " instance_type='fabric.c2.m8.d100',\n", + " image='default_ubuntu_22')\n", + "wash_fab6 = node.add_component(model=\"NIC_Basic\", name=fabnet_name).get_interfaces()[0]\n", + "slice.add_l3network(name=fabnet_name, interfaces=[wash_fab6], type=\"IPv6\")\n", + "slice.submit()" + ] + }, + { + "cell_type": "markdown", + "id": "28fa0a04-6b2f-407a-991f-5bc6e09bb1fa", + "metadata": { + "tags": [] + }, + "source": [ + "## Configuring the FABnet Interface\n", + "\n", + "By default, the FABnetv6 interface we just created is not configured. We will set it up with the first IP address in the its allocated subnet, and then create a route for the /40 superprefix." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "19d803b3-8f95-4d02-a286-3685b60de7b7", + "metadata": {}, + "outputs": [], + "source": [ + "slice = manager.get_slice(name=slice_name)\n", + "node = slice.get_node(name=node_name)\n", + "fab6iface = node.get_interface(network_name=fabnet_name)\n", + "fab6net = slice.get_network(name=fabnet_name)\n", + "fab6pfx = fab6net.get_subnet()\n", + "fab6ips = fab6net.get_available_ips()\n", + "# Verifying assumptions (.1 is gateway and omitted from get_available_ips):\n", + "generator = fab6pfx.hosts()\n", + "assert fab6net.get_gateway() == next(generator)\n", + "assert fab6ips[0] == next(generator)\n", + "print(f\"Our FABnetv6's prefix is {fab6pfx}\")\n", + "print(f\"Our FABnetv6's IP will be {fab6ips[0]}\")\n", + "print(f\"Our FABnetv6's gateway is {fab6net.get_gateway()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "42332f87-967e-49c9-a7c8-d3e0466b0382", + "metadata": {}, + "outputs": [], + "source": [ + "node.get_ssh_command()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e9193811-dcb3-4d05-8fb0-11f190c6fcdc", + "metadata": {}, + "outputs": [], + "source": [ + "fab6json = json.loads(fab6iface.show(output=\"json\"))\n", + "node.execute(f\"sudo ip link set dev {fab6json['dev']} up\")\n", + "fab6iface.ip_addr_add(fab6ips[0], subnet=fab6pfx)\n", + "node.ip_route_add(subnet=FablibManager.FABNETV6_SUBNET, gateway=fab6net.get_gateway());" + ] + }, + { + "cell_type": "markdown", + "id": "36655389-0452-4183-a111-7f16911d25d7", + "metadata": {}, + "source": [ + "If you want to check configuration is correct, the commands below should:\n", + "\n", + "1. Show that the interface is `up`\n", + "2. That the IPv6 address is configured\n", + "3. That a route to the /40 superprefix exists\n", + "4. And that `fabwash.atlas.peering.ee.columbia.edu` is reachable" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5e1e7c0a-7186-41f3-875d-0f6a0c392a79", + "metadata": {}, + "outputs": [], + "source": [ + "node.execute(f\"ip link show dev {fab6json['dev']}\")\n", + "node.execute(f\"ip addr show dev {fab6json['dev']}\")\n", + "node.execute(f\"ip -6 route\")\n", + "node.execute(f\"ping -c1 fabwash.atlas.peering.ee.columbia.edu\")" + ] + }, + { + "cell_type": "markdown", + "id": "5f821434-84a9-4fbd-a9e8-75787334aa03", + "metadata": {}, + "source": [ + "## One-time Configuration of the PEERING Client\n", + "\n", + "Now that your node can communicate with the PEERING slice, you will need to configure the PEERING client on your node. This is a one-time process once the VM is created (although you will need to reexecute it if the VM is recreated). First, configure [SSH access][fabric-ssh] and [NAT64][fabric-nat64] on your node following FABRIC's instructions. With SSH access to your node and NAT64 working, follow these steps:\n", + "\n", + "[fabric-ssh]: ../../configure.ipynb\n", + "[fabric-nat64]: ../fablib_api/accessing_ipv4_services_from_ipv6_nodes/accessing_ipv4_services_from_ipv6_nodes.ipynb\n", + "1. Install required dependencies\n", + "\n", + " ```bash\n", + " apt update\n", + " apt install git openvpn bird socat psmisc ipcalc\n", + " systemctl disable bird bird6 openvpn\n", + " ```\n", + "\n", + "\n", + "2. Clone the `fabric` branch of the PEERING client repository\n", + "\n", + " ```bash\n", + " git clone https://github.com/PEERINGTestbed/client.git --branch fabric\n", + " ```\n", + "\n", + " > The FABRIC branch is identical to the main branch, it just changes the configuration files to allow connection to the FABRIC routers instead of PEERING's normal routers.\n", + "\n", + "\n", + "3. Configure the PEERING client\n", + "\n", + " a. Copy your certificates into the `client/certs` directory\n", + " \n", + " b. Ensure certificates do not have group or other read permissions (run `chmod 400` to fix)\n", + " \n", + " c. Add your PEERING-allocated IPv4 and IPv6 prefixes to `client/prefixes.txt` and `client/prefixes6.txt`, respectively, one prefix per line.\n", + " \n", + " > Your certificates and your assigned prefixes can be obtained from the [PEERING Dashboard](https://peering.ee.columbia.edu/account/) page after logging in.\n", + "\n", + "\n", + "4. Start using your PEERING client" + ] + }, + { + "cell_type": "markdown", + "id": "fc8074bd-e502-4214-90fb-f0bfeef839ab", + "metadata": {}, + "source": [ + "### Node Autoconfiguration\n", + "\n", + "The code cells below will attempt to automatically perform steps 1-3 above. If something breaks, debugging over SSH may be necessary. Please update the `PREFIXES4` and `PREFIXES6` variables in the first cell, and upload your certificate into `peering_files/client.crt` and your key to `peering_files/client.key`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3f78c22e", + "metadata": {}, + "outputs": [], + "source": [ + "PREFIXES4 = [\n", + " \"10.0.0.0/24\",\n", + " \"10.0.1.0/24\",\n", + "]\n", + "PREFIXES6 = [\n", + " \"dead:beef:0000::/48\",\n", + " \"dead:beef:0001::/48\",\n", + "]\n", + "for filename in [\"client.crt\", \"client.key\"]:\n", + " if not os.path.exists(f\"peering_files/{filename}\"):\n", + " assert False, \"Please upload your certificate and key into peering_files\"\n", + "NAT64_SCRIPT = pathlib.Path(\"../fablib_api/accessing_ipv4_services_from_ipv6_nodes/nat64.sh\").expanduser()\n", + "if not os.path.exists(NAT64_SCRIPT):\n", + " assert False, f\"NAT64 script not found at {NAT64_SCRIPT}\"\n", + "os.makedirs(\"peering_files/generated\", exist_ok=True)\n", + "with open(\"peering_files/generated/prefixes.txt\", \"w\", encoding=\"utf8\") as fd:\n", + " fd.write(\"\\n\".join(PREFIXES4))\n", + "with open(\"peering_files/generated/prefixes6.txt\", \"w\", encoding=\"utf8\") as fd:\n", + " fd.write(\"\\n\".join(PREFIXES6))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "436382f6-82c3-4899-bd6b-7bd64498196e", + "metadata": {}, + "outputs": [], + "source": [ + "home, _stderr = node.execute(\"echo -n $HOME\")\n", + "node = slice.get_node(name=node_name)\n", + "node.upload_file(NAT64_SCRIPT, f\"{home}/nat64.sh\")\n", + "node.upload_file('./script/node-autoconfig.sh', f'{home}/node-autoconfig.sh')\n", + "node.execute(f\"chmod 755 /home/ubuntu/nat64.sh {home}/node-autoconfig.sh\")\n", + "node.execute(f\"{home}/nat64.sh\")\n", + "node.execute(\"git clone --branch fabric https://github.com/PEERINGTestbed/client.git\")\n", + "node.upload_file('peering_files/client.crt', f'{home}/client/certs/client.crt')\n", + "node.upload_file('peering_files/client.key', f'{home}/client/certs/client.key')\n", + "node.upload_file('peering_files/generated/prefixes.txt', f'{home}/client/prefixes.txt')\n", + "node.upload_file('peering_files/generated/prefixes6.txt', f'{home}/client/prefixes6.txt')\n", + "node.execute(f\"{home}/node-autoconfig.sh\")" + ] + }, + { + "cell_type": "markdown", + "id": "f2c63b39-3c69-4b15-a9ca-f62259d276e3", + "metadata": {}, + "source": [ + "## Start using your PEERING client\n", + "\n", + "Once your client is configured you should be able to run experiments. Here are some steps to get you started:\n", + "\n", + "- Access the VM using SSH (see [FABRIC's documentation](../../configure.ipynb))\n", + "\n", + "- `cd` into the client directory\n", + "\n", + "- Use the `./peering` script to establish an OpenVPN connection to a PEERING router, e.g., `./peering openvpn up fabwash`\n", + "\n", + "- Use the `./peering` script to establish a BGP session for exchanging routes with the router, e.g., `./peering bgp start`.\n", + "\n", + " > Warning: When using IPv6, you need to edit the `client/bird6/bird6.conf` file and set a valid router ID (search for the line starting with `router id`) and use a unique IP address (e.g., one allocated to your experiment)\n", + "\n", + "Read the [PEERING client's README file](https://github.com/PEERINGTestbed/client/blob/master/README.md) and the output of `./peering prefix` to find out how to make and control announcements. For example, to announce your prefix out of all PEERING routers you are connected to, use `./peering prefix announce `.\n", + "\n", + "You can check that your prefix is propagating by using Looking Glass servers from multiple providers:\n", + "\n", + "- [Spring](https://www.sprint.net/tools/looking-glass)\n", + "- [NTT](https://www.gin.ntt.net/looking-glass-landing/)\n", + "- [Level3/Lumen](https://lookingglass.centurylink.com/)\n", + "\n", + "> Troubleshooting: If the prefix does not seem to be propagating, check that the OpenVPN tunnel is up, that the BGP session is established, and that your client is exporting the prefix to the desired router:\n", + ">\n", + "> ```bash\n", + "> ./peering openvpn status\n", + "> ./peering bgp status\n", + "> ./peering bgp adv \n", + "> ```\n", + "\n", + "To shut everything up, stop the BGP server and OpenVPN tunnels:\n", + "\n", + "```bash\n", + "./peering bgp stop\n", + "./peering openvpn down all\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "25e0a785-6972-498f-b1e5-2a0675a7a70b", + "metadata": {}, + "source": [ + "## PEERING Deployment on Fabric\n", + "\n", + "These are the currently-deploymed PEERING routers on FABRIC. The provided hostnames resolve to the IPv6 FABnet6 addresses your client's OpenVPN uses to connect to the router from within FABRIC. The FABnet6 addresses are not reachable from outside FABRIC by default. These hostnames do not have IPv4 addresses; use `dig AAAA ` to resolve them.\n", + "\n", + "* Washington, DC: `fabwash.atlas.peering.ee.columbia.edu`\n", + "* StarLight, IN: `fabstar.atlas.peering.ee.columbia.edu`" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/fabric_examples/peering_client/script/node-autoconfig.sh b/fabric_examples/peering_client/script/node-autoconfig.sh new file mode 100644 index 00000000..e69f12c0 --- /dev/null +++ b/fabric_examples/peering_client/script/node-autoconfig.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -eu + +sudo apt-get -y update +sudo apt-get -y install --no-install-recommends openvpn bird socat psmisc ipcalc +sudo systemctl disable bird bird6 openvpn +sudo systemctl stop bird bird6 openvpn + +chmod 400 /home/ubuntu/client/certs/client.* diff --git a/start_here.ipynb b/start_here.ipynb index 90f315a8..0856bafb 100644 --- a/start_here.ipynb +++ b/start_here.ipynb @@ -84,6 +84,9 @@ " - [Post Boot Tasks with Templates](./fabric_examples/fablib_api/post_boot_task_templates/post_boot_task_templates.ipynb): Define templated post boot tasks to upload files and execute commands.\n", "- Docker\n", " - [Deploy Docker Containers](./fabric_examples/fablib_api/docker_containers/docker_containers.ipynb): Docker\n", + "\n", + "## PEERING Client on FABRIC\n", + "- [Peering Client](./fabric_examples/peering_client/peering-client-setup.ipynb): setting up a PEERING client on FABRIC\n", " \n", "## Complex Recipes\n", "\n",