diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml new file mode 100644 index 00000000..a69e1d33 --- /dev/null +++ b/.github/workflows/checks.yml @@ -0,0 +1,35 @@ +# Run some code checks with GitHub Actions. + +name: Code checks + +on: + push: + branches: + - "main" + pull_request: + pull_request_target: + +permissions: + contents: read + # Allow 1Password/check-signed-commits-action to leave comments on + # pull requests. + pull-requests: write + +jobs: + + checks: + runs-on: ubuntu-latest + steps: + - name: Check out sources + uses: actions/checkout@v3 + with: + fetch-depth: 0 # since we need to diff against origin/main. + + # https://github.com/marketplace/actions/check-signed-commits-in-pr + # runs on pull_request_target and pull_request events, but + # pull_request_target is preferred because of the ability to + # leave comments on external PRs created from forks. + - name: Check that commits are signed + if: github.event_name == 'pull_request' || github.event_name == 'pull_request_target' + uses: 1Password/check-signed-commits-action@v1 + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..878bc93c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,20 @@ +# Contributing to Jupyter Examples + +We invite contributions from the community to enrich and broaden the collection of Jupyter Notebook examples. If you have an example or tutorial you'd like to contribute, kindly follow these steps: + +- Fork the repository. +- Create a new branch dedicated to your feature or fix. +- Develop and thoroughly test your changes locally. +- Commit your changes with clear and descriptive commit messages. Please ensure to use signed commits. +- Push your changes to your fork. +- Open a pull request against the main branch of the [repository]. + +- Please make sure that your contributions align with the overarching goals and guidelines of the Fabric Testbed project. + +## Submitting Issues + +If you wish to propose new examples or report any issues, please submit an issue via GitHub. If you encounter a bug, kindly describe it in sufficient detail for us to swiftly reproduce the issue. + +## Submitting Pull Requests + +Contributions to Jupyter Examples are facilitated through GitHub Pull Requests. The FABRIC project adheres to a policy of requiring GPG-signed commits. Commits on the PR branch will be scrutinized to ensure they are signed. Signed commits are compulsory for a PR to be reviewed, approved, and merged. \ No newline at end of file diff --git a/Readme.md b/Readme.md index ac69a8ec..bb593176 100644 --- a/Readme.md +++ b/Readme.md @@ -1,7 +1,47 @@ -# Fabric Jupyter Hub Notebooks -This repository contains Jupyter Notebooks for Fabric Testbed. These notebooks are available to the user when they logon to FABRIC Testbed Jupyter Hub. +# Jupyter Examples for Fabric Testbed +Welcome to the Fabric Testbed Jupyter Examples repository! This repository contains a collection of Jupyter Notebook +examples demonstrating various usage scenarios and features of the Fabric Testbed. -## Basic Notebooks -Following basic notebooks are available: -- tokens : Python notebook depicting how to generate FABRIC Tokens which are a MUST for any experiment on FABRIC Testbed -- orchestrator : Python notebook depicting basic Orchestrator APIS +## About Fabric Testbed +Fabric Testbed is a platform for building and managing virtualized network environments for research, +education, and experimentation. It provides users with access to programmable network resources, +allowing them to create, customize, and experiment with network topologies and configurations. +More details about the testbed can be found [here](https://portal.fabric-testbed.net/) + +## What's Included +This repository includes a variety of Jupyter Notebook examples covering different aspects of the Fabric Testbed, +including: + +- Topology Creation: Examples demonstrating how to create custom network topologies using Fabric Testbed APIs. +- Configuration Management: Examples showcasing how to configure network devices and services within the Fabric Testbed environment. +- Experimentation and Analysis: Examples illustrating how to conduct experiments and analyze network performance and behavior using Fabric Testbed resources. + +More details available in the [Start Here Notebook](./start_here.ipynb) + +## Getting Started + +The easiest way to create experiments on FABRIC is using JupyterHub. +You can create your private JupyterHub environment by logging into the [FABRIC portal](https://portal.fabric-testbed.net/). + +Alternatively you can set up your local environment by following these steps: + +- Clone the Repository: Clone this repository to your local machine using Git: +``` +git clone https://github.com/fabric-testbed/jupyter-examples.git +``` +- Install Dependencies: Install the necessary dependencies to run the Jupyter Notebooks as described [here](https://learn.fabric-testbed.net/knowledge-base/install-the-python-api/). Typically, this involves installing Python and Jupyter Notebook. + +- Explore the Examples: Open the Jupyter Notebooks (*.ipynb files) in your preferred Jupyter environment and explore the examples provided. Each notebook includes detailed instructions and explanations to guide you through the usage scenarios. + +- Experiment and Customize: Experiment with the provided examples, modify them to suit your needs, and explore the capabilities of the Fabric Testbed platform. + +## Contributing +We welcome contributions from the community to enhance and expand the collection of Jupyter Notebook examples. +Please follow the guidelines for contributing as described [here](./CONTRIBUTING.md). + +## License +This repository is licensed under the MIT License. + +## Contact +If you have any questions, feedback, or issues related to the examples in this repository, +please feel free to open an issue or ask questions [here](https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/). diff --git a/configure.ipynb b/configure.ipynb deleted file mode 100644 index 13635587..00000000 --- a/configure.ipynb +++ /dev/null @@ -1,328 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "tags": [] - }, - "source": [ - "# Configure your Jupyter Environment\n", - "\n", - "The following notebook can be used to configure your Jupyter environment. The product of running this notebook are the following:\n", - "\n", - "- `requirements.txt`: Used to upgrade/downgrade your environment with the FABLIB version used by these example notebooks.\n", - "- `fabric_rc`: File used to configure a FABlib application. \n", - "- `ssh_config`: File used to ssh from from a terminal to FABRIC VM by jumping through the FABRIC bastion host. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "tags": [] - }, - "source": [ - "## Copy your Bastion Key to your JupyterLab Environment\n", - "\n", - "In order to minimize security incidents on FABRIC, access to VMs and other resources administered by users is controlled using a bastion host. You will need to set up an ssh keypair that will be used to jump through the bastion host to your VMs and other resources. This keypair is unique to you and is only used to set up ssh proxy connections through the bastion host to your FABRIC resources. \n", - "\n", - "The following notebook walks through creating and submitting a bastion keypair necessary to use FABRIC. More information about how to access your experiment through the bastion host can be found [here](https://learn.fabric-testbed.net/knowledge-base/logging-into-fabric-vms/).\n", - "\n", - "If you have not created a bastion host keypair, create a new one in the [Experiments](https://portal.fabric-testbed.net/experiments#sshKeys) section of the FABRIC Portal.\n", - "\n", - "Copy your bastion private key to your Jupyter container by dragging/droppping your key file from your local machine to the file browser in your Juypyter browser window. __Note the name of your key and location where you put it, it is best to place it under fabric_config/ folder at the top of the folder hierarchy.__" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "tags": [] - }, - "source": [ - "## Set Bastion Username and Project ID\n", - "\n", - "Edit the following cell by entering your FABRIC bastion username and the correct path to the copy of your bastion private key in your Jupyter container. \n", - "\n", - "- Your bastion user name can be found on your [user profile page](https://portal.fabric-testbed.net/user) in the FABRIC portal (click \"My SSH Keys\" and select the \"Bastion\" tab).\n", - "- The Project ID can be from any of your projects. The ID can be found in the 'Basic Info' tab for each of the [projects](https://portal.fabric-testbed.net/projects) in the FABRIC portal.\n", - "- The path to your bastion key was determined when you uploaded it to the Jupyter container. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# your bastion login can be found in your profile page. Remember to omit the < and > when pasting into this\n", - "export FABRIC_BASTION_USERNAME=''\n", - "# the ID of your chosen project. You can change this value any time you change the project. Remember to omit < and >\n", - "export FABRIC_PROJECT_ID=''\n", - "# check the file name under which you uploaded your bastion private key file into and change this variable from fabric_bastion_key_file_name to its actual name\n", - "export FABRIC_BASTION_PRIVATE_KEY_LOCATION=${HOME}/work/fabric_config/fabric_bastion_key_file_name\n", - "\n", - "# generally these variables do not require changing\n", - "export FABRIC_BASTION_SSH_CONFIG_FILE=${HOME}'/work/fabric_config/ssh_config'\n", - "export FABRIC_RC_FILE=${HOME}'/work/fabric_config/fabric_rc'" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Set other Tokens and Keys\n", - "\n", - "The the path to where you put your bastion key.The defaults below should work within the Jupyter environment hosted by FABRIC. Custom installations will require modifications to this configuration.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# this is where your FABRIC API token will be stored, leave unchanged\n", - "export FABRIC_TOKEN_FILE=${HOME}'/.tokens.json'\n", - "# if you want to use an alternative slice/sliver key you can change the name here, otherwise leave unchanged\n", - "export FABRIC_SLICE_PRIVATE_KEY_FILE=${HOME}/work/fabric_config/slice_key\n", - "export FABRIC_SLICE_PUBLIC_KEY_FILE=${FABRIC_SLICE_PRIVATE_KEY_FILE}.pub" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Optionally, create a new slice keypair. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#ssh-keygen -t rsa -b 3072 -f $FABRIC_SLICE_PRIVATE_KEY_FILE -q -N \"\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "\n", - "Set the Permissions on your bastion key and private slice key. Your private bastion key must not have it permissions too open." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "chmod 600 ${FABRIC_BASTION_PRIVATE_KEY_LOCATION}\n", - "chmod 600 ${FABRIC_SLICE_PRIVATE_KEY_FILE}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## FABLIB Version\n", - "\n", - "This package of example notebooks was created using the version of fabrictestbed-extensions (i.e. the FABLIB package) specified in the local `requirements.txt` file. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cat requirements.txt" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "These examples may work with your currently installed default FABLIB version. However, you may want to upgrade/downgrade the FABLIB version to the one used to develop these examples. The following cell will synchronize your current FABLIB version with these examples." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "pip install -r requirements.txt --user --quiet" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Your Jupyter container will often be updated with the current default FABLIB version. You can permanently override these updates by setting your desired version in\n", - "`~/work/fabric_config/requirements.txt`. If you would like to permanently configure your Jupyter environment to use the FABLIB version used for this set of examples, copy the local `requirnments.txt` file to `~/work/fabric_config/requirements.txt` using the following command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#cp requirements.txt ~/work/fabric_config/requirements.txt" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create the FABRIC Config File \n", - "\n", - "\n", - "The following cell creates the `fabric_rc` file using the values configured above. \n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cat < ${FABRIC_RC_FILE}\n", - "export FABRIC_CREDMGR_HOST=cm.fabric-testbed.net\n", - "export FABRIC_ORCHESTRATOR_HOST=orchestrator.fabric-testbed.net\n", - "\n", - "export FABRIC_PROJECT_ID=${FABRIC_PROJECT_ID}\n", - "export FABRIC_TOKEN_LOCATION=${FABRIC_TOKEN_FILE}\n", - "\n", - "export FABRIC_BASTION_HOST=bastion.fabric-testbed.net\n", - "export FABRIC_BASTION_USERNAME=${FABRIC_BASTION_USERNAME}\n", - "\n", - "export FABRIC_BASTION_KEY_LOCATION=${FABRIC_BASTION_PRIVATE_KEY_LOCATION}\n", - "#export FABRIC_BASTION_KEY_PASSPHRASE=\n", - "\n", - "export FABRIC_SLICE_PRIVATE_KEY_FILE=${FABRIC_SLICE_PRIVATE_KEY_FILE}\n", - "export FABRIC_SLICE_PUBLIC_KEY_FILE=${FABRIC_SLICE_PUBLIC_KEY_FILE} \n", - "#export FABRIC_SLICE_PRIVATE_KEY_PASSPHRASE=\n", - "\n", - "export FABRIC_LOG_FILE=${HOME}/fablib.log\n", - "export FABRIC_LOG_LEVEL=INFO \n", - "\n", - "export FABRIC_AVOID=''\n", - "\n", - "export FABRIC_SSH_COMMAND_LINE=\"ssh -i {{ _self_.private_ssh_key_file }} -F ${HOME}/work/fabric_config/ssh_config {{ _self_.username }}@{{ _self_.management_ip }}\"\n", - "EOF" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create the SSH Config File\n", - "\n", - "The following cell creates an SSH config file using the values configure above. The config file allows you to use ssh to jump through the bastion host from a terminal command line. After creating this file, you can ssh and pass the file with the `-F` option.\n", - "\n", - "Alternatively, you can directly edit your [ssh config](~/fabric_config/ssh_config) file.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cat < ${FABRIC_BASTION_SSH_CONFIG_FILE}\n", - "UserKnownHostsFile /dev/null\n", - "StrictHostKeyChecking no\n", - "ServerAliveInterval 120 \n", - "\n", - "Host bastion.fabric-testbed.net\n", - " User ${FABRIC_BASTION_USERNAME}\n", - " ForwardAgent yes\n", - " Hostname %h\n", - " IdentityFile ${FABRIC_BASTION_PRIVATE_KEY_LOCATION}\n", - " IdentitiesOnly yes\n", - "\n", - "Host * !bastion.fabric-testbed.net\n", - " ProxyJump ${FABRIC_BASTION_USERNAME}@bastion.fabric-testbed.net:22\n", - "EOF" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "tags": [] - }, - "source": [ - "## Create a Downloadable Package that Deploys SSH Tunnels" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "export FABRIC_CONFIG_DIR=${HOME}'/work/fabric_config'\n", - "export FABRIC_SSH_TUNNEL_PACKAGE_DIR=${FABRIC_CONFIG_DIR}'/fabric_ssh_tunnel_tools'\n", - "\n", - "#mkdir -p ${FABRIC_SSH_TUNNEL_PACKAGE_DIR} \n", - "cp ${FABRIC_SLICE_PRIVATE_KEY_FILE} ./fabric_ssh_tunnel_tools/slice_key\n", - "cp ${FABRIC_BASTION_PRIVATE_KEY_LOCATION} ./fabric_ssh_tunnel_tools/bastion_key\n", - "\n", - "cat < ./fabric_ssh_tunnel_tools/ssh_config\n", - "UserKnownHostsFile /dev/null\n", - "StrictHostKeyChecking no\n", - "ServerAliveInterval 120 \n", - "\n", - "Host bastion.fabric-testbed.net\n", - " User ${FABRIC_BASTION_USERNAME}\n", - " ForwardAgent yes\n", - " Hostname %h\n", - " IdentityFile bastion_key\n", - " IdentitiesOnly yes\n", - "\n", - "Host * !bastion.fabric-testbed.net\n", - " ProxyJump ${FABRIC_BASTION_USERNAME}@bastion.fabric-testbed.net:22\n", - "EOF\n", - "\n", - "chmod +x ./fabric_ssh_tunnel_tools/start_ssh_tunnel.sh\n", - "chmod +x ./fabric_ssh_tunnel_tools/stop_ssh_tunnel.sh\n", - "chmod 600 ./fabric_ssh_tunnel_tools/slice_key\n", - "chmod 600 ./fabric_ssh_tunnel_tools/bastion_key\n", - "\n", - "\n", - "#zip -r fabric_ssh_tunnel_tools.zip ./fabric_ssh_tunnel_tools\n", - "tar -zcvf fabric_ssh_tunnel_tools.tgz fabric_ssh_tunnel_tools\n", - "cp fabric_ssh_tunnel_tools.tgz fabric_ssh_tunnel_tools.zip ${FABRIC_CONFIG_DIR}/.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" - }, - "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/configure_and_validate.ipynb b/configure_and_validate.ipynb index bd79ddbf..5ac275f1 100644 --- a/configure_and_validate.ipynb +++ b/configure_and_validate.ipynb @@ -80,7 +80,7 @@ "metadata": {}, "outputs": [], "source": [ - "fablib.validate_config()" + "fablib.verify_and_configure()" ] }, { diff --git a/fabric_examples/peering_client/figs/peering-fabric.png b/fabric_examples/complex_recipes/networking/peering_client/figs/peering-fabric.png similarity index 100% rename from fabric_examples/peering_client/figs/peering-fabric.png rename to fabric_examples/complex_recipes/networking/peering_client/figs/peering-fabric.png diff --git a/fabric_examples/peering_client/peering-client-setup.ipynb b/fabric_examples/complex_recipes/networking/peering_client/peering-client-setup.ipynb similarity index 100% rename from fabric_examples/peering_client/peering-client-setup.ipynb rename to fabric_examples/complex_recipes/networking/peering_client/peering-client-setup.ipynb diff --git a/fabric_examples/peering_client/script/node-autoconfig.sh b/fabric_examples/complex_recipes/networking/peering_client/script/node-autoconfig.sh similarity index 100% rename from fabric_examples/peering_client/script/node-autoconfig.sh rename to fabric_examples/complex_recipes/networking/peering_client/script/node-autoconfig.sh diff --git a/start_here.ipynb b/start_here.ipynb index 0856bafb..79250d79 100644 --- a/start_here.ipynb +++ b/start_here.ipynb @@ -85,9 +85,6 @@ "- Docker\n", " - [Deploy Docker Containers](./fabric_examples/fablib_api/docker_containers/docker_containers.ipynb): Docker\n", "\n", - "## PEERING Client on FABRIC\n", - "- [Peering Client](./fabric_examples/peering_client/peering-client-setup.ipynb): setting up a PEERING client on FABRIC\n", - " \n", "## Complex Recipes\n", "\n", "The following notebooks are examples of creating and configuring complex FABRIC topologies. They are intended to serve as examples of techniques that might help you build your own experiments. \n", @@ -106,6 +103,7 @@ " - [Running One-Way Latency Measurements with InfluxDB]()\n", " - Programmable Networking\n", " - [DPDK and Pktgen with Smart NICs](./fabric_examples/complex_recipes/networking/programmable_networking/dpdk_smart_nic/dpdk_packet_gen_smart_nic.ipynb): Program your smart nics with DPDK.\n", + " - [Peering Client](./fabric_examples/complex_recipes/networking/peering_client/peering-client-setup.ipynb): setting up a PEERING client on FABRIC\n", "- Component Benchmarking\n", " - Storage\n", " - [Benchmarking FABRIC Storage: Local disk and NVMe](./fabric_examples/fablib_api/bbenchmarking_storage/enchmarking_storage.ipynb): Create and benchmark a node with local disk and NVMe device.\n",