From 1ba7863adcadf151a980750dced00d17ffec8a44 Mon Sep 17 00:00:00 2001 From: Bobbins228 Date: Fri, 11 Oct 2024 14:14:50 +0100 Subject: [PATCH 1/3] docs: convert auto-generated documentation to from pydoc to sphinx --- .github/workflows/release.yaml | 5 +- docs/authentication.md | 40 - docs/cluster-configuration.md | 46 - docs/detailed-documentation/cluster/auth.html | 742 ------ .../cluster/awload.html | 328 --- .../cluster/cluster.html | 2174 ----------------- .../cluster/config.html | 764 ------ .../detailed-documentation/cluster/index.html | 129 - .../detailed-documentation/cluster/model.html | 531 ---- .../cluster/widgets.html | 758 ------ docs/detailed-documentation/index.html | 106 - docs/detailed-documentation/job/index.html | 72 - docs/detailed-documentation/job/ray_jobs.html | 585 ----- docs/detailed-documentation/utils/demos.html | 138 -- .../utils/generate_cert.html | 451 ---- .../utils/generate_yaml.html | 951 ------- docs/detailed-documentation/utils/index.html | 88 - .../utils/kube_api_helpers.html | 112 - .../utils/pretty_print.html | 491 ---- docs/e2e.md | 133 - docs/generate-documentation.md | 14 + docs/s3-compatible-storage.md | 61 - docs/setup-kueue.md | 66 - docs/sphinx/Makefile | 20 + docs/sphinx/conf.py | 38 + docs/sphinx/index.rst | 32 + docs/sphinx/make.bat | 35 + docs/sphinx/user-docs/authentication.rst | 66 + .../user-docs/cluster-configuration.rst | 72 + docs/sphinx/user-docs/e2e.rst | 210 ++ .../user-docs/s3-compatible-storage.rst | 86 + docs/sphinx/user-docs/setup-kueue.rst | 109 + poetry.lock | 261 +- pyproject.toml | 3 +- 34 files changed, 896 insertions(+), 8821 deletions(-) delete mode 100644 docs/authentication.md delete mode 100644 docs/cluster-configuration.md delete mode 100644 docs/detailed-documentation/cluster/auth.html delete mode 100644 docs/detailed-documentation/cluster/awload.html delete mode 100644 docs/detailed-documentation/cluster/cluster.html delete mode 100644 docs/detailed-documentation/cluster/config.html delete mode 100644 docs/detailed-documentation/cluster/index.html delete mode 100644 docs/detailed-documentation/cluster/model.html delete mode 100644 docs/detailed-documentation/cluster/widgets.html delete mode 100644 docs/detailed-documentation/index.html delete mode 100644 docs/detailed-documentation/job/index.html delete mode 100644 docs/detailed-documentation/job/ray_jobs.html delete mode 100644 docs/detailed-documentation/utils/demos.html delete mode 100644 docs/detailed-documentation/utils/generate_cert.html delete mode 100644 docs/detailed-documentation/utils/generate_yaml.html delete mode 100644 docs/detailed-documentation/utils/index.html delete mode 100644 docs/detailed-documentation/utils/kube_api_helpers.html delete mode 100644 docs/detailed-documentation/utils/pretty_print.html delete mode 100644 docs/e2e.md create mode 100644 docs/generate-documentation.md delete mode 100644 docs/s3-compatible-storage.md delete mode 100644 docs/setup-kueue.md create mode 100644 docs/sphinx/Makefile create mode 100644 docs/sphinx/conf.py create mode 100644 docs/sphinx/index.rst create mode 100644 docs/sphinx/make.bat create mode 100644 docs/sphinx/user-docs/authentication.rst create mode 100644 docs/sphinx/user-docs/cluster-configuration.rst create mode 100644 docs/sphinx/user-docs/e2e.rst create mode 100644 docs/sphinx/user-docs/s3-compatible-storage.rst create mode 100644 docs/sphinx/user-docs/setup-kueue.rst diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 0b9d6bbc4..6e56a3f86 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -51,7 +51,10 @@ jobs: - name: Run poetry install run: poetry install --with docs - name: Create new documentation - run: poetry run pdoc --html -o docs/detailed-documentation src/codeflare_sdk && pushd docs/detailed-documentation && rm -rf cluster job utils && mv codeflare_sdk/* . && rm -rf codeflare_sdk && popd && find docs/detailed-documentation -type f -name "*.html" -exec bash -c "echo '' >> {}" \; + run: | + sphinx-apidoc -o docs/sphinx src/codeflare_sdk "**/*test_*" --force + make clean -C docs/sphinx + make html -C docs/sphinx - name: Copy demo notebooks into SDK package run: cp -r demo-notebooks src/codeflare_sdk/demo-notebooks - name: Run poetry build diff --git a/docs/authentication.md b/docs/authentication.md deleted file mode 100644 index bb27f1716..000000000 --- a/docs/authentication.md +++ /dev/null @@ -1,40 +0,0 @@ -# Authentication via the CodeFlare SDK -Currently there are four ways of authenticating to your cluster via the SDK.
-Authenticating with your cluster allows you to perform actions such as creating Ray Clusters and Job Submission. - -## Method 1 Token Authentication -This is how a typical user would authenticate to their cluster using `TokenAuthentication`. -``` -from codeflare_sdk import TokenAuthentication - -auth = TokenAuthentication( - token = "XXXXX", - server = "XXXXX", - skip_tls=False, - # ca_cert_path="/path/to/cert" -) -auth.login() -# log out with auth.logout() -``` -Setting `skip_tls=True` allows interaction with an HTTPS server bypassing the server certificate checks although this is not secure.
-You can pass a custom certificate to `TokenAuthentication` by using `ca_cert_path="/path/to/cert"` when authenticating provided `skip_tls=False`. Alternatively you can set the environment variable `CF_SDK_CA_CERT_PATH` to the path of your custom certificate. - -## Method 2 Kubernetes Config File Authentication (Default location) -If a user has authenticated to their cluster by alternate means e.g. run a login command like `oc login --token= --server=` their kubernetes config file should have updated.
-If the user has not specifically authenticated through the SDK by other means such as `TokenAuthentication` then the SDK will try to use their default Kubernetes config file located at `"/HOME/.kube/config"`. - -## Method 3 Specifying a Kubernetes Config File -A user can specify a config file via a different authentication class `KubeConfigFileAuthentication` for authenticating with the SDK.
-This is what loading a custom config file would typically look like. -``` -from codeflare_sdk import KubeConfigFileAuthentication - -auth = KubeConfigFileAuthentication( - kube_config_path="/path/to/config", -) -auth.load_kube_config() -# log out with auth.logout() -``` - -## Method 4 In-Cluster Authentication -If a user does not authenticate by any of the means detailed above and does not have a config file at `"/HOME/.kube/config"` the SDK will try to authenticate with the in-cluster configuration file. diff --git a/docs/cluster-configuration.md b/docs/cluster-configuration.md deleted file mode 100644 index 97068b490..000000000 --- a/docs/cluster-configuration.md +++ /dev/null @@ -1,46 +0,0 @@ -# Ray Cluster Configuration - -To create Ray Clusters using the CodeFlare SDK a cluster configuration needs to be created first.
-This is what a typical cluster configuration would look like; Note: The values for CPU and Memory are at the minimum requirements for creating the Ray Cluster. - -```python -from codeflare_sdk import Cluster, ClusterConfiguration - -cluster = Cluster(ClusterConfiguration( - name='ray-example', # Mandatory Field - namespace='default', # Default None - head_cpu_requests=1, # Default 2 - head_cpu_limits=1, # Default 2 - head_memory_requests=1, # Default 8 - head_memory_limits=1, # Default 8 - head_extended_resource_requests={'nvidia.com/gpu':0}, # Default 0 - worker_extended_resource_requests={'nvidia.com/gpu':0}, # Default 0 - num_workers=1, # Default 1 - worker_cpu_requests=1, # Default 1 - worker_cpu_limits=1, # Default 1 - worker_memory_requests=2, # Default 2 - worker_memory_limits=2, # Default 2 - # image="", # Optional Field - machine_types=["m5.xlarge", "g4dn.xlarge"], - labels={"exampleLabel": "example", "secondLabel": "example"}, -)) -``` -Note: 'quay.io/modh/ray:2.35.0-py39-cu121' is the default image used by the CodeFlare SDK for creating a RayCluster resource. If you have your own Ray image which suits your purposes, specify it in image field to override the default image. If you are using ROCm compatible GPUs you can use 'quay.io/modh/ray:2.35.0-py39-rocm61'. You can also find documentation on building a custom image [here](https://github.com/opendatahub-io/distributed-workloads/tree/main/images/runtime/examples). - -The `labels={"exampleLabel": "example"}` parameter can be used to apply additional labels to the RayCluster resource. - -After creating their `cluster`, a user can call `cluster.up()` and `cluster.down()` to respectively create or remove the Ray Cluster. - - -## Deprecating Parameters -The following parameters of the `ClusterConfiguration` are being deprecated in release `v0.22.0`. -| Deprecated Parameter | Replaced By | -| :--------- | :-------- | -| `head_cpus` | `head_cpu_requests`, `head_cpu_limits` | -| `head_memory` | `head_memory_requests`, `head_memory_limits` | -| `min_cpus` | `worker_cpu_requests` | -| `max_cpus` | `worker_cpu_limits` | -| `min_memory` | `worker_memory_requests` | -| `max_memory` | `worker_memory_limits` | -| `head_gpus` | `head_extended_resource_requests` | -| `num_gpus` | `worker_extended_resource_requests` | diff --git a/docs/detailed-documentation/cluster/auth.html b/docs/detailed-documentation/cluster/auth.html deleted file mode 100644 index d35b46093..000000000 --- a/docs/detailed-documentation/cluster/auth.html +++ /dev/null @@ -1,742 +0,0 @@ - - - - - - -codeflare_sdk.cluster.auth API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster.auth

-
-
-

The auth sub-module contains the definitions for the Authentication objects, which represent -the methods by which a user can authenticate to their cluster(s). The abstract class, Authentication, -contains two required methods login() and logout(). Users can use one of the existing concrete classes to -authenticate to their cluster or add their own custom concrete classes here.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The auth sub-module contains the definitions for the Authentication objects, which represent
-the methods by which a user can authenticate to their cluster(s). The abstract class, `Authentication`,
-contains two required methods `login()` and `logout()`. Users can use one of the existing concrete classes to
-authenticate to their cluster or add their own custom concrete classes here.
-"""
-
-import abc
-from kubernetes import client, config
-import os
-import urllib3
-from ..utils.kube_api_helpers import _kube_api_error_handling
-
-from typing import Optional
-
-global api_client
-api_client = None
-global config_path
-config_path = None
-
-WORKBENCH_CA_CERT_PATH = "/etc/pki/tls/custom-certs/ca-bundle.crt"
-
-
-class Authentication(metaclass=abc.ABCMeta):
-    """
-    An abstract class that defines the necessary methods for authenticating to a remote environment.
-    Specifically, this class defines the need for a `login()` and a `logout()` function.
-    """
-
-    def login(self):
-        """
-        Method for logging in to a remote cluster.
-        """
-        pass
-
-    def logout(self):
-        """
-        Method for logging out of the remote cluster.
-        """
-        pass
-
-
-class KubeConfiguration(metaclass=abc.ABCMeta):
-    """
-    An abstract class that defines the method for loading a user defined config file using the `load_kube_config()` function
-    """
-
-    def load_kube_config(self):
-        """
-        Method for setting your Kubernetes configuration to a certain file
-        """
-        pass
-
-    def logout(self):
-        """
-        Method for logging out of the remote cluster
-        """
-        pass
-
-
-class TokenAuthentication(Authentication):
-    """
-    `TokenAuthentication` is a subclass of `Authentication`. It can be used to authenticate to a Kubernetes
-    cluster when the user has an API token and the API server address.
-    """
-
-    def __init__(
-        self,
-        token: str,
-        server: str,
-        skip_tls: bool = False,
-        ca_cert_path: str = None,
-    ):
-        """
-        Initialize a TokenAuthentication object that requires a value for `token`, the API Token
-        and `server`, the API server address for authenticating to a Kubernetes cluster.
-        """
-
-        self.token = token
-        self.server = server
-        self.skip_tls = skip_tls
-        self.ca_cert_path = _gen_ca_cert_path(ca_cert_path)
-
-    def login(self) -> str:
-        """
-        This function is used to log in to a Kubernetes cluster using the user's API token and API server address.
-        Depending on the cluster, a user can choose to login in with `--insecure-skip-tls-verify` by setting `skip_tls`
-        to `True` or `--certificate-authority` by setting `skip_tls` to False and providing a path to a ca bundle with `ca_cert_path`.
-        """
-        global config_path
-        global api_client
-        try:
-            configuration = client.Configuration()
-            configuration.api_key_prefix["authorization"] = "Bearer"
-            configuration.host = self.server
-            configuration.api_key["authorization"] = self.token
-
-            api_client = client.ApiClient(configuration)
-            if not self.skip_tls:
-                _client_with_cert(api_client, self.ca_cert_path)
-            else:
-                urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
-                print("Insecure request warnings have been disabled")
-                configuration.verify_ssl = False
-
-            client.AuthenticationApi(api_client).get_api_group()
-            config_path = None
-            return "Logged into %s" % self.server
-        except client.ApiException as e:
-            _kube_api_error_handling(e)
-
-    def logout(self) -> str:
-        """
-        This function is used to logout of a Kubernetes cluster.
-        """
-        global config_path
-        config_path = None
-        global api_client
-        api_client = None
-        return "Successfully logged out of %s" % self.server
-
-
-class KubeConfigFileAuthentication(KubeConfiguration):
-    """
-    A class that defines the necessary methods for passing a user's own Kubernetes config file.
-    Specifically this class defines the `load_kube_config()` and `config_check()` functions.
-    """
-
-    def __init__(self, kube_config_path: str = None):
-        self.kube_config_path = kube_config_path
-
-    def load_kube_config(self):
-        """
-        Function for loading a user's own predefined Kubernetes config file.
-        """
-        global config_path
-        global api_client
-        try:
-            if self.kube_config_path == None:
-                return "Please specify a config file path"
-            config_path = self.kube_config_path
-            api_client = None
-            config.load_kube_config(config_path)
-            response = "Loaded user config file at path %s" % self.kube_config_path
-        except config.ConfigException:  # pragma: no cover
-            config_path = None
-            raise Exception("Please specify a config file path")
-        return response
-
-
-def config_check() -> str:
-    """
-    Function for loading the config file at the default config location ~/.kube/config if the user has not
-    specified their own config file or has logged in with their token and server.
-    """
-    global config_path
-    global api_client
-    home_directory = os.path.expanduser("~")
-    if config_path == None and api_client == None:
-        if os.path.isfile("%s/.kube/config" % home_directory):
-            try:
-                config.load_kube_config()
-            except Exception as e:  # pragma: no cover
-                _kube_api_error_handling(e)
-        elif "KUBERNETES_PORT" in os.environ:
-            try:
-                config.load_incluster_config()
-            except Exception as e:  # pragma: no cover
-                _kube_api_error_handling(e)
-        else:
-            raise PermissionError(
-                "Action not permitted, have you put in correct/up-to-date auth credentials?"
-            )
-
-    if config_path != None and api_client == None:
-        return config_path
-
-
-def _client_with_cert(client: client.ApiClient, ca_cert_path: Optional[str] = None):
-    client.configuration.verify_ssl = True
-    cert_path = _gen_ca_cert_path(ca_cert_path)
-    if cert_path is None:
-        client.configuration.ssl_ca_cert = None
-    elif os.path.isfile(cert_path):
-        client.configuration.ssl_ca_cert = cert_path
-    else:
-        raise FileNotFoundError(f"Certificate file not found at {cert_path}")
-
-
-def _gen_ca_cert_path(ca_cert_path: Optional[str]):
-    """Gets the path to the default CA certificate file either through env config or default path"""
-    if ca_cert_path is not None:
-        return ca_cert_path
-    elif "CF_SDK_CA_CERT_PATH" in os.environ:
-        return os.environ.get("CF_SDK_CA_CERT_PATH")
-    elif os.path.exists(WORKBENCH_CA_CERT_PATH):
-        return WORKBENCH_CA_CERT_PATH
-    else:
-        return None
-
-
-def get_api_client() -> client.ApiClient:
-    "This function should load the api client with defaults"
-    if api_client != None:
-        return api_client
-    to_return = client.ApiClient()
-    _client_with_cert(to_return)
-    return to_return
-
-
-
-
-
-
-
-

Functions

-
-
-def config_check() ‑> str -
-
-

Function for loading the config file at the default config location ~/.kube/config if the user has not -specified their own config file or has logged in with their token and server.

-
- -Expand source code - -
def config_check() -> str:
-    """
-    Function for loading the config file at the default config location ~/.kube/config if the user has not
-    specified their own config file or has logged in with their token and server.
-    """
-    global config_path
-    global api_client
-    home_directory = os.path.expanduser("~")
-    if config_path == None and api_client == None:
-        if os.path.isfile("%s/.kube/config" % home_directory):
-            try:
-                config.load_kube_config()
-            except Exception as e:  # pragma: no cover
-                _kube_api_error_handling(e)
-        elif "KUBERNETES_PORT" in os.environ:
-            try:
-                config.load_incluster_config()
-            except Exception as e:  # pragma: no cover
-                _kube_api_error_handling(e)
-        else:
-            raise PermissionError(
-                "Action not permitted, have you put in correct/up-to-date auth credentials?"
-            )
-
-    if config_path != None and api_client == None:
-        return config_path
-
-
-
-def get_api_client() ‑> kubernetes.client.api_client.ApiClient -
-
-

This function should load the api client with defaults

-
- -Expand source code - -
def get_api_client() -> client.ApiClient:
-    "This function should load the api client with defaults"
-    if api_client != None:
-        return api_client
-    to_return = client.ApiClient()
-    _client_with_cert(to_return)
-    return to_return
-
-
-
-
-
-

Classes

-
-
-class Authentication -
-
-

An abstract class that defines the necessary methods for authenticating to a remote environment. -Specifically, this class defines the need for a login() and a logout() function.

-
- -Expand source code - -
class Authentication(metaclass=abc.ABCMeta):
-    """
-    An abstract class that defines the necessary methods for authenticating to a remote environment.
-    Specifically, this class defines the need for a `login()` and a `logout()` function.
-    """
-
-    def login(self):
-        """
-        Method for logging in to a remote cluster.
-        """
-        pass
-
-    def logout(self):
-        """
-        Method for logging out of the remote cluster.
-        """
-        pass
-
-

Subclasses

- -

Methods

-
-
-def login(self) -
-
-

Method for logging in to a remote cluster.

-
- -Expand source code - -
def login(self):
-    """
-    Method for logging in to a remote cluster.
-    """
-    pass
-
-
-
-def logout(self) -
-
-

Method for logging out of the remote cluster.

-
- -Expand source code - -
def logout(self):
-    """
-    Method for logging out of the remote cluster.
-    """
-    pass
-
-
-
-
-
-class KubeConfigFileAuthentication -(kube_config_path: str = None) -
-
-

A class that defines the necessary methods for passing a user's own Kubernetes config file. -Specifically this class defines the load_kube_config() and config_check() functions.

-
- -Expand source code - -
class KubeConfigFileAuthentication(KubeConfiguration):
-    """
-    A class that defines the necessary methods for passing a user's own Kubernetes config file.
-    Specifically this class defines the `load_kube_config()` and `config_check()` functions.
-    """
-
-    def __init__(self, kube_config_path: str = None):
-        self.kube_config_path = kube_config_path
-
-    def load_kube_config(self):
-        """
-        Function for loading a user's own predefined Kubernetes config file.
-        """
-        global config_path
-        global api_client
-        try:
-            if self.kube_config_path == None:
-                return "Please specify a config file path"
-            config_path = self.kube_config_path
-            api_client = None
-            config.load_kube_config(config_path)
-            response = "Loaded user config file at path %s" % self.kube_config_path
-        except config.ConfigException:  # pragma: no cover
-            config_path = None
-            raise Exception("Please specify a config file path")
-        return response
-
-

Ancestors

- -

Methods

-
-
-def load_kube_config(self) -
-
-

Function for loading a user's own predefined Kubernetes config file.

-
- -Expand source code - -
def load_kube_config(self):
-    """
-    Function for loading a user's own predefined Kubernetes config file.
-    """
-    global config_path
-    global api_client
-    try:
-        if self.kube_config_path == None:
-            return "Please specify a config file path"
-        config_path = self.kube_config_path
-        api_client = None
-        config.load_kube_config(config_path)
-        response = "Loaded user config file at path %s" % self.kube_config_path
-    except config.ConfigException:  # pragma: no cover
-        config_path = None
-        raise Exception("Please specify a config file path")
-    return response
-
-
-
-

Inherited members

- -
-
-class KubeConfiguration -
-
-

An abstract class that defines the method for loading a user defined config file using the load_kube_config() function

-
- -Expand source code - -
class KubeConfiguration(metaclass=abc.ABCMeta):
-    """
-    An abstract class that defines the method for loading a user defined config file using the `load_kube_config()` function
-    """
-
-    def load_kube_config(self):
-        """
-        Method for setting your Kubernetes configuration to a certain file
-        """
-        pass
-
-    def logout(self):
-        """
-        Method for logging out of the remote cluster
-        """
-        pass
-
-

Subclasses

- -

Methods

-
-
-def load_kube_config(self) -
-
-

Method for setting your Kubernetes configuration to a certain file

-
- -Expand source code - -
def load_kube_config(self):
-    """
-    Method for setting your Kubernetes configuration to a certain file
-    """
-    pass
-
-
-
-def logout(self) -
-
-

Method for logging out of the remote cluster

-
- -Expand source code - -
def logout(self):
-    """
-    Method for logging out of the remote cluster
-    """
-    pass
-
-
-
-
-
-class TokenAuthentication -(token: str, server: str, skip_tls: bool = False, ca_cert_path: str = None) -
-
-

TokenAuthentication is a subclass of Authentication. It can be used to authenticate to a Kubernetes -cluster when the user has an API token and the API server address.

-

Initialize a TokenAuthentication object that requires a value for token, the API Token -and server, the API server address for authenticating to a Kubernetes cluster.

-
- -Expand source code - -
class TokenAuthentication(Authentication):
-    """
-    `TokenAuthentication` is a subclass of `Authentication`. It can be used to authenticate to a Kubernetes
-    cluster when the user has an API token and the API server address.
-    """
-
-    def __init__(
-        self,
-        token: str,
-        server: str,
-        skip_tls: bool = False,
-        ca_cert_path: str = None,
-    ):
-        """
-        Initialize a TokenAuthentication object that requires a value for `token`, the API Token
-        and `server`, the API server address for authenticating to a Kubernetes cluster.
-        """
-
-        self.token = token
-        self.server = server
-        self.skip_tls = skip_tls
-        self.ca_cert_path = _gen_ca_cert_path(ca_cert_path)
-
-    def login(self) -> str:
-        """
-        This function is used to log in to a Kubernetes cluster using the user's API token and API server address.
-        Depending on the cluster, a user can choose to login in with `--insecure-skip-tls-verify` by setting `skip_tls`
-        to `True` or `--certificate-authority` by setting `skip_tls` to False and providing a path to a ca bundle with `ca_cert_path`.
-        """
-        global config_path
-        global api_client
-        try:
-            configuration = client.Configuration()
-            configuration.api_key_prefix["authorization"] = "Bearer"
-            configuration.host = self.server
-            configuration.api_key["authorization"] = self.token
-
-            api_client = client.ApiClient(configuration)
-            if not self.skip_tls:
-                _client_with_cert(api_client, self.ca_cert_path)
-            else:
-                urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
-                print("Insecure request warnings have been disabled")
-                configuration.verify_ssl = False
-
-            client.AuthenticationApi(api_client).get_api_group()
-            config_path = None
-            return "Logged into %s" % self.server
-        except client.ApiException as e:
-            _kube_api_error_handling(e)
-
-    def logout(self) -> str:
-        """
-        This function is used to logout of a Kubernetes cluster.
-        """
-        global config_path
-        config_path = None
-        global api_client
-        api_client = None
-        return "Successfully logged out of %s" % self.server
-
-

Ancestors

- -

Methods

-
-
-def login(self) ‑> str -
-
-

This function is used to log in to a Kubernetes cluster using the user's API token and API server address. -Depending on the cluster, a user can choose to login in with --insecure-skip-tls-verify by setting skip_tls -to True or --certificate-authority by setting skip_tls to False and providing a path to a ca bundle with ca_cert_path.

-
- -Expand source code - -
def login(self) -> str:
-    """
-    This function is used to log in to a Kubernetes cluster using the user's API token and API server address.
-    Depending on the cluster, a user can choose to login in with `--insecure-skip-tls-verify` by setting `skip_tls`
-    to `True` or `--certificate-authority` by setting `skip_tls` to False and providing a path to a ca bundle with `ca_cert_path`.
-    """
-    global config_path
-    global api_client
-    try:
-        configuration = client.Configuration()
-        configuration.api_key_prefix["authorization"] = "Bearer"
-        configuration.host = self.server
-        configuration.api_key["authorization"] = self.token
-
-        api_client = client.ApiClient(configuration)
-        if not self.skip_tls:
-            _client_with_cert(api_client, self.ca_cert_path)
-        else:
-            urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
-            print("Insecure request warnings have been disabled")
-            configuration.verify_ssl = False
-
-        client.AuthenticationApi(api_client).get_api_group()
-        config_path = None
-        return "Logged into %s" % self.server
-    except client.ApiException as e:
-        _kube_api_error_handling(e)
-
-
-
-def logout(self) ‑> str -
-
-

This function is used to logout of a Kubernetes cluster.

-
- -Expand source code - -
def logout(self) -> str:
-    """
-    This function is used to logout of a Kubernetes cluster.
-    """
-    global config_path
-    config_path = None
-    global api_client
-    api_client = None
-    return "Successfully logged out of %s" % self.server
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/cluster/awload.html b/docs/detailed-documentation/cluster/awload.html deleted file mode 100644 index fba18e3ff..000000000 --- a/docs/detailed-documentation/cluster/awload.html +++ /dev/null @@ -1,328 +0,0 @@ - - - - - - -codeflare_sdk.cluster.awload API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster.awload

-
-
-

The awload sub-module contains the definition of the AWManager object, which handles -submission and deletion of existing AppWrappers from a user's file system.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The awload sub-module contains the definition of the AWManager object, which handles
-submission and deletion of existing AppWrappers from a user's file system.
-"""
-
-from os.path import isfile
-import errno
-import os
-import yaml
-
-from kubernetes import client, config
-from ..utils.kube_api_helpers import _kube_api_error_handling
-from .auth import config_check, get_api_client
-
-
-class AWManager:
-    """
-    An object for submitting and removing existing AppWrapper yamls
-    to be added to the Kueue localqueue.
-    """
-
-    def __init__(self, filename: str) -> None:
-        """
-        Create the AppWrapper Manager object by passing in an
-        AppWrapper yaml file
-        """
-        if not isfile(filename):
-            raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filename)
-        self.filename = filename
-        try:
-            with open(self.filename) as f:
-                self.awyaml = yaml.load(f, Loader=yaml.FullLoader)
-            assert self.awyaml["kind"] == "AppWrapper"
-            self.name = self.awyaml["metadata"]["name"]
-            self.namespace = self.awyaml["metadata"]["namespace"]
-        except:
-            raise ValueError(
-                f"{filename } is not a correctly formatted AppWrapper yaml"
-            )
-        self.submitted = False
-
-    def submit(self) -> None:
-        """
-        Attempts to create the AppWrapper custom resource using the yaml file
-        """
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            api_instance.create_namespaced_custom_object(
-                group="workload.codeflare.dev",
-                version="v1beta2",
-                namespace=self.namespace,
-                plural="appwrappers",
-                body=self.awyaml,
-            )
-        except Exception as e:
-            return _kube_api_error_handling(e)
-
-        self.submitted = True
-        print(f"AppWrapper {self.filename} submitted!")
-
-    def remove(self) -> None:
-        """
-        Attempts to delete the AppWrapper custom resource matching the name in the yaml,
-        if submitted by this manager.
-        """
-        if not self.submitted:
-            print("AppWrapper not submitted by this manager yet, nothing to remove")
-            return
-
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            api_instance.delete_namespaced_custom_object(
-                group="workload.codeflare.dev",
-                version="v1beta2",
-                namespace=self.namespace,
-                plural="appwrappers",
-                name=self.name,
-            )
-        except Exception as e:
-            return _kube_api_error_handling(e)
-
-        self.submitted = False
-        print(f"AppWrapper {self.name} removed!")
-
-
-
-
-
-
-
-
-
-

Classes

-
-
-class AWManager -(filename: str) -
-
-

An object for submitting and removing existing AppWrapper yamls -to be added to the Kueue localqueue.

-

Create the AppWrapper Manager object by passing in an -AppWrapper yaml file

-
- -Expand source code - -
class AWManager:
-    """
-    An object for submitting and removing existing AppWrapper yamls
-    to be added to the Kueue localqueue.
-    """
-
-    def __init__(self, filename: str) -> None:
-        """
-        Create the AppWrapper Manager object by passing in an
-        AppWrapper yaml file
-        """
-        if not isfile(filename):
-            raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filename)
-        self.filename = filename
-        try:
-            with open(self.filename) as f:
-                self.awyaml = yaml.load(f, Loader=yaml.FullLoader)
-            assert self.awyaml["kind"] == "AppWrapper"
-            self.name = self.awyaml["metadata"]["name"]
-            self.namespace = self.awyaml["metadata"]["namespace"]
-        except:
-            raise ValueError(
-                f"{filename } is not a correctly formatted AppWrapper yaml"
-            )
-        self.submitted = False
-
-    def submit(self) -> None:
-        """
-        Attempts to create the AppWrapper custom resource using the yaml file
-        """
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            api_instance.create_namespaced_custom_object(
-                group="workload.codeflare.dev",
-                version="v1beta2",
-                namespace=self.namespace,
-                plural="appwrappers",
-                body=self.awyaml,
-            )
-        except Exception as e:
-            return _kube_api_error_handling(e)
-
-        self.submitted = True
-        print(f"AppWrapper {self.filename} submitted!")
-
-    def remove(self) -> None:
-        """
-        Attempts to delete the AppWrapper custom resource matching the name in the yaml,
-        if submitted by this manager.
-        """
-        if not self.submitted:
-            print("AppWrapper not submitted by this manager yet, nothing to remove")
-            return
-
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            api_instance.delete_namespaced_custom_object(
-                group="workload.codeflare.dev",
-                version="v1beta2",
-                namespace=self.namespace,
-                plural="appwrappers",
-                name=self.name,
-            )
-        except Exception as e:
-            return _kube_api_error_handling(e)
-
-        self.submitted = False
-        print(f"AppWrapper {self.name} removed!")
-
-

Methods

-
-
-def remove(self) ‑> None -
-
-

Attempts to delete the AppWrapper custom resource matching the name in the yaml, -if submitted by this manager.

-
- -Expand source code - -
def remove(self) -> None:
-    """
-    Attempts to delete the AppWrapper custom resource matching the name in the yaml,
-    if submitted by this manager.
-    """
-    if not self.submitted:
-        print("AppWrapper not submitted by this manager yet, nothing to remove")
-        return
-
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        api_instance.delete_namespaced_custom_object(
-            group="workload.codeflare.dev",
-            version="v1beta2",
-            namespace=self.namespace,
-            plural="appwrappers",
-            name=self.name,
-        )
-    except Exception as e:
-        return _kube_api_error_handling(e)
-
-    self.submitted = False
-    print(f"AppWrapper {self.name} removed!")
-
-
-
-def submit(self) ‑> None -
-
-

Attempts to create the AppWrapper custom resource using the yaml file

-
- -Expand source code - -
def submit(self) -> None:
-    """
-    Attempts to create the AppWrapper custom resource using the yaml file
-    """
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        api_instance.create_namespaced_custom_object(
-            group="workload.codeflare.dev",
-            version="v1beta2",
-            namespace=self.namespace,
-            plural="appwrappers",
-            body=self.awyaml,
-        )
-    except Exception as e:
-        return _kube_api_error_handling(e)
-
-    self.submitted = True
-    print(f"AppWrapper {self.filename} submitted!")
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/cluster/cluster.html b/docs/detailed-documentation/cluster/cluster.html deleted file mode 100644 index 12865c04b..000000000 --- a/docs/detailed-documentation/cluster/cluster.html +++ /dev/null @@ -1,2174 +0,0 @@ - - - - - - -codeflare_sdk.cluster.cluster API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster.cluster

-
-
-

The cluster sub-module contains the definition of the Cluster object, which represents -the resources requested by the user. It also contains functions for checking the -cluster setup queue, a list of all existing clusters, and the user's working namespace.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The cluster sub-module contains the definition of the Cluster object, which represents
-the resources requested by the user. It also contains functions for checking the
-cluster setup queue, a list of all existing clusters, and the user's working namespace.
-"""
-
-import re
-import subprocess
-from time import sleep
-from typing import List, Optional, Tuple, Dict
-
-from kubernetes import config
-from ray.job_submission import JobSubmissionClient
-
-from .auth import config_check, get_api_client
-from ..utils import pretty_print
-from ..utils.generate_yaml import (
-    generate_appwrapper,
-    head_worker_gpu_count_from_cluster,
-)
-from ..utils.kube_api_helpers import _kube_api_error_handling
-from ..utils.generate_yaml import is_openshift_cluster
-
-from .config import ClusterConfiguration
-from .model import (
-    AppWrapper,
-    AppWrapperStatus,
-    CodeFlareClusterStatus,
-    RayCluster,
-    RayClusterStatus,
-)
-from .widgets import (
-    cluster_up_down_buttons,
-    is_notebook,
-)
-from kubernetes import client, config
-from kubernetes.utils import parse_quantity
-import yaml
-import os
-import requests
-
-from kubernetes import config
-from kubernetes.client.rest import ApiException
-
-
-class Cluster:
-    """
-    An object for requesting, bringing up, and taking down resources.
-    Can also be used for seeing the resource cluster status and details.
-
-    Note that currently, the underlying implementation is a Ray cluster.
-    """
-
-    def __init__(self, config: ClusterConfiguration):
-        """
-        Create the resource cluster object by passing in a ClusterConfiguration
-        (defined in the config sub-module). An AppWrapper will then be generated
-        based off of the configured resources to represent the desired cluster
-        request.
-        """
-        self.config = config
-        self.app_wrapper_yaml = self.create_app_wrapper()
-        self._job_submission_client = None
-        self.app_wrapper_name = self.config.name
-        if is_notebook():
-            cluster_up_down_buttons(self)
-
-    @property
-    def _client_headers(self):
-        k8_client = get_api_client()
-        return {
-            "Authorization": k8_client.configuration.get_api_key_with_prefix(
-                "authorization"
-            )
-        }
-
-    @property
-    def _client_verify_tls(self):
-        if not is_openshift_cluster or not self.config.verify_tls:
-            return False
-        return True
-
-    @property
-    def job_client(self):
-        k8client = get_api_client()
-        if self._job_submission_client:
-            return self._job_submission_client
-        if is_openshift_cluster():
-            self._job_submission_client = JobSubmissionClient(
-                self.cluster_dashboard_uri(),
-                headers=self._client_headers,
-                verify=self._client_verify_tls,
-            )
-        else:
-            self._job_submission_client = JobSubmissionClient(
-                self.cluster_dashboard_uri()
-            )
-        return self._job_submission_client
-
-    def create_app_wrapper(self):
-        """
-        Called upon cluster object creation, creates an AppWrapper yaml based on
-        the specifications of the ClusterConfiguration.
-        """
-
-        if self.config.namespace is None:
-            self.config.namespace = get_current_namespace()
-            if self.config.namespace is None:
-                print("Please specify with namespace=<your_current_namespace>")
-            elif type(self.config.namespace) is not str:
-                raise TypeError(
-                    f"Namespace {self.config.namespace} is of type {type(self.config.namespace)}. Check your Kubernetes Authentication."
-                )
-
-        return generate_appwrapper(self)
-
-    # creates a new cluster with the provided or default spec
-    def up(self):
-        """
-        Applies the Cluster yaml, pushing the resource request onto
-        the Kueue localqueue.
-        """
-
-        # check if RayCluster CustomResourceDefinition exists if not throw RuntimeError
-        self._throw_for_no_raycluster()
-
-        namespace = self.config.namespace
-
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            if self.config.appwrapper:
-                if self.config.write_to_file:
-                    with open(self.app_wrapper_yaml) as f:
-                        aw = yaml.load(f, Loader=yaml.FullLoader)
-                        api_instance.create_namespaced_custom_object(
-                            group="workload.codeflare.dev",
-                            version="v1beta2",
-                            namespace=namespace,
-                            plural="appwrappers",
-                            body=aw,
-                        )
-                else:
-                    aw = yaml.safe_load(self.app_wrapper_yaml)
-                    api_instance.create_namespaced_custom_object(
-                        group="workload.codeflare.dev",
-                        version="v1beta2",
-                        namespace=namespace,
-                        plural="appwrappers",
-                        body=aw,
-                    )
-                print(f"AppWrapper: '{self.config.name}' has successfully been created")
-            else:
-                self._component_resources_up(namespace, api_instance)
-                print(
-                    f"Ray Cluster: '{self.config.name}' has successfully been created"
-                )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-    def _throw_for_no_raycluster(self):
-        api_instance = client.CustomObjectsApi(get_api_client())
-        try:
-            api_instance.list_namespaced_custom_object(
-                group="ray.io",
-                version="v1",
-                namespace=self.config.namespace,
-                plural="rayclusters",
-            )
-        except ApiException as e:
-            if e.status == 404:
-                raise RuntimeError(
-                    "RayCluster CustomResourceDefinition unavailable contact your administrator."
-                )
-            else:
-                raise RuntimeError(
-                    "Failed to get RayCluster CustomResourceDefinition: " + str(e)
-                )
-
-    def down(self):
-        """
-        Deletes the AppWrapper yaml, scaling-down and deleting all resources
-        associated with the cluster.
-        """
-        namespace = self.config.namespace
-        self._throw_for_no_raycluster()
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            if self.config.appwrapper:
-                api_instance.delete_namespaced_custom_object(
-                    group="workload.codeflare.dev",
-                    version="v1beta2",
-                    namespace=namespace,
-                    plural="appwrappers",
-                    name=self.app_wrapper_name,
-                )
-                print(f"AppWrapper: '{self.config.name}' has successfully been deleted")
-            else:
-                self._component_resources_down(namespace, api_instance)
-                print(
-                    f"Ray Cluster: '{self.config.name}' has successfully been deleted"
-                )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-    def status(
-        self, print_to_console: bool = True
-    ) -> Tuple[CodeFlareClusterStatus, bool]:
-        """
-        Returns the requested cluster's status, as well as whether or not
-        it is ready for use.
-        """
-        ready = False
-        status = CodeFlareClusterStatus.UNKNOWN
-        if self.config.appwrapper:
-            # check the app wrapper status
-            appwrapper = _app_wrapper_status(self.config.name, self.config.namespace)
-            if appwrapper:
-                if appwrapper.status in [
-                    AppWrapperStatus.RESUMING,
-                    AppWrapperStatus.RESETTING,
-                ]:
-                    ready = False
-                    status = CodeFlareClusterStatus.STARTING
-                elif appwrapper.status in [
-                    AppWrapperStatus.FAILED,
-                ]:
-                    ready = False
-                    status = CodeFlareClusterStatus.FAILED  # should deleted be separate
-                    return status, ready  # exit early, no need to check ray status
-                elif appwrapper.status in [
-                    AppWrapperStatus.SUSPENDED,
-                    AppWrapperStatus.SUSPENDING,
-                ]:
-                    ready = False
-                    if appwrapper.status == AppWrapperStatus.SUSPENDED:
-                        status = CodeFlareClusterStatus.QUEUED
-                    else:
-                        status = CodeFlareClusterStatus.QUEUEING
-                    if print_to_console:
-                        pretty_print.print_app_wrappers_status([appwrapper])
-                    return (
-                        status,
-                        ready,
-                    )  # no need to check the ray status since still in queue
-
-        # check the ray cluster status
-        cluster = _ray_cluster_status(self.config.name, self.config.namespace)
-        if cluster:
-            if cluster.status == RayClusterStatus.SUSPENDED:
-                ready = False
-                status = CodeFlareClusterStatus.SUSPENDED
-            if cluster.status == RayClusterStatus.UNKNOWN:
-                ready = False
-                status = CodeFlareClusterStatus.STARTING
-            if cluster.status == RayClusterStatus.READY:
-                ready = True
-                status = CodeFlareClusterStatus.READY
-            elif cluster.status in [
-                RayClusterStatus.UNHEALTHY,
-                RayClusterStatus.FAILED,
-            ]:
-                ready = False
-                status = CodeFlareClusterStatus.FAILED
-
-            if print_to_console:
-                # overriding the number of gpus with requested
-                _, cluster.worker_gpu = head_worker_gpu_count_from_cluster(self)
-                pretty_print.print_cluster_status(cluster)
-        elif print_to_console:
-            if status == CodeFlareClusterStatus.UNKNOWN:
-                pretty_print.print_no_resources_found()
-            else:
-                pretty_print.print_app_wrappers_status([appwrapper], starting=True)
-
-        return status, ready
-
-    def is_dashboard_ready(self) -> bool:
-        try:
-            response = requests.get(
-                self.cluster_dashboard_uri(),
-                headers=self._client_headers,
-                timeout=5,
-                verify=self._client_verify_tls,
-            )
-        except requests.exceptions.SSLError:  # pragma no cover
-            # SSL exception occurs when oauth ingress has been created but cluster is not up
-            return False
-        if response.status_code == 200:
-            return True
-        else:
-            return False
-
-    def wait_ready(self, timeout: Optional[int] = None, dashboard_check: bool = True):
-        """
-        Waits for requested cluster to be ready, up to an optional timeout (s).
-        Checks every five seconds.
-        """
-        print("Waiting for requested resources to be set up...")
-        time = 0
-        while True:
-            if timeout and time >= timeout:
-                raise TimeoutError(
-                    f"wait() timed out after waiting {timeout}s for cluster to be ready"
-                )
-            status, ready = self.status(print_to_console=False)
-            if status == CodeFlareClusterStatus.UNKNOWN:
-                print(
-                    "WARNING: Current cluster status is unknown, have you run cluster.up yet?"
-                )
-            if ready:
-                break
-            sleep(5)
-            time += 5
-        print("Requested cluster is up and running!")
-
-        while dashboard_check:
-            if timeout and time >= timeout:
-                raise TimeoutError(
-                    f"wait() timed out after waiting {timeout}s for dashboard to be ready"
-                )
-            if self.is_dashboard_ready():
-                print("Dashboard is ready!")
-                break
-            sleep(5)
-            time += 5
-
-    def details(self, print_to_console: bool = True) -> RayCluster:
-        cluster = _copy_to_ray(self)
-        if print_to_console:
-            pretty_print.print_clusters([cluster])
-        return cluster
-
-    def cluster_uri(self) -> str:
-        """
-        Returns a string containing the cluster's URI.
-        """
-        return f"ray://{self.config.name}-head-svc.{self.config.namespace}.svc:10001"
-
-    def cluster_dashboard_uri(self) -> str:
-        """
-        Returns a string containing the cluster's dashboard URI.
-        """
-        config_check()
-        if is_openshift_cluster():
-            try:
-                api_instance = client.CustomObjectsApi(get_api_client())
-                routes = api_instance.list_namespaced_custom_object(
-                    group="route.openshift.io",
-                    version="v1",
-                    namespace=self.config.namespace,
-                    plural="routes",
-                )
-            except Exception as e:  # pragma: no cover
-                return _kube_api_error_handling(e)
-
-            for route in routes["items"]:
-                if route["metadata"][
-                    "name"
-                ] == f"ray-dashboard-{self.config.name}" or route["metadata"][
-                    "name"
-                ].startswith(
-                    f"{self.config.name}-ingress"
-                ):
-                    protocol = "https" if route["spec"].get("tls") else "http"
-                    return f"{protocol}://{route['spec']['host']}"
-        else:
-            try:
-                api_instance = client.NetworkingV1Api(get_api_client())
-                ingresses = api_instance.list_namespaced_ingress(self.config.namespace)
-            except Exception as e:  # pragma no cover
-                return _kube_api_error_handling(e)
-
-            for ingress in ingresses.items:
-                annotations = ingress.metadata.annotations
-                protocol = "http"
-                if (
-                    ingress.metadata.name == f"ray-dashboard-{self.config.name}"
-                    or ingress.metadata.name.startswith(f"{self.config.name}-ingress")
-                ):
-                    if annotations == None:
-                        protocol = "http"
-                    elif "route.openshift.io/termination" in annotations:
-                        protocol = "https"
-                return f"{protocol}://{ingress.spec.rules[0].host}"
-        return "Dashboard not available yet, have you run cluster.up()?"
-
-    def list_jobs(self) -> List:
-        """
-        This method accesses the head ray node in your cluster and lists the running jobs.
-        """
-        return self.job_client.list_jobs()
-
-    def job_status(self, job_id: str) -> str:
-        """
-        This method accesses the head ray node in your cluster and returns the job status for the provided job id.
-        """
-        return self.job_client.get_job_status(job_id)
-
-    def job_logs(self, job_id: str) -> str:
-        """
-        This method accesses the head ray node in your cluster and returns the logs for the provided job id.
-        """
-        return self.job_client.get_job_logs(job_id)
-
-    @staticmethod
-    def _head_worker_extended_resources_from_rc_dict(rc: Dict) -> Tuple[dict, dict]:
-        head_extended_resources, worker_extended_resources = {}, {}
-        for resource in rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"].keys():
-            if resource in ["memory", "cpu"]:
-                continue
-            worker_extended_resources[resource] = rc["spec"]["workerGroupSpecs"][0][
-                "template"
-            ]["spec"]["containers"][0]["resources"]["limits"][resource]
-
-        for resource in rc["spec"]["headGroupSpec"]["template"]["spec"]["containers"][
-            0
-        ]["resources"]["limits"].keys():
-            if resource in ["memory", "cpu"]:
-                continue
-            head_extended_resources[resource] = rc["spec"]["headGroupSpec"]["template"][
-                "spec"
-            ]["containers"][0]["resources"]["limits"][resource]
-
-        return head_extended_resources, worker_extended_resources
-
-    def from_k8_cluster_object(
-        rc,
-        appwrapper=True,
-        write_to_file=False,
-        verify_tls=True,
-    ):
-        config_check()
-        machine_types = (
-            rc["metadata"]["labels"]["orderedinstance"].split("_")
-            if "orderedinstance" in rc["metadata"]["labels"]
-            else []
-        )
-
-        (
-            head_extended_resources,
-            worker_extended_resources,
-        ) = Cluster._head_worker_extended_resources_from_rc_dict(rc)
-
-        cluster_config = ClusterConfiguration(
-            name=rc["metadata"]["name"],
-            namespace=rc["metadata"]["namespace"],
-            machine_types=machine_types,
-            head_cpu_requests=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["requests"]["cpu"],
-            head_cpu_limits=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["cpu"],
-            head_memory_requests=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["requests"]["memory"],
-            head_memory_limits=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["memory"],
-            num_workers=rc["spec"]["workerGroupSpecs"][0]["minReplicas"],
-            worker_cpu_requests=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["requests"]["cpu"],
-            worker_cpu_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["cpu"],
-            worker_memory_requests=rc["spec"]["workerGroupSpecs"][0]["template"][
-                "spec"
-            ]["containers"][0]["resources"]["requests"]["memory"],
-            worker_memory_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["memory"],
-            worker_extended_resource_requests=worker_extended_resources,
-            head_extended_resource_requests=head_extended_resources,
-            image=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"]["containers"][
-                0
-            ]["image"],
-            appwrapper=appwrapper,
-            write_to_file=write_to_file,
-            verify_tls=verify_tls,
-            local_queue=rc["metadata"]
-            .get("labels", dict())
-            .get("kueue.x-k8s.io/queue-name", None),
-        )
-        return Cluster(cluster_config)
-
-    def local_client_url(self):
-        ingress_domain = _get_ingress_domain(self)
-        return f"ray://{ingress_domain}"
-
-    def _component_resources_up(
-        self, namespace: str, api_instance: client.CustomObjectsApi
-    ):
-        if self.config.write_to_file:
-            with open(self.app_wrapper_yaml) as f:
-                yamls = list(yaml.load_all(f, Loader=yaml.FullLoader))
-                for resource in yamls:
-                    enable_ingress = (
-                        resource.get("spec", {})
-                        .get("headGroupSpec", {})
-                        .get("enableIngress")
-                    )
-                    if resource["kind"] == "RayCluster" and enable_ingress is True:
-                        name = resource["metadata"]["name"]
-                        print(
-                            f"Forbidden: RayCluster '{name}' has 'enableIngress' set to 'True'."
-                        )
-                        return
-                _create_resources(yamls, namespace, api_instance)
-        else:
-            yamls = yaml.load_all(self.app_wrapper_yaml, Loader=yaml.FullLoader)
-            _create_resources(yamls, namespace, api_instance)
-
-    def _component_resources_down(
-        self, namespace: str, api_instance: client.CustomObjectsApi
-    ):
-        cluster_name = self.config.name
-        if self.config.write_to_file:
-            with open(self.app_wrapper_yaml) as f:
-                yamls = yaml.load_all(f, Loader=yaml.FullLoader)
-                _delete_resources(yamls, namespace, api_instance, cluster_name)
-        else:
-            yamls = yaml.safe_load_all(self.app_wrapper_yaml)
-            _delete_resources(yamls, namespace, api_instance, cluster_name)
-
-
-def list_all_clusters(namespace: str, print_to_console: bool = True):
-    """
-    Returns (and prints by default) a list of all clusters in a given namespace.
-    """
-    clusters = _get_ray_clusters(namespace)
-    if print_to_console:
-        pretty_print.print_clusters(clusters)
-    return clusters
-
-
-def list_all_queued(
-    namespace: str, print_to_console: bool = True, appwrapper: bool = False
-):
-    """
-    Returns (and prints by default) a list of all currently queued-up Ray Clusters
-    in a given namespace.
-    """
-    if appwrapper:
-        resources = _get_app_wrappers(namespace, filter=[AppWrapperStatus.SUSPENDED])
-        if print_to_console:
-            pretty_print.print_app_wrappers_status(resources)
-    else:
-        resources = _get_ray_clusters(
-            namespace, filter=[RayClusterStatus.READY, RayClusterStatus.SUSPENDED]
-        )
-        if print_to_console:
-            pretty_print.print_ray_clusters_status(resources)
-    return resources
-
-
-def get_current_namespace():  # pragma: no cover
-    if os.path.isfile("/var/run/secrets/kubernetes.io/serviceaccount/namespace"):
-        try:
-            file = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r")
-            active_context = file.readline().strip("\n")
-            return active_context
-        except Exception as e:
-            print("Unable to find current namespace")
-    print("trying to gather from current context")
-    try:
-        _, active_context = config.list_kube_config_contexts(config_check())
-    except Exception as e:
-        return _kube_api_error_handling(e)
-    try:
-        return active_context["context"]["namespace"]
-    except KeyError:
-        return None
-
-
-def get_cluster(
-    cluster_name: str,
-    namespace: str = "default",
-    write_to_file: bool = False,
-    verify_tls: bool = True,
-):
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        rcs = api_instance.list_namespaced_custom_object(
-            group="ray.io",
-            version="v1",
-            namespace=namespace,
-            plural="rayclusters",
-        )
-    except Exception as e:
-        return _kube_api_error_handling(e)
-
-    for rc in rcs["items"]:
-        if rc["metadata"]["name"] == cluster_name:
-            appwrapper = _check_aw_exists(cluster_name, namespace)
-            return Cluster.from_k8_cluster_object(
-                rc,
-                appwrapper=appwrapper,
-                write_to_file=write_to_file,
-                verify_tls=verify_tls,
-            )
-    raise FileNotFoundError(
-        f"Cluster {cluster_name} is not found in {namespace} namespace"
-    )
-
-
-# private methods
-def _delete_resources(
-    yamls, namespace: str, api_instance: client.CustomObjectsApi, cluster_name: str
-):
-    for resource in yamls:
-        if resource["kind"] == "RayCluster":
-            name = resource["metadata"]["name"]
-            api_instance.delete_namespaced_custom_object(
-                group="ray.io",
-                version="v1",
-                namespace=namespace,
-                plural="rayclusters",
-                name=name,
-            )
-
-
-def _create_resources(yamls, namespace: str, api_instance: client.CustomObjectsApi):
-    for resource in yamls:
-        if resource["kind"] == "RayCluster":
-            api_instance.create_namespaced_custom_object(
-                group="ray.io",
-                version="v1",
-                namespace=namespace,
-                plural="rayclusters",
-                body=resource,
-            )
-
-
-def _check_aw_exists(name: str, namespace: str) -> bool:
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        aws = api_instance.list_namespaced_custom_object(
-            group="workload.codeflare.dev",
-            version="v1beta2",
-            namespace=namespace,
-            plural="appwrappers",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e, print_error=False)
-    for aw in aws["items"]:
-        if aw["metadata"]["name"] == name:
-            return True
-    return False
-
-
-# Cant test this until get_current_namespace is fixed and placed in this function over using `self`
-def _get_ingress_domain(self):  # pragma: no cover
-    config_check()
-
-    if self.config.namespace != None:
-        namespace = self.config.namespace
-    else:
-        namespace = get_current_namespace()
-    domain = None
-
-    if is_openshift_cluster():
-        try:
-            api_instance = client.CustomObjectsApi(get_api_client())
-
-            routes = api_instance.list_namespaced_custom_object(
-                group="route.openshift.io",
-                version="v1",
-                namespace=namespace,
-                plural="routes",
-            )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-        for route in routes["items"]:
-            if (
-                route["spec"]["port"]["targetPort"] == "client"
-                or route["spec"]["port"]["targetPort"] == 10001
-            ):
-                domain = route["spec"]["host"]
-    else:
-        try:
-            api_client = client.NetworkingV1Api(get_api_client())
-            ingresses = api_client.list_namespaced_ingress(namespace)
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-        for ingress in ingresses.items:
-            if ingress.spec.rules[0].http.paths[0].backend.service.port.number == 10001:
-                domain = ingress.spec.rules[0].host
-    return domain
-
-
-def _app_wrapper_status(name, namespace="default") -> Optional[AppWrapper]:
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        aws = api_instance.list_namespaced_custom_object(
-            group="workload.codeflare.dev",
-            version="v1beta2",
-            namespace=namespace,
-            plural="appwrappers",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-    for aw in aws["items"]:
-        if aw["metadata"]["name"] == name:
-            return _map_to_app_wrapper(aw)
-    return None
-
-
-def _ray_cluster_status(name, namespace="default") -> Optional[RayCluster]:
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        rcs = api_instance.list_namespaced_custom_object(
-            group="ray.io",
-            version="v1",
-            namespace=namespace,
-            plural="rayclusters",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-    for rc in rcs["items"]:
-        if rc["metadata"]["name"] == name:
-            return _map_to_ray_cluster(rc)
-    return None
-
-
-def _get_ray_clusters(
-    namespace="default", filter: Optional[List[RayClusterStatus]] = None
-) -> List[RayCluster]:
-    list_of_clusters = []
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        rcs = api_instance.list_namespaced_custom_object(
-            group="ray.io",
-            version="v1",
-            namespace=namespace,
-            plural="rayclusters",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-    # Get a list of RCs with the filter if it is passed to the function
-    if filter is not None:
-        for rc in rcs["items"]:
-            ray_cluster = _map_to_ray_cluster(rc)
-            if filter and ray_cluster.status in filter:
-                list_of_clusters.append(ray_cluster)
-    else:
-        for rc in rcs["items"]:
-            list_of_clusters.append(_map_to_ray_cluster(rc))
-    return list_of_clusters
-
-
-def _get_app_wrappers(
-    namespace="default", filter=List[AppWrapperStatus]
-) -> List[AppWrapper]:
-    list_of_app_wrappers = []
-
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        aws = api_instance.list_namespaced_custom_object(
-            group="workload.codeflare.dev",
-            version="v1beta2",
-            namespace=namespace,
-            plural="appwrappers",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-    for item in aws["items"]:
-        app_wrapper = _map_to_app_wrapper(item)
-        if filter and app_wrapper.status in filter:
-            list_of_app_wrappers.append(app_wrapper)
-        else:
-            # Unsure what the purpose of the filter is
-            list_of_app_wrappers.append(app_wrapper)
-    return list_of_app_wrappers
-
-
-def _map_to_ray_cluster(rc) -> Optional[RayCluster]:
-    if "status" in rc and "state" in rc["status"]:
-        status = RayClusterStatus(rc["status"]["state"].lower())
-    else:
-        status = RayClusterStatus.UNKNOWN
-    config_check()
-    dashboard_url = None
-    if is_openshift_cluster():
-        try:
-            api_instance = client.CustomObjectsApi(get_api_client())
-            routes = api_instance.list_namespaced_custom_object(
-                group="route.openshift.io",
-                version="v1",
-                namespace=rc["metadata"]["namespace"],
-                plural="routes",
-            )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-        for route in routes["items"]:
-            rc_name = rc["metadata"]["name"]
-            if route["metadata"]["name"] == f"ray-dashboard-{rc_name}" or route[
-                "metadata"
-            ]["name"].startswith(f"{rc_name}-ingress"):
-                protocol = "https" if route["spec"].get("tls") else "http"
-                dashboard_url = f"{protocol}://{route['spec']['host']}"
-    else:
-        try:
-            api_instance = client.NetworkingV1Api(get_api_client())
-            ingresses = api_instance.list_namespaced_ingress(
-                rc["metadata"]["namespace"]
-            )
-        except Exception as e:  # pragma no cover
-            return _kube_api_error_handling(e)
-        for ingress in ingresses.items:
-            annotations = ingress.metadata.annotations
-            protocol = "http"
-            if (
-                ingress.metadata.name == f"ray-dashboard-{rc['metadata']['name']}"
-                or ingress.metadata.name.startswith(f"{rc['metadata']['name']}-ingress")
-            ):
-                if annotations == None:
-                    protocol = "http"
-                elif "route.openshift.io/termination" in annotations:
-                    protocol = "https"
-            dashboard_url = f"{protocol}://{ingress.spec.rules[0].host}"
-
-    (
-        head_extended_resources,
-        worker_extended_resources,
-    ) = Cluster._head_worker_extended_resources_from_rc_dict(rc)
-
-    return RayCluster(
-        name=rc["metadata"]["name"],
-        status=status,
-        # for now we are not using autoscaling so same replicas is fine
-        num_workers=rc["spec"]["workerGroupSpecs"][0]["replicas"],
-        worker_mem_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"]["memory"],
-        worker_mem_requests=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["requests"]["memory"],
-        worker_cpu_requests=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["requests"]["cpu"],
-        worker_cpu_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"]["cpu"],
-        worker_extended_resources=worker_extended_resources,
-        namespace=rc["metadata"]["namespace"],
-        head_cpu_requests=rc["spec"]["headGroupSpec"]["template"]["spec"]["containers"][
-            0
-        ]["resources"]["requests"]["cpu"],
-        head_cpu_limits=rc["spec"]["headGroupSpec"]["template"]["spec"]["containers"][
-            0
-        ]["resources"]["limits"]["cpu"],
-        head_mem_requests=rc["spec"]["headGroupSpec"]["template"]["spec"]["containers"][
-            0
-        ]["resources"]["requests"]["memory"],
-        head_mem_limits=rc["spec"]["headGroupSpec"]["template"]["spec"]["containers"][
-            0
-        ]["resources"]["limits"]["memory"],
-        head_extended_resources=head_extended_resources,
-        dashboard=dashboard_url,
-    )
-
-
-def _map_to_app_wrapper(aw) -> AppWrapper:
-    if "status" in aw:
-        return AppWrapper(
-            name=aw["metadata"]["name"],
-            status=AppWrapperStatus(aw["status"]["phase"].lower()),
-        )
-    return AppWrapper(
-        name=aw["metadata"]["name"],
-        status=AppWrapperStatus("suspended"),
-    )
-
-
-def _copy_to_ray(cluster: Cluster) -> RayCluster:
-    ray = RayCluster(
-        name=cluster.config.name,
-        status=cluster.status(print_to_console=False)[0],
-        num_workers=cluster.config.num_workers,
-        worker_mem_requests=cluster.config.worker_memory_requests,
-        worker_mem_limits=cluster.config.worker_memory_limits,
-        worker_cpu_requests=cluster.config.worker_cpu_requests,
-        worker_cpu_limits=cluster.config.worker_cpu_limits,
-        worker_extended_resources=cluster.config.worker_extended_resource_requests,
-        namespace=cluster.config.namespace,
-        dashboard=cluster.cluster_dashboard_uri(),
-        head_mem_requests=cluster.config.head_memory_requests,
-        head_mem_limits=cluster.config.head_memory_limits,
-        head_cpu_requests=cluster.config.head_cpu_requests,
-        head_cpu_limits=cluster.config.head_cpu_limits,
-        head_extended_resources=cluster.config.head_extended_resource_requests,
-    )
-    if ray.status == CodeFlareClusterStatus.READY:
-        ray.status = RayClusterStatus.READY
-    return ray
-
-
-
-
-
-
-
-

Functions

-
-
-def get_cluster(cluster_name: str, namespace: str = 'default', write_to_file: bool = False, verify_tls: bool = True) -
-
-
-
- -Expand source code - -
def get_cluster(
-    cluster_name: str,
-    namespace: str = "default",
-    write_to_file: bool = False,
-    verify_tls: bool = True,
-):
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        rcs = api_instance.list_namespaced_custom_object(
-            group="ray.io",
-            version="v1",
-            namespace=namespace,
-            plural="rayclusters",
-        )
-    except Exception as e:
-        return _kube_api_error_handling(e)
-
-    for rc in rcs["items"]:
-        if rc["metadata"]["name"] == cluster_name:
-            appwrapper = _check_aw_exists(cluster_name, namespace)
-            return Cluster.from_k8_cluster_object(
-                rc,
-                appwrapper=appwrapper,
-                write_to_file=write_to_file,
-                verify_tls=verify_tls,
-            )
-    raise FileNotFoundError(
-        f"Cluster {cluster_name} is not found in {namespace} namespace"
-    )
-
-
-
-def get_current_namespace() -
-
-
-
- -Expand source code - -
def get_current_namespace():  # pragma: no cover
-    if os.path.isfile("/var/run/secrets/kubernetes.io/serviceaccount/namespace"):
-        try:
-            file = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r")
-            active_context = file.readline().strip("\n")
-            return active_context
-        except Exception as e:
-            print("Unable to find current namespace")
-    print("trying to gather from current context")
-    try:
-        _, active_context = config.list_kube_config_contexts(config_check())
-    except Exception as e:
-        return _kube_api_error_handling(e)
-    try:
-        return active_context["context"]["namespace"]
-    except KeyError:
-        return None
-
-
-
-def list_all_clusters(namespace: str, print_to_console: bool = True) -
-
-

Returns (and prints by default) a list of all clusters in a given namespace.

-
- -Expand source code - -
def list_all_clusters(namespace: str, print_to_console: bool = True):
-    """
-    Returns (and prints by default) a list of all clusters in a given namespace.
-    """
-    clusters = _get_ray_clusters(namespace)
-    if print_to_console:
-        pretty_print.print_clusters(clusters)
-    return clusters
-
-
-
-def list_all_queued(namespace: str, print_to_console: bool = True, appwrapper: bool = False) -
-
-

Returns (and prints by default) a list of all currently queued-up Ray Clusters -in a given namespace.

-
- -Expand source code - -
def list_all_queued(
-    namespace: str, print_to_console: bool = True, appwrapper: bool = False
-):
-    """
-    Returns (and prints by default) a list of all currently queued-up Ray Clusters
-    in a given namespace.
-    """
-    if appwrapper:
-        resources = _get_app_wrappers(namespace, filter=[AppWrapperStatus.SUSPENDED])
-        if print_to_console:
-            pretty_print.print_app_wrappers_status(resources)
-    else:
-        resources = _get_ray_clusters(
-            namespace, filter=[RayClusterStatus.READY, RayClusterStatus.SUSPENDED]
-        )
-        if print_to_console:
-            pretty_print.print_ray_clusters_status(resources)
-    return resources
-
-
-
-
-
-

Classes

-
-
-class Cluster -(config: ClusterConfiguration) -
-
-

An object for requesting, bringing up, and taking down resources. -Can also be used for seeing the resource cluster status and details.

-

Note that currently, the underlying implementation is a Ray cluster.

-

Create the resource cluster object by passing in a ClusterConfiguration -(defined in the config sub-module). An AppWrapper will then be generated -based off of the configured resources to represent the desired cluster -request.

-
- -Expand source code - -
class Cluster:
-    """
-    An object for requesting, bringing up, and taking down resources.
-    Can also be used for seeing the resource cluster status and details.
-
-    Note that currently, the underlying implementation is a Ray cluster.
-    """
-
-    def __init__(self, config: ClusterConfiguration):
-        """
-        Create the resource cluster object by passing in a ClusterConfiguration
-        (defined in the config sub-module). An AppWrapper will then be generated
-        based off of the configured resources to represent the desired cluster
-        request.
-        """
-        self.config = config
-        self.app_wrapper_yaml = self.create_app_wrapper()
-        self._job_submission_client = None
-        self.app_wrapper_name = self.config.name
-        if is_notebook():
-            cluster_up_down_buttons(self)
-
-    @property
-    def _client_headers(self):
-        k8_client = get_api_client()
-        return {
-            "Authorization": k8_client.configuration.get_api_key_with_prefix(
-                "authorization"
-            )
-        }
-
-    @property
-    def _client_verify_tls(self):
-        if not is_openshift_cluster or not self.config.verify_tls:
-            return False
-        return True
-
-    @property
-    def job_client(self):
-        k8client = get_api_client()
-        if self._job_submission_client:
-            return self._job_submission_client
-        if is_openshift_cluster():
-            self._job_submission_client = JobSubmissionClient(
-                self.cluster_dashboard_uri(),
-                headers=self._client_headers,
-                verify=self._client_verify_tls,
-            )
-        else:
-            self._job_submission_client = JobSubmissionClient(
-                self.cluster_dashboard_uri()
-            )
-        return self._job_submission_client
-
-    def create_app_wrapper(self):
-        """
-        Called upon cluster object creation, creates an AppWrapper yaml based on
-        the specifications of the ClusterConfiguration.
-        """
-
-        if self.config.namespace is None:
-            self.config.namespace = get_current_namespace()
-            if self.config.namespace is None:
-                print("Please specify with namespace=<your_current_namespace>")
-            elif type(self.config.namespace) is not str:
-                raise TypeError(
-                    f"Namespace {self.config.namespace} is of type {type(self.config.namespace)}. Check your Kubernetes Authentication."
-                )
-
-        return generate_appwrapper(self)
-
-    # creates a new cluster with the provided or default spec
-    def up(self):
-        """
-        Applies the Cluster yaml, pushing the resource request onto
-        the Kueue localqueue.
-        """
-
-        # check if RayCluster CustomResourceDefinition exists if not throw RuntimeError
-        self._throw_for_no_raycluster()
-
-        namespace = self.config.namespace
-
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            if self.config.appwrapper:
-                if self.config.write_to_file:
-                    with open(self.app_wrapper_yaml) as f:
-                        aw = yaml.load(f, Loader=yaml.FullLoader)
-                        api_instance.create_namespaced_custom_object(
-                            group="workload.codeflare.dev",
-                            version="v1beta2",
-                            namespace=namespace,
-                            plural="appwrappers",
-                            body=aw,
-                        )
-                else:
-                    aw = yaml.safe_load(self.app_wrapper_yaml)
-                    api_instance.create_namespaced_custom_object(
-                        group="workload.codeflare.dev",
-                        version="v1beta2",
-                        namespace=namespace,
-                        plural="appwrappers",
-                        body=aw,
-                    )
-                print(f"AppWrapper: '{self.config.name}' has successfully been created")
-            else:
-                self._component_resources_up(namespace, api_instance)
-                print(
-                    f"Ray Cluster: '{self.config.name}' has successfully been created"
-                )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-    def _throw_for_no_raycluster(self):
-        api_instance = client.CustomObjectsApi(get_api_client())
-        try:
-            api_instance.list_namespaced_custom_object(
-                group="ray.io",
-                version="v1",
-                namespace=self.config.namespace,
-                plural="rayclusters",
-            )
-        except ApiException as e:
-            if e.status == 404:
-                raise RuntimeError(
-                    "RayCluster CustomResourceDefinition unavailable contact your administrator."
-                )
-            else:
-                raise RuntimeError(
-                    "Failed to get RayCluster CustomResourceDefinition: " + str(e)
-                )
-
-    def down(self):
-        """
-        Deletes the AppWrapper yaml, scaling-down and deleting all resources
-        associated with the cluster.
-        """
-        namespace = self.config.namespace
-        self._throw_for_no_raycluster()
-        try:
-            config_check()
-            api_instance = client.CustomObjectsApi(get_api_client())
-            if self.config.appwrapper:
-                api_instance.delete_namespaced_custom_object(
-                    group="workload.codeflare.dev",
-                    version="v1beta2",
-                    namespace=namespace,
-                    plural="appwrappers",
-                    name=self.app_wrapper_name,
-                )
-                print(f"AppWrapper: '{self.config.name}' has successfully been deleted")
-            else:
-                self._component_resources_down(namespace, api_instance)
-                print(
-                    f"Ray Cluster: '{self.config.name}' has successfully been deleted"
-                )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-    def status(
-        self, print_to_console: bool = True
-    ) -> Tuple[CodeFlareClusterStatus, bool]:
-        """
-        Returns the requested cluster's status, as well as whether or not
-        it is ready for use.
-        """
-        ready = False
-        status = CodeFlareClusterStatus.UNKNOWN
-        if self.config.appwrapper:
-            # check the app wrapper status
-            appwrapper = _app_wrapper_status(self.config.name, self.config.namespace)
-            if appwrapper:
-                if appwrapper.status in [
-                    AppWrapperStatus.RESUMING,
-                    AppWrapperStatus.RESETTING,
-                ]:
-                    ready = False
-                    status = CodeFlareClusterStatus.STARTING
-                elif appwrapper.status in [
-                    AppWrapperStatus.FAILED,
-                ]:
-                    ready = False
-                    status = CodeFlareClusterStatus.FAILED  # should deleted be separate
-                    return status, ready  # exit early, no need to check ray status
-                elif appwrapper.status in [
-                    AppWrapperStatus.SUSPENDED,
-                    AppWrapperStatus.SUSPENDING,
-                ]:
-                    ready = False
-                    if appwrapper.status == AppWrapperStatus.SUSPENDED:
-                        status = CodeFlareClusterStatus.QUEUED
-                    else:
-                        status = CodeFlareClusterStatus.QUEUEING
-                    if print_to_console:
-                        pretty_print.print_app_wrappers_status([appwrapper])
-                    return (
-                        status,
-                        ready,
-                    )  # no need to check the ray status since still in queue
-
-        # check the ray cluster status
-        cluster = _ray_cluster_status(self.config.name, self.config.namespace)
-        if cluster:
-            if cluster.status == RayClusterStatus.SUSPENDED:
-                ready = False
-                status = CodeFlareClusterStatus.SUSPENDED
-            if cluster.status == RayClusterStatus.UNKNOWN:
-                ready = False
-                status = CodeFlareClusterStatus.STARTING
-            if cluster.status == RayClusterStatus.READY:
-                ready = True
-                status = CodeFlareClusterStatus.READY
-            elif cluster.status in [
-                RayClusterStatus.UNHEALTHY,
-                RayClusterStatus.FAILED,
-            ]:
-                ready = False
-                status = CodeFlareClusterStatus.FAILED
-
-            if print_to_console:
-                # overriding the number of gpus with requested
-                _, cluster.worker_gpu = head_worker_gpu_count_from_cluster(self)
-                pretty_print.print_cluster_status(cluster)
-        elif print_to_console:
-            if status == CodeFlareClusterStatus.UNKNOWN:
-                pretty_print.print_no_resources_found()
-            else:
-                pretty_print.print_app_wrappers_status([appwrapper], starting=True)
-
-        return status, ready
-
-    def is_dashboard_ready(self) -> bool:
-        try:
-            response = requests.get(
-                self.cluster_dashboard_uri(),
-                headers=self._client_headers,
-                timeout=5,
-                verify=self._client_verify_tls,
-            )
-        except requests.exceptions.SSLError:  # pragma no cover
-            # SSL exception occurs when oauth ingress has been created but cluster is not up
-            return False
-        if response.status_code == 200:
-            return True
-        else:
-            return False
-
-    def wait_ready(self, timeout: Optional[int] = None, dashboard_check: bool = True):
-        """
-        Waits for requested cluster to be ready, up to an optional timeout (s).
-        Checks every five seconds.
-        """
-        print("Waiting for requested resources to be set up...")
-        time = 0
-        while True:
-            if timeout and time >= timeout:
-                raise TimeoutError(
-                    f"wait() timed out after waiting {timeout}s for cluster to be ready"
-                )
-            status, ready = self.status(print_to_console=False)
-            if status == CodeFlareClusterStatus.UNKNOWN:
-                print(
-                    "WARNING: Current cluster status is unknown, have you run cluster.up yet?"
-                )
-            if ready:
-                break
-            sleep(5)
-            time += 5
-        print("Requested cluster is up and running!")
-
-        while dashboard_check:
-            if timeout and time >= timeout:
-                raise TimeoutError(
-                    f"wait() timed out after waiting {timeout}s for dashboard to be ready"
-                )
-            if self.is_dashboard_ready():
-                print("Dashboard is ready!")
-                break
-            sleep(5)
-            time += 5
-
-    def details(self, print_to_console: bool = True) -> RayCluster:
-        cluster = _copy_to_ray(self)
-        if print_to_console:
-            pretty_print.print_clusters([cluster])
-        return cluster
-
-    def cluster_uri(self) -> str:
-        """
-        Returns a string containing the cluster's URI.
-        """
-        return f"ray://{self.config.name}-head-svc.{self.config.namespace}.svc:10001"
-
-    def cluster_dashboard_uri(self) -> str:
-        """
-        Returns a string containing the cluster's dashboard URI.
-        """
-        config_check()
-        if is_openshift_cluster():
-            try:
-                api_instance = client.CustomObjectsApi(get_api_client())
-                routes = api_instance.list_namespaced_custom_object(
-                    group="route.openshift.io",
-                    version="v1",
-                    namespace=self.config.namespace,
-                    plural="routes",
-                )
-            except Exception as e:  # pragma: no cover
-                return _kube_api_error_handling(e)
-
-            for route in routes["items"]:
-                if route["metadata"][
-                    "name"
-                ] == f"ray-dashboard-{self.config.name}" or route["metadata"][
-                    "name"
-                ].startswith(
-                    f"{self.config.name}-ingress"
-                ):
-                    protocol = "https" if route["spec"].get("tls") else "http"
-                    return f"{protocol}://{route['spec']['host']}"
-        else:
-            try:
-                api_instance = client.NetworkingV1Api(get_api_client())
-                ingresses = api_instance.list_namespaced_ingress(self.config.namespace)
-            except Exception as e:  # pragma no cover
-                return _kube_api_error_handling(e)
-
-            for ingress in ingresses.items:
-                annotations = ingress.metadata.annotations
-                protocol = "http"
-                if (
-                    ingress.metadata.name == f"ray-dashboard-{self.config.name}"
-                    or ingress.metadata.name.startswith(f"{self.config.name}-ingress")
-                ):
-                    if annotations == None:
-                        protocol = "http"
-                    elif "route.openshift.io/termination" in annotations:
-                        protocol = "https"
-                return f"{protocol}://{ingress.spec.rules[0].host}"
-        return "Dashboard not available yet, have you run cluster.up()?"
-
-    def list_jobs(self) -> List:
-        """
-        This method accesses the head ray node in your cluster and lists the running jobs.
-        """
-        return self.job_client.list_jobs()
-
-    def job_status(self, job_id: str) -> str:
-        """
-        This method accesses the head ray node in your cluster and returns the job status for the provided job id.
-        """
-        return self.job_client.get_job_status(job_id)
-
-    def job_logs(self, job_id: str) -> str:
-        """
-        This method accesses the head ray node in your cluster and returns the logs for the provided job id.
-        """
-        return self.job_client.get_job_logs(job_id)
-
-    @staticmethod
-    def _head_worker_extended_resources_from_rc_dict(rc: Dict) -> Tuple[dict, dict]:
-        head_extended_resources, worker_extended_resources = {}, {}
-        for resource in rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"].keys():
-            if resource in ["memory", "cpu"]:
-                continue
-            worker_extended_resources[resource] = rc["spec"]["workerGroupSpecs"][0][
-                "template"
-            ]["spec"]["containers"][0]["resources"]["limits"][resource]
-
-        for resource in rc["spec"]["headGroupSpec"]["template"]["spec"]["containers"][
-            0
-        ]["resources"]["limits"].keys():
-            if resource in ["memory", "cpu"]:
-                continue
-            head_extended_resources[resource] = rc["spec"]["headGroupSpec"]["template"][
-                "spec"
-            ]["containers"][0]["resources"]["limits"][resource]
-
-        return head_extended_resources, worker_extended_resources
-
-    def from_k8_cluster_object(
-        rc,
-        appwrapper=True,
-        write_to_file=False,
-        verify_tls=True,
-    ):
-        config_check()
-        machine_types = (
-            rc["metadata"]["labels"]["orderedinstance"].split("_")
-            if "orderedinstance" in rc["metadata"]["labels"]
-            else []
-        )
-
-        (
-            head_extended_resources,
-            worker_extended_resources,
-        ) = Cluster._head_worker_extended_resources_from_rc_dict(rc)
-
-        cluster_config = ClusterConfiguration(
-            name=rc["metadata"]["name"],
-            namespace=rc["metadata"]["namespace"],
-            machine_types=machine_types,
-            head_cpu_requests=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["requests"]["cpu"],
-            head_cpu_limits=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["cpu"],
-            head_memory_requests=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["requests"]["memory"],
-            head_memory_limits=rc["spec"]["headGroupSpec"]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["memory"],
-            num_workers=rc["spec"]["workerGroupSpecs"][0]["minReplicas"],
-            worker_cpu_requests=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["requests"]["cpu"],
-            worker_cpu_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["cpu"],
-            worker_memory_requests=rc["spec"]["workerGroupSpecs"][0]["template"][
-                "spec"
-            ]["containers"][0]["resources"]["requests"]["memory"],
-            worker_memory_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-                "containers"
-            ][0]["resources"]["limits"]["memory"],
-            worker_extended_resource_requests=worker_extended_resources,
-            head_extended_resource_requests=head_extended_resources,
-            image=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"]["containers"][
-                0
-            ]["image"],
-            appwrapper=appwrapper,
-            write_to_file=write_to_file,
-            verify_tls=verify_tls,
-            local_queue=rc["metadata"]
-            .get("labels", dict())
-            .get("kueue.x-k8s.io/queue-name", None),
-        )
-        return Cluster(cluster_config)
-
-    def local_client_url(self):
-        ingress_domain = _get_ingress_domain(self)
-        return f"ray://{ingress_domain}"
-
-    def _component_resources_up(
-        self, namespace: str, api_instance: client.CustomObjectsApi
-    ):
-        if self.config.write_to_file:
-            with open(self.app_wrapper_yaml) as f:
-                yamls = list(yaml.load_all(f, Loader=yaml.FullLoader))
-                for resource in yamls:
-                    enable_ingress = (
-                        resource.get("spec", {})
-                        .get("headGroupSpec", {})
-                        .get("enableIngress")
-                    )
-                    if resource["kind"] == "RayCluster" and enable_ingress is True:
-                        name = resource["metadata"]["name"]
-                        print(
-                            f"Forbidden: RayCluster '{name}' has 'enableIngress' set to 'True'."
-                        )
-                        return
-                _create_resources(yamls, namespace, api_instance)
-        else:
-            yamls = yaml.load_all(self.app_wrapper_yaml, Loader=yaml.FullLoader)
-            _create_resources(yamls, namespace, api_instance)
-
-    def _component_resources_down(
-        self, namespace: str, api_instance: client.CustomObjectsApi
-    ):
-        cluster_name = self.config.name
-        if self.config.write_to_file:
-            with open(self.app_wrapper_yaml) as f:
-                yamls = yaml.load_all(f, Loader=yaml.FullLoader)
-                _delete_resources(yamls, namespace, api_instance, cluster_name)
-        else:
-            yamls = yaml.safe_load_all(self.app_wrapper_yaml)
-            _delete_resources(yamls, namespace, api_instance, cluster_name)
-
-

Instance variables

-
-
var job_client
-
-
-
- -Expand source code - -
@property
-def job_client(self):
-    k8client = get_api_client()
-    if self._job_submission_client:
-        return self._job_submission_client
-    if is_openshift_cluster():
-        self._job_submission_client = JobSubmissionClient(
-            self.cluster_dashboard_uri(),
-            headers=self._client_headers,
-            verify=self._client_verify_tls,
-        )
-    else:
-        self._job_submission_client = JobSubmissionClient(
-            self.cluster_dashboard_uri()
-        )
-    return self._job_submission_client
-
-
-
-

Methods

-
-
-def cluster_dashboard_uri(self) ‑> str -
-
-

Returns a string containing the cluster's dashboard URI.

-
- -Expand source code - -
def cluster_dashboard_uri(self) -> str:
-    """
-    Returns a string containing the cluster's dashboard URI.
-    """
-    config_check()
-    if is_openshift_cluster():
-        try:
-            api_instance = client.CustomObjectsApi(get_api_client())
-            routes = api_instance.list_namespaced_custom_object(
-                group="route.openshift.io",
-                version="v1",
-                namespace=self.config.namespace,
-                plural="routes",
-            )
-        except Exception as e:  # pragma: no cover
-            return _kube_api_error_handling(e)
-
-        for route in routes["items"]:
-            if route["metadata"][
-                "name"
-            ] == f"ray-dashboard-{self.config.name}" or route["metadata"][
-                "name"
-            ].startswith(
-                f"{self.config.name}-ingress"
-            ):
-                protocol = "https" if route["spec"].get("tls") else "http"
-                return f"{protocol}://{route['spec']['host']}"
-    else:
-        try:
-            api_instance = client.NetworkingV1Api(get_api_client())
-            ingresses = api_instance.list_namespaced_ingress(self.config.namespace)
-        except Exception as e:  # pragma no cover
-            return _kube_api_error_handling(e)
-
-        for ingress in ingresses.items:
-            annotations = ingress.metadata.annotations
-            protocol = "http"
-            if (
-                ingress.metadata.name == f"ray-dashboard-{self.config.name}"
-                or ingress.metadata.name.startswith(f"{self.config.name}-ingress")
-            ):
-                if annotations == None:
-                    protocol = "http"
-                elif "route.openshift.io/termination" in annotations:
-                    protocol = "https"
-            return f"{protocol}://{ingress.spec.rules[0].host}"
-    return "Dashboard not available yet, have you run cluster.up()?"
-
-
-
-def cluster_uri(self) ‑> str -
-
-

Returns a string containing the cluster's URI.

-
- -Expand source code - -
def cluster_uri(self) -> str:
-    """
-    Returns a string containing the cluster's URI.
-    """
-    return f"ray://{self.config.name}-head-svc.{self.config.namespace}.svc:10001"
-
-
-
-def create_app_wrapper(self) -
-
-

Called upon cluster object creation, creates an AppWrapper yaml based on -the specifications of the ClusterConfiguration.

-
- -Expand source code - -
def create_app_wrapper(self):
-    """
-    Called upon cluster object creation, creates an AppWrapper yaml based on
-    the specifications of the ClusterConfiguration.
-    """
-
-    if self.config.namespace is None:
-        self.config.namespace = get_current_namespace()
-        if self.config.namespace is None:
-            print("Please specify with namespace=<your_current_namespace>")
-        elif type(self.config.namespace) is not str:
-            raise TypeError(
-                f"Namespace {self.config.namespace} is of type {type(self.config.namespace)}. Check your Kubernetes Authentication."
-            )
-
-    return generate_appwrapper(self)
-
-
-
-def details(self, print_to_console: bool = True) ‑> RayCluster -
-
-
-
- -Expand source code - -
def details(self, print_to_console: bool = True) -> RayCluster:
-    cluster = _copy_to_ray(self)
-    if print_to_console:
-        pretty_print.print_clusters([cluster])
-    return cluster
-
-
-
-def down(self) -
-
-

Deletes the AppWrapper yaml, scaling-down and deleting all resources -associated with the cluster.

-
- -Expand source code - -
def down(self):
-    """
-    Deletes the AppWrapper yaml, scaling-down and deleting all resources
-    associated with the cluster.
-    """
-    namespace = self.config.namespace
-    self._throw_for_no_raycluster()
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        if self.config.appwrapper:
-            api_instance.delete_namespaced_custom_object(
-                group="workload.codeflare.dev",
-                version="v1beta2",
-                namespace=namespace,
-                plural="appwrappers",
-                name=self.app_wrapper_name,
-            )
-            print(f"AppWrapper: '{self.config.name}' has successfully been deleted")
-        else:
-            self._component_resources_down(namespace, api_instance)
-            print(
-                f"Ray Cluster: '{self.config.name}' has successfully been deleted"
-            )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-
-def from_k8_cluster_object(rc, appwrapper=True, write_to_file=False, verify_tls=True) -
-
-
-
- -Expand source code - -
def from_k8_cluster_object(
-    rc,
-    appwrapper=True,
-    write_to_file=False,
-    verify_tls=True,
-):
-    config_check()
-    machine_types = (
-        rc["metadata"]["labels"]["orderedinstance"].split("_")
-        if "orderedinstance" in rc["metadata"]["labels"]
-        else []
-    )
-
-    (
-        head_extended_resources,
-        worker_extended_resources,
-    ) = Cluster._head_worker_extended_resources_from_rc_dict(rc)
-
-    cluster_config = ClusterConfiguration(
-        name=rc["metadata"]["name"],
-        namespace=rc["metadata"]["namespace"],
-        machine_types=machine_types,
-        head_cpu_requests=rc["spec"]["headGroupSpec"]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["requests"]["cpu"],
-        head_cpu_limits=rc["spec"]["headGroupSpec"]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"]["cpu"],
-        head_memory_requests=rc["spec"]["headGroupSpec"]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["requests"]["memory"],
-        head_memory_limits=rc["spec"]["headGroupSpec"]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"]["memory"],
-        num_workers=rc["spec"]["workerGroupSpecs"][0]["minReplicas"],
-        worker_cpu_requests=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["requests"]["cpu"],
-        worker_cpu_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"]["cpu"],
-        worker_memory_requests=rc["spec"]["workerGroupSpecs"][0]["template"][
-            "spec"
-        ]["containers"][0]["resources"]["requests"]["memory"],
-        worker_memory_limits=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"][
-            "containers"
-        ][0]["resources"]["limits"]["memory"],
-        worker_extended_resource_requests=worker_extended_resources,
-        head_extended_resource_requests=head_extended_resources,
-        image=rc["spec"]["workerGroupSpecs"][0]["template"]["spec"]["containers"][
-            0
-        ]["image"],
-        appwrapper=appwrapper,
-        write_to_file=write_to_file,
-        verify_tls=verify_tls,
-        local_queue=rc["metadata"]
-        .get("labels", dict())
-        .get("kueue.x-k8s.io/queue-name", None),
-    )
-    return Cluster(cluster_config)
-
-
-
-def is_dashboard_ready(self) ‑> bool -
-
-
-
- -Expand source code - -
def is_dashboard_ready(self) -> bool:
-    try:
-        response = requests.get(
-            self.cluster_dashboard_uri(),
-            headers=self._client_headers,
-            timeout=5,
-            verify=self._client_verify_tls,
-        )
-    except requests.exceptions.SSLError:  # pragma no cover
-        # SSL exception occurs when oauth ingress has been created but cluster is not up
-        return False
-    if response.status_code == 200:
-        return True
-    else:
-        return False
-
-
-
-def job_logs(self, job_id: str) ‑> str -
-
-

This method accesses the head ray node in your cluster and returns the logs for the provided job id.

-
- -Expand source code - -
def job_logs(self, job_id: str) -> str:
-    """
-    This method accesses the head ray node in your cluster and returns the logs for the provided job id.
-    """
-    return self.job_client.get_job_logs(job_id)
-
-
-
-def job_status(self, job_id: str) ‑> str -
-
-

This method accesses the head ray node in your cluster and returns the job status for the provided job id.

-
- -Expand source code - -
def job_status(self, job_id: str) -> str:
-    """
-    This method accesses the head ray node in your cluster and returns the job status for the provided job id.
-    """
-    return self.job_client.get_job_status(job_id)
-
-
-
-def list_jobs(self) ‑> List -
-
-

This method accesses the head ray node in your cluster and lists the running jobs.

-
- -Expand source code - -
def list_jobs(self) -> List:
-    """
-    This method accesses the head ray node in your cluster and lists the running jobs.
-    """
-    return self.job_client.list_jobs()
-
-
-
-def local_client_url(self) -
-
-
-
- -Expand source code - -
def local_client_url(self):
-    ingress_domain = _get_ingress_domain(self)
-    return f"ray://{ingress_domain}"
-
-
-
-def status(self, print_to_console: bool = True) ‑> Tuple[CodeFlareClusterStatus, bool] -
-
-

Returns the requested cluster's status, as well as whether or not -it is ready for use.

-
- -Expand source code - -
def status(
-    self, print_to_console: bool = True
-) -> Tuple[CodeFlareClusterStatus, bool]:
-    """
-    Returns the requested cluster's status, as well as whether or not
-    it is ready for use.
-    """
-    ready = False
-    status = CodeFlareClusterStatus.UNKNOWN
-    if self.config.appwrapper:
-        # check the app wrapper status
-        appwrapper = _app_wrapper_status(self.config.name, self.config.namespace)
-        if appwrapper:
-            if appwrapper.status in [
-                AppWrapperStatus.RESUMING,
-                AppWrapperStatus.RESETTING,
-            ]:
-                ready = False
-                status = CodeFlareClusterStatus.STARTING
-            elif appwrapper.status in [
-                AppWrapperStatus.FAILED,
-            ]:
-                ready = False
-                status = CodeFlareClusterStatus.FAILED  # should deleted be separate
-                return status, ready  # exit early, no need to check ray status
-            elif appwrapper.status in [
-                AppWrapperStatus.SUSPENDED,
-                AppWrapperStatus.SUSPENDING,
-            ]:
-                ready = False
-                if appwrapper.status == AppWrapperStatus.SUSPENDED:
-                    status = CodeFlareClusterStatus.QUEUED
-                else:
-                    status = CodeFlareClusterStatus.QUEUEING
-                if print_to_console:
-                    pretty_print.print_app_wrappers_status([appwrapper])
-                return (
-                    status,
-                    ready,
-                )  # no need to check the ray status since still in queue
-
-    # check the ray cluster status
-    cluster = _ray_cluster_status(self.config.name, self.config.namespace)
-    if cluster:
-        if cluster.status == RayClusterStatus.SUSPENDED:
-            ready = False
-            status = CodeFlareClusterStatus.SUSPENDED
-        if cluster.status == RayClusterStatus.UNKNOWN:
-            ready = False
-            status = CodeFlareClusterStatus.STARTING
-        if cluster.status == RayClusterStatus.READY:
-            ready = True
-            status = CodeFlareClusterStatus.READY
-        elif cluster.status in [
-            RayClusterStatus.UNHEALTHY,
-            RayClusterStatus.FAILED,
-        ]:
-            ready = False
-            status = CodeFlareClusterStatus.FAILED
-
-        if print_to_console:
-            # overriding the number of gpus with requested
-            _, cluster.worker_gpu = head_worker_gpu_count_from_cluster(self)
-            pretty_print.print_cluster_status(cluster)
-    elif print_to_console:
-        if status == CodeFlareClusterStatus.UNKNOWN:
-            pretty_print.print_no_resources_found()
-        else:
-            pretty_print.print_app_wrappers_status([appwrapper], starting=True)
-
-    return status, ready
-
-
-
-def up(self) -
-
-

Applies the Cluster yaml, pushing the resource request onto -the Kueue localqueue.

-
- -Expand source code - -
def up(self):
-    """
-    Applies the Cluster yaml, pushing the resource request onto
-    the Kueue localqueue.
-    """
-
-    # check if RayCluster CustomResourceDefinition exists if not throw RuntimeError
-    self._throw_for_no_raycluster()
-
-    namespace = self.config.namespace
-
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        if self.config.appwrapper:
-            if self.config.write_to_file:
-                with open(self.app_wrapper_yaml) as f:
-                    aw = yaml.load(f, Loader=yaml.FullLoader)
-                    api_instance.create_namespaced_custom_object(
-                        group="workload.codeflare.dev",
-                        version="v1beta2",
-                        namespace=namespace,
-                        plural="appwrappers",
-                        body=aw,
-                    )
-            else:
-                aw = yaml.safe_load(self.app_wrapper_yaml)
-                api_instance.create_namespaced_custom_object(
-                    group="workload.codeflare.dev",
-                    version="v1beta2",
-                    namespace=namespace,
-                    plural="appwrappers",
-                    body=aw,
-                )
-            print(f"AppWrapper: '{self.config.name}' has successfully been created")
-        else:
-            self._component_resources_up(namespace, api_instance)
-            print(
-                f"Ray Cluster: '{self.config.name}' has successfully been created"
-            )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-
-def wait_ready(self, timeout: Optional[int] = None, dashboard_check: bool = True) -
-
-

Waits for requested cluster to be ready, up to an optional timeout (s). -Checks every five seconds.

-
- -Expand source code - -
def wait_ready(self, timeout: Optional[int] = None, dashboard_check: bool = True):
-    """
-    Waits for requested cluster to be ready, up to an optional timeout (s).
-    Checks every five seconds.
-    """
-    print("Waiting for requested resources to be set up...")
-    time = 0
-    while True:
-        if timeout and time >= timeout:
-            raise TimeoutError(
-                f"wait() timed out after waiting {timeout}s for cluster to be ready"
-            )
-        status, ready = self.status(print_to_console=False)
-        if status == CodeFlareClusterStatus.UNKNOWN:
-            print(
-                "WARNING: Current cluster status is unknown, have you run cluster.up yet?"
-            )
-        if ready:
-            break
-        sleep(5)
-        time += 5
-    print("Requested cluster is up and running!")
-
-    while dashboard_check:
-        if timeout and time >= timeout:
-            raise TimeoutError(
-                f"wait() timed out after waiting {timeout}s for dashboard to be ready"
-            )
-        if self.is_dashboard_ready():
-            print("Dashboard is ready!")
-            break
-        sleep(5)
-        time += 5
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/cluster/config.html b/docs/detailed-documentation/cluster/config.html deleted file mode 100644 index b329fb031..000000000 --- a/docs/detailed-documentation/cluster/config.html +++ /dev/null @@ -1,764 +0,0 @@ - - - - - - -codeflare_sdk.cluster.config API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster.config

-
-
-

The config sub-module contains the definition of the ClusterConfiguration dataclass, -which is used to specify resource requirements and other details when creating a -Cluster object.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The config sub-module contains the definition of the ClusterConfiguration dataclass,
-which is used to specify resource requirements and other details when creating a
-Cluster object.
-"""
-
-import pathlib
-import warnings
-from dataclasses import dataclass, field, fields
-from typing import Dict, List, Optional, Union, get_args, get_origin
-
-dir = pathlib.Path(__file__).parent.parent.resolve()
-
-# https://docs.ray.io/en/latest/ray-core/scheduling/accelerators.html
-DEFAULT_RESOURCE_MAPPING = {
-    "nvidia.com/gpu": "GPU",
-    "intel.com/gpu": "GPU",
-    "amd.com/gpu": "GPU",
-    "aws.amazon.com/neuroncore": "neuron_cores",
-    "google.com/tpu": "TPU",
-    "habana.ai/gaudi": "HPU",
-    "huawei.com/Ascend910": "NPU",
-    "huawei.com/Ascend310": "NPU",
-}
-
-
-@dataclass
-class ClusterConfiguration:
-    """
-    This dataclass is used to specify resource requirements and other details, and
-    is passed in as an argument when creating a Cluster object.
-
-    Attributes:
-    - name: The name of the cluster.
-    - namespace: The namespace in which the cluster should be created.
-    - head_info: A list of strings containing information about the head node.
-    - head_cpus: The number of CPUs to allocate to the head node.
-    - head_memory: The amount of memory to allocate to the head node.
-    - head_gpus: The number of GPUs to allocate to the head node. (Deprecated, use head_extended_resource_requests)
-    - head_extended_resource_requests: A dictionary of extended resource requests for the head node. ex: {"nvidia.com/gpu": 1}
-    - machine_types: A list of machine types to use for the cluster.
-    - min_cpus: The minimum number of CPUs to allocate to each worker.
-    - max_cpus: The maximum number of CPUs to allocate to each worker.
-    - num_workers: The number of workers to create.
-    - min_memory: The minimum amount of memory to allocate to each worker.
-    - max_memory: The maximum amount of memory to allocate to each worker.
-    - num_gpus: The number of GPUs to allocate to each worker. (Deprecated, use worker_extended_resource_requests)
-    - template: The path to the template file to use for the cluster.
-    - appwrapper: A boolean indicating whether to use an AppWrapper.
-    - envs: A dictionary of environment variables to set for the cluster.
-    - image: The image to use for the cluster.
-    - image_pull_secrets: A list of image pull secrets to use for the cluster.
-    - write_to_file: A boolean indicating whether to write the cluster configuration to a file.
-    - verify_tls: A boolean indicating whether to verify TLS when connecting to the cluster.
-    - labels: A dictionary of labels to apply to the cluster.
-    - worker_extended_resource_requests: A dictionary of extended resource requests for each worker. ex: {"nvidia.com/gpu": 1}
-    - extended_resource_mapping: A dictionary of custom resource mappings to map extended resource requests to RayCluster resource names
-    - overwrite_default_resource_mapping: A boolean indicating whether to overwrite the default resource mapping.
-    """
-
-    name: str
-    namespace: Optional[str] = None
-    head_info: List[str] = field(default_factory=list)
-    head_cpu_requests: Union[int, str] = 2
-    head_cpu_limits: Union[int, str] = 2
-    head_cpus: Optional[Union[int, str]] = None  # Deprecating
-    head_memory_requests: Union[int, str] = 8
-    head_memory_limits: Union[int, str] = 8
-    head_memory: Optional[Union[int, str]] = None  # Deprecating
-    head_gpus: Optional[int] = None  # Deprecating
-    head_extended_resource_requests: Dict[str, Union[str, int]] = field(
-        default_factory=dict
-    )
-    machine_types: List[str] = field(
-        default_factory=list
-    )  # ["m4.xlarge", "g4dn.xlarge"]
-    worker_cpu_requests: Union[int, str] = 1
-    worker_cpu_limits: Union[int, str] = 1
-    min_cpus: Optional[Union[int, str]] = None  # Deprecating
-    max_cpus: Optional[Union[int, str]] = None  # Deprecating
-    num_workers: int = 1
-    worker_memory_requests: Union[int, str] = 2
-    worker_memory_limits: Union[int, str] = 2
-    min_memory: Optional[Union[int, str]] = None  # Deprecating
-    max_memory: Optional[Union[int, str]] = None  # Deprecating
-    num_gpus: Optional[int] = None  # Deprecating
-    template: str = f"{dir}/templates/base-template.yaml"
-    appwrapper: bool = False
-    envs: Dict[str, str] = field(default_factory=dict)
-    image: str = ""
-    image_pull_secrets: List[str] = field(default_factory=list)
-    write_to_file: bool = False
-    verify_tls: bool = True
-    labels: Dict[str, str] = field(default_factory=dict)
-    worker_extended_resource_requests: Dict[str, Union[str, int]] = field(
-        default_factory=dict
-    )
-    extended_resource_mapping: Dict[str, str] = field(default_factory=dict)
-    overwrite_default_resource_mapping: bool = False
-    local_queue: Optional[str] = None
-
-    def __post_init__(self):
-        if not self.verify_tls:
-            print(
-                "Warning: TLS verification has been disabled - Endpoint checks will be bypassed"
-            )
-
-        self._validate_types()
-        self._memory_to_string()
-        self._str_mem_no_unit_add_GB()
-        self._memory_to_resource()
-        self._cpu_to_resource()
-        self._gpu_to_resource()
-        self._combine_extended_resource_mapping()
-        self._validate_extended_resource_requests(self.head_extended_resource_requests)
-        self._validate_extended_resource_requests(
-            self.worker_extended_resource_requests
-        )
-
-    def _combine_extended_resource_mapping(self):
-        if overwritten := set(self.extended_resource_mapping.keys()).intersection(
-            DEFAULT_RESOURCE_MAPPING.keys()
-        ):
-            if self.overwrite_default_resource_mapping:
-                warnings.warn(
-                    f"Overwriting default resource mapping for {overwritten}",
-                    UserWarning,
-                )
-            else:
-                raise ValueError(
-                    f"Resource mapping already exists for {overwritten}, set overwrite_default_resource_mapping to True to overwrite"
-                )
-        self.extended_resource_mapping = {
-            **DEFAULT_RESOURCE_MAPPING,
-            **self.extended_resource_mapping,
-        }
-
-    def _validate_extended_resource_requests(self, extended_resources: Dict[str, int]):
-        for k in extended_resources.keys():
-            if k not in self.extended_resource_mapping.keys():
-                raise ValueError(
-                    f"extended resource '{k}' not found in extended_resource_mapping, available resources are {list(self.extended_resource_mapping.keys())}, to add more supported resources use extended_resource_mapping. i.e. extended_resource_mapping = {{'{k}': 'FOO_BAR'}}"
-                )
-
-    def _gpu_to_resource(self):
-        if self.head_gpus:
-            warnings.warn(
-                f"head_gpus is being deprecated, replacing with head_extended_resource_requests['nvidia.com/gpu'] = {self.head_gpus}"
-            )
-            if "nvidia.com/gpu" in self.head_extended_resource_requests:
-                raise ValueError(
-                    "nvidia.com/gpu already exists in head_extended_resource_requests"
-                )
-            self.head_extended_resource_requests["nvidia.com/gpu"] = self.head_gpus
-        if self.num_gpus:
-            warnings.warn(
-                f"num_gpus is being deprecated, replacing with worker_extended_resource_requests['nvidia.com/gpu'] = {self.num_gpus}"
-            )
-            if "nvidia.com/gpu" in self.worker_extended_resource_requests:
-                raise ValueError(
-                    "nvidia.com/gpu already exists in worker_extended_resource_requests"
-                )
-            self.worker_extended_resource_requests["nvidia.com/gpu"] = self.num_gpus
-
-    def _str_mem_no_unit_add_GB(self):
-        if isinstance(self.head_memory, str) and self.head_memory.isdecimal():
-            self.head_memory = f"{self.head_memory}G"
-        if (
-            isinstance(self.worker_memory_requests, str)
-            and self.worker_memory_requests.isdecimal()
-        ):
-            self.worker_memory_requests = f"{self.worker_memory_requests}G"
-        if (
-            isinstance(self.worker_memory_limits, str)
-            and self.worker_memory_limits.isdecimal()
-        ):
-            self.worker_memory_limits = f"{self.worker_memory_limits}G"
-
-    def _memory_to_string(self):
-        if isinstance(self.head_memory_requests, int):
-            self.head_memory_requests = f"{self.head_memory_requests}G"
-        if isinstance(self.head_memory_limits, int):
-            self.head_memory_limits = f"{self.head_memory_limits}G"
-        if isinstance(self.worker_memory_requests, int):
-            self.worker_memory_requests = f"{self.worker_memory_requests}G"
-        if isinstance(self.worker_memory_limits, int):
-            self.worker_memory_limits = f"{self.worker_memory_limits}G"
-
-    def _cpu_to_resource(self):
-        if self.head_cpus:
-            warnings.warn(
-                "head_cpus is being deprecated, use head_cpu_requests and head_cpu_limits"
-            )
-            self.head_cpu_requests = self.head_cpu_limits = self.head_cpus
-        if self.min_cpus:
-            warnings.warn("min_cpus is being deprecated, use worker_cpu_requests")
-            self.worker_cpu_requests = self.min_cpus
-        if self.max_cpus:
-            warnings.warn("max_cpus is being deprecated, use worker_cpu_limits")
-            self.worker_cpu_limits = self.max_cpus
-
-    def _memory_to_resource(self):
-        if self.head_memory:
-            warnings.warn(
-                "head_memory is being deprecated, use head_memory_requests and head_memory_limits"
-            )
-            self.head_memory_requests = self.head_memory_limits = self.head_memory
-        if self.min_memory:
-            warnings.warn("min_memory is being deprecated, use worker_memory_requests")
-            self.worker_memory_requests = f"{self.min_memory}G"
-        if self.max_memory:
-            warnings.warn("max_memory is being deprecated, use worker_memory_limits")
-            self.worker_memory_limits = f"{self.max_memory}G"
-
-    def _validate_types(self):
-        """Validate the types of all fields in the ClusterConfiguration dataclass."""
-        for field_info in fields(self):
-            value = getattr(self, field_info.name)
-            expected_type = field_info.type
-            if not self._is_type(value, expected_type):
-                raise TypeError(
-                    f"'{field_info.name}' should be of type {expected_type}"
-                )
-
-    @staticmethod
-    def _is_type(value, expected_type):
-        """Check if the value matches the expected type."""
-
-        def check_type(value, expected_type):
-            origin_type = get_origin(expected_type)
-            args = get_args(expected_type)
-            if origin_type is Union:
-                return any(check_type(value, union_type) for union_type in args)
-            if origin_type is list:
-                return all(check_type(elem, args[0]) for elem in value)
-            if origin_type is dict:
-                return all(
-                    check_type(k, args[0]) and check_type(v, args[1])
-                    for k, v in value.items()
-                )
-            if origin_type is tuple:
-                return all(check_type(elem, etype) for elem, etype in zip(value, args))
-            return isinstance(value, expected_type)
-
-        return check_type(value, expected_type)
-
-
-
-
-
-
-
-
-
-

Classes

-
-
-class ClusterConfiguration -(name: str, namespace: Optional[str] = None, head_info: List[str] = <factory>, head_cpu_requests: Union[int, str] = 2, head_cpu_limits: Union[int, str] = 2, head_cpus: Union[int, str, ForwardRef(None)] = None, head_memory_requests: Union[int, str] = 8, head_memory_limits: Union[int, str] = 8, head_memory: Union[int, str, ForwardRef(None)] = None, head_gpus: Optional[int] = None, head_extended_resource_requests: Dict[str, Union[str, int]] = <factory>, machine_types: List[str] = <factory>, worker_cpu_requests: Union[int, str] = 1, worker_cpu_limits: Union[int, str] = 1, min_cpus: Union[int, str, ForwardRef(None)] = None, max_cpus: Union[int, str, ForwardRef(None)] = None, num_workers: int = 1, worker_memory_requests: Union[int, str] = 2, worker_memory_limits: Union[int, str] = 2, min_memory: Union[int, str, ForwardRef(None)] = None, max_memory: Union[int, str, ForwardRef(None)] = None, num_gpus: Optional[int] = None, template: str = '/home/runner/work/codeflare-sdk/codeflare-sdk/src/codeflare_sdk/templates/base-template.yaml', appwrapper: bool = False, envs: Dict[str, str] = <factory>, image: str = '', image_pull_secrets: List[str] = <factory>, write_to_file: bool = False, verify_tls: bool = True, labels: Dict[str, str] = <factory>, worker_extended_resource_requests: Dict[str, Union[str, int]] = <factory>, extended_resource_mapping: Dict[str, str] = <factory>, overwrite_default_resource_mapping: bool = False, local_queue: Optional[str] = None) -
-
-

This dataclass is used to specify resource requirements and other details, and -is passed in as an argument when creating a Cluster object.

-

Attributes: -- name: The name of the cluster. -- namespace: The namespace in which the cluster should be created. -- head_info: A list of strings containing information about the head node. -- head_cpus: The number of CPUs to allocate to the head node. -- head_memory: The amount of memory to allocate to the head node. -- head_gpus: The number of GPUs to allocate to the head node. (Deprecated, use head_extended_resource_requests) -- head_extended_resource_requests: A dictionary of extended resource requests for the head node. ex: {"nvidia.com/gpu": 1} -- machine_types: A list of machine types to use for the cluster. -- min_cpus: The minimum number of CPUs to allocate to each worker. -- max_cpus: The maximum number of CPUs to allocate to each worker. -- num_workers: The number of workers to create. -- min_memory: The minimum amount of memory to allocate to each worker. -- max_memory: The maximum amount of memory to allocate to each worker. -- num_gpus: The number of GPUs to allocate to each worker. (Deprecated, use worker_extended_resource_requests) -- template: The path to the template file to use for the cluster. -- appwrapper: A boolean indicating whether to use an AppWrapper. -- envs: A dictionary of environment variables to set for the cluster. -- image: The image to use for the cluster. -- image_pull_secrets: A list of image pull secrets to use for the cluster. -- write_to_file: A boolean indicating whether to write the cluster configuration to a file. -- verify_tls: A boolean indicating whether to verify TLS when connecting to the cluster. -- labels: A dictionary of labels to apply to the cluster. -- worker_extended_resource_requests: A dictionary of extended resource requests for each worker. ex: {"nvidia.com/gpu": 1} -- extended_resource_mapping: A dictionary of custom resource mappings to map extended resource requests to RayCluster resource names -- overwrite_default_resource_mapping: A boolean indicating whether to overwrite the default resource mapping.

-
- -Expand source code - -
@dataclass
-class ClusterConfiguration:
-    """
-    This dataclass is used to specify resource requirements and other details, and
-    is passed in as an argument when creating a Cluster object.
-
-    Attributes:
-    - name: The name of the cluster.
-    - namespace: The namespace in which the cluster should be created.
-    - head_info: A list of strings containing information about the head node.
-    - head_cpus: The number of CPUs to allocate to the head node.
-    - head_memory: The amount of memory to allocate to the head node.
-    - head_gpus: The number of GPUs to allocate to the head node. (Deprecated, use head_extended_resource_requests)
-    - head_extended_resource_requests: A dictionary of extended resource requests for the head node. ex: {"nvidia.com/gpu": 1}
-    - machine_types: A list of machine types to use for the cluster.
-    - min_cpus: The minimum number of CPUs to allocate to each worker.
-    - max_cpus: The maximum number of CPUs to allocate to each worker.
-    - num_workers: The number of workers to create.
-    - min_memory: The minimum amount of memory to allocate to each worker.
-    - max_memory: The maximum amount of memory to allocate to each worker.
-    - num_gpus: The number of GPUs to allocate to each worker. (Deprecated, use worker_extended_resource_requests)
-    - template: The path to the template file to use for the cluster.
-    - appwrapper: A boolean indicating whether to use an AppWrapper.
-    - envs: A dictionary of environment variables to set for the cluster.
-    - image: The image to use for the cluster.
-    - image_pull_secrets: A list of image pull secrets to use for the cluster.
-    - write_to_file: A boolean indicating whether to write the cluster configuration to a file.
-    - verify_tls: A boolean indicating whether to verify TLS when connecting to the cluster.
-    - labels: A dictionary of labels to apply to the cluster.
-    - worker_extended_resource_requests: A dictionary of extended resource requests for each worker. ex: {"nvidia.com/gpu": 1}
-    - extended_resource_mapping: A dictionary of custom resource mappings to map extended resource requests to RayCluster resource names
-    - overwrite_default_resource_mapping: A boolean indicating whether to overwrite the default resource mapping.
-    """
-
-    name: str
-    namespace: Optional[str] = None
-    head_info: List[str] = field(default_factory=list)
-    head_cpu_requests: Union[int, str] = 2
-    head_cpu_limits: Union[int, str] = 2
-    head_cpus: Optional[Union[int, str]] = None  # Deprecating
-    head_memory_requests: Union[int, str] = 8
-    head_memory_limits: Union[int, str] = 8
-    head_memory: Optional[Union[int, str]] = None  # Deprecating
-    head_gpus: Optional[int] = None  # Deprecating
-    head_extended_resource_requests: Dict[str, Union[str, int]] = field(
-        default_factory=dict
-    )
-    machine_types: List[str] = field(
-        default_factory=list
-    )  # ["m4.xlarge", "g4dn.xlarge"]
-    worker_cpu_requests: Union[int, str] = 1
-    worker_cpu_limits: Union[int, str] = 1
-    min_cpus: Optional[Union[int, str]] = None  # Deprecating
-    max_cpus: Optional[Union[int, str]] = None  # Deprecating
-    num_workers: int = 1
-    worker_memory_requests: Union[int, str] = 2
-    worker_memory_limits: Union[int, str] = 2
-    min_memory: Optional[Union[int, str]] = None  # Deprecating
-    max_memory: Optional[Union[int, str]] = None  # Deprecating
-    num_gpus: Optional[int] = None  # Deprecating
-    template: str = f"{dir}/templates/base-template.yaml"
-    appwrapper: bool = False
-    envs: Dict[str, str] = field(default_factory=dict)
-    image: str = ""
-    image_pull_secrets: List[str] = field(default_factory=list)
-    write_to_file: bool = False
-    verify_tls: bool = True
-    labels: Dict[str, str] = field(default_factory=dict)
-    worker_extended_resource_requests: Dict[str, Union[str, int]] = field(
-        default_factory=dict
-    )
-    extended_resource_mapping: Dict[str, str] = field(default_factory=dict)
-    overwrite_default_resource_mapping: bool = False
-    local_queue: Optional[str] = None
-
-    def __post_init__(self):
-        if not self.verify_tls:
-            print(
-                "Warning: TLS verification has been disabled - Endpoint checks will be bypassed"
-            )
-
-        self._validate_types()
-        self._memory_to_string()
-        self._str_mem_no_unit_add_GB()
-        self._memory_to_resource()
-        self._cpu_to_resource()
-        self._gpu_to_resource()
-        self._combine_extended_resource_mapping()
-        self._validate_extended_resource_requests(self.head_extended_resource_requests)
-        self._validate_extended_resource_requests(
-            self.worker_extended_resource_requests
-        )
-
-    def _combine_extended_resource_mapping(self):
-        if overwritten := set(self.extended_resource_mapping.keys()).intersection(
-            DEFAULT_RESOURCE_MAPPING.keys()
-        ):
-            if self.overwrite_default_resource_mapping:
-                warnings.warn(
-                    f"Overwriting default resource mapping for {overwritten}",
-                    UserWarning,
-                )
-            else:
-                raise ValueError(
-                    f"Resource mapping already exists for {overwritten}, set overwrite_default_resource_mapping to True to overwrite"
-                )
-        self.extended_resource_mapping = {
-            **DEFAULT_RESOURCE_MAPPING,
-            **self.extended_resource_mapping,
-        }
-
-    def _validate_extended_resource_requests(self, extended_resources: Dict[str, int]):
-        for k in extended_resources.keys():
-            if k not in self.extended_resource_mapping.keys():
-                raise ValueError(
-                    f"extended resource '{k}' not found in extended_resource_mapping, available resources are {list(self.extended_resource_mapping.keys())}, to add more supported resources use extended_resource_mapping. i.e. extended_resource_mapping = {{'{k}': 'FOO_BAR'}}"
-                )
-
-    def _gpu_to_resource(self):
-        if self.head_gpus:
-            warnings.warn(
-                f"head_gpus is being deprecated, replacing with head_extended_resource_requests['nvidia.com/gpu'] = {self.head_gpus}"
-            )
-            if "nvidia.com/gpu" in self.head_extended_resource_requests:
-                raise ValueError(
-                    "nvidia.com/gpu already exists in head_extended_resource_requests"
-                )
-            self.head_extended_resource_requests["nvidia.com/gpu"] = self.head_gpus
-        if self.num_gpus:
-            warnings.warn(
-                f"num_gpus is being deprecated, replacing with worker_extended_resource_requests['nvidia.com/gpu'] = {self.num_gpus}"
-            )
-            if "nvidia.com/gpu" in self.worker_extended_resource_requests:
-                raise ValueError(
-                    "nvidia.com/gpu already exists in worker_extended_resource_requests"
-                )
-            self.worker_extended_resource_requests["nvidia.com/gpu"] = self.num_gpus
-
-    def _str_mem_no_unit_add_GB(self):
-        if isinstance(self.head_memory, str) and self.head_memory.isdecimal():
-            self.head_memory = f"{self.head_memory}G"
-        if (
-            isinstance(self.worker_memory_requests, str)
-            and self.worker_memory_requests.isdecimal()
-        ):
-            self.worker_memory_requests = f"{self.worker_memory_requests}G"
-        if (
-            isinstance(self.worker_memory_limits, str)
-            and self.worker_memory_limits.isdecimal()
-        ):
-            self.worker_memory_limits = f"{self.worker_memory_limits}G"
-
-    def _memory_to_string(self):
-        if isinstance(self.head_memory_requests, int):
-            self.head_memory_requests = f"{self.head_memory_requests}G"
-        if isinstance(self.head_memory_limits, int):
-            self.head_memory_limits = f"{self.head_memory_limits}G"
-        if isinstance(self.worker_memory_requests, int):
-            self.worker_memory_requests = f"{self.worker_memory_requests}G"
-        if isinstance(self.worker_memory_limits, int):
-            self.worker_memory_limits = f"{self.worker_memory_limits}G"
-
-    def _cpu_to_resource(self):
-        if self.head_cpus:
-            warnings.warn(
-                "head_cpus is being deprecated, use head_cpu_requests and head_cpu_limits"
-            )
-            self.head_cpu_requests = self.head_cpu_limits = self.head_cpus
-        if self.min_cpus:
-            warnings.warn("min_cpus is being deprecated, use worker_cpu_requests")
-            self.worker_cpu_requests = self.min_cpus
-        if self.max_cpus:
-            warnings.warn("max_cpus is being deprecated, use worker_cpu_limits")
-            self.worker_cpu_limits = self.max_cpus
-
-    def _memory_to_resource(self):
-        if self.head_memory:
-            warnings.warn(
-                "head_memory is being deprecated, use head_memory_requests and head_memory_limits"
-            )
-            self.head_memory_requests = self.head_memory_limits = self.head_memory
-        if self.min_memory:
-            warnings.warn("min_memory is being deprecated, use worker_memory_requests")
-            self.worker_memory_requests = f"{self.min_memory}G"
-        if self.max_memory:
-            warnings.warn("max_memory is being deprecated, use worker_memory_limits")
-            self.worker_memory_limits = f"{self.max_memory}G"
-
-    def _validate_types(self):
-        """Validate the types of all fields in the ClusterConfiguration dataclass."""
-        for field_info in fields(self):
-            value = getattr(self, field_info.name)
-            expected_type = field_info.type
-            if not self._is_type(value, expected_type):
-                raise TypeError(
-                    f"'{field_info.name}' should be of type {expected_type}"
-                )
-
-    @staticmethod
-    def _is_type(value, expected_type):
-        """Check if the value matches the expected type."""
-
-        def check_type(value, expected_type):
-            origin_type = get_origin(expected_type)
-            args = get_args(expected_type)
-            if origin_type is Union:
-                return any(check_type(value, union_type) for union_type in args)
-            if origin_type is list:
-                return all(check_type(elem, args[0]) for elem in value)
-            if origin_type is dict:
-                return all(
-                    check_type(k, args[0]) and check_type(v, args[1])
-                    for k, v in value.items()
-                )
-            if origin_type is tuple:
-                return all(check_type(elem, etype) for elem, etype in zip(value, args))
-            return isinstance(value, expected_type)
-
-        return check_type(value, expected_type)
-
-

Class variables

-
-
var appwrapper : bool
-
-
-
-
var envs : Dict[str, str]
-
-
-
-
var extended_resource_mapping : Dict[str, str]
-
-
-
-
var head_cpu_limits : Union[int, str]
-
-
-
-
var head_cpu_requests : Union[int, str]
-
-
-
-
var head_cpus : Union[int, str, ForwardRef(None)]
-
-
-
-
var head_extended_resource_requests : Dict[str, Union[str, int]]
-
-
-
-
var head_gpus : Optional[int]
-
-
-
-
var head_info : List[str]
-
-
-
-
var head_memory : Union[int, str, ForwardRef(None)]
-
-
-
-
var head_memory_limits : Union[int, str]
-
-
-
-
var head_memory_requests : Union[int, str]
-
-
-
-
var image : str
-
-
-
-
var image_pull_secrets : List[str]
-
-
-
-
var labels : Dict[str, str]
-
-
-
-
var local_queue : Optional[str]
-
-
-
-
var machine_types : List[str]
-
-
-
-
var max_cpus : Union[int, str, ForwardRef(None)]
-
-
-
-
var max_memory : Union[int, str, ForwardRef(None)]
-
-
-
-
var min_cpus : Union[int, str, ForwardRef(None)]
-
-
-
-
var min_memory : Union[int, str, ForwardRef(None)]
-
-
-
-
var name : str
-
-
-
-
var namespace : Optional[str]
-
-
-
-
var num_gpus : Optional[int]
-
-
-
-
var num_workers : int
-
-
-
-
var overwrite_default_resource_mapping : bool
-
-
-
-
var template : str
-
-
-
-
var verify_tls : bool
-
-
-
-
var worker_cpu_limits : Union[int, str]
-
-
-
-
var worker_cpu_requests : Union[int, str]
-
-
-
-
var worker_extended_resource_requests : Dict[str, Union[str, int]]
-
-
-
-
var worker_memory_limits : Union[int, str]
-
-
-
-
var worker_memory_requests : Union[int, str]
-
-
-
-
var write_to_file : bool
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/cluster/index.html b/docs/detailed-documentation/cluster/index.html deleted file mode 100644 index f8c04fa29..000000000 --- a/docs/detailed-documentation/cluster/index.html +++ /dev/null @@ -1,129 +0,0 @@ - - - - - - -codeflare_sdk.cluster API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster

-
-
-
- -Expand source code - -
from .auth import (
-    Authentication,
-    KubeConfiguration,
-    TokenAuthentication,
-    KubeConfigFileAuthentication,
-)
-
-from .model import (
-    RayClusterStatus,
-    AppWrapperStatus,
-    CodeFlareClusterStatus,
-    RayCluster,
-    AppWrapper,
-)
-
-from .cluster import (
-    Cluster,
-    ClusterConfiguration,
-    get_cluster,
-    list_all_queued,
-    list_all_clusters,
-)
-
-from .widgets import (
-    view_clusters,
-)
-
-from .awload import AWManager
-
-
-
-

Sub-modules

-
-
codeflare_sdk.cluster.auth
-
-

The auth sub-module contains the definitions for the Authentication objects, which represent -the methods by which a user can authenticate to their …

-
-
codeflare_sdk.cluster.awload
-
-

The awload sub-module contains the definition of the AWManager object, which handles -submission and deletion of existing AppWrappers from a user's …

-
-
codeflare_sdk.cluster.cluster
-
-

The cluster sub-module contains the definition of the Cluster object, which represents -the resources requested by the user. It also contains functions …

-
-
codeflare_sdk.cluster.config
-
-

The config sub-module contains the definition of the ClusterConfiguration dataclass, -which is used to specify resource requirements and other details …

-
-
codeflare_sdk.cluster.model
-
-

The model sub-module defines Enums containing information for Ray cluster -states and AppWrapper states, and CodeFlare cluster states, as well as -…

-
-
codeflare_sdk.cluster.widgets
-
-

The widgets sub-module contains the ui widgets created using the ipywidgets package.

-
-
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/cluster/model.html b/docs/detailed-documentation/cluster/model.html deleted file mode 100644 index 7d87e34f8..000000000 --- a/docs/detailed-documentation/cluster/model.html +++ /dev/null @@ -1,531 +0,0 @@ - - - - - - -codeflare_sdk.cluster.model API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster.model

-
-
-

The model sub-module defines Enums containing information for Ray cluster -states and AppWrapper states, and CodeFlare cluster states, as well as -dataclasses to store information for Ray clusters and AppWrappers.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The model sub-module defines Enums containing information for Ray cluster
-states and AppWrapper states, and CodeFlare cluster states, as well as
-dataclasses to store information for Ray clusters and AppWrappers.
-"""
-
-from dataclasses import dataclass, field
-from enum import Enum
-import typing
-from typing import Union
-
-
-class RayClusterStatus(Enum):
-    """
-    Defines the possible reportable states of a Ray cluster.
-    """
-
-    # https://github.com/ray-project/kuberay/blob/master/ray-operator/apis/ray/v1/raycluster_types.go#L112-L117
-    READY = "ready"
-    UNHEALTHY = "unhealthy"
-    FAILED = "failed"
-    UNKNOWN = "unknown"
-    SUSPENDED = "suspended"
-
-
-class AppWrapperStatus(Enum):
-    """
-    Defines the possible reportable phases of an AppWrapper.
-    """
-
-    SUSPENDED = "suspended"
-    RESUMING = "resuming"
-    RUNNING = "running"
-    RESETTING = "resetting"
-    SUSPENDING = "suspending"
-    SUCCEEDED = "succeeded"
-    FAILED = "failed"
-    TERMINATING = "terminating"
-
-
-class CodeFlareClusterStatus(Enum):
-    """
-    Defines the possible reportable states of a Codeflare cluster.
-    """
-
-    READY = 1
-    STARTING = 2
-    QUEUED = 3
-    QUEUEING = 4
-    FAILED = 5
-    UNKNOWN = 6
-    SUSPENDED = 7
-
-
-@dataclass
-class RayCluster:
-    """
-    For storing information about a Ray cluster.
-    """
-
-    name: str
-    status: RayClusterStatus
-    head_cpu_requests: int
-    head_cpu_limits: int
-    head_mem_requests: str
-    head_mem_limits: str
-    num_workers: int
-    worker_mem_requests: str
-    worker_mem_limits: str
-    worker_cpu_requests: Union[int, str]
-    worker_cpu_limits: Union[int, str]
-    namespace: str
-    dashboard: str
-    worker_extended_resources: typing.Dict[str, int] = field(default_factory=dict)
-    head_extended_resources: typing.Dict[str, int] = field(default_factory=dict)
-
-
-@dataclass
-class AppWrapper:
-    """
-    For storing information about an AppWrapper.
-    """
-
-    name: str
-    status: AppWrapperStatus
-
-
-
-
-
-
-
-
-
-

Classes

-
-
-class AppWrapper -(name: str, status: AppWrapperStatus) -
-
-

For storing information about an AppWrapper.

-
- -Expand source code - -
@dataclass
-class AppWrapper:
-    """
-    For storing information about an AppWrapper.
-    """
-
-    name: str
-    status: AppWrapperStatus
-
-

Class variables

-
-
var name : str
-
-
-
-
var status : AppWrapperStatus
-
-
-
-
-
-
-class AppWrapperStatus -(value, names=None, *, module=None, qualname=None, type=None, start=1) -
-
-

Defines the possible reportable phases of an AppWrapper.

-
- -Expand source code - -
class AppWrapperStatus(Enum):
-    """
-    Defines the possible reportable phases of an AppWrapper.
-    """
-
-    SUSPENDED = "suspended"
-    RESUMING = "resuming"
-    RUNNING = "running"
-    RESETTING = "resetting"
-    SUSPENDING = "suspending"
-    SUCCEEDED = "succeeded"
-    FAILED = "failed"
-    TERMINATING = "terminating"
-
-

Ancestors

-
    -
  • enum.Enum
    • -
-

Class variables

-
-
var FAILED
-
-
-
-
var RESETTING
-
-
-
-
var RESUMING
-
-
-
-
var RUNNING
-
-
-
-
var SUCCEEDED
-
-
-
-
var SUSPENDED
-
-
-
-
var SUSPENDING
-
-
-
-
var TERMINATING
-
-
-
-
-
-
-class CodeFlareClusterStatus -(value, names=None, *, module=None, qualname=None, type=None, start=1) -
-
-

Defines the possible reportable states of a Codeflare cluster.

-
- -Expand source code - -
class CodeFlareClusterStatus(Enum):
-    """
-    Defines the possible reportable states of a Codeflare cluster.
-    """
-
-    READY = 1
-    STARTING = 2
-    QUEUED = 3
-    QUEUEING = 4
-    FAILED = 5
-    UNKNOWN = 6
-    SUSPENDED = 7
-
-

Ancestors

-
    -
  • enum.Enum
    • -
-

Class variables

-
-
var FAILED
-
-
-
-
var QUEUED
-
-
-
-
var QUEUEING
-
-
-
-
var READY
-
-
-
-
var STARTING
-
-
-
-
var SUSPENDED
-
-
-
-
var UNKNOWN
-
-
-
-
-
-
-class RayCluster -(name: str, status: RayClusterStatus, head_cpu_requests: int, head_cpu_limits: int, head_mem_requests: str, head_mem_limits: str, num_workers: int, worker_mem_requests: str, worker_mem_limits: str, worker_cpu_requests: Union[int, str], worker_cpu_limits: Union[int, str], namespace: str, dashboard: str, worker_extended_resources: Dict[str, int] = <factory>, head_extended_resources: Dict[str, int] = <factory>) -
-
-

For storing information about a Ray cluster.

-
- -Expand source code - -
@dataclass
-class RayCluster:
-    """
-    For storing information about a Ray cluster.
-    """
-
-    name: str
-    status: RayClusterStatus
-    head_cpu_requests: int
-    head_cpu_limits: int
-    head_mem_requests: str
-    head_mem_limits: str
-    num_workers: int
-    worker_mem_requests: str
-    worker_mem_limits: str
-    worker_cpu_requests: Union[int, str]
-    worker_cpu_limits: Union[int, str]
-    namespace: str
-    dashboard: str
-    worker_extended_resources: typing.Dict[str, int] = field(default_factory=dict)
-    head_extended_resources: typing.Dict[str, int] = field(default_factory=dict)
-
-

Class variables

-
-
var dashboard : str
-
-
-
-
var head_cpu_limits : int
-
-
-
-
var head_cpu_requests : int
-
-
-
-
var head_extended_resources : Dict[str, int]
-
-
-
-
var head_mem_limits : str
-
-
-
-
var head_mem_requests : str
-
-
-
-
var name : str
-
-
-
-
var namespace : str
-
-
-
-
var num_workers : int
-
-
-
-
var status : RayClusterStatus
-
-
-
-
var worker_cpu_limits : Union[int, str]
-
-
-
-
var worker_cpu_requests : Union[int, str]
-
-
-
-
var worker_extended_resources : Dict[str, int]
-
-
-
-
var worker_mem_limits : str
-
-
-
-
var worker_mem_requests : str
-
-
-
-
-
-
-class RayClusterStatus -(value, names=None, *, module=None, qualname=None, type=None, start=1) -
-
-

Defines the possible reportable states of a Ray cluster.

-
- -Expand source code - -
class RayClusterStatus(Enum):
-    """
-    Defines the possible reportable states of a Ray cluster.
-    """
-
-    # https://github.com/ray-project/kuberay/blob/master/ray-operator/apis/ray/v1/raycluster_types.go#L112-L117
-    READY = "ready"
-    UNHEALTHY = "unhealthy"
-    FAILED = "failed"
-    UNKNOWN = "unknown"
-    SUSPENDED = "suspended"
-
-

Ancestors

-
    -
  • enum.Enum
    • -
-

Class variables

-
-
var FAILED
-
-
-
-
var READY
-
-
-
-
var SUSPENDED
-
-
-
-
var UNHEALTHY
-
-
-
-
var UNKNOWN
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/cluster/widgets.html b/docs/detailed-documentation/cluster/widgets.html deleted file mode 100644 index b0334903f..000000000 --- a/docs/detailed-documentation/cluster/widgets.html +++ /dev/null @@ -1,758 +0,0 @@ - - - - - - -codeflare_sdk.cluster.widgets API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.cluster.widgets

-
-
-

The widgets sub-module contains the ui widgets created using the ipywidgets package.

-
- -Expand source code - -
# Copyright 2024 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The widgets sub-module contains the ui widgets created using the ipywidgets package.
-"""
-import contextlib
-import io
-import os
-import warnings
-import time
-import codeflare_sdk
-from kubernetes import client
-from kubernetes.client.rest import ApiException
-import ipywidgets as widgets
-from IPython.display import display, HTML, Javascript
-import pandas as pd
-from .config import ClusterConfiguration
-from .model import RayClusterStatus
-from ..utils.kube_api_helpers import _kube_api_error_handling
-from .auth import config_check, get_api_client
-
-
-def cluster_up_down_buttons(cluster: "codeflare_sdk.cluster.Cluster") -> widgets.Button:
-    """
-    The cluster_up_down_buttons function returns two button widgets for a create and delete button.
-    The function uses the appwrapper bool to distinguish between resource type for the tool tip.
-    """
-    resource = "Ray Cluster"
-    if cluster.config.appwrapper:
-        resource = "AppWrapper"
-
-    up_button = widgets.Button(
-        description="Cluster Up",
-        tooltip=f"Create the {resource}",
-        icon="play",
-    )
-
-    delete_button = widgets.Button(
-        description="Cluster Down",
-        tooltip=f"Delete the {resource}",
-        icon="trash",
-    )
-
-    wait_ready_check = wait_ready_check_box()
-    output = widgets.Output()
-
-    # Display the buttons in an HBox wrapped in a VBox which includes the wait_ready Checkbox
-    button_display = widgets.HBox([up_button, delete_button])
-    display(widgets.VBox([button_display, wait_ready_check]), output)
-
-    def on_up_button_clicked(b):  # Handle the up button click event
-        with output:
-            output.clear_output()
-            cluster.up()
-
-            # If the wait_ready Checkbox is clicked(value == True) trigger the wait_ready function
-            if wait_ready_check.value:
-                cluster.wait_ready()
-
-    def on_down_button_clicked(b):  # Handle the down button click event
-        with output:
-            output.clear_output()
-            cluster.down()
-
-    up_button.on_click(on_up_button_clicked)
-    delete_button.on_click(on_down_button_clicked)
-
-
-def wait_ready_check_box():
-    """
-    The wait_ready_check_box function will return a checkbox widget used for waiting for the resource to be in the state READY.
-    """
-    wait_ready_check_box = widgets.Checkbox(
-        False,
-        description="Wait for Cluster?",
-    )
-    return wait_ready_check_box
-
-
-def is_notebook() -> bool:
-    """
-    The is_notebook function checks if Jupyter Notebook environment variables exist in the given environment and return True/False based on that.
-    """
-    if (
-        "PYDEVD_IPYTHON_COMPATIBLE_DEBUGGING" in os.environ
-        or "JPY_SESSION_NAME" in os.environ
-    ):  # If running Jupyter NBs in VsCode or RHOAI/ODH display UI buttons
-        return True
-    else:
-        return False
-
-
-def view_clusters(namespace: str = None):
-    """
-    view_clusters function will display existing clusters with their specs, and handle user interactions.
-    """
-    if not is_notebook():
-        warnings.warn(
-            "view_clusters can only be used in a Jupyter Notebook environment."
-        )
-        return  # Exit function if not in Jupyter Notebook
-
-    from .cluster import get_current_namespace
-
-    if not namespace:
-        namespace = get_current_namespace()
-
-    user_output = widgets.Output()
-    raycluster_data_output = widgets.Output()
-    url_output = widgets.Output()
-
-    ray_clusters_df = _fetch_cluster_data(namespace)
-    if ray_clusters_df.empty:
-        print(f"No clusters found in the {namespace} namespace.")
-        return
-
-    classification_widget = widgets.ToggleButtons(
-        options=ray_clusters_df["Name"].tolist(),
-        value=ray_clusters_df["Name"].tolist()[0],
-        description="Select an existing cluster:",
-    )
-    # Setting the initial value to trigger the event handler to display the cluster details.
-    initial_value = classification_widget.value
-    _on_cluster_click(
-        {"new": initial_value}, raycluster_data_output, namespace, classification_widget
-    )
-    classification_widget.observe(
-        lambda selection_change: _on_cluster_click(
-            selection_change, raycluster_data_output, namespace, classification_widget
-        ),
-        names="value",
-    )
-
-    # UI table buttons
-    delete_button = widgets.Button(
-        description="Delete Cluster",
-        icon="trash",
-        tooltip="Delete the selected cluster",
-    )
-    delete_button.on_click(
-        lambda b: _on_delete_button_click(
-            b,
-            classification_widget,
-            ray_clusters_df,
-            raycluster_data_output,
-            user_output,
-            delete_button,
-            list_jobs_button,
-            ray_dashboard_button,
-        )
-    )
-
-    list_jobs_button = widgets.Button(
-        description="View Jobs", icon="suitcase", tooltip="Open the Ray Job Dashboard"
-    )
-    list_jobs_button.on_click(
-        lambda b: _on_list_jobs_button_click(
-            b, classification_widget, ray_clusters_df, user_output, url_output
-        )
-    )
-
-    ray_dashboard_button = widgets.Button(
-        description="Open Ray Dashboard",
-        icon="dashboard",
-        tooltip="Open the Ray Dashboard in a new tab",
-        layout=widgets.Layout(width="auto"),
-    )
-    ray_dashboard_button.on_click(
-        lambda b: _on_ray_dashboard_button_click(
-            b, classification_widget, ray_clusters_df, user_output, url_output
-        )
-    )
-
-    display(widgets.VBox([classification_widget, raycluster_data_output]))
-    display(
-        widgets.HBox([delete_button, list_jobs_button, ray_dashboard_button]),
-        url_output,
-        user_output,
-    )
-
-
-def _on_cluster_click(
-    selection_change,
-    raycluster_data_output: widgets.Output,
-    namespace: str,
-    classification_widget: widgets.ToggleButtons,
-):
-    """
-    _on_cluster_click handles the event when a cluster is selected from the toggle buttons, updating the output with cluster details.
-    """
-    new_value = selection_change["new"]
-    raycluster_data_output.clear_output()
-    ray_clusters_df = _fetch_cluster_data(namespace)
-    classification_widget.options = ray_clusters_df["Name"].tolist()
-    with raycluster_data_output:
-        display(
-            HTML(
-                ray_clusters_df[ray_clusters_df["Name"] == new_value][
-                    [
-                        "Name",
-                        "Namespace",
-                        "Num Workers",
-                        "Head GPUs",
-                        "Head CPU Req~Lim",
-                        "Head Memory Req~Lim",
-                        "Worker GPUs",
-                        "Worker CPU Req~Lim",
-                        "Worker Memory Req~Lim",
-                        "status",
-                    ]
-                ].to_html(escape=False, index=False, border=2)
-            )
-        )
-
-
-def _on_delete_button_click(
-    b,
-    classification_widget: widgets.ToggleButtons,
-    ray_clusters_df: pd.DataFrame,
-    raycluster_data_output: widgets.Output,
-    user_output: widgets.Output,
-    delete_button: widgets.Button,
-    list_jobs_button: widgets.Button,
-    ray_dashboard_button: widgets.Button,
-):
-    """
-    _on_delete_button_click handles the event when the Delete Button is clicked, deleting the selected cluster.
-    """
-    cluster_name = classification_widget.value
-    namespace = ray_clusters_df[ray_clusters_df["Name"] == classification_widget.value][
-        "Namespace"
-    ].values[0]
-
-    _delete_cluster(cluster_name, namespace)
-
-    with user_output:
-        user_output.clear_output()
-        print(
-            f"Cluster {cluster_name} in the {namespace} namespace was deleted successfully."
-        )
-
-    # Refresh the dataframe
-    new_df = _fetch_cluster_data(namespace)
-    if new_df.empty:
-        classification_widget.close()
-        delete_button.close()
-        list_jobs_button.close()
-        ray_dashboard_button.close()
-        with raycluster_data_output:
-            raycluster_data_output.clear_output()
-            print(f"No clusters found in the {namespace} namespace.")
-    else:
-        classification_widget.options = new_df["Name"].tolist()
-
-
-def _on_ray_dashboard_button_click(
-    b,
-    classification_widget: widgets.ToggleButtons,
-    ray_clusters_df: pd.DataFrame,
-    user_output: widgets.Output,
-    url_output: widgets.Output,
-):
-    """
-    _on_ray_dashboard_button_click handles the event when the Open Ray Dashboard button is clicked, opening the Ray Dashboard in a new tab
-    """
-    from codeflare_sdk.cluster import Cluster
-
-    cluster_name = classification_widget.value
-    namespace = ray_clusters_df[ray_clusters_df["Name"] == classification_widget.value][
-        "Namespace"
-    ].values[0]
-
-    # Suppress from Cluster Object initialisation widgets and outputs
-    with widgets.Output(), contextlib.redirect_stdout(
-        io.StringIO()
-    ), contextlib.redirect_stderr(io.StringIO()):
-        cluster = Cluster(ClusterConfiguration(cluster_name, namespace))
-    dashboard_url = cluster.cluster_dashboard_uri()
-
-    with user_output:
-        user_output.clear_output()
-        print(f"Opening Ray Dashboard for {cluster_name} cluster:\n{dashboard_url}")
-    with url_output:
-        display(Javascript(f'window.open("{dashboard_url}", "_blank");'))
-
-
-def _on_list_jobs_button_click(
-    b,
-    classification_widget: widgets.ToggleButtons,
-    ray_clusters_df: pd.DataFrame,
-    user_output: widgets.Output,
-    url_output: widgets.Output,
-):
-    """
-    _on_list_jobs_button_click handles the event when the View Jobs button is clicked, opening the Ray Jobs Dashboard in a new tab
-    """
-    from codeflare_sdk.cluster import Cluster
-
-    cluster_name = classification_widget.value
-    namespace = ray_clusters_df[ray_clusters_df["Name"] == classification_widget.value][
-        "Namespace"
-    ].values[0]
-
-    # Suppress from Cluster Object initialisation widgets and outputs
-    with widgets.Output(), contextlib.redirect_stdout(
-        io.StringIO()
-    ), contextlib.redirect_stderr(io.StringIO()):
-        cluster = Cluster(ClusterConfiguration(cluster_name, namespace))
-    dashboard_url = cluster.cluster_dashboard_uri()
-
-    with user_output:
-        user_output.clear_output()
-        print(
-            f"Opening Ray Jobs Dashboard for {cluster_name} cluster:\n{dashboard_url}/#/jobs"
-        )
-    with url_output:
-        display(Javascript(f'window.open("{dashboard_url}/#/jobs", "_blank");'))
-
-
-def _delete_cluster(
-    cluster_name: str,
-    namespace: str,
-    timeout: int = 5,
-    interval: int = 1,
-):
-    """
-    _delete_cluster function deletes the cluster with the given name and namespace.
-    It optionally waits for the cluster to be deleted.
-    """
-    from .cluster import _check_aw_exists
-
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-
-        if _check_aw_exists(cluster_name, namespace):
-            api_instance.delete_namespaced_custom_object(
-                group="workload.codeflare.dev",
-                version="v1beta2",
-                namespace=namespace,
-                plural="appwrappers",
-                name=cluster_name,
-            )
-            group = "workload.codeflare.dev"
-            version = "v1beta2"
-            plural = "appwrappers"
-        else:
-            api_instance.delete_namespaced_custom_object(
-                group="ray.io",
-                version="v1",
-                namespace=namespace,
-                plural="rayclusters",
-                name=cluster_name,
-            )
-            group = "ray.io"
-            version = "v1"
-            plural = "rayclusters"
-
-        # Wait for the resource to be deleted
-        while timeout > 0:
-            try:
-                api_instance.get_namespaced_custom_object(
-                    group=group,
-                    version=version,
-                    namespace=namespace,
-                    plural=plural,
-                    name=cluster_name,
-                )
-                # Retry if resource still exists
-                time.sleep(interval)
-                timeout -= interval
-                if timeout <= 0:
-                    raise TimeoutError(
-                        f"Timeout waiting for {cluster_name} to be deleted."
-                    )
-            except ApiException as e:
-                # Resource is deleted
-                if e.status == 404:
-                    break
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-def _fetch_cluster_data(namespace):
-    """
-    _fetch_cluster_data function fetches all clusters and their spec in a given namespace and returns a DataFrame.
-    """
-    from .cluster import list_all_clusters
-
-    rayclusters = list_all_clusters(namespace, False)
-    if not rayclusters:
-        return pd.DataFrame()
-    names = [item.name for item in rayclusters]
-    namespaces = [item.namespace for item in rayclusters]
-    num_workers = [item.num_workers for item in rayclusters]
-    head_extended_resources = [
-        f"{list(item.head_extended_resources.keys())[0]}: {list(item.head_extended_resources.values())[0]}"
-        if item.head_extended_resources
-        else "0"
-        for item in rayclusters
-    ]
-    worker_extended_resources = [
-        f"{list(item.worker_extended_resources.keys())[0]}: {list(item.worker_extended_resources.values())[0]}"
-        if item.worker_extended_resources
-        else "0"
-        for item in rayclusters
-    ]
-    head_cpu_requests = [
-        item.head_cpu_requests if item.head_cpu_requests else 0 for item in rayclusters
-    ]
-    head_cpu_limits = [
-        item.head_cpu_limits if item.head_cpu_limits else 0 for item in rayclusters
-    ]
-    head_cpu_rl = [
-        f"{requests}~{limits}"
-        for requests, limits in zip(head_cpu_requests, head_cpu_limits)
-    ]
-    head_mem_requests = [
-        item.head_mem_requests if item.head_mem_requests else 0 for item in rayclusters
-    ]
-    head_mem_limits = [
-        item.head_mem_limits if item.head_mem_limits else 0 for item in rayclusters
-    ]
-    head_mem_rl = [
-        f"{requests}~{limits}"
-        for requests, limits in zip(head_mem_requests, head_mem_limits)
-    ]
-    worker_cpu_requests = [
-        item.worker_cpu_requests if item.worker_cpu_requests else 0
-        for item in rayclusters
-    ]
-    worker_cpu_limits = [
-        item.worker_cpu_limits if item.worker_cpu_limits else 0 for item in rayclusters
-    ]
-    worker_cpu_rl = [
-        f"{requests}~{limits}"
-        for requests, limits in zip(worker_cpu_requests, worker_cpu_limits)
-    ]
-    worker_mem_requests = [
-        item.worker_mem_requests if item.worker_mem_requests else 0
-        for item in rayclusters
-    ]
-    worker_mem_limits = [
-        item.worker_mem_limits if item.worker_mem_limits else 0 for item in rayclusters
-    ]
-    worker_mem_rl = [
-        f"{requests}~{limits}"
-        for requests, limits in zip(worker_mem_requests, worker_mem_limits)
-    ]
-    status = [item.status.name for item in rayclusters]
-
-    status = [_format_status(item.status) for item in rayclusters]
-
-    data = {
-        "Name": names,
-        "Namespace": namespaces,
-        "Num Workers": num_workers,
-        "Head GPUs": head_extended_resources,
-        "Worker GPUs": worker_extended_resources,
-        "Head CPU Req~Lim": head_cpu_rl,
-        "Head Memory Req~Lim": head_mem_rl,
-        "Worker CPU Req~Lim": worker_cpu_rl,
-        "Worker Memory Req~Lim": worker_mem_rl,
-        "status": status,
-    }
-    return pd.DataFrame(data)
-
-
-def _format_status(status):
-    """
-    _format_status function formats the status enum.
-    """
-    status_map = {
-        RayClusterStatus.READY: '<span style="color: green;">Ready ✓</span>',
-        RayClusterStatus.SUSPENDED: '<span style="color: #007BFF;">Suspended ❄️</span>',
-        RayClusterStatus.FAILED: '<span style="color: red;">Failed ✗</span>',
-        RayClusterStatus.UNHEALTHY: '<span style="color: purple;">Unhealthy</span>',
-        RayClusterStatus.UNKNOWN: '<span style="color: purple;">Unknown</span>',
-    }
-    return status_map.get(status, status)
-
-
-
-
-
-
-
-

Functions

-
-
-def cluster_up_down_buttons(cluster: codeflare_sdk.cluster.Cluster) ‑> ipywidgets.widgets.widget_button.Button -
-
-

The cluster_up_down_buttons function returns two button widgets for a create and delete button. -The function uses the appwrapper bool to distinguish between resource type for the tool tip.

-
- -Expand source code - -
def cluster_up_down_buttons(cluster: "codeflare_sdk.cluster.Cluster") -> widgets.Button:
-    """
-    The cluster_up_down_buttons function returns two button widgets for a create and delete button.
-    The function uses the appwrapper bool to distinguish between resource type for the tool tip.
-    """
-    resource = "Ray Cluster"
-    if cluster.config.appwrapper:
-        resource = "AppWrapper"
-
-    up_button = widgets.Button(
-        description="Cluster Up",
-        tooltip=f"Create the {resource}",
-        icon="play",
-    )
-
-    delete_button = widgets.Button(
-        description="Cluster Down",
-        tooltip=f"Delete the {resource}",
-        icon="trash",
-    )
-
-    wait_ready_check = wait_ready_check_box()
-    output = widgets.Output()
-
-    # Display the buttons in an HBox wrapped in a VBox which includes the wait_ready Checkbox
-    button_display = widgets.HBox([up_button, delete_button])
-    display(widgets.VBox([button_display, wait_ready_check]), output)
-
-    def on_up_button_clicked(b):  # Handle the up button click event
-        with output:
-            output.clear_output()
-            cluster.up()
-
-            # If the wait_ready Checkbox is clicked(value == True) trigger the wait_ready function
-            if wait_ready_check.value:
-                cluster.wait_ready()
-
-    def on_down_button_clicked(b):  # Handle the down button click event
-        with output:
-            output.clear_output()
-            cluster.down()
-
-    up_button.on_click(on_up_button_clicked)
-    delete_button.on_click(on_down_button_clicked)
-
-
-
-def is_notebook() ‑> bool -
-
-

The is_notebook function checks if Jupyter Notebook environment variables exist in the given environment and return True/False based on that.

-
- -Expand source code - -
def is_notebook() -> bool:
-    """
-    The is_notebook function checks if Jupyter Notebook environment variables exist in the given environment and return True/False based on that.
-    """
-    if (
-        "PYDEVD_IPYTHON_COMPATIBLE_DEBUGGING" in os.environ
-        or "JPY_SESSION_NAME" in os.environ
-    ):  # If running Jupyter NBs in VsCode or RHOAI/ODH display UI buttons
-        return True
-    else:
-        return False
-
-
-
-def view_clusters(namespace: str = None) -
-
-

view_clusters function will display existing clusters with their specs, and handle user interactions.

-
- -Expand source code - -
def view_clusters(namespace: str = None):
-    """
-    view_clusters function will display existing clusters with their specs, and handle user interactions.
-    """
-    if not is_notebook():
-        warnings.warn(
-            "view_clusters can only be used in a Jupyter Notebook environment."
-        )
-        return  # Exit function if not in Jupyter Notebook
-
-    from .cluster import get_current_namespace
-
-    if not namespace:
-        namespace = get_current_namespace()
-
-    user_output = widgets.Output()
-    raycluster_data_output = widgets.Output()
-    url_output = widgets.Output()
-
-    ray_clusters_df = _fetch_cluster_data(namespace)
-    if ray_clusters_df.empty:
-        print(f"No clusters found in the {namespace} namespace.")
-        return
-
-    classification_widget = widgets.ToggleButtons(
-        options=ray_clusters_df["Name"].tolist(),
-        value=ray_clusters_df["Name"].tolist()[0],
-        description="Select an existing cluster:",
-    )
-    # Setting the initial value to trigger the event handler to display the cluster details.
-    initial_value = classification_widget.value
-    _on_cluster_click(
-        {"new": initial_value}, raycluster_data_output, namespace, classification_widget
-    )
-    classification_widget.observe(
-        lambda selection_change: _on_cluster_click(
-            selection_change, raycluster_data_output, namespace, classification_widget
-        ),
-        names="value",
-    )
-
-    # UI table buttons
-    delete_button = widgets.Button(
-        description="Delete Cluster",
-        icon="trash",
-        tooltip="Delete the selected cluster",
-    )
-    delete_button.on_click(
-        lambda b: _on_delete_button_click(
-            b,
-            classification_widget,
-            ray_clusters_df,
-            raycluster_data_output,
-            user_output,
-            delete_button,
-            list_jobs_button,
-            ray_dashboard_button,
-        )
-    )
-
-    list_jobs_button = widgets.Button(
-        description="View Jobs", icon="suitcase", tooltip="Open the Ray Job Dashboard"
-    )
-    list_jobs_button.on_click(
-        lambda b: _on_list_jobs_button_click(
-            b, classification_widget, ray_clusters_df, user_output, url_output
-        )
-    )
-
-    ray_dashboard_button = widgets.Button(
-        description="Open Ray Dashboard",
-        icon="dashboard",
-        tooltip="Open the Ray Dashboard in a new tab",
-        layout=widgets.Layout(width="auto"),
-    )
-    ray_dashboard_button.on_click(
-        lambda b: _on_ray_dashboard_button_click(
-            b, classification_widget, ray_clusters_df, user_output, url_output
-        )
-    )
-
-    display(widgets.VBox([classification_widget, raycluster_data_output]))
-    display(
-        widgets.HBox([delete_button, list_jobs_button, ray_dashboard_button]),
-        url_output,
-        user_output,
-    )
-
-
-
-def wait_ready_check_box() -
-
-

The wait_ready_check_box function will return a checkbox widget used for waiting for the resource to be in the state READY.

-
- -Expand source code - -
def wait_ready_check_box():
-    """
-    The wait_ready_check_box function will return a checkbox widget used for waiting for the resource to be in the state READY.
-    """
-    wait_ready_check_box = widgets.Checkbox(
-        False,
-        description="Wait for Cluster?",
-    )
-    return wait_ready_check_box
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/index.html b/docs/detailed-documentation/index.html deleted file mode 100644 index 450007196..000000000 --- a/docs/detailed-documentation/index.html +++ /dev/null @@ -1,106 +0,0 @@ - - - - - - -codeflare_sdk API documentation - - - - - - - - - - - -
-
-
-

Package codeflare_sdk

-
-
-
- -Expand source code - -
from .cluster import (
-    Authentication,
-    KubeConfiguration,
-    TokenAuthentication,
-    KubeConfigFileAuthentication,
-    AWManager,
-    Cluster,
-    ClusterConfiguration,
-    RayClusterStatus,
-    AppWrapperStatus,
-    CodeFlareClusterStatus,
-    RayCluster,
-    AppWrapper,
-    get_cluster,
-    list_all_queued,
-    list_all_clusters,
-    view_clusters,
-)
-
-from .job import RayJobClient
-
-from .utils import generate_cert
-from .utils.demos import copy_demo_nbs
-
-from importlib.metadata import version, PackageNotFoundError
-
-try:
-    __version__ = version("codeflare-sdk")  # use metadata associated with built package
-
-except PackageNotFoundError:
-    __version__ = "v0.0.0"
-
-
-
-

Sub-modules

-
-
codeflare_sdk.cluster
-
-
-
-
codeflare_sdk.job
-
-
-
-
codeflare_sdk.utils
-
-
-
-
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/job/index.html b/docs/detailed-documentation/job/index.html deleted file mode 100644 index ccfc679de..000000000 --- a/docs/detailed-documentation/job/index.html +++ /dev/null @@ -1,72 +0,0 @@ - - - - - - -codeflare_sdk.job API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.job

-
-
-
- -Expand source code - -
from .ray_jobs import RayJobClient
-
-
-
-

Sub-modules

-
-
codeflare_sdk.job.ray_jobs
-
-

The ray_jobs sub-module contains methods needed to submit jobs and connect to Ray Clusters that were not created by CodeFlare. -The SDK acts as a …

-
-
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/job/ray_jobs.html b/docs/detailed-documentation/job/ray_jobs.html deleted file mode 100644 index 20002e27e..000000000 --- a/docs/detailed-documentation/job/ray_jobs.html +++ /dev/null @@ -1,585 +0,0 @@ - - - - - - -codeflare_sdk.job.ray_jobs API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.job.ray_jobs

-
-
-

The ray_jobs sub-module contains methods needed to submit jobs and connect to Ray Clusters that were not created by CodeFlare. -The SDK acts as a wrapper for the Ray Job Submission Client.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-The ray_jobs sub-module contains methods needed to submit jobs and connect to Ray Clusters that were not created by CodeFlare.
-The SDK acts as a wrapper for the Ray Job Submission Client.
-"""
-
-from ray.job_submission import JobSubmissionClient
-from ray.dashboard.modules.job.pydantic_models import JobDetails
-from typing import Iterator, Optional, Dict, Any, Union, List
-
-
-class RayJobClient:
-    """
-    A class that functions as a wrapper for the Ray Job Submission Client.
-
-    parameters:
-    address -- Either (1) the address of the Ray cluster, or (2) the HTTP address of the dashboard server on the head node, e.g. “http://<head-node-ip>:8265”. In case (1) it must be specified as an address that can be passed to ray.init(),
-    e.g. a Ray Client address (ray://<head_node_host>:10001), or “auto”, or “localhost:<port>”. If unspecified, will try to connect to a running local Ray cluster. This argument is always overridden by the RAY_ADDRESS environment variable.
-    create_cluster_if_needed -- Indicates whether the cluster at the specified address needs to already be running. Ray doesn't start a cluster before interacting with jobs, but third-party job managers may do so.
-    cookies -- Cookies to use when sending requests to the HTTP job server.
-    metadata -- Arbitrary metadata to store along with all jobs. New metadata specified per job will be merged with the global metadata provided here via a simple dict update.
-    headers -- Headers to use when sending requests to the HTTP job server, used for cases like authentication to a remote cluster.
-    verify -- Boolean indication to verify the server's TLS certificate or a path to a file or directory of trusted certificates. Default: True.
-    """
-
-    def __init__(
-        self,
-        address: Optional[str] = None,
-        create_cluster_if_needed: bool = False,
-        cookies: Optional[Dict[str, Any]] = None,
-        metadata: Optional[Dict[str, Any]] = None,
-        headers: Optional[Dict[str, Any]] = None,
-        verify: Optional[Union[str, bool]] = True,
-    ):
-        self.rayJobClient = JobSubmissionClient(
-            address=address,
-            create_cluster_if_needed=create_cluster_if_needed,
-            cookies=cookies,
-            metadata=metadata,
-            headers=headers,
-            verify=verify,
-        )
-
-    def submit_job(
-        self,
-        entrypoint: str,
-        job_id: Optional[str] = None,
-        runtime_env: Optional[Dict[str, Any]] = None,
-        metadata: Optional[Dict[str, str]] = None,
-        submission_id: Optional[str] = None,
-        entrypoint_num_cpus: Optional[Union[int, float]] = None,
-        entrypoint_num_gpus: Optional[Union[int, float]] = None,
-        entrypoint_memory: Optional[int] = None,
-        entrypoint_resources: Optional[Dict[str, float]] = None,
-    ) -> str:
-        """
-        Method for submitting jobs to a Ray Cluster and returning the job id with entrypoint being a mandatory field.
-
-        Parameters:
-        entrypoint -- The shell command to run for this job.
-        submission_id -- A unique ID for this job.
-        runtime_env -- The runtime environment to install and run this job in.
-        metadata -- Arbitrary data to store along with this job.
-        job_id -- DEPRECATED. This has been renamed to submission_id
-        entrypoint_num_cpus -- The quantity of CPU cores to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-        entrypoint_num_gpus -- The quantity of GPUs to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-        entrypoint_memory –- The quantity of memory to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-        entrypoint_resources -- The quantity of custom resources to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it.
-        """
-        return self.rayJobClient.submit_job(
-            entrypoint=entrypoint,
-            job_id=job_id,
-            runtime_env=runtime_env,
-            metadata=metadata,
-            submission_id=submission_id,
-            entrypoint_num_cpus=entrypoint_num_cpus,
-            entrypoint_num_gpus=entrypoint_num_gpus,
-            entrypoint_memory=entrypoint_memory,
-            entrypoint_resources=entrypoint_resources,
-        )
-
-    def delete_job(self, job_id: str) -> (bool, str):
-        """
-        Method for deleting jobs with the job id being a mandatory field.
-        """
-        deletion_status = self.rayJobClient.delete_job(job_id=job_id)
-
-        if deletion_status:
-            message = f"Successfully deleted Job {job_id}"
-        else:
-            message = f"Failed to delete Job {job_id}"
-
-        return deletion_status, message
-
-    def get_address(self) -> str:
-        """
-        Method for getting the address from the RayJobClient
-        """
-        return self.rayJobClient.get_address()
-
-    def get_job_info(self, job_id: str):
-        """
-        Method for getting the job info with the job id being a mandatory field.
-        """
-        return self.rayJobClient.get_job_info(job_id=job_id)
-
-    def get_job_logs(self, job_id: str) -> str:
-        """
-        Method for getting the job logs with the job id being a mandatory field.
-        """
-        return self.rayJobClient.get_job_logs(job_id=job_id)
-
-    def get_job_status(self, job_id: str) -> str:
-        """
-        Method for getting the job's status with the job id being a mandatory field.
-        """
-        return self.rayJobClient.get_job_status(job_id=job_id)
-
-    def list_jobs(self) -> List[JobDetails]:
-        """
-        Method for getting a list of current jobs in the Ray Cluster.
-        """
-        return self.rayJobClient.list_jobs()
-
-    def stop_job(self, job_id: str) -> (bool, str):
-        """
-        Method for stopping a job with the job id being a mandatory field.
-        """
-        stop_job_status = self.rayJobClient.stop_job(job_id=job_id)
-        if stop_job_status:
-            message = f"Successfully stopped Job {job_id}"
-        else:
-            message = f"Failed to stop Job, {job_id} could have already completed."
-        return stop_job_status, message
-
-    def tail_job_logs(self, job_id: str) -> Iterator[str]:
-        """
-        Method for getting an iterator that follows the logs of a job with the job id being a mandatory field.
-        """
-        return self.rayJobClient.tail_job_logs(job_id=job_id)
-
-
-
-
-
-
-
-
-
-

Classes

-
-
-class RayJobClient -(address: Optional[str] = None, create_cluster_if_needed: bool = False, cookies: Optional[Dict[str, Any]] = None, metadata: Optional[Dict[str, Any]] = None, headers: Optional[Dict[str, Any]] = None, verify: Union[str, bool, ForwardRef(None)] = True) -
-
-

A class that functions as a wrapper for the Ray Job Submission Client.

-

parameters: -address – Either (1) the address of the Ray cluster, or (2) the HTTP address of the dashboard server on the head node, e.g. “>:8265”. In case (1) it must be specified as an address that can be passed to ray.init(), -e.g. a Ray Client address (ray://:10001), or “auto”, or “localhost:”. If unspecified, will try to connect to a running local Ray cluster. This argument is always overridden by the RAY_ADDRESS environment variable. -create_cluster_if_needed – Indicates whether the cluster at the specified address needs to already be running. Ray doesn't start a cluster before interacting with jobs, but third-party job managers may do so. -cookies – Cookies to use when sending requests to the HTTP job server. -metadata – Arbitrary metadata to store along with all jobs. New metadata specified per job will be merged with the global metadata provided here via a simple dict update. -headers – Headers to use when sending requests to the HTTP job server, used for cases like authentication to a remote cluster. -verify – Boolean indication to verify the server's TLS certificate or a path to a file or directory of trusted certificates. Default: True.

-
- -Expand source code - -
class RayJobClient:
-    """
-    A class that functions as a wrapper for the Ray Job Submission Client.
-
-    parameters:
-    address -- Either (1) the address of the Ray cluster, or (2) the HTTP address of the dashboard server on the head node, e.g. “http://<head-node-ip>:8265”. In case (1) it must be specified as an address that can be passed to ray.init(),
-    e.g. a Ray Client address (ray://<head_node_host>:10001), or “auto”, or “localhost:<port>”. If unspecified, will try to connect to a running local Ray cluster. This argument is always overridden by the RAY_ADDRESS environment variable.
-    create_cluster_if_needed -- Indicates whether the cluster at the specified address needs to already be running. Ray doesn't start a cluster before interacting with jobs, but third-party job managers may do so.
-    cookies -- Cookies to use when sending requests to the HTTP job server.
-    metadata -- Arbitrary metadata to store along with all jobs. New metadata specified per job will be merged with the global metadata provided here via a simple dict update.
-    headers -- Headers to use when sending requests to the HTTP job server, used for cases like authentication to a remote cluster.
-    verify -- Boolean indication to verify the server's TLS certificate or a path to a file or directory of trusted certificates. Default: True.
-    """
-
-    def __init__(
-        self,
-        address: Optional[str] = None,
-        create_cluster_if_needed: bool = False,
-        cookies: Optional[Dict[str, Any]] = None,
-        metadata: Optional[Dict[str, Any]] = None,
-        headers: Optional[Dict[str, Any]] = None,
-        verify: Optional[Union[str, bool]] = True,
-    ):
-        self.rayJobClient = JobSubmissionClient(
-            address=address,
-            create_cluster_if_needed=create_cluster_if_needed,
-            cookies=cookies,
-            metadata=metadata,
-            headers=headers,
-            verify=verify,
-        )
-
-    def submit_job(
-        self,
-        entrypoint: str,
-        job_id: Optional[str] = None,
-        runtime_env: Optional[Dict[str, Any]] = None,
-        metadata: Optional[Dict[str, str]] = None,
-        submission_id: Optional[str] = None,
-        entrypoint_num_cpus: Optional[Union[int, float]] = None,
-        entrypoint_num_gpus: Optional[Union[int, float]] = None,
-        entrypoint_memory: Optional[int] = None,
-        entrypoint_resources: Optional[Dict[str, float]] = None,
-    ) -> str:
-        """
-        Method for submitting jobs to a Ray Cluster and returning the job id with entrypoint being a mandatory field.
-
-        Parameters:
-        entrypoint -- The shell command to run for this job.
-        submission_id -- A unique ID for this job.
-        runtime_env -- The runtime environment to install and run this job in.
-        metadata -- Arbitrary data to store along with this job.
-        job_id -- DEPRECATED. This has been renamed to submission_id
-        entrypoint_num_cpus -- The quantity of CPU cores to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-        entrypoint_num_gpus -- The quantity of GPUs to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-        entrypoint_memory –- The quantity of memory to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-        entrypoint_resources -- The quantity of custom resources to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it.
-        """
-        return self.rayJobClient.submit_job(
-            entrypoint=entrypoint,
-            job_id=job_id,
-            runtime_env=runtime_env,
-            metadata=metadata,
-            submission_id=submission_id,
-            entrypoint_num_cpus=entrypoint_num_cpus,
-            entrypoint_num_gpus=entrypoint_num_gpus,
-            entrypoint_memory=entrypoint_memory,
-            entrypoint_resources=entrypoint_resources,
-        )
-
-    def delete_job(self, job_id: str) -> (bool, str):
-        """
-        Method for deleting jobs with the job id being a mandatory field.
-        """
-        deletion_status = self.rayJobClient.delete_job(job_id=job_id)
-
-        if deletion_status:
-            message = f"Successfully deleted Job {job_id}"
-        else:
-            message = f"Failed to delete Job {job_id}"
-
-        return deletion_status, message
-
-    def get_address(self) -> str:
-        """
-        Method for getting the address from the RayJobClient
-        """
-        return self.rayJobClient.get_address()
-
-    def get_job_info(self, job_id: str):
-        """
-        Method for getting the job info with the job id being a mandatory field.
-        """
-        return self.rayJobClient.get_job_info(job_id=job_id)
-
-    def get_job_logs(self, job_id: str) -> str:
-        """
-        Method for getting the job logs with the job id being a mandatory field.
-        """
-        return self.rayJobClient.get_job_logs(job_id=job_id)
-
-    def get_job_status(self, job_id: str) -> str:
-        """
-        Method for getting the job's status with the job id being a mandatory field.
-        """
-        return self.rayJobClient.get_job_status(job_id=job_id)
-
-    def list_jobs(self) -> List[JobDetails]:
-        """
-        Method for getting a list of current jobs in the Ray Cluster.
-        """
-        return self.rayJobClient.list_jobs()
-
-    def stop_job(self, job_id: str) -> (bool, str):
-        """
-        Method for stopping a job with the job id being a mandatory field.
-        """
-        stop_job_status = self.rayJobClient.stop_job(job_id=job_id)
-        if stop_job_status:
-            message = f"Successfully stopped Job {job_id}"
-        else:
-            message = f"Failed to stop Job, {job_id} could have already completed."
-        return stop_job_status, message
-
-    def tail_job_logs(self, job_id: str) -> Iterator[str]:
-        """
-        Method for getting an iterator that follows the logs of a job with the job id being a mandatory field.
-        """
-        return self.rayJobClient.tail_job_logs(job_id=job_id)
-
-

Methods

-
-
-def delete_job(self, job_id: str) ‑> (, ) -
-
-

Method for deleting jobs with the job id being a mandatory field.

-
- -Expand source code - -
def delete_job(self, job_id: str) -> (bool, str):
-    """
-    Method for deleting jobs with the job id being a mandatory field.
-    """
-    deletion_status = self.rayJobClient.delete_job(job_id=job_id)
-
-    if deletion_status:
-        message = f"Successfully deleted Job {job_id}"
-    else:
-        message = f"Failed to delete Job {job_id}"
-
-    return deletion_status, message
-
-
-
-def get_address(self) ‑> str -
-
-

Method for getting the address from the RayJobClient

-
- -Expand source code - -
def get_address(self) -> str:
-    """
-    Method for getting the address from the RayJobClient
-    """
-    return self.rayJobClient.get_address()
-
-
-
-def get_job_info(self, job_id: str) -
-
-

Method for getting the job info with the job id being a mandatory field.

-
- -Expand source code - -
def get_job_info(self, job_id: str):
-    """
-    Method for getting the job info with the job id being a mandatory field.
-    """
-    return self.rayJobClient.get_job_info(job_id=job_id)
-
-
-
-def get_job_logs(self, job_id: str) ‑> str -
-
-

Method for getting the job logs with the job id being a mandatory field.

-
- -Expand source code - -
def get_job_logs(self, job_id: str) -> str:
-    """
-    Method for getting the job logs with the job id being a mandatory field.
-    """
-    return self.rayJobClient.get_job_logs(job_id=job_id)
-
-
-
-def get_job_status(self, job_id: str) ‑> str -
-
-

Method for getting the job's status with the job id being a mandatory field.

-
- -Expand source code - -
def get_job_status(self, job_id: str) -> str:
-    """
-    Method for getting the job's status with the job id being a mandatory field.
-    """
-    return self.rayJobClient.get_job_status(job_id=job_id)
-
-
-
-def list_jobs(self) ‑> List[ray.dashboard.modules.job.pydantic_models.JobDetails] -
-
-

Method for getting a list of current jobs in the Ray Cluster.

-
- -Expand source code - -
def list_jobs(self) -> List[JobDetails]:
-    """
-    Method for getting a list of current jobs in the Ray Cluster.
-    """
-    return self.rayJobClient.list_jobs()
-
-
-
-def stop_job(self, job_id: str) ‑> (, ) -
-
-

Method for stopping a job with the job id being a mandatory field.

-
- -Expand source code - -
def stop_job(self, job_id: str) -> (bool, str):
-    """
-    Method for stopping a job with the job id being a mandatory field.
-    """
-    stop_job_status = self.rayJobClient.stop_job(job_id=job_id)
-    if stop_job_status:
-        message = f"Successfully stopped Job {job_id}"
-    else:
-        message = f"Failed to stop Job, {job_id} could have already completed."
-    return stop_job_status, message
-
-
-
-def submit_job(self, entrypoint: str, job_id: Optional[str] = None, runtime_env: Optional[Dict[str, Any]] = None, metadata: Optional[Dict[str, str]] = None, submission_id: Optional[str] = None, entrypoint_num_cpus: Union[int, float, ForwardRef(None)] = None, entrypoint_num_gpus: Union[int, float, ForwardRef(None)] = None, entrypoint_memory: Optional[int] = None, entrypoint_resources: Optional[Dict[str, float]] = None) ‑> str -
-
-

Method for submitting jobs to a Ray Cluster and returning the job id with entrypoint being a mandatory field.

-

Parameters: -entrypoint – The shell command to run for this job. -submission_id – A unique ID for this job. -runtime_env – The runtime environment to install and run this job in. -metadata – Arbitrary data to store along with this job. -job_id – DEPRECATED. This has been renamed to submission_id -entrypoint_num_cpus – The quantity of CPU cores to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0. -entrypoint_num_gpus – The quantity of GPUs to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0. -entrypoint_memory –- The quantity of memory to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0. -entrypoint_resources – The quantity of custom resources to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it.

-
- -Expand source code - -
def submit_job(
-    self,
-    entrypoint: str,
-    job_id: Optional[str] = None,
-    runtime_env: Optional[Dict[str, Any]] = None,
-    metadata: Optional[Dict[str, str]] = None,
-    submission_id: Optional[str] = None,
-    entrypoint_num_cpus: Optional[Union[int, float]] = None,
-    entrypoint_num_gpus: Optional[Union[int, float]] = None,
-    entrypoint_memory: Optional[int] = None,
-    entrypoint_resources: Optional[Dict[str, float]] = None,
-) -> str:
-    """
-    Method for submitting jobs to a Ray Cluster and returning the job id with entrypoint being a mandatory field.
-
-    Parameters:
-    entrypoint -- The shell command to run for this job.
-    submission_id -- A unique ID for this job.
-    runtime_env -- The runtime environment to install and run this job in.
-    metadata -- Arbitrary data to store along with this job.
-    job_id -- DEPRECATED. This has been renamed to submission_id
-    entrypoint_num_cpus -- The quantity of CPU cores to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-    entrypoint_num_gpus -- The quantity of GPUs to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-    entrypoint_memory –- The quantity of memory to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it. Defaults to 0.
-    entrypoint_resources -- The quantity of custom resources to reserve for the execution of the entrypoint command, separately from any tasks or actors launched by it.
-    """
-    return self.rayJobClient.submit_job(
-        entrypoint=entrypoint,
-        job_id=job_id,
-        runtime_env=runtime_env,
-        metadata=metadata,
-        submission_id=submission_id,
-        entrypoint_num_cpus=entrypoint_num_cpus,
-        entrypoint_num_gpus=entrypoint_num_gpus,
-        entrypoint_memory=entrypoint_memory,
-        entrypoint_resources=entrypoint_resources,
-    )
-
-
-
-def tail_job_logs(self, job_id: str) ‑> Iterator[str] -
-
-

Method for getting an iterator that follows the logs of a job with the job id being a mandatory field.

-
- -Expand source code - -
def tail_job_logs(self, job_id: str) -> Iterator[str]:
-    """
-    Method for getting an iterator that follows the logs of a job with the job id being a mandatory field.
-    """
-    return self.rayJobClient.tail_job_logs(job_id=job_id)
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/utils/demos.html b/docs/detailed-documentation/utils/demos.html deleted file mode 100644 index e0dc5a8e7..000000000 --- a/docs/detailed-documentation/utils/demos.html +++ /dev/null @@ -1,138 +0,0 @@ - - - - - - -codeflare_sdk.utils.demos API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.utils.demos

-
-
-
- -Expand source code - -
import pathlib
-import shutil
-
-package_dir = pathlib.Path(__file__).parent.parent.resolve()
-demo_dir = f"{package_dir}/demo-notebooks"
-
-
-def copy_demo_nbs(dir: str = "./demo-notebooks", overwrite: bool = False):
-    """
-    Copy the demo notebooks from the package to the current working directory
-
-    overwrite=True will overwrite any files that exactly match files written by copy_demo_nbs in the target directory.
-    Any files that exist in the directory that don't match these values will remain untouched.
-
-    Args:
-        dir (str): The directory to copy the demo notebooks to. Defaults to "./demo-notebooks". overwrite (bool):
-        overwrite (bool): Whether to overwrite files in the directory if it already exists. Defaults to False.
-    Raises:
-        FileExistsError: If the directory already exists.
-    """
-    # does dir exist already?
-    if overwrite is False and pathlib.Path(dir).exists():
-        raise FileExistsError(
-            f"Directory {dir} already exists. Please remove it or provide a different location."
-        )
-
-    shutil.copytree(demo_dir, dir, dirs_exist_ok=True)
-
-
-
-
-
-
-
-

Functions

-
-
-def copy_demo_nbs(dir: str = './demo-notebooks', overwrite: bool = False) -
-
-

Copy the demo notebooks from the package to the current working directory

-

overwrite=True will overwrite any files that exactly match files written by copy_demo_nbs in the target directory. -Any files that exist in the directory that don't match these values will remain untouched.

-

Args

-
-
dir : str
-
The directory to copy the demo notebooks to. Defaults to "./demo-notebooks". overwrite (bool):
-
overwrite : bool
-
Whether to overwrite files in the directory if it already exists. Defaults to False.
-
-

Raises

-
-
FileExistsError
-
If the directory already exists.
-
-
- -Expand source code - -
def copy_demo_nbs(dir: str = "./demo-notebooks", overwrite: bool = False):
-    """
-    Copy the demo notebooks from the package to the current working directory
-
-    overwrite=True will overwrite any files that exactly match files written by copy_demo_nbs in the target directory.
-    Any files that exist in the directory that don't match these values will remain untouched.
-
-    Args:
-        dir (str): The directory to copy the demo notebooks to. Defaults to "./demo-notebooks". overwrite (bool):
-        overwrite (bool): Whether to overwrite files in the directory if it already exists. Defaults to False.
-    Raises:
-        FileExistsError: If the directory already exists.
-    """
-    # does dir exist already?
-    if overwrite is False and pathlib.Path(dir).exists():
-        raise FileExistsError(
-            f"Directory {dir} already exists. Please remove it or provide a different location."
-        )
-
-    shutil.copytree(demo_dir, dir, dirs_exist_ok=True)
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/utils/generate_cert.html b/docs/detailed-documentation/utils/generate_cert.html deleted file mode 100644 index 01084d84f..000000000 --- a/docs/detailed-documentation/utils/generate_cert.html +++ /dev/null @@ -1,451 +0,0 @@ - - - - - - -codeflare_sdk.utils.generate_cert API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.utils.generate_cert

-
-
-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import base64
-import os
-from cryptography.hazmat.primitives import serialization, hashes
-from cryptography.hazmat.primitives.asymmetric import rsa
-from cryptography import x509
-from cryptography.x509.oid import NameOID
-import datetime
-from ..cluster.auth import config_check, get_api_client
-from kubernetes import client, config
-from .kube_api_helpers import _kube_api_error_handling
-
-
-def generate_ca_cert(days: int = 30):
-    # Generate base64 encoded ca.key and ca.cert
-    # Similar to:
-    # openssl req -x509 -nodes -newkey rsa:2048 -keyout ca.key -days 1826 -out ca.crt -subj '/CN=root-ca'
-    # base64 -i ca.crt -i ca.key
-
-    private_key = rsa.generate_private_key(
-        public_exponent=65537,
-        key_size=2048,
-    )
-
-    key = base64.b64encode(
-        private_key.private_bytes(
-            serialization.Encoding.PEM,
-            serialization.PrivateFormat.PKCS8,
-            serialization.NoEncryption(),
-        )
-    ).decode("utf-8")
-
-    # Generate Certificate
-    one_day = datetime.timedelta(1, 0, 0)
-    public_key = private_key.public_key()
-    builder = (
-        x509.CertificateBuilder()
-        .subject_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "root-ca"),
-                ]
-            )
-        )
-        .issuer_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "root-ca"),
-                ]
-            )
-        )
-        .not_valid_before(datetime.datetime.today() - one_day)
-        .not_valid_after(datetime.datetime.today() + (one_day * days))
-        .serial_number(x509.random_serial_number())
-        .public_key(public_key)
-    )
-    certificate = base64.b64encode(
-        builder.sign(private_key=private_key, algorithm=hashes.SHA256()).public_bytes(
-            serialization.Encoding.PEM
-        )
-    ).decode("utf-8")
-    return key, certificate
-
-
-def get_secret_name(cluster_name, namespace, api_instance):
-    label_selector = f"ray.openshift.ai/cluster-name={cluster_name}"
-    try:
-        secrets = api_instance.list_namespaced_secret(
-            namespace, label_selector=label_selector
-        )
-        for secret in secrets.items:
-            if (
-                f"{cluster_name}-ca-secret-" in secret.metadata.name
-            ):  # Oauth secret share the same label this conditional is to make things more specific
-                return secret.metadata.name
-            else:
-                continue
-        raise KeyError(f"Unable to gather secret name for {cluster_name}")
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-def generate_tls_cert(cluster_name, namespace, days=30):
-    # Create a folder tls-<cluster>-<namespace> and store three files: ca.crt, tls.crt, and tls.key
-    tls_dir = os.path.join(os.getcwd(), f"tls-{cluster_name}-{namespace}")
-    if not os.path.exists(tls_dir):
-        os.makedirs(tls_dir)
-
-    # Similar to:
-    # oc get secret ca-secret-<cluster-name> -o template='{{index .data "ca.key"}}'
-    # oc get secret ca-secret-<cluster-name> -o template='{{index .data "ca.crt"}}'|base64 -d > ${TLSDIR}/ca.crt
-    config_check()
-    v1 = client.CoreV1Api(get_api_client())
-
-    # Secrets have a suffix appended to the end so we must list them and gather the secret that includes cluster_name-ca-secret-
-    secret_name = get_secret_name(cluster_name, namespace, v1)
-    secret = v1.read_namespaced_secret(secret_name, namespace).data
-
-    ca_cert = secret.get("ca.crt")
-    ca_key = secret.get("ca.key")
-
-    with open(os.path.join(tls_dir, "ca.crt"), "w") as f:
-        f.write(base64.b64decode(ca_cert).decode("utf-8"))
-
-    # Generate tls.key and signed tls.cert locally for ray client
-    # Similar to running these commands:
-    # openssl req -nodes -newkey rsa:2048 -keyout ${TLSDIR}/tls.key -out ${TLSDIR}/tls.csr -subj '/CN=local'
-    # cat <<EOF >${TLSDIR}/domain.ext
-    # authorityKeyIdentifier=keyid,issuer
-    # basicConstraints=CA:FALSE
-    # subjectAltName = @alt_names
-    # [alt_names]
-    # DNS.1 = 127.0.0.1
-    # DNS.2 = localhost
-    # EOF
-    # openssl x509 -req -CA ${TLSDIR}/ca.crt -CAkey ${TLSDIR}/ca.key -in ${TLSDIR}/tls.csr -out ${TLSDIR}/tls.crt -days 365 -CAcreateserial -extfile ${TLSDIR}/domain.ext
-    key = rsa.generate_private_key(
-        public_exponent=65537,
-        key_size=2048,
-    )
-
-    tls_key = key.private_bytes(
-        serialization.Encoding.PEM,
-        serialization.PrivateFormat.PKCS8,
-        serialization.NoEncryption(),
-    )
-    with open(os.path.join(tls_dir, "tls.key"), "w") as f:
-        f.write(tls_key.decode("utf-8"))
-
-    one_day = datetime.timedelta(1, 0, 0)
-    tls_cert = (
-        x509.CertificateBuilder()
-        .issuer_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "root-ca"),
-                ]
-            )
-        )
-        .subject_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "local"),
-                ]
-            )
-        )
-        .public_key(key.public_key())
-        .not_valid_before(datetime.datetime.today() - one_day)
-        .not_valid_after(datetime.datetime.today() + (one_day * days))
-        .serial_number(x509.random_serial_number())
-        .add_extension(
-            x509.SubjectAlternativeName(
-                [x509.DNSName("localhost"), x509.DNSName("127.0.0.1")]
-            ),
-            False,
-        )
-        .sign(
-            serialization.load_pem_private_key(base64.b64decode(ca_key), None),
-            hashes.SHA256(),
-        )
-    )
-
-    with open(os.path.join(tls_dir, "tls.crt"), "w") as f:
-        f.write(tls_cert.public_bytes(serialization.Encoding.PEM).decode("utf-8"))
-
-
-def export_env(cluster_name, namespace):
-    tls_dir = os.path.join(os.getcwd(), f"tls-{cluster_name}-{namespace}")
-    os.environ["RAY_USE_TLS"] = "1"
-    os.environ["RAY_TLS_SERVER_CERT"] = os.path.join(tls_dir, "tls.crt")
-    os.environ["RAY_TLS_SERVER_KEY"] = os.path.join(tls_dir, "tls.key")
-    os.environ["RAY_TLS_CA_CERT"] = os.path.join(tls_dir, "ca.crt")
-
-
-
-
-
-
-
-

Functions

-
-
-def export_env(cluster_name, namespace) -
-
-
-
- -Expand source code - -
def export_env(cluster_name, namespace):
-    tls_dir = os.path.join(os.getcwd(), f"tls-{cluster_name}-{namespace}")
-    os.environ["RAY_USE_TLS"] = "1"
-    os.environ["RAY_TLS_SERVER_CERT"] = os.path.join(tls_dir, "tls.crt")
-    os.environ["RAY_TLS_SERVER_KEY"] = os.path.join(tls_dir, "tls.key")
-    os.environ["RAY_TLS_CA_CERT"] = os.path.join(tls_dir, "ca.crt")
-
-
-
-def generate_ca_cert(days: int = 30) -
-
-
-
- -Expand source code - -
def generate_ca_cert(days: int = 30):
-    # Generate base64 encoded ca.key and ca.cert
-    # Similar to:
-    # openssl req -x509 -nodes -newkey rsa:2048 -keyout ca.key -days 1826 -out ca.crt -subj '/CN=root-ca'
-    # base64 -i ca.crt -i ca.key
-
-    private_key = rsa.generate_private_key(
-        public_exponent=65537,
-        key_size=2048,
-    )
-
-    key = base64.b64encode(
-        private_key.private_bytes(
-            serialization.Encoding.PEM,
-            serialization.PrivateFormat.PKCS8,
-            serialization.NoEncryption(),
-        )
-    ).decode("utf-8")
-
-    # Generate Certificate
-    one_day = datetime.timedelta(1, 0, 0)
-    public_key = private_key.public_key()
-    builder = (
-        x509.CertificateBuilder()
-        .subject_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "root-ca"),
-                ]
-            )
-        )
-        .issuer_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "root-ca"),
-                ]
-            )
-        )
-        .not_valid_before(datetime.datetime.today() - one_day)
-        .not_valid_after(datetime.datetime.today() + (one_day * days))
-        .serial_number(x509.random_serial_number())
-        .public_key(public_key)
-    )
-    certificate = base64.b64encode(
-        builder.sign(private_key=private_key, algorithm=hashes.SHA256()).public_bytes(
-            serialization.Encoding.PEM
-        )
-    ).decode("utf-8")
-    return key, certificate
-
-
-
-def generate_tls_cert(cluster_name, namespace, days=30) -
-
-
-
- -Expand source code - -
def generate_tls_cert(cluster_name, namespace, days=30):
-    # Create a folder tls-<cluster>-<namespace> and store three files: ca.crt, tls.crt, and tls.key
-    tls_dir = os.path.join(os.getcwd(), f"tls-{cluster_name}-{namespace}")
-    if not os.path.exists(tls_dir):
-        os.makedirs(tls_dir)
-
-    # Similar to:
-    # oc get secret ca-secret-<cluster-name> -o template='{{index .data "ca.key"}}'
-    # oc get secret ca-secret-<cluster-name> -o template='{{index .data "ca.crt"}}'|base64 -d > ${TLSDIR}/ca.crt
-    config_check()
-    v1 = client.CoreV1Api(get_api_client())
-
-    # Secrets have a suffix appended to the end so we must list them and gather the secret that includes cluster_name-ca-secret-
-    secret_name = get_secret_name(cluster_name, namespace, v1)
-    secret = v1.read_namespaced_secret(secret_name, namespace).data
-
-    ca_cert = secret.get("ca.crt")
-    ca_key = secret.get("ca.key")
-
-    with open(os.path.join(tls_dir, "ca.crt"), "w") as f:
-        f.write(base64.b64decode(ca_cert).decode("utf-8"))
-
-    # Generate tls.key and signed tls.cert locally for ray client
-    # Similar to running these commands:
-    # openssl req -nodes -newkey rsa:2048 -keyout ${TLSDIR}/tls.key -out ${TLSDIR}/tls.csr -subj '/CN=local'
-    # cat <<EOF >${TLSDIR}/domain.ext
-    # authorityKeyIdentifier=keyid,issuer
-    # basicConstraints=CA:FALSE
-    # subjectAltName = @alt_names
-    # [alt_names]
-    # DNS.1 = 127.0.0.1
-    # DNS.2 = localhost
-    # EOF
-    # openssl x509 -req -CA ${TLSDIR}/ca.crt -CAkey ${TLSDIR}/ca.key -in ${TLSDIR}/tls.csr -out ${TLSDIR}/tls.crt -days 365 -CAcreateserial -extfile ${TLSDIR}/domain.ext
-    key = rsa.generate_private_key(
-        public_exponent=65537,
-        key_size=2048,
-    )
-
-    tls_key = key.private_bytes(
-        serialization.Encoding.PEM,
-        serialization.PrivateFormat.PKCS8,
-        serialization.NoEncryption(),
-    )
-    with open(os.path.join(tls_dir, "tls.key"), "w") as f:
-        f.write(tls_key.decode("utf-8"))
-
-    one_day = datetime.timedelta(1, 0, 0)
-    tls_cert = (
-        x509.CertificateBuilder()
-        .issuer_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "root-ca"),
-                ]
-            )
-        )
-        .subject_name(
-            x509.Name(
-                [
-                    x509.NameAttribute(NameOID.COMMON_NAME, "local"),
-                ]
-            )
-        )
-        .public_key(key.public_key())
-        .not_valid_before(datetime.datetime.today() - one_day)
-        .not_valid_after(datetime.datetime.today() + (one_day * days))
-        .serial_number(x509.random_serial_number())
-        .add_extension(
-            x509.SubjectAlternativeName(
-                [x509.DNSName("localhost"), x509.DNSName("127.0.0.1")]
-            ),
-            False,
-        )
-        .sign(
-            serialization.load_pem_private_key(base64.b64decode(ca_key), None),
-            hashes.SHA256(),
-        )
-    )
-
-    with open(os.path.join(tls_dir, "tls.crt"), "w") as f:
-        f.write(tls_cert.public_bytes(serialization.Encoding.PEM).decode("utf-8"))
-
-
-
-def get_secret_name(cluster_name, namespace, api_instance) -
-
-
-
- -Expand source code - -
def get_secret_name(cluster_name, namespace, api_instance):
-    label_selector = f"ray.openshift.ai/cluster-name={cluster_name}"
-    try:
-        secrets = api_instance.list_namespaced_secret(
-            namespace, label_selector=label_selector
-        )
-        for secret in secrets.items:
-            if (
-                f"{cluster_name}-ca-secret-" in secret.metadata.name
-            ):  # Oauth secret share the same label this conditional is to make things more specific
-                return secret.metadata.name
-            else:
-                continue
-        raise KeyError(f"Unable to gather secret name for {cluster_name}")
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/utils/generate_yaml.html b/docs/detailed-documentation/utils/generate_yaml.html deleted file mode 100644 index c2a7bb347..000000000 --- a/docs/detailed-documentation/utils/generate_yaml.html +++ /dev/null @@ -1,951 +0,0 @@ - - - - - - -codeflare_sdk.utils.generate_yaml API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.utils.generate_yaml

-
-
-

This sub-module exists primarily to be used internally by the Cluster object -(in the cluster sub-module) for AppWrapper generation.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-This sub-module exists primarily to be used internally by the Cluster object
-(in the cluster sub-module) for AppWrapper generation.
-"""
-
-import json
-from typing import Optional
-import typing
-import yaml
-import sys
-import os
-import argparse
-import uuid
-from kubernetes import client, config
-from .kube_api_helpers import _kube_api_error_handling
-from ..cluster.auth import get_api_client, config_check
-from os import urandom
-from base64 import b64encode
-from urllib3.util import parse_url
-from kubernetes.client.exceptions import ApiException
-import codeflare_sdk
-
-
-def read_template(template):
-    with open(template, "r") as stream:
-        try:
-            return yaml.safe_load(stream)
-        except yaml.YAMLError as exc:
-            print(exc)
-
-
-def gen_names(name):
-    if not name:
-        gen_id = str(uuid.uuid4())
-        appwrapper_name = "appwrapper-" + gen_id
-        cluster_name = "cluster-" + gen_id
-        return appwrapper_name, cluster_name
-    else:
-        return name, name
-
-
-# Check if the routes api exists
-def is_openshift_cluster():
-    try:
-        config_check()
-        for api in client.ApisApi(get_api_client()).get_api_versions().groups:
-            for v in api.versions:
-                if "route.openshift.io/v1" in v.group_version:
-                    return True
-        else:
-            return False
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-def is_kind_cluster():
-    try:
-        config_check()
-        v1 = client.CoreV1Api()
-        label_selector = "kubernetes.io/hostname=kind-control-plane"
-        nodes = v1.list_node(label_selector=label_selector)
-        # If we find one or more nodes with the label, assume it's a KinD cluster
-        return len(nodes.items) > 0
-    except Exception as e:
-        print(f"Error checking if cluster is KinD: {e}")
-        return False
-
-
-def update_names(
-    cluster_yaml: dict,
-    cluster: "codeflare_sdk.cluster.Cluster",
-):
-    metadata = cluster_yaml.get("metadata")
-    metadata["name"] = cluster.config.name
-    metadata["namespace"] = cluster.config.namespace
-
-
-def update_image(spec, image):
-    containers = spec.get("containers")
-    if image != "":
-        for container in containers:
-            container["image"] = image
-
-
-def update_image_pull_secrets(spec, image_pull_secrets):
-    template_secrets = spec.get("imagePullSecrets", [])
-    spec["imagePullSecrets"] = template_secrets + [
-        {"name": x} for x in image_pull_secrets
-    ]
-
-
-def update_env(spec, env):
-    containers = spec.get("containers")
-    for container in containers:
-        if env:
-            if "env" in container:
-                container["env"].extend(env)
-            else:
-                container["env"] = env
-
-
-def update_resources(
-    spec,
-    cpu_requests,
-    cpu_limits,
-    memory_requests,
-    memory_limits,
-    custom_resources,
-):
-    container = spec.get("containers")
-    for resource in container:
-        requests = resource.get("resources").get("requests")
-        if requests is not None:
-            requests["cpu"] = cpu_requests
-            requests["memory"] = memory_requests
-        limits = resource.get("resources").get("limits")
-        if limits is not None:
-            limits["cpu"] = cpu_limits
-            limits["memory"] = memory_limits
-        for k in custom_resources.keys():
-            limits[k] = custom_resources[k]
-            requests[k] = custom_resources[k]
-
-
-def head_worker_gpu_count_from_cluster(
-    cluster: "codeflare_sdk.cluster.Cluster",
-) -> typing.Tuple[int, int]:
-    head_gpus = 0
-    worker_gpus = 0
-    for k in cluster.config.head_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type == "GPU":
-            head_gpus += int(cluster.config.head_extended_resource_requests[k])
-    for k in cluster.config.worker_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type == "GPU":
-            worker_gpus += int(cluster.config.worker_extended_resource_requests[k])
-
-    return head_gpus, worker_gpus
-
-
-FORBIDDEN_CUSTOM_RESOURCE_TYPES = ["GPU", "CPU", "memory"]
-
-
-def head_worker_resources_from_cluster(
-    cluster: "codeflare_sdk.cluster.Cluster",
-) -> typing.Tuple[dict, dict]:
-    to_return = {}, {}
-    for k in cluster.config.head_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type in FORBIDDEN_CUSTOM_RESOURCE_TYPES:
-            continue
-        to_return[0][resource_type] = cluster.config.head_extended_resource_requests[
-            k
-        ] + to_return[0].get(resource_type, 0)
-
-    for k in cluster.config.worker_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type in FORBIDDEN_CUSTOM_RESOURCE_TYPES:
-            continue
-        to_return[1][resource_type] = cluster.config.worker_extended_resource_requests[
-            k
-        ] + to_return[1].get(resource_type, 0)
-    return to_return
-
-
-def update_nodes(
-    ray_cluster_dict: dict,
-    cluster: "codeflare_sdk.cluster.Cluster",
-):
-    head = ray_cluster_dict.get("spec").get("headGroupSpec")
-    worker = ray_cluster_dict.get("spec").get("workerGroupSpecs")[0]
-    head_gpus, worker_gpus = head_worker_gpu_count_from_cluster(cluster)
-    head_resources, worker_resources = head_worker_resources_from_cluster(cluster)
-    head_resources = json.dumps(head_resources).replace('"', '\\"')
-    head_resources = f'"{head_resources}"'
-    worker_resources = json.dumps(worker_resources).replace('"', '\\"')
-    worker_resources = f'"{worker_resources}"'
-    head["rayStartParams"]["num-gpus"] = str(head_gpus)
-    head["rayStartParams"]["resources"] = head_resources
-
-    # Head counts as first worker
-    worker["replicas"] = cluster.config.num_workers
-    worker["minReplicas"] = cluster.config.num_workers
-    worker["maxReplicas"] = cluster.config.num_workers
-    worker["groupName"] = "small-group-" + cluster.config.name
-    worker["rayStartParams"]["num-gpus"] = str(worker_gpus)
-    worker["rayStartParams"]["resources"] = worker_resources
-
-    for comp in [head, worker]:
-        spec = comp.get("template").get("spec")
-        update_image_pull_secrets(spec, cluster.config.image_pull_secrets)
-        update_image(spec, cluster.config.image)
-        update_env(spec, cluster.config.envs)
-        if comp == head:
-            # TODO: Eventually add head node configuration outside of template
-            update_resources(
-                spec,
-                cluster.config.head_cpu_requests,
-                cluster.config.head_cpu_limits,
-                cluster.config.head_memory_requests,
-                cluster.config.head_memory_limits,
-                cluster.config.head_extended_resource_requests,
-            )
-        else:
-            update_resources(
-                spec,
-                cluster.config.worker_cpu_requests,
-                cluster.config.worker_cpu_limits,
-                cluster.config.worker_memory_requests,
-                cluster.config.worker_memory_limits,
-                cluster.config.worker_extended_resource_requests,
-            )
-
-
-def del_from_list_by_name(l: list, target: typing.List[str]) -> list:
-    return [x for x in l if x["name"] not in target]
-
-
-def get_default_kueue_name(namespace: str):
-    # If the local queue is set, use it. Otherwise, try to use the default queue.
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        local_queues = api_instance.list_namespaced_custom_object(
-            group="kueue.x-k8s.io",
-            version="v1beta1",
-            namespace=namespace,
-            plural="localqueues",
-        )
-    except ApiException as e:  # pragma: no cover
-        if e.status == 404 or e.status == 403:
-            return
-        else:
-            return _kube_api_error_handling(e)
-    for lq in local_queues["items"]:
-        if (
-            "annotations" in lq["metadata"]
-            and "kueue.x-k8s.io/default-queue" in lq["metadata"]["annotations"]
-            and lq["metadata"]["annotations"]["kueue.x-k8s.io/default-queue"].lower()
-            == "true"
-        ):
-            return lq["metadata"]["name"]
-
-
-def local_queue_exists(namespace: str, local_queue_name: str):
-    # get all local queues in the namespace
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        local_queues = api_instance.list_namespaced_custom_object(
-            group="kueue.x-k8s.io",
-            version="v1beta1",
-            namespace=namespace,
-            plural="localqueues",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-    # check if local queue with the name provided in cluster config exists
-    for lq in local_queues["items"]:
-        if lq["metadata"]["name"] == local_queue_name:
-            return True
-    return False
-
-
-def add_queue_label(item: dict, namespace: str, local_queue: Optional[str]):
-    lq_name = local_queue or get_default_kueue_name(namespace)
-    if lq_name == None:
-        return
-    elif not local_queue_exists(namespace, lq_name):
-        raise ValueError(
-            "local_queue provided does not exist or is not in this namespace. Please provide the correct local_queue name in Cluster Configuration"
-        )
-    if not "labels" in item["metadata"]:
-        item["metadata"]["labels"] = {}
-    item["metadata"]["labels"].update({"kueue.x-k8s.io/queue-name": lq_name})
-
-
-def augment_labels(item: dict, labels: dict):
-    if not "labels" in item["metadata"]:
-        item["metadata"]["labels"] = {}
-    item["metadata"]["labels"].update(labels)
-
-
-def notebook_annotations(item: dict):
-    nb_prefix = os.environ.get("NB_PREFIX")
-    if nb_prefix:
-        if not "annotations" in item["metadata"]:
-            item["metadata"]["annotations"] = {}
-        item["metadata"]["annotations"].update(
-            {"app.kubernetes.io/managed-by": nb_prefix}
-        )
-
-
-def wrap_cluster(cluster_yaml: dict, appwrapper_name: str, namespace: str):
-    return {
-        "apiVersion": "workload.codeflare.dev/v1beta2",
-        "kind": "AppWrapper",
-        "metadata": {"name": appwrapper_name, "namespace": namespace},
-        "spec": {"components": [{"template": cluster_yaml}]},
-    }
-
-
-def write_user_yaml(user_yaml, output_file_name):
-    # Create the directory if it doesn't exist
-    directory_path = os.path.dirname(output_file_name)
-    if not os.path.exists(directory_path):
-        os.makedirs(directory_path)
-
-    with open(output_file_name, "w") as outfile:
-        yaml.dump(user_yaml, outfile, default_flow_style=False)
-
-    print(f"Written to: {output_file_name}")
-
-
-def generate_appwrapper(cluster: "codeflare_sdk.cluster.Cluster"):
-    cluster_yaml = read_template(cluster.config.template)
-    appwrapper_name, _ = gen_names(cluster.config.name)
-    update_names(
-        cluster_yaml,
-        cluster,
-    )
-    update_nodes(cluster_yaml, cluster)
-    augment_labels(cluster_yaml, cluster.config.labels)
-    notebook_annotations(cluster_yaml)
-    user_yaml = (
-        wrap_cluster(cluster_yaml, appwrapper_name, cluster.config.namespace)
-        if cluster.config.appwrapper
-        else cluster_yaml
-    )
-
-    add_queue_label(user_yaml, cluster.config.namespace, cluster.config.local_queue)
-
-    if cluster.config.write_to_file:
-        directory_path = os.path.expanduser("~/.codeflare/resources/")
-        outfile = os.path.join(directory_path, appwrapper_name + ".yaml")
-        write_user_yaml(user_yaml, outfile)
-        return outfile
-    else:
-        user_yaml = yaml.dump(user_yaml)
-        print(f"Yaml resources loaded for {cluster.config.name}")
-        return user_yaml
-
-
-
-
-
-
-
-

Functions

-
-
-def add_queue_label(item: dict, namespace: str, local_queue: Optional[str]) -
-
-
-
- -Expand source code - -
def add_queue_label(item: dict, namespace: str, local_queue: Optional[str]):
-    lq_name = local_queue or get_default_kueue_name(namespace)
-    if lq_name == None:
-        return
-    elif not local_queue_exists(namespace, lq_name):
-        raise ValueError(
-            "local_queue provided does not exist or is not in this namespace. Please provide the correct local_queue name in Cluster Configuration"
-        )
-    if not "labels" in item["metadata"]:
-        item["metadata"]["labels"] = {}
-    item["metadata"]["labels"].update({"kueue.x-k8s.io/queue-name": lq_name})
-
-
-
-def augment_labels(item: dict, labels: dict) -
-
-
-
- -Expand source code - -
def augment_labels(item: dict, labels: dict):
-    if not "labels" in item["metadata"]:
-        item["metadata"]["labels"] = {}
-    item["metadata"]["labels"].update(labels)
-
-
-
-def del_from_list_by_name(l: list, target: List[str]) ‑> list -
-
-
-
- -Expand source code - -
def del_from_list_by_name(l: list, target: typing.List[str]) -> list:
-    return [x for x in l if x["name"] not in target]
-
-
-
-def gen_names(name) -
-
-
-
- -Expand source code - -
def gen_names(name):
-    if not name:
-        gen_id = str(uuid.uuid4())
-        appwrapper_name = "appwrapper-" + gen_id
-        cluster_name = "cluster-" + gen_id
-        return appwrapper_name, cluster_name
-    else:
-        return name, name
-
-
-
-def generate_appwrapper(cluster: codeflare_sdk.cluster.Cluster) -
-
-
-
- -Expand source code - -
def generate_appwrapper(cluster: "codeflare_sdk.cluster.Cluster"):
-    cluster_yaml = read_template(cluster.config.template)
-    appwrapper_name, _ = gen_names(cluster.config.name)
-    update_names(
-        cluster_yaml,
-        cluster,
-    )
-    update_nodes(cluster_yaml, cluster)
-    augment_labels(cluster_yaml, cluster.config.labels)
-    notebook_annotations(cluster_yaml)
-    user_yaml = (
-        wrap_cluster(cluster_yaml, appwrapper_name, cluster.config.namespace)
-        if cluster.config.appwrapper
-        else cluster_yaml
-    )
-
-    add_queue_label(user_yaml, cluster.config.namespace, cluster.config.local_queue)
-
-    if cluster.config.write_to_file:
-        directory_path = os.path.expanduser("~/.codeflare/resources/")
-        outfile = os.path.join(directory_path, appwrapper_name + ".yaml")
-        write_user_yaml(user_yaml, outfile)
-        return outfile
-    else:
-        user_yaml = yaml.dump(user_yaml)
-        print(f"Yaml resources loaded for {cluster.config.name}")
-        return user_yaml
-
-
-
-def get_default_kueue_name(namespace: str) -
-
-
-
- -Expand source code - -
def get_default_kueue_name(namespace: str):
-    # If the local queue is set, use it. Otherwise, try to use the default queue.
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        local_queues = api_instance.list_namespaced_custom_object(
-            group="kueue.x-k8s.io",
-            version="v1beta1",
-            namespace=namespace,
-            plural="localqueues",
-        )
-    except ApiException as e:  # pragma: no cover
-        if e.status == 404 or e.status == 403:
-            return
-        else:
-            return _kube_api_error_handling(e)
-    for lq in local_queues["items"]:
-        if (
-            "annotations" in lq["metadata"]
-            and "kueue.x-k8s.io/default-queue" in lq["metadata"]["annotations"]
-            and lq["metadata"]["annotations"]["kueue.x-k8s.io/default-queue"].lower()
-            == "true"
-        ):
-            return lq["metadata"]["name"]
-
-
-
-def head_worker_gpu_count_from_cluster(cluster: codeflare_sdk.cluster.Cluster) ‑> Tuple[int, int] -
-
-
-
- -Expand source code - -
def head_worker_gpu_count_from_cluster(
-    cluster: "codeflare_sdk.cluster.Cluster",
-) -> typing.Tuple[int, int]:
-    head_gpus = 0
-    worker_gpus = 0
-    for k in cluster.config.head_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type == "GPU":
-            head_gpus += int(cluster.config.head_extended_resource_requests[k])
-    for k in cluster.config.worker_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type == "GPU":
-            worker_gpus += int(cluster.config.worker_extended_resource_requests[k])
-
-    return head_gpus, worker_gpus
-
-
-
-def head_worker_resources_from_cluster(cluster: codeflare_sdk.cluster.Cluster) ‑> Tuple[dict, dict] -
-
-
-
- -Expand source code - -
def head_worker_resources_from_cluster(
-    cluster: "codeflare_sdk.cluster.Cluster",
-) -> typing.Tuple[dict, dict]:
-    to_return = {}, {}
-    for k in cluster.config.head_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type in FORBIDDEN_CUSTOM_RESOURCE_TYPES:
-            continue
-        to_return[0][resource_type] = cluster.config.head_extended_resource_requests[
-            k
-        ] + to_return[0].get(resource_type, 0)
-
-    for k in cluster.config.worker_extended_resource_requests.keys():
-        resource_type = cluster.config.extended_resource_mapping[k]
-        if resource_type in FORBIDDEN_CUSTOM_RESOURCE_TYPES:
-            continue
-        to_return[1][resource_type] = cluster.config.worker_extended_resource_requests[
-            k
-        ] + to_return[1].get(resource_type, 0)
-    return to_return
-
-
-
-def is_kind_cluster() -
-
-
-
- -Expand source code - -
def is_kind_cluster():
-    try:
-        config_check()
-        v1 = client.CoreV1Api()
-        label_selector = "kubernetes.io/hostname=kind-control-plane"
-        nodes = v1.list_node(label_selector=label_selector)
-        # If we find one or more nodes with the label, assume it's a KinD cluster
-        return len(nodes.items) > 0
-    except Exception as e:
-        print(f"Error checking if cluster is KinD: {e}")
-        return False
-
-
-
-def is_openshift_cluster() -
-
-
-
- -Expand source code - -
def is_openshift_cluster():
-    try:
-        config_check()
-        for api in client.ApisApi(get_api_client()).get_api_versions().groups:
-            for v in api.versions:
-                if "route.openshift.io/v1" in v.group_version:
-                    return True
-        else:
-            return False
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-
-
-
-def local_queue_exists(namespace: str, local_queue_name: str) -
-
-
-
- -Expand source code - -
def local_queue_exists(namespace: str, local_queue_name: str):
-    # get all local queues in the namespace
-    try:
-        config_check()
-        api_instance = client.CustomObjectsApi(get_api_client())
-        local_queues = api_instance.list_namespaced_custom_object(
-            group="kueue.x-k8s.io",
-            version="v1beta1",
-            namespace=namespace,
-            plural="localqueues",
-        )
-    except Exception as e:  # pragma: no cover
-        return _kube_api_error_handling(e)
-    # check if local queue with the name provided in cluster config exists
-    for lq in local_queues["items"]:
-        if lq["metadata"]["name"] == local_queue_name:
-            return True
-    return False
-
-
-
-def notebook_annotations(item: dict) -
-
-
-
- -Expand source code - -
def notebook_annotations(item: dict):
-    nb_prefix = os.environ.get("NB_PREFIX")
-    if nb_prefix:
-        if not "annotations" in item["metadata"]:
-            item["metadata"]["annotations"] = {}
-        item["metadata"]["annotations"].update(
-            {"app.kubernetes.io/managed-by": nb_prefix}
-        )
-
-
-
-def read_template(template) -
-
-
-
- -Expand source code - -
def read_template(template):
-    with open(template, "r") as stream:
-        try:
-            return yaml.safe_load(stream)
-        except yaml.YAMLError as exc:
-            print(exc)
-
-
-
-def update_env(spec, env) -
-
-
-
- -Expand source code - -
def update_env(spec, env):
-    containers = spec.get("containers")
-    for container in containers:
-        if env:
-            if "env" in container:
-                container["env"].extend(env)
-            else:
-                container["env"] = env
-
-
-
-def update_image(spec, image) -
-
-
-
- -Expand source code - -
def update_image(spec, image):
-    containers = spec.get("containers")
-    if image != "":
-        for container in containers:
-            container["image"] = image
-
-
-
-def update_image_pull_secrets(spec, image_pull_secrets) -
-
-
-
- -Expand source code - -
def update_image_pull_secrets(spec, image_pull_secrets):
-    template_secrets = spec.get("imagePullSecrets", [])
-    spec["imagePullSecrets"] = template_secrets + [
-        {"name": x} for x in image_pull_secrets
-    ]
-
-
-
-def update_names(cluster_yaml: dict, cluster: codeflare_sdk.cluster.Cluster) -
-
-
-
- -Expand source code - -
def update_names(
-    cluster_yaml: dict,
-    cluster: "codeflare_sdk.cluster.Cluster",
-):
-    metadata = cluster_yaml.get("metadata")
-    metadata["name"] = cluster.config.name
-    metadata["namespace"] = cluster.config.namespace
-
-
-
-def update_nodes(ray_cluster_dict: dict, cluster: codeflare_sdk.cluster.Cluster) -
-
-
-
- -Expand source code - -
def update_nodes(
-    ray_cluster_dict: dict,
-    cluster: "codeflare_sdk.cluster.Cluster",
-):
-    head = ray_cluster_dict.get("spec").get("headGroupSpec")
-    worker = ray_cluster_dict.get("spec").get("workerGroupSpecs")[0]
-    head_gpus, worker_gpus = head_worker_gpu_count_from_cluster(cluster)
-    head_resources, worker_resources = head_worker_resources_from_cluster(cluster)
-    head_resources = json.dumps(head_resources).replace('"', '\\"')
-    head_resources = f'"{head_resources}"'
-    worker_resources = json.dumps(worker_resources).replace('"', '\\"')
-    worker_resources = f'"{worker_resources}"'
-    head["rayStartParams"]["num-gpus"] = str(head_gpus)
-    head["rayStartParams"]["resources"] = head_resources
-
-    # Head counts as first worker
-    worker["replicas"] = cluster.config.num_workers
-    worker["minReplicas"] = cluster.config.num_workers
-    worker["maxReplicas"] = cluster.config.num_workers
-    worker["groupName"] = "small-group-" + cluster.config.name
-    worker["rayStartParams"]["num-gpus"] = str(worker_gpus)
-    worker["rayStartParams"]["resources"] = worker_resources
-
-    for comp in [head, worker]:
-        spec = comp.get("template").get("spec")
-        update_image_pull_secrets(spec, cluster.config.image_pull_secrets)
-        update_image(spec, cluster.config.image)
-        update_env(spec, cluster.config.envs)
-        if comp == head:
-            # TODO: Eventually add head node configuration outside of template
-            update_resources(
-                spec,
-                cluster.config.head_cpu_requests,
-                cluster.config.head_cpu_limits,
-                cluster.config.head_memory_requests,
-                cluster.config.head_memory_limits,
-                cluster.config.head_extended_resource_requests,
-            )
-        else:
-            update_resources(
-                spec,
-                cluster.config.worker_cpu_requests,
-                cluster.config.worker_cpu_limits,
-                cluster.config.worker_memory_requests,
-                cluster.config.worker_memory_limits,
-                cluster.config.worker_extended_resource_requests,
-            )
-
-
-
-def update_resources(spec, cpu_requests, cpu_limits, memory_requests, memory_limits, custom_resources) -
-
-
-
- -Expand source code - -
def update_resources(
-    spec,
-    cpu_requests,
-    cpu_limits,
-    memory_requests,
-    memory_limits,
-    custom_resources,
-):
-    container = spec.get("containers")
-    for resource in container:
-        requests = resource.get("resources").get("requests")
-        if requests is not None:
-            requests["cpu"] = cpu_requests
-            requests["memory"] = memory_requests
-        limits = resource.get("resources").get("limits")
-        if limits is not None:
-            limits["cpu"] = cpu_limits
-            limits["memory"] = memory_limits
-        for k in custom_resources.keys():
-            limits[k] = custom_resources[k]
-            requests[k] = custom_resources[k]
-
-
-
-def wrap_cluster(cluster_yaml: dict, appwrapper_name: str, namespace: str) -
-
-
-
- -Expand source code - -
def wrap_cluster(cluster_yaml: dict, appwrapper_name: str, namespace: str):
-    return {
-        "apiVersion": "workload.codeflare.dev/v1beta2",
-        "kind": "AppWrapper",
-        "metadata": {"name": appwrapper_name, "namespace": namespace},
-        "spec": {"components": [{"template": cluster_yaml}]},
-    }
-
-
-
-def write_user_yaml(user_yaml, output_file_name) -
-
-
-
- -Expand source code - -
def write_user_yaml(user_yaml, output_file_name):
-    # Create the directory if it doesn't exist
-    directory_path = os.path.dirname(output_file_name)
-    if not os.path.exists(directory_path):
-        os.makedirs(directory_path)
-
-    with open(output_file_name, "w") as outfile:
-        yaml.dump(user_yaml, outfile, default_flow_style=False)
-
-    print(f"Written to: {output_file_name}")
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/utils/index.html b/docs/detailed-documentation/utils/index.html deleted file mode 100644 index 4a65cc393..000000000 --- a/docs/detailed-documentation/utils/index.html +++ /dev/null @@ -1,88 +0,0 @@ - - - - - - -codeflare_sdk.utils API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.utils

-
-
-
-
-

Sub-modules

-
-
codeflare_sdk.utils.demos
-
-
-
-
codeflare_sdk.utils.generate_cert
-
-
-
-
codeflare_sdk.utils.generate_yaml
-
-

This sub-module exists primarily to be used internally by the Cluster object -(in the cluster sub-module) for AppWrapper generation.

-
-
codeflare_sdk.utils.kube_api_helpers
-
-

This sub-module exists primarily to be used internally for any Kubernetes -API error handling or wrapping.

-
-
codeflare_sdk.utils.pretty_print
-
-

This sub-module exists primarily to be used internally by the Cluster object -(in the cluster sub-module) for pretty-printing cluster status and details.

-
-
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/utils/kube_api_helpers.html b/docs/detailed-documentation/utils/kube_api_helpers.html deleted file mode 100644 index 6bf6fe817..000000000 --- a/docs/detailed-documentation/utils/kube_api_helpers.html +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - -codeflare_sdk.utils.kube_api_helpers API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.utils.kube_api_helpers

-
-
-

This sub-module exists primarily to be used internally for any Kubernetes -API error handling or wrapping.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-This sub-module exists primarily to be used internally for any Kubernetes
-API error handling or wrapping.
-"""
-
-import executing
-from kubernetes import client, config
-from urllib3.util import parse_url
-
-
-# private methods
-def _kube_api_error_handling(
-    e: Exception, print_error: bool = True
-):  # pragma: no cover
-    perm_msg = (
-        "Action not permitted, have you put in correct/up-to-date auth credentials?"
-    )
-    nf_msg = "No instances found, nothing to be done."
-    exists_msg = "Resource with this name already exists."
-    if type(e) == config.ConfigException:
-        raise PermissionError(perm_msg)
-    if type(e) == executing.executing.NotOneValueFound:
-        if print_error:
-            print(nf_msg)
-        return
-    if type(e) == client.ApiException:
-        if e.reason == "Not Found":
-            if print_error:
-                print(nf_msg)
-            return
-        elif e.reason == "Unauthorized" or e.reason == "Forbidden":
-            if print_error:
-                print(perm_msg)
-            return
-        elif e.reason == "Conflict":
-            raise FileExistsError(exists_msg)
-    raise e
-
-
-
-
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/detailed-documentation/utils/pretty_print.html b/docs/detailed-documentation/utils/pretty_print.html deleted file mode 100644 index f2a8d7db9..000000000 --- a/docs/detailed-documentation/utils/pretty_print.html +++ /dev/null @@ -1,491 +0,0 @@ - - - - - - -codeflare_sdk.utils.pretty_print API documentation - - - - - - - - - - - -
-
-
-

Module codeflare_sdk.utils.pretty_print

-
-
-

This sub-module exists primarily to be used internally by the Cluster object -(in the cluster sub-module) for pretty-printing cluster status and details.

-
- -Expand source code - -
# Copyright 2022 IBM, Red Hat
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-This sub-module exists primarily to be used internally by the Cluster object
-(in the cluster sub-module) for pretty-printing cluster status and details.
-"""
-
-from rich import print
-from rich.table import Table
-from rich.console import Console
-from rich.layout import Layout
-from rich.panel import Panel
-from rich import box
-from typing import List
-from ..cluster.model import RayCluster, AppWrapper, RayClusterStatus
-
-
-def print_no_resources_found():
-    console = Console()
-    console.print(Panel("[red]No resources found, have you run cluster.up() yet?"))
-
-
-def print_app_wrappers_status(app_wrappers: List[AppWrapper], starting: bool = False):
-    if not app_wrappers:
-        print_no_resources_found()
-        return  # shortcircuit
-
-    console = Console()
-    table = Table(
-        box=box.ASCII_DOUBLE_HEAD,
-        title="[bold] :rocket: Cluster Queue Status :rocket:",
-    )
-    table.add_column("Name", style="cyan", no_wrap=True)
-    table.add_column("Status", style="magenta")
-
-    for app_wrapper in app_wrappers:
-        name = app_wrapper.name
-        status = app_wrapper.status.value
-        if starting:
-            status += " (starting)"
-        table.add_row(name, status)
-        table.add_row("")  # empty row for spacing
-
-    console.print(Panel.fit(table))
-
-
-def print_ray_clusters_status(app_wrappers: List[AppWrapper], starting: bool = False):
-    if not app_wrappers:
-        print_no_resources_found()
-        return  # shortcircuit
-
-    console = Console()
-    table = Table(
-        box=box.ASCII_DOUBLE_HEAD,
-        title="[bold] :rocket: Cluster Queue Status :rocket:",
-    )
-    table.add_column("Name", style="cyan", no_wrap=True)
-    table.add_column("Status", style="magenta")
-
-    for app_wrapper in app_wrappers:
-        name = app_wrapper.name
-        status = app_wrapper.status.value
-        if starting:
-            status += " (starting)"
-        table.add_row(name, status)
-        table.add_row("")  # empty row for spacing
-
-    console.print(Panel.fit(table))
-
-
-def print_cluster_status(cluster: RayCluster):
-    "Pretty prints the status of a passed-in cluster"
-    if not cluster:
-        print_no_resources_found()
-        return
-
-    console = Console()
-    status = (
-        "Active :white_heavy_check_mark:"
-        if cluster.status == RayClusterStatus.READY
-        else "Inactive :x:"
-    )
-    name = cluster.name
-    dashboard = cluster.dashboard
-
-    #'table0' to display the cluster name, status, url, and dashboard link
-    table0 = Table(box=None, show_header=False)
-
-    table0.add_row("[white on green][bold]Name")
-    table0.add_row("[bold underline]" + name, status)
-    table0.add_row()
-    # fixme harcded to default for now
-    table0.add_row(
-        f"[bold]URI:[/bold] ray://{cluster.name}-head-svc.{cluster.namespace}.svc:10001"
-    )  # format that is used to generate the name of the service
-    table0.add_row()
-    table0.add_row(f"[link={dashboard} blue underline]Dashboard:link:[/link]")
-    table0.add_row("")  # empty row for spacing
-
-    # table4 to display table0 and table3, one below the other
-    table4 = Table(box=None, show_header=False)
-    table4.add_row(table0)
-
-    # Encompass all details of the cluster in a single panel
-    table5 = Table(box=None, title="[bold] :rocket: CodeFlare Cluster Status :rocket:")
-    table5.add_row(Panel.fit(table4))
-    console.print(table5)
-
-
-def print_clusters(clusters: List[RayCluster]):
-    if not clusters:
-        print_no_resources_found()
-        return  # shortcircuit
-
-    console = Console()
-    title_printed = False
-
-    for cluster in clusters:
-        status = (
-            "Active :white_heavy_check_mark:"
-            if cluster.status == RayClusterStatus.READY
-            else "Inactive :x:"
-        )
-        name = cluster.name
-        dashboard = cluster.dashboard
-        workers = str(cluster.num_workers)
-        memory = f"{cluster.worker_mem_requests}~{cluster.worker_mem_limits}"
-        cpu = f"{cluster.worker_cpu_requests}~{cluster.worker_cpu_limits}"
-        gpu = str(cluster.worker_extended_resources.get("nvidia.com/gpu", 0))
-
-        #'table0' to display the cluster name, status, url, and dashboard link
-        table0 = Table(box=None, show_header=False)
-
-        table0.add_row("[white on green][bold]Name")
-        table0.add_row("[bold underline]" + name, status)
-        table0.add_row()
-        # fixme harcded to default for now
-        table0.add_row(
-            f"[bold]URI:[/bold] ray://{cluster.name}-head-svc.{cluster.namespace}.svc:10001"
-        )  # format that is used to generate the name of the service
-        table0.add_row()
-        table0.add_row(f"[link={dashboard} blue underline]Dashboard:link:[/link]")
-        table0.add_row("")  # empty row for spacing
-
-        #'table1' to display the worker counts
-        table1 = Table(box=None)
-        table1.add_row()
-        table1.add_column("# Workers", style="magenta")
-        table1.add_row()
-        table1.add_row(workers)
-        table1.add_row()
-
-        #'table2' to display the worker resources
-        table2 = Table(box=None)
-        table2.add_column("Memory", style="cyan", no_wrap=True, min_width=10)
-        table2.add_column("CPU", style="magenta", min_width=10)
-        table2.add_column("GPU", style="magenta", min_width=10)
-        table2.add_row()
-        table2.add_row(memory, cpu, gpu)
-        table2.add_row()
-
-        # panels to encompass table1 and table2 into separate cards
-        panel_1 = Panel.fit(table1, title="Workers")
-        panel_2 = Panel.fit(table2, title="Worker specs(each)")
-
-        # table3 to display panel_1 and panel_2 side-by-side in a single row
-        table3 = Table(box=None, show_header=False, title="Cluster Resources")
-        table3.add_row(panel_1, panel_2)
-
-        # table4 to display table0 and table3, one below the other
-        table4 = Table(box=None, show_header=False)
-        table4.add_row(table0)
-        table4.add_row(table3)
-
-        # Encompass all details of the cluster in a single panel
-        if not title_printed:
-            # If first cluster in the list, then create a table with title "Codeflare clusters".
-            # This is done to ensure the title is center aligned on the cluster display tables, rather
-            # than being center aligned on the console/terminal if we simply use console.print(title)
-
-            table5 = Table(
-                box=None, title="[bold] :rocket: CodeFlare Cluster Details :rocket:"
-            )
-            table5.add_row(Panel.fit(table4))
-            console.print(table5)
-            title_printed = True
-        else:
-            console.print(Panel.fit(table4))
-
-
-
-
-
-
-
-

Functions

-
-
-def print_app_wrappers_status(app_wrappers: List[AppWrapper], starting: bool = False) -
-
-
-
- -Expand source code - -
def print_app_wrappers_status(app_wrappers: List[AppWrapper], starting: bool = False):
-    if not app_wrappers:
-        print_no_resources_found()
-        return  # shortcircuit
-
-    console = Console()
-    table = Table(
-        box=box.ASCII_DOUBLE_HEAD,
-        title="[bold] :rocket: Cluster Queue Status :rocket:",
-    )
-    table.add_column("Name", style="cyan", no_wrap=True)
-    table.add_column("Status", style="magenta")
-
-    for app_wrapper in app_wrappers:
-        name = app_wrapper.name
-        status = app_wrapper.status.value
-        if starting:
-            status += " (starting)"
-        table.add_row(name, status)
-        table.add_row("")  # empty row for spacing
-
-    console.print(Panel.fit(table))
-
-
-
-def print_cluster_status(cluster: RayCluster) -
-
-

Pretty prints the status of a passed-in cluster

-
- -Expand source code - -
def print_cluster_status(cluster: RayCluster):
-    "Pretty prints the status of a passed-in cluster"
-    if not cluster:
-        print_no_resources_found()
-        return
-
-    console = Console()
-    status = (
-        "Active :white_heavy_check_mark:"
-        if cluster.status == RayClusterStatus.READY
-        else "Inactive :x:"
-    )
-    name = cluster.name
-    dashboard = cluster.dashboard
-
-    #'table0' to display the cluster name, status, url, and dashboard link
-    table0 = Table(box=None, show_header=False)
-
-    table0.add_row("[white on green][bold]Name")
-    table0.add_row("[bold underline]" + name, status)
-    table0.add_row()
-    # fixme harcded to default for now
-    table0.add_row(
-        f"[bold]URI:[/bold] ray://{cluster.name}-head-svc.{cluster.namespace}.svc:10001"
-    )  # format that is used to generate the name of the service
-    table0.add_row()
-    table0.add_row(f"[link={dashboard} blue underline]Dashboard:link:[/link]")
-    table0.add_row("")  # empty row for spacing
-
-    # table4 to display table0 and table3, one below the other
-    table4 = Table(box=None, show_header=False)
-    table4.add_row(table0)
-
-    # Encompass all details of the cluster in a single panel
-    table5 = Table(box=None, title="[bold] :rocket: CodeFlare Cluster Status :rocket:")
-    table5.add_row(Panel.fit(table4))
-    console.print(table5)
-
-
-
-def print_clusters(clusters: List[RayCluster]) -
-
-
-
- -Expand source code - -
def print_clusters(clusters: List[RayCluster]):
-    if not clusters:
-        print_no_resources_found()
-        return  # shortcircuit
-
-    console = Console()
-    title_printed = False
-
-    for cluster in clusters:
-        status = (
-            "Active :white_heavy_check_mark:"
-            if cluster.status == RayClusterStatus.READY
-            else "Inactive :x:"
-        )
-        name = cluster.name
-        dashboard = cluster.dashboard
-        workers = str(cluster.num_workers)
-        memory = f"{cluster.worker_mem_requests}~{cluster.worker_mem_limits}"
-        cpu = f"{cluster.worker_cpu_requests}~{cluster.worker_cpu_limits}"
-        gpu = str(cluster.worker_extended_resources.get("nvidia.com/gpu", 0))
-
-        #'table0' to display the cluster name, status, url, and dashboard link
-        table0 = Table(box=None, show_header=False)
-
-        table0.add_row("[white on green][bold]Name")
-        table0.add_row("[bold underline]" + name, status)
-        table0.add_row()
-        # fixme harcded to default for now
-        table0.add_row(
-            f"[bold]URI:[/bold] ray://{cluster.name}-head-svc.{cluster.namespace}.svc:10001"
-        )  # format that is used to generate the name of the service
-        table0.add_row()
-        table0.add_row(f"[link={dashboard} blue underline]Dashboard:link:[/link]")
-        table0.add_row("")  # empty row for spacing
-
-        #'table1' to display the worker counts
-        table1 = Table(box=None)
-        table1.add_row()
-        table1.add_column("# Workers", style="magenta")
-        table1.add_row()
-        table1.add_row(workers)
-        table1.add_row()
-
-        #'table2' to display the worker resources
-        table2 = Table(box=None)
-        table2.add_column("Memory", style="cyan", no_wrap=True, min_width=10)
-        table2.add_column("CPU", style="magenta", min_width=10)
-        table2.add_column("GPU", style="magenta", min_width=10)
-        table2.add_row()
-        table2.add_row(memory, cpu, gpu)
-        table2.add_row()
-
-        # panels to encompass table1 and table2 into separate cards
-        panel_1 = Panel.fit(table1, title="Workers")
-        panel_2 = Panel.fit(table2, title="Worker specs(each)")
-
-        # table3 to display panel_1 and panel_2 side-by-side in a single row
-        table3 = Table(box=None, show_header=False, title="Cluster Resources")
-        table3.add_row(panel_1, panel_2)
-
-        # table4 to display table0 and table3, one below the other
-        table4 = Table(box=None, show_header=False)
-        table4.add_row(table0)
-        table4.add_row(table3)
-
-        # Encompass all details of the cluster in a single panel
-        if not title_printed:
-            # If first cluster in the list, then create a table with title "Codeflare clusters".
-            # This is done to ensure the title is center aligned on the cluster display tables, rather
-            # than being center aligned on the console/terminal if we simply use console.print(title)
-
-            table5 = Table(
-                box=None, title="[bold] :rocket: CodeFlare Cluster Details :rocket:"
-            )
-            table5.add_row(Panel.fit(table4))
-            console.print(table5)
-            title_printed = True
-        else:
-            console.print(Panel.fit(table4))
-
-
-
-def print_no_resources_found() -
-
-
-
- -Expand source code - -
def print_no_resources_found():
-    console = Console()
-    console.print(Panel("[red]No resources found, have you run cluster.up() yet?"))
-
-
-
-def print_ray_clusters_status(app_wrappers: List[AppWrapper], starting: bool = False) -
-
-
-
- -Expand source code - -
def print_ray_clusters_status(app_wrappers: List[AppWrapper], starting: bool = False):
-    if not app_wrappers:
-        print_no_resources_found()
-        return  # shortcircuit
-
-    console = Console()
-    table = Table(
-        box=box.ASCII_DOUBLE_HEAD,
-        title="[bold] :rocket: Cluster Queue Status :rocket:",
-    )
-    table.add_column("Name", style="cyan", no_wrap=True)
-    table.add_column("Status", style="magenta")
-
-    for app_wrapper in app_wrappers:
-        name = app_wrapper.name
-        status = app_wrapper.status.value
-        if starting:
-            status += " (starting)"
-        table.add_row(name, status)
-        table.add_row("")  # empty row for spacing
-
-    console.print(Panel.fit(table))
-
-
-
-
-
-
-
- -
- - - diff --git a/docs/e2e.md b/docs/e2e.md deleted file mode 100644 index 83d8ae4e7..000000000 --- a/docs/e2e.md +++ /dev/null @@ -1,133 +0,0 @@ -# Running e2e tests locally -#### Pre-requisites -- We recommend using Python 3.9, along with Poetry. - -## On KinD clusters -Pre-requisite for KinD clusters: please add in your local `/etc/hosts` file `127.0.0.1 kind`. This will map your localhost IP address to the KinD cluster's hostname. This is already performed on [GitHub Actions](https://github.com/project-codeflare/codeflare-common/blob/1edd775e2d4088a5a0bfddafb06ff3a773231c08/github-actions/kind/action.yml#L70-L72) - -If the system you run on contains NVidia GPU then you can enable the GPU support in KinD, this will allow you to run also GPU tests. -To enable GPU on KinD follow [these instructions](https://www.substratus.ai/blog/kind-with-gpus). - -- Setup Phase: - - Pull the [codeflare-operator repo](https://github.com/project-codeflare/codeflare-operator) and run the following make targets: - ``` - make kind-e2e - export CLUSTER_HOSTNAME=kind - make setup-e2e - make deploy -e IMG=quay.io/project-codeflare/codeflare-operator:v1.3.0 - - For running tests locally on Kind cluster, we need to disable `rayDashboardOAuthEnabled` in `codeflare-operator-config` ConfigMap and then restart CodeFlare Operator - ``` - - - **(Optional)** - Create and add `sdk-user` with limited permissions to the cluster to run through the e2e tests: - ``` - # Get KinD certificates - docker cp kind-control-plane:/etc/kubernetes/pki/ca.crt . - docker cp kind-control-plane:/etc/kubernetes/pki/ca.key . - - # Generate certificates for new user - openssl genrsa -out user.key 2048 - openssl req -new -key user.key -out user.csr -subj '/CN=sdk-user/O=tenant' - openssl x509 -req -in user.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out user.crt -days 360 - - # Add generated certificated to KinD context - user_crt=$(base64 --wrap=0 user.crt) - user_key=$(base64 --wrap=0 user.key) - yq eval -i ".contexts += {\"context\": {\"cluster\": \"kind-kind\", \"user\": \"sdk-user\"}, \"name\": \"sdk-user\"}" $HOME/.kube/config - yq eval -i ".users += {\"name\": \"sdk-user\", \"user\": {\"client-certificate-data\": \"$user_crt\", \"client-key-data\": \"$user_key\"}}" $HOME/.kube/config - cat $HOME/.kube/config - - # Cleanup - rm ca.crt - rm ca.srl - rm ca.key - rm user.crt - rm user.key - rm user.csr - - # Add RBAC permissions to sdk-user - kubectl create clusterrole list-ingresses --verb=get,list --resource=ingresses - kubectl create clusterrolebinding sdk-user-list-ingresses --clusterrole=list-ingresses --user=sdk-user - kubectl create clusterrole appwrapper-creator --verb=get,list,create,delete,patch --resource=appwrappers - kubectl create clusterrolebinding sdk-user-appwrapper-creator --clusterrole=appwrapper-creator --user=sdk-user - kubectl create clusterrole namespace-creator --verb=get,list,create,delete,patch --resource=namespaces - kubectl create clusterrolebinding sdk-user-namespace-creator --clusterrole=namespace-creator --user=sdk-user - kubectl create clusterrole list-rayclusters --verb=get,list --resource=rayclusters - kubectl create clusterrolebinding sdk-user-list-rayclusters --clusterrole=list-rayclusters --user=sdk-user - kubectl config use-context sdk-user - - ``` - - - Install the latest development version of kueue - ``` - kubectl apply --server-side -k "github.com/opendatahub-io/kueue/config/rhoai?ref=dev" - ``` - -- Test Phase: - - Once we have the codeflare-operator, kuberay-operator and kueue running and ready, we can run the e2e test on the codeflare-sdk repository: - ``` - poetry install --with test,docs - poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_kind_test.py - ``` - - If the cluster doesn't have NVidia GPU support then we need to disable NVidia GPU tests by providing proper marker: - ``` - poetry install --with test,docs - poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_kind_test.py -m 'kind and not nvidia_gpu' - ``` - - -## On OpenShift clusters -- Setup Phase: - - Pull the [codeflare-operator repo](https://github.com/project-codeflare/codeflare-operator) and run the following make targets: - ``` - - make setup-e2e - make deploy -e IMG=quay.io/project-codeflare/codeflare-operator:v1.3.0 - ``` - - - Install the latest development version of kueue - ``` - kubectl apply --server-side -k "github.com/opendatahub-io/kueue/config/rhoai?ref=dev" - ``` - -If the system you run on contains NVidia GPU then you can enable the GPU support on OpenShift, this will allow you to run also GPU tests. -To enable GPU on OpenShift follow [these instructions](https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/introduction.html). -Currently the SDK doesn't support tolerations, so e2e tests can't be executed on nodes with taint (i.e. GPU taint). - -- Test Phase: - - Once we have the codeflare-operator, kuberay-operator and kueue running and ready, we can run the e2e test on the codeflare-sdk repository: - ``` - poetry install --with test,docs - poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_test.py - ``` - - To run the multiple tests based on the cluster environment, we can run the e2e tests by marking -m with cluster environment (kind or openshift) - ``` - poetry run pytest -v -s ./tests/e2e -m openshift - ``` - - By default tests configured with timeout of `15 minutes`. If necessary, we can override the timeout using `--timeout` option - ``` - poetry run pytest -v -s ./tests/e2e -m openshift --timeout=1200 - ``` - -## On OpenShift Disconnected clusters - -- In addition to setup phase mentioned above in case of Openshift cluster, Disconnected environment requires following pre-requisites : - - Mirror Image registry : - - Image mirror registry is used to host set of container images required locally for the applications and services. This ensures to pull images without needing an external network connection. It also ensures continuous operation and deployment capabilities in a network-isolated environment. - - PYPI Mirror Index : - - When trying to install Python packages in a disconnected environment, the pip command might fail because the connection cannot install packages from external URLs. This issue can be resolved by setting up PIP Mirror Index on separate endpoint in same environment. - - S3 compatible storage : - - Some of our distributed training examples require an external storage solution so that all nodes can access the same data in disconnected environment (For example: common-datasets and model files). - - Minio S3 compatible storage type instance can be deployed in disconnected environment using `/tests/e2e/minio_deployment.yaml` or using support methods in e2e test suite. - - The following are environment variables for configuring PIP index URl for accessing the common-python packages required and the S3 or Minio storage for your Ray Train script or interactive session. - ``` - export RAY_IMAGE=quay.io/project-codeflare/ray@sha256: (prefer image digest over image tag in disocnnected environment) - PIP_INDEX_URL=https:///root/pypi/+simple/ \ - PIP_TRUSTED_HOST= \ - AWS_DEFAULT_ENDPOINT= \ - AWS_ACCESS_KEY_ID= \ - AWS_SECRET_ACCESS_KEY= \ - AWS_STORAGE_BUCKET= - AWS_STORAGE_BUCKET_MNIST_DIR= - ``` - Note : When using the Python Minio client to connect to a minio storage bucket, the `AWS_DEFAULT_ENDPOINT` environment variable by default expects secure endpoint where user can use endpoint url with https/http prefix for autodetection of secure/insecure endpoint. diff --git a/docs/generate-documentation.md b/docs/generate-documentation.md new file mode 100644 index 000000000..75b5c7c6b --- /dev/null +++ b/docs/generate-documentation.md @@ -0,0 +1,14 @@ +# Generate CodeFlare Documentation with Sphinx +The following is a short guide on how you can use Sphinx to auto-generate code documentation. Documentation for the latest SDK release can be found [here](https://project-codeflare.github.io/codeflare-sdk/index.html). + +1. Clone the CodeFlare SDK +``` bash +git clone https://github.com/project-codeflare/codeflare-sdk.git +``` +2. [Install Sphinx](https://www.sphinx-doc.org/en/master/usage/installation.html) +3. Run the below command to generate code documentation +``` bash +sphinx-apidoc -o docs/sphinx src/codeflare_sdk "**/*test_*" --force # Generates RST files +make html -C docs/sphinx # Builds HTML files +``` +4. You can access the docs locally at `docs/sphinx/_build/html/index.html` diff --git a/docs/s3-compatible-storage.md b/docs/s3-compatible-storage.md deleted file mode 100644 index 919ce8151..000000000 --- a/docs/s3-compatible-storage.md +++ /dev/null @@ -1,61 +0,0 @@ -# S3 compatible storage with Ray Train examples -Some of our distributed training examples require an external storage solution so that all nodes can access the same data.
-The following are examples for configuring S3 or Minio storage for your Ray Train script or interactive session. - -## S3 Bucket -In your Python Script add the following environment variables: -``` python -os.environ["AWS_ACCESS_KEY_ID"] = "XXXXXXXX" -os.environ["AWS_SECRET_ACCESS_KEY"] = "XXXXXXXX" -os.environ["AWS_DEFAULT_REGION"] = "XXXXXXXX" -``` -Alternatively you can specify these variables in your runtime environment on Job Submission. -``` python -submission_id = client.submit_job( - entrypoint=..., - runtime_env={ - "env_vars": { - "AWS_ACCESS_KEY_ID": os.environ.get('AWS_ACCESS_KEY_ID'), - "AWS_SECRET_ACCESS_KEY": os.environ.get('AWS_SECRET_ACCESS_KEY'), - "AWS_DEFAULT_REGION": os.environ.get('AWS_DEFAULT_REGION') - }, - } -) -``` -In your Trainer configuration you can specify a `run_config` which will utilise your external storage. -``` python -trainer = TorchTrainer( - train_func_distributed, - scaling_config=scaling_config, - run_config = ray.train.RunConfig(storage_path="s3://BUCKET_NAME/SUB_PATH/", name="unique_run_name") -) -``` -To learn more about Amazon S3 Storage you can find information [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). - -## Minio Bucket -In your Python Script add the following function for configuring your run_config: -``` python -import s3fs -import pyarrow - -def get_minio_run_config(): - s3_fs = s3fs.S3FileSystem( - key = os.getenv('MINIO_ACCESS_KEY', "XXXXX"), - secret = os.getenv('MINIO_SECRET_ACCESS_KEY', "XXXXX"), - endpoint_url = os.getenv('MINIO_URL', "XXXXX") - ) - custom_fs = pyarrow.fs.PyFileSystem(pyarrow.fs.FSSpecHandler(s3_fs)) - run_config = ray.train.RunConfig(storage_path='training', storage_filesystem=custom_fs) - return run_config -``` -You can update the `run_config` to further suit your needs above. -Lastly the new `run_config` must be added to the Trainer: -``` python -trainer = TorchTrainer( - train_func_distributed, - scaling_config=scaling_config, - run_config = get_minio_run_config() -) -``` -To find more information on creating a Minio Bucket compatible with RHOAI you can refer to this [documentation](https://ai-on-openshift.io/tools-and-applications/minio/minio/).
-Note: You must have `sf3s` and `pyarrow` installed in your environment for this method. diff --git a/docs/setup-kueue.md b/docs/setup-kueue.md deleted file mode 100644 index c8fffa10e..000000000 --- a/docs/setup-kueue.md +++ /dev/null @@ -1,66 +0,0 @@ -# Basic Kueue Resources configuration - -## Introduction: - -This document is designed for administrators who have Kueue installed on their cluster. We will walk through the process of setting up essential Kueue resources, namely Cluster Queue, Resource Flavor, and Local Queue. - -## 1. Resource Flavor: -Resource Flavors allow the cluster admin to define different types of resources with specific characteristics, such as CPU, memory, GPU, etc. These can then be assigned to workloads to ensure they are executed on appropriate resources. - -The YAML configuration provided below creates an empty Resource Flavor named default-flavor. It serves as a starting point and does not specify any detailed resource characteristics. -```yaml -apiVersion: kueue.x-k8s.io/v1beta1 -kind: ResourceFlavor -metadata: - name: default-flavor -``` -For more detailed information on Resource Flavor configuration options, refer to the Kueue documentation: [Resource Flavor Configuration](https://kueue.sigs.k8s.io/docs/concepts/resource_flavor/) - -## 2. Cluster Queue: -A Cluster Queue represents a shared queue across the entire cluster. It allows the cluster admin to define global settings for workload prioritization and resource allocation. - -When setting up a Cluster Queue in Kueue, it's crucial that the resource specifications match the actual capacities and operational requirements of your cluster. The example provided outlines a basic setup; however, each cluster may have different resource availabilities and needs. -```yaml -apiVersion: kueue.x-k8s.io/v1beta1 -kind: ClusterQueue -metadata: - name: "cluster-queue" -spec: - namespaceSelector: {} # match all. - resourceGroups: - - coveredResources: ["cpu", "memory", "pods", "nvidia.com/gpu"] - flavors: - - name: "default-flavor" - resources: - - name: "cpu" - nominalQuota: 9 - - name: "memory" - nominalQuota: 36Gi - - name: "pods" - nominalQuota: 5 - - name: "nvidia.com/gpu" - nominalQuota: '0' -``` - -For more detailed information on Cluster Queue configuration options, refer to the Kueue documentation: [Cluster Queue Configuration](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/) - -## 3. Local Queue (With Default Annotation): -A Local Queue represents a queue associated with a specific namespace within the cluster. It allows namespace-level control over workload prioritization and resource allocation. -```yaml -apiVersion: kueue.x-k8s.io/v1beta1 -kind: LocalQueue -metadata: - namespace: team-a - name: team-a-queue - annotations: - kueue.x-k8s.io/default-queue: "true" -spec: - clusterQueue: cluster-queue -``` - -In the LocalQueue configuration provided above, the annotations field specifies `kueue.x-k8s.io/default-queue: "true"`. This annotation indicates that the team-a-queue is designated as the default queue for the team-a namespace. When this is set, any workloads submitted to the team-a namespace without explicitly specifying a queue will automatically be routed to the team-a-queue. - -For more detailed information on Local Queue configuration options, refer to the Kueue documentation: [Local Queue Configuration](https://kueue.sigs.k8s.io/docs/concepts/local_queue/) - -## Conclusion: -By following the steps outlined in this document, the cluster admin can successfully create the basic Kueue resources necessary for workload management in the cluster. For more advanced configurations and features, please refer to the comprehensive [Kueue documentation](https://kueue.sigs.k8s.io/docs/concepts/). diff --git a/docs/sphinx/Makefile b/docs/sphinx/Makefile new file mode 100644 index 000000000..d4bb2cbb9 --- /dev/null +++ b/docs/sphinx/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/sphinx/conf.py b/docs/sphinx/conf.py new file mode 100644 index 000000000..75f6f16fd --- /dev/null +++ b/docs/sphinx/conf.py @@ -0,0 +1,38 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +import os +import sys + +sys.path.insert(0, os.path.abspath("..")) + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = "CodeFlare SDK" +copyright = "2024, Project CodeFlare" +author = "Project CodeFlare" +release = "v0.21.1" + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = [ + "sphinx.ext.autodoc", + "sphinx.ext.todo", + "sphinx.ext.viewcode", + "sphinx.ext.autosummary", + "sphinx_rtd_theme", +] + +templates_path = ["_templates"] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] + + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = "sphinx_rtd_theme" +html_static_path = ["_static"] diff --git a/docs/sphinx/index.rst b/docs/sphinx/index.rst new file mode 100644 index 000000000..fdf4c15b0 --- /dev/null +++ b/docs/sphinx/index.rst @@ -0,0 +1,32 @@ +.. CodeFlare SDK documentation master file, created by + sphinx-quickstart on Thu Oct 10 11:27:58 2024. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +CodeFlare SDK documentation +=========================== + +The CodeFlare SDK is an intuitive, easy-to-use python interface for batch resource requesting, access, job submission, and observation. Simplifying the developer's life while enabling access to high-performance compute resources, either in the cloud or on-prem. + + +.. toctree:: + :maxdepth: 2 + :caption: Code Documentation: + + modules + +.. toctree:: + :maxdepth: 2 + :caption: User Documentation: + + user-docs/authentication + user-docs/cluster-configuration + user-docs/e2e + user-docs/s3-compatible-storage + user-docs/setup-kueue + +Quick Links +=========== +- `PyPi `__ +- `GitHub `__ +- `OpenShift AI Documentation `__ diff --git a/docs/sphinx/make.bat b/docs/sphinx/make.bat new file mode 100644 index 000000000..32bb24529 --- /dev/null +++ b/docs/sphinx/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/sphinx/user-docs/authentication.rst b/docs/sphinx/user-docs/authentication.rst new file mode 100644 index 000000000..d07063d91 --- /dev/null +++ b/docs/sphinx/user-docs/authentication.rst @@ -0,0 +1,66 @@ +Authentication via the CodeFlare SDK +==================================== + +Currently there are four ways of authenticating to your cluster via the +SDK. Authenticating with your cluster allows you to perform actions such +as creating Ray Clusters and Job Submission. + +Method 1 Token Authentication +----------------------------- + +This is how a typical user would authenticate to their cluster using +``TokenAuthentication``. + +:: + + from codeflare_sdk import TokenAuthentication + + auth = TokenAuthentication( + token = "XXXXX", + server = "XXXXX", + skip_tls=False, + # ca_cert_path="/path/to/cert" + ) + auth.login() + # log out with auth.logout() + +Setting ``skip_tls=True`` allows interaction with an HTTPS server +bypassing the server certificate checks although this is not secure. You +can pass a custom certificate to ``TokenAuthentication`` by using +``ca_cert_path="/path/to/cert"`` when authenticating provided +``skip_tls=False``. Alternatively you can set the environment variable +``CF_SDK_CA_CERT_PATH`` to the path of your custom certificate. + +Method 2 Kubernetes Config File Authentication (Default location) +----------------------------------------------------------------- + +If a user has authenticated to their cluster by alternate means e.g. run +a login command like ``oc login --token= --server=`` +their kubernetes config file should have updated. If the user has not +specifically authenticated through the SDK by other means such as +``TokenAuthentication`` then the SDK will try to use their default +Kubernetes config file located at ``"/HOME/.kube/config"``. + +Method 3 Specifying a Kubernetes Config File +-------------------------------------------- + +A user can specify a config file via a different authentication class +``KubeConfigFileAuthentication`` for authenticating with the SDK. This +is what loading a custom config file would typically look like. + +:: + + from codeflare_sdk import KubeConfigFileAuthentication + + auth = KubeConfigFileAuthentication( + kube_config_path="/path/to/config", + ) + auth.load_kube_config() + # log out with auth.logout() + +Method 4 In-Cluster Authentication +---------------------------------- + +If a user does not authenticate by any of the means detailed above and +does not have a config file at ``"/HOME/.kube/config"`` the SDK will try +to authenticate with the in-cluster configuration file. diff --git a/docs/sphinx/user-docs/cluster-configuration.rst b/docs/sphinx/user-docs/cluster-configuration.rst new file mode 100644 index 000000000..1fe28c643 --- /dev/null +++ b/docs/sphinx/user-docs/cluster-configuration.rst @@ -0,0 +1,72 @@ +Ray Cluster Configuration +========================= + +To create Ray Clusters using the CodeFlare SDK a cluster configuration +needs to be created first. This is what a typical cluster configuration +would look like; Note: The values for CPU and Memory are at the minimum +requirements for creating the Ray Cluster. + +.. code:: python + + from codeflare_sdk import Cluster, ClusterConfiguration + + cluster = Cluster(ClusterConfiguration( + name='ray-example', # Mandatory Field + namespace='default', # Default None + head_cpu_requests=1, # Default 2 + head_cpu_limits=1, # Default 2 + head_memory_requests=1, # Default 8 + head_memory_limits=1, # Default 8 + head_extended_resource_requests={'nvidia.com/gpu':0}, # Default 0 + worker_extended_resource_requests={'nvidia.com/gpu':0}, # Default 0 + num_workers=1, # Default 1 + worker_cpu_requests=1, # Default 1 + worker_cpu_limits=1, # Default 1 + worker_memory_requests=2, # Default 2 + worker_memory_limits=2, # Default 2 + # image="", # Optional Field + machine_types=["m5.xlarge", "g4dn.xlarge"], + labels={"exampleLabel": "example", "secondLabel": "example"}, + )) + +Note: ‘quay.io/modh/ray:2.35.0-py39-cu121’ is the default image used by +the CodeFlare SDK for creating a RayCluster resource. If you have your +own Ray image which suits your purposes, specify it in image field to +override the default image. If you are using ROCm compatible GPUs you +can use ‘quay.io/modh/ray:2.35.0-py39-rocm61’. You can also find +documentation on building a custom image +`here `__. + +The ``labels={"exampleLabel": "example"}`` parameter can be used to +apply additional labels to the RayCluster resource. + +After creating their ``cluster``, a user can call ``cluster.up()`` and +``cluster.down()`` to respectively create or remove the Ray Cluster. + +Deprecating Parameters +---------------------- + +The following parameters of the ``ClusterConfiguration`` are being deprecated. + +.. list-table:: + :header-rows: 1 + :widths: auto + + * - Deprecated Parameter + - Replaced By + * - ``head_cpus`` + - ``head_cpu_requests``, ``head_cpu_limits`` + * - ``head_memory`` + - ``head_memory_requests``, ``head_memory_limits`` + * - ``min_cpus`` + - ``worker_cpu_requests`` + * - ``max_cpus`` + - ``worker_cpu_limits`` + * - ``min_memory`` + - ``worker_memory_requests`` + * - ``max_memory`` + - ``worker_memory_limits`` + * - ``head_gpus`` + - ``head_extended_resource_requests`` + * - ``num_gpus`` + - ``worker_extended_resource_requests`` diff --git a/docs/sphinx/user-docs/e2e.rst b/docs/sphinx/user-docs/e2e.rst new file mode 100644 index 000000000..e64032e20 --- /dev/null +++ b/docs/sphinx/user-docs/e2e.rst @@ -0,0 +1,210 @@ +Running e2e tests locally +========================= + +Pre-requisites +^^^^^^^^^^^^^^ + +- We recommend using Python 3.9, along with Poetry. + +On KinD clusters +---------------- + +Pre-requisite for KinD clusters: please add in your local ``/etc/hosts`` +file ``127.0.0.1 kind``. This will map your localhost IP address to the +KinD cluster’s hostname. This is already performed on `GitHub +Actions `__ + +If the system you run on contains NVidia GPU then you can enable the GPU +support in KinD, this will allow you to run also GPU tests. To enable +GPU on KinD follow `these +instructions `__. + +- Setup Phase: + + - Pull the `codeflare-operator + repo `__ + and run the following make targets: + + :: + + make kind-e2e + export CLUSTER_HOSTNAME=kind + make setup-e2e + make deploy -e IMG=quay.io/project-codeflare/codeflare-operator:v1.3.0 + + For running tests locally on Kind cluster, we need to disable `rayDashboardOAuthEnabled` in `codeflare-operator-config` ConfigMap and then restart CodeFlare Operator + + - **(Optional)** - Create and add ``sdk-user`` with limited + permissions to the cluster to run through the e2e tests: + + :: + + # Get KinD certificates + docker cp kind-control-plane:/etc/kubernetes/pki/ca.crt . + docker cp kind-control-plane:/etc/kubernetes/pki/ca.key . + + # Generate certificates for new user + openssl genrsa -out user.key 2048 + openssl req -new -key user.key -out user.csr -subj '/CN=sdk-user/O=tenant' + openssl x509 -req -in user.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out user.crt -days 360 + + # Add generated certificated to KinD context + user_crt=$(base64 --wrap=0 user.crt) + user_key=$(base64 --wrap=0 user.key) + yq eval -i ".contexts += {\"context\": {\"cluster\": \"kind-kind\", \"user\": \"sdk-user\"}, \"name\": \"sdk-user\"}" $HOME/.kube/config + yq eval -i ".users += {\"name\": \"sdk-user\", \"user\": {\"client-certificate-data\": \"$user_crt\", \"client-key-data\": \"$user_key\"}}" $HOME/.kube/config + cat $HOME/.kube/config + + # Cleanup + rm ca.crt + rm ca.srl + rm ca.key + rm user.crt + rm user.key + rm user.csr + + # Add RBAC permissions to sdk-user + kubectl create clusterrole list-ingresses --verb=get,list --resource=ingresses + kubectl create clusterrolebinding sdk-user-list-ingresses --clusterrole=list-ingresses --user=sdk-user + kubectl create clusterrole appwrapper-creator --verb=get,list,create,delete,patch --resource=appwrappers + kubectl create clusterrolebinding sdk-user-appwrapper-creator --clusterrole=appwrapper-creator --user=sdk-user + kubectl create clusterrole namespace-creator --verb=get,list,create,delete,patch --resource=namespaces + kubectl create clusterrolebinding sdk-user-namespace-creator --clusterrole=namespace-creator --user=sdk-user + kubectl create clusterrole list-rayclusters --verb=get,list --resource=rayclusters + kubectl create clusterrolebinding sdk-user-list-rayclusters --clusterrole=list-rayclusters --user=sdk-user + kubectl config use-context sdk-user + + - Install the latest development version of kueue + + :: + + kubectl apply --server-side -k "github.com/opendatahub-io/kueue/config/rhoai?ref=dev" + +- Test Phase: + + - Once we have the codeflare-operator, kuberay-operator and kueue + running and ready, we can run the e2e test on the codeflare-sdk + repository: + + :: + + poetry install --with test,docs + poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_kind_test.py + + - If the cluster doesn’t have NVidia GPU support then we need to + disable NVidia GPU tests by providing proper marker: + + :: + + poetry install --with test,docs + poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_kind_test.py -m 'kind and not nvidia_gpu' + +On OpenShift clusters +--------------------- + +- Setup Phase: + + - Pull the `codeflare-operator + repo `__ + and run the following make targets: + + :: + + + make setup-e2e + make deploy -e IMG=quay.io/project-codeflare/codeflare-operator:v1.3.0 + + - Install the latest development version of kueue + + :: + + kubectl apply --server-side -k "github.com/opendatahub-io/kueue/config/rhoai?ref=dev" + +If the system you run on contains NVidia GPU then you can enable the GPU +support on OpenShift, this will allow you to run also GPU tests. To +enable GPU on OpenShift follow `these +instructions `__. +Currently the SDK doesn’t support tolerations, so e2e tests can’t be +executed on nodes with taint (i.e. GPU taint). + +- Test Phase: + + - Once we have the codeflare-operator, kuberay-operator and kueue + running and ready, we can run the e2e test on the codeflare-sdk + repository: + + :: + + poetry install --with test,docs + poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_test.py + + - To run the multiple tests based on the cluster environment, we can + run the e2e tests by marking -m with cluster environment (kind or + openshift) + + :: + + poetry run pytest -v -s ./tests/e2e -m openshift + + - By default tests configured with timeout of ``15 minutes``. If + necessary, we can override the timeout using ``--timeout`` option + + :: + + poetry run pytest -v -s ./tests/e2e -m openshift --timeout=1200 + +On OpenShift Disconnected clusters +---------------------------------- + +- In addition to setup phase mentioned above in case of Openshift + cluster, Disconnected environment requires following pre-requisites : + + - Mirror Image registry : + + - Image mirror registry is used to host set of container images + required locally for the applications and services. This + ensures to pull images without needing an external network + connection. It also ensures continuous operation and deployment + capabilities in a network-isolated environment. + + - PYPI Mirror Index : + + - When trying to install Python packages in a disconnected + environment, the pip command might fail because the connection + cannot install packages from external URLs. This issue can be + resolved by setting up PIP Mirror Index on separate endpoint in + same environment. + + - S3 compatible storage : + + - Some of our distributed training examples require an external + storage solution so that all nodes can access the same data in + disconnected environment (For example: common-datasets and + model files). + + - Minio S3 compatible storage type instance can be deployed in + disconnected environment using + ``/tests/e2e/minio_deployment.yaml`` or using support methods + in e2e test suite. + + - The following are environment variables for configuring PIP + index URl for accessing the common-python packages required and + the S3 or Minio storage for your Ray Train script or + interactive session. + + :: + + export RAY_IMAGE=quay.io/project-codeflare/ray@sha256: (prefer image digest over image tag in disocnnected environment) + PIP_INDEX_URL=https:///root/pypi/+simple/ \ + PIP_TRUSTED_HOST= \ + AWS_DEFAULT_ENDPOINT= \ + AWS_ACCESS_KEY_ID= \ + AWS_SECRET_ACCESS_KEY= \ + AWS_STORAGE_BUCKET= + AWS_STORAGE_BUCKET_MNIST_DIR= + + Note : When using the Python Minio client to connect to a minio + storage bucket, the ``AWS_DEFAULT_ENDPOINT`` environment + variable by default expects secure endpoint where user can use + endpoint url with https/http prefix for autodetection of + secure/insecure endpoint. diff --git a/docs/sphinx/user-docs/s3-compatible-storage.rst b/docs/sphinx/user-docs/s3-compatible-storage.rst new file mode 100644 index 000000000..60937441b --- /dev/null +++ b/docs/sphinx/user-docs/s3-compatible-storage.rst @@ -0,0 +1,86 @@ +S3 compatible storage with Ray Train examples +============================================= + +Some of our distributed training examples require an external storage +solution so that all nodes can access the same data. The following are +examples for configuring S3 or Minio storage for your Ray Train script +or interactive session. + +S3 Bucket +--------- + +In your Python Script add the following environment variables: + +.. code:: python + + os.environ["AWS_ACCESS_KEY_ID"] = "XXXXXXXX" + os.environ["AWS_SECRET_ACCESS_KEY"] = "XXXXXXXX" + os.environ["AWS_DEFAULT_REGION"] = "XXXXXXXX" + +Alternatively you can specify these variables in your runtime +environment on Job Submission. + +.. code:: python + + submission_id = client.submit_job( + entrypoint=..., + runtime_env={ + "env_vars": { + "AWS_ACCESS_KEY_ID": os.environ.get('AWS_ACCESS_KEY_ID'), + "AWS_SECRET_ACCESS_KEY": os.environ.get('AWS_SECRET_ACCESS_KEY'), + "AWS_DEFAULT_REGION": os.environ.get('AWS_DEFAULT_REGION') + }, + } + ) + +In your Trainer configuration you can specify a ``run_config`` which +will utilise your external storage. + +.. code:: python + + trainer = TorchTrainer( + train_func_distributed, + scaling_config=scaling_config, + run_config = ray.train.RunConfig(storage_path="s3://BUCKET_NAME/SUB_PATH/", name="unique_run_name") + ) + +To learn more about Amazon S3 Storage you can find information +`here `__. + +Minio Bucket +------------ + +In your Python Script add the following function for configuring your +run_config: + +.. code:: python + + import s3fs + import pyarrow + + def get_minio_run_config(): + s3_fs = s3fs.S3FileSystem( + key = os.getenv('MINIO_ACCESS_KEY', "XXXXX"), + secret = os.getenv('MINIO_SECRET_ACCESS_KEY', "XXXXX"), + endpoint_url = os.getenv('MINIO_URL', "XXXXX") + ) + custom_fs = pyarrow.fs.PyFileSystem(pyarrow.fs.FSSpecHandler(s3_fs)) + run_config = ray.train.RunConfig(storage_path='training', storage_filesystem=custom_fs) + return run_config + +You can update the ``run_config`` to further suit your needs above. +Lastly the new ``run_config`` must be added to the Trainer: + +.. code:: python + + trainer = TorchTrainer( + train_func_distributed, + scaling_config=scaling_config, + run_config = get_minio_run_config() + ) + +To find more information on creating a Minio Bucket compatible with +RHOAI you can refer to this +`documentation `__. +Note: You must have ``sf3s`` and ``pyarrow`` installed in your +environment for this method. diff --git a/docs/sphinx/user-docs/setup-kueue.rst b/docs/sphinx/user-docs/setup-kueue.rst new file mode 100644 index 000000000..86956e011 --- /dev/null +++ b/docs/sphinx/user-docs/setup-kueue.rst @@ -0,0 +1,109 @@ +Basic Kueue Resources configuration +=================================== + +Introduction: +------------- + +This document is designed for administrators who have Kueue installed on +their cluster. We will walk through the process of setting up essential +Kueue resources, namely Cluster Queue, Resource Flavor, and Local Queue. + +1. Resource Flavor: +------------------- + +Resource Flavors allow the cluster admin to define different types of +resources with specific characteristics, such as CPU, memory, GPU, etc. +These can then be assigned to workloads to ensure they are executed on +appropriate resources. + +The YAML configuration provided below creates an empty Resource Flavor +named default-flavor. It serves as a starting point and does not specify +any detailed resource characteristics. + +.. code:: yaml + + apiVersion: kueue.x-k8s.io/v1beta1 + kind: ResourceFlavor + metadata: + name: default-flavor + +For more detailed information on Resource Flavor configuration options, +refer to the Kueue documentation: `Resource Flavor +Configuration `__ + +2. Cluster Queue: +----------------- + +A Cluster Queue represents a shared queue across the entire cluster. It +allows the cluster admin to define global settings for workload +prioritization and resource allocation. + +When setting up a Cluster Queue in Kueue, it’s crucial that the resource +specifications match the actual capacities and operational requirements +of your cluster. The example provided outlines a basic setup; however, +each cluster may have different resource availabilities and needs. + +.. code:: yaml + + apiVersion: kueue.x-k8s.io/v1beta1 + kind: ClusterQueue + metadata: + name: "cluster-queue" + spec: + namespaceSelector: {} # match all. + resourceGroups: + - coveredResources: ["cpu", "memory", "pods", "nvidia.com/gpu"] + flavors: + - name: "default-flavor" + resources: + - name: "cpu" + nominalQuota: 9 + - name: "memory" + nominalQuota: 36Gi + - name: "pods" + nominalQuota: 5 + - name: "nvidia.com/gpu" + nominalQuota: '0' + +For more detailed information on Cluster Queue configuration options, +refer to the Kueue documentation: `Cluster Queue +Configuration `__ + +3. Local Queue (With Default Annotation): +----------------------------------------- + +A Local Queue represents a queue associated with a specific namespace +within the cluster. It allows namespace-level control over workload +prioritization and resource allocation. + +.. code:: yaml + + apiVersion: kueue.x-k8s.io/v1beta1 + kind: LocalQueue + metadata: + namespace: team-a + name: team-a-queue + annotations: + kueue.x-k8s.io/default-queue: "true" + spec: + clusterQueue: cluster-queue + +In the LocalQueue configuration provided above, the annotations field +specifies ``kueue.x-k8s.io/default-queue: "true"``. This annotation +indicates that the team-a-queue is designated as the default queue for +the team-a namespace. When this is set, any workloads submitted to the +team-a namespace without explicitly specifying a queue will +automatically be routed to the team-a-queue. + +For more detailed information on Local Queue configuration options, +refer to the Kueue documentation: `Local Queue +Configuration `__ + +Conclusion: +----------- + +By following the steps outlined in this document, the cluster admin can +successfully create the basic Kueue resources necessary for workload +management in the cluster. For more advanced configurations and +features, please refer to the comprehensive `Kueue +documentation `__. diff --git a/poetry.lock b/poetry.lock index 3b65c16b4..5c4ce93f8 100644 --- a/poetry.lock +++ b/poetry.lock @@ -151,6 +151,17 @@ files = [ [package.dependencies] frozenlist = ">=1.1.0" +[[package]] +name = "alabaster" +version = "0.7.16" +description = "A light, configurable Sphinx theme" +optional = false +python-versions = ">=3.9" +files = [ + {file = "alabaster-0.7.16-py3-none-any.whl", hash = "sha256:b46733c07dce03ae4e150330b975c75737fa60f0a7c591b6c8bf4928a28e2c92"}, + {file = "alabaster-0.7.16.tar.gz", hash = "sha256:75a8b99c28a5dad50dd7f8ccdd447a121ddb3892da9e53d1ca5cca3106d58d65"}, +] + [[package]] name = "anyio" version = "4.6.0" @@ -862,6 +873,17 @@ files = [ {file = "distlib-0.3.8.tar.gz", hash = "sha256:1530ea13e350031b6312d8580ddb6b27a104275a31106523b8f123787f494f64"}, ] +[[package]] +name = "docutils" +version = "0.20.1" +description = "Docutils -- Python Documentation Utilities" +optional = false +python-versions = ">=3.7" +files = [ + {file = "docutils-0.20.1-py3-none-any.whl", hash = "sha256:96f387a2c5562db4476f09f13bbab2192e764cac08ebbf3a34a95d9b1e4a59d6"}, + {file = "docutils-0.20.1.tar.gz", hash = "sha256:f08a4e276c3a1583a86dce3e34aba3fe04d02bba2dd51ed16106244e8a923e3b"}, +] + [[package]] name = "exceptiongroup" version = "1.2.2" @@ -1257,6 +1279,17 @@ files = [ [package.extras] all = ["flake8 (>=7.1.1)", "mypy (>=1.11.2)", "pytest (>=8.3.2)", "ruff (>=0.6.2)"] +[[package]] +name = "imagesize" +version = "1.4.1" +description = "Getting image size from png/jpeg/jpeg2000/gif file" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" +files = [ + {file = "imagesize-1.4.1-py2.py3-none-any.whl", hash = "sha256:0d8d18d08f840c19d0ee7ca1fd82490fdc3729b7ac93f49870406ddde8ef8d8b"}, + {file = "imagesize-1.4.1.tar.gz", hash = "sha256:69150444affb9cb0d5cc5a92b3676f0b2fb7cd9ae39e947a5e11a36b4497cd4a"}, +] + [[package]] name = "importlib-metadata" version = "8.5.0" @@ -1744,43 +1777,6 @@ websocket-client = ">=0.32.0,<0.40.0 || >0.40.0,<0.41.dev0 || >=0.43.dev0" [package.extras] adal = ["adal (>=1.0.2)"] -[[package]] -name = "mako" -version = "1.3.5" -description = "A super-fast templating language that borrows the best ideas from the existing templating languages." -optional = false -python-versions = ">=3.8" -files = [ - {file = "Mako-1.3.5-py3-none-any.whl", hash = "sha256:260f1dbc3a519453a9c856dedfe4beb4e50bd5a26d96386cb6c80856556bb91a"}, - {file = "Mako-1.3.5.tar.gz", hash = "sha256:48dbc20568c1d276a2698b36d968fa76161bf127194907ea6fc594fa81f943bc"}, -] - -[package.dependencies] -MarkupSafe = ">=0.9.2" - -[package.extras] -babel = ["Babel"] -lingua = ["lingua"] -testing = ["pytest"] - -[[package]] -name = "markdown" -version = "3.7" -description = "Python implementation of John Gruber's Markdown." -optional = false -python-versions = ">=3.8" -files = [ - {file = "Markdown-3.7-py3-none-any.whl", hash = "sha256:7eb6df5690b81a1d7942992c97fad2938e956e79df20cbc6186e9c3a77b1c803"}, - {file = "markdown-3.7.tar.gz", hash = "sha256:2ae2471477cfd02dbbf038d5d9bc226d40def84b4fe2986e49b59b6b472bbed2"}, -] - -[package.dependencies] -importlib-metadata = {version = ">=4.4", markers = "python_version < \"3.10\""} - -[package.extras] -docs = ["mdx-gh-links (>=0.2)", "mkdocs (>=1.5)", "mkdocs-gen-files", "mkdocs-literate-nav", "mkdocs-nature (>=0.6)", "mkdocs-section-index", "mkdocstrings[python]"] -testing = ["coverage", "pyyaml"] - [[package]] name = "markupsafe" version = "2.1.5" @@ -2484,21 +2480,6 @@ files = [ qa = ["flake8 (==5.0.4)", "mypy (==0.971)", "types-setuptools (==67.2.0.1)"] testing = ["docopt", "pytest"] -[[package]] -name = "pdoc3" -version = "0.10.0" -description = "Auto-generate API documentation for Python projects." -optional = false -python-versions = ">= 3.6" -files = [ - {file = "pdoc3-0.10.0-py3-none-any.whl", hash = "sha256:ba45d1ada1bd987427d2bf5cdec30b2631a3ff5fb01f6d0e77648a572ce6028b"}, - {file = "pdoc3-0.10.0.tar.gz", hash = "sha256:5f22e7bcb969006738e1aa4219c75a32f34c2d62d46dc9d2fb2d3e0b0287e4b7"}, -] - -[package.dependencies] -mako = "*" -markdown = ">=3.0" - [[package]] name = "pexpect" version = "4.9.0" @@ -3556,6 +3537,17 @@ files = [ {file = "sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc"}, ] +[[package]] +name = "snowballstemmer" +version = "2.2.0" +description = "This package provides 29 stemmers for 28 languages generated from Snowball algorithms." +optional = false +python-versions = "*" +files = [ + {file = "snowballstemmer-2.2.0-py2.py3-none-any.whl", hash = "sha256:c8e1716e83cc398ae16824e5572ae04e0d9fc2c6b985fb0f900f5f0c96ecba1a"}, + {file = "snowballstemmer-2.2.0.tar.gz", hash = "sha256:09b16deb8547d3412ad7b590689584cd0fe25ec8db3be37788be3810cbf19cb1"}, +] + [[package]] name = "soupsieve" version = "2.6" @@ -3567,6 +3559,169 @@ files = [ {file = "soupsieve-2.6.tar.gz", hash = "sha256:e2e68417777af359ec65daac1057404a3c8a5455bb8abc36f1a9866ab1a51abb"}, ] +[[package]] +name = "sphinx" +version = "7.4.7" +description = "Python documentation generator" +optional = false +python-versions = ">=3.9" +files = [ + {file = "sphinx-7.4.7-py3-none-any.whl", hash = "sha256:c2419e2135d11f1951cd994d6eb18a1835bd8fdd8429f9ca375dc1f3281bd239"}, + {file = "sphinx-7.4.7.tar.gz", hash = "sha256:242f92a7ea7e6c5b406fdc2615413890ba9f699114a9c09192d7dfead2ee9cfe"}, +] + +[package.dependencies] +alabaster = ">=0.7.14,<0.8.0" +babel = ">=2.13" +colorama = {version = ">=0.4.6", markers = "sys_platform == \"win32\""} +docutils = ">=0.20,<0.22" +imagesize = ">=1.3" +importlib-metadata = {version = ">=6.0", markers = "python_version < \"3.10\""} +Jinja2 = ">=3.1" +packaging = ">=23.0" +Pygments = ">=2.17" +requests = ">=2.30.0" +snowballstemmer = ">=2.2" +sphinxcontrib-applehelp = "*" +sphinxcontrib-devhelp = "*" +sphinxcontrib-htmlhelp = ">=2.0.0" +sphinxcontrib-jsmath = "*" +sphinxcontrib-qthelp = "*" +sphinxcontrib-serializinghtml = ">=1.1.9" +tomli = {version = ">=2", markers = "python_version < \"3.11\""} + +[package.extras] +docs = ["sphinxcontrib-websupport"] +lint = ["flake8 (>=6.0)", "importlib-metadata (>=6.0)", "mypy (==1.10.1)", "pytest (>=6.0)", "ruff (==0.5.2)", "sphinx-lint (>=0.9)", "tomli (>=2)", "types-docutils (==0.21.0.20240711)", "types-requests (>=2.30.0)"] +test = ["cython (>=3.0)", "defusedxml (>=0.7.1)", "pytest (>=8.0)", "setuptools (>=70.0)", "typing_extensions (>=4.9)"] + +[[package]] +name = "sphinx-rtd-theme" +version = "2.0.0" +description = "Read the Docs theme for Sphinx" +optional = false +python-versions = ">=3.6" +files = [ + {file = "sphinx_rtd_theme-2.0.0-py2.py3-none-any.whl", hash = "sha256:ec93d0856dc280cf3aee9a4c9807c60e027c7f7b461b77aeffed682e68f0e586"}, + {file = "sphinx_rtd_theme-2.0.0.tar.gz", hash = "sha256:bd5d7b80622406762073a04ef8fadc5f9151261563d47027de09910ce03afe6b"}, +] + +[package.dependencies] +docutils = "<0.21" +sphinx = ">=5,<8" +sphinxcontrib-jquery = ">=4,<5" + +[package.extras] +dev = ["bump2version", "sphinxcontrib-httpdomain", "transifex-client", "wheel"] + +[[package]] +name = "sphinxcontrib-applehelp" +version = "2.0.0" +description = "sphinxcontrib-applehelp is a Sphinx extension which outputs Apple help books" +optional = false +python-versions = ">=3.9" +files = [ + {file = "sphinxcontrib_applehelp-2.0.0-py3-none-any.whl", hash = "sha256:4cd3f0ec4ac5dd9c17ec65e9ab272c9b867ea77425228e68ecf08d6b28ddbdb5"}, + {file = "sphinxcontrib_applehelp-2.0.0.tar.gz", hash = "sha256:2f29ef331735ce958efa4734873f084941970894c6090408b079c61b2e1c06d1"}, +] + +[package.extras] +lint = ["mypy", "ruff (==0.5.5)", "types-docutils"] +standalone = ["Sphinx (>=5)"] +test = ["pytest"] + +[[package]] +name = "sphinxcontrib-devhelp" +version = "2.0.0" +description = "sphinxcontrib-devhelp is a sphinx extension which outputs Devhelp documents" +optional = false +python-versions = ">=3.9" +files = [ + {file = "sphinxcontrib_devhelp-2.0.0-py3-none-any.whl", hash = "sha256:aefb8b83854e4b0998877524d1029fd3e6879210422ee3780459e28a1f03a8a2"}, + {file = "sphinxcontrib_devhelp-2.0.0.tar.gz", hash = "sha256:411f5d96d445d1d73bb5d52133377b4248ec79db5c793ce7dbe59e074b4dd1ad"}, +] + +[package.extras] +lint = ["mypy", "ruff (==0.5.5)", "types-docutils"] +standalone = ["Sphinx (>=5)"] +test = ["pytest"] + +[[package]] +name = "sphinxcontrib-htmlhelp" +version = "2.1.0" +description = "sphinxcontrib-htmlhelp is a sphinx extension which renders HTML help files" +optional = false +python-versions = ">=3.9" +files = [ + {file = "sphinxcontrib_htmlhelp-2.1.0-py3-none-any.whl", hash = "sha256:166759820b47002d22914d64a075ce08f4c46818e17cfc9470a9786b759b19f8"}, + {file = "sphinxcontrib_htmlhelp-2.1.0.tar.gz", hash = "sha256:c9e2916ace8aad64cc13a0d233ee22317f2b9025b9cf3295249fa985cc7082e9"}, +] + +[package.extras] +lint = ["mypy", "ruff (==0.5.5)", "types-docutils"] +standalone = ["Sphinx (>=5)"] +test = ["html5lib", "pytest"] + +[[package]] +name = "sphinxcontrib-jquery" +version = "4.1" +description = "Extension to include jQuery on newer Sphinx releases" +optional = false +python-versions = ">=2.7" +files = [ + {file = "sphinxcontrib-jquery-4.1.tar.gz", hash = "sha256:1620739f04e36a2c779f1a131a2dfd49b2fd07351bf1968ced074365933abc7a"}, + {file = "sphinxcontrib_jquery-4.1-py2.py3-none-any.whl", hash = "sha256:f936030d7d0147dd026a4f2b5a57343d233f1fc7b363f68b3d4f1cb0993878ae"}, +] + +[package.dependencies] +Sphinx = ">=1.8" + +[[package]] +name = "sphinxcontrib-jsmath" +version = "1.0.1" +description = "A sphinx extension which renders display math in HTML via JavaScript" +optional = false +python-versions = ">=3.5" +files = [ + {file = "sphinxcontrib-jsmath-1.0.1.tar.gz", hash = "sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8"}, + {file = "sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl", hash = "sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178"}, +] + +[package.extras] +test = ["flake8", "mypy", "pytest"] + +[[package]] +name = "sphinxcontrib-qthelp" +version = "2.0.0" +description = "sphinxcontrib-qthelp is a sphinx extension which outputs QtHelp documents" +optional = false +python-versions = ">=3.9" +files = [ + {file = "sphinxcontrib_qthelp-2.0.0-py3-none-any.whl", hash = "sha256:b18a828cdba941ccd6ee8445dbe72ffa3ef8cbe7505d8cd1fa0d42d3f2d5f3eb"}, + {file = "sphinxcontrib_qthelp-2.0.0.tar.gz", hash = "sha256:4fe7d0ac8fc171045be623aba3e2a8f613f8682731f9153bb2e40ece16b9bbab"}, +] + +[package.extras] +lint = ["mypy", "ruff (==0.5.5)", "types-docutils"] +standalone = ["Sphinx (>=5)"] +test = ["defusedxml (>=0.7.1)", "pytest"] + +[[package]] +name = "sphinxcontrib-serializinghtml" +version = "2.0.0" +description = "sphinxcontrib-serializinghtml is a sphinx extension which outputs \"serialized\" HTML files (json and pickle)" +optional = false +python-versions = ">=3.9" +files = [ + {file = "sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl", hash = "sha256:6e2cb0eef194e10c27ec0023bfeb25badbbb5868244cf5bc5bdc04e4464bf331"}, + {file = "sphinxcontrib_serializinghtml-2.0.0.tar.gz", hash = "sha256:e9d912827f872c029017a53f0ef2180b327c3f7fd23c87229f7a8e8b70031d4d"}, +] + +[package.extras] +lint = ["mypy", "ruff (==0.5.5)", "types-docutils"] +standalone = ["Sphinx (>=5)"] +test = ["pytest"] + [[package]] name = "stack-data" version = "0.6.3" @@ -4025,4 +4180,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.0" python-versions = "^3.9" -content-hash = "a54e3ebe29255d397651cea6d849ada39f03565a1a7bf13084092be3600a77f0" +content-hash = "4463099e8d145fd823f523b134f18d48766038cc3d2ad466864e5a2debcc3479" diff --git a/pyproject.toml b/pyproject.toml index 37eb17a44..17b598804 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,7 +34,8 @@ ipywidgets = "8.1.2" optional = true [tool.poetry.group.docs.dependencies] -pdoc3 = "0.10.0" +sphinx = "7.4.7" +sphinx-rtd-theme = "2.0.0" [tool.poetry.group.test] optional = true From 7f4b9b745986bb277df4fd4936518c7a2730797a Mon Sep 17 00:00:00 2001 From: Bobbins228 Date: Mon, 14 Oct 2024 14:30:36 +0100 Subject: [PATCH 2/3] ci: add publish documentation workflow update release workflow --- .github/workflows/publish-documentation.yaml | 45 ++++++++++++++++++++ .github/workflows/release.yaml | 29 ++----------- .gitignore | 1 + 3 files changed, 50 insertions(+), 25 deletions(-) create mode 100644 .github/workflows/publish-documentation.yaml diff --git a/.github/workflows/publish-documentation.yaml b/.github/workflows/publish-documentation.yaml new file mode 100644 index 000000000..80afe7d6e --- /dev/null +++ b/.github/workflows/publish-documentation.yaml @@ -0,0 +1,45 @@ +name: Publish Documentation + +on: + workflow_dispatch: + inputs: + codeflare_sdk_release_version: + type: string + required: true + description: 'Version number (for example: 0.1.0)' + +permissions: + contents: write + +jobs: + docs: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install Python + uses: actions/setup-python@v5 + with: + python-version: 3.9 + - name: Install Sphinx + run: | + sudo apt-get update + sudo apt-get install python3-sphinx + - name: Install Poetry + uses: abatilo/actions-poetry@v2 + with: + poetry-version: 1.8.3 + - name: Create new documentation + run: | + python3 -m venv .venv + source .venv/bin/activate + poetry install --with docs + sed -i 's/release = "v[0-9]\+\.[0-9]\+\.[0-9]\+"/release = "${{ github.event.inputs.codeflare_sdk_release_version }}"/' docs/sphinx/conf.py + sphinx-apidoc -o docs/sphinx src/codeflare_sdk "**/*test_*" --force # Generate docs but ignore test files + make html -C docs/sphinx + - name: Deploy to GitHub Pages + uses: peaceiris/actions-gh-pages@v3 + with: + publish_branch: gh-pages + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: docs/sphinx/_build/html + force_orphan: true diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 6e56a3f86..16b5aac43 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -33,8 +33,6 @@ jobs: permissions: contents: write id-token: write # This permission is required for trusted publishing - env: - PR_BRANCH_NAME: adjustments-release-${{ github.event.inputs.release-version }} steps: - name: Checkout the repository uses: actions/checkout@v4 @@ -52,35 +50,16 @@ jobs: run: poetry install --with docs - name: Create new documentation run: | - sphinx-apidoc -o docs/sphinx src/codeflare_sdk "**/*test_*" --force - make clean -C docs/sphinx - make html -C docs/sphinx + gh workflow run publish-documentation.yaml \ + --repo ${{ github.event.inputs.codeflare-repository-organization }}/codeflare-sdk \ + --ref ${{ github.ref }} \ + --field codeflare_sdk_release_version=${{ github.event.inputs.release-version }} - name: Copy demo notebooks into SDK package run: cp -r demo-notebooks src/codeflare_sdk/demo-notebooks - name: Run poetry build run: poetry build - - name: Commit changes in docs - uses: stefanzweifel/git-auto-commit-action@v4 - with: - file_pattern: 'docs' - commit_message: "Changes in docs for release: v${{ github.event.inputs.release-version }}" - create_branch: true - branch: ${{ env.PR_BRANCH_NAME }} - - name: Create a PR with code changes - run: | - if git branch -a | grep "${{ env.PR_BRANCH_NAME }}"; then - GIT_BRANCH=${GITHUB_REF#refs/heads/} - gh pr create --base "$GIT_BRANCH" --fill --head "${{ env.PR_BRANCH_NAME }}" --label "lgtm" --label "approved" - fi env: GITHUB_TOKEN: ${{ secrets.CODEFLARE_MACHINE_ACCOUNT_TOKEN }} - - name: Wait until PR with code changes is merged - run: | - if git branch -a | grep "${{ env.PR_BRANCH_NAME }}"; then - timeout 3600 bash -c 'until [[ $(gh pr view '${{ env.PR_BRANCH_NAME }}' --json state --jq .state) == "MERGED" ]]; do sleep 5 && echo "$(gh pr view '${{ env.PR_BRANCH_NAME }}' --json state --jq .state)"; done' - fi - env: - GITHUB_TOKEN: ${{ github.TOKEN }} - name: Create Github release uses: ncipollo/release-action@v1 with: diff --git a/.gitignore b/.gitignore index 2940f885b..9ac5d6873 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,4 @@ node_modules ui-tests/playwright-report ui-tests/test-results /src/codeflare_sdk.egg-info/ +docs/sphinx/_build From a921928e0ef88d33ff0b70d16e8931628f424b00 Mon Sep 17 00:00:00 2001 From: Bobbins228 Date: Tue, 15 Oct 2024 10:44:30 +0100 Subject: [PATCH 3/3] docs: add new documentation and fix existing docs --- README.md | 5 +- docs/sphinx/index.rst | 4 +- docs/sphinx/user-docs/authentication.rst | 4 +- .../user-docs/cluster-configuration.rst | 18 ++-- docs/sphinx/user-docs/e2e.rst | 19 ++-- docs/sphinx/user-docs/images/ui-buttons.png | Bin 0 -> 22385 bytes .../user-docs/images/ui-view-clusters.png | Bin 0 -> 28767 bytes .../user-docs/ray-cluster-interaction.rst | 90 ++++++++++++++++++ .../user-docs/s3-compatible-storage.rst | 2 +- docs/sphinx/user-docs/setup-kueue.rst | 7 +- docs/sphinx/user-docs/ui-widgets.rst | 55 +++++++++++ 11 files changed, 176 insertions(+), 28 deletions(-) create mode 100644 docs/sphinx/user-docs/images/ui-buttons.png create mode 100644 docs/sphinx/user-docs/images/ui-view-clusters.png create mode 100644 docs/sphinx/user-docs/ray-cluster-interaction.rst create mode 100644 docs/sphinx/user-docs/ui-widgets.rst diff --git a/README.md b/README.md index e166b4f53..ffc226268 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ For guided demos and basics walkthroughs, check out the following links: - these demos can be copied into your current working directory when using the `codeflare-sdk` by using the `codeflare_sdk.copy_demo_nbs()` function - Additionally, we have a [video walkthrough](https://www.youtube.com/watch?v=U76iIfd9EmE) of these basic demos from June, 2023 -Full documentation can be found [here](https://project-codeflare.github.io/codeflare-sdk/detailed-documentation) +Full documentation can be found [here](https://project-codeflare.github.io/codeflare-sdk/index.html) ## Installation @@ -32,11 +32,10 @@ It is possible to use the Release Github workflow to do the release. This is gen The following instructions apply when doing release manually. This may be required in instances where the automation is failing. - Check and update the version in "pyproject.toml" file. -- Generate new documentation. -`pdoc --html -o docs src/codeflare_sdk && pushd docs && rm -rf cluster job utils && mv codeflare_sdk/* . && rm -rf codeflare_sdk && popd && find docs -type f -name "*.html" -exec bash -c "echo '' >> {}" \;` (it is possible to install **pdoc** using the following command `poetry install --with docs`) - Commit all the changes to the repository. - Create Github release (). - Build the Python package. `poetry build` - If not present already, add the API token to Poetry. `poetry config pypi-token.pypi API_TOKEN` - Publish the Python package. `poetry publish` +- Trigger the [Publish Documentation](https://github.com/project-codeflare/codeflare-sdk/actions/workflows/publish-documentation.yaml) workflow diff --git a/docs/sphinx/index.rst b/docs/sphinx/index.rst index fdf4c15b0..3c6fe876f 100644 --- a/docs/sphinx/index.rst +++ b/docs/sphinx/index.rst @@ -16,14 +16,16 @@ The CodeFlare SDK is an intuitive, easy-to-use python interface for batch resour modules .. toctree:: - :maxdepth: 2 + :maxdepth: 1 :caption: User Documentation: user-docs/authentication user-docs/cluster-configuration + user-docs/ray-cluster-interaction user-docs/e2e user-docs/s3-compatible-storage user-docs/setup-kueue + user-docs/ui-widgets Quick Links =========== diff --git a/docs/sphinx/user-docs/authentication.rst b/docs/sphinx/user-docs/authentication.rst index d07063d91..82441d564 100644 --- a/docs/sphinx/user-docs/authentication.rst +++ b/docs/sphinx/user-docs/authentication.rst @@ -39,7 +39,7 @@ a login command like ``oc login --token= --server=`` their kubernetes config file should have updated. If the user has not specifically authenticated through the SDK by other means such as ``TokenAuthentication`` then the SDK will try to use their default -Kubernetes config file located at ``"/HOME/.kube/config"``. +Kubernetes config file located at ``"$HOME/.kube/config"``. Method 3 Specifying a Kubernetes Config File -------------------------------------------- @@ -62,5 +62,5 @@ Method 4 In-Cluster Authentication ---------------------------------- If a user does not authenticate by any of the means detailed above and -does not have a config file at ``"/HOME/.kube/config"`` the SDK will try +does not have a config file at ``"$HOME/.kube/config"`` the SDK will try to authenticate with the in-cluster configuration file. diff --git a/docs/sphinx/user-docs/cluster-configuration.rst b/docs/sphinx/user-docs/cluster-configuration.rst index 1fe28c643..6d27b0f41 100644 --- a/docs/sphinx/user-docs/cluster-configuration.rst +++ b/docs/sphinx/user-docs/cluster-configuration.rst @@ -29,13 +29,14 @@ requirements for creating the Ray Cluster. labels={"exampleLabel": "example", "secondLabel": "example"}, )) -Note: ‘quay.io/modh/ray:2.35.0-py39-cu121’ is the default image used by -the CodeFlare SDK for creating a RayCluster resource. If you have your -own Ray image which suits your purposes, specify it in image field to -override the default image. If you are using ROCm compatible GPUs you -can use ‘quay.io/modh/ray:2.35.0-py39-rocm61’. You can also find -documentation on building a custom image -`here `__. +.. note:: + `quay.io/modh/ray:2.35.0-py39-cu121` is the default image used by + the CodeFlare SDK for creating a RayCluster resource. If you have your + own Ray image which suits your purposes, specify it in image field to + override the default image. If you are using ROCm compatible GPUs you + can use `quay.io/modh/ray:2.35.0-py39-rocm61`. You can also find + documentation on building a custom image + `here `__. The ``labels={"exampleLabel": "example"}`` parameter can be used to apply additional labels to the RayCluster resource. @@ -46,7 +47,8 @@ After creating their ``cluster``, a user can call ``cluster.up()`` and Deprecating Parameters ---------------------- -The following parameters of the ``ClusterConfiguration`` are being deprecated. +The following parameters of the ``ClusterConfiguration`` are being +deprecated. .. list-table:: :header-rows: 1 diff --git a/docs/sphinx/user-docs/e2e.rst b/docs/sphinx/user-docs/e2e.rst index e64032e20..846536f11 100644 --- a/docs/sphinx/user-docs/e2e.rst +++ b/docs/sphinx/user-docs/e2e.rst @@ -11,7 +11,7 @@ On KinD clusters Pre-requisite for KinD clusters: please add in your local ``/etc/hosts`` file ``127.0.0.1 kind``. This will map your localhost IP address to the -KinD cluster’s hostname. This is already performed on `GitHub +KinD cluster's hostname. This is already performed on `GitHub Actions `__ If the system you run on contains NVidia GPU then you can enable the GPU @@ -91,7 +91,7 @@ instructions `__. poetry install --with test,docs poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_kind_test.py - - If the cluster doesn’t have NVidia GPU support then we need to + - If the cluster doesn't have NVidia GPU support then we need to disable NVidia GPU tests by providing proper marker: :: @@ -124,8 +124,8 @@ If the system you run on contains NVidia GPU then you can enable the GPU support on OpenShift, this will allow you to run also GPU tests. To enable GPU on OpenShift follow `these instructions `__. -Currently the SDK doesn’t support tolerations, so e2e tests can’t be -executed on nodes with taint (i.e. GPU taint). +Currently the SDK doesn't support tolerations, so e2e tests can't be +executed on nodes with taint (i.e. GPU taint). - Test Phase: @@ -203,8 +203,9 @@ On OpenShift Disconnected clusters AWS_STORAGE_BUCKET= AWS_STORAGE_BUCKET_MNIST_DIR= - Note : When using the Python Minio client to connect to a minio - storage bucket, the ``AWS_DEFAULT_ENDPOINT`` environment - variable by default expects secure endpoint where user can use - endpoint url with https/http prefix for autodetection of - secure/insecure endpoint. + .. note:: + When using the Python Minio client to connect to a minio + storage bucket, the ``AWS_DEFAULT_ENDPOINT`` environment + variable by default expects secure endpoint where user can use + endpoint url with https/http prefix for autodetection of + secure/insecure endpoint. diff --git a/docs/sphinx/user-docs/images/ui-buttons.png b/docs/sphinx/user-docs/images/ui-buttons.png new file mode 100644 index 0000000000000000000000000000000000000000..a274929203e5cbad91aa2a3dc21a08363f9a9b73 GIT binary patch literal 22385 zcmeIac{rAB`!;&3P|_fYM2ZFt5|TtTpinZ;88c_dJkyMlCR9R^GGv}-r6OZw$V@`! zd7i%ge1G5iXKmYWec$@y`)jRd+urwi((Ssh>painIQD%%_T%!Em%F%$j+Kr=p=^@A zB%w&5tjWWF53Z-d|36HS+>d{)vpX-XydM9!ufOhtKku}cRJT{MHnw-tw>6@eSXo;d z@z@#K8W~yHnOfVAt}PV9hYpYrU9dIMw>Pu4+M{e{X+%+WFxtZ>v`5LvW)J^y{u6un z_)iN8pFS=a?e^W2LfJ!+mN=*E{AQ@lNnN>QMSN^l@qOysHMgUEqNCRzmOSxY=F?}L z$Ye?X{jU3iSn8tHvKxLIz7M+gcq?{8MW|@9azdDv_^qKlwra zckf{GfhpGi?GH#%E135F@=IL^_OXwY=T2K$99hLRFDc!k&7*hqosfDbe^#Pyr9R^k zuA6q`i8r$|l8|^|M}3kP!$jF*x!;3Fp zzTEC$=Od}~mO%(6j8KL_i2 z`krx{88cH;-Z$07rj_T+r<3n`FR@@jX8Q28oD;@XZ#3$zaB*^;DHh4g-pP`|PAwV+bZO-IM?J`7wo{k8j*KdCcN^z`&7T#>W7 zu`4lsRXydwU-&~R>)n!4QU-rZ zpGIHHJz21_V3C!T^*86!wOmL2nW3iRCr(H#FHF~@7|Fi>__2FodhmCO(MHFi-?wgd ze}5+zdU`&SYP9Cv1zP%DCm&HA=xK85@OY4vbUZyx^5G8sng~f&mj|2KV;2eo1DVcW zxRA;d(zm&@ukXQC@#XE5C$C@oy?n{?=G4tKxUJhCKOQ`A<@LHKMGbDty_^d_k9Src zJa{$iYmCyzpRp=Oruu);W@KcXj_26D`>{r<@#m(doxeU_y%?hyRTp#-&oK3`X8nc@ zpZ%DQ$VW){F5M_lRZvjy2?(gpoKRF$l>elj?0zIb?3k|CvY_Kg>wJ6Xw56Ds*xa9o zyeXB%c72=F)YQ_<>t%Q>JBx1p&T|piaZKBXY>QoA)xGoYN;4f97(9-$jP-Ice16O> zV$)efWk1-kDfF~M*?3U~wsZ~Uz~z^3$M0>TFdbGdddX*KGdt3D)a$jV!@&0Mk9OXv ziM$x;GT9qD_2R|$6V~l)Jep}m(F&0i%7zUa)RPUU<-$aL@w`Q?t!zw3R3!p-pXu!H z_Z75g&>R2rjp8l+`~a2rKwn2)o$@D*R2tgNY(>Qnw&)uh(`?zYg)EMhl~t^3;qr`@ zsMiYHg$J9zJ=!TSHqj${qb*19V)#Y5>({UE*|UdY*_?Lk!2?>=L>>Qs|4h}@)!*tD z-=A9;OcYKUTl=TCmj>%lzlh_!_t2pYnbsXr#rHSeT$nO@`TBK)!8gBCo=c}FML5)g z=5?hRtzX2PIJvlz-REt!vu&xX!_M4MjFE`fO#hjvEB?Ewsi!>q*L!KkH0zGtSfTF| zJ>^|JJyC~cnd$DWVHa`yiX)$7*>aLi*mm9YV51sO2TuOVT=$k%#1TGXR30Q5a?1 zN8->UV!lR5J{+5F)GaP4xyW?+@|#l(AFqZ>1s}U6pK5&NYe|VALYL*VJ-6q|yh;5@ zmR-BL>*Lhalk}+8tX-Q65)l z$;s#J?N6v82fcl}=aY6ezit&f7Z+D+j>ATMef@tOh2n2QLnrG-@NIf68J4G9r=kyc zUQ4;YfgBe^f23NXjy(7F->D{Bj4NN?o11gCA8ltZC@45((@C*v&AL?e+x(0 zM@v*h>b31_NOs*RV5~RUTiLYvMy}K7ZV8F$dx}3v0_^%ioo4on=GLuSvG(HP;x}5e z7!Dmel$4&X;vpS;ScMMZN1?nC^I$Cr;3)B?TKhBILajarfka{|$>1AZeJJ;Z3m3kA z{o0v3TA;M+<%^EpV(vcKc73s)u_HB+v&n%84;$n#s1C9#9c= zo1s5V{f;SR$&TC z>N|PvtxQb!uH&Trkh)0e|*i9uRgS~K|I!GmilM~@z* zi#qeokLixT|E86dMQf5?`}_MT6gxXR)D78b>i9~eR0;*5aZX0YZ@|dR%=fTLY;?g? zJ9at$cGHtprDOw1f6rH8VPRVtTC;9Oo*n!Uq4K-$aen@p?R3;9Et~I_mR?fH2?^Pi z<1`vJ%%PSOCX$elFuxGz6B83NFzM&xbJo>0M{|~@^U4#=k8U4OHD1t*`2V=_3^R2?c}`i zB`OHM?6Ko;^XK2c)h%}K+O>QKL&3h@rx$=g@t#1`0#v%cpl0}tCx80l#ruKwv5~jg70Z(+anDt{auW9* zQ(05-@$;WYsEzjo>wYOKJKU1@`0-pal$a5a&EEj#HTND%&WZvIgAS%F7R+N2l zT1(-L$d%V8&gHvKlk1T5g0ewEd1+~BEPQpTFTX@vt~&Hh#hEus$X4fFT~C(<^Ckv` zJ$p9M?%?6!k?gtbN>atuFwtA*-+4cH@L-_pAoDS8Z^f9e51DzD+jjZ*h@Bug6_|lf zuZWWFJCbUsuTO=;|K_v<9rk%_Vxsmnuc#;+(k+)%!0t^bUj99=bkjz~ZXJp1e@ahJ z-`UkgMM~Om3$sae*tgQBT*)`PrRaByst=xiC7#^;`}cK`OPAD8PiVYv<4N&BppTo0^cIuqO~%QQS_QpjOLbSJ8u5@xq3gUT|1i>{$Jd zB|5RD;PNY{XW^nQ5kX;1PGiF<2d;*FS}-X8+uy$diC9I*yCU4Aqu|bK7h03Xglp!5 ze(QG}Rrf|Y&$m`eN0@GpRZD!#cu>xi{iiLEAX!$VUsk7Z{5AtSdwh^1YBg{tQ?uTI z!>8}LjIfD1Zxt}E_&KMt*Y2#jvA6ciJ~>B;bp-GNk<=vVUr@&FQ%Ka1QiNinSRVfL z>8;3%0XUB`)9-Up_1~ewt>3bxq$$;;{)6x2)D$_MTNxN`y%x0S9lU178L{*=PD#wh z-dPO)(-CK?Y*CDSYog+)z5NVk1|jfv)0MUi3A4<34swZFSu|0cz)1iK5ZW` z$M4_2^?-h(0nKHCj{1&`*$lO0UbU=s6%i4+d+(lpU9`f#-^pA6(PMwUZ5(RLWk&Ij z9WEUz&vY8qVPR#Bu>MdkV*QtNEGAvtqvl7i6Sx=aH%LE?QC$$~06gNPNT=W7(lRzJ^hm6P~e}8{I zhXLiKrO9d*LG$g1NclKbo<|(wyVugtpT$~HiW(Yrkn>LNpu=mG9Vs*^E$wbx+>sA* zp?=7AYbmj+@qX1}i}Vk+a7bOhzVG&(J6)Zf-oA_nN$w(GVP$FZ>dRNJK37(5Lh5`F z5O6givjeMN9>i0OE8HCV%}1~S4GoQrt?l{I_WYU{rGqNE9hrf=y1S9aKBpL!+l;oK zApq8}h9t0*}p%Y$F5qpWNa!5Id^}{Zc+KDzA4{(_Z&u?-P{bZgX)Ez zXL_r{1CUg|;7L&BULM!KJ<#|7VAfm6DNZn;xi?nfGMutv`UEnh)_WU*XrvtGDQ|2bYOvpuAd?8fSnU4eMgUG9(e9ulGq9$C&0 z)Gqq73Z4g=?fUmmz3$51g@4F~osGI)lG@tZ8A?irw$tHYi|Tp`gM1ql#YJL4cQ`Qm zh?!%}V|P*@70|6UG28QBnuJK+lma-;SgSC2whQ&pYD=uxk z=X1%E`89ZCTG|O<3_@9c{P^KKKe1n@z}*)lfpV5gS@yje|D8s%h+n>}*|B5CaY4a* zuU_p0R6v$qhx6~Bn@?^$p=e}`m4&6~_iq*iW$db!rY5+}g}{JwyiXmiX%nz6$NBkh zCnqNlbY!l*Mw5$Qg4QAOpQn=GpsOnWZ=Uymxgh`T2O=Y^DE*0P^qq13Mfcb55muR< zX8+`^A5Qt~KUrvr@1LrEx0T`0fYcMuf%M(#JA-CzgY4HVEcE8HVc!-^41#G6AK%4k zuUqo{vcPaaxr_Bi?8LGiUuV>sNDCpF3v2Zn)rxw<&hiznMit7HcYx$F)QE3ftFxe2 zvBEaSyD@E+vU#&RBZU!v(Zh6sZefqR0was+sMlyfD^0C{U&5K}anL%^_(F@uV+gPd zG0G9p2C`7H#nO+Cg3@#imOrk;o!&Gkl9fzwJ)~#$aW!AuhH-s+;Nk?B7ah1xhv(vz z!3j{@bMNFk0k~>&+;MUVF!orme>~U!IuXjMox!c6KH#`q;5!rZo}N$Wj{I)6DOurtEKY6tE-zT)0Qaq-SNd8Rh75 znCMQN7|AwKPR_&o=$&IO9lCOh-Gl1+YRI0^X2Z9|GJCkT*gb$SKnIZFHak+E4ucJA;#aSI{``3k z;BX?n0y$u6q-_^e5EfDA;~%bs+($0&dcfgHINwD2G3>a_&D}bOPpmax`@o8-k)zPX z`Yq+8=Y|*QE|V)7{9Ci=FI9B=3(?Zj&dtwvq12z3lA@ufUzv4k17N)gf@42Ww*id% zJXoSk@Ui%emG*o$m2-A-f@ZbraPZB|pa>8e&u3ima<)A_-);67C?Bnz-6UJ%D`tRy z)0PY#$SpQNA$8CxfB4FYtK~=&_Cye zsY74B)E!*2dDAAz{@N%$%jOe4>rv}IWGvyXvcMWO)6E0LH~rYlP;;oLDh_&GH*OvU zV40R-Xvh1^J8hR89UYJJ@!fj&?p=@|Q3Q`8G}OulDvP=JSG}CKj#+wiE=sMIPcA12 zc{okV>gnyF=s~71@5gTtRS$foYk1hdGDIkjrvqRb`qO<<`6y$5e=uGSIk^eyAGbop zx*L|33Wc5?fMfEO`H;xCx6=)^W;0T@Zr^@4;act*N^4si$llM7S2^M>Rk*jSXIil* z=;R|hs1P7Pu`xm75O%;m$HvCIQP-hK@L9KuUcVk4FhGP&pejC-DmKJmF)D)B@^o;+ zX~1~Dnc8+J&&9M)``IRLT$Z`OAR!{Kjbs~PN9I<;2vyn5I$*T z%1!TV&398vUqOMcDfIHn%Fh0bE91bjdVE6KfO0q{q& zj)o=znpSqZC(fjElI&WhtOwP%P9E|8spFlzx6^df(%k%NP~sL^W45x6(`U|{QAp4h z2Hx9(!%p$O7_hr*ZoHcW3gxZRM>!S|$D@D#{89F>hAeZ3W)o}lLbcSzi-uqsruDH* z*y;y&@2)$ll|hePxTvnq0sx=ZAa43|54g}8;I!|zsW*0xjBLJm@nXEo>}W>-7M%0g zvCWIKqqJimv}nc-{EpX1J)o$h#Ap9c{sp(f8WE?FuVY=s3_Esw;7j_1$`77*HcmC3 zD3M^TW_7bm^L8YBmltOnrwuCu%Y%6%amu6B5_f-p%>MA%e%U`bX@EuRD1az>K*a>c z{dl%t#p9EJ_#NDDU98GRBnt^J&dZlC6J5f6;>STGytL+%E3<{G-@g0rs>!-(_=rt- z8H6x4l_ zyYPU1CnuzU=TInod`ka(dng&BEF)7EBGsDbqC_dhO-e}Y6BgD$DZD+i23%OPz0 zp7I2gA-47?4-d7sxm)V9rUqp7;r!vUsrkO}hXDZrnwp5auT4!FT%)=D;D3>nCy=m= zwI1MV_QTC8TxYnsFR`(+_W<|CrlF$8sK&pBMshCRpYj)rsnGyIdE-r6+h;AA)+Q}G zAQkZr)t#ri)26nTSa#%Uq z8$2L3N03d-odvSV=y0J{%5`a|-b8Gp{xz1lgxe)dV`zA|zP7}F7cIgM{3|AE3;XHq zfDcVU{eS*hUpqO)#!wy$y{BO~Jp%jWHiGsDHGCQrk{9`E4MjPB{Li(y2iNP=QA)RcO$dt?M0 zsB>`eAqb3Nd!7sU=0y-FEY4Qu;Hye9*MSX+aF0UP=8t=2uUr9+Wdk985tq`EZt(yd zKp*+Q#avfLM&|KE#*L;Ko>rb9=ThyLphK_joZV?%6lzL3X0|# zqUx3x0HnMxWEoSub+fcLdqcJTON-3wGlf4*jvPO}m8>am;nHn%C`?5w2y1~dJcmw- z&G;Wl_m%lR^Ky_tI*{Ql-jrF*4}8( z)8I+$bJ-{&Ak=pM1?-J@Y4aMnCpY16NEJjS(EY4V_PeHG_u5lmg zI@Xco{rB(RcOs%=jlO^Xt|=Y|g0A0J6$)@(svwdYAE5SJ2192%7B_y$oxl`r6BdN*!7LY0W&nVgv+O2y9)S9TrMNLh!xb}`p+ z7zzphQ+DV$h$602r;w3{if*l?wYOTWi&4@Cmh$rQx_kGo9v2Le3ws0Hqgv12ckEhDQAGtk z>5>qE8`}3f)CCBoqR*Z^Qyx49RgBy^GW%9mRzIYgO}OU$_;?-&4UuJiqoVb@XEo?&7#kfbMsnp3QTim9BOc?^w>2tG(L|9-x^BWILfj+_Dk?WKUp=FJ>pk5Ih#qJqiE$w8w$hdQt9Qn)xG z@1~vHm&vzI2jLNp7Wb2~~MKlkoM=ZCHM7vX~#fq2aCsFR{>vPaE;#U2{Vq6{O zk4_x2dHwUT_T7(!fVzp)y*gj*^;Y>XpWDn0qE!PbWkFm-4BQ3$)q^yFx<58EBZVVr zmzPg^E$=Qom|8iv3y~zfcgZ6F$I)ksCHDlRqW>pC_}}uM|BEsHAN~OC{W`P^{>uyS zzr|qx58u!K2kwC|TLt|r@Q2#kS_n_xIGfB4cF`0Y4n_tARl1(d)<_-zqjfs>@86Hs zOg~JtrMkMhFQ!rjzw_OMh_L47=7u8FJy|VY0yz)#P+PmeU4v)z#UZ7-4B-(M_Y&Gk-5duLQuW; zWgOeLzxcRdX93{~V*cIp??|@vd%x+0h$I8Y->K#>UAq09`8Hi^Nb9kv=Um32yYXE+Ws@q?%k^M@!oc>}x&`kn*v)9Tk8{ zA#Sw~k~OqFWd)(gOUlvm=Y~51B%#28N@&(q%G*u-Q$QXt5#aM=)Y00j0Q@D$nn~AxY!h{zT2JInfFmeyA4LP>2AwVfgA*izn>PG%cY{l; z2ruckzJI?Dw310<7q^oir&wu!`kBJ*GQl)xwQWD5N{?Lx&!VCbO&)425oOWqf+J-; z^4tz7JKG+#1l6-`_g96BZ-M5k;)=znNi$P|u|Z+b^;L_7si`{m@V;UGJA2gB8ab^t z^xQ|+@X@1`LhYVeF;(vE&~++kv-;a@m!o6?zo6L2!19AXlZH9&CR`)4L-Jh^JdkaY_L`|4+&!?C_U0m8e(ng zjTW*2Xr&X00m24~!rEinS#Jfcfx=@eU>MjVD9E&T?_O@D5A=S_ypQ1>i5?EQ42|5h zDP=!V)N#D~rVM{c%Z;p9LL7yf1$`w}m;Izw>w3t?>YADnxCfjGd9A*$zKq1w0zHLh zbJ)pr*bwTDeT^!n}%O{;wf^7x)4MP9Ml8Mok~y>}Gt0 zbaEWWUpZcr32Hb&RYX-qp*KV!oUdJZGt`ZC@)tl02s&slD@CD?6$!G6)reO5__$$0 z7trye2F*m~l{KnYJat>JW^ZF6#|N*v_dzH8UR&v$4e(Imj3lgzg5`3YQub8P;m zIH#AH;RTP@4qP3$={#b!zUp3Pp~-};CGUFZibL_(3>(nDx&UgB6}WhKuJGt0rkgV?AL3W`95_I8 z#&zl}6c8e^?>^&-NTPLVueyBkG21zi*3O|JdYrPlcrE5QwM1g*7>72YQh=sL5hPJa z41(>Q0IH_VX@}51?i?LWWMGVniqc2xW^8u$L3+9%81?b%rCZZ&I~-S6Jdk-JAyyGJ zoSZfEAPHYLpP$&qvArSvOvk0Y8qi(REL&8$&Kx)(h0h!Vg;tRb@b?cyTlNQX%f8d6 zH4O|5u48@vbam~uX!umaZG!KsMMFwjR#r*&>eZ{kpdw{x;M=y%01qc8B|SltNIRHe z>()JpiE>*PTicYZoSZWB9_{2Z{Qjs$-2{JUU}AC*9j>PqjR`+s7%_tArwE0i zzb5j5W;gpYmvBT)KSuk%W{~8CK-m=g;R5A1`|x z(<~b0&YwU3wXsnRk`*s{2SJDy=~Uy&zYtzn2j84=Q>Cn@jL*zeO$d}dL=$N}r3ncn zV_v7wlgF;N;wdB?X$VJAK-soqo#$M}I&aal96t;3KR(C{xH$zTnr!Fs2slK_(PuNN zeJ`z;ZENITJ&lG&g!17Y^vYPLM?jTgm180tmkt~}SYGi;APSMmx_I`E3GW$kGMz(F={{E>Kx}pZw*1SOc zK~Oh;q+2vz2Y@DDY*rI-@A51D{pFU`{bCK0%E~Vg@t07Pd%$pm(AX&hYLCLc{^{%c zINe&)^KvW020FTJAPG%54sjBalF||qcX5XoFJHN01P_x-JI%F3MwH?p^svp}uYza) z!BJCKff)53t(4kaCu?Dcffu443(A1^N4vlKGlEd{gFA;!47$?$`an@sDwzATQP`%_ z`|J1bWhnYau(hl?Yt^3js&DC4-xrf`j{`VMKW17hD$|!OM;Aw&8-WI{&y96y9P?UP z%7zo>2RN@_&>7Dqb#gF~pnlx+Xoh2;91#5wdbxq%wk~sH@7{7+A}T7On!`ZOBpB<`TnO6cAJt7A0Xpwq-48J$7-&?<+WksMn-04W{oVH zchHzip)3AH&f|x~mp&{fBeM;yLbIlnD7G`M%3!(sI62Rwb#wsk0o(5HoBU_MY0sm} z#S2T>i>$0u**ClI!%3hG#W-qwzVCn-cZG$U=b<|9OfZROK|u$g-EL!N=fUDJKtTL74Tq8;E7V(U5EhEGXEG&+|B!u#)lfMlTay+es+!^;dc@!(V{DMnW$27qvSeYdm zo`M6J<#T%?Th>C&^q)dZm7*bVqP99*oEIMwHXlXjF$!C*c_$UU>`N$=f3cUEjl;i_ zBhkt&gLTbTSxrjnK8cXLyu9nEw8EMf)gai^f?^zPS<0Z^xWniCyC=9%Ig*;KiY~OZ z9%!Ac{SeFe!C3sK;B)$c3ME*JBy$8zs$3y7yU8SH_XSSQlP6DZV`7?apMf&)Fh4*4 zji`&Fp$2voSs~V=A3jc9w5Ruiwy@3v{v4*HLccg0f`(=@;|?m!-LzVH1V%a0?1Atd(uFSZtXP>8z~ymE&OHKlMt$ zL*k3POR`ZATrYE73!ZzMlsG&eCBL@4BG>s9FPh!{YmX}_Dn28jZ{G0f zkW7>!GC&kC*B_YwZmMzqLb1RXA2T;Y7`*@r?~NHobHTve{PjnEI!=O%Z$pGV4|03} z75X{i(f84#iWK*2oBu)RDg%Apj~X}_hliU0;043fl9Cqadb%a(@QiwwOLQz;D!%KbCjO_l+D>Up`rWt@0Sd8go74lTGiYmPo;ys zg@hI_HyU2r*GDyq^AiR2Pb0$wotcSxHQktrRhT7Noo3rrQCiPua~JU7Ox(C}BkkDa z`1n(N*6jVD>*Xbw2Mi1ix$x&e*aAt(o0zhw&9!eU$yt{N9AKcYzZZrh&2qnhfMBRB zeWwc(O5{J_oIH$+i$l9zCx}<~7#tDL(M2}NuFUDJo=T?4gRDqu0RdiVC}i-MJkrjx z*<0dZ^sS>q7tj5{K?ANXxHV%}jyF1RbmTb5l9*pG>ThdASs_tp?r>n=KA)#g6@o15 z9W%9Y+;Yc{SfG!`n~k+VG_ zwD{?5z&lVSlcq4ZDAP|T=oUT(4>QpS3l3HbYgS^hwYv;6@>*nGiC6qaTVb{W=y?YS z3&qW{pyIy5lf^mg^Kk)za@)5-@adtTKO0;^VtVDdJgr6=8K@tpoX7SQz|IQ_(uG6b zc>3|##w89B_}-8dDUbt|qvEgS68i!gLZ!i{Q%(oNy3sW-5c`@+veCmG&oAmex0Rgr z$wu@VNE(CWitPP*I6V&VO=I)lRMQtmi}N9fBw}(YL-J++~+zw;!m85$N*n~_7y%{-Z_W9b#^`k^5P&KEx1;F zM1upviAf0hpXRg}7uQ;-d1p}xZsJb0g9k@O9wVv@VAr0nxtza!+qT;{-*H#sOyJRS zb6dP|aW-J501Y1GBGTpfSl5LisQNAEOoanmJXD{Q$U`(s0w`naNmP#6wt}EO8r%V7 zawlB%SZ0X?NZfJrI_~;KK)nbkkB~K^8*b&~9FF!F`uq1Tf|84mk3lAizhe^EBtIUG z;Aq{#f`R2w5hrDV0od1ss^`TW(XWe!&lWZ>dJ_|q3LGTlr_#)c#A~@91kZr8p`og} zX`;b_16>eu;9y$ksJO8eJrK1v({gLVVL<8N#gCjR(R0Mg+(!mhHr(or^FG;QTt~vJ zHu4)9b6A;TqI}3bhCS&mjvK2}9vXVd2+JGguvqUSzF$0vKI96Qg4_@@cwt_iv?P|? z6G6V~iTI2K?2FkfD=X_SavBHa3;rxPVSN2MVeG72KAY(2f5xftl4AfsU|x>2PMZI0 z={*m53yPrPr?OEIH^$V|6kFi580T&95V)Xe(XXiv=HFNhZ9A=flLq`ZKrAWRSOZ54 zjufE+9EYM!maM#RY@=;>PcKVt)D+3`@Aj%+3#2!xPe1psJEBfGI4y z47GP%Uba@;us#oQ3ILjYH;J`aY5m#LtfLNESp&rd*@_lrnHa`3i^A0q292k=zT>9~ z=o)~GP=0!Fe|jS5G%*0=G#!22+waAW87x1TOLz`2}$H!r+8c zrcm7APw;Uv%Mow%Z{pfyjZVwPykB?o`Y#m_gYw&Srz>y^!$JC{7A2)mP5q3grSYS zXi^2DsG8I7aLv6c3w&*a@)|qayzl^>x=cd7gkeO6dw}R^L6m!q|zTT5U zIu}hz`gcf21C+G~QTdEqY|3bd%P+60Z{H-Lgd9OTB@l3v5G+!|kl8q45BrZsbr1%K zAPC2$=$Y;p6Vs6k7vq6q@#pW~edwUoaf_p-6K)0KrqHr7wEg?hO~PTHWcsO;+{uAS^|*;NJkQ#F4IS49xbSi-(bCGV`KY^^QY09Yun=o zH&+=DH~9f@-+pNS!N|a}4|njKm>5i{wzUJhaG&o9uFbI25_TMVxo_XTQpgh5Ki{VQ zGdOqz-=b2U3svr>z5NSN@N(ECdJr78fDIZc*L`75yZ|>Vh#XKT;Rvv?J;!)Ow%cq1 zVP2rm@6qYuM^lY$I=+e*`^XTb7~?R*47NnrjAl+k$ycOSYHz|597o%E0yxC~L$W|R zzWw+ijcJsZA7}cEIabF@IQ|R`9fi2?44VChO`8HC$orszLPavIkbsMDKS%&S?6<;B zBX3Z_WkAD@YNnCaY7}siIj{|>H6Y6jXi%X-`7IT$&aUnQ{6jk24>Z7tA|q_qdpT_4 z;nJvPBWNsQcG)eFDM!0S(tdTM^u|?9&DWP-pO6OS_~07e=G24GmWWaID=aL84{?9F zd(I_U+2^?ThaW$F%*xFzM~v?~c1#Xkoqb4&fByZ8=5tm;m4Zk9kXFVgm_y_N*5aB? zV5^t_Gqjiz40!mE5#phLwJ@T_yfJ|XmdYQKjsd%ba}Jc^kQv3bn!fDTGM zW)qGW7Y;>72Icr;Mq&DJ2OyGJYgV%ROp}Q%CU$%uK75Wv6Sn{Ngr4=pCG>FWXwho83b@A>4Ro9--F4o8W5&XhUjTX~-pQAu5Kee#A*|yU z)+ikgR75TzEbqId5r78kovC)urluw{dPP_8RU}00+0SVU8^;EE`WM^w%lM!eE5|5G(rjkqhJxJq zdKJ_&2=%}>H8u5RNQf+2jXi^jg+Xvz#PrBsy0kS)@m0?ZSh#3;^$9^i6+&_$r9MHQ zjVCvt@T?%r73DJ)yLwij1MfulACi|Uv?HRAtSpau$wFCqaacu$snqnjFx2g}wZ!l_ zJ~_#VES;R5&IIJhBrFM#p-Tz1*!2AEX5 z+)E7RmTuVd>(FY}oFymh=_(`{emF*mQ2}OF-;j_H;#U}aThRKqv4>>=p&A7VY8wj+ zH~N=MXoQ1f1fiCiRs4A#vIH2zh^dVD#tm@O%%c-=qM@dzIbqPLB)6nHmlwG=KC)lH zp}$s=^tmnd-Y6jtG2ydRd%(w@1N+LHJ?o9s=7%qy8to7-*Wk6!m&5#*HVn7lT3WQo zj1UTK``G|bI7A~BVC(Gyl`%}gtQNFzI%8AT_w3=BANt76SFN;8nl z$SB?t2B?}Gdk-F#0q3Vs1O!q!t$bJJekKe9{a-}h{#od?8ev7JaBJasRhIinWGi>n zFpdi9{X(|dwuGhzK&2 z2!FPhJoAnSj9U7v=jZ3|?CH6OYTothsyNSpF>H~SdIi^0pa`u2l6%B1GL!%7_%@HB zDbCJi?47dRQh_DmAZXV3MXN}xB8fi4ERyM?+shN>ypu-FhcL)WnziccFEurhBFKCY zP5%2vT;M;Gzhz|)_$&kHl}$+5daz@e1$ScDRVnzLSsME%mem5+5;e0TZt| z1bdK=&%{*>0VAhEmG?#y>-@X3I0!1`#94;pSf9dgRAwqF8nI`2ta!^Q+rO9I)z9uF zM-d|M7KAFg0$;v;qlRw5Jz%V``;7A#1*#u!-Dw<7eKWHS@WwoO^5lGch?pEPi&3B_d!%`h2#Zk;_4S^3hf=+YsWI9e!~K03JL-K4UTu{jXlvS$TQ+cMY-}iRiI&0=r{AMzVnh z!Z;MyZXinDl#|;7BeLYSGNrHC;(SXefG5v=o*&*U3I%(D#aX{FGrSQc6Qi(PCr|D` zNLL0gpmKB#4ZYztGBdQYB2qY$`aJb8cn;~ih{2bw@>((G=$9QeB~_)R>!qab*$VG1 z6B5N?7N(2Zl9-w*Fw#+&=nDVL4fn}Pq1cgie0+SwFbKi#L)qlQv;{$|KyLS*KWD_c z5wjtHl(EKDkS&!qE(~M=7?a%GiKMs6^-g%do4lVzi8zM|yaiaP|A4q3p2klDSW~I&ip1p8$xY%674% z52jnv&?KY2fJm{HlmXARLB*iCq~EuzzDn`#4!n+O{ND3M6k4QcFr@CKK#_rv_$;EMIBGS*`qAyQ79FEoHy@8>i9ogi)0IKnH$j_)_bYyS}c$u6&l>%hp z&+r$c){>5C{Y*TfiZtn#w863cfW}B0%Lc<_n0>Z^vGwR=IEL_`g5Jdd9a8ySaK$b# z8;#yZNvp?uug%^>)0t+=?nf|&e6Fk8j$(WPl4Vzaf6UD#%2^Bykaf#-v{cv9DhAU= z$^Qx}Qa{nt`_g%=lM1Qxtvt7Ufx8Q(2n3e&ErErLF#Q7CQl059L~ISJCtw(9Xpwmb zn9j%y45+I4Ux-B3&1oAT1K(4OxeEi0ySw|(C|PDSc}%k9(s^R#LgW;WM3{*89vjMt22RZ^N!VfI;9j<%|NVFj@C#)o= ztlwod13kDr?Srr{w-(0-X~ z9tIB4J&D>OvbD!|`)x3LS*Qgnqeu%;m|x)sU1wk*=p9NzY&`Ps5u$A2Z=~(*(=a+w z3bFWGaq)$QPa2WwVeo^aRTA{~?{&TtCtkpi89$GtPwbF5`9^4_#X4Zx6zbsq7Rw(M zjmSAoAO}J7Gou~6AdHId&QtF^W%CxZNz%}bm*CiBLfeJV`Em{p4qmXQC$PJfBhvPx z*9EM#A82wLdRHCplm3`aGH%J3{n0vtdQ=W`Kz)V?lv%D-RMl^&*q-w}%-IfuL4<6n zep26yb}7*}d-{GQTob`wFc9^>qadI79SEjIzI=}Q)n~Trf1}-0<~H0cc+_S zLYM$^c#4kVFb4y&m*Yb9WnwU*g5h4Wu}RX-v6sec6^xK@NEZzx#`lDI-6hLVtldoq zhd`V~3wIjuiKlQCMMoWkN|>0K2s4!;?1G08!k7oy3v|s0|1If5e4nAAruNOu%*1HA z3W_i3T|krI0xC;E*El^7{M>fpuQaZ$(vk^#{-4aUr$#W`6?51$^bFWGHxaP-PX_Tni_B|5V*`Zm;k9_pNQSqgx(? zHsn6kmY>SZ*A=L+sTOvd9}06DS+>dU8!KErCn2w(pkFhCe!^E6r^kL92J96QVs>zF zD6Ot8f%N9Q?lPqa4qOF!d2bj-@P>vR*pFg}A_{7154MOeZG}HEMRKvV zRogUOlW2&Nl9K-!cqEpOef!QzKxtrX3~GpY@U!UZYQQHm=?P7F;qpc1$PF2Z7ZEmAeG($@s9` zI%2nl{Pgx$KbdX*H|_c&IM@)2|L@nwfIk8Q6{l2XE3;_prf-^5F`DBm2q(kjo63t- zK)tOD_{P`es~A{rMPY$+_P*T53#zI+%q=X)#HQ%#(&QM%4zZy*Y|ILo`|?y&RA4=$ zp^ynkK-J%nT3~UbqL3FlY-MI{eCJ_=QNx&zVLAFDu)&6Guns}S!ih`2AU^Ag>SHR@ zMeU6UfbT{_A?(Q$Z!E6X3xs>uE0ZvVv@;$HCtPQSe1mmdW0Y=cK8I};>8+scWKT~I zL2(KSPa_g?a#)?(9GF3Fji=Me99v{07i4=+r9b&PP_7|CY~|nx2nyN}GCYz$tMf@S zT}Hu*|J7DNLv(8GUe%a>ZFA~~zqEt$HX~yPt3M6yOTO;j-Mie+c2U5fmG%iG=FhM~ zE&HUSOvddHVBDl_=t(@J3M*wt67?X7`AwjO%DHc!go0Hi8YEj$*+ECIDL~DoMWR6b z{?JxcQK9yiGMxUCia{Np(ks%=w;pp`FkfJyybcM0yuO7v8OL8oM|0QJ*J}{&vjtu2 zH5HNWcA)V%zBg=^P5iX9;ieS_0uyvZc|&;UKn+TG5bRCo(Pifl66tOT0O zfiW>Tplse&jZ2?fIxXZWHfj&14l?Rc$ZX=m>}UX5Cr5N9 zM{-AB+fhd({0q9UV2FTH1DKUYXg2h~#4j z^Hx??<{)vzf4751k6=|+QPIdVhGc{T{e2^Eo;MiObrX|r)Pj@U%65B0;Q16=_{SAD zjg6>IOrw7JH9DT`0+-@t-L>&$<6d}_StfF`$ zhkCxNU`k4g4RmP7ZpFCB4c6z*QQ4mkty~x_Slo=Xhu51A$%xM$C{g-nBxW-;l%nxCIf&yBb!E$v&^(BSv*;X@aLvckfV z?7rPZFLN<_@8?GcRV_Wc5WS{j-&$&GIdL1PHfDmm0XAde;!>Swlq9ZWDV~Lg^TG@F z?Dp+7(y*(LKBd2Z`uro;i-9j*ltP&`72J*7)0FR)DJhBuqM4J*a|qtYzS-sPV!wbe zsYJXO0aw-{{)C;!-X7J?lE*auUw@(YBK5lPEn25A?C}9@ zd_^6dW1OKJCFU_ZXG8nDMV`is{cB47i0NW!BO@cBixM**T0VT-jP^VFo?JK`6A-fe zd`i(F+%F=cMf@3C&P+YYwmf(47GBmNi*)!+;v`HvJ5inzdG(sbR!&9d5fjGXOP zk==~Y;Y$G!+^}Vf+H^B9XAGw|RNZd2yH0jC)h%H2ubXi}J6OmhWVc35(C50dFf?;@? z;BT0I`t*oMONkS!mWSSP7Ky~hj*04!A>GX7kLmD$mZ69ogM&n<;~NPPH{o&hR>^mM)$kOZDIFNxf-l-9h_x`;E-qSSLvaK3$HeTn3 zxw-j10RbgwgAKzs9hBh*T(@rBC3q()PsYc`_l`Q84h6lsD7HMMB$-P*OUA|@>ODXP zj*Mg>GhWiye-|;#cvW;SJU~01|FAE1mXPqot7%^G8QzZwMJ@zA*2+(f>U|j|Xc2^T zUE5nR0n3HvQsx%|Uhz<$sz$`!=aS%t$wF~8R%j{VY#bR$fEA+V*zx1?yFKUhA6D1F zONG9HWy0mLDv{ArG+%+mizBov`Q9}kNkwI3r~+DBrx)bqwfXfcZJ2<2pIQUil`**Y1@t_01z2E8QKlTx$7qstVC3aAFh4IRo2vH#+H9tSUqbn)V zj%;yJj;(RShEKJ;hEq?{NW5?JnwmnH(Ym(32d{%59<#&WHA28cf85=`cpZunyWg;C z=0_2Ar!yks022|YTF%IDl}1vL3@FjCm8Lj{4Puh|9%=`0J$1fL020h~V`ENk?)B(c zyu>=yzsg?qwEBTUG(A1tYiWUwj5m3wK~pO2rcIloqoYaJYpj$jKo1>*;Sn!1U4CcV zeRNq|=#%!B;)0GtnnT|_F)DKxW}t^D_wUo-9T+S)Wf&1Vfm}xhXTZ_8PM==*AX;p} z{|cuq=|&U(QT3#vKg;UBGGs0o8y|q1HUK?VJs><-`AS4a^CC^HPq6AAX8}lh0F|@V zW;N8`3p@&w1y!476B`GK%@AH*%sNFRCo_PE5*)VM87}WwamHf#8tg5M+wz zxVU0WgrR;O$GkyY^%z{KTWe}+PPt6%D=RN2ukOH2h8$#F5y|jcnWsgvD@G0($I?d@ z6zmoh6g-8OwjtNQt2gI4a0LZv3+HVZU-V&>!?iD1TJmIAcWlOt1Q4$(LqLtG1V%91 z-6yXg`9gRv=g*(s$bcJAF7Jegb0A?G)_8)4 z5GN_LV;?d!fr|+c-=qg&Cn2LMPMvj;--z=vCP zJKw9zWR&c<cyPo0moh@QMgaJ$Z#NY)?+pk(4vkKCo zP4_$NCfD}>jxA_|Kq=jW9%}NJuhW<$hI;1_8o%|FT**(13cW8TJtJ6#Pm0mSo zrQfn8cw)kq08_w)j**`$&#}vU@iqLIw1+Okfav;BUY^^gkC#dm28V|7pg2BTT5^AN z!t5D*OFsmvMeTBnIYqvi{O2X7r3rYs2WfCL2_E>*TQjaVzWlG3gu@R3l5!FW=k;#?57KJkwTY?b3XH3zG|xH$afvuMM6SCt{^X~K|-=M z1pl3~b36WhM6|F_lQjDprq{Lg*o4IliO*759RM@<`JM<;!IBN7v98!IDj2Sa-! zBWnj!8^_5lWm5PM3-KWtdn0{EGaG9LEi)@45-ro4419tN=M5bh1o#Am82ClRg@nZg z#AXXG6p)ZGkSNHW)^dJ2*6pOFRo(f1YVz%Y_h&3$_%`Kl-@k2y&CgHg^^2w#Y*i8M zMW(v@-O784)D6cDx%rYC1gf|Cgtxm*?)*4W>Sa5>ewm$zu3<*3+Tqs(_q%rvll2}G z_1D66Gun>d-+N{!oN_oov{*fj4RLDQSdY{jC zYsi0p^S*)r)8p)0Du%~tr?esvMnB(x?hkNK08)lj1Q(LBfR{A@NUDOC? z*W7MlF)G{U%e$6hNfUoH&$@4LuxfQ_CbZZ}vDj@P=+B=&j*=<=`^7K4x%Q!{Y2V3{ zCtJ;hnH9sXuPx6xySQxnGe3THJNcpesi~~t92XQFKYa^4ENa@4LVon<(bhE0hoP+J zIpRy)mqI+(mis0qRF|d)!mlgzR|SgQ8at>{bSv4S?Ty_$EhXida&K}wC#O_=PHtta zGmp0=^?a(#!#%Ap)bB}oEbpbGqqFI_d+%PFUYXRy#6-45ThI=Qqa=KGKV=J@Cf~dk zHTitRb#^#m%xk-O!<$x2V0y;8rWU+o_oQ*MYBCf&8mZ^5nO z`}e8UC5M5@!G;*BBS*;If4H|JO*3^59#+V#`Tna*8TK3N9ukt0q!bj@-QBUwN)aT! z8xbBW^F~zxw2O<2Uy>B4NIG(@l&)XDUTpSHJjo7*jGYR3_;3d!BO{k?ab%3BP1TY+ z9{sYmHsz5c^+9weFPFIHP)7Qwy!zUlrpchIO*SMv@^IP+l-hAMoigiOEOw(c(6^d&=28rQChlXS~> z9HGS48P!Mf+s*SE)l$2=yPt5KWvh$e*1UU~UP-!Q>m6?j`p>xY7nie+X{IQvOO1Yi z$GWz%V6w5k7C`ABQ)bsRjjT;nuie0`Jy9ieL?O!onmJ2$P zww2Ai^~DEV&ShaDcDmrzD^8B^T{+q+_(Xy8l)+OE%kOU=AXTm#8g8kWZ7y&a+Jg;Z zviSB|IZEI`Q{uUnGA}PgNjZ(4`#uwTV`Jm4!khm1$e#w$-aq9O(ve|hdj(q?nwu$a zb6j}IW=2LqF@#Oc&(CM;9r*qAm0hm%&EM2|?o-=tj($6vtkL)DSH)!6Mv!*8h!{D^ zmA5zgdc8K7#l;Wt^71CPeSb)~!cG4BTe^Q>Ai0?RFOp~2&em6#cojzW{Vj8{Ri8pa z0{yAkKQ>BlkiEaRgJN;2ntti0Kkp0O5^;5Pbwo7VsNC7J>d$3;Ppss7ZAi7f(MdFJ zh^~G6=Z`JcuA#sB(WbELiW+0RW$}8ROAmZ}b~=jFiO23K(WRd(#0Kz)@DHV%a(Jd=3F0SFG#D@Z}IoSJ?k0Q7s=K<5+&=SQ7peg{k6x*={{e13QBe~- z-3=6jrZsAwYmsk_>Np&>R2m<=JBK2eyu)8LwY;8dDu^iD&z&ZF#<~hZpP%*L%vRzT z^4zxX|-bxwCoO<)T~M*b|SZH+NC9JtI>nvLD!`5+jm~gs9mS zR4Iazrk!W4a{W5Pp+N7c{_5&0snb>g_RlZ=st4f$mLRZ zvAu#$EqJ{4OnsTB9!&FBcl~{~pDj zk4Q^PpOKO2A7jmZ@q&$HdZ>}2LRmRh%_gZr$h_63Ax3ltfw=k4La$d#mf<-P__4$bCS(NPWg*Q1@=En{D zK0d(OpZF}!!^2bTJat#nZT?x?XaN=ueTPLrK;UzhQaDHD^4w_Ch~u?<+dHFe8C&>6 zW1Ys@)t5SvJG*!LJGWr-qaNe2-nlJI+!(m?PpfObgg)x&>4`0j{TzEzGuQHp=jzls zcMp%L`T6YzEH$2se;Co{o}>Q$s(bQi*M1HXeus_KrJ;DT!y*BwqkLZL?w(5nG}o?O zGyVE+E%GmYf-OhQ7pYVX$bj zv>ZEB{*a<2+e}mZ=I^ki$;+=V^Kag~dAO(KBpy0|?&L)m&CWl4J@^>LDi1qB5WhG-c5sOnsoGZ_*?S^xQkMlC;%?MX~ z5EVsBq-SmI@6dJb8^^5w2KX@tC3S`o%{|)gwI~_YdR}V}^72Gcfx{KgrfDRHF`C44 z$hd^u{s*)Li!r;IUUiRAxURr-?#!Lzq->;ruN9@__b}JvphU>^5PAH*k(RN+jz_bu zC0j%Pxjkhjbe49}4bn9|pClhTErG3%Z{)}PjiX?fq}n42zS}~WGMNknnHg^EWK)=A z`g{3f4hrgzk|?$)h{t{&o?Oh`w;@R%v!$lKo~x&k$(iZzciiA%KiZMfG7|Po!~H|@ z{amUIa{ArJD)$^r)8eSFuMhBkSGk7;cmKEBeY(s(5uAk2dy(RcotfUBctL&g@r|7f zw5}{3`Egqx|GltJJ>&N~OtGpDFTNUBoa#@DIyyyv2_e~PoVf-btU|Fc?noLdYEuf9_5-ya@rw(l+~K(c6vcKYWZ zt$%NA@3CA|*5`?d%#WEgbhNan&2zM)ME|+-y;0}hPPXP~r;}MxrSYIarC)rd+{5$V z`z6}=@UST9wi?TjP*$24LRlt<=VNtyn+uP9~YGJ+=kVhnEU zjCam|e-b$>PZ^~+cJ#mS{?oM(I{D_&Is13K8nI!V|7_KaqB?^@r1AXN_xs-7DbCW} zg*R8u*B@pw;4SlJzR?%HJl#P?LfB?S25=lFO7 zU~ZK7O^(B&*1tXlAE##5xDTN3F#Ltlb7RekL+iyeOKGbPA(Fwy_{OiVE-eB572Fy> zw4XzZ;n=ae0Ksn!DtFx)?>L)l`TdKj5Xn<92X7FJtec}3gXpDIky*|g>yle`>|v6S zcnX-CaPd`ER#wepcLfXF01MD2`sIE>!FP=*JHCx~=9x7;-$sI}_oLIMY@pJgW_G;u z2~c>J!;os}8u}FtPkpXc=jD7`#+$!e)pIA?GU#1P;%?t2K{c+vvt`E#uXPFhgc08c z_$vVLfAP&V1#A!d)!f(iU4XQFfVzO~=g=s!va^#ELRpGkX1sxjf|->L)P{4O13-QO z4uR@l4e0Rt%DcBtle*Eu77>gt5eEP*1jI@IgMX$JLXRr==3_lT46fr#ZZyWl9zPfK z1ps{TbF9)kYmu$x@A1po)yNXPGIaW0E{FTUthjc zP$1TlqGH&UZ+GN`?d^9ajhi!V`_8y7Of=voOuxOReMB$KqF3r(n?-h-duqgG>gP6> z>0kONkXkRbe8H9W?cMu+eQjl=BZpJQn{3lFG$CTA0jWrOO5Cb}r_;1^I6y+GK7S5k z*G%C9EGOB#WlL6O=5_SW*E&TPNgX%WM+rVed&Ku29juQk2mE43?ZwZxT<wjg57gAEONp z4kobk+ZzhZ2OXDZ&1%EgWw3PpW16WgcC*9k(V{j%DB@xcgGs10wSi^Y>4Tskra9Wc zFt)a~@~JkQ&C%iE`^gUpF(8A`Uba&l4Mr1R}-n^NbPiZpGAu!ibN!Mox!?$mrUwLPYzn~&PrNXYcDsW`80v3!~5+Cd<9lau!~{!?jXr{x)HWc&L1h)qWGH?*=Mza1YR zKV9JOznk;Yn_qxp*9skdd2|XZis(5xIe}rT&;x2hn62lZAYQ?`Hu290rC-jvia@zhAoRrfUnQC`Ww+ZXv}ofdx)Yuwu7}3?Xt&*y7uX z_@Tylz3=mA$Jth$5y+v(+*fm*7n}eO5LeYe^R|wT$|G&mf72~0IKhg=FDPic!RFH> zZ5eu^t|}!D9&8g45qYgw7Kd9s)cvA*!q>pqSnk2D{q6a7$3e*!CVK^u@_?T%EA!lt z8wSD2NpK&M9}as?nfd$qZDv4Hol@p~);_?F><{C-6olyeF)?8+q%3gDMdQ`i);Bsa zs6zX&l{aoMA-DRlCM0BZ;$g`4XDuUu{;c3>Q`?`Cy-K$R|1IqmN}Cl} zYGY4|0=yel`0P`T5w6pur+Z%7!U$yDSFJ>o$jtrzIs%LdOb3LxEk`>rG4VL|@kVtJ9f@Uo<{bz{O#r0; zd*NP}GIUi^ul*&8!Qaznvx-hLeSj80>;gf&#)UlI>J~eH0M3sTwb8v>A|?CNkDC1q z%5P?7CIdQyVq@&ab4U*T{r%-wou9G!I!roo5-$I<04v>kl+;*|CvUG)?7WYLhKBT5 z^5d}dgQ=>qAM5Ir?#lmVssiLumM6ns>y||0>6$b2$^umV7Ri+3rDCx-{2t4*`14B^SVdYOvujZ3m0Sh2UdnKSc?$4h{}(EG1YSo2&UE@*e;oA4N@W70M`oEQhu> zJdFFlupl@u*H>BT?mqxRcJP>Fh-?b}^*1Jb>N(HeVGz&QN)V|01q4k+T0caZ@4hlm=)HIzC#H+?DwsAGQb`B1dAYWZYot zMD{kB3W1hCK=N_cp#YoUr$8>B-?w@c3cS6BVvbJxP%g2L5T>q`y3+>A-Me>hi)+cz z+rSFXnoLc>q~8V!g5a_rC;hXq@JQ4Kj0xW-H>7R8IGRz0{|q?Df9%7}^|b{(2z;s# z1K2OUIieCRq!0B_ZZzApX>*`l;wtBbm(+;dek2%)a;*6NK)G_1VT0`0*jTzlfh-rw zG%dYnMhFV~7Q7et2}-6k*WVDy-=ApY20SNZ)AQ7RptcWS>4kdYP6~=02A@B)FTH3o z&CJV7wd%~hR^qx}URU>XytMRZPb*8y7dk~E$O)xnoA^vT0r51zT*dJVqiq!UcA%OT z3GP2-%cJG?y(Ml=_)21hULKt^9J)ftpS=U<*0s~%Y@Bt{m7~e&=~hqehlYoJ%F3hx zu}AFP4z)T;nQ)T&&4hM~>2!!LhLbtSx{XNwLRD1k{h~CMHi_ z7fS&dog4KI6Sw_AV%3rTp|WxtdR0fBO=C?cYj|VJZ`VkTc%^mLi!>jqN9T|zRc?Fm zE2Ow-@TL7jk=}(r%DsU7u0xkGo7a(+@HBqmFZECaVn#`LuPx9^?R2|UP&QY&8S|}f zi$xO)3;&Fa40XELi)2d>V4pfmXF%)=3JNld3u;B>=R`W+z4&TvxDb_^^?ckP01rp` zE58*Mjw;}4f`ztT=0s=q+s`Q=P8%N^z(!-96HTycw!j2bP!ioT&j$3YP5HKcoA){} zQpM*H)EM%{t;rr0$Ov|}whUZc0Vp2=c#;#NZZ0l;{rxh&l+1)cNl2r>7POD2>&*bj z$c{=x`qS_l*w{3prCtGlMZF#XDy?;eI`*Nu`k^=Z;SfLz)TlGi3d6jO?7w$({KN$j za(0f6zaZ8m+w_*Q_t+>FufdC%%EXYJ5|6#po!G; z@dB>1rXN3k)X44)Q*108cCc{pvc5@k3VAd&HI)Eii_owPtAqIUKi&^eJ%Hs)fc8!s z8^E*zIRkuNbY$6XyhFIFyPHpD;c*R}k|KsiYZ37QR%3YI2{CKsUz z3W&4E2L^D^#KpxyzkdXML9@W#2x)U31vcMpfoJG*Y=4YRsX9loVmL>#MThLNUqHZ( zPmdXky*A>IHktqxBuAy(bWU-lqt{37zErg9mnKved~5x+SiT zlRb1kE*&a=b&w(UNk{GCR56Ph999>KB2s0RR{h-qR~x=KA_fMyJhj4FY6 z1jWs2(2tF&9d!#G*$t}#)PDyDgh+d>JKyVz-z>w>Umnv4W*CkAY?H1Mw8d4~4D zfhvHqtlV5<)EsJVZR*=ptg2rMZ(11FN1jHl4VxqK^IE*?dW;55<-}TuT8ZupwS*g~ zgkB>z)C+<1Q>n*FKBVNkB|BSWT-(mZE5&p|k^z$E@YIHztI2CcXl@ zaB%1yNGdWy4n->~C#SNrGX_I~EsJ`2ZIIuaU#K4+8yh18r-XzA_l>oMb__?>7J6sU zEJ!|kz<3o`_b!WDp}?m;OB_`MK!3o!8j zYEgTZA=3I8YBI5nQ2l@Q_1%eyiBZ4QhHyoC-$Q9fxawm!YfYflqk*MGTZ@(TmAX- z9vd4QP!&UsXpLi8_WB2<-hml14CVz&YE^#;oZDER+#qHdh^2r-!hX^ltD#_E`lu$r zI6ud=7yCkUQL~KNAM}wDU4UR9&^Aeqh&%eEsKgkUn4AkdBn;KmyE%O2iJs@_#HK&! zbki74379oACH9vf4Iox)f;;V@6W{tCi1)>ncPzJV-O9G^)*kujiQ0f}f^4|aoTN~z zvVhGYJ^;*!QCya3({t)ytEr4x_D|ZyQoowhN_$FJTU!(50ML-tIRGnfj|IR$ER2Ej z6jV6k(La3n5E&E0n(u*{N%B*oTUl9dHG;lKgy(8583qN=@B>p*Q=4)o5H*0V+rSOA zQ8_1<%rk9zr6AyymX?AelKB+W92GPsqhe8h-_s-2GmaTaZJVAKITJUpNO`;UPELhS{c=fa*0LAED5$VYPS z+&O40!KU9;xJT^bDhG#$f!9oKZ%8)Xw6*0EQ&DKhOVTYO?_MJ>X9R~;Lgh63^6XBH zi}~=`>G8bY&HQsV=-8;>Nd}dE^|2=}T@i&^v+uavkDou!ARogEtw3Q|=y@6G_gE>J zyD=RP=ej~^VPTQHPzUI+^!rr?21kBx$I5ldM6J7?A+s@dA*PGl$qv3pGr4nr^jkY{ z5ipNFW;myQ4t!~93dWcw)3ACE=7Ig$!cOBI4=%n+MhMG^?m&G%a`G0}hrT0^i$@`v z7(o?qc6L^S;0&T9tETo?p|grLXJ*52oJqcW^}Myt)|yvsD*E|62j@5R()=}w8oHB} zXb=rM_8cK*mPaIM}vUK^L$`>ppSwl7UrvlCD>8v5O3JDHLeT z1e>w=15f>VyS*6$LCmz~_Z>VKnPfks-yMFr=XAY?Yoc5rG+|>DTq3~&U{h66Lj^Bm^wdG6VISNosC=HGiT~Pp(km&En#2f}Y^le$=!}uM*mk`yV57j|b zCpm~w^Xb#4F<5(q@NgJw=SR|Ruh25a;Nlmw7i4xcsBR0c3#1oj8m+;B((@9q6GM4# z?Q~;6F@oG_Br663nV_jrPgZIsHmT*?>JvBe?;!^&1%YZHxe*mM&!)Eul^3eV`qB_R zf&b7+|IE!XgYF^B??D=>L169pLMUK{@~%X2K!hcvX++8?i{Cu{KesvE~6)#?^8C ztXcL8Sq|19m|WV~J{hIUyG%@mwr+WUn^Kwk!ppF?mcY)7D=UJx#xAF*#ztRXXhnzR zBtIxq{>QL+I!~ycKe@^Dg=T6{lZmg+;f>`{Ub(2)Gog0&M_IR>JGI-_qLz)Kah-nPMkbh{T-vFeHf=BUOq(&bDs61R#2B3>nSj?~&k^|;5i4T-(2Xjg#y8m~w~dZ;6$oPgoLyZ*FmA&j#3*}`QO5fRW~H@l4;JN1dvc4!2QflH1*Zfn~k(>W6Vkj-X zK|zD@Uh92zyk!*~9UVzRw2{#^H3hd&We&bK4GJ3`9+v%=jE!4dU0WN0F#^RW8GVJA z;XrFA%mniCJCRcfU}Va15>W(}zwJxKN=Ub*OM{1@tA)jw-cx$Q17YYi26thH8{WS^ zW!+n9rQum!9X2m-tcDS5oV3?xD9t;`$t%$c-Q3&=kqC3S0zhx*kRQH%i$GvCV@OW8 zFfe<|*q`N3jVc-uQ5PORe)B!Dy@TT>judb$? zj}JngVQ8WD$m_tUq?wv zckkAGvm1P#nT^c{S!7^g5%aUq6uuF~$0JagU|r!kyKvI%3Qh8b7`Hz^K1g`39^Acq zx6u{z>;kHvAXpf9fHWzRjFpSID;H1*&nfPfALo|w;bHlv z=V!$*gKJj$jR80lu&5Rw>W4{T@K$UFE#Gw##0vBII0?llWv)pAx)$9U&v%{U=;`Ts zD&-NanW{=Veln+1Ii+e#vxdR=BGGcj$H^}jpwqY0Chxgp?nPjwn@0o%3nzZj3$x_eI0m!3YDiCg&^N^ zO&ogl59|xU4#A!b4GqgNM5Cyf6=^SUU?tR#%dro**jQXhD-b_+PQ-S~Myj0Mlw5_A)e`M6SfDR%0}R0WZt3 zW1FEdL95r-I!(A{u+4xmFD0z5y}dySpt%YoQ7w5b7CPMj)aR#SF6*meklP1wxo2>9 zoT?9f2LaN+$mlF!3Fa2~&Y7>5^!U+A5F0Nfj8?P?3JRw1SL|MsT?LxH`5Dj>KG=&w zg_dnB*w1;9Z4-rqg9D}zVKKQB1D}Go@Oe%<&yQWf80Z-y4(J+Y5H>Hb6UIf?3haRx zB4;ZmApD!nUv+hJ7H9mR?NvsxKZh5BJ+P#r_QI=+ z=_nD&_?_CNP7rY;z4P%>Mo`i*!6i&W05fPxH&6q7IRVg6oSUrz=Ihk4hPJZn14 z>cI|p*2g(|3&LK8mo3-^XNjPHS^ zs;lcMdL|KJMm#YUgXUjG=j7VRe8lK?Q?s2?Q`?KK^BhWfYx-3h^mI(9K7V|$dundZ z2)a8VtD=L2qKxC_D=`lbUqhrEg6R$votVTU+X?d(;Vlx%pfXT&V}fv8)8l2Hh8TkK zf07`uC?JbVOF;eS9UX;;zd$rVKRz6;Scx|141WL;!U$h>s{YRsa5s3_2HTL?Sm9UO z76;5jh*^4QC>3IPyE5lPhzUl}7EXwXRVui-->d_}3p=_EEfs6R5uZHky7;SxiUjdM zXi;Dra9?nJ(BGdT->@p`haAJ-)$BtTG)5fWQ= z?6iL4S)^0)ISm7sg~cgD>^kiUTLu*H0aUR3!a|PYr<TTA*6MfZ*(W0<)(b>uoxne}1(! zG?0@Z$!jCH>j+)0>hLM7S536=t0ucMXU?dlX@MD#VLXp185-`zLwmEj^X+8PS~)Jh zexXGM@Q(P)@>sbA#ImzOs0y(1N!aUJN8mgg>QiKU_FUYoK!1{kQhW#<{f(BfH=`Mr z`~08nZE9Cwf;&>8`$#d_0H~grZ{VTP{%B1OBV9@tf9{646MIYy0mBuOg)SSG-BkUT zQ%?QW*Hazq5zsmq(d|&^)MOY&4BO-mA3h9EWgTK>YIau0)z#AZrNu#M3VsG$pj|(RnQCa?NXKE=AMr~n>d*d#xvn3ibole*G`1uFdInrN6{v3b z`M&-8lUrN75AQ6uA?J5-SzWY4RZYG&YS@;(9rZ>gS;G!)0aWobVDKs&4>1b!6ZP4& z@#<>OQOTDQ2_o6tzgyGFvFZWe&X3G}ft?67nD{ElXAiW~Q*b%b`C|UF=7VLY-N%!% zKY3~Xe%@Y3^|FmzGFHkx=5^m>#qUI53--2(-84L3^6d;^KMHuWyC%(_!T5wN)VV*i zvo{w1^aapM>(UxMK4PXrlYBO&2)deGQ0O76iTf!3iY@ijQ7esHiIT!8hC}rOrNjXWm{ZvI8s-uRf;bf#t1C1DYdM>l8I`x1V+pR6% zByI$v?oYBBzVHN#;=H^($`4+vi~JZAsoL$Wd4|yzG2R43X0N+@_wHqUAEd71fWyWZ zR4D#oP9y8phb{qy;@Ma80+$(RYkL56%vb6|ch;X=5_8d9!o|yLelzf0V!G=LI z8tDK!4w%Obq;Nb-5LhSCM~IGzo?A;LSiUtIh3c96()9iN_y0iB>>nSG1qmdI$roDz ztwULEFOt2n7YQ0^?XbWd3fv9}+0x1i35kjpPFNH%eE;D^^9JFpo~lX*M<8L6K>H-B zRjpLNg<7^vZyfF=%dTGzIN}t%jBIMpNYG@UK*9n^keEkDC65u-19IIl(P!|T?uHZw z*{&Mskub?(VT_=*v)p{JO3i$6g7V}LS}0;^5Y-ZlJ_KY|R?iNE`U)gLzyl*NbFkL^ zKD4TD(GH#Sd?ndUEWJnTL3L`*ORsIM=#s_naRhb*EOJZVvU&>NFSwjO3O1B1S`}N< z;i-d^l+U@8xTFEFJ`?s$=s2c5#RUSq>hMG2!~DDCD&nrq%cT4osD??PxTJ(u#4=K# z_;{ARk!kVa!O2biqS@|ib8TdYgm&IG|N4@)A#E;AF=K{H1l1Qy_&wh)vH9R5I(fwg zan^#i3GhmytYUEg5p0(rzRFT5IXS`vpDT5bf+2^k*948db~N9nyYPW(YNA@g8AN&@ zL_c_Nz$jm-N8O7gbtMOme$3vJ5~}+;Jsox!{n2mfrP z{#gy5?iu0IJby9BF|V%(g49$2#zu+Xe5o+Nmn9l5Y}x)8we=kcBnZw!7{p14_<=pr zuJ)Q@#zSwhn;=x50Tvbqp6Ee1-2n`l1dK+SBFLW>wc(w6?Oz$bk<@zjv2BUMmZsYJtMTGW3Lzf^2|mexq!4TFL19!;$s*P8)XUb`S_)U;qw2%FM9Jk37+Obukca znIm+SHNwezM>kP}7D~PQ{I%1+nZ9r_E}*Z^ zbaK4)yVV!ZM62>G8cY(BBW6HD(12<+OWGZnqK=^>5r$2062cHv-_UR++jQTzZ{KQv zb+d<`bh5^=0DU3Sh(4yC919vu*c5L_rj6_%95-ca^M@g)YOy>%Y5!{{sE`JW7^gHJJ_+^uPgBo6#3c07OoS`^J!zQc6WX48Wu}IC*>ZAYxEBe%E4_C7Gt1!m@oc$MXe4}^v+_Vm`vLgXdRx6U8hh8HZkQt zCpg0Wf{&m32}?ejYdg{(1Ayb8#xOIpj948(bB0tAO=$5WBMO-9gn4K8k+aiIUG*I6 zFm>J~1`8}?^l~uB+mWk|j*d*&pd*s5VQ@0QEkkEQ=5C2eL{(ec6M}xB(!foce{(cq zIwG1PzL~iqtXSwl-)cZWz*eZm82Emus`3W7gi|VANH?^Wk5|spduh8s%&d5$)Gr7z zV908(zSdIdYc_|q{wtMXzyk}b?8vcDPt%%`T2c*2(Ai5%tAHUKF@XWI1{ZXI77jFC z>D23S4kZOEID|twzM}L@y*j3ahNE z^T(bS-WpdVc2EBcLQObYb5;h(%m9Z&KwBST^E`zb4OTK6!Y=m+<7^9hntb?}V>)8C&a%JIvA1i@DdJ8nc zR2W-cYvg^>OI@K-n|H zO~;WbX2|(u#c+e=*^yth;r^H{y@h7=67Mk8B4hle)WdCgVUiGb z3Aa54DA$1-NL9y%M@N;=v*G#P1YSaj;YcS@m1(AT@~^u*#-1J)Hvg9^0vT{b%>F)v z7G_s2jN%~CDaMN5B!+&(90SaaknliZIlGIb2mN_V2~Qk6`#WP*&*4xMR5{`>RD#S7 zhJMewiZ_9DCvPPIbHEI1Kl+V(fSb~7e(VF52RDc_X{w;$$n`$B*p+54W5jm@N0^8f zOpL$?@i0Z@@Wl*WEBggQ&FN-pFf&3^!x5}UC_GHsZvc{sK1Q6;f#wRM9AqJr=`1p` z8H0*-8aMm;dM3%nckN{5>KIGWr79;G$A)%U5K1=G8)&*ImA6dDpFs#-tFtZVVW2YJ!n^z5xA1dq!|(yee` zo%-4Wp~3*V0i@rzgxwVfkq{D>-`?<55Y@o+SFr%*J0@@mcAkegGEgg9h-Xn}k&P3v zidOHCt_mCN1MS4Dy$@kTIFZm>*=*bueb0++XeRP(j+As0fqRS5Ykl>p_)WjN+jciY z6ku8);k!lyB)=U0+FvSM#L0lnVHu8ZY!S`SOlvVkEaCtoHyHAy0a^DzSHX&|H7&mY zl16886z@LwFvsK5m(CF&3DcYo*_jDw%Qt}Hptd|EDA?H8Yi0isYRt1AX{s8Y&wBjb z{WV?h^Rv5&^ExoX!qMyTe4!Tu@gx)mA`%h&23A(EIg((X2k|&qlS3tZ4DY%qUd~1> z(NNYjV~|bAEr@lM(y9Gtb_Qo6eEAHk$WZ1^!K@%4E_RI0sw+R3uxg+oGtJ#qO2cr* z9~HO>$$P?qVHe|iOcD>Jf_k#zerQEFAQ)Lg^*MEW0nM7tr_2Pae|f#4z7zgnjNS1~ zqS5!%V0`@a8Wn|5Bl4jIAh;FV7(x~&)gJk*?V_2w;^XJ{5&eU39Q;<3V!XeY(LCY^ znX)jcj4Ushe`|NP*RD%}V z-n?l9!-5^lO;rnf-sN5fna=-OweBk<#s6~#;QyCZt0ttG4t`<=;pV^YG^~@x6=b>VH*?2rt4RGp{yZNt3Q0WLrhU{0Wu|jc@LPa zk^rp;11GZ49^PEX?-$+0F%y%;<7T~kmxy@9^%{m}0Ld|#lLc@gOk=$`+X=&8gJZQk zx9?YOptSPJ%7MgDqwU0?6jQtMpbg?|5^-$m>C@q_uV^retj)~!sV86$EQ+iaooHT3 z_U08s$IwvF-w$#7!@B|uMI3U4>W=x`euAx{(1hF!K&_hwSW7fQjW$5MpihTm&`BI# z!+<0J2Ylz9_9@ILYz;xaf#8Y?xYNgnf+-QMezM+&^3lMQ#L+LLt(fy42BMS_zHv!c zA)OXtiUSQosxS`Z1SLcDAvJR&4o@OB8#4Q0$%}^pE(C)K0-hnbn?~noBaQ~U0=f`q zW1?`(3u7HtRBpKMMH}+=#|OcyBWU`C0gBQmOwm4I2B3h|4R0j)G3esrbR92&si7Vw z!VN=!Ol*%yv|$p9=I8;jnWL4!;7!r!)Xd#80;tI{lMsfm*a{b%Oj9fo(#A6QiwOFWSrvIR{rIS-&pC z@XU@d;gs0-&cdRi(F}l<=l!WwQc|n{-Z7<+0ccApf`!gKI{Fpz#pq4(*thk)y?W9p zSY*lp6tGa*I`Esv%m6D=_cCBuMoL>}YQ&C@9)+5f5~iSqN;t@45lBXMwWog{tLw zLEMRZF>w@7ZU!1Cx%jH#v0(;WnY~z#fk~{=u$7h79JqcBzBxb?t}hRK#T@ui8$IT` zzs8Bt!(?Y(Mo&|dx*m*XLdpq?T2|eK7xbV@dB86bZiH^Tj3E9^FXe8UZSCxw-_y{b zQierlqBBJJE<0Y=eH_M;+C7bmdR#+t))f}1 z)I6evv z0FF;sQ{HcQD~Z9Lf~JN$dhxmpJ|Oe9H!4vAM#Rw(466K`=#Py<>hnSCdajWcX4Duj zoqFod=9(T8G&_O@K&OQvs&90(4kh6-l80!5Na^lEOLRdPl-SF6fyx5CrNWa!obUiL zXD>0w2aJTW8*C>q4JJ+?uP={gIJZ#m+jko67qpMC-9iszUvWo~Bt$BxYlQm;M@a_W z0R!+uH-*C|6V(udVvvd;{o#QZwE5gn-#wc8Hfc*q<0 zVHOU=z-4p?rcCe#&2Ju9Uudfz>$y^VW7WEeQ$j$~=a2^IH05=5buW1M$@YmJ1~_AZx49apl{I*o%DY20Ee%WR;>&T??EI3c9brrzqia1%9G!6v@pw;;O z?M-H30#oc5upEUt#BV;c8AH!|sAfQ1R@q&^*2n}_Gp6^TCxj6L?lZffru3_#IIh=Z zkIDeK`|g7WXDQ19`31vim$;^b8;{X1anJN${uHb`;AS`WIg5H+lzXo0D;1{$XRY=V zl{~|gr7yWC5X8ZWj%>4X%(MzHw1)1A@}3X`suz{B6GR+gNdCAv(C-Z)IO6im!WwarQG+ed^klrF3c;Q+!} zNXc?2OSgKcaRw_4Apm(-WL{t`-Dn@8xc;x}%0L0b6!YFBJ)2^AUudmoJ`-Fvre1s>32uK{l z0z-88SwU(PGcqy)bJ$}P^C>th>~P-)c&@!@$)dwbp-7!g=`Cqm8m{9XkN*)2@y5Q-Q` zJp2a6aD{`g$`0ik;bb_$3;{IWnKm8sT9==|fKMYaXnOIpJ za2+Vt=YSsIsWC>^BEuU4^7Bs^t)WWZLkW9}%OJ1hFqWQ{XD!+B;uH8r1kE8FVoYUbOw&5=%hAGb*@{)oRW=jkbhbxBH1 zCC-;ad);zMTl-K?PY+zBiZxlYwkLR~b?d-qst~Kfo*OrA+_3Gh!f`m$XUcUoHR`o^ z`9k40hqbeyg&a{AHKaCNr!F1hwz)sRrBrA=R$E!L@k?y&)<#~d?&ZV%HUU{zt*d)1 zHY6h3-K_;hlq}g!zT;Z+964uftv`_(-?*;azkfqwt6E~p^6GEd3MomGJ{uaZKaKL(}^dChbiKcqo zCij2ufzV%pcRE0EBlKS7Xdz$1@CWP}8X78NZ!Z8n8U4o=2LO%*rm>mBsfeH@a42+u zurxLwuUtf3%#1T%`)qwuUkIeUTNLH1We z7ec|(C~>_REn@ZZnghN)oTLkG9Ku75i81mFApuG3zFaKaH1dEU#BnUhsb7~S--GfX zifN+O^A8ItP%ie`V{V73Ix*0+w#M`{->!c<&K98W#bD-**OgpjB2CV{S?N`c1X_e~ zAII_f(AN-4yPzEk2nwRFss>w(qy3x0Lp#mO!m^#PXqK)oTNpRS5ob?{$v!MV?B4|9 z&YeEJiE!IjRBXL^^(tXSRaEp-wNk=SrIWYDe!#Ar0(rT3p~nrUbIw?Z73Ol|H~YZS z0**>XBNPa&h@kw$FE~vk{hKZ9h82-`h`hYEPgGw4)|6FL2*VJFU3qo&4k{`txSRO+ z`PDljpi_e4ZIdXm;ujP&gcupdef0p$Xi)5AOihP1AEhTGFhYgmny%RBvuk7F<(*9H z+&>w|sv~{h`6*!pg_RB8f+hIY$%*>2%6RyRp4Wi^|AG!T`_GB1hck(IZxNc3(g*&gz?B1HXobGz%S} zlwK)ENi_TZmKn<(CMsHN3_?@Y(h>@-j`%9zz^RX&?m{m&Oj~>LVhenViNQ zjqdyRw_>f6-QQf!dMuQx0cKqQWpY4VGsgwb zuEL*X1kDE;);BSEn3l$VZ#((sU2dhVFE99mUYTAqB&Md8mRuO+5z)vd@q~y*T;Y4R z85OL;H*njEjbDGV^`DjK$dz#2Bb{#LvS^%kE(5mIZyuInh1uNrSQ0YlkPMElOAw{P z*Vp%uktEj&*1dL!HHVv`vjLS+6O?SgY_X*y0@}7|b^)gNgLq`^(G4!Ey0YG9OjiD1 z2F=}+gm9U({_GqO6Xrsofu-w?`cts~qzhzdfhBvb@+ z)ok;xyRfW4!A&hKi2&h>v`b4%)YQ~!XxoZmYzgY#IMk)#T~xHPd$NCgM;?E?e$Wz` z1DCP(^`iO%TO_wg<#D>#l>erBHFB}+@aLOPv)If&v|lt-Pc)wVT9W;uKgam`}m{m?ubFlgQN7b@0;60Nav|Cd9I!i|JeG4dPw}UrKH~Cv*n?8 z(k+RnA07L^C~W?AW<%k!)Z`ZlZ;``_3b(RIsL(f)X^Ittk$Bn^YihkRjr9)tZ`my z>6g*u;MH_=DB=}f0Nb1`DD(XM{EW=Z+cY&bF*c}aXfOiqM@{vE*i>9~sI17|-hQ~Z zOd39ZxN*#WDZ1`^)ER^~x_~+nZX-a<>85k^`>zn?5(fw`L_LX!sDj$z$CU@BMoC2l zH)|r!QxLC?(T?YfiHaiFEXdzk%)sw_)OA2}EQWgqA^h<5Yc9-yNU67<$&n2A2ti%F zZf2HtEuJ{=0^KKU>HGKZhr~Kb;6F?0qOPp0RL4NbXzg75tkjXXSs~~Y#QZuSfNUVG zmT>A!O}V6a?6XNB?C@x)Z-H$=pc45rh;&qWm7bnnm9sz~ydbDy$6DYCa&d8i-IxTe z3*mZ+iJ}juz`}bhFl|BC*^IlXfqd;(67ZzY%*r3frtkwGFY|A&NuTX&bqRZ8r35A= zudJN(?AfUry=;Jd_ykatI6^M5OG)V(I(zc~iX~3P%m9d?iPqHBY2H=9k`w*6cGWA7 z8+hK$oDSoc_H47z=_SbFzy|Eq>OK$SZmsP-69XPiMNfYo@KHF$2AW|xka0=$zWnSnr#LqY89Y$TWze8h>ZS`6Uv3Ibxra(0?KIVHvZ z*Qb4ftFIdqpPQYv4qQ%c@Kx|jScvFZEqk$1ePl?if~EScw6*lt!AmWdCr?}Vtu?lm zz}0!=^peJauy=X4eAx%Li0I>QENeU2HYO4NUs*5qcYb`*FeF0NeEh!GRR?==-vz3u z>(Y1Jq84@rk`A;#Gvv)KYaC5pkayKGH|I!6Nr9?Q9HN8C8^?%i-NDKUx(+D0B`{9; zw!L8oUa3TQhKZL-ojJ1^K!TtX!y_a7Fw0`U&Y>4?-@g5Zk&&6u#|~@jUL%U?XSrrA z-cbE!aQG-BF77i35(qmX%>haihY^S)kk-~b+qP}1waq)1$EX_!15R*MlrQ8NlpZD~ zrpJ#T1Gjc4v1ec)BxKzciI`}?nUwTRsQ^3gZq!f`GfBWa;@xkaNzxp`!u#PwMu9o4 zp+U)c@ihY{r;^wc9OP_(9t_)?<5+tz=o@UibhyAnxWtK9PGLSk9G*bjhDJx%LiJx- zUzyCA4|5HHM2dGngg3$!wu&(irltO~vyP~i=X7=Bs@6Rhx}BI|s*&*{t!Y$h z2N!oyP_QyHKOkC|QQd*SxAPO-!~q}w)&xu!iTsBl4yldzsC>lbEi%D}h<66UBgTfq z3e!%}j~_qNoc>93Z5rkB`4eAB?~yk zk&(@fQ|;i1xM(?C0AjAQFWz>8!Vq7SJoGIQ&p^EM1+Ob)MUlkA*2roIU_teGW22%!tvW!Q28N@k8(amKFdvgIU-V${s)FKIoYw zx3Py%&#N)%2V@S3jQj*dvDCYi*^4&D3wfaiOX|;_r}IiBQXUlpFM#=oB}fYR3Eq8~Yr$3Fq@8!Wsr} zJy7F|1Be7s_s%IODk2mEya%G;OM=YOaN?ukmQ)HTF24o`2{ZV!KS{euNfY2O#;LU1 z*!XK^UmzUZ#)dfHY#uB~7#x|PR1wAiyeUr*18n$nR6!)tF1$KGuoVnMp*Rq}OGs;Y zuS8orbO#>YV)#s*guEFc=4iNt*46!hbZ==nW_V8+=QW`$N&g@Z96lE8Y1mv#6(dPG5tZE+D)62iY=U~mkg zhdy|-`}}v4#x8Ow>T8E;j@SMB070af0}EJpDp)*T4yS9#b`RtZhvW!7k>Em@E>Paa z7|+1kS`~^baRL+~!673LBpmGgDi)H&8^sX3i&((+T&u(Q=C6nDQ`tOOU3Eum753Jb zUR{`EWUVZC<6H#M2S8Sj548%fhEWBl$Hy;tZUrABY)_b6pT#f+Ep-Q+JH#+L&qgn{ z=jQ@Pt+VcVi+`Obfl(GWKx_$Mote&rf>j;pKB0r4gUIOV#bf4V2Q36(xCSJWcxMmP z^&}YRanK+FTnJG112lKcc54LZe!4`e%{ToTz{NqiPl2oxdJ@h}1so9zfrWC@1Jb4Q zvEmPKWCY#-jm^qxC+=J0r61Z3m!qPiabDmMaRvtPgpG~O*>d6x<9#-3uvg;jfcta} zE4<89P%L3>^#dRV6m1$dnk{~LF})VLNYI@-o1k9f%)>j^IdkH?1o)mG6AAC00p;4@ zkPDF^&mURQA6U!sdGSh}$%I*O;mftId)-Z!Mna!P1t^eGkzeTX7zCkuH zx`*72ijwMML?m~TQEJxWqe?~FUGJYKZL>1Z)^~2Vbrl^;&pDdnokC_RaQpW0O{?zn zCaxmGSH@Oea)vLHD;{%%M+TZJUbOQ^u>S%MQO^Y zPL&RC4nBso~K0)L9PS8`E$m#z@zpzyb@x~ z)MSWhv@*7DyVP6JsyZ@lsySdOJPW z+HIEb=*;AmQ45)#NGDS4z&jj|0{EjMJx0o?%MJlMKAs-l5aOjlukrdD=r0C%3!$s) z^5~nmy=Q##zJzuhz+>1PrLcH(>@?GD5#f%X(uf#h?z75H(bkts=7Y7-b1#ImwdN$$4LS&g+bS zpV#xp`7^!ze(wAJey{I!eYX3Xz?a`jjV@_a7ep|Lp=bZoc?(+GRAaO~As+ zsB>|3e+JRy_#pN_U-zbAS4AgdI8kxJ?c1lKg!~lSSK_#5C>3)ro{m%NA}%B32(JeZ>*Dx&-M zFzP9@=gu89t|TYN8_wR~^-ly>rK2EW{MqGG+vkQ}SwrcL8wAJFj8?-te>Hu2a{6wz zr(k5(NgwRON^WRPn4qAmZ3Hz}_v<%gi1_U>vQ>|YDuh}&($2$2-Q3tNfLM7_&%hh( zKMiy?JtM=Z_NtDK&a(Zpg~URx%o*6y(h`P$5fy1DoBi%eYeY04>yHWx_g3uyKNjPz znE24HN=^hY+QO0f11+X2qhVtfLzu`lrGV5)TGm$NJpDkN~;NNIyx=zxP8=$oBQ-fDE$14;W$s) zxy=c=Kws-IP%pA5(i3DGaiisk6zo`Zk|Oye_9R--e%lT>#y)*xcpPg8JtYlub$Hf8 zx+MOD^#OV)I>E$eEMD9fqTd$w{u{i?Jq+Fi7S$5fHf!zLD*K3CI2Gi?BG*guzhOGd zfeQiXqKY;*SqHnQ%%4ABq!j>^BEwRWL7jk>eoKGi@Pw~bAQVIa)x*}x;lQ1$yOq(@pQdyB2Qyhl7xw#2~eDXyx?djNw6OAGk zs}a}1(OVR2XBoXob`@z^N(XNjuwKTxd~mFZZ4Zu5U0xgEY`2_aLxLfc(5fm2(y=Vt zQlJ@%#RPO4awKBx8bU44paw1_xPZnIJUg!7RsM!=`C{6iGJZ;1lK$|a&e*ZNXz~DU z*V9x%$NlBYLsi{0r)luGP-|wQH;mn^r>onOC5f*Xj3S6{lPpz!D0R0kPC;>JAbybvYePu^BWn2KhyfS1J9po%q}0P--Vs2OQD)|ssQSg?2HdHz z%T2Lg1^y%Pl3*$##@qJnMuGj0|44(RU0xNK6>9hxpEdFqbhg-!?FcUU`CRPan-JHG`PElzVo$$S8-dOFT&9BxG7qQw(|6oFhYN%KyB@L-WI=RtQYx*}%4`l!hk z&fj74&YBxMV?l7F^j23F!MPwyh{se{F>h-O9N0{rX(N)%7(o>RD_PCF9b*+mnCB#> zl7LoXarUDxbVCN_%`nadGm8!#H%<`SXcBhXw{5f7envnWo-Y;2dd;t_>PN6Uk{Oef1WRS$^{erF4go<#HecbA z-Xd)V!ZGu)&CAQfaNrPK@6X7CaQ3S)W4zefqi)Anho2|GgnXadsj0zBYIVMxJay_Q zq8TIPl*Q_}Z5+c~(f_?cT>=pQm~$6VV*fGpP_-v;MVP7@ zL?ctoAtTouR3d0f8c8?e_FPAWb!>1^?C^ig-1gv~GqAc>ll5=Wg7%vAtgQJClbf^U*TzCMX1yNF|4$F)E>|daG+}TwFo=5nDUE zxe(=wl~+jn7v7YD&)k2EG{1*whV3OMK7ZhOtr}a3(n1x%1_e{qpQ)M1viH$jsI8+j z&c4b7PEbj`Rfl@<=-ua`AV-KpFO7Y!rKQ!0myrz3I&=E8w~tTbo%f$fTO(3M&>F?_ z1i$)Jlp08rO=?eBc;m~E!Q!nsO%1+lH;H(#}J<9*snityuvdc7wX zXGBn@C%7N!KVU$4Y3VJxunE2y+1a>2X>9|B7LQu+i${T#EeNG!x$X)+o;7xt7Bf97 zE2X%2x!);&|GD(U*CKyhw=V3_<@#qdn6wsa$w9%gvdYfx^8GHv-*X{QkXT2(*sJCm zsobA3G@8Qb%nv_z8WRD}75{ZbCV4EP`!)%fa+0@$fMqR^CDl0FN1=&ute`qFVm6q}lv0r$|UqbNmi z@YkE4-VEPn^V%D7mM%E6`iEV!C7%RfSLI-eM4Scr^r&^J9rm*&A0RRd6j^fd=;Bw@ zZ8v*=4D9D2;;9U&%*5?Xo$3!P3#%2|-44Se|6ZQah-_fL+jnKQGQYp-3K03q-fLs}fg9uY}Nf(vR#yKMjX2Psb-h0!JvAAUsT;B5tJKywIAZhF)= z$-vZ+*2xc9Y}Z=$t3lk6yfgpk zAINeS|0&C9wAFP({=?kSo{n#;w(-^I=ti3@Rp&!oF1(?1qe=XHyS+WH?MW)IJ=Z!I z$H3qmKUn`DrJ18bQVZF*MCMF^=(E`Ffz+lKx#&U>T+Wx7fg9pk zYD8t{e=v2Qf+7`RtdeLckaNLUD85-LuLM!I^RC!&2`>rtoq6I4kJ>g7Gk0$fy0eA9 zL}B>#_U>t0vbPRyfHG}^nw#UAkib2#e5ruCZ&A9Otjx{L#YFHuLxGxFihy(|9Vx36 zAW?eJ*wVS*qmyKwOwMsx9A>VJqE`Z30$(m$&lDit?PEnF-f#T6e{^eX#pz_w|fSw(4g?p&Dt2i@oc<|#>awkB3G0Lh;BqHZuN zFa>>%B~G+n5EJ24Jo8B#?=df+?L5w$$uaLo-eZ?SGj+}6a;vl_Py7(g_Nnn}8s4eX z{$VaZVERP5GF*A^@cUkBg9k_K2(v(eFHxWf^k~!aZtUEmT`EdrQ4aD4&7L#o4qis! zc*gdVuLzT!$^0NMvf&qG_?I|uT^e-4R|MNfyE(@!G=7C7%%O}Rd@U#alp1fZVuW$+ zeZc+BfqW$vWWki*{sHGGK{T)pj}q*Ev+hKSF%)lL$5sD+qNh0 zrEMoMDgb4}-J#fRGC7hzB-A?D1)BIE86KjB2j~$>f}tgZbBKdZcs4e!JCa^>OMF~Y zbV_ZY?ugcuL|*je4nES1bW&wnw1*64oCER#7LGQOb|j=^%0Ca0r-l{G{8$N{^n_z2 z=z%a2+)YnEObr?7PfI0i`7ls`s7+o@p(yDGZleNR(E#7iuVG<@6;S}BOg?Tf^ zBK&;yqJ>_LeSWos(P%0Pn(nm>&kiNEac-2CqtnO1sm>>yf1n(a#g@el6Gso-Res`c z)L&6fimC|ZqeAuO7Vm2PWZ~`M0M4Jb``v==45INOp?T)rw6yW7ZYllr;PWwSuT%Y#1kxCssWo$(oX1eT$S+Es?|oMj z8g*e#c^~P%06F%QTzUZiXb^QS4J@);#fi1vd2g7 z-!I$wSp4+opiSi^jXFIxsco4)IXTte$UW{;zVqSu2UA}c{5xD)aNOPhXYBL&!45Ie nPrK!6{VCr0zW|MG8`5XpWBoBP{S>_XPsw=JyqS^TuiE!739Z;b literal 0 HcmV?d00001 diff --git a/docs/sphinx/user-docs/ray-cluster-interaction.rst b/docs/sphinx/user-docs/ray-cluster-interaction.rst new file mode 100644 index 000000000..8e7929b4d --- /dev/null +++ b/docs/sphinx/user-docs/ray-cluster-interaction.rst @@ -0,0 +1,90 @@ +Ray Cluster Interaction +======================= + +The CodeFlare SDK offers multiple ways to interact with Ray Clusters +including the below methods. + +get_cluster() +------------- + +The ``get_cluster()`` function is used to initialise a ``Cluster`` +object from a pre-existing Ray Cluster/AppWrapper. Below is an example +of it's usage: + +:: + + from codeflare_sdk import get_cluster + cluster = get_cluster(cluster_name="raytest", namespace="example", is_appwrapper=False, write_to_file=False) + -> output: Yaml resources loaded for raytest + cluster.status() + -> output: + 🚀 CodeFlare Cluster Status 🚀 + ╭─────────────────────────────────────────────────────────────────╮ + │ Name │ + │ raytest Active ✅ │ + │ │ + │ URI: ray://raytest-head-svc.example.svc:10001 │ + │ │ + │ Dashboard🔗 │ + │ │ + ╰─────────────────────────────────────────────────────────────────╯ + (, True) + cluster.down() + cluster.up() # This function will create an exact copy of the retrieved Ray Cluster only if the Ray Cluster has been previously deleted. + +| These are the parameters the ``get_cluster()`` function accepts: +| ``cluster_name: str # Required`` -> The name of the Ray Cluster. +| ``namespace: str # Default: "default"`` -> The namespace of the Ray Cluster. +| ``is_appwrapper: bool # Default: False`` -> When set to +| ``True`` the function will attempt to retrieve an AppWrapper instead of a Ray Cluster. +| ``write_to_file: bool # Default: False`` -> When set to ``True`` the Ray Cluster/AppWrapper will be written to a file similar to how it is done in ``ClusterConfiguration``. + +list_all_queued() +----------------- + +| The ``list_all_queued()`` function returns (and prints by default) a list of all currently queued-up Ray Clusters in a given namespace. +| It accepts the following parameters: +| ``namespace: str # Required`` -> The namespace you want to retrieve the list from. +| ``print_to_console: bool # Default: True`` -> Allows the user to print the list to their console. +| ``appwrapper: bool # Default: False`` -> When set to ``True`` allows the user to list queued AppWrappers. + +list_all_clusters() +------------------- + +| The ``list_all_clusters()`` function will return a list of detailed descriptions of Ray Clusters to the console by default. +| It accepts the following parameters: +| ``namespace: str # Required`` -> The namespace you want to retrieve the list from. +| ``print_to_console: bool # Default: True`` -> A boolean that allows the user to print the list to their console. + +.. note:: + + The following methods require a ``Cluster`` object to be + initialized. See :doc:`./cluster-configuration` + +cluster.up() +------------ + +| The ``cluster.up()`` function creates a Ray Cluster in the given namespace. + +cluster.down() +-------------- + +| The ``cluster.down()`` function deletes the Ray Cluster in the given namespace. + +cluster.status() +---------------- + +| The ``cluster.status()`` function prints out the status of the Ray Cluster's state with a link to the Ray Dashboard. + +cluster.details() +----------------- + +| The ``cluster.details()`` function prints out a detailed description of the Ray Cluster's status, worker resources and a link to the Ray Dashboard. + +cluster.wait_ready() +-------------------- + +| The ``cluster.wait_ready()`` function waits for the requested cluster to be ready, up to an optional timeout and checks every 5 seconds. +| It accepts the following parameters: +| ``timeout: Optional[int] # Default: None`` -> Allows the user to define a timeout for the ``wait_ready()`` function. +| ``dashboard_check: bool # Default: True`` -> If enabled the ``wait_ready()`` function will wait until the Ray Dashboard is ready too. diff --git a/docs/sphinx/user-docs/s3-compatible-storage.rst b/docs/sphinx/user-docs/s3-compatible-storage.rst index 60937441b..0ca2cc0d1 100644 --- a/docs/sphinx/user-docs/s3-compatible-storage.rst +++ b/docs/sphinx/user-docs/s3-compatible-storage.rst @@ -82,5 +82,5 @@ Lastly the new ``run_config`` must be added to the Trainer: To find more information on creating a Minio Bucket compatible with RHOAI you can refer to this `documentation `__. -Note: You must have ``sf3s`` and ``pyarrow`` installed in your +Note: You must have ``s3fs`` and ``pyarrow`` installed in your environment for this method. diff --git a/docs/sphinx/user-docs/setup-kueue.rst b/docs/sphinx/user-docs/setup-kueue.rst index 86956e011..1f2bdc041 100644 --- a/docs/sphinx/user-docs/setup-kueue.rst +++ b/docs/sphinx/user-docs/setup-kueue.rst @@ -11,10 +11,9 @@ Kueue resources, namely Cluster Queue, Resource Flavor, and Local Queue. 1. Resource Flavor: ------------------- -Resource Flavors allow the cluster admin to define different types of -resources with specific characteristics, such as CPU, memory, GPU, etc. -These can then be assigned to workloads to ensure they are executed on -appropriate resources. +Resource Flavors allow the cluster admin to reflect differing resource capabilities +of nodes within a clusters, such as CPU, memory, GPU, etc. These can then be assigned +to workloads to ensure they are executed on nodes with appropriate resources. The YAML configuration provided below creates an empty Resource Flavor named default-flavor. It serves as a starting point and does not specify diff --git a/docs/sphinx/user-docs/ui-widgets.rst b/docs/sphinx/user-docs/ui-widgets.rst new file mode 100644 index 000000000..6c797e043 --- /dev/null +++ b/docs/sphinx/user-docs/ui-widgets.rst @@ -0,0 +1,55 @@ +Jupyter UI Widgets +================== + +Below are some examples of the Jupyter UI Widgets that are included in +the CodeFlare SDK. + +.. note:: + To use the widgets functionality you must be using the CodeFlare SDK in a Jupyter Notebook environment. + +Cluster Up/Down Buttons +----------------------- + +The Cluster Up/Down buttons appear after successfully initialising your +`ClusterConfiguration `__. +There are two buttons and a checkbox ``Cluster Up``, ``Cluster Down`` +and ``Wait for Cluster?`` which mimic the +`cluster.up() `__, +`cluster.down() `__ and +`cluster.wait_ready() `__ +functionality. + +After initialising their ``ClusterConfiguration`` a user can select the +``Wait for Cluster?`` checkbox then click the ``Cluster Up`` button to +create their Ray Cluster and wait until it is ready. The cluster can be +deleted by clicking the ``Cluster Down`` button. + +.. image:: images/ui-buttons.png + :alt: An image of the up/down ui buttons + +View Clusters UI Table +---------------------- + +The View Clusters UI Table allows a user to see a list of Ray Clusters +with information on their configuration including number of workers, CPU +requests and limits along with the clusters status. + +.. image:: images/ui-view-clusters.png + :alt: An image of the view clusters ui table + +Above is a list of two Ray Clusters ``raytest`` and ``raytest2`` each of +those headings is clickable and will update the table to view the +selected Cluster's information. There are three buttons under the table +``Cluster Down``, ``View Jobs`` and ``Open Ray Dashboard``. \* The +``Cluster Down`` button will delete the selected Cluster. \* The +``View Jobs`` button will try to open the Ray Dashboard's Jobs view in a +Web Browser. The link will also be printed to the console. \* The +``Open Ray Dashboard`` button will try to open the Ray Dashboard view in +a Web Browser. The link will also be printed to the console. + +The UI Table can be viewed by calling the following function. + +.. code:: python + + from codeflare_sdk import view_clusters + view_clusters() # Accepts namespace parameter but will try to gather the namespace from the current context