feat: lgen operator #204

samika98 · 2024-07-17T22:41:18Z

Features in this pr :

Generate crds with vars specific to load generation
Add reconciliation controller for load generation specific resources
Change operator yaml to give custom api access for generation load generation specific crds

Tests :

Test 1 : Create a load generation and apply it.

Create a network (similar to basic.yaml) in the docs - network.yaml
Create load generation yaml for configuring load generation

  # load-generator.yaml
apiVersion: "keramik.3box.io/v1alpha1"
kind: LoadGenerator
metadata:
  name: load-gen
  namespace: keramik-small
spec:
  scenario: "model-load-sync"
  runTime: 1
  image: "samika98/runner:final-1"
  imagePullPolicy: "IfNotPresent"
  throttleRequests: 100
  tasks: 10

Result : Success - Was able to generate load successfully against a ceramic node. Model got indexed on the ceramic peer.

dav1do

Mostly questions, am happy to approve if needed but was leaving the review to the folks more familiar with the k8s requirements.

dav1do · 2024-07-18T17:57:07Z

operator/src/lgen/job.rs

+        },
+        EnvVar {
+            name: "RUST_LOG".to_owned(),
+            value: Some("info,keramik_runner=trace".to_owned()),


do we really want to default to trace here? I don't think we actually have tracing yet, but that will probably cause a lot of logs/memory consumption 💸

This is just for future iterations. Cause we need tracing one less thing to do I guess when we enable it. We just need to figure out what's causing high memory consumption in tracing.
For now this wont cause a memory consumption increase

operator/src/lgen/spec.rs

runner/src/main.rs

dav1do · 2024-07-18T18:02:11Z

operator/src/lgen/controller.rs

+}
+
+/// Start a controller for the LoadGenerator CRD.
+pub async fn run() {


what do you see as the future of the simulation crd? will that likely go away? seems like there is a lot of duplication here, which is fine for now but we may want to think about how to share some things if we're going to keep both around (and/or add other variations)

I see load generation completely separate from simulations.
Simulations will be short lived, more flexible, produce burst traffic and offer way more flexibility and coverage in terms of use cases it covers.
Load generation is for steady non bursty load on the network.
I believe both can be used together as well. Have a steady load over the network, on top of that load we can run simulations to test out specific use-cases.

operator/src/lgen/job.rs

dav1do · 2024-07-18T18:23:51Z

operator/src/lgen/job.rs

+            metadata: Some(ObjectMeta {
+                labels: Some(BTreeMap::from_iter(vec![(
+                    "name".to_owned(),
+                    "load-gen-job".to_owned(),


should we use the LOAD_GENERATOR_JOB_NAME const for name in this file as well?

dav1do · 2024-07-18T18:24:27Z

operator/src/lgen/job.rs

+                ..Default::default()
+            }),
+            spec: Some(PodSpec {
+                hostname: Some("job".to_owned()),


I'm not super familiar with the hostname/subdomain settings here.. do we want something more specific?

Naming is hard. We can probably be more specific here, load-gen-job sound better?

Kubernetes uses the hostname and subdomain fields to create a fully qualified domain name (FQDN) for the pod. The FQDN will be of the form hostname.subdomain.namespace.svc.cluster.local
I feel given the subdomain, this should be fine. Any thoughts?

smrz2001 · 2024-07-26T19:02:16Z

k8s/operator/manifests/operator.yaml

@@ -87,7 +87,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: keramik-operator
-  namespace: default
+  namespace: keramik


I believe this was intentionally default. IIUC, the operator doesn't itself run in the keramik namespace, though @nathanielc might have a better idea if it matters much.

smrz2001

Only minor comments from my side. LGTM!

smrz2001 · 2024-07-28T23:38:40Z

operator/src/lgen/controller.rs

+        job::{job_spec, JobConfig, JobImageConfig},
+        spec::{LoadGenerator, LoadGeneratorState},
+    },
+    simulation::controller::monitoring_ready,


Should this be moved somewhere common? Maybe it's ok but feels odd to pull in simulation-related code in lgen.

smrz2001 · 2024-07-28T23:46:43Z

operator/src/lgen/mod.rs

+pub mod job;
+/// Spec module for creating load generator specs.
+#[cfg(feature = "controller")]
+pub mod spec;


Want to also add these couple of lines at the end? Very minor, just for consistency with the other mods:

#[cfg(feature = "controller")] pub use controller::run;

smrz2001 · 2024-07-28T23:47:36Z

operator/src/main.rs

@@ -38,7 +38,8 @@ async fn main() -> Result<()> {
        Command::Daemon => {
            tokio::join!(
                keramik_operator::network::run(),
-                keramik_operator::simulation::run()
+                keramik_operator::simulation::run(),
+                keramik_operator::lgen::controller::run(),


nit: if you also export run, you'll be able to use the same syntax as the other modes.

Samika Kashyap added 3 commits July 17, 2024 15:29

feat: operator to create a separate job for load generation

0210290

feat: add api access for load gen

08d6c03

fix: command name

8248082

samika98 marked this pull request as ready for review July 17, 2024 22:42

samika98 changed the title ~~Feat/lgen operator~~ feat: lgen operator Jul 17, 2024

samika98 requested review from dav1do, 3benbox and nathanielc July 17, 2024 22:45

dav1do reviewed Jul 18, 2024

View reviewed changes

samika98 requested review from smrz2001 and dav1do July 25, 2024 19:38

smrz2001 reviewed Jul 26, 2024

View reviewed changes

fix: naming + clenaups

0bdd901

samika98 requested a review from smrz2001 July 26, 2024 20:26

smrz2001 approved these changes Jul 28, 2024

View reviewed changes

chore: review comments

ec53b9c

samika98 merged commit bff0702 into feat/lgen-runner Jul 30, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: lgen operator #204

feat: lgen operator #204

samika98 commented Jul 17, 2024 •

edited

Loading

dav1do left a comment

dav1do Jul 18, 2024

samika98 Jul 25, 2024

dav1do Jul 18, 2024

samika98 Jul 25, 2024

dav1do Jul 18, 2024

dav1do Jul 18, 2024

samika98 Jul 25, 2024

samika98 Jul 26, 2024

smrz2001 Jul 26, 2024

smrz2001 left a comment

smrz2001 Jul 28, 2024

smrz2001 Jul 28, 2024

smrz2001 Jul 28, 2024

feat: lgen operator #204

feat: lgen operator #204

Conversation

samika98 commented Jul 17, 2024 • edited Loading

dav1do left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smrz2001 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samika98 commented Jul 17, 2024 •

edited

Loading