implementing vuln scanner and h100 cluster deployment

Author: clubanderson

.tekton/README.md

@@ -1,6 +1,10 @@
 ## 🛠️ CI/CD Pipeline Overview – Your Project
 
-This pipeline is designed to support safe, efficient, and traceable development and deployment workflows using OpenShift Pipelines-as-Code, GitHub, and Quay.io.
+<!-- NOTE TO CONTRIBUTORS: every repo in the hc4ai organization is intended to have the same contents in this file. The origin is the copy in https://github.ibm.com/mspreitz/hc4ai-hello-neural/blob/dev/.tekton/README.md; submit PRs against that one -->
+
+This pipeline is designed to support safe, efficient, and traceable development and deployment workflows using [OpenShift Pipelines-as-Code](https://pipelinesascode.com/), [Tekton](https://tekton.dev/), [buildah](https://buildah.io/), GitHub, and Quay.io.
+
+This pipeline is used for CI/CD of the `dev` and `main` branches. This pipeline runs from source through container image build to deployment and testing in the hc4ai cluster.
 
 ---
 
@@ -24,19 +28,28 @@ Each repo includes a `.version.json` file at its root. This file controls:
 
 #### 🔑 Fields:
 - **dev-version**: Current version of the dev branch. Used to tag dev images.
-- **dev-registry**: Container registry location for development image pushes.
+- **dev-registry**: Container repository location for development image pushes.
 - **prod-version**: Managed by automation. Updated during promotion to match the dev-version.
-- **prod-registry**: Container registry for production image pushes. The promoted dev image is re-tagged and pushed here.
+- **prod-registry**: Container repository for production image pushes. The promoted dev image is re-tagged and pushed here.
 
 The pipeline reads this file to:
 - Extract the appropriate version tag
-- Determine the correct registry for image pushes
+- Determine the correct repository for image pushes
 - Promote and tag dev images for prod
 
 ---
 
+### Container Repositories
+
+This pipeline maintains two container repositories for this GitHub repository, as follows.
+
+- `quay.io/vllm-d/<repoName>-dev`. Hold builds from the `dev` branch as described below.
+- `quay.io/vllm-d/<repoName>`. Holds promotions to prod, as described below.
+
+---
+
 ### ⚙️ Pipeline Triggers
-Triggered on `push` and `pull_request` events targeting the `dev` or `main` branches.
+Triggered on `push` and `pull_request` events targeting the `dev` or `main` branches. The following workflows are the two behaviors of this pipeline.
 
 ### 🔧 dev Branch Workflow
 1. Checkout repository
@@ -47,20 +60,20 @@ Triggered on `push` and `pull_request` events targeting the `dev` or `main` bran
     - prod-version
     - prod-registry
 4. Build and push container image to:
-   → `<dev-registry>:<dev-version>`
+   → `<dev-repository>:<dev-version>`
 5. Tag the Git commit using the `dev-version`
-6. Optionally redeploy objects to OpenShift in `hc4ai-operator-dev`
+6. Optionally redeploy objects to OpenShift in the `hc4ai-operator-dev` namespace.
 
 ✅ This process ensures that all code merged into dev is validated and deployed for testing.
 
 ### 🚀 main Branch Workflow
 1. Checkout, lint, test, and parse `.version.json`
 2. Skip image rebuild
 3. Promote image by copying from:
-   → `<dev-registry:<dev-version>` → `<prod-registry>:<prod-version>`
+   → `<dev-repository:<dev-version>` → `<prod-repository>:<prod-version>`
 4. Tag the Git commit using the `prod-version`
 5. Update the upstream repo’s submodule to reference the new tag
-6. Redeploy to OpenShift in `hc4ai-operator`
+6. Redeploy to OpenShift in the `hc4ai-operator` namespace.
 
 ✅ No image rebuilds occur on main. Only validated dev images are promoted, ensuring reproducibility.
 
@@ -84,8 +97,8 @@ Tags are created using the configured Git credentials and pushed to the remote r
 
 ### ☸️ OpenShift Deployment
 The pipeline includes automated deployment:
-- On `dev`: Deploys to `hc4ai-operator-dev`
-- On `main`: Deploys to `hc4ai-operator`
+- On `dev`: Deploys to the `hc4ai-operator-dev` namespace. The Pod is named `<repoName>-major-minor`, using the `dev-version` from `.version.json`.
+- On `main`: Deploys to `hc4ai-operator` namespace. The Pod is named `<repoName>-major-minor`, using the `prod-version` from `.version.json`.
 
 Using `make uninstall-openshift` and `make install-openshift`, resources are cleanly reset.
 
@@ -112,6 +125,7 @@ After deployment, the pipeline:
 
 ### 🧠 Why `.version.json` Matters
 - Decouples versioning from Git commit hashes
-- Provides a single source of truth for version and registry info
+- Provides a single source of truth for version and repository info
 - Enables deterministic builds and controlled releases
 - Simplifies debugging and auditing across environments
+

.tekton/pipelinerun.yaml

@@ -12,14 +12,6 @@ metadata:
       (!has(body.ref) || body.ref == 'refs/heads/main' || body.ref == 'refs/heads/dev') &&
       (!has(body.head_commit) || !has(body.head_commit.author) || !body.head_commit.author.name.matches("(?i).*ci-tag-bot.*")) &&
       (!has(body.pull_request) || (body.pull_request.base.ref == 'main' || body.pull_request.base.ref == 'dev'))
-    results.tekton.dev/columns: |
-      [
-        {
-          "name": "Vulnerabilities",
-          "type": "string",
-          "jsonPath": ".status.pipelineResults[?(@.name==\"vulnerabilities\")].value"
-        }
-      ]
 spec:
   podTemplate:
     serviceAccountName: pipeline
@@ -39,6 +31,10 @@ spec:
     - name: source_branch
       value: "{{ source_branch }}"
   pipelineSpec:
+    results:
+      - description: The common vulnerabilities and exposures (CVE) result
+        name: SCAN_OUTPUT
+        value: $(tasks.vulnerability-scan.results.SCAN_OUTPUT)
     params:
       - name: repo_url
       - name: revision
@@ -162,6 +158,90 @@ spec:
           - name: source
             workspace: source
 
+      - name: openshift-redeploy-h100
+        when:
+          - input: "$(params.runOptional)"
+            operator: in
+            values: ["true"]
+          - input: "$(params.source_branch)"
+            operator: in
+            values: ["dev", "main"]
+          - input: "$(tasks.read-cluster-name.results.cluster-name)"
+            operator: notin
+            values: ["cluster-platform-eval"]
+        taskRef:
+          name: openshift-redeploy-task
+        params:
+          - name: source-branch
+            value: "$(params.source_branch)"
+          - name: prod-version
+            value: "$(tasks.extract-version-and-registry.results.prod-version)"
+          - name: dev-version
+            value: "$(tasks.extract-version-and-registry.results.dev-version)"
+          - name: prod_image_tag_base
+            value: "$(tasks.extract-version-and-registry.results.prod-image-tag-base)"
+          - name: dev_image_tag_base
+            value: "$(tasks.extract-version-and-registry.results.dev-image-tag-base)"
+        runAfter:
+          - extract-version-and-registry
+        workspaces:
+          - name: source
+            workspace: source
+
+      - name: go-test-post-deploy-h100
+        when:
+          - input: "$(params.runOptional)"
+            operator: in
+            values: ["true"]
+          - input: "$(params.source_branch)"
+            operator: in
+            values: ["dev", "main"]
+        taskRef:
+          name: go-test-post-deploy-task
+        params:
+          - name: source-branch
+            value: "$(params.source_branch)"
+          - name: prod-version
+            value: "$(tasks.extract-version-and-registry.results.prod-version)"
+          - name: dev-version
+            value: "$(tasks.extract-version-and-registry.results.dev-version)"
+          - name: prod_image_tag_base
+            value: "$(tasks.extract-version-and-registry.results.prod-image-tag-base)"
+          - name: dev_image_tag_base
+            value: "$(tasks.extract-version-and-registry.results.dev-image-tag-base)"
+        runAfter:
+          - openshift-redeploy-h100
+        workspaces:
+          - name: source
+            workspace: source
+
+      - name: benchmark-h100
+        when:
+          - input: "$(params.source_branch)"
+            operator: in
+            values: ["dev"]
+        continueOn:
+          errors: true
+        params:
+          - name: openshift_host
+            value: "https://api.fmaas-vllm-d.fmaas.res.ibm.com:6443"
+          - name: openshift_namespace
+            value: "hc4ai-operator-dev"
+        taskRef:
+          name: benchmark-task 
+        runAfter:
+          - go-test-post-deploy-h100
+
+      - name: pipeline-complete-dev-h100
+        when:
+          - input: "$(params.source_branch)"
+            operator: in
+            values: ["dev"]
+        runAfter:
+          - benchmark-h100
+        taskRef:
+          name: noop-task
+
       - name: promote-to-prod
         when:
           - input: "$(params.runOptional)"
@@ -234,7 +314,7 @@ spec:
           - name: IMAGE_URL
             value: "$(tasks.buildah-build.results.image-url)"
           - name: SEVERITY
-            value: "CRITICAL,HIGH"
+            value: "CRITICAL,HIGH,MEDIUM,LOW"
           - name: ARGS
             value: "--exit-code 0"
         workspaces:
@@ -250,7 +330,6 @@ spec:
             values: ["cluster-platform-eval"]
         runAfter:
           - promote-to-prod
-          # - buildah-build
           - vulnerability-scan
         taskRef:
           name: noop-task

.tekton/vuln-scan-trivy.yaml

@@ -2,26 +2,30 @@ apiVersion: tekton.dev/v1
 kind: Task
 metadata:
   name: trivy-scan
+  annotations:
+    task.output.location: results
+    task.results.format: application/json
+    task.results.key: SCAN_OUTPUT
 spec:
   params:
     - name: IMAGE_URL
       type: string
       description: Full image URL (e.g., quay.io/org/image:tag)
     - name: SEVERITY
       type: string
-      default: "CRITICAL,HIGH"
+      default: "CRITICAL,HIGH,MEDIUM"
       description: Comma-separated severity levels
     - name: ARGS
       type: string
       default: ""
       description: Additional Trivy arguments
+  results:
+    - name: SCAN_OUTPUT
+      description: CVE result format
   workspaces:
     - name: registry-secret
       description: Workspace with Docker config.json (auth for private registries)
     - name: output
-  results:
-    - name: vulnerabilities
-      type: string
   steps:
     - name: trivy-scan
       image: docker:20.10.24-dind
@@ -57,11 +61,29 @@ spec:
           $(params.ARGS) \
           "$IMAGE" > /workspace/output/trivy-results.json; then
           echo "❌ Trivy scan failed"
-          echo -n "-1" > $(results.vulnerabilities.path)
+          echo -n "-1" > /tekton/results/vulnerabilities
           exit 1
         fi
 
-        echo "📊 Counting vulnerabilities..."
-        vuln_count=$(jq '[.Results[].Vulnerabilities[]?] | length' /workspace/output/trivy-results.json)
+        echo "📋 Trivy scan result:"
+        cat /workspace/output/trivy-results.json
+
+        echo "📊 Parsing vulnerabilities..."
+
+        vuln_count=$(jq '[.Results[].Vulnerabilities[]?] | length // 0' /workspace/output/trivy-results.json)
         echo "📊 Found $vuln_count vulnerabilities"
-        echo -n "$vuln_count" > /tekton/results/vulnerabilities
+
+        if [ "$vuln_count" -gt 0 ]; then
+          # Parse the vulnerabilities and ensure that missing categories are assigned zero count
+          jq -rce '
+            {
+              vulnerabilities: {
+                critical: ([.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length),
+                high: ([.Results[].Vulnerabilities[]? | select(.Severity == "HIGH")] | length),
+                medium: ([.Results[].Vulnerabilities[]? | select(.Severity == "MEDIUM")] | length),
+                low: ([.Results[].Vulnerabilities[]? | select(.Severity == "LOW")] | length)
+              }
+            }' /workspace/output/trivy-results.json > /tekton/results/SCAN_OUTPUT
+        else
+          echo "📊 No vulnerabilities found, skipping parsing."
+        fi