This is a companion project to openshift-by-example and the associated article that now tackle the topic of Operators development and packaging using tools and practices from the Operator Framework.
More precisely, this repository contains samples on how to develop Operators using the Operator SDK and the 3 different technologies embedded: Helm, Ansible and Go language. We hope it will help you understand the pro and cons of different approaches and how they can map on the Operators Maturity Model phases.
The operators we have developed here are made to deploy the fruits-catalog application we usually use. This application is made of 2 components:
- A WebApp component made using Quarkus. You can find the code within the
fruits-catalog/folder, - A MongoDB database.
The container image for the WebApp part of the application was made available as quay.io/lbroudoux/fruits-catalog:latest.
operator-sdkshould be installed and present into your$PATH. We use latest1.0.0version and previous0.19.2version. Just check https://master.sdk.operatorframework.io/docs/installation/install-operator-sdk/ on how to install it,docker(or another tool compatible with multi-stage Dockerfiles) should be installed and present into your$PATH. Minimum version is17.03+,goshould be installed and present into your$PATH. Minimum version is Go1.13,kubectlshoud be installed and present into your$PATH.v1.16.0+is the minimum version,
Operator SDK allows creating Operator using/re-using Helm Charts. Whilst they're great starting points, we tend to think that Helm only allows you to support the first phases of the maturity model.
Discover how to develop such Operator on the dedicated Helm page.
| Pros | Cons |
|---|---|
Templating and basic controls (if) |
Unable to discover cluster capabilities |
| Simplicity | Custom Resource is not typed |
| Great to start with | YAML hell ;-) |
Discover how to develop such Operator on the dedicated Ansible page.
| Pros | Cons |
|---|---|
| Templating + Control flow structures | Role or Playbook is fully evaluated at each reconciliation |
| Ability to discover cluster capabilities | Custom Resource is not typed |
| Hybrid Operators: in and out the cluster | YAML hell ;-) |
| Development loop and CI/CD with modelcule | |
| Huge Ansible ecosystem for any equipment, device, stack, ... |
Discover how to develop such Operator on the dedicated Go page.
| Pros | Cons |
|---|---|
| Go is full language | Complex Kube resources created with Go APIs are verbose |
| Ability to access and leverage Kubernetes API (discovery, types, schemes, ...) | Personal note: coming from Java, I hate Go error management style ;-) |
| Custom Resource are fully typed | |
| Development loop with IDE support, local testing, CI/CD | |
| Very fine grained control on reconciliation loop | |
| Optimized for few resources (CPU, memory) |
The Operator Lifecycle Manager (OLM) is a set of cluster resources that manage the lifecycle of an Operator. The Operator SDK supports both creating manifests for OLM deployment, and testing your Operator on an OLM-enabled Kubernetes cluster.
The OLM defines some mandatory metadata manifests that can be generated and then completed before having them packages into what will define an Operator bundle.
So start, generating this famous manifests and everything we'll need to create a bundle!
As an example, we did this from within the fruits-catalog-ansible-operator-1.0.0 folder
$ export IMG=quay.io/lbroudoux/fruits-catalog-ansible-operator:0.0.2
$ make bundle IMG=$IMG
operator-sdk generate kustomize manifests -q
Display name for the operator (required):
> fruits-catalog-ansible-operator-1.0.0
Description for the operator (required):
> Operator for the FruitsCatalog app using Operator Ansible SDK 1.0.0
Provider's name for the operator (required):
> Laurent Broudoux
Any relevant URL for the provider name (optional):
> https://github.com/redhat-france-sa/openshift-by-example-operators
Comma-separated list of keywords for your operator (required):
> sample,operator-sdk,fruits-catalog
Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required):
> lbroudoux:laurent.broudoux@redhat.com
cd config/manager && /Users/lbroudou/Development/go-workspace/bin/kustomize edit set image controller=quay.io/lbroudoux/fruits-catalog-ansible-operator:0.0.2
/Users/lbroudou/Development/go-workspace/bin/kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1
INFO[0000] Building annotations.yaml
INFO[0000] Writing annotations.yaml in /Users/lbroudou/Development/github/openshift-by-example-operators/fruits-catalog-ansible-operator-1.0.0/bundle/metadata
INFO[0000] Building Dockerfile
INFO[0000] Writing bundle.Dockerfile in /Users/lbroudou/Development/github/openshift-by-example-operators/fruits-catalog-ansible-operator-1.0.0
operator-sdk bundle validate ./bundle
INFO[0000] Found annotations file bundle-dir=bundle container-tool=docker
INFO[0000] Could not find optional dependencies file bundle-dir=bundle container-tool=docker
INFO[0000] All validation tests have completed successfully
The main metadata manifests generated is a Cluster Service Version. It is indeed an aggregate of your Operator CRD, deployment informations, RBAC permissions + a number of metadata allowing to identify owner, provider and categorize operators. More information on CSV here.
The previous command generates 2 things:
- Aggregation rules of elements using
kustomizewithin theconfig/manifestsfolder of the project, - Result of aggregation of elements within the
bundle/folder of the project.
If you need to make adjustements to the generated CSV, you'll have to edit the config/manifests/base file and then re-run the following:
$ operator-sdk generate bundle -q --overwrite --version 0.0.1
0.0.1being here the version of the generated CSV.
The Scorecard command within Operator SDK executes tests on your operator manifests and future bundle based upon a configuration file and test images. Tests are implemented within test images that are configured and constructed to be executed by scorecard.
In order to run, Scorecard need you to be connected to a Kubernetes cluster. You just ask Scorecard to eevaluate the manifest that have been previously generated and are present into the bundle/ folder of your project like this:
$ operator-sdk scorecard bundle
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test olm-crds-have-resources]
Labels:
"suite":"olm"
"test":"olm-crds-have-resources-test"
Results:
Name: olm-crds-have-resources
State: pass
Log:
Loaded ClusterServiceVersion: fruits-catalog-ansible-operator.v0.0.1
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
"test":"olm-crds-have-validation-test"
"suite":"olm"
Results:
Name: olm-crds-have-validation
State: fail
Suggestions:
Add CRD validation for spec field `name` in FruitsCatalogA1/v1beta1
Add CRD validation for spec field `webapp` in FruitsCatalogA1/v1beta1
Log:
Loaded 1 Custom Resources from alm-examples
Loaded CustomresourceDefinitions: [&CustomResourceDefinition{ObjectMeta:{fruitscataloga1s.redhat.com 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:CustomResourceDefinitionSpec{Group:redhat.com,Names:CustomResourceDefinitionNames{Plural:fruitscataloga1s,Singular:fruitscataloga1,ShortNames:[],Kind:FruitsCatalogA1,ListKind:FruitsCatalogA1List,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1beta1,Served:true,Storage:true,Schema:&CustomResourceValidation{OpenAPIV3Schema:&JSONSchemaProps{ID:,Schema:,Ref:nil,Description:FruitsCatalogA1 is the Schema for the fruitscataloga1s API,Type:object,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{apiVersion: { <nil> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources string nil <nil> false <nil> false <nil> <nil> <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},kind: { <nil> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds string nil <nil> false <nil> false <nil> <nil> <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},metadata: { <nil> object nil <nil> false <nil> false <nil> <nil> <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},spec: { <nil> Spec defines the desired state of FruitsCatalogA1 object nil <nil> false <nil> false <nil> <nil> <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false 0xc000236f10 false false [] <nil> <nil>},status: { <nil> Status defines the observed state of FruitsCatalogA1 object nil <nil> false <nil> false <nil> <nil> <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false 0xc000236f11 false false [] <nil> <nil>},},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},},Subresources:&CustomResourceSubresources{Status:&CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test olm-status-descriptors]
Labels:
"suite":"olm"
"test":"olm-status-descriptors-test"
Results:
Name: olm-status-descriptors
State: fail
Log:
Loaded ClusterServiceVersion: fruits-catalog-ansible-operator.v0.0.1
Loaded 1 Custom Resources from alm-examples
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test basic-check-spec]
Labels:
"suite":"basic"
"test":"basic-check-spec-test"
Results:
Name: basic-check-spec
State: pass
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test olm-spec-descriptors]
Labels:
"suite":"olm"
"test":"olm-spec-descriptors-test"
Results:
Name: olm-spec-descriptors
State: fail
Suggestions:
Add a spec descriptor for webapp
Errors:
webapp does not have a spec descriptor
Log:
Loaded ClusterServiceVersion: fruits-catalog-ansible-operator.v0.0.1
Loaded 1 Custom Resources from alm-examples
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test olm-bundle-validation]
Labels:
"suite":"olm"
"test":"olm-bundle-validation-test"
Results:
Name: olm-bundle-validation
State: pass
Log:
time="2020-08-31T07:52:00Z" level=debug msg="Found manifests directory" name=bundle-test
time="2020-08-31T07:52:00Z" level=debug msg="Found metadata directory" name=bundle-test
time="2020-08-31T07:52:00Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test
time="2020-08-31T07:52:00Z" level=info msg="Found annotations file" name=bundle-test
time="2020-08-31T07:52:00Z" level=info msg="Could not find optional dependencies file" name=bundle-test
The Scorecard output gives you advices and suggestions for making your Cluster Service Version and aggregated manifests compliant with best practices.
Once you're happy with the Scorecard results, it's time to package and distribute your Operator bundle. The previous commands have also generated a bundle.Dockerfile at project root and you can simply use it for building a contaienr image for your bundle:
$ docker build -f bundle.Dockerfile -t quay.io/lbroudoux/fruits-catalog-ansible-operator-bundle:v0.0.1 .
$ docker push quay.io/lbroudoux/fruits-catalog-ansible-operator-bundle:v0.0.1
Last step before going is to validate the published bundle with this final command:
$ operator-sdk bundle validate quay.io/lbroudoux/fruits-catalog-ansible-operator-bundle:v0.0.1
INFO[0000] Unpacking image layers
INFO[0000] running /usr/local/bin/docker pull quay.io/lbroudoux/fruits-catalog-ansible-operator-bundle:v0.0.1
INFO[0001] running docker create
INFO[0001] running docker cp
INFO[0002] running docker rm
INFO[0002] Found annotations file bundle-dir=/var/folders/wy/lb2qd3m51ld_k8ds2b6xkz180000gn/T/bundle-845300542 container-tool=docker
INFO[0002] Could not find optional dependencies file bundle-dir=/var/folders/wy/lb2qd3m51ld_k8ds2b6xkz180000gn/T/bundle-845300542 container-tool=docker
INFO[0002] All validation tests have completed successfully
The very last part of this sample/tutorials projets is on the topic of easily distributing your Operator to customers, partners through the concept of market place as we have in OpenShift ;-)
For this, just have a look at the Fruits Catalog Registry.