feat(template/helm): Don't include product repo in core operators

README.adoc → README.md

@@ -1,37 +1,38 @@
-= Templates for Common Files in Operator Repositories
+# Templates for Common Files in Operator Repositories
 
 Stackable develops and maintains a growing number of operators for open source software.
 
 The high level structure of the repository is consistent across all repositories, which in the past has led to repetitive maintenance tasks in many repositories even for small changes.
 
 This repository is intended to help with these changes by keeping common files as templates and offering tooling to roll these out to all repositories that are under management by this tool.
 
-== Structure of this repository
+## Structure of this repository
 
-=== GitHub Actions
+### GitHub Actions
 
 The definition files for GitHub actions that in turn execute the playbooks doing the actual work are kept in `.github`
 
-=== Playbook
+### Playbook
 
 `playbook` contains the ansible playbook which is executed to perform the needed changes.
 
-=== Templates
+### Templates
 
 Everything under the top level folder `template`  is replicated to the target repositories.
 The folder and file structure under `template` is considered as source structure which will be synchronized across all repositories in the following manner:
 
-* Regular files and folders are simply copied
-* Files with an extension of `.j2` will be processed as jinja2 templates and copied to the target repositories with the `.j2` extension removed. The default jinja2 variable delimiter has been replaced with `{[ }]` since a lot of the template files contain `{{  }}` and caused issues.
-* File and directory names which contain `\[[product]]` will have this substituted for the value of the `product_string` variable. For example: "stackable-\[[product]]-operator.service" would become `stackable-kafka-operator.service`.
+- Regular files and folders are simply copied
+- Files with an extension of `.j2` will be processed as jinja2 templates and copied to the target repositories with the `.j2` extension removed. The default jinja2 variable delimiter has been replaced with `{[ }]` since a lot of the template files contain `{{  }}` and caused issues.
+- File and directory names which contain `\[[product]]` will have this substituted for the value of the `product_string` variable. For example: "stackable-\[[product]]-operator.service" would become `stackable-kafka-operator.service`.
 
 To remove files or directories that already exist in the target repositories these need to be configured in `repositories.yaml` under the `retired_files` key.
 
 Anything that is listed here will be deleted from the target repositories.
 
-NOTE: Deletion is the last step that is performed, so if there is an overlap between files existing in the template folder and this setting, the files would not be rolled out, since they'd get deleted before creating the pull request.
+> [!NOTE]
+> Deletion is the last step that is performed, so if there is an overlap between files existing in the template folder and this setting, the files would not be rolled out, since they'd get deleted before creating the pull request.
 
-=== Configuration
+### Configuration
 
 All user-facing configuration is kept in `repositories.yaml`.
 
@@ -41,62 +42,51 @@ So in principle it is as simple as pushing any commit to `main` which will trigg
 
 Target repositories are configured in `repositories.yaml` in the following form:
 
-----
+```yaml
 - name: kafka-operator
   url: stackabletech/kafka-operator.git
   product_string: kafka
   pretty_string: Kafka
-----
-
-|===
-|Field |Description
-
-|name
-|This is only used internally to name working directories and the like.
-
-|url
-|The github repository for this operator. Need to be in the form of `<org>/<repo>.git`
-
-|product_string
-| A lower case string to use in config files, file names and the like. Should not contain whitespaces. This can sometimes be a shortened version of the full name, for example for Open Policy Agent this would be "opa"
-
-|pretty_string
-| The actual name of the product, including whitespaces and proper capitalization. This is intended to be used in doc or man files or similar things.
-
-|include_productconfig
-| Whether to include files from the `deploy/config-spec` folder into the os package.
-
-*Default*: true
-|===
-
+```
+
+| Field | Description |
+| ----- | ----------- |
+| `name` | This is only used internally to name working directories and the like. |
+| `url` | The github repository for this operator. Need to be in the form of `<org>/<repo>.git`. |
+| `product_string` | A lower case string to use in config files, file names and the like. Should not contain whitespaces. This can sometimes be a shortened version of the full name, for example for Open Policy Agent this would be "opa" |
+| `pretty_string` | The actual name of the product, including whitespaces and proper capitalization. This is intended to be used in doc or man files or similar things. |
+| `config.include_productconfig` | Whether to include files from the `deploy/config-spec` folder into the os package. *Default*: true |
+| `config.has_product` | Indicates that the operator manages a product. This is particularly useful to differentiate between core and product operators. *Default*: true |
+| `config.run_as` | Whether to run the operator as a default Deployment, or something custom. |
 
 These are the only variables currently being used on the playbooks, but can be extended easily as more are needed.
 
-NOTE: If a new variable is introduced, it needs to be added to all repository objects!
+> [!NOTE]
+> If a new variable is introduced, it needs to be added to all repository objects!
 
 Additional settings can be found in `playbook/group_vars/all`, but these are not intended to be freely changed and should be treated with care.
 
-
-== Making changes to the template
+## Making changes to the template
 
 If you want to make a change that should be rolled out to all operators, make the change in the `template` directory.
 Consult the section above to learn more about the structure of the template.
 
-=== Test changes locally
+### Test changes locally
 
-1. Run the `test.sh` script. To limit the run to a single operator, add `--extra-vars "shard_repositories=['nifi-operator']"` (or your choice of operator).
+1. Run the `test.sh` script. To limit the run to a single operator, add `--extra-vars "shard_repositories#['nifi-operator']"` (or your choice of operator).
    It will automatically delete and recreate a `work` directory.
 2. The changes can be examined with `git status`.
    When the pull request is later merged into the `main` branch then pull requests with these changes will be created automatically.
 3. Depending on the change, it makes sense to run the integration tests for all changed operators.
    If the tests are not run in this stage and if there is even just one integration test failing in the subsequential generated pull requests then the operator-templating must be adapted which creates again pull requests for all operators.
    Changes in the GitHub workflow actions cannot be tested until finally merged.
 
-== Deploying changes
+## Deploying changes
 
 Changes are rolled out via GitHub actions.
 
-=== Authentication
+### Authentication
+
 Since this tool needs to authenticate itself in order to push commits it needs credentials of a user.
 This is currently solved via a personal access token that needs to be provided as a repository (or org) secret with the name `GH_ACCESS_TOKEN`.
 
@@ -106,9 +96,11 @@ The personal access token needs to have the following permissions for this to wo
 - org:read
 - workflow
 
-== Limitations
+## Limitations
+
 There is currently no synchronization with existing PRs on the target repositories whatsoever. A new pull request will be created for every commit made to this repository.
 
 To update a PR that was created via this tool, it will have to be closed and necessary changes pushed here, which will result in a new PR.
 
-WARNING: The Helm Chart files that are rolled out by the templates in their current form do not include a ClusterRole object which may be needed for this to work with RBAC.
+> [!WARNING]
+> The Helm Chart files that are rolled out by the templates in their current form do not include a ClusterRole object which may be needed for this to work with RBAC.

config/repositories.yaml

@@ -6,72 +6,91 @@ repositories:
     pretty_string: Apache Airflow
     product_string: airflow
     url: stackabletech/airflow-operator.git
+
   - name: commons-operator
-    include_productconfig: false
-    has_product: false
     pretty_string: Stackable Commons
     product_string: commons
     url: stackabletech/commons-operator.git
+    config:
+      include_productconfig: false
+      has_product: false
+
   - name: druid-operator
     pretty_string: Apache Druid
     product_string: druid
     url: stackabletech/druid-operator.git
+
   - name: hbase-operator
     pretty_string: Apache HBase
     product_string: hbase
     url: stackabletech/hbase-operator.git
+
   - name: hdfs-operator
     pretty_string: Apache HDFS
     product_string: hdfs
     url: stackabletech/hdfs-operator.git
+
   - name: hive-operator
     pretty_string: Apache Hive
     product_string: hive
     url: stackabletech/hive-operator.git
+
   - name: kafka-operator
     pretty_string: Apache Kafka
     product_string: kafka
     url: stackabletech/kafka-operator.git
+
   - name: nifi-operator
     pretty_string: Apache NiFi
     product_string: nifi
     url: stackabletech/nifi-operator.git
+
   - name: listener-operator
-    include_productconfig: false
-    has_product: false
     pretty_string: Stackable Listener Operator
     product_string: listener-operator
-    run_as: custom
     url: stackabletech/listener-operator.git
+    config:
+      include_productconfig: false
+      has_product: false
+      run_as: custom
+
   - name: opa-operator
-    extra_crates:
-      - stackable-opa-bundle-builder
     pretty_string: OpenPolicyAgent
     product_string: opa
     url: stackabletech/opa-operator.git
+    config:
+      extra_crates:
+        - stackable-opa-bundle-builder
+
   - name: opensearch-operator
     pretty_string: OpenSearch
     product_string: opensearch
     url: stackabletech/opensearch-operator.git
+
   - name: secret-operator
-    include_productconfig: false
-    has_product: false
     pretty_string: Stackable Secret Operator
     product_string: secret-operator
-    run_as: custom
     url: stackabletech/secret-operator.git
+    config:
+      include_productconfig: false
+      has_product: false
+      run_as: custom
+
   - name: spark-k8s-operator
     pretty_string: Apache Spark-on-Kubernetes
     product_string: spark-k8s
     url: stackabletech/spark-k8s-operator.git
+
   - name: superset-operator
     pretty_string: Apache Superset
     product_string: superset
     url: stackabletech/superset-operator.git
+
   - name: trino-operator
     pretty_string: Trino
     product_string: trino
     url: stackabletech/trino-operator.git
+
   - name: zookeeper-operator
     pretty_string: Apache ZooKeeper
     product_string: zookeeper

template/.github/ISSUE_TEMPLATE/02-bug_report.yml.j2

@@ -12,7 +12,7 @@ body:
       label: Affected Stackable version
       description: Which version of the Stackable Operator do you see this bug in?
 
-  # {[%- if operator.has_product | default(true) +%}]
+  # {[%- if operator.config.has_product | default(true) +%}]
   - type: input
     attributes:
       label: Affected {[ operator.pretty_string }] version

template/Tiltfile → template/Tiltfile.j2

@@ -12,9 +12,11 @@ registry = settings.get('default_registry', 'oci.stackable.tech')
 operator_repository = settings.get('default_operator_repository', registry + '/' + 'sandbox')
 operator_image_name = operator_repository + '/' + operator_name
 
-# For the product image, we wanna use the images in the "sdp" namespace, because "sandbox" doesn't
+{[% if operator.config.has_product | default(true) %}]
+# For the product image, we want to use the images in the "sdp" namespace, because "sandbox" doesn't
 # contain those images (by default).
 product_repository = settings.get('default_product_repository', registry + '/' + 'sdp')
+{[%- endif %}]
 
 # Configure default registry either read from config file above, or with default value of
 # "oci.stackable.tech"
@@ -41,15 +43,19 @@ k8s_kind('DaemonSet', image_json_path='{.spec.template.metadata.annotations.inte
 helm_values = settings.get('helm_values', None)
 
 helm_override_operator_image_repository = 'image.repository=' + operator_repository
+{[% if operator.config.has_product | default(true) %}]
 helm_override_product_image_repository = 'image.productRepository=' + product_repository
+{[% endif %}]
 
 k8s_yaml(helm(
    'deploy/helm/' + operator_name,
    name=operator_name,
    namespace="stackable-operators",
    set=[
       helm_override_operator_image_repository,
+{[% if operator.config.has_product | default(true) %}]
       helm_override_product_image_repository,
+{[% endif %}]
    ],
    values=helm_values,
 ))

template/deploy/helm/[[operator]]/templates/deployment.yaml.j2

@@ -1,4 +1,4 @@
-{[% if operator.run_as is undefined or operator.run_as == "deployment" %}]
+{[% if operator.config.run_as is undefined or operator.config.run_as == "deployment" %}]
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -71,9 +71,15 @@ spec:
             - name: OPERATOR_SERVICE_NAME
               value: {{ include "operator.fullname" . }}
 
+{[% if operator.config.has_product | default(true) %}]
             # The product image repository, like "oci.stackable.tech/sdp".
             - name: IMAGE_REPOSITORY
               value: {{ .Values.image.productRepository | default .Values.image.repository }}
+{[% else %}]
+            # The image repository, like "oci.stackable.tech/sdp".
+            - name: IMAGE_REPOSITORY
+              value: {{ .Values.image.repository }}
+{[% endif %}]
 
             # Operators need to know the node name they are running on, to e.g. discover the
             # Kubernetes domain name from the kubelet API.

template/docker/Dockerfile.j2

@@ -191,7 +191,7 @@ COPY LICENSE /licenses/LICENSE
 
 COPY --from=builder --chown=${STACKABLE_USER_UID}:0 /app/* /usr/local/bin/
 
-{[% if operator.include_productconfig is undefined or operator.include_productconfig == true %}]
+{[% if operator.config.include_productconfig is undefined or operator.config.include_productconfig == true %}]
 COPY deploy/config-spec/properties.yaml /etc/stackable/{[ operator.name }]/config-spec/properties.yaml
 {[% endif %}]