diff --git a/README.adoc b/README.md similarity index 66% rename from README.adoc rename to README.md index 393382b7..e1b14618 100644 --- a/README.adoc +++ b/README.md @@ -1,4 +1,4 @@ -= Templates for Common Files in Operator Repositories +# Templates for Common Files in Operator Repositories Stackable develops and maintains a growing number of operators for open source software. @@ -6,32 +6,33 @@ The high level structure of the repository is consistent across all repositories This repository is intended to help with these changes by keeping common files as templates and offering tooling to roll these out to all repositories that are under management by this tool. -== Structure of this repository +## Structure of this repository -=== GitHub Actions +### GitHub Actions The definition files for GitHub actions that in turn execute the playbooks doing the actual work are kept in `.github` -=== Playbook +### Playbook `playbook` contains the ansible playbook which is executed to perform the needed changes. -=== Templates +### Templates Everything under the top level folder `template` is replicated to the target repositories. The folder and file structure under `template` is considered as source structure which will be synchronized across all repositories in the following manner: -* Regular files and folders are simply copied -* Files with an extension of `.j2` will be processed as jinja2 templates and copied to the target repositories with the `.j2` extension removed. The default jinja2 variable delimiter has been replaced with `{[ }]` since a lot of the template files contain `{{ }}` and caused issues. -* File and directory names which contain `\[[product]]` will have this substituted for the value of the `product_string` variable. For example: "stackable-\[[product]]-operator.service" would become `stackable-kafka-operator.service`. +- Regular files and folders are simply copied +- Files with an extension of `.j2` will be processed as jinja2 templates and copied to the target repositories with the `.j2` extension removed. The default jinja2 variable delimiter has been replaced with `{[ }]` since a lot of the template files contain `{{ }}` and caused issues. +- File and directory names which contain `\[[product]]` will have this substituted for the value of the `product_string` variable. For example: "stackable-\[[product]]-operator.service" would become `stackable-kafka-operator.service`. To remove files or directories that already exist in the target repositories these need to be configured in `repositories.yaml` under the `retired_files` key. Anything that is listed here will be deleted from the target repositories. -NOTE: Deletion is the last step that is performed, so if there is an overlap between files existing in the template folder and this setting, the files would not be rolled out, since they'd get deleted before creating the pull request. +> [!NOTE] +> Deletion is the last step that is performed, so if there is an overlap between files existing in the template folder and this setting, the files would not be rolled out, since they'd get deleted before creating the pull request. -=== Configuration +### Configuration All user-facing configuration is kept in `repositories.yaml`. @@ -41,50 +42,38 @@ So in principle it is as simple as pushing any commit to `main` which will trigg Target repositories are configured in `repositories.yaml` in the following form: ----- +```yaml - name: kafka-operator url: stackabletech/kafka-operator.git product_string: kafka pretty_string: Kafka ----- - -|=== -|Field |Description - -|name -|This is only used internally to name working directories and the like. - -|url -|The github repository for this operator. Need to be in the form of `/.git` - -|product_string -| A lower case string to use in config files, file names and the like. Should not contain whitespaces. This can sometimes be a shortened version of the full name, for example for Open Policy Agent this would be "opa" - -|pretty_string -| The actual name of the product, including whitespaces and proper capitalization. This is intended to be used in doc or man files or similar things. - -|include_productconfig -| Whether to include files from the `deploy/config-spec` folder into the os package. - -*Default*: true -|=== - +``` + +| Field | Description | +| ----- | ----------- | +| `name` | This is only used internally to name working directories and the like. | +| `url` | The github repository for this operator. Need to be in the form of `/.git`. | +| `product_string` | A lower case string to use in config files, file names and the like. Should not contain whitespaces. This can sometimes be a shortened version of the full name, for example for Open Policy Agent this would be "opa" | +| `pretty_string` | The actual name of the product, including whitespaces and proper capitalization. This is intended to be used in doc or man files or similar things. | +| `config.include_productconfig` | Whether to include files from the `deploy/config-spec` folder into the os package. *Default*: true | +| `config.has_product` | Indicates that the operator manages a product. This is particularly useful to differentiate between core and product operators. *Default*: true | +| `config.run_as` | Whether to run the operator as a default Deployment, or something custom. | These are the only variables currently being used on the playbooks, but can be extended easily as more are needed. -NOTE: If a new variable is introduced, it needs to be added to all repository objects! +> [!NOTE] +> If a new variable is introduced, it needs to be added to all repository objects! Additional settings can be found in `playbook/group_vars/all`, but these are not intended to be freely changed and should be treated with care. - -== Making changes to the template +## Making changes to the template If you want to make a change that should be rolled out to all operators, make the change in the `template` directory. Consult the section above to learn more about the structure of the template. -=== Test changes locally +### Test changes locally -1. Run the `test.sh` script. To limit the run to a single operator, add `--extra-vars "shard_repositories=['nifi-operator']"` (or your choice of operator). +1. Run the `test.sh` script. To limit the run to a single operator, add `--extra-vars "shard_repositories#['nifi-operator']"` (or your choice of operator). It will automatically delete and recreate a `work` directory. 2. The changes can be examined with `git status`. When the pull request is later merged into the `main` branch then pull requests with these changes will be created automatically. @@ -92,11 +81,12 @@ Consult the section above to learn more about the structure of the template. If the tests are not run in this stage and if there is even just one integration test failing in the subsequential generated pull requests then the operator-templating must be adapted which creates again pull requests for all operators. Changes in the GitHub workflow actions cannot be tested until finally merged. -== Deploying changes +## Deploying changes Changes are rolled out via GitHub actions. -=== Authentication +### Authentication + Since this tool needs to authenticate itself in order to push commits it needs credentials of a user. This is currently solved via a personal access token that needs to be provided as a repository (or org) secret with the name `GH_ACCESS_TOKEN`. @@ -106,9 +96,11 @@ The personal access token needs to have the following permissions for this to wo - org:read - workflow -== Limitations +## Limitations + There is currently no synchronization with existing PRs on the target repositories whatsoever. A new pull request will be created for every commit made to this repository. To update a PR that was created via this tool, it will have to be closed and necessary changes pushed here, which will result in a new PR. -WARNING: The Helm Chart files that are rolled out by the templates in their current form do not include a ClusterRole object which may be needed for this to work with RBAC. +> [!WARNING] +> The Helm Chart files that are rolled out by the templates in their current form do not include a ClusterRole object which may be needed for this to work with RBAC. diff --git a/config/repositories.yaml b/config/repositories.yaml index ddea14ca..9a62c9f1 100644 --- a/config/repositories.yaml +++ b/config/repositories.yaml @@ -6,72 +6,91 @@ repositories: pretty_string: Apache Airflow product_string: airflow url: stackabletech/airflow-operator.git + - name: commons-operator - include_productconfig: false - has_product: false pretty_string: Stackable Commons product_string: commons url: stackabletech/commons-operator.git + config: + include_productconfig: false + has_product: false + - name: druid-operator pretty_string: Apache Druid product_string: druid url: stackabletech/druid-operator.git + - name: hbase-operator pretty_string: Apache HBase product_string: hbase url: stackabletech/hbase-operator.git + - name: hdfs-operator pretty_string: Apache HDFS product_string: hdfs url: stackabletech/hdfs-operator.git + - name: hive-operator pretty_string: Apache Hive product_string: hive url: stackabletech/hive-operator.git + - name: kafka-operator pretty_string: Apache Kafka product_string: kafka url: stackabletech/kafka-operator.git + - name: nifi-operator pretty_string: Apache NiFi product_string: nifi url: stackabletech/nifi-operator.git + - name: listener-operator - include_productconfig: false - has_product: false pretty_string: Stackable Listener Operator product_string: listener-operator - run_as: custom url: stackabletech/listener-operator.git + config: + include_productconfig: false + has_product: false + run_as: custom + - name: opa-operator - extra_crates: - - stackable-opa-bundle-builder pretty_string: OpenPolicyAgent product_string: opa url: stackabletech/opa-operator.git + config: + extra_crates: + - stackable-opa-bundle-builder + - name: opensearch-operator pretty_string: OpenSearch product_string: opensearch url: stackabletech/opensearch-operator.git + - name: secret-operator - include_productconfig: false - has_product: false pretty_string: Stackable Secret Operator product_string: secret-operator - run_as: custom url: stackabletech/secret-operator.git + config: + include_productconfig: false + has_product: false + run_as: custom + - name: spark-k8s-operator pretty_string: Apache Spark-on-Kubernetes product_string: spark-k8s url: stackabletech/spark-k8s-operator.git + - name: superset-operator pretty_string: Apache Superset product_string: superset url: stackabletech/superset-operator.git + - name: trino-operator pretty_string: Trino product_string: trino url: stackabletech/trino-operator.git + - name: zookeeper-operator pretty_string: Apache ZooKeeper product_string: zookeeper diff --git a/template/.github/ISSUE_TEMPLATE/02-bug_report.yml.j2 b/template/.github/ISSUE_TEMPLATE/02-bug_report.yml.j2 index 6b999c01..6a78858f 100644 --- a/template/.github/ISSUE_TEMPLATE/02-bug_report.yml.j2 +++ b/template/.github/ISSUE_TEMPLATE/02-bug_report.yml.j2 @@ -12,7 +12,7 @@ body: label: Affected Stackable version description: Which version of the Stackable Operator do you see this bug in? - # {[%- if operator.has_product | default(true) +%}] + # {[%- if operator.config.has_product | default(true) +%}] - type: input attributes: label: Affected {[ operator.pretty_string }] version diff --git a/template/Tiltfile b/template/Tiltfile.j2 similarity index 89% rename from template/Tiltfile rename to template/Tiltfile.j2 index 02508e50..826d23bf 100644 --- a/template/Tiltfile +++ b/template/Tiltfile.j2 @@ -12,9 +12,11 @@ registry = settings.get('default_registry', 'oci.stackable.tech') operator_repository = settings.get('default_operator_repository', registry + '/' + 'sandbox') operator_image_name = operator_repository + '/' + operator_name -# For the product image, we wanna use the images in the "sdp" namespace, because "sandbox" doesn't +{[% if operator.config.has_product | default(true) %}] +# For the product image, we want to use the images in the "sdp" namespace, because "sandbox" doesn't # contain those images (by default). product_repository = settings.get('default_product_repository', registry + '/' + 'sdp') +{[%- endif %}] # Configure default registry either read from config file above, or with default value of # "oci.stackable.tech" @@ -41,7 +43,9 @@ k8s_kind('DaemonSet', image_json_path='{.spec.template.metadata.annotations.inte helm_values = settings.get('helm_values', None) helm_override_operator_image_repository = 'image.repository=' + operator_repository +{[% if operator.config.has_product | default(true) %}] helm_override_product_image_repository = 'image.productRepository=' + product_repository +{[% endif %}] k8s_yaml(helm( 'deploy/helm/' + operator_name, @@ -49,7 +53,9 @@ k8s_yaml(helm( namespace="stackable-operators", set=[ helm_override_operator_image_repository, +{[% if operator.config.has_product | default(true) %}] helm_override_product_image_repository, +{[% endif %}] ], values=helm_values, )) diff --git a/template/deploy/helm/[[operator]]/templates/deployment.yaml.j2 b/template/deploy/helm/[[operator]]/templates/deployment.yaml.j2 index f92b41c9..62f637cc 100644 --- a/template/deploy/helm/[[operator]]/templates/deployment.yaml.j2 +++ b/template/deploy/helm/[[operator]]/templates/deployment.yaml.j2 @@ -1,4 +1,4 @@ -{[% if operator.run_as is undefined or operator.run_as == "deployment" %}] +{[% if operator.config.run_as is undefined or operator.config.run_as == "deployment" %}] --- apiVersion: apps/v1 kind: Deployment @@ -71,9 +71,15 @@ spec: - name: OPERATOR_SERVICE_NAME value: {{ include "operator.fullname" . }} +{[% if operator.config.has_product | default(true) %}] # The product image repository, like "oci.stackable.tech/sdp". - name: IMAGE_REPOSITORY value: {{ .Values.image.productRepository | default .Values.image.repository }} +{[% else %}] + # The image repository, like "oci.stackable.tech/sdp". + - name: IMAGE_REPOSITORY + value: {{ .Values.image.repository }} +{[% endif %}] # Operators need to know the node name they are running on, to e.g. discover the # Kubernetes domain name from the kubelet API. diff --git a/template/docker/Dockerfile.j2 b/template/docker/Dockerfile.j2 index 31bdc067..0ccef5d2 100644 --- a/template/docker/Dockerfile.j2 +++ b/template/docker/Dockerfile.j2 @@ -191,7 +191,7 @@ COPY LICENSE /licenses/LICENSE COPY --from=builder --chown=${STACKABLE_USER_UID}:0 /app/* /usr/local/bin/ -{[% if operator.include_productconfig is undefined or operator.include_productconfig == true %}] +{[% if operator.config.include_productconfig is undefined or operator.config.include_productconfig == true %}] COPY deploy/config-spec/properties.yaml /etc/stackable/{[ operator.name }]/config-spec/properties.yaml {[% endif %}]