Using the CLI
1. Initialize a values repo
APL needs a git repo to store its configuration. We call it a values repo.
In order to quickly get up and running it is advised to download the chart values and fill in the values.
The following commands bootstrap the values repo:
# map the chart values.yaml as initial input:
export VALUES_INPUT=$PWD/values.yaml
# point to a folder that is or will become the values repo
export ENV_DIR=$PWD/otomi-values
# and bootstrap all the files in there
otomi bootstrap
2. Customize configuration
The essential otomi platform configurations is stored in env/cluster.yaml
, env/settings.yaml
and env/secrets.settings.yaml
files. Inspect them and customize values to match your environment.
The minimum required input is found in the chart values.
3. Configure credentials from a KMS (optional)
If you don't need encryption straight away please continue to the next step
APL will encrypt any secrets.*.yaml
files with sops, but only if it finds sops:
configuration details. In order to have access to the KMS credentials to encrypt/decrypt, a .secrets
file needs to exist and have those credentials. Please copy .secrets.sample .secrets
and fill it in with the KMS credentials.
Then you can run otomi bootstrap
, which will result in the encryption and decryption of the secrets files.
Note to developers:
To allow git diff
to show unencrypted values, you must register the sops diffing routine once with git. To register it:
git config diff.sopsdiffer.textconv 'sops -d'
This only registers the sops differ, which is responsible for invoking sops. But sops still needs the credentials to the KMS service. Again, your AWS profile is always pointed and loaded, but in case of Google KMS you will need to point GOOGLE_APPLICATION_CREDENTIALS to the gcp-key.json
file holding your account information:
export GOOGLE_APPLICATION_CREDENTIALS=$PWD/gcp-key.json
Now try a diff:
git diff
4. Start APL Console on your local machine (optional)
Bootstrap again and start the console:
otomi bootstrap
otomi console
The console allows for easy configuration of many settings but not all. Assuming the setup steps are completed, you need to now configure the APL values repository. This repo is the source configuration for APL.
5. Configuration
Configuration can be performed much easier through the APL Console, so please refer to the APL Console documentation.
However, chart configuration is not (yet) exposed through the console, so please look at the values repo's env/charts/*
files to edit the configuration files.
Important things to note:
- Every configuration file can have a
secrets.*.yaml
counterpart, but these are optional. - A json schema and vscode settings are imported by the bootrap (in
.vscode/*
), so you will have automatic linting and hinting for the configuration when vscode is used (try CTRL+SPACE in the yaml). - If
.secrets
is correctly configured then automatic de-/en-cryption will also be performed when in vscode and editing asecrets.*.yaml
file.
Please follow the guidance of the yaml hinting, as it has all the descriptions and example values you need to operate on these files.
VSCode automatically loads the '.vscode/values-schema.yaml' schema provided. Please inspect it or wire it up manually when using another editor.
If you wish to be sure of your changes, you can always do a git diff
. When you chose to use encryption and have correctly followed the corresponding instructions, then you should see a diff with the unencrypted values. That is, if you modified any ;)
6. Validation
When you are done with the configuration you can validate the results:
otomi validate-values
If you have made an error in the format of the values this will be reported.
To check if all the output manifests are valid for the target cluster's k8s version, and following best practices you can run another variation:
otomi validate-templates
7. Deployment
Charted vs uncharted resources
The output manifests generated by otomi are deployed in two ways:
- Uncharted: some base manifests are applied directly with
kubectl apply
- Charted: manfests that are packaged up in helm charts.
Ideally, we would like to deploy as helm chart as it has many benefits such as rollback. But in some cases we can't or we don't wish to. The reasons for that are the following:
- Some resources we don't want governed by charts (as charts might get accidentally removed, erasing everything that was deployed with it).
- Some existing resources have to be patched (like pull secrets in service accounts), which helmfile won't do as it will not modify existing resources not annotated to be under control by a chart.
- Some resources need to exist before the charts are deployed (such as CRDs).
The manifests that are currently not charted are:
k8s/base
(unparameterized, mostly rbac roles)values/cloud
(applies cloud specific "normalization" patterns, such as for storageclasses)values/k8s
(team resources, such as namespaces, service accounts, pull secrets)
Working with uncharted resources
Currently we don't have any subcommand that only works on uncharted resources, but we have the following commands that target the entire bundle.
otomi test
: does a dry run, showing all manifests that will be deployed, and will also show any errors in the output manifests.otomi apply
: deploys all the manifests (uncharted first, then charted)
So after doing otomi test
, if all looks ok, go ahead and do the initial deployment of all resources:
otomi apply
This command executes two stages (please see binzx/apply.ts
). The first stage will deploy all uncharted resources with kubectl apply
, and the second stage will deploy all the charted resources with helmfile apply
.
Whenever you add a team, or change or add to these uncharted resources, you have to run otomi apply
to apply them. When you let Drone do the syncing for you, it will invoke that command to synchronize the cluster.
Working with charted resources
During development iterations you will probably not touch uncharted resources often, but instead you will add features in charts.
APL has these subcommands that only target charted resources:
otomi (diff|apply|sync|template)
You can always target a single chart like this:
otomi (diff|apply|sync|template) -l name=prometheus-operator
(For a list of all supported flags to use those subcommands, we defer to the helmfile documentation, as those are deferred to the helmfile cli.)
Let's do a diff of all the charts that are enabled:
otomi diff
Need to know quirks
Whenever you modify resources without using helm, its internal bookkeeping (the versioned secrets in the namespaces) will not change, and any subsequent otomi apply
commands will not modify anything. If you notice this, and want to overwrite with the output manifests, you can use otomi sync
, which will skip doing a diff, and instead apply all charted manifests as a new version.
8. Committing values
To commit values and run post processing tasks:
otomi commit
This will detect any version changes and then commit all files with a standardized message "Manual commit". (We believe all values repo configuration changes are equally meaningful and don't need explicit commit messages.) Directly doing a git commit
is discouraged with a git hook saying so, but whenever you did not touch any versions in env/clusters.yaml
you may bypass with git commit -m "Manual commit" --no-verify
to save development time.