Skip to main content

Using the CLI

1. Initialize a values repo​

APL needs a git repo to store its configuration. We call it a values repo.

In order to quickly get up and running it is advised to download the chart values and fill in the values.

The following commands bootstrap the values repo:

# map the chart values.yaml as initial input:
export VALUES_INPUT=$PWD/values.yaml
# point to a folder that is or will become the values repo
export ENV_DIR=$PWD/otomi-values
# and bootstrap all the files in there
otomi bootstrap

2. Customize configuration​

The essential otomi platform configurations is stored in env/cluster.yaml, env/settings.yaml and env/secrets.settings.yaml files. Inspect them and customize values to match your environment.

note

The minimum required input is found in the chart values.

3. Configure credentials from a KMS/Age (optional)​

No encryption needed?

If you don't need encryption straight away please continue to the next step

APL will encrypt any secrets.*.yaml files with sops, but only if it finds sops: configuration details. In order to have access to the KMS/Age credentials to encrypt/decrypt, a .secrets file needs to exist and have those credentials. Please copy .secrets.sample .secrets and fill it in with the KMS credentials.

Then you can run otomi bootstrap, which will result in the encryption and decryption of the secrets files.

Note to developers:

To allow git diff to show unencrypted values, you must register the sops diffing routine once with git. To register it:

git config diff.sopsdiffer.textconv 'sops -d'

This only registers the sops differ, which is responsible for invoking sops. But sops still needs the credentials to the KMS service. Again, your AWS profile is always pointed and loaded, but in case of Google KMS you will need to point GOOGLE_APPLICATION_CREDENTIALS to the gcp-key.json file holding your account information:

export GOOGLE_APPLICATION_CREDENTIALS=$PWD/gcp-key.json

Now try a diff:

git diff

4. Start APL Console on your local machine (optional)​

Bootstrap again and start the console:

otomi bootstrap
otomi console

The console allows for easy configuration of many settings but not all. Assuming the setup steps are completed, you need to now configure the APL values repository. This repo is the source configuration for APL.

5. Configuration​

Configuration can be performed much easier through the APL Console, so please refer to the APL Console documentation.

However, chart configuration is not (yet) exposed through the console, so please look at the values repo's env/charts/* files to edit the configuration files.

Important things to note:

  • Every configuration file can have a secrets.*.yaml counterpart, but these are optional.
  • A json schema and vscode settings are imported by the bootrap (in .vscode/*), so you will have automatic linting and hinting for the configuration when vscode is used (try CTRL+SPACE in the yaml).
  • If .secrets is correctly configured then automatic de-/en-cryption will also be performed when in vscode and editing a secrets.*.yaml file.

Please follow the guidance of the yaml hinting, as it has all the descriptions and example values you need to operate on these files.

APL YAML hinting only works in vscode

VSCode automatically loads the '.vscode/values-schema.yaml' schema provided. Please inspect it or wire it up manually when using another editor.

If you wish to be sure of your changes, you can always do a git diff. When you chose to use encryption and have correctly followed the corresponding instructions, then you should see a diff with the unencrypted values. That is, if you modified any ;)

6. Validation​

When you are done with the configuration you can validate the results:

otomi validate-values

If you have made an error in the format of the values this will be reported.

To check if all the output manifests are valid for the target cluster's k8s version, and following best practices you can run another variation:

otomi validate-templates

7. Deployment​

Charted vs uncharted resources​

The output manifests generated by otomi are deployed in two ways:

  • Uncharted: some base manifests are applied directly with kubectl apply
  • Charted: manfests that are packaged up in helm charts.

Ideally, we would like to deploy as helm chart as it has many benefits such as rollback. But in some cases we can't or we don't wish to. The reasons for that are the following:

  1. Some resources we don't want governed by charts (as charts might get accidentally removed, erasing everything that was deployed with it).
  2. Some existing resources have to be patched (like pull secrets in service accounts), which helmfile won't do as it will not modify existing resources not annotated to be under control by a chart.
  3. Some resources need to exist before the charts are deployed (such as CRDs).

The manifests that are currently not charted are:

  • k8s/base (unparameterized, mostly rbac roles)
  • values/cloud (applies cloud specific "normalization" patterns, such as for storageclasses)
  • values/k8s (team resources, such as namespaces, service accounts, pull secrets)

Working with uncharted resources​

Currently we don't have any subcommand that only works on uncharted resources, but we have the following commands that target the entire bundle.

  • otomi test: does a dry run, showing all manifests that will be deployed, and will also show any errors in the output manifests.
  • otomi apply: deploys all the manifests (uncharted first, then charted)

So after doing otomi test, if all looks ok, go ahead and do the initial deployment of all resources:

otomi apply

This command executes two stages (please see binzx/apply.ts). The first stage will deploy all uncharted resources with kubectl apply, and the second stage will deploy all the charted resources with helmfile apply.

Whenever you add a team, or change or add to these uncharted resources, you have to run otomi apply to apply them. When you let Drone do the syncing for you, it will invoke that command to synchronize the cluster.

Working with charted resources​

During development iterations you will probably not touch uncharted resources often, but instead you will add features in charts.

APL has these subcommands that only target charted resources:

otomi (diff|apply|sync|template)

You can always target a single chart like this:

otomi (diff|apply|sync|template) -l name=prometheus-operator

(For a list of all supported flags to use those subcommands, we defer to the helmfile documentation, as those are deferred to the helmfile cli.)

Let's do a diff of all the charts that are enabled:

otomi diff

Need to know quirks​

Whenever you modify resources without using helm, its internal bookkeeping (the versioned secrets in the namespaces) will not change, and any subsequent otomi apply commands will not modify anything. If you notice this, and want to overwrite with the output manifests, you can use otomi sync, which will skip doing a diff, and instead apply all charted manifests as a new version.

8. Committing values​

To commit values and run post processing tasks:

otomi commit

This will detect any version changes and then commit all files with a standardized message "Manual commit". (We believe all values repo configuration changes are equally meaningful and don't need explicit commit messages.) Directly doing a git commit is discouraged with a git hook saying so, but whenever you did not touch any versions in env/clusters.yaml you may bypass with git commit -m "Manual commit" --no-verify to save development time.