If there is some work-flow, configuration or tweak to a system to be made you best believe it’s going into a playbook because otherwise I will forget. Becoming a yaml-farmer is not what I envisioned for myself but here we are. Almost every time I use Ansible to implement something I learn some new trick, which can be annoying or cool depending on my mood. Tools from Redhat tend to be pretty bleeding-edge so this is no exception. When I first started writing simple playbooks ansible-galaxy had just become a thing, but now there is so much more, sheesh.
Here’s my setup, which you can easily repurpose for your own projects.
ansible-navigator + ansible-builder
It feels a little silly that now there are even more cli tools to juggle, instead of the usual ansible-playbook task.yml. Now we have… ’execution environments’? I LOVE learning new jargon, I LOVE having to memorize words for things that already exist.
Anyway, an execution environment is just a container image which comes with all the ansible-core and python dependencies. Much in the same way that it doesn’t really make sense to run builds or batch jobs on your own development laptop, ansible-builder allows you to create an execution environment for a control node on a remote server with all the ansible-galaxy collections and python dependencies setup and ready to go. Share with other yaml-farmers on your team as you please. Python dependencies for me have always been pure pain to configure so I am willing to take the assist.
ansible-navigator is the UI that wraps everything together and becomes the new entrypoint for executing commands against the execution environment. For example:
ansible-vault encrypt group_vars/all/vault_passwords.yml
becomes
ansible-navigator exec -- ansible-vault encrypt group_vars/all/vault_passwords.yml
or alternatively
ansible-navigator exec "ansible-vault encrypt group_vars/all/vault_passwords.yml"
and
ansible-playbook website-deploy.yml
becomes
ansible-navigator run website-deploy.yml
How delightfully madenning it is having to memorize even more wrappers and abstractions. Furthermore, one annoying detail I discovered is that you have to escape whitespace if you’re running ad-hoc commands.
So
ansible-navigator exec -- ansible gwt -m file -a "path=/path/to/some/file.txt state=absent"
will error out but
ansible-navigator exec -- ansible gwt -m file -a "path=/path/to/some/file.txt\ state=absent"
succeeds. Most uncool.
Configuring the configurators
You can get a good overview of what exactly navigator is capable of doing by checking out its interactive mode ansible-navigator --mode interactive via the :settings.
Name Default Source Current
0│Ansible runner artifact dir False Settings file /run/user/1000
1│Ansible runner rotate artifacts count False Settings file 10
2│Ansible runner timeout True Not set Not set
3│Ansible runner write job events True Defaults False
4│App False Command line collections
5│Cmdline True Not set Not set
6│Collection doc cache path True Defaults /home/programmist/.cache/ansible-navigator/collection_doc_cache.db
7│Config True Not set Not set
8│Container engine False Settings file podman
9│Container options False Settings file ['--user=0', '--user=root']
10│Current settings file False Search path /home/programmist/.ansible-navigator.yml
11│Display color True Defaults True
12│Editor command True Defaults nvim {filename}
13│Editor console True Defaults True
14│Enable prompts True Defaults False
15│Exec command True Defaults /bin/bash
16│Exec shell True Defaults True
17│Execution environment True Settings file True
18│Execution environment image False Settings file custom-ee:latest
19│Execution environment volume mounts True Not set Not set
20│Format True Settings file yaml
21│Help builder True Defaults False
22│Help config True Defaults False
23│Help doc True Defaults False
24│Help inventory True Defaults False
25│Help playbook True Defaults False
26│Images details True Defaults ['everything']
27│Inventory False Ansible configuration file ['/home/programmist/projects/mine/generic-wizard-tower/hosts.yml']
28│Inventory column True Not set Not set
29│Lint config True Not set Not set
30│Lintables True Not set Not set
31│Log append True Defaults True
32│Log file False Settings file /home/programmist/projects/mine/ansible-navigator.log
33│Log level False Settings file critical
34│Mode True Command line interactive
35│Osc4 True Defaults True
36│Pass environment variable False Settings file ['ANSIBLE_VAULT_PASSWORD_FILE']
37│Playbook True Not set Not set
38│Playbook artifact enable True Settings file True
39│Playbook artifact replay False Settings file /tmp/test_artifact.json
40│Playbook artifact save as False Settings file /tmp/test_artifact.json
41│Plugin name True Not set Not set
42│Plugin type True Defaults module
43│Pull arguments True Not set Not set
44│Pull policy False Settings file missing
45│Set environment variable False Settings file {'ANSIBLE_STDOUT_CALLBACK': 'yaml'}
46│Settings effective True Defaults False
47│Settings sample True Defaults False
48│Settings schema True Defaults json
49│Settings sources True Defaults False
50│Time zone True Defaults UTC
51│Workdir True Defaults /home/programmist/projects/mine/generic-wizard-tower
As you can see its a mix of config values from differents sources; environment variables, defaults and files. Note the execution environment image, which is created with ansible-builder build -t custom-ee:latest --prune-images -v3. The config for the image references the requirements.yml and requirements.txt for any collections from galaxy you might want to use in your playbooks, such as Podman modules:
~/projects/mine/gwt/execution-environment.yml
version: 3
images:
base_image:
name: ghcr.io/ansible/community-ansible-dev-tools:latest
dependencies:
ansible_core:
package_pip: ansible-core
ansible_runner:
package_pip: ansible-runner
galaxy: requirements.yml
python: requirements.txt
system:
- openssh-clients
- sshpass
options:
package_manager_path: /usr/bin/dnf5
Of course, its own config is also found at ~/.ansible-navigator.yml. Other important things you’ll probably want to set are your vault password file, log levels and log file location. I like to see prettier yaml so I use the stdout callback function.
---
ansible-navigator:
ansible-runner:
artifact-dir: /run/user/1000
rotate-artifacts-count: 10
execution-environment:
container-engine: podman
container-options: ["--user=0"]
enabled: true
environment-variables:
pass:
- ANSIBLE_VAULT_PASSWORD_FILE
set:
ANSIBLE_STDOUT_CALLBACK: yaml
image: custom-ee:latest
pull:
policy: missing
logging:
file: /home/programmist/projects/mine/ansible-navigator.log
level: critical
playbook-artifact:
enable: True
replay: /tmp/test_artifact.json
save-as: /tmp/test_artifact.json
mode: stdout
format: yaml
Project directory structure
There are a multitude of ways to structure an Ansible project but here is what works for me. My extremely generalized, happy-path organization is as follows.
.
├── files
│ ├── grafana
│ │ ├── dashboards
│ │ │ ├── dashboard.yml
│ │ │ ├── victoriametrics.json
│ │ │ ├── vmagent.json
│ │ │ └── vmalert.json
│ │ └── prometheus-datasource
│ │ └── single.yml
│ ├── hugo
│ │ ├── rpavlov.com
│ │ └── othersites.com
│ ├── playlists
│ ├── samba
│ ├── vm-agent
│ ├── vm-alert
│ │ └── rules
│ ├── vm-alertmanager
│ └── zsh
├── group_vars
│ ├── all
│ │ ├── common.yml
│ │ └── vault_passwords.yml
│ └── bunker
│ ├── authelia.yml
│ └── htpasswd.yml
├── host_vars
│ ├── arcane_sanctum.yml
│ ├── gwt.yml
│ ├── secret_hetzner_tunnel.yml
│ └── secret_oracle_tunnel.yml
├── roles
│ ├── dns
│ │ └── tasks
│ ├── podman_container
│ │ ├── defaults
│ │ └── tasks
│ ├── podman_pod
│ │ ├── defaults
│ │ └── tasks
│ ├── redis
│ │ ├── defaults
│ │ └── tasks
│ ├── ssh
│ │ ├── handlers
│ │ └── tasks
│ ├── victoriametrics
│ │ ├── files
│ │ ├── tasks
│ │ ├── templates
│ │ └── vars
│ └── wireguard
│ ├── handlers
│ │ └── main.yml
│ ├── meta
│ │ └── main.yml
│ ├── tasks
│ │ ├── client.yml
│ │ └── server.yml
│ └── templates
│ ├── add-nat-rules.sh.j2
│ ├── client.conf.j2
│ ├── remove-nat-rules.sh.j2
│ └── wg0.conf.j2
├─ templates
│ ├── haproxy
│ │ └── haproxy.cfg.j2
│ ├── podman
│ │ └── storage.conf.j2
│ └── traefik
│ ├── conf.d
│ │ ├── anubis.yml.j2
│ │ ├── authelia.yml.j2
│ │ ├── crowdsec.yml.j2
│ │ ├── dashboard.yml.j2
│ │ ├── jellyfin.yml.j2
│ │ ├── sws.yml.j2
│ │ ├── middlewares.yml.j2
│ │ └── ...
│ └── traefik.yaml.j2
├── traefik-config.yml
├── traefik.yml
├── jellyfin.yml
├── uptimekuma.yml
├── victoria-metrics.yml
├── vm-agent.yml
├── vm-alertmanager.yml
├── vm-alert.yml
├── website-deploy.yml
├── wg-gwt-client.yml
├── wg-hetzner-server.yml
├── wg-oracle-client.yml
└── ...
files/
Any flat config file, generated asset or image. Basically anything without injected variables, however, these folders might also reflect the logical grouping for configs for that particular service (ie rules/ for monitoring).
templates/
Config files with injected variables used directly by playbooks. Otherwise they belong in the appropriate role’s templates/ dir. The only weird, non-standard thing I do here is keep each Traefik service config separate. Its just easier to see at a glance the current state of whats deployed and how its configured instead of wading through giant blocks of label definitions.
group_vars/
My rule of thumb is to scope variables as narrowly as possible because it quickly becomes a nightmare trying to figure out precedence and where values are coming from. The only exception in this case is vault_passwords.yml where I encrypt all the secrets in one place.
Here I’ll also store variables for different services (which could be across different hosts) in separate .yml files, or variables that are relevant for an entire network or site.
host_vars/
Variables specific to each physical host or VPS.
roles/
Ideally everything should be consolidated into a role. My typical workflow usually starts with writing everything in a single playbook, then if I have the patience or free time extract and generalize as much as possible into a role. Clearly that has not fully transpired judging by the 20+ dangling playbooks for all my self-hosted stuff but then again every project is a WIP :)
