Using multiple var files for icinga2_objects

[Wade9320]

Hi,

We are currently in the processing of migrating to use this collection to manage our icinga set up.

I want to know if its possible manage the icinga2_objects in separate var_files. We want to manage it in separate var files so teams how their own var file to put their hosts in.

This what our playbook looks like:

---
- name: Build Icinga Image
  hosts: all
  become: yes
  vars_files:
    - vars/mariadb.yml
    - vars/icinga-check-commands.yml
    - vars/icinga-repos.yml
    - vars/icinga.yml
    - vars/icingadb.yml
    - vars/icingaweb.yml
    - vars/icinga-config/main.yml
    - vars/icinga-config/team1.yml
  collections:
    - icinga.icinga

  pre_tasks:
    - ansible.builtin.include_role:
        name: repos
    - ansible.builtin.include_role:
        name: geerlingguy.mysql        
    - ansible.builtin.include_role:
        name: icinga2        
    - ansible.builtin.include_role:
        name: icingadb        
    - ansible.builtin.include_role:
        name: monitoring_plugins

  post_tasks:
    - ansible.builtin.include_role:
        name: icingaweb2      

[mkayontour]

Hi it is possible if you use host_vars/group_vars.

just add for every host a variable file with the following information:

vim host_vars/host1

icinga2_objects:
  yourconfigmaster.fqdn:
    - name: host1.fqdn
       type: Host
       file: zones.d/main/host1.fqdn.conf
       address: 127.0.0.1

vim host_vars/host2

icinga2_objects:
  yourconfigmaster.fqdn:
    - name: host2.fqdn
       type: Host
       file: zones.d/main/host2.fqdn.conf
       address: 127.0.0.2

If you run the playbook on the host "yourconfigmaster.fqdn" the role will look for all hostvars and merge them into one list.

I'm not sure if the var_files solution is working, I'll check

[Wade9320]

Can you provide an example of what the playbook would look like @mkayontour ?

[Wade9320]

Bumping this, can anyone help?

[lucagubler]

@Wade9320 You want to manage multiple icinga2_objects on the same host in different YAML files. Ansible won't let you do that. If you have the same variable name defined multiple times, e.g. icinga2_objects, Ansible will simply override them. It doesn't merge them.

I have a similar dilemma. We are also migrating to Icinga2 using this collection, but we have several hundred endpoints and several hundred services that we monitor. Defining and managing all these objects in a single icinga2_objects list is going to be a pain.

The only solution I have right now is to define the objects in multiple files, e.g. host_vars/<host>/endpoints.yml, host_vars/<host>/services.yml and so on. In the playbook you can then use include_vars to register them in a temporary variable and use set_fact to merge them together. But I'm not 100% convinced that this is the best approach yet...

[lucagubler]

I was able to build a solution where multiple var files are possible.

First you define multiple files inside the inventory/host_vars/<icinga2-master-host> host_vars file. Either define YAML files directly (e.g. hostgroups.yml, timeperiods.yml etc.) or create folders and add YAML files to that folder. For example I created an endpoints/ folder and added multiple files to that folder (e.g. host1.example.com.yml, host2.example.com.yml etc.)

At the Playbook level I added two new tasks. The first one will use include_vars to include all the new YAML files and register them to the icinga2_vars variable. This now contains all the different Icinga2 objects. In the second task I simply append them to the icinga2_objects list.

- name: Include all Icinga2 objects from YAML files
  include_vars:
    dir: "../../inventory/host_vars/{{ inventory_hostname }}/"
  register: icinga2_vars

- name: Combine all Icinga2 objects into one list
  set_fact:
    icinga2_objects: "{{ icinga2_objects | default([]) + (item.value | default([])) }}"
  with_dict: "{{ icinga2_vars.ansible_facts }}"

There's one caveat. When you create different YAML files in the host_vars, every file has to have a different name for the list for the Icinga2 Objects. If they all have the same name, Ansible will override the lists...
I just use the filename as list name.

Here's an example for the timeperiods.yml file

timeperiods:
  - name: 24x7
    type: TimePeriod
    file: "zones.d/main/timeperiods.conf"
    ranges:
      monday: "00:00-24:00"
      tuesday: "00:00-24:00"
      wednesday: "00:00-24:00"
      thursday: "00:00-24:00"
      friday: "00:00-24:00"

And here's another example for hostgroups.yml

hostgroups:
  - name: linux-hosts
    type: HostGroup
    file: zones.d/main/hostgroups.conf
    display_name: Linux Server
    assign:
      - host.vars.os == Linux

@Wade9320 I think you could use the same approach, very team can manage their own team_xyz.yml file.

[Wade9320]

@lucagubler I'll try this out and report back. Thanks for the advice.

[Wade9320]

@lucagubler are you able to provide a more in-depth example based on my code?

We aren't currently using inventory/host_vars

Currently my playbook looks like this:

---
- name: Build Icinga Image
  hosts: all
  become: yes
  vars_files:
    - vars/mariadb.yml
    - vars/icinga-check-commands.yml
    - vars/icinga-repos.yml
    - vars/icinga.yml
    - vars/icingadb.yml
    - vars/icingaweb.yml
    - vars/icinga-config/stage.yml
  collections:
    - icinga.icinga

  pre_tasks:
    - ansible.builtin.include_role:
        name: repos
    - ansible.builtin.include_role:
        name: geerlingguy.mysql        
    - ansible.builtin.include_role:
        name: icinga2        
    - ansible.builtin.include_role:
        name: icingadb        
    - ansible.builtin.include_role:
        name: monitoring_plugins

  post_tasks:
    - ansible.builtin.include_role:
        name: icingaweb2      

And the var files look like:

---
icinga2_objects:
  internal.icinga:
    ##############
    ## Timeperiods
    - name: yum_check_period
      type: TimePeriod
      file: conf.d/timeperiods/yum_check.conf
      ranges:
        monday: "08:30-17:00"
        tuesday: "08:30-17:00"
        wednesday: "08:30-17:00"
        thursday: "08:30-17:00"
        friday: "08:30-17:00"
        saturday: "08:30-17:00"
        sunday: "08:30-17:00"

[Wade9320]

@lucagubler any update on this?

[telution]

This works for me.

tasks:
   - name: "Find all directories under {{playbbook_dir}}/host_vars/"
     command: "find {{playbook_dir}}/host_vars/ -type f"
     register: host_var_dirs
     delegate_to: localhost

   - name: Include variable files
     include_vars:
       file: "{{ item }}"
     loop: "{{ host_var_dirs.stdout_lines }}"
     register: included_vars

   - name: Combine dictionaries dynamically
     set_fact:
       combined_icinga2_objects: |
         {%- set ns = namespace(combined = []) -%}
         {%- for item in included_vars.results -%}
          {%- if item.ansible_facts.icinga2_var is defined -%} 
           {%- for k,v in item.ansible_facts.icinga2_var.items() -%}
             {%- if k not in ns.combined -%}
               {%- set ns.combined = ns.combined | combine({k: v}) -%}  {# Add the key with its value #}
             {%- else -%}
               {%- set ns.combined = ns.combined | combine({k: ns.combined[k] + v}) -%}  {# Merge lists for existing keys #}
              {%- endif -%}
           {%- endfor -%}
           {%- endif -%}
         {%- endfor -%}
         {{ ns.combined }}

   - name: Display merged objects
     debug:
       msg: "{{ combined_icinga2_objects }}"

#apiusers.yml
icinga2_var:
hostname:
- name: root
type: ApiUser
file: 'local.d/apiusers.conf'
password: 1234
permissions:
- "*"

#host.yml
icinga2_var:
hostname:
- name: root
type: Host
...

[lucagubler]

@Wade9320 Here's how we solved it. Inside of the host_vars, we have a folder icinga2. This folder contains other folders or YAML files with the icinga2_objects.

tree inventory/host_vars/icinga2-server.example.com
inventory/host_vars/icinga2-server.example.com
.
├── manual_vars.yml
├── icinga2
│   ├── commands
│   │   ├── general_commands.yml
│   │   ├── mpp_commands.yml
│   │   ├── nas_commands.yml
│   │   ├── network_commands.yml
│   │   └── vpn_commands.yml
│   ├── eod.yml
│   ├── hostgroups.yml
│   ├── services
│   │   ├── cep_services.yml
│   │   ├── general_services.yml
│   │   ├── mpp_services.yml
│   │   ├── nas_services.yml
│   │   ├── network_services.yml
│   │   ├── od_service.yml
│   │   └── vpn_services.yml
│   └── timeperiods.yml
└── vault.yml

I did this to make the config clearer. Otherwise, there would quickly be several thousand lines of config, which would make it pretty confusing. You can also create a folder for each team here. The team can then create their own icinga2 objects in this folder.

But in order for this config to load correctly, we need to add the new objects to the icinga2_object list. I do this with a new task in the playbook

# playbooks/icinga2_server.yml

- name: Install Icinga2 Main Setup
  hosts: icinga2-server.example.com
  become: true

  pre_tasks:
    - name: Include all Icinga2 objects from YAML files
      ansible.builtin.include_vars:
        dir: "../../inventory/host_vars/{{ inventory_hostname }}/icinga2"
      register: icinga2_vars
      tags:
        - icinga2

    - name: Combine all Icinga2 objects into one list
      ansible.builtin.set_fact:
        icinga2_objects: "{{ icinga2_objects | default([]) + (item.value | default([])) }}"
      with_dict: "{{ icinga2_vars.ansible_facts }}"
      tags:
        - icinga2

The first task loads all variable files from the host_vars and the second task adds all items to the icinga2_object.

There is something important to note. The host_vars files are separate lists with icinga2 objects. In order for these to all be loaded correctly, each list in each file needs a unique name. If several files have the same list name, the lists are overwritten and the icinga2 objects are removed or not written at all. To prevent this from happening, I use the file path and filename as the list name.

Here you can see two example files. The list name, e.g. eod or hostgroups must be unique. If both files had the same list name, it wouldn't work.

# eod.yml
eod:
  - name: EOD
    type: UserGroup
    display_name: EOD
    file: zones.d/main/groups.conf

  - name: admins
    type: UserGroup
    display_name: admins
    file: zones.d/main/groups.conf

# hostgroups.yml
hostgroups:
  - name: linux-hosts
    type: HostGroup
    display_name: Linux Server
    file: zones.d/main/hostgroups.conf
    assign:
      - match(Ubuntu*, host.vars.nb.platform)
  - name: mpp
    type: HostGroup
    display_name: MPPs
    file: zones.d/main/hostgroups.conf
    assign:
      - host.vars.nb.appliance_type == mpp

Based on @Wade9320's provided variables, you can try something like this

---
- name: Build Icinga Image
  hosts: all
  become: yes
  vars_files:
    - vars/mariadb.yml
    - vars/icinga-check-commands.yml
    - vars/icinga-repos.yml
    - vars/icinga.yml
    - vars/icingadb.yml
    - vars/icingaweb.yml
    - vars/icinga-config/stage.yml
  collections:
    - icinga.icinga

  pre_tasks:
    - name: Include variable files individually
      ansible.builtin.include_vars:
        file: "{{ item }}"
      with_items:
        - vars/mariadb.yml
        - vars/icinga-check-commands.yml
        - vars/icinga-repos.yml
        - vars/icinga.yml
        - vars/icingadb.yml
        - vars/icingaweb.yml
        - vars/icinga-config/stage.yml
      register: included_vars

    - name: Combine all variables into icinga2_objects list
      ansible.builtin.set_fact:
        icinga2_objects: "{{ icinga2_objects | default([]) + (item.ansible_facts | dict2items | map(attribute='value') | list | flatten) }}"
      with_items: "{{ included_vars.results }}"

    - ansible.builtin.include_role:
        name: repos
    - ansible.builtin.include_role:
        name: geerlingguy.mysql        
    - ansible.builtin.include_role:
        name: icinga2        
    - ansible.builtin.include_role:
        name: icingadb        
    - ansible.builtin.include_role:
        name: monitoring_plugins

  post_tasks:
    - ansible.builtin.include_role:
        name: icingaweb2

The disadvantage is that you have to adapt the playbook for each new file and include the new file. With my approach, this works automatically. I hope this helps. Please try it out and let me know if it works.