How to manage hundreds of IPMI BMCs

configuration-managementipmi

I have over 200 computers which can provide IPMI services. The servers are manufactured by several different companies (SuperMicro, Dell, etc.), and there are 6-7 BMC models from about 5 different vendors, and each model has it's own idiosyncrasies.

So far we have been configuring the BMCs by using a combination of DHCP and manually configuring each BMC. The manual configuration might be done using a bootable CD-ROM, configuration from the BIOS (If supported), from the host operating system with a utility like ipmitool, freeipmi, etc. or remotely using ipmitool if we can determine the network address of the device.

However, this manual configuration is rather tedious. In some cases we want to change a setting globally on all BMCs, which requires that an administrator run a command against dozens of boxes. Since the BMCs are provided by different vendors and each model of BMC might have it's own idiosyncrasies, the same command does not always work on all BMCs.

Are there any utilities which allow me to mass configure the BMCs on dozens of boxes? Say that I want to query a parameter on dozens of different BMCs, or change the password, disable HTTP access to the WebUI or disable the infamous cipher zero security hole.

Bonus points for any utility which would allow me to update the BMC firmware, which is necessary to mitigate several security vulnerabilities

Best Answer

I'd probably use Ansible. It's a very simple configuration management / orchestration engine that's far simpler to get started with than Puppet (Puppet used to be my go-to choice for this, but not always now, having discovered Ansible).

The benefit of Ansible here is that it communicates directly over SSH, so you'd be able to get started using just your existing SSH credentials and workflow.

If you're currently configuring your BMCs with ipmitool, you'd be able to do something like:

Define a Hosts file -- This tells Ansible which hosts are in the bmc group (in this case), and which to run stuff on.

[bmc]
192.168.1.100
192.168.1.101
192.168.1.102

And so on... You can also use hostnames in that file, as long as they're resolvable.

Then create a "playbook", which is the set of commands to run on each host in a host-group. You want to have this kind of top-down directory layout:

ansible/
   playbooks/
      bmc.yml
      roles/
        bmcconfig/
           files/
           handlers/
             main.yml
           tasks/
             main.yml
           templates/
   group_vars/
      all

A playbook has Roles, which are little sections of configuration that you can break down and reuse.

So I'd create a file called bmc.yml (All Ansible configuration is in YAML files)

---
- name: Configure BMC on the hosts
  hosts: bmc
  user: root
  roles: 
    - bmcconfig

Then inside roles/bmcconfig/tasks/main.yml you can start listing the commands that are to be run on each host, to communicate with ipmi.

---
  - name: Install ipmitool
    apt: pkg=ipmitool state=installed
  - name: Run ipmitool config
    shell: ipmitool -your -options -go -here

When you run the playbook, with ansible-playbook -i hosts bmc.yml the commands listed in tasks/main.yml for each role will be executed in top-down order on each host found in the bmc hostgroup in hosts

group_vars/all is an interesting file, it allows you to define key-value pairs of variables and values that can be used in your playbooks.

so you could define something like 

ipmitool_password: $512315Adb

in your group_vars/all and as a result, you'd be able to have something like:

shell: ipmitool -your -options -go -here --password=${ipmitool_password}

in the playbook.

You can find out way more information about how to use the "modules" - the components of Ansible that allow you to do stuff, how to write your own :D, and so on at the Ansible Documentation Pages.

Related Topic