Linux – How to configure Munin to give me a “total” view of all the cloud servers

cloudlinuxmonitoringmuninnetworking

I have a dozen load balanced cloud servers all monitored by Munin.

I can track each one individually just fine. But I'm wondering if I can somehow bundle them up to see just how much collective CPU usage (for example) there is among the cloud cluster as a whole.

How can I do this?

The munin.conf file makes it easy enough to handle this for subdomains, but I'm not sure how to configure this for simple web nodes. Assume my web nodes are named, web_node_1 – web_node_10.

My conf looks something like this right now:

[web_node_1]
    address 10.1.1.1
    use_node_name yes
...
[web_node_10]
    address 10.1.1.10
    use_node_name yes

Your help is much appreciated.

Best Answer

Yes, Munin supports virtual nodes which can combine data from other defined nodes.

See here for a good example: ~~http://waste.mandragor.org/munin_tutorial/munin.html#sev_nodes_single_graph~~

update: Actually that was a terrible example, considering it didn't work. Here's the official example, which ought to work a lot better. :-) http://munin-monitoring.org/wiki/aggregate_examples

I guess that's what I get for firing off a poorly researched answer that kinda looked right late on a Friday afternoon. :-o

Related Solutions

Is it possible to match an internal IP address to a switch port

Given an IP address, you should be able to find the MAC address of the corresponding host.

arp -a

On both Windows and Linux will show you the arp cache of that host, mapping IPs to MAC addresses. (Note that this will need to be run on a machine that is on the same IP subnet as the machine you are trying to find).

Once you have the MAC address, log on to the switch you suspect the rogue host is connected to, and search the MAC address table for that address. (The MAC address table is also called the bridging table, or the CAM table).

For example, on Cisco IOS based switches, the following command:

show mac-address-table address <MAC address>

Will show you the port that a given MAC address was last seen on. If the resulting port is a link to another switch, log on to that switch and run the command again. Repeat until you end up with a host port, and you should have your culprit.

Note that this approach will only work if you have a managed switch that allows its MAC address table to be queried. Failing that, it's going to be a case of manual elimination; find each port that you know isn't the rogue machine, until you're left with one port your can't account for. Good luck.

How to get a total bandwidth graph in munin

Yes, it is done on the server side by creating a combined graph.

Basically it is done by describing a new graph in munin server configuration (munin.conf) where the data graphed is the sum of two data sources :

[some.machine.boo]
    total_http_bw.graph_order total_received
    total_http_bw.graph_title HTTP traffic recived by machine1 and machine2
    total_http_bw.graph_vlabel Kb/s
    total_http_bw.total_received.label HTTP received traffic
    total_http_bw.total_received.sum \
            machine1.machine.boo:http_bw:received \
            machine2.machine.boo:http_bw:received

You can find an example of stacking (graphing two data sources one on the top of the other) on munin web site. If you want to graph the sum of two data sources you can use the total_received.sum function instead of total_received.stack.

[edit] ive added the stanzas required to have the net interface bandwidth to this answer, as it makes for a more complete answer to the original question (and I can't select 2 correct answers etc)

I had a lot of trouble getting the net interfaces to give up their values as a SUM, so here is the end result of lots of random trial and error changes to get a SUM line chart of multiple munin nodes

    [mydomain.com;aggregates]
    total_bandwidth.graph_args --base 1000 -l 0
    total_bandwidth.cdef 0
    total_bandwidth.graph_category Network
    total_bandwidth.graph_title Aggregated bandwidth
    total_bandwidth.graph_vlabel Bits/sec
    total_bandwidth.upload.label upload
    total_bandwidth.total.graph yes
    total_bandwidth.upload.sum \
    mybox1.mydomain.com:if_eth0.up \
    mybox2.mydomain.com:if_eth0.up
    total_bandwidth.upload.type COUNTER
    total_bandwidth.download.type COUNTER
    total_bandwidth.download.label download
    total_bandwidth.graph_order upload download
    total_bandwidth.total.graph no
    total_bandwidth.download.sum \
    mybox1.mydomain.com:if_eth0.down \
    mybox2.mydomain.com:if_eth0.down

I suspect a few of those directives are either unnecessary or plain wrong, but I hope it is a working starting point for anyone trying to achieve the same.

I also suspect that it works better if you explicitly make sure that all the hosts are in the aggregates group, eg mydomain.com even if your hosts are incorrectly named locally eg hostname -f is wrong

Best Answer

Related Solutions

Is it possible to match an internal IP address to a switch port

How to get a total bandwidth graph in munin

Related Topic