Wifi – Who’s using our bandwidth

You can monitor network traffic using ntop tool in realtime. ntop clears its logs on restart, this is annoying a bit, but you still can run it for a period of time to classify your traffic. As I recall it can classify the traffic by client IPs, ports and protocols. If you know a port you will be able to find a related service using fuser command line tool.

Unless you have a very good reason not to, please firewall all non-public ports on your Mac Minis immediately and only expose what you need to! I did a quick check and all of your services currently appear to be completely open to the internet (SSH, SMTP, Tomcat, VNC, MySQL and apple-xsrvr-admin (TCP625) certainly are), which is a very bad idea.

The large amount of inbound traffic you're seeing may quite possibly be hackers/bots trying to brute-force logins and passwords for your services – I see this on servers a lot, particularly against SSH (as it's often public-facing) and directly against popular web software, such as WordPress's /wp-login.php page. From your netstat I can see that IPs from Israel and China were trying to access TCP625 (apple-xsrvr-admin, used for DirectoryService, Open Directory Assistant & Workgroup Manager), which isn't a good sign. I hope all of your usernames and passwords are strong, because – not trying to sound alarmist here, but – 7GB's worth of brute-forcing may have let some bad people get to things they shouldn't have access to, regardless of the security/patch state of any software.

Check with your hosting provider/colo that you have some sort of remote serial console and/or VNC access should you accidentally block yourself from accessing your servers, then add the absolute minimum required for you to remote admin the servers yourself to the rulesets and turn the firewalls on, which I'm assuming they're not already. By default, OS X Server's firewall blocks all incoming ports except those used to configure the server remotely (TCP22, TCP311, TCP626, TCP625, ICMP standard ping (in & out), UDP53 DNS name resolution) so you should be OK to turn it straight on, though you want to lock these down more once you've done so. Create Address Groups that are specific to your office IP (or your ISP's IP netblock(s) if your office doesn't have a static IP address) and use these to open admin access (e.g. SSH, OpenDirectory, VNC) to these only. Create another Group with single IP access for 122.99.117.18 and 122.99.117.19 (or 122.99.117.18/31 mask) to talk to each other and allow this for the MySQL replication. Open the SMTP, HTTP, HTTPS ports to the world, assuming they're public-facing. Lock everything down tight, granting permission only to the IPs that need access to each port. Consider doing this for outbound traffic, too. You want to plan this in advance and make sure you get it all right, but do it soon rather than leaving the server an open sitting duck.

Review your servers' logs and look for suspicious activity. In particular look for any successful logins to services from odd locations or at odd times. Establish some procedures to do this regularly.

I'm not sure which version of OS X Server you have, but guessing at 10.6 or 10.7, this Peachpit document may help get you started on the firewalling front.

As may this Apple support PDF on Networking Services (see chapter 4 for firewalling).

For more advanced firewall configuration, try Waterroof or Icefloor, which provide a simple GUI rather than having to mess with pf on the commandline.

(Edit to address the ipfilter rules pasted from each server)

OK, let's start with venus1 (122.99.117.18). The obvious problem here is that there's no catch-all deny ip from any to any, so we need to fix that immediately. From the commandline, you can issue: sudo ipfw add 65534 deny log logamount 1000 ip from any to any

Or do it from the Server Admin's Settings tab, under the Advanced sub-tab if you're not too comfortable with manipulating ipfw rules from the commandline (which should make you nervous as it can end badly if you slip up). There should be a list of Advanced Rules somewhat similar to the following image. Assuming that rule is there already at the bottom, tick it to enable it:

Remember that the ipfw rules flow down the priority list, so a priority 1 rule takes precedence over everything else. So adding a 'deny everything' rule down at 65534 is last on the list, meaning it'll only deny a connection if everything above it doesn't allow something first.

OK, with that out of the way, let's deal with some Address Groups. You've defined an Address Group that's 122.99.117.16/29, so 122.99.177.16-22 (where one IP's the gateway), which is your Mac Minis and other co-lo IP addresses. You probably want to define another Address Group for your office location, which I'm guessing at a static IP of 59.167.152.67 (or if your work gets a dynamic IP then the whole netblock of Internode is 59.167.0.0/16). Do this for both venus1 & venus2 and add more for any other offices you have.

Once you've definied your office Groups then give each of these permission to access TCP22, TCP311, TCP626, TCP625, TCP3283 and TCP5900, so you can always admin your Minis even when you disable world-access to these ports. Using the GUI, go to the Services sub-tab of the Server Admin's Settings. Change the "Editing Services for:" to your office Address Group(s), then tick the relevant boxes for the above ports.

Now let's address some more problematic venus1 rules. For instance:

12321    1570      78548 allow tcp from any to any dst-port 3306

Rule 12321 allows everything to connect to MySQL. Disable this rule for the "any" Address Group (using the Services sub-tab again) and while you're there, any other services that the world doesn't need access to. You probably want to keep TCP25 (smtp mail), TCP/UDP53 (dns), TCP80 (HTTP), TCP443 (HTTPS) world-accessible, but do you need world-access to TCP8080 (tomcat's server status)? Anything else? Lock it all down and only grant access to your office Address Group(s) and co-lo where appropriate (e.g. the MySQL replication).

Remember to test everything! Learn to use nmap to scan your server ports so you can see what's open and closed. Scan from different locations (e.g. your office, ssh'ed into your other server, at home through another ISP, etc.) to make sure the rules are working as they should. Be careful not to lock yourself out, but do lock everything down as tight as you can. Re-read the Peachpit doc and Apple support PDF linked above and really learn the material, it'll stand you in good stead. Oh, and keep a regular eye on your logs!

OK, hopefully that should see you right, or at least put you on a better path. Best of luck.