TLDR: how can I dig deeper into my Mac mini OSX server to determine what processes are consuming so much bandwidth, or where all the inbound traffic is coming from?
Resource caching is on, all resources are minified or images crushed, our pages are less bandwidth consuming then the website last month (www.vulytrampolines.com), and our traffic in analytics is the same.
Back story: We have 2x mac mini servers running our website/staging/databases etc. Since moving from a dedicated server in America to a colocation place in the city, our bandwidth consumption has somehow quadrupled. There is a database replication process setup between the two, as well as DNS and various websites (e.g. large files, database tasks, intranet packages etc are on staging server, website and database are on production server)
Our staging server has had 7GB of inbound traffic in 3 DAYS. Does anyone know how to check for inbound traffic sources to see potentially where this consistent stream of inbound traffic at a 200k connection is coming from? We have no idea. We aren't sending it files at all, the only thing that should be active is SSH and a database replication process. netstat [see below] shows we have about 20+ established and 30+ close_weight connections to port 625. We haven't the foggiest how this could be happening.
The annoying thing is, webstats show we haven't used anywhere near 11.66GB for HTTP traffic (it says we used 22GB in the last month, yet our outbound traffic was recorded as over 100GB). Database stats say we haven't used anywhere near enough bandwidth to cause the issue either.
This is our staging server, venus1 (it has been like this for a few weeks):
This is our production server, venus2:
netstat -anp tcp output on venus1. Most of the foreign addresses are from our work IP addresses. 11211 is memcached.
tcp4 0 0 122.99.117.18.49712 204.93.223.143.80 ESTABLISHED
tcp4 0 0 122.99.117.18.11211 122.99.117.18.49711 ESTABLISHED
tcp4 0 0 122.99.117.18.49711 122.99.117.18.11211 ESTABLISHED
tcp4 0 52 122.99.117.18.22 59.167.152.67.56106 ESTABLISHED
tcp4 0 0 122.99.117.18.11211 122.99.117.18.49588 ESTABLISHED
tcp4 0 0 122.99.117.18.49588 122.99.117.18.11211 ESTABLISHED
tcp4 0 0 122.99.117.18.49410 122.99.117.19.3306 ESTABLISHED
tcp4 0 0 122.99.117.18.5432 122.99.117.18.58704 ESTABLISHED
tcp4 0 0 122.99.117.18.58704 122.99.117.18.5432 ESTABLISHED
tcp4 0 0 127.0.0.1.58699 *.* LISTEN
tcp4 0 0 122.99.117.18.625 110.142.234.238.62055 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.71.206.61838 ESTABLISHED
tcp4 5 0 122.99.117.18.625 203.206.171.34.61741 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.61270 ESTABLISHED
tcp4 0 0 127.0.0.1.54 *.* LISTEN
tcp4 0 0 122.99.117.18.53 *.* LISTEN
tcp4 0 0 127.0.0.1.53 *.* LISTEN
tcp4 0 0 122.99.117.18.625 110.142.234.238.63980 ESTABLISHED
tcp4 0 0 122.99.117.18.625 203.206.171.34.55282 ESTABLISHED
tcp46 0 0 *.80 *.* LISTEN
tcp4 0 0 *.* *.* CLOSED
tcp46 0 0 *.443 *.* LISTEN
tcp4 0 0 *.* *.* CLOSED
tcp4 0 0 122.99.117.18.625 122.99.117.19.50766 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 122.99.117.19.63981 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.60214 ESTABLISHED
tcp4 0 0 122.99.117.18.625 27.33.32.204.65196 ESTABLISHED
tcp4 0 0 122.99.117.18.625 110.142.234.238.60274 ESTABLISHED
tcp4 0 0 122.99.117.18.625 122.99.117.19.53201 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 203.206.171.34.59662 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 122.99.117.19.49869 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 122.99.117.19.53827 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.64678 ESTABLISHED
tcp4 0 0 122.99.117.18.625 122.99.117.19.52810 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.62510 ESTABLISHED
tcp4 0 0 122.99.117.18.625 122.99.117.19.49909 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.56096 ESTABLISHED
tcp4 0 0 122.99.117.18.625 203.206.171.34.53247 ESTABLISHED
tcp4 0 0 122.99.117.18.625 203.206.171.34.62051 ESTABLISHED
tcp4 0 0 122.99.117.18.625 58.111.93.92.59123 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 122.99.117.19.57173 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 203.206.171.34.49352 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.64362 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.59772 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.59528 CLOSE_WAIT
tcp4 0 0 *.3306 *.* LISTEN
tcp4 0 0 122.99.117.18.625 27.33.32.204.56812 ESTABLISHED
tcp4 0 0 122.99.117.18.625 110.142.234.238.52987 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.50598 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.63339 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.63283 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.61312 ESTABLISHED
tcp4 0 0 122.99.117.18.625 110.142.234.238.52045 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 122.99.117.19.49172 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 203.206.171.34.50501 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.93.92.56042 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.55882 ESTABLISHED
tcp4 0 0 122.99.117.18.311 58.111.93.92.55875 ESTABLISHED
tcp4 0 0 122.99.117.18.625 203.206.171.34.58776 ESTABLISHED
tcp6 0 0 *.5432 *.* LISTEN
tcp4 0 0 *.5432 *.* LISTEN
tcp4 0 0 122.99.117.18.625 58.111.93.92.52692 ESTABLISHED
tcp4 0 0 122.99.117.18.625 203.206.171.34.57121 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.54673 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.53915 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.52109 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.51807 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.65049 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.93.92.64442 ESTABLISHED
tcp4 0 0 122.99.117.18.311 203.206.171.34.51628 ESTABLISHED
tcp4 0 0 122.99.117.18.625 203.206.171.34.51594 ESTABLISHED
tcp4 0 0 122.99.117.18.625 58.111.79.42.62597 ESTABLISHED
tcp4 0 0 122.99.117.18.625 58.111.79.42.62454 ESTABLISHED
tcp4 0 0 122.99.117.18.625 58.111.79.42.58088 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.57305 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.53724 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.62224 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.62064 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.58236 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 203.206.171.34.51320 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 203.206.171.34.51297 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 203.206.171.34.50864 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.49800 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 27.33.32.204.50894 ESTABLISHED
tcp4 0 0 122.99.117.18.625 59.167.152.67.50411 ESTABLISHED
tcp4 0 0 122.99.117.18.625 27.33.32.204.54446 ESTABLISHED
tcp4 0 0 122.99.117.18.625 58.111.79.42.51680 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.60797 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.60729 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.49209 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.63371 CLOSE_WAIT
tcp4 81 0 122.99.117.18.625 113.128.44.66.3842 CLOSE_WAIT
tcp4 11 0 122.99.117.18.625 213.57.189.27.55646 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 110.142.234.238.53655 ESTABLISHED
tcp4 0 0 122.99.117.18.625 110.142.234.238.53644 ESTABLISHED
tcp4 0 0 122.99.117.18.625 58.111.79.42.52146 CLOSE_WAIT
tcp4 0 0 127.0.0.1.8005 *.* LISTEN
tcp46 0 0 *.8009 *.* LISTEN
tcp46 0 0 *.8080 *.* LISTEN
tcp4 0 0 122.99.117.18.625 58.111.79.42.50716 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.49872 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.63218 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.62471 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.64758 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.64646 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.56788 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.56770 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.56017 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.53131 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.52519 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.51215 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.51131 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.57058 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 59.167.152.67.56711 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.50975 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.57209 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.54753 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.60786 CLOSE_WAIT
tcp4 0 0 122.99.117.18.625 58.111.79.42.56174 CLOSE_WAIT
tcp4 0 0 *.11212 *.* LISTEN
tcp6 0 0 *.11212 *.* LISTEN
tcp4 0 0 127.0.0.1.5348 127.0.0.1.49167 ESTABLISHED
tcp4 0 0 127.0.0.1.49167 127.0.0.1.5348 ESTABLISHED
tcp4 0 0 122.99.117.18.5218 122.99.117.18.49166 ESTABLISHED
tcp4 0 0 122.99.117.18.49166 122.99.117.18.5218 ESTABLISHED
tcp46 0 0 *.5268 *.* LISTEN
tcp46 0 0 *.5218 *.* LISTEN
tcp4 0 0 127.0.0.1.5348 127.0.0.1.49163 ESTABLISHED
tcp4 0 0 127.0.0.1.49163 127.0.0.1.5348 ESTABLISHED
tcp4 0 0 127.0.0.1.5348 127.0.0.1.49162 ESTABLISHED
tcp4 0 0 127.0.0.1.49162 127.0.0.1.5348 ESTABLISHED
tcp4 0 0 127.0.0.1.5348 127.0.0.1.49161 ESTABLISHED
tcp4 0 0 127.0.0.1.49161 127.0.0.1.5348 ESTABLISHED
tcp4 0 0 127.0.0.1.5348 *.* LISTEN
tcp4 0 0 *.11211 *.* LISTEN
tcp6 0 0 *.11211 *.* LISTEN
tcp4 0 0 *.88 *.* LISTEN
tcp6 0 0 *.88 *.* LISTEN
tcp6 0 0 *.2000 *.* LISTEN
tcp4 0 0 *.2000 *.* LISTEN
tcp6 0 0 *.4190 *.* LISTEN
tcp4 0 0 *.4190 *.* LISTEN
tcp4 0 0 *.464 *.* LISTEN
tcp6 0 0 *.464 *.* LISTEN
tcp6 0 0 *.25 *.* LISTEN
tcp4 0 0 *.25 *.* LISTEN
tcp4 0 0 *.749 *.* LISTEN
tcp6 0 0 *.749 *.* LISTEN
tcp4 0 0 *.22 *.* LISTEN
tcp6 0 0 *.22 *.* LISTEN
tcp4 0 0 *.5900 *.* LISTEN
tcp6 0 0 *.5900 *.* LISTEN
tcp4 0 0 *.625 *.* LISTEN
tcp6 0 0 *.625 *.* LISTEN
tcp4 0 0 127.0.0.1.631 *.* LISTEN
tcp6 0 0 ::1.631 *.* LISTEN
tcp4 0 0 *.311 *.* LISTEN
Best Answer
Unless you have a very good reason not to, please firewall all non-public ports on your Mac Minis immediately and only expose what you need to! I did a quick check and all of your services currently appear to be completely open to the internet (SSH, SMTP, Tomcat, VNC, MySQL and apple-xsrvr-admin (TCP625) certainly are), which is a very bad idea.
The large amount of inbound traffic you're seeing may quite possibly be hackers/bots trying to brute-force logins and passwords for your services – I see this on servers a lot, particularly against SSH (as it's often public-facing) and directly against popular web software, such as WordPress's /wp-login.php page. From your netstat I can see that IPs from Israel and China were trying to access TCP625 (apple-xsrvr-admin, used for DirectoryService, Open Directory Assistant & Workgroup Manager), which isn't a good sign. I hope all of your usernames and passwords are strong, because – not trying to sound alarmist here, but – 7GB's worth of brute-forcing may have let some bad people get to things they shouldn't have access to, regardless of the security/patch state of any software.
Check with your hosting provider/colo that you have some sort of remote serial console and/or VNC access should you accidentally block yourself from accessing your servers, then add the absolute minimum required for you to remote admin the servers yourself to the rulesets and turn the firewalls on, which I'm assuming they're not already. By default, OS X Server's firewall blocks all incoming ports except those used to configure the server remotely (TCP22, TCP311, TCP626, TCP625, ICMP standard ping (in & out), UDP53 DNS name resolution) so you should be OK to turn it straight on, though you want to lock these down more once you've done so. Create Address Groups that are specific to your office IP (or your ISP's IP netblock(s) if your office doesn't have a static IP address) and use these to open admin access (e.g. SSH, OpenDirectory, VNC) to these only. Create another Group with single IP access for 122.99.117.18 and 122.99.117.19 (or 122.99.117.18/31 mask) to talk to each other and allow this for the MySQL replication. Open the SMTP, HTTP, HTTPS ports to the world, assuming they're public-facing. Lock everything down tight, granting permission only to the IPs that need access to each port. Consider doing this for outbound traffic, too. You want to plan this in advance and make sure you get it all right, but do it soon rather than leaving the server an open sitting duck.
Review your servers' logs and look for suspicious activity. In particular look for any successful logins to services from odd locations or at odd times. Establish some procedures to do this regularly.
I'm not sure which version of OS X Server you have, but guessing at 10.6 or 10.7, this Peachpit document may help get you started on the firewalling front.
As may this Apple support PDF on Networking Services (see chapter 4 for firewalling).
For more advanced firewall configuration, try Waterroof or Icefloor, which provide a simple GUI rather than having to mess with pf on the commandline.
(Edit to address the ipfilter rules pasted from each server)
OK, let's start with venus1 (122.99.117.18). The obvious problem here is that there's no catch-all
deny ip from any to any, so we need to fix that immediately. From the commandline, you can issue:sudo ipfw add 65534 deny log logamount 1000 ip from any to anyOr do it from the Server Admin's Settings tab, under the Advanced sub-tab if you're not too comfortable with manipulating ipfw rules from the commandline (which should make you nervous as it can end badly if you slip up). There should be a list of Advanced Rules somewhat similar to the following image. Assuming that rule is there already at the bottom, tick it to enable it:
Remember that the ipfw rules flow down the priority list, so a priority 1 rule takes precedence over everything else. So adding a 'deny everything' rule down at 65534 is last on the list, meaning it'll only deny a connection if everything above it doesn't allow something first.
OK, with that out of the way, let's deal with some Address Groups. You've defined an Address Group that's 122.99.117.16/29, so 122.99.177.16-22 (where one IP's the gateway), which is your Mac Minis and other co-lo IP addresses. You probably want to define another Address Group for your office location, which I'm guessing at a static IP of 59.167.152.67 (or if your work gets a dynamic IP then the whole netblock of Internode is 59.167.0.0/16). Do this for both venus1 & venus2 and add more for any other offices you have.
Once you've definied your office Groups then give each of these permission to access TCP22, TCP311, TCP626, TCP625, TCP3283 and TCP5900, so you can always admin your Minis even when you disable world-access to these ports. Using the GUI, go to the Services sub-tab of the Server Admin's Settings. Change the "Editing Services for:" to your office Address Group(s), then tick the relevant boxes for the above ports.
Now let's address some more problematic venus1 rules. For instance:
Rule 12321 allows everything to connect to MySQL. Disable this rule for the "any" Address Group (using the Services sub-tab again) and while you're there, any other services that the world doesn't need access to. You probably want to keep TCP25 (smtp mail), TCP/UDP53 (dns), TCP80 (HTTP), TCP443 (HTTPS) world-accessible, but do you need world-access to TCP8080 (tomcat's server status)? Anything else? Lock it all down and only grant access to your office Address Group(s) and co-lo where appropriate (e.g. the MySQL replication).
Remember to test everything! Learn to use nmap to scan your server ports so you can see what's open and closed. Scan from different locations (e.g. your office, ssh'ed into your other server, at home through another ISP, etc.) to make sure the rules are working as they should. Be careful not to lock yourself out, but do lock everything down as tight as you can. Re-read the Peachpit doc and Apple support PDF linked above and really learn the material, it'll stand you in good stead. Oh, and keep a regular eye on your logs!
OK, hopefully that should see you right, or at least put you on a better path. Best of luck.