network-monitoring
I apologize in advance if this is out of topic.
I am currently using Icinga and Cacti to monitor machines and the network respectively. While I have no big issues with this setup, I would really like to have the option to monitor network traffic BY PORT in real time.
Is there a utility that does this? I only know of Paesler and Solar Winds, but anything non open source is out of the question for now.
Any ideas?
"During that time switches hanged 2 times and required hard resets"
I'm not trying to be elitist here, but Linksys/D-Link/Netgear isn't even mid-size grade hardware. iSCSI and Virtualization requires a very stable and quick network to perform properly.
I strongly suggest you buy better networking gear (Cisco, HP etc).
These are my general suggestions for this kind of process. I appreciate you'll have covered some of them already but its better to be told something twice than miss something important. These notes are orientated towards malware that's spreading on a LAN but could easily be scaled back to deal with more minor infections.
Make sure you have an up to date backup of every system and every bit of data on this network that the business cares about. Make sure you note that this restore media may be compromised, so that people don't try and restore from it in 3 months time while your back is turned and infect the network again. If you have a backup from before the infection happened, put this safely to one side too.
Shut down the live network, if you possibly can (you will probably need to do this as part of the cleanup process, at least). At the very least, seriously consider keeping this network, including servers, off the internet until you know what is going on - what if this worm is stealing info?
Don't get ahead of yourself. It's tempting to just say clean build everything at this point, force everyone to change passwords, etc, and call that 'good enough'. While you will probably need to do this sooner or later, it's likely to leave you with pockets of infection if you don't understand what is happening on your LAN. (If you don't want to investigate the infection further go to step 6)
Copy an infected machine to a virtual environment of some kind, isolate this virtual environment from everything else including the host machine before you boot the compromised guest.
Create another couple of clean virtual guest machines for it to infect then isolate that network and use tools like wireshark to monitor the network traffic (time to take advantage of that linux background and create another guest on this virtual LAN that can watch all this traffic without being infected by any Windows worm!) and Process Monitor to monitor changes happening on all these machines. Also consider that the issue may be a well hidden rootkit - try using a reputable tool for finding these but remember that this is a bit of an uphill struggle so finding nothing doesn't mean there is nothing there.
(Assuming you haven't / can't shut down the main LAN) Use wireshark on the main LAN to look at traffic being sent to/from the infected machines. Treat any unexplainable traffic from any machine as potentially suspicious - absence of visible symptoms is not evidence of an absence of any compromise. You should be especially worried about servers and any workstations running business critical information.
Once you have isolated any infected processes on the virtual guests, you should be able to send a sample to the company that made the antivirus software you're using on these machines. They will be keen to examine samples and produce fixes for any new malware they see. In fact, if you have not done so already, you should contact them with your tale of woe as they might have some way of helping.
Try very hard to work out what the original infection vector was - this worm may be an exploit that was hidden inside a compromised website that someone visited, it may have been brought in from someone's home on a memory stick or received by email, to name but a few ways. Did the exploit compromise these machines via a user with admin rights? If so, don't give users admin rights in future. You need to try and make sure the infection source is fixed and you need to see if there is any procedural change you can make to make that infection route more difficult for exploits in the future.
Some of these steps will seem over the top. Heck some of them probably are over the top, especially if you determine that only a few machines are actually compromised, but they should guarantee your network is as clean as it can be. Bosses won't be keen on some of these steps either, but there's not much to be done about that.
Shutdown all machines on the network. All workstations. All servers. Everything. Yes, even the bosses' teenage son's laptop which the son uses to sneak onto the network while waiting for dad to finish work so the son can play 'dubious-javascript-exploit-Ville' on whatever the current social media site du-jour is. In fact, thinking about it, shut this machine down especially. With a brick if that's what it takes.
Start up each server in turn. Apply any fix you've discovered for yourself or have been given by an AV company. Audit the users and groups for any unexplained accounts (both local accounts and AD accounts), audit installed software for anything unexpected and use wireshark on another system to watch traffic coming from this server (If you find any issues at this point then seriously consider rebuilding that server). Shut each system down before you start the next one, so that a compromised machine can't attack the others. Or unplug them from the network, so you can do several at once but they can't talk to each other, its all good.
Once you're as sure as you can be that all your servers are clean, start them up and using wireshark, process monitor, etc. again observe them again for any strange behaviour.
Reset every single user password. And if possible, service account passwords, too. Yeah I know its a pain. We're about to head into "possibly over the top" territory at this point. Your call.
Rebuild all the workstations. Do so one at a time, so that possibly infected machines aren't sitting there idle on the LAN attacking freshly rebuilt ones. Yes this will take a while, sorry about that.
If that's not possible then:
Carry out the steps I outlined above for servers on all the "hopefully clean" workstations.
Rebuild all the ones that showed any hint of suspicious activity, and do so while all the "hopefully clean" machines are powered off.
If you haven't already then consider centralised AV that will report problems back to a server where you can watch for problems, centralised event logging, network monitoring, etc. Obviously pick and choose which of these are right for this network's needs and budgets, but there's clearly a problem here, right?
Review user rights and software installs on these machines, and set up a periodic audit to make sure things are still how you expect them to be. Also make sure that users are encouraged to report things asap without being moaned at, encourage a business culture of fixing IT problems rather than shooting the messenger, etc.
Best Answer
You asked for ideas and... here is mine.
To solve your problem, you've two very limiting conditions:
You're unable to take hands on your Cisco ('cause it's not yours and its configuration cannot be changed to suite your needs);
You cannot change (at least, not easily) the way Zeroshell is working (due to the very nature of Zeroshell itself [it's quite complex to rebuild Zeroshell to suite your needs [see below]).
On the other end as you want REAL_TIME_MONITORING and PER-PORT-TRAFFIC-ACCOUNTING you're mostly forced to have at least one point (one network interface) where:
What I have done in such situations is to REPLACE the existing appliance (in your case: Zeroshell; in my case various hardware appliances from various vendors) with something I can fully manage without constraints: a common linux box with at least two interfaces properly configured to route/firewall traffic.
Let's suppose this could be OK for you (...even tough I understand that could be an issue, for you, due to the initial setup-efforts).
IF such a machine is available, THEN I'd add to the set of software to install on it:
IPTRAF: despite its age, it's still perfectly able to gives out REAL-TIME data from your network interfaces. It provides a character-user-interface, so it can be launched remotely, within a simple SSH connection (no web, no big GUI libraries, etc.);
NTOP: from the official website: "...a network traffic probe that shows the network usage, similar to what the popular top Unix command does...". NTOP is way much feature-rich than IPTRAF. Definitely more powerful (but more complex to configure/install than a single "apt-get install" or "yum install")
As clearly stated, both tools above provides good REAL-TIME data (as you asked in your question). Anyway, I'm quite confident that you need ALSO asynchronous data: I'm sure you want also to be able to check something like: "who were the hosts/MACs that generated/consumed most of the traffic, yesterday? And for which protocols?", probably drilling down such data back to a single IP/MAC/PORT, and down to a granularity of.... 1 minute. Don't you? In such a case I strongly reccomend:
-
it can easily keep track of traffic flowing along eth0 in a mysql table, so for you to easily check what happened on your network with a common/simple SQL-query.
Just to give you some real numbers, I've succesfully used PMACCT on a server with a XEON X3350; 4GB of RAM; 4 broadcom GigaEth interfaces; nearly 70 VLANs configured on eth0 and pmacct listening on all of them; +/- 300GB of various IP traffic routed on a daily basis; PMACCT generating accounting EVERY_MINUTE, for EVERY_VLAN, for EVERY tuple (src_mac, dst_mac, src_ip, dst_ip, src_port, dst_port); +/- 60.000.000 accounting records per day. All of this, without any issue (but writing on text-files, not in MySQL). In smaller environments, anyway, there are no problems in writing directly to MySQL.
Also, please note that thanks to PMACCT I keep track of EACH IP addresses seen on my networks, on a daily basis (in other words: I know that 10.29.19.89 have not been seen since july 16th 2014; 172.17.1.45 have never been seen [since the start of PMACCT accounting]; etc.).
Also on PMACCT: I have configured the ethernet switch connecting my main Internet-gateway, to "mirror" its traffic to a free port, where I've plugged an ad-hoc linux-box accounting all the Internet traffic (a 1GEth link). No issue at all.
A final note about PMACCT: should you (or some other readers) wonder why I NOT choosed some more common NETFLOW/IPFIX probe/collector, the reason is very simple: PMACCT is the only one I've found being able to account also MAC-addresses.