I'm wondering if there is a heuristic plug-in, software or other process for determining protocol when a non-standard TCP port is used.
Wireshark will decode packets based on the well-known port number. For example, TCP 80 will be decoded as HTTP. There is some level of flexibility, such as the ability to define TCP 8080 traffic as HTTP and to be decoded accordingly. My question surrounds decoding non-standard port numbers. You can use the "Decode As…" functionality to do this, provided that the protocol is known.
Below is a session with the server listening on TCP 4443 which is officially Individual Zone Solutions P/L Remote Mail Checking Protocol, for which there is no decode and I was pretty sure is not what it is:
I took a wild guess, based on the similarity of 443 and 4443 and did a "Decode As…" SSL and was rewarded with:
This worked, however it could be lots of guessing/trial and error. Since I can analyze the payload and most likely find clues as to the protocol, is there an automated method to do the same within Wireshark or another tool?
Best Answer
You need software that does DPI (Deep Packet Inspection). Some companies have made standard rules for some protocols, where in a TCP or UDP packet or some sequence, it can identify the protocol or application that is running.
This kind of application realizes an analysis of Layer 7 of a packet, matching the packet content with a huge rules database.
Like @ojs said, you can try with nDPI from ntop, test it and see the results.
Cisco also has its own database that can be used with the OPNManager, but requires a license.