Wireshark and IPSec

Ping packets should use an ICMP type of 8 (echo) or 0 (echo reply), so you could use a capture filter of:

icmp

and a display filter of:

icmp.type == 8 || icmp.type == 0

For HTTP, you can use a capture filter of:

tcp port 80

or a display filter of:

tcp.port == 80

or:

http

Note that a filter of http is not equivalent to the other two, which will include handshake and termination packets.

If you want to measure the number of connections rather than the amount of data, you can limit the capture or display filters to one side of the communication. For example, to capture only packets sent to port 80, use:

dst tcp port 80 

Couple that with an http display filter, or use:

tcp.dstport == 80 && http

For more on capture filters, read "Filtering while capturing" from the Wireshark user guide, the capture filters page on the Wireshark wiki, or pcap-filter (7) man page. For display filters, try the display filters page on the Wireshark wiki. The "Filter Expression" dialog box can help you build display filters.

For normal ethernet, your snaplen (-s option) should be 1500 if you want the entire packet. That'll give you the entire packet, and allow full protocol decodes when loaded up into WireShark itself. Depending on what you're sniffing, you'll probably want to increase your filesize as well.

To get at least the headers of most packets, a snaplen of 200 is usually sufficient. It'll get the E_II, IP, TCP/UDP, and some of the higher level protocol headers.

There are a few types of connection resets, and each has its own meaning.

Resetting all existing connections immediately

This style of reset shows up in sniffs as a big block of RSTs being sent by the server, probably with very little traffic between the packets. This can be caused by an application above the TCP/IP stack resetting itself, which in turn tells the TCP/IP stack to kill all connection-handles associated with that module.

Resetting all existing connections when they have traffic

This shows up in the sniff with a more sneaky pattern. After a certain point, any time an existing connection receives traffic from a client station a RST packet is issued. What the client does then depends on the higher level protocol, perhaps a reconnection attempt is issued. This is usually caused by the TCP/IP stack itself silently losing connection state. As far as the stack is concerned, that packet from the client is not associated with any open connections so it just issues a RST packet and ignores it.

Resetting new connection attempts

Existing connections continue to work, but SYN packets are replied to with an RST. In normal operation this is done because there is no open port for the port listed on the SYN packet. This is typically caused by a fault in the higher level application.

As for your retransmits, those are caused by clients not receiving packets they expect to get. This can be caused by outright packet loss, or it could be that the server just isn't sending those packets at all. If your sniff on the server shows the retransmit packets, and listing the conversation shows no packets being sent by the server, it's a sign that something has gone wrong on the server itself. This can be related to faults in the higher level application, the TCP/IP stack, or the NIC driver itself. I've seen NIC driver updates fix this, but in a VM that's less likely to be the source of the problem. Mass resets like that could be caused by resource exhaustion on the part of the server.