we found the root cause of this. We had an acceptCount of 25 in our tomcat server.xml.

acceptCount is documented like this:

acceptCount

The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.

But this is not the whole story about acceptCount. Short: acceptCount is the backlog Parameter when opening the socket. So this value is important for the listen backlog, even if not all threads are busy. It is important if request are faster coming in then tomcat can accept and delegate them to waiting threads. The default acceptCount is 100. This is still a small value to feed a sudden peak in requests.

We checked the same thing with apache and nginx and had the same strange packet loss but with higher concurrency values. The corresponding value in apache is ListenBacklog which defaults to 511.

BUT, with debian (and other linux based os) the default max value for the backlog paramter is 128.

$ sysctl -a | grep somaxc
net.core.somaxconn = 128

So whatever you type in acceptCount or ListenBacklog it will not be over 128 until you change net.core.somaxconn

For a very busy webserver 128 is not enough. You should change it to something like 500, 1000 or 3000, depending on your needs.

After setting acceptCount to 1000 and net.core.somaxconn to 1000 we no longer had those dropped packets. (Now we have a bottleneck somewhere else, but this is another story..)

Default Gateways work a bit differently on IPv6.

What I suggest you do is fire up tcpdump and watch it for Router Advertisements (you might want to analyse the capture in WireShark - much easier). This is how your IPv6 host knows how to get out to the internet. Also watch for Neighbor Solicitation as this is the replacement for ARP.

If you see these requests coming in, but not going back out, then your internal firewall is blocking IPv6 traffic. If you see them going out (requests), but not back in, then you need to make sure that your prefix length is correct (that's the /64 part. Although I've seen a lot of /64s being passed out that are actually a /48).

Big Important Bold Bit

ICMP is absolutally critical for IPv6 functionality. In IPv4 it can be blocked without huge impact, but in IPv6, the whole network relies on ICMP for all its functionality. Don't block it.

In reference to

My provider gave me a "/64" (whatever that is - I'm still confused) of IPv6 addresses

That's your Prefix Length, previously known as your Subnet Mask that's used for CIDR. The lower the number, the more IP addresses you have. For IPv6, /64 and /48 are very common and should give you more IP addresses than you will ever need in your entire life.