Variance in ping times can be indicative of several things, but the fact that you get the same behaviour regardless of which server in that data centre you ping seems to indicate that there may be an issue with a link or device along the path.

It's worth doing repeated traceroutes to the sever(s) in question, to see if you can identify a particular network device where latency is varying; or alternatively, it could be that there are multiple paths to this location, one of which experiences generally higher latency.

Are you seeing any packet loss when you perform extended pings? If so, this may be indicative of a congested link.

Could you provide a bit more detail about the connectivity between your office and your data centres? Are you using leased lines? Internet VPN? Have you spoken to your company's NOC/network administrators? They may be aware of this issue already.

Default Gateways work a bit differently on IPv6.

What I suggest you do is fire up tcpdump and watch it for Router Advertisements (you might want to analyse the capture in WireShark - much easier). This is how your IPv6 host knows how to get out to the internet. Also watch for Neighbor Solicitation as this is the replacement for ARP.

If you see these requests coming in, but not going back out, then your internal firewall is blocking IPv6 traffic. If you see them going out (requests), but not back in, then you need to make sure that your prefix length is correct (that's the /64 part. Although I've seen a lot of /64s being passed out that are actually a /48).

Big Important Bold Bit

ICMP is absolutally critical for IPv6 functionality. In IPv4 it can be blocked without huge impact, but in IPv6, the whole network relies on ICMP for all its functionality. Don't block it.

In reference to

My provider gave me a "/64" (whatever that is - I'm still confused) of IPv6 addresses

That's your Prefix Length, previously known as your Subnet Mask that's used for CIDR. The lower the number, the more IP addresses you have. For IPv6, /64 and /48 are very common and should give you more IP addresses than you will ever need in your entire life.