Your best bet is to tap the links themselves. Use an in-line optical tap or a powered copper tap. This will split the connection off to a secondary cable, which you can then plug into your server for monitoring. You'll have an instantly-accurate, infinitely-granular picture of what's going across the wire.
Edit: If you've got too many links to tap, your only other option I can think of is to SPAN all the ports to a single 10G or 40G interface and monitor that. Keep in mind that you're adding in the processing time for the SPAN, and you may drop traffic depending on oversubscription ratios, etc. Just depends on how much accuracy you really need.
The only way to get a 100% complete, guaranteed measurement of the bandwidth crossing a given link is to tap the link.
Does it mean it's electric circuitry is capable of serializing/deserializing (SERDES) 400G bits per second of data onto a wire while maintaining a relativley clean signal (low SNR)?
Yes, that's what 400GE is designed for. The physical coding sublayer (PCS) uses forward error correction (FEC) to achieve a block error rate of 10-13 or better. The acceptable SNR varies with the different PHYs.
400G Ethernet uses multiple 25, 50, or 100 Gbit/s lanes (with up to 53 GBd using PAM-4), so it requires multiple fiber strands or wavelengths (or differential pairs in a backplane). 400GBASE-LR4 currently provides up to 10 km reach (nominally), the upcoming 400GBASE-ER8 is going to support up to 40 km.
Propagation in fiber is generally limited by the fiber's velocity factor, ca. .67 (the reciprocal of the refractive index), so .67 x c0 ≈ 200,000 km/s. Accordingly, on the fiber the PAM-4 symbols at 53 GBd are 3.8 mm "long".
For Ethernet, the nominal bandwidth is present at the top of the physical layer. It includes "high-level" signaling like the preamble and inter-packet gap (IPG) and of course, L2 frame header and footer. However, it excludes line code overhead from PCS.
That way, the exact, usable bandwidth can be very easily calculated: for maximum-sized frames there's 1500 bytes L3 payload and 38 bytes overhead for L1 & L2 in total. Thus, 400GE provides a usable bandwidth for L3 of 400 Gbit/s / 8 bit/byte / 1538 bytes * 1500 bytes = 48,76 GB/s.
Incidentally, Broadcom just announced their new Tomahawk 4 switch chip, sporting 64x 400 Gbit/s ports (or 256x 100 Gbit/s) for a total backplane capacity of 25.6 Tbit/s. The chip with 31 billion transistors must be huge, even in 7 nm. Truly mindblowing...
Best Answer
The difference between maximum link capacity and link utilization defines your headroom for future growth or additional workloads. The smaller this headroom the shorter is your time before you have to upgrade the network.
If you manage to distribute the load more evenly and lower the peak network utilization (with the same overall/daily throughput) the longer the time you can use the network without problems and the lower the overall cost. Of course, lowering the overall throughput serves the same purpose.