Mixing jumbo frames and normal frames on one server

The key here is that you can control your little segment of the network and enable large MTUs but you can't control the path your packets take over the internet and certainly can't control the configuration of the routers your packets will pass through. Most internet routers aren't configured above 1500 so you aren't going to have much luck with this solution. Worse, sometimes larger packets will actually get dropped by routers that don't support jumbo frames so I think you'd actually find things are worse if you try sending jumbo frames to the internet.

Jumbo frames are great on your internal network - especially for networks doing streaming or iSCSI.

Make sure that your NICs exist in separate netblocks when doing this. If you use Linux, packets are routed via the first NIC in the system in the netblock, so, even though eth1 has an MTU of 9000, it could end up routing those packets through eth0.

We set up a separate VLAN to our storage network and had to set up a separate netblock on eth1 to avoid that behavior. Increasing the MTU to 9000 easily increased throughput as that particular system deals with streaming a number of rather large files.