Say I had two servers which needed super-low latency (Database, file etc.). Would it be possible to directly connect the two servers with 10GbE, so each server had 1 (in the real world it would have 2) connections to the 'main' network, but 1 network card with an ethernet cable that connected directly to the second server, no switches or routers, just a direct connection
Internet/Datacenter
|
|
|
|
|
|
|
--------------------
| |
------------| Switch |-----------
| | | |
| -------------------- |
| |
| |
| |
| |
| |
| |
| |
Network Card 1 (eth0) Network Card 1 (eth0)
| |
-------------------- --------------------
| | | |
| Server 1 | | Server 2 |
| | | |
-------------------- --------------------
| |
Network Card 2 (eth1) Network Card 2 (eth1)
| |
| |
| Direct 10GbE |
-------------------------------------------
My first question is, would this even be possible? Would they need any unusual/special services configured to let them talk over this network other than a standard file in /etc/sysconfig/network-scripts/? They would both have static IPs on eth1 but how would things like routing work? I'm not an expert on networking so this is probably a n00b-ish question
Second question, is there any point? Would there be any advantages doing this over just letting them communicate over the standard network connection via the switch, or giving them a second dedicated network just for communicating intra-server (Since bandwidth would be used on the standard network by clients accessing the servers). Assuming latency was the priority.
I know there are some issues with this method, like when we came to add a 3rd server we'd ether have to give every server another network card and probably set up some very complicated replication triangle thingy but since this is hypothetical lets ignore that.
And since latency is the key issue, would fiber be better over ethernet (Speed isn't important so long as it can do a couple of Gb/sec)
I phrased this question from a linux POV, because that's my background, but it could apply to any server/device
Best Answer
There's no reason why you technically can't do this.
I'd probably do something similar, under the circumstances, actually. From a purely linux point of view, it's really easy, just give the connection an IP address with a /30 bitmask, giving you 2 IP addresses, then it's a simple Point-to-Point link.
If you wanted to grow the network, you could get a 10GE switch, and then have a seperate VLAN for traffic between servers. There's some very shiny gear in the Force10 range of switches that can do line-rate 10GE switching, with enormous buffers.