We currently have a 2-node Oracle 11g RAC using 1GigE as the private interconnect. At peak update time, the traffic between nodes reaches 35 MByte/second, so we are still doing good in terms of bandwidth capacity. We are considering adding a third node to the cluster to support our growth in transaction volume. I am wondering whether somebody out there has already seen how adding nodes to the cluster would impact private interconnect traffic and what would trigger the use of Infiniband. From what i read, Infiniband not only can increase bandwidth but also has much shorter network latency. Can anybody who use Infiniband for RAC private interconnect share your experience with us? Thanks in advance.
Infiniband for Oracle 11g RAC private interconnect
oracleoracle-rac
Best Answer
I've a 4 node oltp rac database and a 3 node dw rac database using IB as interconnects.
We use cisco IB switches and cisco IB cards. We use IP over IB as the interconnect protocol. So far we are quite please with the results. Never had an issue with the interconnect traffic. We've come close to saturating our gigE public interface but no where close to the IB bandwidth. All 7 nodes are connected to 2 infiniband switches setup in active/passive failover configuration.