One way of doing this kind of thing with pure routing is to have the "service" IP addresses be secondaries on all the servers, and have routes as appropriate.
Consider this straightforward network where the clients AC and BC go through a router to get to the servers A1, A2, B1, B2. Everything has default gateway aimed at their network's .1. RA and RB also have the obvious inter-site routes: RA sends 10.0.0.2/24 and 10.1.2.0/24 to RB; and RB sends 10.0.1.0/24 and 10.1.1.0/24 to RA.
servers servers
A1 A2 B1 B2
|.4 |.5 |.4 |.5
===+===+===+===+=== ===+===+===+===+====
10.1.1.0/24 |.1 |.1 10.1.2.0/24
RA---------RB
10.0.1.0/24 |.1 |.1 10.0.2.0/24
===+===========+=== ===+===========+====
|.64 |.64
AC BC
clients clients
If the servers are also given addresses in 10.0.0.0/24 as aliases on loopback interfaces (such as lo:1) A1=10.0.0.4, B1 also 10.0.0.4, A2=10.0.0.5, B2 also 10.0.0.5.
Then you control which server gets the work by the routing on RA and RB.
If the whole network moves from site A to B, you do routes on 10.0.0.0/24; if it's one host at a time you do host routes eg 10.0.0.4/32.
This structure is useful in two cases:
- Where you want to use a local server in a content distribution method (like Google's DNS on 8.8.8.8) you can have different routes on RA and RB. (10.0.0.4->10.0.1.4 on RA, 10.0.0.4->10.0.2.4 on RB).
- Where you want a given server in only one place at a time, you have 10.0.0.4->10.0.1.4 on RA, 10.0.0.4->RA on RB.
The scheme lends itself naturally to some virtualisation schemes, where the service networks are inside the virtual host.
Does that help at all?
Other methods include
- MPLS, as you've said
- VPN tunnels
EDIT: there's certainly some horrible way to do this with NAT, and there's probably a horrible way to do it with proxy ARP. The pure routing method has the advantage that you can manage the routes by whatever method you have available, such as the BGP you mentioned, but any routing update method will work. Additionally, the routing method is (I think) the only one which supports "send to my most local" CDP-type functionality, if that's something which might be helpful.
PPP is a protocol for transporting higher-layer protocols (like IP) over a simple serial interface. Since you'd already be running Ethernet, PPP doesn't work and is of no use anyway. You could use PPPoE but still without gain.
Simply run Ethernet and IP on top and you're fine. All you need are routers or switches with SFP interfaces, suitable SFP transceivers and the right cable. Of course, you need to use IP addresses on the link but private addresses are free to use. Config, stir, enjoy.
In case you're worried about the security of the fiber outside a building: PPP or PPPoE doesn't help with that at all. You'd need an encrypted tunnel link, usually called VPN.
Is PPP a useful choice for fiber connections?
Not really. PPP requires an underlying hardware interface that most likely is expensive and hard to acquire - also think of a spare or replacement in a few years.
Ethernet is available in an extremely wide variety of speeds and distances and it's very low priced. By performance and by budget there's no real alternative for lighting fiber.
Best Answer
There are a number of things ISP's do:
The last option is most common. There is only a limited number of transit providers who don't buy transit from another ISP, they're usually called Tier 1. Most ISP's combine IXP connectivity and IP transit for their global connectivity.
Edit: Here's a real-world example: I used NLNOG Ring's
.
ring-traceto create a graph of how networks around the world reach Facebook's network.As you can see from this example, a lot of networks reach Facebook via DE-CIX (the IXP in Frankfurt, Germany, one of the largest in the world), but there are also a large number of networks which use Telia (AS1299) and NTT (AS2914) to reach Facebook. Telia and NTT are tier 1 transit providers.
Edit 2: Since the image is downscaled it's hard to read. here is a full size version.