Routing – How to setup self hosting with redundant Internet connections

This might be a bit long winded, but I'll try to address most of your issues in one go. Wish me luck!

You can have your upstream service providers advertise a default route (0.0.0.0) through BGP to your routers. When you configure OSPF on your BGP routers, you can use the command "default-information originate." As long as you're getting the default route from your ISP (which only happens when you have connectivity to that ISP), you'll advertise the default route into OSPF. As soon as you lose that default route from your ISP, you'll lose the OSPF default route.

You'll need to establish BGP peering between all of your internet routers (the ones connected to your ISPs). This is an internal BGP or iBGP relationship and needs to be a full mesh. Per your diagram, you only have two internet routers, so a full mesh is simple. There are no tricky steps to peer on iBGP versus eBGP; just a simple neighbor statement. When BGP sees that your neighbor has the same AS number, it forms an iBGP relationship instead of an eBGP one.

Preferring one route over another is a bit tricky, particularly where BGP is concerned. Load balancing is done using route maps and traditionally with as-path prepending. There are a few ways to do this. I'd like to add a caveat that if you aren't filtering outgoing route advertisements, you'll end up advertising ISP A's routes to ISP B and C and so forth which will turn you into a transit AS and you'll end up piping some A <-> B, A <-> C, and B <-> C traffic through your network which is probably not what you want.

Here's one go at your load balancing:

! Set an IP access list that matches your BGP-advertised network

ip access-list standard 1 permit a.b.c.d mask a.b.c.d

! Set an IP as-path access list to only allow advertising of YOUR network

ip as-path access-list 1 permit ^$

! Make a route-map for ISP C, match the as-path and access-list above, then make it look
! less appealing than going through ISP A/B

route-map ISP_C 10
match ip address 1
match as-path 1
set as-path prepend <Your AS> <Your AS> <Your AS>

! The more times you prepend your AS to a route, the less desirable it looks, so traffic
! will be more likely to come in via ISP A/B than C.  The last step is to add it to your
! ISP C neighbor statement in BGP

router bgp <Your AS>
neighbor <ISP C> route-map ISP_C out

If you have more than one subnet, you can even things out a little more and use router B for site B and router A for site A by selectively prepending your AS path to individual subnets. Here's an example:

Router A:

ip access-list standard 1
permit <site B subnets>

ip as-path 1 permit ^$

route-map ISP_AB 10
match ip address 1
match as-path 1
set as-path prepend <Your AS> <Your AS>
route-map ISP AB 20
match as-path 1

router bgp <Your AS>
neighbor <ISP A> route-map ISP_AB out
neighbor <ISP B> route-map ISP_AB out

Router B:

ip access-list standard 1
permit <site A subnets>

ip as-path 1 permit ^$

route-map ISP_C 10
match ip address 1
match as-path 1
set as-path prepend <Your AS> <Your AS>
route-map ISP C 20
match as-path 1

router bgp <Your AS>
neighbor <ISP C> route-map ISP_C out

What you've effectively done is make site B's subnets look less appealing coming from Router A than they do from ISP C and make site A's subnets look less appealing coming from Router B than they do from ISP A/B. You might have to play around with your AS path prepending some to get the right amount of prepending in.

I hope this helps! BGP is a bit of a monster, but once you understand the parts, it's fun to play with. I highly recommend the CBT Nuggets series on BGP if you feel a bit shaky on the subject and I always use GNS3 as a testbed to verify big network changes before I implement them.

1. The networks you connect to should set up filters to accept routes from you only if they are "yours" or those of your customer(s). Some ISPs build filters from various databases (RIR or IRR). Not all networks do this.
2. BGP tables (RIB) do contain ASNs. Forwarding tables (FIB) do not.
3. Not necessarily. It depends on your needs. You could also accept just a default route from each provider, or a subset of the routes your upstreams know, or a complete table form one upstream and partial routes from another.