Linux – Routing multiple IPs from different subnets (ISPs) with Linux

I had/have an interesting problem today regarding routing with linux.

My expectation: to be able to assign multiple public ips to an interface and have one default gateway. packets sent to a particular ip will respond with that ip through the particular gateway the packet was received on, while any origination traffic would go through the default gateway. Here is the setup

ifconfig eth0 x.x.x.x/29
ifconfig eth0:1 y.y.y.y/29
route add default gateway y.y.y.g

Packets originating at this machine going out would have ip y.y.y.y and exit through y.y.y.g. If a packet was received on x.x.x.x it would be replied to via x.x.x.g even though x.x.x.g is not the default route.

Actual results: This worked fine for me until today. Today based my experience with above setup I added z.z.z.z to the mix thinking I could just assign another ip and the same results would apply. Totally not what happened and I do not understand why. As soon as I added z.z.z.z with

ifconfig eth0:2 z.z.z.z

i was able to manually ping the new network gateway at z.z.z.g, but external packets were not responded to. The other two routes worked fine as described before. In my experimentation I set z.z.z.g as my default route. This caused all originating traffic to originate from z.z.z.z as expected and external traffic sent to z.z.z.z worked fine. However x.x.x.x and y.y.y.y completely stopped responding to external traffic, but I could manually originate traffic from either and it would route fine through their respective gateways. In no way shape or form could I get z.z.z.z to play nice with any other ip or route combinations, although the original two worked just fine together without z.z.z.z.

What could cause this behavior ? (FYI iptables was not the issue I made sure of that. All ips are public and no NAT involved)