You need to weigh several factors to determine which method to use in different places on your network. Different network engineers will come up with different recommendations. This is really leading to opinion-based answers, and it is probably better suited for a discussion on Network Engineering Chat.
Load-Balancing and Forwarding Methods
EtherChannel balances the traffic load across the links in a channel
by reducing part of the binary pattern formed from the addresses in
the frame to a numerical value that selects one of the links in the
channel. EtherChannel load-balancing can use MAC addresses or IP
addresses, source or destination addresses, or both source and
destination addresses. The selected mode applies to all EtherChannels
configured on the switch. You configure the load-balancing and
forwarding method by using the port-channel load-balance global
configuration command.
With source-MAC address forwarding, when packets are forwarded to an
EtherChannel, they are distributed across the ports in the channel
based on the source-MAC address of the incoming packet. Therefore, to
provide load-balancing, packets from different hosts use different
ports in the channel, but packets from the same host use the same port
in the channel.
With destination-MAC address forwarding, when packets are forwarded to
an EtherChannel, they are distributed across the ports in the channel
based on the destination host's MAC address of the incoming packet.
Therefore, packets to the same destination are forwarded over the same
port, and packets to a different destination are sent on a different
port in the channel.
With source-and-destination MAC address forwarding, when packets are
forwarded to an EtherChannel, they are distributed across the ports in
the channel based on both the source and destination MAC addresses.
This forwarding method, a combination source-MAC and destination-MAC
address forwarding methods of load distribution, can be used if it is
not clear whether source-MAC or destination-MAC address forwarding is
better suited on a particular switch. With source-and-destination
MAC-address forwarding, packets sent from host A to host B, host A to
host C, and host C to host B could all use different ports in the
channel.
With source-IP address-based forwarding, when packets are forwarded to
an EtherChannel, they are distributed across the ports in the
EtherChannel based on the source-IP address of the incoming packet.
Therefore, to provide load-balancing, packets from different IP
addresses use different ports in the channel, but packets from the
same IP address use the same port in the channel.
With destination-IP address-based forwarding, when packets are
forwarded to an EtherChannel, they are distributed across the ports in
the EtherChannel based on the destination-IP address of the incoming
packet. Therefore, to provide load-balancing, packets from the same IP
source address sent to different IP destination addresses could be
sent on different ports in the channel. But packets sent from
different source IP addresses to the same destination IP address are
always sent on the same port in the channel.
With source-and-destination IP address-based forwarding, when packets
are forwarded to an EtherChannel, they are distributed across the
ports in the EtherChannel based on both the source and destination IP
addresses of the incoming packet. This forwarding method, a
combination of source-IP and destination-IP address-based forwarding,
can be used if it is not clear whether source-IP or destination-IP
address-based forwarding is better suited on a particular switch. In
this method, packets sent from the IP address A to IP address B, from
IP address A to IP address C, and from IP address C to IP address B
could all use different ports in the channel.
Different load-balancing methods have different advantages, and the
choice of a particular load-balancing method should be based on the
position of the switch in the network and the kind of traffic that
needs to be load-distributed. In Figure 40-5, an EtherChannel of four
workstations communicates with a router. Because the router is a
single-MAC-address device, source-based forwarding on the switch
EtherChannel ensures that the switch uses all available bandwidth to
the router. The router is configured for destination-based forwarding
because the large number of workstations ensures that the traffic is
evenly distributed from the router EtherChannel.
Use the option that provides the greatest variety in your
configuration. For example, if the traffic on a channel is going only
to a single MAC address, using the destination-MAC address always
chooses the same link in the channel. Using source addresses or IP
addresses might result in better load-balancing.
Figure 40-5 Load Distribution and Forwarding Methods
Best Answer
There's really no reason not to put multiple subnets on the same VLAN, but there's also probably no reason to do it.
Pro:
Con:
We generally don't do it because of the messiness and lack of security. One VLAN = one subnet is easier to document and easier to troubleshoot and there's usually not a good reason to complicate things.
The only reason I can think of to do it is company mergers or network upgrades and for both of those I'd prefer it to be temporary.
Edit to clarify, for the hosts on different subnets but the same VLAN to talk directly you'd need to either make them their own default gateway or add a route to the "other" subnet that connects it to the interface.
In the gateway case if the host IP was 10.1.1.2 then the gateway would also be 10.1.1.2. This will cause the host to ARP for everything on or off it's subnet. This would allow it to talk to the second subnet on that VLAN but the only way it'll be able to talk to anything else is if there's a router/firewall running proxy arp that can help it out.
In the route out the interface case you'd add something like "route add -net 192.56.76.0 netmask 255.255.255.0 eth0" to the device and then 10.1.1.2 will ARP directly on eth0 when it wants to reach 192.56.76.*.