in my experience having truly separate AGND and DGND nets almost never works out well in practice. 90% of the designs i see that try to do this end up with current loops that introduce EMI issues and can generate more noise in the analog portions of the circuit than using a single ground with careful part placement would.
Having two GND planes also creates a problem for routing in that signals referenced to a particular ground should only ever be run on layers that are adjacent to this plane or its associate power plane. This can result is a pretty funky stack up that can limit where you can run traces. Your best answer would be AGND,signal,?GND,POWER,signal,DGND but thats funky to layout, uses lots of vias, only gives 2 signal layers to route on.
What i would recommend is a single solid ground plane and careful part placement. High speed digital signals and noise will follow the path of least inductance to ground not the path of least resistance. The path of least inductance is the smallest loop area, for signals this is directly under the trace on the adjacent ground plane. In some cases a ground pour on top, bottom, or both can be helpful in reducing noise pick up as well. This is dependent on the components and the design layout.
Create virtual partitions, keep out areas, where you only run either analog or digital signals, keeping in mind that the return current path for the low frequency analog signals is the path of least resistance, while the return path for the high speed digital signals is the path of least inductance. As long as your careful to ensure that the return current paths don't cross, especially a digital return running under your analog sections. You shouldn't get much noise pick up at all.
If your have a particular device that is very sensitive to noise, such as a high resolution ADC, you can use a ground island to increase noise immunity, like this:
alt text http://www.hottconsultants.com/techtips/a-d%20gnd%20plane.gif
In cases where i have some sensitive analog circuitry i will usually also use a power island that is separated from the digital power supply by an LC filter of some sort, depending on the digital frequencies i'm wishing to block.
Look for application notes for your PHY and magnetics. The manufacturer would know best in regards to what works with their parts.
Generally there is no ground/power or routing under the magnetics and try to avoid ground/power under the TX/RX pairs. If you can't route the whole trace without a ground/power plane under it, leave the plane under it. It is worse if you go over a break in the plane.
For termination, check with manufacturers of the PHY and magnetics. Like you said, there are a few different schemes, the manufacturer should know best about their device.
We follow what I described above at work and don't have any problems with ethernet.
Best Answer
OK, here are my tips
0) As has been noted, do not place vias within SMT pads.
1) Start with ground and power. Ground is the most important single entity on your board. The higher your signal frequencies the more important a ground plane is. This means that USB (480 MHz) is a real stressor and Ethernet (100 MHz) is only marginally better. Power planes are nice but not as important. Make all power traces as thick as you can, and provide decoupling capacitors (0.1 uF ceramics are a good default value) as close to the ICs as possible. In general, try for one decoupling capacitor per power pin. The consequences of inadequate ground and power routing/decoupling can be bizarre beyond belief. If your voltage regulators are located off the board, include at least one larger decoupling capacitor per voltage, 10 uF to 100 uF is a good start. For preference, tantalum or MLCC. Avoid aluminum electrolytics in digital applications - they tend to have lousy high-frequency characteristics.
2) Only after you have a good solid ground plane established should you start running signal traces. Give first priority to high speed and analog signals. Signals like audio need some care to keep them away from digital lines as much as possible: the human ear can detect distortions in sound in a very sensitive manner.
3) Be extremely wary of thinking you can scrimp on layers to save money, especially for either high-frequency or mixed analog/digital boards. USB or Ethernet counts as high frequency.
4) Put reference designators on every part.
5) Learn how to design controlled-impedance lines, both single and differential.
6) Unless you will be soldering parts on the board by hand, avoid placing parts on both sides. This will increase assembly costs substantially.