How to modulate Transceiver module for use with ATmega164PA

Your option 2 is only viable if your microcontroller has built in Ethernet hardware. Most controllers don't, but they exist especially at the higher end of the spectrum.

Your option 1 with a separate Ethernet controller is pretty common. You may want to check several open hardware microcontroller boards and get yourself inspired by how they solve Ethernet (and which chips they use). Most of the time circuit diagrams and libraries are fully available, but mind the license that they come with. I know there exist Arduino compatible boards with Ethernet, but I'm not sure of its name.

IrDA uses a short transmit pulse (2uS from memory) which is the main way it varies to normal serial data at the physical level. For example you can see the following in the datasheet:

This Schmitt-Trigger input is used to transmit serial data when SD is low. An on-chip protection circuit disables the LED driver if the TXD pin is asserted for longer than 50 μs (max. 300 μs).

Other than being able to send and receive data that roughly conforms to that they aren't particular about the protocol and I've used a similar part from a different manufacturer in the past with a non-standard protocol. It doesn't seem to be documented in the datasheet but something to be aware of is that because IrDA is half-duplex some modules disable the receiver while transmitting to avoid getting a reflected signal back, so that may put an end to being able to use one for proximity detection.

I'm not sure if the Vishay modules do that but if you don't get an answer from someone that knows for sure you could do a pretty simple test by feeding in say a 10 kHz signal into the TXD and check with a scope or microcontroller interrupt if you get anything on RXD when pointed at a reflective surface. Ideally the signal should have the short pulse duration although for a quick test you can probably rely on the in-built protection. Also make sure to take notice of the information on capacitor selection and component placement, they can be pretty fussy in that regard.

As you've confirmed that you can receive back reflected data you may be able to use it for proximity detection by limiting the current through the VCC2 pin to reduce the range. The datasheet mentions using an external resistor when less than 300 mA is required, so a simple way might be to include a resistor and then use say a reed relay to short it out to go back to full power while transmitting data. If you wanted something programmable you could probably take a look at some programmable constant current LED drivers.

Because the data output is strictly digital and you don't any other access to the photodiode output the only method I can think of for proximity detection is to vary the output power to control the range and use the fact that you're receiving back the same signal to indicate there's an object in front. That's assuming the reflected data is the result of the light being reflected back from another object, if not you may need to add a physical barrier between the LED and photodiode to try and isolate them. They also tend to have fairly wide angle lenses so you may want to add something to narrow the beam regardless so you're not detecting the floor etc.