The audio processing algorithm you are interested in is called "Acoustic Echo Cancellation", or AEC. It is most commonly used in speakerphones to remove the output of the speaker from the mic signal. Most of this benefit is to the person on the other end of the phone call, since he won't be hearing echos of himself.
Some cheap and not so cheap speakerphones don't use AEC. I have a Polycom speakerphone which is "half duplex". Meaning that when one side is talking, the other side is muted. Because of this, there is no chance for echos or feedback to happen. Unfortunately, this also allows for a "filibuster"-- if one side never shuts up then the other side can never interrupt.
There are many types of AEC algorithms, and almost every type is patented. Most of them involve some form of modeling, where a model of the "speaker to mic acoustic signal path" is created. Once created, we can predict how the speaker output will be picked up by the mic, and thus remove that signal from the mic, leaving only the intended sounds in the mic signal.
This model would thus figure out how the sounds reflect off of the walls and other things in the room, etc. The patents for AEC usually center around exactly how this model is initially created and later updated as things change in the room (mic position, position of people and furniture, etc).
In addition to the "room model", there are other noise-reduction algorithms used. While these algorithms are not technically part of AEC, there are no useful implementations of AEC that don't use these. Normally there is some sort of simple noise-gate (or a multi-band noise gate). Other algorithms are also typically used, but are either patented or treated as a "trade secret"-- which is why I can't tell you about them! :(
Most AEC algorithms operate on a limited frequency range, 300 Hz to 3 KHz, which is the same frequency range as most telephones. Increasingly, wide-band AEC is becoming popular with the advent of higher-bandwidth teleconferencing/telepresence systems.
AEC algorithms are very computationally expensive, and the wide-band AEC requires several times more horsepower than the more limited versions. It is not uncommon for a single "run-of-the-mill" DSP to only be able to do 1 or 2 channels of AEC. For a high quality wide-band AEC, a single high-powered DSP might be required for a single channel.
AEC algorithms are also very difficult to implement. In the entire USA, there are perhaps only 10 or 20 people who have the ability to write a good one. One very smart person that I know just wrote a wide-band AEC algorithm and it took him over a year!
For a 2-way baby monitor, I highly recommend using a half-duplex approach!
Best Answer
Like Dave said, you need to detect the state of the phone, and you need to send ringing voltage /24 VAC should do fine, but I needed 48 VAC for some Panasonic phones/, then you have to trip this voltage off as soon as someone has picked up the phone...comes out lot of logic for what looks like simple task, hard to do without micro.
I have actually designed exactly what you need, long time ago, it was even a commercial product, but it's long history know. I was thinking of making it public anyway, but it is a bit over designed, we used the current loop for phone supply (and it was a bad choice of uC). You might find some parts useful, or try to replace the micro with some simpler logic while taking the phone supply, of / on hook detector and some other stuff from our design (us as 'me and my good friend Mario', his part was FW and analog HW design, my part was CPU HW design, PCB design, testing, and documentation).
Here you can see the circuit reponsible for powering the phone, line state detection (on/off hook, but also the pulse dialing detection), ring voltage control, everything regarding the single phone. You could build two of those circuits, and you have got yourself an intercom, but with many options you might not need :-).
I can post the complete schematic, code everything, if anyone was still interested for analog phones made useful. Personally, I still use similar switch at home. And yes, our exchange (two designers, remember?) has four internal extensions, one external line, phones galvanically isolated from the external line, 'do not disturb' mode, distinct ringing for internal and external line...and many more options pushed into 4K of PIC assembler code. It has got also a complete users manual (in Croatian only ;-) )
A should mentioned that phones were interconnected using the analog crosspoint switch, not relays or discrete components. Music on hold was for example also routed to the relevant phone through the crosspoint patrix, as well as the dial internal dial tone, ring tone, external party...
I should make some open hardware / open source product out of it. Bad news? I have mentioned the micro. PIC16C57, all assembler, it was nightmare /my friend has pulled it off, micro choice was mine which was as stupid as it could be considering I was already familiar with 8051, and the price of a micro was not relevant anyway. If we designed in the 8051 back than, it would have 64K Flash today in a same DIP package. May be you could do it now ;-)). The boards are separate, so even Arduino could be used for the CPU :-)