I understand that a DSP is optimized for digital signal processing, but I'm not sure how that impacts to the task of choosing an IC. Almost everything I do with a microcontroller involves the processing of digital signals!
For example, let's compare the popular Microchip dsPIC30 or 33 DSP and their other 16-bit offering, the PIC24 general purpose microcontroller. The dsPIC and the PIC can be configured to have the same memory and speed, they have similar peripherial sets, similar A/D capability, pin counts, current draw, etc. The only major difference that appears on Digikey's listing is the location of the oscillator. I can't tell the difference by looking at the prices (or any other field, for that matter.)
If I want to work with a couple of external sensors using various protocols (I2C, SPI, etc.), do some A/D conversions, store some data on some serial flash, respond to some buttons, and push data out to a character LCD and over an FT232 (a fairly generic embedded system), which chip should I use? It doesn't appear that the DSP will lag behind the PIC in any way, and it offers this mysterious "DSP Engine." My code always does math, and once in a while I need floating point or fractional numbers, but I don't know if I'll benefit from using a DSP.
A more general comparison between another vendor's DSPs and microcontrollers would be equally useful; I'm just using these as a starting point for discussion.
Best Answer
To be honest the line between the two is almost gone nowadays and there are processors that can be classified as both (AD Blackfin for instance).
Generally speaking:
Microcontrollers are integer math processors with an interrupt sub system. Some may have hardware multiplication units, some don't, etc. Point is they are designed for simple math, and mostly to control other devices.
DSPs are processors optimized for streaming signal processing. They often have special instructions that speed common tasks such as multiply-accumulate in a single instruction. They also often have other vector or SIMD instructions. Historically they weren't interrupt based systems and operated with non-standard memory systems optimized for their purpose making them more difficult to program. They were usually designed to operate in one big loop processing a data stream. DSP's can be designed as integer, fixed point or floating point processors.
Historically if you wanted to process audio streams, video streams, do fast motor control, anything that required processing a stream of data at high speed you would look to a DSP.
If you wanted to control some buttons, measure a temperature, run a character LCD, control other ICs which are processing things, you'd use a microcontroller.
Today, you mostly find general purpose microcontroller type processors with either built in DSP-like instructions or with on chip co-processors to deal with streaming data or other DSP operations. You don't see pure DSP's used much anymore except in specific industries.
The processor market is much broader and more blurry than it used to be. For instance i hardly consider a ARM cortex-A8 SoC a micro-controller but it probably fits the standard definition, especially in a PoP package.
EDIT: Figured i'd add a bit to explain when/where i've used DSPs even in the days of application processors.
A recent product i designed was doing audio processing with X channels of input and X channels of output per 'zone'. The intended use for the product meant that it would often times sit there doing its thing, processing the audio channels for years without anyone touching it. The audio processing consisted of various acoustical filters and functions. The system also was "hot plugable" with the ability to add some number of independent 'zones' all in one box. It was a total of 3 PCB designs (mainboard, a backplane and a plug in module) and the backplane supported 4 plug in modules. Quite a fun project as i was doing it solo, i got to do the system design, schematic, PCB layout and firmware.
Now i could have done the entire thing with an single bulky ARM core, i only needed about 50MIPS of DSP work on 24bit fixed point numbers per zone. But because i knew this system would operate for an extremely long time and knew it was critical that it never click or pop or anything like that. I chose to implement it with a low power DSP per zone and a single PIC microcontroller that played the system management role. This way even if one of the uC functions crashed, maybe a DDOS attack on its Ethernet port, the DSP would happily just keep chugging away and its likely no one would ever know.
So the microcontroller played the role of running the 2 line character LCD, some buttons, temperature monitoring and fan control (there were also some fairly high power audio amplifiers on each board) and even served an AJAX style web page via ethernet. It also managed the DSPs via a serial connection.
So thats a situation where even in the days where i could have used a single ARM core to do everything, the design dictated a dedicated signal processing IC.
Other areas where i've run into DSPs:
*High End audio - Very high end receivers and concert quality mixing and processing gear
*Radar Processing - I've also used ARM cores for this in low end apps.
*Sonar Processing
*Real time computer vision
For the most part, the low and mid ends of the audio/video/similar space have been taken over by application processors which combine a general purpose CPU with co-proc offload engines for various applications.