Many people seem to be afraid of mixing stable with testing, but frankly, testing is fairly stable in its own right, and with proper preferences and solution checking, you can avoid the "stability drift" that puts your core packages on the unstable path.
"Testing is fairly stable??", you ask. Yes. In order for a package to migrate from unstable to testing, it has to have zero open bugs for 10 consecutive days. Chances are that, especially for the more popular packages, somebody is going to submit a bug report for an unstable version if something is wrong.
Even if you don't want to mix the environments, it's still nice to have the option there in case you run into something that requires a newer version than what is in stable.
Here's what I recommend for setting this up:
First, create the following files in /etc/apt/preferences.d:
stable.pref:
# 500 <= P < 990: causes a version to be installed unless there is a
# version available belonging to the target release or the installed
# version is more recent
Package: *
Pin: release a=stable
Pin-Priority: 900
testing.pref:
# 100 <= P < 500: causes a version to be installed unless there is a
# version available belonging to some other distribution or the installed
# version is more recent
Package: *
Pin: release a=testing
Pin-Priority: 400
unstable.pref:
# 0 < P < 100: causes a version to be installed only if there is no
# installed version of the package
Package: *
Pin: release a=unstable
Pin-Priority: 50
experimental.pref:
# 0 < P < 100: causes a version to be installed only if there is no
# installed version of the package
Package: *
Pin: release a=experimental
Pin-Priority: 1
(Don't be afraid of the unstable/experimental stuff here. The priorities are low enough that it's never going to automatically install any of that stuff. Even the testing branch will behave, as it's only going to install the packages you want to be in testing.)
Now, creating a matching set for /etc/apt/sources.list.d:
stable.list: Copy from your original /etc/apt/sources.list. Rename the old file to something like sources.list.orig.
testing.list: Same as stable.list, except with testing.
unstable.list: Same as stable.list, except with unstable, and remove the security lists.
experimental.list: Same as unstable.list, except with experimental.
You can also add a oldstable in sources.lists.d and preferences.d (use a priority of 1), though this moniker will tend to expire and disappear before the next stable cycle. In cases like that, you can use http://archive.debian.org/debian/ and "hardcode" the Debian version (etch, lenny, etc.).
To install the testing version of a package, simply use aptitude install lib-foobar-package/testing, or just jump into aptitude's GUI and select the version inside of the package details (hit enter on the package you're looking at).
If you get complaints of package conflicts, look at the solutions first. In most cases, the first one is going to be "don't install this version". Learn to use the per-package accept/reject resolver choices. For example, if you're installing foobar-package/testing, and the first solution is "don't install foobar-package/testing", then mark that choice as rejected, and the other solutions will never veer to that path again. In cases like these, you'll probably have to install a few other testing packages.
If it's getting too hairy (like it's trying to upgrade libc or the kernel or some other huge core system), then you can either reject those upgrade paths or just back out of the initial upgrade altogether. Remember that it's only going to upgrade stuff to testing/unstable if you allow it to.
EDIT: Fixed some priority pins, and updated the list.
Best Answer
Will remove a package and all configuration files.
If it is still giving you issues, try reinstalling then purging:
You still might need to:
Reboot and if it started again run:
Story Time: Once for some unknown reason, when I purge removed ejabberd; it continued running. I trusted apt-get to kill it and remove it. It didn't. Needless to say, it eventually filled up my entire /var with a ramdisk log file. It continued until I killed all of the ejabberd processes and that quickly resolved the issue. So it taught me to always be sure that apt-get worked and killed all needed processes.