Domain – GoDaddy DNS point the Sub-domain to another Server?

First: @ is a standard "master file format" thing. It means "the current zone."

Your problem is simple. Add a '.' to the last line. that is, you want:

* IN CNAME domain.com.

The tailing '.' at the end of the name means "do not append the current origin name to this"

If you leave it off, you get (in this case) "domain.com.domain.com."

OK, first things first: Go to you local book store (or library if it's got a decent selection of technical books) and pick up a copy of DNS & BIND - any edition will be sufficient, though if you're buying one buy the latest. Then read this book cover to cover, or at least read through chapters 1, 2, 3, 5 and 6.

I am absolutely serious about this - If you try to set up DNS without a solid understanding of what's going on you are in for a wold of pain, suffering and mysterious breakage. Spending a day with a good book on DNS will pay for itself the first time you have a problem.

Now to actually answer your questions :-)

Re: the issue of being flagged as spam: Hosting your own DNS may or may not solve the problem of your system being flagged as a spam source -- The question you haven't asked/answered is WHY you are being flagged as a spam source (Is it your IP, the lack of SPF records, a bad reverse-DNS entry, or is your server perhaps misconfigured and really being used to send spam?).
You need to answer that question first, then pursue solutions based on what you discover.

If after investigating the incident problem you still want to host your own DNS (either as part of a solution to the incident problem, or just for the experience) . . .

I'm pretty sure the error you're getting from GoDaddy is their way of saying you're missing glue records -- In plain English "You want us to use ns1.xxx.com as a nameserver for xxx.com, but we have no way of finding that server".

Prior to using a host within in your own zone as an NS you need to create glue records for it. You can do this in GoDaddy's domain manager's "Host" box -- Add a domain host for each NS you want to use, and the system should then let you specify those hosts as the domain's nameservers.

BIG IMPORTANT WARNING
Before you flip the switch check, double-check and triple-check that the servers you're about to set as the authoritative NS are working properly (they resolve all the names they're supposed to resolve, you can query them from a machine off your network, etc.).

Many admins (probably every one who has ever set up DNS) probably has a horror story about screwing up and knocking their domain off the internet for a while -- Don't be a statistic like the rest of us :-)