I would use a standardized body and not use the header. For example, every body can be JSON and include an ID field. As long as it's standardized across your platform it's guaranteed to work.
I would not use a custom header because of the risk of it getting stripped. I would not use a standard header because I don't know of one that fits perfectly (e.g. the ETag you mention is used to determine file changes and caching keys.)
Don't invent status codes
You are not expected to invent your own response codes, since the point of the API is to use a standard interface any developer can understand.
The fact that you maintain both the API and its client is irrelevant: since everyone can trace the calls to the API, everyone can implement a different client. The point of using standard interfaces is also:
To facilitate maintenance. If your app should later be maintained by your coworker, he can find his way with ease. He will be lost if it appears that HTTP 306 means success, HTTP 500 is a “Not found” and HTTP 909 is “Internal Server Error”, unless the error is a FileNotFound exception, in which case it's HTTP 404.
To facilitate the work of your system administrator. Tools which deal with server logs use status codes to determine whether this is a error or not. Two concrete examples are:
The displaying of the number of errors (HTTP 4xx and HTTP 5xx) for a given server and:
The tracking of hacking attempts. For instance, when I see a hacking attempt which results in HTTP 200, this immediately attracts my attention: it should be HTTP 4xx instead.
To avoid messing with your HTTP server. IIS, for instance, can easily be offended by some status codes, will catch them and generate its own responses. Configuring both your app and IIS is not always easy (and I've spent several days banging my head with this issue when I started programming ASP.NET websites).
To avoid problems with proxies. For instance, if your API is hosted on a server accessed through Nginx, I'm not sure sysadmins will be happy to spend a few hours redoing all the configuration for your app.
A basic example. When configuring failover, I use proxy_next_upstream directive which indicates in which cases Nginx should switch to the failover machine. Looking at the documentation, I have an impression that you can't set the directive to, say, HTTP 200, so if your API uses HTTP 200 as “Fatal error, all data was corrupted so you may be better using another mirror”, you're out of luck.
To avoid reinventing the wheel. Why would you waste your time standardizing your set codes, while there is already one?
To leverage support from many tools. For instance, Fiddler relies on HTTP response code to colorize the items. I'm not sure if it's easy to configure it to use different statuses. In the same way, different API testing tools may rely on status codes as well (while CURL, conveniently, couldn't care less about the response status, Python's requests package, for instance, will do a redirection in a presence of HTTP 3xx code).
By the way, browsers themselves interpret the response code:
Provide additional information
The response code other than HTTP 200 doesn't mean that you can't feed the client with JSON as well (unless it's HTTP 204, in which case there should be no content). This means that you're free to give as much information as you want to the client through JSON. Personally, I include:
The ID of the error in a form of a string. For instance price-range-invalid or product-not-found.
The URI containing additional help, when relevant (when the API is large enough to contain individual pages for every error message). For instance http://example.com/api/v1/errors/price-range-invalid.
The description. For instance “The current price is outside the allowed range. It should be superior to 0 and inferior or equal to 50000.”.
See also how other APIs handle the errors and what sort of information they provide. For instance, this is how Google+ API errors look like:
HTTP 403
{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "ipRefererBlocked",
"message": "There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed.",
"extendedHelp": "https://console.developers.google.com"
}
],
"code": 403,
"message": "There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed."
}
}
Don't provide too much information
Finally, make sure you don't include the exception itself in the details, since it may contain sensitive information and represent a path for a hacker to explore (including the stack trace).
For instance, some PHP websites, when encountering a database related error, just show the SQL query and the error to the user. For a hacker, this is very convenient. For a legitimate user, this is just unfriendly and unhelpful. Don't do that.
Best Answer
Any HTTP error code would be inappropriate. There is no error or problem of any sort from an HTTP perspective so it should be something in the 200 range. You politely inform some of your users that they will not be serviced by sending back a document that tells them so. And this all goes well.
The user will not be able to use your application. That is a conscious decision made by your business logic, not a mishap. On the HTTP level everything is honky dory.
Edit
It looks like what we are looking at here is a clash of old school versus new school. When HTTP was designed, there were no web services, there was no SOAP, no JSON, no REST principles. As a protocol above TCP this was already considered (close to) application level and many high level status codes were defined. When the web started to be used for richer, high level services and a common means to transport "envelopes" was required, designers hi-jacked HTTP rather than defining a newer and cleaner protocol, just because HTTP was ubiquitous.
So in a modern web service context, HTTP is indeed little more than a dumb transport layer and most of its codes may be considered not applicable or obsolete. Just picking one because it comes close to your application state and happens to be in that list that once meant something may seem harmless, but I think it would send a wrong message. You do not want HTTP to play that regulating role in a web service context.