304 with CORS on apache

I have an REST API that returns a 304 Not Modified status code for some request (that have If-Modified-Since header). The problem is that the apache2 software strips any CORS header prior to the response being sent to the browser.

This happens only when the status code is 304. Any other end-point works great with CORS. All the pre-flight requests work also great.

I have read on the internet that Apache does this in order to comply with some specs, but I can't believe that CORS with 304 is not supposed to work in specs.

Is there a way I could achieve this with apache?

Update:

My javascript is

var xhr = new XMLHttpRequest();
xhr.open('GET', 'http://api.domain.com/api/endpoint?token='+localStorage.getItem('token'));
xhr.setRequestHeader("If-Modified-Since", "Mon, 11 Jan 2016 15:46:54 GMT");
xhr.send(null);

I get an error:

No 'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://xxx.domain.com' is therefore not allowed
access.