C# – Server cannot append header after HTTP headers have been sent c# pdf inline

So I have been having this problem for a while now for trying to return a pdf inline response. I searched all through the help coding pages for answers to this and can not find something that works yet. I made a simplified example of what I am doing below. But the core is all the same as real production. This issue doesn't always happen, but its annoying when it does. It's almost like it's some weird timing issue or something else I am not seeing is going on.

I get this error System.Web.HttpException (0x80004005): File Not Found —> System.Web.HttpException (0x80004005): Server cannot append header after HTTP headers have been sent.

Does anyone know how to fix this??

    [HttpGet]
    public ActionResult PDF(string id, bool inline = false)
    {
        try
        {
            MemoryStream PDFStream = GetPdfStream(id);

            PDFStream.Position = 0;

            string PDFFile = "Test.pdf";


            if (inline)
            {
                Response.AppendHeader("Content-Disposition", string.Format("inline; filename={0}", PDFFile));
                Response.BufferOutput = true;
                return File(PDFStream, MediaTypeNames.Application.Pdf);
            }
            else
                return File(PDFStream, MediaTypeNames.Application.Pdf, PDFFile);
        }
        catch (Exception Ex)
        {
            throw new HttpException(404, "File Not Found", Ex);
        }
    }