Static file download from browser breaking in varnish but works fine in Apache

I would at first like to thank everyone at serverfault for this great website and I also come to this site while searching in google for various server related issues and setups.

I also have an issue today and so I am posting here and hope that the seniors would help me out. I had setup a website on a dedicated server a few days ago and I used Varnish 3 as the frontend to Apache2 on a Debian Lenny server as the traffic was a bit high. There are several static file downloads of around 10-20 MB in size in the website. The website looked fine in the last few days after I setup. I was checking from a 5mbps + broadband connection and the file downloads were also completed in seconds and working fine.

But today I realized that on a slow internet connection the file downloads were breaking off. When I tried to download the files from the website using a browser then it broke off after a minute or so. It kept on happening again and again and so it had nothing to do with the internet connection. The internet connection was around 512 kbps and so it was not dial up level speed too but decent speed where files should easily download though not that fast.

Then I thought of trying out with the apache backend port and used the port number to check out if the problem occurs. But then on adding the apache port in the static file download url, the files got downloaded easily and did not break even once. I tried it several times to make sure that it was not a coincidence but every time I was using the apache port in the file download url then it was downloading fine while it was breaking each time with the normal link which was routed through Varnish I suppose. So, it seems Varnish has somehow resulted in the broken file downloads.

Could anyone give any idea as to why it is happening and how to fix the problem.

For more clarification, take this example:

Apache backend set on port 8008, Varnish frontend set on port 80

Now when I download say

http://mywebsite.com/directory/filename.extension

Then the download breaks off after a minute or so. I cannot be sure it is due to the time or size though and I am just assuming. May be some other reason too.

But when I download using:

http://mywebsite.com:8008/directory/filename.extension

Then the file download does not break at all and it gets download fine.

So, it seems that varnish is somehow creating the file download breaking and not apache. Does anybody have any idea as to why it is happening and how can it be fixed. Any help would be highly appreciated.

And my varnish default.vcl is

    backend apache {
      set backend.host = "127.0.0.1";
      set backend.port = "8008";
    }

    sub vcl_deliver {
      remove resp.http.X-Varnish;
      remove resp.http.Via;
      remove resp.http.Age;
      remove resp.http.Server;
      remove resp.http.X-Powered-By;
    }