What should I be looking for, when wget returns 500 but the same URL works fine in my web browser? I don't see any access_log entries that seem to be related to the error.
DEBUG output created by Wget 1.14 on linux-gnu.
<SSL negotiation info stripped out>
---request begin---
GET /survey/de/tools/clear-caches/password/<some-token> HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: testing.thesurveylab.net
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 500 Internal Server Error
Date: Wed, 12 Dec 2012 14:53:07 GMT
Server: Apache/2.2.3 (CentOS)
Set-Cookie: blueprint2-staging=8jnbmkqapl30hjkgo0u6956pd1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Strict-Transport-Security: max-age=8640000;includeSubdomains
X-UA-Compatible: IE=Edge,chrome=1
Content-Length: 5
Connection: close
Content-Type: text/html; charset=UTF-8
---response end---
500 Internal Server Error
Stored cookie testing.thesurveylab.net -1 (ANY) / <session> <insecure>
[expiry none] blueprint2-staging 8jnbmkqapl30hjkgo0u6956pd1
Closed 3/SSL 0x0000000001f33430
2012-12-12 15:53:07 ERROR 500: Internal Server Error.
Best Answer
There are probably a gazillion possible reasons for this but whenever I've struck the same issue, whether it be using wget or some other downloader, it's usually because there is code on the site which is designed to respond in certain ways to "normal" web browsers. Along comes wget, or similar, and the code breaks because it doesn't have code for it, nor a proper fallback. Such site usually break for very old browsers as well. In such cases you have to get the downloader to mimic a common browser to get it to work correctly.