I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be "crawled".
My problem was simple: after a migration from a propietary, customised CMS to WordPress we had a lot of URLs that Google didn't find on the new site, and that went to 404 page. This is bad for our Pagerank and the searches, because Google still thinks those pages are alive.
We have a list of the URLs that don't work, and I tried to redirect to the good ones. The problem is, there are 20.000 of them, and there's no chance to solve the problem with a regular expression. We had to do 301 redirects ONE BY ONE, and that was a hell of a task.
But I was wondering: Could we just list those all bad URLs on our robots.txt with the "Disallow:" prefix, so Google does not index them? Is this a bad idea?
Best Answer
If Google thinks that your 404 page is valid then you need to be returning a 404 response code on that page. Fix that, and the rest will be fine.