Is there a problem in disabling curl and wget

curlwget

Is it recommended to disable curl and wget to disallow anyone from accessing my websites using those commands?

Or is it needed for some search engines to access my websites?

What are your recommendations regarding curl and wget?

Best Answer

Depends what you want to do, and why - you could implement a ban based on user-agent, that's the easiest way, but it won't stop a determined "scraper", but then little will.

It may be that enforcing referrer checks could achieve what you want as well.

Google certainly doesn't scrape with a curl-ush or wget-ish useragent, so it shouldn't affect that.

What web server are you using? Why are you trying to achieve this? These answers whill help us help you :)

Related Solutions

Wget and pretty urls

Well, how wget is supposed to know if index.php/my/pretty is actually not a directory? This is not obvious at all from HTTP client's perspective.

Maybe you can wget --exclude-directories to work around this? Or maybe check wget -nd, which will create a flat set of files (not a directory tree). Check these out.

Ubuntu – Wget a script and run it

Careful

Before running the script, do you trust the person who wrote it?

For example, did you expect the script to contain this?

echo "brain1" > /etc/hostname

That will try to change your hostname.

For future reference, if, after verifying the script is correct and not malicious, you can run it in one line like this:

wget -O - http://dl.dropbox.com/u/11210438/flockonus-stack.sh | bash

But download it separately and read it before running it the first time.

Also note that interactive prompts inside the downloaded script may not work properly using this method.

Best Answer

Related Solutions

Wget and pretty urls

Ubuntu – Wget a script and run it

Related Topic