Massive Crawling requests from Google Apps Engine useragent

google-app-enginegoogle-cloud-platform

I'm badly affected with 'Google AppEngine-Google' UserAgent.. receiving 5/6 requests per second on http server. This bot is crawling my site just like GoogleBot does.

Following is the sample of url in my access logs.

72.14.192.3 – – [19/May/2010:01:27:06 +0000] "GET /some-url/etc-123.htm HTTP/1.1" 200 4707 "-" "AppEngine-Google; (+http://code.google.com/appengine; appid: harpy000)"

I have checked the ip address it is registered with Google Inc.

Can anyone tell me where i can report Abuse to Google Inc. Or any information about this issue.

Thank you!

Best Answer

You can report Google App Engine abuse using the following form: http://code.google.com/support/bin/request.py?contact_type=AppEngineContact. The App has a web page at this address: http://harpy000.appspot.com/. There is also a link to a Twitter page (the owner?), maybe you can contact him.

Related Solutions

Using Google App Engine for static files

The documentation on how to get GAE to serve static content is here: Using Static Files.

It's rather simple, you just define a static directory in your app.yaml file.

handlers:
- url: /stylesheets
  static_dir: stylesheets

Google App Engine – How to Troubleshoot 500 Internal Server Error

The previous answers are out of date now (some three years later). Look for the Error Reporting panel in the google cloud platform dashboard.

You will get a stack trace to the problem in your code (if the problem is in your code and I suppose you probably need to be logging errors to std err). A stack trace should help you debug a 500 error. Here's a stack trace that I just received ...

ImportError: No module named flask_assistant

at <module> (/base/data/home/apps/f~XXXX/20180209t180735.407529010683953771/main.py:31)
at LoadObject (/base/alloc/tmpfs/dynamic_runtimes/python27/a7637d5531ec9deb_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:85)
at _LoadHandler (/base/alloc/tmpfs/dynamic_runtimes/python27/a7637d5531ec9deb_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:299)
at Handle (/base/alloc/tmpfs/dynamic_runtimes/python27/a7637d5531ec9deb_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:240)

That's how to get a stack trace in the event of a 500 error.

Best Answer

Related Solutions

Using Google App Engine for static files

Google App Engine – How to Troubleshoot 500 Internal Server Error

Related Topic