Logging – How to Extract Raw Plain Text Logs from Stackdriver

google-cloud-platformgoogle-kubernetes-enginegoogle-stackdriverlogging

We need to extract the logs of GKE-hosted reverse proxies in a raw format that can be parsed by SEO log analysis tools. Unfortunately these tools flat out refuse to ingest CSV and JSON files, they only accept raw text as it would be in .log files generated by Nginx on a physical server.

Downloading them from the GCP GUI doesn't work, as it only allows JSON and CSV exports. The exports GCP allows do not really work either – we have been able to save plaintext logs in google storage using a cloud function script, but only in an inconvenient way (1 file per hour, seemingly no way to change this), and only for future logs, as log export only apply to incoming logs.

What we need is to extract plaintext logs from the last month, and there seems to be no documented and/or supported way to do this.

Best Answer

Sorry to answer my own question, but I spent a lot of time researching this and hopefully by posting it here I can save others the hassle.

The only way to achieve what I need seems to be use google cloud sdk from the command line and dump the result into a local text file, like so:

$ gcloud logging read 'resource.type=container resource.labels.cluster_name=your_cluster logName=the_specific_deployment_you_want_logs_from timestamp>="2015-05-01T00:00:00Z" (whatever text search filters are needed)' --limit 1000000000000 --order asc --format "value(textPayload)" > total.log
Related Topic