Logging

DISCLAIMER: This post is not about log storage billing or managing log sinks.

Have you ever read or heard the phrase “Write everything to logs”? This is good advice. You never know what information can be useful or when. It is easy to do in Google Cloud. With help of audit logs all infrastructure, security and other cloud internal events are stored in Cloud Logging. And you can write application logs by simply printing them to stdout. However, there are situations when you may need to prevent some log entries from being stored:

Google Cloud provides efficient and not expensive storage for application and infrastructure logs. Logs stored in Google Cloud can be queried and analyzed using the analytical power of BigQuery. However, there are scenarios when Google Cloud customers may need to export log data from Google Cloud to third party (3P) solutions. This post reviews two main use cases of log exporting: exporting already stored logs and exporting logs while they are being ingested. The post focuses on how to configure and implement the part of the exporting process that handles extracting logs from Google Cloud. The part of loading the data into 3P solutions is not explored because of the variety of requirements and constraints that different 3P solutions expose.