Getting started with BigQuery
We will stream our App Engine logs into BigQuery and query the data.
We will use the App Engine project created in #14 to store queries.
- Confirm that you have a running instance of your app at App Engine (opens in a new tab)
- Visit Cloud Logging (opens in a new tab) and select your currently running application
- Click 'Create Export' to create a dataset to write to
- Add a meaningful name
- Select 'BigQuery' for sink service
- Create a sink destination (or select one if already created)
- Create sink
- Visit Exports (opens in a new tab) from the sidebar
- Visit BigQuery (opens in a new tab) where the dataset should be visible (you may need to select the correct project)
- Create a few requests in your application
- Back in BigQuery, click on the table generated by Google below the export name in the sidebar
- In the table details click a few fields (only leaf nodes can be selected for the query) and make the query
Get the top 10 requests ordered by latency
SELECT protoPayload.resource, protoPayload.latency FROM <your-table-id> ORDER BY protoPayload.latency DESC LIMIT 10
Get all status codes and the number of times those status codes were returned in a request:
SELECT protoPayload.status, COUNT(protoPayload.status) AS COUNT FROM [<table-di>] ORDER BY protoPayload.latency DESC GROUP BY protoPayload.status
- Under Log Exports (opens in a new tab) delete the export
- In BigQuery (opens in a new tab) delete your dataset
Make sure to also delete the compute instance and storage created in the earlier demos.