Getting started with BigQuery

We will stream our App Engine logs into BigQuery and query the data.

We will use the App Engine project created in #14 to store queries.

Setup Cloud Logging

1. Confirm that you have a running instance of your app at App Engine (opens in a new tab)
2. Visit Cloud Logging (opens in a new tab) and select your currently running application
3. Click 'Create Export' to create a dataset to write to
   1. Add a meaningful name
   2. Select 'BigQuery' for sink service
   3. Create a sink destination (or select one if already created)
   4. Create sink
4. Visit Exports (opens in a new tab) from the sidebar

Making queries

1. Visit BigQuery (opens in a new tab) where the dataset should be visible (you may need to select the correct project)
2. Create a few requests in your application
3. Back in BigQuery, click on the table generated by Google below the export name in the sidebar
4. In the table details click a few fields (only leaf nodes can be selected for the query) and make the query

Make a custom query

1. Get the top 10 requests ordered by latency

   SELECT protoPayload.resource, protoPayload.latency
   FROM <your-table-id>
   ORDER BY protoPayload.latency DESC
   LIMIT 10

2. Get all status codes and the number of times those status codes were returned in a request:

   SELECT
     protoPayload.status,
     COUNT(protoPayload.status) AS COUNT
   FROM
     [<table-di>]
   ORDER BY
     protoPayload.latency DESC
   GROUP BY
     protoPayload.status

Cleanup

1. Under Log Exports (opens in a new tab) delete the export
2. In BigQuery (opens in a new tab) delete your dataset

Make sure to also delete the compute instance and storage created in the earlier demos.