Google Big Query

Prerequisites

---

In order for you to be able to export data from Big Query, you will need to ensure that the service account has permission to access the table, export data, create/list jobs and also create objects in the GCS bucket. This will require the following individual permissions:

• bigquery.tables.export
• bigquery.jobs.create
• storage.objects.create
• storage.objects.delete

Alternatively, you can apply these predefined IAM roles:

• bigquery.dataViewer
• bigquery.jobUser
• storage.objectAdmin

Set up

---

Whitelisting

You will need to whitelist the following IPs:

• 54.78.204.135
• 34.242.207.164

Connection

Running Extracts

---

When running an extract you will be asked to select a schema, then a table to extract.

You will also have the option to select the fields you wish to pull back.

🚧

Changing selected fields

If you change the fields you have selected after the extract has been running, any further extracts on append will fail, due to the column mismatch.

If you would like to add new fields, please run a new extract on Drop, Create, Load, to a new destination table. You can then set your extract to append onto this new table.

Please then select your load method and Extract Type.

If you select incremental, you will need to select your incremental key, to see how incremental works, you can head here.

Lastly, in order for Kleene to successfully pull the data without any duplicates or missing rows, we have selected the primary key, if for any reason this primary key selection is wrong or empty, please override it correctly.

Limitations

• You cannot export nested and repeated data
• When you export data to a Cloud Storage bucket configured with a retention policy, BigQuery might fail to write the files to the bucket. Consider relaxing the retention policy for the duration of the export jobs.
• When you export data, the regional or multi-regional Cloud Storage bucket must be in the same location as the BigQuery dataset. For example, if your BigQuery dataset is in the EU multi-regional location, the Cloud Storage bucket containing the data you're exporting must be in a regional or multi-regional location in the EU.
• If your dataset is in a regional location, your Cloud Storage bucket must be a regional bucket in the same location. For example, if your dataset is in the Tokyo region, your Cloud Storage bucket must be a regional bucket in Tokyo.
• Exception: If your dataset is in the US multi-regional location, you can export data into a Cloud Storage bucket in any regional or multi-regional location.