Quickstart for BigQuery on Ascend

Scale up using BigQuery with Ascend.

Prerequisites

Before you begin, ensure you have the following:

• An Ascend Instance
• Google Cloud Platform (GCP) project: sufficient permissions to manage IAM roles
  • We recommend using an isolated GCP project per Ascend Environment

Grant your Ascend Environment access

You'll need to get your Ascend Environment's service account email and grant it sufficient BigQuery permissions.

warning

We use admin privileges here for simplicity. In production, consider using the principle of least privilege.

• Navigate to IAM & Admin > IAM in the GCP Console
• Select your GCP project for use with Ascend (top left)
• Click Grant access in the View by principals section
• Paste the Ascend Environment's service account email in the New principals field
• Click the Select a role dropdown and choose BigQuery admin
• Click Save to apply the changes

Create a BigQuery Data Plane Connection

Navigate to your Workspace. Click the + button next to Connections from the Super Graph. You should only need to fill out the GCP project ID field, then you can Test Read Connection.

tip

If you don't see the Connections section in the Resources tab of the Toolbar, you are likely in a Flow Graph. Use the search bar and type "Super Graph" or click back to all resources from the Toolbar (Resources tab).

Alternatively, create the file connections/data_plane_bigquery.yaml in your Project's files:

connection:
  bigquery:
    project: MY_PROJECT
    dataset: MY_PROJECT_DEVELOPMENT
    create_dataset_if_not_exists: true

Use BigQuery by default for all Flows

Ask Otto, or find and edit your ascend_project.yaml at the root of your Project's files. You can set the defaults section to use your new BigQuery Data Plane Connection, assuming you named it data_plane_bigquery, like:

project:
  ...
  defaults:
    - kind: Flow
      name:
        regex: .*
      spec:
        data_plane:
          connection_name: data_plane_bigquery
    - kind: Flow
      name:
        regex: .*-duckdb
      spec:
        data_plane:
          connection_name: data_plane_duckdb

Any Flows that end in -duckdb will still use the default Ascend-managed DuckDB Data Plane Connection.

Next steps

Try running the demo sales Flow or use Otto to create your first data pipeline with BigQuery on Ascend.