Skip to main content
Version: 3.0.0

Quickstart for Ascend on BigQuery

Introduction​

In this quickstart guide, you'll learn how to use Ascend with Google BigQuery. You will be guided through the process of:

  • Creating a GCP project
  • Connecting Ascend to BigQuery
  • Creating a developer Environment
  • Setting up a Git repository
  • Creating an Ascend Project
  • Creating your developer Workspace
  • Building & running your first flow

Prerequisites​

Create a new GCP Project​

  1. Set the following variables:

    # The ID of your GCP project
    PROJECT_ID="<project-id>"

    # If you don't know your billing account, run `gcloud billing accounts list`
    BILLING_ACCOUNT_ID="<billing-account-id>"

  2. Run the following commands:

    # Make sure you're logged in and have access
    gcloud auth login

    # Create your project
    gcloud projects create \
      --name="Ascend Quickstart on BigQuery" \
      ${PROJECT_ID}

    # Set your billing account... you'll need this to create a project with API access
    gcloud billing projects link \
      --billing-account=${BILLING_ACCOUNT_ID} \
      ${PROJECT_ID}

    # Set the project
    gcloud config set project ${PROJECT_ID}

Connect your Ascend Instance to BigQuery​

info

The Instance Store is a BigQuery dataset that your Ascend Instance will use to store operational data, such as logs, metrics, and other metadata.

It is kept separate from your project's data, and uses a different service account to ensure security.

warning

This quickstart grants elevated permissions for the sake of simplicity. For production use cases, we recommend isolating your Ascend Instance and each Environment's resources in separate GCP projects.

  1. In the Ascend UI, navigate to the Settings page by clicking on the instance name and your icon in the top right corner of the screen, then click on the Settings button.

  2. Click on the Instance settings tab.

  3. Copy the value for GCP Service Account Email listed under Instance Identity Details. Then set the ASCEND_INSTANCE_SERVICE_ACCOUNT variable by running the command below in your terminal. (Remember to replace <your-ascend-instance-gcp-service-account-email> with the email address you copied.) This is the email address of the service account that your Ascend Instance will use to access your BigQuery Instance Store.

    ASCEND_INSTANCE_SERVICE_ACCOUNT=<your-ascend-instance-gcp-service-account-email>

  4. Update your Instance Store settings

    • GCP Project ID: The ID of your GCP project.
    • Dataset: The name of the dataset you want to use for your Instance Store. We recommend using ASCEND_INSTANCE as the name.

  5. Click Get Setup Script and copy & paste the commands into your terminal. Alternatively, you can run the commands below (after setting the variables).

    # The GCP Project ID if you've not already done so
    PROJECT_ID=<project-id>

    # The name of the Instance Store dataset.
    INSTANCE_DATASET="ASCEND_INSTANCE"
    # Enable the BigQuery API
    gcloud services enable \
      bigquery.googleapis.com \
      secretmanager.googleapis.com \
      --project ${PROJECT_ID}

    # Create the Instance Store dataset
    bq mk \
      --dataset \
      --description="Ascend Instance Store" \
      ${PROJECT_ID}:${INSTANCE_DATASET}

    # Grant the BigQuery Job User role to the Ascend Instance service account
    gcloud projects add-iam-policy-binding \
      --member="serviceAccount:${ASCEND_INSTANCE_SERVICE_ACCOUNT}" \
      --role="roles/bigquery.admin" \
      --condition=None \
      ${PROJECT_ID}

  6. Click Check and Update Instance Store

Your Ascend Instance is now connected to BigQuery! 🎉 You can view the Instance Store dataset in the BigQuery UI or by running bq ls ${PROJECT_ID}:${INSTANCE_DATASET}.

Create your Dev Environment​

info

Each Ascend Instance can have multiple environments. Environments are secure, isolated spaces that physically separate your data and resources.

For this quickstart, we'll just create a single environment. In a real-world scenario, you'll want to create a common set of environments: Dev, Staging, and Prod. Each of these environments will have separate GCP Projects, BigQuery datasets, secrets, and service accounts.

  1. Create a GCP Service Account (such as ascend-quickstart-dev@<project-id>.iam.gserviceaccount.com)

    ENVIRONMENT_SERVICE_ACCOUNT_NAME="ascend-quickstart-dev"
    ENVIRONMENT_SERVICE_ACCOUNT_EMAIL="${ENVIRONMENT_SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

    gcloud iam service-accounts create \
      --display-name="Ascend Quickstart Dev" \
      --description="Service account for Ascend Quickstart Dev environment" \
      --project=${PROJECT_ID} \
      ${ENVIRONMENT_SERVICE_ACCOUNT_NAME}

  2. Grant the Service Account access to bigquery

    gcloud projects add-iam-policy-binding \
      --member="serviceAccount:${ENVIRONMENT_SERVICE_ACCOUNT_EMAIL}" \
      --role="roles/bigquery.admin" \
      --condition=None \
      ${PROJECT_ID}

  3. Create a JSON Key for the Service Account

    mkdir -p ~/.ascend
    gcloud iam service-accounts keys create \
      --iam-account=${ENVIRONMENT_SERVICE_ACCOUNT_EMAIL} \
      --project=${PROJECT_ID} \
      ${HOME}/.ascend/${ENVIRONMENT_SERVICE_ACCOUNT_NAME}.json

  4. Store the JSON Key in Secret Manager (such as ascend-quickstart-dev-service-account)

    ENVIRONMENT_SERVICE_ACCOUNT_SECRET_NAME="${ENVIRONMENT_SERVICE_ACCOUNT_NAME}-service-account"

    # Enable Secret Manager
    gcloud services enable \
      secretmanager.googleapis.com \
      --project ${PROJECT_ID}

    # Create the Secret Manager Secret
    gcloud secrets create \
      --project ${PROJECT_ID} \
      ${ENVIRONMENT_SERVICE_ACCOUNT_SECRET_NAME}

    # Add the JSON Key to the Secret Manager Secret
    gcloud secrets versions add \
      --project ${PROJECT_ID} \
      --data-file=${HOME}/.ascend/${ENVIRONMENT_SERVICE_ACCOUNT_NAME}.json \
      ${ENVIRONMENT_SERVICE_ACCOUNT_SECRET_NAME}
    Coming Soon!

    Environment secrets will be available shortly, removing the requirement to use GCP Secret Manager.

  5. Grant Ascend access to Google Secret Manager

    1. Navigate to your Ascend Settings (top right with your instance name and user icon)

    2. Click Environments

    3. Select the Dev environment

    4. Copy the GCP Service Account Email for the environment

    5. Run the script below:

      ASCEND_ENVIRONMENT_SERVICE_ACCOUNT_EMAIL=<ascend-environment-gcp-service-account-email>
      gcloud projects add-iam-policy-binding \
        --member="serviceAccount:${ASCEND_ENVIRONMENT_SERVICE_ACCOUNT_EMAIL}" \
        --role="roles/secretmanager.viewer" \
        --condition=None \
        ${PROJECT_ID}
      gcloud projects add-iam-policy-binding \
        --member="serviceAccount:${ASCEND_ENVIRONMENT_SERVICE_ACCOUNT_EMAIL}" \
        --role="roles/secretmanager.secretAccessor" \
        --condition=None \
        ${PROJECT_ID}

  6. In the Ascend UI, update the Environment Vault from above

    1. Click Edit Vault
    2. Set GCP Project ID to your project ID
    3. Click Check and Update Vault
Helpful Links

If you need to go back and look at the GCP resources you created, here are some helpful links:

Set up a Git Repository​

When you develop in Ascend, you will need to provide a Git repository in which to store your code. This step will guide you through how to set up a Git repository and add it to your Ascend Instance.

  1. Create a new repository
  • Navigate to GitHub's New Repository page
  • Set Owner to either your personal GitHub account, or your organization.
    tip

    If you have the appropriate permissions to your company's GitHub Organization, you can create a new repository within your organization. If you do not have admin access, you can create a new repository within your personal account, which will show up in the list as your GitHub name.

  • Set Repository name to ascend-quickstart-bigquery (or your preferred name)
  • Set Description to Ascend Quickstart BigQuery
  • Set Public/Private to Private
  • Click Create repository

  1. Next, we'll clone the Ascend Community Repository and push it to your new repository.

    # Set the variables
    GITHUB_ORG="<your-github-org>" # If you are using your personal account, set this to your username
    GITHUB_REPO="ascend-quickstart-bigquery"
    # Clone the Ascend Community Repository
    git clone git@github.com:ascend-io/ascend-community.git ${GITHUB_REPO}
    cd ${GITHUB_REPO}

    # Set the remote URL to your new repository
    git remote set-url origin git@github.com:${GITHUB_ORG}/${GITHUB_REPO}.git

    # Push the code to your new repository
    git push -u origin main

Connect Ascend to your Git Repository​

Create the SSH key pair​

  1. On your local machine, open a terminal and run the following command:
mkdir -p ~/.ssh && ssh-keygen -t ed25519 -f ~/.ssh/ascend_quickstart_repo_key -q -N ""
  1. This command will generate a new SSH key pair in the ~/.ssh directory.
  2. The public key will be saved in ~/.ssh/ascend_quickstart_repo_key.pub
  3. The private key will be saved in ~/.ssh/ascend_quickstart_repo_key.
    tip

    You can copy the contents of the public key to your clipboard by running:

    cat ~/.ssh/ascend_quickstart_repo_key.pub | pbcopy

Add the public key to your repository​

Please follow the instructions here to setup the public ssh key in your Github repository.

Add the private key to Ascend​

  1. In the Ascend UI, navigate to the instance settings by clicking on the button in the top right with your instance name and your user icon.
  2. Click on the Repositories settings tab.
  3. Click the Add Repository button.
  4. Set Repository URL to the URL of your repository. This should be in the format git@github.com:<github-org>/<github-repo>.git.
  5. Set SSH Private Key to the private key you generated in the previous step.
  6. Click Create.

Create an Ascend Project​

With your Git Repository connected, you can now create an Ascend Project.

  1. In the Ascend UI, navigate to the instance settings by clicking on the button in the top right with your instance name and your user icon.
  2. Click on the Projects settings tab.
  3. Click the Add Project button.
  4. Set Title to Ascend Quickstart BigQuery
  5. Select Repository from the pulldown options.
  6. Set Project Root to 'ottos_expeditions/'
  7. Click Save.

Create your developer Workspace​

Now, let's create a workspace for us to develop in.

  1. In the Ascend UI, navigate to the instance settings by clicking on the button in the top right with your instance name and your user icon.
  2. Click on the Workspaces settings tab.
  3. Click the Add Workspace button.
  4. Set Title to Ascend Quickstart BigQuery
  5. Select Environment as Dev
  6. Set Project to the project you created above (Ascend Quickstart BigQuery)
  7. Set Git Branch to my-first-branch (You can type the branch name and click "New Branch:my-first-branch" to create it if it doesn't already exist)
  8. Set Profile as prod
  9. Select Size as X-Small
  10. Set Storage to 8GB
  11. Click Save.

Build & run your first flow​

Modify your flow and run it again​

Deploy your Project​

Last updated on