Skip to main content

Set up a Databricks Instance Store

This guide outlines the steps to set up an Ascend Instance Store with Databricks.

Prerequisites​

Create Databricks resources for the Ascend Instance​

Create a Databricks service principal for the Ascend Instance:

INSTANCE_SP_APP_ID=$(databricks service-principals create \
    --display-name "ascend-instance-sp" \
    | jq -r '.applicationId')
echo $INSTANCE_SP_APP_ID

In this step, you will run SQL commands to create a catalog and schema for the Ascend Instance and give permissions to the corresponding service principal.

tip

If you don't manage Unity Catalog resources and permissions in your Databricks workspace, reach out to your Databricks administrator to run the SQL commands below.

info

For ease of use purposes, we grant ALL PRIVILEGES on catalogs to ensure Ascend has the necessary permissions to operate without permission errors. In production environments, you may want to implement more granular permissions based on your security requirements.

danger

Ensure you are seeing "SUCCEEDED" as the status.state after running each SQL command below.

If you prefer, you can run these SQL commands in a query editor or notebook in the Databricks UI. Using the CLI avoids copying and pasting Databricks service principal application IDs and reduces the risk of error.

First, verify the current metastore is setup correctly:

SQL="select current_metastore()"
databricks api post /api/2.0/sql/statements --json \
    '{"warehouse_id": "'"$WH_ID_INSTANCE"'", "statement": "'"$SQL"'"}'

Set up Ascend Instance catalog and schema​

Set the variables for the Databricks Unity Catalog catalog and schema the Ascend Instance Store will use:

ASCEND_INSTANCE_CATALOG="ascend_instance_data"
ASCEND_INSTANCE_SCHEMA="instance_data"

Create a Databricks Unity Catalog catalog for the Ascend Instance Store to use:

SQL="CREATE CATALOG IF NOT EXISTS $ASCEND_INSTANCE_CATALOG"
databricks api post /api/2.0/sql/statements --json \
    '{"warehouse_id": "'"$WH_ID_INSTANCE"'", "statement": "'"$SQL"'"}'

And a schema:

SQL="CREATE SCHEMA IF NOT EXISTS $ASCEND_INSTANCE_CATALOG.$ASCEND_INSTANCE_SCHEMA"
databricks api post /api/2.0/sql/statements --json \
    '{"warehouse_id": "'"$WH_ID_INSTANCE"'", "statement": "'"$SQL"'"}'

Give the Ascend Instance's corresponding Databricks service principal access to the Databricks Unity Catalog catalog:

SQL="GRANT ALL PRIVILEGES ON CATALOG $ASCEND_INSTANCE_CATALOG TO \`$INSTANCE_SP_APP_ID\`"
databricks api post /api/2.0/sql/statements --json \
    '{"warehouse_id": "'"$WH_ID_INSTANCE"'", "statement": "'"$SQL"'"}'

And schema:

SQL="GRANT ALL PRIVILEGES ON SCHEMA $ASCEND_INSTANCE_CATALOG.$ASCEND_INSTANCE_SCHEMA TO \`$INSTANCE_SP_APP_ID\`"
databricks api post /api/2.0/sql/statements --json \
    '{"warehouse_id": "'"$WH_ID_INSTANCE"'", "statement": "'"$SQL"'"}'

Give Databricks service principals access to the Databricks compute​

tip

If you don't manage compute permissions in your Databricks workspace, reach out to your Databricks administrator to grant the service principals access to the compute resources.

In the Databricks UI, navigate to Compute > SQL warehouses, select the warehouse you created above, and add the service principal for the Ascend Instance as a user.

Create Ascend Vault secret for Databricks service principal​

  1. In the Databricks UI, navigate to Settings > Identity and access > Service principals. Click on the service principal for the Ascend Instance. Navigate to Secrets, click Generate secret, and copy the secret value.
  2. In the Ascend UI, navigate to Settings > Secrets & Vaults > Vault > Manage Secrets. Add the secret value as a new secret with the name app-password (or whatever you prefer).
tip

Keep the secret names as app-password for simple configuration that works with the Otto's Expeditions project.

Connect your Ascend Instance to Databricks​

In the Ascend UI, navigate to Settings > Instance > Instance Store and click Edit Instance Store. Choose Databricks and fill out the form using the following details:

WH_HOSTNAME=$(databricks warehouses get $WH_ID_INSTANCE | jq -r '.odbc_params.hostname')

echo "\n\n---\n"
echo "Server (Ascend Instance Store setup):"
echo "$WH_HOSTNAME"
echo "Service Principal Client ID (Ascend Instance Store setup):"
echo "$INSTANCE_SP_APP_ID"
echo "SQL Warehouse ID (Ascend Instance Store setup):"
echo "$WH_ID_INSTANCE"
echo "Catalog (Ascend Instance Store setup):"
echo "$ASCEND_INSTANCE_CATALOG"
echo "Schema (Ascend Instance Store setup):"
echo "$ASCEND_INSTANCE_SCHEMA"
echo "\n---"

🎉 Congratulations! Your Ascend Instance is now connected to Databricks.

Last updated on