Instance Store
Your Instance Store is where Ascend stores instance level data such as flow runs, tasks, and logs.
- Snowflake Instance Store
- BigQuery Instance Store
Prerequisites​
- You have setup an Instance Vault in Ascend.
- You have access to a Snowflake account.
- You have SYSADMIN and SECURITYADMIN roles available to the user executing the setup script.
Creating your Snowflake Instance Store​
- In the Ascend UI, navigate to the instance settings (Ascend.io Instance: <instance-name> -> Settings -> Instance)
- Under Instance Store, select Snowflake
- Fill in the following fields with the Snowflake resources you want to use for your Instance Store. These values will be used to generate the setup script you will run in the next step:
- Account - Your Snowflake Account Identifier.
- User - The user to create to access the Instance Store.
- Secret name in Vault for the password - The name of the secret in your Vault that contains the password for your user. We will add the password to your Vault using this name in the next step. You can use the value
ascend-instance-snowflake-password
as the secret name. - Database - The database to create in Snowflake for your Instance Store.
- Schema - The schema to create in Snowflake.
- Warehouse - The warehouse to use for your Instance Store.
- Role - The role you will create to access your Instance Store Database.
- Click
Get Setup Script
button to generate the Snowflake script to set up the needed resources for your Instance Store.
What does this script do?
The generated setup script will use the values populated in the previous step to create the necessary resources for your Instance Store.
- Set some variables to be used in the next steps
SET WAREHOUSE='<your-snowflake-warehouse>';
SET INSTANCE_DB='<your-snowflake-database>';
SET INSTANCE_FULL_SCHEMA=$INSTANCE_DB||'.'||'<your-snowflake-schema>';
SET INSTANCE_USER='<your-snowflake-user>';
SET INSTANCE_ROLE='<your-snowflake-role>';
- Switch to the SYSADMIN role and create the database, schema, and warehouse that will be used as the Instance Store.
USE ROLE SYSADMIN;
CREATE DATABASE identifier($INSTANCE_DB);
CREATE WAREHOUSE identifier($WAREHOUSE) WAREHOUSE_SIZE='XSMALL' WAREHOUSE_TYPE='STANDARD' AUTO_SUSPEND=60 AUTO_RESUME=TRUE INITIALLY_SUSPENDED=TRUE;
CREATE SCHEMA identifier($INSTANCE_FULL_SCHEMA);
- Switch to the SECURITYADMIN role and create the role and user that will be used to access the Instance Store, also grants the necessary permissions.
USE ROLE SECURITYADMIN;
CREATE ROLE identifier($INSTANCE_ROLE);
CREATE USER identifier($INSTANCE_USER) PASSWORD='<CHANGE ME>' DEFAULT_ROLE=$INSTANCE_ROLE;
GRANT ROLE identifier($INSTANCE_ROLE) TO USER identifier($INSTANCE_USER);
GRANT ALL ON WAREHOUSE identifier($WAREHOUSE) TO ROLE identifier($INSTANCE_ROLE);
GRANT ALL ON DATABASE identifier($INSTANCE_DB) TO ROLE identifier($INSTANCE_ROLE);
GRANT ALL ON SCHEMA identifier($INSTANCE_FULL_SCHEMA) TO ROLE identifier($INSTANCE_ROLE);
GRANT OWNERSHIP ON ALL TABLES IN SCHEMA identifier($INSTANCE_FULL_SCHEMA) TO ROLE identifier($INSTANCE_ROLE);
GRANT OWNERSHIP ON FUTURE TABLES IN SCHEMA identifier($INSTANCE_FULL_SCHEMA) TO ROLE identifier($INSTANCE_ROLE);
- In your Snowflake console, create a new worksheet, copy in your generated script. In the script, you will need to replace the placeholder with a password of your choice.
CREATE USER identifier($INSTANCE_USER) PASSWORD='<CHANGE ME>' DEFAULT_ROLE=$INSTANCE_ROLE;
You will add it to your Instance Vault in the next step.
- Run all the commands to create your Instance Store resources.
Adding your Snowflake password to your Instance Vault​
In order for Ascend to access your Snowflake Instance Store, we need to add the password to your Instance Vault.
- Azure Key Vault
- GCP Secret Manager
- In your terminal, ensure you have the Azure CLI installed. Run
az login
to log into the subscription where you set up your Instance Vault.
The following commands can be found in the comments of the generated Snowflake script, already populated with your Azure vault details.
- If you don't already have write access to your vault, you can grant it with the following command:
az role assignment create \
--role "Key Vault Secrets Officer" \
--assignee "$(az ad signed-in-user show --query id -o tsv)" \
--scope "/subscriptions/<your-instance-subscription-id>/resourcegroups/<your-instance-resource-group>/providers/Microsoft.KeyVault/vaults/<your-instance-vault-name>";
- Create a secret in your vault:
az keyvault secret set --vault-name "<your-instance-vault-name>" --name "ascend-instance-snowflake-password" --value "<your-snowflake-password>"
- In your terminal, ensure you have the GCP CLI installed. Run
gcloud auth login
to log into your GCP account. - If you don't already have write access to your project, you can grant it with the following command:
gcloud projects add-iam-policy-binding <your-gcp-project-id> \
--member="user:<your-email>" \
--role="roles/secretmanager.admin"
- Set your GCP Project ID and the name of the Secret Key to store your Snowflake password.
ENV_VAULT_PROJECT_ID=<your-instance-project-id>
ENV_VAULT_SECRET_NAME=ascend-instance-snowflake-password
- Create a secret in your project:
gcloud secrets create $ENV_VAULT_SECRET_NAME --data-file=<your-snowflake-password> --project $ENV_VAULT_PROJECT_ID
- Add your Snowflake password as the value for the secret:
echo "<your-instance-snowflake-password>" | gcloud secrets versions add $ENV_VAULT_SECRET_NAME --project $ENV_VAULT_PROJECT_ID --data-file=-
Validating your Instance Store Connection​
- Go back into the Instance Settings page where you entered your Instance Store details.
- Click Check and Add Instance Store to validate your connection.
- If the connection is successful, you will see a message indicating that the Instance Store was added successfully.
Prerequisites​
- You have setup an Instance Vault in Ascend.
- You have a Google Cloud Project where you will create your BigQuery Instance Store.
- You have the
Owner
role, or at least have the following roles:serviceusage.serviceUsageAdmin
bigquery.admin
resourcemanager.projectIamAdmin
Provision BigQuery Instance Store​
- In the Ascend UI, navigate to the instance settings (Ascend.io Instance: <instance-name> -> Settings -> Instance)
- Under Instance Store, select BigQuery
- Fill in the following fields. These values will be used to generate the setup script you will run in the next step:
- GCP Project ID - The Google Cloud Project ID where you want to create your BigQuery Instance Store.
- Dataset - The BigQuery Dataset to create for your Instance Store.
- Click Get Setup Script button. This will pop up a window with the generated script to create your BigQuery Instance Store, and grant your Ascend Instance access to it.
What does the generated script do?
- Sets some environment variables to be used in the next steps, and enables the BigQuery API (if it is not already enabled).
GCP_PROJECT_ID=<your-gcp-project-id>
BQ_DATASET=<your-bigquery-dataset-id>
GCP_SERVICE_ACCOUNT=<ascend-instance-service-account-email>
gcloud config set project $GCP_PROJECT_ID
gcloud services enable bigquery.googleapis.com
- Creates the dataset that will be used as the Instance Store.
bq mk \\
--dataset \\
--description="Ascend Instance Store" $GCP_PROJECT_ID:$BQ_DATASET
- Grants the necessary permissions to the Ascend Instance service account, it gives access to run jobs in bigquery and access to the dataset created in the previous step.
gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \\
--member="serviceAccount:$GCP_SERVICE_ACCOUNT" \\
--role="roles/bigquery.jobUser" \\
--condition=None
gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \\
--member="serviceAccount:$GCP_SERVICE_ACCOUNT" \\
--role="roles/bigquery.dataEditor" \\
--condition="title=dataset:$BQ_DATASET,expression=resource.name.startsWith('projects/$GCP_PROJECT_ID/datasets/$BQ_DATASET')"
- In you terminal, run
gcloud auth login
to log into your GCP account that has access to the Instance Store Project. - Copy the generated script from Ascend and run.
Validating your Instance Store Connection​
- Go back into the Instance Settings page in Ascend.
- Click Check and Add Instance Store to validate your connection.
- If the connection is successful, you will see a message indicating that the Instance Store was added successfully.
Next Steps​
Now that you have set up your Instance setup with an Instance Store and a Vault, you can now move on to setting up your first environment! Your Ascend Instance already comes configured with a Production
environment, so we will need to create a Vault for that environment, as well as set up the Dataplane.