Skip to main content
Version: 3.0.0

Quickstart 🚀

This guide will walk you through setting up a sample project in Ascend and running your first flow.

Prerequisites​

Before getting started, you will need to make sure you have the following configured in your Ascend Instance:

  1. Your Instance Vault and Instance Store configured
  2. Your Environment Vault and Data Plane configured
  3. A Git Repository, Project, and Workspace

Cloning the Quickstart Project into your Repository​

To get you started on setting up your Ascend project, we have provided a quickstart project that you can clone and copy into your repository. This will give you a good starting point for your own project. The following steps in this guide will reference this quickstart project.

  1. In your terminal, run the following commands to clone the quickstart project
git clone git@github.com:ascend-io/ascend-community.git
  1. Clone your repository in the same directory:
git clone {your-repo-ssh-url} # make sure you have set up local ssh keys
cd {your-repo-name}
  1. Copy the quickstart project folder into your own repository
cp -r ../ascend-community/quickstart quickstart
git add .
git commit -m "Initial commit"
git push

Setting up your Project and Workspace​

Let's make sure the your Project is pointing to the correct folder in you repository.

  1. Log in to your Ascend Instance and navigate to your Project Settings (Ascend.io Instance: <instance-name> -> Settings -> Project).
  2. You should see the project you created as part of setting up your Project. Click on the project to open the project settings.
  3. For the project root, ensure that it is quickstart. Note: If you used a different directory for your project in the above step, ensure that directory is specified here.

Now let's ensure that your Workspace is using the correct profile.

  1. In the settings page, click on the Workspace tab.
  2. You should see a the Workspace you created as part of setting up your Workspace. Click on the Workspace to open the workspace settings.
  3. Under profile, you should already see the value prod. If not, click on the field and a drop down menu will list the available profiles in your project. The Quickstart project will have a single profile called prod. Select prod and then Save.
  1. Navigate back to your homepage by click the Ascend icon in the top left corner.
  2. Here you can see all your Ascend Projects and Workspaces. Click on your Workspace to open it.
  3. Welcome to your workspace! This is where you'll be building and running your Ascend Flows.
  4. On the left, you'll see your Project file tree, including all the folders and files needed for your Quickstart project. Let's take a look at each of the folders:
    • flows: This is where you will store your actual flow definitions. There is one flow in here called quickstart. Each flow has yaml definition file with the same name and a components folder that contains component files that can either be yaml, sql, or python, depending on whether the
flows
└── quickstart/
    ├── components/
    │   ├── lake_reader.yaml
    │   ├── my_python_transform.py
    │   └── sample_sql_transform.sql.jinja
    └── quickstart.yaml
vaults/
└── my_vault.yaml
  • connections: This is where you will store configuration for any connections to systems outside of Ascend, including the connection to your Data Plane.
connections/
├── local_files.yaml
└── my_data_plane.yaml
  • profiles: This is where you will store your Profile configurations. Profiles are used to defines the parameters and Data Plane with which your flows will run.
profiles/
└── prod.yaml

Configure your Project Vault​

  1. Click into the vaults folder.
  2. You will see a file called my_vault.yaml. This will be the file where we will define your vault. Click on this file to open it in the editor.
  3. You will see two options for you vault, azure_key_vault and gcp_secret_manager. Each of these values also has a vault or project name, depending on the vault type. These values have been set using Ascend Parameters, which use the syntax ${parameter.<parameter-name>} and are defined in your profile. You will set these in the next step.
  4. Delete the irrelevant Vault definition, keeping only the definition for the Vault you configured during the Environment Vault Setup. (If you are unsure which Vault is configured, you can check in your Environment Settings.)
  5. Click the Save button to save the file.

Setting up your Data Plane Connection​

Next, you'll set up the connection to use for your Data Plane. This will determine where your data will be stored and processed.

  1. Navigate to the connections folder and click my_data_plane.yaml to open the file in the editor.
  2. You will again see two options here, one for a BigQuery Data Plane under the bigquery key, and the other for a Snowflake Data Plane, under the snowflake key. You'll notice again that the values for the connection have been added as parameters that you will be setting in your profile. You will also notice another type of parameter used for injecting secrets into our configuration. This uses the syntax ${secret.<vault-name>.<secret-name>}.
  3. Delete the connection configuration for the connection you will not be using, and click Save to save your changes.
Using secrets

For the secret values, in this case bigquery.key and snowflake.password, ensure that the key here matches the key under which the secret is stored in your vault.

For example, if you are using Snowflake for your Data Plane, ensure that the secret is stored in your vault with the key ascend-env-prod-snowflake-password. If the key is different, you can update the value in the connection yaml.

Configuring your Profile​

Finally, you will set up your profile. Profiles are used to store run-time configuration of your Ascend Flows, such as parameters to use and which connection to use as a Data Plane. By defining different profiles, you have the ability to run dynamic flows with different configurations.

  1. Navigate to the profiles folder and click on prod.yaml to open the file in the editor.
  2. You will see that the parameters mentioned earlier are defined here under profile.parameters. The Data Plane for your flow is also configured to use the connection my_data_plane.
  3. There are two possible configuration options presented: Azure + Snowflake, which sets parameters for your Azure Key Vault and Snowflake Data Plane, and GCP + BigQuery, which sets parameters for your GCP Secret Manager and BigQuery Data Plane. You will delete the parameters you do not need and then update the values of the parameters you will be using. You can see a sample configuration below:
profiles/prod.yaml
profile:
  parameters:
    # Azure + Snowflake
    azure_key_vault_name: env-prod-vault
    snowflake_account: my_account
    snowflake_database: ASCEND_ENV_PROD
    snowflake_schema: quickstart
    snowflake_user: ASCEND_ENV_PROD
    snowflake_role: ASCEND_ENV_PROD
    snowflake_warehouse: ASCEND_ENV_PROD
  defaults:
    - kind: Flow
      name:
        regex: .*
      spec:
        data_plane:
          connection_name: my_data_plane
  1. Click Save to save your changes.

Build and Run your Flow!​

It is now time to run your flow!

  1. Click the Build Project button in the build panel on the bottom of the workspace page. This will compile your project, substituting the profile details into the flows. You should now see a visual representation of your flow, with the status of each component in the flow.

  2. Select on the quickstart flow in the build panel. Here you can see your built Flow DAG. To run the built flow, click the Play Button on the right hand side of the DAG.

  3. You should now see your Flow go into a running state. As components complete, you will see the record count pop up on each component. You can right-click on a component and click Records to see its records.

✨Congratulations!✨ You have just successfully run your first flow in Ascend. This is a great starting point for you to start building your own flows and use cases. Check out our ❓ What's Next? section to continue building and learning!

Last updated on