Profile

Profiles are configuration settings within Ascend Projects that define runtime parameters for your Ascend Flows. They allow you to parameterize the Connections, Components, and Vaults in your Flow without altering the core code, enabling their reuse across different Workspaces and Deployments.

Key Features

There are two main types of configurations defined in your Profile:

1. Default Values: Profiles must include a default configuration for your Flows. Within a Profile, it is necessary to specify the default Connection to use as your Data Plane. If you'd like to use specific Data Plane Connections for specific Flows, you can do so using regular expressions. Each default configuration has a kind (which determines the kind of resource being configured), a name (which is used to match the resource to be configured), and a spec (which contains the configuration for the resource).

2. Execution Parameters: These are used to configure the runtime behavior of your Flow. Parameters are defined as key-value pairs within your Profile, under the parameters field.

Here is an example of a Profile with both Parameters and Defaults. In this example, Snowflake Connection parameters are set for a development Snowflake environment and the default Data Plane for all Flows in this Profile is set to the snowflake Connection.

dev-snowflake.yaml

profile:
  parameters:
    vault_name: dev-vault
    snowflake_database: dev-db
    snowflake_user: dev-user
    snowflake_schema: dev-schema
    snowflake_warehouse: dev-warehouse

  defaults:
    - kind: Flow
      name:
        regex: .*
      spec:
        data_plane:
          connection_name: snowflake

CI/CD with Profiles

Profiles allow for seamless promotion of your Flows from your Ascend Development Environment to your Staging or Production Environment as you can create separate Profiles for each stage in the Software Development Lifecycle (SDLC). For example, the most common scenario is using a Profile to parameterize a Connection, so you can ensure you are reading from development data sources in Development, and reading from production data sources in Production.

Workspaces or Deployments run in a single Ascend Environment and are bound by the security limitations of that Environment. For example, a Workspace running in Development cannot access resources in Production. A Profile can be used to determine what those resources are. For example, the Profile for Development may specify the name of the Secrets Vault to use to get secrets, or it may specify the specific S3 bucket to read from.

Profiles and Environments

It is important remember that while Profiles can determine what resources your Flows are connecting to, access to those resources are still constrained by the permissions granted to the Environment where your Workspace or Deployment is running.

Profiles at Runtime

Let's walk through how Profiles work in a simple example where we have a Project with a Snowflake Connection, and a Development and Production Profile.

my-project
├── connections
│   └── snowflake.yaml
├── flows
├── profiles
│   ├── dev-snowflake.yaml
│   └── prod-snowflake.yaml
└── vaults
    └── vault.yaml

Within this Project there is a snowflake.yaml Connection, where the values for the Connection are parameterized.

snowflake.yaml

connection:
  snowflake:
    account: snowflakeaccount
    database: ${parameters.snowflake_database}
    user: ${parameters.snowflake_user}
    schema: ${parameters.snowflake_schema}
    password: ${secret.vault.snowflake_password}
    warehouse: ${parameters.snowflake_warehouse}

There is also an Azure Key Vault defined in vault.yaml, where the vault name is parameterized. This means that the same vault definition can be used for both development and production vaults.

vault.yaml

vault:
  azure_key_vault:
    vault_name: ${parameters.vault_name}

Finally, there are two Profiles, dev-snowflake.yaml and prod-snowflake.yaml. For both Profiles, Parameters have been set for the Snowflake Connection and the vault name. The default Data Plane has also been set to use the snowflake.yaml Connection in both Profiles.

dev-snowflake.yaml

profile:
  parameters:
    vault_name: dev-vault
    snowflake_database: dev-db
    snowflake_user: dev-user
    snowflake_schema: dev-schema
    snowflake_warehouse: dev-warehouse

  defaults:
    - kind: Flow
      name:
        regex: .*
      spec:
        data_plane:
          connection_name: snowflake

prod-snowflake.yaml

profile:
  parameters:
    vault_name: prod-vault
    snowflake_database: prod-db
    snowflake_user: prod-user
    snowflake_schema: prod-schema
    snowflake_warehouse: prod-warehouse

  defaults:
    - kind: Flow
      name:
        regex: .*
      spec:
        data_plane:
          connection_name: snowflake

During the Development phase, a Workspace is created that uses the dev-snowflake Profile. When a Flow is run in this Workspace, the following will happen:

1. The parameters in the snowflake.yaml Connection are substituted for the parameter values in the dev-snowflake Profile. That means that the Snowflake database and schema that will be used will be dev-db and dev-schema, the Snowflake user will be dev-user, and the Snowflake warehouse will be dev-warehouse.
2. The password for the Snowflake user will be retrieved from the dev-vault vault, as that is the value of the vault_name parameter in the dev-snowflake Profile.
3. The Flow is then run using the snowflake Connection as the Data Plane.

When changes have been made, they can be pushed to the Production Deployment. This Deployment will use the prod-snowflake Profile, and the Flow will run with the parameters defined in that Profile.

Other Use Cases

While it is most common to use Profiles to allow for more seamless CI/CD workflows, the flexible nature of Profiles allow you to parameterize any part of your Flow. Below we will walk through two other common use cases for Profiles.

Parameterized Components

While the above example shows how to parameterize Connections, you can also parameterize Components themselves. Any field in the YAML definition of a component can be parameterized. For example, you can limit the amount of data a Flow ingests by setting a parameter for the glob pattern used to ingest data from S3, like so:

s3-read-component.yaml

component:
  read:
    connection: lake_on_s3
    s3:
      path: listing/binary/
      include:
        - glob: ${parameter.s3_glob_pattern}

In your Profile, you can set the s3_glob_pattern to **/*.csv to ingest all CSV files in the listing/binary/ directory, or set it to **/year=2024/month=01/**/*.csv to only ingest a subset of the CSV files.

Isolated Workspaces for Developers

When you have multiple developers working on the same Project in Ascend using the same Data Plane, it is important to ensure that they are not writing over each other's data when running their Flows. Using Profiles, you can parameterize the Data Plane Connection for each developer, so that, when they run a Flow, the data is written to a location that is isolated from other developers. In a Snowflake Data Plane, for example, this can be separate schemas for each user.

A typical profile in this scenario might look like this:

dev-snowflake-jane.yaml

profile:
  parameters:
    vault_name: dev-vault
    snowflake_database: dev-db
    snowflake_user: dev-user
    snowflake_schema: jane-schema
    snowflake_warehouse: dev-warehouse

  defaults:
    - kind: Flow
      name:
        regex: .*
      spec:
        data_plane:
          connection_name: snowflake

In this example, when Jane runs a Flow in her Workspace using the dev-snowflake-jane Profile, the data will be written to an isolated schema in Snowflake.