Skip to main content

Retry Components on failure

Implement retry strategies in your Components to handle transient failures automatically.

tip

New to retry strategies? Start with the Retry Strategies concept guide to understand when and how to use retry configurations.

The following examples show how to configure retry strategies in different component types. For details on configuration levels and strategy parameters, see the Retry Strategies concept guide.

YAML Components​

Basic retry with attempts​

Simple retry configuration for HTTP Read Components:

read-component-retry.yaml
component:
  read:
    http:
      url: "http://api.example.com/data"
      parser: "json"
  retry_strategy:
    stop_after_attempt: 3

Combined retry limits​

Robust retry configuration with both attempt and time limits:

combined-retry.yaml
component:
  read:
    http:
      url: "http://api.example.com/data"
      parser: "json"
  retry_strategy:
    stop_after_attempt: 5
    stop_after_delay: 600

SQL Components​

Basic retry with attempt limit​

This example retries up to 2 times before failing:

simple_retry.sql
{{ config(partitioned=True, retry_strategy=retry_strategy(stop_after_attempt=2)) }}

SELECT  *
FROM {{ ref("read_local") }}

Combined retry limits​

Robust retry configuration with both attempt and time limits:

combined_retry.sql
{{ config(partitioned=True, retry_strategy=retry_strategy(stop_after_attempt=5, stop_after_delay=300)) }}

SELECT  *
FROM {{ ref("your_data") }}

Python Components​

Python Components configure retry strategies using parameters inside their respective decorators (e.g. @read, @transform), providing the same flexibility as YAML configurations.

Python Read Component​

This example shows retry configuration for a Python Read Component with Incremental processing:

retry.py
from datetime import datetime, timedelta

from ascend.application.context import IncrementalComponentExecutionContext
from ascend.resources import read, retry_strategy
from pandas import to_datetime


@read(strategy="incremental", incremental_strategy="append", retry_strategy=retry_strategy(stop_after_attempt=5))
def market_data_incremental(context: IncrementalComponentExecutionContext):
    import yfinance as yf

    if context.is_incremental:
        current_data = context.current_data()
        date_column = context.data_plane.normalize_identifier("Date")  # This normalizes the letter casing between DuckDB and Snowflake
        start_date = to_datetime(current_data[date_column].max().execute()) + timedelta(days=1)
    else:
        current_data = None
        start_date = datetime.now() - timedelta(days=1)

    if start_date < datetime.now():
        data = yf.download("^GSPC", start=start_date).reset_index().rename(columns={"Adj Close": "Adj_Close"})
        if data.count().iloc[0] == 0:
            # if the data is empty, set the type of the date column to datetime (otherwise it will resolve to type NULL, which causes an error)
            data["Date"] = to_datetime(data["Date"])
        return data
    else:
        return None

Python Transform​

Transforms can also benefit from retry strategies for handling temporary processing failures:

python-transform-retry.py
import ibis
from ascend.application.context import IncrementalComponentExecutionContext
from ascend.common.events import log
from ascend.models.shared.base import RetryStrategy
from ascend.resources import ref, transform


@transform(inputs=[ref("read_local")], retry_strategy=RetryStrategy(stop_after_attempt=2))
def test_py(read_local: ibis.Table, context: IncrementalComponentExecutionContext) -> ibis.Table:
    # Your transform logic here
    log.info("Reading data from source")
    return read_local

Project-level defaults​

You can set default retry behavior for all Components in your Project, which individual Components can override as needed:

ascend_project.yaml
project:
  description: A collection of flows showcasing a variety of capabilities
  version: 0.0.1
  connections: ["connections/"]
  flows: ["flows/"]
  profiles: ["profiles/"]
  tests: ["tests/"]
  vaults: ["vaults/"]
  defaults:
    - kind: Component
      name:
        regex: .*
      spec:
        retry_strategy:
          stop_after_attempt: 3

Individual Components can override these defaults by specifying their own retry_strategy configuration.

Next steps​

Last updated on