HTTP Read

For ingesting data from an HTTP endpoint.

Examples

http_read_csv_uppercase.yaml

component:
  read:
    http:
      url: "http://example.com/data.csv"
      parser: "csv"
    uppercase: true

http_read_component_retry.yaml

component:
  read:
    http:
      url: "http://api.example.com/data"
      parser: "json"
  retry_strategy:
    stop_after_attempt: 5

http_read_component_custom_columns.yaml

component:
  read:
    http:
      url: "http://example.com/data"
      parser: "json"
    columns:
      - id
      - name
      - email

HttpReadComponent

info

HttpReadComponent is defined beneath the following ancestor nodes in the YAML structure:

• Component
• ReadComponent

Below are the properties for the HttpReadComponent. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this component runs.
event_time		string	No	Timestamp column in the component output used to represent event time.
connection		string	No	The name of the connection to use for reading data.
columns		array[None]	No	A list specifying the columns to read from the source and transformations to make during read.
normalize		boolean	No	A boolean flag indicating if the output column names should be normalized to a standard naming convention after reading.
preserve_case		boolean	No	A boolean flag indicating if the case of the column names should be preserved after reading.
uppercase		boolean	No	A boolean flag indicating if the column names should be transformed to uppercase after reading.
http			Yes

Property Details

Component

A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.

Property	Default	Type	Required	Description
component		One of:   CustomPythonReadComponent   ApplicationComponent   AliasedTableComponent   ExternalTableComponent	Yes	Configuration options for the component.

ReadComponent

A component that reads data from a data system.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DatabricksDataPlane	No	Data Plane-specific configuration options for a component.
skip		boolean	No	A boolean flag indicating whether to skip processing for the component or not.
retry_strategy			No	The retry strategy configuration options for the component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	The name of the flow that the component belongs to.
data_maintenance			No	The data maintenance configuration options for the component.
tests			No	Defines tests to run on the data of this component.
read		One of:   GenericFileReadComponent   LocalFileReadComponent   SFTPReadComponent   S3ReadComponent   GcsReadComponent   AbfsReadComponent   HttpReadComponent   MSSQLReadComponent   MySQLReadComponent   OracleReadComponent   PostgresReadComponent   SnowflakeReadComponent   BigQueryReadComponent   DatabricksReadComponent	Yes	The read component that reads data from a data system.

HttpReadComponentOptions

Options for reading data from an HTTP endpoint.

Property	Default	Type	Required	Description
parser		string ("json", "csv")	Yes	Parser to use for HTTP-based data. Can be one of 'json' or 'csv'.
url		string	Yes	URL to retrieve the data from.

ComponentColumn

Component column expression definition.

No properties defined.

InputComponent

Specification for input components, including how partitioning behaviors should be handled. This additional metadata is required when a component is used as an input to other components in a flow.

Property	Default	Type	Required	Description
flow		string	Yes	Name of the parent flow that the input component belongs to.
name		string	Yes	The input component name.
alias		string	No	The alias to use for the input component.
partition_spec		Any of:   string ("full_reduction", "map")	No	The type of partitioning to apply to the component's input data before processing the component's logic. Input partitioning is applied before the component's logic is executed.
where		string	No	An optional filter condition to apply to the input component's data.
partition_binding		Any of:   string	No	An optional partition binding specification to apply to the component on a per-output-partition basis against other inputs' partitions.

PartitionBinding

Property	Default	Type	Required	Description
logical_operator	logical_operator	string ("AND", "OR")	No	The logical operator to use to combine the partition binding predicates provided
predicates	predicates	array[string]	No	The list of partition binding predicates to apply to the input component's data

RepartitionSpec

Specification for repartitioning operations on input component's data

Property	Default	Type	Required	Description
repartition			No	Options for repartitioning the input component's data.

RepartitionOptions

Options for repartitioning the input component's data.

Property	Default	Type	Required	Description
partition_by		string	Yes	The column to partition by.
granularity		string	Yes	The granularity to use for the partitioning.