Data Sources

Airflow

Apache Airflow

Overview

Use the Apache Airflow connector to extract metadata, execution lineage, and pipeline run histories. Connetra supports two connection methods for Airflow: REST API and OpenLineage.

Connection Method: REST API

The REST API method connects directly to your Airflow webserver to pull metadata. When adding this data connector in Connetra, provide the following details:

  • Configuration name: A user-friendly label for this connection to help identify the Airflow instance within Connetra.
  • Host/server: The full URL of your Airflow webserver/API (e.g., https://airflow.yourdomain.com).
  • User: The username of an account with sufficient API access to read DAG and run data.
  • Password: The password associated with the Airflow user account.

Before you can save the data connector you must first test the connection.

Connection Method: OpenLineage

The OpenLineage connection method allows your Airflow instance to push lineage events dynamically to Connetra as tasks execute. This is the recommended approach for real-time observability.

When you select the OpenLineage method in Connetra, it will automatically generate the credentials necessary to configure your Airflow environment:

  • Configuration name: A user-friendly label for this connection.
  • API Token: A generated token used by Airflow to authenticate its requests to Connetra. You will need to copy this value into your Airflow configuration.
  • Events Endpoint: The endpoint path Connetra provides (e.g., /v1/openlineage/events).
  • Events URL: The base URL where Connetra listens for incoming OpenLineage events.

Step-by-Step OpenLineage Configuration

Follow these steps to set up the OpenLineage integration on your Airflow instance:

Step 1: Install the OpenLineage Provider

Ensure that the OpenLineage provider package is installed in your Airflow environment. You can install it using pip:

pip install apache-airflow-providers-openlineage

Step 2: Configure OpenLineage Transport

Provide a Transport configuration so that OpenLineage knows where to send the events. Keep the API Token, Events URL, and Events Endpoint handy from the previous step.

Within airflow.cfg file:

[openlineage] transport = {"type": "http", "url": "<Events URL>", "endpoint": "<Events Endpoint>", "auth": {"type": "api_key", "api_key": "<API Token>"}}

Or with AIRFLOW__OPENLINEAGE__TRANSPORT environment variable:

export AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "http", "url": "<Events URL>", "endpoint": "<Events Endpoint>", "auth": {"type": "api_key", "api_key": "<API Token>"}}'

Once your Airflow environment is configured and restarted, trigger a DAG run in Airflow. You will begin to see metadata and lineage events populating automatically within Connetra.