Creating a Data Connector

A quick guide on how to create a Data Connector using DT Studio or our APIs.

Overview

To illustrate the process, we will create a new Data Connector that forwards all events to webhook.site, a free service that fulfills the same role as a server that lets you inspect the contents of each request. If you already have an HTTPS POST endpoint set up, feel free to use that instead.

Prerequisites

Service Account Creating, updating, and deleting Data Connectors requires that your User or Service Account has been granted the role Project Developer or higher.
HTTPS Endpoint Data Connectors sends events as POST requests over HTTPS. In order to receive these events you need to have a service up and running that can accept incoming HTTPS connections.HTTP is not supported. In this guide, we will use webhook.site as an example.

Please note that Disruptive Technologies and webhook.site are in no way affiliated and that the data sent there is publicly available.

Creating a New Data Connector

Data Connectors can be created using either the DT Studio interface or our REST API. For simplicity, most of the configurations will be left as default in this example. If you're interested in a more custom Data Connector, see the Configuring a Data Connector page for more information.

In DT Studio, navigate to your Project. In the left menu, locate Data Connectors under API Integrations and press Add Data Connector. This will open the Data Connector configuration menu.

Edit the following parameters.

Display name Give the Data Connector some identifiable display name.
Endpoint URL The URL the Data Connector will forward events to. As we use webhook.site in this example, navigate to https://webhook.site to generate a new URL on the form https://webhook.site/<RANDOM_ID>. Copy this URL, and paste it into the Endpoint URL field of the Data Connector config.

Remember to save your Data Connector configuration at the bottom of the page.

Send a POST request to:

https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors

A request body with the following parameters is required.

{
    "type": "HTTP_PUSH",
    "displayName": "Example",
    "httpConfig": {
        "url": "<ENDPOINT_URL>"
    }
}

A list of all available parameters can be found in our REST API Reference.

Example Usage

Using cURL with a Service Account for authentication, the following example creates a new Data Connector with a given name "Example" and a webhook.site endpoint which can be monitored at https://webhook.site/<RANDOM_ID>.

curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"displayName": "Example", "type": "HTTP_PUSH", "httpConfig": {"url": "https://webhook.site/<RANDOM_ID>"}}'

Once the package is installed and authenticated as described in the Python API Reference, a new Data Connector can be created by calling the following resource method.

disruptive.DataConnector.create_data_connector()

Example Usage

Using our Python API with Service Account credentials for authentication, the following example creates a new Data Connector with a given name "Example" and a webhook.site endpoint.

import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Create a new HTTP Push Data Connector called "Example".
new_dcon = dt.DataConnector.create_data_connector(
    project_id='<PROJECT_ID>',
    config=dt.DataConnector.HttpPushConfig(
        url='https://webhook.site/<RANDOM_ID>',
    ),
    display_name='Example',
)

# Print the newly created Data Connector.
print(new_dcon)

Test Your Data Connector

The simplest way of testing your connection is by forwarding an event in one of the following ways.

Wait for a sensor in the project to naturally emit an event by its periodic heartbeat.
Touching a sensor to force an emitted event.
Use the Sensor Emulator to emulate an emitted event.

Once an event has been forwarded by the Data Connector, each event will be visible as individual requests at the generated webhook.site URL.

You can verify that an event has been successfully forwarded by looking at the Data Connector metrics.

The Data Connector metrics can be found at the top of the Data Connector page in Studio.

Send a GET request to:

https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATACONNECTOR>:metrics

The response contains the metrics for the specified Data Connector over the last 3 hours.

{
  "metrics": {
    "successCount": 44,
    "errorCount": 0,
    "latency99p": "0.663s"
  }
}

Example

Using cURL with a Service Account for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.

curl -X GET "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:metrics" \
    --user "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"

Once the package is installed and authenticated as described in the Python API Reference, Data Connector metrics can be fetched by calling the following resource method.

disruptive.DataConnector.get_metrics()

Example Usage

Using our Python API with Service Account credentials for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.

import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Fetch the metrics of the specified Data Connector.
metrics = dt.DataConnector.get_metrics(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
)

# Print the content of metrics.
print(metrics)

If the Error counter is incrementing, it indicates that DT Cloud is unable to forward events to your endpoint. This can be for any of the following reasons:

You have entered an incorrect Endpoint URL in the Data Connector config.
An invalid SSL certificate has been configured on the server. See Server Configuration for more details.
If you're forwarding events to a cloud function, make sure the cloud function is publicly available and does not require authentication. Otherwise, DT Cloud will receive 401 or 403 error codes from your endpoint.
Check your error logs for any other issues.

Last updated 1 month ago