Data Connectors can be configured for different use-cases.
Last updated
Was this helpful?
Preliminaries
Before diving into how to configure a Data Connector, there are a few things worth keeping in mind when working with them.
Multiple Projects
Data Connectors are created and configured on a per-project basis. If you want multiple projects to send data to the same Endpoint URL, you have to create one Data Connector per project.
Multiple Data Connectors
All Data Connectors in a project will receive events from all the devices in that project. It is recommended to filter on event types when possible.
Access Control
To create, update, or delete a Data Connector, your DT Studio User or Service Account must have the role of Project Developer or higher.
Enable and Disable
A Data Connector can be enabled or disabled at any time. A Data Connector that is disabled will not forward events to its Endpoint URL until it is re-enabled. Any events that were generated while the Data Connector was disabled will not be forwarded once the Data Connector is re-enabled.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then toggle the Data Connector enabled switch. Remember to click Update Data Connector.
A request body with the following parameter should be included.
{
"status": "ACTIVE"
}
The status field of a Data Connector can be set to either "ACTIVE" or "USER_DISABLED". If the Data Connector gets automatically disabled by DT Cloud, the status will be set to SYSTEM_DISABLED. See the Auto-Disabling of Data Connectors section for more information.
Example Usage
Using cURL with a Service Account for authentication, the following example disables the speficied Data Connector by setting status to "USER_DISABLED".
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example disables the specified Data Connector by setting status to "USER_DISABLED".
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Disabled Data Connector by updating the status.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
status='USER_DISABLED',
)
# Print the updated Data Connector.
print(dcon)
The status field of a Data Connector can be set to either "ACTIVE" or "USER_DISABLED". If the Data Connector gets automatically disabled by DT Cloud, the status will be set to SYSTEM_DISABLED. See the Auto-Disabling of Data Connectors section for more information.
Disabled Data Connectors have no at-least-once guarantee.
When a Data Connector is disabled, undelivered events and events generated after being disabled will not be sent. Re-enabling the Data Connector will not backfill data from the period it was disabled and must be fetched programmatically using our REST API.
Auto-Disabling of Data Connectors
A Data Connector will be automatically disabled by DT Cloud if it has failed consistently for an extended period of time (currently 5 days) without any success. A push-attempt from a Data Connector is considered a failure if the receiver responded with a non-200 status code, or didn’t respond at all. A Data Connector that is disabled by DT Cloud will get the status SYSTEM_DISABLED.
Filtering on Event Types
By default, all event types are forwarded by a Data Connector. If you want to avoid unnecessary traffic on your endpoint, events can be filtered by type.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then toggle the Forward All Events followed by a selection. Remember to click Update Data Connector.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example sets a filter so that only temperature- and humidity events are forwarded.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector to only forward specified events.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
event_types=[
dt.events.TEMPERATURE,
dt.events.HUMIDITY,
],
)
# Print the updated Data Connector.
print(dcon)
Including Labels
A Data Connector can optionally forward device labels along with each event. By default, no device labels will be included, but you can specify a list of label keys that should be included in each request. See the Request Contents section for more details about how these device labels are included in the requests.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then append the label keys to the list. Remember to click Update Data Connector.
A request body with the following parameter should be included.
{
"labels": ["name", "floor", ...]
}
Example Usage
Using cURL with a Service Account for authentication, the following example configures the specified Data Connector to include device labels called "floor" and "customer-ID".
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example configured a Data Connector to include device labels called "name" and "floor".
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector to include device labels.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
labels=['name', 'floor'],
)
# Print the updated Data Connector.
print(dcon)
DT Studio Specific Labels
DT Studio uses labels with the keys name and description as the display name and the description of a device. It is recommended that an integration uses these labels in the same way when displaying or updating a device.
Adding HTTP Headers
Custom HTTP headers can be included with each event forwarded by a Data Connector.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then add a key- and value pairs for your custom headers. Remember to click Update Data Connector.
Using cURL with a Service Account for authentication, the following example adds the header "my-header" with value "my-value" to every event forwarded by a Data Connector.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example adds the header "my-header" with value "my-value" to every forwarded event.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector to include custom HTTP headers.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
config=dt.DataConnector.HttpPushConfig(
headers={
'my-header': 'my-value',
},
),
)
# Print the updated Data Connector.
print(dcon)
Signing Events
For increased security in a production integration, we recommend configuring a Signature Secret which can be used to verify the signed events on the receiving end.
The HTTP header of the webhook request includes a JSON Web Token (JWT) that is signed with the Signature Secret.
The JWT payload contains the checksum of the request body.
This allows you to validate the origin and the content integrity of the received request for a more secure integration. The process is two-fold and must be configured for both the Data Connector and the receiving endpoint.
Signing Events Sent by the Data Connector
Adding a signature secret to your Data Connector will automatically sign each forwarded event.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then add a strong and unique signature secret. Remember to click Update Data Connector.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example adds a signature secret to the specified Data Connector.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector with a signature secret.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
config=dt.DataConnector.HttpPushConfig(
signature_secret='<YOUR_SECURE_SECRET>',
),
)
# Print the updated Data Connector.
print(dcon)
Verifying Signed Events
To handle incoming events that have been signed by a signature secret, see Verifying Signed Events.
Synchronize
When synchronizing a Data Connector, the most recent event of each event type of each device currently in the project will be published to the Data Connector. This can be useful when a new Data Connector has been set up, and you need to get an up-to-date snapshot of each device in the project.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then click Synchronize Data Connector towards the bottom of the page.
Using cURL with a Service Account for authentication, the following example syncs the specified Data Connector.
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:sync" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be synced by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example syncs the specified Data Connector.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Synchronize the specified Data Connector.
dt.DataConnector.sync_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
)
