Data Connectors can through configuration be tailored to fit more custom use-cases.
Preliminaries
Before diving into how to configure a Data Connector, there are a few things worth keeping in mind when working with them.
Multiple Projects
Data Connectors are created and configured on a per-project basis. If you want multiple projects to send data to the same Endpoint URL, you have to create one Data Connector per project.
Access Control
To create, update, and delete a Data Connector, your DT Studio User or Service Account must have the role of Project Developer or higher.
Multiple Data Connectors
All Data Connectors in a project will receive events from all the devices in that project. It is recommended to filter to event types when possible.
Enable and Disable
It is possible to enabled and disable a Data Connector at will.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then toggle the Data Connector enabled switch. Remember to click Update Data Connector.
A request body with the following parameter should be included.
{"status": "ACTIVE"}
The status field takes either "ACTIVE" or "USER_DISABLED".
Example Usage
Using cURL with a Service Account for authentication, the following example disables the speficied Data Connector by setting status to "USER_DISABLED".
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example disables the specified Data Connector by setting status to "USER_DISABLED".
import disruptive as dt# Authenticate the package using Service Account credentials.dt.default_auth = dt.Auth.service_account( key_id='<SERVICE_ACCOUNT_KEY_ID>', secret='<SERVICE_ACCOUNT_SECRET>', email='<SERVICE_ACCOUNT_EMAIL>',)# Disabled Data Connector by updating the status.dcon = dt.DataConnector.update_data_connector( data_connector_id='<DATA_CONNECTOR_ID>', project_id='<PROJECT_ID>', status='USER_DISABLED',)# Print the updated Data Connector.print(dcon)
Disabled Data Connectors have no at-least-once guarantee.
When a Data Connector is disabled, undelivered events and events generated after being disabled will not be sent. Re-enabling the Data Connector will not backfill data from the period it was disabled and must be fetched programmatically using our REST API.
Auto-Disabled Data Connectors
A Data Connector will be automatically disabled by DT Cloud if it has failed for an extended period of time without any success. A push-attempt from a Data Connector is considered a failure if the receiver responded with a non-200 status code, or didn’t respond at all. A Data Connector that is disabled by DT Cloud will get the status SYSTEM_DISABLED.
Filtering on Event Types
By default, all event types are forwarded by a Data Connector. If you want to avoid uneccesary traffic on your endpoint, events can be filtered by type.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then toggle the Forward All Events followed by a selection. Remember to click Update Data Connector.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example sets a filter so that only temperature- and humidity events are forwarded.
import disruptive as dt# Authenticate the package using Service Account credentials.dt.default_auth = dt.Auth.service_account( key_id='<SERVICE_ACCOUNT_KEY_ID>', secret='<SERVICE_ACCOUNT_SECRET>', email='<SERVICE_ACCOUNT_EMAIL>',)# Update the Data Connector to only forward specified events.dcon = dt.DataConnector.update_data_connector( data_connector_id='<DATA_CONNECTOR_ID>', project_id='<PROJECT_ID>', event_types=[ dt.events.TEMPERATURE, dt.events.HUMIDITY, ],)# Print the updated Data Connector.print(dcon)
Including Labels
You can include labels held by a device with each event forwarded by the Data Connector.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then append the label keys to the list. Remember to click Update Data Connector.
A request body with the following parameter should be included.
{"labels": ["label-01","label-02",...]}
Example Usage
Using cURL with a Service Account for authentication, the following example configures the specified Data Connector to include device labels called "floor" and "customer-ID".
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example configured a Data Connector to include device labels called "floor" and "customer-ID".
import disruptive as dt# Authenticate the package using Service Account credentials.dt.default_auth = dt.Auth.service_account( key_id='<SERVICE_ACCOUNT_KEY_ID>', secret='<SERVICE_ACCOUNT_SECRET>', email='<SERVICE_ACCOUNT_EMAIL>',)# Update the Data Connector to include device labels.dcon = dt.DataConnector.update_data_connector( data_connector_id='<DATA_CONNECTOR_ID>', project_id='<PROJECT_ID>', labels=['floor', 'customer-ID'],)# Print the updated Data Connector.print(dcon)
DT Studio Specific Labels
DT Studio uses labels with the keys name and description to show as the display name and the description of a device. It is recommended that an integration uses these labels in the same way when displaying or updating a device.
Adding HTTP Headers
Custom HTTP headers can be included with each event forwarded by a Data Connector.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then add a key- and value pairs for your custom headers. Remember to click Update Data Connector.
Using cURL with a Service Account for authentication, the following example adds the header "my-header" with value "my-value" to every event forwarded by a Data Connector.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example adds the header "my-header" with value "my-value" to every forwarded event.
import disruptive as dt# Authenticate the package using Service Account credentials.dt.default_auth = dt.Auth.service_account( key_id='<SERVICE_ACCOUNT_KEY_ID>', secret='<SERVICE_ACCOUNT_SECRET>', email='<SERVICE_ACCOUNT_EMAIL>',)# Update the Data Connector to include custom HTTP headers.dcon = dt.DataConnector.update_data_connector( data_connector_id='<DATA_CONNECTOR_ID>', project_id='<PROJECT_ID>', config=dt.DataConnector.HttpPushConfig( headers={'my-header': 'my-value', }, ),)# Print the updated Data Connector.print(dcon)
Signing Events
For increased security in a production integration, we recommend signing each Event with a Signature Secret that can be verified on the receiving end.
The HTTP header includes a JSON Web Token (JWT) signed with the Signature Secret.
The JWT payload contains the checksum of the request body.
This allows you to both validate the origin and content integrity of the received request for a more secure connection. The process is two-fold and must be configured for both the Data Connector and endpoint.
Signing Events Sent by the Data Connector
Adding a signature secret to your Data Connector will automatically sign each forwarded event.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then add a strong and unique signature secret. Remember to click Update Data Connector.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example adds a signature secret to the specified Data Connector.
import disruptive as dt# Authenticate the package using Service Account credentials.dt.default_auth = dt.Auth.service_account( key_id='<SERVICE_ACCOUNT_KEY_ID>', secret='<SERVICE_ACCOUNT_SECRET>', email='<SERVICE_ACCOUNT_EMAIL>',)# Update the Data Connector with a signature secret.dcon = dt.DataConnector.update_data_connector( data_connector_id='<DATA_CONNECTOR_ID>', project_id='<PROJECT_ID>', config=dt.DataConnector.HttpPushConfig( signature_secret='<YOUR_SECURE_SECRET>', ),)# Print the updated Data Connector.print(dcon)
Verifying Signed Events
To handle incoming events that have been signed by a signature secret, see Verifying Signed Events.
Synchronize
When synchronizing the Data Connector, the state of each device currently in the project will be sent as events. These events will be dated back in time to when they were reported from the device. Note, however, that the event id will be different for the synchronized events.
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then click Synchronize Data Connector towards the bottom of the page.
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be synced by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example syncs the specified Data Connector.
import disruptive as dt# Authenticate the package using Service Account credentials.dt.default_auth = dt.Auth.service_account( key_id='<SERVICE_ACCOUNT_KEY_ID>', secret='<SERVICE_ACCOUNT_SECRET>', email='<SERVICE_ACCOUNT_EMAIL>',)# Synchronize the specified Data Connector.dt.DataConnector.sync_data_connector( data_connector_id='<DATA_CONNECTOR_ID>', project_id='<PROJECT_ID>',)
