An example integration on forwarding Data Connector events to a server hosted on Heroku.
Overview
This example uses a Data Connector to forward the events of all devices in a project to a server hosted on Heroku. When receiving the HTTPS POST request, our application will verify both the origin and content of the request using a Signature Secret.
Prerequisites
The following points are assumed.
You have the role of Project Developer or higher in your DT Studio project.
A functioning local environment for the language of your choice.
Create a New App Locally
On your machine, create and enter a new directory for your app.
mkdir my-app
cd my-app
In the directory, create a new file app.py where you paste the following snippet which contains the Flask server code for receiving and validating the Data Connector request.
import osimport hashlibimport jwtfrom flask import Flask, requestapp =Flask(__name__)# Fetch environment variables.SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')defverify_request(body,token):# Decode the token using signature secret.try: payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])exceptExceptionas err:print(err)returnFalse# Verify the request body checksum. m = hashlib.sha1() m.update(body) checksum = m.digest().hex()if payload["checksum"]!= checksum:print('Checksum Mismatch')returnFalsereturnTrue@app.route('/', methods=['POST'])defendpoint():# Extract necessary request information. body = request.data token = request.headers['x-dt-signature']# Validate request origin and content integrity.ifnotverify_request(body, token):return ('Could not verify request.',400)# Print the request body.print(body)return ('OK',200)
In the same directory, create a new file requirements.txt with the following contents. Heroku will install these dependencies in its environment before starting the server.
requirements.txt
gunicorn==20.1.0
flask==2.3.2
pyjwt==2.7.0
Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where Gunicorn should be used to serve the app using a variable called app in a module called app (app.py)
Procfile
web: gunicorn app:app
In the directory, create a new file app.py where you paste the following snippet which contains the Flask server code for receiving, validating, and decoding the Data Connector request.
app.py
import osfrom dtintegrations import data_connector, providerfrom flask import Flask, requestapp =Flask(__name__)@app.route('/', methods=['POST'])defendpoint():# Use the provider-specific validation function. payload = data_connector.HttpPush.from_provider( request=request, provider=provider.FLASK, secret=os.getenv('DT_SIGNATURE_SECRET'), )# Print the payload data.print(payload)# If all is well, return 200 response.return'Success'
In the same directory, create a new file requirements.txt with the following contents. Heroku will install these dependencies in its environment before starting the server.
Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where Gunicorn should be used to serve the app using a variable called app in a module called app (app.py)
Procfile
web: gunicorn app:app
In the directory, initialize a node application with the following command. This generates the file package.json which contains information used by Heroku when deploying the app.
npm init
Create a new file index.js where you paste the following snippet which contains the Express server code for receiving and validating the Data Connector request.
constexpress=require("express")constbodyParser=require("body-parser")constcrypto=require('crypto')constjwt=require('jsonwebtoken')// Create the express server.constapp=express()app.use(bodyParser.urlencoded({ extended:false }))app.use(bodyParser.json())// Fetch environment variables.constsignatureSecret=process.env.DT_SIGNATURE_SECRETfunctionverifyRequest(body, token) {// Decode the token using signature secret.let decoded;try { decoded =jwt.verify(token, signatureSecret) } catch(err) {console.log(err)returnfalse }// Verify the request body checksum.let shasum =crypto.createHash('sha1')let checksum =shasum.update(JSON.stringify(body)).digest('hex')if (checksum !==decoded.checksum) {console.log('Checksum Mismatch')returnfalse }returntrue}app.post("/", (req, res) => {// Extract necessary request information.let body =req.bodylet token =req.get('X-Dt-signature')// Validate request origin and content integrity.if (verifyRequest(body, token) ===false) {res.sendStatus(400)return } // Log the request body.console.log(body)res.sendStatus(200)})app.listen(process.env.PORT)
In the same directory, install the following packages. This updates package.json so that Heroku can replicate the environment when deploying.
Create a file Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where node index.js should be used to serve the app.
web: node index.js
In the directory, initialize a go application with the following command. This generates the file go.mod which specifies version and requirements. Give any name you like.
go mod init example
Create a new file main.go where you paste the following snippet which contains the server code for receiving and validating the Data Connector request.
packagemainimport ("crypto/sha1""encoding/hex""fmt""io/ioutil""log""net/http""os" jwt "github.com/golang-jwt/jwt/v5")// Environment variables.var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")// verifyRequest validates the request origin and content integrity.funcverifyRequest(bodyBytes []byte, tokenString string) error {// Decode the token using signature secret. claims :=jwt.MapClaims{} _, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {return []byte(signatureSecret), nil })if err !=nil {return err }// Verify the request body checksum. sha1Bytes := sha1.Sum(bodyBytes) sha1String := hex.EncodeToString(sha1Bytes[:])if sha1String != claims["checksum"] {return fmt.Errorf("Checksum mismatch.") }returnnil}funcendpoint(w http.ResponseWriter, r *http.Request) {if r.URL.Path !="/" { http.Error(w, "404 not found.", http.StatusNotFound)return }// Take an action depending on the request method.switch r.Method {case"POST":if err := r.ParseForm(); err !=nil { fmt.Fprintf(w, "ParseForm() err: %v", err)return }// Attempt to extract the body bytes of the request. body, err := ioutil.ReadAll(r.Body)if err !=nil { log.Printf("Error reading body: %v", err) http.Error(w, "Can't read body", http.StatusBadRequest)return }// Validate the origin and contents of the request.if err = verifyRequest(body, r.Header.Get("X-Dt-Signature")); err !=nil { log.Printf("Could not validate request: %v", err) http.Error(w, "Could not validate request", http.StatusBadRequest)return }//// Further processing here.// log.Println("Success")default: http.Error(w, "Only POST methods are allowed.", http.StatusMethodNotAllowed) }}funcmain() { http.HandleFunc("/", endpoint) port :=":"+string(os.Getenv("PORT")) log.Printf("Listening on port %s.", port)if err := http.ListenAndServe(port, nil); err !=nil { log.Fatal(err) }}
In the same directory, run the following command to install the required package.
go get github.com/golang-jwt/jwt/v5@v5.0.0
Build your application into a new directory bin.
GOOS=linux GOARCH=amd64 go build -o bin/main -v .
Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where the app should be server from the binary bin/main that we built in the previous step.
Procfile
web: bin/main
Deploy the App
The following steps will deploy your application to a new app in Heroku. Take note of the app name given to the application in step 4.
Initialize your app's directory as a Git repository: git init
Add all changes: git add .
Commit the changes: git commit -m "initial commit"
Create the Heroku app, and add it as a Git remote: heroku create <APP_NAME>. You can pick a unique app name yourself or let Heroku create a random one by omitting <APP_NAME>. This will be used as a part of the URL for your server.
Push your application to Heroku: git push heroku main
Post Deployment Configuration
Add a new signature secret configuration variable to your app using the following command. This can be read by the python instance and imported as an environment variable to use. Use a strong secret.
heroku config:set -a <APP_NAME> DT_SIGNATURE_SECRET=<YOUR_SECRET>
Your app is now ready to receive requests, but we need to know the endpoint URL. This can be acquired by looking at the Web URL field in the results of the following command. Save it for later.
heroku apps:info -a <APP_NAME>
Create a Data Connector
To continuously forward the data to our newly created Heroku server, a Data Connector with almost all default settings is sufficient. Refer to Creating a Data Connector if you are unfamiliar with the process. The following configurations should be set.
Endpoint URL: The Web URL found in the previous step.
Signature Secret: The value of DT_SIGNATURE_SECRET environment variable.
Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.
Test the Integration
If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each Periodic Heartbeat or by touching a sensor to force a new event.
If instead the Error counter increments, a response containing a non-2xx status code is returned.
Verify that the Data Connector endpoint URL is correct.
You can view the logs of your Heroku app with the following command.