AWS Lambda

An example integration where a Data Connector forwards events to an AWS Lambda Function.

Overview

This example uses a Data Connector to forward the events of all devices in a project to an AWS Lambda Function. When receiving the HTTPS POST request, our function will verify both the origin and content of the request using a Signature Secret, then decode the data.

Prerequisites

We will create, configure, and deploy our Lambda function using the AWS Console, but the deployment package that contains the code and related runtime must be built locally on your machine. You must therefore have a working development environment.

The following points are assumed.

• You have a Role of Project Developer or higher in your DT Studio project.

• You are familiar with the Introduction to Data Connectors and know how to Create a Data Connector.

• You are familiar with the AWS Lambda documentation.

Create a Local Deployment Package

AWS Lambda Functions can be created and deployed using either a Docker Image or a .zip archive. Both will achieve the same result, but we will here use a .zip archive in the interest of simplicity.

Source Code

Create and enter a new root directory in which your function will be developed.

mkdir my-lambda-function
cd my-lambda-function

Python 3.10

In your directory root, create a new file lambda_function.py with the following content. The implementation is explained in detail on the Data Connector Advanced Configurations page.

import os
import json
import hashlib
import jwt

# Fetch secret environment variable.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')


def verify_request(body, token):
    # Decode the token using signature secret.
    payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(body.encode('ascii'))
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        raise ValueError('Checksum Mismatch')


def lambda_handler(event, context):
    # Extract necessary request information.
    body = event['body']
    token = event['headers']['X-Dt-Signature']

    # Validate request origin and content integrity.
    try:
        verify_request(body, token)
    except Exception as e:
        return {
            'statusCode': 400,
            'body': json.dumps(str(e)),
        }

    #
    # Further processing here.
    #

    return {
        'statusCode': 200,
        'body': json.dumps('Success'),
    }

Node.js 18

In your directory root, create a file index.js in which you paste the following snippet. The implementation is explained in detail on the Data Connector Advanced Configurations page.

var crypto = require('crypto');
var jwt = require('jsonwebtoken');

// Fetch environment variables.
const signatureSecret = process.env.DT_SIGNATURE_SECRET

function verifyRequest(body, token) {
    // Decode the token using signature secret.
    let decoded;
    try {
        decoded = jwt.verify(token, signatureSecret)
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(JSON.stringify(body)).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

exports.handler = async function(event, context) {
    // Extract necessary request information.
    let body  = JSON.parse(event.body)
    let token = event.multiValueHeaders['X-Dt-Signature'][0]

    // Validate request origin and content integrity.
    if (verifyRequest(body, token) === false) {
        const response = {
            statusCode: 400,
            body: JSON.stringify('could not verify request'),
        };
        return response; 
    }

    //
    // Further processing here.
    //
    const response = {
        statusCode: 200,
        body: JSON.stringify('success'),
    };
    return response; 
};

Go 1.x

In your directory root, initialize a new go module.

go mod init example

Create a file main.go in which you paste the following snippet.

package main

import (
	"context"
	"crypto/sha1"
	"encoding/hex"
	"log"
	"os"

	jwt "github.com/dgrijalva/jwt-go"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
)

// Environment variables.
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")

// verifyRequest validates the request origin and content integrity.
func verifyRequest(bodyBytes []byte, tokenString string) bool {
	// Decode the token using signature secret.
	claims := jwt.MapClaims{}
	_, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
		return []byte(signatureSecret), nil
	})
	if err != nil {
		log.Println(err)
		return false
	}

	// Verify the request body checksum.
	sha1Bytes := sha1.Sum(bodyBytes)
	sha1String := hex.EncodeToString(sha1Bytes[:])
	if sha1String != claims["checksum"] {
		log.Println("Checksum mismatch.")
		return false
	}

	return true
}

func handleRequest(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	// Extract necessary request information.
	tokenString := req.Headers["X-Dt-Signature"]
	bodyBytes := []byte(req.Body)

	// Validate request origin and content integrity.
	if !verifyRequest(bodyBytes, tokenString) {
		return events.APIGatewayProxyResponse{StatusCode: 400}, nil
	}

	return events.APIGatewayProxyResponse{StatusCode: 200}, nil
}

func main() {
	lambda.Start(handleRequest)
}

Dependencies and Compilation

From inside your function root directory, install the following dependencies.

Python 3.10

pip install --target . pyjwt==2.7.0

Your directory tree should now look something like this.

.
├── lambda_function.py/
├── jwt/
└── PyJWT-2.7.0.dist-info

Node.js 18

npm install aws-xray-sdk@3.5.0
npm install jsonwebtoken@9.0.1

Your directory tree should now look something like this.

.
├── index.js
├── node_modules
│   ├── aws-xray-sdk
│   ├── jsonwebtoken
│   ├── ...
│   ├── ...
├── package.json
└── package-lock.json

Go 1.x

go get github.com/aws/aws-lambda-go@v1.41.0
go getgithub.com/dgrijalva/jwt-go@v3.2.0

Your directory tree should now look something like this.

.
├── go.mod
├── go.sum
└── main.go

Create a .zip Archive

The following steps create the deployment package .zip archive.

Python 3.10

From your directory root, create the deployment package with the following command.

zip -r deployment-package.zip .

Node.js 18

From your directory root, create the deployment package with the following command.

zip -r deployment-package.zip .

Go 1.x

The following steps are for Linux and MacOS. If you're on Windows or want more details about the process, refer to the official AWS Guide.

Compile your executable.

GOOS=linux GOARCH=amd64 go build main.go

From your directory root, create the deployment package with the following command.

zip deployment-package.zip main

Create a New Lambda Function

We are now ready to create our new Lambda function. In the AWS web Console, find the Lambda service and click Create function and set the following parameters.

Python 3.10

Select Author from scratch and set the following Basic Information.

• Function name: As desired.

• Runtime: Python 3.10

Node.js 18

Select Author from scratch and set the following Basic Information.

• Function name: As desired.

• Runtime: Node.js 18

Go 1.x

Select Author from scratch and set the following Basic Information.

• Function name: As desired.

• Runtime: Go 1.x

Click Create function to move onward to configuration.

Configure Lambda Function

Anything not mentioned here can be left default or changed as desired.

New Trigger

Under the Configuration tab, click Triggers and then Add trigger. Choose the API Gateway source with the following parameters.

• Intent: Create a new API

• API type: HTTP API

• Security: Open

Click Save and note the URL. We will need it later.

Environment Variables

Under the Configuration tab, click Environment variables and then Edit. Add a signature secret.

• Name: DT_SIGNATURE_SECRET

• Value: A unique password. We will need it later, so write it down.

Leaving Encryption configuration default will let AWS handle encryption at rest, but you can choose to handle the master key if you so desire.

Create a New Data Connector

To continuously forward the data to our newly created Lambda Function, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our Creating a Data Connector guide. The following configurations should be set.

• Endpoint URL: The API endpoint URL found in the previous step.

• Signature Secret: The value of DT_SIGNATURE_SECRET parameter set earlier.

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.

Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each Periodic Heartbeat or by touching a sensor to force a new event.

If instead the Error counter increments, a response containing a non-200 status code is returned.

• Verify that the Data Connector endpoint URL is correct.

• AWS provides a host of tools that can be used to monitor Lambda Functions. Check the logs for any tracebacks that could explain why an error is returned.

Next steps

Your sensor data is now in the AWS environment, and you can start using it in their various services. Fortunately, AWS has some well-documented guides to get you started.

PostgreSQL Database

A database should be tailored to each specific use case. However, if you're uncertain, PostgreSQL (Postgres) is a good place to get started. The following guides will show you how to create a new Postgres database, then connect your Lambda Function to execute queries.

• Create and Connect to a PostgreSQL Database.

• Configuring a Lambda Function to Access Amazon RDS.