Upload Data to S3 Using Funnel and Amazon Data Firehose

Introduction

In this guide, we create an Amazon Data Firehose instance and an S3 bucket on AWS CloudFormation, then use the Soracom Funnel Amazon Data Firehose adapter to send data directly from your device to an S3 bucket.

We will be using Soracom Funnel along with the following AWS resources:

• Amazon Data Firehose
• Amazon S3
• AWS IAM
• Amazon CloudWatch Logs
• AWS CloudFormation

Click the links to read about these services in more depth.

Data Flow

1. The Data sent from your devices will be received by Soracom Funnel.
2. Funnel uses IAM Role authentication to send your data over to Amazon Data Firehose.
3. Amazon Data Firehose relays the data to S3.
4. When the data transfer to S3 fails, the error log is recorded on Amazon CloudWatch Logs, which will be available for 14 days after the error occurs.

Step 1: Set up AWS Resources

You will first use AWS CloudFormation to configure and prepare all the AWS Resources used on this guide.

Prepare AWS Resources Using CloudFormation

1. Click here to Use the AWS CloudFormation Template.

2. Configure the settings below:

   Parameter	Content
   CoverageAWSAccountID	This parameter depends on your IoT SIM's coverage: JP Coverage: 762707677580 Global Coverage: 950858143650
   DeliveryStreamName	Enter any stream name
   IamRoleExternalIdForFunnel	Enter any external id, then save it somewhere since it will be used later (e.g. external-id-0123456789)
   IamRoleNameForFunnel	Enter any IAM Role name
   S3BucketName	Enter any S3 Bucket name

![Security](./step1-2.png)

3. Click the check box next to I acknowledge that AWS CloudFormation..., then click Create stack

![Security](./step1-3.png)

Retrieve IAM Role ARN

1. Navigate to AWS IAM Roles

![Security](./step1-4.png)

2. Search for and click the IAM Role you just created (enter the IAM Role Name you entered when you created the CloudFormation Resource Stack)

![Security](./step1-5.png)

3. Copy the Role ARN and save it for later

![Security](./step1-6.png)

Step 2: Configure Soracom Funnel

Now that you have the AWS Resources set up, you will set up and enable Soracom Funnel.

Register Your AWS IAM Role Credentials

1. Sign in to your Soracom console , then click the top right corner

![Security](./step2-1.png)

2. Click Security

![Credentials](./step2-2.png)

3. Click the Credentials tab, then click the Register credentials button

![Credentials](./step2-3.png)

4. Enter your IAM credentials as follows, then click Register:

| Parameter               | Content                                                                          |

|--------------------|----------------------------------------------------------------------------------|
| **Credential set ID** | Any name for this credential set (i.e. AWS-IAM-Role-Soracom-Funnel-s3)          |
| **Type**               | AWS IAM Role                                                                     |
| **Role ARN**           | Role ARN of the IAM Role you created in Step 1　　　　　　　　　　　　　　　　 　|
| **External ID**        | Your IAM Role's external id from Step 1                                    |

![Credentials](./step2-4.png)

Enable Soracom Funnel

1. Add all desired SIMs to a group, then click SORACOM Funnel.

![Credentials](./step2-5.png)

2. Turn Soracom Funnel on by clicking the switch

3. Configure all the below settings for Funnel

| Parameter    | Content                          |

|--------------|----------------------------------|
| **Service**      | Amazon Data Firehose             |
| **Destination**  | https://firehose.<Your Region code\>.amazonaws.com/<DeliveryStreamName\> (e.g. https://firehose.ap-northeast-1.amazonaws.com/funnel-delivery-stream)|
| **Credentials**  | The credentials you just created |
| **Content Type** | JSON                             |
<br>

![Credentials](./step2-6.png)

4. Click Save

![Credentials](./step2-7.png)

Step 3: Send Data to Soracom Funnel

Now that you have all the set up done, send a request to Funnel's entry point. We have provided the following examples for connecting with TCP, UDP, and HTTP.

[ui-tabs theme=code] [ui-tab title=TCP]

nc funnel.soracom.io 23080
|{"temperature":20} [Enter]
>200
|[Ctrl+C]

[/ui-tab]

[ui-tab title=UDP]

nc -u funnel.soracom.io 23080
|{"temperature":20} [Enter]
>200
|[Ctrl+C]

[/ui-tab]

[ui-tab title=HTTP]

curl -vX POST http://funnel.soracom.io \
|  -d '{
|        "temperature": 20
|      }' \
|  -H "Content-Type: application/json"
>
>* Note: Unnecessary use of -X or --request, POST is already inferred.
>*   Trying 100.127.65.43:80...
>* Connected to funnel.soracom.io (100.127.65.43) port 80 (#0)
>> POST / HTTP/1.1
>> Host: funnel.soracom.io
>> User-Agent: curl/7.74.0
>> Accept: */*
>> Content-Type:application/json
>> Content-Length: 18
>>
>* upload completely sent off: 18 out of 18 bytes
>< HTTP/1.1 204 No Content
>< Date: Fri, 28 Oct 2022 12:54:03 GMT
>< Connection: keep-alive
>< Keep-Alive: timeout=5
><
>* Connection #0 to host funnel.soracom.io left intact

[/ui-tab] [/ui-tabs]

Step 4: Confirm Your Data in S3

Now that you've sent data through Soracom Funnel, let's actually confirm that the data arrived in S3.

1. Open and sign in to the AWS S3 Console

2. Search for and click the S3 Bucket you created in Step 1

![Credentials](./step4-1.png)

3. Click the latest year, month, day, hour folders

![Credentials](./step4-2.png)

4. Click the newly created Object

![Credentials](./step4-3.png)

5. Click Download

![Credentials](./step4-4.png)

5. Open the file and confirm that the data you provided through Soracom Funnel is in the file {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"} {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"} {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}

References

AWS CloudFormation Resource Stack Details

Below are the details of the CloudFormation Resource Stack you created in Step 1.

Terminate Usage

When you want to stop using Funnel to store your data in S3, disable the Soracom Funnel service you enabled in Step 2, then simply delete the CloudFormation Resource Stack you made in Step 1.

1. Navigate to the Amazon CloudFormation Resource Stack you created in Step 1
2. Click Delete

![Credentials](./ref-1.png)