Inspecting Traffic with Junction

Junction's Inspection mode will parse all packets passing through the VPG and send statistical information to a specified cloud service using a service adapter. From the cloud service, the data can then be connected with a tool in order to build visualizations and perform network analysis.

The following cloud service adapters are available:

• Amazon Kinesis Data Streams
• Amazon Data Firehose
• AWS IoT
• Microsoft Azure Event Hubs
• Google Cloud Pub/Sub

Example Implementation

As each adapter provides access to different cloud services, implementation instructions will vary according to the data analysis tools you use. This guide demonstrates a practical implementation of Junction Inspection with the following architecture:

• Deliver packet statistics to Amazon Kinesis Data Streams.
• Add an AWS Lambda function to format the data and transfer it to Elastic Cloud.
• Analyze the data using Elasticsearch.
• Build packet visualization with Kibana.

Requirements

• A Virtual Private Gateway
• An AWS account
• An Elastic Cloud account

Elastic Cloud Setup

Let's first start by setting up Elasticsearch and Kibana on Elastic Cloud. If you don't have an account, you can sign up for a 14-day trial .

1. Log in to Elastic Cloud.

2. Click Create Cluster.

3. Enter the following options:

   
   • Cluster Size - 1GB memory and 24GB storage.
   • Region - Select a region near you.

   Keep the defaults for the remaining options, then create the cluster.

4. Once the cluster is created, a elastic user and its password will be displayed. Make a note of this information, as we will need it later.

5. Next, from the Kibana section, click the Enable button.

   

Now we need to test that we can connect to Elasticsearch.

6. Open the cluster Overview page. Then from the Endpoints section, click HTTPS. Use the elastic username and password noted earlier to authenticate the connection. If successful, you should see a JSON response similar to the following:

   

7. We also need to check that we can access Kibana. Follow the same process, using the same elastic username and password from earlier.

   

Last, we need to register a template in Elasticsearch, which will tell Elasticsearch about the format of the statistical data that Junction will be sending.

8. Download the Soracom Realtime VPG Metrics template file to your computer.

9. Register it to Elasticsearch by making an HTTP PUT request to your cluster's HTTPS endpoint, specifying the elastic username and password, and using the template file as the HTTP request body:

   curl -X PUT \
   |  -H 'Content-Type:application/json' \
   |  --user elastic:<PASSWORD> \
   |  -d '@/path/to/soracom-realtime-vpg-metrics-template.json' \
   |  "https://xxxxxxxxxxxxxxxxxx.ap-northeast-1.aws.found.io:9243/_template/soracom-vpg"

   Elasticsearch will return a {"acknowledged":true} response when the template is successfully registered.

Amazon Kinesis Data Streams Setup

Next, we need to set up an Amazon Kinesis Data Stream, which is where Junction will send Inspection data.

1. Sign in to the AWS Management Console . From the Services menu, open the Kinesis dashboard. Then click the Data streams button.

   

2. Click the Create data stream button.

   

3. Enter a name for this stream, such as junction-inspection-escloud. We will keep the default settings for this stream. Then click the Create data stream button.

After the Kinesis Data Stream has been created, we need to create credentials that Junction can use in order to connect to Kinesis Data Streams. For security, we will create a new AWS IAM role that only has access to Kinesis Data Streams.

4. From the AWS Services menu, open the IAM dashboard. Then from the Roles, click the Create role button.

   

5. Select AWS account as the trusted entity type and add an External ID. This External ID will be needed for our Soracom Credential Set later. Then click Next.

   

6. Select AmazonKinesisFullAccess as the permission policy and click Next.

   

7. Enter a descriptive name for the role, then click Create role.

8. From the main Roles screen, click on your newly created role and make a note of the Role ARN. This will be needed for the creation of our Soracom Credential Set later.

Lambda Setup

Now with Elasticsearch and Kinesis Data Streams set up, we can create a Lambda function that will take care of sending our Inspection data from Kinesis Data Streams to Elasticsearch.

In order to let Lambda access Kinesis Data Streams, we need to set up a Role.

1. In the IAM dashboard, click the Roles section. Then click the Create role button.

   

2. In the Select role type screen, select AWS service. Under Use case, select Lambda. Then click Next.

   

3. In the Add permissions screen, search for the AWSLambdaKinesisExecutionRole policy and click its to attach it to the new role. Then click Next.

   

4. Last, enter a Role name. We will use this role when setting up Lambda next. Here, we use the name lambda_kinesis_execution. Leave the rest of the settings as default and click Create role.

Now we can create the Lambda function.

5. From the AWS Services menu, open the Lambda dashboard. Click the Create function button.

   

6. Using the Author from scratch option, enter a descriptive function name, then choose Python 3.8 under Runtime and x86_64 under Architecture. Under Permissions click Use an existing role, then select the lambda_kinesis_execution we created earlier. Finally, click Create function.

   

7. In the Code source section copy and paste the Soracom Junction Lambda function code.

   

8. The Elasticsearch cluster endpoint and credentials can now be configured as environment variables. Under the Configuration tab click Environment variables, then click Edit.

   

9. Add the following environment variables then click Save:

   Key	Value
   ES_BASE_URL	Your Elasticsearch cluster HTTPS endpoint, without the trailing /.
   ES_USERNAME	elastic
   ES_PASSWORD	The password created earlier for the elastic user.
   ES_INDEX	A prefix added to the data. Set this as soracom, as this is the format the Elasticsearch template expects.
   ES_TYPE	Stats

Configure Junction

So far, we've completed the following configuration:

• Elasticsearch - We registered the packet inspection template, and our cluster is ready to accept data at its endpoint.
• Kinesis Data Stream - Our Kinesis Data Stream is set up, and the IAM user we created to access it is ready.
• Lambda - Our function is ready to send data from our Kinesis Data Stream to Elasticsearch.

Let's configure our VPG to send Inspection data to our Kinesis Data Stream.

Follow the Inspection configuration instructions from the Junction Configuration documentation, providing the following parameters:

• Service - Amazon Kinesis Data Streams
• Destination - The endpoint of the Kinesis Data Stream we created, in the format of https://kinesis.<YOUR-AWS-REGION>.amazonaws.com/<DELIVERY-STREAM-NAME>, providing the AWS region where your Kinesis Data Stream was created, and the junction-inspection-escloud delivery stream name we set earlier.
• Credential set - A credential set containing the IAM role Role ARN and External ID we created during Kinesis Data Stream setup. For more information on creating credential sets see the Credential Sets documentation.

Once we save these settings, Junction will immediately begin sending Inspection data to our Kinesis Data Stream. Our Lambda trigger will then send the data to Elasticsearch, and we can start to visualize the VPG traffic.

Kibana Configuration

Now that everything is hooked up, we just need to configure our Kibana visualization.

We need to define an Index Pattern that will match the data being sent from Lambda function.

1. Log in to Kibana and click Management, then click Index Pattern.

2. Enter soracom-* as the Index name or pattern. Then click the Create button.

   

Next, let's import a dashboard template that is pre-configured.

3. Download the Soracom Realtime VPG Metrics Dashboard template file.

4. From Kibana, click Management Saved Objects. Then click the Import button, and import the dashboard template.

   

Our Inspection configuration is now fully configured. As traffic passes through our Soracom VPG, Junction will take care of parsing the packets for statistical data. Kinesis Data Streams, Lambda, and Elasticsearch will take care of handling the data, and we can now view realtime statistics from Kibana.

From Kibana, click Dashboard Soracom.

The dashboard template will include information about throughput, protocol, packet destination, and so on.