Menu
Streaming Analytics Pipeline
Streaming Analytics Pipeline

Appendix B: YAML File Configuration

The Streaming Analytics Pipeline includes a YAML file that contains configuration information for the Amazon Kinesis Analytics application that the solution creates. Review the parameters in the YAML file and modify them as necessary for your implementation. Then, upload the file to an Amazon S3 bucket.


        Streaming Analytics Pipeline on AWS view YAML config button
      streaming-analytics-pipeline-config.yaml: Use this file to specify your Amazon Kinesis Analytics application configuration.

Parameter Default Description
Input Format Type CSV The format of the records of the source stream. Choose CSV or JSON.
Record Column Delimiter “,”

The column delimiter of CSV-formatted data from the source stream. For example, “|” or “,”.

Note

Leave this parameter blank if you chose JSON as your Input Format Type.

Record Row Delimiter “/n”

The row delimiter of CSV-formatted data from the source stream. For example, “/n”.

Note

Leave this parameter blank if you chose JSON as your Input Format Type.

Record Row Path “$”

The path to the top-level parent that contains the records.

Note

Leave this parameter blank if you chose CSV as your Input Format Type.

Output Format Type CSV The format of the analyzed data that is put in the output stream. Choose CSV or JSON.
Columns <Requires Input> A list of dictionary values specifying the name, SQL type, and (if necessary) record row path mapping. For example, CSV: {Name: pressure, SqlType: DOUBLE} or JSON: {Name: pressure, SqlType: DOUBLE, Mapping: $.pressure}.
SQL Code <Requires Input> The Amazon Kinesis Analytics application code. The code will be copied to the application.

The YAML configuration file is not required to run this solution. If you do not specify a file location, the solution will launch an Kinesis Analytics application with the following configuration with a “catch all” schema.

Note

If you provide a YAML configuration file location, you must complete the Format section of the file.

Copy
# Update this file according to your Input Schema and application code # Note: pay attention to indentation - it matters format: InputFormatType: CSV RecordColumnDelimiter: "," RecordRowDelimiter: "\n" RecordRowPath: "$" OutputFormatType: CSV columns: - {Name: temp, SqlType: TINYINT} - {Name: segmentId, SqlType: CHAR(4)} - {Name: sensorIp, SqlType: VARCHAR(15)} - {Name: pressure, SqlType: DOUBLE} - {Name: incline, SqlType: DOUBLE} - {Name: flow, SqlType: BIGINT} - {Name: captureTs, SqlType: TIMESTAMP} - {Name: sensorId, SqlType: CHAR(4)} sql_code: | -- Paste your SQL code here CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" ( temp TINYINT, sensorIp VARCHAR(15), sensorId CHAR(4), captureTs TIMESTAMP, pressure DOUBLE); CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM" SELECT STREAM "temp", "sensorIp", "sensorId", "captureTs", "pressure" FROM "SOURCE_SQL_STREAM_001";