AWS::OSIS::Pipeline

The AWS::OSIS::Pipeline resource creates an Amazon OpenSearch Ingestion pipeline.

Syntax

To declare this entity in your AWS CloudFormation template, use the following syntax:

JSON

{
  "Type" : "AWS::OSIS::Pipeline",
  "Properties" : {
      "BufferOptions" : BufferOptions,
      "EncryptionAtRestOptions" : EncryptionAtRestOptions,
      "LogPublishingOptions" : LogPublishingOptions,
      "MaxUnits" : Integer,
      "MinUnits" : Integer,
      "PipelineConfigurationBody" : String,
      "PipelineName" : String,
      "Tags" : [ Tag, ... ],
      "VpcOptions" : VpcOptions
    }
}

YAML

Type: AWS::OSIS::Pipeline
Properties:
  BufferOptions: 
    BufferOptions
  EncryptionAtRestOptions: 
    EncryptionAtRestOptions
  LogPublishingOptions: 
    LogPublishingOptions
  MaxUnits: Integer
  MinUnits: Integer
  PipelineConfigurationBody: String
  PipelineName: String
  Tags: 
    - Tag
  VpcOptions: 
    VpcOptions

Properties

BufferOptions

Options that specify the configuration of a persistent buffer. To configure how OpenSearch Ingestion encrypts this data, set the EncryptionAtRestOptions. For more information, see Persistent buffering.

Required: No

Type: BufferOptions

Update requires: No interruption

EncryptionAtRestOptions

Options to control how OpenSearch encrypts buffer data.

Required: No

Type: EncryptionAtRestOptions

Update requires: No interruption

LogPublishingOptions

Key-value pairs that represent log publishing settings.

Required: No

Type: LogPublishingOptions

Update requires: No interruption

MaxUnits

The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

Required: Yes

Type: Integer

Minimum: 1

Maximum: 384

Update requires: No interruption

MinUnits

The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

Required: Yes

Type: Integer

Minimum: 1

Maximum: 384

Update requires: No interruption

PipelineConfigurationBody

The Data Prepper pipeline configuration in YAML format.

Required: Yes

Type: String

Minimum: 1

Maximum: 24000

Update requires: No interruption

PipelineName

The name of the pipeline.

Required: Yes

Type: String

Pattern: [a-z][a-z0-9\-]+

Minimum: 3

Maximum: 28

Update requires: Replacement

Tags

List of tags to add to the pipeline upon creation.

Required: No

Type: Array of Tag

Update requires: No interruption

VpcOptions

Options that specify the subnets and security groups for an OpenSearch Ingestion VPC endpoint.

Required: No

Type: VpcOptions

Update requires: No interruption

Return values

Ref

When the logical ID of this resource is provided to the Ref intrinsic function, Ref returns the resource name, such as mystack-abc1d2efg3h4. For more information about using the Ref function, see Ref.

Fn::GetAtt

GetAtt returns a value for a specified attribute of this type. For more information, see Fn::GetAtt. The following are the available attributes and sample return values.

IngestEndpointUrls

A list of the ingestion endpoints for the pipeline that you can send data to. Currently, only a single ingestion endpoint is supported for a pipeline. For example, my-pipeline-123456789012.us-east-1.osis.amazonaws.com.

PipelineArn

The Amazon Resource Name (ARN) of the pipeline.

VpcEndpoints

The VPC interface endpoints that have access to the pipeline.

VpcEndpointService

The VPC endpoint service name for the pipeline.

Examples

Create an OpenSearch Ingestion pipeline

The following example creates an OpenSearch Ingestion pipeline running version 2.x of Data Prepper with log publishing enabled.

JSON

{
   "Parameters": {
      "PipelineConfigurationBody": {
         "Type": "String"
      }
   },
   "Resources": {
      "OpenSearchIngestionPipeline": {
         "Type": "AWS::OSIS::Pipeline",
         "Properties": {
            "LogPublishingOptions": {
               "IsLoggingEnabled": true,
               "CloudWatchLogDestination": {
                  "LogGroup": "/aws/OpenSearchService/IngestionService/test-pipeline"
               }
            },
            "MinUnits": 3,
            "MaxUnits": 9,
            "PipelineConfigurationBody": {
               "Value": {
                  "Ref": "PipelineConfigurationBody"
               }
            },
            "PipelineName": "test-pipeline"
         }
      }
   },
   "Outputs": {
      "PipelineArn": {
         "Value": {
            "Ref": "OpenSearchIngestionPipeline"
         }
      },
      "IngestEndpointUrls": {
         "Value": {
            "Fn::Select": [
               0,
               {
                  "Fn::GetAtt": [
                     "OpenSearchIngestionPipeline",
                     "IngestEndpointUrls"
                  ]
               }
            ]
         }
      }
   }
}

YAML

Parameters:
  PipelineConfigurationBody:
    Type: String
Resources:
  OpenSearchIngestionPipeline:
    Type: 'AWS::OSIS::Pipeline'
    Properties:
      LogPublishingOptions:
        IsLoggingEnabled: true
        CloudWatchLogDestination:
          LogGroup: /aws/vendedlogs/test-pipeline
      MinUnits: 3
      MaxUnits: 9
      PipelineConfigurationBody:
        Ref: PipelineConfigurationBody
      PipelineName: test-pipeline
Outputs:
  PipelineArn:
    Value:
      Ref: OpenSearchIngestionPipeline
  IngestEndpointUrls:
    Value: !Select [0, !GetAtt OpenSearchIngestionPipeline.IngestEndpointUrls]