Setting up roles and users in Amazon OpenSearch Ingestion

Amazon OpenSearch Ingestion uses a variety of permissions models and IAM roles to allow source applications to write to pipelines, and to allow pipelines to write to sinks. Before you can start ingesting data, you must create one or more IAM roles with specific permissions based on your use case.

At minimum, you require the following roles to set up a successful pipeline.

Name	Description
Management role	Any principal that's managing pipelines (generally a "pipeline admin") needs management access, which includes permissions like osis:CreatePipeline and osis:UpdatePipeline. These permissions allow a user to administer pipelines but not necessarily write data to them.
Pipeline role	The pipeline role, which you specify within the pipeline's YAML configuration, provides the required permissions for a pipeline to write to the domain or collection sink and read from pull-based sources. For more information, see the following topics: Granting Amazon OpenSearch Ingestion pipelines access to domains Granting Amazon OpenSearch Ingestion pipelines access to collections
Ingestion role	The ingestion role contains the osis:Ingest permission for the pipeline resource. This permission allows push-based sources to ingest data into a pipeline.

The following image demonstrates a typical pipeline setup, where a data source such as Amazon S3 or Fluent Bit is writing to a pipeline in a different account. In this case, the client needs to assume the ingestion role in order to access the pipeline. For more information, see Cross-account ingestion.

For a simple setup guide, see Tutorial: Ingesting data into a domain using Amazon OpenSearch Ingestion.

Management role

In addition to the basic osis:* permissions needed to create and modify a pipeline, you also need the iam:PassRole permission for the pipeline role resource. Any AWS service that accepts a role must use this permission. OpenSearch Ingestion assumes the role every time it needs to write data to a sink. This helps administrators ensure that only approved users can configure OpenSearch Ingestion with a role that grants permissions. For more information, see Granting a user permissions to pass a role to an AWS service.

If you're using the AWS Management Console (using blueprints and later checking on your pipeline), you need the following permissions to create and update a pipeline:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Resource":"*",
         "Action":[
            "osis:CreatePipeline",
            "osis:GetPipelineBlueprint",
            "osis:ListPipelineBlueprints",
            "osis:GetPipeline",
            "osis:ListPipelines",
            "osis:GetPipelineChangeProgress",
            "osis:ValidatePipeline",
            "osis:UpdatePipeline"
         ]
      },
      {
         "Resource":[
            "arn:aws:iam::your-account-id:role/pipeline-role"
         ],
         "Effect":"Allow",
         "Action":[
            "iam:PassRole"
         ]
      }
   ]
}

If you're using the AWS CLI (not prevalidating your pipeline or using blueprints), you need the following permissions to create and update a pipeline:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Resource":"*",
         "Action":[
            "osis:CreatePipeline",
            "osis:UpdatePipeline"
         ]
      },
      {
         "Resource":[
            "arn:aws:iam::your-account-id:role/pipeline-role"
         ],
         "Effect":"Allow",
         "Action":[
            "iam:PassRole"
         ]
      }
   ]
}       

Pipeline role

A pipeline needs certain permissions to write to its sink. These permissions depend on whether the sink is an OpenSearch Service domain or an OpenSearch Serverless collection.

In addition, a pipeline might need permissions to pull from the source application (if the source is a pull-based plugin), and permissions to write to an S3 dead letter queue, if configured.

Writing to a domain sink

An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Service domain that is configured as its sink. These permissions include the ability to describe the domain and send HTTP requests to it.

To provide your pipeline with the required permissions to write to a sink, first create an AWS Identity and Access Management (IAM) role with the required permissions. These permissions are the same for public and VPC pipelines. Then, specify the pipeline role in the domain access policy so that the domain can accept write requests from the pipeline.

Finally, specify the role ARN as the value of the sts_role_arn option within the pipeline configuration:

version: "2"
source:
  http:
    ...
processor:
  ...
sink:
  - opensearch:
      ...
      aws:
        sts_role_arn: arn:aws:iam::your-account-id:role/pipeline-role

For instructions to complete each of these steps, see Allowing pipelines to access domains.

Writing to a collection sink

An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Serverless collection that is configured as its sink. These permissions include the ability to describe the collection and send HTTP requests to it.

First, create an IAM role that has the aoss:BatchGetCollection permission against all resources (*). Then, include this role in a data access policy and provide it permissions to create indexes, update indexes, describe indexes, and write documents within the collection. Finally, specify the role ARN as the value of the sts_role_arn option within the pipeline configuration.

For instructions to complete each of these steps, see Allowing pipelines to access collections.

Writing to a dead-letter queue

If you configure your pipeline to write to a dead-letter queue (DLQ), you must include the sts_role_arn option within the DLQ configuration. The permissions included in this role allow the pipeline to access the S3 bucket that you specify as the destination for DLQ events.

You must use the same sts_role_arn in all pipeline components. Therefore, you must attach a separate permissions policy to your pipeline role that provides DLQ access. At minimum, the role must be allowed the S3:PutObject action on the bucket resource:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteToS3DLQ",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-dlq-bucket/*"
    }
  ]
}

You can then specify the role within the pipeline's DLQ configuration:

  ...
  sink:
    opensearch:
      dlq:
        s3:
          bucket: "my-dlq-bucket"
          key_path_prefix: "dlq-files"
          region: "us-west-2"
          sts_role_arn: "arn:aws:iam::your-account-id:role/pipeline-role"

Ingestion role

All source plugins that OpenSearch Ingestion currently supports, with the exception of S3, use a push-based architecture. This means that the source application pushes the data to the pipeline, rather than the pipeline pulling the data from the source.

Therefore, you must grant your source applications the required permissions to ingest data into an OpenSearch Ingestion pipeline. At minimum, the role that signs the request must be granted permission for the osis:Ingest action, which allows it to send data to a pipeline. The same permissions are required for public and VPC pipeline endpoints.

The following example policy allows the associated principal to ingest data into a single pipeline called my-pipeline:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PermitsWriteAccessToPipeline",
      "Effect": "Allow",
      "Action": "osis:Ingest",
      "Resource": "arn:aws:osis:region:your-account-id:pipeline/pipeline-name"
    }
  ]
}

For more information, see Integrating Amazon OpenSearch Ingestion pipelines with other services and applications.

Cross-account ingestion

You might need to ingest data into a pipeline from a different AWS account, such as an application account. To configure cross-account ingestion, define an ingestion role within the same account as the pipeline and establish a trust relationship between the ingestion role and the application account:

{
  "Version": "2012-10-17",
  "Statement": [{
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::external-account-id:root"
      },
     "Action": "sts:AssumeRole"
  }]
}

Then, configure your application to assume the ingestion role. The application account must grant the application role AssumeRole permissions for the ingestion role in the pipeline account.

For detailed steps and example IAM policies, see Providing cross-account ingestion access.