Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”

Prerequisites for creating knowledge base evaluations in Amazon Bedrock

Focus mode
Prerequisites for creating knowledge base evaluations in Amazon Bedrock - Amazon Bedrock

To create an evaluation job that uses knowledge bases, you need access to specific service level resources, and Amazon Bedrock foundation models. Use the linked topics to learn more about getting setting up.

Prior to starting the model evaluation job, check to make sure that you have ingested and synced all the data in your knowledge base.

Required service level resources to start a model evaluation job that uses a Amazon Bedrock Knowledge Bases
  1. You need access to at least one of the following Amazon Bedrock foundation models. To learn more about gaining access to models, see Access Amazon Bedrock foundation models.

    • Mistral Large – mistral.mistral-large-2402-v1:0

    • Anthropic Claude 3.5 Sonnet – anthropic.claude-3-5-sonnet-20240620-v1:0

    • Anthropic Claude 3 Haiku – anthropic.claude-3-haiku-20240307-v1:0

    • Meta Llama 3.1 70B Instruct – meta.llama3-1-70b-instruct-v1:0

  2. Create a prompt dataset. Your prompt dataset represents the user queries you want to use to see how well the knowledge base retrieves information and generates responses. For more information, see Use a prompt dataset for a knowledge base evaluation in Amazon Bedrock.

  3. To create a model evaluation job that uses a Amazon Bedrock Knowledge Bases you need access to the https://console.aws.amazon.com/bedrock/, AWS Command Line Interface, or a supported AWS SDK. For more information about the required IAM actions and resources, see the Required permissions to create a Amazon Bedrock Knowledge Bases evaluation job section that follows.

  4. When the model evaluation job starts, a service role is used to perform actions on your behalf. To learn more about required IAM actions and trust policy requirements, see Service role requirements for knowledge base evaluation jobs.

  5. Amazon Simple Storage Service – All data used in the model evaluation job must be placed in a Amazon S3 bucket. Model evaluation job created using the Amazon Bedrock console require that you specify the correct CORS permissions on the bucket.

  6. During a knowledge base evaluation job, Amazon Bedrock makes a temporary copy of your data, which Amazon Bedrock encrypts by using an AWS KMS key. You can have Amazon Bedrock encrypt this data with a key that Amazon Bedrock owns, or you can provide a key that you own. If you want to use your own key, you must add the required permissions to the KMS key policy. For more information see Data encryption for knowledge base evaluation jobs.

Required permissions to create a Amazon Bedrock Knowledge Bases evaluation job

This section covers the required IAM policy requirements for the user, group or role who wants to create a Amazon Bedrock Knowledge Bases evaluation job. If you are looking for details about the service role and trust policy requirements, see Service role requirements for knowledge base evaluation jobs.

The following policy contains the minimum set of IAM actions and resources in Amazon Bedrock and Amazon S3 that are required to create an Amazon Bedrock Knowledge Bases evaluation job using the Amazon Bedrock console.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "BedrockConsole", "Effect": "Allow", "Action": [ "bedrock:CreateEvaluationJob", "bedrock:GetEvaluationJob", "bedrock:ListEvaluationJobs", "bedrock:StopEvaluationJob", "bedrock:GetCustomModel", "bedrock:ListCustomModels", "bedrock:CreateProvisionedModelThroughput", "bedrock:UpdateProvisionedModelThroughput", "bedrock:GetProvisionedModelThroughput", "bedrock:ListProvisionedModelThroughputs", "bedrock:GetImportedModel", "bedrock:ListImportedModels", "bedrock:ListTagsForResource", "bedrock:UntagResource", "bedrock:TagResource" ], "Resource": [ "arn:aws:bedrock:us-west-2::foundation-model/model-id-of-foundational-model", "arn:aws:bedrock:us-west-2:account-id:inference-profile/*", "arn:aws:bedrock:us-west-2:account-id:provisioned-model/*", "arn:aws:bedrock:us-west-2:account-id:imported-model/*" ] }, { "Sid": "BedrockKnowledgeBaseConsole", "Effect": "Allow", "Action": [ "bedrock:GetKnowledgeBase", "bedrock:ListKnowledgeBases" ], "Resource": [ "arn:aws:bedrock:us-west-2:account-id:knowledge-base/*" ] }, { "Sid": "AllowConsoleS3AccessForModelEvaluation", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetBucketCORS", "s3:ListBucket", "s3:ListBucketVersions", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::my_output_bucket", "arn:aws:s3:::input_datasets/prompts.jsonl" ] } ] }
PrivacySite termsCookie preferences
© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved.