Amazon OpenSearch Trending Queries with AWS Glue and Amazon Bedrock

Publication date: December 18, 2024 (Diagram history)

This architecture demonstrates how to use AWS services like AWS Glue, Amazon Bedrock, Amazon OpenSearch Service, vector embedding, K-means clustering, and LLMs to identify top trending search queries for optimizing content strategy and improving user experience.

Amazon OpenSearch Trending Queries with AWS Glue and Amazon Bedrock

The following steps describe the architecture:

1. End users search articles on the search page. Queries are sent to Amazon OpenSearch Service for results retrieval.

2. Search query logs are streamed through Amazon Kinesis Data Streams using an API Gateway proxy.

3. Amazon Data Firehose consolidates search query logs every 15 minutes at the maximum buffer limit.

4. An Lambda function compresses search query logs for Amazon S3 storage optimization.

5. EventBridge daily scheduler triggers Step Functions for trending query identification.

6. An AWS Glue crawler creates catalog tables for search query logs stored in Amazon S3.

7. An AWS Glue job consolidates and transforms query logs to Parquet files to boost query performance.

8. An AWS Glue crawler creates a catalog table for Parquet files stored in Amazon S3.

9. An AWS Glue job processes data using K-means clustering, creates search query clusters, and stores them in Amazon S3.

10. An AWS Glue crawler creates a catalog table for search query clusters stored in Amazon S3.

11. Athena queries the top n queries per cluster and passes a CSV file as input to a Lambda function.

12. Lambda processes the CSV file in a loop, invokes Amazon Bedrock to identify the most relevant search query per cluster, and stores results in DynamoDB.

13. When the user opens the search page, application logic uses API Gateway to retrieve top trending queries using Lambda and the DynamoDB table for display on the search page.

14. Business analysts use a trending query API to analyze trending search queries and define content strategy.

Further reading

For additional information, refer to the following resources:

• AWS Architecture Icons

• AWS Architecture Center

• AWS Well-Architected

Diagram history

To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.

Change	Description	Date
Initial publication	Reference architecture diagram first published.	December 18, 2024

Note

To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.