Real-time streaming for RAG
Publication date: August 12, 2024 (Diagram history)
This architecture demonstrates the integration of streaming data services on AWS with Retrieval Augmented Generation (RAG) in generative AI applications.
Real-time streaming for RAG diagram
The following steps describe the architecture:
-
Data sources for change data capture (CDC) include on-premises or AWS databases such as Oracle, SQL Server, MySQL, PostgreSQL, Amazon Aurora, and Amazon Relational Database Service, all funneling data into your Retrieval Augmented Generation (RAG) model.
-
AWS Database Migration Service and Amazon Managed Streaming for Apache Kafka (Amazon MSK) Connect with the Debezium connector help with one-time data migration of databases and continuous data replication. AWS DMS and Amazon MSK Connect with the Debezium connector capture and stream changes from source databases and apply them in the same order they are captured to the target.
-
Using AWS DMS and Amazon MSK Connect with the Debezium connector enables the streaming of data to Amazon Kinesis Data Streams or Amazon MSK, facilitating the collection and processing of large streams.
-
By using AWS Glue Spark Streaming and Amazon Managed Service for Apache Flink, you can construct specialized data processing pipelines to cater to your data consumption requirements.
-
AWS Lambda, Amazon Data Firehose, and Amazon MSK Connect, which serve as data sink services, enable the direct transfer of source data into destinations like a data lake and Amazon Redshift, among others.
-
Using Aurora PostgreSQL with pgvector, Amazon OpenSearch Service, and Amazon DocumentDB allows the generation of vector embeddings for streamlined data retrieval, vector representation management, scalability, and real-time inference capabilities.
-
Amazon SageMaker AI and Amazon Bedrock offer the means to discover pertinent information within a corpus, conduct similarity search on vectorized domain-specific datasets, and use this data as input for generation models.
-
To preserve user profiles and conversation history, Amazon DocumentDB, Amazon DynamoDB, and Amazon MemoryDB provide suitable options.
-
Use the Amazon Redshift service to ensure data persistence, thereby augmenting the data inputs for the generative AI RAG model.
-
For details on the workings of RAG applications, refer to Retrieval-Augmented Generation (RAG)
.
Further reading
For additional information, see the following resources: