AWS Well-Architected design considerations
This solution was designed with best practices from the AWS Well-Architected Framework which helps customers design and operate reliable, secure, efficient, and cost-effective workloads in the cloud.
This section describes how the design principles and best practices of the Well-Architected Framework were applied when building this solution.
Ingesting and processing real-time streaming data requires scalability and low latency to support a variety of applications such as activity tracking, transaction order processing, click-stream analysis, data cleansing, metrics generation, log filtering, indexing, social media analysis, and IoT device data telemetry and metering. These applications are often spiky and process thousands of events per second.
Operational excellence
This section describes how we architected this solution using the principles and best practices of the operational excellence pillar.
The Streaming Data Solution for Amazon Kinesis solution pushes metrics to Amazon CloudWatch to provide observability into the infrastructure; AWS Lambda functions, Amazon Kinesis Streams, Kinesis Data Analytics, Kinesis Data Firehose, S3 buckets, and the rest of the solution components. Errors and failed records that occur during processing are annotated, and the events are stored in Amazon Simple Queue Service.
Security
This section describes how we architected this solution using the principles and best practices of the security pillar.
-
All data storage including Amazon S3 buckets have encryption at rest.
-
All inter-service communications use AWS IAM roles.
-
Communications between end user and Amazon API Gateway uses Bearer token generated and handed by Amazon Cognito.
-
All roles used by the solution follows least-privilege access. That is, it only contains minimum permissions required so the service can function properly.
Reliability
This section describes how we architected this solution using the principles and best practices of the reliability pillar.
The Streaming Data Solution for Amazon Kinesis solution uses AWS Serverless Services wherever possible (examples include AWS Lambda, Amazon API Gateway, S3, Kinesis Data Stream) to ensure high availability and recovery from service failure.
Performance efficiency
This section describes how we architected this solution using the principles and best practices of the performance efficiency pillar.
-
Using serverless architecture throughout this solution.
-
Automatic scaling is turned on by default in Kinesis Data Analytics.
-
The ability to launch this solution in any region that supports AWS services in this solution such as: Amazon API Gateway, Kinesis Data Stream, Kinesis Data Analytics, Kinesis Data Firehose, EC2, S3 Bucket, CloudWatch, and AWS Lambda.
-
Multiple options are available to quickly carry out comparative testing using different types of service configurations.
Cost optimization
This section describes how we architected this solution using the principles and best practices of the cost optimization pillar.
-
Using serverless architecture so that customers only get charged for what they use.
-
Providing an option to the user on whether or not to enable enhanced monitoring (shard-level) for Amazon Kinesis Data Streams. This option is turned off by default to reduce the cost for users who don’t need shard-level data monitoring.
Sustainability
This section describes how we architected this solution using the principles and best practices of the sustainability pillar.
The solution utilizes managed and serverless services, to minimize the environmental impact of the backend services. The solution Serverless design (using Lambda, SQS, API Gateway, and S3) and the use of managed services (such as Kinesis Data Streams) are aimed at reducing carbon footprint compared to the footprint of continually operating on-premises servers.
