AWS services in this solution
The following AWS services are included in this solution:
AWS service | Description |
---|---|
Amazon
Elastic Load Balancing |
Core. To distribute network traffic to ingestion fleet. |
Amazon ECS |
Core. To run the ingestion module fleet. |
Amazon EC2 |
Core. To provide the underlying computing resources for ingestion fleet. |
Amazon ECR |
Core. To host the container images used by ingestion fleet. |
Amazon S3 |
Core. To store the ingested and processed Clickstream data. And it also stores the service logs and static web assets (frontend user interface). |
AWS Global Accelerator |
Supporting. To improve the availability, performance, and security of the ingestion service in AWS Regions. |
AWS CloudWatch |
Supporting. To monitor the metrics, logs and trace of data pipeline. |
Amazon SNS |
Supporting. To provide topic and email subscription notifications for the alarms of data pipeline. |
Amazon Kinesis Data Streams |
Supporting. To provide the ingestion buffer. |
AWS Lambda |
Supporting. To integrate with kinds of AWS services. For example, sink ingestion data to S3, manage the lifecycle of AWS resources. |
Amazon Managed Streaming for Apache Kafka (MSK) |
Supporting. To provide the ingestion buffer with Apache Kafka. |
Amazon EMR Serverless |
Supporting. To process the ingested data. |
Amazon
Glue |
Supporting. To manage the data catalog of ingested data. |
Amazon EventBridge |
Supporting. To integrate with AWS services with events or schedule. |
Amazon Redshift |
Supporting. To analyze your Clickstream data in data warehouse. |
Amazon Athena |
Supporting. To analyze your Clickstream data in data lake. |
AWS Step Functions |
Supporting. To orchestrate the lifecycle management of project's pipeline. Also it manages the workflow to load data into data warehouse. |
AWS Secrets Manager |
Supporting. To store the credential for OIDC credentials and BI user in Redshift. |
QuickSight |
Supporting. Visual your analysis reporting of your Clickstream data. |
Amazon CloudFront |
Supporting. To made available the static web assets (frontend user interface) and proxy the backend in the same origin. |
Amazon Cognito |
Supporting. To authenticate users (in AWS Regions). |
Amazon API Gateway |
Supporting. To provide the backend APIs. |
Amazon DynamoDB |
Supporting. To store projects data. |
AWS CloudFormation |
Supporting. To provision the AWS resources for the modules of data pipeline. |