Cost - Discovering Hot Topics using Machine Learning

Cost

You are responsible for the cost of the AWS services used while running this solution, which can vary based on the following factors:

  • The volume of data ingested (based on the configuration for the ingestion source — Twitter, RSS feeds, or both).

    When invoking the Search API for Twitter, the search query string and the language filter both contribute to the volume of feeds that the solution ingests.

    • Search query string: Broader and more generic terms result in a larger volume of ingested data. Specific and precise terms result a smaller volume of ingested data.

    • Language filter: The broader list of languages (for example, de,en,es,it,pt,fr,ja,ko,hi,ar,zh-cn,zh-tw) can potentially result in a larger volume of ingested data. Narrow language filters (for example, only en,fr,es) will result in a smaller volume of ingested data.

When ingesting RSS feeds, the number of news sites and the search query string both contribute to the volume of data that the solution ingests.

  • Search query string: Broader and more generic terms result in a larger volume of ingested data. Specific and precise terms result a smaller volume of ingested data.

  • News feed configuration: A broad configuration (topics, languages, and countries) increases the volume of ingested data.

  • The number of queries for visualization.

  • The number of records that require language detection (RSS feeds, and in some cases tweets, which do not have associated languages. When this occurs, the solution uses Amazon Comprehend to detect the data’s source language before processing.

  • The number of records that must be translated into English.

  • The number of images (media assets) ingested with tweets and/or RSS feeds.

As of February 2021, the cost for running this solution in the US East (N. Virginia) Region using Twitter and RSS news feed ingestion, with the default values, and reports queried sporadically, is approximately $300 per week. We recommend creating a budget through AWS Cost Explorer to help manage costs.

Example cost tables

The following tables provide an example cost breakdown for deploying this solution with the default parameters in the US East (N. Virginia) Region for one week with different volume scenarios (excludes free tier).

Example 1: Ingesting – RSS news feeds and 4,000 tweets a week

AWS service Dimensions Cost
Amazon Athena 70 queries/week and 100 GB data scanned/query $40.00/week
Amazon CloudWatch Event Rule – 1 5,040 events/week (runs every 2 mins) $0.01/week
Amazon CloudWatch Event Rule – 2 14 events/week (runs every day) $0.000084/week
Amazon Comprehend (Pre-trained, Sync APIs) 3 Sync APIs, 280 characters long, 4,000 (tweets) requests/API + news feeds $120.00/week
Amazon Comprehend (Pre-trained, Topic Modeling Job) 7 Twitter jobs/week + 7 news feed jobs with 25 KB data/job ~$14.00/week
Amazon DynamoDB Records inserted with TTL 7-day expiry to keep state for Twitter search and news feeds, on-demand capacity ~$0.16/week
Amazon Kinesis Data Firehose 13 firehose, total 3 GB/week $0.26/week
Amazon Kinesis Data Streams 4,000 records/week, 80 MB + news feeds $2.52/week
Amazon Simple Queue Service (Amazon SQS) 14 queues (regular queues + DLQ) processing news feed information + 4000 tweets every week $0.67
Amazon Rekognition – Label Detection 4,000 tweets, 60% of tweets have images = 2,400 tweets 2 = images a tweet on average = 4,800 images + images associated with news feeds $24.00/week
Amazon Rekognition – Text Detection 4,000 tweets, 60% of tweets have images = 2,400 tweets = 2 images a tweet on average = 4,800 images + images associated with news feeds $24.00/week
Amazon Simple Storage Service (Amazon S3) 5 buckets, 75 GB ~$5.00/week
Amazon Translate 280 characters/tweet with 4,000 tweets, assumes 50% of 4,000 tweets in non-English language = 2,000 tweets = 560,000 characters + news feed from different languages translated to English $70.00/week
AWS Lambda (128 MB)
  • 128 MB - 15 functions (total of 89,529 invocations/week and duration of 4,739 millisecond/week)

  • 256 MB - 5 functions (total of 31,238 invocations/week and duration of 2,644 millisecond/week)

  • 512 MB - 2 function (total of 84 invocations/week and duration of 717 milliseconds/ week)

~$1.30/week
AWS Step Functions 2 workflow definitions ~ 30 states (in all) $21.00/week
AWS Key Management Service Using AWS managed keys with DynamoDB, S3 (SSE-S3), Kinesis Data Streams, SQS $5.00
Total ~$330.00/week

Example 2 – Ingesting 4,000 tweets per week

AWS service Dimensions Cost
Amazon Athena 70 queries/week and 100 GB data scanned/query $40.00/week
Amazon CloudWatch Events – 1 5,040 events/week (runsevery 2 mins) $0.01/week
Amazon CloudWatch Events – 2 7 events/week (runs every 2 hours) $0.000084/week
Amazon Comprehend (Pre-trained, Sync APIs) 3 Sync APIs, 280 characters long, 4,000 (tweets) requests/API $1.20/week
Amazon Comprehend (Pre-trained, Topic Modeling Job) 7 jobs/week + 25 KB data/job ~$7.00/week
Amazon DynamoDB Records inserted with TTL 7-day expiry, on-demand capacity ~$0.08/week
Amazon Kinesis Data Firehose 13 firehose, total 3 GB/week ~$0.26/week
Amazon Kinesis Data Streams 4,000 records/week, 4 GB $1.00/week
Amazon Simple Queue Service (Amazon SQS) 14 queues (regular queues + DLQ) processing + 4000 tweets every week $0.67
Amazon Rekognition – Label Detection 4,000 tweets, assume 60% of tweets have images = 2,400 tweets = 2 images a tweet on average = 4,800 images $4.80/week
Amazon Rekognition – Text Detection 4,000 tweets, 60% of tweets have images = 2,400 tweets = 2 images a tweet on average = 4,800 images $4.80/week
Amazon Simple Storage Service (Amazon S3) 5 buckets, 75 GB ~$5.00/week
Amazon Translate 280 characters/tweet with 4,000 tweets, assumes 50% of 4,000 tweets in non-English language = 2,000 tweets = 560,000 characters $8.40/week
AWS Lambda (128 MB)
  • 128 MB - 15 functions (total of 89,529 invocations/week and duration of 4,739 millisecond/week)

  • 256 MB - 5 functions (total of 31,238 invocations/week and duration of 2,644 millisecond/week)

  • 512 MB - 2 function (total of 84 invocations/week and duration of 717 milliseconds/week)

~$1.30/week
AWS Step Functions 2 workflow definitions ~ 30 states (in all) $2.00/week
AWS Key Management Service Using AWS managed keys with DynamoDB, S3 (SSE-S3), Kinesis Data Streams, SQS $5.00
Total ~$100.00/week

Example 3 – Ingesting 400,000 tweets per week

AWS service Dimensions Cost
Amazon Athena 70 queries/week and 100 GB data scanned/query $40.00/week
Amazon CloudWatch Events – 1 5,040 events/week (trigger every 2 mins) $0.01/week
Amazon CloudWatch Events – 2 7 events/week (runs every day) $0.000084/week
Amazon Comprehend (Pre-trained, Sync APIs) 3 Sync APIs, 280 characters long, 400,000 (tweets) requests/API $150.00/week
Amazon Comprehend (Pre-trained, Topic Modeling Job) 7 jobs/week + 1 MB data/job ~$7.00/week
Amazon DynamoDB Records inserted with TTL 7-day expiry, on-demand capacity ~$0.08/week
Amazon Kinesis Data Firehose 13 firehose, total 40 GB/week ~$1.50/week
Amazon Simple Queue Service 14 queues (regular queues + DLQ) processing news feed information + 4,000 tweets every week $0.67
Amazon Kinesis Data Streams 400,000 records/week, 4 GB $11.12/week
Amazon Rekognition – Label Detection 400,000 tweets, assumes 60% of tweets have images = 240,000 tweets = 2 images a tweet on average = 480,000 images $480.00/week
Amazon Rekognition – Text Detection 400,000 tweets, assumes 60% of tweets have images = 240,000 tweets = 2 images a tweet on average = 480,000 images $480.00/week
Amazon Simple Storage Service (Amazon S3) 5 buckets, 750 GB ~$20.00/week
Amazon Translate 280 characters/tweet with 400,000 tweets, assumes 50% tweets are in a non-English language = 200,000 tweets = 56 M characters $840.00/week
AWS Lambda (128 MB)
  • 128 MB - 15 functions (total of 89,529 invocations/week and duration of 4739 millisecond/week)

  • 256 MB - 5 functions (total of 31,238 invocations/week and duration of 2644 millisecond/week)

  • 512 MB - 512 MB - 2 function (total of 84 invocations/week and duration of 717 milliseconds/ week)

~$100.00/week
AWS Step Functions 2 workflow definitions ~ 30 states (in all) $400.00/week
Total ~$2,100.00/week