Getting started with Amazon Managed Service for Apache Flink (DataStream API)

This section introduces you to the fundamental concepts of Managed Service for Apache Flink and the DataStream API. It describes the available options for creating and testing your applications. It also provides instructions for installing the necessary tools to complete the tutorials in this guide and to create your first application.

Topics

• Components of the Managed Service for Apache Flink application
• Prerequisites for completing the exercises
• Step 1: Set up an AWS account and create an administrator user
• Step 2: Set up the AWS Command Line Interface (AWS CLI)
• Step 3: Create and run a Managed Service for Apache Flink application
• Step 4: Clean up AWS resources
• Step 5: Next steps

Components of the Managed Service for Apache Flink application

To process data, your Managed Service for Apache Flink application uses a Java/Apache Maven or Scala application that processes input and produces output using the Apache Flink runtime.

An Managed Service for Apache Flink application has the following components:

• Runtime properties: You can use runtime properties to configure your application without recompiling your application code.

• Source: The application consumes data by using a source. A source connector reads data from a Kinesis data stream, an Amazon S3 bucket, etc. For more information, see Sources.

• Operators: The application processes data by using one or more operators. An operator can transform, enrich, or aggregate data. For more information, see DataStream API operators.

• Sink: The application produces data to external sources by using sinks. A sink connector writes data to a Kinesis data stream, a Firehose stream, an Amazon S3 bucket, etc. For more information, see Sinks.

After you create, compile, and package your application code, you upload the code package to an Amazon Simple Storage Service (Amazon S3) bucket. You then create a Managed Service for Apache Flink application. You pass in the code package location, a Kinesis data stream as the streaming data source, and typically a streaming or file location that receives the application's processed data.

Prerequisites for completing the exercises

To complete the steps in this guide, you must have the following:

• Java Development Kit (JDK) version 11. Set the JAVA_HOME environment variable to point to your JDK install location.

• We recommend that you use a development environment (such as Eclipse Java Neon or IntelliJ Idea) to develop and compile your application.

• Git client. Install the Git client if you haven't already.

• Apache Maven Compiler Plugin. Maven must be in your working path. To test your Apache Maven installation, enter the following:

  $ mvn -version

To get started, go to Step 1: Set up an AWS account and create an administrator user.