Update Python applications - Managed Service for Apache Flink

Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.

Update Python applications

Python applications can use connectors in 2 different ways: packaging connectors and other Java dependencies as part of single uber-jar, or use connector jar directly. To fix applications affected by Async Sink deadlock:

  • If the application uses an uber jar, follow the instructions for Update Java applications .

  • To rebuild connector jars from source, use the following steps:

Building connectors from source:

Prerequisites, similar to Flink build requirements:

  • Java 11

  • Maven 3.2.5

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-src.tgz

  2. Uncompress source code:

    tar -xvf flink-1.15.4-src.tgz

  3. Navigate to kinesis connector directory

    cd flink-1.15.4/flink-connectors/flink-connector-kinesis/

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Daws.sdkv2.version=2.20.144

  5. Navigate to kinesis connector directory

    cd ../flink-sql-connector-kinesis

  6. Compile and install sql connector jar:

    mvn clean install -DskipTests -Dfast

  7. Resulting jar will be available at:

    target/flink-sql-connector-kinesis-1.15.4.jar

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-src.tgz

  2. Uncompress source code:

    tar -xvf flink-1.15.4-src.tgz

  3. Navigate to kinesis connector directory

    cd flink-1.15.4/flink-connectors/flink-connector-aws-kinesis-streams/

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Daws.sdk.version=2.20.144

  5. Navigate to kinesis connector directory

    cd ../flink-sql-connector-aws-kinesis-streams

  6. Compile and install sql connector jar:

    mvn clean install -DskipTests -Dfast

  7. Resulting jar will be available at:

    target/flink-sql-connector-aws-kinesis-streams-1.15.4.jar

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-src.tgz

  2. Uncompress source code:

    tar -xvf flink-1.15.4-src.tgz

  3. Navigate to connector directory

    cd flink-1.15.4/flink-connectors/flink-connector-aws-kinesis-firehose/

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Daws.sdk.version=2.20.144

  5. Navigate to sql connector directory

    cd ../flink-sql-connector-aws-kinesis-firehose

  6. Compile and install sql connector jar:

    mvn clean install -DskipTests -Dfast

  7. Resulting jar will be available at:

    target/flink-sql-connector-aws-kinesis-firehose-1.15.4.jar

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-connector-aws-3.0.0/flink-connector-aws-3.0.0-src.tgz

  2. Uncompress source code:

    tar -xvf flink-connector-aws-3.0.0-src.tgz

  3. Navigate to connector directory

    cd flink-connector-aws-3.0.0

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Dflink.version=1.15.4 -Daws.sdk.version=2.20.144

  5. Resulting jar will be available at:

    flink-sql-connector-dynamodb/target/flink-sql-connector-dynamodb-3.0.0.jar