Memecahkan masalah Amazon EMR pada operator EKS Spark

Lihat bagian berikut jika Anda mengalami masalah dengan Amazon EMR di operator EKS Spark. Untuk informasi selengkapnya termasuk langkah-langkah untuk menyelesaikan instalasi, lihatMenjalankan pekerjaan Spark dengan operator Spark.

Kesalahan pada instalasi bagan Helm

Jika Anda mengikuti langkah-langkah Instal operator Spark dan mengembalikan INSTALLATION FAILED kesalahan seperti di bawah ini ketika Anda mencoba menginstal atau memverifikasi bagan Helm, Anda mungkin tidak memperoleh token otentikasi ke repositori Amazon ECR untuk operator.

Untuk mengatasi masalah ini, ulangi langkah Instal operator Spark untuk mengautentikasi klien Helm Anda ke registri Amazon ECR. Kemudian, coba langkah instalasi lagi.

Error: INSTALLATION FAILED: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

UnsupportedFileSystemException: Tidak FileSystem untuk skema “s3"

Anda mungkin menemukan pengecualian berikut di utas “main”:

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

Jika ini terjadi, tambahkan pengecualian berikut ke SparkApplication spesifikasi:

 hadoopConf:
   # EMRFS filesystem
   fs.s3.customAWSCredentialsProvider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
   fs.s3.impl: com.amazon.ws.emr.hadoop.fs.EmrFileSystem
   fs.AbstractFileSystem.s3.impl: org.apache.hadoop.fs.s3.EMRFSDelegate
   fs.s3.buffer.dir: /mnt/s3
   fs.s3.getObject.initialSocketTimeoutMilliseconds: "2000"
   mapreduce.fileoutputcommitter.algorithm.version.emr_internal_use_only.EmrFileSystem: "2"
   mapreduce.fileoutputcommitter.cleanup-failures.ignored.emr_internal_use_only.EmrFileSystem: "true"
 sparkConf:
   # Required for EMR Runtime
   spark.driver.extraClassPath: /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/home/hadoop/extrajars/*
   spark.driver.extraLibraryPath: /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native
   spark.executor.extraClassPath: /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/home/hadoop/extrajars/*
   spark.executor.extraLibraryPath: /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native