AWS Data Pipeline n'est plus disponible pour les nouveaux clients. Les clients existants de AWS Data Pipeline peuvent continuer à utiliser le service normalement. En savoir plus
Les traductions sont fournies par des outils de traduction automatique. En cas de conflit entre le contenu d'une traduction et celui de la version originale en anglais, la version anglaise prévaudra.
Associer EBS des volumes aux nœuds du cluster
Vous pouvez associer EBS des volumes à n'importe quel type de nœud du EMR cluster au sein de votre pipeline. Pour attacher EBS des volumes à des nœuds coreEbsConfiguration
masterEbsConfiguration
, utilisez et TaskEbsConfiguration
dans votre EmrCluster
configuration.
Cet exemple de EMR cluster Amazon utilise des EBS volumes Amazon pour ses nœuds principaux, de tâches et principaux. Pour plus d'informations, consultez la section Amazon EBS Volumes in Amazon EMR dans le guide EMR de gestion Amazon.
Ces configurations sont facultatives. Vous pouvez les utiliser dans n'importe quel pipeline utilisant un objet EmrCluster
.
Dans le pipeline, cliquez sur la configuration de l'EmrCluster
objet, choisissez EBSConfiguration principale, EBSConfiguration principale ou EBSConfiguration des tâches, puis entrez les détails de configuration similaires à ceux de l'exemple suivant.
{ "objects": [ { "output": { "ref": "S3BackupLocation" }, "input": { "ref": "DDBSourceTable" }, "maximumRetries": "2", "name": "TableBackupActivity", "step": "s3://dynamodb-emr-#{myDDBRegion}/emr-ddb-storage-handler/2.1.0/emr-ddb-2.1.0.jar,org.apache.hadoop.dynamodb.tools.DynamoDbExport,#{output.directoryPath},#{input.tableName},#{input.readThroughputPercent}", "id": "TableBackupActivity", "runsOn": { "ref": "EmrClusterForBackup" }, "type": "EmrActivity", "resizeClusterBeforeRunning": "false" }, { "readThroughputPercent": "#{myDDBReadThroughputRatio}", "name": "DDBSourceTable", "id": "DDBSourceTable", "type": "DynamoDBDataNode", "tableName": "#{myDDBTableName}" }, { "directoryPath": "#{myOutputS3Loc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}", "name": "S3BackupLocation", "id": "S3BackupLocation", "type": "S3DataNode" }, { "name": "EmrClusterForBackup", "coreInstanceCount": "1", "taskInstanceCount": "1", "taskInstanceType": "m4.xlarge", "coreInstanceType": "m4.xlarge", "releaseLabel": "emr-4.7.0", "masterInstanceType": "m4.xlarge", "id": "EmrClusterForBackup", "subnetId": "#{mySubnetId}", "emrManagedMasterSecurityGroupId": "#{myMasterSecurityGroup}", "emrManagedSlaveSecurityGroupId": "#{mySlaveSecurityGroup}", "region": "#{myDDBRegion}", "type": "EmrCluster", "coreEbsConfiguration": { "ref": "EBSConfiguration" }, "masterEbsConfiguration": { "ref": "EBSConfiguration" }, "taskEbsConfiguration": { "ref": "EBSConfiguration" }, "keyPair":
"user-key-pair"
}, { "name": "EBSConfiguration", "id": "EBSConfiguration", "ebsOptimized": "true", "ebsBlockDeviceConfig" : [ { "ref": "EbsBlockDeviceConfig" } ], "type": "EbsConfiguration" }, { "name": "EbsBlockDeviceConfig", "id": "EbsBlockDeviceConfig", "type": "EbsBlockDeviceConfig", "volumesPerInstance" : "2", "volumeSpecification" : { "ref": "VolumeSpecification" } }, { "name": "VolumeSpecification", "id": "VolumeSpecification", "type": "VolumeSpecification", "sizeInGB": "500", "volumeType": "io1", "iops": "1000" }, { "failureAndRerunMode": "CASCADE", "resourceRole": "DataPipelineDefaultResourceRole", "role": "DataPipelineDefaultRole", "pipelineLogUri": "#{myPipelineLogUri}", "scheduleType": "ONDEMAND", "name": "Default", "id": "Default" } ], "parameters": [ { "description": "Output S3 folder", "id": "myOutputS3Loc", "type": "AWS::S3::ObjectKey" }, { "description": "Source DynamoDB table name", "id": "myDDBTableName", "type": "String" }, { "default": "0.25", "watermark": "Enter value between 0.1-1.0", "description": "DynamoDB read throughput ratio", "id": "myDDBReadThroughputRatio", "type": "Double" }, { "default": "us-east-1", "watermark": "us-east-1", "description": "Region of the DynamoDB table", "id": "myDDBRegion", "type": "String" } ], "values": { "myDDBRegion": "us-east-1", "myDDBTableName": "ddb_table", "myDDBReadThroughputRatio": "0.25", "myOutputS3Loc":"s3://s3_path"
, "mySubnetId":"subnet_id"
, "mySlaveSecurityGroup":"slave security group"
, "myMasterSecurityGroup":"master security group"
, "myPipelineLogUri":"s3://s3_path"
} }