CreateDevEndpoint - AWS Glue

CreateDevEndpoint

Creates a new development endpoint.

Request Syntax

{ "Arguments": { "string" : "string" }, "EndpointName": "string", "ExtraJarsS3Path": "string", "ExtraPythonLibsS3Path": "string", "GlueVersion": "string", "NumberOfNodes": number, "NumberOfWorkers": number, "PublicKey": "string", "PublicKeys": [ "string" ], "RoleArn": "string", "SecurityConfiguration": "string", "SecurityGroupIds": [ "string" ], "SubnetId": "string", "Tags": { "string" : "string" }, "WorkerType": "string" }

Request Parameters

For information about the parameters that are common to all actions, see Common Parameters.

The request accepts the following data in JSON format.

Arguments

A map of arguments used to configure the DevEndpoint.

Type: String to string map

Map Entries: Minimum number of 0 items. Maximum number of 100 items.

Required: No

EndpointName

The name to be assigned to the new DevEndpoint.

Type: String

Required: Yes

ExtraJarsS3Path

The path to one or more Java .jar files in an S3 bucket that should be loaded in your DevEndpoint.

Type: String

Required: No

ExtraPythonLibsS3Path

The paths to one or more Python libraries in an Amazon S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.

Note

You can only use pure Python libraries with a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.

Type: String

Required: No

GlueVersion

Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for running your ETL scripts on development endpoints.

For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.

Development endpoints that are created without specifying a Glue version default to Glue 0.9.

You can specify a version of Python support for development endpoints by using the Arguments parameter in the CreateDevEndpoint or UpdateDevEndpoint APIs. If no arguments are provided, the version defaults to Python 2.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 255.

Pattern: ^\w+\.\w+$

Required: No

NumberOfNodes

The number of AWS Glue Data Processing Units (DPUs) to allocate to this DevEndpoint.

Type: Integer

Required: No

NumberOfWorkers

The number of workers of a defined workerType that are allocated to the development endpoint.

The maximum number of workers you can define are 299 for G.1X, and 149 for G.2X.

Type: Integer

Required: No

PublicKey

The public key to be used by this DevEndpoint for authentication. This attribute is provided for backward compatibility because the recommended attribute to use is public keys.

Type: String

Required: No

PublicKeys

A list of public keys to be used by the development endpoints for authentication. The use of this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.

Note

If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys. Call the UpdateDevEndpoint API with the public key content in the deletePublicKeys attribute, and the list of new keys in the addPublicKeys attribute.

Type: Array of strings

Array Members: Maximum number of 5 items.

Required: No

RoleArn

The IAM role for the DevEndpoint.

Type: String

Pattern: arn:aws:iam::\d{12}:role/.*

Required: Yes

SecurityConfiguration

The name of the SecurityConfiguration structure to be used with this DevEndpoint.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 255.

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*

Required: No

SecurityGroupIds

Security group IDs for the security groups to be used by the new DevEndpoint.

Type: Array of strings

Required: No

SubnetId

The subnet ID for the new DevEndpoint to use.

Type: String

Required: No

Tags

The tags to use with this DevEndpoint. You may use tags to limit access to the DevEndpoint. For more information about tags in AWS Glue, see AWS Tags in AWS Glue in the developer guide.

Type: String to string map

Map Entries: Minimum number of 0 items. Maximum number of 50 items.

Key Length Constraints: Minimum length of 1. Maximum length of 128.

Value Length Constraints: Minimum length of 0. Maximum length of 256.

Required: No

WorkerType

The type of predefined worker that is allocated to the development endpoint. Accepts a value of Standard, G.1X, or G.2X.

  • For the Standard worker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker.

  • For the G.1X worker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.

  • For the G.2X worker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.

Known issue: when a development endpoint is created with the G.2X WorkerType configuration, the Spark drivers for the development endpoint will run on 4 vCPU, 16 GB of memory, and a 64 GB disk.

Type: String

Valid Values: Standard | G.1X | G.2X | G.025X | G.4X | G.8X | Z.2X

Required: No

Response Syntax

{ "Arguments": { "string" : "string" }, "AvailabilityZone": "string", "CreatedTimestamp": number, "EndpointName": "string", "ExtraJarsS3Path": "string", "ExtraPythonLibsS3Path": "string", "FailureReason": "string", "GlueVersion": "string", "NumberOfNodes": number, "NumberOfWorkers": number, "RoleArn": "string", "SecurityConfiguration": "string", "SecurityGroupIds": [ "string" ], "Status": "string", "SubnetId": "string", "VpcId": "string", "WorkerType": "string", "YarnEndpointAddress": "string", "ZeppelinRemoteSparkInterpreterPort": number }

Response Elements

If the action is successful, the service sends back an HTTP 200 response.

The following data is returned in JSON format by the service.

Arguments

The map of arguments used to configure this DevEndpoint.

Valid arguments are:

  • "--enable-glue-datacatalog": ""

You can specify a version of Python support for development endpoints by using the Arguments parameter in the CreateDevEndpoint or UpdateDevEndpoint APIs. If no arguments are provided, the version defaults to Python 2.

Type: String to string map

Map Entries: Minimum number of 0 items. Maximum number of 100 items.

AvailabilityZone

The AWS Availability Zone where this DevEndpoint is located.

Type: String

CreatedTimestamp

The point in time at which this DevEndpoint was created.

Type: Timestamp

EndpointName

The name assigned to the new DevEndpoint.

Type: String

ExtraJarsS3Path

Path to one or more Java .jar files in an S3 bucket that will be loaded in your DevEndpoint.

Type: String

ExtraPythonLibsS3Path

The paths to one or more Python libraries in an S3 bucket that will be loaded in your DevEndpoint.

Type: String

FailureReason

The reason for a current failure in this DevEndpoint.

Type: String

GlueVersion

Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for running your ETL scripts on development endpoints.

For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 255.

Pattern: ^\w+\.\w+$

NumberOfNodes

The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.

Type: Integer

NumberOfWorkers

The number of workers of a defined workerType that are allocated to the development endpoint.

Type: Integer

RoleArn

The Amazon Resource Name (ARN) of the role assigned to the new DevEndpoint.

Type: String

Pattern: arn:aws:iam::\d{12}:role/.*

SecurityConfiguration

The name of the SecurityConfiguration structure being used with this DevEndpoint.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 255.

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*

SecurityGroupIds

The security groups assigned to the new DevEndpoint.

Type: Array of strings

Status

The current status of the new DevEndpoint.

Type: String

SubnetId

The subnet ID assigned to the new DevEndpoint.

Type: String

VpcId

The ID of the virtual private cloud (VPC) used by this DevEndpoint.

Type: String

WorkerType

The type of predefined worker that is allocated to the development endpoint. May be a value of Standard, G.1X, or G.2X.

Type: String

Valid Values: Standard | G.1X | G.2X | G.025X | G.4X | G.8X | Z.2X

YarnEndpointAddress

The address of the YARN endpoint used by this DevEndpoint.

Type: String

ZeppelinRemoteSparkInterpreterPort

The Apache Zeppelin port for the remote Apache Spark interpreter.

Type: Integer

Errors

For information about the errors that are common to all actions, see Common Errors.

AccessDeniedException

Access to a resource was denied.

HTTP Status Code: 400

AlreadyExistsException

A resource to be created or added already exists.

HTTP Status Code: 400

IdempotentParameterMismatchException

The same unique identifier was associated with two different records.

HTTP Status Code: 400

InternalServiceException

An internal service error occurred.

HTTP Status Code: 500

InvalidInputException

The input provided was not valid.

HTTP Status Code: 400

OperationTimeoutException

The operation timed out.

HTTP Status Code: 400

ResourceNumberLimitExceededException

A resource numerical limit was exceeded.

HTTP Status Code: 400

ValidationException

A value could not be validated.

HTTP Status Code: 400

See Also

For more information about using this API in one of the language-specific AWS SDKs, see the following: