Using AWS CloudFormation to Set Up Neptune-to-Neptune Replication with the Streams Consumer Application - Amazon Neptune

Using AWS CloudFormation to Set Up Neptune-to-Neptune Replication with the Streams Consumer Application

You can use an AWS CloudFormation template to set up the Neptune streams consumer application to support Neptune-to-Neptune replication.

Choose an AWS CloudFormation Template for Your Region

To launch the appropriate AWS CloudFormation stack on the AWS CloudFormation console, choose one of the Launch Stack buttons in the following table, depending on the AWS Region that you want to use.

Region View View in Designer Launch
US East (N. Virginia) View View in Designer
US East (Ohio) View View in Designer
US West (N. California) View View in Designer
US West (Oregon) View View in Designer
Canada (Central) View View in Designer
South America (São Paulo) View View in Designer
Europe (Stockholm) View View in Designer
Europe (Ireland) View View in Designer
Europe (London) View View in Designer
Europe (Paris) View View in Designer
Europe (Frankfurt) View View in Designer
Middle East (Bahrain) View View in Designer
Asia Pacific (Tokyo) View View in Designer
Asia Pacific (Hong Kong) View View in Designer
Asia Pacific (Seoul) View View in Designer
Asia Pacific (Singapore) View View in Designer
Asia Pacific (Sydney) View View in Designer
Asia Pacific (Mumbai) View View in Designer
China (Ningxia) View View in Designer
AWS GovCloud (US-West) View View in Designer
AWS GovCloud (US-East) View View in Designer

On the Create Stack page, choose Next.

Add Details About the Neptune Streams Consumer Stack You Are Creating

The Specify Stack Details page provides properties and parameters that you can use to control the setup of the application:

Stack Name – The name of the new AWS CloudFormation stack that you're creating. You can generally use the default value, NeptuneStreamPoller.

Under Parameters, provide the following:

Network Configuration for the VPC Where the Streams Consumer Runs

  • VPC – Provide the name of the VPC where the polling Lambda function will run.

  • List of Subnet IDs – The subnets to which a network interface is established. Add subnets corresponding to your Neptune cluster.

  • List of Security Group Ids – Provide the IDs of security groups that grant write inbound access to your source Neptune DB cluster.

  • List of Route Table Ids – This is needed to create an Amazon DynamoDB endpoint in your Neptune VPC, if you do not already have one. You must provide a comma-separated list of route table IDs associated with the subnets.

  • Require to create Dynamo DB VPC Endpoint – A Boolean value that defaults to true. You only need to change it to false if you have already created a DynamoDB endpoint in your VPC.

  • Require to create Monitoring VPC Endpoint – A Boolean value that defaults to true. You only need to change it to false if you have already created a monitoring endpoint in your VPC.

Stream Poller

  • Application Name – You can generally leave this set to the default (NeptuneStream). If you use a different name, it must be unique.

  • Memory size for Lambda Poller – Used to set the memory size available to the lambda poller function. The default value is 2,048 megabytes.

  • Lambda Runtime – The language used in the Lambda function that retrieves items from the Neptune stream. You can set this either to python3.6 or to java8.

  • S3 Bucket having Lambda code artifacts – Leave this blank unless you are using a custom Lambda polling function that loads from a different S3 bucket.

  • S3 Key corresponding to Lambda Code artifacts – Leave this blank unless you are using a custom Lambda polling function.

  • Logging level for Lambda – In general, leave this set to the default value, INFO.

  • Managed Policies for Lambda Execution – In general, leave this blank unless you are using a custom Lambda polling function.

  • Stream Records Handler – In general, leave this blank unless you are using a custom handler for the records in Neptune streams.

  • Maximum records Fetched from Stream – You can use this parameter to tune performance. The default (100) is a good place to start. The maximum allowable is 10,000. The higher the number, the fewer network calls are needed to read records from the stream, but the more memory is required to process the records.

  • Max wait time between two Polls (in Seconds) – Determines how frequently the Lambda poller is invoked to poll the Neptune streams. Set this value to 0 for continuous polling. The maximum value is 3,600 seconds (1 hour). The default value (60 seconds) is a good place to start, depending on how fast your graph data changes.

  • Maximum Continuous polling period (in Seconds) – Used to set a timeout for the Lambda polling function. It should be between 5 seconds and 900 seconds. The default value (600 seconds) is a good place to start.

  • Step Function Fallback Period – The number of step-function-fallback-period units to wait for the poller, after which the step function is called through Amazon CloudWatch Events to recover from a failure. The default (5 minutes) is a good place to start.

  • Step Function Fallback Period Unit – The time units used to measure the preceding Step Function Fallback Period (minutes, hours, days). The default (minutes) is generally sufficient.

  • Data replication scope – Determines whether to replicate both nodes and edges, or only nodes to Elastic search (this applies to Gremlin engine data only). The default value (All) is generally a good place to start.

  • Ignore Elastic Search missing document error – Flag to determine whether a missing document error in Elastic Search can be ignored. Missing document errors can occur rarely but will need manual intervention if not ignored. The default value (True) is generally a good place to start.

Neptune Stream

  • Endpoint of source Neptune Stream – (Required) This takes one of two forms:

    • https://your DB cluster:port/gremlin/stream

    • https://your DB cluster:port/sparql/stream

  • Neptune Query Engine – Choose Gremlin or SPARQL.

  • Is IAM Auth Enabled? – If your Neptune DB cluster is using IAM authentication, set this parameter to true.

  • Neptune Cluster Resource Id – If your Neptune DB cluster is using IAM authentication, set this parameter to the cluster resource ID. The resource ID is not the same as the cluster ID. Rather, it takes the form: cluster- followed by 28 alpha-numeric characters. It can be found under Cluster Details in the Neptune console.

If you are creating an Elasticsearch streams consumer, complete the following fields:

Target Elasticsearch Cluster

  • Endpoint for Elastic Search Service – Provide the endpoint for the Elasticsearch service in your VPC.

  • Number of Shards for Elastic Search Index – The default value (5) is generally a good place to start.

  • Number of Replicas for Elastic Search Index – The default value (1) is generally a good place to start.

  • Geo Location Fields for Mapping – – If you are using geolocation fields, list the property keys here.

Alarm

  • Require to create Cloud watch Alarm – Set this to true if you want to create a CloudWatch alarm for the new stack.

  • Email for Alarm Notifications – The email address to which alarm notifications should be sent (only needed if alarms are enabled).

Run the AWS CloudFormation Template

Now you can complete the process of provisioning a Neptune streams consumer application instance as follows:

  1. In AWS CloudFormation, on the Specify Stack Details page, choose Next.

  2. On the Options page, choose Next.

  3. On the Review page, select the first check box to acknowledge that AWS CloudFormation will create IAM resources. Select the second check box to acknowledge CAPABILITY_AUTO_EXPAND for the new stack.

    Note

    CAPABILITY_AUTO_EXPAND explicitly acknowledges that macros will be expanded when creating the stack, without prior review. Users often create a change set from a processed template so that the changes made by macros can be reviewed before actually creating the stack. For more information, see the AWS CloudFormation CreateStack API in the AWS CloudFormation API Reference.

    Then choose Create.