Using an AWS CloudFormation template to start Neptune-to-OpenSearch replication

Launch an AWS CloudFormation stack specific to your region

Each of the AWS CloudFormation templates below creates a streams-consumer application instance in a specific AWS region. To launch the corresponding stack using the AWS CloudFormation console, choose one of the Launch Stack buttons in the following table, depending on the AWS Region that you want to use.

Region	View	View in Designer	Launch
US East (N. Virginia)	View	View in Designer
US East (Ohio)	View	View in Designer
US West (N. California)	View	View in Designer
US West (Oregon)	View	View in Designer
Canada (Central)	View	View in Designer
South America (São Paulo)	View	View in Designer
Europe (Stockholm)	View	View in Designer
Europe (Ireland)	View	View in Designer
Europe (London)	View	View in Designer
Europe (Paris)	View	View in Designer
Europe (Frankfurt)	View	View in Designer
Middle East (Bahrain)	View	View in Designer
Middle East (UAE)	View	View in Designer
Israel (Tel Aviv)	View	View in Designer
Africa (Cape Town)	View	View in Designer
Asia Pacific (Hong Kong)	View	View in Designer
Asia Pacific (Tokyo)	View	View in Designer
Asia Pacific (Seoul)	View	View in Designer
Asia Pacific (Singapore)	View	View in Designer
Asia Pacific (Mumbai)	View	View in Designer
China (Beijing)	View	View in Designer
China (Ningxia)	View	View in Designer
AWS GovCloud (US-West)	View	View in Designer
AWS GovCloud (US-East)	View	View in Designer

On the Create Stack page, choose Next.

Add Details About the new OpenSearch stack you are creating

The Specify Stack Details page provides properties and parameters that you can use to control the setup of full-text search:

Stack Name –  The name of the new AWS CloudFormation stack that you're creating. You can generally use the default value, NeptuneStreamPoller.

Under Parameters, provide the following:

Network Configuration for the VPC Where the Streams Consumer Runs

• VPC –  Provide the name of the VPC where the polling Lambda function will run.

• List of Subnet IDs –  The subnets to which a network interface is established. Add subnets corresponding to your Neptune cluster.

• List of Security Group Ids –  Provide the IDs of security groups that grant write inbound access to your source Neptune DB cluster.

• List of Route Table Ids –  This is needed to create an Amazon DynamoDB endpoint in your Neptune VPC, if you do not already have one. You must provide a comma-separated list of route table IDs associated with the subnets.

• Require to create Dynamo DB VPC Endpoint –  A Boolean value that defaults to true. You only need to change it to false if you have already created a DynamoDB endpoint in your VPC.

• Require to create Monitoring VPC Endpoint –  A Boolean value that defaults to true. You only need to change it to false if you have already created a monitoring endpoint in your VPC.

Stream Poller

• Application Name –  You can generally leave this set to the default (NeptuneStream). If you use a different name, it must be unique.

• Memory size for Lambda Poller –  Used to set the memory size available to the Lambda poller function. The default value is 2,048 megabytes.

• Lambda Runtime –  The language used in the Lambda function that retrieves items from the Neptune stream. You can set this either to python3.9 or to java8.

• S3 Bucket having Lambda code artifacts –  Leave this blank unless you are using a custom Lambda polling function that loads from a different S3 bucket.

• S3 Key corresponding to Lambda Code artifacts –  Leave this blank unless you are using a custom Lambda polling function.

• StartingCheckpoint –  the starting checkpoint for the stream poller. The default is 0:0, which signifies starting from the beginning of the Neptune stream.

• StreamPollerInitialState –  The initial state of the poller. The default is ENABLED, which means that the stream replication will start as soon as the entire stack creation is complete.

• Logging level for Lambda –  In general, leave this set to the default value, INFO.

• Managed Policies for Lambda Execution –  In general, leave this blank unless you are using a custom Lambda polling function.

• Stream Records Handler –  In general, leave this blank unless you are using a custom handler for the records in Neptune streams.

• Maximum records Fetched from Stream –  You can use this parameter to tune performance. The default (100) is a good place to start. The maximum allowable is 10,000. The higher the number, the fewer network calls are needed to read records from the stream, but the more memory is required to process the records.

• Max wait time between two Polls (in Seconds) –  Determines how frequently the Lambda poller is invoked to poll the Neptune streams. Set this value to 0 for continuous polling. The maximum value is 3,600 seconds (1 hour). The default value (60 seconds) is a good place to start, depending on how fast your graph data changes.

• Maximum Continuous polling period (in Seconds) –  Used to set a timeout for the Lambda polling function. It should be between 5 seconds and 900 seconds. The default value (600 seconds) is a good place to start.

• Step Function Fallback Period –  The number of step-function-fallback-period units to wait for the poller, after which the step function is called through Amazon CloudWatch Events to recover from a failure. The default (5 minutes) is a good place to start.

• Step Function Fallback Period Unit –  The time units used to measure the preceding Step Function Fallback Period (minutes, hours, days). The default (minutes) is generally sufficient.

• Data replication scope –  Determines whether to replicate both nodes and edges, or only nodes to OpenSearch (this applies to Gremlin engine data only). The default value (All) is generally a good place to start.

• Ignore OpenSearch missing document error –  Flag to determine whether a missing document error in OpenSearch can be ignored. Missing document errors occur rarely but need manual intervention if not ignored. The default value (True) is generally a good place to start.

• Enable Non-String Indexing –  Flag to enable or disable indexing of fields that do not have string content. If this flag is set to true, non-string fields are indexed in OpenSearch, or if false, only string fields are indexed. The default is true.

• Properties to exclude from being inserted into OpenSearch –  A comma-delimited list of property or predicate keys to exclude from OpenSearch indexing. If this CFN parameter value is left blank, all the property keys are indexed.

• Datatypes to exclude from being inserted into OpenSearch –  A comma-delimited list of property or predicate datatypes to exclude from OpenSearch indexing. If this CFN parameter value is left blank, all the property values that can safely be converted to OpenSearch datatypes are indexed.

Neptune Stream

• Endpoint of source Neptune Stream –  (Required) This takes one of two forms:

  • https://your DB cluster:port/propertygraph/stream (or its alias, https://your DB cluster:port/pg/stream).

  • https://your DB cluster:port/sparql/stream

• Neptune Query Engine –  Choose Gremlin or SPARQL.

• Is IAM Auth Enabled? –  If your Neptune DB cluster is using IAM authentication, set this parameter to true.

• Neptune Cluster Resource Id –  If your Neptune DB cluster is using IAM authentication, set this parameter to the cluster resource ID. The resource ID is not the same as the cluster ID. Instead, it takes the form: cluster- followed by 28 alpha-numeric characters. It can be found under Cluster Details in the Neptune console.

Target OpenSearch cluster

• Endpoint for OpenSearch service –  (Required) Provide the endpoint for the OpenSearch service in your VPC.

• Number of Shards for OpenSearch Index –  The default value (5) is generally a good place to start.

• Number of Replicas for OpenSearch Index –  The default value (1) is generally a good place to start.

• Geo Location Fields for Mapping –  If you are using geolocation fields, list the property keys here.

Alarm

• Require to create Cloud watch Alarm –  Set this to true if you want to create a CloudWatch alarm for the new stack.

• SNS Topic ARN for Cloudwatch Alarm Notifications   –   The SNS topic ARN where CloudWatch alarm notifications should be sent (only needed if alarms are enabled).

• Email for Alarm Notifications –  The email address to which alarm notifications should be sent (only needed if alarms are enabled).

For destination of the alarm notification, you can add SNS only, email only, or both SNS and email.

Run the AWS CloudFormation Template

Now you can complete the process of provisioning a Neptune streams consumer application instance as follows:

1. In AWS CloudFormation, on the Specify Stack Details page, choose Next.

2. On the Options page, choose Next.

3. On the Review page, select the first check box to acknowledge that AWS CloudFormation will create IAM resources. Select the second check box to acknowledge CAPABILITY_AUTO_EXPAND for the new stack.

   Note

   CAPABILITY_AUTO_EXPAND explicitly acknowledges that macros will be expanded when creating the stack, without prior review. Users often create a change set from a processed template so that the changes made by macros can be reviewed before actually creating the stack. For more information, see the AWS CloudFormation CreateStack API operation in the AWS CloudFormation API Reference.

   Then choose Create.