Access an Amazon Neptune database from an Amazon EKS container - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Access an Amazon Neptune database from an Amazon EKS container

Created by Ramakrishnan Palaninathan (AWS)

Environment: Production

Technologies: Containers & microservices; Databases

Workload: All other workloads

AWS services: Amazon EKS; Amazon Neptune

Summary

This pattern establishes a connection between Amazon Neptune, which is a fully managed graph database, and Amazon Elastic Kubernetes Service (Amazon EKS), a container orchestration service, to access a Neptune database. Neptune DB clusters are confined within a virtual private cloud (VPC) on AWS. For this reason, accessing Neptune requires careful configuration of the VPC to enable connectivity.

Unlike Amazon Relational Database Service (Amazon RDS) for PostgreSQL, Neptune doesn't rely on typical database access credentials. Instead, it uses AWS Identity and Access Management (IAM) roles for authentication. Therefore, connecting to Neptune from Amazon EKS involves setting up an IAM role with the necessary permissions to access Neptune.

Furthermore, Neptune endpoints are accessible only within the VPC where the cluster resides. This means that you have to configure network settings to facilitate communication between Amazon EKS and Neptune. Depending on your specific requirements and networking preferences, there are various approaches to configuring the VPC to enable seamless connectivity between Neptune and Amazon EKS. Each method offers distinct advantages and considerations, which provide flexibility in designing your database architecture to suit your application's needs.

Prerequisites and limitations

Prerequisites

  • Install the latest version of kubectl (see instructions). To check your version, run: 

    kubectl version --short

  • Install the latest version of eksctl (see instructions). To check your version, run: 

    eksctl info

  • Install the latest version of the AWS Command Line Interface (AWS CLI) version 2 (see instructions). To check your version, run: 

    aws --version

  • Create a Neptune DB cluster (see instructions). Make sure to establish communications between the cluster's VPC and Amazon EKS through VPC peering, AWS Transit Gateway, or another method. Also make sure that the status of the cluster is “available” and that it has an inbound rule on port 8182 for the security group.

  • Configure an IAM OpenID Connect (OIDC) provider on an existing Amazon EKS cluster (see instructions).

Product versions

Architecture

The following diagram shows the connection between Kubernetes pods in an Amazon EKS cluster and Neptune to provide access to a Neptune database.

Connecting pods in a Kubernetes node with Amazon Neptune.

Automation and scale

You can use the Amazon EKS Horizontal Pod Autoscaler to scale this solution.

Tools

Services

Best practices

For best practices, see Identity and Access Management in the Amazon EKS Best Practices Guides.

Epics

TaskDescriptionSkills required

Verify the cluster context.

Before you interact with your Amazon EKS cluster by using Helm or other command-line tools, you must define environment variables that encapsulate your cluster's details. These variables are used in subsequent commands to ensure that they target the correct cluster and resources.

First, confirm that you are operating within the correct cluster context. This ensures that any subsequent commands are sent to the intended Kubernetes cluster. To verify the current context, run the following command.

kubectl config current-context
AWS administrator, Cloud administrator

Define the CLUSTER_NAME variable.

Define the CLUSTER_NAME environment variable for your Amazon EKS cluster. In the following command, replace the sample value us-west-2 with the correct AWS Region for your cluster. Replace the sample value eks-workshop with your existing cluster name.

export CLUSTER_NAME=$(aws eks describe-cluster --region us-west-2 --name eks-workshop --query "cluster.name" --output text)
AWS administrator, Cloud administrator

Validate output.

To validate that the variables have been set properly, run the following command.

echo $CLUSTER_NAME

Verify that the output of this command matches the input you specified in the previous step.

AWS administrator, Cloud administrator
TaskDescriptionSkills required

Create a service account.

You use IAM roles for service accounts to map your Kubernetes service accounts to IAM roles, to enable fine-grained permissions management for your applications that run on Amazon EKS. You can use eksctl to create and associate an IAM role with a specific Kubernetes service account within your Amazon EKS cluster. The AWS managed policy NeptuneFullAccess allows write and read access to your specified Neptune cluster.

Important: You must have an OIDC endpoint associated with your cluster before you run these commands.

Create a service account that you want to associate with an AWS managed policy named NeptuneFullAccess.

eksctl create iamserviceaccount --name eks-neptune-sa --namespace default --cluster $CLUSTER_NAME --attach-policy-arn arn:aws:iam::aws:policy/NeptuneFullAccess --approve --override-existing-serviceaccounts

where eks-neptune-sa is the name of the service account that you want to create.

Upon completion, this command displays the following response:

2024-02-07 01:12:39 [ℹ] created serviceaccount "default/eks-neptune-sa"
AWS administrator, Cloud administrator

Verify that the account is set up properly.

Make sure that the eks-neptune-sa service account is set up correctly in the default namespace in your cluster.

kubectl get sa eks-neptune-sa -o yaml

The output should look like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789123:role/eksctl-eks-workshop-addon-iamserviceaccount-d-Role1-Q35yKgdQOlmM
  creationTimestamp: "2024-02-07T01:12:39Z"
  labels:
    app.kubernetes.io/managed-by: eksctl
  name: eks-neptune-sa
  namespace: default
  resourceVersion: "5174750"
  uid: cd6ba2f7-a0f5-40e1-a6f4-4081e0042316
AWS administrator, Cloud administrator

Check connectivity.

Deploy a sample pod called pod-util and check connectivity with Neptune.

apiVersion: v1
kind: Pod
metadata:
  name: pod-util
  namespace: default
spec:
  serviceAccountName: eks-neptune-sa
  containers:
  - name: pod-util
    image: public.ecr.aws/patrickc/troubleshoot-util
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
kubectl apply -f pod-util.yaml
kubectl exec --stdin --tty pod-util -- /bin/bash
bash-5.1# curl -X POST -d '{"gremlin":"g.V().limit(1)"}' https://db-neptune-1.cluster-xxxxxxxxxxxx.us-west-2.neptune.amazonaws.com:8182/gremlin
{"requestId":"a4964f2d-12b1-4ed3-8a14-eff511431a0e","status":{"message":"","code":200,"attributes":{"@type":"g:Map","@value":[]}},"result":{"data":{"@type":"g:List","@value":[]},"meta":{"@type":"g:Map","@value":[]}}}
bash-5.1# exit
exit
AWS administrator, Cloud administrator
TaskDescriptionSkills required

Enable IAM database authentication.

By default, IAM database authentication is disabled when you create a Neptune DB cluster. You can enable or disable IAM database authentication by using the AWS Management Console.

Follow the steps in the AWS documentation to enable IAM database authentication in Neptune.

AWS administrator, Cloud administrator

Verify connections.

In this step, you interact with the pod-util container, which is already in running status, to install awscurl and verify the connection.

  1. Run the following command to find the pod.

    kubectl get pods

    The output should look like this:

    NAME READY STATUS RESTARTS AGE
pod-util 1/1 Running 0 50m

  2. Run the following command to install awscurl.

    kubectl exec --stdin --tty pod-util -- /bin/bash bash-5.1#pip3 install awscurl Installing collected packages: idna, configparser, configargparse, charset-normalizer, certifi, requests, awscurl Successfully installed awscurl-0.32 certifi-2024.2.2 charset-normalizer-3.3.2 configargparse-1.7 configparser-6.0.0 idna-3.6 requests-2.31.0 bash-5.1# awscurl https://db-neptune-1.cluster-xxxxxxxxxxxx.us-west-2.neptune.amazonaws.com:8182/status --region us-west-2 --service neptune-db {"status":"healthy","startTime":"Thu Feb 08 01:22:14 UTC 2024","dbEngineVersion":"1.3.0.0.R1","role":"writer","dfeQueryEngine":"viaQueryHint","gremlin":{"version":"tinkerpop-3.6.4"},"sparql":{"version":"sparql-1.1"},"opencypher":{"version":"Neptune-9.0.20190305-1.0"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"},"features":{"SlowQueryLogs":"disabled","ResultCache":{"status":"disabled"},"IAMAuthentication":"enabled","Streams":"disabled","AuditLog":"disabled"},"settings":{"clusterQueryTimeoutInMs":"120000","SlowQueryLogsThreshold":"5000"}}
AWS administrator, Cloud administrator

Troubleshooting

IssueSolution

Can't access the Neptune database.

Review the IAM policy that's attached to the service account. Make sure that it allows the necessary actions (for example, neptune:Connec,neptune:DescribeDBInstances) for the operations you want to run.

Related resources