Automatically generate a PynamoDB model and CRUD functions for Amazon DynamoDB by using a Python application

Created by Vijit Vashishtha (AWS), Dheeraj Alimchandani (AWS), and Dhananjay Karanjkar (AWS)

Code repository: amazon-reverse-engineer-dynamodb	Environment: PoC or pilot	Technologies: DevelopmentAndTesting; Databases; DevOps
Workload: Open-source	AWS services: Amazon DynamoDB

Summary

It's common to require entities and create, read, update, and delete (CRUD) operations functions to efficiently perform Amazon DynamoDB database operations. PynamoDB is a Python-based interface that supports Python 3. It also provides features such as support for Amazon DynamoDB transactions, automatic attribute value serialization and deserialization, and compatibility with common Python frameworks, such as Flask and Django. This pattern helps developers working with Python and DynamoDB by providing a library that streamlines the automatic creation of PynamoDB models and CRUD operation functions. While it generates essential CRUD functions for database tables, it can also reverse engineer PynamoDB models and CRUD functions from Amazon DynamoDB tables. This pattern is designed to simplify database operations by using a Python-based application.

The following are the key features of this solution:

• JSON schema to PynamoDB model – Automatically generate PynamoDB models in Python by importing a JSON schema file.

• CRUD function generation – Automatically generate functions to perform CRUD operations on DynamoDB tables.

• Reverse engineering from DynamoDB – Use PynamoDB object-relational mapping (ORM) to reverse engineer PynamoDB models and CRUD functions for existing Amazon DynamoDB tables.

Prerequisites and limitations

Prerequisites

• An active AWS account

• Python version 3.8 or later, downloaded and installed

• Jinja2 version 3.1.2 or later, downloaded and installed

• Amazon DynamoDB tables for which you want to generate ORM

• AWS Command Line Interface (AWS CLI), installed and configured

• PynamoDB version 5.4.1 or later, installed

Architecture

Target technology stack

• JSON script

• Python application

• PynamoDB model

• Amazon DynamoDB database instance

Target architecture

1. You create an input JSON schema file. This JSON schema file represents the attributes of the respective DynamoDB tables that you want to create PynamoDB models from and CRUD functions for. It contains the following three important keys:

   • name –The name of the target DynamoDB table.

   • region – The AWS region where the table is hosted

   • attributes – The attributes that are part of the target table, such as the partition key (also known as a hash attribute), sort key, local secondary indexes, global secondary indexes, and any non-key attributes. This tool expects the input schema to only provide the non-key attributes as the application fetches the key attributes directly from the target table. For an example of how to specify attributes in the JSON schema file, see the Additional information section of this pattern.

2. Run the Python application and provide the JSON schema file as an input.

3. The Python application reads the JSON schema file.

4. The Python application connects to the DynamoDB tables to derive the schema and data types. The application runs the describe_table operation and fetches the key and index attributes for the table.

5. The Python application combines the attributes from the JSON schema file and DynamoDB table. It uses the Jinja template engine to generate a PynamoDB model and corresponding CRUD functions.

6. You access the PynamoDB model to perform CRUD operations on the DynamoDB table.

Tools

AWS services

• Amazon DynamoDB is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.

Other tools

• Jinja is an extensible templating engine that compiles templates into optimized Python code. This pattern uses Jinja to to generate dynamic content by embedding placeholders and logic within templates.

• PynamoDB is a Python-based interface for Amazon DynamoDB.

• Python is a general-purpose computer programming language.

Code repository

The code for this pattern is available in the GitHub Auto-generate PynamoDB models and CRUD functions repository. The repository is divided into two main parts: the controller package and the templates.

Controller package

The controller Python package contains the main application logic that helps generate the PynamoDB model and the CRUD functions. It contains the following:

• input_json_validator.py – This Python scripts validates the input JSON schema file and creates the Python objects that contain the list of target DynamoDB tables and the required attributes for each.

• dynamo_connection.py – This script establishes a connection to the DynamoDB table and uses the describe_table operation to extract the attributes that are necessary to create the PynamoDB model.

• generate_model.py – This script contains a Python class GenerateModel that creates the PynamoDB model based on the input JSON schema file and the describe_table operation.

• generate_crud.py – For the DynamoDB tables that are defined in the JSON schema file, this script uses the GenerateCrud operation to create the Python classes.

Templates

This Python directory contains the following Jinja templates:

• model.jinja – This Jinja template contains the template expression for generating the PynamoDB model script.

• crud.jinja – This Jinja template contains the template expression for generating the CRUD functions script.

Epics

Set up the environment

Task	Description	Skills required
Clone the repository.	Enter the following command to clone the Auto-generate PynamoDB models and CRUD functions repository. git clone https://github.com/aws-samples/amazon-reverse-engineer-dynamodb.git	App developer
Set up the Python environment.	Navigate into the top-level directory in the cloned repository. cd amazon-reverse-engineer-dynamodb Enter the following command to install the required libraries and packages. pip install -r requirements.txt	App developer

Generate the PynamoDB model and CRUD functions

Task	Description	Skills required
Modify the JSON schema file.	Navigate into the top-level directory in the cloned repository. cd amazon-reverse-engineer-dynamodb Open the test.json file in your preferred editor. You can use this file as a reference to create your own JSON schema file, or you can update the values in this file to match your environment. Modify the name, AWS Region, and attributes values for your target DynamoDB tables. Note: If you define a table that does not exist in the JSON schema file, this solution does not generate models or CRUD functions for that table. Save and close the test.json file. We recommend that you save this file with a new name.	App developer
Run the Python application.	Enter the following command to generate the PynamoDB models and CRUD functions, where <input_schema.json> is the name of your JSON schema file. python main.py --file <input_schema.json>	App developer

Verify the PynamoDB model and CRUD functions

Task	Description	Skills required
Verify the generated PynamoDB model.	In the top-level directory of the cloned repository, enter the following command to navigate into the models repository. cd models By default, this solution names the PynamoDB model file demo_model.py. Validate that this file is present.	App developer
Verify the generated CRUD functions.	In the top-level directory of the cloned repository, enter the following command to navigate into the crud repository. cd crud By default, this solution names the script demo_crud.py. Validate that this file is present. Use the Python classes in the demo_crud.py file to perform a CRUD operation on the target DynamoDB table. Confirm that the operation completed successfully.	App developer

Related resources

• Core components of Amazon DynamoDB (DynamoDB documentation)

• Improving data access with secondary indexes (DynamoDB documentation)

Additional information

Sample attributes for the JSON schema file

[
{
"name": "test_table",
"region": "ap-south-1",
"attributes": [
{
"name": "id",
"type": "UnicodeAttribute"
},
{
"name": "name",
"type": "UnicodeAttribute"
},
{
"name": "age",
"type": "NumberAttribute"
}
]
}
]