Tenant onboarding in SaaS architecture for the silo model using C# and AWS CDK

Created by Tabby Ward (AWS), Susmitha Reddy Gankidi (AWS), and Vijai Anand Ramalingam (AWS)

Code repository: Tennat Onboarding Silo	Environment: PoC or pilot	Technologies: Modernization; DevOps
Workload: Open-source	AWS services: AWS CloudFormation; Amazon DynamoDB; Amazon DynamoDB Streams; AWS Lambda; Amazon API Gateway

Summary

Software as a service (SaaS) applications can be built with a variety of different architectural models. The silo model refers to an architecture where tenants are provided dedicated resources.

SaaS applications rely on a frictionless model for introducing new tenants into their environment. This often requires the orchestration of a number of components to successfully provision and configure all the elements needed to create a new tenant. This process, in SaaS architecture, is referred to as tenant on-boarding. On-boarding should be fully automated for every SaaS environment by utilizing infrastructure as code in your on-boarding process.

This pattern guides you through an example of creating a tenant and provisioning a basic infrastructure for the tenant on Amazon Web Services (AWS). The pattern uses C# and the AWS Cloud Development Kit (AWS CDK).

Because this pattern creates a billing alarm, we recommend deploying the stack in the US East (N. Virginia), or us-east-1, AWS Region. For more information, see the AWS documentation.

Prerequisites and limitations

Prerequisites 

• An active AWS account.

• An AWS Identity and Access Management (IAM) principal with sufficient IAM access to create AWS resources for this pattern. For more information, see IAM roles.

• Install Amazon Command Line Interface (AWS CLI) and configure AWS CLI to perform AWS CDK deployment.

• Visual Studio 2022 downloaded and installed or Visual Studio Code downloaded and installed.

• AWS Toolkit for Visual Studio set up.

• .NET Core 3.1 or later (required for C# AWS CDK applications)

• Amazon.Lambda.Tools installed.

Limitations 

• AWS CDK uses AWS CloudFormation, so AWS CDK applications are subject to CloudFormation service quotas. For more information, see AWS CloudFormation quotas. 

• The tenant CloudFormation stack is created with a CloudFormation service role infra-cloudformation-role with wildcard characters on actions (sns* and sqs*) but with resources locked down to the tenant-cluster prefix. For a production use case, evaluate this setting and provide only required access to this service role. The InfrastructureProvision Lambda function also uses a wildcard character (cloudformation*) to provision the CloudFormation stack but with resources locked down to the tenant-cluster prefix.

• This example code's docker build uses --platform=linux/amd64 to force linux/amd64 based images. This is to ensure that the final image artifacts will be suitable for Lambda, which by default uses x86-64 architecture. If you need to change the target Lambda architecture, be sure to change both the Dockerfiles and the AWS CDK codes. For more information, see the blog post Migrating AWS Lambda functions to Arm-based AWS Graviton2 processors.

• The stack deletion process will not clean up CloudWatch Logs (log groups and logs) generated by the stack. You must manually clean up the logs through the AWS Management Console Amazon CloudWatch console or the through the API.

This pattern is set up as an example. For production use, evaluate the following setups and make changes based on your business requirements:

• The AWS Simple Storage Service (Amazon S3) bucket in this example does not have versioning enabled for simplicity. Evaluate and update the setup as needed.

• This example sets up Amazon API Gateway REST API endpoints without authentication, authorization, or throttling for simplicity. For production use, we recommend integrating the system with the business security infrastructure. Evaluate this setting and add required security settings as needed.

• For this tenant infrastructure example, Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) have only minimum setups. The AWS Key Management Service (AWS KMS) for each tenant opens to Amazon CloudWatch and Amazon SNS services in the account to consume based on the AWS KMS key policy. The setup is only an example placeholder. Adjust the setups as needed based on your business use case.

• The entire setup, which includes but isn’t limited to API endpoints and backend tenant provisioning and deletion by using AWS CloudFormation, covers only the basic happy path case. Evaluate and update the setup with the necessary retry logic, additional error handling logic, and security logic based on your business needs.

• The example code is tested with up-to-date cdk-nag to check for policies at the time of this writing. New policies might be enforced in the future. These new policies might require you to manually modify the stack based on the recommendations before the stack can be deployed. Review the existing code to ensure that it aligns with your business requirements.

• The code relies on the AWS CDK to generate a random suffix instead of relying on static assigned physical names for most created resources. This setup is to ensure that these resources are unique and do not conflict with other stacks. For more information, see the AWS CDK documentation. Adjust this based on your business requirements.

• This example code packages .NET Lambda artifacts into Docker based images and runs with the Lambda provided Container image runtime. The container image runtime has advantages for standard transfer and store mechanisms (container registries) and more accurate local test environments (through the container image). You can switch the project to use Lambda provided .NET runtimes to reduce the build time of the Docker images, but you will then need to set up transfer and store mechanisms and ensure that the local setup matches the Lambda setup. Adjust the code to align with users' business requirements.

Product versions

• AWS CDK version 2.45.0 or later

• Visual Studio 2022

Architecture

Technology stack

• Amazon API Gateway

• AWS CloudFormation

• Amazon CloudWatch

• Amazon DynamoDB

• AWS Identity and Access Management (IAM)

• AWS KMS

• AWS Lambda

• Amazon S3

• Amazon SNS

• Amazon SQS

Architecture

The following diagram shows the tenant stack creation flow. For more information about the control-plane and tenant technology stacks, see the Additional information section.

Tenant stack creation flow

1. User sends a POST API request with new tenant payload (tenant name, tenant description) in JSON to a REST API hosted by Amazon API Gateway. The API Gateway processes the request and forwards it to the backend Lambda Tenant On-boarding function. In this example, there is no authorization or authentication. In a production setup, this API should be integrated with the SaaS infrastructure security system.

2. The Tenant On-boarding function verifies the request. Then it attempts to store the tenant record, which includes the tenant name, generated tenant universally unique identifier (UUID), and tenant description, into the Amazon DynamoDB Tenant On-boarding table. 

3. After DynamoDB stores the record, a DynamoDB stream initiates the downstream Lambda Tenant Infrastructure function.

4. The Tenant Infrastructure Lambda function acts based the on received DynamoDB stream. If the stream is for the INSERT event, the function uses the stream's NewImage section (latest update record, Tenant Name field) to invoke CloudFormation to create a new tenant infrastructure using the template that is stored in the S3 bucket. The CloudFormation template requires the Tenant Name parameter. 

5. AWS CloudFormation creates the tenant infrastructure based on the CloudFormation template and input parameters.

6. Each tenant infrastructure setup has a CloudWatch alarm, a billing alarm, and an alarm event.

7. The alarm event becomes a message to an SNS topic, which is encrypted by the tenant's AWS KMS key.

8. The SNS topic forwards the received alarm message to the SQS queue, which is encrypted by the tenant's AWS KMS for encryption key.

Other systems can be integrated with Amazon SQS to perform actions based on messages in queue. In this example, to keep the code generic, incoming messages remain in queue and require manual deletion.

Tenant stack deletion flow

1. User sends a DELETE API request with new tenant payload (tenant name, tenant description) in JSON to the REST API hosted by Amazon API Gateway, which will process the request and forward to Tenant On-boarding function. In this example, there is no authorization or authentication. In a production setup, this API will be integrated with the SaaS infrastructure security system.

2. The Tenant On-boarding function will verify the request and then attempt to delete the tenant record (tenant name) from the Tenant On-boarding table. 

3. After DynamoDB deletes the record successfully (the record exists in the table and is deleted), a DynamoDB stream initiates the downstream Lambda Tenant Infrastructure function.

4. The Tenant Infrastructure Lambda function acts based on the received DynamoDB stream record. If the stream is for the REMOVE event, the function uses the record's OldImage section (record information and Tenant Name field, before the latest change, which is delete) to initiate deletion of an existing stack based on that record information.

5. AWS CloudFormation deletes the target tenant stack according to the input.

Tools

AWS services

• Amazon API Gateway helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.

• AWS Cloud Development Kit (AWS CDK) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.

• AWS CDK Toolkit is a command line cloud development kit that helps you interact with your AWS Cloud Development Kit (AWS CDK) app.

• AWS Command Line Interface (AWS CLI) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.

• AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.

• Amazon DynamoDB is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.

• AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

• AWS Key Management Service (AWS KMS) helps you create and control cryptographic keys to help protect your data.

• AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

• Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

• Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.

• Amazon Simple Queue Service (Amazon SQS) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.

• AWS Toolkit for Visual Studio is a plugin for the Visual Studio integrated development environment (IDE). The Toolkit for Visual Studio supports developing, debugging, and deploying .NET applications that use AWS services.

Other tools

• Visual Studio is an IDE that includes compilers, code completion tools, graphical designers, and other features that support software development.

Code

The code for this pattern is in the Tenant onboarding in SaaS Architecture for Silo Model APG Example repository.

Epics

Set up AWS CDK

Task	Description	Skills required
Verify Node.js installation.	To verify that Node.js is installed on your local machine, run the following command. node --version	AWS administrator, AWS DevOps
Install AWS CDK Toolkit.	To install AWS CDK Toolkit on your local machine, run the following command. npm install -g aws-cdk If npm is not installed, you can install it from the Node.js site.	AWS administrator, AWS DevOps
Verify the AWS CDK Toolkit version.	To verify that the AWS CDK Toolkit version is installed correctly on your machine, run the following command.   cdk --version	AWS administrator, AWS DevOps

Review the code for the tenant onboarding control plane

Task	Description	Skills required
Clone the repository.	Clone the repository, and navigate to the \tenant-onboarding-in-saas-architecture-for-silo-model-apg-example folder. In Visual Studio 2022, open the \src\TenantOnboardingInfra.sln solution. Open the TenantOnboardingInfraStack.cs file and review the code. The following resources are created as part of this stack: DynamoDB table S3 bucket (Upload the CloudFormation template to the S3 bucket.) Lambda execution role Lambda function API Gateway API Event source to Lambda function	AWS administrator, AWS DevOps
Review the CloudFormation template.	In the \tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\template folder, open infra.yaml, and review the CloudFormation template. This template will be hydrated with the tenant name retrieved from the tenant onboarding DynamoDB table. The template provisions the tenant-specific infrastructure. In this example, it provisions the AWS KMS key, Amazon SNS , Amazon SQS, and the CloudWatch alarm.	App developer, AWS DevOps
Review the tenant onboarding function.	Open Function.cs, and review the code for the tenant onboarding function, which is created with the Visual Studio AWS Lambda Project (.NET Core- C#) template with the .NET 6 (Container Image) blueprint. Open the Dockerfile, and review the code. The Dockerfile is a text file that consists of instructions for building the Lambda container image. Note that the following NuGet packages are added as dependencies to the TenantOnboardingFunction project: Amazon.Lambda.APIGatewayEvents AWSSDK.DynamoDBv2 Newtonsoft.Json	App developer, AWS DevOps
Review the Tenant InfraProvisioning function.	Navigate to \tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\src\InfraProvisioningFunction. Open Function.cs, and review the code for the tenant infrastructure provisioning function, which is created with the Visual Studio AWS Lambda Project (.NET Core- C#) template with the .NET 6 (Container Image) blueprint. Open the Dockerfile, and review the code. Note that the following NuGet packages are added as dependencies to the InfraProvisioningFunction project: Amazon.Lambda.DynamoDBEvents AWSSDK.DynamoDBv2 AWSSDK.Cloudformation	App developer, AWS DevOps

Deploy the AWS resources

Task	Description	Skills required
Build the solution.	To build the solution, perform the following steps: In Visual Studio 2022, open the \tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\src\TenantOnboardingInfra.sln solution.  Open the context (right-click) menu for the solution, and choose Build solution. Note: Make sure that you update the Amazon.CDK.Lib NuGet package to the latest version in \tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\src\TenantOnboardingInfra project before you build the solution.	App developer
Bootstrap the AWS CDK environment.	Open the Windows command prompt and navigate to the AWS CDK app root folder where the cdk.json file is available (\tenant-onboarding-in-saas-architecture-for-silo-model-apg-example). Run the following command for bootstrapping. cdk bootstrap If you have created an AWS profile for the credentials, use the command with your profile. cdk bootstrap --profile <profile name>	AWS administrator, AWS DevOps
List the AWS CDK stacks.	To list all the stacks to be created as part of this project, run the following command. cdk ls cdk ls --profile <profile name> If you have created an AWS profile for the credentials, use the command with your profile. cdk ls --profile <profile name>	AWS administrator, AWS DevOps
Review which AWS resources will be created.	To review all the AWS resources that will be created as part of this project, run the following command. cdk diff If you have created an AWS profile for the credentials, use the command with your profile. cdk diff --profile <profile name>	AWS administrator, AWS DevOps
Deploy all the AWS resources by using AWS CDK.	To deploy all the AWS resources run the following command. cdk deploy --all --require-approval never If you have created an AWS profile for the credentials, use the command with your profile. cdk deploy --all --require-approval never --profile <profile name> After the deployment is complete, copy the API URL from the outputs section in the command prompt, which is shown in the following example. Outputs: TenantOnboardingInfraStack.TenantOnboardingAPIEndpoint42E526D7 = https://j2qmp8ds21i1i.execute-api.us-west-2.amazonaws.com/prod/	AWS administrator, AWS DevOps

Verify the functionality

Task	Description	Skills required
Create a new tenant.	To create the new tenant, send the following curl request. curl -X POST <TenantOnboardingAPIEndpoint* from CDK Output>tenant -d '{"Name":"Tenant123", "Description":"Stack for Tenant123"}' Change the place holder <TenantOnboardingAPIEndpoint* from CDK Output> to the actual value from AWS CDK, as shown in the following example. curl -X POST https://j2qmp8ds21i1i.execute-api.us-west-2.amazonaws.com/prod/tenant -d '{"Name":"Tenant123", "Description":"test12"}' The following example shows the output. {"message": "A new tenant added - 5/4/2022 7:11:30 AM"}	App developer, AWS administrator, AWS DevOps
Verify the newly created tenant details in DynamoDB.	To verify the newly created tenant details in DynamoDB, perform the following steps. Open AWS Management Console, and navigate to the Amazon DynamoDB service. In the left navigation, choose Explore items, and choose the TenantOnboarding table. Note: The tenant name will be prepended with tenantcluster-. For more information, see the Additional information section. Verify that a new item is created with the tenant details.	App developer, AWS administrator, AWS DevOps
Verify the stack creation for the new tenant.	Verify that the new stack was successfully created and provisioned with infrastructure for the newly created tenant according to the CloudFormation template. Open the CloudFormation console. In the left navigation, choose Stacks, and verify that a stack with the tenant name was successfully created. Choose the newly created tenant stack, and then choose the Resources tab. Note the alarm resource and the Amazon SQS resource. Open a new terminal with AWS credentials configured, and point to the correct Region. To raise a test alarm, enter the following code, replacing <alarm resource name> with the alarm resource name noted in step 3. aws cloudwatch set-alarm-state --alarm-name <alarm resource name> --state-value ALARM --state-reason 'Test setup' The following example shows the code with an alarm resource name. aws cloudwatch set-alarm-state --alarm-name tenantcluster-tenant123-alarm --state-value ALARM --state-reason 'Test setup' Open the console and navigate to the Amazon SQS console. Choose the Amazon SQS resource name identified in step 3. Follow the AWS documentation instructions to receive and delete the test message from the alarm that was raised in step 4.	App developer, AWS administrator, AWS DevOps
Delete the tenant stack.	To delete the tenant stack, send the following curl request. curl -X DELETE <TenantOnboardingAPIEndpoint* from CDK Output>tenant/<Tenant Name from previous step> Change the place holder <TenantOnboardingAPIEndpoint* from CDK Output> to the actual value from AWS CDK, and change <Tenant Name from previous step> to the actual value from the previous tenant creation step, as shown in the following example. curl -X DELETE https://j2qmp8ds21i1i.execute-api.us-west-2.amazonaws.com/prod/tenant/Tenant123 The following example shows the output. {"message": "Tenant destroyed - 5/4/2022 7:14:48 AM"}	App developer, AWS DevOps, AWS administrator
Verify the stack deletion for the existing tenant.	To verify that the existing tenant stack got deleted, perform the following steps: Open console and navigate to the CloudFormation console. In the left navigation, verify that the existing stack with the tenant name is no longer in console (if the CloudFormation console is set up to show only Active stacks) or is in the process of being deleted. If the stack is no longer in the CloudFormation console, use the dropdown list to change the console's setting from Active to Deleted to see the deleted stack and verify that the stack was deleted successfully.	App developer, AWS administrator, AWS DevOps

Clean up

Task	Description	Skills required
Destroy the environment.	Before the stack clean up, ensure the following: All records in DynamoDB are removed either through the previous tenant deletion operation or through the DynamoDB console or API. Each tenant record deletion will initiate the cleanup of its AWS CloudFormation counterpart.  All tenant-based AWS CloudFormation stacks are cleaned up (in case the DynamoDB trigger cleanup logic fails) on the AWS CloudFormation console. After testing is done, AWS CDK can be used to destroy all the stacks and related resources by running the following command. cdk destroy --all; If you created an AWS profile for the credentials, use the profile. Confirm the stack deletion prompt to delete the stack.	AWS administrator, AWS DevOps
Clean up Amazon CloudWatch Logs.	The stack deletion process will not clean up CloudWatch Logs (log groups and logs) that were generated by the stack. Manually clean up the CloudWatch resources by using the CloudWatch console or the API.	App developer, AWS DevOps, AWS administrator

Related resources

• AWS CDK .NET Workshop

• Working with the AWS CDK in C#

• CDK .NET Reference

Additional information

Control-plane technology stack

The CDK code written in .NET is used to provision the control-plane infrastructure, which consists of the following resources:

1. API Gateway

   Serves as the REST API entry point for the control-plane stack.

2. Tenant on-boarding Lambda function

   This Lambda function is initiated by API Gateway using the m method.

   A POST method API request results in (tenant name, tenant description) being inserted into the DynamoDB Tenant Onboarding table.

   In this code example, the tenant name is also used as part of the tenant stack name and the names of resources within that stack. This is to make these resources easier to identify. This tenant name must be unique across the setup to avoid conflicts or errors. Detailed input validation setup is explained in the IAM roles documentation and the Limitations section.

   The persistence process to the DynamoDB table will succeed only if the tenant name is not used in any other record in the table.

   The tenant name in this case is the partition key for this table, because only the partition key can be used as a PutItem condition expression.

   If the tenant name was never recorded before, the record will be saved into the table successfully.

   However, if the tenant name is already used by an existing record in the table, the operation will fail and initiate a DynamoDB ConditionalCheckFailedException exception. The exception will be used to return a failure message (HTTP BadRequest) indicating that the tenant name already exists.

   A DELETE method API request will remove the record for a specific tenant name from the Tenant Onboarding table.

   The DynamoDB record deletion in this example will succeed even if the record does not exist.

   If the target record exists and is deleted, it will create a DynamoDB stream record. Otherwise, no downstream record will be created.

3. Tenant on-boarding DynamoDB, with Amazon DynamoDB Streams enabled

   This records the tenant metadata information, and any record save or deletion will send a stream downstream to the Tenant Infrastructure Lambda function. 

4. Tenant infrastructure Lambda function

   This Lambda function is initiated by the DynamoDB stream record from the previous step. If the record is for an INSERT event, it invokes AWS CloudFormation to create a new tenant infrastructure with the CloudFormation template that is stored in an S3 bucket. If the record is for REMOVE, it initiates deletion of an existing stack based on the stream record's Tenant Name field.

5. S3 bucket

   This is for storing the CloudFormation template.

6. IAM roles for each Lambda function and a service role for CloudFormation

   Each Lambda function has its unique IAM role with least-privilege permissions to achieve its task. For example, the Tenant On-boarding Lambda function has read/write access to DynamoDB, and the Tenant Infrastructure Lambda function can only read the DynamoDB stream.

   A custom CloudFormation service role is created for tenant stack provisioning. This service role contains additional permissions for CloudFormation stack provisioning (for example, the AWS KMS key). This divides roles between Lambda and CloudFormation to avoid all permissions on a single role (Infrastructure Lambda role).

   Permissions that allow powerful actions (such as creating and deleting CloudFormation stacks) are locked down and allowed only on resources that start with tenantcluster-. The exception is AWS KMS, because of its resource naming convention. The ingested tenant name from the API will be prepended with tenantcluster- along with other validation checks (alphanumeric with dash only, and limited to less than 30 characters to fit into most AWS resource naming). This ensures that the tenant name will not accidentally result in disruption of core infrastructure stacks or resources.

Tenant technology stack

A CloudFormation template is stored in the S3 bucket. The template provisions the tenant-specific AWS KMS key, a CloudWatch alarm, an SNS topic, an SQS queue, and an SQS policy.

The AWS KMS key is used for data encryption by Amazon SNS and Amazon SQS for their messages. The security practices for AwsSolutions-SNS2 and AwsSolutions-SQS2 recommend that you set up Amazon SNS and Amazon SQS with encryption. However, CloudWatch alarms don’t work with Amazon SNS when using an AWS managed key, so you must use a customer managed key in this case. For more information, see the AWS Knowledge Center.

The SQS policy is used on the Amazon SQS queue to allow the created SNS topic to deliver the message to the queue. Without the SQS policy, the access will be denied. For more information, see the Amazon SNS documentation.