Developing with a serverless workflow

Revised: 2024-02-15

In previous chapters, we presented serverless development from a conceptual point of view. You are likely wondering how serverless development compares to traditional development at a practical level.

Fundamentally, the workflow is the same. You create and deploy code, invoke and test features, update and fix bugs, package code with assets and libraries, apply security and secrets, and release to production. After release, you monitor results, and then repeat the cycle.

Tip

For a hands-on approach, check out the Serverless Patterns Workshop.

In the workshop, you will experience a typical serverless development lifecycle, including creating and deploying code and infrastructure through the web-based management console, the CLI, and with Infrastructure as Code (IaC) automation.

Serverless fundamentals learning path as text

The following is a text representation of the key concepts in the precedingdiagram.

Event Driven Architecture

• Events are sent to services. Services produce additional events.

• Event 1 is processed by two services (A and B). Service A produces 'event 2' which is processed by Service B.

Workshop: Intro to Serverless

• Connect a REST API to a Lambda function to retrieve data from DynamoDB

• Services: API Gateway, Lambda, DynamoDB

Development Workflow

Create & Deploy

• Management Console (essential)

• Infrastructure as Code (IaC)

• Note: The console gives you full manual control.

• Infrastructure as code topics and services: AWS SAM (important), AWS CloudFormation(essential), AWS CDK

• Note: Automate provisioning, code deploy, and more!

Invoke & Test

• Synchronous (essential)

• Asynchronous

Update & Package

• AWS SAM CLI can automate updates, packaging, and local testing.

• Packaging can include containers and layers - see Lambda for more info

Secure

• Least-priviledge

• Execution Role

• Resource-based Policy (RBAC)

• Services: IAM

• Note: If you start with less restrictive development permissions, always audit and use least-privilege for production.

Monitor

• Logs, Errors

• Services: CloudWatch, X-Ray

Workshop: Synchronous Invoke

• Note: Dive deeper into synchronous invocation with Serverless Workshop Module 2 and IaC using Serverless Application Model (AWS SAM).

• Services: API Gateway, Lambda, DynamoDB + AWS SAM CLI

Development workflow

Development workflow is similar regardless of programming language, tooling, or architecture. After you pick a language, app framework, and set up your local workstation, the DevSecOps cycle begins:

• Plan new features

• Code new features/fixes — write code, build, test locally, commit

• Build code continuously

• Test in staging environment

• Release to production

• Deploy infrastructure and updated code package

• Operate - sanity test, configure

• Monitor - scale, optimize, debug

Your unique business process may vary, but developing code for features (step 2) is nearly universal. Developers write code, save, build and test locally. They commit after everything works as planned. This mini-cycle is sometimes referred to as the inner loop. You can think of it as a constantly spinning flywheel, with releases flying off to be deployed into production.

Security (Sec) is, or should be, part of the process at every step, as shown in the diagram:

For traditional applications, developers commonly use a local development environment to test their code end-to-end before committing. This local environment needs to maintain parity with staging and production environments. That can be a full-time job in itself and prone to differences and drift.

With serverless, more resources are in the cloud. Replicating the production environment on your workstation is more challenging. Fortunately, spinning up developer sandboxes that replicate production infrastructure is easier with serverless.

The essential skill you need is how to create infrastructure programmatically, known as infrastructure as code or IaC. With IaC, you can automate the creation of service resources, code, security settings, and connections between services. Because serverless can scale down to zero, every developer can have an environment that is an exact replica of production. A developer could spin up a dozen such instances, if necessary. For example, perhaps you want to stress test with various allocations of memory and CPU.

You may be wondering, “Can’t I do this with containers?” That is similar, but serverless IaC brings the benefits of creating resources and code all the way down to the individual function level. You can deploy new code for a specific function without disrupting existing infrastructure or other functions.

Create & deploy your solution

Nearly all learning resources for AWS begin with the AWS Management Console, referred to as simply “the console”.

With the web-based console, you can create and deploy services, resources, and code into the cloud. Wizards and automated workflows simplify the process. You can get a Management Console mobile application to monitor resources, view alarms, and access billing information. The following screenshot shows favorited and recently visited services:

Create infrastructure with code

The AWS Management Console is essential. It provides full manual control and access to all of your services. However, most developers build from within a local integrated development environment (IDE) or by running commands on the command line.

When you need to create and deploy more than a few resources, tools are essential to automate the process. You will need to understand Infrastructure as Code and tools to automate creating and deploying your code and infrastructure resources.

These are some of the tools used to automate creation and maintenance of your cloud infrastructure and code:

• AWS CLI tool can control and automate AWS services through the same service API used by the language SDKs

• AWS Serverless Application Model uses higher level templates and CLI to speed up the code development cycle

• AWS Cloud Development Kit (AWS CDK) is a code-first approach to defining infrastructure

A brief history of AWS SAM, AWS CloudFormation, and AWS CDK

Before we dive into AWS SAM command line tools, we need to introduce SAM templates. And before that, you need to know about AWS CloudFormation and AWS CloudFormation templates.

In 2006, there were only a few services, including S3 and EC2. Managing these services was not complex, but as more services were added to the AWS portfolio, creating and managing services manually became cumbersome.

In 2011, AWS released AWS CloudFormation as a service to create and configure resources. With AWS CloudFormation, you create text templates to define infrastructure and code. The service provisions and configures those resources. Adding resources became as easy as copy/paste!

AWS CloudFormation provisions your resources in a safe, repeatable manner, enabling you to frequently build your infrastructure and applications without manual steps. When you change the configuration, AWS CloudFormation determines the right operations to perform to update your stack. AWS CloudFormation can even roll back changes.

As new services were created, AWS CloudFormation template complexity increased. Large customer environments are defined by templates with tens of thousands of lines. While better than manual creation, at scale the templates can become difficult to manage.

In Nov 2016, AWS introduced the Serverless Application Model or AWS SAM, an open-source framework for defining serverless applications. AWS SAM templates use a shorthand syntax to define functions, APIs, databases, and event source mappings with just a few lines of text (YAML) per resource.

During deployment, AWS SAM transforms and expands the AWS SAM syntax into AWS CloudFormation syntax. Because of this, any AWS CloudFormation syntax can be added to AWS SAM templates. This gives AWS SAM all the power of AWS CloudFormation, but with fewer lines of configuration.

In July 2019, AWS launched AWS Cloud Development Kit (AWS CDK) a code-first approach to defining cloud application infrastructure. With AWS CDK, you define and provision infrastructure using code constructs. This brings the power of programming to IaC. Imagine you need to create hundreds of queues named according to your product categories. No problem! You could write a loop to provision the queues.

To review:

• AWS CloudFormation is a service that uses text templates to reliably automate provisioning infrastructure stacks.

• AWS SAM templates are a higher level of abstraction YAML that transform into AWS CloudFormation templates, and can embed AWS CloudFormation template syntax.

• AWS Cloud Development Kit (AWS CDK) is a code-first approach to defining infrastructure.

Invoke & test your solution

After you write code, you will want to verify that it works. Like traditional development, unit and integration tests serve this purpose.

Remember, serverless services rely on an event-driven architecture. Invoking your code to test requires creating and sending events to drive actions. AWS SAM CLI can invoke functions locally, either directly or inside a container.

You can run code locally during the development of a feature, using a test harness. This sets up an inner loop where you can iterate and test quickly. The following diagram illustrates the “Inner Loop” with a TestHarness to verify your Business Logic locally. It also shows how you can deploy AWS SAM templates to testing and production environments with the --profile setting.

For workflows that require resources that only exist in the cloud, you need a cloud-based environment for testing.

Since serverless services like Lambda and Amazon API Gateway are pay-per-use, developers no longer need to share environments. Every developer can have their own sandboxed AWS developer account. You do not need to replicate all resources in a local environment, but instead you can test with real resources in the cloud.

To review:

• You can invoke and test code locally that connects to some cloud resources.

• For cloud native tests, developers can set up sandboxed infrastructure environments that better mimic production.

Related resource(s):

• Learn how to write and run unit and integration tests in the Serverless Patterns Workshop. See Module 2 - Unit Test and Module 3 - Set up tests to test Lambda services with AWS SAM and Python.

• In the Lambda developer guide, see testing serverless functions and applications for practical test strategies.

Update & package your solution

After creating function(s) for your features, you’ll iterate. Updates and packaging go hand-in-hand. Depending on how your code is packaged, you have different options for making updates.

For scripting languages (Node.js, Python, and Ruby), you can edit your function code in the embedded Lambda console code editor. If your code is larger than 3MB, or you need to use libraries, or you use languages that the editor doesn't support (Java, Go, C#), then you must upload your function code as a .zip archive.

When you add libraries, make code changes, you will need to create a new package or container to upload to the cloud. This is a slow process that AWS SAM CLI can automate and accelerate. AWS SAM CLI has a deploy command that will push updates to AWS, and a feature called AWS SAM Accelerate that speeds up deployments by using service APIs instead of AWS CloudFormation templates. AWS SAM can also watch your local environment and automatically deploy changes to the cloud.

You need to learn about Lambda layers, another feature that speeds up update/deploy cycle. Lambda layers package libraries and dependencies to reduce deployment archive size. Layers promote code sharing and separation of responsibilities so that you can upload changes and iterate faster on the business logic.

Related resource(s):

• AWS SAM Accelerate - Quickly update serverless applications in AWS Cloud during development

• Lambda Containers - A container image, compatible with the Open Container Initiative (OCI) specification, is one way to bring all your function code and dependencies to the cloud.

• Lambda .zip archives - A .zip file archive includes your application code and its dependencies.

• Lambda Layers - A layer is a .zip file archive that can contain additional code or data. A layer can contain libraries, a custom runtime, data, or configuration files.

Secure your solution

Cloud security at AWS is the highest priority. As an AWS customer, you benefit from a data center and network architecture that is built to meet the requirements of the most security-sensitive organizations.

Security is a shared responsibility between AWS and you. The shared responsibility model describes this as security of the cloud and security in the cloud:

• Security of the cloud – AWS is responsible for protecting the infrastructure that runs AWS services in the AWS Cloud. AWS also provides you with services that you can use securely. Third-party auditors regularly test and verify the effectiveness of our security as part of the AWS compliance programs.

• Security in the cloud – Your responsibility is determined by the AWS service that you use. You are also responsible for other factors including the sensitivity of your data, your company’s requirements, and applicable laws and regulations.

From a developer perspective, you will use the Identity and Access Management service (IAM) to grant least-privilege access to users, services, and resources. For example, you will provide each of your functions with an execution role which defines specifically what actions the function can take. Permissions can be created and applied granularly using resource and attribute based access control policies.

Monitor your solution

All services, resources, and your application functions will record actions in CloudWatch so that you can monitor activity. Think of CloudWatch as a centralized place for logs, plus tools to view and notify you based on alert conditions.

• Monitoring provides the information about a system’s state and can help you discover that there is an issue.

• Observability uses instrumentation to provide context and insights into what is causing issues.

You can use AWS X-Ray to trace user requests through your application to identify bottlenecks and debug problems.

Instrumenting your application involves sending trace data for events within your application, along with metadata about each request. Many AWS services are already integrated with X-Ray, so for any traced request to your application, you can see detailed information for calls that your application makes to downstream AWS resources, microservices, databases, and web APIs.

In the following example, you can see the durations of various requests and updates saving state in an app called Scorekeep:

Summary

• The development lifecycle for serverless is similar to traditional for the “inner loop” to develop features.

• The management console gives you full manual control, but learning how to use infrastructure as code tools will help you spin up multiple environments quickly to replicate production for testing.

• AWS SAM, AWS CDK, and AWS CloudFormation are essential infrastructure as code tools.

• AWS SAM CLI provides tools to accelerate creation, invocation, testing, and deploy of code and resources to the AWS cloud.