Testing techniques for serverless applications - AWS Prescriptive Guidance
Mock testingTesting in the cloudEmulation testing

Testing techniques for serverless applications

There are three main approaches to running tests against serverless application code: mock testing, testing in the cloud, and emulation testing. You can generally find a mix of these approaches in use within the scope of a single project. For example, you might use mock testing when you’re developing your code locally, but your code might be tested in the cloud as part of a nightly continuous integration and continuous delivery (CI/CD) process.

Mock testing

Mock testing is a strategy where you create replacement objects in your code that simulate the behavior of cloud services. For example, you can write a test that uses a mock of the Amazon S3 service, and you can configure that mock to return a specific response object whenever the CreateObject method is called. When the test runs, the mock returns the response without calling Amazon S3 or any other service endpoints.

These replacement objects are often generated by a mock framework to reduce the amount of implementation necessary for a given test. Some mock frameworks are generic and others are designed specifically for AWS SDKs.

Mocks, along with stubs, fall into the broader category of fakes. Mocks differ from emulators (discussed later in this section) in that mocks are typically created or configured by a developer as part of the test code, whereas emulators are standalone applications that expose APIs (such as REST APIs) in the same manner as the systems they emulate.

The advantages of using mocks include the following:

  • Mocks can simulate third-party services that are beyond the control of your application, such as APIs and software as a service (SaaS) providers, without needing direct access to those services.

  • Mocks are also useful for testing failure conditions, especially when such conditions (such as service outages) are hard to simulate.

  • Like emulators, mock frameworks can provide fast local development after they’ve been configured.

  • Mocks can provide substitute behavior for virtually any kind of object, so mocking strategies can create coverage for a wider variety of services than emulators.

  • When new features or behaviors become available, mock testing can react more quickly by using (or falling back to) a generic mock framework, which can simulate the new features as soon as the updated AWS SDK becomes available.

Mock testing has these disadvantages:

  • Mocks generally require a non-trivial amount of setup and configuration effort, specifically when trying to determine return values from different services in order to properly mock responses.

  • Because mocks are written or configured by the developers who write the tests, this is an added responsibility for developers.

  • You might need to have access to the cloud in order to understand the APIs and return values of services.

  • Mocks can also be difficult to maintain, because they require updates whenever the mocked cloud API signatures change, return value schemas evolve, or the tested application logic is extended to make calls to new APIs. These changes create additional test development workload for developers.

  • Mock tests might pass in desktop environments but fail in the cloud.

  • Mock frameworks, like emulators, are limited in testing or detecting AWS Identity and Access Management (IAM) policy or quota limitations.

  • Although mocks are better at simulating what an application will do when authorization fails or a quota is exceeded, mock testing can’t determine which outcome will actually occur when the code is deployed to the production environment.

Testing in the cloud

Testing in the cloud is the process of running tests against code that is deployed to a cloud environment instead of a desktop environment. The value of testing in the cloud increases with cloud-native application development. For example:

  • You can run tests in the cloud against the most recent APIs and return values.

  • Your tests can cover IAM policies, service quotas, configurations, and all services.

  • Your cloud test environment most closely resembles your production runtime environment, so tests that you run in the cloud are likely to achieve the most consistent results as they progress through environments.

Drawbacks to testing in the cloud include the following:

  • Deployments to cloud environments typically take more time than deployments to desktop environments. Tools such as AWS Serverless Application Model (AWS SAM) Accelerate, AWS Cloud Development Kit (AWS CDK) watch mode, and SST help decrease the latency involved with cloud deployment iterations.

  • Cloud testing can incur additional service costs, unlike local testing, which uses your local development environment.

  • Testing in the cloud might be less feasible if you don’t have high-speed internet access.

  • Testing in the cloud typically requires cloud environments that are isolated from one another. Environment boundaries are often drawn at the stack level in shared accounts for developer environments, sometimes by using namespace type strategies such as using prefixes to identify ownership. For pre-production and production environments, boundaries are typically drawn at the account level to insulate workloads from noisy neighbor problems, to support least privilege security controls, and to protect sensitive data. The requirement to create isolated environments might place additional burdens on DevOps teams, especially if they’re in an enterprise that has strict controls around accounts and infrastructure.

Emulation testing

Emulation testing is enabled by locally running applications known as emulators that resemble AWS services. Emulators have APIs that are similar to their cloud counterparts and provide similar return values. They can also simulate state changes that are initiated by API calls. For example, an Amazon S3 emulator might store an object on the local disk when the PutObject method is called. When GetObject is called with the same key, the emulator reads the same object from disk and returns it.

The advantages of emulation testing include the following:

  • It provides the most familiar environment for developers who are used to developing code exclusively in a local environment. For example, if you’re familiar with the development of an n-tier application, you might have a database engine and web server, similar to those running in production, running on your local machine to provide quick, local, isolated test capability.

  • It doesn’t require any changes to cloud infrastructure (such as developer cloud accounts), so it’s easy to implement with existing testing patterns. Emulation testing provides the benefits of fast, local development iterations.

However, emulators have several drawbacks:

  • They’re often hard to set up and replicate, especially when they’re used as a part of CI/CD pipelines. This might increase the workload and maintenance for IT staff or for developers who manage their own software installations.

  • Emulated features and APIs typically lag behind services changes and might hinder the adoption of new features until support is added.

  • Emulators might require support, updates, bug fixes, or feature parity enhancements, which are the responsibility of the emulator author (which is often a third-party company).

  • Tests that rely on emulators can provide successful results locally, but they might fail in the cloud due to interactions with other aspects of AWS such as IAM policies and quotas, which are generally not emulated.

  • Some AWS services don’t have emulators available, so if you rely only on emulation, you might not have a satisfactory testing option for portions of your application.