Define success criteria

Any performance testing initiative needs to start with a clear understanding of what you are trying to achieve. This section provides advice on setting goals for performance testing and how to communicate objectives such as performance and cost optimization to business stakeholders.

Improving customer experience

Customer experience is an aspect of competitive differentiation for ISV applications. Examples of customer experience outcomes such as faster response time, lower latency, and reduced error rates due to lower saturation and less requests dropped. These outcomes result in higher customer satisfaction with a service, leading to improved retention and higher lifetime value.

Graviton’s performance advantage translates into higher throughput and lower latency for common cloud workloads such as databases and search clusters. This means your application can serve more user requests and complete them in less time, leading to improved customer experience and retention. The introduction of mixed instance policies (instances with different characteristics such as the processor type) in Amazon EC2 Auto Scaling enables you to phase in instances powered by the Graviton processor and to monitor their performance in a real-world setting (a practice known as A/B testing). When instances underperform, you phase them out.

Reducing computing cost

Computing costs are a key input into lowering COGS, an important measure of SaaS profitability. Industry benchmarks for publicly listed SaaS companies are in the 60–80% range measured as gross profit margin (SaaS revenues less COGS divided by SaaS revenues). This means that lower unit costs per compute instance and smaller instance footprints are important for SaaS providers that are publicly listed and report their financial data in earnings calls.

Graviton’s competitive pricing and performance advantage means you can lower computing cost by reducing unit costs and running smaller server footprints. Graviton instances offer up to 40% better price performance compared to current generation instance types. Instances are on average 10–20% cheaper than alternatives in the same instance family.

Graviton’s performance advantage also enables you to run smaller server footprints, leading to less instance to maintain and pay for in production. The combination of these benefits leads to lower COGS, an important measure of profitability and internal efficiency for SaaS providers that are publicly listed.

Measuring price performance

Price performance is the ability of a system to deliver performance at a particular price. The price performance metrics is often used in performance engineering to compare different systems. In the context of modeling the impact of unit cost reduction and performance-related reductions resulting from the introduction of Graviton, use the following definition:

For example, if you migrate from c5.2xlarge to c6g.2xlarge and see a 5% performance increase, you would observe a 34.6% performance/price improvement (1.05/0.78 – or 5% performance increase, with c6g.2xlarge being 78% the cost of c5.2xlarge per hour). If the intent of your migration is exclusively to lower the cost of operating the workload without impacting the current user experience, then you will have accomplished this goal.