Step 1: Deploy the base architecture Step 2: Scale out to multiple clusters Step 3: Tune PBS communication Further reading

Scaling Synopsys SiliconSmart on AWS

Publication date: April 1, 2021 (Diagram history)

With these three architectures, you can scale Synopsys SiliconSmart characterization jobs on AWS. First, deploy the base Scale-Out Computing on AWS architecture. Next, scale out to multiple clusters across Availability Zones. Finally, tune PBS communication and the PostgreSQL database for large-scale job execution.

Step 1: Deploy the base architecture

With this architecture, you can scale 100,000 or more concurrent Synopsys SiliconSmart jobs by using Scale-Out Computing on AWS. This first step deploys the base architecture by using AWS CloudFormation.

Reference architecture diagram showing step 1 of scaling Synopsys SiliconSmart by deploying Scale-Out Computing on AWS with CloudFormation, Amazon EC2 Spot Fleet, and FSx for Lustre.

The following steps describe the deployment and configuration for this architecture:

Set up a Amazon VPC with private subnets and access from on-premises through AWS Direct Connect or AWS Site-to-Site VPN before launching Scale-Out Computing on AWS.
Deploy Scale-Out Computing on AWS as a baseline by using CloudFormation.
Enable license server access with one of two methods. Use the VM UUID (which is the AWS instance ID), or use an Elastic IP address attached to the instance.
Use AWS Auto Scaling groups for coordinator instances.
Use Amazon EC2 Spot Fleet to launch a compute fleet for running jobs.
Use POSIX-compliant file systems for the Scale-Out Computing on AWS environment. Use Amazon Elastic File System for cluster automation, and FSx for Lustre for SiliconSmart binaries and job data.
Move data to and from Amazon Simple Storage Service by using an Amazon S3 endpoint.
Enable encryption for intellectual property (IP) data and multiple AWS services by using AWS KMS.

Step 2: Scale out to multiple clusters

With this architecture, you can scale the previous Scale-Out Computing on AWS deployment to multiple clusters and Availability Zones. Limit total SiliconSmart jobs to fewer than 40,000 for each scheduler instance and keep each cluster within the same Availability Zone.

Reference architecture diagram showing step 2 of scaling Synopsys SiliconSmart by deploying multiple clusters across Availability Zones with FSx for Lustre and Amazon EC2 Spot Fleet.

The following steps describe the scale-out configuration for this architecture:

Limit total SiliconSmart jobs to fewer than 40,000 for each scheduler instance. Keep each cluster within the same Availability Zone.
Add 1 communication instance for every 1,000 compute instances. See Step 3 for tuning details.
Create an FSx for Lustre file system for each cluster in each Availability Zone. Move the data needed to run your jobs.
Launch a license server in each Availability Zone to reduce or eliminate inter-Availability Zone traffic.
Use the scheduler instance to launch SiliconSmart coordinators through an AWS Auto Scaling group. Scale to 50 or more coordinators.
Launch Amazon Elastic Compute Cloud instances in the compute fleet to meet the cores required by the coordinators.
Each coordinator invokes 500 to 2,000 job submissions by using qsub commands. Jobs then run on the compute fleet.

Step 3: Tune PBS communication

With this architecture, you can move the PBS Communication process from the PBS Scheduler to multiple communication instances. This step tunes the scheduler and database configuration for large-scale job execution.

Reference architecture diagram showing step 3 of scaling Synopsys SiliconSmart by moving PBS communication to multiple instances and tuning PostgreSQL.

The following steps describe the PBS tuning configuration for this architecture:

Increase the number of threads for the pbs_sched process. Use the maximum processes for each scheduler instance.
Tune the PostgreSQL database (for example, max memory, number of connections, processes).
Launch communication instances (which run the pbs_comm process) for every 200 to 1,000 compute instances.
Update PBS configuration in coordinators and compute instances (in the AMI) to use multiple communication instances.

General recommendations:

Create AMIs for the license manager, scheduler, communication nodes, and coordinators with all software and EDA base tooling baked in. This reduces ramp-up time.
Reduce inter-Availability Zone traffic by keeping all components of a cluster within the same Availability Zone.
Monitor throughput, IOPS, and metadata server operations to optimize file system performance.