Manual set up to develop custom components with AWSTOE
The AWS Task Orchestrator and Executor (AWSTOE) application is a standalone application that creates, validates, and runs commands within a component definition framework. AWS services can use AWSTOE to orchestrate workflows, install software, modify system configurations, and test image builds.
Follow these steps to manually install the AWSTOE application and use it as a stand-alone application to develop custom components. Image Builder takes care of these steps for you, if you use the Image Builder console or AWS CLI commands to create custom components. For more information, see Create custom components with Image Builder.
Get started steps
Step 1: Install AWSTOE
To develop components locally, download and install the AWSTOE application.
-
Download the AWSTOE application
To install AWSTOE, choose the appropriate download link for your architecture and platform. For the full list of application download links, see AWSTOE downloads
Important
AWS is phasing out support for TLS versions 1.0 and 1.1. To access the S3 bucket for AWSTOE downloads, your client software must use TLS version 1.2 or later. For more information, see this AWS Security Blog post
. -
Verify the signature
The steps for verifying your download depend on the server platform where you run the AWSTOE application after you install it. To verify your download on a Linux server, see Verify the signature on Linux. To verify your download on a Windows server, see Verify the signature on Windows.
Important
AWSTOE is invoked directly from its download location. There is no need for a separate install step. This also means that AWSTOE can make changes to the local environment.
To ensure that you isolate changes during component development, we recommend that you use an EC2 instance to develop and test AWSTOE components.
Step 2: Set AWS credentials
AWSTOE requires AWS credentials to connect to other AWS services, such as Amazon S3 and Amazon CloudWatch, when running tasks, such as:
-
Downloading AWSTOE documents from a user-provided Amazon S3 path.
-
Running
S3Download
orS3Upload
action modules. -
Streaming logs to CloudWatch, when enabled.
If you are running AWSTOE on an EC2 instance, then running AWSTOE uses the same permissions as the IAM role attached to the EC2 instance.
For more information about IAM roles for EC2, see IAM roles for Amazon EC2.
The following examples show how to set AWS credentials using the
AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables.
To set these variables on Linux, macOS, or Unix, use export
.
$
export AWS_ACCESS_KEY_ID=your_access_key_id
$
export AWS_SECRET_ACCESS_KEY=your_secret_access_key
To set these variables on Windows using PowerShell, use $env
.
C:\>
$env:AWS_ACCESS_KEY_ID=your_access_key_id
C:\>
$env:AWS_SECRET_ACCESS_KEY=your_secret_access_key
To set these variables on Windows using the command prompt, use
set
.
C:\>
set AWS_ACCESS_KEY_ID=your_access_key_id
C:\>
set AWS_SECRET_ACCESS_KEY=your_secret_access_key
Step 3: Develop component documents locally
Components are authored with plaintext YAML documents. For more information about document syntax, see Use the AWSTOE component document framework for custom components.
The following are example Hello World component documents to help you get started.
hello-world-windows.yml
.
name: Hello World description: This is Hello World testing document for Windows. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the test phase.'
hello-world-linux.yml
.
name: Hello World description: This is hello world testing document for Linux. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the test phase.'
Step 4: Validate AWSTOE components
You can validate the syntax of AWSTOE components locally with the AWSTOE
application. The following examples show the AWSTOE application
validate
command to validate the syntax of a component
without running it.
Note
The AWSTOE application can validate only the component syntax for the
current operating system. For example, when running
awstoe.exe
on Windows, you cannot validate the syntax
for a Linux document that uses the ExecuteBash
action
module.
Windows
C:\>
awstoe.exe validate --documents C:\Users\user\Documents\hello-world.yml
Linux
$
awstoe validate --documents /home/user/hello-world.yml
Step 5: Run AWSTOE components
The AWSTOE application can run one or more phases of specified documents
using the --phases
command line argument. Supported values for
--phases
are build
, validate
, and
test
. Multiple phase values can be entered as comma
separated values.
When you provide a list of phases, the AWSTOE application sequentially runs
the specified phases of each document. For example, AWSTOE runs the
build
and validate
phases of
document1.yaml
, followed by the build
and
validate
phases of document2.yaml
.
To ensure that your logs are stored securely and retained for troubleshooting, we recommend configuring log storage in Amazon S3. In Image Builder, the Amazon S3 location for publishing logs is specified in the infrastructure configuration. For more information about infrastructure configuration, see Manage Image Builder infrastructure configuration
If a list of phases is not provided, the AWSTOE application runs all phases in the order listed in the YAML document.
To run specific phases in single or multiple documents, use the following commands.
Single phase
awstoe run --documents hello-world.yml --phases build
Multiple phases
awstoe run --documents hello-world.yml --phases build,test
Document run
Run all phases in a single document
awstoe run --documents
documentName
.yaml
Run all phases in multiple documents
awstoe run --documents
documentName1
.yaml,documentName2
.yaml
Enter Amazon S3 information to upload AWSTOE logs from a user-defined local path (recommended)
awstoe run --documents
documentName
.yaml --log-s3-bucket-name <S3Bucket> --log-s3-key-prefix <S3KeyPrefix> --log-s3-bucket-owner <S3BucketOwner> --log-directory <local_path>
Run all phases in a single document, and display all logs on the console
awstoe run --documents
documentName
.yaml --trace
Example command
awstoe run --documents s3://bucket/key/doc.yaml --phases build,validate
Run document with unique ID
awstoe run --documents <documentName>.yaml --execution-id <user provided id> --phases <comma separated list of phases>
Get help with AWSTOE
awstoe --help