Use the AWSTOE component document framework for custom components
To build a component using the AWS Task Orchestrator and Executor (AWSTOE) component framework, you must provide a YAML-based document that represents the phases and steps that apply for the component you create. AWS services use your component when they create a new Amazon Machine Image (AMI) or container image.
Topics
- Component document workflow
- Component logging
- Input and output chaining
- Document schema and definitions
- Document examples
- Use variables in your custom component document
- Use conditional constructs in AWSTOE
- Use comparison operators in AWSTOE component documents
- Use logical operators in AWSTOE component documents
- Use looping constructs in AWSTOE
Component document workflow
The AWSTOE component document uses phases and steps to group related tasks, and organize those tasks into a logical workflow for the component.
Tip
The service that uses your component to build an image might implement rules about what phases to use for their build process, and when those phases are allowed to run. This is important to consider when you design your component.
Phases
Phases represent the progression of your workflow through the image build
process. For example, the Image Builder service uses build and
validate phases during its build stage for
the images it produces. It uses the test and
container-host-test phases during its test stage
to ensure that the image snapshot or container image produces the expected results
before creating the final AMI or distributing the container image.
When the component runs, the associated commands for each phase are applied in the order that they appear in the component document.
Rules for phases
-
Each phase name must be unique within a document.
-
You can define many phases in your document.
-
You must include at least one of the following phases in your document:
-
build – for Image Builder, this phase is generally used during the build stage.
-
validate – for Image Builder, this phase is generally used during the build stage.
-
test – for Image Builder, this phase is generally used during the test stage.
-
-
Phases always run in the order that they are defined in the document. The order in which they are specified for AWSTOE commands in the AWS CLI has no effect.
Steps
Steps are individual units of work that define the workflow within each phase. Steps run in sequential order. However, input or output for one step can also feed into a subsequent step as input. This is called "chaining".
Rules for steps
-
The step name must be unique for the phase.
-
The step must use a supported action (action module) that returns an exit code.
For a complete list of supported action modules, how they work, input/output values, and examples, see Action modules supported by AWSTOE component manager.
Component logging
AWSTOE creates a new log folder on the EC2 instances that are used for building and testing a new image, each time your component runs. For container images, the log folder is stored in the container.
To assist with troubleshooting if something goes wrong during the image creation process, the input document and all of the output files AWSTOE creates while running the component are stored in the log folder.
The log folder name is comprised of the following parts:
-
Log directory – when a service runs a AWSTOE component, it passes in the log directory, along with other settings for the command. For the following examples, we show the log file format that Image Builder uses.
-
Linux and macOS:
/var/lib/amazon/toe/ -
Windows:
$env:ProgramFiles\Amazon\TaskOrchestratorAndExecutor\
-
-
File prefix – This is a standard prefix used for all components: "
TOE_". -
Run time – This is a timestamp in YYYY-MM-DD_HH-MM-SS_UTC-0 format.
-
Execution ID – This is the GUID that is assigned when AWSTOE runs one or more components.
Example: /var/lib/amazon/toe/TOE_2021-07-01_12-34-56_UTC-0_a1bcd2e3-45f6-789a-bcde-0fa1b2c3def4
AWSTOE stores the following core files in the log folder:
Input files
-
document.yaml – The document that is used as input for the command. After the component runs, this file is stored as an artifact.
Output files
-
application.log – The application log contains timestamped debug level information from AWSTOE about what's happening as the component is running.
-
detailedoutput.json – This JSON file has detailed information about run status, inputs, outputs, and failures for all documents, phases, and steps that apply for the component as it runs.
-
console.log – The console log contains all of the standard out (stdout) and standard error (stderr) information that AWSTOE writes to the console while the component is running.
-
chaining.json – This JSON file represents optimizations that AWSTOE applied to resolve chaining expressions.
Note
The log folder might also contain other temporary files that are not covered here.
Input and output chaining
The AWSTOE configuration management application provides a feature for chaining inputs and outputs by writing references in the following formats:
{{ phase_name.step_name.inputs/outputs.variable
}}
or
{{ phase_name.step_name.inputs/outputs[index].variable
}}
The chaining feature allows you to recycle code and improve the maintainability of the document.
Rules for chaining
-
Chaining expressions can be used only in the inputs section of each step.
-
Statements with chaining expressions must be enclosed in quotes. For example:
-
Invalid expression:
echo {{ phase.step.inputs.variable }} -
Valid expression:
"echo {{ phase.step.inputs.variable }}" -
Valid expression:
'echo {{ phase.step.inputs.variable }}'
-
-
Chaining expressions can reference variables from other steps and phases in the same document. However, the calling service might have rules that require chaining expressions to operate only within the context of a single stage. For example, Image Builder does not support chaining from the build stage to the test stage, as it runs each stage independently.
-
Indexes in chaining expressions follow zero-based indexing. The index starts with zero (0) to reference the first element.
Examples
To refer to the source variable in the second entry of the following example step,
the chaining pattern is {{
build..SampleS3Download.inputs[1].source
}}
phases: - name: 'build' steps: - name:SampleS3Downloadaction: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://sample-bucket/sample1.ps1' destination: 'C:\sample1.ps1' - source: 's3://sample-bucket/sample2.ps1' destination: 'C:\sample2.ps1'
To refer to the output variable (equal to "Hello") of the following example step,
the chaining pattern is {{
build..SamplePowerShellStep.outputs.stdout
}}
phases: - name: 'build' steps: - name:SamplePowerShellStepaction: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: commands: - 'Write-Host "Hello"'
Document schema and definitions
The following is the YAML schema for a document.
name: (optional) description: (optional) schemaVersion: "string" phases: - name: "string" steps: - name: "string" action: "string" timeoutSeconds: integer onFailure: "Abort|Continue|Ignore" maxAttempts: integer inputs:
The schema definitions for a document are as follows.
| Field | Description | Type | Required |
|---|---|---|---|
| name | Name of the document. | String | No |
| description | Description of the document. | String |
No |
| schemaVersion | Schema version of the document, currently 1.0. | String |
Yes |
| phases | A list of phases with their steps. |
List |
Yes |
The schema definitions for a phase are as follows.
| Field | Description | Type | Required |
|---|---|---|---|
| name | Name of the phase. | String | Yes |
| steps | List of the steps in the phase. | List |
Yes |
The schema definitions for a step are as follows.
| Field | Description | Type | Required | Default value |
|---|---|---|---|---|
| name | User-defined name for the step. | String | ||
| action | Keyword pertaining to the module that runs the step. | String | ||
| timeoutSeconds |
Number of seconds that the step runs before failing or retrying. Also, supports -1 value, which indicates infinite timeout. 0 and other negative values are not allowed. |
Integer |
No |
7,200 sec (120 mins) |
| onFailure |
Specifies what the step should do in case of failure. Valid values are as follows:
|
String |
No |
Abort |
| maxAttempts | Maximum number of attempts allowed before failing the step. | Integer |
No |
1 |
| inputs | Contains parameters required by the action module to run the step. | Dict |
Yes |
Document examples
The following examples show AWSTOE component documents that perform tasks for the target operating system.
