Pipelining and $AWSHistory
For AWS service calls that return collections, the objects within the collection are
enumerated to the pipeline. Result objects that contain additional fields beyond the
collection and which are not paging control fields have these fields added as Note
properties for the calls. These Note properties are logged in the new
$AWSHistory
session variable, should you need to access this data. The
$AWSHistory
variable is described in the next section.
In versions of the Tools for Windows PowerShell prior to v1.1, the collection object itself was emitted, which required the use of foreach {$_.getenumerator()} to continue pipelining.
Examples
The following example returns a list of AWS Regions and your Amazon EC2 machine images (AMIs) in each Region.
PS >
Get-AWSRegion | % { Echo $_.Name; Get-EC2Image -Owner self -Region $_ }
The following example stops all Amazon EC2 instances in the current default region.
PS >
Get-EC2Instance | Stop-EC2Instance
Because collections enumerate to the pipeline, the output from a given cmdlet might be
$null
, a single object, or a collection. If it is a collection, you can use
the .Count
property to determine the size of the collection. However, the
.Count
property is not present when only a single object is emitted. If
your script needs to determine, in a consistent way, how many objects were emitted, you can
check the EmittedObjectsCount
property of the last command value in
$AWSHistory
.
$AWSHistory
To better support pipelining, output from AWS cmdlets is not reshaped to include the service response and result instances as Note properties on the emitted collection object. Instead, for those calls that emit a single collection as output, the collection is now enumerated to the PowerShell pipeline. This means that the AWS SDK response and result data cannot exist in the pipe, because there is no containing collection object to which it can be attached.
Although most users probably won't need this data, it can be useful for diagnostic purposes, because you can see exactly what was sent to and received from the underlying AWS service calls made by the cmdlet.
Starting with version 1.1, this data and more is now available in a new shell variable
named $AWSHistory
. This variable maintains a record of AWS cmdlet
invocations and the service responses that were received for each invocation.
Optionally, this history can be configured to also record the service requests that each
cmdlet made. Additional useful data, such as the overall execution time of the cmdlet,
can also be obtained from each entry.
Each entry in the $AWSHistory.Commands
list is of type
AWSCmdletHistory
. This type has the following useful members:
- CmdletName
-
Name of the cmdlet.
- CmdletStart
-
DateTime that the cmdlet was run.
- CmdletEnd
-
DateTime that the cmdlet finished all processing.
- Requests
-
If request recording is enabled, list of last service requests.
- Responses
-
List of last service responses received.
- LastServiceResponse
-
Helper to return the most recent service response.
- LastServiceRequest
-
Helper to return the most recent service request, if available.
Note that the $AWSHistory
variable is not created until an AWS cmdlet
making a service call is used. It evaluates to $null until that time.
Earlier versions of the Tools for Windows PowerShell emitted data related to service responses as
Note
properties on the returned object. These are now found on the
response entries that are recorded for each invocation in the list.
Set-AWSHistoryConfiguration
A cmdlet invocation can hold zero or more service request and response entries. To
limit memory impact, the $AWSHistory
list keeps a record of only the
last five cmdlet executions by default; and for each, the last five service
responses (and if enabled, last five service requests). You can change these default
limits by running the Set-AWSHistoryConfiguration
cmdlet. It allows you
to both control the size of the list, and whether service requests are also
logged:
PS >
Set-AWSHistoryConfiguration -MaxCmdletHistory <value> -MaxServiceCallHistory <value> -RecordServiceRequests
The -MaxCmdletHistory
parameter sets the maximum number of cmdlets
that can be tracked at any time. A value of 0 turns off recording of AWS cmdlet
activity. The -MaxServiceCallHistory
parameter sets the maximum number
of service responses (and/or requests) that are tracked for each cmdlet. The
-RecordServiceRequests
parameter, if specified, turns on tracking
of service requests for each cmdlet. All parameters are optional.
If run with no parameters, Set-AWSHistoryConfiguration
simply turns
off any prior request recording, leaving the current list sizes unchanged.
To clear all entries in the current history list, run the
Clear-AWSHistory
cmdlet.
$AWSHistory
Examples
Enumerate the details of the AWS cmdlets that are being held in the list to the pipeline.
$AWSHistory.Commands
PS >
Access the details of the last AWS cmdlet that was run:
PS >
$AWSHistory.LastCommand
Access the details of the last service response received by the last AWS cmdlet that was run. If an AWS cmdlet is paging output, it may make multiple service calls to obtain either all data or the maximum amount of data (determined by parameters on the cmdlet).
PS >
$AWSHistory.LastServiceResponse
Access the details of the last request made (again, a cmdlet may make more than one request if it is paging on the user's behalf). Yields $null unless service request tracing is enabled.
PS >
$AWSHistory.LastServiceRequest
Automatic Page-to-Completion for Operations that Return Multiple Pages
For service APIs that impose a default maximum object return count for a given call or that support pageable result sets, all cmdlets "page-to-completion" by default. Each cmdlet makes as many calls as necessary on your behalf to return the complete data set to the pipeline.
In the following example, which uses Get-S3Object
, the
$c
variable contains S3Object
instances for
every key in the bucket test
, potentially a
very large data set.
PS >
$c = Get-S3Object -BucketName test
If you want to retain control of the amount of data returned, you can use
parameters on the individual cmdlets (for example, MaxKey
on
Get-S3Object
) or you can explicitly handle paging yourself by using
a combination of paging parameters on the cmdlets, and data placed in the
$AWSHistory
variable to get the service's next token data. The
following example uses the MaxKeys parameter to limit the number of
S3Object
instances returned to no more than the first 500 found in
the bucket.
PS >
$c = Get-S3Object -BucketName test -MaxKey 500
To know if more data was available but not returned, use the
$AWSHistory
session variable entry that recorded the service calls
made by the cmdlet.
If the following expression evaluates to $true, you can find the next
marker for the next set of results using
$AWSHistory.LastServiceResponse.NextMarker
:
$AWSHistory.LastServiceResponse -ne $null && $AWSHistory.LastServiceResponse.IsTruncated
To manually control paging with Get-S3Object
, use a combination of
the MaxKey
and Marker
parameters for the cmdlet and the
IsTruncated
/NextMarker
notes on the last recorded
response. In the following example, the variable $c
contains up to a
maximum of 500 S3Object
instances for the next 500 objects that are
found in the bucket after the start of the specified key prefix marker.
PS >
$c = Get-S3Object -BucketName test -MaxKey 500 -Marker $AWSHistory.LastServiceResponse.NextMarker