Synopsis
Calls the Amazon Kinesis Firehose CreateDeliveryStream API operation.
Syntax
New-KINFDeliveryStream-DeliveryStreamName <String>-DeliveryStreamType <DeliveryStreamType>-ElasticsearchDestinationConfiguration_DomainARN <String>-RetryOptions_DurationInSecond <Int32>-CloudWatchLoggingOptions_Enabled <Boolean>-ProcessingConfiguration_Enabled <Boolean>-ExtendedS3DestinationConfiguration <ExtendedS3DestinationConfiguration>-ElasticsearchDestinationConfiguration_IndexName <String>-ElasticsearchDestinationConfiguration_IndexRotationPeriod <ElasticsearchIndexRotationPeriod>-BufferingHints_IntervalInSecond <Int32>-KinesisStreamSourceConfiguration_KinesisStreamARN <String>-CloudWatchLoggingOptions_LogGroupName <String>-CloudWatchLoggingOptions_LogStreamName <String>-ProcessingConfiguration_Processor <Processor[]>-RedshiftDestinationConfiguration <RedshiftDestinationConfiguration>-ElasticsearchDestinationConfiguration_RoleARN <String>-KinesisStreamSourceConfiguration_RoleARN <String>-ElasticsearchDestinationConfiguration_S3BackupMode <ElasticsearchS3BackupMode>-ElasticsearchDestinationConfiguration_S3Configuration <S3DestinationConfiguration>-BufferingHints_SizeInMBs <Int32>-SplunkDestinationConfiguration <SplunkDestinationConfiguration>-Tag <Tag[]>-ElasticsearchDestinationConfiguration_TypeName <String>-S3DestinationConfiguration <S3DestinationConfiguration>-Force <SwitchParameter>
Description
Creates a Kinesis Data Firehose delivery stream.
By default, you can create up to 50 delivery streams per AWS Region.
This is an asynchronous operation that immediately returns. The initial status of the delivery stream is
CREATING. After the delivery stream is created, its status is ACTIVE and it now accepts data. Attempts to send data to a delivery stream that is not in the ACTIVE state cause an exception. To check the state of a delivery stream, use DescribeDeliveryStream.
A Kinesis Data Firehose delivery stream can be configured to receive records directly from providers using PutRecord or PutRecordBatch, or it can be configured to use an existing Kinesis stream as its source. To specify a Kinesis data stream as input, set the DeliveryStreamType parameter to KinesisStreamAsSource, and provide the Kinesis stream Amazon Resource Name (ARN) and role ARN in the KinesisStreamSourceConfiguration parameter.
A delivery stream is configured with a single destination: Amazon S3, Amazon ES, Amazon Redshift, or Splunk. You must specify only one of the following destination configuration parameters: ExtendedS3DestinationConfiguration, S3DestinationConfiguration, ElasticsearchDestinationConfiguration, RedshiftDestinationConfiguration, or SplunkDestinationConfiguration.
When you specify S3DestinationConfiguration, you can also provide the following optional values: BufferingHints, EncryptionConfiguration, and CompressionFormat. By default, if no BufferingHints value is provided, Kinesis Data Firehose buffers data up to 5 MB or for 5 minutes, whichever condition is satisfied first. BufferingHints is a hint, so there are some cases where the service cannot adhere to these conditions strictly. For example, record boundaries might be such that the size is a little over or under the configured buffering size. By default, no encryption is performed. We strongly recommend that you enable encryption to ensure secure data storage in Amazon S3.
A few notes about Amazon Redshift as a destination: - An Amazon Redshift destination requires an S3 bucket as intermediate location. Kinesis Data Firehose first delivers data to Amazon S3 and then uses
COPYsyntax to load data into an Amazon Redshift table. This is specified in theRedshiftDestinationConfiguration.S3Configurationparameter. - The compression formats
SNAPPYorZIPcannot be specified inRedshiftDestinationConfiguration.S3Configurationbecause the Amazon RedshiftCOPYoperation that reads from the S3 bucket doesn't support these compression formats. - We strongly recommend that you use the user name and password you provide exclusively with Kinesis Data Firehose, and that the permissions for the account are restricted for Amazon Redshift
INSERTpermissions.
Parameters
-BufferingHints_IntervalInSecond <Int32>
Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination. The default value is 300 (5 minutes).
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_BufferingHints_IntervalInSeconds |
-BufferingHints_SizeInMBs <Int32>
Buffer incoming data to the specified size, in MBs, before delivering it to the destination. The default value is 5.We recommend setting this parameter to a value greater than the amount of data you typically ingest into the delivery stream in 10 seconds. For example, if you typically ingest data at 1 MB/sec, the value should be 10 MB or higher.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_BufferingHints_SizeInMBs |
-CloudWatchLoggingOptions_Enabled <Boolean>
Enables or disables CloudWatch logging.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_CloudWatchLoggingOptions_Enabled |
-CloudWatchLoggingOptions_LogGroupName <String>
The CloudWatch group name for logging. This value is required if CloudWatch logging is enabled.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_CloudWatchLoggingOptions_LogGroupName |
-CloudWatchLoggingOptions_LogStreamName <String>
The CloudWatch log stream name for logging. This value is required if CloudWatch logging is enabled.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_CloudWatchLoggingOptions_LogStreamName |
-DeliveryStreamName <String>
The name of the delivery stream. This name must be unique per AWS account in the same AWS Region. If the delivery streams are in different accounts or different Regions, you can have multiple delivery streams with the same name.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-DeliveryStreamType <DeliveryStreamType>
The delivery stream type. This parameter can be one of the following values:
DirectPut: Provider applications access the delivery stream directly.KinesisStreamAsSource: The delivery stream uses a Kinesis data stream as a source.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_DomainARN <String>
The ARN of the Amazon ES domain. The IAM role must have permissions for
DescribeElasticsearchDomain, DescribeElasticsearchDomains, and DescribeElasticsearchDomainConfig after assuming the role specified in RoleARN. For more information, see Amazon Resource Names (ARNs) and AWS Service Namespaces. | Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_IndexName <String>
The Elasticsearch index name.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_IndexRotationPeriod <ElasticsearchIndexRotationPeriod>
The Elasticsearch index rotation period. Index rotation appends a timestamp to the
IndexName to facilitate the expiration of old data. For more information, see Index Rotation for the Amazon ES Destination. The default value is OneDay. | Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_RoleARN <String>
The Amazon Resource Name (ARN) of the IAM role to be assumed by Kinesis Data Firehose for calling the Amazon ES Configuration API and for indexing documents. For more information, see Grant Kinesis Data Firehose Access to an Amazon S3 Destination and Amazon Resource Names (ARNs) and AWS Service Namespaces.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_S3BackupMode <ElasticsearchS3BackupMode>
Defines how documents should be delivered to Amazon S3. When it is set to
FailedDocumentsOnly, Kinesis Data Firehose writes any documents that could not be indexed to the configured Amazon S3 destination, with elasticsearch-failed/ appended to the key prefix. When set to AllDocuments, Kinesis Data Firehose delivers all incoming records to Amazon S3, and also writes failed documents with elasticsearch-failed/ appended to the prefix. For more information, see Amazon S3 Backup for the Amazon ES Destination. Default value is FailedDocumentsOnly. | Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_S3Configuration <S3DestinationConfiguration>
The configuration for the backup Amazon S3 location.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ElasticsearchDestinationConfiguration_TypeName <String>
The Elasticsearch type name. For Elasticsearch 6.x, there can be only one type per index. If you try to specify a new type for an existing index that already has another type, Kinesis Data Firehose returns an error during run time.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ExtendedS3DestinationConfiguration <ExtendedS3DestinationConfiguration>
The destination in Amazon S3. You can specify only one destination.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-Force <SwitchParameter>
This parameter overrides confirmation prompts to force the cmdlet to continue its operation. This parameter should always be used with caution.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-KinesisStreamSourceConfiguration_KinesisStreamARN <String>
The ARN of the source Kinesis data stream. For more information, see Amazon Kinesis Data Streams ARN Format.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-KinesisStreamSourceConfiguration_RoleARN <String>
The ARN of the role that provides access to the source Kinesis data stream. For more information, see AWS Identity and Access Management (IAM) ARN Format.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ProcessingConfiguration_Enabled <Boolean>
Enables or disables data processing.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_ProcessingConfiguration_Enabled |
-ProcessingConfiguration_Processor <Processor[]>
The data processors.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_ProcessingConfiguration_Processors |
-RedshiftDestinationConfiguration <RedshiftDestinationConfiguration>
The destination in Amazon Redshift. You can specify only one destination.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-RetryOptions_DurationInSecond <Int32>
After an initial failure to deliver to Amazon ES, the total amount of time during which Kinesis Data Firehose retries delivery (including the first attempt). After this time has elapsed, the failed documents are written to Amazon S3. Default value is 300 seconds (5 minutes). A value of 0 (zero) results in no retries.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | ElasticsearchDestinationConfiguration_RetryOptions_DurationInSeconds |
-S3DestinationConfiguration <S3DestinationConfiguration>
[Deprecated] The destination in Amazon S3. You can specify only one destination.This parameter is deprecated.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-SplunkDestinationConfiguration <SplunkDestinationConfiguration>
The destination in Splunk. You can specify only one destination.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-Tag <Tag[]>
A set of tags to assign to the delivery stream. A tag is a key-value pair that you can define and assign to AWS resources. Tags are metadata. For example, you can add friendly names and descriptions or other types of information that can help you distinguish the delivery stream. For more information about tags, see Using Cost Allocation Tags in the AWS Billing and Cost Management User Guide.You can specify up to 50 tags when creating a delivery stream.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
| Aliases | Tags |
Common Credential and Region Parameters
-AccessKey <String>
The AWS access key for the user account. This can be a temporary access key
if the corresponding session token is supplied to the -SessionToken parameter.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-Credential <AWSCredentials>
An AWSCredentials object instance containing access and secret key information,
and optionally a token for session-based credentials.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ProfileLocation <String>
Used to specify the name and location of the ini-format credential file (shared with the AWS CLI and other AWS SDKs)
If this optional parameter is omitted this cmdlet will search the encrypted credential file used by the AWS SDK for .NET and AWS Toolkit for Visual Studio first. If the profile is not found then the cmdlet will search in the ini-format credential file at the default location: (user's home directory)\.aws\credentials. Note that the encrypted credential file is not supported on all platforms. It will be skipped when searching for profiles on Windows Nano Server, Mac, and Linux platforms.
If this parameter is specified then this cmdlet will only search the ini-format credential file at the location given.
As the current folder can vary in a shell or during script execution it is advised that you use specify a fully qualified path instead of a relative path.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-ProfileName <String>
The user-defined name of an AWS credentials or SAML-based role profile containing
credential information. The profile is expected to be found in the secure credential
file shared with the AWS SDK for .NET and AWS Toolkit for Visual Studio. You can also
specify the name of a profile stored in the .ini-format credential file used with
the AWS CLI and other AWS SDKs.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-NetworkCredential <PSCredential>
Used with SAML-based authentication when ProfileName references a SAML role profile.
Contains the network credentials to be supplied during authentication with the
configured identity provider's endpoint. This parameter is not required if the
user's default network identity can or should be used during authentication.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-SecretKey <String>
The AWS secret key for the user account. This can be a temporary secret key
if the corresponding session token is supplied to the -SessionToken parameter.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-SessionToken <String>
The session token if the access and secret keys are temporary session-based
credentials.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-Region <String>
The system name of the AWS region in which the operation should be invoked. For example, us-east-1, eu-west-1 etc.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
-EndpointUrl <String>
The endpoint to make the call against.
Note: This parameter is primarily for internal AWS use and is not required/should not be specified for normal usage. The cmdlets normally determine which endpoint to call based on the region specified to the -Region parameter or set as default in the shell (via Set-DefaultAWSRegion). Only specify this parameter if you must direct the call to a specific custom endpoint.
| Required? | False |
| Position? | Named |
| Accept pipeline input? | False |
Inputs
This cmdlet does not accept pipeline input.
Outputs
This cmdlet returns a System.String object.
The service call response (type Amazon.KinesisFirehose.Model.CreateDeliveryStreamResponse) can also be referenced from properties attached to the cmdlet entry in the $AWSHistory stack.
Related Links
Supported Version
AWS Tools for PowerShell: 2.x.y.z
