Broadcasting to Amazon IVS: Mixer Guide - Amazon Interactive Video Service

Broadcasting to Amazon IVS: Mixer Guide

The mixer is an audio and video processing unit that takes multiple input sources and generates a single output. It is a powerful feature that lets you define and manage multiple on-screen (video) elements and audio tracks. You can combine video and audio from multiple sources such as cameras, microphones, screen captures, and audio and video generated by your app. You can use transitions to move these sources around the video that you stream to Amazon IVS and add and to remove them mid-stream.

To access the mixer, call:

BroadcastSession.getMixer() on Android

IVSBroadcastSession.mixer on iOS

Terminology


                IVS broadcasting mixer terminology.
Term Description

Binding

To associate an input device with a slot, the device must be bound to the mixer slot. This is done with the Mixer.bind() method. A slot can have one image input and one audio input bound to it at a time. You can unbind a device from the slot by calling Mixer.unbind().

Canvas

The display extent of the video defined in your BroadcastSession configuration. The canvas is equal in size to your video settings and runs at the same frame rate as specified in your configuration.

Device

A hardware or software component that produces audio or image input to the BroadcastSession. Examples of devices are microphones, cameras, Bluetooth headsets, and virtual devices such as screen captures or custom-image inputs. With the exception of custom inputs, you generally do not need to keep a reference to the device object; instead, you can keep a copy of the device descriptor.

Device descriptor

A structure with information about an input device; e.g., its type, system address, a human-readable "friendly" name, and physical position on the mobile device. This information lets you decide if you want to use the referenced device and lets Amazon IVS access it.

Slot

A container that defines a visual element’s position on screen and an audio track’s properties in the audio mix. A mixer can be configured with zero or more slots. Slots are given a string name that can be used to bind devices and execute transitions. The image above shows four slots:

  • Bottom left with camera input

  • Top right with movie input

  • Bottom right with the Amazon IVS logo

  • A full-screen background image

After configuring a session, you can add and remove slots with the addSlot and removeSlot mixer methods.

Transition

To move a slot to a new position or change some of its properties, use Mixer.transition(). This method takes:

  • A new slot structure that represents the next state for the slot

  • A duration that specifies how long the animation should take, relative to the timeline of the video. If the duration is set to 0, the transition happens on the next frame that is mixed.

  • An optional callback that informs you when the animation is complete. The callback may be useful for chaining animations.

Canvas Properties

Canvas properties are set based on the BroadcastConfiguration you provide when creating the BroadcastSession. Several properties in the Audio and Video structures affect the canvas:

Name Type Description

Audio.channels

Integer

Number of output channels from the audio mixer. Valid values: 1, 2. 1 channel is mono audio; 2, stereo audio. Default: 2.

Audio.sampleRate

AudioSampleRate

Number of audio samples per second from the audio mixer. This value should be at least twice the highest frequency in your audio signal. People can hear up to about 20 kHz, so 44.1 kHz and 48 kHz generally suffice. Default: 48 kHz.

Video.defaultAspectMode

AspectMode

Default aspect-ratio mode for slots. Valid values:

  • Fill — Maintain the aspect ratio of the image but fill the slot. The image will be cropped if needed.

  • Fit — Maintain the aspect ratio of the image but fit the entire image into the slot. The slot may have a letterbox or pillarbox if necessary. The letter/pillarbox will be of the fillColor if that value was set; otherwise, transparent (which may appear black if the canvas color behind the image is black).

  • None — Do not maintain the aspect ratio of the image. The image will be scaled to match the dimensions of the slot.

Video.size

Vec2

Size of the video canvas.

Video.targetFramerate

Integer

Number of target frames per second for the canvas. On average this value should be met, but the system may drop frames under certain circumstances (e.g., high CPU load or network congestion).

Slot Properties

Slots have several configurable properties that you can use to customize your scenes and animate. Any value that is Float or Vector is animated using linear interpolation for transitions with a duration longer than 0 seconds.

Name Type Description

aspect

AspectMode

Aspect-ratio mode for any image rendered in the slot. Valid values:

  • Fill — Maintain the aspect ratio of the image but fill the slot. The image will be cropped if needed.

  • Fit — Maintain the aspect ratio of the image but fit the entire image into the slot. The slot may have a letterbox or pillarbox if necessary. The letter/pillarbox will be of the fillColor if that value was set; otherwise, transparent (which may appear black if the canvas color behind the image is black).

  • None — Do not maintain the aspect ratio of the image. The image will be scaled to match the dimensions of the slot.

Default: same as the Canvas aspect if matchCanvasAspectMode is true; otherwise, Fill. Setting this value also sets matchCanvasAspectMode to false.

fillColor

Vec4

Fill color to be used with Aspect Fit when the slot and image aspect ratios do not match. The format is (red, green, blue, alpha). Valid value (for each channel): 0 - 1. Default: (0, 0, 0, 0).

gain

Float

Audio gain. This is a multiplier, so any value above 1 increases the gain; any value below 1, decreases it. Valid values: 0 - 2. Default: 1.

matchCanvasAspectMode

Boolean

If true, use the canvas’s Video.defaultAspectMode value. This is set to false if you set the slot’s aspect property. Default: true.

matchCanvasSize

Boolean

If true, the slot’s size is adjusted to be equal to the size of the canvas and its position is set to (0, 0). This is set to false if you set the slot’s size property. Default: true.

name

String

Name of the slot. This is used to reference the slot for bindings and transitions. Default: "default".

position

Vec2

Slot position (in pixels), relative to the top-left corner of the canvas. The origin of the slot also is top-left.

preferredAudioInput

DeviceType

Preferred audio-input-device type. If this slot is unbound and an audio device of the specified type is attached to the session, the device automatically binds to this slot. Valid values:

  • Microphone — Audio hardware such as on-board microphone, pluggable headphones, or Bluetooth headphones.

  • System Audio — Audio captured from the operating system, usually accompanied by a screen recording.

  • User Audio — Custom audio inputs that you create.

  • Unknown — There is no preferred device; the slot will always be bound manually.

preferredVideoInput

DeviceType

Preferred video-input-device. If this slot is unbound and a video device of the specified type is attached to the session, the device automatically binds to this slot. Valid values:

  • Camera — On-board camera devices such as the front-facing, back-facing, or wide-angle camera.

  • Screen — Screen capture from the operating system.

  • User Image — Custom image and video inputs that you create.

  • Unknown — There is no preferred device; the slot will always be bound manually.

size

Vec2

Size of the slot, in pixels. Setting this value also sets matchCanvasSize to false. Default: (0, 0); however, because matchCanvasSize defaults to true, the rendered size of the slot is the canvas size, not (0, 0).

transparency

Float

Transparency of the slot. This is multiplicative with any alpha values in the image. Opacity is 1 - transparency. Valid values: 0-1, where 0 is fully opaque and 1 is fully transparent. Default: 0.

zIndex

Float

Relative ordering of slots. Slots with higher zIndex values are drawn on top of slots with lower zIndex values.

Configuring a Broadcast Session for Mixing


                Configuring a broadcast session for mixing.

Here, we create a scene similar to the one at the beginning of this guide, with three on-screen elements:

  • Bottom-left slot for a camera.

  • Bottom-right slot for a logo overlay.

  • Top-right slot for a movie.

Note that the origin for the canvas is the top-left corner and this is the same for the slots. Hence, positioning a slot at (0, 0) puts it in the top-left corner with the entire slot visible.

iOS

let config = IVSBroadcastConfiguration() try config.video.setSize(CGSize(width: 1280, height: 720)) try config.video.setTargetFramerate(60) config.video.enableTransparency = true // Bottom Left var cameraSlot = IVSMixerSlotConfiguration() cameraSlot.size = CGSize(width: 320, height: 180) cameraSlot.position = CGPoint(x: 20, y: 1280 - 200) cameraSlot.preferredVideoInput = .camera cameraSlot.preferredAudioInput = .microphone cameraSlot.matchCanvasAspectMode = false cameraSlot.zIndex = 2 try cameraSlot.setName("camera") // Top Right var streamSlot = IVSMixerSlotConfiguration() streamSlot.size = CGSize(width: 640, height: 320) streamSlot.position = CGPoint(x: 1280 - 660, y: 20) streamSlot.preferredVideoInput = .userImage streamSlot.preferredAudioInput = .userAudio streamSlot.matchCanvasAspectMode = false streamSlot.zIndex = 1 try streamSlot.setName("stream") // Bottom Right var logoSlot = IVSMixerSlotConfiguration() logoSlot.size = CGSize(width: 320, height: 180) logoSlot.position = CGPoint(x: 1280 - 340, y: 720 - 200) logoSlot.preferredVideoInput = .userImage logoSlot.preferredAudioInput = .unknown logoSlot.matchCanvasAspectMode = false logoSlot.zIndex = 3 try logoSlot.setTransparency(0.7) try logoSlot.setName("logo") config.mixer.slots = [ cameraSlot, streamSlot, logoSlot ]

Android

// Bottom Left val cameraSlot = BroadcastConfiguration.Mixer.Slot.with { s -> s.setSize(320, 180) s.position = BroadcastConfiguration.Vec2(20, 1280 - 200) s.preferredVideoInput = Device.Descriptor.DeviceType.CAMERA s.preferredAudioInput = Device.Descriptor.DeviceType.MICROPHONE s.matchCanvasAspectMode = false s.zIndex = 2 s.name = "camera" s } // Top Right val streamSlot = BroadcastConfiguration.Mixer.Slot.with { s -> s.setSize(640, 320) s.position = BroadcastConfiguration.Vec2(1280 - 660, 20) s.preferredVideoInput = Device.Descriptor.DeviceType.USER_IMAGE s.preferredAudioInput = Device.Descriptor.DeviceType.USER_AUDIO s.matchCanvasAspectMode = false s.zIndex = 1 s.name = "stream" s } // Bottom Right val logoSlot = BroadcastConfiguration.Mixer.Slot.with { s -> s.setSize(320, 180) s.position = BroadcastConfiguration.Vec2(1280 - 340, 720 - 200) s.preferredVideoInput = Device.Descriptor.DeviceType.USER_IMAGE s.preferredAudioInput = Device.Descriptor.DeviceType.UNKNOWN s.matchCanvasAspectMode = false s.zIndex = 3 s.name = "logo" s.transparency = 0.7 s } val config = BroadcastConfiguration.with { c -> c.mixer.slots = listOf(cameraSlot, streamSlot, logoSlot) c.video.targetFramerate = 60 c.video.setSize(1280, 720) c }

Adding Slots

Once you’ve created a BroadcastSession with your configuration, you can add slots to and remove slots from the mixer. Here, we add to the mixer a large background slot for an image.

iOS

// Background. We will use most of the defaults for this slot. var backgroundSlot = IVSMixerSlotConfiguration() backgroundSlot.preferredVideoInput = .userImage backgroundSlot.preferredAudioInput = .unknown backgroundSlot.matchCanvasAspectMode = false try backgroundSlot.setName("background") session.mixer.addSlot(backgroundSlot)

Android

// Background. We will use most of the defaults for this slot. val backgroundSlot = BroadcastConfiguration.Mixer.Slot.with { s -> s.preferredVideoInput = Device.Descriptor.DeviceType.USER_IMAGE s.preferredAudioInput = Device.Descriptor.DeviceType.UNKNOWN s.matchCanvasAspectMode = false s.name = "background" s } session.mixer.addSlot(backgroundSlot)

Removing Slots

To remove a slot, call BroadcastSession.Mixer.removeSlot with the name of the slot you want to remove. Any devices bound to that slot are automatically unbound, so you must rebind them to different slots if you want to continue using them.

Animations with Transitions

The mixer transition method replaces a slot’s configuration with a new configuration. This replacement can be animated over time by setting a duration higher than 0, in seconds.

Which Properties Can Be Animated?

Not all properties in the slot structure can be animated. Any properties based on Float types can be animated; other properties take effect at either the start or the end of the animation.

Name Can Be Animated? Impact Point

aspect

No

End

fillColor

Yes

Interpolated

gain

Yes

Interpolated

matchCanvasAspectMode

No

Start

matchCanvasSize

No

Start

name

Note: You cannot change the name of the slot.

No

N/A

position

Yes

Interpolated

preferredAudioInput

No

End

preferredVideoInput

No

End

size

Yes

Interpolated

transparency

Yes

Interpolated

zIndex

Note: The zIndex moves 2D planes through 3D space, so the transition happens when the two planes cross at some point in the middle of the animation. This could be computed, but it depends on the starting and ending zIndex values. For a smoother transition, combine this with transparency.

Yes

Unknown

Simple Examples

Below are examples of a full-screen camera takeover using the configuration defined above in Configuring a Broadcast Session for Mixing. This is animated over 0.5 seconds.

iOS

// Bottom Left var bigCameraSlot = cameraSlot bigCameraSlot.size = CGSize(width: 1280, height: 720) bigCameraSlot.position = CGPoint(x: 0, y: 0) session.mixer.transition("camera", bigCameraSlot, 0.5) { println("animation completed!") }

Android

// Bottom Left val bigCameraSlot = cameraSlot.changing { s -> s.setSize(1280, 720) s.position = BroadcastConfiguration.Vec2(0, 0) s } session.mixer.transition("camera", bigCameraSlot, 0.5) { print("animation completed!") }