IVS Broadcast SDK: Mixer Guide (Low-Latency Streaming)

The mixer is an audio and video processing unit that takes multiple input sources and generates a single output. It is a powerful feature that lets you define and manage multiple on-screen (video) elements and audio tracks. You can combine video and audio from multiple sources such as cameras, microphones, screen captures, and audio and video generated by your app. You can use transitions to move these sources around the video that you stream to Amazon IVS, and add to and remove them mid-stream.

To access the mixer, call:

BroadcastSession.getMixer() on Android

IVSBroadcastSession.mixer on iOS

Terminology

Term	Description
Binding	To associate an input device with a slot, the device must be bound to the mixer slot. This is done with the Mixer.bind() method. A slot can have one image input and one audio input bound to it at a time. You can unbind a device from the slot by calling Mixer.unbind().
Canvas	The display extent of the video defined in your BroadcastSession configuration. The canvas is equal in size to your video settings and runs at the same frame rate as specified in your configuration.
Device	A hardware or software component that produces audio or image input to the BroadcastSession. Examples of devices are microphones, cameras, Bluetooth headsets, and virtual devices such as screen captures or custom-image inputs. With the exception of custom inputs, you generally do not need to keep a reference to the device object; instead, you can keep a copy of the device descriptor.
Device descriptor	A structure with information about an input device; e.g., its type, system address, a human-readable "friendly" name, and physical position on the mobile device. This information lets you decide if you want to use the referenced device and lets Amazon IVS access it.
Slot	A container that defines a visual element’s position on screen and an audio track’s properties in the audio mix. A mixer can be configured with zero or more slots. Slots are given a string name that can be used to bind devices and execute transitions. The image above shows four slots: Bottom left with camera input Top right with movie input Bottom right with the Amazon IVS logo A full-screen background image After configuring a session, you can add and remove slots with the addSlot and removeSlot mixer methods.
Transition	To move a slot to a new position or change some of its properties, use Mixer.transition(). This method takes: A new slot structure that represents the next state for the slot A duration that specifies how long the animation should take, relative to the timeline of the video. If the duration is set to 0, the transition happens on the next frame that is mixed. An optional callback that informs you when the animation is complete. The callback may be useful for chaining animations.

Canvas Properties

Canvas properties are set based on the BroadcastConfiguration you provide when creating the BroadcastSession. Several properties in the Audio and Video structures affect the canvas:

Name	Type	Description
Audio.channels	Integer	Number of output channels from the audio mixer. Valid values: 1, 2. 1 channel is mono audio; 2, stereo audio. Default: 2.
Audio.sampleRate	AudioSampleRate	Number of audio samples per second from the audio mixer. This value should be at least twice the highest frequency in your audio signal. People can hear up to about 20 kHz, so 44.1 kHz and 48 kHz generally suffice. Default: 48 kHz.
Video.defaultAspectMode	AspectMode	Default aspect-ratio mode for slots. Valid values: Fill — Maintain the aspect ratio of the image but fill the slot. The image will be cropped if needed. Fit — Maintain the aspect ratio of the image but fit the entire image into the slot. The slot may have a letterbox or pillarbox if necessary. The letter/pillarbox will be of the fillColor if that value was set; otherwise, transparent (which may appear black if the canvas color behind the image is black). None — Do not maintain the aspect ratio of the image. The image will be scaled to match the dimensions of the slot.
Video.size	Vec2	Size of the video canvas.
Video.targetFramerate	Integer	Number of target frames per second for the canvas. On average this value should be met, but the system may drop frames under certain circumstances (e.g., high CPU load or network congestion).

Slot Properties

Slots have several configurable properties that you can use to customize your scenes and animate. Any value that is Float or Vector is animated using linear interpolation for transitions with a duration longer than 0 seconds.

Name	Type	Description
aspect	AspectMode	Aspect-ratio mode for any image rendered in the slot. Valid values: Fill — Maintain the aspect ratio of the image but fill the slot. The image will be cropped if needed. Fit — Maintain the aspect ratio of the image but fit the entire image into the slot. The slot may have a letterbox or pillarbox if necessary. The letter/pillarbox will be of the fillColor if that value was set; otherwise, transparent (which may appear black if the canvas color behind the image is black). None — Do not maintain the aspect ratio of the image. The image will be scaled to match the dimensions of the slot. Default: same as the Canvas aspect if matchCanvasAspectMode is true; otherwise, Fill. Setting this value also sets matchCanvasAspectMode to false.
fillColor	Vec4	Fill color to be used with Aspect Fit when the slot and image aspect ratios do not match. The format is (red, green, blue, alpha). Valid value (for each channel): 0 - 1. Default: (0, 0, 0, 0).
gain	Float	Audio gain. This is a multiplier, so any value above 1 increases the gain; any value below 1, decreases it. Valid values: 0 - 2. Default: 1.
matchCanvasAspectMode	Boolean	If true, use the canvas’s Video.defaultAspectMode value. This is set to false if you set the slot’s aspect property. Default: true.
matchCanvasSize	Boolean	If true, the slot’s size is adjusted to be equal to the size of the canvas and its position is set to (0, 0). This is set to false if you set the slot’s size property. Default: true.
name	String	Name of the slot. This is used to reference the slot for bindings and transitions. Default: "default".
position	Vec2	Slot position (in pixels), relative to the top-left corner of the canvas. The origin of the slot also is top-left.
preferredAudioInput	DeviceType	Preferred audio-input-device type. If this slot is unbound and an audio device of the specified type is attached to the session, the device automatically binds to this slot. Valid values: Microphone — Audio hardware such as on-board microphone, pluggable headphones, or Bluetooth headphones. System Audio — Audio captured from the operating system, usually accompanied by a screen recording. User Audio — Custom audio inputs that you create. Unknown — There is no preferred device; the slot will always be bound manually.
preferredVideoInput	DeviceType	Preferred video-input-device. If this slot is unbound and a video device of the specified type is attached to the session, the device automatically binds to this slot. Valid values: Camera — On-board camera devices such as the front-facing, back-facing, or wide-angle camera. Screen — Screen capture from the operating system. User Image — Custom image and video inputs that you create. Unknown — There is no preferred device; the slot will always be bound manually.
size	Vec2	Size of the slot, in pixels. Setting this value also sets matchCanvasSize to false. Default: (0, 0); however, because matchCanvasSize defaults to true, the rendered size of the slot is the canvas size, not (0, 0).
transparency	Float	Transparency of the slot. This is multiplicative with any alpha values in the image. Opacity is 1 - transparency. Valid values: 0-1, where 0 is fully opaque and 1 is fully transparent. Default: 0.
zIndex	Float	Relative ordering of slots. Slots with higher zIndex values are drawn on top of slots with lower zIndex values.

Configuring a Broadcast Session for Mixing

Here, we create a scene similar to the one at the beginning of this guide, with three on-screen elements:

• Bottom-left slot for a camera.

• Bottom-right slot for a logo overlay.

• Top-right slot for a movie.

Note that the origin for the canvas is the top-left corner and this is the same for the slots. Hence, positioning a slot at (0, 0) puts it in the top-left corner with the entire slot visible.

iOS

let config = IVSBroadcastConfiguration()
try config.video.setSize(CGSize(width: 1280, height: 720))
try config.video.setTargetFramerate(60)
config.video.enableTransparency = true

// Bottom Left
var cameraSlot = IVSMixerSlotConfiguration()
cameraSlot.size = CGSize(width: 320, height: 180) 
cameraSlot.position = CGPoint(x: 20, y: 1280 - 200)
cameraSlot.preferredVideoInput = .camera
cameraSlot.preferredAudioInput = .microphone
cameraSlot.matchCanvasAspectMode = false
cameraSlot.zIndex = 2
try cameraSlot.setName("camera")

// Top Right
var streamSlot = IVSMixerSlotConfiguration()
streamSlot.size = CGSize(width: 640, height: 320) 
streamSlot.position = CGPoint(x: 1280 - 660, y: 20)
streamSlot.preferredVideoInput = .userImage
streamSlot.preferredAudioInput = .userAudio
streamSlot.matchCanvasAspectMode = false
streamSlot.zIndex = 1
try streamSlot.setName("stream")

// Bottom Right
var logoSlot = IVSMixerSlotConfiguration()
logoSlot.size = CGSize(width: 320, height: 180) 
logoSlot.position = CGPoint(x: 1280 - 340, y: 720 - 200)
logoSlot.preferredVideoInput = .userImage
logoSlot.preferredAudioInput = .unknown
logoSlot.matchCanvasAspectMode = false
logoSlot.zIndex = 3
try logoSlot.setTransparency(0.7)
try logoSlot.setName("logo")

config.mixer.slots = [ cameraSlot, streamSlot, logoSlot ]

Android

// Bottom Left
val cameraSlot = BroadcastConfiguration.Mixer.Slot.with { s ->
    s.setSize(320, 180)
    s.position = BroadcastConfiguration.Vec2(20, 1280 - 200)
    s.preferredVideoInput = Device.Descriptor.DeviceType.CAMERA
    s.preferredAudioInput = Device.Descriptor.DeviceType.MICROPHONE
    s.matchCanvasAspectMode = false
    s.zIndex = 2
    s.name = "camera"
    s
}

// Top Right
val streamSlot = BroadcastConfiguration.Mixer.Slot.with { s ->
    s.setSize(640, 320)
    s.position = BroadcastConfiguration.Vec2(1280 - 660, 20)
    s.preferredVideoInput = Device.Descriptor.DeviceType.USER_IMAGE
    s.preferredAudioInput = Device.Descriptor.DeviceType.USER_AUDIO
    s.matchCanvasAspectMode = false
    s.zIndex = 1
    s.name = "stream"
    s
}

// Bottom Right
val logoSlot = BroadcastConfiguration.Mixer.Slot.with { s ->
    s.setSize(320, 180)
    s.position = BroadcastConfiguration.Vec2(1280 - 340, 720 - 200)
    s.preferredVideoInput = Device.Descriptor.DeviceType.USER_IMAGE
    s.preferredAudioInput = Device.Descriptor.DeviceType.UNKNOWN
    s.matchCanvasAspectMode = false
    s.zIndex = 3
    s.name = "logo"
    s.transparency = 0.7
    s
}

val config = BroadcastConfiguration.with { c ->
    c.mixer.slots = listOf(cameraSlot, streamSlot, logoSlot)
    c.video.targetFramerate = 60
    c.video.setSize(1280, 720)
    c
}

Adding Slots

Once you’ve created a BroadcastSession with your configuration, you can add slots to and remove slots from the mixer. Here, we add to the mixer a large background slot for an image.

iOS

// Background. We will use most of the defaults for this slot.
var backgroundSlot = IVSMixerSlotConfiguration()
backgroundSlot.preferredVideoInput = .userImage
backgroundSlot.preferredAudioInput = .unknown
backgroundSlot.matchCanvasAspectMode = false
try backgroundSlot.setName("background")

session.mixer.addSlot(backgroundSlot)

Android

// Background. We will use most of the defaults for this slot.
val backgroundSlot = BroadcastConfiguration.Mixer.Slot.with { s ->
    s.preferredVideoInput = Device.Descriptor.DeviceType.USER_IMAGE
    s.preferredAudioInput = Device.Descriptor.DeviceType.UNKNOWN
    s.matchCanvasAspectMode = false
    s.name = "background"
    s
}

session.mixer.addSlot(backgroundSlot)

Removing Slots

To remove a slot, call BroadcastSession.Mixer.removeSlot with the name of the slot you want to remove. Any devices bound to that slot are automatically unbound, so you must rebind them to different slots if you want to continue using them.

Animations with Transitions

The mixer transition method replaces a slot’s configuration with a new configuration. This replacement can be animated over time by setting a duration higher than 0, in seconds.

Which Properties Can Be Animated?

Not all properties in the slot structure can be animated. Any properties based on Float types can be animated; other properties take effect at either the start or the end of the animation.

Name	Can Be Animated?	Impact Point
aspect	No	End
fillColor	Yes	Interpolated
gain	Yes	Interpolated
matchCanvasAspectMode	No	Start
matchCanvasSize	No	Start
name Note: You cannot change the name of the slot.	No	N/A
position	Yes	Interpolated
preferredAudioInput	No	End
preferredVideoInput	No	End
size	Yes	Interpolated
transparency	Yes	Interpolated
zIndex Note: The zIndex moves 2D planes through 3D space, so the transition happens when the two planes cross at some point in the middle of the animation. This could be computed, but it depends on the starting and ending zIndex values. For a smoother transition, combine this with transparency.	Yes	Unknown

Simple Examples

Below are examples of a full-screen camera takeover using the configuration defined above in Configuring a Broadcast Session for Mixing. This is animated over 0.5 seconds.

iOS

// Bottom Left
var bigCameraSlot = cameraSlot
bigCameraSlot.size = CGSize(width: 1280, height: 720) 
bigCameraSlot.position = CGPoint(x: 0, y: 0)

session.mixer.transition("camera", bigCameraSlot, 0.5) {
    println("animation completed!")
}

Android

// Bottom Left
val bigCameraSlot = cameraSlot.changing { s ->
    s.setSize(1280, 720)
    s.position = BroadcastConfiguration.Vec2(0, 0)
    s
}

session.mixer.transition("camera", bigCameraSlot, 0.5) {
    print("animation completed!")
}

Mirroring the Broadcast

To mirror an attached image device in the broadcast in this direction …	Use a negative value for the …
Horizontally	Width of the slot
Vertically	Height of the slot
Both horizontally and vertically	Both the width and height of the slot

The position will need to be adjusted by the same value, to put the slot in the correct position when mirrored.

Below are examples for mirroring the broadcast horizontally and vertically.

iOS

Horizontal mirroring:

var cameraSlot = IVSMixerSlotConfiguration
cameraSlot.size = CGSize(width: -320, height: 720)
// Add 320 to position x since our width is -320
cameraSlot.position = CGPoint(x: 320, y: 0)

Vertical mirroring:

var cameraSlot = IVSMixerSlotConfiguration
cameraSlot.size = CGSize(width: 320, height: -720)
// Add 720 to position y since our height is -720
cameraSlot.position = CGPoint(x: 0, y: 720)

Android

Horizontal mirroring:

cameraSlot = BroadcastConfiguration.Mixer.Slot.with {
   it.size = BroadcastConfiguration.Vec2(-320f, 180f)
   // Add 320f to position x since our width is -320f
   it.position = BroadcastConfiguration.Vec2(320f, 0f)
   return@with it
}

Vertical mirroring:

cameraSlot = BroadcastConfiguration.Mixer.Slot.with {
   it.size = BroadcastConfiguration.Vec2(320f, -180f)
   // Add 180f to position y since our height is -180f
   it.position = BroadcastConfiguration.Vec2(0f, 180f)
   return@with it
}

Note: This mirroring is different than the setMirrored method on ImagePreviewView (Android) and IVSImagePreviewView (iOS). That method affects only the local preview view on the device and does not impact the broadcast.