Publishing and Subscribing

Concepts

Three core concepts underlie real-time functionality: stage, strategy, and renderer. The design goal is minimizing the amount of client-side logic necessary to build a working product.

Stage

The IVSStage class is the main point of interaction between the host application and the SDK. The class represents the stage itself and is used to join and leave the stage. Creating or joining a stage requires a valid, unexpired token string from the control plane (represented as token). Joining and leaving a stage are simple.

let stage = try IVSStage(token: token, strategy: self)

try stage.join()

stage.leave()

The IVSStage class also is where the IVSStageRenderer and IVSErrorDelegate can be attached:

let stage = try IVSStage(token: token, strategy: self)
stage.errorDelegate = self
stage.addRenderer(self) // multiple renderers can be added

Strategy

The IVSStageStrategy protocol provides a way for the host application to communicate the desired state of the stage to the SDK. Three functions need to be implemented: shouldSubscribeToParticipant, shouldPublishParticipant, and streamsToPublishForParticipant. All are discussed below.

Subscribing to Participants

func stage(_ stage: IVSStage, shouldSubscribeToParticipant participant: IVSParticipantInfo) -> IVSStageSubscribeType

When a remote participant joins a stage, the SDK queries the host application about the desired subscription state for that participant. The options are .none, .audioOnly, and .audioVideo. When returning a value for this function, the host application does not need to worry about the publish state, current subscription state, or stage connection state. If .audioVideo is returned, the SDK waits until the remote participant is publishing before subscribing, and it updates the host application through the renderer throughout the process.

Here is a sample implementation:

func stage(_ stage: IVSStage, shouldSubscribeToParticipant participant: IVSParticipantInfo) -> IVSStageSubscribeType {
    return .audioVideo
}

This is the complete implementation of this function for a host application that always wants all participants to see each other; e.g., a video-chat application.

More advanced implementations also are possible. Use the attributes property on IVSParticipantInfo to selectively subscribe to participants based on server-provided attributes:

func stage(_ stage: IVSStage, shouldSubscribeToParticipant participant: IVSParticipantInfo) -> IVSStageSubscribeType {
    switch participant.attributes["role"] {
    case "moderator": return .none
    case "guest": return .audioVideo
    default: return .none
    }
}

This can be used to create a stage where moderators can monitor all guests without being seen or heard themselves. The host application could use additional business logic to let moderators see each other but remain invisible to guests.

Publishing

func stage(_ stage: IVSStage, shouldPublishParticipant participant: IVSParticipantInfo) -> Bool

Once connected to the stage, the SDK queries the host application to see if a particular participant should publish. This is invoked only on local participants that have permission to publish based on the provided token.

Here is a sample implementation:

func stage(_ stage: IVSStage, shouldPublishParticipant participant: IVSParticipantInfo) -> Bool {
    return true
}

This is for a standard video chat application where users always want to publish. They can mute and unmute their audio and video, to instantly be hidden or seen/heard. (They also can use publish/unpublish, but that is much slower. Mute/unmute is preferable for use cases where changing visibility often is desirable.)

Choosing Streams to Publish

func stage(_ stage: IVSStage, streamsToPublishForParticipant participant: IVSParticipantInfo) -> [IVSLocalStageStream]

When publishing, this is used to determine what audio and video streams should be published. This is covered in more detail later in Publish a Media Stream.

Updating the Strategy

The strategy is intended to be dynamic: the values returned from any of the above functions can be changed at any time. For example, if the host application does not want to publish until the end user taps a button, you could return a variable from shouldPublishParticipant (something like hasUserTappedPublishButton). When that variable changes based on an interaction by the end user, call stage.refreshStrategy() to signal to the SDK that it should query the strategy for the latest values, applying only things that have changed. If the SDK observes that the shouldPublishParticipant value has changed, it will start the publish process. If the SDK queries and all functions return the same value as before, the refreshStrategy call will not make any modifications to the stage.

If the return value of shouldSubscribeToParticipant changes from .audioVideo to .audioOnly, the video stream will be removed for all participants with changed returned values, if a video stream existed previously.

Generally, the stage uses the strategy to most efficiently apply the difference between the previous and current strategies, without the host application needing to worry about all the state required to manage it properly. Because of this, think of calling stage.refreshStrategy() as a cheap operation, because it does nothing unless the strategy changes.

Renderer

The IVSStageRenderer protocol communicates the state of the stage to the host application. Updates to the host application’s UI usually can be powered entirely by the events provided by the renderer. The renderer provides the following functions:

func stage(_ stage: IVSStage, participantDidJoin participant: IVSParticipantInfo)

func stage(_ stage: IVSStage, participantDidLeave participant: IVSParticipantInfo)

func stage(_ stage: IVSStage, participant: IVSParticipantInfo, didChange publishState: IVSParticipantPublishState)

func stage(_ stage: IVSStage, participant: IVSParticipantInfo, didChange subscribeState: IVSParticipantSubscribeState)

func stage(_ stage: IVSStage, participant: IVSParticipantInfo, didAdd streams: [IVSStageStream])

func stage(_ stage: IVSStage, participant: IVSParticipantInfo, didRemove streams: [IVSStageStream])

func stage(_ stage: IVSStage, participant: IVSParticipantInfo, didChangeMutedStreams streams: [IVSStageStream])

func stage(_ stage: IVSStage, didChange connectionState: IVSStageConnectionState, withError error: Error?)

It is not expected that the information provided by the renderer impacts the return values of the strategy. For example, the return value of shouldSubscribeToParticipant is not expected to change when participant:didChangePublishState is called. If the host application wants to subscribe to a particular participant, it should return the desired subscription type regardless of that participant’s publish state. The SDK is responsible for ensuring that the desired state of the strategy is acted on at the correct time based on the state of the stage.

Note that only publishing participants trigger participantDidJoin, and whenever a participant stops publishing or leaves the stage session, participantDidLeave is triggered.

Publish a Media Stream

Local devices such as built-in microphones and cameras are discovered via IVSDeviceDiscovery. Here is an example of selecting the front-facing camera and default microphone, then returning them as IVSLocalStageStreams to be published by the SDK:

let devices = IVSDeviceDiscovery().listLocalDevices()

// Find the camera virtual device, choose the front source, and create a stream
let camera = devices.compactMap({ $0 as? IVSCamera }).first!
let frontSource = camera.listAvailableInputSources().first(where: { $0.position == .front })!
camera.setPreferredInputSource(frontSource)
let cameraStream = IVSLocalStageStream(device: camera)

// Find the microphone virtual device and create a stream
let microphone = devices.compactMap({ $0 as? IVSMicrophone }).first!
let microphoneStream = IVSLocalStageStream(device: microphone)

// Configure the audio manager to use the videoChat preset, which is optimized for bi-directional communication, including echo cancellation.
IVSStageAudioManager.sharedInstance().setPreset(.videoChat)

// This is a function on IVSStageStrategy
func stage(_ stage: IVSStage, streamsToPublishForParticipant participant: IVSParticipantInfo) -> [IVSLocalStageStream] {
    return [cameraStream, microphoneStream]
}

Display and Remove Participants

After subscribing is completed, you will receive an array of IVSStageStream objects through the renderer’s didAddStreams function. To preview or receive audio level stats about this participant, you can access the underlying IVSDevice object from the stream:

if let imageDevice = stream.device as? IVSImageDevice {
    let preview = imageDevice.previewView()
    /* attach this UIView subclass to your view */
} else if let audioDevice = stream.device as? IVSAudioDevice {
    audioDevice.setStatsCallback( { stats in
        /* process stats.peak and stats.rms */
    })
}

When a participant stops publishing or is unsubscribed from, the didRemoveStreams function is called with the streams that were removed. Host applications should use this as a signal to remove the participant’s video stream from the view hierarchy.

didRemoveStreams is invoked for all scenarios in which a stream might be removed, including:

• The remote participant stops publishing.

• A local device unsubscribes or changes subscription from .audioVideo to .audioOnly.

• The remote participant leaves the stage.

• The local participant leaves the stage.

Because didRemoveStreams is invoked for all scenarios, no custom business logic is required around removing participants from the UI during remote or local leave operations.

Mute and Unmute Media Streams

IVSLocalStageStream objects have a setMuted function that controls whether the stream is muted. This function can be called on the stream before or after it is returned from the streamsToPublishForParticipant strategy function.

Important: If a new IVSLocalStageStream object instance is returned by streamsToPublishForParticipant after a call to refreshStrategy, the mute state of the new stream object is applied to the stage. Be careful when creating new IVSLocalStageStream instances to make sure the expected mute state is maintained.

Monitor Remote Participant Media Mute State

When a participant changes the mute state of its video or audio stream, the renderer didChangeMutedStreams function is invoked with an array of streams that have changed. Use the isMuted property on IVSStageStream to update your UI accordingly:

func stage(_ stage: IVSStage, participant: IVSParticipantInfo, didChangeMutedStreams streams: [IVSStageStream]) {
    streams.forEach { stream in 
        /* stream.isMuted */
    }
}

Create a Stage Configuration

To customize the values of a stage’s video configuration, use IVSLocalStageStreamVideoConfiguration:

let config = IVSLocalStageStreamVideoConfiguration()
try config.setMaxBitrate(900_000)
try config.setMinBitrate(100_000)
try config.setTargetFramerate(30)
try config.setSize(CGSize(width: 360, height: 640))
config.degradationPreference = .balanced

Get WebRTC Statistics

To get the latest WebRTC statistics for a publishing stream or a subscribing stream, use requestRTCStats on IVSStageStream. When a collection is completed, you will receive statistics through the IVSStageStreamDelegate which can be set on IVSStageStream. To continually collect WebRTC statistics, call this function on a Timer.

func stream(_ stream: IVSStageStream, didGenerateRTCStats stats: [String : [String : String]]) {
    for stat in stats {
      for member in stat.value {
         print("stat \(stat.key) has member \(member.key) with value \(member.value)")
      }
   }
}

Get Participant Attributes

If you specify attributes in the CreateParticipantToken endpoint request, you can see the attributes in IVSParticipantInfo properties:

func stage(_ stage: IVSStage, participantDidJoin participant: IVSParticipantInfo) {
    print("ID: \(participant.participantId)")
    for attribute in participant.attributes {
        print("attribute: \(attribute.key)=\(attribute.value)")
    }
}

Continue Session in the Background

When the app enters the background, you can continue to be in the stage while hearing remote audio, though it is not possible to continue to send your own image and audio. You will need to update your IVSStrategy implementation to stop publishing and subscribe to .audioOnly (or .none, if applicable):

func stage(_ stage: IVSStage, shouldPublishParticipant participant: IVSParticipantInfo) -> Bool {
    return false
}
func stage(_ stage: IVSStage, shouldSubscribeToParticipant participant: IVSParticipantInfo) -> IVSStageSubscribeType {
    return .audioOnly
}

Then make a call to stage.refreshStrategy().

Enable/Disable Layered Encoding with Simulcast

When publishing a media stream, the SDK transmits high-quality and low-quality video streams, so remote participants can subscribe to the stream even if they have limited downlink bandwidth. Layered encoding with simulcast is on by default. You can disable it with IVSSimulcastConfiguration:

// Disable Simulcast
let config = IVSLocalStageStreamVideoConfiguration()
config.simulcast.enabled = false

let cameraStream = IVSLocalStageStream(device: camera, configuration: config)

// Other Stage implementation code

Broadcast the Stage to an IVS Channel

To broadcast a stage, create a separate IVSBroadcastSession and then follow the usual instructions for broadcasting with the SDK, described above. The device property on IVSStageStream will be either an IVSImageDevice or IVSAudioDevice as shown in the snippet above; these can be connected to the IVSBroadcastSession.mixer to broadcast the entire stage in a customizable layout.

Optionally, you can composite a stage and broadcast it to an IVS low-latency channel, to reach a larger audience. See Enabling Multiple Hosts on an Amazon IVS Stream in the IVS Low-Latency Streaming User Guide.