Publishing and Subscribing

Concepts Publish a Media Stream Display and Remove Participants Mute and Unmute Media Streams Monitor Remote Participant Media Mute State Get WebRTC Statistics Get Participant Attributes Continue Session in the Background Enable/Disable Layered Encoding with Simulcast Video-Configuration Limitations Handling Network Issues Using Bluetooth Microphones

Three core concepts underlie real-time functionality: stage, strategy, and renderer. The design goal is minimizing the amount of client-side logic necessary to build a working product.

The Stage class is the main point of interaction between the host application and the SDK. It represents the stage itself and is used to join and leave the stage. Creating and joining a stage requires a valid, unexpired token string from the control plane (represented as token). Joining and leaving a stage are simple.


Stage stage = new Stage(context, token, strategy);

try {
	stage.join();
} catch (BroadcastException exception) {
	// handle join exception
}

stage.leave();

The Stage class is also where the StageRenderer can be attached:


stage.addRenderer(renderer); // multiple renderers can be added

The Stage.Strategy interface provides a way for the host application to communicate the desired state of the stage to the SDK. Three functions need to be implemented: shouldSubscribeToParticipant, shouldPublishFromParticipant, and stageStreamsToPublishForParticipant. All are discussed below.


Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

When a remote participant joins the stage, the SDK queries the host application about the desired subscription state for that participant. The options are NONE, AUDIO_ONLY, and AUDIO_VIDEO. When returning a value for this function, the host application does not need to worry about the publish state, current subscription state, or stage connection state. If AUDIO_VIDEO is returned, the SDK waits until the remote participant is publishing before subscribing, and it updates the host application through the renderer throughout the process.

Here is a sample implementation:


@Override
Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return Stage.SubscribeType.AUDIO_VIDEO;
}

This is the complete implementation of this function for a host application that always wants all participants to see each other; e.g., a video chat application.

More advanced implementations also are possible. Use the userInfo property on ParticipantInfo to selectively subscribe to participants based on server-provided attributes:


@Override
Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	switch(participantInfo.userInfo.get(“role”)) {
		case “moderator”:
			return Stage.SubscribeType.NONE;
		case “guest”:
			return Stage.SubscribeType.AUDIO_VIDEO;
		default:
			return Stage.SubscribeType.NONE;
	}
}

This can be used to create a stage where moderators can monitor all guests without being seen or heard themselves. The host application could use additional business logic to let moderates see each other but remain invisible to guests.


boolean shouldPublishFromParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

Once connected to the stage, the SDK queries the host application to see if a particular participant should publish. This is invoked only on local participants that have permission to publish based on the provided token.

Here is a sample implementation:


@Override
boolean shouldPublishFromParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return true;
}

This is for a standard video chat application where users always want to publish. They can mute and unmute their audio and video, to instantly be hidden or seen/heard. (They also can use publish/unpublish, but that is much slower. Mute/unmute is preferable for use cases where changing visibility often is desirable.)


@Override
List<LocalStageStream> stageStreamsToPublishForParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);
}

When publishing, this is used to determine what audio and video streams should be published. This is covered in more detail later in Publish a Media Stream.

The strategy is intended to be dynamic: the values returned from any of the above functions can be changed at any time. For example, if the host application does not want to publish until the end user taps a button, you could return a variable from shouldPublishFromParticipant (something like hasUserTappedPublishButton). When that variable changes based on an interaction by the end user, call stage.refreshStrategy() to signal to the SDK that it should query the strategy for the latest values, applying only things that have changed. If the SDK observes that the shouldPublishFromParticipant value has changed, it will start the publish process. If the SDK queries and all functions return the same value as before, the refreshStrategy call will not perform any modifications to the stage.

If the return value of shouldSubscribeToParticipant changes from AUDIO_VIDEO to AUDIO_ONLY, the video stream will be removed for all participants with changed returned values, if a video stream existed previously.

Generally, the stage uses the strategy to most efficiently apply the difference between the previous and current strategies, without the host application needing to worry about all the state required to manage it properly. Because of this, think of calling stage.refreshStrategy() as a cheap operation, because it does nothing unless the strategy changes.

The StageRenderer interface communicates the state of the stage to the host application. Updates to the host application’s UI usually can be powered entirely by the events provided by the renderer. The renderer provides the following functions:


void onParticipantJoined(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

void onParticipantLeft(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

void onParticipantPublishStateChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull Stage.PublishState publishState);

void onParticipantSubscribeStateChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull Stage.SubscribeState subscribeState);

void onStreamsAdded(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams);

void onStreamsRemoved(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams);

void onStreamsMutedChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams);

void onError(@NonNull BroadcastException exception);

void onConnectionStateChanged(@NonNull Stage stage, @NonNull Stage.ConnectionState state, @Nullable BroadcastException exception);

For most of these methods, the corresponding Stage and ParticipantInfo are provided.

It is not expected that the information provided by the renderer impacts the return values of the strategy. For example, the return value of shouldSubscribeToParticipant is not expected to change when onParticipantPublishStateChanged is called. If the host application wants to subscribe to a particular participant, it should return the desired subscription type regardless of that participant’s publish state. The SDK is responsible for ensuring that the desired state of the strategy is acted on at the correct time based on the state of the stage.

The StageRenderer can be attached to the stage class:


stage.addRenderer(renderer); // multiple renderers can be added

Note that only publishing participants trigger onParticipantJoined, and whenever a participant stops publishing or leaves the stage session, onParticipantLeft is triggered.

Local devices such as built-in microphones and cameras are discovered via DeviceDiscovery. Here is an example of selecting the front-facing camera and default microphone, then return them as LocalStageStreams to be published by the SDK:


DeviceDiscovery deviceDiscovery = new DeviceDiscovery(context);

List<Device> devices = deviceDiscovery.listLocalDevices();
List<LocalStageStream> publishStreams = new ArrayList<LocalStageStream>();

Device frontCamera = null;
Device microphone = null;

// Create streams using the front camera, first microphone
for (Device device : devices) {
	Device.Descriptor descriptor = device.getDescriptor();
	if (!frontCamera && descriptor.type == Device.Descriptor.DeviceType.Camera && descriptor.position = Device.Descriptor.Position.FRONT) {
		front Camera = device;
	}
	if (!microphone && descriptor.type == Device.Descriptor.DeviceType.Microphone) {
		microphone = device;
	}
}

ImageLocalStageStream cameraStream = new ImageLocalStageStream(frontCamera);
AudioLocalStageStream microphoneStream = new AudioLocalStageStream(microphoneDevice);

publishStreams.add(cameraStream);
publishStreams.add(microphoneStream);

// Provide the streams in Stage.Strategy
@Override
@NonNull List<LocalStageStream> stageStreamsToPublishForParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return publishStreams;
}

After subscribing is completed, you will receive an array of StageStream objects through the renderer’s onStreamsAdded function. You can retrieve the preview from an ImageStageStream:


ImagePreviewView preview = ((ImageStageStream)stream).getPreview();

// Add the view to your view hierarchy
LinearLayout previewHolder = findViewById(R.id.previewHolder);
preview.setLayoutParams(new LinearLayout.LayoutParams(
		LinearLayout.LayoutParams.MATCH_PARENT,
		LinearLayout.LayoutParams.MATCH_PARENT));
previewHolder.addView(preview);

You can retrieve the audio-level stats from an AudioStageStream:


((AudioStageStream)stream).setStatsCallback((peak, rms) -> {
	// handle statistics
});

When a participant stops publishing or is unsubscribed from, the onStreamsRemoved function is called with the streams that were removed. Host applications should use this as a signal to remove the participant’s video stream from the view hierarchy.

onStreamsRemoved is invoked for all scenarios in which a stream might be removed, including:

The remote participant stops publishing.
A local device unsubscribes or changes subscription from AUDIO_VIDEO to AUDIO_ONLY.
The remote participant leaves the stage.
The local participant leaves the stage.

Because onStreamsRemoved is invoked for all scenarios, no custom business logic is required around removing participants from the UI during remote or local leave operations.

LocalStageStream objects have a setMuted function that controls whether the stream is muted. This function can be called on the stream before or after it is returned from the streamsToPublishForParticipant strategy function.

Important: If a new LocalStageStream object instance is returned by streamsToPublishForParticipant after a call to refreshStrategy, the mute state of the new stream object is applied to the stage. Be careful when creating new LocalStageStream instances to make sure the expected mute state is maintained.

When a participant changes the mute state of their video or audio stream, the renderer onStreamMutedChanged function is invoked with a list of streams that have changed. Use the getMuted method on StageStream to update your UI accordingly.


@Override
void onStreamsMutedChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams) {
	for (StageStream stream : streams) {
		boolean muted = stream.getMuted();
		// handle UI changes
	}
}

To get the latest WebRTC statistics for a publishing stream or a subscribing stream, use requestRTCStats on StageStream. When a collection is completed, you will receive statistics through the StageStream.Listener which can be set on StageStream.


stream.requestRTCStats();

@Override
void onRTCStats(Map<String, Map<String, String>> statsMap) {
	for (Map.Entry<String, Map<String, string>> stat : statsMap.entrySet()) {
		for(Map.Entry<String, String> member : stat.getValue().entrySet()) {
			Log.i(TAG, stat.getKey() + “ has member “ + member.getKey() + “ with value “ + member.getValue());
		}
	}
}

If you specify attributes in the CreateParticipantToken endpoint request, you can see the attributes in ParticipantInfo properties:


@Override
void onParticipantJoined(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	for (Map.Entry<String, String> entry : participantInfo.userInfo.entrySet()) {
		Log.i(TAG, “attribute: “ + entry.getKey() + “ = “ + entry.getValue());
	}
}

When the app enters the background, you may want to stop publishing or subscribe only to other remote participants’ audio. To accomplish this, update your Strategy implementation to stop publishing, and subscribe to AUDIO_ONLY (or NONE, if applicable).


// Local variables before going into the background
boolean shouldPublish = true;
Stage.SubscribeType subscribeType = Stage.SubscribeType.AUDIO_VIDEO;

// Stage.Strategy implementation
@Override
boolean shouldPublishFromParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return shouldPublish;
}

@Override
Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return subscribeType;
}

// In our Activity, modify desired publish/subscribe when we go to background, then call refreshStrategy to update the stage
@Override
void onStop() {
	super.onStop();
	shouldPublish = false;
	subscribeTpye = Stage.SubscribeType.AUDIO_ONLY;
	stage.refreshStrategy();
}

When publishing a media stream, the SDK transmits high-quality and low-quality video streams, so remote participants can subscribe to the stream even if they have limited downlink bandwidth. Layered encoding with simulcast is on by default. You can disable it by using the StageVideoConfiguration.Simulcast class:


// Disable Simulcast
StageVideoConfiguration config = new StageVideoConfiguration();
config.simulcast.setEnabled(false);

ImageLocalStageStream cameraStream = new ImageLocalStageStream(frontCamera, config);

// Other Stage implementation code

The SDK does not support forcing portrait mode or landscape mode using StageVideoConfiguration.setSize(BroadcastConfiguration.Vec2 size). In portrait orientation, the smaller dimension is used as the width; in landscape orientation, the height. This means that the following two calls to setSize have the same effect on the video configuration:


StageVideo Configuration config = new StageVideo Configuration();

config.setSize(BroadcastConfiguration.Vec2(720f, 1280f);
config.setSize(BroadcastConfiguration.Vec2(1280f, 720f);

When the local device’s network connection is lost, the SDK internally tries to reconnect without any user action. In some cases, the SDK is not successful and user action is needed. There are two main errors related to losing the network connection:

Error code 1400, message: "PeerConnection is lost due to unknown network error"
Error code 1300, message: "Retry attempts are exhausted"

If the first error is received but the second is not, the SDK is still connected to the stage and will try to reestablish its connections automatically. As a safeguard, you can call refreshStrategy without any changes to the strategy method’s return values, to trigger a manual reconnect attempt.

If the second error is received, the SDK’s reconnect attempts have failed and the local device is no longer connected to the stage. In this case, try to rejoin the stage by calling join after your network connection has been reestablished.

In general, encountering errors after joining a stage successfully indicates that the SDK was unsuccessful in reestablishing a connection. Create a new Stage object and try to join when network conditions improve.

To publish using Bluetooth microphone devices, you must start a Bluetooth SCO connection:


Bluetooth.startBluetoothSco(context);
// Now bluetooth microphones can be used
…
// Must also stop bluetooth SCO
Bluetooth.stopBluetoothSco(context);

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Getting Started

Known Issues and Workarounds

Publishing and Subscribing

Concepts

Stage

Strategy

Subscribing to Participants

Publishing

Choosing Streams to Publish

Updating the Strategy

Renderer

Publish a Media Stream

Display and Remove Participants

Mute and Unmute Media Streams

Monitor Remote Participant Media Mute State

Get WebRTC Statistics

Get Participant Attributes

Continue Session in the Background

Enable/Disable Layered Encoding with Simulcast

Video-Configuration Limitations

Handling Network Issues

Using Bluetooth Microphones