MediaEngine

createCustomAudioTrack

Creates a custom audio track.

Future<int> createCustomAudioTrack(
      {required AudioTrackType trackType, required AudioTrackConfig config});

To publish custom captured audio in a channel, follow these steps:

Call this method to create an audio track and get the audio track ID.
When calling joinChannel to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the audio track ID you want to publish, and set publishCustomAudioTrack to true.
Call pushAudioFrame and set trackId to the audio track ID specified in step 2 to publish the corresponding custom audio source in the channel.

Note: This method must be called before joining a channel.

Parameters

trackType: Custom audio track type. See AudioTrackType.
Note: If audioTrackDirect is specified, you must set publishMicrophoneTrack to false in ChannelMediaOptions when calling joinChannel; otherwise, joining the channel will fail and return error code -2.
config: Custom audio track configuration. See AudioTrackConfig.

Return Values

If the method call succeeds, it returns the audio track ID as the unique identifier of the audio track.
If the method call fails, it returns 0xffffffff. See Error Codes for details and resolution suggestions.

destroyCustomAudioTrack

Destroys the specified audio track.

Future<void> destroyCustomAudioTrack(int trackId);

Parameters

trackId: The custom audio track ID returned by the createCustomAudioTrack method.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

pullAudioFrame

Pulls remote audio data.

Future<void> pullAudioFrame(AudioFrame frame);

After calling this method, the app actively pulls the decoded and mixed remote audio data for playback.

Note: This method and the onPlaybackAudioFrame callback can both be used to get the mixed remote audio playback data. After calling setExternalAudioSink to enable external audio rendering, the app can no longer get data from the onPlaybackAudioFrame callback. Therefore, choose between this method and the onPlaybackAudioFrame callback based on your actual business needs. The processing mechanisms differ, with the following distinctions:

After calling this method, the app actively pulls the audio data. By setting the audio data, the SDK can adjust the buffer to help the app handle latency and effectively avoid audio playback jitter.
After registering the onPlaybackAudioFrame callback, the SDK pushes the audio data to the app through the callback. When handling audio frame latency, the app may cause audio playback jitter.

This method is only used to pull the mixed remote audio playback data. To obtain the captured raw audio data or the raw audio playback data of each stream before mixing, you can register the corresponding callback by calling registerAudioFrameObserver.

Timing

This method must be called after joining a channel. Before calling this method, you need to call setExternalAudioSink(enabled: true) to enable and set external rendering.

Parameters

frame: Pointer to AudioFrame.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

pushAudioFrame

Pushes external audio frames.

Future<void> pushAudioFrame({required AudioFrame frame, int trackId = 0});

Call this method to push external audio frames through an audio track.

Timing

Before pushing external audio data using this method, perform the following steps:

Call createCustomAudioTrack to create an audio track and obtain the audio track ID.
When calling joinChannel to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the ID of the audio track you want to publish, and set publishCustomAudioTrack to true.

Parameters

frame: External audio frame. See AudioFrame.
trackId: Audio track ID. If you want to publish a custom external audio source, set this parameter to the custom audio track ID you want to publish.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

pushVideoFrame

Publishes external raw video frames to the channel through a custom video track.

Future<void> pushVideoFrame(
    {required ExternalVideoFrame frame, int videoTrackId = 0});

When you need to publish custom captured video in a channel, follow these steps:

Call the createCustomVideoTrack method to create a video track and get the video track ID.
When calling joinChannel to join a channel, set customVideoTrackId in ChannelMediaOptions to the video track ID you want to publish and set publishCustomVideoTrack to true.
Call this method and specify videoTrackId as the video track ID from step 2 to publish the corresponding custom video source in the channel.

Note: After calling this method, even if you stop pushing external video frames to the SDK, the custom captured video stream will still be counted in the video duration usage and incur charges. Agora recommends taking appropriate measures based on your actual situation to avoid such video billing:

If you no longer need to capture external video data, call destroyCustomVideoTrack to destroy the custom captured video track.
If you only want to use the captured external video data for local preview and not publish it in the channel, call muteLocalVideoStream to stop sending the video stream, or call updateChannelMediaOptions and set publishCustomVideoTrack to false.

Parameters

frame: The video frame to be pushed. See ExternalVideoFrame.
videoTrackId: The video track ID returned by the createCustomVideoTrack method.
Note: If you only need to push one external video stream, set videoTrackId to 0.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

registerAudioFrameObserver

Registers an audio observer object.

void registerAudioFrameObserver(AudioFrameObserver observer);

This method registers an audio observer object, i.e., registers callbacks. When you need the SDK to trigger onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onEarMonitoringAudioFrame callbacks, you need to call this method to register the callbacks.

Timing

This method must be called before joining a channel.

Parameters

observer: Instance of the interface object. See AudioFrameObserver. It is recommended to call this after receiving onLeaveChannel to release the audio observer object.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

registerFaceInfoObserver

Registers a face information observer.

void registerFaceInfoObserver(FaceInfoObserver observer);

You can call this method to register the onFaceInfo callback to obtain the face information processed by the Agora voice driver plugin. When registering the face information observer using this method, you can register callbacks in the FaceInfoObserver class as needed. After successfully registering the face information observer, the SDK triggers the registered callback when it captures face information converted by the voice driver plugin.

Note:

You must call this method before joining a channel.
Before calling this method, make sure you have called enableExtension to enable the voice driver plugin.

Scenario

The face information processed by the Agora voice driver plugin conforms to the ARKit standard BS (Blend Shape) data. You can use third-party 3D rendering engines to further process the BS data, such as driving a virtual avatar's facial movements to match speech.

Parameters

observer: Face information observer. See FaceInfoObserver.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly.

registerVideoEncodedFrameObserver

Registers a video frame observer for encoded video images.

void registerVideoEncodedFrameObserver(VideoEncodedFrameObserver observer);

If you only want to observe encoded video frames (such as H.264 format) without decoding and rendering, Agora recommends that you register a VideoEncodedFrameObserver class using this method.

Note: This method must be called before joining a channel.

Parameters

observer: Video frame observer. See VideoEncodedFrameObserver.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

registerVideoFrameObserver

Registers a raw video frame observer object.

void registerVideoFrameObserver(VideoFrameObserver observer);

If you want to observe raw video frames (such as YUV or RGBA format), Agora recommends that you register a VideoFrameObserver class using this method. When registering a video frame observer using this method, you can register callbacks in the VideoFrameObserver class as needed. After successful registration, the SDK triggers the registered callbacks each time a video frame is captured.

Note: When handling the callback, you need to consider changes in the width and height parameters of the video frame, as the observed video frames may vary due to the following situations:

When the network condition is poor, the resolution will drop step by step.
When the user manually adjusts the resolution, the resolution reported in the callback will also change.

Scenario

After registering a raw video frame observer, you can use the raw video data for various video preprocessing scenarios, such as implementing virtual backgrounds or beauty effects.

Timing

This method must be called before joining a channel.

Parameters

observer: Instance of the interface object. See VideoFrameObserver.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

< 0: Method call failed. See Error Codes for details and resolution suggestions.

setExternalAudioSink

Sets external audio rendering.

Future<void> setExternalAudioSink(
    {required bool enabled, required int sampleRate, required int channels});

After calling this method to enable external audio rendering, you can call pullAudioFrame to pull remote audio data. The app can process the pulled raw audio data before rendering to achieve the desired audio effect.

Note: After calling this method to enable external audio rendering, the app can no longer get data from the onPlaybackAudioFrame callback.

Scenario

This method is suitable for scenarios where you need to render audio yourself.

Timing

This method must be called before joining a channel.

Parameters

enabled

Whether to enable external audio rendering:

true: Enable external audio rendering.
false: (Default) Disable external audio rendering.

sampleRate

The sample rate (Hz) for external audio rendering. You can set it to 16000, 32000, 44100, or 48000.

channels

The number of channels for external audio rendering:

1: Mono
2: Stereo

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

setExternalAudioSource

Sets the external audio source parameters.

Future<void> setExternalAudioSource(
    {required bool enabled,
    required int sampleRate,
    required int channels,
    bool localPlayback = false,
    bool publish = true});

Deprecated: Deprecated: This method is deprecated. Use createCustomAudioTrack instead.

Timing

This method must be called before joining a channel.

Parameters

enabled

Whether to enable the external audio source:

true: Enable the external audio source.
false: (Default) Disable the external audio source.

sampleRate

The sample rate (Hz) of the external audio source. You can set it to 8000, 16000, 32000, 44100, or 48000.

channels

The number of channels of the external audio source. You can set it to 1 (mono) or 2 (stereo).

localPlayback

Whether to play the external audio source locally:

true: Play locally.
false: (Default) Do not play locally.

publish

Whether to publish the audio to the remote:

true: (Default) Publish to the remote.
false: Do not publish to the remote.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

< 0: Failure. See Error Codes for details and resolution suggestions.

setExternalVideoSource

Sets the external video source.

Future<void> setExternalVideoSource(
    {required bool enabled,
    required bool useTexture,
    ExternalVideoSourceType sourceType = ExternalVideoSourceType.videoFrame,
    SenderOptions encodedVideoOption = const SenderOptions()});

After calling this method to enable the external video source, you can call pushVideoFrame to push external video data to the SDK.

Note: Switching video sources dynamically within a channel is not supported. If you have already called this method to enable an external video source and joined a channel, to switch to an internal video source, you must first leave the channel, then call this method to disable the external video source, and rejoin the channel.

Timing

This method must be called before joining the channel.

Parameters

enabled

Whether to enable the external video source:

true: Enable the external video source. The SDK is ready to receive external video frames.
false: (Default) Do not enable the external video source.

useTexture

Whether to use external video frames in Texture format:

true: Use Texture format for external video frames.
false: Do not use Texture format for external video frames.

sourceType

Whether the external video frame is encoded. See ExternalVideoSourceType.

encodedVideoOption

Video encoding options. If sourceType is encodedVideoFrame, this parameter must be set. You can contact technical support to learn how to configure this parameter.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

unregisterAudioFrameObserver

Unregisters the audio frame observer.

void unregisterAudioFrameObserver(AudioFrameObserver observer);

Parameters

observer: The audio frame observer that monitors the reception of each audio frame. See AudioFrameObserver.

Return Values

This method returns no value if the call succeeds. If the method call fails, it throws an AgoraRtcException, which you need to catch and handle. See Error Codes for details and troubleshooting suggestions.

unregisterFaceInfoObserver

Unregisters the face information observer.

void unregisterFaceInfoObserver(FaceInfoObserver observer);

Parameters

observer: Face information observer. See FaceInfoObserver.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly.

unregisterVideoEncodedFrameObserver

Unregisters the video frame observer for encoded video frames.

void unregisterVideoEncodedFrameObserver(VideoEncodedFrameObserver observer);

Parameters

observer: The video frame observer that observes each received video frame. See VideoEncodedFrameObserver.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.

unregisterVideoFrameObserver

Unregisters the video frame observer.

void unregisterVideoFrameObserver(VideoFrameObserver observer);

Parameters

observer: Video frame observer that observes each received video frame. See VideoFrameObserver.

Return Values

When the method call succeeds, there is no return value; when fails, the AgoraRtcException exception is thrown. You need to catch the exception and handle it accordingly. See Error Codes for details and resolution suggestions.