IMediaEngine

addListener

addListener?<EventType extends keyof IMediaEngineEvent>(
      eventType: EventType,
      listener: IMediaEngineEvent[EventType]
    ): void;

After successfully calling this method, you can listen to events of the corresponding IMediaEngine object and retrieve data through IMediaEngineEvent. You can add multiple listeners for the same event as needed.

Parameters

eventType

The name of the target event to listen to. See IMediaEngineEvent.

listener

The callback function corresponding to eventType. For example, to add onPlaybackAudioFrameBeforeMixing:

const onPlaybackAudioFrameBeforeMixing = (channelId: string, uid: number, audioFrame: AudioFrame) => {};
engine.addListener('onPlaybackAudioFrameBeforeMixing', onPlaybackAudioFrameBeforeMixing);

createCustomAudioTrack

Creates a custom audio capture track.

abstract createCustomAudioTrack(
    trackType: AudioTrackType,
    config: AudioTrackConfig
  ): number;

To publish custom captured audio in a channel, follow these steps:

Call this method to create an audio track and obtain the audio track ID.
When calling joinChannel to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the audio track ID you want to publish, and set publishCustomAudioTrack to true.
Call pushAudioFrame and set trackId to the audio track ID specified in step 2 to publish the corresponding custom audio source in the channel.

Note: This method must be called before joining a channel.

Parameters

trackType: Custom audio track type. See AudioTrackType.
Note: If AudioTrackDirect is specified, you must set publishMicrophoneTrack in ChannelMediaOptions to false when calling joinChannel, otherwise joining the channel will fail and return error code -2.
config: Custom audio track configuration. See AudioTrackConfig.

Return Values

If the method call succeeds, returns the audio track ID as the unique identifier of the audio track.
If the method call fails, returns 0xffffffff. See Error Codes for details and resolution suggestions.

destroyCustomAudioTrack

Destroys the specified audio track.

abstract destroyCustomAudioTrack(trackId: number): number;

Parameters

trackId: Custom audio track ID returned by the createCustomAudioTrack method.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

pullAudioFrame

Pulls remote audio data.

abstract pullAudioFrame(): AudioFrame;

After calling this method, the app actively pulls the decoded and mixed remote audio data for audio playback.

Note: This method and the onPlaybackAudioFrame callback can both be used to obtain the mixed remote audio playback data. After calling setExternalAudioSink to enable external audio rendering, the app will no longer receive data from the onPlaybackAudioFrame callback. Therefore, choose between this method and the onPlaybackAudioFrame callback based on your actual business needs. The two have different handling mechanisms. The differences are as follows:

After calling this method, the app actively pulls audio data. By setting the audio data, the SDK can adjust the buffer to help the app handle latency, effectively avoiding audio playback jitter.
After registering the onPlaybackAudioFrame callback, the SDK delivers audio data to the app through the callback. When the app handles audio frame latency, it may cause audio playback jitter.

This method is only used to pull mixed remote audio playback data. To obtain the original captured audio data, or the original playback data of each stream before mixing, call registerAudioFrameObserver to register the corresponding callback.

Timing

This method must be called after joining a channel. Before calling this method, you need to call setExternalAudioSink(enabled: true) to enable and set external rendering.

Return Values

If the method call succeeds, returns an AudioFrame object.
If the method call fails, returns an error code.

pushAudioFrame

Pushes external audio frames.

abstract pushAudioFrame(frame: AudioFrame, trackId?: number): number;

Call this method to push external audio frames through an audio track.

Timing

Before calling this method to push external audio data, perform the following steps:

Call createCustomAudioTrack to create an audio track and obtain the audio track ID.
When calling joinChannel, set publishCustomAudioTrackId in ChannelMediaOptions to the ID of the audio track you want to publish, and set publishCustomAudioTrack to true.

Parameters

frame: The external audio frame. See AudioFrame.
trackId: The audio track ID. If you want to publish a custom external audio source, set this parameter to the custom audio track ID you want to publish.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

pushVideoFrame

Publishes external raw video frames to the channel through a custom video track.

abstract pushVideoFrame(
    frame: ExternalVideoFrame,
    videoTrackId?: number
  ): number;

When you need to publish a custom captured video in the channel, follow these steps:

Call createCustomVideoTrack to create a video track and get the video track ID.
When calling joinChannel to join the channel, set customVideoTrackId in ChannelMediaOptions to the video track ID you want to publish, and set publishCustomVideoTrack to true.
Call this method and specify videoTrackId as the video track ID from step 2 to publish the corresponding custom video source in the channel.

Note: After calling this method, even if you stop pushing external video frames to the SDK, the custom captured video stream will still be counted in video duration usage and incur charges. Agora recommends taking appropriate actions based on your actual needs to avoid such video billing:

If you no longer need to capture external video data, call destroyCustomVideoTrack to destroy the custom captured video track.
If you only want to use the captured external video data for local preview and not publish it in the channel, call muteLocalVideoStream to stop sending the video stream, or call updateChannelMediaOptions and set publishCustomVideoTrack to false.

Parameters

frame: The video frame to be pushed. See ExternalVideoFrame.
videoTrackId: The video track ID returned by the createCustomVideoTrack method.
Note: If you only need to push a single external video stream, set videoTrackId to 0.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

registerAudioFrameObserver

Registers an audio frame observer object.

abstract registerAudioFrameObserver(observer: IAudioFrameObserver): number;

This method registers an audio frame observer object, i.e., registers callbacks. You need to call this method to register callbacks if you want the SDK to trigger the onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onEarMonitoringAudioFrame callbacks.

Timing

This method must be called before joining a channel.

Parameters

observer: Instance of the interface object. See IAudioFrameObserver. It is recommended to call this after receiving onLeaveChannel to release the audio frame observer object.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

registerFaceInfoObserver

Registers a face information observer.

abstract registerFaceInfoObserver(observer: IFaceInfoObserver): number;

You can call this method to register the onFaceInfo callback to obtain face information processed by the Agora voice driver plugin. When registering a face information observer using this method, you can register the callbacks in the IFaceInfoObserver class as needed. After successful registration, the SDK triggers the registered callback when face information converted by the voice driver plugin is detected.

Note:

This method must be called before joining a channel.
Before calling this method, make sure you have called enableExtension to enable the voice driver plugin.

Scenario

The face information processed by the Agora voice driver plugin conforms to the ARKit standard BS (Blend Shape) data. You can use third-party 3D rendering engines to further process the BS data, such as driving a virtual avatar's facial movements to match speech.

Parameters

observer: The face information observer. See IFaceInfoObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

registerVideoEncodedFrameObserver

Registers a video frame observer for encoded video frames.

abstract registerVideoEncodedFrameObserver(
    observer: IVideoEncodedFrameObserver
  ): number;

If you only want to observe encoded video frames (e.g., H.264 format) and do not need to decode or render them, Agora recommends using this method to register an IVideoEncodedFrameObserver class.

Note: This method must be called before joining a channel.

Parameters

observer: Video frame observer. See IVideoEncodedFrameObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

registerVideoFrameObserver

Registers a raw video frame observer object.

abstract registerVideoFrameObserver(observer: IVideoFrameObserver): number;

If you want to observe raw video frames (such as YUV or RGBA format), Agora recommends registering an IVideoFrameObserver class using this method. When registering the video observer, you can choose to register callbacks from the IVideoFrameObserver class as needed. Once registered successfully, the SDK triggers the registered callbacks whenever a video frame is captured.

Note: When handling callbacks, you need to consider changes in the width and height parameters of the video frame, as the observed video frame may vary due to the following conditions:

When the network condition is poor, the resolution may drop in steps.
When the user adjusts the resolution manually, the resolution reported in the callback will also change.

Scenario

After registering a raw video frame observer, you can use the raw video data for various video preprocessing scenarios, such as implementing virtual backgrounds or beauty effects.

Timing

Call this method before joining a channel.

Parameters

observer: Instance of the interface object. See IVideoFrameObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

removeAllListeners

Removes all listeners for the specified event.

removeAllListeners?<EventType extends keyof IMediaEngineEvent>(
      eventType?: EventType
    ): void;

Parameters

eventType: The name of the target event to listen to. See IMediaEngineEvent.

removeListener

Removes the specified IMediaEngineEvent listener.

removeListener?<EventType extends keyof IMediaEngineEvent>(
      eventType: EventType,
      listener: IMediaEngineEvent[EventType]
    ): void;

Parameters

eventType

The name of the target event to listen to. See IMediaEngineEvent.

listener

The callback function corresponding to eventType. You must pass in the same function object that was passed to addListener. For example, to remove the onJoinChannelSuccess listener:

const onPlaybackAudioFrameBeforeMixing = (channelId: string, uid: number, audioFrame: AudioFrame) => {};
engine.addListener('onPlaybackAudioFrameBeforeMixing', onPlaybackAudioFrameBeforeMixing);
engine.removeListener('onPlaybackAudioFrameBeforeMixing', onPlaybackAudioFrameBeforeMixing);

setExternalAudioSink

Sets external audio rendering.

abstract setExternalAudioSink(
    enabled: boolean,
    sampleRate: number,
    channels: number
  ): number;

After calling this method to enable external audio rendering, you can call pullAudioFrame to pull remote audio data. The app can process the pulled raw audio data before rendering to achieve the desired audio effect.

Note: After calling this method to enable external audio rendering, the app will no longer receive data from the onPlaybackAudioFrame callback.

Scenario

This method is suitable for scenarios where you need to render audio yourself.

Timing

This method must be called before joining a channel.

Parameters

enabled

Whether to enable external audio rendering:

true: Enable external audio rendering.
false: (Default) Disable external audio rendering.

sampleRate

The sample rate (Hz) for external audio rendering. Can be set to 16000, 32000, 44100, or 48000.

channels

The number of channels for external audio rendering:

1: Mono
2: Stereo

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

setExternalAudioSource

Sets external audio capture parameters.

abstract setExternalAudioSource(
    enabled: boolean,
    sampleRate: number,
    channels: number,
    localPlayback?: boolean,
    publish?: boolean
  ): number;

Deprecated: Deprecated: This method is deprecated. Use createCustomAudioTrack instead.

Timing

This method must be called before joining a channel.

Parameters

enabled

Whether to enable the use of external audio sources:

true: Enable external audio source.
false: (Default) Disable external audio source.

sampleRate

Sampling rate (Hz) of the external audio source. Can be set to 8000, 16000, 32000, 44100, or 48000.

channels

Number of channels of the external audio source. Can be set to 1 (mono) or 2 (stereo).

localPlayback

Whether to play the external audio source locally:

true: Play locally.
false: (Default) Do not play locally.

publish

Whether to publish the audio to the remote end:

true: (Default) Publish to remote.
false: Do not publish to remote.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

setExternalVideoSource

Sets the external video source.

abstract setExternalVideoSource(
    enabled: boolean,
    useTexture: boolean,
    sourceType?: ExternalVideoSourceType,
    encodedVideoOption?: SenderOptions
  ): number;

After calling this method to enable the external video source, you can call pushVideoFrame to push external video data to the SDK.

Note: Dynamic switching of video sources within a channel is not supported. If you have enabled the external video source and joined a channel, to switch to the internal video source, you must leave the channel first, then call this method to disable the external video source, and rejoin the channel.

Timing

You must call this method before joining a channel.

Parameters

enabled

Whether to enable the external video source:

true: Enable the external video source. The SDK is ready to receive external video frames.
false: (Default) Do not enable the external video source.

useTexture

Whether to use external video frames in Texture format:

true: Use external video frames in Texture format.
false: Do not use external video frames in Texture format.

sourceType

Whether the external video frame is encoded. See ExternalVideoSourceType.

encodedVideoOption

Video encoding options. If sourceType is EncodedVideoFrame, you need to set this parameter. You can contact technical support to learn how to configure this parameter.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

unregisterAudioFrameObserver

Unregisters the audio frame observer.

abstract unregisterAudioFrameObserver(observer: IAudioFrameObserver): number;

Parameters

observer: The audio frame observer that monitors each received audio frame. See IAudioFrameObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and troubleshooting.

unregisterFaceInfoObserver

Unregisters the face information observer.

abstract unregisterFaceInfoObserver(observer: IFaceInfoObserver): number;

Parameters

observer: The face information observer. See IFaceInfoObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

unregisterVideoEncodedFrameObserver

Unregisters the video frame observer for encoded video frames.

abstract unregisterVideoEncodedFrameObserver(
    observer: IVideoEncodedFrameObserver
  ): number;

Parameters

observer: Video frame observer that observes the reception of each video frame. See IVideoEncodedFrameObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.

unregisterVideoFrameObserver

Unregisters the video frame observer.

abstract unregisterVideoFrameObserver(observer: IVideoFrameObserver): number;

Parameters

observer: The video frame observer that observes the reception of each video frame. See IVideoFrameObserver.

Return Values

0: Success.
< 0: Failure. See Error Codes for details and resolution suggestions.