IMediaEngine

IMediaEngine class.

addListener

Adds an IMediaEngineEvent listener.

addListener?<EventType extends keyof IMediaEngineEvent>(
      eventType: EventType,
      listener: IMediaEngineEvent[EventType]
    ): void;

After successfully calling this method, you can listen to events and retrieve data from the IMediaEngine object via IMediaEngineEvent. You can add multiple listeners for the same event as needed.

Parameters

eventType
Name of the target event to be listened to. See IMediaEngineEvent.
listener
Callback function corresponding to eventType. For example, to add onPlaybackAudioFrameBeforeMixing: 
const onPlaybackAudioFrameBeforeMixing = (channelId: string, uid: number, audioFrame: AudioFrame) => {};
engine.addListener('onPlaybackAudioFrameBeforeMixing', onPlaybackAudioFrameBeforeMixing);

createCustomAudioTrack

Creates a custom audio capture track.

abstract createCustomAudioTrack(
    trackType: AudioTrackType,
    config: AudioTrackConfig
  ): number;
To publish custom captured audio in a channel, refer to the following steps:
  1. Call this method to create an audio track and obtain the audio track ID.
  2. When calling joinChannel to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the audio track ID you want to publish, and set publishCustomAudioTrack to true.
  3. Call pushAudioFrame and specify the trackId as the audio track ID specified in step 2 to publish the corresponding custom audio source in the channel.
Note: You need to call this method before joining a channel.

Parameters

trackType
Custom audio track type. See AudioTrackType.
Note: If AudioTrackDirect is specified, you must set publishMicrophoneTrack to false in ChannelMediaOptions when calling joinChannel, otherwise joining the channel will fail and return error code -2.
config
Custom audio track configuration. See AudioTrackConfig.

Return Values

  • On success, returns the audio track ID as the unique identifier of the audio track.
  • On failure, returns 0xffffffff. See Error Codes for details and resolution suggestions.

destroyCustomAudioTrack

Destroys the specified audio track.

abstract destroyCustomAudioTrack(trackId: number): number;

Parameters

trackId
Custom audio track ID returned by the createCustomAudioTrack method.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

pullAudioFrame

Pulls remote audio data.

abstract pullAudioFrame(): AudioFrame;

After calling this method, the app actively pulls the decoded and mixed remote audio data for playback.

Note: This method and the onPlaybackAudioFrame callback can both be used to obtain remote mixed audio playback data. After enabling external audio rendering by calling setExternalAudioSink, the app will no longer receive data from the onPlaybackAudioFrame callback. Therefore, choose between this method and the onPlaybackAudioFrame callback based on your actual business needs. The two have different processing mechanisms, detailed as follows:
  • After calling this method, the app actively pulls audio data. By setting the audio data, the SDK can adjust the buffer to help the app handle latency, effectively avoiding audio playback jitter.
  • After registering the onPlaybackAudioFrame callback, the SDK pushes audio data to the app through the callback. When the app handles audio frame latency, it may cause audio playback jitter.
This method is only used to pull remote mixed audio playback data. To obtain raw captured audio data or raw audio playback data of each individual stream before mixing, you can register the corresponding callback by calling registerAudioFrameObserver.

Timing

You need to call this method after joining a channel. Before calling this method, you need to call setExternalAudioSink(enabled: true) to enable and configure external rendering.

Return Values

  • On success, returns an AudioFrame object.
  • On failure, returns an error code.

pushAudioFrame

Pushes external audio frames.

abstract pushAudioFrame(frame: AudioFrame, trackId?: number): number;

Call this method to push external audio frames through an audio track.

Timing

Before calling this method to push external audio data, perform the following steps:
  1. Call createCustomAudioTrack to create an audio track and obtain the audio track ID.
  2. When calling joinChannel to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the audio track ID you want to publish, and set publishCustomAudioTrack to true.

Parameters

frame
External audio frame. See AudioFrame.
trackId
Audio track ID. If you want to publish a custom external audio source, set this parameter to the custom audio track ID you want to publish.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

pushVideoFrame

Publishes an external raw video frame to the channel through a custom video track.

abstract pushVideoFrame(
    frame: ExternalVideoFrame,
    videoTrackId?: number
  ): number;
When you need to publish a custom captured video in the channel, refer to the following steps:
  1. Call the createCustomVideoTrack method to create a video track and obtain the video track ID.
  2. When calling joinChannel to join the channel, set customVideoTrackId in ChannelMediaOptions to the video track ID you want to publish, and set publishCustomVideoTrack to true.
  3. Call this method and specify videoTrackId as the video track ID specified in step 2 to publish the corresponding custom video source in the channel.
Note: After calling this method, even if you stop pushing external video frames to the SDK, the custom captured video stream will still be counted in video duration usage and incur charges. Agora recommends taking appropriate measures based on your actual situation to avoid such charges:
  • If you no longer need to capture external video data, call destroyCustomVideoTrack to destroy the custom captured video track.
  • If you only want to use the captured external video data for local preview and not publish it in the channel, call muteLocalVideoStream to stop sending the video stream, or call updateChannelMediaOptions and set publishCustomVideoTrack to false.

Parameters

frame
The video frame to be pushed. See ExternalVideoFrame.
videoTrackId
The video track ID returned by the createCustomVideoTrack method.
Note: If you only need to push one external video stream, set videoTrackId to 0.

Return Values

  • 0: The method call succeeds.
  • < 0: The method call fails. See Error Codes for details and resolution suggestions.

registerAudioFrameObserver

Registers an audio frame observer.

abstract registerAudioFrameObserver(observer: IAudioFrameObserver): number;

This method registers an audio frame observer, i.e., registers callbacks. You need to call this method to register callbacks when the SDK needs to trigger onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onEarMonitoringAudioFrame.

Timing

This method must be called before joining a channel.

Parameters

observer
Instance of the interface object. See IAudioFrameObserver. It is recommended to call this after receiving onLeaveChannel to release the audio frame observer.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

registerFaceInfoObserver

Registers a face information observer.

abstract registerFaceInfoObserver(observer: IFaceInfoObserver): number;

You can call this method to register the onFaceInfo callback and receive face information processed by the Agora voice driver extension. When registering the face info observer, you can register callbacks in the IFaceInfoObserver class as needed. After successful registration, the SDK triggers the registered callback when face information processed by the voice driver extension is captured.

Note:
  • This method must be called before joining a channel.
  • Before calling this method, make sure you have called enableExtension to enable the voice driver extension.

Scenario

The face information processed by the Agora voice driver plugin conforms to the ARKit standard BS (Blend Shape) data. You can use third-party 3D rendering engines to further process the BS data, such as driving a virtual avatar's facial movements to match speech.

Parameters

observer
Face information observer. See IFaceInfoObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

registerVideoEncodedFrameObserver

Registers a video frame observer for encoded video images.

abstract registerVideoEncodedFrameObserver(
    observer: IVideoEncodedFrameObserver
  ): number;

If you only want to observe encoded video frames (e.g., H.264 format) without decoding and rendering them, Agora recommends registering an IVideoEncodedFrameObserver class using this method.

Note: This method must be called before joining a channel.

Parameters

observer
Video frame observer. See IVideoEncodedFrameObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

registerVideoFrameObserver

Registers a raw video frame observer.

abstract registerVideoFrameObserver(observer: IVideoFrameObserver): number;

If you want to observe raw video frames (such as YUV or RGBA format), Agora recommends registering an IVideoFrameObserver class using this method. When registering the video observer via this method, you can choose to register callbacks from the IVideoFrameObserver class as needed. After successful registration, the SDK triggers the registered callbacks when each video frame is captured.

Note: When handling the callback, you need to consider changes in the width and height parameters of the video frame, as the observed video frames may vary in the following scenarios:
  • Resolution may decrease stepwise when network conditions are poor.
  • When the user manually adjusts the resolution, the resolution reported in the callback also changes.

Scenario

After registering a raw video frame observer, you can use the raw video data for various video preprocessing scenarios, such as implementing virtual backgrounds or beauty effects.

Timing

This method must be called before joining a channel.

Parameters

observer
Instance of the interface object. See IVideoFrameObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

removeAllListeners

Removes all listeners for the specified event.

removeAllListeners?<EventType extends keyof IMediaEngineEvent>(
      eventType?: EventType
    ): void;

Parameters

eventType
The target event name to listen to. See IMediaEngineEvent.

removeListener

Removes a specified IMediaEngineEvent listener.

removeListener?<EventType extends keyof IMediaEngineEvent>(
      eventType: EventType,
      listener: IMediaEngineEvent[EventType]
    ): void;

Parameters

eventType
Name of the target event to be listened to. See IMediaEngineEvent.
listener
Callback function corresponding to eventType. You must pass the same function object that was passed to addListener. For example, to remove the listener for onJoinChannelSuccess: 
const onPlaybackAudioFrameBeforeMixing = (channelId: string, uid: number, audioFrame: AudioFrame) => {};
engine.addListener('onPlaybackAudioFrameBeforeMixing', onPlaybackAudioFrameBeforeMixing);
engine.removeListener('onPlaybackAudioFrameBeforeMixing', onPlaybackAudioFrameBeforeMixing);

setExternalAudioSink

Sets the external audio rendering.

abstract setExternalAudioSink(
    enabled: boolean,
    sampleRate: number,
    channels: number
  ): number;

After calling this method to enable external audio rendering, you can call pullAudioFrame to pull remote audio data. The app can process the pulled raw audio data before rendering to achieve the desired audio effect.

Note: After calling this method to enable external audio rendering, the app will no longer receive data from the onPlaybackAudioFrame callback.

Scenario

This method is suitable for scenarios where you need to render audio yourself.

Timing

This method must be called before joining a channel.

Parameters

enabled
Sets whether to enable external audio rendering:
  • true: Enable external audio rendering.
  • false: (Default) Disable external audio rendering.
sampleRate
The sample rate (Hz) for external audio rendering. Can be set to 16000, 32000, 44100, or 48000.
channels
The number of channels for external audio rendering:
  • 1: Mono
  • 2: Stereo

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

setExternalAudioSource

Sets external audio capture parameters.

abstract setExternalAudioSource(
    enabled: boolean,
    sampleRate: number,
    channels: number,
    localPlayback?: boolean,
    publish?: boolean
  ): number;
Deprecated
Deprecated: This method is obsolete. Use createCustomAudioTrack instead.

Timing

You need to call this method before joining a channel.

Parameters

enabled
Whether to enable the use of external audio sources:
  • true: Enable external audio source.
  • false: (Default) Disable external audio source.
sampleRate
Sampling rate (Hz) of the external audio source. Can be set to 8000, 16000, 32000, 44100, or 48000.
channels
Number of channels of the external audio source. Can be set to 1 (mono) or 2 (stereo).
localPlayback
Whether to play the external audio source locally:
  • true: Play locally.
  • false: (Default) Do not play locally.
publish
Whether to publish the audio to the remote end:
  • true: (Default) Publish to remote.
  • false: Do not publish to remote.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

setExternalVideoSource

Sets the external video source.

abstract setExternalVideoSource(
    enabled: boolean,
    useTexture: boolean,
    sourceType?: ExternalVideoSourceType,
    encodedVideoOption?: SenderOptions
  ): number;

After enabling an external video source by calling this method, you can call pushVideoFrame to push external video data to the SDK.

Note: Switching video sources dynamically within the channel is not supported. If you have enabled an external video source and joined a channel, to switch to an internal video source, you must first leave the channel, then call this method to disable the external video source, and rejoin the channel.

Timing

This method must be called before joining a channel.

Parameters

enabled
Whether to enable the external video source:
  • true: Enable the external video source. The SDK prepares to receive external video frames.
  • false: (default) Do not enable the external video source.
useTexture
Whether to use external video frames in Texture format:
  • true: Use external video frames in Texture format.
  • false: Do not use external video frames in Texture format.
sourceType
Whether the external video frame is encoded. See ExternalVideoSourceType.
encodedVideoOption
Video encoding options. If sourceType is EncodedVideoFrame, this parameter must be set. You can contact technical support to learn how to set this parameter.

Return Values

  • 0: The method call succeeds.
  • < 0: The method call fails. See Error Codes for details and resolution suggestions.

unregisterAudioFrameObserver

Unregisters the audio frame observer.

abstract unregisterAudioFrameObserver(observer: IAudioFrameObserver): number;

Parameters

observer
The audio frame observer that monitors the reception of each audio frame. See IAudioFrameObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and troubleshooting.

unregisterFaceInfoObserver

Unregisters the face information observer.

abstract unregisterFaceInfoObserver(observer: IFaceInfoObserver): number;

Parameters

observer
Face information observer. See IFaceInfoObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

unregisterVideoEncodedFrameObserver

Unregisters the video frame observer for encoded video images.

abstract unregisterVideoEncodedFrameObserver(
    observer: IVideoEncodedFrameObserver
  ): number;

Parameters

observer
Video frame observer that observes each received video frame. See IVideoEncodedFrameObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

unregisterVideoFrameObserver

Unregisters the video frame observer.

abstract unregisterVideoFrameObserver(observer: IVideoFrameObserver): number;

Parameters

observer
Video frame observer that observes each received video frame. See IVideoFrameObserver.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.