IMediaEngine

IMediaEngine class.

CreateCustomAudioTrack

Creates a custom audio capture track.

public abstract uint CreateCustomAudioTrack(AUDIO_TRACK_TYPE trackType, AudioTrackConfig config);
To publish custom captured audio in a channel, follow these steps:
  1. Call this method to create an audio track and obtain the audio track ID.
  2. When calling JoinChannel [2/2] to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the audio track ID you want to publish, and set publishCustomAudioTrack to true.
  3. Call PushAudioFrame and set trackId to the audio track ID specified in step 2 to publish the corresponding custom audio source in the channel.
Note: This method must be called before joining a channel.

Parameters

trackType
The type of custom audio track. See AUDIO_TRACK_TYPE.
Note: If AUDIO_TRACK_DIRECT is specified, you must set publishMicrophoneTrack in ChannelMediaOptions to false when calling JoinChannel [2/2], otherwise joining the channel will fail and return error code -2.
config
Configuration of the custom audio track. See AudioTrackConfig.

Return Values

  • If the method call succeeds, returns the audio track ID as the unique identifier of the audio track.
  • If the method call fails, returns 0xffffffff. See Error Codes for details and resolution suggestions.

DestroyCustomAudioTrack

Destroys the specified audio track.

public abstract int DestroyCustomAudioTrack(uint trackId);

Parameters

trackId
The custom audio track ID returned by the CreateCustomAudioTrack method.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

PullAudioFrame

Pulls remote audio data.

public abstract int PullAudioFrame(AudioFrame frame);

After calling this method, the app actively pulls the decoded and mixed remote audio data for playback.

Note: This method and the OnPlaybackAudioFrame callback can both be used to obtain the mixed remote audio playback data. After enabling external audio rendering using SetExternalAudioSink, the app will no longer receive data from the OnPlaybackAudioFrame callback. Therefore, choose between this method and the OnPlaybackAudioFrame callback based on your actual business needs. They differ in processing mechanisms as follows:
  • After calling this method, the app actively pulls audio data. By setting audio data, the SDK can adjust the buffer to help the app handle latency and effectively avoid audio playback jitter.
  • After registering the OnPlaybackAudioFrame callback, the SDK pushes audio data to the app through the callback. When the app processes audio frame latency, it may cause audio playback jitter.
This method is only used to pull mixed remote audio playback data. To obtain raw captured audio data, or raw playback data of each stream before mixing, you can register the corresponding callback by calling RegisterAudioFrameObserver.

Timing

This method must be called after joining a channel. Before calling this method, you need to call SetExternalAudioSink(enabled: true) to enable and configure external rendering.

Parameters

frame
Pointer to AudioFrame.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

PushAudioFrame

Pushes external audio frames.

public abstract int PushAudioFrame(AudioFrame frame, uint trackId = 0);

Call this method to push external audio frames through an audio track.

Timing

Before calling this method to push external audio data, perform the following steps:
  1. Call CreateCustomAudioTrack to create an audio track and obtain the audio track ID.
  2. When calling JoinChannel [2/2] to join a channel, set publishCustomAudioTrackId in ChannelMediaOptions to the ID of the audio track you want to publish, and set publishCustomAudioTrack to true.

Parameters

frame
External audio frame. See AudioFrame.
trackId
Audio track ID. If you want to publish a custom external audio source, set this parameter to the ID of the custom audio track you want to publish.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

PushVideoFrame

Publishes external raw video frames to the channel through a custom video track.

public abstract int PushVideoFrame(ExternalVideoFrame frame, uint videoTrackId = 0);
When you need to publish a custom captured video in the channel, refer to the following steps:
  1. Call the CreateCustomVideoTrack method to create a video track and get the video track ID.
  2. When calling JoinChannel [2/2] to join the channel, set customVideoTrackId in ChannelMediaOptions to the video track ID you want to publish, and set publishCustomVideoTrack to true.
  3. Call this method and specify videoTrackId as the video track ID set in step 2 to publish the corresponding custom video source in the channel.
Note: After calling this method, even if you stop pushing external video frames to the SDK, the custom captured video stream will still be counted in video usage duration and incur charges. Agora recommends taking appropriate measures based on your actual situation to avoid such video billing:
  • If you no longer need to capture external video data, call DestroyCustomVideoTrack to destroy the custom captured video track.
  • If you only want to use the captured external video data for local preview and not publish it in the channel, call MuteLocalVideoStream to stop sending the video stream, or call UpdateChannelMediaOptions and set publishCustomVideoTrack to false.

Parameters

frame
The video frame to be pushed. See ExternalVideoFrame.
videoTrackId
The video track ID returned by the CreateCustomVideoTrack method.
Note: If you only need to push one external video stream, you can set videoTrackId to 0.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

RegisterAudioFrameObserver

Registers an audio frame observer object.

public abstract int RegisterAudioFrameObserver(IAudioFrameObserver audioFrameObserver, AUDIO_FRAME_POSITION position, OBSERVER_MODE mode = OBSERVER_MODE.INTPTR);

This method registers an audio frame observer object, i.e., registers callbacks. You need to call this method to register the callbacks if you want the SDK to trigger the OnMixedAudioFrame, OnRecordAudioFrame, OnPlaybackAudioFrame, OnPlaybackAudioFrameBeforeMixing, and OnEarMonitoringAudioFrame callbacks.

Timing

You need to call this method before joining a channel.

Parameters

audioFrameObserver
Instance of the interface object. See IAudioFrameObserver. Passing NULL means unregistering. It is recommended to call this method after receiving OnLeaveChannel to release the audio frame observer object.
mode
Audio data callback mode. See OBSERVER_MODE.
position
Audio observation position.
  • AUDIO_FRAME_POSITION_PLAYBACK (0x0001): Observes the playback audio after mixing from all remote users. Corresponds to the OnPlaybackAudioFrame callback.
  • AUDIO_FRAME_POSITION_RECORD (0x0002): Observes the captured audio of the local user. Corresponds to the OnRecordAudioFrame callback.
  • AUDIO_FRAME_POSITION_MIXED (0x0004): Observes the audio after mixing from the local and all remote users. Corresponds to the OnMixedAudioFrame callback.
  • AUDIO_FRAME_POSITION_BEFORE_MIXING (0x0008): Observes the audio of a single remote user before mixing. Corresponds to the OnPlaybackAudioFrameBeforeMixing callback.
  • AUDIO_FRAME_POSITION_EAR_MONITORING (0x0010): Observes the ear monitoring audio of a single local user. Corresponds to the OnEarMonitoringAudioFrame callback.

Return Values

  • 0: Method call succeeds.
  • < 0: Method call fails. See Error Codes for details and suggested solutions.

RegisterFaceInfoObserver

Registers a face info observer.

public abstract int RegisterFaceInfoObserver(IFaceInfoObserver observer);

You can call this method to register the OnFaceInfo callback to obtain the face information processed by the Agora voice driver extension. When registering the face info observer, you can register the callbacks in the IFaceInfoObserver class as needed. After successful registration, the SDK triggers the registered callback when it captures face information processed by the voice driver extension.

Note:
  • You must call this method before joining a channel.
  • Before calling this method, make sure you have called EnableExtension to enable the voice driver extension.

Scenario

The face information processed by the Agora voice driver plugin conforms to the ARKit standard BS (Blend Shape) data. You can use third-party 3D rendering engines to further process the BS data, such as driving a virtual avatar's facial movements to match speech.

Parameters

observer
Face info observer. See IFaceInfoObserver.

Return Values

  • 0: Method call succeeds.
  • < 0: Method call fails. See Error Codes for details and resolution suggestions.

RegisterVideoEncodedFrameObserver

Registers a video frame receiver observer for encoded video images.

public abstract int RegisterVideoEncodedFrameObserver(IVideoEncodedFrameObserver videoEncodedImageReceiver, OBSERVER_MODE mode = OBSERVER_MODE.INTPTR);

If you only want to observe encoded video frames (such as H.264 format) and do not need to decode and render the video, Agora recommends that you register an IVideoEncodedFrameObserver class using this method.

Note: This method must be called before joining a channel.

Parameters

videoEncodedImageReceiver
Video frame receiver observer. See IVideoEncodedFrameObserver.
mode
Video data callback mode. See OBSERVER_MODE.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

RegisterVideoFrameObserver

Registers a raw video frame observer object.

public abstract int RegisterVideoFrameObserver(IVideoFrameObserver videoFrameObserver, VIDEO_OBSERVER_FRAME_TYPE formatPreference, VIDEO_OBSERVER_POSITION position, OBSERVER_MODE mode = OBSERVER_MODE.INTPTR);

If you want to observe raw video frames (such as YUV or RGBA format), Agora recommends registering an IVideoFrameObserver class using this method. When registering the video observer, you can choose to register the callbacks in the IVideoFrameObserver class as needed. After successful registration, the SDK triggers the registered callbacks whenever a video frame is captured.

Note: When handling callbacks, you need to consider changes in the width and height parameters of the video frame, as the observed video frames may vary due to the following reasons:
  • When the network condition is poor, the resolution may drop in steps.
  • When the user manually adjusts the resolution, the resolution reported in the callback will also change.

Scenario

After registering a raw video frame observer, you can use the raw video data for various video preprocessing scenarios, such as implementing virtual backgrounds or beauty effects.

Timing

This method must be called before joining a channel.

Parameters

videoFrameObserver
Instance of the interface object. See IVideoFrameObserver. Pass NULL to unregister.
mode
Video data callback mode. See OBSERVER_MODE.
formatPreference
Video data type. See VIDEO_OBSERVER_FRAME_TYPE.
position
Bit mask of the video observation position. See VIDEO_MODULE_POSITION.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

SetExternalAudioSink

Sets external audio rendering.

public abstract int SetExternalAudioSink(bool enabled, int sampleRate, int channels);

After calling this method to enable external audio rendering, you can call PullAudioFrame to pull remote audio data. The app can process the pulled raw audio data before rendering to achieve the desired audio effects.

Note: After calling this method to enable external audio rendering, the app will no longer receive data from the OnPlaybackAudioFrame callback.

Scenario

This method is suitable for scenarios where you need to render audio yourself.

Timing

This method must be called before joining a channel.

Parameters

enabled
Whether to enable external audio rendering:
  • true: Enable external audio rendering.
  • false: (Default) Disable external audio rendering.
sampleRate
Sample rate (Hz) for external audio rendering. Can be set to 16000, 32000, 44100, or 48000.
channels
Number of channels for external audio rendering:
  • 1: Mono
  • 2: Stereo

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

SetExternalAudioSource

Sets the external audio capture parameters.

public abstract int SetExternalAudioSource(bool enabled, int sampleRate, int channels, bool localPlayback = false, bool publish = true);
Deprecated
Deprecated: This method is deprecated. Use CreateCustomAudioTrack instead.

Timing

This method must be called before joining a channel.

Parameters

enabled
Whether to enable the use of an external audio source:
  • true: Enable external audio source.
  • false: (Default) Disable external audio source.
sampleRate
The sample rate (Hz) of the external audio source. Can be set to 8000, 16000, 32000, 44100, or 48000.
channels
The number of channels of the external audio source. Can be set to 1 (mono) or 2 (stereo).
localPlayback
Whether to play the external audio source locally:
  • true: Play locally.
  • false: (Default) Do not play locally.
publish
Whether to publish the audio to the remote end:
  • true: (Default) Publish to the remote end.
  • false: Do not publish to the remote end.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

SetExternalVideoSource

Sets the external video source.

public abstract int SetExternalVideoSource(bool enabled, bool useTexture, EXTERNAL_VIDEO_SOURCE_TYPE sourceType, SenderOptions encodedVideoOption);

After calling this method to enable the external video source, you can call PushVideoFrame to push external video data to the SDK.

Note: Switching video sources dynamically in the channel is not supported. If you have called this method to enable the external video source and joined a channel, to switch to an internal video source, you must first leave the channel, then call this method to disable the external video source, and rejoin the channel.

Timing

You need to call this method before joining the channel.

Parameters

enabled
Whether to enable the external video source:
  • true: Enable the external video source. The SDK is ready to receive external video frames.
  • false: (Default) Do not enable the external video source.
useTexture
Whether to use Texture format for external video frames:
  • true: Use Texture format for external video frames.
  • false: Do not use Texture format for external video frames.
sourceType
Whether the external video frames are encoded. See EXTERNAL_VIDEO_SOURCE_TYPE.
encodedVideoOption
Video encoding options. If sourceType is ENCODED_VIDEO_FRAME, this parameter must be set. You can contact technical support to learn how to set this parameter.

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

UnregisterFaceInfoObserver

Unregisters the face info observer.

public abstract int UnRegisterFaceInfoObserver();

Parameters

observer
Face info observer. See IFaceInfoObserver.

Return Values

  • 0: Method call succeeds.
  • < 0: Method call fails. See Error Codes for details and resolution suggestions.

UnregisterVideoEncodedFrameObserver

Unregisters the video frame receiver observer for encoded video images.

public abstract int UnRegisterVideoEncodedFrameObserver();

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.

UnregisterVideoFrameObserver

Unregisters the video frame observer.

public abstract int UnRegisterVideoFrameObserver();

Return Values

  • 0: Success.
  • < 0: Failure. See Error Codes for details and resolution suggestions.