IMediaEngine

createCustomAudioTrack

Creates a custom audio track.

public abstract int createCustomAudioTrack(
      Constants.AudioTrackType trackType, AudioTrackConfig config);

Details

Since: v4.2.0

Attention: Call this method before joining a channel.

To publish a custom audio source, see the following steps:

Call this method to create a custom audio track and get the audio track ID.
Call joinChannel [2/2] to join the channel. In ChannelMediaOptions, set publishCustomAudioTrackId to the audio track ID that you want to publish, and set publishCustomAudioTrack to true.
Call pushExternalAudioFrame and specify trackId as the audio track ID set in step 2. You can then publish the corresponding custom audio source in the channel.

Parameters

trackType: The type of the custom audio track. See AudioTrackType.
Attention: If AUDIO_TRACK_DIRECT is specified for this parameter, you must set publishMicrophoneTrack to false in ChannelMediaOptions when calling joinChannel [2/2] to join the channel; otherwise, joining the channel fails and returns the error code -2.
config: The configuration of the custom audio track. See AudioTrackConfig.

Returns

If the method call is successful, the audio track ID is returned as the unique identifier of the audio track.
If the method call fails, 0xffffffff is returned.

destroyCustomAudioTrack

Destroys the specified audio track.

public abstract int destroyCustomAudioTrack(int trackId);

Details

Since: v4.2.0

Parameters

trackId: The custom audio track ID returned in createCustomAudioTrack.

Returns

0: Success.
< 0: Failure.

pullPlaybackAudioFrame [1/2]

Pulls the remote audio data.

public abstract int pullPlaybackAudioFrame(byte[] data, int lengthInByte);

After a successful call of this method, the app pulls the decoded and mixed audio data for playback.

Call timing

Call this method after joining a channel.

Before calling this method, call setExternalAudioSink(enabled: true) to notify the app to enable and set the external audio rendering.

Restrictions

Both this method and the onPlaybackAudioFrame callback can be used to get audio data after remote mixing. After calling setExternalAudioSink to enable external audio rendering, the app will no longer be able to obtain data from the onPlaybackAudioFrame callback. Therefore, you should choose between this method and the onPlaybackAudioFrame callback based on your actual business requirements. The specific distinctions between them are as follows:

After calling this method, the app automatically pulls the audio data from the SDK. By setting the audio data parameters, the SDK adjusts the frame buffer to help the app handle latency, effectively avoiding audio playback jitter.
After registering the onPlaybackAudioFrame callback, the SDK sends the audio data to the app through the callback. Any delay in processing the audio frames may result in audio jitter.

This method is only used for retrieving audio data after remote mixing. If you need to get audio data from different audio processing stages such as capture and playback, you can register the corresponding callbacks by calling registerAudioFrameObserver.

Parameters

data: The remote audio data to be pulled. The data type is byte[].
lengthInByte: The data length (byte). The value of this parameter is related to the audio duration, and the values of the sampleRate and channels parameters that you set in setExternalAudioSink. lengthInByte = sampleRate/1000 × 2 × channels × audio duration (ms).

Returns

0: Success.
< 0: Failure.

pullPlaybackAudioFrame [2/2]

Pulls the remote audio data.

public abstract int pullPlaybackAudioFrame(ByteBuffer data, int lengthInByte);

Details

Before calling this method, call the setExternalAudioSink(enabled: true) method to notify the app to enable and set the external audio sink.

After a successful method call, the app pulls the decoded and mixed audio data for playback.

Attention:

Call this method after joining a channel.
The difference between this method and the onPlaybackAudioFrame callback is as follows:
- onPlaybackAudioFrame: The SDK sends the audio data to the app through this callback. Any delay in processing the audio frames may result in audio jitter.
- pullPlaybackAudioFrame [1/2]: The app pulls the remote audio data. After setting the audio data parameters, the SDK adjusts the frame buffer and avoids problems caused by jitter in the external audio playback.

Parameters

data: The remote audio data to be pulled. The data type is ByteBuffer.
lengthInByte: The length (in bytes) of the remote audio data. The value of this parameter is related to the audio duration,and the values of the sampleRate and channels parameters that you set in setExternalAudioSink. lengthInByte = sampleRate/1000 × 2 × channels × audio duration (ms).

Returns

0: Success.
< 0: Failure.

pushExternalAudioFrame

Pushes the external audio frame to the SDK.

public abstract int pushExternalAudioFrame(byte[] data, long timestamp, int sampleRate,
      int channels, Constants.BytesPerSample bytesPerSample, int trackId);

Call this method to push external audio frames through the audio track.

Call timing

Before calling this method to push external audio data, perform the following steps:

Call createCustomAudioTrack to create a custom audio track and get the audio track ID.
Call joinChannel [2/2] to join the channel. In ChannelMediaOptions, set publishCustomAudioTrackId to the audio track ID that you want to publish, and set publishCustomAudioTrack to true.

Restrictions

None.

Parameters

data: The external audio data.
timestamp: The timestamp (ms) of the external audio frame. This parameter is required. You can use it to restore the order of the captured audio frames, or synchronize audio and video frames in video-related scenarios (including scenarios where external video sources are used).
sampleRate: The sample rate (Hz) of the external audio source which can be set as 8000, 16000, 32000, 44100, or 48000.
channels: The number of channels of the external audio source, which can be set as 1 (Mono) or 2 (Stereo).
bytesPerSample: The number of bytes per sample. For PCM, this parameter is generally set to 16 bits (2 bytes).
trackId: The audio track ID. Set this parameter to the custom audio track ID returned in createCustomAudioTrack.

Returns

0: Success.
< 0: Failure.

registerFaceInfoObserver

Registers or unregisters a facial information observer.

public abstract int registerFaceInfoObserver(IFaceInfoObserver receiver);

Details

Since: v4.3.1

You can call this method to register the onFaceInfo callback to receive the facial information processed by Agora speech driven extension. When calling this method to register a facial information observer, you can register callbacks in the IFaceInfoObserver class as needed. After successfully registering the facial information observer, the SDK triggers the callback you have registered when it captures the facial information converted by the speech driven extension.

Attention:

Call this method before joining a channel.
Before calling this method, you need to make sure that the speech driven extension has been enabled by calling enableExtension.

Applicable scenarios

Facial information processed by the Agora speech driven extension is BS (Blend Shape) data that complies with ARkit standards. You can further process the BS data using third-party 3D rendering engines, such as driving avatar to make mouth movements corresponding to speech.

Parameters

observer: Facial information observer, see IFaceInfoObserver. If you need to unregister a facial information observer, pass in NULL.

Returns

0: Success.
< 0: Failure.

registerAudioFrameObserver

Registers an audio frame observer object.

public abstract int registerAudioFrameObserver(IAudioFrameObserver observer);

Call this method to register an audio frame observer object (register a callback). When you need the SDK to trigger the onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing or onEarMonitoringAudioFrame callback, you need to use this method to register the callbacks.

Call timing

Call this method before joining a channel.

Restrictions

None.

Parameters

observer: The observer instance. See IAudioFrameObserver.Set the value as NULL to release the instance. Agora recommends calling this method after receiving onLeaveChannel to release the audio observer object.

Returns

0: Success.
< 0: Failure.

registerFaceInfoObserver

Registers or unregisters a facial information observer.

public abstract int registerFaceInfoObserver(IFaceInfoObserver receiver);

Details

Since: v4.3.1

You can call this method to register the onFaceInfo callback to receive the facial information processed by Agora speech driven extension. When calling this method to register a facial information observer, you can register callbacks in the IFaceInfoObserver class as needed. After successfully registering the facial information observer, the SDK triggers the callback you have registered when it captures the facial information converted by the speech driven extension.

Attention:

Call this method before joining a channel.
Before calling this method, you need to make sure that the speech driven extension has been enabled by calling enableExtension.

Applicable scenarios

Facial information processed by the Agora speech driven extension is BS (Blend Shape) data that complies with ARkit standards. You can further process the BS data using third-party 3D rendering engines, such as driving avatar to make mouth movements corresponding to speech.

Parameters

observer: Facial information observer, see IFaceInfoObserver. If you need to unregister a facial information observer, pass in NULL.

Returns

0: Success.
< 0: Failure.

registerVideoEncodedFrameObserver

Registers a receiver object for the encoded video image.

public abstract int registerVideoEncodedFrameObserver(IVideoEncodedFrameObserver receiver);

Details

If you only want to observe encoded video frames (such as H.264 format) without decoding and rendering the video, Agora recommends that you implement one IVideoEncodedFrameObserver class through this method. For detailed implementation steps, please refer to .

Attention:

Call this method before joining a channel.

Parameters

receiver: The video frame observer object. See IVideoEncodedFrameObserver.

Returns

0: Success.
< 0: Failure.

registerVideoFrameObserver

Registers a raw video frame observer object.

public abstract int registerVideoFrameObserver(IVideoFrameObserver observer);

If you want to observe raw video frames (such as YUV or RGBA format), Agora recommends that you implement one IVideoFrameObserver class with this method.

When calling this method to register a video observer, you can register callbacks in the IVideoFrameObserver class as needed. After you successfully register the video frame observer, the SDK triggers the registered callbacks each time a video frame is received.

Applicable scenarios

After registering the raw video observer, you can use the obtained raw video data in various video pre-processing scenarios, such as virtual backgrounds and image enhacement by yourself. For detailed implementation steps, please refer to .

Agora provides an open source sample project beautyapi on GitHub for your reference.

Call timing

Call this method before joining a channel.

Restrictions

When handling the video data returned in the callbacks, pay attention to the changes in the width and height parameters, which may be adapted under the following circumstances:

When network conditions deteriorate, the video resolution decreases incrementally.
If the user adjusts the video profile, the resolution of the video returned in the callbacks also changes.

Parameters

observer: The observer instance. See IVideoFrameObserver.To release the instance, set the value as NULL.

Returns

0: Success.
< 0: Failure.

setExternalAudioSink

Sets the external audio sink.

public abstract int setExternalAudioSink(boolean enabled, int sampleRate, int channels);

After enabling the external audio sink, you can call pullPlaybackAudioFrame [1/2] to pull remote audio frames. The app can process the remote audio and play it with the audio effects that you want.

Applicable scenarios

This method applies to scenarios where you want to use external audio data for playback.

Call timing

Call this method before joining a channel.

Restrictions

Once you enable the external audio sink, the app will not retrieve any audio data from the onPlaybackAudioFrame callback.

Parameters

enabled

Whether to enable or disable the external audio sink:

true: Enables the external audio sink.
false: (Default) Disables the external audio sink.

sampleRate

The sample rate (Hz) of the external audio sink, which can be set as 16000, 32000, 44100, or 48000.

channels

The number of audio channels of the external audio sink:

1: Mono.
2: Stereo.

Returns

0: Success.
< 0: Failure.

setExternalAudioSource [2/2]

Sets the external audio source parameters.

public abstract int setExternalAudioSource(boolean enabled, int sampleRate, int channels, boolean localPlayback, boolean publish);

Deprecated:: This method is deprecated, use createCustomAudioTrack instead.

Call timing

Call this method before joining a channel.

Restrictions

None.

Parameters

enabled

Whether to enable the external audio source:

true: Enable the external audio source.
false: (Default) Disable the external audio source.

sampleRate

The sample rate (Hz) of the external audio source which can be set as 8000, 16000, 32000, 44100, or 48000.

channels

The number of channels of the external audio source, which can be set as 1 (Mono) or 2 (Stereo).

localPlayback

Whether to play the external audio source:

true: Play the external audio source.
false: (Default) Do not play the external source.

publish

Whether to publish audio to the remote users:

true: (Default) Publish audio to the remote users.
false: Do not publish audio to the remote users.

Returns

0: Success.
< 0: Failure.

setExternalVideoSource

Configures the external video source.

public abstract int setExternalVideoSource(
    boolean enable, boolean useTexture, Constants.ExternalVideoSourceType sourceType);

After calling this method to enable an external video source, you can call pushExternalVideoFrameById [1/2] to push external video data to the SDK.

Call timing

Call this method before joining a channel.

Restrictions

Dynamic switching of video sources is not supported within the channel. To switch from an external video source to an internal video source, you must first leave the channel, call this method to disable the external video source, and then rejoin the channel.

Parameters

enable

Whether to use the external video source:

true: Use the external video source. The SDK prepares to accept the external video frame.
false: (Default) Do not use the external video source.

useTexture

Whether to use the external video frame in the Texture format.

true: Use the external video frame in the Texture format.
false: (Default) Do not use the external video frame in the Texture format.

sourceType

Whether the external video frame is encoded. See ExternalVideoSourceType.

Returns

0: Success.
< 0: Failure.