Raw Audio Data

getEarMonitoringAudioParams

Sets the data format for the onEarMonitoringAudioFrame callback.

@CalledByNative public abstract AudioParams getEarMonitoringAudioParams();

When calling registerAudioFrameObserver to register an audio frame observer, you need to implement this callback. The SDK triggers this callback, and you can set the audio format in the return value of this callback.

Note: The SDK triggers the onEarMonitoringAudioFrame callback based on the sampling interval calculation formula you set in the return value. Sampling interval (seconds) = samplePerCall / (sampleRate × channel). Make sure the sampling interval is ≥ 0.01 seconds.

Return Values

Audio format for ear monitoring. See AudioParams.

getMixedAudioParams

Sets the data format for the onMixedAudioFrame callback.

@CalledByNative public abstract AudioParams getMixedAudioParams();

When calling registerAudioFrameObserver to register the audio frame observer, you need to implement this callback. After successfully registering the audio frame observer, the SDK triggers this callback, and you can set the audio format in the return value of this callback.

Note: The SDK triggers the onMixedAudioFrame callback based on the sampling interval calculation formula set in the return value. Sampling interval (seconds) = samplePerCall / (sampleRate × channel). Make sure the sampling interval is ≥ 0.01 seconds.

Return Values

Returns an AudioParams object upon successful method call. See AudioParams.

getObservedAudioFramePosition

Sets the audio data observation position.

@CalledByNative public abstract int getObservedAudioFramePosition();

After successfully registering the audio frame observer, the SDK calls this callback at each specific audio frame processing node to determine whether to trigger the following callbacks:

You can set one or more positions to observe by modifying the return value of getObservedAudioFramePosition according to your scenario needs.To observe multiple positions, use the | (bitwise OR) operator. To conserve system resources, it is recommended to minimize the number of observed frame positions.

Return Values

Returns a bitmask to set observation positions. The possible values are:

POSITION_PLAYBACK (0x0001): Observes the playback audio after mixing from all remote users, corresponding to the onPlaybackAudioFrame callback.
POSITION_RECORD (0x0002): Observes the audio captured by the local user, corresponding to the onRecordAudioFrame callback.
POSITION_MIXED (0x0004): Observes the playback audio after mixing the local user and all remote users, corresponding to the onMixedAudioFrame callback.
POSITION_BEFORE_MIXING (0x0008): Observes the audio of a single remote user before mixing, corresponding to the onPlaybackAudioFrameBeforeMixing callback.
POSITION_EAR_MONITORING (0x0010): Observes the local user's in-ear monitoring audio, corresponding to the onEarMonitoringAudioFrame callback.

getPlaybackAudioParams

Sets the data format for the onPlaybackAudioFrame callback.

public abstract AudioParams getPlaybackAudioParams()

You need to register the callback when calling the registerAudioFrameObserver method. After successfully registering the audio observer, the SDK triggers this callback. You can set the audio format in the return value of this callback.

Note: The SDK calculates the sampling interval based on the AudioParams you set in the return value and triggers the onPlaybackAudioFrame callback accordingly. The formula for calculating the sampling interval is: Sampling interval (seconds) = samplePerCall / (sampleRate × channel). Make sure the sampling interval is ≥ 0.01 seconds.

Return Values

Returns an AudioParams object upon successful method call. See AudioParams.

getRecordAudioParams

Sets the audio format for the onRecordAudioFrame callback.

@CalledByNative public abstract AudioParams getRecordAudioParams();

After calling registerAudioFrameObserver to register the callback, the SDK triggers this callback. You can set the audio format in the return value of this callback.

Note: The SDK calculates the sampling interval and triggers the onRecordAudioFrame callback based on the AudioParams you set in the return value. Sampling interval (seconds) = samplePerCall / (sampleRate × channel). Make sure the sampling interval is ≥ 0.01 seconds.

Return Values

If the method call succeeds, returns an AudioParams object.

registerAudioFrameObserver

Registers an audio frame observer.

public abstract int registerAudioFrameObserver(IAudioFrameObserver observer);

After calling this method to register an audio frame observer, the SDK triggers the onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, or onEarMonitoringAudioFrame callback.

Timing

Call this method before joining a channel.

Related Callbacks

Parameters

observer: The audio frame observer instance. See IAudioFrameObserver. Set to null to release the instance. It is recommended to call this method after receiving the onLeaveChannel callback to release the audio frame observer.

Return Values

0: Success.
< 0: Failure.

setEarMonitoringAudioFrameParameters

Sets the format of the ear monitoring audio data.

public abstract int setEarMonitoringAudioFrameParameters(int sampleRate, int channel, int mode, int samplesPerCall);

This method sets the format of the ear monitoring audio data reported in the onEarMonitoringAudioFrame callback.

Note:

Before calling this method, you must call enableInEarMonitoring and set includeAudioFilters to EAR_MONITORING_FILTER_BUILT_IN_AUDIO_FILTERS or EAR_MONITORING_FILTER_NOISE_SUPPRESSION.
The SDK calculates the sampling interval based on the samplesPerCall, sampleRate, and channel parameters. Sampling interval (seconds) = samplesPerCall / (sampleRate × channel). Ensure the sampling interval is ≥ 0.01 seconds. The SDK triggers the onEarMonitoringAudioFrame callback based on the sampling interval.

Parameters

sampleRate

The sampling rate reported in the onEarMonitoringAudioFrame callback. Can be set to 8000, 16000, 32000, 44100, or 48000 Hz.

channel

The number of audio channels reported in the onEarMonitoringAudioFrame callback:

1: Mono.
2: Stereo.

mode

Usage mode of the audio frame:

RAW_AUDIO_FRAME_OP_MODE_READ_ONLY (0): (Default) Read-only mode. For example, the user retrieves data via the Agora SDK and pushes RTMP or RTMPS streams.
RAW_AUDIO_FRAME_OP_MODE_READ_WRITE (2): Read-write mode. The user reads data from AudioFrame, modifies it, and plays it back. For example, the user has a custom audio effects module and performs voice preprocessing such as voice changing.

samplesPerCall

The number of samples per call reported in the onEarMonitoringAudioFrame callback, e.g., 1024 for media streaming.

Return Values

0: Success.
< 0: Failure.

setMixedAudioFrameParameters

Sets the format of the mixed raw audio data used for capture and playback.

public abstract int setMixedAudioFrameParameters(int sampleRate, int channel, int samplesPerCall);

The SDK calculates the sampling interval based on the samplesPerCall, sampleRate, and channel parameters. Sampling interval (seconds) = samplesPerCall / (sampleRate × channel). Ensure the sampling interval is ≥ 0.01 seconds. The SDK triggers the onMixedAudioFrame callback based on the sampling interval.

Timing

Call this method before joining a channel.

Parameters

sampleRate

The sampling rate returned in the callback. Can be set to 8000, 16000, 32000, 44100, or 48000 Hz.

channel

Number of audio channels. Can be set to:

1: Mono.
2: Stereo.

samplesPerCall

Number of samples per callback, e.g., 1024 for media streaming.

Return Values

0: Success.
< 0: Failure.

setPlaybackAudioFrameBeforeMixingParameters [1/2]

Sets the format of raw audio playback data before audio mixing.

public abstract int setPlaybackAudioFrameBeforeMixingParameters(int sampleRate, int channel);

The SDK triggers the onPlaybackAudioFrameBeforeMixing callback based on the sampling interval.

Timing

Call this method before joining a channel.

Related Callbacks

Related callback: onPlaybackAudioFrameBeforeMixing.

Parameters

sampleRate

The sample rate returned in the callback. You can set it to:

8000: 8 kHz.
16000: 16 kHz.
32000: 32 kHz.
44100: 44.1 kHz.
48000: 48 kHz.

channel

The number of audio channels. You can set it to:

1: Mono.
2: Stereo.

Return Values

0: Success.
< 0: Failure.

setPlaybackAudioFrameBeforeMixingParameters [2/2]

Sets the format of audio data returned in the onPlaybackAudioFrameBeforeMixing callback.

public abstract int setPlaybackAudioFrameBeforeMixingParameters(int sampleRate, int channel, int samplesPerCall);

Parameters

sampleRate

Sets the sample rate returned in the onPlaybackAudioFrameBeforeMixing callback. You can set it to (unit: Hz):

8000
16000
32000
44100
48000

channel

Sets the number of audio channels returned in the onPlaybackAudioFrameBeforeMixing callback. You can set it to:

1: Mono.
2: Stereo.

samplesPerCall

Sets the number of samples returned in the onMixedAudioFrame callback.

Note: In RTMP streaming scenarios, it is recommended to set this to 1024.

Return Values

0: Success.
< 0: Failure.

setPlaybackAudioFrameParameters

Sets the format of raw audio playback data.

public abstract int setPlaybackAudioFrameParameters(int sampleRate, int channel, int mode, int samplesPerCall);

The SDK calculates the sampling interval based on the samplesPerCall, sampleRate, and channel parameters. Sampling interval (in seconds) = samplesPerCall / (sampleRate × channel). Ensure that the sampling interval is ≥ 0.01 seconds. The SDK triggers the onPlaybackAudioFrame callback based on the sampling interval.

Timing

Call this method before joining a channel.

Related Callbacks

This method triggers the onPlaybackAudioFrame callback based on the sampling interval.

Parameters

sampleRate

Sampling rate returned in the callback. Can be set to 8000, 16000, 24000, 32000, 44100, or 48000 Hz.

channel

Number of audio channels. Can be set to:

1: Mono.
2: Stereo.

mode

Usage mode of the audio frame:

RAW_AUDIO_FRAME_OP_MODE_READ_ONLY (0): (Default) Read-only mode. For example, the user obtains data through the Agora SDK and pushes RTMP or RTMPS streams.
RAW_AUDIO_FRAME_OP_MODE_READ_WRITE (2): Read-write mode. The user reads data from AudioFrame, modifies it, and then plays it. For example, the user has their own audio effect processing module and performs voice preprocessing, such as voice changing.

samplesPerCall

Number of data samples returned per call, e.g., 1024 for media streaming.

Return Values

0: Success.
< 0: Failure.

setRecordingAudioFrameParameters

Sets the format of the captured raw audio data.

public abstract int setRecordingAudioFrameParameters(int sampleRate, int channel, int mode, int samplesPerCall);

The SDK calculates the sampling interval based on the samplesPerCall, sampleRate, and channel parameters. Sampling interval (in seconds) = samplesPerCall / (sampleRate × channel). Ensure the sampling interval is ≥ 0.01 seconds. The SDK triggers the onRecordAudioFrame callback based on the sampling interval.

Timing

Call this method before joining a channel.

Parameters

sampleRate

The sampling rate returned in the callback. Can be set to 8000, 16000, 32000, 44100, or 48000 Hz.

channel

Number of audio channels:

1: Mono.
2: Stereo.

mode

Usage mode of the audio frame:

0: Read-only mode. For example, the user retrieves data via the Agora SDK and pushes RTMP or RTMPS streams.
2: Read-write mode. The user reads data from AudioFrame, modifies it, and plays it back. For example, the user has a custom audio effects module and performs voice preprocessing such as voice changing.

samplesPerCall

Number of samples per call, e.g., 1024 for media streaming.

Return Values

0: Success.
< 0: Failure.

onEarMonitoringAudioFrame

Callback for ear monitoring audio frames.

public abstract boolean onEarMonitoringAudioFrame(int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type);

This callback is used to retrieve ear monitoring audio data. To ensure the retrieved data meets expectations, Agora recommends setting the audio data format using one of the following methods:

Method 1: Call setEarMonitoringAudioFrameParameters to set the audio data format, then call registerAudioFrameObserver to register the audio frame observer. The SDK will calculate the sampling interval based on the set parameters and trigger the onEarMonitoringAudioFrame callback accordingly.
Method 2: After calling registerAudioFrameObserver to register the audio frame observer, set the audio data format in the return value of getObservedAudioFramePosition. The SDK will then calculate the sampling interval based on the return value of getEarMonitoringAudioParams and trigger the onEarMonitoringAudioFrame callback accordingly.

Note: Method 1 takes precedence over Method 2. If you use Method 1 to set the audio data format, the settings in Method 2 will be ignored.

Parameters

type

Audio frame type.

samplesPerChannel

Number of samples per channel.

bytesPerSample

Number of bytes per audio sample. For example, each PCM audio sample typically occupies 16 bits (2 bytes).

channels

Number of channels:

1: Mono.
2: Stereo. If stereo, the data is in interleaved format.

samplesPerSec

Recording sample rate (Hz).

buffer

Audio buffer. Buffer size = samplesPerChannel × channels × bytesPerSample. See ByteBuffer.

renderTimeMs

Timestamp of the external audio frame (milliseconds). You can use this parameter to synchronize audio and video frames in scenarios such as using custom video sources.

avsync_type

Reserved parameter.

Return Values

true: Callback handled successfully.
false: Callback handling failed.

onMixedAudioFrame

Callback for retrieving the audio frame mixed from capture and playback.

public abstract boolean onMixedAudioFrame(String channelId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type);

Agora recommends setting the audio data format in one of the following two ways to ensure the format of the mixed audio frame from capture and playback meets expectations:

Method 1: Call setMixedAudioFrameParameters to set the audio data format, and then call registerAudioFrameObserver to register the audio frame observer. The SDK calculates the sampling interval based on the parameters and triggers this callback accordingly.
Method 2: Call registerAudioFrameObserver to register the audio frame observer, and then set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK calculates the sampling interval based on the return value of the getMixedAudioParams callback and triggers this callback accordingly.

Note: The settings of setMixedAudioFrameParameters take precedence over getObservedAudioFramePosition. If you use Method 1 to set the audio data format, the settings in Method 2 will be ignored.

Parameters

channelId

Channel ID.

type

Audio frame type.

samplesPerChannel

Number of samples per channel.

bytesPerSample

Number of bytes per audio sample. For example, each PCM audio sample typically occupies 16 bits (2 bytes).

channels

Number of channels:

1: Mono.
2: Stereo. When using stereo, the data is in interleaved format.

samplesPerSec

Recording sample rate (Hz).

buffer

Audio buffer. Buffer size = samplesPerChannel × channels × bytesPerSample. See ByteBuffer.

renderTimeMs

Timestamp of the external audio frame (in milliseconds). You can use this parameter to synchronize audio and video frames in scenarios involving video or audio (including scenarios with external video sources).

avsync_type

Reserved parameter.

Return Values

true: Callback handled successfully.
false: Callback handling failed.

onPlaybackAudioFrame

Callback for retrieving the raw audio frame for playback.

public abstract boolean onPlaybackAudioFrame(String channelId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type);

You can set the audio data format in one of the following two ways:

Method 1: Call setPlaybackAudioFrameParameters to set the audio data format, and then call registerAudioFrameObserver to register the audio frame observer. The SDK calculates the sampling interval based on the parameters and triggers this callback accordingly.
Method 2: Call registerAudioFrameObserver to register the audio frame observer, and then set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK calculates the sampling interval based on the return value of the getPlaybackAudioParams callback and triggers this callback accordingly.

Note: The settings of setPlaybackAudioFrameParameters take precedence over getObservedAudioFramePosition. If you use Method 1 to set the audio data format, the settings in Method 2 will be ignored.

Parameters

channelId

Channel ID.

type

Audio frame type.

samplesPerChannel

Number of samples per channel.

bytesPerSample

Number of bytes per audio sample. For example, each PCM audio sample typically occupies 16 bits (2 bytes).

channels

Number of channels:

1: Mono.
2: Stereo. For stereo, the data is interleaved.

samplesPerSec

Recording sample rate (Hz).

buffer

Audio buffer. Buffer size = samplesPerChannel × channels × bytesPerSample.

renderTimeMs

Timestamp of the external audio frame (in milliseconds). You can use this parameter to synchronize audio and video frames in scenarios involving video or audio (including scenarios with external video sources).

avsync_type

Reserved parameter.

Return Values

true: Callback handled successfully.
false: Callback handling failed.

onPlaybackAudioFrameBeforeMixing

Callback for retrieving audio frames of subscribed remote users before audio mixing.

public abstract boolean onPlaybackAudioFrameBeforeMixing(String channelId, int uid, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type, int rtpTimestamp, long presentationMs);

Trigger Timing

This callback is triggered after retrieving the remote user's audio frame and before mixing.

Parameters

channelId

Channel ID.

uid

User ID of the subscribed remote user.

type

Audio frame type.

samplesPerChannel

Number of samples per channel.

bytesPerSample

Number of bytes per audio sample. For example, each PCM audio sample typically occupies 16 bits (2 bytes).

channels

Number of channels:

1: Mono.
2: Stereo. Stereo data is interleaved.

samplesPerSec

Recording sample rate (Hz).

buffer

Audio buffer. Buffer size = samplesPerChannel × channels × bytesPerSample.

renderTimeMs

Timestamp of the external audio frame (milliseconds). You can use this parameter to synchronize audio and video in scenarios such as using custom video sources.

avsync_type

Reserved parameter.

rtpTimestamp

RTP timestamp.

presentationMs

Presentation timestamp (milliseconds).

Return Values

true: Processing succeeded.
false: Processing failed.

onRecordAudioFrame

Callback for retrieving the captured audio frame.

public abstract boolean onRecordAudioFrame(String channelId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type);

You can set the audio data format using one of the following methods:

Method 1: Call setRecordingAudioFrameParameters to set the audio data format, and then call registerAudioFrameObserver to register the audio frame observer. The SDK calculates the sampling interval based on the parameters set in the method and triggers this callback accordingly.
Method 2: Call registerAudioFrameObserver to register the audio frame observer, and set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK calculates the sampling interval based on the return value of the getRecordAudioParams callback and triggers this callback accordingly.

Note: The settings in setRecordingAudioFrameParameters take precedence over those in getObservedAudioFramePosition. If you set the audio data format using Method 1, the settings in Method 2 will be ignored.

Parameters

channelId

Channel ID.

type

Audio frame type.

samplesPerChannel

Number of samples per channel.

bytesPerSample

Number of bytes per audio sample. For example, each PCM audio sample typically takes 16 bits (2 bytes).

channels

Number of channels:

1: Mono.
2: Stereo. If stereo is used, the data is interleaved.

samplesPerSec

Recording sample rate (Hz).

buffer

Audio buffer. Buffer size = samplesPerChannel × channels × bytesPerSample.

renderTimeMs

Timestamp of the external audio frame (milliseconds). You can use this parameter to synchronize audio and video frames in scenarios such as using external video sources.

avsync_type

Reserved parameter.

Return Values

true: Callback handled successfully.
false: Callback handling failed.