Raw Video Data

getMirrorApplied

Callback triggered each time the SDK receives a video frame, used to determine whether to mirror the captured video.

@CalledByNative boolean getMirrorApplied();

If you want the obtained video data to be a mirrored image of the original video, you need to register this callback when calling registerVideoFrameObserver. After successfully registering the video frame observer, the SDK triggers this callback each time it receives a video frame. You must set whether to mirror the video frame in the return value of this callback.

Note:
  • On Android, this callback supports the following video data formats: I420, RGBA, and Texture.
  • This method and setVideoEncoderConfiguration both support setting mirror effects. It is recommended to use only one method. Using both may result in overlapping mirror effects, causing the setting to fail.

Timing

This callback is triggered each time the SDK receives a video frame.

Return Values

Whether to mirror the captured video:
  • true: Mirror the captured video.
  • false: (default) Do not mirror the captured video.

getObservedFramePosition

Sets the frame position for the video observer.

@CalledByNative int getObservedFramePosition();
After successfully registering the video data observer, the SDK uses this callback to determine whether to trigger the onCaptureVideoFrame, onRenderVideoFrame, and onPreEncodeVideoFrame callbacks at specific video frame processing positions. You can set one or more positions to observe by modifying the return value based on your business scenario:
  • VIDEO_MODULE_POSITION_POST_CAPTURER (1 << 0): After video capture, corresponds to onCaptureVideoFrame callback.
  • VIDEO_MODULE_POSITION_PRE_RENDERER (1 << 1): Before rendering remote video, corresponds to onRenderVideoFrame callback.
  • VIDEO_MODULE_POSITION_PRE_ENCODER (1 << 2): Before encoding, corresponds to onPreEncodeVideoFrame callback.

Return Values

If the method call succeeds, returns the bitmask used to control the frame position for the video observer:
  • VIDEO_MODULE_POSITION_POST_CAPTURER (1 << 0): After video capture.
  • VIDEO_MODULE_POSITION_PRE_RENDERER (1 << 1): Before rendering remote video.
  • VIDEO_MODULE_POSITION_PRE_ENCODER (1 << 2): Before encoding.

getRotationApplied

getRotationApplied callback is triggered each time the SDK receives a video frame, used to determine whether to rotate the captured video.

@CalledByNative boolean getRotationApplied();

If you want to rotate the captured video based on the rotation member in VideoFrame, you need to register this callback when calling registerVideoFrameObserver. After successful registration, the SDK triggers this callback each time it receives a video frame. You must set whether to rotate the video frame in the return value of this callback.

Note: On Android, this callback supports the following video data formats: I420, RGBA, and Texture.

Timing

This callback is triggered each time the SDK receives a video frame.

Return Values

  • true: Rotate the captured video.
  • false: (default) Do not rotate the captured video.

getVideoFormatPreference

Sets the format of the raw video data output by the SDK.

@CalledByNative int getVideoFormatPreference();

After calling registerVideoFrameObserver to register the callback, the SDK triggers this callback each time it receives a video frame. You need to set the expected video data format in the return value of this callback.

Note: VIDEO_PIXEL_DEFAULT indicates the default pixel format of the raw video, as follows:
  • On Android, the default video frame type may be I420Buffer or TextureBuffer. The texture format of TextureBuffer may be OES or RGB. If the video frame type returned by getVideoFormatPreference is VIDEO_PIXEL_DEFAULT, you must handle both I420Buffer and TextureBuffer.
Cases where the video frame type is fixed to I420Buffer include but are not limited to:
  • Specific devices, such as: LG G5 SE (H848), Google Pixel 4a, Samsung Galaxy A7, or Xiaomi Max.
  • When beauty effect extension is integrated and video denoising or low light enhancement is enabled.

Return Values

If the method call succeeds, returns the raw data format output by the SDK:
  • VIDEO_PIXEL_DEFAULT (0): Original video format, must support I420Buffer or TextureBuffer.
  • VIDEO_PIXEL_I420 (1): I420 format.
  • VIDEO_PIXEL_RGBA (4): RGBA format.
  • VIDEO_PIXEL_I422 (16): I422 format.

getVideoFrameProcessMode

getVideoFrameProcessMode callback. Triggered each time the SDK receives a video frame, prompting you to set the processing mode.

@CalledByNative int getVideoFrameProcessMode();

After successfully registering a video frame observer, the SDK triggers the getVideoFrameProcessMode callback each time it receives a video frame. You need to set the preferred processing mode in the return value of this callback.

Return Values

Video frame processing mode:
  • PROCESS_MODE_READ_ONLY (0): (default) Read-only.
  • PROCESS_MODE_READ_WRITE (1): Read and write.

registerVideoFrameObserver

Registers the raw video frame observer object.

public abstract int registerVideoFrameObserver(IVideoFrameObserver observer);

It is recommended to register an IVideoFrameObserver instance using this method and implement the callbacks as needed. After successful registration, the SDK triggers the corresponding callbacks whenever a video frame is received. You can use this raw video data in video preprocessing scenarios such as virtual background and beauty effects.

Note: When processing the video data returned in the callbacks, note that the width and height parameters may change under the following conditions:
  • When network conditions deteriorate, the video resolution will gradually decrease.
  • When the user adjusts the video configuration, the resolution in the callback will also change accordingly.

Timing

Call this method before joining a channel.

Parameters

observer
Observer instance. See IVideoFrameObserver.
Note: To release this instance, set this parameter to null.

Return Values

  • 0: Success.
  • < 0: Failure.

onCaptureVideoFrame

Callback triggered when a video frame is captured by the local device.

@CalledByNative boolean onCaptureVideoFrame(int sourceType, VideoFrame videoFrame);

You can use this callback to obtain the raw video data captured by the local device and preprocess it as needed. After preprocessing, you can directly modify the videoFrame in this callback and return true to send the modified video data to the SDK. If you want to send the preprocessed video data to the SDK, you must first call getVideoFrameProcessMode to set the video processing mode to read-write (PROCESS_MODE_READ_WRITE).

Note:
  • If the obtained video data type is RGBA, the SDK does not support processing alpha channel data.
  • It is recommended to ensure that the modified parameters in videoFrame match the actual video frame in the buffer. Otherwise, issues such as abnormal rotation or stretching may occur in local preview and remote video display.
  • The default video format obtained through this callback may be I420Buffer or TextureBuffer. The texture format of TextureBuffer may be OES or RGB. If you need other video formats, you can set the desired data format in the return value of the getVideoFormatPreference callback.

Trigger Timing

This callback is triggered each time the SDK captures a video frame after successfully registering the video data observer.

Parameters

sourceType
Video source type, including camera, screen, or media player. See VideoSourceType.
videoFrame
Video frame.
Note: The default format of the video frame data obtained through this callback is as follows:
  • Android: I420 or RGB (GLES20.GL_TEXTURE_2D)

Return Values

  • When the video processing mode is PROCESS_MODE_READ_ONLY:
    • true: Retain for subsequent use.
    • false: Retain for subsequent use.
  • When the video processing mode is PROCESS_MODE_READ_WRITE:
    • true: The SDK accepts the video frame.
    • false: The SDK discards the video frame.

onPreEncodeVideoFrame

Callback triggered each time the SDK receives a video frame before encoding.

@CalledByNative boolean onPreEncodeVideoFrame(int sourceType, VideoFrame videoFrame);

After you successfully register a video frame observer, the SDK triggers this callback each time it receives a video frame. In this callback, you can obtain the video data before encoding and process it according to your specific scenario. After processing, you can return the processed video data to the SDK.

Note:
  • If you want to send the preprocessed video data back to the SDK, you must first call getVideoFrameProcessMode to set the video processing mode to read-write (PROCESS_MODE_READ_WRITE).
  • To obtain pre-encoded video data from the second screen capture, you need to set the frame position to VIDEO_MODULE_POSITION_PRE_ENCODER (1 << 2) via getObservedFramePosition.
  • The video data obtained from this callback has already been preprocessed, including cropping, rotation, and beauty effects.
  • It is recommended to ensure that the modified parameters in videoFrame match the actual video frame in the buffer. Otherwise, issues such as abnormal rotation or image stretching may occur in local preview and remote video display.

Trigger Timing

This callback is triggered each time the SDK receives a video frame before encoding.

Parameters

sourceType
Video source type. See VideoSourceType.
videoFrame
Video frame.
Note: The default format of the video frame data obtained through this callback is as follows:
  • Android: I420 or RGB (GLES20.GL_TEXTURE_2D).

Return Values

  • When the video processing mode is PROCESS_MODE_READ_ONLY:
    • true: Retain for subsequent use.
    • false: Retain for subsequent use.
  • When the video processing mode is PROCESS_MODE_READ_WRITE:
    • true: The SDK accepts the video frame.
    • false: The SDK discards the video frame.

onRenderVideoFrame

Callback triggered when a remote user sends a video frame.

boolean onRenderVideoFrame(String channelId, int uid, VideoFrame videoFrame)

After you successfully register a video frame observer, the SDK triggers this callback each time it receives a video frame sent by a remote user. You can obtain the video data sent by the remote user before rendering and process it according to your specific scenario. The default format of the video frame obtained via this callback may be I420Buffer or TextureBuffer, and the texture format of TextureBuffer may be OES or RGB. If you need other video formats, you can set the desired data format in the getVideoFormatPreference callback.

Note:
  • If you want to send the preprocessed video data back to the SDK, you must first call getVideoFrameProcessMode to set the video processing mode to read-write (PROCESS_MODE_READ_WRITE).
  • If the obtained video data type is RGBA, the SDK does not support processing alpha channel data.
  • It is recommended to ensure that the modified parameters in videoFrame match the actual video frame in the buffer. Otherwise, issues such as abnormal rotation or image stretching may occur in local preview and remote video display.

Trigger Timing

This callback is triggered when the SDK receives a video frame sent by a remote user.

Parameters

channelId
Channel ID.
uid
Remote user ID who sent the current video frame.
videoFrame
Video frame.

Return Values

  • When the video processing mode is PROCESS_MODE_READ_ONLY:
    • true: Retain for subsequent use.
    • false: Retain for subsequent use.
  • When the video processing mode is PROCESS_MODE_READ_WRITE:
    • true: The SDK accepts the video frame.
    • false: The SDK discards the video frame.