[ad_1]
I am constructing a video streaming app utilizing Agora SDK (for Flutter). I additionally use a native-layer Uncooked Video Knowledge plugin as a dependency, which provides me entry to uncooked YUV420 body buffers earlier than they’re encoded. I can entry them by way of a Swift & Kotlin interface.
I want to write down code that intercepts a uncooked video body, shops it, after which overwrites the next body with the beforehand saved picture knowledge. Then intercepts the third body, and writes it over the 4th, and so forth.
The plugin repo has a demo of the right way to work with uncooked frames — this code simply colours the captured video inexperienced:
iOS (supply):
public func onCapture(_ videoFrame: AgoraVideoFrame) -> Bool {
memset(videoFrame.uBuffer, 0, Int(videoFrame.uStride * videoFrame.peak) / 2)
memset(videoFrame.vBuffer, 0, Int(videoFrame.vStride * videoFrame.peak) / 2)
return true
}
videoFrame itself is outlined as follows in AgoraVideoFrame.h, and
videoFrame.uBuffer is an UnsafeMutableRawPointer.
Android (supply):
override enjoyable onCaptureVideoFrame(videoFrame: VideoFrame): Boolean {
Arrays.fill(videoFrame.getuBuffer(), 0)
Arrays.fill(videoFrame.getvBuffer(), 0)
return true
}
videoFrame.uBuffer is a byte[], and there is a setuStride() methodology that accepts a byte[] argument, so maybe Android might be simpler.
First attempt with Swift (paints half of the 2nd body into the colour of the top-left pixel):
// first body callback
bufferedFrameY = videoFrame.yBuffer.load(as: Int32.self)
bufferedFrameU = videoFrame.uBuffer.load(as: Int32.self)
bufferedFrameV = videoFrame.vBuffer.load(as: Int32.self)
//.. 2nd body callback
memset(videoFrame.yBuffer, bufferedFrameY, Int(videoFrame.yStride * videoFrame.peak))
memset(videoFrame.uBuffer, bufferedFrameU, Int(videoFrame.uStride * videoFrame.peak / 4))
memset(videoFrame.vBuffer, bufferedFrameV, Int(videoFrame.vStride * videoFrame.peak / 4))
Second attempt with Swift (does nothing in any respect):
// first body callback
bufferedFrameY = videoFrame.yBuffer.load(as: Int32.self)
bufferedFrameU = videoFrame.uBuffer.load(as: Int32.self)
bufferedFrameV = videoFrame.vBuffer.load(as: Int32.self)
//.. 2nd body callback
videoFrame.yBuffer.storeBytes(of: bufferedFrameY, as: Int32.self)
videoFrame.uBuffer.storeBytes(of: bufferedFrameU, as: Int32.self)
videoFrame.vBuffer.storeBytes(of: bufferedFrameV, as: Int32.self)
How ought to I method this to resolve the issue?
[ad_2]
