使用脚本压测相机录像,暂停和恢复间隔3s反复调用,相机出现录制失败
压测视频录制或者音频录制无该问题,感觉是音视频同步问题,但是日志并没有相关错误
代码版本是4.1release,社区相关暂停恢复的错误已经合入
定位到原因了,修改后,我这测试暂停恢复几分钟是正常的。排查不易,如果能解决你的问题,麻烦采纳一下该答案。
修改如下:
foundation/multimedia/player_framework/services/engine/gstreamer/plugins/source/videocapture/src/gst_video_capture_src.cpp
gst_video_capture_src_pause 函数,所有场景下pause后都需要清空视频buffer
GstBufferPool *bufferpool = GST_BUFFER_POOL(surfacesrc->pool);
if (src->stream_type != VIDEO_STREAM_TYPE_ES_AVC) {
gst_buffer_pool_set_flushing(bufferpool, TRUE);
gst_buffer_pool_set_flushing(bufferpool, FALSE);
}
GST_DEBUG_OBJECT(src, "video capture src pause");
修改成
GstBufferPool *bufferpool = GST_BUFFER_POOL(surfacesrc->pool);
gst_buffer_pool_set_flushing(bufferpool, TRUE);
gst_buffer_pool_set_flushing(bufferpool, FALSE);
GST_DEBUG_OBJECT(src, "video capture src pause");
看如下打印,第二行是第10次恢复录制,第三行是恢复后拿到的首帧,比上一帧(暂停前)的pts大了3.9s,但是奇怪的是没有走到gst_video_capture_deal_with_pts 函数后面的逻辑了,此时的状态既不是RECORDER_RESUME也不是RECORDER_PAUSED,导致没有减去暂停的3.9s时间,导致了后续异常。理论上此时状态应该是RECORDER_RESUME,走line 361~368逻辑,没看懂。
可以在foundation/multimedia/player_framework/services/engine/gstreamer/plugins/source/videocapture/src/gst_video_capture_src.cpp 的gst_video_capture_deal_with_pts 函数中加打印,看此时的状态和paused_time 等。
还可以兼容操作:如果当前帧的pts比上一帧大2s以上,此时不是paused或resume状态也认为暂停了,增加total_pause_time的值
07-18 20:59:12.618 788 12878 D C02b2b/videocapturesrc: {gst_video_capture_src_pause():457} [gst::VideoSource:6AAEA0] video capture src pause
07-18 20:59:15.804 788 12878 D C02b2b/videocapturesrc: {gst_video_capture_src_resume():471} [gst::VideoSource:6AAEA0] video capture src resume
07-18 20:59:16.262 788 12882 D C02b2b/videocapturesrc: {gst_video_capture_deal_with_pts():337} [gst::VideoSource:6AAEA0] video capture buffer size is: 32064, pts: 812450800057
07-18 20:59:16.262 788 12882 I C02b2b/videocapturesrc: {gst_video_capture_deal_with_pts():341} [gst::VideoSource:6AAEA0] videoCapturer timestamp has increased: 3918618561
出问题时音视频的pts差异4s,但是此时从src获取的音视频pts基本对齐。因为从src获取的音视频pts需要减去暂停期间的时长,怀疑恢复录制时音视频的暂停时间计算出现了偏差。
看最后3行,第10次暂停videocapturesrc未响应,导致视频pts直接比音频大了4s(未减去本次暂停时间),导致的报错。
解决方案:分析中
07-18 20:59:18.043 788 12923 D C02b2b/AudioCaptureAsImpl: {GetSegmentInfo():213} timestamp from audioCapturer: 814235163860
07-18 20:59:18.070 788 12882 D C02b2b/videocapturesrc: {gst_video_capture_deal_with_pts():337} [gst::VideoSource:6AAEA0] video capture buffer size is: 34179, pts: 814260656113
07-18 20:59:18.141 788 12882 W C02b2b/splitmuxsink: {check_completed_gop():2760} [gst::MuxSinkBin:F1C0D0] error: Timestamping error on input streams
07-18 20:59:18.141 788 12882 W C02b2b/splitmuxsink: {check_completed_gop():2760} [gst::MuxSinkBin:F1C0D0] error: Context 0x7fa1ade330 sink pad <queue_audio_0:sink> running time +0:13:01.638298898, next gop start: +0:13:05.685712872, diff +0:00:04.047413974
Line 46549: 07-18 20:59:09.424 788 12882 I C02b2b/videocapturesrc: {gst_video_capture_deal_with_pts():368} [gst::VideoSource:6AAEA0] video has 9 times pause, total PauseTime: 28595869323
Line 47157: 07-18 20:59:09.434 788 12889 D C02b2b/AudioCaptureAsImpl: {GetBuffer():357} audio has 9 times pause, total PauseTime: 29344133673
Line 118562: 07-18 20:59:15.812 788 12889 D C02b2b/AudioCaptureAsImpl: {GetBuffer():357} audio has 10 times pause, total PauseTime: 32611328920
07-18 20:59:18.141 788 12882 W C02b2b/structure: {priv_gst_structure_append_to_gstring():2093} [gst:: :0] No value transform to serialize field 'gerror' of type 'GError'
07-18 20:59:18.142 788 12881 E C02b2b/RecMsgProc: {ProcessErrorMsgDefault():62} [ERROR] domain:0x7ba, code:0x1, msg:Timestamping error on input streams, ../../../third_party/gstreamer/gstplugins_good/gst/multifile/gstsplitmuxsink.c(2760): check_completed_gop (): /GstPipeline:recorder-pipeline/GstSplitMuxSink:MuxSinkBin:
07-18 20:59:18.142 788 12881 E C02b2b/RecMsgProc: Context 0x7fa1ade330 sink pad <queue_audio_0:sink> running time +0:13:01.638298898, next gop start: +0:13:05.685712872, diff +0:00:04.047413974.
是时间戳相差超过3s出现录像异常,可以分析代码rk_4.1/foundation/multimedia/player_framework/services/engine/gstreamer/plugins/source下audio的时间戳计算和video的时间戳计算