1.介绍

文章主要介绍在openharmony上如何实现同时启动多个摄像头预览录像。

2.环境

oh版本：4.1 release
开发板：RK3568
2个USB摄像头

3.当前录像框架

如下图所示，录像时主要涉及应用，Camera Framework，Player Framework，Audio Framework和图形显示等几个部分。

经过分析和测试，在camera hdi层，是已经支持同时启动多个实例录像的，问题主要阻塞在framework层。

4.framework流程分析

根据当前提供的api接口，录像的大致开发流程如下：
1.调用media接口创建CameraManager.
2.通过CameraManager来获取所有Camera设备.
3.用指定的Camera来获取支持的场景和Camera规格.
4.调用media接口创建AVRecorder，根据video和audio配置情况，调用prepare准备AVRecorder.
5.通过XComponent和AVRecorder分别获取预览和录像的SurfaceId.
6.通过CameraManager以获取的Camera规格等属性和SurfaceId来创建PreviewOutput，VideoOutput和CameraInput.
7.通过CameraInput打开设备.
8.通过CameraManager创建VideoSession，并调用BeginConfig开始配置Session.
9.将PreviewOutput，VideoOutput和CameraInput分别加入VideoSession.
10.调用VideoSession的CommitConfig提交配置.
11.分别调用VideoSession，VideoOutput，AVRecorder的start开始录像.
12.录像完成后，分别调用VideoOutput，AVRecorder，VideoSession的stop结束录像.
13.调用CameraInput的Close关闭设备.
14.释放创建的各种资源.

5.修改方案

根据上面流程调用的各个接口，分析框架代码，在启动摄像头录像时，类资源和实例都是单独创建，不互相依赖和影响，但是openharmony在录像时对系统资源占用做了限制，具体如下:
1.多路audio录制
2.多路AVRecorder实例
3.Camera device管控
4.Camera session管控

5.1.多audio录制问题

1.在AVRecorderNapi中，会检测是否有配置audio，来确定在录制时是否也启动audio的录制，在多摄像头同时录像的情况下，也会启动多个audio实例录音。RK3568目前采用的是多路实例单设备的方式。如果在其它的开发板上适配时不支持，可以在hap开发时，传参不设置 audioSourceType ，或是在 avrecorder_napi.cpp 文件的 GetSourceType 函数中将第一个 if 中的 getValue 在之前直接赋值为 false。

具体代码:

// foundation/multimedia/player_framework/frameworks/js/avrecorder/avrecorder_napi.cpp

int32_t AVRecorderNapi::GetSourceType(std::unique_ptr<AVRecorderAsyncContext> &asyncCtx， napi_env env， napi_value args)
{
    std::shared_ptr<AVRecorderConfig> config = asyncCtx->config_;
    int32_t audioSource = AUDIO_SOURCE_INVALID;
    int32_t videoSource = VIDEO_SOURCE_BUTT;

    bool getValue = false;
    int32_t ret = AVRecorderNapi::GetPropertyInt32(env， args， "audioSourceType"， audioSource， getValue);
    CHECK_AND_RETURN_RET(ret == MSERR_OK，
        (asyncCtx->AVRecorderSignError(ret， "getaudioSourceType"， "audioSourceType")， ret));
    // 不设置 audioSourceType 属性，或是在if判断前将 getValue设置为 false
    if (getValue) {
        config->audioSourceType = static_cast<AudioSourceType>(audioSource);
        config->withAudio = true;
        MEDIA_LOGI("audioSource Type %{public}d!"， audioSource);
    }

    ret = AVRecorderNapi::GetPropertyInt32(env， args， "videoSourceType"， videoSource， getValue);
    CHECK_AND_RETURN_RET(ret == MSERR_OK，
        (asyncCtx->AVRecorderSignError(ret， "getvideoSourceType"， "videoSourceType")， ret));
    if (getValue) {
        config->videoSourceType = static_cast<VideoSourceType>(videoSource);
        config->withVideo = true;
        MEDIA_LOGI("videoSource Type %{public}d!"， videoSource);
    }

    CHECK_AND_RETURN_RET(config->withAudio || config->withVideo，
        (asyncCtx->AVRecorderSignError(MSERR_INVALID_VAL， "getsourcetype"， "SourceType")， MSERR_INVALID_VAL));

    return MSERR_OK;
}

5.2.多AVRecorder实例问题

在media_server_manager.cpp中，在创建Recorder实例时，在SA服务端，目前限制的实例数 recorderMax 是 2 个，如果有更多需求可以修改 recorderMax 的数量。

// foundation/multimedia/player_framework/services/services/sa_media/server/media_server_manager.cpp
sptr<IRemoteObject> MediaServerManager::CreateRecorderStubObject()
{
    // 限制录像实例个数为2，可根据需要增大
    constexpr uint32_t recorderMax = 2;
    CHECK_AND_RETURN_RET_LOG(recorderStubMap_.size() < recorderMax，
        nullptr， "The number of recorder services(%{public}zu) has reached the upper limit."
        "Please release the applied resources."， recorderStubMap_.size());

    sptr<RecorderServiceStub> recorderStub = RecorderServiceStub::Create();
    CHECK_AND_RETURN_RET_LOG(recorderStub != nullptr， nullptr， "failed to create RecorderServiceStub");

    sptr<IRemoteObject> object = recorderStub->AsObject();
    CHECK_AND_RETURN_RET_LOG(object != nullptr， nullptr， "failed to create RecorderServiceStub");

    pid_t pid = IPCSkeleton::GetCallingPid();
    recorderStubMap_[object] = pid;

    Dumper dumper;
    dumper.entry_ = [recorder = recorderStub](int32_t fd) -> int32_t {
        return recorder->DumpInfo(fd);
    };
    dumper.pid_ = pid;
    dumper.uid_ = IPCSkeleton::GetCallingUid();
    dumper.remoteObject_ = object;
    dumperTbl_[StubType::RECORDER].emplace_back(dumper);
    MEDIA_LOGD("The number of recorder services(%{public}zu) pid(%{public}d)."，
        recorderStubMap_.size()， pid);
    (void)Dump(-1， std::vector<std::u16string>());
    return object;
}

5.3.Camera device管控问题

在hcamera_device.cpp文件的 OpenDevice 函数中，会判断是否可以打开camera，调用流程：OpenDevice()->CanOpenCamera()->GetConflictDevices()，在hcamera_device_manager.cpp的函数 GetConflictDevices 会检测已打开的Camera device，对相同/不同进程，前/后台状态，打开相同/不同camera等情况关闭之前已打开的Camera device。这里可以直接去掉 CanOpenCamera 的调用，来取消管控限制。

// foundation/multimedia/camera_framework/services/camera_service/src/hcamera_device.cpp
int32_t HCameraDevice::OpenDevice()
{
    MEDIA_DEBUG_LOG("HCameraDevice::OpenDevice start");
    CAMERA_SYNC_TRACE;
    int32_t errorCode;
    MEDIA_INFO_LOG("HCameraDevice::OpenDevice Opening camera device: %{public}s", cameraID_.c_str());

    // 注释 CanOpenCamera() 调用，取消限制
    // bool canOpenDevice = CanOpenCamera();
    // if (!canOpenDevice) {
    //     MEDIA_ERR_LOG("refuse to turning on the camera");
    //     return CAMERA_DEVICE_CONFLICT;
    // }

    errorCode = cameraHostManager_->OpenCameraDevice(cameraID_, this, hdiCameraDevice_);
    if (errorCode != CAMERA_OK) {
        MEDIA_ERR_LOG("HCameraDevice::OpenDevice Failed to open camera");
    } else {
        isOpenedCameraDevice_.store(true);
        HCameraDeviceManager::GetInstance()->AddDevice(IPCSkeleton::GetCallingPid(), this);
    }
    errorCode = InitStreamOperator();
    if (errorCode != CAMERA_OK) {
        MEDIA_ERR_LOG("HCameraDevice::OpenDevice InitStreamOperator fail err code is:%{public}d", errorCode);
    }

    ..........

}

bool HCameraDevice::CanOpenCamera()
{
    sptr<HCameraDevice> cameraNeedEvict;
    bool ret = HCameraDeviceManager::GetInstance()->GetConflictDevices(cameraNeedEvict, this);
    if (cameraNeedEvict != nullptr) {
        MEDIA_DEBUG_LOG("HCameraDevice::CanOpenCamera open current device need to close other devices");
        cameraNeedEvict->OnError(DEVICE_PREEMPT, 0);
        cameraNeedEvict->CloseDevice();
    }
    return ret;
}

// foundation/multimedia/camera_framework/services/camera_service/src/hcamera_device_manager.cpp
// 在 GetConflictDevices 函数中，进行各种判断来关闭之前打开的 camera device
bool HCameraDeviceManager::GetConflictDevices(sptr<HCameraDevice> &cameraNeedEvict，
                                              sptr<HCameraDevice> cameraIdRequestOpen)
{
    ADD_TAG();
    pid_t activeClient = GetActiveClient();
    pid_t pidOfOpenRequest = IPCSkeleton::GetCallingPid();
    if (stateOfACamera_.Size() != 0) {
        if (activeClient != -1) {
            MEDIA_ERR_LOG("HCameraDeviceManager::GetConflictDevices A and OH camera is turning on in the same time");
            return false;
        }
        return isAllowOpen(pidOfOpenRequest);
    } else {
        MEDIA_INFO_LOG("HCameraDeviceManager::GetConflictDevices no A camera active");
    }
    if (activeClient == -1) {
        return true;
    }
    sptr<HCameraDevice> activeCamera = GetCameraByPid(activeClient);
    if (activeCamera == nullptr) {
        return true;
    }
    int32_t priorityOfOpenRequestPid = 1001;
    int32_t result = Memory::MemMgrClient::GetInstance().
                    GetReclaimPriorityByPid(pidOfOpenRequest， priorityOfOpenRequestPid);
    MEDIA_INFO_LOG("HCameraDeviceManager::GetConflictDevices callerPid:%{public}d， priority score: %{public}d"，
                   pidOfOpenRequest， priorityOfOpenRequestPid);
    if (!result) {
        if (activeClient == pidOfOpenRequest) {
            MEDIA_INFO_LOG("HCameraDeviceManager::GetConflictDevices is same pid");
            if (!activeCamera->GetCameraId().compare(cameraIdRequestOpen->GetCameraId())) {
                cameraNeedEvict = activeCamera;
                MEDIA_INFO_LOG("HCameraDeviceManager::GetConflictDevices is same pid， return ture");
                return true;
            } else {
                MEDIA_INFO_LOG("HCameraDeviceManager::GetConflictDevices is same pid， return false");
                return false;
            }
        }
        int32_t priorityOfIterPid = 1001;
        int32_t iterResult = Memory::MemMgrClient::GetInstance().
                            GetReclaimPriorityByPid(activeClient， priorityOfIterPid);
        MEDIA_INFO_LOG("HCameraDeviceManager::canOpenCamera pid:%{public}d， priority score: %{public}d"，
                       activeClient， priorityOfIterPid);
        if (!iterResult && priorityOfOpenRequestPid <= priorityOfIterPid) {
            cameraNeedEvict= activeCamera;
            return true;
        } else {
            return false;
        }
    } else {
        MEDIA_ERR_LOG("HCameraDeviceManager::GetConflictDevices falied to get priority");
        return false;
    }
}

5.4.Camera session管控问题

在hcapture_session.cpp中创建 HCaptureSession 实例时，会检测当前调用进程是否已有创建Session实例，会将已创建的实例关闭.

// foundation/multimedia/camera_framework/services/camera_service/src/hcapture_session.cpp
HCaptureSession::HCaptureSession(const uint32_t callingTokenId， int32_t opMode)
{
    pid_ = IPCSkeleton::GetCallingPid();
    uid_ = IPCSkeleton::GetCallingUid();
    MEDIA_DEBUG_LOG("HCaptureSession: camera stub services(%{public}zu) pid(%{public}d)."， TotalSessionSize()， pid_);
    auto pidSession = TotalSessionsGet(pid_);   // 获取当前进程已经创建的 session 实例
    if (pidSession != nullptr) {
        auto disconnectDevice = pidSession->GetCameraDevice();
        if (disconnectDevice != nullptr) {
            disconnectDevice->OnError(HDI::Camera::V1_0::DEVICE_PREEMPT， 0);
        }
        MEDIA_ERR_LOG("HCaptureSession::HCaptureSession doesn't support multiple sessions per pid");
        pidSession->Release();  // 关闭释放当前进程已经创建的 session 实例
    }
    TotalSessionsInsert(pid_， this);
    callerToken_ = callingTokenId;
    opMode_ = opMode;
    SetOpMode(opMode_);
    MEDIA_INFO_LOG(
        "HCaptureSession: camera stub services(%{public}zu). opMode_= %{public}d"， TotalSessionSize()， opMode_);
}

在修改限制时，要注意 session 的释放问题，具体修改如下：

修改 g_totalSessions 的类型，由之前 static std::map<pid_t, sptr<HCaptureSession>> g_totalSessions; 改为 static std::map<pid_t, std::vector<sptr<HCaptureSession>>> g_totalSessions; ，由一对一改为一对多，同时将增，删，获取方式等也同步做修改。

// foundation/multimedia/camera_framework/services/camera_service/src/hcapture_session.cpp
static std::map<pid_t, std::vector<sptr<HCaptureSession>>> g_totalSessions;
static std::mutex g_totalSessionLock;
static size_t TotalSessionSize()
{
    std::lock_guard<std::mutex> lock(g_totalSessionLock);
    int size = 0;
    for (auto &pair : g_totalSessions) {
        size += pair.second.size();
    }
    return size;
}

static const std::map<pid_t, std::vector<sptr<HCaptureSession>>> TotalSessionsCopy()
{
    std::lock_guard<std::mutex> lock(g_totalSessionLock);
    return g_totalSessions;
}

static void TotalSessionsInsert(pid_t pid, sptr<HCaptureSession> session)
{
    std::lock_guard<std::mutex> lock(g_totalSessionLock);
    auto it = g_totalSessions.find(pid);
    if (it != g_totalSessions.end()) {
        MEDIA_INFO_LOG("HCaptureSession TotalSessionsInsert insert session, pid already exist! pid is:%{public}d", pid);
        it->second.push_back(session);
        return;
    }
    g_totalSessions[pid] = {session};
}

static std::vector<sptr<HCaptureSession>> TotalSessionsGet(pid_t pid)
{
    std::lock_guard<std::mutex> lock(g_totalSessionLock);
    auto it = g_totalSessions.find(pid);
    if (it != g_totalSessions.end()) {
        return it->second;
    }
    return {};
}

static void TotalSessionErase(pid_t pid, const HCaptureSession* ptr)
{
    std::lock_guard<std::mutex> lock(g_totalSessionLock);
    if (g_totalSessions.count(pid) != 0) {
        auto it = g_totalSessions[pid].begin();
        for (; it != g_totalSessions[pid].end(); ++it) {
            if (it->GetRefPtr() == ptr) {
                g_totalSessions[pid].erase(it);
                break;
            }
        }
        if (g_totalSessions[pid].empty()) {
            g_totalSessions.erase(pid);
        }
    }
}

在 HCaptureSession::HCaptureSession 构造函数里去掉限制。

// foundation/multimedia/camera_framework/services/camera_service/src/hcapture_session.cpp

HCaptureSession::HCaptureSession(const uint32_t callingTokenId, int32_t opMode)
{
    pid_ = IPCSkeleton::GetCallingPid();
    uid_ = IPCSkeleton::GetCallingUid();
    MEDIA_DEBUG_LOG("HCaptureSession: camera stub services(%{public}zu) pid(%{public}d).", TotalSessionSize(), pid_);
    // auto pidSession = TotalSessionsGet(pid_);
    // if (pidSession != nullptr) {
    //     auto disconnectDevice = pidSession->GetCameraDevice();
    //     if (disconnectDevice != nullptr) {
    //         disconnectDevice->OnError(HDI::Camera::V1_0::DEVICE_PREEMPT, 0);
    //     }
    //     MEDIA_ERR_LOG("HCaptureSession::HCaptureSession doesn't support multiple sessions per pid");
    //     pidSession->Release();
    // }
    TotalSessionsInsert(pid_, this);
    callerToken_ = callingTokenId;
    opMode_ = opMode;
    SetOpMode(opMode_);
    MEDIA_INFO_LOG(
        "HCaptureSession: camera stub services(%{public}zu). opMode_= %{public}d", TotalSessionSize(), opMode_);
}

在 HCaptureSession::Release 中将实例从 map 中删除，减少引用计数，确保资源正确释放。

// foundation/multimedia/camera_framework/services/camera_service/src/hcapture_session.cpp

int32_t HCaptureSession::Release(CaptureSessionReleaseType type)
{
    CAMERA_SYNC_TRACE;
    int32_t errorCode = CAMERA_OK;
    stateMachine_.StateGuard([&errorCode, this, type](CaptureSessionState currentState) {
        MEDIA_INFO_LOG("HCaptureSession::Release pid(%{public}d). release type is:%{public}d", pid_, type);
        bool isTransferSupport = stateMachine_.CheckTransfer(CaptureSessionState::SESSION_RELEASED);
        if (!isTransferSupport) {
            MEDIA_ERR_LOG("HCaptureSession::Release error, this session is already released!");
            errorCode = CAMERA_INVALID_STATE;
            return;
        }

        // Clear outputs
        ReleaseStreams();

        // Clear inputs
        auto cameraDevice = GetCameraDevice();
        if (cameraDevice != nullptr) {
            cameraDevice->Release();
            POWERMGR_SYSEVENT_CAMERA_DISCONNECT(cameraDevice->GetCameraId().c_str());
            SetCameraDevice(nullptr);
        }
        StopUsingPermissionCallback(callerToken_, OHOS_PERMISSION_CAMERA);
        UnregisterPermissionCallback(callerToken_);

        // Clear current session
        MEDIA_DEBUG_LOG(
            "ClearCaptureSession: camera stub services(%{public}zu) pid(%{public}d).", TotalSessionSize(), pid_);
        TotalSessionErase(pid_, this);  // 移除当前 session，减少引用计数
        MEDIA_DEBUG_LOG("ClearCaptureSession: camera stub services(%{public}zu).", TotalSessionSize());

        sptr<ICaptureSessionCallback> emptyCallback = nullptr;
        SetCallback(emptyCallback);
        stateMachine_.Transfer(CaptureSessionState::SESSION_RELEASED);
        isSessionStarted_ = false;
    });

    return errorCode;
}

在 HCaptureSession::DestroyStubObjectForPid 中，退出进程时，将对应所有打开 session 实例都释放。

// foundation/multimedia/camera_framework/services/camera_service/src/hcapture_session.cpp

void HCaptureSession::DestroyStubObjectForPid(pid_t pid)
{
    MEDIA_DEBUG_LOG("camera stub services(%{public}zu) pid(%{public}d).", TotalSessionSize(), pid);
    // 一对一改为一对多之后，需要依次调用。
    std::vector<sptr<HCaptureSession>> sessions = TotalSessionsGet(pid);
    // std::string info;
    for (auto &session : sessions) {
        session->Release(CaptureSessionReleaseType::RELEASE_TYPE_CLIENT_DIED);
    }
    MEDIA_DEBUG_LOG("camera stub services(%{public}zu).", TotalSessionSize());
}

在 HCaptureSession::dumpSessions 中，dump出所有 session 信息。

// foundation/multimedia/camera_framework/services/camera_service/src/hcapture_session.cpp

void HCaptureSession::dumpSessions(std::string& dumpString)
{
    auto totalSession = TotalSessionsCopy();
    for (auto &pair : totalSession) {
        // 一对一改为一对多之后，需要依次调用。
        for (auto &session : pair.second) {
            dumpString += "No. of sessions for client:[" + std::to_string(1) + "]:\n";
            session->dumpSessionInfo(dumpString);
        }
    }
}

上面所有修改完成后，编译完重烧镜像，效果如下：

6.注意事项

1.在社区openharmony 4.1release上，RK3568已经适配好USB Camera，可以即插即用，无须另外再适配USB Camera。

2.如果是要接入多个USB Camera，需要注意：目前基本都是使用linux中提供的usb camera驱动，但是usb bus的 bandwidth 是有限的，如果将多个USB Camera接入一路usb bus，后打开的camera就会出现 No space left on device 的错误，网络上有比较多的解决方法，可以参考下。