Our hospital's application of Virtual Reality (VR) and Augmented Reality (AR) technology in surgical navigation, through the "Smart Operating Room Environment Intelligence System - Surgical Flow Behavior Recognition" project, has successfully enhanced intelligent management and accuracy during surgical procedures. This system is composed of five major modules: real-time image streaming, skin detection, behavior recognition, de-identification, and API interfacing. Two cameras are installed in two operating rooms, utilizing image streaming technology to perform surgical flow behavior recognition, specifically monitoring the movement status of operating room beds and accurately recording the time nodes of patient entry and transfer.
The system, combined with a finite state machine calibration mechanism, can dynamically correct recognition results, ensuring that abnormal data generated due to image blur or occlusion are filtered out, improving recognition accuracy and stability. Through deep learning models ResNet and Sequencer, the system can automatically identify key behaviors in the operating room. It utilizes time matching and error correction mechanisms to address background complexity and data imbalance issues, achieving an entry time recognition accuracy of 91.72% and a transfer time recognition accuracy of 90.83%.
To ensure privacy protection, the system uses the HSV color space for skin detection, masking the skin areas in the images to prevent leakage of patient and staff personal information. Furthermore, the de-identification module automatically removes features that could identify individuals within three minutes before and after each key time node and stores the processed images, complying with medical privacy regulations. Statistics show that from January to November 2024,each transfer and entry process saved an average of 4 minutes, accumulating a saving of 548 minutes of nursing time, reducing manual error rates in surgical records, and alleviating the workload of nursing staff.
The system also interfaces with hospital systems through APIs, enabling real-time matching of patient data with recognition results and storage in the medical database, ensuring the transparency and traceability of the surgical process and further enhancing the degree of automation of operating room operations. After implementation, the surgical flow connection delay rate decreased by 2% from March to April 2024, effectively shortening surgical preparation time and improving operating room utilization efficiency.
Our hospital is continuously optimizing this system and will further expand to automated recognition of more surgical behaviors in the future, such as time stamping of key intraoperative operations (e.g., incision time), and enhance the model's adaptability in low recognition rate scenarios, improving flexibility across operating room environments. This project successfully applies intelligent behavior recognition technology to surgical navigation, improves surgical management efficiency, ensures the accuracy of medical processes, and lays the foundation for the development of future smart operating rooms.