Non-contact respiration monitoring using optical sensors

The main goal of this paper is to develop classification of non-contact respiration monitoring approaches and proposal of structure for system with facial artifacts rejection. All available techniques were divided into two main groups: based on reconstruction of respiration from 3-D image of object and based on 2-D image processing of techniques. Structure of system for respiration monitoring using optical sensors with facial artifacts removing was developed. New approach allows improving of respiration monitoring for objects in supine position and in a sitting position. References 26, figures 16.


Introduction
Monitoring of vital parameters such as heart rate and breathing parameters is an important task for medical scientists and engineers. Today's medicine has good results in vital parameter estimation with invasive methods. Such methods typicaly involve contact to patient's body, and require preliminary patient preparation which might not be convenient for patients and medical personnel in various clinical situations.
Respiration monitoring is an essential aspect in removal of moving artifacts for computer tomography [1], magnetic resonance imaging [2], imageguided radiation therapy [3], in neonatal applications [4], in monitoring applications for elderly people [5] and for automotive and aviation applications. Significant efforts are focused on the design of non-contact real-time breathing monitoring systems. Absence of direct contact between patient and monitoring device does not deliver discomfort and extends the areas of possible applications. There are plenty of approaches to noncontact respiration monitoring, such as ultrasonic, radarbased, capacitive ECG-based etc., but the most promising are the techniques based on the analysis of natural images of patients under monitoring, which are easily achievable. This is due to the wide availability of cameras, low cost and good operability.
The main goal of this work is to give thorough review of optical systems for respiration monitoring, and to propose the new method for improving existent solutions. The new design of non-contact respiration monitoring system is developed, in which the unit for artifact removal is included. The paper is organized as follows. In first section, techniques of respiration monitoring with 3-D scanning sensors are described. Second section presents monitoring approaches using 2-D image processing techniques. In the third section, the new structure of respiration monitoring system is proposed and justified.

General approaches to respiration monitoring in optical domain
If detection in optical domain is used for retrieval of respiration parameters, the movements of human body are usually thought to be directly connected to respiration. To detect respiration, the system should be designed to reconstruct body surface movement from the video sequence, and to estimate the magnitude of surface excursions in real time. Techniques of respiration monitoring differ by the general approach, which is used to extract movements.
In this paper, we propose to divide all optical methods into two groups depending on the dimensionality of captured data used for respiration extraction ( Fig. 1).
In the first approach, reconstruction of respiration from 3-D image of object is used. It requires a specialized hardware, such as stereoscopic cameras and/or additional lightening equipment. This approach can be subdivided into two classes, first one using stereoscopic imaging, the second one based on projection methods.
Second approach is based on 2-D image processing techniques with using a single optical sensor, and can be subdivided into optical flow methods and frame subtraction methods.

3-D reconstruction based methods
Many kinds of systems for 3-D objects scanning are available in the market [6]. The taxonomy of optical 3-D scanners is given in Fig 2. In biomedical applications it is reasonable to use noncontact systems based on reflective principles. This approach provides patient's comfort and has no in-fluence on his conditions. Two techniques of 3-D reconstruction are employed for respiration monitoring in most cases. Passive stereoscopic method based only on acquisition of reflected light from external sources, and active method called an active triangulation based on projection of specific markers on object before acquisition.

Stereoscopic methods of respiratory monitoring
Stereoscopic system contains two or more cameras that capture video of the same scene from different locations. It is possible to estimate distance to the object using the system demonstrated in Fig. 3. Depth of 3-D image is directly proportional to focal length of camera lense f and distance between cameras optical axes b , and inversely proportional to disparity of the same object point in pixels d : The points of object for different images of the same scene should be robustly identified for correct 3-D reconstruction.

Projection-based systems for respiration monitoring
The main difference of active systems is the usage of light emmiter to create auxillary projection markers on object surface (Fig. 4). 3-D position of spot on the object's surface can be computed as intersection between l 1 and l 2 by triangulation [6].

Respiratory monitoring using 3-D techniques
Li et al. [3] used stereoscopic camera manufactured by Xigen for respiration tracking tasks in radiotherapy applications. Authors calculated 2-D depth image from 3-D reconstructed surface. After conversion of 3-D surface images to 2-D depth images, they used principal component analysis (PCA) to perform an unsupervised learning to extract different surface motion patterns from the 2-D depth image sequence. For the surface movements' patterns classification the support vector machine (SVM) technique is used (Fig. 5). Experiments have been conducted on phantom and 4 volunteers. The majority of researchers [5], [7]- [9] uses active 3-D systems based on single CCD camera and fiber grating device (FGD). The FG device gener-ate field of dots or stripes for three-dimensional surface of object reconstruction Fig. 7. FG devices are available on the market, Aoki et al. [8] used solution manufactured by Sumitomo Osaka Cement Co, ltd.
Appearance of common construction of experimental system shown in Fig. 8. System included bed with overhead support, where CCD camera and FGD are installed. In the most cases, resolution of CCD cameras was equal to 640x480 pixels. The relationship between vertical motion, represented by a symbol Z ∆ , in the thoracicoabdominal part and shift length of the pattern light, denoted by a symbol d , in the image is given by following equation: where Z is the vertical distance from the height, at which the sensor was installed, to the body surface, d is the focal distance of lens, and l is the horizontal distance from the center of lens to the FG element. Thus the measurement principle of this method is based on the triangulation [5].
Aoki et al. [5], [8] used active triangle system for respiration parameters obtaining (Fig. 6). Authors have shown that waveforms measured by spirometer correspond to the respiratory flow measured by non-contact method. The same authors achieved better results with stripe projections then in the case of dot field projections [8]. They also obtained similar results with and without top covered quilt.
Povsic et al. [10] proposed non-contact realtime system for teaching and correcting respiration. Tamagawa et al. [9] presented system based on active triangle system for respiration estimation during tomography examination.
The Kinect system (9) for 3-D reconstruction is prevalent as well. Kinect was initially developed as gaming device but this system is used in respiratory applications [11]- [13]. Kinect includes RGB camera and couple of IR projector and IR camera, which are used for estimation of depth distance. System captures depth map of image with 320x240 pixel resolution. Yu et al. [14] developed Kinectbased system for monitoring position, body movements, and breathing rate during sleep. Lodovico et al. [15] and Povsic et al. [10] proposed complex systems for whole human trunk surface vizualization during respiration analysis.
Systems in Figures 10 and 11 allow calculation of not only respiratory rate but also of expiratory volume.

2-D reconstruction based methods
Systems with 2-D reconstruction of respiration use single camera to capture video from the object, and consequent processing of two-dimensional images for localization of the object points.
2-D techniques employ single optical sensor for image acquisition. Sensors with various wavelength range can be used. Visible range cameras and near infrared cameras are used usually for respiration monitoring. Both types of cameras can be manufactured by CCD or CMOS technology. Modern CCD and CMOS sensors have comparable characteristics but CMOS is inexpensive and has lower power consumption. Therefore, CMOS cameras are usually used for general purpose and costumer electronics (web-cams, smartphones, etc.) and CCD are used in scientific and industrial applications.
Visible range cameras for respiration monitoring with good scene illumination can be employed. In the work [16], a system of NIR camera and NIR projector are implemented for the respiration monitoring in the scene with low brightness (monitoring during night). Fig. 12 shows sample frame of night respiration monitoring of patient in supine position.

Respiration extraction by optical flow estimation
The most widespread technique that implements object movement estimation in video sequences is based on assessment of optical flow coming from the patient. For optical flow estimation, a few assumptions have to be accepted. First is that intensity of pixel for the same object on the video does not change between consecutive frames (1). This means that one gets the same intensity of the pixel on the next frame and this pixel is only shifted by x u and y u distance in x and y direction respectively. ) ) ( , , ( , , ( , , 0 x y I x y t I x y t I x y t u u t x y Expression (3) has been obtained using Taylor series expansion and ignoring high-order terms. Since only one time instance lies between two consecutive frames, the ( ) ( , , ) , , 1 I x y t I x y t − + part of equation (3) can be regarded as time derivative Having this assumption accepted, it is possible to solve system of equation (6) There are many approaches to solve system (6) but the most widespread are Lukas-Kanade [17] and Horn-Schunk [18] methods of optical flow estimation. Both methods assess displacement magnitude → u which minimizes error (7) Nakajima et al. [21], [22] used optical flow technique for estimation of human respiration parameters and posture changes. Authors proposed 2 v parameter for evaluation of movement using video sequence. Proposed system detects posture change and respiration of the subject in bed by observing chest or blanket movement. Kuo et al. [16] proposed a visual sleepingrespiration estimation system for monitoring and measuring the respiration parameters of sleeping people. Proposed system was built using near-infrared camera with a NIR projector. In their paper, authors evaluate new technique that combine Horn-Schunk optical flow estimation method and finite-state controlled hidden Markov model. The developed system can distinguish respiratory and non-respiratory body movement.

Respiration estimation by frame subtraction method
Frame subtraction that is usually applied for background removal [19], [20] can be used for respiration parameters estimation. The main feature of this method is removing regions that do not change from scene to scene. This technique allows deleting invariable background and leaving part of frame that has changed. Frame without background is computed using formula (9) ( ) ( ) , , ( , , ) , , 1 I x y t I x y t I x y t = − + As seen from formula (9) the resulting frame ∆ ( , , ) I x y t contains region that was changed, on the black background (invariable pixel are canceled). Example of two frames subtraction is shown in Fig. 14. Tan et al. [23] presents system for respiration monitoring based on frame subtraction technique. Authors developed system for assessment breathing parameters. They made experiments to study dependence of respiration monitoring quality on the distance between object and camera. Dependence of closing contrast and the monitoring accuracy was researched.
Weixing et al. [24] and Ji et al. [25] proposed system for animal respiration monitoring based on frame subtraction technique. The proposed techniques can be used for identifying animal health in real-time by detecting breathing parameters.

Artefact removal in 2-D respiration extraction techniques
Despite the availability of 3-D and 2-D systems for non-contact respiration monitoring, they are far from wide commercial use. In particular, these sys-tems are prone to artifacts of various origin, such as patient and background movement, ambient light change etc. Thus they cannot be used in clinical conditions without the guidance of qualified personnel. For accurate respiration monitoring of patient in supine position the facial artifacts removal is important as well. Mimics and eyes movement are the usual source of artifacts that should be canceled, since they introduce substantial distortion in scene leading to decrease of quality of respiration parameters extraction.
Systems for non-contact respiration monitoring using single optical sensor include one camera with required wavelength range, hardware for collecting and processing raw sensor data and software for video processing. Software implements estimation of movement in video sequence (hence possibility to compute respiratory parameters). Typical block diagram of respiratory monitoring system is shown in Fig. 15 Optical sensor Preprocessing Motion estimation Target parameter estimation

Fig. 15. Typical respiration monitoring system
In this paper, facial artifacts removing is proposed to improve existent systems. Separate additional block is included in the block diagram. This block implements face recognition and exclusion of the face region from image to be analyzed at next stage. Algorithm is based on optical flow technique for motion estimation and Viola-Jones algorithm [26] for face region localization, and following removal of this area from video frame. The proposed method allows to remove artifacts induced by mimics movement and eye blinking. Modified block diagram of proposed system is shown in Fig. 16. Procedure of data processing have the following stages: video flow capturing, motion estimation using optical flow technique, face recognition and removal of facial region from analysis, obtaining of respiration curve, target parameter calculation.

Conclusion
In this paper, the classification of respiration monitoring methods using optical sensors was proposed. Structure of respiration monitoring system with facial artifacts removing is developed, using the facial mimics and eyes blinking artefacts removal. New system can improve respiration monitoring for human positioned in frontal plane.