EYE PHONE TECHNOLOGY

EYEPHONE TECHNOLOGY

ABSTRACT

Eye phone is a hands free interfacing system that is used for activating mobile phone by eye. Here the the functions of the phone can be drive easily. The phone functions  activated by blinking of the eye. The navigation  key fuctions are done by the movement of eye. The principle behind in eye phone technology is Eye tracking systems.There is no need for any other devices that placed in the eye for tracking the movements It is done by the movement of pupil in the eye. The  device sense the movement of the eye using the pupil movement. Normal devices are used front camera to sense the eye  movement. But in modern phones uses sensors used to track eye movements. The details , design and working of the eye phone shown below report.

TABLE OF CONTENT

NUMBER                                      TITLE                                       PAGE

1.      INTRODUCTION                                                            

2.      Eye phone Technology                                                   

        3.   Human Phone Intraction                                               

                    What is eye tracking?                                                                

                    Eye tracking Technology                                                           

      4.    EyePhone Design                                                            

  An eye detection                                                                            

  An open eye template creation phase                                            

  An eye tracking phase                                                                    

  A blink detection phase                                                                 

        5.   Evaluvation                                                                     

               Daylight Exposure Analysis for a Stationary  Subject                      

                          Artificial Light Exposure For A Stationary Subject                        

  6.     Applications                                                          

  7.     Conclusion                                                                      

  8       Reference                                                                        

INTRODUCTION

 As smartphones evolve researchers are studying new techniques to ease the human-mobile interaction. We propose EyePhone, a novel“hand-free”interfacing system capable of driving mobile applications/functions using only the user’s eyes movement and actions (e.g., wink). EyePhone tracks the user’s eye movement across the phone’s display using the camera mounted on the front of the phone; more specifically, machine learning algorithms are used to: i) track the eye and infer its position on the mobile phone display as a user views a particular application; and ii) detect eye blinks that emulate mouse clicks to activate the target application under view. We present a prototype implementation of EyePhone on a Nokia N810, which is capable of tracking the position of the eye on the display, mapping this positions to an application that is activated by a wink. At no time does the user have to physically touch the phone display.

Eye phone Technology

EyePhone tracks the user’s eye movement across the phone’s display using the camera mounted on the front of the phone. It a novel“hand-free”interfacing system capable of driving mobile applications/functions using only the user’s eyes movement and actions (e.g., wink). EyePhone tracks the user’s eye movement across the phone’s display using the camera mounted on the front of the phone.Eye phone  track the eye and infer its position on the mobile phone display as a user views a particular application; and eye phone detect eye blinks that emulate mouse clicks to activate the target application under view. We present a prototype implementation of EyePhone.Eye phone is capable of tracking the position of the eye on the display, mapping this positions to an application that is activated by a wink. At no time does the user have to physically touch the phone display. EyePhone is the first system capable of tracking a user’s eye and mapping its current position on the display to a function/application on the phone using the phone’s front-facing camera. EyePhone allows the user to activate an application by simply“blinking at the app”, emulating a mouse click. While other interfaces could be used in a hand-free manner, such as voice recognition, we focus on exploiting the eye as a driver of the HPI. We believe EyePhone technology is an important alternative to, for example, voice activation systems based on voice recognition, since the performance of a voice recognition system tends to degrade in noisy environments.

v  The front camera is the only requirement in EyePhone. Most of the smartphones today are equipped with a front camera and we expect that many more will be introduced inthe future (e.g., Apple iPhone 4G ) in support of video conferencing on the phone. The EyePhone system uses machine learning techniques that afterdetecting the eyecreate a template of the open eye and use template matching foreye tracking. Correlation matching is exploited for eye wink detection. We implement EyePhone on the Nokia N810 tablet and present experimental results in different settings. These nitial results demonstrate that EyePhone is capable of driving the mobile phone.

Human Computer Interaction(HCI)

             “HCI (human-computer interaction) is the study of how people interact with computers and to what extent computers are or are not developed for successful interaction with human beings”. Most HCI technology addresses the interaction between people and computers in “ideal” environments, i.e., where people sit in front of a desktop machine with specialized sensors and cameras centered on them

Human Phone Interaction(HPI)

Human-Computer Interaction (HCI) researchers and phone vendors are continuously searching for new approaches to reduce the effort users exert when accessing applications on limited form factor devices such as mobile phones.

Human-phone interaction (HPI) extends the challenges not typically found in HCI research, more specially related to the phone and how we use it. In order to address these goals HPI technology should be less intrusive; that is ,

•      it should not rely on any external devices other than the mobile phone itself;

•      it should be readily usable with minimum user dependency as possible;

•      it should be fast in the inference phase;

•      it should be lightweight in terms of computation;

•      it should preserve the phone user experience, e.g., it should not deplete the phone battery over normal operations

HUMAN-PHONE INTERACTION

                        Human-Phone Interaction represents an extension of the field of HCI since HPI presents new challenges that need to be addressed specifically driven by issues of mobility, the form factor of the phone, and its resource limitations (e.g., energy and computation). More specifically, the distinguishing factors of the mobile phone environment are mobility and the lack of sophisticated hardware support, i.e., specialized headsets, overhead cameras, and dedicated sensors, that are often required to realize HCI applications. In what follows, we discuss these issues.

Mobility Challenges:

                        One of the immediate products of mobility is that a mobile phone is moved around through unpredicted context, i.e., situations and scenarios that are hard to see or predict during the design phase of a HPI application. A mobile phone is subject to uncontrolled movement, i.e., people interact with their mobile phones while stationary, on the move, etc. It is almost impossible to predict how and where people are going to use their mobile phones. A HPI application should be able to operate reliably in any encountered condition. Consider the following examples

Two HPI applications, one using the accelerometer, the other relying on the phone’s camera. Imagine exploiting the accelerometer to infer some simple gestures a person can perform with the phone in their hands, e.g., shake the phone to initiate a phone call, or tap the phone to reject a phone call . What is challenging is being able to distinguish between the gesture itself and any other action the person might be performing. For example, if a person is running or if a user tosses their phone down on a sofa, a sudden shake of the phone could produce signatures that could be easily confused with a gesture. There are many examples where a classifier could be easily confused. In response, erroneous action could be triggered on the phone. Similarly, if the phone’s camera is used to infer a user action , it becomes important to make the inference algorithm operating on the video captured by the camera robust against lighting conditions, which can vary from place to place. In addition, video frames blur due to the phone movement. Because HPI application developers cannot assume any optimal operating conditions (i.e., users operating in some idealized manner) before detecting gestures in this example, (e.g., requiring a user to stop walking or running before initiating a phone call by a shaking movement), then the effects of mobility must be taken into account in order for the HPI application to be reliable and scalable.

 Hardware Challenges:

 As opposed to HCI applications, any HPI implementation should not rely on any external hardware. Asking people to carry or wear additional hardware in order to use their phone might reduce the penetration of the technology. Moreover, state-of-the art HCI hardware, such as glass mounted cameras, or dedicated helmets are not yet small enough to be conformably worn for long periods of time by people. Any HPI application should rely as much as possible on just the phone’s on-board sensors. Although modern smartphones are becoming more computationally capable, they are still limited when running complex machine learning algorithms . HPI solutions should adopt lightweight machine learning techniques to run properly and energy efficiently on mobile phones.

What is eye tracking?

Eye tracking refers simply to recording eye movements whilst a participant examines a visual stimulus (Collewijn, 1991). Accurate eye tracking must account for both the position of the head and the position of the eyes relative to the head. The earliest eye trackers were mechanical devices that must have caused participants great discomfort due to their size and invasiveness (see Collewijn, 1991 and Hayoe, & Ballard, 2005 for descriptions of various techniques). The technology has progressed significantly since then, such that systems are now available that do not require headgear or any physical attachments to be worn by the participant.

Two important aspects of eye tracking are calibrating the system to specific participants and managing eye drift, via drift correction. Calibration normally involves participants looking at an image (e.g., a dot or a fixation cross) in a known location. The eye tracking system compares the true location of the image to where it detects the participant’s gaze on the screen, and applies

a suitable correction for future fixations. Drift correction measures how much the difference between a participant’s gaze and a central point “drifts” over a short time period. Drift can occur because of factors such as fatigue and changes in body (head) position. Moreover, the longer a viewing session, the more drift that occurs, and thus the less precise the gaze recording.

Fixation duration, frequency, location and sequencing are the primary measures of visual behaviour used to study face processing. Fixation duration provides an index of the speed with which information is processed. Increasing fixation duration is associated with tasks that require more detailed visual analysis (e.g., Xiaohua, C., & Liren, 2007). Frequency of fixations often serves as a measure of sampling quantity. Fixation location and sequencing provide information regarding the regions of the face to which participants are attending and information about the order in which stimulus properties are sampled.

EYE TRACKING TECHNOLOGY

                                       Eye tracking is a technique where by an individual’s eye movements are measured so that the researcher knows both where a person is looking at any given time and the sequence in which their eyes are shifting from one location to another. Tracking people’s eye movements can help HCI researchers understand visual and display-based information processing and the factors that may impact upon the usability of system interfaces. In this way, eye movement recordings can provide an objective source of interface-evaluation data that can inform the design of improved interfaces. Eye movements can also be captured and used as control signals to enable people to interact with interfaces directly without the need for mouse or keyboard input, which can be a major advantage for certain populations of users such as disabled individuals. We begin this chapter with an overview of eye-tacking technology, and progress toward a detailed discussion of the use of eye tracking in HCI and usability research. A key element of this discussion is to provide a practical guide to inform researchers of the various eye-movement measures that can be taken, and the way in which these metrics can address questions about system usability. We conclude by considering the future prospects for eye-tracking research in HCI and usability testing.

    Most commercial eye-tracking systems available today measure point-of-regard by the “corneal-reflection/pupil centre” method . These kinds of trackers usually consist of a standard desktop computer with an infrared camera mounted beneath (or next to) a display monitor, with image processing software to locate and identify the features of the eye used for tracking. In operation, infrared light from an LED embedded in the infrared camera is first directed into the eye to create strong reflections in target eye features to make them easier to track (infrared light is used to avoid dazzling the user with visible light). The light enters the retina and a large proportion of it is reflected back, making the pupil appear as a bright, well defined disc (known as the “bright pupil” effect). The corneal reflection (or first Purkinje image) is also generated by the infrared light, appearing as a small, but sharp, glint (see Figure 1).

 

Figure 1. Corneal reflection and bright pupil as seen in the infrared camera image.

Once the image processing software has identified the centre of the pupil and the location of the corneal reflection, the vector between them is measured, and, with further trigonometric calculations, point-of-regard can be found. Although it is possible to determine approximate point-of-regard by the corneal reflection alone (as shown in Figure 2), by tracking both features eye movements can, critically, be disassociated from head movements

Figure 2. Corneal reflection position changing according to point of regard (cf. Redline & Lankford, 2001).

Video-based eye trackers need to be fine-tuned to the particularities of each person’seye movements by a “calibration” process. This calibration works by displaying a dot on the screen, and if the eye fixes for longer than a certain threshold time and within a certain area, the system records that pupil centre/corneal-reflection relationship as corresponding to a specific x,y coordinate on the screen.

EyePhone Design

o   An eye detection phase

o   An open eye template creation phase

o   An eye tracking phase

o   A blink detection phase

Eye Detection

            By applying a motion analysis technique which operates on consecutive frames, this phase consists on finding the contour of the eyes. The eye pair is identified by the left and right eye contours. While the original algorithm  identifies the eye pair with almost no error when running on a desktop computer with a fixed camera (see the left image in Figure 1).

Figure 1: Left figure: example of eye contour pair returned by the original algorithm running on a desktop with a USB camera. The two white clusters identify the eye pair. Right figure: example of number of contours returned by EyePhone on the Nokia N810. The smaller dots are erroneously interpreted as eye contours.

We obtain errors when the algorithm is implemented on the phone due to the quality of the N810 camera compared to the one on the desktop and the unavoidable movement of the phone while in a person’s hand (refer to the right image in Figure 1). Based on these experimental observations, we modify the original algorithm by:

 i) reducing the image resolution, which according to the authors in reduces the eye detection error rate, and

ii) adding two more criteria to the original heuristics that filter out the false eye contours. In particular, we filter out all the contours for which their width and height in pixels are such that widthmin ≤ width ≤ widthmax and heightmin ≤ height ≤ heightmax. The widthmin, widthmax, heightmin, and heightmax thresholds, which identify the possible sizes for a true eye contour, are determined under various experimental conditions (e.g., bright, dark, moving, not moving) and with different people.

Open Eye Template Creation

 While the authors in adopt an online open eye template creation by extracting the template every time the eye pair is lost (this could happen because of lighting condition changes or movement in the case of a mobile device), EyePhone does not rely on the same strategy. The reduced computation speed compared to a desktop machine and therestricted battery requirements imposed by the N810 dictate a different approach. EyePhone creates a template of a user’s open eye once at the beginning when a person uses the system for the first time using the eye detection algorithm described above1. The template is saved in the persistent memory of the device and fetched when EyePhone is invoked. By taking this simple approach, we drastically reduce the runtime inference delay of EyePhone, the application memory footprint, and the battery drain. The downside of this off-line template creation approach is that a template created in certain lighting conditions might not be perfectly suitable for other environments. We intend to address this problem as part of our future work. In the current implementation the system is trained individually, i.e., the eye template is created by each user when the application is used for the first time. In the future, we will investigate eye template training by relying on pre-collected data from multiple individuals. With this supervised learning approach users can readily use EyePhone without going through the initial eye template creation phase

Eye Tracking:

 The eye tracking algorithm is based on the template matching.

Figure 2: Eye capture using the Nokia N810 front camera running the EyePhone system. The inner white box surrounding the right eye is used to discriminate the nine positions of the eye on the phone’s display. The outer box encloses the template matching region.

The template matching function calculates a correlation score between the open eye template, created the first time the application is used, and a search window. In order to reduce the computation time of the template matching function and save resources, the search window is limited to a region which is twice the size of a box enclosing the eye. These regions are shown in Figure 2, where the outer box around the left eye encloses the region where the correlation score is calculated. The correlation coefficient we rely on, which is often used in template matching problems, is the normalized correlation coefficient defined in . This coefficient ranges between -1 and 1. From our experiments this coefficient guarantees better performance than the one used in. If the normalized correlation coefficient equals 0.4 we conclude that there is an eye in the search window. This threshold has been verified accurate by means of multiple experiments under different conditions.

Blink Detection

To detect blinks we apply a thresholding technique for the normalized correlation coefficient returned by the template matching function as suggested in. However, our algorithm differs from the one proposed in. In the authors introduce a single threshold T and the eye is deemed to be open if the correlation score is greater than T, and closed vice versa. In the EyePhone system, we have two situations to deal with: the quality of the camera is not the same as a good USB camera, and the phone’s camera is generally closer to the person’s face than is the case of using a desktop and USB camera. Because of this latter situation the camera can pick up iris movements, i.e., the interior of the eye, due to Eyeball rotation.

Table 1: EyePhone average eye tracking accuracy for different positions of the eye in different lighting and movement conditions and blink detection average accuracy. Legend: DS = eye tracking accuracy measured in daylight exposure and being steady; AS = eye tracking accuracy measured in artificial light exposure and being steady; DM = eye tracking accuracy measured in daylight exposure and walking; BD = blink detection accuracy in daylight exposure

Figure 3: Eye tracking accuracy for the middlecenter position as a function of different distances between the phone and the eyes when the person is steady and walking.

EVALUATION

In this section, we discuss initial results from the evaluation of the EyePhone prototype. We implement EyePhone on the Nokia N810 [19]. The N810 is equipped with a 400 MHz processor and 128 MB of RAM2. The N810 operat-

ing system is Maemo 4.1, a Unix based platform on which we can install both the C OpenCV (Open Source Computer Vision) library [20] and our EyePhone algorithms which are cross compiled on the Maemo scratchbox. To intercept the video frames from the camera we rely on GStreamer [21], the main multimedia framework on Maemo platforms. In what follows, we _rst present results relating to average accuracy for eye tracking and blink detection for di_erent lighting and user movement conditions to show the performance of Eye- Phone under di_erent experimental conditions. We also report system measurements, such as CPU and memory usage, battery consumption and computation time when running EyePhone on the N810. All experiments are repeated fivetimes and average results are shown

Daylight Exposure Analysis for a Stationary   Subject

The first experiment shows the performance of Eye-Phone when the person is exposed to bright daylight, i.e., in a bright environment, and the person is stationary. The eye tracking results are shown in Figure 2. The inner white box

in each picture, which is a frame taken from the front camera when the person is looking at the N810 display while holding the device in their hand, represents the eye position on the phone display. It is evident that nine di_erent positions

for the eye are identi_ed. These nine positions of the eye can be mapped to nine di_erent functions and applications as shown in Figure 4. Once the eye locks onto a position (i.e., the person is looking at one of the nine buttons on the display), a blink, acting as a mouse click, launches the application corresponding to the button. The accuracy of the eye tracking and blink detection algorithms are reported in Table 1. The results show we obtain good tracking accuracy of the user's eye. However, the blink detection algorithms accuracy oscillates between _67 and 84%. We are studying further improvements in the blink detection as part of future work.

Impact of Distance Between Eye and Tablet.

Since in the current implementation the open eye template is created once at a fixed distance, we evaluate the eye tracking performance when the distance between the eye and the tablet is varied while using EyePhone. We carry out the measurements for the middle-center position in the display (similar results are obtained for the remaining eight positions) when the person is steady and walking. The results are shown in Figure 3. As expected, the accuracy degrades for distances larger than 18-20 cm (which is the distance be tween the eye and the N810 we currently use during the eye template training phase). The accuracy drop becomes severe when the distance is made larger (e.g., _45 cm). These results indicate that research is needed in order to design eye template training techniques which are robust against distance variations between the eyes and the phone.

System Measurements.

 In Table 2 we report the average CPU usage, RAM usage, battery consumption, and computation time of the EyePhone system when processing one video frame { the N810 camera is able to produce up to 15 frames per second. EyePhone is quite lightweight in terms of CPU and RAM needs. The computation takes 100 msec/frame, which is the delay between two consecutive inference results. In addition, the EyePhone runs only when the eye pair is detected implying that the phone resources

are used only when a person is looking at the phone's display and remain free otherwise. The battery drain of the N810 when running the EyePhone continuously for three hours is shown in the 4th column of Table 3. Although this is not a realistic use case, since a person does not usually continuously interact with their phone for three continuous hours,the result indicates that the EyePhone algorithms need to be further optimized to extend the battery life as much as possible.

 Table3; Average CPU usage, RAM usage, and computation time for one video frame. The front camera supports up to 15 frames per second. The last column reports the percentage of used battery by EyePhone after a three hour run of the system

Artificial Light Exposure For A Stationary Subject

In this experiment, the person is again not moving but in an artificially lit environment (i.e., a room with very low daylight penetration from the windows). We want to verify if different lighting conditions impact the system’s performance. The results, shown in Table 1, are comparable to the daylight scenario in a number of cases. However, the accuracy drops. Given the poorer lighting conditions, the eye tracking algorithm fails to locate the eyes with higher frequency. Daylight Exposure for Person Walking. We carried out an experiment where a person walks outdoors in a bright environment to quantify the impact of the phone’s natural movement; that is, shaking of the phone in the hand induced by the person’s gait. We anticipate a drop in the accuracy of the eye tracking algorithm because of the phone movement. This is confirmed by the results shown in Table 1, column 4. Further research is required to make the eye tracking algorithm more robust when a person is using the system on the move.

ADVANTAGE

·        Simply used

·        Hands free interfacing system

DISADVANTAGES

o   Light dependent

o   Accuracy less in night

APPLICATIONS

v EyeMenu:

An example of an EyePhone application is EyeMenu as shown in belowfigure. EyeMenu is a way to shortcut the access to some of the phone’s functions. The set of applications in the menu can be customized by the user. The idea is the following: the position of a person’s eye is mapped to one of the nine buttons. A button is highlighted when EyePhone detects the eye in the position mapped to the button. If a user blinks their eye, the application associated with the button is lunched. Driving the mobile phone user interface with the eyes can be used as a way to facilitate the interaction with mobile phones or in support of people with disabilities.

                                   

 Car Driver Safety:

 EyePhone could also be used to detect drivers drowsiness and distraction in cars. While car manufactures are developing technology to improve drivers safety by detecting drowsiness and distraction using dedicated sensors and cameras , EyePhone could be readily usable for the same purpose even on low-end cars by just clipping the phone on the car dashboard.

Phone face detection(Smart stay):

The Smart stay feature in Samsung Galaxy Series phones uses the front camera to detect when you are looking at your device so that the screen stays on regardless of the screen timeout setting and it auto adjust screen for better using phone. Smart stay technology is works by using the technology of eye phone. Smart stay is a feature on the Samsung Galaxy in which it detects if you are looking at the phone. By doing so, it keeps the screen on while you are using it, but turns if off when you are not using it.

Fig:  Activating smart stay function in Mobile phones.

Smart stay is a feature on the Samsung Galaxy S4,S5 series phones

Conclusion

In this paper, we have focused on developing a HPI technology solely using one of the phone’s growing number of onboard sensors, i.e., the front-facing camera. We presented the implementation and evaluation of the EyePhone prototype. The EyePhone relies on eye tracking and blink detection to drive a mobile phone user interface and activate different applications or functions on the phone. Although preliminary, our results indicate that EyePhone is a promising approach to driving Mobile applications in a hand-free manner.

·