Haytham is an open source gaze tracker suited for head-mounted or remote setups. It provides real-time gaze estimation in the user’s field of view or in a computer display by analyzing eye movement. Haytham offers gaze-based interaction with computer screens in fully mobile situations. The software is built by C#, using Emgu and AForge image processing libraries.
The name Haytham comes from the "father of optics" Ibn al Haytham. He was "the first scientist" who described the eye and its functioning, and he made mathematical descriptions of the properties of light. He is commonly referred to as Ibn al-Haytham or Alhazen who was born in 965 in Basra (in present day Iraq). He made significant contributions to the principles of optics, as well as to astronomy, mathematics, visual perception, and to the scientific method. Ibn al-Haytham’s most famous work is his seven volume treatise on optics, Kitab al-Manazir (Book of Optics, written from 1011 to 1021). He was the first scientist in history to insist that everything be proven through a given method for discovering new information. Ibn al-Haytham is regarded as the father of the modern scientific method.
- Real-time gaze estimation in the user's field of view;
- Remote camera gaze tracking for remote setup;
- Supporting the video files as input for offline processing;
- Exporting the eye/scene videos;
- Exporting the eye tracking data as a .txt file;
- Undistorting the scene image (distortion can occur when you are using wide-angle lenses);
- Real-time plotting the eye data;
- Blink and double blink detection;
- Head gesture recognition. Haytham detects head movements (pitch, yaw, and roll) through the eye movements. There are some predefined gestures in the software and it also lets the users to define custom head gestures;
- Streaming the eye and gaze tracking data (e.g. pupil position, gaze point, pupil diameter, and iris torsion) to the network;
- Streaming the head gestures and other gaze-based commands as well as gaze data through the network via TCP/IP;
- Detecting the visual markers and displays in the scene image;
- Mobile gaze-based interaction with multiple displays in environment;
- Mobile gaze-based interaction with objects in environment.
The Haytham gaze tracker also works with OGAMA. If you need a software for analysing eye and mouse movements check out the OGAMA. The new version works with the Haytham gaze tracker. OGAMA is an open source software for recording and analyzing eye and mouse tracking data.
The Haytham gaze tracker is divided into the main gaze tracker software (server) and variety of application softwares (Haytham_Clients). The Haytham gaze tracker does the eye tracking and scene image processing and streams the data through network via TCP/IP. Haytham_Clients are basically different applications that use gaze tracking and the other data provided by the Haytham server. Haytham is hosted in sourceforge where you can download the binaries or check out the source code.
Download the source code from here. You can download the source code using a svn client (e.g, TortoiseSVN) or you can download the revision that you're looking at as a zip file just by clicking on the "Download Snapshot" link in the sourceForge page.
How to compile the haytham_server source code: There is a file called "compile.bat" in the haytham_server folder. Click on this file and it generates the binary version for you and puts that into the folder called "Haytham_server.bin".
- Windows 7 (32 & 64 bit) or higher;
- .NET Framework 4; and
- Head-mounted or remote eye tracker hardware (check build your own head-mounted eye tracker).
The Haytham is released under a dual license:
The source code is released under GPL version 3.0. In short, it means that any distributed project that includes or links any portion of Haytham source code has to be released with the source code under a license compatible with GPLv3.
Mobile Gaze Tracking
Head-mounted gaze tracking
Launch the Haytham gaze tracker and choose the mobile gaze tracking.
In the "Camera" tab, choose the eye camera and the scene camera and press start. The images will be displayed in the window, as show in Figure 1.
Figure 1: Configuration Tabs.
Setup Eye Tracking
By default the software is able to detect the pupil when the eye image is like Figure 2 (normal eye), if not go to the "Eye" tab and adjust the iris diameter until the gray circle in the image fits to the iris border. Then Adjust the sensitivity/threshold until the color of the pupil is green and the pupil is tracked well. Do the same for the glint until you see the line from the pupil center to the center of the glint. Using glint detection helps to keep the accuracy when the eye camera is not completely fixed relative to the eye. Be sure that the glint is detected precisely when moving the eye, otherwise disable the glint detection.
Figure 2: Detected pupil and glint (left) a normal infrared eye image (right).
A calibration process is needed for mapping the pupil position into a point in the field of view. You can either use 4 points (easier) or 9 points for calibration. For calibration, go to the "Calibration" tab. Press the "4 Point Homography" button. Then you need to click on 4 points in the scene image while you are looking at the corresponding points in front of you (in your field of view). All the points should be on a fronto-parallel plane with the distance dc from you. That’s it! Now you see the gaze point shown in the scene image by a red cross, as shown in Figure 3.
Figure 3: Haytham window showing the eye and the scene images. The gaze point is shown by a red cross in the scene image.
Sometimes because of the relative movements between head-mounted camera and the head, a small constant offset will be observed in the gaze estimation (especially when not using the glint), in this case you can easily look at a point in the calibration distance and click on the corresponding point in the scene image. Therefore, after the main calibration the scene image will be active and if you click on a point the eye tracker correct the gaze estimation offset.
Interact with Computer Display
Launch the Haytham_Monitor on a computer connected to your local network. Input the IP address of the server and press "start". Once the connection is established, you should be able to see your gaze point shown by an icon in the display when the client screen is detected in the scene image, if not adjust the screen detection sensitivity in the "scene" tab of the server.
Figure 4: Detected screen in the scene image.
You can also move the mouse cursor in the client screen by clicking on the "mouse control" button. The cursor will follow the gaze position on the screen.
A common problem of monocular head-mounted eye trackers is that they introduce gaze estimation errors when the distance between the point of regard and the user is different than when the system was calibrated (dc). This error is due to the scene camera and the eye are not co-axial (a.k.a. parallax error). Therefore you need to do the calibration in a distance in which you need the estimated gaze later. Generally, the calibration done in a closer distance is valid for a smaller range, as shown in Figure 5.
Figure 5: The range of the acceptable accuracy relative to the calibration distance.
Remote gaze tracking
Contact-free gaze tracking.
Launch the Haytham gaze tracker and choose the remote gaze tracking. Adjust the pupil/glint tracking as it is described above for mobile gaze tracking. For calibration go to the calibration tab and press "4 points calibration" button. Then you just need to look at the red marker shown in the screen after starting the calibration and follow the marker.
For Interact with computer display, launch the Haytham_Monitor on the same screen that you have calibrated the system on it. Input the IP address of the server and press "start". Once the connection is established, you should be able to see your gaze point shown by an icon in the screen. For moving the cursor, click on the "mouse control" button.
Haytham gaze tracker detects a variety of commands that you can use together with gaze for interaction with computer. Blinking, dwell-time, and the eye-based head gestures are detected by the Haytham gaze tracker. The video below shows how to setup and use these commands for interaction with computer.
Build Your Own Head-Mounted Eye Tracker
A head-mounted eye tracker consists of two cameras, one for capturing the eye image (eye camera) and the other one for capturing the user's field of view (scene camera). Haytham works based on the dark pupil method and it needs an infrared (IR) image of the eye, therefore the eye camera should be infrared (night vision). You probably need some filters that can block all visible light but don't block infrared light. A cheep solution would be to use photography film negatives (Figure 6). Find some unexposed (black) processed film (there should be some black bits at the end of the old negatives), and use put them in front of the lens or the CCD of the camera. You can also buy some visible light cut off filters that look like black pieces of glass that is transparent for the light of certain wavelength range. I found these in eBay. Just remember to know the wavelength of the infrared leds that you are using to buy a appropriate filter.
Figure 6: Example of low-cost IR filter block.
Wireless Head-Mounted Eye Tracker
If you would like to have a wireless eye tracker, there are some wireless night vision IR pinhole cameras (e.g., link1 and link2) that work fine for the distance about 10 meters (Figure 7). I have used this cameras couple of years ago on Dias eye tracker, and they worked pretty well. You just need some AV adapter (EasyCap USB) that capture the analog image from the wireless receiver and convert it to digital.
Figure 7: Example of two wireless night-vision cameras.
I have also found some very tiny infrared cameras (Figure 8) from a Chinese Company and they can be used as both eye camera and the scene camera:
Figure 8: Other infrared cameras that you can used to build your now head-mounted eye tracker.
Find a plastic glasses frame and a small telescopic antenna (used as an arm) and mount the cameras on the frame (Figure 9). The good point of using the antenna is that it is very solid against the shakes and vibration, and on the other hand you can use the telescopic feature of the antenna (extensibility and twisting the element) to give some degree of freedom to the eye camera. Therefore the eye camera can be easily adjusted in front of the eye.
Figure 9: Example of the use of a telescopic antenna to mount the cameras on head-mounted eye tracker.
In Figure 10, you can see the images captured by the wireless cameras in the Haytham software, and the video of the wireless head-mounted gaze tracker.
Figure 10: Examples of some images captured during an eye tracking session.
Wired Head-Mounted Eye Tracker
Genius islim 321R is a light weight night vision webcam that provides a sharp IR image of the eye and is good for making a head mounted eye tracker. This webcam has the infrared leds on it and you don't need to add a light source for the eye. You can either use a cap or a glasses frame to mount the cameras on it. Use a piece of wire and mount the eye camera in front of one of the eyes (distance about 10cm). Place the scene camera as close as possible to the eye in order to reduce the parallax error, as shown in Figure 11.
Figure 11: An wired head-mounted eye tracker.
You can actually use the ball joint of the webcam on the eye tracker. Disassemble the joint, use the glue and attach the lower part of the joint on the cap, as show in Figures 12 and 13. :)
Figure 12: Example of a scene camera used by a wired head-mounted eye tracker.
Figure 13: Cameras used by the wired head-mounted eye tracker.
Some Useful Links
For more information about how to build your own head-mounted eye tracker, please access the following links:
- Make a Lightweight Aluminium Frame for Head-Mounted Eye Tracking;
- How to Build Low Cost Eye Tracking Glasses for Head Mounted System; and
- Build an Eyewriter.
Connecting to the gaze tracker and receiving the gaze data
The Haytham_SimpleClient is a simple program that shows how to connect to the Haytham server through TCP/IP, and receive the eye tracking and scene image data.
Download: source code
Interaction with computer displays
Haytham_Monitor allows you to interact with your computer using gaze and gestures detected by the Haytham gaze tracker. Showing the gaze point in the screen, moving the cursor with gaze, performing different actions by commands provided by the Haytham gaze tracker, and exporting the gaze data are the main features of this client.When the Haytham gaze tracker is working in remote gaze tracking mode, the Haytham_Monitor should be executed in the same display that the system is calibrated on it. In the mobile gaze tracking mode, you can run this client in any display connected to the network. The Haytham gaze tracker detects and recognizes the display that you are looking at and automatically transfer the gaze data to that display. It is possible to run multiple instances of the Haytham_Monitor in different displays in the network (in mobile gaze tracking mode). The video below shows a quick demo of how to use the eye tracker for interaction with multiple displays in the environment.
You can also export the gaze data in the coordinates system of the display (in pixels) as a text file.
The Haytham_RecipeDemo is intended for a hands-free interaction with a recipe when cooking. The user interface consists of three areas: the recipe slides frame, a simple music player and the volume frame. The interface is fully operated by looking at these predefined regions while doing head gestures. This demo is intended to show the potential of using gaze trackers for fully mobile hands-free interaction with display. The video below shows a quick demo of the Haytham_RecipeDemo works.
The Haytham_Roomba is an application example of how to use gaze and only eye tracker for controlling a robot in the environment. The roomba vacuum cleaner robot is used because it actually has a useful function (cleaning) and also it is a good example of a mobile robot. With this client you can either control and drive the roomba with head gestures (measured by the Haytham gaze tracker) or with your gaze. In the gaze mode, roomba will go to the point that you are looking at on the floor. A visual marker is used for detecting the roomba in the scene image of the head-mounted eye tracker. There is also a simple application (written in C#) in the source code folder which can be used to understand more about controlling roomba via Bluetooth. You may need that if you are interested to extend the abilities of the Haytham_Roomba. The video below shows a quick demo of how this client works. There is also another demo about gaze-based controlling the roomba in the video made for the Haytham_SerialPortSwitch client.
The Haytham_SerialPortSwitch is intended to show how to use a mobile eye tracker for controlling real objects in the environment when the user is fully mobile. This client receives the eye tracking data and the eye-based commands provided by the Haytham gaze tracker, and uses them for controlling the objects connected to the client computer via a serial port. The first version of this client is designed for switching 10 different appliances in the home. Some visual markers are used for detecting objects in the user’s field of view by the eye tracker. The video bellow shows a quick demo of this client software.
The Haytham_SerialPortTV is a client software for the Haytham gaze tracker that can be used for controlling TV with gaze. TV is an example of a complex appliance that is more challenging to be control only by an eye tracker. This client is actually a combination of the Haytham_Monitor and the Haytham_SerialPort clients. The Haytham gaze tracker detects the TV screen like a computer display and sends the point that the user is looking (in the TV screen) to the Haytham_SerialPortTV. This client uses the eye-based commands and gaze for controlling the TV as a serial device. All the basic actions for controlling the TV can be done in this application e.g., looking at an area in the TV screen and adjusting the volume by continuous vertical head movement.
This version of the software is designed for LG TV panel of model LG55LE550, but you can easily modify the source code to work on you TV (with a serial connection). The video bellow shows a quick demo of this client.
Low-Cost Eye Trackers for Sports Training
This research project is being developed by Camilo Rodegheri Mendes dos Santos and Martina Navarro, under orientation of Prof. Ronald Ranvaud at the Laboratory of Physyology of Behavior, Department of Physiology and Biophysics, Institute of Biomedical Sciences of the University of Sao Paulo, São Paulo, Brazil.
Figure 14: Player using a head-mounted eye tracker.
The main goal is to develop training strategies to improve perceptual-cognitive skills of athletes incorporating low cost eye-tracker equipment. More specifically, it aims to provide for any athlete an accessible (i.e. very low investment) and reliable equipment that can robustly evaluate individual gaze patterns. Based on these individual evaluations, each athlete could be further instructed according appropriate training strategies. Additionally, these low cost eye-trackers can also offer a method in monitoring the adoption and the effectiveness of the specific training strategy.
This project uses Haytham gaze tracker as a low cost solution for mobile gaze estimation, as shown in Figures 14 and 15.
Figure 15: The player using the head-mounted eye tracker during a training session.
Visual search of taekwondo expert before the opponent’s attack. The athlete had two very long fixations before being attacked, first (Figure 15.1) in the head and the second (Figure 15.2) in the shoulder region.
Eye Tracking Experiment at ARKEN Museum of Copenhagen
There was an exhibition in the ARKEN art museum of Copenhagen and there was a very cool installation there designed by the artist Olafur Eliasson. The installation was called "Your blind passenger" and it was basically a 90-metre-long tunnel with a dense fog inside, and some white and yellow color lights at the beginning and the end of the tunnel (Figure 16). It is interesting that when you go through the tunnel and go from the yellow color area to the white area you can actually see the afterimage effect and instead of seeing the white light you see a purple background in the field of view (Lateral Inhibition Phenomena). The eye tracking experiment was conducted to see how the eyes behave inside the tunnel when there is no stimuli in the field of view. In the video below you can see the recorded eye and the scene videos with the gaze point in three different trials.
Figure 16: Your blind passage exposition.
Interacting with Objects in the Environment by Gaze and Hand Gestures
This project was done by Jeremy Hales under supervision of David Rozado at ICT Centre of CSIRO, Australia. The project was about using gaze and hand gestures in a multimodal interaction senario for controlling the environment, as shown in Figure 17. A wireless head-mounted gaze tracker made from low-cost hardware was used for both estimating the point regard and for detecting the hand gestures in the scene image. The haytham gaze tracker was used for continuous and mobile monitoring of a subject’s point of regard and for recognizing the visual markers attached to the objects in the surrounding environment. The emerging system permitted a subject to move freely in an environment, select the object he wants to interact with using gaze (identification) and transmit a command to it by performing a hand gesture (control).
Figure 17: Multimodal interaction using gaze and hand gestures for controlling an environment.
Mobile Gaze-Based Controlling a Vehicle
This was a bachelor project done by Per Mortensen and Jacob Rasmussen at IT University of Copenhagen. In essence the project concerns a product that allows a user to control a Roomba vacuum cleaning robot inside a given room using data from a mobile gaze tracker (Figure 18). When the user looks at a point in the room and performs a certain gesture, the Roomba will attempt to navigate towards the given destination point and optionally start cleaning on arrival.
Figure 18: Mobile gaze-based controlling a Roomba vacuum.
This was a bachelor project done by Per Mortensen and Jacob Rasmussen at IT University of Copenhagen. In essence the project concerns a product that allows a user to control a Roomba vacuum cleaning robot inside a given room using data from a mobile gaze tracker. When the user looks at a point in the room and performs a certain gesture, the Roomba will attempt to navigate towards the given destination point and optionally start cleaning on arrival, as show in Figure 19.
Figure 19: Setup of Roomba vacuum controlled by eye movements.
The software developed in this project communicate with the Haytham gaze tracking server and receives the gaze data and the recognized visual marker in the scene image. It finds the path for the roomba to go from the current position to the gazed destination point and sends the control commands to the Roomba.
Haytham with Oculus Rift
Recently we have been playing around with the oculus rift and the Haytham gaze tracker. The idea was to some how simulate a see through HMD and combining it with the haytham gaze tracker. The only thing we did was to display the scene image on the rift screen and attaching a eye camera to the head gear. The user will see his front view through the rift and we do the gaze estimation inside that image. We decided to share the code here so that other people that are interested in combining gaze tracking in the rift can use the code. In fact we didn’t spend too much time on it and we just wanted to make something that works on C#. We used the openCV for doing the barrel distortion and showing the stereo images but it was very slow. So we ended up using the OculusArDroneKinect project developed by Alessandro Colla which is using a simple pixel shader in XNA for rendering the views. Check out this interesting project that uses Oculus Rift and a Kinect to drive a Parrot Ar.Drone! Anyway, we modified parts of his code so that we could use in our prototype. Instead of making a client for the rift that connects to the haytham_server, all the changes are directly done in the server side. There is no stereo camera and we only attached a webcam to the rift for both left and right views.
An infrared wireless camera (from 3RD Eye) is used for capturing the eye. You might need to cut and remove a small piece of plastic of the headset to make some room for the camera. I use an tick aluminium wire for mounting the eye camera, as shown in Figures 20 e 21.
Figure 20: Head-mounted eye tracker used in the Oculus Rift project.
Figure 21: Internal structure of head-mounted eye tracker.
After attaching the eye camera next to the eye and the front view camera in from of the headset you are ready to use the system. Download the source code from here (if you are interested in improving the code make a branch of the Haytham project in svn!). You also need to install the XNA framework from <https://msxna.codeplex.com/releases>. After running the program start the eye and the scene cameras. If you want to try the system on your eyes you need someone else to help you adjusting the eye tracking sliders and calibrating the gaze tracker. Remember to connect the oculus to your computer (since we are not gonna use the sensors of the rift only connecting the VGA is sufficient) and make sure that the oculus is being detected as the second monitor of the system. You need to extend the main screen of you computer to the secondary monitor (oculus) inside Windows. You can see a button called Oculus at the top/right of the main window of the Haytham which is for showing the eye or scene image (you can choose which one) on the oculus. There is also a slider next to this button that allows you to change the disparity. Below you can see the front view image with gaze tracking inside the image shown on oculus rift:
Let your Google Glass know what exactly your eyes are looking at!
What is Gaze Glass?
GlassGaze is an android app designed for Google Glass that makes it possible to do gaze tracking on the Glass. It allows the Google Glass to know where exactly you are looking at. You can interact with the head-mounted display with you eyes. With GlassGaze, your google Glass not only knows where and what point in the display you are looking at but also knows where you are looking in your field view. Now, not only you can take picture or video with your Glass but also you can record your gaze point inside the image and video and share them with your friends. It would be very interesting for your friends to know what you where you were watching in front of you!
GlassGaze is actually a client for the Haytham gaze tracker. Both the Haytham gaze tracker and the GlassGaze client are freeware and open-source. You just need to attach a small camera to the Google Glass frame to capture the eye image. The camera send the image to a remote computer that the Haytham gaze tracker is runnign on it. All the eye tracking processing will be done in the computer and the gaze data will be sent to the Glass through Wifi. The need for a remote computer for the main processing in our first prototype is due to the low processing power and limited battery life of the current Google Glasses.
GlassGaze actually provides a basic framework for experiencing the eye and gaze tracking on the Google Glass. The gaze data streamed by the Haytham gaze tracker will be received by the GlassGaze app. The app facilitate the user to calibrate the system for both gaze tracking while watching the display and while viewing the scene. Besides the demo apps inside the application that show the main features of the app, as an developer you have access to the gaze tracking data from other applications running on the Glass!
What Does It Do?
- Gaze estimation inside the head-mounted display that can for example be used for interaction with the Glass; and
- Gaze estimation inside the image captured by the front view camera of the device.
How to Use GlassGaze?
The Figure 22 shows the main components of the system. Following on this page, is more detailed information you need to know about setting up the system.
Figure 22: Components of the GlassGaze system.
In terms of software you basically just need to download and install the Haytham_server (currently only available for windows) on your computer, and download the GlassGaze apk version and install it on your Glass. Both computer and Glass need to be in the same local network. All the communications are handled through TCP sockets.
Installing the Haytham_server on the windows is very straightforward. Once you have installed the software on your machine launch the application and select the Google Glass mode from the first window that pops up. After you see the main window of the Haytham, you need to select the eye camera in the camera tab and start the capturing. You can then adjust the pupil tracking to get a relatively robust pupil detection in the image.
Download the apk version of the GlassGaze app and install it on your Glass using adb (steps are described below). Alternatively, if you are familiar with android programming, get the source code from GitHub and compile the code to your Glass using Android studio.
Installing the GlassGaze.apk on Google Glass. An easy way of doing this (for beginners) has been described in the following steps:
- Connect your Google Glass to the usb port of your computer;
- Download the latest usb driver for Google Glass from here;
- Extract the folder. Before installing the driver open the android_winusb.inf which is inside the folder because you probably need to modify this file as follows:
- Go to device manager and copy the VID and PID of you Glass: (Control Panel/System/Device Manager/Portable Devices/Glass 1/Properties/Details/Hardware Ids).
- Look inside the android_winusb.inf file and see if you can find similar lines as below. The VID and PID values should match what you have for your Glass. If it does not match copy corrected version of these two lines to these two sections: [Google.NTx86] and [Google.NTamd64];
- ;Google Glass
- %SingleAdbInterface% = USB_Install, USB\VID_18D1&PID_4E11&REV_0216
- %CompositeAdbInterface% = USB_Install, USB\VID_18D1&PID_4E11&MI_01
- Make sure that Driver Signature Enforcement has been disabled in Windows;
- Install the modified version of the driver for you Glass. (Control Panel/System/Device Manager/Other Devices/Glass 1/ Update Driver Software/Browse my computer for driver software/);
- Install the Android Debug Bridge (ADB) on your computer. (There is a good guide in this page)
- Open CMD (Command Prompt) or Terminal (for Macs) and type adb.exe devices;
- You should see the Google Glass device listed. If it is listed as an unauthorized device make sure that you have Debug Mode Enabled on your Glass;
- Navigate the command prompt to where you have the file GlassGaze.apk;
- Install the GlassGaze by typing adb install GlassGaze.apk
- Google Glass;
- A night-vision camera (you can just remove the infrared filter of a regular webcam. It usually sits behind the lens);
- An infrared LED; and
- A windows machine!
You can use any kind of camera as long as it’s night vision, as shown in Figures 23 and 24. You need to attach the camera to the frame of the Glass. Figure below shows a wired version using Genius "islim 321R" webcam attached to the Glass and also a wireless version which is more convenient.
Figure 23: Prototype with a webcam (Genius "islim 321R").
Figure 24: Prototype with a wireless camera.
*NOTE: You don’t actually get a clean image from this wireless cameras and the image sometimes gets very noisy, however, this is a very cheap and small camera that works and it can easily be mounted on a Google Glass.
Please post your questions to the discussion forum of the Haytham project: