Replace your mouse with your face. Control the cursor just by moving your head. Click by winking your eyes, scroll by raising and lowering your eyebrows. All of that is possible now with ExpressionMouse Kinect.
During the usage of our KinectMouse, we figured out that it is very exhausting for your arm, if you have to control the mouse cursor all the time with your hand. So we were looking for a more easier method to control the mouse cursor with Kinect. All you have to do is moving your head and using some facial expressions for certain mouse actions.
See how this new kind of mouse control works:
How does our application work?
It is really easy: Just move your head to control the cursor. Make sure that Kinect can see your face as well as your chest. Sometimes the inital recognition is better when you are waving. It is normal that Kinect needs a few seconds to identify your face correctly. In contrast to KinectMouse, ExpressionMouse Kinect is more precise when you are more close to the sensor (but not too close) as the sensor has a more detailed view on your face this way. One meter should be a sufficient distance.
Just wink with your right eye about a second. At this point you may ask why you have to use your right eye for a left click and not your left eye. During testing we figured out that the sensor detected our right eyes much more accurate than our left eyes. So we decided to swap left and right. As left clicks are much more frequent than right clicks, we think it is a good idea to use the most sensitive wink for the left click.
Just wink with your left eye about a second.
Wink with both of your eyes at the same time. Then a double click will be executed.
Raise your eyebrows for scrolling up and lower it for scrolling down.
Drag & Drop
Open your mouth for starting drag & drop. Move your head to move the cursor and keep your mouth open. For dropping, just close your mouth.
Find the correct settings for yourself
Every face is different. It could be that the preselected settings in ExpressionMouse Kinect are not optimal for you. Just play a bit with the thresholds until you are satisfied.
- ClickDelay: Timespan in frames (Normally Kinect works with 30 fps) which have to elapse between two mouse actions.
- Headrotation Smoothing Filter Values: Frameweights for calculating weighted average of your head rotation. Used for smoothing the cursor motion. If you enter the following the cursor will become more precise, but it will also a bit more delayed: “2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1”
- Percentage of horizontal edge Pixels: Used for differentiating between open and closed eye. A higher value means that the closed eye detection is less sensitive. A lower value makes it more sensitive.
- Used frames for closed eye detection: More frames increases accuracy in closed eye detection, but it also increases the timespan between closing the eye and the execution of the mouseclick.
- Eye closed filter threshold: Used for differentiating between open and closed eye. A higher value means that the closed eye detection is less sensitive. A lower value makes it more sensitive.
- Double click second eye threshold: Threshold for differentiating between a normal click and a double click. If doubleclicks are not recognized correctly, you should decrease this value. If normal clicks are recognized as doubleclicks, you should increase this value.
- Brow raiser start threshold: Threshold for raising your brow. Decrease value, if raising your brow is not recognized. If your computer is scrolling up, even if you are not raising your brow, increase this value.
- Brow lowerer start threshold: Same as Brow raiser start threshold, but for lowering your brow and scrolling down.
- Mouth open start threshold: Threshold for opening your mouth (executing MouseDown Event). Increase if opening your mouth is not recognized correctly. Decrease if MouseDown is executed, even if your mouth is closed.
- Mouth open confirmation: Decrease this value, if MouseUp is executed, even if your mouth is still open.
- Mouth open end threshold: Threshold for closing your mouth (executing MouseUp Event). Increase if closing your mouth is not recognized correctly. Decrease if MouseUp is executed, even if your mouth is still open.
- Scroll multiplier up: A higher value means that scrolling up is faster.
- Scroll multiplier down: A higher value means that scrolling down is faster.
- Head to Screen relation X – Width: Sensitivity of the mouse cursor in horizontal direction. A higher value means less sensitivity.
- Head to Screen relation Y – Height: Sensitivity of the mouse cursor in vertical direction. A higher value means less sensitivity.
The free version only works with Kinect for Windows sensor. The PRO version saves your changed settings automatically and also works with Xbox 360 Kinect sensor.
I’m always happy to get some feedback. So please comment and let me know, if you are satisfied with this app. If you have any questions, feel free to ask.
As the stores for the PRO version are no longer available since a while a go and as I don’t have much time currently to maintain this project, I decided to make this app Open Source. You get the “PRO version” via the download link, now.
Check out the Source here:
ExpressionMouse on GitHub
- I’ve uploaded a new version 1.0.2 of ExpressionMouse Kinect with several minor bugfixes.
- There is a PRO version available, now. The PRO version saves your settings automatically. It supports also Kinect sensor for Xbox 360 as well as Kinect for Windows sensor. You can download it via the Motionfair Store.
- Due to a Trademark complaint this app is called ExpressionMouse by now.
- Users of PRO version: If you use Kinect for Xbox sensor, you have to install the Kinect SDK. It is not sufficient to run ExpressionMouse with Kinect runtime and a Xbox Kinect sensor.