Rss

  • youtube

Archives for : Kinect for Windows

Use your face as mouse control (with Kinect)

Replace your mouse with your face. Control the cursor just by moving your head. Click by winking your eyes, scroll by raising and lowering your eyebrows. All of that is possible now with ExpressionMouse Kinect.

During the usage of our KinectMouse, we figured out that it is very exhausting for your arm, if you have to control the mouse cursor all the time with your hand. So we were looking for a more easier method to control the mouse cursor with Kinect. All you have to do is moving your head and using some facial expressions for certain mouse actions.

See how this new kind of mouse control works:

How does our application work?

Cursor Moving

It is really easy: Just move your head to control the cursor. Make sure that Kinect can see your face as well as your chest. Sometimes the inital recognition is better when you are waving. It is normal that Kinect needs a few seconds to identify your face correctly. In contrast to KinectMouse, ExpressionMouse Kinect is more precise when you are more close to the sensor (but not too close) as the sensor has a more detailed view on your face this way. One meter should be a sufficient distance.

Left Click

Just wink with your right eye about a second. At this point you may ask why you have to use your right eye for a left click and not your left eye. During testing we figured out that the sensor detected our right eyes much more accurate than our left eyes. So we decided to swap left and right. As left clicks are much more frequent than right clicks, we think it is a good idea to use the most sensitive wink for the left click.

Right Click

Just wink with your left eye about a second.

Double Click

Wink with both of your eyes at the same time. Then a double click will be executed.

Scrolling

Raise your eyebrows for scrolling up and lower it for scrolling down.

Drag & Drop

Open your mouth for starting drag & drop. Move your head to move the cursor and keep your mouth open. For dropping, just close your mouth.

Find the correct settings for yourself

Every face is different. It could be that the preselected settings in ExpressionMouse Kinect are not optimal for you. Just play a bit with the thresholds until you are satisfied.

  • ClickDelay: Timespan in frames (Normally Kinect works with 30 fps) which have to elapse between two mouse actions.
  • Headrotation Smoothing Filter Values: Frameweights for calculating weighted average of your head rotation. Used for smoothing the cursor motion. If you enter the following the cursor will become more precise, but it will also a bit more delayed: “2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1”
  • Percentage of horizontal edge Pixels: Used for differentiating between open and closed eye. A higher value means that the closed eye detection is less sensitive. A lower value makes it more sensitive.
  • Used frames for closed eye detection: More frames increases accuracy in closed eye detection, but it also increases the timespan between closing the eye and the execution of the mouseclick.
  • Eye closed filter threshold: Used for differentiating between open and closed eye. A higher value means that the closed eye detection is less sensitive. A lower value makes it more sensitive.
  • Double click second eye threshold: Threshold for differentiating between a normal click and a double click. If doubleclicks are not recognized correctly, you should decrease this value.  If normal clicks are recognized as doubleclicks, you should increase this value.
  • Brow raiser start threshold:  Threshold for raising your brow. Decrease value, if raising your brow is not recognized. If your computer is scrolling up, even if you are not raising your brow, increase this value.
  • Brow lowerer start threshold: Same as Brow raiser start threshold, but for lowering your brow and scrolling down.
  • Mouth open start threshold: Threshold for opening your mouth (executing MouseDown Event). Increase if opening your mouth is not recognized correctly. Decrease if MouseDown is executed, even if your mouth is closed.
  • Mouth open confirmation: Decrease this value, if MouseUp is executed, even if your mouth is still open.
  • Mouth open end threshold: Threshold for closing your mouth (executing MouseUp Event). Increase if closing your mouth is not recognized correctly. Decrease if MouseUp is executed, even if your mouth is still open.
  • Scroll multiplier up: A higher value means that scrolling up is faster.
  • Scroll multiplier down: A higher value means that scrolling down is faster.
  • Head to Screen relation X – Width: Sensitivity of the mouse cursor in horizontal direction. A higher value means less sensitivity.
  • Head to Screen relation Y – Height: Sensitivity of the mouse cursor in vertical direction. A higher value means less sensitivity.

The free version only works with Kinect for Windows sensor. The PRO version saves your changed settings automatically and also works with Xbox 360 Kinect sensor.

I’m always happy to get some feedback. So please comment and let me know, if you are satisfied with this app. If you have any questions, feel free to ask.


Update (20.01.2017):

As the stores for the PRO version are no longer available since a while a go and as I don’t have much time currently to maintain this project, I decided to make this app Open Source. You get the “PRO version” via the download link, now.
Check out the Source here:
ExpressionMouse on GitHub

Update:
  1. I’ve uploaded a new version 1.0.2 of ExpressionMouse Kinect with several minor bugfixes.
  2. There is a PRO version available, now. The PRO version saves your settings automatically. It supports also Kinect sensor for Xbox 360 as well as Kinect for Windows sensor. You can download it via the Motionfair Store.
  3. Due to a Trademark complaint this app is called ExpressionMouse by now.
  4. Users of PRO version: If you use Kinect for Xbox sensor, you have to install the Kinect SDK. It is not sufficient to run ExpressionMouse with Kinect runtime and a Xbox Kinect sensor.

Use your hand instead a mouse (with Kinect for Windows)

Within an university programming project we tried out several methods to control the mouse cursor with Kinect for Windows. One of the results is a small application, which lets you control the Windows mouse cursor with your hands. Generally the cursor moves like the Kinect cursor on Xbox, but in our opinion there are several disadvantages on the Xbox Kinect mouse control:

  1. On Xbox you need to hold the cursor for about a second over a tile if you want to click on it. This is an unnecessary delay. So we were looking for a clicking method, which is as fast as a click on a PC mouse.
  2. With the Xbox Kinect control you only have the opportunity to make a standard click. There is neither a possibility for a right click nor a possibility for a double click and also no possibilty for Drag & Drop. But you need all of these special mouse actions, when you are working with your PC.

How does our application work?

Cursor Moving

Just move your hand for controlling the cursor. The recognition of your hand is a bit insensitive when you are too close to the sensor. If the cursor is too much jumping, just go one or two steps back. When you start the application it takes a moment for the sensor to recognize your hand. Make sure that Kinect can see your face as well as your chest and both of your hands. Sometimes the inital recognition is better when you are waving.

Left Click

As Kinect recognizes the distance from your hand to the sensor, clicking is possible by moving your hand just a bit forward and then directly backwards (see picture below). Just make a little motion, the sensor should recognize that and executes the click on the position where the mouse cursor was, when you started the motion.

Click

Double Click

Double clicking works the same way like normal clicking. The difference to a normal click is that you have to move your hand just a bit more forward and backward.

Right Click

For a right click the motion is quite similar to the motion for the left click. Use your left hand and move it just a bit backward and then directly forward (instead forward and backward as at the left click).
It’s important that you are working with your left hand when using right click, as there is an incompatibility with the distinction between right click and Drag & Drop.

Drag & Drop

Just move your right hand a bit back. This motion will execute a Mouse Down Event. Then you can move the cursor. When you move your hand back forward, the Mouse Up Event is executed. Drag & Drop ends at this point.

Drag and Drop