Rss

  • youtube

Use your face as mouse control (with Kinect)

Replace your mouse with your face. Control the cursor just by moving your head. Click by winking your eyes, scroll by raising and lowering your eyebrows. All of that is possible now with ExpressionMouse Kinect.

During the usage of our KinectMouse, we figured out that it is very exhausting for your arm, if you have to control the mouse cursor all the time with your hand. So we were looking for a more easier method to control the mouse cursor with Kinect. All you have to do is moving your head and using some facial expressions for certain mouse actions.

See how this new kind of mouse control works:

How does our application work?

Cursor Moving

It is really easy: Just move your head to control the cursor. Make sure that Kinect can see your face as well as your chest. Sometimes the inital recognition is better when you are waving. It is normal that Kinect needs a few seconds to identify your face correctly. In contrast to KinectMouse, ExpressionMouse Kinect is more precise when you are more close to the sensor (but not too close) as the sensor has a more detailed view on your face this way. One meter should be a sufficient distance.

Left Click

Just wink with your right eye about a second. At this point you may ask why you have to use your right eye for a left click and not your left eye. During testing we figured out that the sensor detected our right eyes much more accurate than our left eyes. So we decided to swap left and right. As left clicks are much more frequent than right clicks, we think it is a good idea to use the most sensitive wink for the left click.

Right Click

Just wink with your left eye about a second.

Double Click

Wink with both of your eyes at the same time. Then a double click will be executed.

Scrolling

Raise your eyebrows for scrolling up and lower it for scrolling down.

Drag & Drop

Open your mouth for starting drag & drop. Move your head to move the cursor and keep your mouth open. For dropping, just close your mouth.

Find the correct settings for yourself

Every face is different. It could be that the preselected settings in ExpressionMouse Kinect are not optimal for you. Just play a bit with the thresholds until you are satisfied.

  • ClickDelay: Timespan in frames (Normally Kinect works with 30 fps) which have to elapse between two mouse actions.
  • Headrotation Smoothing Filter Values: Frameweights for calculating weighted average of your head rotation. Used for smoothing the cursor motion. If you enter the following the cursor will become more precise, but it will also a bit more delayed: “2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1″
  • Percentage of horizontal edge Pixels: Used for differentiating between open and closed eye. A higher value means that the closed eye detection is less sensitive. A lower value makes it more sensitive.
  • Used frames for closed eye detection: More frames increases accuracy in closed eye detection, but it also increases the timespan between closing the eye and the execution of the mouseclick.
  • Eye closed filter threshold: Used for differentiating between open and closed eye. A higher value means that the closed eye detection is less sensitive. A lower value makes it more sensitive.
  • Double click second eye threshold: Threshold for differentiating between a normal click and a double click. If doubleclicks are not recognized correctly, you should decrease this value.  If normal clicks are recognized as doubleclicks, you should increase this value.
  • Brow raiser start threshold:  Threshold for raising your brow. Decrease value, if raising your brow is not recognized. If your computer is scrolling up, even if you are not raising your brow, increase this value.
  • Brow lowerer start threshold: Same as Brow raiser start threshold, but for lowering your brow and scrolling down.
  • Mouth open start threshold: Threshold for opening your mouth (executing MouseDown Event). Increase if opening your mouth is not recognized correctly. Decrease if MouseDown is executed, even if your mouth is closed.
  • Mouth open confirmation: Decrease this value, if MouseUp is executed, even if your mouth is still open.
  • Mouth open end threshold: Threshold for closing your mouth (executing MouseUp Event). Increase if closing your mouth is not recognized correctly. Decrease if MouseUp is executed, even if your mouth is still open.
  • Scroll multiplier up: A higher value means that scrolling up is faster.
  • Scroll multiplier down: A higher value means that scrolling down is faster.
  • Head to Screen relation X – Width: Sensitivity of the mouse cursor in horizontal direction. A higher value means less sensitivity.
  • Head to Screen relation Y – Height: Sensitivity of the mouse cursor in vertical direction. A higher value means less sensitivity.

The free version only works with Kinect for Windows sensor. The PRO version saves your changed settings automatically and also works with Xbox 360 Kinect sensor.

I’m always happy to get some feedback. So please comment and let me know, if you are satisfied with this app. If you have any questions, feel free to ask.


Update:
  1. I’ve uploaded a new version 1.0.2 of ExpressionMouse Kinect with several minor bugfixes.
  2. There is a PRO version available, now. The PRO version saves your settings automatically. It supports also Kinect sensor for Xbox 360 as well as Kinect for Windows sensor. You can download it via the mageca Store.
  3. Due to a Trademark complaint this app is called ExpressionMouse by now.
  4. Users of PRO version: If you use Kinect for Xbox sensor, you have to install the Kinect SDK. It is not sufficient to run ExpressionMouse with Kinect runtime and a Xbox Kinect sensor.

Comments (24)

  1. MP

    Hi,

    nice project! I allready download it, but it is not working. If I use the “Fachemouse Kinect.exe” nothing, and I mean nothing, happens. What’s wrong, may be a nother programm is needed?
    I use a Win7 64bit System and a windows-Kinect. Pleas tell me what to do, I would like to use my head ;-)

    Thanks

  2. Arjan

    Hi Tom,

    I’m looking forward to try your application, but unfortunately it doesn’t work.

    When clicking ‘Start’ I can see the Kinect’s IR-depthsensor start operating, but your application says: “Error during Kinect initialization. …”

    I’ve tried SDK 1.6 and 1.7. I have all the components installed and all other apps are working fine with the Kinect. Using Win7 x64.

    Any clue?

    Thanks

  3. Hi,

    Good job Tom, cool idea with the eyeblinks. We just finished betatesting of our facecontrolled mouse, called KinesicMouse.
    A free trial is available from our website at kinesicmouse.com

    Best,
    Markus

  4. Jackie

    Hi Tom,

    I just purchased the pro app, but I get an error when I click “start”. I have the xbox kinect and installed kinect SDK 1.8 and .NET 4.0. The kinect is working correctly with other programs.

    This is the error I get: http://imgur.com/imrCFVe

    Thanks in advance!

    • Hi Jackie,
      The message “Kinect is already in use by another process” indicates that you used another Kinect app at your computer before you have started ExpressionMouse. It seems that this app hasn’t released the Kinect resources properly.
      If you try to restart your computer and then start ExpressionMouse directly after startup: Keeps the error occuring?

      • Jackie

        Hi Tom,

        Thank you for the quick reply. It seems that a restart is helping, also uninstalling the SDK’s used for other programs. Anyway, we got it working. :)

  5. Blake

    Hi Tom,
    I recently bought the pro version of this program and own an Xbox Kinect. I have the sdk installed but it doesn’t seem to be working. I tried restarting m computer but it still wont work. Also, this error popped up in my folder: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index
    at System.ThrowHelper.ThrowArgumentOutOfRangeException()
    at System.Collections.Generic.List`1.get_Item(Int32 index)
    at FaceMouseConfig.FaceMouseConfig.OnAllFramesReady(Object sender, AllFramesReadyEventArgs allFramesReadyEventArgs)

    Any suggestions?

    Thank you!

    Blake

  6. Bian

    it can not work. Other Apps can work.
    ExpressionMouse Kinect-1.0.2
    win7 32bit,xbox 360,SDK 1.8.
    error:
    Error during Kinect Initialization. Ensure that Kinect sensor is connected correctly.
    Error Message:
    System.InvalidOperationExcepthion:The feature is not supported by this version of the hardware
    at Microsoft.Kinect.KinectExceptionHelper.CheckHr(Int32 hr)
    at
    Microsoft.Kinect.ImageStream.set_ImageStreamFlags(ImageStreamFlags value)
    at Microsoft.Kinect.DepthImageStream.set_Range(DepthRange value)
    at FaceMouseConfig.FaceMouseConfig.Init()

    • Hi,
      It seems that you are using Kinect for XBox sensor. Only the Pro version of ExpressionMouse Kinect works with the Xbox sensor. For the free version you need the special windows hardware.

  7. Benjamin Hamrick

    What are the differences, if any, of using your app with the Kinect for the 360?

    • Hi,
      The difference is that Kinect for Windows sensor is supporting a so called “Near mode”. This means it is possible to come a bit closer to the sensor, than with the Xbox sensor.
      The Xbox sensor was designed in a way to use it a few meters away from the sensor. Furthermore you need a special adapter to conect the Xbox sensor to your PC.

      • Benjamin Hamrick

        Thanks for the quick reply. I have currently ordered the 360 Kinect since it is much cheaper. I am planning helping my friend who can not move his arms use his laptop. What is the closest distance that it can be properly used?

  8. I am developing a same application like this for my final year project at university. can I know you developed eye blinking methods using openCV or something else. Can you give me some references that you followed when build this app. It will be a great help for me

  9. jacques lambert

    Hi
    i just but the pro vertion from mageca how long it take to get the version pro after it paid
    Thank you
    Jacques

    • Hi, you should be able to download it immediately from Mageca. Please contact Mageca support, if you have problems with the download.
      Best regards,
      Tom

Leave a Reply

Your email address will not be published. Required fields are marked *


× 7 = forty nine