Kinect multipoint was a project intended to work with Kinect multipoint, mouse mischief, and other multi-mouse applications and some desktop touch software. It was intended to map Kinect user coordinates
of multiple people to multiple mouse coordinates (x,y), move a mouse, or simulate multiple mouse clicking behavior. The project started out with vbscript, and then I progressed to working in vb.net and c++ to create a mouse emulator library for the project.
I am currently working on a full vb.net solution but currently only have Kinect code available to share with everyone in vb.net. The mouse emulator code is not available yet and I will update the article whenever I get it working. Please bear with me below
as the code may work better on your Kinect for windows or the newer Kinect for windows 2.0. This starts part I of II in my Kinect multipoint article session. Part II will be written once I finish source code for the mouse emulator.
1. DOTNET framework 4.5 or higher
2. Kinect SDK 1.7 or higher
3. Kinect Multipoint project (http://kinectmultipoint.codeplex.com)
1. DSF runtime 1.0 from wdk 7.1 (need the one from wdk 7.1!) <- for the mouse emulation code (will post that section later once finished with it).
The full requirements for development are:
First, I must describe how the Kinect interacts with users before proceeding into the code. This process involves an infrared beam that hits the user/s and returns raw data about each user back to
the Kinect software. Furthermore, the Kinect software API then takes the data and creates a depth image and skeleton of it based on the position of the human in front of the Kinect. In addition, WPF (windows presentation foundation) is used to host the controls
the Kinect needs to fire its events on the vb.net side. A picture box is used in WPF to capture the depth and skeleton parts used to represent a person. Skeleton parts are represented by dots that represent the center of mass on a person. For example, there
are dots for the foot, head, hands, and many other skeleton parts. Think of rigging a 3D model with a skeleton as an example for the skeleton involved with the Kinect. Depth is the data representing what the Kinect detected whether humans or inanimate objects.
It is used in most instances for identification of human users or players in a Kinect game. Sometimes it can be used for mapping inanimate objects and 3D reconstruction. Note: A human is usually color coded in a depth image based on his distance from the
Kinect (a future sdk may change this but as far as I know this is true). A good tip to remember: if the player is black in the depth stream they are too far away for the Kinect to recognize. If the player is white they are too near the sensor. Please refer
to figure 2 for a little picture of the color range to figure out about how far a user is from the Kinect. In the picture, the Default range is for the Xbox Kinect and near range is only for Kinect for windows (KFW). Let's dive into some code now. The code
in figure 1 shows how my Kinect multipoint application is initialized with certain settings (color camera resolution of 640*480, depth resolution of 320*240, depth enabled, skeleton stream, and color stream enabled, and skeleton is set for smoothing to allow
better precision on mouse coordinates). As you can see it looks like it is adding smoothing or precision to the initialization routine and different resolutions for Color and depth stream.
Figure1 – Kinect initialization routine
'If old Is Nothing Then
parameters.Smoothing = 0.7F
parameters.Correction = 0.3F
parameters.Prediction = 0.4F
parameters.JitterRadius = 1.0F
parameters.MaxDeviationRadius = 0.5F
'another app is using Kinect
I have to identify players by looping through them as the Kinect sdk does not recognize more than 2 players at a time. First, I recognize players by retrieving all people recognized from the skeleton
stream and copy to a skeleton array. Refer to figure 3. Second, I assign a random player ID used to identify the player by the Kinect system. Kinect requires a random ID used for custom identification can you believe that? For instance, player 1 gets a number
between 5 and 9, player 2 a number between 10 and 14, and so on. As you can see every player is assigned a random id number within a 4 number range depending on what player they are. I show some sample ids up to player 6 in a table in Figure 6 to show how
the id process works for the players. This identification tells us who a specific player is so we can send input to that player.
Figure 4 contains the player identification code. I combine the players identification Third, I retrieve the x and y coordinates of a player using the CursorX and CursorY variables and find out how many players the
Kinect recognizes using activeCount. The activeCount variable stores the number of people during each loop that the Kinect recognizes. I use it to figure out how many people need to have coordinates sent to the emulated mouse devices created. Notice the
scaledright and scaledleft variables? They are there to give a hand that moves better precision when moving their specific emulated mouse. In the original application that gave me inspiration for this project there was a button to switch between left and right
handed people that’s why they the variables are there. Figure 5 was the place I retrieved the x,y coordinates of the hand and whether the gesture to left click was made. Please refer to notes for information about the limitations of the code, more information
about the project, and hints about what will be in part II.
Figure 3 – retrieve players
' Dim j As Integer
' Dim s(5) As Skeleton
' For Each k In Microsoft.Kinect.KinectSensor.KinectSensors
' k.SkeletonStream.AppChoosesSkeletons = True
' sensor.SkeletonStream.AppChoosesSkeletons = True
SkeletonFrame = e.OpenSkeletonFrame
' sensor.SkeletonStream.AppChoosesSkeletons = True
allSkeletons(skeletonFrameData.SkeletonArrayLength - 1)
Figure 4 – Identifying players by their specific random ID range (still sensor_SkeletonFrameReady)
'identify specific players by ID
' Using frame = k.SkeletonStream.OpenNextFrame(200)
'If IsNothing(Frame) = False Then
' Dim s(Frame.SkeletonArrayLength - 1) As Skeleton
'player tracking id initialization
' For j = 0 To 5
'allSkeletons.TrackingId = 0
(Rnd() * 4)
' playerid(0) = 1
allSkeletons(0).TrackingId = playerid(0)
"player#1 tracking: "
(Rnd() * 4 + 5)
' 5 - 9
'playerid(1) = 2
allSkeletons(1).TrackingId = playerid(1)
' playerid(2) = 3
(Rnd() * 4 + 10)
' 10 - 14
allSkeletons(2).TrackingId = playerid(2)
(Rnd() * 4 + 15)
' 15 - 19
allSkeletons(3).TrackingId = playerid(3)
(Rnd() * 4 + 20)
' 20 - 24
allSkeletons(4).TrackingId = playerid(4)
(Rnd() * 4 + 25)
' 25 - 29
allSkeletons(5).TrackingId = playerid(5)
'force each player to be moved to first blank spot
allSkeletons = tempList.ToArray
"Tracking id for player#1: "
"Tracking id for player#2: "
"Tracking id for player#3: "
'If allSkeletons.Length < 6 Then
' For j = 0 To allSkeletons.Length - 1
' ReDim allSkeletons(5)
' Next j
' For j = 0 To 5
Figure 5 – retrieve x,y coordinates,do a left click, and drop players not being used. (still sensor_SkeletonFrameReady)
' Dim d As Integer
+ i.ToString +
" trackingstate: "
' Dim s(5) As Skeleton
's(j) = allSkeletons(j)
s.TrackingState = SkeletonTrackingState.Tracked
activeCount = activeCount + 1
s.TrackingState = SkeletonTrackingState.PositionOnly
passiveCount = passiveCount + 1
s.TrackingState = SkeletonTrackingState.NotTracked
nottracked = nottracked + 1
totalplayers = activeCount + passiveCount + nottracked
'Log("passive count: " + passiveCount.ToString + " date: " + Now.ToString)
s.TrackingId > 0
'Select Case s.TrackingId
' Case Is = 1
' Case Is = 2
' Case Is = 3
' Case Is = 4
' Case Is = 5
' Case Is = 6
' the first found/tracked skeleton moves the mouse cursor
' make sure both hands are tracked
'If Skeleton.Joints(JointType.HandLeft).TrackingState = JointTrackingState.Tracked AndAlso Skeleton.Joints(JointType.HandRight).TrackingState
= JointTrackingState.Tracked Then
' get the left and right hand Joints
Joint = s.Joints(JointType.HandRight)
Joint = s.Joints(JointType.HandLeft)
' scale those Joints to the primary screen width and height
Joint = jointRight.ScaleTo(
(Fix(SystemParameters.PrimaryScreenHeight)), SkeletonMaxX, SkeletonMaxY)
Joint = jointLeft.ScaleTo(
' relativemouselocation.Content = jointRight.Position
' figure out the cursor position based on left/right handedness
' figure out whether the mouse button is down based on where the opposite hand is
jointRight.Position.Y > ClickThreshold)
jointLeft.Position.Y > ClickThreshold)
g = 0
allSkeletons(g).TrackingId = 0
f = 0
activeCount - 1
s.TrackingId = playerid(0)
s.TrackingId = playerid(1)
s.TrackingId = playerid(2)
s.TrackingId = playerid(3)
s.TrackingId = playerid(4)
'if i is less then the total amount of players then send players coordinates for person that is active.
i <= 5
= s.TrackingId >= 1
s.TrackingId <= 4
i = 0
player1xy.Content = cursorX &
& cursorY &
& cursorX.ToString &
& cursorY.ToString &
' sensor.SkeletonStream.ChooseSkeletons(1, 2)
= s.TrackingId >= 5
s.TrackingId <= 9
i = 1
player2xy.Content = cursorX &
= s.TrackingId >= 10
s.TrackingId <= 14
i = 2
player3xy.Content = cursorX &
= s.TrackingId >= 15
s.TrackingId <= 19
i = 3
player4xy.Content = cursorX &
= s.TrackingId >= 20
s.TrackingId <= 24
i = 4
player5xy.Content = cursorX &
= s.TrackingId >= 25
s.TrackingId <= 29
i = 5
player6xy.Content = cursorX &
"person #: "
"Tracking ID: "
currentperson.Content = i.ToString +
'If playeractive(i) = True And i >= 0 And frame.SkeletonArrayLength > 0 Then
'NativeMethods.SendMouseInput(cursorX, cursorY, CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)),
leftClick, totalplayers, i)
'if total players is 1 or greater send coordinate data.
totalplayers >= 1
DefineMouseData(cursorX, cursorY, leftClick)
'make the below code active when I get multiple player tracking working
' Environment.SetEnvironmentVariable("Player" + i.ToString + "xcoords", player(i).bytex.ToString, EnvironmentVariableTarget.Machine)
' Environment.SetEnvironmentVariable("Player" + i.ToString + "ycoords", player(i).bytey.ToString, EnvironmentVariableTarget.Machine)
'Environment.SetEnvironmentVariable("Player" + i.ToString + "leftclick", player(i).leftclick.ToString, EnvironmentVariableTarget.Machine)
'if 6th player exit for loop and open next frame.
'd = d + 1
"passive count: "
"active count: "
"not tracked players: "
h = 0
h >= 2
activeCount - 1 > 0
sensor.SkeletonStream.ChooseSkeletons(playerid(h), playerid(h + 1))
Figure 6 - Table of Sample player identification numbers.
Below is a list of extra information I may have forgot to add in this article and some hints for the code later for part II which is still in progress:
1. Both the Xbox Kinect and Kinect for windows recognize 6 players total per Kinect (excluding skeleton tracking for Xbox 360 Kinect hence the weird looping of players above) and the number
of Kinects depends on CPU processing power. The new Kinect for Windows rec and Xbox One Kinect recognize and identify 6 players.
2. The mouse emulator code helper dll is almost finished. If someone wants to take a shot at it and get the whole Kinect multipoint project working efficiently they can. I only need to figure
out the reason why the queue input function to softhidreceiver.dll does not take my data from vb.net or c++ side. It wants a com safe array of variants the data passed is correct but the format of the com data is not.
3. As it stands now you can change the mouse to any device you need to emulate with the Kinect but require you to have hardware and low level knowledge of the touch or mouse related device.
Use Microsoft’s device simulation framework to do so. I’ll post an article on that side later when I finish it and link the two articles together in this one.
4. There is no right click functionality yet. That will happen after I finish the vb.net emulator dll helper code.
5. Kinect is not recommended on tablets as tablets might not have the proper processing power to handle many Kinect's (Mini-desktop towers are okay).
6. Ignore commented out code as chances are I will not use it in the future.
1. KinectContrib –
http://kinectcontrib.codeplex.com (Optional) – These are some nice open source templates for coding with the Kinect
sdk until official ones are replace by Microsoft.
2. Article Source code –
3. Kinect Mouse project –
4. WDK 7.1 (only for part II) – http://www.microsoft.com/en-us/download/details.aspx?id=11800.
5. Device simulation framework (latest version is obtained from WDK 7.1).
6. WDK 8 (Optional and for part II) – this is optional improvement to help with running through wdk and dsf source code from wdk 7.1 –
7. Kinect SDK 8.0 –
8. Kinect Toolkit 8.0 -
9. Human Interface Guidelines - these are the guidelines Microsoft made for development with the Kinect sdk.
Follow them and you will have nice applications that run well with the sdk-http://go.microsoft.com/fwlink/?LinkID=247735