Multipoint with the Kinect - Part I

Multipoint with the Kinect - Part I


Introduction

Kinect multipoint was a project intended to work with Kinect multipoint, mouse mischief, and other multi-mouse applications and some desktop touch software. It was intended to map Kinect user coordinates of multiple people to multiple mouse coordinates (x,y), move a mouse, or simulate multiple mouse clicking behavior. The project started out with vbscript, and then I progressed to working in vb.net and c++ to create a mouse emulator library for the project.  I am currently working on a full vb.net solution but currently only have Kinect code available to share with everyone in vb.net.  The mouse emulator code is not available yet and I will update the article whenever I get it working. Please bear with me below as the code may work better on your Kinect for windows or the newer Kinect for windows 2.0. This starts part I of II in my Kinect multipoint article session. Part II will be written once I finish source code for the mouse emulator.

  1. Requirements

Current requirements

1. DOTNET framework 4.5 or higher
2. Kinect SDK 1.7 or higher
3. Kinect Multipoint project (http://kinectmultipoint.codeplex.com)

Future requirements

1. DSF runtime 1.0 from wdk 7.1 (need the one from wdk 7.1!) <- for the mouse emulation code (will post that section later once finished with it).

The full requirements for development are:

  1. WDK 7.1 sample code with header files
  2. The softhidreceiver.dll file
  3. Full Kinect SDK with source code
  4. Optional – download KinectContrib for easier development of Kinect projects.
 II. Kinect Interaction with Users

First, I must describe how the Kinect interacts with users before proceeding into the code. This process involves an infrared beam that hits the user/s and returns raw data about each user back to the Kinect software. Furthermore, the Kinect software API then takes the data and creates a depth image and skeleton of it based on the position of the human in front of the Kinect. In addition, WPF (windows presentation foundation) is used to host the controls the Kinect needs to fire its events on the vb.net side. A picture box is used in WPF to capture the depth and skeleton parts used to represent a person. Skeleton parts are represented by dots that represent the center of mass on a person. For example, there are dots for the foot, head, hands, and many other skeleton parts. Think of rigging a 3D model with a skeleton as an example for the skeleton involved with the Kinect. Depth is the data representing what the Kinect detected whether humans or inanimate objects. It is used in most instances for identification of human users or players in a Kinect game. Sometimes it can be used for mapping inanimate objects and 3D reconstruction.  Note: A human is usually color coded in a depth image based on his distance from the Kinect (a future sdk may change this but as far as I know this is true). A good tip to remember: if the player is black in the depth stream they are too far away for the Kinect to recognize. If the player is white they are too near the sensor.  Please refer to figure 2 for a little picture of the color range  to figure out about how far a user is from the Kinect. In the picture, the Default range is for the Xbox Kinect and near range is only for Kinect for windows (KFW).  Let's dive into some code now. The code in figure 1 shows how my Kinect multipoint application is initialized with certain settings (color camera resolution of 640*480, depth resolution of 320*240, depth enabled, skeleton stream,  and color stream enabled, and skeleton is set for smoothing to allow better precision on mouse coordinates). As you can see it looks like it is adding smoothing or precision to the initialization routine and different resolutions for Color and depth stream.

 Figure1 – Kinect initialization routine

 

Private Sub kinectSensorChooser_KinectSensorChanged(ByVal sender As Object, ByVal e As DependencyPropertyChangedEventArgs)
        Dim old As KinectSensor = CType(e.OldValue, KinectSensor)
        'If old Is Nothing Then
        '    Return
        'End If
        StopKinect(old)
        sensor = CType(e.NewValue, KinectSensor)
        If sensor Is Nothing Then
            Return
        End If
 
        Dim parameters As New TransformSmoothParameters()
        parameters.Smoothing = 0.7F
        parameters.Correction = 0.3F
        parameters.Prediction = 0.4F
        parameters.JitterRadius = 1.0F
        parameters.MaxDeviationRadius = 0.5F
 
        sensor.SkeletonStream.Enable(parameters)
        sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30)
        sensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30)
 
        AddHandler sensor.AllFramesReady, AddressOf sensor_AllFramesReady
        Try
            sensor.Start()
        Catch e1 As System.IO.IOException
            'another app is using Kinect
 
            kinectSensorChooser.AppConflictOccurred()
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
    End Sub

   
Figure 2 - Kinect sensor depth stream distance colors (How far is a person away from the Kinect?).

  

   

III. Kinect Player identification and coordinate location

I have to identify players by looping through them as the Kinect sdk does not recognize more than 2 players at a time.  First, I recognize players by retrieving all people recognized from the skeleton stream and copy to a skeleton array. Refer to figure 3. Second, I assign a random player ID used to identify the player by the Kinect system. Kinect requires a random ID used for custom identification can you believe that? For instance, player 1 gets a number between 5 and 9, player 2 a number between 10 and 14, and so on. As you can see every player is assigned a random id number within a 4 number range depending on what player they are.  I show some sample ids up to player 6 in a table in Figure 6  to show how the id process works for the players.  This identification tells us who a specific player is so we can send input to that player. Figure 4 contains the player identification code.    I combine the players identification Third, I retrieve the x and y coordinates of a player using the CursorX and CursorY variables and find out how many players the Kinect recognizes using activeCount. The activeCount variable stores the number of people during each loop that the Kinect recognizes.   I use it to figure out how many people need to have coordinates sent to the emulated mouse devices created.  Notice the scaledright and scaledleft variables? They are there to give a hand that moves better precision when moving their specific emulated mouse. In the original application that gave me inspiration for this project there was a button to switch between left and right handed people that’s why they the variables are there. Figure 5 was the place I retrieved the x,y coordinates of the hand and whether the gesture to left click was made. Please refer to notes for information about the limitations of the code, more information about the project,  and hints about what will be in part II.

 Figure 3 – retrieve players

  
Private Sub sensor_SkeletonFrameReady(ByVal e As AllFramesReadyEventArgs)
            ' Dim j As Integer
            ' Dim s(5) As Skeleton
            Dim playerid(6) As Integer
            If IsNothing(sensor) Then
                Application.Exit()
            End If
            '  For Each k In Microsoft.Kinect.KinectSensor.KinectSensors
            ' k.SkeletonStream.AppChoosesSkeletons = True
            ' sensor.SkeletonStream.AppChoosesSkeletons = True
            Using skeletonFrameData As SkeletonFrame = e.OpenSkeletonFrame
 
                If skeletonFrameData Is Nothing Then
                    Exit Sub
                End If
                '  sensor.SkeletonStream.AppChoosesSkeletons = True
                Dim allSkeletons(skeletonFrameData.SkeletonArrayLength - 1) As Skeleton
                skeletonFrameData.CopySkeletonDataTo(allSkeletons)

Figure 4 – Identifying players by their specific random ID range (still sensor_SkeletonFrameReady)

 

'identify specific players by ID
'ResetValues()
 
'    Using frame = k.SkeletonStream.OpenNextFrame(200)
'If IsNothing(Frame) = False Then
'    Dim s(Frame.SkeletonArrayLength - 1) As Skeleton
'    Frame.CopySkeletonDataTo(s)
'player tracking id initialization
' For j = 0 To 5
'allSkeletons.TrackingId = 0
'Next
 
'player #1
playerid(0) = CInt(Rnd() * 4)
' playerid(0) = 1
allSkeletons(0).TrackingId = playerid(0)
Log("player#1 tracking: " + allSkeletons(0).TrackingId.ToString)
'player #2
playerid(1) = CInt(Rnd() * 4 + 5)  ' 5  -  9
'playerid(1) = 2
allSkeletons(1).TrackingId = playerid(1)
'player #3
' playerid(2) = 3
playerid(2) = CInt(Rnd() * 4 + 10) ' 10 - 14
allSkeletons(2).TrackingId = playerid(2)
'player #4
playerid(3) = CInt(Rnd() * 4 + 15) ' 15 - 19
allSkeletons(3).TrackingId = playerid(3)
'player #5
playerid(4) = CInt(Rnd() * 4 + 20) ' 20 - 24
allSkeletons(4).TrackingId = playerid(4)
'player #6
playerid(5) = CInt(Rnd() * 4 + 25) ' 25 - 29
allSkeletons(5).TrackingId = playerid(5)
'force each player to be moved to first blank spot
Dim tempList As New List(Of Skeleton)(allSkeletons)
tempList.RemoveAll(Function(sk) IsNothing(sk))
allSkeletons = tempList.ToArray
Log("Tracking id for player#1: " + allSkeletons(0).TrackingId.ToString)
Log("Tracking id for player#2: " + allSkeletons(1).TrackingId.ToString)
Log("Tracking id for player#3: " + allSkeletons(2).TrackingId.ToString)
'MsgBox(allSkeletons.Length)
'If allSkeletons.Length < 6 Then
'    For j = 0 To allSkeletons.Length - 1
'        ReDim allSkeletons(5)
'    Next j
'End If
'  For j = 0 To 5
 

 Figure 5 – retrieve x,y coordinates,do a left click, and drop players not being used. (still sensor_SkeletonFrameReady)

 

For Each s As Skeleton In allSkeletons
                   ' Dim d As Integer
                   Log("person#: " + i.ToString + " trackingstate: " + allSkeletons(i).TrackingState.ToString)
                   '  Dim s(5) As Skeleton
                   's(j) = allSkeletons(j)
                   If IsNothing(s) Then
                       Exit Sub
                   End If
                   If s.TrackingState = SkeletonTrackingState.Tracked Then
                       activeCount = activeCount + 1
                   End If
                   If s.TrackingState = SkeletonTrackingState.PositionOnly Then
                       passiveCount = passiveCount + 1
                   End If
                   If s.TrackingState = SkeletonTrackingState.NotTracked Then
                       nottracked = nottracked + 1
                   End If
                   totalplayers = activeCount + passiveCount + nottracked
                   'Log("passive count: " + passiveCount.ToString + " date: " + Now.ToString)
                   If s.TrackingId > 0 Then
 
                       'sensor.SkeletonStream.ChooseSkeletons(s.TrackingId)
                   End If
                   'Select Case s.TrackingId
                   '    Case Is = 1
                   '        sensor.SkeletonStream.ChooseSkeletons(1)
                   '    Case Is = 2
                   '        sensor.SkeletonStream.ChooseSkeletons(2)
                   '    Case Is = 3
                   '        sensor.SkeletonStream.ChooseSkeletons(3)
                   '    Case Is = 4
                   '        sensor.SkeletonStream.ChooseSkeletons(4)
                   '    Case Is = 5
                   '        sensor.SkeletonStream.ChooseSkeletons(5)
                   '    Case Is = 6
                   '        sensor.SkeletonStream.ChooseSkeletons(6)
                   'End Select
 
 
                   ' the first found/tracked skeleton moves the mouse cursor
                   If s.TrackingState = SkeletonTrackingState.Tracked Then
 
                       ' make sure both hands are tracked
                       'If Skeleton.Joints(JointType.HandLeft).TrackingState = JointTrackingState.Tracked AndAlso Skeleton.Joints(JointType.HandRight).TrackingState = JointTrackingState.Tracked Then
                       Dim cursorX, cursorY As Integer
 
                       ' get the left and right hand Joints
                       Dim jointRight As Joint = s.Joints(JointType.HandRight)
                       Dim jointLeft As Joint = s.Joints(JointType.HandLeft)
 
                       ' scale those Joints to the primary screen width and height
                       Dim scaledRight As Joint = jointRight.ScaleTo(CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)), SkeletonMaxX, SkeletonMaxY)
                       Dim scaledLeft As Joint = jointLeft.ScaleTo(CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)), SkeletonMaxX, SkeletonMaxY)
                       ' relativemouselocation.Content = jointRight.Position
                       ' figure out the cursor position based on left/right handedness
                       If LeftHand.IsChecked.GetValueOrDefault() Then
                           cursorX = CInt(Fix(scaledLeft.Position.X))
                           cursorY = CInt(Fix(scaledLeft.Position.Y))
                       Else
                           cursorX = CInt(Fix(scaledRight.Position.X))
                           cursorY = CInt(Fix(scaledRight.Position.Y))
                       End If
 
                       Dim leftClick As Boolean
                       ' figure out whether the mouse button is down based on where the opposite hand is
                       If (LeftHand.IsChecked.GetValueOrDefault() AndAlso jointRight.Position.Y > ClickThreshold) OrElse ((Not LeftHand.IsChecked.GetValueOrDefault()) AndAlso jointLeft.Position.Y > ClickThreshold) Then
                           leftClick = True
                           '  MsgBox("clicked")
                       Else
                           leftClick = False
                       End If
                       For g = 0 To 5
                           allSkeletons(g).TrackingId = 0
 
                           For f = 0 To activeCount - 1
                              
                               Select Case f
                                   Case Is = 0
                                       playerid(0) = CInt(Rnd() * 4)
                                       s.TrackingId = playerid(0)
                                   Case Is = 1
                                       playerid(1) = CInt(Rnd() * 4 + 5)  ' 5  -  9
                                       'playerid(1) = 2
                                       s.TrackingId = playerid(1)
                                   Case Is = 2
                                       ' playerid(2) = 3
                                       playerid(2) = CInt(Rnd() * 4 + 10) ' 10 - 14
                                       s.TrackingId = playerid(2)
                                   Case Is = 3
                                       playerid(3) = CInt(Rnd() * 4 + 15) ' 15 - 19
                                       s.TrackingId = playerid(3)
                                   Case Is = 4
                                       playerid(4) = CInt(Rnd() * 4 + 20) ' 20 - 24
                                       s.TrackingId = playerid(4)
                                   Case Is = 5
                                       playerid(5) = CInt(Rnd() * 4 + 25) ' 25 - 29
                                       allSkeletons(5).TrackingId = playerid(5)
                               End Select
                             
                           Next
                       Next
                       'if i is less then the total amount of players then send players coordinates for person that is active.
                       If i <= 5 Then
                           Select Case True
                               Case Is = s.TrackingId >= 1 And s.TrackingId <= 4
                                   i = 0
                                   playeractive(i) = True
                                   player1xy.Content = cursorX & ", " & cursorY & ", " & leftClick
                                   Status.Text = "player1 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
                                   ' sensor.SkeletonStream.ChooseSkeletons(1, 2)
                               Case Is = s.TrackingId >= 5 And s.TrackingId <= 9
                                   i = 1
                                   playeractive(i) = True
                                   player2xy.Content = cursorX & ", " & cursorY & ", " & leftClick
                                   Status.Text = "player2 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
 
                               Case Is = s.TrackingId >= 10 And s.TrackingId <= 14
                                   i = 2
                                   playeractive(i) = True
                                   player3xy.Content = cursorX & ", " & cursorY & ", " & leftClick
                                   Status.Text = "player3 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
 
                               Case Is = s.TrackingId >= 15 And s.TrackingId <= 19
                                   i = 3
                                   playeractive(i) = True
                                   player4xy.Content = cursorX & ", " & cursorY & ", " & leftClick
                                   Status.Text = "player4 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
                               Case Is = s.TrackingId >= 20 And s.TrackingId <= 24
                                   i = 4
                                   playeractive(i) = True
                                   player5xy.Content = cursorX & ", " & cursorY & ", " & leftClick
                                   Status.Text = "player5 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
                               Case Is = s.TrackingId >= 25 And s.TrackingId <= 29
                                   i = 5
                                   playeractive(i) = True
                                   player6xy.Content = cursorX & ", " & cursorY & ", " & leftClick
                                   Status.Text = "player6 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
                           End Select
                           Log("person #: " + i.ToString + "Tracking ID: " + s.TrackingId.ToString)
                           currentperson.Content = i.ToString + "Tracking ID: " + s.TrackingId.ToString
                       End If
                       'If playeractive(i) = True And i >= 0 And frame.SkeletonArrayLength > 0 Then
                       'NativeMethods.SendMouseInput(cursorX, cursorY, CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)), leftClick, totalplayers, i)
 
                       'End If
 
                       'if total players is 1 or greater send coordinate data.
                       If totalplayers >= 1 Then
                           DefineMouseData(cursorX, cursorY, leftClick)
                       End If
                       'make the below code active when I get multiple player tracking working
                       If playeractive(i) = True Then
                           ' Environment.SetEnvironmentVariable("Player" + i.ToString + "xcoords", player(i).bytex.ToString, EnvironmentVariableTarget.Machine)
                           ' Environment.SetEnvironmentVariable("Player" + i.ToString + "ycoords", player(i).bytey.ToString, EnvironmentVariableTarget.Machine)
                           'Environment.SetEnvironmentVariable("Player" + i.ToString + "leftclick", player(i).leftclick.ToString, EnvironmentVariableTarget.Machine)
                           'System.Threading.Thread.Sleep(300)
                       End If
                       If i <= 5 Then
                           If playeractive(5) = True Then
                               'if 6th player exit for loop and open next frame.
                               playeractive(5) = False
                               Exit For
                           Else
                               'd = d + 1
                           End If
 
                       End If
                   End If
                   NumPassive.Content = "passive count: " + passiveCount.ToString
                   Numactive.Content = "active count: " + activeCount.ToString
                   nottrackedplayers.Content = "not tracked players: " + nottracked.ToString
                   playeractive(i) = False
                   ' Next
                   For h = 0 To activeCount - 1
                       If h >= 2 Then
                           If activeCount - 1 > 0 Then
                               sensor.SkeletonStream.ChooseSkeletons(playerid(h), playerid(h + 1))
                           End If
                       End If
                   Next
               Next
 
               ResetValues()
 
           End Using
 
           'Next k
 
 
 
 
       End Sub

 

   Figure  6 - Table of Sample player identification numbers.

 Player #  Identification #
 1  5-9
 2  10-14
 3  15-19
 4  20-24
 5  25-29
 6  30-34

  IV. Notes for this project

                                                    Below is a list of extra information I may have forgot to add in this article and some hints for the code later for part II which is still in progress:

1. Both the Xbox Kinect and Kinect for windows recognize 6 players total per Kinect (excluding skeleton tracking  for Xbox 360 Kinect hence the weird looping of players above) and the number of Kinects depends on CPU processing power. The new Kinect for Windows rec and Xbox One Kinect recognize and identify 6 players.

2. The mouse emulator code helper dll is almost finished. If someone wants to take a shot at it and get the whole Kinect multipoint project working efficiently they can. I only need to figure out the reason why the queue input function to softhidreceiver.dll does not take my data from vb.net or c++ side. It wants a com safe array of variants the data passed is correct but the format of the com data is not.

3. As it stands now you can change the mouse to any device you need to emulate with the Kinect but require you to have hardware and low level knowledge of the touch or mouse related device. Use Microsoft’s device simulation framework to do so.  I’ll post an article on that side later when I finish it and link the two articles together in this one.

4. There is no right click functionality yet. That will happen after I finish the vb.net emulator dll helper code.
5. Kinect is not recommended on tablets as tablets might not have the proper processing power to handle many Kinect's (Mini-desktop towers are okay).
6. Ignore commented out code as chances are I will not use it in the future.

 

       VI. Links for development with Kinect and Source Code

  1. KinectContrib – http://kinectcontrib.codeplex.com (Optional) – These are some nice open source templates for coding with the Kinect sdk until official ones are replace by Microsoft.

  2.  Article Source code – http://kinectmultipoint.codeplex.com

  3. Kinect Mouse project – http://kinectmouse.codeplex.com

  4. WDK 7.1 (only for part II) – http://www.microsoft.com/en-us/download/details.aspx?id=11800.

  5. Device simulation framework (latest version is obtained from WDK 7.1).

  6. WDK 8 (Optional and for part II) – this is optional improvement to help with running through wdk and dsf source code from wdk 7.1 – http://go.microsoft.com/fwlink/p/?LinkID=324284.

  7. Kinect SDK 8.0 – http://go.microsoft.com/fwlink/?LinkID=323588

  8. Kinect Toolkit 8.0 - http://go.microsoft.com/fwlink/?LinkID=3235899
  9. Human Interface Guidelines - these are the guidelines Microsoft made for development with the Kinect sdk.
Follow them and you will have nice applications that run well with the sdk-
http://go.microsoft.com/fwlink/?LinkID=247735

Sort by: Published Date | Most Recent | Most Useful
Comments
  • Please fix all code samples - don't add line numbers. Also please fix formatting of the article - big indent is hard to read

  • I can fix the formatting of the article but a portion of code samples require line numbers to understand where I am going with it.

  • I will try fixing the broken code sections but I have a lot of code that the editor breaks once I paste it. Some code might have bad formatting issues.

  • Thanks, Naomi  I will try my best on the code! If it does not break it I will try editing it.

  • I will perform so edits on the x,y coordinate retrieval section as I get close to part II. In addition, I will post my form load section which has code for initializing the device simulator.

  • Congratulations on winning a TechNet Guru medal! blogs.technet.com/.../technet-guru-awards-october-2013.aspx

Page 1 of 1 (6 items)