Simulating Touch Input in Windows 8 Using Touch Injection API

Simulating Touch Input in Windows 8 Using Touch Injection API

 

Introduction

Windows 8 comes with multiple input types. Windows 8 devices often have multi-touch screens that enable users to use multiple fingers simultaneously to produce different input interactions such as tapping, dragging, or pinching. The Windows Runtime has a number of different mechanisms for handling touch input, enabling us to create an immersive experience that we can explore with confidence. This Article covers the basics of using touch input (simulating the touch input via VC++ code) in Windows Developer Preview using C++ by Touch Injection API. For other (consumer preview and release preview) you may need to enable UI access permission( check references for this).

Using this article will help all to create application that will help in simulating the Touch inputs like Tap, Drag, Rotate etc. , Consider an application that will convert users mouse event to touch event for those users who does not have Touch screen monitor. Also this will be of great use for Automating any touch based application on windows 8.

Prerequisites

This Article assumes that Reader is updated with these basics requirements

  • Understandings of Microsoft Visual C++
  • Using the code on Windows developer preview
  • Using Visual studio 11 developer preview to compile the code
  • Reader has knowledge of Touch Injection API structure and flags. If not please refer doc1, doc2 or read the references

 

Touch Interactions

Touch Interactions are a high-level way of interpreting touch input data into a set of common motions such as tapping, dragging, and pinching. Lets Understand some of the common interactions used in Windows Developer Preview which are as follow:

Interaction

  Input Type 

Description

Tap Single Input One finger touches the screen and lifts up.
Hold Single Input One finger touches the screen and stays in place.
Drag Single Input One or more fingers touch the screen and move in the same direction.
Pinch/Pan Multiple Input Two or more fingers touch the screen and move closer together (Zoom -out) or farther apart (Zoom-in).
Rotate Multiple Input Two or more fingers touch the screen and move in a clockwise or counter-clockwise arc.
Cross-slide Multiple Input  Two finger touches an object and drags it at a right angle to the panning direction.

For windows 8 developer preview this will work as it is but for other windows 8 versions you may need to change UI access permission. Lets see how to simulate these Touch interaction via VC++ code using Touch Injection API.

 

Simulating Tap

By Tap we mean that it has simply touched any portion of screen. In this use case actor takes two action
 

  1. Touch down on screen co-ordinate input
  2. Lifts Up Touch input

To get above done you need to following 5 steps.

1] Create a Win32 application using visual studio 2011 developer preview. ( with No CLR support and without ATL support)

2] In the WinMain Method Initialize the Touch Injection API refer code below.
POINTER_TOUCH_INFO contact;
InitializeTouchInjection(1, TOUCH_FEEDBACK_DEFAULT); // Here number of contact point is declared as 1.
memset(&contact, 0, sizeof(POINTER_TOUCH_INFO));

 

3] Define the contact Point, this definition contains contact type, contact location ,contact area, pressure and orientation .
contact.pointerInfo.pointerType = PT_TOUCH;
contact.pointerInfo.pointerId = 0;          //contact 0
contact.pointerInfo.ptPixelLocation.y = 200; // Y co-ordinate of touch on screen
contact.pointerInfo.ptPixelLocation.x = 300; // X co-ordinate of touch on screen
     
contact.touchFlags = TOUCH_FLAG_NONE;
contact.touchMask = TOUCH_MASK_CONTACTAREA | TOUCH_MASK_ORIENTATION | TOUCH_MASK_PRESSURE;
contact.orientation = 90; // Orientation of 90 means touching perpendicular to screen.
contact.pressure = 32000;

// defining contact area (I have taken area of 4 x 4 pixel)
contact.rcContact.top = contact.pointerInfo.ptPixelLocation.y - 2;
contact.rcContact.bottom = contact.pointerInfo.ptPixelLocation.y + 2;
contact.rcContact.left = contact.pointerInfo.ptPixelLocation.x  - 2;
contact.rcContact.right = contact.pointerInfo.ptPixelLocation.x  + 2;

 

4] Implementing use case 1 , Injecting Touching down on screen.
contact.pointerInfo.pointerFlags = POINTER_FLAG_DOWN | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
InjectTouchInput(1, &contact); // Injecting the touch down on screen

 

5] Implementing use case 2 , Injecting Touching Up from screen.
contact.pointerInfo.pointerFlags = POINTER_FLAG_UP;
InjectTouchInput(1, &contact); // Injecting the touch Up from screen

Simulating Hold

 By Hold it mean that Touch is continuously in contact to the screen. In this use case actor takes two action

  • Touches the screen
  • Keep holding the touch at same  place

    To follow the above scenario repeat all 4 steps mentioned in TAP Simulation and then add the below code as last steps

  • Implementing Use case 2 , holding the touch for few second and then lifting the touch up.
    //POINTER_FLAG_UPDATE when use with POINTER_FLAG_INRANGE and POINTER_FLAG_INCONTACT keeps the touch in drag mode with respect to last down Input.

    contact.pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    for(int i=0;i<100000;i++){        //loops for approx. 1 second causing 1 second HOLD effect
    InjectTouchInput(1, &contact);
    }
    contact.pointerInfo.pointerFlags = POINTER_FLAG_UP;       // Moving the touch Up after Hold Complete
    InjectTouchInput(1, &contact);

     

    Simulating Drag

     By Drag it mean that the user has touched down at any Screen co-ordinate and has moved his finger to another co-ordinate. In this use case the actor has three actions.

    1. Touch down at desired co-ordinate
    2. Dragging from last down co-ordinate to some other.
    3. Lifts Touch Up.

    To follow above scenario repeat all 4 steps of TAP Simulation and then add the below code as last step.

  • Implementing use case 2 and 3, wherein we are dragging from one co-ordinate to another and lifting up the touch input. 
    //Setting the Pointer Flag to Drag
    contact.pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    for(int i=0;i<100;i++){
    contact.pointerInfo.ptPixelLocation.x--; // updating the X Co-ordinate to x-100 pixels
    InjectTouchInput(1, &contact);
    }
    // Lifts the touch input UP
    contact.pointerInfo.pointerFlags = POINTER_FLAG_UP;
    InjectTouchInput(1, &contact);

     

    Simulating Pinch/Pan

    By Pinch it means that we inject two contact point and drag both towards each other( Zoom-out) or drag both the contact point apart (Zoom-in). In this use case  actor has to following three actions

    1. Injecting two Touch down contact point simultaneously on the screen.
    2. Drag both points inward/outward with respect to each other.
    3. Lifts up both the touch contact point.

    To get the above scenario done follow the following steps.

    1] Initializing the Touch injection API for implementing multiple(in our case two) contact points
    //Taking two contact points and initializing the Touch API accordingly
    POINTER_TOUCH_INFO contact[2];
    InitializeTouchInjection(2, TOUCH_FEEDBACK_DEFAULT);
    memset(&contact[0], 0, sizeof(POINTER_TOUCH_INFO));
    memset(&contact[1], 0, sizeof(POINTER_TOUCH_INFO));
     
    //Check Pointer Id is taken as 0 for contact 0
    contact[0].pointerInfo.pointerType = PT_TOUCH;
    contact[0].pointerInfo.pointerId = 0;          //Id 0 for contact 0
    contact[0].pointerInfo.ptPixelLocation.y = 200;
    contact[0].pointerInfo.ptPixelLocation.x = 300;
    //Defining Touch flag and touchmask for contact 0
    contact[0].touchFlags = TOUCH_FLAG_NONE;
    contact[0].touchMask = TOUCH_MASK_CONTACTAREA | TOUCH_MASK_ORIENTATION | TOUCH_MASK_PRESSURE;
    contact[0].orientation = 90;
    contact[0].pressure = 32000;
    contact[0].rcContact.top = contact[0].pointerInfo.ptPixelLocation.y - 2;
    contact[0].rcContact.bottom = contact[0].pointerInfo.ptPixelLocation.y + 2;
    contact[0].rcContact.left = contact[0].pointerInfo.ptPixelLocation.x  - 2;
    contact[0].rcContact.right = contact[0].pointerInfo.ptPixelLocation.x  + 2;
     
    //Check Pointer Id is taken as 1 for contact 1
    contact[1].pointerInfo.pointerType = PT_TOUCH;
    contact[1].pointerInfo.pointerId = 1;          //Id 0 for contact 1
    contact[1].pointerInfo.ptPixelLocation.y = 200;
    contact[1].pointerInfo.ptPixelLocation.x = 180;
    //Defining Touch flag and touchmask for contact 1
    contact[1].touchFlags = TOUCH_FLAG_NONE;
    contact[1].touchMask = TOUCH_MASK_CONTACTAREA | TOUCH_MASK_ORIENTATION | TOUCH_MASK_PRESSURE;
    contact[1].orientation = 90;
    contact[1].pressure = 32000;
    contact[1].rcContact.top = contact[1].pointerInfo.ptPixelLocation.y - 2;
    contact[1].rcContact.bottom = contact[1].pointerInfo.ptPixelLocation.y + 2;
    contact[1].rcContact.left = contact[1].pointerInfo.ptPixelLocation.x  - 2;
    contact[1].rcContact.right = contact[1].pointerInfo.ptPixelLocation.x  + 2;

     

    2] Injecting both contact point to Down, Implementing use case 1
    //Implementing two touch down for both array of contact point in one go
    contact[0].pointerInfo.pointerFlags = POINTER_FLAG_DOWN | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    contact[1].pointerInfo.pointerFlags = POINTER_FLAG_DOWN | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    InjectTouchInput(2, contact);

     

    3] Dragging both the contact point apart causing he zoom-in , implementing use case 2
    //Setting both the contact point for Drag
    contact[0].pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    contact[1].pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    for(int i=0; i<100;i++)
    {
    contact[0].pointerInfo.ptPixelLocation.x =300+i;//Zooming in by dragging apart both contact points w.r.t each other
    contact[1].pointerInfo.ptPixelLocation.x =180-i;
    InjectTouchInput(2, contact);
    }

     

    4] Lifts Up both the contact both contact point to Up, Implementing use case 3
    contact[0].pointerInfo.pointerFlags = POINTER_FLAG_UP;
    contact[1].pointerInfo.pointerFlags = POINTER_FLAG_UP;
    InjectTouchInput(2, contact);

     

    Simulating Rotate

    By rotate it mean that two contact points will be dragged in clock wise or anti-clock wise direction with respect to each other. In this use case again actor has three actions.
    1. Injecting two Touch down contact point simultaneously on the screen.
    2. Drag both points clock wise/ anti-clock wise with respect to each other.
    3. Lifts up both the touch contact points.

    To implement the above scenario repeat all the steps in shown in Pinch Simulation , but change the step 3 to the code shown below.

  • In this step we will rotate the both the touch points to anti-clock wise direction. implementing use case 2
    contact[0].pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    contact[1].pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    for(int i=0; i<100;i++)
    {
        // implementing the rotation
        if(i%3==0)
        {
        contact[0].pointerInfo.ptPixelLocation.x -= 3;
        contact[1].pointerInfo.ptPixelLocation.x += 3;
         
        contact[0].pointerInfo.ptPixelLocation.y = 200+i;
        contact[1].pointerInfo.ptPixelLocation.y = 200-i;
        }
        InjectTouchInput(2, contact);
    }

     

    Simulating Cross-Slide

    By Cross-Slide we mean two contact points moves in any one direction with respect to screen, simulating a situation where use puts both his figure to touch any picture and drag it to any direction using both of his fingure. In this use case actor has three action.
    1. Injecting two Touch down contact point simultaneously on the screen.
    2. Drag both points to any 1 direction.
    3. Lifts up both the touch contact points.

    To implement the above scenario repeat all the steps in shown in Pinch Simulation , but change the step 3 to the code shown below

  • Drag Both Touch points to any one direction, implementing use case 2.
    contact[0].pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    contact[1].pointerInfo.pointerFlags = POINTER_FLAG_UPDATE | POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT;
    for(int i=0; i<100;i++)
    {   
        contact[0].pointerInfo.ptPixelLocation.y = 200+i;
        contact[1].pointerInfo.ptPixelLocation.y = 200+i;
        InjectTouchInput(2, contact);
    }

     

    References

     

Sort by: Published Date | Most Recent | Most Useful
Comments
  • Thanks Sanjay for the reply.

    Important question is from do I get those x, y and timeSpan?? I want these values to be auto generated while I am doing touch, pinch or swipe.

  • getting x,y values must not be a big deal... try out this link msdn.microsoft.com/.../dd371581(v=vs.85).aspx ,

    and when ever you are getting the OnTouch event (from the code from last link shown) take the "timeSpan" which can be "lastTimeSpan" - "currentTimeSpan". ..

    try it out.. you can do it easily dude.. :D

  • Hi, good examples, and works very well in C++. Is it possible that we can do the same in C#? I tried using [DllImport("User32.dll")] but the touch did not work for me

  • Hi,

        yes we can easily convert it to managed code.... I think you are not providing proper type marshaling in managed structures like POINTER_TOUCH_INFO,etc ... please use Pinvoke.net library for that use any from the below link to get the managed code

    http://www.pinvoker.com/

    clrinterop.codeplex.com

    www.paulyao.com/.../pinvoke.aspx

  • Hi , Great work..thanks a lot :) ,

    I understand that we need windows 8 and visual studio 11 developer edition to compile the code . But will the compiled code be able to work on windows 7 ? And if it will not work , then does it have some other API to inject touch events .

  • For "Simulating Hold", when running this code, sometimes the "right click menu"should show up ,but most time it only hold the icon but not show "right click menu"? The expected phenomenon is show "right click menu" everytime.

    Also for the sample " Simulating Pinch/Pan", it never work well.

    Could any one tell me the reason for above questions ? Does some parameters setting error?

  • Hello,

    At first thanks for your tutorials - without them I wouldn't have made it this far!

    Now I have a problem with the UIAccess permission - I can inject touch input to the desktop, or metro UI fine (either from a C#, or a C++application). But neither the UAC, nor the logon screen respond to my artificial touch input. Even with UIAccess=true, running from a trusted location and the executable being signed with a trusted certificate, the touch input has no effect.

    (I've validated this with a simple drag-up simulation from a C++ application...)

    Do you know why the touch injection is ignored by the system in these cases?

    regards,

    Michael O.

  • Hi Michael,

                     try running your exe from administrator permission.. Or if you are running via Visual studio then open the project as administrator.

  • Hi  tzhang4x,

            Please add/increase some delay between the two consecutive touch injections.

  • Hello,

    After I run these code to simulate the Pinch/Pan, the cursor just disappear unless I move the mouse, can you help me ?

Page 2 of 3 (28 items) 123