Oculus Rift, Kinect and Drone completed

Recently I completed my experiment with the Oculus Rift and a Kinect to drive a Parrot Ar.Drone.
Using Kinect I implemented both voice commands and gestures to drive the drone. With voice commands, I issue the takeoff, land, emergency and change camera commands, while with gestures I drive the drone.
The nice thing about the voice recognition of the Kinect is that it works quite well with all the noise made by the drone. As you will see in the video I raised a bit my voice, but I still find it awesome.
The gestures implemented are the following:

  • Both arms forward -> move the drone forward
  • Both hands near the shoulders -> move the drone backward
  • Left arm extended to the left and right arm along the body -> move the drone to the left
  • Left arm extended to the left and right arm forward -> move forward and to the left the drone
  • Left arm extended to the left and right hand near the right shoulder -> move backward and to the left the drone

The right movements are simply the left ones mirrored.
The Oculus, instead, was used to view the live feed of both drone’s cameras (with the stereoscopic filter applied, of course) and the head tracking to move the drone up/down and turn left/right.

Here is a video of it working. In the first part you can see the head movements, while in the second part the gestures.

It was fun to put together all these gadgets and try them out (I mean, having the drone crash everywhere :D)
Here you can find the full source code if you want to try it out.
As a disclaimer, I did this to learn something new and just for fun in my spare time, so the code is nothing to be proud of πŸ˜€

This is a simple summary, but, if you are interested, I will make some posts about the code of every part involved (gestures, voice commands, head tracking)

  • Jake Edwards

    Hello Sir,

    The Backward and Forward command was not functioning well. I was able to fix it and change some of the codes. Here is how I made it work. Thanks again Sir for everything you are
    awesome. I guess this is my last comment πŸ™‚ here. tomorrow is my Judgment (Panel) for
    my capstone. Good luck Sir with your future plan and keep the good works up.


    In the >>>>>>>>> OculusParrotKinect.cs file fix this.

    case KinectClient.GestureCommandType.Backward:

    if (lastCommandSent != CommandType.Backward)


    kinectMessage = “Forward”; <<<<<<<< <<<< #change the forward to Backward#

    lastCommandSent = CommandType.Backward;

    droneClient.Progress(Drone.Commands.FlightMode.Progressive, pitch: 0.1f);




    change these in the KinectClient.cs file

    if (!forwardLeft && !forwardRight && left && right) <<<<< <<<<<< # change to that (or simply copy and past it) so that when you rise your both hands to your right and left it goes forward#


    OnGestureCommandRecognized(new GestureCommandRecognizedEventArgs(GestureCommandType.Forward));



    if (backwardRight && backwardLeft && !left && !right) <<<<<<< <<<<<<<< #change to that (or copy past it) so that once you put Both of your hands near to your shoulders it goes backward#


    OnGestureCommandRecognized(new GestureCommandRecognizedEventArgs(GestureCommandType.Backward));



    • Thank you for your bug report. I found this issue too, but it seems I forgot to upload the fix πŸ™‚
      Good luck to you too for your life.

  • Phillip Brown

    Hello, im trying to do a similar thing with a DK2 and using phidget controls instead of a Kinect. However being a pretty shitty programmer and needing things really dumbed down i have no idea where to start. I have downloaded everything you did and the C# and FF thing but have no idea how or where to start. Even the simple connecting to the drone camera like in your last blog post is stumping me, i downloaded everything extracted the folder and went to the Win7 cmd and type in what you did and nothing (yes i was connect to the drone wifi) so i have no hope of trying to wack through your code and finding what i need. Any help would be great cause this inabilty to do anything pushes me away from programming and it sucks.

    • Hi, for the DK2 consider to rewrite the PixelShader, because is not working correctly with it.
      For the video part in my previous post, there is no programming skills involved, becuase once you connect to the drone wifi and excute the command from a command prompt everything should works out of the box. What error exaclty are you getting?

      • Phillip Brown

        command FFmeg (or whatever it was) is not a command. i downloaded the folder and extracted it, and its jsut sitting on the desktop.

        • Connect to the drone wifi and launch this command: ffplay tcp:// (the ip should be the one of your drone obviously)
          You should see the video feed of your drone

          • Phillip Brown

            OK i got that working and i started on the program, but im getting this error and both ema nd my programing teacher don’t know why: http://imgur.com/LzYhslv am i missing something here?

          • In your screenshot I do not see the detail of the exception, but my spidey sense is telling me that you did not copied the haar cacade xml file in the bin directory. Be sure to copy also the other Emgu libs. Let me know if it solved your problem

  • Dimas Angger

    hello I would like to introduce myself, I dimas from Indonesia . I want to ask you about these drones , do you use unity for programming? because I wanted to learn it for my thesis. I hope we can learn together about it. πŸ™‚

    • I use Unity3D at work and the use of kinect and oculus with it is a trivial thing, you just need to import the plugins and you are ready to go.
      Learn the basics of Unity3d and you will see how easy it is to implement the two gadgets. For the drone part I think you can implement the code quite easily, at the moment no specific problem comes to mind. I used XNA framework just because I wanted to try something different.

      • Dimas Angger

        I want to implement it in google cardboard, is this same with your project? I mean in implementation

        • The implemenattion is the same, there is a plugin for unity that works out of the box.

          • Dimas Angger

            does it mean i just set on the head recognition if I use google cardboard for it’s implementation (Controlling drone)?

          • You have to write/adapt the drone code, but you do not have to learn how the cardboard works,because the plugin do all the works for you

          • Dimas Angger

            Oh,I see. in my perspective,I think I must write / adapt drone code in unity3D first and after that I also set the plugin for cardboard in unity3D, doesn’t it?

            I’m sorry if I’m bothering you with our discussion, I am very interested in the topic of our discussion. I hope we can exchange ideas / discussion on this interesting topic. πŸ™‚

          • Yes exactly, at least what I would do.
            Feel free to write me an email if you prefer.

          • Dimas Angger

            Can I get your email address sir? for ask about this theme πŸ™‚
            And Do you make a journal / paper about this research? if there is a journal, may I ask for a reference in my research?

          • Hi, you can find my email in the about page. I just made this project for fun and to try something new, so no specific paper, sorry

          • Dimas Angger

            It’s okay sir, I think you also keep a journal about your project hehe. can discuss with you it was more than enough sir
            If I have trouble I will certainly contact you

          • Dimas Angger

            Hello Sir, it’s me again. πŸ™‚
            I have successfully made drone control applications using keyboard control. the application is able to control the movement of drones, reading navigation data, read the wifi signal strength and battery power. but, I still have a problem, I create applications that have not been able to display a stream of camera drones. can you help me about this problem?
            thank you

          • If you are usign C# looking at my code should help you solve your problem.
            Basically I use ffmpeg as an external library to decode the stream via wifi

          • Dimas Angger

            I use the C # programming language in Unity3D, and I also use the library ff mpeg, but for the plane that I use to display the images streaming from the camera drones texture can not be updated according to the data stream from the camera drone. The main problems that exist in my program is not able to display streaming cameras on drones. in the drone control I feel I have no problem

          • I do not know how to help you more from here. Video Texture in unity3d are tricky (at least for me)

  • Lorenzo Canci

    Ciao sono un ragazzo italiano che vorrebbe portare questo progetto all’esame, credi sia fattibile per un ragazzo che Γ¨ in quinta superiore indirizzo informatica? potrei farti delle domande in privato magari via mail?Grazie