April 6, 2011

webcam head tracking using FaceAPI

FaceAPI is a real-time head-tracking engine which uses webcam input to acquire 3D position and orientation coordinates per frame of video. And it works!

I implemented it in Blender Game Engine for cockpit view of BGE Air Race game.

and here is how I got it working in BGE

1. Downloaded FaceAPI here: LINK
2. Downloaded FaceApiStreamer here: LINK
(exports 6 degrees of freedom head tracking to a UDP socket connection)
3. Acquire the values from FaceApiStramer in BGE with Python code like this (not sure if it is quite right, but it kinda works):

from socket import *
controller = GameLogic.getCurrentController()
own = controller.owner

if own["once"] == True:
# Set the socket parameters
host = ""
port = 29129
addr = (host,port)
# Create socket and bind to address
GameLogic.UDPSock = socket(AF_INET,SOCK_DGRAM)
own["once"] = False


data,svrip = GameLogic.UDPSock.recvfrom(1024)
str = data.split(' ')
own['xPos'] = float(str[0])
own['zPos'] = float(str[1])
own['yPos'] = float(str[2])
own['pitch'] = float(str[3])
own['yaw'] = float(str[4])
own['roll'] = float(str[5])

Here is a blend file: LINK
you will need FaceAPI instaled and FaceApiStreamer running in background.


  1. Nicely done! You don't seem to need the faceAPI binary. The latest release of the Streamer comes with the faceAPI binary.

  2. Won't work for me.
    I downloaded & installed the non-commercial version of FaceAPI, then used FaceApiStreamer to track my head and then opened the attached .blend-file.
    What am I supposed to do then?
    Starting the game will just do nothing and running the script won't work (error) either. I'm using 2.57.
    The above demo was pretty damn cool so please help ;)

  3. It was made in Blender 2.49, I haven`t tried it in 2.57 yet. Also I suggest you take a look in this BlenderArtists thread

    D-Man has made an easier and artist friendly face tracking setup.

  4. is it possible to have this software start up when along with a blender runtime?

    1. hmm, you might need to make a batch file that opens blender binary and faceAPI at the same time or something..