Object Detection with HTML5 getUserMedia

Recently, I have seen a couple different JavaScript face detection demos based on the HTML5 getUserMedia API. I even made my own face detection demo based on the HTML5 video tag. Beyond human faces, we also have face detection for cats in JavaScript, nudity detection, and much more with js-objectdetect, a JavaScript library for real-time object detection.

JS-objectdetect let’s you detect eyes, hands, and faces in JavaScript and all of these JavaScript object detection scripts are based on OpenCV, the defacto computer vision library. But OpenCV only has C++, C, Python and Java interfaces so how are these demos working with just JavaScript? JS-objectdetect, similarly to HAAR.js, are using feature detection based on Haar Cascades in JavaScript (Viola-Jones Algorithm). I believe that this has the most reliable detection based on a partial OpenCV port.

CCV was the first port of OpenCV into JavaScript and it powers most of the demos I have found online, but its classifier seems to be less reliable than the one provided by OpenCV (HAAR classifier). Finally, headtrackr, a JavaScript library for real-time face and head tracking with a webcam (webRTC/getUserMedia), implements the camshift algorithm also found in OpenCV. If you want to learn more about how OpenCV and object detection works, you should check out these papers on Fast Face Recognition and Rapid Object Detection using a Boosted Cascade of Simple Features.

Beyond detecting certain objects from pixel data coming from images, videos, and webcams, I have also seen many demos that manipulate the getUserMedia feed in various ways. Here are some cool getUserMedia demos:

WebCamToy – a toy camera using getUserMedia

Photo Booth – one of many cool examples from shinydemos

ASCIIGRAM – create ASCII images and videos via getUserMedia.

PixelRoulette – grabs your picture via getUserMedia and randomly matches you with people on the internet

Easy Mustache – put a mustache on your face.

WebCamMesh – a HTML5 demo that projects webcam video onto a WebGL 3D Mesh.

SimpleCV JS – computer-vision library that brings functionality from the SimpleCV python library to the browser. It uses getUserMedia and then allows you to apply some filters on the images.

JS Motion Detection – playing xylophone via motion detection in getUserMedia.

Displace – a really cool getUserMedia() and WebGL demo with awesome effect

ProcessingJS and GetUserMedia – awesome demo of using the getUserMedia API with ProcessingJS

Photobooth – capture your webcam picture and apply CSS filters.

WebcamSwiper lets you turn pages of a book by swiping your hand in front of your webcam.

Head tracking with WebRTC – a game where movement is controlled by tracking your head with webRTC and getUserMedia.

Cross-browser camera capture with getUserMedia/WebRTC

The following demos from Opera are also interesting:

Warholiser
Instant Camera
Explode
RLColorPicker
Magic HTML5 Moustache

Final Thoughts

This stuff is really cool and I expect that we will see many more cool websites, libraries, and demos in the future around getUserMedia API and object detection. If you are looking for some good tutorials, I have found the following two articles very informative:

Web Gestures with getUserMedia

Face detection using HTML5, javascript, webrtc, websockets, Jetty and OpenCV

I should also mention that there is a face detection jQuery plugin for all the jQuery lovers (here is an example of using that plugin) and node-opencv contains OpenCV bindings for NodeJS if you want to play around with OpenCV in Node.

Realtime Responsive Typography is a project by Marko Dugonjić which uses the headtrackr library mentioned above to adjust the font on the browser screen to the distance that your face is from the monitor (or webcam).

Check out this article and demo on quickly building an HTML photo booth app with filters.

External:

Capture mobile camera picture data with pure JavaScript and HTML
Opera Demo of steering a game with your webcam
Masking Video with HTML5 Canvas