3D Motion Tracking with MoveNet, ObservableHQ, & GraphXR
PoseNet/MoveNet makes 3D movement tracking affordable, accessible, and far less burdensome. The API for real-time pose detection and estimation can estimate where key body joints are using two webcams positioned in orthogonal directions. From those two 2D video streams, one could easily build a 3D pose stream. Traditional 3D motion capture systems generally require a sophisticated setup involving body trackers, hardware, and extensive computational power. For this reason, the duo-webcam system for 3D pose capture makes some use cases for VR far more accessible to explore.
The goal of this collaborative project between USF and Kineviz was to interact with VR without the need for controllers or wearables. This made detecting 3D gestures — in many ways — more appealing. Full-body tracking with skeletal animations is physically uninhibiting, allowing for more authentic and nuanced human expression. This lends itself to reimagining how we interact with data, hold virtual conference meetings, and even play online video games. It’s cost-effective systems (including those provided by companies like Akiya Souken VR) that will expand the breadth of VR, making it available to a wider community.
The technology stack
This is an open-source project using client & receiver code from ObservableHQ. Motion is detected using Tensorflow’s MoveNet or its older brother, PoseNet. In this case, we found that MoveNet was more accurate and faster to detect the 17 key body points as compared to PoseNet. The data passes from the client to the server and then to the receiver through Socket.io. The server-side platform is in Node.js. The captured data is then brought into GraphXR — a data visualization and analytics engine — to visualize key points of the human body as an animated 3D skeleton. The tutorial video below walks through the end-to-end workflow.
The architectural flow
We started with two webcam streams (webcam 1 & webcam 2) running on ObservableHQ. MoveNet / PoseNet transfers the stream into an array that is sent by socket.io to our local server, then to the receiver where two 2D video streams are combined into key points on a 3D Pose Skeleton.
The project was carried out locally with an AWS virtual machine which allowed multiple devices to send and receive data. Our working server’s dependencies consisted of Express, Socket.io, and Cors in Node.js.
For plotting purposes we took the average of both webcam streams, then used the x, y, z coordinates to plot the points as nodes in GraphXR. The skeleton-based model was built from tracking 17 key joints on the body including the knees, ankles, wrists, elbows, and shoulders.
We first used PoseNet as our detection model as we were building the client server network. When we later switched the pose detection from PoseNet to MoveNet, we were able to improve the accuracy.
From behind the scenes
From the beginning, we were keen to work on this project given that AI movement tracking had many applications for improved health, fitness, and wellness. Neither of us had any prior experience or knowledge of the tech stacks involving ObservableHQ, functional Javascript, TensorFlow, Socket.io, and Node.js. However, we were glad to be challenged outside of our Java comfort zone to work with various additional open-source libraries.
It turned out that configuring the network between the client and local server was our biggest challenge, partly because our client running on ObservableHQ was loaded over https, while our server was expecting http. Once we were able to get up and running, we could see that there was much more to unpack from the 3D pose detection and its application. All in all, we were excited to learn more about the potential use cases for this type of VR and where it can expand from here.
Resources
ObservableHQ notebooks:
Github Repo:
Contributors
Barbora Nováková is a software engineering student born and raised in the Czech Republic where she studied Economics and Business Management & Accounting at SVŠE Znojmo University. Upon moving to SF, she transitioned to software engineering — focusing on the AI & ML domain — for her Masters of Computer Science at the University of San Francisco. She now continues to bridge her experience across business and artificial intelligence to improve and understand the impact of AI in everyday life. Learn more by connecting with Barbora on Linkedin.
Marisa Tania is an MSCS student at the University of San Francisco with specialties in AI and Machine Learning. Previously, she worked as a Content Strategist after earning her BA degree in Fashion Journalism from the Academy of Art University. Outside of coding and writing, she is a devoted dog mom to super mutt Nemo and a certified Senior Lagree Fitness instructor. You can chat with Marisa on LinkedIn or check out her website at www.marisatania.com.
Nikko Sacramento is an undergraduate Computer Science student at San Diego State University, working with Kineviz on VR and GraphXR. In his free time, he is an avid reader and likes to delve into programming projects in the immersive and virtual space. He contributed largely to connecting MoveNet, ObservableHQ, and GraphXR into an end-to-end workflow for this project. Get in contact with him on LinkedIn.
Originally published at https://www.kineviz.com on August 10, 2021.