Ken Pfeuffer, a young pioneering computer scientist, has embarked on a remarkable research journey that reached new heights during Apple's presentation of the Vision Pro headset*, as it turns out that the device utilizes a technology similar to the “gaze+pinch” interaction principle that he published back in 2017. Ken is now an Assistant Professor at the Department of Computer Science at Aarhus University.
‘I didn’t know that a user interface so close to my ideas had been integrated into Vision Pro before the product was announced by Apple. There had been some rumors on the internet, but I thought it was a mere phantasy. Before Apple, it was always hard to explain what I do. But suddenly it is very visual, accepted, and understandable’, Ken explains.
Ken’s research revolves around the convergence of human-computer interaction, augmented reality, and gesture recognition. He envisioned a future where intuitive gestural commands would seamlessly control digital devices, a concept almost reminiscent of science fiction films. Ken’s pioneering 2017 paper ‘Gaze + pinch interaction in virtual reality’ introduced the concept of gaze and pinch interaction, aiming to simplify user interactions by combining eye tracking and a simple hand gesture.
While researching for that paper, Ken and his team tested their concept using early VR headsets, combining eye and hand tracking hardware with custom-built software. As one of few researchers in this field, Ken persisted, driven by his conviction that his research held the key to a transformative technological shift, however, he didn’t believe it would happen so soon.
‘For ten years, I have been doing this research and I wouldn’t be doing this if I didn’t believe that at some point it would become reality. But I didn’t expect it to happen anytime soon, maybe in 20-30 years and then I would be happy if I had been part of laying the foundation via our prototypes and research. I really didn’t believe it would be a reality so soon.’
Apple's integration of a ‘pinch and gaze’ technology in the Vision Pro headset is a significant validation of Ken’s research. It gives strong support to his vision of effortless gestural control is realized by users simply looking at objects and pinching with their fingers to interact - a great leap in user experience. However, this implementation not only simplifies interactions but also holds promise for enhanced accessibility across many industries.
When Apple picks something up it usually gets exposed to a world-wide audience and thus Ken’s research ideas are brought to the forefront. Several international media - such as AppleInsider, Medium and iPhonesoft.fr to name a few - have already picked up and commented on the close connection between Ken’s research and Vision Pro. He envisions a future where the technology could extend beyond headsets to influence mainstream devices such as smartphones, benefiting users worldwide.
‘I am happy and excited that there’s a big tech company who bets on the research concepts that I introduced, and I know this is just one single feature of a huge development. If this takes off and Apple sees that it works well, they and other tech companies might be interested in integrating it into the next generation of phones, which would impact the lives of millions of people around the world.’
Over the past 10 years, Ken has been continuously working with many research papers on the "Eye Hand Symbiosis". This is an overall vision that investigates how any hand-based interface can be enhanced by using our eyes. What this could mean, is that headsets are just the beginning - many computer interfaces could be expanded in future, which is described in greater details in the ‘Eye-Hand Symbiosis’. As Ken continues to refine his concepts and explore new avenues, his research journey underscores the dynamic relationship between visionary research and its real-world impact.
*Please note that due to Apple’s usual secrecy around their products, it is unknown to what degree Apple is familiar with the research conducted by Ken Pfeuffer, but the technology used in Vision Pro is extremely similar to the principles explained in Pfeuffer’s 2017 research paper Gaze + pinch interaction in virtual reality
In 2017, Ken graduated as a PhD from Lancaster university with Professor Hans Gellersen as his supervisor. His PhD explored how our eyes and hands can work together on touchscreen devices. During his PhD he also interned at both Microsoft and Google Research in the US. Since then, he has had research positions at Bundeswehr University Munich and Lancaster University before he joined Department of Computer Science at Aarhus University as an assistant professor in august 2021.
If you want to know more about Ken and his research, you should follow him on X or LinkedIn, where he shares the latest advances within his research.
Vision Pro is Apple's wearable "spatial computing" device worn over the head. The headset is designed to seamlessly blend digital content with physical space.
Vision Pro was announced on June 5, 2023, at Apple's Worldwide Developers Conference, with availability scheduled for early 2024 in the United States and later that year internationally.
Read more about Vision Pro on Apple’s website or view the introduction video.
