Menu Close

Swing when you’re winning: do gesture-based controllers push the right buttons for gamers?

Sure, gesture-based gaming can be fun, but is it really “better” than using a button-based controller? Ludovic Maillard

Researchers at Baylor University in the US have conducted a study that reportedly shows players get more pleasure from videogames that use motion and gesture controllers (such as the Nintendo Wiimote, Microsoft Kinect and Playstation Move) than from videogames that use more traditional button- and trigger-based controllers.

The paper hasn’t yet been published, but the researchers have stated in a press release that their findings prove “moving toward a more natural user interface between the player and the game world can create a more immersive, realistic, and fun experience”.

I do not want to be hasty and attack the research before the full findings are released, but I can’t help but be highly sceptical of any research that seemingly sets up a dichotomy of gesture-based and button-based controllers to see which one is “better”. This without seemingly appreciating the vastly different strengths and weaknesses of each.

Swinging a tennis racquet is intuitive with a gesture-based controller. Bruce A Stockwell

A videogame’s interface is very much its language. The controller is a mediator between the game and the player, much like the written word is a mediator between the meaning an author puts on paper and the meaning the reader gets out of them.

The controller sits in the middle. To understand how to play a videogame, how to “read” meaning from it, players and developers alike must understand the language of the controller. This is as true for gesture-based controllers as it is for button-based ones.

Of course, gesture-based controllers have a language that will be vastly more accessible to the novice player who has not spent years learning how to use a button-based controller. In the same way, a book that only uses pictures will be more accessible to a person who has not spent years learning how to read words.

Take two people who have never played a videogame: give one a traditional controller covered in buttons and triggers, and give the other a gesture-control wand such as the Wiimote. Tell them both to “swing a tennis racquet” and, undoubtedly, the player with the Wii-mote will get it first.

My concern with this research is test subjects unfamiliar with button-based controllers got pleasure out of gesture-based controllers more readily and therefore the researchers inferred that gesture-based controllers afford more pleasure.

There are plenty of pleasures to be had in games via button-based controllers, but they are pleasures that take years of practice to access.

In all videogame play, there is a distance and a dissonance between the player’s actual world action (pressing a button, swinging an arm, touching a screen) and the player-character’s virtual world action (shooting a gun, swinging a tennis racquet, flinging a bird). It’s not just a configurative act but a figurative act where one action stands in for another.

This is as true for gesture-based controllers as it is for button-based controllers. The difference is how the different control types are figurative.

Gesture-based controls are what I would call “synecdochic” – the actual action of the player stands in for a similar, virtual action performed by the playable character. Swinging an arm typically represents swinging an arm; jumping typically represents jumping.

Learning how to use multi-button controllers can be difficult, but the rewards are great. C. Strife

Button-based controls, on the other hand, are “metonymic” — the actual action of the player acts as a substitution for a different action by the playable character.

Pressing a button can represent shooting a grappling hook or saying “I love you” or accelerating a car or destroying a planet or playing tag with a child.

Synecdochic controls are, of course, easier to immediately understand. You don’t have to learn anything you don’t already know. You already know how to swing your arm. Metonymic controls, on the other hand, will take much longer to comprehend, but still have their own strengths.

The written language you are reading this in is metonymic. The word “controller” does not directly represent what it is standing in for; it is just a series of signifiers strung together that you, having committed years to learning how to read, can understand as an actual, tangible object.

This kind of language takes much, much longer to learn but is able to do so much.

Ultimately, gesture-based and synecdochic controllers restrict the player to the motions their own body is capable of.

It’s worth noting that if you go through the questionnaire the study used, all the games the tests subjects played were sports videogames.

Their fictional content and goals closely represented actual world, gesture-based sports – swinging an arm in Tiger Woods Golf, throwing a punch in Kinect Sports Boxing, and so on.

In such games, of course motion controls are more immersive. But what of the rich, deep, fictional worlds of Grand Theft Auto IV, Bioshock, Minecraft, Portal, Skyrim, or Mass Effect?

The things players do in these worlds, the meanings and experiences players get out of these words, would be vastly hindered by the inaccuracy and lack of tactile feedback inherent in gesture-based controls.

A simple example: how do you represent the action of “walking forward” in a synecdochic, gesture-based control interface without walking into your television or the coffee table? You can’t. You would have to walk on the spot, at which stage you might as well just be pushing forward on a thumbstick.

According to the questionnaire, the study has not accounted for the pleasures of such videogames and, as such, seems more concerned not with proving that gesture-based controls offer more pleasure to the player, but simply an easier, more accessible pleasure.

Still, it is certainly true that videogames – both as an industry and a culture – are not as accessible or as inclusive as they could be. It’s also true that there still exists an exclusionary culture of “gamer or non-gamer” that has affected and continues to affect the public discourses surrounding videogames.

It can’t be denied that a large part of the segregation of videogames on to the cultural sideline comes down to the fact many people are not willing to commit the years required to learn the language of the traditional controller.

The controller is the language through which we understand videogames. josh-ferris

But then again, why should they when the content of so many of the videogames that use traditional controllers still seem to focus solely on a white, male teenage audience?

Gesture-based games can be incredibly fun and clearly revel in their own unique pleasures. Further, they can be a great introduction to videogames for someone intimidated by the myriad buttons and triggers of a traditional controller.

But to imply that one controller type is “better” or “more immersive and realistic” than another plays into the same obsessive futurism that has infected games studies specifically and new media studies broadly for years.

It continues the blind, idealistic, market-driven pilgrimage towards tomorrow’s mythical, all-encompassing “holodecks"— all-encompasing virtual worlds that we will supposedly one day be able to just step into in our own bodies — instead of focusing on what the videogames of today, in their myriad forms, actually do and how players actually engage with them.

With all its buttons and triggers, the traditional controller is a language barrier that prevents many people from fully experiencing or comprehending the pleasure of most videogames.

The solution here, I think, is not to abolish the language, but to abolish the barrier. Instead of looking at how we might do away with button-based controllers, let’s focus on how we might teach new players their unique language so they too can step into and appreciate the myriad worlds and experiences button-based controllers afford.

And, no less importantly, we must make sure there are worlds worth experiencing when they get there.

Want to write?

Write an article and join a growing community of more than 181,000 academics and researchers from 4,921 institutions.

Register now