I have a new paper out, just presented at the CHI Play’21 conference:
“The Game of Video Game Objects: A Minimal Theory of When We See Pixels as Objects Rather than Pictures.” In Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play. CHI PLAY ’21. https://www.jesperjuul.net/text/gameofobjects/
We’ve discussed immersion (for and against), but I argue that we’ve overlooked a much more fundamental question: Why and when do we think of pixels on the screen as objects, rather than as pictures of objects?
During the pandemic, I built a game for exploring this question and wrote an accompanying essay. This extends my previous paper Virtual Reality: Fictional all the Way Down (and that’s OK).
The game presents a series of game objects, and asks the player to consider their status:
- When would you describe something as an object, rather than a picture?
- When do you think of it as the type of object it represents – like a rock or a lamp?
- And when would you argue that an onscreen object is the type of object it claims to be, such as a calculator?
The conclusion is not just that games and VR are cultural forms (obviously), but that we judge game objects based on what we are trying to use them for, and game worlds are always designed for particular kinds of uses. And we know this. There thus can be no universal metaverse, only different worlds built for different purposes.
From the abstract:
“While looking to the future, we have overlooked what is right before us. With new technology, haptics, rendering, virtual reality, we have spent much energy discussing immersion and presence, thinking sometimes about current technology, but often about a hypothetical perfect experience or future perfect technology.
In this, we have forgotten something rather fundamental: How do we in the first place decide to see a group of pixels on a screen as an object to which we have access, rather than as a picture of an object? This paper explores this question through a playable essay. At first, we may think that we will identify anything interactive as an object, but the playable essay demonstrates that this is much more complex and pragmatic, and that this identification has three steps – identifying pixels as an object rather than a picture, reasoning about the object as a specific type of object (such as a ball), and identifying it as a real instance of a type of object (such as a calculator).”
Thanks to IVD at The Royal Danish Academy, Nick Montfort, Stefano Gualeni, Pawel Grabarczyk, Dooley Murphy, and Jan-Noël Thon for comments; to Andrés Cabrero Rodríguez-Estecha for visual design; Stephane Bersot for the calculator asset. The project was made with Unity3D and Low Poly Game Kit by JayAnAm.