Seeing as a machine: augmentic vision in gaming

Machines see the world differently from humans; how is this changing the way we play games? Meg Jayanth investigates what machine vision means to the modern gamer.

Kate Moss hologram

A virtual Kate Moss models clothes at the 2006 Alexander McQueen collection show

It’s not the Singularity, but you can see it from here with your machine-enhanced eyes.

In Part One, Playing Like A Machine I talked generally about the New Aesthetic as an art movement: interested in digital incursions into the physical world and machine-readability. And then I turned that on its side a little bit and asked – if we’re seeing like digital devices are we also playing like machines? Games consoles have been making our living rooms coded spaces for decades. The people making these devices have clear corporate visions of how humans and technology are going to interact – but they’re not the only ones.

It seems disingenuous to ignore actual videogames – many of which are set in near futures and deal with the situations of humans in an increasingly technological and digitised world. But are the images and ideas produced by games New Aesthetic? After all, it’s meant to be about what is happening all around us – of the moment, rather than science fiction futures.

To answer that question it’s a good idea to make the distinction between computer vision and computer graphics. Basically – computer vision is how machines see stuff and computer graphics are how we can use machines to make stuff look.

In use – both Wonderbook and Kinect are about computer graphics, but the nature of the technology is that we think about how it’s made. It’s still strange enough and new enough and imperfect enough that we can see the grain. There are glitches and distortions and failures of magic that expose the magician – no matter what the press department tells you. The E3 reveal of Book of Spells ran into a bug and froze during the demo, resulting in relieved applause when it started to work again. My Kinect still doesn’t recognise my face though it manages others – I get welcomed as a “guest” in my own living room, constantly. Which is kind of galling.

(Full disclosure here: my partner works on Wonderbook’s Book of Spells, so you know. Take that as you will.)

Dan Catt argues that NA isn’t about computer graphics – but I have to disagree. If it’s about new ways of seeing, then the special effects department is where it begins. Early film shocked and amazed audiences with new visual perspectives – seeing at the speed of a moving car if you’d never been in one must have been exhilarating and disorienting. Special effects, post-production, animation and motion-capture all contribute to expanding, documenting and – significantly – sharing our visual imaginations.

Blade Runner brought grimy, gritty punk back into fashion and design. Seeing Minority Report’s swipable surfaces and interactive screens in the cinema brings us alive to the possibilities of interactive whiteboards and Kinect-capable game-interfaces. Some of these possibilities may disappoint you.

Minority Report

Tom Cruise fights crime using hairless women and a floaty computer

My point is that the New Aesthetic is often about both computer graphics and computer vision – if the aim is to disseminate imagery and ideas into a culture, then computer graphics are the way to spread the meme. (Extending the metaphor, the cult film is a cultural dirty bomb.)

The trailer for Beyond and David Cage’s E3 intro of the game are all about computer-vision under the user friendly cloak of computer-graphics:

Pay attention around 2.22 in – where Cage reveals that Ellen Page is playing the lead character (and please note: playing, not voicing). The camera focuses on a lovely publicity shot and then pulls out – her photo is surrounded by other images of Page wearing reference dots for motion tracking. This tells us that Beyond is incredible because of the way it is made – realtime 3D motion-capture and the perfectly represented twitch of Ellen Page’s eyebrow, an incursion of humanity into the digitally rendered world of the game. A new kind of emotional intimacy for the player – possible because of a technological act of vampirism, stealing a little bit of Page’s soul and infusing it into Jodie Holmes.

Each of the images in the trailer is thick with the conditions of its production – we are meant to “see” the reference dots on Ellen Page’s meat face as we watch the expressions flicker across her unblemished digital one. (The question here: will we be able to enjoy the story without having to enjoy the story of the press-release? In all this looking and seeing, is there enough space for fun?)

Coming at this from a different angle, we already “see like machines” in videogames – we can activate detective mode in Arkham Asylum and see the environment through enhanced senses, all murky blue see-through and perversely ambulatory skeletons. David Hego, the art director, said he wanted to “cry a little bit” whenever he heard that players went through the game entirely in detective mode. Seeing like a machine is powerful, it’s addictive. Are the players who don’t see the atmospheric (read: filmic) art of the game missing its beauty? – or is there a new, distanced, grotesque beauty in their x-ray vision? Forget the London-by-night bus tour, it’s the London by infrared goggles you want – you can hang the heatmap of the Thames on your wall as a souvenir.

Talking about Arkham City, the sequel, Hego said detective vision would be more like “augmented reality” to avoid incentivising players to spend the entire game in this mode. (The art is so pretty! Look at it with your otherwise useless human eyes!) He’s not the only one making that distinction, or trying for a compromise between the two.

Batman Detective Mode

Bats swaps his boring old human eyes for shiny robot ones

Deus Ex is all about augmented vision (and machine-augmented selfhood) – human vision mixed seamlessly with the cybernetic, as we can see from this excellent cinematic trailer:

Deus Ex: Human Revolution is set in 2027 – augmentation of human abilities has started to become mainstream. Augmentations vary – from the physical abilities to jump from tall buildings or punch through walls to sensory enhancements which can allow you to hear nearby heartbeats or analyse pheromones to predict human behaviour. The trailer specifically marks the connection between our understanding, our reading of our own bodies to the desire to alter it – the transhumanist desire to be more and better. “What a piece of work is man […] in action how like an angel, in apprehension how like / a god!” Hamlet says that we’re like gods and angels; Sarif Industries wants to make us living gods.

Back to that sense of “new wonder” and “magic” that I talked about in Part 1- the slightly shopsoiled secret is that the machine is divine. It’s the cybernetic parts that make man the angel – Lazarus was a cyborg, and you can be one too. Maybe, just maybe, god sees like a digital device?

(And of course, unlike David Sarif – Hamlet was being ironic. All of that angelic action and godly apprehension doesn’t make us happy, or good. The same’s true of Jensen.)

Deus Ex Smashed Mirror

Adam Jensen: not a fan of mirrors

The first words of the trailer are Adam Jensen’s: “I never asked for this”. By and large, that’s true of all of us. Some of us may have helped build it but the connected world is changing rapidly all around us – if James Bridle’s New Aesthetic blog does nothing else, it assures us that like it or not machine-vision is a thing. It exists outside of our acknowledgement. Society’s changes get writ large upon Jensen’s body without his consent – he is augmented as a lifesaving procedure: he is objectified, subjected, scrutinised. And that is why he is our hero.

Probably one of the best moments of storytelling in any game is in Deus Ex: HR: when we return to Jensen’s flat after his first day back at work. Everything appears to be in place. Cosmopolitan, comfortable, plush. Then we enter the bathroom and the mirror is shattered – punched, at about the right height at which Jensen might see his unexpectedly augmented face. Looking like a machine has its problems.

Like all great scifi, it’s as much about the present and how we feel about the digital future and new technologies. If Deus Ex: HR is any indice: exhilarated and deeply, deeply frightened. And not (just) about the technology, but who is using it, and why. The angel escapes the autopsy table and heads for the sky – but heaven is too far. The angel is Icarus. The sun burns his wings. He falls to the earth and wakes up machine-enhanced. You know, the classic story.

Syndicate screenshot

Syndicate 2012: nice ideas, shoddy game

The opening to Syndicate (2012) informs the player that a new “biodigital implant” in human heads has rendered “all digital devices obsolete”. In the opening gameplay – check it out here your Dart biodigital chip literally boots you up courtesy of a soothingly bland female voice. You are punched and your vision doesn’t haze or blur, it pixellates like a badly set up TV screen. Why see like a digital device when you can be one? (Whether the game lives up to its story claims is a different kettle of motherboards.)

Similarly, Watch Dogs, which was revealed at E3 2012 – starts from the premise of smart cities and centralised, computerised control: “what would you do if you could hack the planet?” Ubisoft’s text under the gameplay footage video is even clearer in its sociological focus: ‘a glimpse at the future’ – the imperative to buy the game is clear, play the future before it plays you.

Personal data is the commodity in Watch Dogs – the trailer’s stentorian predictions are meant to frighten and dislocate: you are no longer an individual. You are a data cluster bound to a vast global network. You are endlessly surveilled by machine-vision, and it has the ability to see through your skin right into your data shadow, your soul. You can lie to everyone else, but the quantified self is pure as data-driven snow. (Peter Pan is looking prescient. All we need to catch you is your shadow.)

The trailer poses that question that comes up again and again in these scifi visions of the digital future: who controls the computer? Syndicate and Watch Dogs and Deus Ex are all about corporate control of life-altering technology and a distinct lack of sufficient regulation or oversight. It isn’t just about the (potential) evils of innovation or the (potential) evils of business – it’s about that hybrid of technological innovation and capitalism that results in technological autonomy. Which is when advances in tech start taking on a life of their own – placing the economic well-being of corporations or nations in opposition to people’s (or rather – their consumers’) physical, psychological or emotional well-being.

Nobody is driving this bus. But we’re all going to the future it’s taking us to.

In Part III – the last part, I promise – I’m going to talk about humans, politics and power – some of these science fiction questions that the New Aesthetic and its tumblrised, collate-and-curate approach seems to sidestep. Who is doing all of this looking and why? What’s it like being the object of the technological gaze? And is Google’s Project Glass dorky or awesome, or both? (Answer: yes.)