Extenuating Circumstances is a weblog by Dan Hon

The future is Movie OS

The tl;dr headline/controversial summary:

Remember Mark Coleran’s (and others) Movie OS motion graphics? Like it or not, I think they’re the real future, and point towards interface and interaction design for the rest of us. Movie OS is right, and everything else we have is wrong.

Thesis:

Normal people ‘don’t understand’ computers. Of course, the position is more nuanced than that, and you can do everything from point to documents like Apple’s HIG (for both the OS X platform for Mac computers and the iPhone and iPad platforms), Microsoft’s Windows User Experience Interaction Guidelines and Windows Phone 7 Series UI Design & Interaction Guide to the trainwreck that was the Read/Write Web Facebook Login Fiasco.

But let me focus on one, small aspect, and see if I can persuade you that there’s a genuine insight into how we design interfaces.

Action and Reaction, Cause and Effect

By an intuitive interface, in some ways what we’re describing is an interface which makes it abundantly clear what the consequences of an action are. Ideally, we’d like those consequences to be communicated in a subtle but understandable manner, and also in a way that requires the minimum amount of high-level cognitive processing in order to get across the meaning of the message.

Some examples:

• one-way destruction of data with no undo
• tagging of an image that will propagate through a social graph, publishing information to an audience of hundreds (thousands?)
• moving, not copying, files

The tools that we have to get across these messages – which, in their own ways, are all incredibly important – are a very limited palette. One that looks like this:

A typical Windows Aero modal warning, from the Microsoft Windows UX Guide

A Windows Aero risky action confirmation, from the Microsoft Windows UX Guide

A Standard Alert on Mac OS X, from Apple's Human Interface Guidelines

A well-written alert message on Mac OS X, from Apple's Human Interface Guidelines

A determinate progress bar on Mac OS X, from Apple's Human Interface Guidelines

An action sheet, modal view and alert on iPhone OS, from the Apple iPhone Human Interface Guidelines

A bar-style progress view in a toolbar on iPhone OS, from Apple's iPhone Human Interface Guidelines

A network activity indicator and a toolbar activity indicator in iPhone OS, from Apple's iPhone Human Interface Guidelines

Why do we have these UI palettes? Well, mainly because of Xerox. And they haven’t changed in over 20 years.

So how do we get across this incredibly critical information more clearly?

Luckily, a whole industry has already done it for us, the problem is that we (the geeks, the professional computer users, the internet commenters) have been laughably dismissive of that work, because it wasn’t “realistic”.

But just because something isn’t realistic doesn’t mean it can’t be better. And I think it is. Substantially.

Movie OS: it makes sense

Think of a situation you’ve probably been in recently. You’re in a darkened room with at least a hundred other people. Someone in front of you is using a computer. Something incredibly tense and relevant is happening, or is about to happen. Something that could potentially dramatically, critically change the course of someone’s life.

Hugh Jackman, or Tom Cruise, or Angelina Jolie, or Brad Pitt, or Daniel Craig is  pretending to use a pretend computer. And it’s all made up. In fact, it’s hilarious. It’s not even a real computer! It’s not even Windows! Or a Mac! Or it looks like a normal computer on the outside, but what’s on the screen bears no resemblance to what you see in your office.

Real computers don’t look anything like that! It’s like a Fisher Price interface! What’s all this stuff whizzing about? Why, when Meg Ryan uses AOL to send an email, does it animate a letter-fold, put itself in an envelope and fly off the screen? Why, when Hugh Jackman is messing around with RSA encryption, is he moving 3D objects around a screen? Why is the encrypted email displayed as gibberish, when it would just be a normal email with, say, a PGP icon next to it?

Because if none of those things happened, if it wasn’t shown that way, you’d miss it. Forget the 10 foot interface. This is the 50 foot interface for brain dead people who like explosions. It has to be abundantly clear. You can’t miss it.

And you can’t afford to miss it, because Meg sending Tom an email is important. Angelina destructively erasing hard drives is important. Wayne Knight (Dennis Nedry) in shutting down Jurassic Park’s security systems and displaying a wagging finger is important. Ariana Richards, playing “Lex” “This is UNIX!” Murphy, navigating a 3D filesystem to do device management and open a door, or turn on a security system, is important. Incrdibly important. And you need to know when, or if, these things have happened.

"Access Granted" by Mark Coleran from Mr & Mrs Smith

"Lockdown Alert" by Mark Coleran, from xXx: The State of the Union

"Disc Wipe / Erasure Protocol ZERO" by Mark Coleran from xXx: The State of the Union

OK, so the main problem here is that these are all static screenshots. But.

In Movie OS, visual storytelling is used to make the system’s important, critical reaction to a user’s action abundantly clear. In Movie OS, you know if you’re logging into Facebook.

I’d argue that visual storytelling doesn’t exist – if it does, it hardly exists at all – in computer or consumer eletronics user interfaces. The entire palette of visual storytelling in terms of interface, through accident of history, is purely engineering and control-led.

This is where, I’d say, Apple is grasping when it says that interfaces should sometimes look toward real-life objects. Real-life physical objects have affordances that are used in effective visual storytelling – and animation – that can be used well to make clear the consequences of actions. It’s more complicated than that, though, and it can go horribly wrong as well as right.

I’m a big Pixar fan, and learning more and more every day about animation. Animation, and the kind of animation that Pixar praises, worships and strives for, in terms of conveying weight and emotion and meaning, is what I’d argue our next-generation storytelling interfaces need.

Expert animators know how to visually convey, unambigously, the weight, the heft, the sheer effort (or lack of effort) Mr. Incredible is expending in picking up a car. A car: something that’s heavy. Something that’s heavy like 20GB worth of files, as opposed to something light, like only 20MB worth. But one can see how being instantly shown the heft, the feel, the weight of something in a file operation is helpful and intuitive in a way that text displaying “You are moving 20GB of files” isn’t.

File Copy Progress Indicator from Windows Aero

I give our industry slack. For most of the last twenty years, we haven’t had the luxury of a stupendously powerful graphics processing unit sitting idle while you’re not running around a gorgeously rendered 60 frames per second photorealistic 3D environment. With ample processing cycles spare for physics modelling. Do you see what I’m getting at? We need storytellers and animators to get across the cause from our effect as a user. Frameworks like Core Animation and Windows Presentation Foundation are the first steps to providing ways in which we can use animation and storytelling to enhance the user experience.

I’m not arguging for a virtual 3D interface where you walk into your 3D office and you pick up a piece of paper in a filing cabinet and it flops around in the air convincingly. I’m arguing for taking the visual storytelling talents, the learnings from evolutionary cognitive psychology in terms of the affordances our brains give us in understanding a phsyical environment, and letting them loose on how we use our WIMPy environment, at the very least.

There’s a reason why Apple has a patent on rubberband scrolling. It is not a coincidence that Steve Jobs is CEO of Apple, as well as having been CEO of Pixar.

I caveat all of this with the note that I’ve had no formal training, and many people more qualified than me have probably thought a lot more about this kind of thing. Some of my best friends are, as they say, interaction designers. But at the very least, I’m pleased that I’ve finally worked out why over the last decade or so I’ve been collecting DVDs of movies for no reason other than their vision of what computing interfaces will look like in the future.

So: am I wrong? Does this make sense? Please comment below.

Update: Everything old is new again. A kind comment from Hacker News points out a paper from 15 years ago entitled Animation: From Cartoons to the User Interface.

Update: From another kind comment at OSNews, a collection of Movie OSs at TV Tropes, entitled the Viewer Friendly Interface. TV Tropes lists Key Tenets of a Movie OS / Viewer Friendly Interface. One of them might sound familiar.

• All applications must be run full screen – there is no multitasking on television. Windows may show in the background, but they might as well be wallpaper for all anyone uses them.

Remind you of anything?

Update: Edited for spelling and grammar.

Nice article. I’ve done work in both interaction design and in entertainment. I agree with you about taking some ideas from movie UIs. However, many of the large alerts, strong feedback mechanisms, etc become extremely annoying when performed many times. Typically the user expects a certain action to happen (unless movie viewers), and is attuned to it, so only a relatively subtle cue is needed.

Arguably, these stronger cues would make sense the first few times. But, you don’t want to change your feedback mechanism mid-stream or people will think their actions have started failing. They might fade over time, but that’s an order of magnitude up in complexity…

Your article is interesting, but I disagree with the basic premise that the future of UI is going to resemble what we see in movies at all.

Movie UIs look and behave the way they do to communicate what’s happening not to the user, who, by the way is usually frantically mashing the keyboard to convey a deep knowledge of the software he or she is using. But they exist to convey what is happening to the people observing the user.

They aren’t usable. They are watchable.

@Mike But what I’m arguing for is learning from what works in Movie OS. I’m not for a second saying that Movie OS is usable – but that it has solved, in its own way, the specific problem of linking cause to effect in an unambiguous manner, and that’s something we can learn from.

@Dan I agree that the “look” would most likely get annoying after a while. Movie OS only deals in the needs of the instant scene, not Meg Ryan sending her 200th email to Tom Hanks. So yes: once a user has learned the (predictable, usual, complex) implications of an action, we should be able to tone down the response.

http://tvtropes.org/pmwiki/pmwiki.php/Main/ExtremeGraphicalRepresentation

Agree with Dan. A consistent feedback mechanism is important. Basically this is what a tutorial at the beginning of a game or use of a system does. Separating it out so you’re in “tutorial mode” etc. makes sense because you can show what the actual alerts would look like in that mode but you’re giving them added emphasis.

Real life interfaces are complex, because they display complex things. Movie interfaces are simple because they display simple things.

Your eye is guided easily to the important element of a movie interface because it was predetermined that this is the important thing of the story at this point. As software designers, we don’t have that luxury

@Zeus: I disagree. When you’re designing a tool, you should know what the most important element is.

@ted Agreed. I have no problem with a tutorial mode.

how about what Firefox does when installing updates/ update availability. for the most part, like you point out, osx has taken the lead (i am a fairly new convert and i am loving it).

I like the thinking here, but I think its an incredibly hard thing to do well and very easy to do badly. I would argue that some of the most ridiculed UI experiments of the past attempted this kind of narrative interface: Microsoft Bob anyone or everyone’s favourite the ms office paper-clip.

Still I like the championing of things the majority sneers at, and in the often insular world of IT, its often much needed!

I agree we can learn things from movie’s. Something I think some movie UI’s do really well is finding visual metaphors for technical models and abstract processes. I’m a real fan of that as long as the metaphor is really good. Because then we understand what is technically going on. Good visual models capitalize on what we already know or instantly understand. They model exactly on to the technology on an abstract level.

Examples:

The Disc Wipe in xXx: The State of the Union shows in a very visible way how hard drives are being emptied. We get this because the cylinders are being emptied. We get it’s a network of disks because there are more disks on the screen.

When we defragment our hard drives we see colored bars scattered all over the place which then slowly line up neatly to fill up all the gaps. Representing how our data is grouped together.

[...] Design: Useful Design Tips For Your iPad App Storytelling in an OS: The future is Movie OS Stupid floaty bar: Some Gaps To Fill Between Web Apps and Desktop/Mobile Apps CSS3 Generator Web [...]

the 3d file manager in jurassic park is real.

http://en.wikipedia.org/wiki/Fsn

http://web.archive.org/web/20070409024417/http://www.sgi.com/fun/freeware/3d_navigator.html

I have always felt for the plain, sharp 2D interfaces of the movies. I’ve tried more than once to resemble some of that in my own computers, with not that much success. Contrary to the vast amount of rebuke from the OSNews comments(a proud member) I believe the UX/interaction designers won’t lose anything by testing some of this concepts in certain environments. Maybe not the OS(at the end, the OS is just a platform for the actually apps you use) but for some data view/presentation/driven apps, and why not some military software too just like the movies.

I confess that would be quite delighted to have the same UI in my computer as that kid in Code Lyoko, though.

[this is good]

There’s a common basic semiotics of HollywoodOS, and it’s one that has evolved in parallel with the UI but also with the shorthand understanding of that UI. HollywoodOS is both an idealisation and an encapsulation: a simile of a metaphor.

I’d argue that visual storytelling doesn’t exist – if it does, it hardly exists at all – in computer or consumer eletronics user interfaces.

As ted notes upthread, though, games do it all the time, and ‘tutorial mode’ has become increasingly subtle and less teachy. The first rooms in Portal; the introductory gameplay in the various GTAs. (Not so much the sporting games, though, which is interesting.) There’s an assumption on both sides that the control model is unfamiliar and that familiarisation is necessary in order to advance, and so it’s taught, but in ways that are part of the gameplay narrative. AchievementUnlockedOS.

Everything you wrote I find it really good… but the part on only-fullscreen applications it sounds a bit useless to me. Unless the system you have in mind is only for a certain kind of people and a certain type of applications.

[...] Hon New blog post: the future is Movie OS – http://danhon.com/2010/04/16/the-future-is-movie-os/ [...]

Honestly, I took the article as sarcasm.

As long as you get your keyboard shortcuts designed by someone who can touchtype – and you don’t use the Mac ones for anything else – and you don’t use Appple-left cursor for “exit this page without warning or saving changes”… you can create as many pretty pictures as you like.

Real software is ALREADY like movie mockups. Back when I was a nipper it was obvious that something was a mockup when it had fully formed and justified text, because real computers didn’t waste time on typesetting. Or when you had text of different sizes and colors on the screen at once.

Then it was the idea that different programs that did the same thing could have radically different looks, like Facebook and Google Reader. What a waste of computer time.

Oh, and that 3d filesystem from Jurassic Park? That was a real UNIX user interface developed by SGI.

But having a huge red warning come up every time you delete a file? Can you imagine how ANNOYING that would become? Moving 20GB of files should look “heavy”? But if we’d been doing this all along, it would be showing that big truck every time you saved a Word document to disk and we’d be groaning about how camp that was, because that used to be a heavy file.

And our computers DO have better things to do when they’re not rendering graphics. The most important thing they should be doing is NOT RUNNING OUR BLOODY BATTERIES DOWN.

It’s an interesting idea, and I think it will work that way for people who use the iPad and it’s progenitors, it may even make Microsoft relevant again. I do think however that you’re going to catch a lot of flak from geeks, who tend to be very wedded to their interface, and since they understand the OS like to keep it as minimal/functional as possible.

You can see this dichotomy in action between Gnome and KDE, one is heavily typed, even the screensaver is locked down, the other is massively configurable.

For the average n00b MovieOS is probably the only way they will ever understand what their computer is actually doing.

[...] Extenuating Circumstances ? The future is Movie OS"Real computers don?t look anything like that! It?s like a Fisher Price interface! What?s all this stuff whizzing about? Why, when Meg Ryan uses AOL to send an email, does it animate a letter-fold, put itself in an envelope and fly off the screen? Why, when Hugh Jackman is messing around with RSA encryption, is he moving 3D objects around a screen? Why is the encrypted email displayed as gibberish, when it would just be a normal email with, say, a PGP icon next to it? Because if none of those things happened, if it wasn?t shown that way, you?d miss it. Forget the 10 foot interface. This is the 50 foot interface for brain dead people who like explosions. It has to be abundantly clear."(tags:design computing ) [...]

I was with you until “learnings”. Now I’m hoping that someone bites you on the nose *really hard*. “Lessons”, mofo, the word is “lessons”. Diarize that going forward, in this space, across the piece.

Sure, Coleran is cool, but most of his ideas look like they were inspired by Masamune Shirow’s Ghost in the Shell (the two manga more so than the movie). Which is totally awesome, but we should visit the source for some more direct insperation. http://www.genjipress.com/img/159307204X.jpg is one example. Google “man machine interface manga” and browse for more. There may be naked anime women so NSFW beware.

“Because if none of those things happened, if it wasn’t shown that way, you’d miss it.”

Not necessarily true. It would have to be shown in a different way. For instance, the movie Office Space. While it wasn’t an actual OS or anything like that, it mimicked them very closely and managed to keep the audience informed through close ups on the monitor’s information. Did it look as cool? Well, no. Did it look like something we could all actually relate to? Yes, yes it did. And the slowly moving progress bar added tension to the scene and allowed all the relevant information to be absorbed. You don’t have to know what commands are being performed to know that “computer stuff is happening.”

The Meg Ryan example could have been performed thusly:
INTERIOR. Mail is being typed. Meg exhibits the body motions characteristic of whatever emotion she’s putting into her email. She’s done. Sits back, grabs the mouse.
CLOSE UP, SCREEN: the mouse travels up towards the send button as the camera pans with the cursor. It hovers over the send button.
CLOSE UP, MEG: She bites her lower lip in anticipation, eyes shining.
CLOSE UP, SEND BUTTON: You hear the mouse button click, and AOL announces the mail has been sent. Meg sighs in relief.

But no, an animated envelope is way easier and looks ‘cool.’ Why do we assume everyone should be led like idiots through computers? Maybe back in the 80s and early 90s before everyone’s mom was on facebook, but nowdays how many people don’t know what a send button is besides maybe grandma and grandpa? Are they watching Swordfish?

I like fancy interfaces, don’t get me wrong. I just don’t think designers should needlessly waste space (or time) by a crap but ‘pretty’ interface. The interface should be useful, minimal, and fade into the background of the actual operation of the computer. Otherwise you are using Duplo blocks instead of Legos. Hardly an effecient use of space. Anyone remember Kai’s Power Tools? http://veerle.duoh.com/images/uploads/KPTexture.jpg

One future is Movie OS, but for me, the answer will always be Efficient OS. But that’s just me, and everyone’s answer is going to be different.

P.S. Thanks for mentioning Xerox.

Oh, I’d also like to note that as displays get bigger the size of the interface objects will be less of an issue – and as our computers become more aware of the world around them (they’re already getting the sense of feel in every day use) they will be more reactive to our presence – hopefully the candy piece interfaces of the future will rescale themselves relative to your viewing position, becoming less dense and more visual as you go further away and as you get closer displaying more and more data in a denser visual architecture.

See http://www.informationisbeautiful.net for some great examples of information interface design, especially the Billion Dollar o Gram and Billion Pound o Gram (US and UK budgets respectively. Also check out the book “The Visual Display of Quantitative Information by Edward Tufte” who is like the go-to guy on efficiently representing data in a visual way.

And for an example of where these interfaces are actually going Right Now please check out BumpTop: http://bumptop.com/ I expect you’ll have an interfacegasm over that one. :D

I agreed with Peter da Silva and the rest that: Computer shouldn’t waste it’s computing time for something ridiculous like that. Wiping hdd on xXx is user friendly you’d say? Think again: How MUCH wasted RAM for that kind of UI? I hope our current OS’s doesn’t mimic ‘that’.
PS: Microsoft has done their help section pretty well. We don’t need tutorial like what you’ve said (well, if you really want one that bad, you can search on the site like youtube)

if there’s anything I’ve learned since joining a software development program with no real software design experience, it’s that consumer GUI and the interfaces that people use to actually design and use real software are always going to be completely different things. For instance, I was noticing that the movie, “Angles and Demons” contained a scene taking place at CERN where all the computers had really flashy GUIs that would show everything happening in a way that was visually comprehensible at a glance. This would never really happen, but, like the article points out, is necessary for the storytelling. The REASON, however, that it is unlikely to occur that way in real life, is that the applications measuring the particals generated by collisions in the LHC are probably written specifically FOR the LHC and CERN by scientists who work there and don’t have A: time, or B: interest in designing flashy GUIs for their software. So quit griping, your integrated programming environment will alwasys be just as boring as ever, and you can always just run your machine from a command prompt if you are that turned on by green alpha-numeric characters flashing incomprehensibly on the screen. Lets face it, complaining about the perceived future of GUIs is like complaining about the clothes in your closet, if you don’t like it, go get a new one.

[...] a further link about interface design: The future is Movie OS(?) On the one hand, I’m tempted to roll my eyes at this notion; on the other hand, [...]

Concept videos of MovieOS: videos at http://youtube.com/automenta

An insightful idea. The problem with common dialogs, of course, is basic and fixable without a flashy UI: Give confirm/cancel buttons names that match the action. How many times have you had to pause to determine if it said “Exit Yes/No?” or “Save Yes/No”? Why can’t the message be “My uncle Hank thinks this is a bad idea. Save/Discard Changes?” That alone would to worlds for everyday users.

The problem with movie UIs is that they’re *unlikely* and/or single-purpose. It’s true that normal people prefer modal UIs–even though UI wonks tell us that modal is bad for usability (because users get confused about what mode they’re in? I guess.) Making that kind of flashy UI in a way that is functional (as opposed to just pretty) is hard, and making it able to do all the things it needs to do but that you *don’t* see in the 15 seconds it’s on camera is also hard.

I had a phone that did the Meg-Ryan-AOL-Mail-Folding animation for every SMS message sent. It was annoying! The best part was that it didn’t start sending until the entire animation (5-8s) was over.

On the other hand, zooming (the basic idea of 3d on a 2d display) could easily implement complexities of animated events with “skill levels” of scaling down the animation as it’s frequented, allowing multi-tasking to become simpler. I’m looking at Gnome Shell for this.