I'm using the title of a popular song to describe how Virtual Reality is, among other things, a stupid idea. I think that's called an Allegory.
I’m sure most of you are used to a mouse. Some of you may be used to an old style graphics tablet. Fewer still of you will be used to a new style graphics tablet. Humour me while I describe the differences.
A mouse is a little doo-hickey which sits on your desk. As you move it around, a pointer moves around on your screen at the same time. You use it to get work done. It seems to do the job and people are fairly used to it. No one would say it’s unnatural except for those crazy fundamentalists.
An oldschool digitizer / graphics tablet is like a mouse, only you use a pen to move things around instead. The other major difference is that you can set the digitizer to have be “absolute” with respect to the screen (as opposed to “relative”, which is what a mouse does). That is, there is (roughly speaking) a special “mouse pad” (really a digitizer) which corresponds exactly to the screen. If you move the pen to the left hand side of the digitizer, the mouse pointer moves to the left hand side of the screen, and similarly if you move it to the right hand side. A lot of digitizers also have fancy features like pressure sensitivity and tilt sensitivity, but this is not necessary. This digitizer is better than the mouse at things like writing, or selecting things in an image editing program. It sometimes has issues, but overall, you can’t call it unnatural.
You can see that this is getting closer to actual hand-writing, so a tablet PC (or a new-school graphics tablet) is basically a graphics tablet with a monitor stuck in front of it. Now, when you move your pen around, the pointer moves in the same spot as your pen. Magic! You would think that this would make writing on your PC easier and more natural than having a graphics tablet, and far and above better than a mouse. In some ways, this is true, and in other ways, this is not.
You can write much faster and more accurately with a tablet PC than with a graphics tablet, but there are two fundamental problems: The first is, because you can see the screen and your hand at the same time, you can actually write at about double the speed compared to the graphics tablet, and this means the CPU finds it hard to keep up. More importantly though, when using the graphics tablet, the movements of the cursor and the physical movements of your hand are a fair distance apart. In fact, you can’t even see the pen when you’re looking at the screen. Any delay, therefore, is completely unnoticeable.
On a tablet PC, however, because the pen is right there in front of the screen, even the tiniest bit of lag is noticeable. Even though the lag can be as little as 10 or 15 milliseconds, this makes the difference between writing naturally on a piece of paper and the entire experience being awkward.
This idea of lag and the different amounts of it that we can tolerate based on the situation is critical on the success of various technologies. Another example is talking on the telephone: You can actually withstand up-to 500 milliseconds of delay between you talking and the other person talking before it’s very hard to have a conversation. However, you are far more sensitive to your own echo, and any echo greater than 50 milliseconds can confuse you so far as to completely stop you from speaking.
Lag also has important properties in gaming. If the lag time between you moving your mouse and the character on screen reacting is too high, it can cause headaches and nausea for the player. I had issues playing half life on an older PC under DirectX because the delay between the mouse movements and the character’s movements were too great. The (then) more responsive OpenGL games did not cause these issues. Note that this has nothing to do with frame-rate. You could have a million frames per second, but if you’re using a mouse with too much lag (like, say, earlier wireless mice) you’re still going to have issues.
This is all a round-about way of getting to Virtual Reality, and why it hasn’t taken off as a technology. A lot of people seem very surprised, but they really shouldn’t be, considering the one unique problem it has: lag!
When looking at a monitor, you can look at different parts of the scene. That is to say, you can alter your gaze at will. The control you have over your monitor also happens indirectly, through the mouse. Your mind adjusts to enable you to experience the 3D without feeling ill. However, when you look around using your neck or your eyes, a computer needs to respond immediately. You have far less tolerance to lag when it’s your head and eyes doing the looking as compared to your monitor. The real reason virtual reality has failed is that there’s no way of going all the way from receiving a signal for your head moving, getting it to a PC, getting that PC to re-orient itself, re-render the scene, and get it to the VR display without you noticing. More advanced rendering technology doesn’t really help, because this is related to lag, not to processing power.
In video games, we often need 60FPS instead of the 30 odd we get on TV and in movies. In non-interactive content, the lower frame rate is acceptable because the camera has control of the shot. In a game, it is the player which is in control. With VR, the player has more control than ever, and the system needs to get a higher frame rate still, and all that while ensuring that every frame is based on up-to-date info from your inputs. This is all a big ask, and one arguably not worth the effort.