What next for VR technology ?

As of right now (Summer 2015), the VR industry is in the crazy try-it-all and see what sticks phase. It’ll be some time before the dust settles and a set of core features emerges, and the killer use-cases are nailed with the sillier aspects left behind reminding us only of how young and innocent we were. Which affords us time now to pontificate. I warn you there is much pontification to follow.  Let’s gaze into the near-future to see what tech is likely to follow the rebirth of consumer VR.

Where is VR taking us?

Where on earth is VR taking us and do we want to go there? What are the benefits for humanity beyond entertainment and industrial use? Could it be used to increase empathy, facilitate therapy, and help us connect to those at a distance (culturally or geographically), placing us into current or past conflicts so we can understand the situation directly? With a new media technology, the tendency is to begin with translating our existing media into the new ‘format’. I imagine surrond video is going to be huge. I’m more interested in what VR can do that no other media can. To that end, I talk about body transfer and gender-swapping in VR, role-play, PTSD and combatting anxiety. Darn, now I’ve ruined the surprise.

Frankenstein VR: Rebooting a dead (virtual) DJ … 15 years on

I often wonder about the lengths we ‘grown ups’ go to to recall the imagination we once had…

In late 1999 and for most of 2000 I spent almost every waking hour laboring on a virtual reality project in a basement at the back of the Trinity College campus, in Dublin.  I rarely contacted my supervisor to check in on progress and in a way fell deeper and deeper into my own little world, by the end of which I don’t think he or anyone beyond my two lab buddies had any idea what I was at.  Certainly the examiners didn’t, and left the room puzzled after my demo, which was partly due to the difficulty of introducing three people to a VR experience in under 30 minutes, and partly due to the massive array of options and ‘modes’ that I had incorporated (more on that later). Needless to say, the presentation was a disaster. Shamelessly, I’m rebooting the concept behind the project fifteen years later with the tech of today, and dusting off the report on the off-chance there’s something in there worth salvaging or at least laughing at…

The concept itself was inspired by a lot of cool stuff that was going on in M.I.T. MediaLab in terms of fusing technology and instruments. I was reading about the HyperCello developed for Yo Yo Ma which allowed him to lift the bow off the strings to hold and filter the sound with bow gestures and being the type of excitable fellow I was, got quite excited. I had heard that the Graphics Group at the college had a broken VR system and eventually got permission to use it to create a virtual instrument as long as I didn’t ask them to help with fixing it.  I instantly agreed.

Despite the fancy graphic the gloves readings were not entirely trustworthy. low pass filter anyone?

The system comprised of a glove, goggles, a globe and a giant ball of wires in a sub-pavement lab next door to a mysterious lab with pacemaker warnings on the door.  (I later learned that the labs used to be a row of houses dating from the 1830s, where Oscar Wilde had been raised, but from inside it looked like your average college lab.) The glove, from 5th dimension, tracked the curvature of the fingers and 3 degrees of freedom (tilt, yaw, and roll of the hand). It worked okay, although I assigned a fairly low ‘trust value’ to the orientation values.  The V8 goggles worked fine and were driven by two separate graphics cards, but to enter VR mode required updating the monitor settings by extending the desktop cross two monitors, like so:

Extending the desktop across the two graphics cards and ‘screens’.

Since I was developing and debugging the code in visual studio, whenever the code would crash (hey it was C++), I’d be treated to an eyeball full of the debugger – you haven’t lived until you’ve seen Visual Studio in VR, people…

To render the world in stereo I basically drew everything twice in OpenGL, with a gap in between, and wasn’t even embarrassed about it:

The draw method draws the entire scene once for the left eye, then translates to the bottom corner of the right hand screen and redraws the scene for the right eye.  In practice we actually translate a little past the right hand screen to compensate for the distance between the pupils.   Because of the limited field of vision in the world (approximately two to three meters in each direction) there is no reason to implement any stereo -perspective correction algorithms  (i.e.  it is acceptable that both eyes see the same perspective although in reality they would not).

The ‘Long Ranger’ globe emitted a weak electromagnetic field which the sensors used to determine their position relative to the globe, which they returned as three floating point numbers representing their x,y,z values in the american co-ordinate system. Of course, there was the issue of it being ‘broken’. Sure enough, the values returned were gibberish, or so it seemed. Having no tools or support to debug it, I started to output values from a sensor as I walked in straight lines carrying the sensors along the x and y axes and plotting the results. Instead of a nice neat grid I saw the lines converging on some point past the wall. The magnetic field was being warped into the room with the mysterious pacemaker sign. There’s some sort of magnetic black hole next door, I concluded.  Or as I eventually discovered: an electron-microscope.

As an aside, weak magnetic fields are so hot right now, forming the basis for the STEM controller system for VR input. Here’s an extract from the STEM FAQ explaining why they don’t wreck your credit cards:

The STEM System uses A/C electromagnetic motion tracking technology, one of the advantages of which is that it can operate at very low field levels. So low, in fact, that the peak field strength from the Sixense system is 40 times less than that of the Earth’s magnetic field. Therefore, any system or material which is not affected by the Earth’s magnetic field will not be affected by the field used with a STEM System.

Additional reason the STEM System cannot magnetize materials is that because the field alternates between positive and negative polarity with time, the average field strength is zero. The magnetic source used in the STEM System operates at or above ~8,000 Hz, so the field polarity changes faster than every ~125 microseconds. Any material will see a zero average magnetic field and no magnetization will take place.

The concept

Ah yes, the concept – this is the alleged point of the blog post. Rebooting the concept.  Well, as with most of my projects the concept changed daily. Some of the initial designs had grand names, here’s some of the grandest, combined with earnest descriptions from the report:

The Squeezebox

Controlling the sound using the distance between the hands to alter the pitch or pan is an intuitive interface.  Some forms of meditation involve imagining a malleable ball of energy between the hands, which can be rolled around.  When the hands are quite close, the power of the imagination can be very convincing, making it feel like there is actually a real force flowing between the hands.  This feeling could be amplified by the presence of a sound which is actually changing with the relative motion.

Musical Beams

When the hand breaks the vertical beam it becomes selected, and perhaps changes colour to reflect the fact.  The height at which the beam was broken would determine its volume and as in the other models, its pan and frequency could also be altered by various glove gestures.

Conjurer

…various music loops [are] situated in boxes on the floor.  By pointing directly at a particular box the player could then raise it from the ground at which time its volume would increment with respect to its displacement.  The player would play the role of conductor

I toyed with the even grander idea of creating a virtual instrument but ended up settling on a concept where the user was surrounded by four ‘walls’ of semi transparent bricks which could be touched or punched depending on your disposition to trigger one time samples, or loops, and select effects to manipulate those sounds by squeezing the glove (the one reliable gesture).

Here’s a top down view, taken from the report, featuring a little clip art man in the middle – that’s you, pretending to be a DJ:

Each wall was composed of a number of bricks, each ‘containing’ a wave sample. A sample tracking system meant that new samples always came in on the beat. In this way the user could gradually build up layers upon layers of looping beats and textures.  The beats were directly in front of the user, with ambient pads to the left and one-time samples (which don’t loop) to the right.  Behind the performer was a fourth wall containing a bank of effects that could be grabbed and applied to the current sample. The effect parameters can then be controlled through the glove. For example. the DJ could activate a sample by punching a brick in the sample wall. spin around, punch the pitch-down effect and squeeze the right hand to drop the pitch of the sample right to the floor.

Did I mention there were options for everything ? I had recently discovered MFC Dialog boxes, and let’s just say that I made ‘extensive use of them’.

Remember when interfaces looked like this?

Even the effects had MFC dialog boxes:

Muse would have loved my arpegiator

The arpeggiator doubled as a drum loop programmer too:

The sequences you created in the MFC program could be saved or loaded into bricks in the wall, which would play it back when triggered.

The soflware that drove the system was actually a distributed affair. One program ran on the ultraTrak machine (connected to giant web of wires running to the magnetic sensors) and pumped the filtered data across to a machine where a multithreaded application dedicated one thread to receiving the data and another to rendering the sound and graphics. The sound engine itself was based on DirectX and amusingly the delay based effects were achieved using MFC’s very own WM_TIMER messages, which probably accounted for the odd wobble in the delay.

The reboot

Rebooting this project is going to be fun. The next post will talk about a design for a prototype using Unity, the Leap Motion for hand tracking and Google Cardboard. I make no guarantees that any of the original features will make the cut. I might even drop the dialog boxes.

If you want further giggles, here’s the project report.

VR is happening people …

And it’s happening now. Like this year. 2015. For realz… Valve and HTC dropped a surprise bomb at GDC by unveiling the Vive.  Those who tried it, have gushed about the sustained sense of presense they felt – the sense that you are inhabiting the virtual world you see all around you.  People are reporting not just brief moments of presence, but full-on total immersion for twenty plus minutes (or whenever they are kindly asked to leave). Nathan Grayson describes laughing like an idiot (and not caring), and the fear of moving from his safe spot in case he fell in the virtual world.  The Valve demos  hint at what is to come, and the sense that it’s game-changing.  The best news – the Vive is due to launch Nov 2015 and is actively pursuing developers to get kits into their hands.  Better still – it leads the race to market of a series of units which are all being developed in parallel, with many as loss leaders, hoping to regain on software post-sale.  Sony’s Morpheus for PS4 + Move controllers and the Rift are one’s to watch in 2016.  Developers keen to get started can buy the OSVR Dev Kit from June this year. Sign me up.

It’s the Vive that has me excited right now – in particular the reports of its presence, which is of course the whole point of VR.  Some attributed it not only to the headset’s tech but also to its unique controllers which are tracked to sub-millimeter accuracy by one or more laser base stations. The tracking system is genius – An alternating horizontal and vertical sweep of a band-laser allows sensors embedded in the headset and controllers to report timings used to determine an exact position and orientation in a 15 x 15 foot space.  Seriously – that’s a huge volume, and any tech with these open-standard sensors can be tracked!  So, seated, standing and moving experiences are now all possible, with the ‘Chaperone’ system fading in to alert you as you near a wall.  Moving around the space may be another key ingredient in the sustained presence, although I imagine since few people have 15×15 foot to spare most experiences will work with less.

Okay, I know, I know .. we had this in the 90s, and we had the virtual boy flop in 95, but that was all before the mass production of smart phones drove down the cost of all the components required to build a convincing VR experience.  The adrenaline shot in the arm for VR may have been Facebook’s acquisition of Oculus, however it remains to be seen what devices and experiences the general public will be attracted to.  Kudus goes to Oculus and Valve for tackling the hard problems and attempting to set the bar for ‘immersion’.  Michael Abrash’s fascinating blog gives a great insight into the problems of perception and the limitations of the current generation screens.  Many of the issues they are tackling relate to the queasy feeling of ‘VR sickness’ that users of the current systems experience.  Ultimately reducing the interval between detecting head movements and updating the scene that the brain groks are key to reducing this effect.  Oculus are keenly aware that its critical that devices don’t go to market before this is solved as it could halt the entire rebirth of VR, with the meme of ‘VR would rock, if it didn’t make you puke’ killing it stone dead.  Abrash argues that ideally we’d use screens with a refresh rate of 120Hz+ and zero persistence, combined with 1K resolution and Oculus laid down the gauntlet by specifying the minimal requirements for presence which includes 6DOF tracking with submillimeter accuracy, >90Hz refresh rate, <20ms latency and so on.  Rendering games at 120Hz would work for simple graphics but there are a number of tricks discussed that allow rendering the game run at sixty HZ and interpolating the scene based on the latest head tracking data to simulate a true 120Hz (aka timewarp). In terms of screens and lenses various solutions from double lenses to Fresnel lenses to RGB strips vs PenTile displays are being flouted as well as techniques such as casting light into the eye using lasers and micro mirrors (arrays of one million mirrors delivering 720p e.g. the Glyph and Microsoft’s HoloLens), which bring us to what makes this explosion in VR so exciting – there are suddenly so many ‘new’ hard problems to solve ! And to do so we need to fully understand how the brain interprets light and audio to build its mental model of the world. We need to consider what parts of an image the brain attends to, and how moving the eyes creates a temporary blind state in the brain (a.k.a saccadic masking). On the audio side, we need to understand how the brain localises audio sources. If you are moving your head, sound sources need to be ‘stable’ in 3D space – they shouldn’t feel like they are moving (unless they are).  This ‘simple’ little nuance is being investigated in both Trinity College Dublin and York University, and is just one of the many many problems to solve.

Having said all that, there is an alternative perspective, which isn’t driven by the desire to present a perfect experience. Almost as a joke, Google responded to Facebook at Google IO by gifting attendees with Google Cardboard, a VR mount for your phone so DIY that you can print it yourself. A Google 20% time project that has since shipped 40,000 units and spawned a Cardboard team at Google. Not bad.

You’ll need a set of lenses though, so it’s best to splash out buy a pack off amazon for 15euro with the full DIY kit.  Also, Google released a set of demos for Cardboard which use WebVR (VR in the browser) to deliver some interesting experiences.  Despite the extra latency, the music experience alone was enough to convince me, that the web and mobile have a lot to offer.  The web is so easy to share (no installs required) and unless you obfuscate your javascript you are sharing your code and knowledge to other developers.  As for graphics, the Mr. Doobs Three.js library shows it is well up to task. Unfortunately the same can not be said for audio, but surely that will come in time. Phones seem ideal for more casual content and right now are the only solution that doesn’t require being tethered to a machine.  (With the Rift/Vive you still need to plug into a PC and have wires running from one to the other, which for me really feel like a deal breaker for a world where convenience is king.)

Phones are iterating fast and a hotbed for new tech: Two interesting changes are Google’s Project Tango which is attempting to bring Kinect-like computer vision processing to phones. This is powered by a custom chip from Movidius which can take the inputs from stereoscopic or multi-camera phones and create a three dimensional depth map of the environment in real time. Imagine the Kinect2 inside your phone.  The chip is designed with the kind of parallel processing required to do computer vision well and at a low power consumption. Key to the design is a shared memory which offers a single address space to the 12 cores which access it.  The challenge is ensure that this new programming model is made as painless as possible for developers unaccustomed to it.

A second innovation for phones, is the possibility of a built in pico projector. Imagine combining a Tango phone and a pico projector in a phone. It would finally unlock the type of Augmented Reality experiences that the promo video <br>

<br> for the Sony Vita envisaged but never delivered on.  My experience with AR apps on mobile was that many of the experiences fall down on the ability of the computer vision algorithms to track the fiducial markers in a stable manner, but these are all based on running on a phones general purpose engine – the CPU. Movidius’s chip could be the gateway to solid AR experiences as well as VR.  AR game designs open up a huge set of challenges but this videos such as ‘Speed of light’ hint at the potential:

However, we need to address the elephant in the room. Oculus have been very tightlipped on the question of input devices, apart from mentioning that input is a very important aspect of VR and anything important is something they will address….okay, that’s almost CIA-level evasive phraseology.  Personally, I think hand tracking or VR-native controllers are key to offering experiences that are fundamentally different than just playing games in 3D.  One potential solution way of getting your hands in the game is to use stereoscopic camera solutions such as the Leap Motion.  The Leap is a small monochromatic infra-red stereoscopic camera with a fast frame rate that you can connect via USB2 to your PC or (using an OTP adapter) to your phone (beta).

It shoots at 240 frames/sec, pumps the image data back to the device/pc for converting to a depth map. At this point their software extracts the arms, hands and fingers and furthermore detects some basic gestures – such as swiping, circling, tapping etc.  The device itself does not have a custom chip so the processing is all done in software on your CPU.  However, it can also track tools and the accuracy is good enough for air drumming for example.

There’s an obvious limit to camera based tech – which is occlusion (when hands are overlapping you lose tracking of the parts not visible to the cameras) but there’s no wires or gloves and you could combine with optional controllers such as a nunchuck.

One of the great things about the Leap is their encouragement of developers. They gifted the devices to many developers who pitched them ideas, and were rewarded with an explosion of code and projects which you can checkout here.  If you need convincing that VR is worth the hype, check out the sublime weightless project which pits you as an astronaut floating in a capsule in zero gravity.

Leap are tackling the problems of hand tracking based UI in VR experimenting with Tony Stark like interfaces such as the Arm HUD which superimpose a smart-watch like interface onto your forearm. Rotating your forearm, brings up a menu system. Hey yeah. Great for controller-less experiences which still need input beyond gestures and hand waving.  Other approaches such as the Hovercast UI hint at the interactions to come. And maybe soon, we will finally get to experience the video scrubbing interface from Minority Report…

<i> Rotating your forearm … reveals a different interface: </i>

There are a ton of other players in the input field from camera tech (NimbleVR), gloves to track bending in the fingers or holding tracker controllers (STEM, lighthouse).  Perhaps the best general solution to hand tracking would be a hardware accelerated Leap, which could for example be something that the Movidius chip could be programmed to do, as well as being re-programmable for more general computer vision purposes.  However, for games the nunchuck like controllers are probably best as ultimately pressing buttons is faster than gestures and controllers like the Valve controller also allow ‘squeezing’ which turns out to be a surprisingly convincing proxy for picking up virtual objects. Haptic devices have the plus that they counter the ghost-like feeling VR users report (I’m there, but I can’t touch anything). On the other hand, the camera based solutions can also offer image-passthrough (i.e. blend the virtual world and the real world) offering Augmented Reality experiences, or simply allowing us to not bump into stuff.

So, what’s my own stake in this game ? Well, back in my college days circa 2000, I developed a Virtual DJ experience in VR using the hardware of that era. The three components were a 5th dimension VR glove, a stereo HMD (head mounted display) and a  Polhemus Motion Tracking System which comprised of a giant beachball sized globe, 12 sensors and a mess of wires.  The globe emitted an electromagnetic field which the sensors use to determine their position.  I recall I had great difficulty getting accurate readings from the sensors and determined to map out the field itself by moving the sensor along straight lines and periodically emitting values.  Instead of the expected cube, I could see that the field was being warped into some point in the room next-door. I’d always been curious as to the warnings on the adjoining lab door, and investigated. In turns out, they were running an electron microscope … which as you may have guessed, was interfering with the Polhemus ‘long ranger’ globe.

Back then, I was lucky to get to work with these tools, which were out of reach of the budget of a frugal student. These days, the tech which colleges could barely afford then is cheap as chips. Which is why, I’m rebooting the Virtual DJ project.  I pitched Leap and they delivered a device to experiment with.  I’ve currently got a prototype in Unity using the Leap Motion + Durovis Dive plugins to create both a VR and a tablet version (after all, only devs really have the VR HMDs at present).  I’ll save the project for another blog post, but for now you can have a chuckle at my original project from 2000 here.

Further reading for the curious:

• The Vive is a super exciting high-end VR system designed to allow for full movement, complete with tracked controllers, and of course a software distribution platform – Steam/SteamVR.  The evolution of the Vive shows the attempts at camera+machine vision based approaches versus laser based (time of flight, I guess).  I like the way they hacked together two hard disks to prototype the laser solution. I also love the fact that the design for the tracking system is open source and Valve are encouraging other peripheral makers to use it. It’s currently the forerunner for presence, but I expect it may cost up to 1000$ once you factor in upgrading your graphics card too to replicate the experiences insiders have been gibbering about.

• My favorite description of the Vive demos.

• Michael Abrash’s blog on vision and HMDs getting deep into perceptual problems, and in depth tales from the trenches on the various plans to overcome the dreaded latency.

• Intro to Project Tango which is a google led consortium aiming to nail computer vision within phones. The movidius chip is a key part.  The chip has been integrated into a few tablets and phones.

• PS Vita AR Suite … what ever happened AR flickball … table top diving … and W.A.A.R (wide area augmented reality) ? These game designs look really fresh.  Mixing in reality and the game world is what kids do naturally. Imagine building a game world on the floor that the virtual characters actually walk around.

• Sensor fusion google tech talk is a technique of combining different sensors and some signal processing to achieve greater accuracy than the sensors themselves provide. e.g. Combining the drifty accelerometer data, gyro data, and compass data to provide a cheap and cheerful 6 degree of freedom approximation … i.e. head tracking.

• weightless – zero gravity physics experience for Oculus Rift + Leap Motion hand tracking. To me, I’m way mroe excited by these non-game experiences as oppose to translating existing game designs into HMD-format.

• Leap Motion air drumming demo showing the accuracy of its tool tracking software algorithms.

• HMD mounted Leap Motion A (pricey) mount to attach your leap motion to your headset. I’m a cheapo, so I used glue instead.

• Leap Motion VR project gallery A great gallery with videos and often links to code of a ton of projects from VR entuasitists to small studios. A powerful demonstration of the power of harassing the commuity.  If you build it … they will come.

• OSVR With the rapid proliferation of headsets and input devices, it’s going to be a nightmare for developers to make experiences that run on a wide range of designs unless projects such as OSVR take off.  It will be interesting to see how this plays out, or if developers will have to design for a select few devices however the incentives all stack up to make OSVR a success.  OSVR will come with a Leap Motion built into the front plate of the headset. It has SDKs for Unity, Unreal etc. and their own headset with interesting optics and also the optic to wirelessly send to the headset instead of tethering. It adds a measley 1ms latency – come on Vive – are you listening ?

• Oculus purchased NimbleVR which developed a camera based hand tracker but don’t plan to use it on first few iterations of the consumer kits (or perhaps ever).

• Leap Motion UI interaction with the hovercast menu. The hovercast VR menu system is available as a library.

• The Virtual DJ project from the year 2000.