The incredibly fast progress of “virtual production” techniques in filmmaking is just astounding and will soon drive radical innovation in the broader built environment.

Virtual production uses a combination of large format direct-view LED display walls with live-rendered content and precision tracking of camera position & viewpoint data to capture “in camera” real actors on largely virtual sets. Only a few critical physical props are generally built that the actors need to interact with.

Below is a sizzle reel from Unreal Engine showcasing a huge assortment of different productions and how they are using such tech:

Jon Favreau and his team are enthusiastically driving the development of virtual production with the hugely successful Disney/Lucasfilm “The Mandalorian”, supported by the cutting edge “Stagecraft” tech developed by the famed ILM division of Lucasfilm. Their progress even from Season 1 to Season 2 is astounding, as you can see in the video below:

While the fundamental concept of a virtual reality “CAVE” style environment has been around since the early ’90s, the technology was always a highly limiting factor, either through poor visual quality, encumbrance, cost, or some mix of all three.

What has happened in the past couple years is clearly a “tipping point”: LED screens have reached a resolution and image quality that is nearly indistinguishable from reality; live-rendering engines are highly sophisticated in generating massively complex environments and realistic animation; and the hardware to render such life-like imagery at such ridiculous high quality levels has dropped in price to be commonly available. Add to that the digital camera gear and positional tracking systems, and you have the foundational infrastructure for profound change, and not just in the singular act of “filming” actors: In the ILM video above, the key point is not just the usefulness of the LED “volume” as they call it; it is the radical digital transformation of the entire production workflow. From the initial art development, to location scouting, to capturing performance, to post production – everything is put into one seamless “pipeline” which radically speeds up the overall development process.

And the AEC world is immediately benefiting, with technologies like the Unreal Engine being quickly adapted into systems like Twinmotion to realize the promise and power of seamless digital design/visualization workflows.

We have to realize the profound changes these technologies can bring to our built environments if the creators of those environments step up to the opportunities and challenges poised by such transformation. While the slow moving AEC world can only talk about vague concepts of “digital twin” technologies after spending two decades struggling to implement basic BIM, we have to realize that because of the plummeting cost of large format DVLED screens, many project customers in the real world will simply jump ahead and turn their spaces into extended-reality portals to the virtual world, before architects and designers can even wrap their heads around the concept.

For example, if a space is bordered purely by DVLED surfaces, of such high resolution and quality that an occupant of the space can’t distinguish the “screen” surface from reality at arms length, what is the “architecture” of that space? The digital content becomes the architecture – or maybe better phrased – the digital architectural experience. Such is why Gensler – the largest architectural design firm in the world – is developing a division they call DXD – short for Digital Experience Design. They don’t want to be left behind as their leading clients demand evermore sophisticated crossover digital experiential concepts.

So while the AEC world plods on, artists like Refik Anadol, Leo Villareal, and TeamLab are pushing the concept of “architectural experience” far into the future, with projects like this gorgeous TeamLab space in Miami:

The scary part of digital screens is that if the architects of a space don’t have their act together in designing the digital character of a space, you can bet that advertisers will eagerly fill the void with marketing messaging at every opportunity.

But designing the digital personality of a space doesn’t require Hollywood-grade direction. For example, simple biomimetic effects, like waterfalls, are captivating and powerful spatial features. For example, the work of Fusion CI Studios with their iconic lobby feature in the Salesforce HQ and other hypnotic effects:

Adding interactivity to such “XR” or extended reality spaces offers another mind-blowing level of development opportunity and project complexity. Such interaction design in physical spaces requires a new mix of skillsets not at all common in the traditional AEC world, such as UX/UI, game design, character animation, etc., leading to the fast growth of firms like Moment Factory or Hello IO, with their wild mix of technical and artistic skill sets. For one example, check out the delightful interactivity rampant throughout the Hello Park family entertainment centers:

You’ve probably heard parents whine: “These kids today! They think every surface is a touchscreen!”

Ummm….in an environment like Hello Park, every surface IS A TOUCHSCREEN!

I lamented in a previous post that smartphones delayed innovation in physical interactivity (“phygital” as some groups are now trying to label it) by over a decade. Investment dollars were sucked into apps for teenagers and vanity mongers staring at their phones like zombies. But it wasn’t preordained that smartphones are the best way to interact with the virtual world – nor are they the end-all be-all interface to digital experiences. In fact, there is far greater creativity and innovation potential to be had in connecting our large-format physical realms to the virtual realm, if for no other reason than it creates the opportunity for shared virtual experiences – and we are very social creatures, after all. We will see tremendous investment and innovation going into large format DVLED screens in retail, hospitality and cultural application, along with innovation in the associated content development and live XR experiences, now that the fundamental technologies have become almost unbelievably compelling.

You have to wonder what schools of architecture are doing to recognize and embrace such profound change? Perhaps they need to install walls of 0.5mm pixel pitch DVLED, instead of the traditional homasote pinup boards in the crit rooms?