Active Objects, Surfaces and Zones
Exploring the potential of luminous surfaces, interactive controls, and digital content for retail applications
Way back in 1998-2000, I explored the potential of interactive controls for retail lighting during my master’s design thesis. For the first time I will dig into my archives and publish my research online, much of which I believe is still very relevant in today’s hype around the “Internet of Things”. Many of the fundamental concepts I explored in employing active sensing and luminous surfaces may help those of you working on IoT, embedded lighting, or retail design.
A PDF of my final thesis document can be downloaded here: 20160902_BKoerner_HarvardThesis_ActiveObjectsSurfacesZones_2001
Beyond the final thesis document, I would like to expand on the context, content and relevance of the research. Here goes….
Part I: Preface – The Story of My Thesis
A long time ago, in a land far away…
…such as the year 1998 in Cambridge, Massachusetts….
I earned two degrees in architectural design: So how exactly did I wind up researching interactive retail lighting controls for my final design thesis?
During my undergraduate program at University of Virgina’s School of Architecture, I became fascinated with theatrical lighting. I took several courses on theatrical lighting design with professor Lee Kennedy at the School of Drama. Most of the theatrical lighting technologies were quite advanced and basically unheard of in architectural applications at that time. Lee’s enthusiasm for the art and science of lighting opened my eyes to a mix of technologies that I realized would clearly impact the world of architectural design.
When I attended Harvard University’s Graduate School of Design, I continued to explore advanced concepts in architectural lighting, fusing technologies from the theatrical world into the architectural world. Instead of the typical final thesis in which I would design a fantasy building, I opted to explore advanced lighting technologies and how they would impact the future of architecture. Michelle Addington, who is now a tenured professor at Yale’s School of Architecture, was my thesis adviser.
You need to understand how limited architectural lighting was at that time. To set the historical context a bit: My top-of-the-line PC in 1998 ran Windows NT on a 800mhz processor with 128mb of RAM. Digital cameras were expensive, poor-quality novelties (so I had to shoot on Kodachrome and scan the slides). Steve Jobs just released the original iMac, which made a big splash by ditching the 3.5″ floppy disk drive, but promised to make it easier for average folks to connect to the internet…by modem, of course! The World Wide Web was only about 5 years old at this point and in the midst of the infamous dot-com bubble, but early sites like Amazon.com or Outpost.com clearly showed the immense promise of e-commerce. Architectural lighting had only very limited, strictly proprietary control systems, all based on turning lighting on or off, or somewhere in between.
Part II: A confluence of technologies…
Around 1999, several nascent technologies emerged that together, would lead to a profound change in architectural lighting design.
Digitally Controlled Lighting
By 1999, the DMX controls standard along with robotic automated lighting fixtures had already radically changed entertainment lighting. Some of this spilled over into the architectural world, a category that came to be known as “architainment” lighting. Color Kinetics was born in 1997 when robotics engineers from Carnegie Mellon combined RGB LEDs with PMW and digital controls. With the first “solid state” digitally controlled lighting fixture, the promise of no maintenance finally made advanced lighting more suitable to permanent installations.
Networking and the “Internet of Things” in 1998
A couple early networked lighting technologies launched around this time, like ETC’s Unison system or Strand’s system (such as the ParkNet installation at Islands of Adventure), which incorporated IP networking to push DMX over long distances. It was clearly obvious that the potential existed to push digital control and digital content throughout a large building or even urban-scaled project.
Lee Kennedy at UVA experimented with a Rosco Horizon controller, an early theatrical DMX control system that replaced the typical lighting console with a PC-based application. One aspect of the Horizon was especially unique: The Horizon software provided an interface for HTML triggering of its channels and cues. Lee built HTML “magic sheets” that gave him a graphical, web-based control of his lighting system. The “Internet of Things” long before the concept even existed!
Light as a Material
LED’s for architectural use were just birthing onto the market. Blue LED’s only became “practical” light sources around 1997-1998; two of the first big examples of their implementation was the massive LED screen used in U2’s Popmart concert tour in 1997 and the original curving NASDAQ sign in Times Square in 1999. LED screen technology held great promise at that time, even given the embryonic level of the technology. I attended the LDI conference for theatrical lighting technologies either 1997 or 1998; at that conference was an early LED screen (I can’t remember the manufacturer who made it) that consisted of printed circuit boards crudely connected by flexible wires, allowing the screen to be rolled up for transport.
Once I saw that screen, plus the architectural scale of the U2 stage design, the future of architectural lighting was obvious to me: Lighting will become a material, one filled with digital content. It was just a matter of time.
And in 1999, during a design studio in at Harvard taught by Sheila Kennedy, we explored that very notion, that light could become a material. I took the topic in a more theoretical direction, proposing a soft fabric screen that emitted digital content for use in a modern theater.
I made a conceptual model that attached 5mm LEDs onto a flexible, clear poly-carbonate sheet; I proposed that in the future, the wiring would be fundamentally integrated into the material substrate – PCB’s would disappear. In hindsight, I was right-on-the-money with that prediction – as evidenced by the emergence of flexible, printed electronics. Below are the only images I have of the model (sorry for the bad images – remember – I was shooting on film! Film!!!!!!).
At the time of my thesis, internet retailing was also just in its infancy but the potential to transform “brick-and-mortar” retail was obvious. But the challenge was how to fuse the ethereal digital world into the physical retail world?
Experiments by multi-media artists, mostly using the MIDI communications protocol for digital instruments, showed the tremendous promise of using electronic sensors for interacting with digital media. An early example was the 1995 I-CubeX system from Infusion Systems (which amazingly, is still around after all these years), which created an amazing catalogue of various sensor technologies that could be implemented in the system.
Part III: The technologies come together…now what are the implications?
OK, so let’s summarize: In 1999, we had intelligent digitally controlled LED fixtures, the ability to network these, the ability to connect them to the Internet, the potential for light to become a building-material, and the potential to use sensors to physically interact with digital media…potentially in an application like a retail store. So from this technological base, I started to explore how to use interactive control of lighting and digital media for retail environments.
Part IV: Exploring the concept of dynamically controlled space
As you can see in the front section of the thesis book, I explored and diagrammed a language of “active” objects, surfaces and zones.
I realized that what we now call “web 1.0” e-commerce sites all had a distinct spatial progression: A splash screen, a category page, then a product page. This could be translated into spatial zoning and a progression in a physical retail store.
Proximity sensors could to detect where a person was in this progression; dynamic luminous surfaces and shelving could act as the visual guide. Below is an example of an interactive kiosk I proposed:
The technology platform was fairly easy to define:
Part V: Prototype #1
One of the mandates of the technical thesis track was to prototype. Luckily, I reached out to Kevin Dowling of Color Kinetics to pitch my thesis, and Kevin was happy to support by lending me some of CK’s early products, small color changing LED spot lights, flood lights, and cove lighting. I also was able to borrow a Horizon controller to experiment with HTML control. The problem for me was how to get these point/linear format light fixtures to emulate a plane of light and how to simulate sensor-control from a theatrical console.
The first prototype setup used 4 CK color changing spots in a typical light box to emulate a low-resolution luminous surface, next to two distinct glowing shelves with CK cove lights.
The Horizon controller luckily included two contact closure inputs, which I hacked with an IR near-range sensor and an ultrasonic far-range sensor, to at least demo some “live” proximity interactions.
The architectural professors at the jury though didn’t like how fragmented that presentation was; they wanted me to somehow unify the design.
Part VI: Exploring the concept of layered dynamic presentations
Michelle Addington pressed me though to take the concept even further. She was researching a variety of material developments, including variable translucency LCD glass, that hinted at a future where more sophisticated layering of information could be conceivably implemented- selectively showing or revealing space, visual data, or even physical merchandise. In 1999 I got as far at these sketches:
With today’s technologies, this concept of “layering visual information” is a bit more feasible. Very large LCD monitors, LED screens with very fine pixel pitch, transparent LCD screens, variable opacity LCD glass, and digital projections/projection mapping technologies could be combined to create some interesting plays of depth and focus control. I think there is a lot to be explored here, still to this day…
But back in 1999, this “layered” concept led me to the second prototype:
Part VII: Prototype #2
I devised a configuration using corrugated plastic sheeting to create layered front and back translucent surfaces, with shelves “hidden in plain sight” in the middle aperture. I conceived of the effect very much like a James Turrell “ganzfeld” piece. Between scripting the effects and the two triggers I could use, I was able to convincingly “fake” the final effect for the jury.
Inside the prototype wall.
I also programmed the wall to show emotive capability, such as a dramatic sunrise effect:
This seems so simple nowadays, but back in 1999 this was very unique. Most of the professors and critics had never seen an effect such as this outside of a theater performance, and certainly not from something that seemed like an architectural surface.
Part VIII: Awesome concepts! Soooo…what happened???
Long story short, my timing was impeccably awful: The dot-com bubble burst right when I graduated. No one wanted to hear anything about e-commerce, much more some crazy ideas about integrating e-commerce into physical stores. Since I had no programming abilities myself, I couldn’t start creating the “lumiSense” interactive store control system as I conceived of it. And my job offer at CK evaporated as they had to suddenly cut jobs (which is why I didn’t work there until 2005). Then 9/11 happened, which really threw cold water on the idea of a startup.
I did win some prestigious recognition for the project: Awards in ID Magazine and Interiors, publication in Professional Lighting Design, and I was asked to present the topic to the European Lighting Design Association in Milan.
Kevin Dowling realized the long-term value of the concepts and was a big supporter. After I graduated I offered him a design concept for an interactive tradeshow exhibition for CK’s light fixtures, which he pushed ahead with me. The display had recessed pockets sized for each product, internally lit around the perimeter with CK cove lighting, with the products mounted so they appeared to float in the middle. Kevin used industrial IR sensors to detect when someone reached for a product; this triggered a light show that focused on the product being touched and an LCD screen in the middle displayed corresponding product data.
Later, while I worked at Color Kinetics, I pitched the concept for a live interactive controller (as opposed to a pre-scripted setup like most lighting controllers), but unfortunately it was never deemed a priority and never made it to the product roadmap. When Color Kinetics was bought by Philips in 2007, I soon had enough and quit. Ironically, about the same time, an internal Philips venture in Europe was launched called Philips Retail Solutions – focusing on interactive lighting and digital content control for retail applications. After 3-4 years of trying, the venture never made much money and was shuttered. They simply struggled to convince retailers of the potential of interactive presentations.
Part IX: Conclusion
There is obvious potential for interactive lighting, whether for retail, or entertainment, or for creating a more pleasant, tailored environment in general. There remains clear opportunity to fuse the digital world with the physical world, to treat the physical built environment as a portal to the virtual world. And now with all the recent technology developments like smartphones, coded light, etc. there are many additional opportunities to create personalized interactions with “smart” architectural environments. I’m hoping that some application will “catch” and deliver value to at least one market segment, driving further exploration and development in the area overall.