The work on this page requires a little context. These projects were all made at Dynamicland. The space uses cameras, projectors, and a custom language to facilitate turning the environment (walls, tables, anything you can stick a piece of paper onto) into a programmable space. Really, though, the research is more focused on the space, and the people: what kinds of interactions can you have when computing is easy, accessible, and ubiquitous? It's about democratized control & access, through the lens of computation. For more detail I'd encourage you to read The "Next Big Thing" is a Room or Omar Rizwan's project-based explanation of the space. Omar's piece goes a little into Realtalk, the superset of Lua that Dynamicland is built with. It's a research lab started by Bret Victor, an intentional community, a gathering space, a learning space. Here are some things I've made there in the context of research, education, art, and exploration.
The video above demonstrates how the inherently spatial design of the programming system paired with live editing makes for a powerful combination. First, some context:
x, y coordinates.So, in the video linked above I used my nails as dot input to a program that can live edit the numbers in a Realtalk program (specifically, whichever number on the page of the program is closest to a given dot). With this powerful chain of simple tools in Realtalk I alter the color, radius & stroke width of the circle (the output of the patched program) using only my fingers.
The prototype I created takes a page it's pointing at & randomizes all the numbers in the program. I'm working toward being able to chain or show various branches of randomly chosen numbers for the program, with a hope this this could be:
Originally I was inspired to work with anaglyph projection in Dynamicland to get a slight 3D effect on the 2D programs. Then, I got distracted by a much more novel concept: can you use just the red & blue glasses to hide UI in Dynamicland? This is interesting because everything in Dynamicland is, on principle, public & thus shared information. This could be a model for a kind of opt-in privacy! First, I started by swapping the blue/red lenses in two pairs of glasses:
When you put on the single-channel color glasses the opposing color light effectively disappears:
Note that the necker cubes above disappear almost completely while the blue/red dots, because they're opaque objects & not just projected light, are less well hidden. Designing experiences for a game or private messaging application in this way could be an interesting way to keep all Dynamicland programs public by default but facilitate privacy & privileged information when necessary. Future work in this direction would include using gray-light frequency-shift keying (FSK) to track & encode messages using publicly available projectors. This would build on the technique outlined in Moveable Interactive Projected Displays Using Projector Based Tracking (Johnny C. Lee, et al.).
I wrapped up this experiment & left it to the community with this little illustration:
One of the most powerful consequences of Realtalk having first class concepts of space in the language is that a lot of things we'd normally have to simulate come "for free" in the physical world. A good example of this is rotation: instead of calculating translations to simulate rotation we can stick pages on turntables ("lazy susans") & rotate them for real. Ricardo Cabello used atan2() to calculate the rotation of the paper using a corner of the program's quadrilateral against the center. With this method we get dials that control everything from text spacing, to animations, to music. Some examples I've developed:
When broadcasting in practicedial, but the way it calculates its value is novel. It compares the relative length of the page with the width of the supporter (in general this is the larger surface that's being projected on, in the video it's a table, but can be a wall, the floor, etc.) Normally this length is constant because the paper is resting on the supporter surface (in fact, the project mapping depends on this fact, which is why you'll notice the labelled value (~1.3) appears to drift in the video above - we could calculate the drift based the amount the paper is raised), but when you raise the page off the table, assuming at least one corner of dots its still visible to the camera, this value changes. We can use this as a proxy value for height from the table & normalize this to produce a reliable height-based "dial" (more accurately called a slider perhaps?)All the other experiments above save the height-base dial, and indeed most programs in Dynamicland, are spatial but naturally limited to 2D because it's an easier starting point than 3D (nearly trivial projection mapping, easy spatial input tracking with an RGB camera, etc.) Along with Toby Schachman, one of the researchers, I've been exploring using ultra-high frequency (UHF) RFID to get low-cost 3D tracking to be accessible in Dynamicland. There are no visuals for now but we can get a stream of UHF RFID signals into Realtalk as JSON, the next natural step is to transform that into Lua tables with which to trigger/animate programs.