The Extension of Man – Hacking the Body

What is a prosthetic?  We have seen artificial limbs for amputees and the Cheetah (R) foot for track and field athletes. You can have your fingertips replaced if you accidentally slice them off while making dinner. Cancer survivors can have their breasts reconstructed if they’ve lost them in treatment.

But what if you wanted a super-human digit? Or what if you want to shave off minutes from your current marathon time?  Where is that fine line between restoring and enhancing?

As part of the TinkTank labs workshop series I helped lead participants through a 4-hour workshop that highlighted how artists and scientists are currently hacking the body and discussed the socio-technical, cultural and ethical issues we will be facing as these hacks become widely possible, prevalent and undetectable.

Hacking the Body Workshop

As part of the workshop we engaged participants in building physical prototypes of their own prosthetic designs using construction materials, microcontrollers, electronics, and Arduino code. We wanted participants to focus on the design and critical theory behind the changing landscape of prosthetics and human augmentation (rather than messing around with and debugging code). To this end I develop several flexible and reusable snippets of Arduino code that participants could use and combine to get their prosthetics working with minimal coding knowledge (the code can be found here).

Overall, the workshop was a great success, with participants trying out and refining designs, and talking and thinking critically about the future of prosthetics, human augmentation, and how these are changing how we connect and relate to the world around us.

Click here to see some of the workshop slides

Hunger Games

Hunger Games is a learning environment designed to support upper elementary learners’ construction of understandings of animal foraging behaviors. It is an educational research project lead by Tom Moher’s research group at the University or Illinois Chicago (UIC). In Hunger Games, learners enact animal foraging within the context of a sequence of increasingly complex simulated scenarios involving varying conditions of competition, resource depletion, sociality, and predation. The instructional unit is designed to develop understandings of the factors that foraging animals use to guide their decisions in selecting food patches, as well as the ways in which populations of animals distribute themselves (e.g., resource matching and ideal free distributions) among available resources. The record of students’ (individual and aggregate) behaviors during enactment of the foraging simulations serves as the object of inquiry for reflective activities.


In Hunger Games, the classroom is “transformed” into a natural habitat in which students embody the role of squirrels foraging for food. This was inspired by a longstanding practitioner tradition of using embodied activities with physical materials (e.g., chickpeas, M&Ms) to introduce foraging concepts, and the feeling that an embodied approach had several potential advantages over a distributed screen-based approach (Moher et al., submitted).

In Hunger Games, each student in the classroom is provided with a small stuffed animal (“squirrel”) that serves as his or her “avatar” during the activity. Students forage by physically moving their squirrels among a set of “food patches” of varying quality distributed around the classroom, gaining energy as a function of the elapsed time in the patch, patch quality, and competition within the patch (i.e., the presence of other squirrels). Avatars may also fall victim to predation (signaled on smaller displays adjacent to each food patch). Avatars who are “caught” are considered “injured,” and given a short “time out” period in which their squirrel cannot gain calories even if located in a patch; this allows for the introduction of concepts of predation without forcing children out of the game prematurely.


Working closely with the UIC research lab we developed a community knowledge-building application to support individual student, small group, and whole class reflection and discourse. Within the application, students are provided with representations of both individual and aggregate data reflecting their performance during foraging bouts. At the aggregate level, students have access to an interactive version of the Harvest Graph (see image on right) that allows them to sort the distribution of individual caloric gains according to various factors including patch quality, competition strategies, and frequency of moves. At an individual level, students are provided with a “Move Tracker” (see image on left) that enables them to replay the step-by-step patch moves that they made during game play; this tool is used to support reflection on the effectiveness of their moves and to prepare students for subsequent foraging bouts. Finally, the application provides a threaded discussion tool to support development of community knowledge guided by a series of embedded inquiry prompts.

Hunger Games was successfully enacted in three grade 5 classrooms as part of a four-week curricula . Analysis of the findings is currently being conducted and will be reported on in future publications.

RPA – Developing a Framework for Tangible and Embodied Interactions

RPA – Developing a Framework for Tangible and Embodied Interactions

Rock, Paper, Awesome!

Rock, Paper, Awesome (RPA) was Encore Lab’s initial foray into developing the means for tangible and embodied interactions that would connect to our S3 technology framework. The goal for RPA was simple; individual labs could create their own unique tangible or embodied interactions through which they played rock, paper, scissors with other labs that were physically distributed around the world.

The Theory

We chose rock, paper, and scissors as our test-bed because it provided us with not only a well-defined set of semantics (i.e., win, lose, draw, player ready, player gone), but also a very loose coupling in how we enacted those semantics. For instance how a player chose “rock” in one space could be entirely different from how a lab chose it in another (e.g., standing in a particular spot in the room, versus pushing a button). This allowed us to think deeply about what it meant to convey the same message through various tangible and embodied interactions, and to begin building an understanding of how these different interactions affected the meaning making of the participants. In essence we built a “reverse tower of babel” where multiple languages could all be interpreted through S3, allowing recipients at both ends to effectively communicate through their own designs.

Screen Shot 2013-12-02 at 10.14.47 PM

In this way, RPA is more than just a game of rock, paper, scissors – it is an avenue for us to begin investigating novel ways for users to interact with the world, and for connecting these investigations within a broader knowledge community. We aim to not only connect these communities, but also to add a layer of user-contributed design to their interactions, where community members engage in creative fabrication and exchange of tangible, interactive media that reflect their ideas, workflow or presence, bridging the distances and connecting the community.

Three critical questions guided our development of RPA and this component of S3 in general:

  • How can we bring distributed communities together through tangible and embodied interactions?
  • What are the possible roles for tangible and physical computing, and ambient or interactive media that are deeply connected to the semantics, workflow, physical presence, ideas, activities, and interests of the distributed communities?
  • How does the temporality of the interactions (synchronous versus asynchronous) determine the selection of appropriate kinds of interactions and representations?

We are currently sending out kits, first versions of the code, and design documents to labs at the Learning Technologies Group at the University of Chicago, and Intermedia at the University of Oslo. We are excited to see how they will develop and contribute new interactive designs that represent their own representations of space and meaning within the game.

The Technology

The physical interactions and ambient feedback is handled by an Arduino microcontroller. The Arduino allows users to develop a wide array of inputs (e.g. proximity, light, and sound sensors, buttons and levers), and outputs (e.g. sound, light, movement). Using the S3 framework, RPA facilitates different game “events” (e.g., joining the game, choosing Rock) by sending messages over an XMPP chatroom (conference). We originally attempted to implement these messages over the XMPP server only using the Arduino  – however, given the relatively limited amount of RAM on the Arduino board (2KB) this turned out to be overly restrictive and we started looking at other solutions.

As a solution to this issue, we made a simplified set of event messages (i.e., single text characters) that were sent over the Arduino’s serial port to a connected computer. For testing purposes we used a laptop. However, in permanent installations, we envision RPA having a more compact and flexible setup. In order to achieve this, we connected the Arduino board to a Raspberry Pi. The benefits of the Raspberry Pi is that it is small and cheap, allowing us to dedicate a Pi for each game installation, and to have the “brains” of RPA be as unobtrusive as possible.

In order to connect the various RPA installations we use node.js as an intermediary between the XMPP chatroom and RaspberryPI. Messages that are posted to the XMPP chatroom are picked up by the node.js server and sent over serial port to the Arduino, which then executes the user-designed action, such as turning on a light or playing a chime. Respectively, any event trigger on the Arduino (e.g. a button is pressed), is sent over the serial port to node.js and translated into a XMPP message.

Sample Arduino code for RPA and the node.js setup code can all be freely downloaded, tinkered with and customized from github.

The Run

We set up two “stations” at OISE, one on the third floor and one on the 11th floor. Players challenged each other to a game of rock, paper, scissors (see the video below).

Each location had different tangible, audible, and visual inputs and outputs providing players unique multi-modal experiences that conveyed the same message. At the third floor location, a “servo motor” swung a dial to let the player know a challenger was waiting to play. At the eleventh floor location, an LED flashed to convey the challenge. We have tested other designs (not shown here) that used proximity sensors to detect where players were within a room, using their location to trigger an event (such as choosing rock). In another instance, a light sensor conveyed one player’s availability to other players (in remote locations) when the lights in the original player’s room were on.

Going Live! RPA at TEI 2013

We submitted RPA to TEI 2013′s student design challenge. The conference was held in Barcelona Spain and provided an ideal opportunity for us to try out RPA (and S3) in a live setting with users who had no experience with it. We had stations running at the site and at labs site running in Toronto allowing us to observe a wide range of interactions and gain feedback from participants. We also added a new layer to RPA which connected a real-time visualization of win/lose/draw results to the game – although this visualization duplicated some of the functionality of the tangible RPA elements it did represent a significant step in merging the tangible elements of S3 with a key element of the existing architecture.


PLACE.Web – Orchestrating Smart Classroom Knowledge Communities

PLACE.Web – Orchestrating Smart Classroom Knowledge Communities

PLACE.web (Physics Learning Across Contexts and Environments) is a 13-week high school physics curriculum in which students capture examples of physics in the world around them (through pictures, videos, or open narratives), which they then explain, tag, and upload to a shared social space. Within this knowledge community, peers are free to respond, debate, and vote on the ideas presented within the examples towards gaining consensus about the phenomena being shown, empowering students to drive their own learning and sense making. We also developed a visualization of student work that represented student ideas as a complex interconnected web of social and semantic relations, allowing students to filter the information to match their own interests and learning needs, and a teacher portal for authoring tasks (such as multiple choice homework) and reviewing and assessing individual student work. Driven by the KCI Model the goal of PLACE.Web was to create an environment in which the class’ collective knowledge base was ubiquitously accessible – allowing students to engage with the ideas of their peers spontaneously and across multiple contexts (at home, on the street, in class, in a smart classroom). The PLACE.web curriculum culminated in a 1-week smart classroom activity (described in depth below).

To leverage this student contributed content towards productive opportunities for learning, we developed several micro-scripts that focused student interactions, and facilitated collaborative knowledge construction:

  • Develop-Connect-Explain: A student captures an example of physics in the real world (Develop), tags the example with principles (Connect), and provides a rationale for why the tag applies to the example (Explain).
  • Read-Vote-Connect-Critique: A student reads a peers’ published artifact (Read), votes on the tags (Vote), adds any new tags they feel apply (Connect), and adds their own critique to the collective knowledge artifact (Critique).
  • Revisit-Revise-Vote: A student revisits one of their earlier contributions (Revisit), revises their own thinking and adds their new understanding to the knowledge base (Revise), and votes on ideas and principles that helped in generating their new understanding (Vote).
  • Group-Collective-Negotiate-Develop-Explain: Students are grouped based on their “principle expertise” during the year (Group), browse the visualization to find artifacts in the knowledge base that match their expertise (Collective), negotiate which examples to inform their design of a challenge problem (Negotiate), create the problem (Develop), and finally explains how their principles are reflected in the problem (Explain).

Over the twelve weeks 179 student examples were created with 635 contributed discussion notes, 1066 tags attached, and 2641 votes cast.

Culminating Smart Classroom Activity

The curriculum culminated in a one-week activity where students solved ill-structured physics problems based on excerpts from Hollywood films. The script for this activity consisted of three phases: (1) at home solving and tagging of physics problems; (2) in-class sorting and consensus; and (3) smart classroom activity.

PLACE Culminating Script
PLACE.web Culminating Script (click to enlarge)

In the smart classroom, students were heavily scripted and scaffolded to solve a series of ill-structured physics problems using Hollywood movie clips as the domain for their investigations (i.e., could IronMan Survive a shown fall). Four videos were presented to the students, with the room physically mapped into quadrants (one for each video). The activity was broken up into four different steps: (1) Principle Tagging; (2) Principle Negotiation and Problem Assignment; (3) Equation Assignment, and Assumption and Variable Development; and (4) Solving and Recording (Figure 3).

PLACE smart classroom imagesAt the beginning of Step 1, each student was given his or her own Android tablet, which 
displayed the same subset of principles assigned from the homework activity. Students freely chose a video location in the room and watched a Hollywood video clip, “flinging” (physically “swiping” from the tablet) any of their assigned principles “onto” the video wall that they felt were illustrated or embodied in that clip. They all did this four times, thus adding their tags to all four videos.

In Step 2, students were assigned to one video (a role for the S3 agents, using their tagging activity as a basis for sorting), and tasked with coming to a consensus (i.e., a “consensus script”) concerning all the tags that had been flung onto their video in Step 1 – using the large format displays. Each group was then given a set of problems, drawn from the pool of problems that were tagged during the in-class activity (selected by an S3 agent, according to the tags that group had settled on – i.e., this was only “knowable” to the agents in real-time). The group’s task was to select from that set of problems any that might “help in solving the video clip problem.”

In Step 3, students were again sorted and tasked with collaboratively selecting equations (connected to the problems chosen in Step 2), for approaching and solving the problem, and developing a set of assumptions and variables to “fill in the gaps”. Finally in Step 4, students actually “solved” the problem, using the scaffolds developed by groups who had worked on their video in the preceding steps, and recording their answer using one of the tablets’ video camera – which was then uploaded.

Orchestrating Real-Time Enactment
PLACEweb Students At Board

Several key features (as part of the larger S3 framework) were developed in order to support the orchestration of the live smart classroom activity – below I describe each and their implementation within the PLACE.web culminating activity:

Ambient Feedback: A large Smartboard screen at the front of the room (i.e, not one of the 4 Hollywood video stations) provided a persistent, passive representation of the state of individual, small group, and whole class progression through each step of the smart classroom activity. This display showed and dynamically updated all student location assignments within the room, and tracked the timing of each activity, using three color codes (a large color band around the whole board that reflected how much time was remaining): “green” (plenty of time remaining), “yellow” (try to finish up soon), and “red” (you should be finished now)

Scaffolded Inquiry Tools and Materials: In order for students to effectively engage in the activity and with peers, there is a need for specific scaffolding tools and interfaces through which students interact, build consensus, and generate ideas as a knowledge community (i.e., personal tablets, interactive whiteboards). Two main tools were provided to students, depending on their place in the script: individual tablets connected to their S3 user accounts; and four large format interactive displays that situated the context (i.e., the Hollywood video), providing location specific aggregates of student work, and served as the primary interface for collaborative negotiation

Real-Time Data Mining and Intelligent Agency:To orchestrate the complex flow of materials and students within the room, a set of intelligent agents were developed. The agents, programmed as active software routines, responded to emergent patterns in the data, making orchestration decisions “on-the-fly,” and providing teachers and students with timely information. Three agents in particular were developed: (1) The Sorting agent sorted students into groups and assigned room locations. The sorting was based on emergent patterns during enactment (2) The Consensus Agent monitored groups requiring consensus to be achieved among members before progression to the next step; (3) The Bucket Agent coordinated the distribution of materials to ensure all members of a group received an equal but unique set of materials (i.e., problems and equations in Steps 2 & 3).

Locational and Physical Dependencies: Specific inquiry objects and materials could be mapped to the physical space itself (i.e., where different locations could have context specific materials, simulations, or interactions), allowing for unique but interconnected interactions within the smart classroom. Students “logged into” one of four spaces in our room (one for each video), and their actions, such as “flinging” a tag, appeared on that location’s collaborative display. Students’ location within the room also influenced the materials that were sent to their tablet. In Step 2, students were provided with physics problems based on the tags that had been assigned to their video wall, and in Step 3 they were provided with equations based on their consensus about problems in Step 2.

Teacher Orchestration: The teacher plays a vital role in the enactment of such a complex curriculum. Thus, it is critical to provide him or her with timely information and tools with which to understand the state of the class and properly control the progression of the script. We provided the teacher with an “orchestration tablet” that updated him in real-time on individual groups’ progress within each activity. Using his tablet, the teacher also controlled when students were re-sorted – i.e., when the script moved on to the next step. During Step 3, the teacher was alerted on his tablet whenever the students in a group had submitted their work (variables and assumptions)

Non-Standard Bodies

Non-Standard Bodies

Non-Standard Bodies Dress Front

This project was part of a 4 month exhibition at the Ontario Science Center’s !dea Gallery as part of Mirror, Mirror…Reflections on Body Image

We are constantly assailed with body images and standards in which we as individuals and as a society must fit – are you a small, a medium, a large? How short is your dress? What does your neckline say about you? These questions are at the heart of Non-Standard Bodies. The project, initially conceived as a way of exploring the influence of external factors on the comfort and presentation of the individual, has come to tell a story about the impact of standards on our daily lives and the impact of remote decisions on our perception and presentation of self.

Except in the case of bespoke clothing, the garments we wear are traditionally based on standardized sizes, sometimes reflecting national or international decisions, sometimes reflecting the decisions of individual clothing designers and manufacturers. These garments cannot, by necessity, be a perfect fit for each wearer. They are, instead, good enough, aiming to reflect a reasonably popular or common set of measurements.

This idea of garment sizes being imposed from the outside, by invisible hands, is one persistent with the principles of standardization. Standards setting has, historically, been the province of experts, educated individuals situated within official organizations. These twin ideas of outside influences and “one size fits none” standards are the themes running through Non-Standard Bodies. The project, from an abstract viewpoint, is a physical manifestation of the invisible hands of standardization making decisions about the appearance, presentation and bodyimage of its wearer. Practically speaking, Non-Standard Bodies is an adjustable dress. It is, in its ground state, large and voluminous, beige cotton cloth, fashioned after a monk’s habit and worn over a structural plastic frame.

The dress has both a wearer and a user. The two functions, unlike with normal clothing, are distinct. The user becomes the subject, the wearer the object. This metaphorical representation of standards setting (with the user as subject at a distance) takes place through the manipulation of the fit of the dress. The fit of the dress is manipulated through the adjustment of a series of controls, arrayed along its spine (and therefor inaccessible by the wearer). These controls provide input to an Arduino microcontroller, which manipulates a number of motors. Those motors wind up spools of cord, which lift the hem of the dress, shorten its sleeves and adjust the fit of its waist. Thus, through activities unseen by the wearer (because, in fact, the user is behind her), her appearance and presentation of self are changed. This is the metaphorical representation which runs through the heart of the work and presents, in an evocative and whimsical way, the issue of the politics of standardized clothing sizes.

Radical Design Workshop

Radical Design Workshop

This workshop was designed as part of the Knowledge Media Design Institute‘s (KMDI) Radical Design Series entitled: Malleable Designs – Using Play-Doh to Design the Future.

The goal of the workshop was to engage the participants in thinking about how the future of communicative technologies will provide new ways for people to connect, share, and grow in their communities. Participants used Play-Doh as a medium to articulate and develop their ideas both tactilely and visually. Play-Doh was especially effective for this as the medium itself is so malleable and flexible that the participants didn’t have to conform their thinking to rigid structural limitations.

The workshop combined open, guided discussion and hands-on investigations with the Play-Doh. As a group we looked at 4 main themes: 1) What does it mean to communicate? 2) How do we define our communication networks and communities? 3) What is a disruptive technology? And how are current disruptive technologies changing the way we connect and relate to each other? 4) What do we envision the future landscape of communicative and community technologies to be?



Developed primarily as a tool to engage participants at poster sessions at conferences, Roadshow provides participants and attendees a way to more collectively engage with the ideas and discussion generated during the session, and to generate more collaborative and connected ideas for follow-up discussion among the whole session.

As an extension to the Sail Smart Space (S3) Framework, Roadshow enables users to create a pop-up social collaboration space that can be indexed to physical locations, facilitating discussion, idea exchange, and the development of a shared taxonomy (tags) within an emergent knowledge base. All of these interactions are then broadcast on an interactive aggregated screen that shows all of the contributed work within the network and filter items by location, contribution type, and tags.

In essence Roadshow can let you quickly author and deploy and set of ad-hoc social networks that aggregate information both within their individual defined social spaces and across the spaces in a shared central pool.

When an individual creates an instance of Roadshow they have several options that they can customize to help guide the interactions. The author can define the number of locations that exist within the network, give them each unique names (e.g., “Poster 1” or “Collaborative Tablet Applications Talk”); define the types of

contributions that participants can make when contributing (e.g., “Question”, “Comment” or “Critique”); and pre-seed tags that may help focus participants thinking (e.g., “Collaboration”, “Learning Goals”, “HCI Considerations”, or “Key Points”).

Roadshow was designed to allow for maximum flexibility in regards to devices that could be used – using responsive web techniques we made it possible to use Roadshow on any mobile device (except Blackberry) or laptop. This mean that users weren’t confined to specific technologies, thus reducing barriers to participation.

Once logged in users could see the contributions of every other member within the network filterable by location. Users could then add their own contributions to the collective knowledge base. Each contribution was also tagged by the user – these tags were a combination of the pre-seeded tags described above and tags organically added by users. When users added their own tags to a post that tag was propagated to every other tablet in the space in real-time helping participants made new connections between their ideas and spaces (users who came later on would also see all the emergent tags).

Finally discussion could take place using the large aggregate display as a mediator and avenue for organizing and filetering ideas. Here users (or a central mediator) could drag the different contributions around the board making “collections” of ideas in order to find themes, topics of interest, or points of conflict. The final layout of the interactive board could be saved and recalled later for future discussion or reworking.

Although Roadshow is still very much in its infancy there are several avenues that we are exploring for future iterations. Primary amongst these are creating more dynamic interactions patterns between individual contributors and their both immediate social spaces (individual locations) and the broader network (whole room), thinking about how we can get individual from one space to connect and build on the ideas of participants in others to get a greater sense of how their ideas connect and contrast towards building new opportunities for knowledge construction. Additionally, similar to other work with S3, we want to think about how the inclusion of Ambient technologies can give participants a greater sense of community belonging, and feelings of spatial relevance and embodiment; and how the inclusion of intelligent software agents can help in the spatial coordination of people in these spaces and in facilitating the productive interaction patterns described above.



There have been ongoing discussions amongst educational researchers concerning how teachers can support students in making connections between mathematics topics (The National Council of Teachers of Mathematics, 2000). Conventional instruction, with its a sequential presentation of materials in textbooks and the rote completion of problem sets, often fails to help students develop a deep understanding. This is particularly true in regard to the interconnections amongst mathematical concepts, which often come across to students as completely separate topics (Hiebert, 1984).

MathRepoVizIn response, working closely with a school math teacher, we co-designed (Penuel, et al., 2007) a curriculum to engage several small groups of students working in parallel as they “tagged” a common set of math problems. In so doing, a collaborative visualization emerged as the curriculum synthesized the combined tags from all groups. A set of thirty problems developed by the teacher belonged to one or more of four category groups: Algebra & Polynomials, Functions & Relations, Trigonometry, and Graphing Functions. The basic goal of this activity was to help students understand the relationships between these four aspects of mathematics by having them visualize the association of math problems with multiple categories.

Within our S3 classroom, students were automatically grouped and placed at one of the room’s visualization displays, and usinglaptops were asked to “tag” (label) a total of 30 questions. Each group’s display showed a graphical visualization of their collective responses. Students were then asked to collaboratively solve their tagged questions and vote and comment on the validity of other groups’ tags. A central display showed a larger real-time aggregate of the all groups’ tags as a collective association of links. As students voted on these tags, agreements resulted in thicker link lines than those that fostered disagreement.

Preliminary findings, while representing only a small number of participants, showed an upward trend of increasing accuracy and structuredness for the experimental condition. The improved accuracy from the pre-test to the curriculum activity and post-test suggests the importance of how we ask students to make connections to problems, with greater accuracy derived from a collaborative design which shares responsibility. The structuredness, which measured students’ recognition of the connections, shows increasing willingness to characterize math problems from different perspectives.

Overall, students found the visualizations useful in showing different mathematical themes from which a problem could be approached. One student indicated that the visualization was helpful when he could not solve a problem. Students also stated that, over time and with more contributors, the system would become increasingly valuable for studying purposes.

Students also commented that they became more cognizant of the connections amongst mathematics ideas and themes. It is noteworthy that students gained awareness that one could discuss properties of math problems and their relevant themes rather than simply answer them.

Watch the video below: