Story Time

Gene Dreyband | Dennis Ellis Manali Gortekar | Jonathan Partlow

Project OverviewOur first path was to learn about our technology with which we have been given a task to design. We first learned through the exemplars mentioned in class during the October 21st Make Session. We then read, watched, and listened to examples from the Internet to gain more insights on how the camera functioned. We also asked more knowledge colleagues about certain functions and then getting into specifics of coding.

After getting a handle on the specifics of the camera and trying to focus on the unique capabilities, we started to think of different design domains. We first thought of music and using sound as starting off point. It was a struggle to learn how to problem frame and problem set, but not problem solve. We thought it would be best to keep music or sound as part of our concept but to move on to new areas for exploration.

We considered domains like the kitchen and cooking as well as physical therapy. Here we focused on discrete actions and how the camera could be utilized to improve the experience. This was a step closer, but we were still trying to problem solve rather than explore a new area. With the knowledge we had gained over the past weeks about the camera itself as well as different domains, we decided to choose some of the capabilities of the camera to explore and then shape a context around the design.

Voice recognition and sound output were selected as a focus of our design. We decided to reintroduce an earlier domain, that of turning a child’s room into a jungle, or some other environment. Immediately the idea of storytelling came up in a conversation and we began to iterate on the idea of trigger words and images, sounds, and gestures the could enhance the storytelling

experience. We also incorporated gestures into the storytelling. When teachers, parents, or other children tell stories, they act out the adventures of the story. We utilized the gesture imputs of the camera to produce sounds that reflected actions or environments in the story.

Make Session 10.28Our concept from the original In-tel make session needed rework-ing. We were able to accomplish a proof of concept, but we wanted to change our direction. Jeff Bardzell gave us some advice about trying to truly capture the unique capabilities of this technology with our design. We started to consider other areas for which we could design. It was a lesson to learn how to navigate the line between problem framing, problem setting, and problem solv-ing. For this project we needed to avoid problem solving, concentrat-ing more on the other two aspects of design.

We explored different areas for a setting. We proposed working with sound and music, facial recognition for the seeing-impaired, and physi-cal rehabilitation. Physical rehabil-itation sounded the most intriguing space at the time, so we started iterating on what type of feedback could be utilized by the camera. We

learned that we were too focused on a solution, and not focused enough on exploring the capabilities of the camera to the greatest extent.

A - Team members discuss directions reflecting ideas generated during the make session.

B - Other teams critique and consider our directions and give feedback as how to continue.

C - Other teams critique and consider our directions and give feedback as how to continue.

(A) (B)

(C)

Affinity Diagram InsightsSeveral areas arose from this that intrigued us most.

Empowerment and removing barriers came up regularly from the discussions that we had had about rehabilitation, in the sense that we wondered how we could use the camera to augment someone’s senses. Could the functions of the camera be helpful for those with mobility issues, visual or hearing impairment? Could it be useful for the colorblind?

Creating wonder was a space we thought was also exciting, and we came up with ideas such as turning a child’s bedroom into a fantasy world, using the camera to perform magic tricks, or just using the device for undirected play. Rather than trying to solve a problem, play could be used as an interesting way to explore what the device could do, rather than creating a solution and expecting the camera to conform.

-Empowerment/Removing Barriers-Ease of Access to Information-Sensory-Creating Wonder

-Connecting People-Bridging Digital/Physical-Distributed Cognition-Sensing Emotion

Why can’t I just...-Grab something from one screen and put it onto another

-Tell my computer to turn off the lights

-Use simple thumbs up/down gestures to vote for songs on Pandora/Spotify

-Use voice commands to look for a specific file

-Delete things with a unique gesture like squishing my hands together

-Record and “play back” gestures that I just performed

-Make drawable shapes with my hands and use

them in existing art programs instead of using a tablet or a stylus

-Detect user facial expressions and cheer them up if they’re sad-Skip through images and presentations using a finger swipe gesture in the air

-Tell the computer to start recording video and audio on command and do simple edits on the recorded footage

-Push and pull objects in real space and reflect their real depth on camera

-Look out the window and use my computer

-Workout while using an interface

-Draw on my computer like I can in my sketchbook.

-Use public media players rather than having to buy my own.

-Control the lights in my house from anywhere

-Do more in my car than I currently can

-Change the pitch of my drums digitally without losing any of the acoustic properties

-Listen to music in my own space without earphones

-More passively digest newsConduct a digital orchestra

-Be more grateful

-Make people think differently

-Let people find the “new” by exploration

-See the world from a different perspective

-Help people understand that we are all part of one system

-Motivate others to take action rather than just talk about it-Remove barriers to access

Why can’t I just...-Re-ignite / ignite a new spark

-Make people thirsty for knowledge

-Give people a tool to impact lives

-Create new awareness

-feel data

-get sensory feedback in the form of touch

-smell data

-get sensory feedback in the form of taste

-not have people getting stuck in their selfish lit-tle bubble of augmented reality

-come across meaningful information that makes sense

-invent something that helps me document my thinking process

-physically move things as I am thinking

-have a system that could tell me where I could have been produc-tive

-travel places around the world virtually if I can’t go there physically

-edits on the recorded footage

Wouldn’t it be cool if I could...-Do magic tricks without any props

-Make hand shadow puppets that “talk” to each other

-Put together blocks and record the instructions

-Recreate the lightning scene with Elmer Fudd from the “What’s Opera Doc” cartoon

-Have a series of mov-able blocks that corre-spond to things in the real world like lights, TV, or appliances

-Change the camera color scheme with my fingers (b/w, color, depth, etc)

-Play catch with a digital ball with my computer

-Connect multiple cameras and take 3D photos of objects and convert them to models

-Make a rainbow in between my hands

-Detect different objects like using a blue vs. a green pen or detect all of the different sides of an object

-Video chat with someone at the front door and unlock the door remotely so they can come in if you’re not able to do so (like working in the back yard)

-Just utter the title of a movie and it shows up on your TV screen.

-Never have to adjust the thermostat again.

-Never have to use a cookbook again, just had the recipes in your kitchen

-Read a book on any surface

-Have more private interactions in a public space-Have more public interactions I a private space

-Turn your child’s room into a jungle

-Walk down the same

hallway a different way each time

-Have friends experience your travels in more immersive way than through photographs

-Talk to anyone, regardless of language

-Smile at a blind person and have them know it

-I could if the ability to give was always available

-I knew someone was hurting inside and could reach out to them

-I could pay it forward from anywhere at anytime

Wouldn’t it be cool if I could...-I could practice random acts of kindness in the world from my living room

-I could be in touch but out of reach

-I had time to do what I wanted to do as well as needed to do

-I could rearrange my house without actually touching anything

-I could hug my kids while I am away

-I could design in 3d without manipulating a ton of buttons and controls

-I could get these thoughts out of my head

and into the world

-consume data in literal sense

-give me new/ fresh data that smells nice and raw/outdated data that smells like garbage

-give me feedback by hugging/ shaking hands/patting my back

-touch it and play with it as I am getting informa-tion

-socialize with people in realbalance physical and mental activity together while working

-get information I need instead of displaying

facts

-get a readymade doc-ument/video/image for reflection

-simulate what would be like if I had been to plac-es like Paris, Slovenia

-make sure if I am living a good life.

Brainstorming body mechanics-Eating a chicken wing-Picking up a box-Rolling a barrel-Putting a beverage up to your mouth-Smoking a rolled cigarette-Eating a brownie-Shopping for twinkies bending and putting in cart and then removing and eating...-Stretching your arms-Praying-Lifting weights-Tuning the strings on a guitar-Sanding a piece of wood-Stroking a piece of wood-Displaying your o-face-Turning off an alarm clock-Looking at your phone/watch-Holding onto the headboard -Playing a drumset -Putting on a new drumhead-Eating corn on the cob-Putting a vinyl record on the turntable-Rolling a cigarette-Getting freaky-Hugging

-Playing Guitar-Paddling (Both versions - in water and out of… ;-)-Opening a bottle of wine-Washing dishes-Moving Furniture-Serving in Tennis-SkyDiving-Swimming - Freestyle, -Backstroke -Replacing a flat tire-Eating a chicken wrap -Pushing the walk-Crying-Rolling out in a laughter -Rolling tongue upside down-Striking a match-Sharpening a pencil-Rolling eyes-Driving a stick shift-Nodding -Typing on keyboard (two handed and one handedBoxing)-Carving a pumpkin-High-fiving a bro-Fixing hair while walking-Sneezing-Crowd surfing-Skipping-Blowing up a balloon

-Tying a balloon-Fire by friction-Applying ointment-Getting a shot-Taking a medication-Suppository-Bungee Jumping

Final List

1. Praying2. Sanding a piece of wood3.Putting on a new drumhead4.Hugging 5. Opening a bottle of wine6. Washing dishes7. Replacing a flat tire8.Fire by friction9. Getting a shot10. Putting a vinyl record on the turntable11. Crowd surfing

Body-storming activitiesWe came up with a large number of ideas for bodystorming activities, from the smallest activities like picking up a box, to extremely complex activities like starting a fire. The activities that we chose to actually body storm once we were finished coming up ideas were the following:

PrayingSanding a piece of woodPutting on a new drumheadHugging Opening a bottle of wineWashing dishesReplacing a flat tireFire by frictionGetting a shot

Praying - This was an interesting activity, because while praying often consists of slow, repetitive gestures which the camera is good at identifying, there were so many different types of stances and positions and activities that even something so “simple” had a lot of very complex gestures embedded in them.

Sanding a piece of wood - While sanding a piece of wood has relatively few activities like the sanding process itself and examining the wood, the camera was something that could be useful to identify the smoothness/roundedness of the wood, however it would require the camera to be positioned in such a way to pick it up. Certain angles would make this difficult.

Putting on a new drumhead - The activity of tuning a drumhead was helpful because it contained several things the camera could identify: key locations to hit the drum using the camera and using the microphone to pick up timbre/intonation.

Hugging - While hugging is a common activity, body language is a very subtle but key consideration that brings up a lot of questions. Is the hug between friends/family/strangers? What aspects make a hug appropriate or uncomfortable (length of time, familiarity with the person, appropriateness of the context)? Was it more joking (like in our own activities), or is the hug in earnest? What can we do with this kind of information?

Opening a bottle of wine - Opening a bottle of wine is tricky since different openers have different optimal positions. Some are so unwieldy that they need to be held between the legs, others it’s a matter of knowing how far to twist the screw, and what about removing the cork on champagne and worrying about hitting someone?

Washing dishes - Washing dishes is a repetitive activity, but it’s only so similar in the abstract sense. Small dishes require different motions to wash than pots and pans, or cups or utensils. Things with lots of food stuck on them require vigorous scrubbing that maybe a plate with crumbs would not. Even the difference between whether dishes would be washed and stuck on a drying rack versus washed to be placed in a dishwasher significantly changed the specific context.

Body-storming activitiesReplacing a flat tire - With its many steps and trouble spots, it would be easy to envision ways that technology could improve the tire changing process. From finding the right spot to place the jack, the properly tightening the lug nuts. We found that each activity in the process was relatively simple, but the emotional state of having a flat tire is probably worse than the reality so it was definitely an interesting activity to bodystorm.

Fire by friction - Starting a fire was such a complex process to explain and navigate that just the process alone took something like 5-10 minutes to even effectively pantomime, let alone the description of it to people that hadn’t done it before in the group. It was fun and involved, but ultimately way too sophisticated for the camera.

Getting a shot - One would think getting a shot would be simple, but there is the preparation of the syringe, preparing the spot in the arm, giving the shot and finally treating the area after the shot. Identifying a key area here might be how to help find a vein more effectively, but that is just one particular area.

The key takeaway from the bodystorming activity is just how involved almost any activity we engage in truly is. Those minor adjustments we do that are contextual are so subject to change, that it becomes difficult to make assumptions about what the person will do. With current technology it is difficult to create something that accounts for those use cases without forcing the person using them to conform to some pattern, rather than having the device adapt to their existing patterns.

Make Session 11.11During this Make session, we be-gan to focus on a particular space. We needed to scale down our ideas so that our design had a core idea from which we could build our new functionality for the camera. From the six design directions we found for the Make session we dissected and compared the different charac-teristics of the designs. The idea of adding ambiance to an environment stuck out to us.

We began with the idea of working the the realm of music reappropri-ation. We explored use of sound, movement, and dance. Yet, we weren’t able to identify how we would use the unique capabilties of the camers so we decided to explore other arenas. We started talking about the space of the kitchen and the many differen

(A) (B)

Make Session 11.18During this make session, we began to explore the different capabilities of the camera. We still needed to find a way to problem set, and we thought that focusing on the unique characteristics of the camera was the way to do it. The past 3 weeks of research gave us knowledge that we want to use, primarily that we wanted to use sound in our design. We were intrigued by the use of voice recognition and music output, as well as possible facial recogni-tion.

21 BalançoiresExemplar description:The musical swing interactive installation called ‘21 Balançoires,’ consists of a number of swings that both lit up and play a tune.Each swing plays a different pre-recorded sound, some being piano notes and sounds from a xylophone. When all 21 swings are in use and even greater song is created encouraging cooperation among people. This installation was put in Montreal, Quebec in the Quartier des Spectacles.

Design Direction: “Adult Playground Installation” - An installation of playful objects like swings, spun chair , see-saws, etc. for adults that will lit up and play a tune. Adults will be given a task of playing together a melody or completing a picture together by playing with installation objects. Like the exemplar, all the objects in installation will be associated with a particular sound note and a light color in order to incorporate audio and visual feedback. The installation intends to offer a fresh look on the idea of cooperation - the notion that together we can achieve more than alone and it is specifically directed to adults to encourage playful learning among them.

Inspiration:The idea of “playfulness” and “ co-operation” are inspirational elements of the exemplar implied to the “Adult playground installation” design. Playing is a psychological necessity and so it is not restricted only to children. We wanted to give an experience to adults where they could re-live their childhood by playing objects like swings and spun chair in public installations. We intend the individuals to un-mask their identity and escape from reality for a while in the world of playfulness and fun.

Context of Use:We see this design being used as public installation like Amusement park in cities. Adults could purchase tickets and enter the installation to join the fun activity. Through visual and audio feedback we expect adults becoming more engaged with the exhibit objects and also realizing the notion of “oneness” among people.

The Novel Experience:One of the important feature of this design is that it involves an entire body experience. All part of body are engaged while playing spun chairs . Also everyone’s minds get in sync to create a melody or picture. Thus, we intend to provide the “Whole body experience” in this design that mainly includes audio and visual feedback. Through perceptual computing technology, the playful experience could be enhanced and made more seamless.

Talk to Dead ArtistsExemplar Description : This interactive art exhibit allows visitors to have interactive chats with the works of art. Its is a novel way to give information about a classic painting to visitors where the painting could talk.For example a visitor can ask Mona Lisa why doesn’t she has eyebrows and have a conversation with her or listen to Jesus and his disciples in The Last Supper.

Design Direction:“Talking objects” - The everyday object like cell phones, books and computer talking for themselves. The basic idea is that of an ensoulment of a non-animated object. To de-familiarize a commonly used object and make people “think” about them. This critical design aims to encourage people to imagine a life without these objects and appreciate their presence. It could facilitate a conversation between an object and its owner so that the owner knows more about it.

Inspiration:The notion of “de-familiarization” and “collaborative learning” are inspirational elements of the exemplar implied to the “Talking objects” design. In the exemplar, people have conversation with an exhibit object and know more about its rich history. Through perceptual computing, we intend to bring similar learning experience by letting people know more about their everyday objects. The conversational way of learning can be very effective as it gives people right information at the time as and when needed. The everyday object could talk about its history

and designer and make people more aware of its rich background. One of our goals is to give information to the people in a way that makes sense to them and the “conversational” element of the exemplar seems to be a perfect design fit.

Context of Use:The “Talking object” is more of a critical design. Its only purpose is to make people think and appreciate the presence of everyday objects. Thus, we don’t have a specific usage of the design. We wanted it to be open ended, exploratory and playful to people.

The Novel Experience: “Talking objects” - Design that incorporates the notion of ensoulment as the objects speak for themselves in their usual environment. It is a critical design in daily lives of people. A normal object like computer could just start a friendly conversation with its user and ask him about his day. It could introduce itself by telling about its background in brief. The object could also respond depending on how its user is handling it by enhancing the tactual experience of the user.

Nervous StructureExemplar description: Nervous structure is an exciting design because it uses the body to create an experience that is many things: playful, delicate, collaborative, and easy to understand. The wave-like motion of the threads creates an easy sense of motion that translates nicely into full body motions.

Design Direction:“full body motion” - Using the full body as an interface as a way of getting people to be active. The goal of such an exhibit is to create an easy to understand interface that invites play and interaction. Creating an experience that is inviting and soothing. Semi-directed activity works well when cameras are not totally precise, as fluid-like motion can mask some of the imperfections of a time delay as well as issues with detection.

Inspiration: In the artists own words, it is a synthesis between the physical, digital, and viewer’s perception. The art resembles physical fabric (through using a moire pattern of the projection on the fixed grid), but the perception is that of a real flowing material. By creating a relatively simple design, people are able to understand almost immediately in an embodied way how to use the design, even if they don’t understand exactly how it works.

Context of use: Using our camera, people can interact with the threads in 3 dimensions, pushing them out and side to side. The soothing motion of the design is something that could be seen in a home as a way to encourage someone to get out of their seat for a few minutes just to move around, even aimlessly. In close quarters, it could also be used like a conventional blind to change colors or adjust lights in a room.

The Novel Experience: The novelty lies in the simplicity of the actions. While the “wow” factor of this type of design may be lower than others, it makes up for it in being an interaction that can remain fun and novel for long periods of time. Being able to express yourself freely in an undirected way could lead to some interesting interactions and gestures that people do regularly that make sense as a standard for perceptual computing.

Concept Map

Final ConceptDesign Rationale

Our design concept is based on creating a space for possibility. The overall concept is to aid in augmenting, expanding the sensory input / output and increasing engagement using the specificity of the Intel Technology.

For the proof of concept of this application of the design we placed it in a classroom setting with a storyteller / teacher as this happens millions of times a day in classrooms all over the world.

Specificity of the deviceVoice and Language Recognition

We utilized the ability of the voice and word recognition of the device as a tool to aid the storyteller while telling a simple childrens story. Typically, this would be done by holding a book and reading to the children, flipping the book around

or passing the book around to show the images to the children to assist in meaning making and understanding the relationships of Size, Color, Location, Proximity, etc.

By utilizing the capabilities of the device the speaker reads the story and the images are immediately triggered along with the additional sound appropriate sound. The rationale for this was the increased engagement and understanding (mapping) provided by engaging additional senses. We chose this structured storytelling approach as a starting point as it allows the audience to ‘discover’ what is happening on their own creating an additional level of learning :“the cause and effect relationship”. Although this highly valued learning concept may be valuable be incorporated into the story being

told it is not typically incorporated into the traditional storytelling process itself.

Using a “build upon prior learning” system we introduced the gestures and their functionality into the storytelling process. This introduction served multiple purposes. The audience had already figured out that the images and sounds were reacting to the audio storytelling and recognized the characters and concepts individually. (ie: Blue, Dog, In) At this point the children were already starting to tell the story out loud and anticipating what the next slide would say. They were saying it out loud before it appeared. By the storyteller not speaking the words aloud, but instead advancing to the next logical step of combining elements in the story by gesture the audience was ‘invited

Final Conceptto participate’ and also quickly associated and mapped the cause and effect of the gesture: moving the hand in for ‘in’ and out for ‘out.The other purpose of using this structure was introduce another way of storytelling and communication available - using body language and gestures rather than just words.

With the concepts of the design introduced and discovered the dynamic of the storytelling experience itself changed from a one way exchange, where the the storyteller simply reads and the audience listens; or not as typically happens with young children in this situation. The experience has now become a participatory session due the audiences ability to advance the story by using the words recognized by the Intel SDK, patterns, process and story logic introduced earlier. This participation is enabled by the

unique features and separation of controls by input type of the Intel camera. The storyteller in front of the camera has the ultimate control of the images, pace and additional sounds but the audience, by virtue of the word recognition, is able to also have limited control simply by speaking the words.

We found that once the concepts were introduced the audience wanted to engage and play with the system. Without technical knowledge or special skills the children can (and did) create their own stories either by triggering an image and / or sound and then building off of what they saw / heard, or by intentionally using words that they knew in advance would display / play what they wanted.

Target Audience

Storytelling is a formative part of the human experience, and one that colors each and every transition we go through. Particularly, we focus on primary school children aged 6-11. During this period, storytelling permeates education. Not only do children learn through stories, they learn how to tell stories and how to communicate their emotions, thoughts, and desires through storytelling. We hope to enhance this experience and thus inspire future generations to think, dream, make, and believe.

Expected Outcomes

Bringing the story out of the book is a primary goal for our design. We hope to engender a spirit of exploration and wonder with the effects our design creates. What will occur in the mind of a child when she hears a lion roar when she says ‘Lion!’? We expect for children to use our design in an

Final Conceptimprovisational way, crafting their own stories and adventures as they see fit.

- Increased understanding and engagement of the storyteller and the audience.- Additional sensory stimulation providing expanded association of concepts from different domains.- Introduction to other communication tools and methods beyond voice. - Convergence of meaning making and making meaning.- Changing the dynamic from a one way communication to a participatory event.- Children using the design as a tool to express and create their own stories. - The design speaks to the same issues that we had to deal with in this project: Understanding cause and effect but we want also want the participants to be able to think of “the possibilities” without

assesment.

Evaluation Approaches

Our design engenders imagination, storytelling, adventuring, and wonder. Of course, each of these is notoriously difficult to evaluate. After some time with our design, we could examine how children take to storytelling. Is their imagination different? We could observe how children create new stories our observe how they physically interact with the device. We could record the number and time value of each visit, and how that changes over time. We could make more subjection judgements about emotions and such, but measuring change is a primary objective.

- Direct Evaluation in Situ: Observation of participation and engagement in storytelling- Direct Evaluation Individual: Testing of progress in concept

understanding- Indirect effects: Tracking progress in areas outside of storytelling sessions that involve communication, creativity, imagination, cause and effect and logic. - Direct Evaluation: Interviews with participants and those within their ecosystem - Parents, Teachers.- Video and Photo Ethnographical Studies- Do the participants relate the cause and effect and still create their own original stories?

Final ConceptVideo: