Takao Umehara
Takao Umehara
product design director / creative strategist / design thinking coach

Immersive Learning Experience Design with AR

Perception Academy Quest

Color-chip-blue 2.png

PROJECT TYPE

PLATFORM

DATE

ROLE

Learning Product for Middle School Students

Web Based Application

Last updated Nov 2017

Creative lead, contributed building product strategy, lead entire creative effort from product branding, ux design, and final interface design.

DESCRIPTION

The Perception Academy Quest takes students through a series of linked activities that focus on brain disorders and how they affect what we perceive and how we respond to the world around us. Students move through the periods of a school day as though they have one of the perception disorders detailed in the Oliver Sacks book The Man Who Mistook His Wife for a Hat. Students master excerpts from this difficult non-fiction text, building on their close reading and detailed writing skills.

Introduction

Oliver Sacks

Project started when a group of designers, software engineers, project managers, and playwrights were gathered in one team. We called it “Quest Team'“. There was virtually no direction given to this team. We were informed

The challenge : “Make something very unique out of this book.” and we had complete autonomy to set our product vision and experience strategy.

That was it. It was a blue-sky project. This is the challenge we were given to this team.


RESEARCH

Each person in our team have read the text material (book) and start imagining what kind of learning experience we would design for middle school children.

We also conducted interview sessions with middle schoolers to learn about their interest, aspirations, studies and struggles. It became clear that traditional lessons of reading and writing has not been inspiring for the learners.

I co-crafted research questions to learn about students and conducted interviews.

InterviewwithKids.png

Through research, we surfaced one big question to tackle.

How might we enable students to gain deep reading experience in fun and engaging way?


 

Another big finding we discovered is: NO WIFI CONNECTIVITY. There was no reliable network available, and we are forced to build our product without relying on connectivity.


Synthesize and brainstorm in the co-creation workshop

Designers, software engineers, playwrights and education experts all gathered and collaborated to build, share ideas for possibility of new learning experience. I facilitated several co-creation workshops for the team.

Through this workshop, we came up strategy together.

What if we maximize engagement using different types of senses and activity to create rich in-class inquiry based learning experience?


DEFINING EXPERIENCE STRATEGY

After the workshop, I contributed solidifying the experience design strategy as a direction to build our learning experience for this project.



design sprint: Visual EXPLORATION

Design team worked on developing ideas through visual research, sketch, motion sketch. Through these efforts we push our product concept.

I designed and built these simple motion graphics.

neuro-visualsketch.png

DESIGN SPRINT: AR EXPERIMENT

While software engineering team is also doing on their TECH SPRINT to explore implementable technology ideas, design team also spend time exploring desired tech solution. The team decided to build AR experience to depict different brain disorder symptom for the role-play experience. We build movie for AR concept building and also used LAYAR system to study how AR would work in the learning experience.

I did food photography, implemented AR experiments with LAYAR, art directed a 3D designer for final 3D animation production, worked side by side with software engineers to implement the AR as part of the app.


wireframing / prototyping

We gathered as a whole team and I lead a sketching session to explore possible experience flow and then translated into rough digital sketch. Then we used the sketch to develop low fidelity wireframe to start testing out with internal staff then actual students.


INFORMATION ARCHITECTURE

As we continue to build the digital experience, I lead designing the information architecture.


LEARNING EXPERIENCE DESIGN: quest method cards

As the learning experience gets dense enough to explain easily with stakeholders, I proposed and implemented a Quest Method Card, which enables to stripped down the content layer and show the activity layer of the learning experience. This set of cards became highly useful not only to explain new teammates, internal stakeholders, and teachers, we also utilized to design another Quest products.


USER INTERFACE DESIGN: phase 1

After testing few digital prototype with wireframes, we started to solidify the learning experience. It was the time to develop the user interface that is useful and delightful by the middle schoolers. This is one of the version that I designed for prototype version 4.


AR Development

I art directed and collaborated with 3D designer to implement AR breakfast experience.


USER INTERFACE DESIGN: phase 2

After several user testing, we found that interaction is too complicated and was not achieving best user experience, we changed direction and I proposed and implemented much simpler design.

The color coding per character that each student would play a role made so much more sense and it became very clear from the far distance which character students are playing in the classroom. This add another layer of benefit for teachers.

Through usability test, we proved this UI brings much better experience and leas confusion while navigating within the app.


Interaction Design Flow

This is the final interaction flow that I designed. We made interaction simple and identical for all students regardless of which role they are playing. Only difference they experience are in contents such as audio, AR, texts (story), challenges for debate, challenges for the drama class, and picture for final assembly.

Perception Academy_Interaction Flow.png

Final OUTCOME 

We successfully launched the app to the Amplify customers. This Perception Academy Quest is consist of 6 experience.


PART 1: AUDIO EXPERIENCE

The Quest is structured like a medical thriller: On an ordinary morning, students are involved in a minor school bus fender bender on the way to school. They experience the crash via audio recording though their headphones. While “walking” the rest of the way to school, they hear narration detailing the eerie “symptom” of altered perception—something is clearly amiss, but students don’t yet understand what it is.

01.png

PART 2: BREAKFAST (AUGMENTED REALITY THREE-DIMENSIONAL SIMULATION)

At school, augmented reality simulation distorts students’ breakfast on their tablet, offering more evidence of the mind-bending changes going on inside them.

02.png

PART 3: ENGLISH LANGUAGE ARTS (ELA)

In ELA, they read case studies in Sacks’s book, which detail the powerful forces that have commandeered their perception—they are suffering from neurological disorders, the result of brain trauma. 

03.png

PART 4: RECESS (DEBATE & DISCUSSION GAME)

During Recess, they play a card game that helps them confront their new limitations. One student picks a card, and determine YES or No whether he/she can perform that task on the card with restriction he/she has as a role. Then, remaining students would determine to AGREE or DISAGREE and they go through few rounds of debate/discussion.

04.png

PART 5: DRAMA (PRACTICING PROBLEM SOLVING SKILLS)

Next, students pair up to imagine how people with different disorders might solve problems together, and dramatize these interactions as a scene of dialogue.

05.png

PART 6: FINAL ASSEMBLY (COLLABORATE WRITING ASSIGNMENT)

The Quest concludes with a collaborative writing assignment in which students use their knowledge of all five disorders and create a scenario that explains the circumstances of that morning’s bus accident.

06.png

RESULT / LEARNING FROM THE PROJECt

This learning experience product leveraged the power of digital technology to provide teachers with a resource that they wouldn’t have the time, specialized knowledge, or technical know-how to create on their own. This app facilitates students tackle challenging texts by creating a prepackaged, collaborative learning experience that sharpens critical English Language Arts skills for middle schoolers.

With a challenge of no connectivity, we designed simple product that allows humans to play a vital role for interaction with other peers in the classroom.

The learning is that in the context of ‘immersive learning experience design’, what we want people to immerse is not an app. Instead of making students to focus on the screen, we needed to position the app as a catalyst to enhance human interaction. This brings high students engagement overall.

CREDIT

Company:
Amplify Education

Takao Umehara, Experience Design Strategy,  Lead Visual Design, UX/UI Design Direction
Jean Pierre Dillard:  Concept Development, Art Direction
Chris Kalb: Illustration, Concept Development
Chris Leathers: 3D Modeling
Anna Ziegler, Nicco Moretti: Content Authoring, Lesson Guide, Overall Concept
Nicole Stein, Dan Russo, Te Yi Liu, Eli Burmin, Beth Goebel : Engineering
Sarah Smirnoff: Project Management