Immersive Learning Experience Design with AR
Perception Academy Quest
Learning Product for Middle School Students
Web Based Application
Last updated Nov 2017
Creative lead, contributed building product strategy, lead entire creative effort from product branding, ux design, and final interface design.
The Perception Academy Quest takes students through a series of linked activities that focus on brain disorders and how they affect what we perceive and how we respond to the world around us. Students move through the periods of a school day as though they have one of the perception disorders detailed in the Oliver Sacks book The Man Who Mistook His Wife for a Hat. Students master excerpts from this difficult non-fiction text, building on their close reading and detailed writing skills.
Project started when a group of designers, software engineers, project managers, and playwrights were gathered in one team. We called it “Quest Team'“. There was virtually no direction given to this team. We were informed
The challenge : “Make something very unique out of this book.” and we had complete autonomy to set our product vision and experience strategy.
That was it. It was a blue-sky project. This is the challenge we were given to this team.
Each person in our team have read the text material (book) and start imagining what kind of learning experience we would design for middle school children.
We also conducted interview sessions with middle schoolers to learn about their interest, aspirations, studies and struggles. It became clear that traditional lessons of reading and writing has not been inspiring for the learners.
I co-crafted research questions to learn about students and conducted interviews.
Through research, we surfaced one big question to tackle.
Designers, software engineers, playwrights and education experts all gathered and collaborated to build, share ideas for possibility of new learning experience. I facilitated several co-creation workshops for the team.
Through this workshop, we came up strategy together.
After the workshop, I contributed solidifying the experience design strategy as a direction to build our learning experience for this project.
Design team worked on developing ideas through visual research, sketch, motion sketch. Through these efforts we push our product concept.
I designed and built these simple motion graphics.
While software engineering team is also doing on their TECH SPRINT to explore implementable technology ideas, design team also spend time exploring desired tech solution. The team decided to build AR experience to depict different brain disorder symptom for the role-play experience. We build movie for AR concept building and also used LAYAR system to study how AR would work in the learning experience.
I did food photography, implemented AR experiments with LAYAR, art directed a 3D designer for final 3D animation production, worked side by side with software engineers to implement the AR as part of the app.
We gathered as a whole team and I lead a sketching session to explore possible experience flow and then translated into rough digital sketch. Then we used the sketch to develop low fidelity wireframe to start testing out with internal staff then actual students.
As we continue to build the digital experience, I lead designing the information architecture.
As the learning experience gets dense enough to explain easily with stakeholders, I proposed and implemented a Quest Method Card, which enables to stripped down the content layer and show the activity layer of the learning experience. This set of cards became highly useful not only to explain new teammates, internal stakeholders, and teachers, we also utilized to design another Quest products.
After testing few digital prototype with wireframes, we started to solidify the learning experience. It was the time to develop the user interface that is useful and delightful by the middle schoolers. This is one of the version that I designed for prototype version 4.
I art directed and collaborated with 3D designer to implement AR breakfast experience.
After several user testing, we found that interaction is too complicated and was not achieving best user experience, we changed direction and I proposed and implemented much simpler design.
The color coding per character that each student would play a role made so much more sense and it became very clear from the far distance which character students are playing in the classroom. This add another layer of benefit for teachers.
Through usability test, we proved this UI brings much better experience and leas confusion while navigating within the app.
This is the final interaction flow that I designed. We made interaction simple and identical for all students regardless of which role they are playing. Only difference they experience are in contents such as audio, AR, texts (story), challenges for debate, challenges for the drama class, and picture for final assembly.
We successfully launched the app to the Amplify customers. This Perception Academy Quest is consist of 6 experience.
The Quest is structured like a medical thriller: On an ordinary morning, students are involved in a minor school bus fender bender on the way to school. They experience the crash via audio recording though their headphones. While “walking” the rest of the way to school, they hear narration detailing the eerie “symptom” of altered perception—something is clearly amiss, but students don’t yet understand what it is.
At school, augmented reality simulation distorts students’ breakfast on their tablet, offering more evidence of the mind-bending changes going on inside them.
In ELA, they read case studies in Sacks’s book, which detail the powerful forces that have commandeered their perception—they are suffering from neurological disorders, the result of brain trauma.
During Recess, they play a card game that helps them confront their new limitations. One student picks a card, and determine YES or No whether he/she can perform that task on the card with restriction he/she has as a role. Then, remaining students would determine to AGREE or DISAGREE and they go through few rounds of debate/discussion.
Next, students pair up to imagine how people with different disorders might solve problems together, and dramatize these interactions as a scene of dialogue.
The Quest concludes with a collaborative writing assignment in which students use their knowledge of all five disorders and create a scenario that explains the circumstances of that morning’s bus accident.
This learning experience product leveraged the power of digital technology to provide teachers with a resource that they wouldn’t have the time, specialized knowledge, or technical know-how to create on their own. This app facilitates students tackle challenging texts by creating a prepackaged, collaborative learning experience that sharpens critical English Language Arts skills for middle schoolers.
With a challenge of no connectivity, we designed simple product that allows humans to play a vital role for interaction with other peers in the classroom.
Takao Umehara, Experience Design Strategy, Lead Visual Design, UX/UI Design Direction
Jean Pierre Dillard: Concept Development, Art Direction
Chris Kalb: Illustration, Concept Development
Chris Leathers: 3D Modeling
Anna Ziegler, Nicco Moretti: Content Authoring, Lesson Guide, Overall Concept
Nicole Stein, Dan Russo, Te Yi Liu, Eli Burmin, Beth Goebel : Engineering
Sarah Smirnoff: Project Management