AR-based learning tool for Luv Michael's autistic employees (In Progress)

AR-based learning tool for Luv Michael's autistic employees

(In Progress)

Creating an accessible, inclusive learning experience for Luv Michael's Neurodivergent employees to learn and practice hygiene practices in their kitchens.

Creating an accessible, inclusive learning experience for Luv Michael's Neurodivergent employees to learn and practice hygiene practices in their kitchens.

Overview

Luv Michael's autistic employees are in need of an accessible tool to learn about safe practices in their kitchen

Luv Michael's autistic employees are in need of an accessible tool to learn about safe practices in their kitchen

Luv Michael's employees currently receive training from instructors on safe kitchen practices through lessons and physical games. Given the diverse learning abilities of the employees, Luv Michael tasked my team and I with creating a more accessible and inclusive learning tool to better address their needs.


Luv Michael is a granola making non-profit company that provides employment for neurodiverse adults.

CLIENT

  • Luv Michael

  • NYU school project. (Assistive Technology Course)

Role

  • UX Designer

  • AR Developer

Team & Duration

  • 3 UX Designers,

  • 2 AR Developers.

  • Six Weeks & ongoing

Tools

  • Figma

  • Snapchat's Lens Studio

  • Unity

Problem

How can we make Luv Michael's Serv Safe Handbook, particularly the personal hygiene section, more accessible for autistic employees?

How can we better address Luv Michael's autistic employees' unique needs by making the Serv Safe Handbook's personal hygiene section more accessible?

How can we better address Luv Michael's autistic employees' unique needs by making the Serv Safe Handbook's personal hygiene section more accessible?

action

  • Field research

  • Competitive Analysis

  • Usability Testing

  • UI Kit

  • Wireframing & Prototyping

  • AR tool development

  • Field research

  • Competitive Analysis

  • Usability Testing

  • UI Kit

  • Wireframing & Prototyping

  • AR tool development

Solution

An AR-based learning tool is being developed on Unity AR to be used by Luv Michael employees in the kitchen to revise and apply their learnings in quick 10 minute exercises.

An AR-based learning tool is being developed on Unity AR to be used by Luv Michael employees in the kitchen to revise and apply their learnings in quick 10 minute exercises.

impact

A 60% increase in user engagement with the AR-based tool, with additional user testing scheduled for the latest prototype to further evaluate retention rates.

UX Research

  1. Client Visits

Field Study: Our visits to Luv Michael kitchen helped us understand our users and their needs better

Field Study: Our visits to Luv Michael kitchen helped us understand our users and their needs better

  • Instructors at Luv Michael use a curriculum called ServSafe to train their employees on safe kitchen practices.

  • Lessons are conducted by the instructor using a whiteboard, where she writes and draws as she speaks. To reinforce these lessons, employees engage in physical card sorting activities.

Why is the current learning system not effective?

Why is the current learning system not effective?

  • Retention of information is low.

  • Multiple repetition of lessons is often required.

  • Varied learning abilities becomes a challenge in a group learning setting.

  • Employees often find it hard to relate what they learn to their work, making it difficult to apply the lessons effectively.

What might help?

What might help?

What might help?

  • Simple, quick exercises to revise lessons.

  • Some degree of customized learning.

  • Employees need to connect learning to their environment; an immersive, practical experience could help.

  1. Research and Brainstorming

Secondary Research: Understanding the learning needs and available approaches of Neurodiverse Individuals

Secondary Research: Understanding the learning needs and available approaches of Neurodiverse Individuals

  • Understanding Luv Michael's curriculum - I reviewed Luv Michael's ServSafe curriculum to see their material, understand the content that is being taught and how.

  • Available tools/approaches in the market for neurodiverse population - Most learning tools on the market are designed for autistic children, with very few available for autistic adults.

  • Research papers on Autism and Learning suggest that - augmented reality is quite effective for learning in autistic individuals.

  • Research papers on Autism and Learning suggest that - augmented reality is quite effective for learning in autistic individuals.

Ideation: My team and I came up with different ideas for the curriculum and below are some of the ideas we discussed with our Luv Michael's POC

An Interactive Web Game?

Why?

Why?

  • Engaging

  • User can control the pace.

  • Can use audio, visual, and text catering to different learning styles.

  • Engaging

  • User can control the pace.

  • Can use audio, visual, and text catering to different learning styles.

Why not?

Why not?

  • The current material used to teach at Luv Michael is similar.

  • The current material used to teach at Luv Michael is similar.

A Tactile Game?

Why?

Why?

  • Touching physical cards and moving them around can be an effective learning technique.

Why not?

Why not?

  • Currently being experimented with.

  • Want something that does not require assistance from another person.

  • Currently being experimented with.

  • Want something that does not require assistance from another person.

An Augmented Reality based tool!

Why?

Why?

  • Studies show that AR works well for people with Autism.

Why?

  • Studies show that AR works well for people with Autism.

Why might it work?

Why might it work?

  • Integrating the work environment into training helps employees relate learning to their job.

  • Our Luv Michael contact, found this approach promising and wanted us to explore AR further.

Why might it work?

  • Integrating the work environment into training helps employees relate learning to their job.

  • Our Luv Michael contact, found this approach promising and wanted us to explore AR further.

  1. User Testing with AR

Observing how seven Luv Michael employees reacted to AR as a learning tool

Observing how seven Luv Michael employees reacted to AR as a learning tool

My team and I conducted another client visit where 7 employees tested the below AR application for 15 minutes each. We conducted fly on the wall observations noting how employees interacted with their phones while using the AR applications.

My team and I conducted another client visit where 7 employees tested the below AR application for 15 minutes each. We conducted fly on the wall observations noting how employees interacted with their phones while using the AR applications.

Chalk

Chalk

Chalk was chosen to test employees' interaction with a free-flowing, minimal-constraint AR experience.

Chalk was chosen to test employees' interaction with a free-flowing, minimal-constraint AR experience.

AR Basketball

AR Basketball

AR Basketball was chosen to observe employee reactions to a simple, constrained AR experience.

AR Basketball was chosen to observe employee reactions to a simple, constrained AR experience.

AR Paddle Ball

AR Paddle Ball

AR Paddle Ball was chosen to see how users respond to a more flexible, less constrained experience.

AR Paddle Ball was chosen to see how users respond to a more flexible, less constrained experience.

Key Observations

Insights

1) Participation Engagement:

  • All 7 participants stayed engaged throughout the 15 minutes session.

1) Participation Engagement:

  • All 7 participants stayed engaged throughout the 15 minutes session.

  • AR effectively captured their attention.

2) Effective Features:

  • AR Basketball app - demonstrating clear understanding, high engagement, and ease of use

2) Effective Features:

  • AR Basketball app - demonstrating clear understanding, high engagement, and ease of use

  • Ease of use is a key factor.

3) User Feedback:

  • One employee found the AR Basketball app easy to understand, unlike other apps.

  • Most employees enjoyed seeing motion on the screen. (e.g., watching a basketball move)

  • Constraint and simplicity are valuable.

  • Touch-triggered motion is engaging.

4) Interface Clarity:

  • The split-screen interface with two directions was clear to all 7 participants.

  • Split screen is not confusing.

Prototyping: 1st Iteration

Prototyping: 1st Iteration

  1. User flows, Wireframes, and UI Kit

  1. User flows, Wireframes, and UI Kit

I designed scenarios, user flows, and wireframes for the AR feature

I designed scenarios, user flows, and wireframes for the AR feature

Why Snapchat's Lens Studio? - With the project's initial 4-week timeline, Lens Studio was chosen as it offered the shortest learning curve for building the AR feature.

Why Snapchat's Lens Studio? - With the project's initial 4-week timeline, Lens Studio was chosen as it offered the shortest learning curve for building the AR feature.

User Flow

User Flow

User Flow

Low-fi wireframes

Low-fi wireframes

Low-fi wireframes

Designed the UI kit with color theory and accessibility principles tailored for Autism to improve the learning experience.

Designed the UI kit with color theory and accessibility principles tailored for Autism to improve the learning experience.

Based on our research on how colors affect people with autism, we discovered the following:

  • A huge percentage of the population finds certain bright colors like reds and neon colors intense.

  • Blues, greens, yellows, and pastel colors have a more positive effect on them.

Based on our research on how colors affect people with autism, we discovered the following:

  • A huge percentage of the population finds certain bright colors like reds and neon colors intense.

  • Blues, greens, yellows, and pastel colors have a more positive effect on them.

UI Kit

UI Kit

UI Kit

  1. First Prototype

I refined my team's sketches into a simple high-fidelity prototype on Lens Studio to test the concept

I refined my team's sketches into a simple high-fidelity prototype on Lens Studio to test the concept

We used Snapchat's Lens Studio to build our initial prototype to conduct a round of usability testing with Luv Michael's employees and move to the next iteration.

We used Snapchat's Lens Studio to build our initial prototype to conduct a round of usability testing with Luv Michael's employees and move to the next iteration.

  • Readable text

  • Readable text

  • Text accompanied by slow speed audio

  • Text accompanied by slow speed audio

  • Images to support the text

  • Images to support the text

  • Adhering to WCAG Accessibility guidelines

  • Adhering to WCAG Accessibility guidelines

Sample Scenario #1: Which sink do we wash our hands in?

Sample Scenario #1: Which sink do we wash our hands in?

Uh-oh! Looks like there's something on the floor!

Uh-oh! Looks like there's something on the floor!

After the first scenario, the AR tool would ask the user to tilt their phone towards the floor and say, 'Uh-oh! Looks like there is something on the floor!' prompting the user to tilt their phone upon which they would see virtual granola on the phone.

(This prototype has not been built in this iteration)

After the first scenario, the AR tool would ask the user to tilt their phone towards the floor and say, 'Uh-oh! Looks like there is something on the floor!' prompting the user to tilt their phone upon which they would see virtual granola on the phone.

(This prototype has not been built in this iteration)

Sample Scenario #2: What do we do after picking something off the floor?

Sample Scenario #2: What do we do after picking something off the floor?

User Testing with First Prototype

User Testing with First Prototype

With our initial prototype, user testing with 10 Luv Michael employees helped us understand the following

With our initial prototype, user testing with 10 Luv Michael employees helped us understand the following

Surprise Iteration #2 - Pivot!

Surprise Iteration #2 - Pivot!

  1. Brainstorming

Luv Michael wanted us to continue working on the prototype beyond the initial project timeline and we were thrilled!

Luv Michael wanted us to continue working on the prototype beyond the initial project timeline and we were thrilled!

Pivoting - adapting to user testing insights and fresh client demands!

Since our decision to choose Snapchat's Lens Studio was made based on our initially discussed project timeframe and project scope, we had to now pivot based on the new timeframe and already made progress.

Since our decision to choose Snapchat's Lens Studio was made based on our initially discussed project timeframe and project scope, we had to now pivot based on the new timeframe and already made progress.

First Iteration

Second Iteration

  1. Snapchat's Lens Studio: Limited scope that doesn't suit new iteration.

  1. Snapchat's Lens Studio: Limited scope that doesn't suit new iteration.

Unity AR: Higher technical feasibility

Unity AR: Higher technical feasibility

  1. Camera open throughout: distracting

  1. Camera open throughout: distracting

Camera open only while setting the scenario: clear indication

Camera open only while setting the scenario: clear indication

  1. Split screen approach: Confusing

  1. Split screen approach: Confusing

One screen with 2 options side-by-side: clear and simple decision making

One screen with 2 options side-by-side: clear and simple decision making

  1. AR focus throughout: Doesn't help in effective scenario building

  1. AR focus throughout: Doesn't help in effective scenario building

AR Focus only at essential aspects: for effective scenario building with emphasis on understanding the question at hand.

AR Focus only at essential aspects: for effective scenario building with emphasis on understanding the question at hand.

  1. New user flows, wireframes, and UI Kit

  1. New user flows, wireframes, and UI Kit

Our content creation and design team members developed new user flows and wireframes to match the new requirements.

Our content creation and design team members developed new user flows and wireframes to match the new requirements.

The below user flow diagram was created for the same scenario from the first iteration matching the changes based on previous user feedback, and language changes based on feedback from Luv Michael's instructors.

The below user flow diagram was created for the same scenario from the first iteration matching the changes based on previous user feedback, and language changes based on feedback from Luv Michael's instructors.

As mentioned above, based on user testing results, we decided to focus the user flow primarily on the screens, incorporating minimal AR only for essential parts of the experiment. This approach allows users to concentrate on questions and answers while effectively using AR to build scenario steps.

As mentioned above, based on user testing results, we decided to focus the user flow primarily on the screens, incorporating minimal AR only for essential parts of the experiment. This approach allows users to concentrate on questions and answers while effectively using AR to build scenario steps.

Revised Low-fi Wireframes

Revised Low-fi Wireframes

Revised Low-fi Wireframes

Revised UI Kit

Revised UI Kit

Revised UI Kit

  1. Final Prototyping

We created our high fidelity prototype screens on figma which were later exported to Unity AR to develop the application

We created our high fidelity prototype screens on figma which were later exported to Unity AR to develop the application

High fidelity screens were made on Figma using the UI Kit designed in the first iteration, detailing each scenario and various outputs leading the user through the First Chapter of the ServSafe book - Personal Hygiene.

High fidelity screens were made on Figma using the UI Kit designed in the first iteration, detailing each scenario and various outputs leading the user through the First Chapter of the ServSafe book - Personal Hygiene.

The below prototype is the current version of the app that is still undergoing work since this is an ongoing project.

The below prototype is the current version of the app that is still undergoing work since this is an ongoing project.

Showcasing the work!

Showcasing the work!

Our team qualified to showcase our work in Tandon's Research Excellence Exhibit 2024 where we received valuable feedback

Our team qualified to showcase our work in Tandon's Research Excellence Exhibit 2024 where we received valuable feedback

At Tandon's Research Excellence Exhibit, we got a chance to present our research and prototype to numerous attendees and judges. The feedback that the participants gave us based on their use of the prototype and the research was helpful in thinking about future considerations.

At Tandon's Research Excellence Exhibit, we got a chance to present our research and prototype to numerous attendees and judges. The feedback that the participants gave us based on their use of the prototype and the research was helpful in thinking about future considerations.

My Learnings and Next Steps

My Learnings and Next Steps

Working on this project with Luv Michael has been an incredibly joyful and rewarding experience for me as a designer. Collaborating with a disability audience and leading the end-to-end product design and development process has greatly enhanced my ability to be flexible and adapt to evolving requirements, and thinking about accessibility and inclusivity. A key takeaway for me was the importance of being ready to "pivot" and make changes as conditions shift.

Next steps for this product includes:

  • Conduct a round of usability testing with the latest prototype on Luv Michael's iPad to understand pain points, and effective approaches of learning or using the application that can inform our decisions for future iterations.

  • Take 3D scans of the equipment in the kitchen and use them in the app to increase the relatibility factor for employees.

  • Extend our app to other chapters of the book.