LIFESAVaR - Oct 2020
Victim triaging and rescuing reimagined
"LifesavAR" is an augmented reality concept application for saving lives (victim triage) in an emergency. It is aimed to improve the overall rescue operation and reduces response time by overcoming the information gap between the EMS, fire, law enforcement officials, and the incident command.

Challenge

Improve a first responder's situational awareness in a emergency using AR
Design an augmented reality interface leveraging heads-up display, holographic interfaces, and IoT based sensor streams, to convey actionable information and improve situational awareness to first responders without distractions or cognitive overload.

Project Specifics

Deliverables
Concept Sketches
Personas
10x10 sketches
High-fidelity Prototype
Unity Project file + MPK
Tools
Adobe XD
‍Illustrator
Photoshop
Unity
MagicLeap Lab
Team
Abhinav Sikharam
Akshay Yelmar
RaviTeja Jorigay
Sathwik Aala
Ying-Hsuan (Irene) Kuo
My Role
As a Product Designer, I contributed to research, ideation, concept sketching, prototyping, and collaborated with the developers to build the product. I led the team in creating the interactive prototype and cross-functional communications with the developers.
First Responder's Interface
A heads-up display for making quick decision on-site with the available information
Incident Commander's Interface
A hologram projection display with curated data from sensors on-site to guide the public safety officials.

Introduction

Background
Public safety officials perform tasks in various environments that severely limit their senses and ability to communicate. The aftermath of any emergency is filled with chaos, and the public safety personnel needs to coordinate and rescue the victims as quickly as possible effectively. They follow traditional and standardized procedures for logistics, assessment, triaging, etc. They are using regular radio communication, information from witnesses, and visuals from the site.
Problem Space
All emergencies require immediate action to rescue a victim and it is one of the utmost priorities. But there is a significant gap in action, response time, and lack of standardized system while responding to an emergency situation that is well-validated, reliable, and uniformly accepted.
Problem Statement
We converted the challenge description provided to us into a how might we statement to define our design direction and focus on the user’s most pressing needs. The problem statement we followed is:
Design Process
I collaborated with two designers, two developers, and a project manager for this project in researching to designing and developing a working prototype. We followed a combination of a Double diamond and Lean UX design processes for this project and a high-level process and cross-functional team involvement is represented in the diagram below.

Research

Public Safety Personnel
Public safety personnel are any law enforcement officer, fire department member, emergency medical response personnel, or other individuals responding in the aid of public safety or rescue.  Their job is to rescue and attend to those injured, suppress fires, secure and police the disaster area and to begin the process of restoring order.

First Responders

are often the primary line of defense, responding to an evolving spectrum of natural and man-made threats.

Incident Commander

would help the agencies responding to a disaster to work together in a coordinated and systematic approach.

Operating Scenarios
There are four typical scenarios to explore the professions and the kinds of tasks they do and information that will be helpful for them to operate in an emergency.
1
Mass-Transit
Accident
2
Active
Shooter
3
Flood
4
Wildfire
Task Analysis
Firefighters respond to a variety of emergency situations, including fires, natural disasters, hazardous waste spills and medical emergencies. Below is a list of key tasks that the first responders and incident commanders are responsible in an emergency situation.

First Responder's Tasks

  • Locating and evacuating victims
  • Preparing for staging/triage/ treatment
  • Identifying and mitigating risks on-site
  • Mapping hazardous materials 
  • Maintaining safety of personnel and team
  • Identifying safe locations

Incident Commander's Tasks

  • Identify victims or assets at risk and assist
  • Track resources (e.g., personnel movement)
  • Advising/drawing feedback to inform responders
  • Establish geographic knowledge of area
  • Monitor evacuation routes
  • Detect injured individuals
Victim Triaging
Triage is the process of examining patients and assessing how urgently they need care, and determining who will get treated first. Traditionally, victims are triaged using the physical tags and color codes with manual entries of information.
START (Simple Triage and Rapid Treatment)
START helps responders rapidly assess victims’ ability to walk and their mental status, blood flow, and respiration. Based on the assessment, patients are placed in 1 of 4 color-coded categories:
IMMEDIATE: severely injured patients.
DELAYED: stable patients.
MINOR: patients with minor injuries.
DECEASED: victims who have been killed during the incident.
AR Platform
Spatial Computing enables the blending of digital contentand the physical environment

VIRTUAL REALITY

Digital environment isolated from the physical.

AUGMENTED REALITY

Digital content on top of the physical world

SPATIAL COMPUTING

Digital content interacts with the physical world
Input Methods
The input method for the device is fundamentally different from the way users are used to interact with digital content, especially compared to interactions within handheld augmented reality apps.
Fiducial Markers
Fiducial markers are visible objects placed in an imaging system (camera) as reference points or measuring units. In VR and AR, fiducial markers are used for positional tracking and overlaying digital objects in reference to the physical space.
Findings
Fiducial markers are visible objects placed in an imaging system (camera) as reference points or measuring units. In VR and AR, fiducial markers are used for positional tracking and overlaying digital objects in reference to the physical space.
Lack of centralized communication.
Fragmented tracking of triage statuses.
Difficult to track identity of victims.

Ideation

Personas
Public safety personnel are any law enforcement officer, fire department member, emergency medical response personnel, or other individuals responding in the aid of public safety or rescue.  Their job is to rescue and attend to those injured, suppress fires, secure and police the disaster area and to begin the process of restoring order.
" I’d like to know the patient’s medical information while triaging to  provide treatment. "
" I need accurate information to determine severity and guide incident response activities. "
Ryan Morris
Emergency Medical
Technician
Keenan Gill
Incident Commander
- Fire Dept.
Click on the persona’s picture to view the complete version
10x10 Sketches
We brainstormed different concepts and ideas using the 10x10 method with the personas in the context. We developed ideas, which are both practical and radical. These concept sketches gave us a direction towards converging feasible ideas into an appropriate solution.
Storyboard
We created two storyboards around scenarios that the first responder and incident command personas would operate.  We explored how they shall use an AR solution to rescue and triage victims in an emergency.
First Responder’s Scenario
Paramedic’s response in a mass accident scenario, identifying, triaging, and taking victims to a hospital for treatment.
Incident Commander’s Scenario
Incident Commander’s response in a mass accident scenario, analyzing the situation and planning, and deploying resources.

Design

Product concepts
Traditionally, victims are triaged using the physical tags and color codes with manual entries of information. We propose to replace the tags with fiducial markers which can be scanned using the AR goggles and victim status can be tagged to it using the START system of triage algorithm.
Traditional
Triage Tag
LifeSavAR
Triage Tag
Victim Triaging Task Flow
‘LifeSavAR’ System Overview
Data is collected from the site sensors, cameras, and drones using the IoT sensors. This data is provided to the incident command to make necessary decisions while tackling the emergency situation.
Low-Fidelity Prototype
Using the storyboards and product concept sketches we designed low-fidelity prototypes to demonstrate the concept and the flow of things.
‘LifeSavAR’ Triage Marker
A triage tag with a fiducial marker is tagged to the victim. This is the first responder’s view through the AR headset.
Triaging Identified Victim
After scaning the tag, the first responder has 4 options to chooce from based on the victim’s status.
Adding Victim Details
The first responder can enter additional details about the victim and also mark the injuries on the body.
Victims' Status
The data entered is shared across all the other first responders. They would know the total number of victims on site and their status.
Incident Command
IC has all the information relayed from the first responders on site. The victim details and site information helps them manage the resouces.
High-Fidelity Prototype
We used Adobe XD to design the high-fidelity prototype. There were several iterations of the designs for the final screens bsed on the usability, accesability, and aestheics. Below are a few screens from the design.
Scenario 1: Mass Transit Accident
First Responder Interface - Assigning triage tag
First Responder Interface - Visualizing patient assessment
First Responder Interface - Triage tag expanded view
First Responder Interface - Triage status overview
Heads-up display interface for Emergency Doctor
Incident Commander Interface
Scenario 2: Flood
First Responder Interface
Incident Commander Interface
Scenario 3: Wildfire
First Responder Interface
Incident Commander Interface
Scenario 4: Active Shooter
First Responder Interface
Incident Commander Interface
Final Design
After multiple design iterations and testing with a few target users, we finalized our designs and collaborated with the developers to build our AR app. Keeping the time and budget constraints in mind, we prioritized a list of features that we would develop as an MVP. The final design iteration of the product is presented below.
First Responder Interface - Mass Transit Accident (AR view)
Hover to play the video
First Responder Interface - Mass Transit Accident  (Prototype)
First Responder Interface - Active Shooter Scenario
First Responder Interface - Wildfire Scenario
First Responder Interface - Flood Scenario
Incident Commander Interface

Evaluation

User Testing
We conducted usability study sessions with two public safety personnel, one each from Indy EMS and Indy Fire to validate our designs and the interactions. Both the participants have served for more than 6 years in their respective fields of operations and well aware of all the typical tasks that comes with an emergency.
Results and Findings
The overall experience of the prototype was well received by both the public safety personnel. There were a few suggestions that were made by the participants to improve the experience.
For Mass-Transit Accident Scenarios
1. The system could display the location of victims that are tagged either by the drone or by the LifeSavAR system.
2. The resources section could also include things like tools, air bags, oxygen, doctor / surgeon.
For Wildfire Scenarios
1. It is ideal to have access to immediate information with a quick gesture instead of having it always present on the screen as it could be distracting.
2. Crossover communication between the fire department/ police/ EMS have to streamlined as it is difficult to use a radio with the AR device on.
This usability study confirmed that the basic usability aspects of the application are useful, usable, and functional. The evaluators felt that the system is consistent in terms of design which they liked. They also said that this system would definitely solve the problem of centralized communication and tracking of triage status.

Reflection

Next Iteration
We learned that AR technology combining with IoT sensor data can greatly improve the gap in current Triage process. If we had more time we would iterate our designs to accommodate:
1. voice recording and live interactions with the incident command.
2. Automating resource allocation using AI to make the system more efficient.
3. Assisted communication to make our architecture robust to use in real-time situation.
Key learnings
Collaborate & Empathize with devs
Apart from the design team, I got to collaborate with the Developers as well. I explained my thoughts regarding a particular concept followed by the designs and what we were expecting them to do. But somewhere I felt that there was a small communication gap in between, this taught me that you should be able to explain your views according to the other person.
Predict User Behavior
A product designer must understand user's behavior in such depth that they can predict their behavior. An in-depth understanding of the user’s workflow will assist in the ideation and execution of a seamless UX/UI design that the user will not even know exists!