Jennifer Story
PROJECT
VAUX (Visual Automation UX)
Create your own routine tasks automation by recording usage of the phone
ROLE
User scenarios, user flow, wireframe, visual mockup, demo prototype
As a lead designer, worked with SW engineers
DURATION
Design Research and development April. 2018 ~ Aug 2018
Project Introduction
-
Smart and intuitive task automation by recording the user behavioral App usage.
-
Teach my device what to do by demonstrating the tasks and the events with Machine learning (semantic segmentation of screen images and OCR)
Challenges
-
Make easy of repeated routine App usage with no training or required model
-
Smart and intuitive interface to operate
Motivation
-
User perform numerous tasks on their mobile device. Some tasks are repetitive, tedious, and time-consuming.
-
AI assistants activate a series of actions one by one, what if make task automation in one command
-
Existing automation approaches are not user friendly
-
AI assistants (Bixby, Google Siri, etc.) have limitations.
- Understand context but lack support for deep navigation inside of the App.
- Requires 3rd-party app support.
User experience
Overall user Interface
List of recorded tasks/preview of tasks
Task recording in progress
Actions in a task can be modified
Detail view of user actions and corresponding screen
Set up the task-trigger conditions to start automatically
Time trigger
User flow / wireframe
User scenarios
Distinctive Features
-
Multi-commands at one time
VAUX
Spotify >>> Waze. >>> Spotify
-
Open Waze and set the destination
-
Open Spotify and shuffle play Favorite Playlist
-
Go back to Waze and start Navigation
VS
Bixby
Set pre-defined command >>. bring Bixby. >> command to action
-
Set pre-defined commands, one at a time.
2. Deep Operations
VAUX
Finance App >>> Stock >> Detail
-
Open Finance App
-
View the detail of stock
Bixby
VS
Bring Bixby. >> command to action
-
lacks support for deep operations inside Apps. 3rd party Apps need to provide Bixby support
4. Other virtual assistants
OK Google
Apple Siri
VS
More focus on the search
Poor support on multiple commands
3. Off-track Actions
VAUX
S-Health App >>> Set daily calories
-
Open S-Health App
-
Set daily calories to 1200
like operating as naturally as human user
Bixby
VS
Bring Bixby. >> Options >> Select one >> Set daily calories
-
Getting confused on multiple choices, still require further action by the user
Recording task:
The user defines tasks by simply user interactive demonstration of taps, gestures or other actions along with screenshots when each action occurs
Inferred task:
Analyze patterns in user behavior based on the combination of ML and deep learning techniques including activity recognition, routine detection, and contextual recognition.
i.e. user actions, snapshots of the screen, SW data, HW sensor data, etc
Manual start:
Users can manually select and start a pre-defined task (recorded or inferred), by UI clicks, voice command and etc.
User-defined triggers:
The user specifies the set of conditions for the task to be automatically performed when the condition is met
Inferred triggers:
The set of conditions inferred by the system upon which users usually perform a specific task. These triggers are learned along with the inferred tasks.
i.e. system detect that the user performs her commute+music every weekday when entering her car: trigger is weekday around
Automatic start:
If triggers are assigned to a task and when the condition of the triggers is met, the task will be automatically started
Prototype Demo Video
Value of VAUX
-
Freestyle/unlimited task definition by demonstration
-
Context-aware task automation with flexible triggers and learning user activity patterns.
-
Intuitive interface leveraging existing mobile user behavior: Recording, Editing, and Playing
-
Simple user experience with multiple commands at one time
-
Deeper engagement with manipulation of activities
