Paint Adjustment Usability Test
Paint technicians add colorants to paint cans based on customer specifications. Usability testing was conducted on the new, modernized paint tinting system that was replacing a decades-old legacy system at 1700+ stores nationwide.
Why It Maters
When we buy paint from a home improvement store, the paint technician adds colorant to the paint cans based on customer specifications. Often, this is just the color code from the provided swatch, but additional customizations could include using a custom color formula, lightening a color by 25%, or reducing the amount of yellow colorant in spring green. These customizations require system flexibility that must be consistent with paint technician mental models and mitigate error. Operationally, paint mis-tints lead to wasted labor efforts, lost revenue and erode customer trust. Additionally, any mis-tints not sold at a discount must be safely disposed following expensive hazmat procedures.
Business Obectives
-
Reduce operating expenses by deprecating an expensive external legacy tinting system.
-
Reduce mis-tints that result in lost revenue.
-
Raise customer trust resulting in increased likelihood to recommend (LTR) scores.
Research Method
​​​​Moderated Usability Test with paint technicians.
-
N = 8 to allow for metrics stabilization.
​
-
Collected UMUX (usability metric for user experience) to assess overall system usability. ​
​
- Collected SEQ (single ease question) metrics to obtain subjective perception of task ease (a task-level metric).
​
- Used TCR (task completion rating) to asses task success.
- TCR scores are assigned by the researcher and can be used in conjunction with user-perception based UMUX and SEQ to get a fuller picture of usability. For example, a user might fail at a task, require assistance, or not know they failed a task and still rate the system or task as highly usable or very easy.​​
- If overall UMUX scores average 6.5/7 (excellent) but TCR is 50%, this signals to the researcher and designers that usability problems exist.
-
Paint technician users were tested at 3 stores to control for store similarities and a range of tenure (few months to several years) to ensure less experienced and experiences technicians gave feedback.
​
-
Testing script with detailed instructions was provided to moderators.​
​​
-
There was one high-fidelity prototype and users were shown 2 different task flows. Counterbalancing was done to control for possible order effects. ​
​​​​​
Data Analytic Approach
-
Reviewed test transcripts and video recordings of paint technicians interacting with the prototypes.
​​​
-
Extracted direct quotes from paint technicians to convey sentiments to stakeholders.
​
-
Categorized errors into meaningful groups, including misinterpretations, navigation errors, and system feedback issues.
​
-
Grouped general observations into themes, including navigation confusion, feature discoverability, and mental model mismatch.
​
-
Prioritized usability issues based on frequency and task criticality and triangulated with test metrics (UMUX, SEQ, TCR) to make recommendations.
​
​
​
​
​​​​
Insights
Entry Point​​
-
The entry screen was inconsistent with mental models for how paint orders were completed.
-
The color tinting entry screen gave paint technicians 3 options: "New Order," "Buy it Again," and "Color Adjustment."
-
To make a color adjustment, technicians needed to select the "color adjustment" option. However, all users overlooked this option and selected the "New Order" or "Buy it Again" option.​
-
Likely because all adjustments are either a New or Buy It Again order, making these options the superordinate order category, and then "adjust" is a subordinate category of new or buy it again orders.
-
-
​
Language Clarity​​​
-
The term "adjustment" confused paint technicians when applying it to a new order-- how can you make an adjustment to something that hasn't been created yet?
-
The term "adjustment" is more aligned to re-orders or to changes after the initial tint, suggesting that the entry point for the color adjustment flow may differ based on order type.
-
Both order types needed the adjustment option before and after tinting, but re-orders are more commonly adjusted before tinting and new orders are more commonly adjusted after tinting.
-
​​​
Required Selection
-
When tinting paint, the paint technician must select one of 3 bases to ensure correct tint configurations and prevent overflow.
-
Several paint technicians overlooked the base selection drop-down menu when inputting color information, which would result in tint errors.
​
Learning Curve
-
Paint technicians struggled to perform task scenarios that deviated from their current experience.
-
They expected the new tinting experience to have the same functionality as their current legacy system, and any changes in the new experience will result in an initial learning curve until technicians become familiar with the new system.
Recommendations
-
​The color adjustment option needed to be shown to paint technicians after they selected the order type.
​​
-
Both order types needed the adjustment option before and after tinting, but re-orders are more commonly adjusted before tinting and new orders are more commonly adjusted after tinting.
​​
-
​Move base selection drop-down to the top of the screen to establish visual hierarchy and ensure this selection is required before proceeding through the flow. Provide instructional error messaging to paint technicians who try to proceed without making a selection. ​
​​
-
Provide paint technicians with contextual help to ensure any deviations in functionality in the new tinting system compared to the current tinting system are more easily learned to ensure they can perform their tasks.