Home Energy Management Challenges System

Challenges System

The Powerley Challenges system utilizes a gamified approach to motivate users to take specific Energy Efficiency (EE) actions. Upon completing tasks, users receive both intrinsic and extrinsic rewards to increase engagement and retention.

However, the original Challenges system required significant effort and resources to maintain the gamification aspect, which was not proportional to the desired impact of the feature. When the gamification was removed, negative reviews increased and engagement metrics decreased, indicating that gamification had a positive impact on the feature's effectiveness. As a result, an updated Challenges system was developed with gamification, which involved extensive research and user testing to create a low-effort yet high-value rewards system for both end-users and utility customers.

My Roles

Strategic Direction
Creative Direction
User Research
User Testing
Product Design
User Experience Design

Cross-functional teams

Product Design
Product Management
Voice of the Customer
Engineering (Mobile & Web)
Data Science
Quality Assurance
External Research Consultants

The Problem

The Challenges feature has received a significant number of negative reviews, which could impact user adoption and engagement levels.
Despite being frequently visited, the feature has low engagement, indicating a potential loss of interest among users. This highlights the need to identify and address underlying issues to improve the feature's effectiveness and user experience.

Hypothesis

The lack of sufficient incentives, buggy app experiences, and ongoing performance issues may be contributing to the loss of interest and engagement in the Challenges feature. To address these challenges, a successful Challenges system must be engaging and appropriately tailored to the target audience. Personalization of Challenges should be prioritized to maximize engagement, but should also be designed with scalability in mind to ensure widespread applicability. By addressing these issues, we can create a more effective and user-friendly Challenges feature that is capable of sustaining user engagement over time.

The Goals

Us: Increase user engagement with the app and improve review sentiments around the Challenges feature. Additionally, we aim to gain insights into user behavior by creating a measurable path towards milestones.
Home Resident: Save energy and reduce expenses through an engaging and personalized experience that is tailored to their specific needs and preferences.
Energy Utility: Increase platform engagement, promote energy efficiency, and reduce customer power consumption to achieve sustainability goals and enhance customer satisfaction.

Success Metrics

kWh reduction
App engagement rate
Challenge completion rate
Feature retention rate

Challenges

The Challenges feature encountered multiple challenges during its design and implementation. Substantial external research and user testing were required, along with a significant budget to support the rewards system. These resource-intensive activities caused the project to undergo several reprioritizations with other initiatives within the company, leading to delays and a need to resume the design process from where it was halted.

Approach

Fundamental flaws in the original feature's usability led us to research multiple behavioral studies on engagement. This research supported our decision to utilize gamification, with new approaches, to incentivize EE behaviors and improve user engagement.

Our core team developed a phased feature release approach to address primary pain points, starting with beta testing to collect crucial field data. The team's ambitions for the feature included a strong foundation with a scalable approach that leveraged user innovation.

Additionally, we collaborated with the University of Michigan's Science in Information department on a research project that provided valuable consultation from two Masters students in the program.

The beta test was initially launched at one utility and monitored by the Development and Data Science teams to ensure proper functionality. We then rolled out a 10-week test at other utilities, requiring users to opt-in to the beta experience with gamified features.

Internal Research

Approximately 75% of users engage with the Challenges feature during any given month.
The original feature was only being engaged with 2-3 times per week, which is significantly lower than engagement rates for other core features (which were approximately 7 times higher).
Despite the existing Energy Efficiency (EE) challenges, we observed a 30% increase in customer usage between week 1 and 20.
Negative customer reviews validated our hypotheses and reinforced the need for redesigning the Challenges feature with a new gamification approach.
These insights from internal research were critical in informing our decision to incorporate gamification into the feature and our ongoing efforts to improve user engagement and promote sustainable behaviors.

Discovery (Related Industries)

Our research revealed that several utilities have deployed gamification-related EE programs. In addition, several direct competitors have invested resources into building their own gamification platforms.

Despite these efforts, we found that challenge and reward features are relatively new to the Home Energy Management (HEM) industry. Most existing systems offer only Demand Response (DR) challenges and cash rewards, with limited emphasis on personalized and engaging Challenges to incentivize energy efficiency. Therefore, our efforts to incorporate gamification into the Challenges feature were an innovative and unique approach to promoting sustainable behaviors and reducing energy consumption.

Discovery (Unrelated Industries)

To better understand behavior change techniques used in consumer industries outside of HEM, our team conducted market research analysis with the assistance of U of M Masters Students.

We investigated successful apps that use persuasive technology to gamify their product, such as Duolingo, Headspace, Spotify, Starbucks, and Forest. We used behavior change techniques and theoretical frameworks to standardize the languages while evaluating their techniques.

Our findings revealed that successful gamification features included hyper-personalization, social influence and comparison, gamified levels, intrinsic and extrinsic rewards, and goal-setting via alerts. Incorporating these techniques into the Challenges feature helped to maximize user engagement and promote sustainable behaviors.

User Interviews

To understand user pain points and preferences, we conducted interviews with frequent users, declining users, and non-users of the existing Challenges feature.

Key findings:

Users care about energy conservation and want to contribute to reducing their carbon footprint.
The current approach to levels is unclear and needs to be simplified to better engage users.
Tasks need to be personalized to reflect user interests and preferences.
Immediate feedback upon completion is necessary to help users feel a sense of accomplishment and progress.
An entertaining progress visualization would be a nice addition to the feature and could help sustain user engagement over time.

Focus Groups

To expose existing users' mental models, we hosted a small guided workshop in collaboration with U of M. The results helped us prioritize opportunity areas for prototyping.

Through the use of Lightning Talks, How Might We? (HMW) Affinity Mapping, Crazy Eights, design discussion, and concept definition exercises, we gathered further insights. Key findings from the focus groups include:

Tasks need to be personalized
Users like seeing a positive impact on the environment
Customers want both virtual and tangible rewards
Social influence features would increase engagement
Real-time progress visualization is crucial
Use Challenges as a channel to promote new features

Card Sorting

Open Card Sorting was conducted to allow testers to decipher categories that make sense to them from the cards provided. By ranking category importance, testers created a multi-level hierarchy according to how they would like to engage with the feature.

Findings to support the information architecture showed:

Testers anticipate a full-fledged Reward Center or 'Shop'.
Testers expect a User Profile and social comparison abilities.
Status and "bragging rights" were clearly prioritized.

Feedback Survey

Using the existing Challenges feature, we prompted a utility's customers to provide feedback on challenges they would be interested in completing and the level of effort they are willing to expend. Key findings from the survey include:

Users aim to lower their utility bill and reduce overall energy use.
Users would like to participate in a mix of one-step and multi-step challenges.
Users are highly motivated by utility bill discounts, status levels, and badges.
Some users expressed interest in receiving free smart devices, particularly smart plugs, to better track their energy usage.

Ideation

The Design Team experimented with many quick iterations of wireframes, flows and mocks before arriving at the prototype for a user test.

Mid-Fi Invision Prototype

To validate that the proposed features and tasks would improve engagement and retain users, we asked participants to complete tasks within the prototype and provide feedback on ease of use. Key findings from the prototype testing include:

Testers would like to participate in challenges and reward systems provided by their utility.
The most-wanted reward is a discount on their utility bill, and we need more general reward options outside of smart devices, such as cash or gift cards.
Testers are motivated by positive reinforcement and intrinsic rewards, and do not care much about seeing a 'set' of intrinsic rewards all in one place.
No testers would share their results on social media, and there was a lot of uncertainty around 'level' status.
Overall architecture and concepts resonated with users, and participants were able to complete tasks from end-to-end.
The point system was well-understood, though the difficulty of a challenge needs to be reflected in the point value.
Users liked the idea of 'leveling up' but were unsure of the structure and benefit, and the majority of testers would not 'save' challenges they would like to complete later.

Solution

Due to technical constraints with bandwidth, we launched the new Challenges feature in two phases. Phase 1 involved launching a bank of 20 reconstructed Challenges that were personalized with home energy, customer journey, and engagement data relevant to each specific user. These Challenges were also feature-toggle aware for differing utility rate types and subscription plans between each participating customer.

To ensure that current users were aware of the changes, we created an opt-in experience that informed them that their existing points and achievements would be temporarily removed as they helped in testing the new feature. It was important for current users to know that their progress and points would be converted into new rewards later once the new Challenges feature was fully implemented.

These measures helped us successfully launch the new Challenges feature and ensure that users were able to participate in personalized challenges that effectively motivated them to reduce their energy consumption and achieve sustainability goals.

Challenges: Goal-based games designed to achieve a specific objective (e.g. complete profile, stay under budget). Some challenges consist of one task, while others contain several. Completion of a challenge gives the user a specific amount of points.

Points: Accrual of points allows the user to level up, collect badges, and redeem for rewards.

Levels: A new level will unlock as the user reaches new milestones in the experience.

Descriptions: Detailed explanations as well as progress tracking were added to each challenge, giving the user a better idea of how to do the task and what to expect.

Success: Once the user completes a Challenge, gets a new badge, or levels up, they’ll get a fun animation. Cyclical motivational elements, such as success states and notifications were included to boost engagement and let the user know how they did or are doing.

Intrinsic Rewards: Badges are a motivation tool, achieved by completing special Challenges, hitting new milestones, or being applicable for certain statuses. Users can visualize all badges together on one page, in the order they were earned. The ‘Potential’ tab lets users see what badges they can work towards earning.

Extrinsic Rewards: Utility Rewards are the goods, services, and discounts that the user can exchange points for. Within the Reward Center, users can window shop by tapping individual utility rewards they’re interested in. The four most recent badges earned are displayed under the Trophy Case.

Phased Development Rollout

The proposed Challenges system was going to be a large undertaking and to ensure that our proposed solution was going in the right direction we decided to create a phased rollout as a means to gather feedback on specific components so that we felt confident in continuing to bulid on and support this feature long term.

Phase1

The MVP release was created to focus on the challenges themselves. We organized different challenges within specific categories. These categories included, Power User, Energy Saver, Money Saver, and Understand what challenges were of interest

The Results

70% of users are more likely to complete challenges under a new system
On a scale from 1-5, users ranked level of effort to complete challenges at an avg of 2.7
For most, detailed descriptions made it easier to complete the challenge than before
The majority of users enjoy tracking with the progress bar, though some did not find the bar to be the most helpful UI element. 10% don’t care about tracking progress at all.
While completing challenges, 30% of users discovered other features in the app that they were previously unaware of, increasing overall app engagement.
20% of users had too much difficulty completing challenges to explore further

Phase 2

Now that we have a better understanding of what challenges resonated with users and the level of difficultly. We now wanted to test the implementation of extrinsic rewards. Introducing rewards came with a new challenge as it called for significant funding from the utility customer.

After we were able to obtain funding we needed an approach that allowed us to test extrinsic rewards with minimal development effort. The team decided that it was going to be the least amount of effort to send an Amazon giftcard via email each time the user reached a new milestone.