Watts - Automation and Data Analysis with Advance GUI

Introduction

If you work in operations, you're likely familiar with the ubiquitous gray GUIs—those nondescript interfaces that have become the backbone of countless workflows. Often initially crafted in Excel and VBA, and now frequently built with Python's tkinter, these gray boxes are quick to create and serve as essential tools for automating manual tasks. A simple button here, a dropdown menu there, and suddenly, hours of tedious Excel work are automated, freeing you to focus on more critical tasks. However, despite their utility, these interfaces often fall short in delivering engaging visuals or generating actionable insights that help users truly understand and analyze data. Users often find themselves rushing through tasks, disengaged from the potential insights these gray boxes could offer.

In my current professional role, I inherited workflows heavily dependent on a series of semi-autonomous gray boxes to calculate customer participation in an energy savings program called Connected Solutions. While the program itself is straightforward—customers take actions to provide the grid with more usable energy and are financially rewarded for their efforts—the processes behind the scenes were far from simple. The gray boxes we relied on were cumbersome, prone to failures, and often took weeks to produce results. The reports they generated were so unappealing that our sales director eventually stopped sharing them with customers altogether. Their only real value lay in post-season settlements and advocacy, with my team rarely using them to derive insights or engage meaningfully with customers throughout the season.

To address these challenges, I developed Watts, a modern GUI that has transformed our data processing and visualization capabilities. Watts leverages a suite of Python modules to efficiently extract, transform, load, and visualize data for analysis. The core problem addressed by Watts is the inefficiency and ineffectiveness of the gray box tools in engaging and providing actionable insights for both users and customers. These tools failed to deliver the level of visual engagement necessary to encourage meaningful data analysis. This shortcoming hindered our ability to engage with customers and make data-driven decisions, ultimately causing the Connected Solutions program to miss out on some potential participation.

Designing and Developing Simplicity

As Edward Tufte emphasized in The Visual Display of Quantitative Information, "The excellence of a visualization is related to how many ideas it can give its viewer while using the least amount of ink" (Tufte, 2001). I designed every aspect of Watts with this principle in mind. From my personal experience, a complex assortment of buttons and widgets can be intimidating, hard to remember, and prone to user errors. Some of the gray boxes I encountered featured date selectors, variable identifiers, and other metrics that cluttered the interface and detracted from the data. Complex button options distract users and diminish a GUI's storytelling potential.

In Watts, I significantly reduced the number of buttons and options an analyst needs to click through to complete tasks, compared to the gray boxes that came before me. For example, in one workflow an analyst had to make over 10 clicks to complete a calculation and a few more to get a report. All of these steps are now wrapped into just a few buttons. Doing this reduces the cognitive load on analysts, freeing their minds to focus on analyzing data. It might not seem like it, but the strain from deciphering buttons can greatly impact one's desire to engage with the tool fully.

The dashboards in Watts followed the same principle: reduce ink, increase encoding.

Watts was developed using PyQt5, a comprehensive framework for creating graphical user interfaces in Python. This choice allowed for the integration of multiple interactive elements—a data viewer for visualizing processed information, an integrated web browser for accessing external resources, and a data table for displaying and interacting with program data. These components work together to provide a cohesive and user-friendly interface for managing energy savings program data and workflows. Unlike VBA-based GUIs, PyQt5 offers greater flexibility, better performance, and a more modern look and feel, significantly enhancing the user experience. My idea was to allow a user to painlessly do as much work as possible inside Watts. If analysts have to leave Watts, they are distracting themselves and may not uncover everything the data has to tell.

Extracting, Processing, and Saving Data for Reusability

In the Energy Industry, the data of interest we are processing is known as load data, a customer's demand for energy. This data is collected in 1-15 minute intervals using specialized equipment and saved in the cloud somewhere. My company's IT team manages pipelines that feed copies of that data into our database.

To handle data retrieval and storage, Watts integrates with our database using cx_Oracle. This integration enables efficient querying and retrieval of meter data based on specific parameters such as asset IDs, date ranges, and meter tags. The retrieved data is then processed and analyzed using Pandas and NumPy. These tools facilitate complex data transformations, including filtering, grouping, aggregation, and time series analysis, which are crucial for accurate performance calculations in energy savings programs.

Watts utilizes a custom set of modules I developed to autonomously process data. Storing code in separate files and creating modules or your own folders of code is useful for modularizing your code and keeping it fresh and up to date.

Most energy saving programs work something like this: An event is called, and customers are asked to curtail their load. Baseline data is collected to gauge how much power a customer typically uses. The customer's load during the event is compared to the baseline to find performance. The specific details and flavors vary from region to region, but that is the gist.

Watts incorporates robust file and directory management using Python's OS and Glob modules. This ensures efficient handling of output files, maintaining an organized structure for the large volumes of data and reports generated by the system. The data is saved in files that are reusable for all work operations and can be easily shared with applicable third parties.

Data to PDFs

Watts was meant to improve customer engagement and participation with Connected Solutions. This required reviving the preliminary reporting process as something fresh, beautiful, and interesting. To facilitate this, I wanted to provide a few different reporting options. Watts can generate single-event, monthly, seasonal, and custom reports. These reports combine text and charts to present data to the customer in either a one-page summary or a full detailed brief. This provides sales reps flexibility to uniquely engage with customers at varying depths.

The summary chart as displayed above provides the customer with the most relevant information they need to understand how customers are doing at a macro level. One way to divide the type of summary pages is into single event summaries and multi-event summaries. Single event summaries compare assets to each other. However, multi-event summaries make a switch. They sum the assets and show the sum across time.

As Nathan Yau emphasizes in his work on visualizing uncertainty, 'Statistics is a game where you figure out these uncertainties and make estimated judgements based on your calculations. But standard errors, confidence intervals, and likelihoods often lose their visual space in data graphics, which leads to judgements based on simplified summaries expressed as means, medians, or extremes' (Yau, 2021). This observation is particularly relevant in the energy industry, where uncertainty is a pervasive challenge. Whether due to variations in customer behavior or fluctuations in energy supply and demand, it is often difficult for entities to replicate each other's work consistently. To address this, Watts integrates uncertainty into the performance PDFs by providing detailed ranges and confidence intervals. This approach ensures that our customers are not only informed of the central trends but are also aware of the inherent variability and potential outcomes, aligning with Yau's advice to avoid overly simplistic interpretations of complex data.

In the chart below this approach is particularly valuable in the narrative of the data. There are moment when the error ranges overlap between the adjusted baseline and the demand curve. This means that we cannot say with certainty that the utility will calculate performance during those times of overlap.

To present the processed data and visualizations in a polished format, Watts employs the reportlab and PyPDF2 libraries for PDF report generation and manipulation. This allows for the creation of detailed, customized reports tailored to different reporting needs, such as event-specific, monthly, or custom date range reports. These reports serve as powerful tools for communicating performance results to customers and stakeholders, significantly enhancing the actionability of the data compared to the previous Excel-based system. The visually appealing nature of these reports directly addresses the issue of unappealing outputs mentioned earlier, making the data more engaging and easier to interpret for customers.

Narrating with Dashboards

I developed an event summary dashboard and a geo-analysis dashboard with the goal of helping analysts and sales representatives understand what their customers' data is trying to convey. The idea here is simple: where is the most potential being lost? After working with the sales team for a while, I know they can get distracted by percentages and the quantity of some portfolios over the quality of others. I want to help them stay focused on the most valuable gains available.

The design of the Watts dashboards was heavily influenced by best practices in dashboard design, particularly those outlined by Plecto. Their advice to 'keep it simple' and 'use colors to improve understanding' (Plecto, 2023) guided the decision to employ smart coloring across all dashboards. This makes Watts both intuitive and immediately actionable. By simplifying the interface and focusing on essential data points, the dashboards not only enhance user engagement but also ensure that critical information is quickly and easily interpreted, reflecting Plecto's principles of effective dashboard design.

The Event Summary dashboard creates a heatmap of assets, allowing users to easily compare performance in the smallest units available across all other assets. This list is sorted by greater loss in potential performance and allows the analyst to easily see when a customer was participating. When a row in the map is clicked, detailed performance and baseline charts are displayed.

The Geo Analysis dashboard tells a similar story but with more emphasis on the differences in opportunities. The Geo analysis uses a map of the US to plot assets. The markers are encoded with color and size. Colors range from reds to yellows to greens. And the size indicates the loss in potential. This means the larger the circle, the more the customer could have curtailed. The advantage of this visual is simple, big circles demand more attention from the user than the sorting of a list (Tufte, 2001).

Early Adoption, Results, and Other Discussions

The success of Watts was defined by several key benchmarks: reducing data processing time and making preliminary reports actionable for Connected Solutions. These benchmarks were chosen to address the specific inefficiencies and user engagement issues identified in the legacy systems.

The implementation of the Watts application has resulted in significant improvements to our workflow processes. The time required to generate performance results has been drastically reduced, with processing times cut by approximately 80% compared to the previous Excel-based system. The improved workflow process has allowed the sales director to reapprove preliminary reporting for Connected Solution. Both major objectives completed.

The improved reports have had a notable impact on data analysis and sales strategies. For example, the highlighting of event times in the load charts has provided sales teams with valuable insights. This feature has enabled them to identify patterns in customer energy usage during peak demand periods, allowing for more targeted and effective recommendations. In one instance, this led to a 15% increase in customer participation in demand response events after tailored strategies were implemented based on the visualized data.

One ethical consideration in the development of Watts is the potential impact of increased automation on jobs within the organization. As more processes become automated, there may be a reduced need for certain manual tasks, which could affect employment. Additionally, the integration of AI and predictive analytics raises concerns about data privacy and the potential for biased decision-making, which must be carefully managed to avoid unintended consequences.

Future Work

Looking ahead, several enhancements are planned for Watts:

1. CO2 and Emissions Data: I aim to incorporate CO2 and emissions data into our reports, providing a more comprehensive view of the environmental impact of energy-saving efforts. This addition will allow customers to see not just their energy savings but also their contribution to reducing carbon emissions. I have completed a project combining LSTM models and random forest models to predict the State of New York's Energy demand with over 98% accuracy.
2. AI Integration: I plan to integrate chat functionality powered by artificial intelligence. This feature will allow users to ask questions about our industry, work functions, or even complete specific workflows through natural language interactions. This AI assistant could significantly enhance user productivity and provide instant access to relevant information and guidance.
3. Machine Learning for Predictive Analytics: I am exploring the use of machine learning algorithms to predict future energy usage patterns and potential savings opportunities. This predictive capability could help customers proactively manage their energy consumption and maximize their participation in energy-saving programs.

Statement of Work

I designed and developed all parts to this project. Shamelessly with the aid of AI. I have a public repo with a demo of some of the features of the application. There is dummy code available to play with as well. tsims2/Watts-MADS-Capstone (github.com).

APA Citation List

Evans, M. (2023, September 25). The 15 golden rules of dashboard design. Plecto. Retrieved August 12, 2024, from https://www.plecto.com/blog/dashboard-design/the-15-golden-rules-of-dashboard-design

Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.). Graphics Press.

Yau, N. (n.d.). Visualizing the uncertainty in data. FlowingData. Retrieved August 12, 2024, from https://flowingdata.com/uncertainty

OpenAI. (2024). ChatGPT (August 13 version) [Large language model]. Retrieved from [https://chatgpt.com]

Anthropic. (2024). Claude (Version 3.5) [Large language model]. Retrieved from [https://claude.ai/chat]

Appendix