For the past three centuries, charts, graphs, and other visual depictions have become essential tools for conveying quantitative information.
In numerous fields, from climate science to public health, effective data visualization is essential for conveying complex information clearly.
This challenge involves not only the proficient use of specialized software tools but, more importantly, a deep understanding of the data and design principles.
Edward Tufte, a leading expert in data visualization, emphasized this in his seminal work The Visual Display of Quantitative Information (1983):
"Graphical excellence is the well-designed presentation of interesting data—a matter of substance, of statistics, and of design."
Consider the example depicted in the figure, extracted from Savva (2011).
The pie chart presents data on the 2005 research budget of the National Institutes of Health (NIH).
There are several areas for improvement: the slices are randomly ordered, labels are inconsistently placed, and the text clashes with the vibrant background colors.
Additionally, the chart uses angular extents, which are less effective for accurate comparisons than position encodings.
The plot on the right presents the same data in a redesigned visualization: a sorted bar chart for easier comparison and ranking, using a color palette based on perceptual principles.
Creating effective visualizations requires a series of expert design choices,
yet ongoing efforts aim to develop systems that provide automatic support.
This trend is gaining momentum, as evidenced by the 2023 IEEE Visualization Conference,
which showcased techniques integrating AI and data visualization
Similarly, commercial software such as Microsoft’s Power BI and Einstein Copilot for Tableau are incorporating AI assistance in their interfaces.
However, they are primarily geared towards design applications and operate with their own data structures, preventing them from processing arbitrary visualization image formats.
This project aims to create an AI tool capable of automatically interpreting and providing feedback on scientific plots.
Our system leverages state-of-the-art visual language understanding techniques, such as developing a custom Vision Transformer (ViT) model trained on a
dataset specifically created for this task, as well as utilizing well-known datasets such as PlotQA,
to analyze and evaluate raster images of various plots, such as bar charts, line charts, and scatter plots.
No existing ViT models in the state-of-the-art can directly retrieve the design decisions made in a chart, such as variable-to-visual attribute mapping.
To bridge this gap, we developed a custom solution that leverages the MatCha model, pre-trained on chart derendering and Q&A tasks.
This combination has enabled us to extract both design attributes and data effectively. Our first prototype has shown promising results, demonstrating the capability to accurately distill design decisions.
VisDecode extracts key visual attributes including color, shape, positioning, and encoded data, which are critical for chart understanding and re-rendering.
Based on these outputs and best practices compiled from data visualization literature, the system will offer actionable suggestions to enhance plot design.
Additionally, users could apply these suggestions and re-render the plot within the tool, facilitating iterative improvements.
A significant advantage of this approach would be its framework-free nature, allowing scientists to continue using their preferred visualization tools while still benefiting from AI-driven enhancements.