Since the publication of the NSF landmark report "Visualization in Scientific Computing" in 1987, computer-aided visualization has been recognized as one of the most potent tool sets for scientific discovery. However, discoveries based on data displays are often criticized because they are not secured by statistical inference. The team of researchers from Iowa State University, Rice University and University of Pennsylvania is addressing exactly this issue by bringing the rigors of statistical inference to visual data exploration. Statistical inference for plots are cast as comparison of a plot of the actual data with plots of null data simulated under a null hypothesis. If the actual plot stands out from a background of "null plots", it amounts to the rejection of the null hypothesis. Executing this idea leads to rigorous protocols that can confer proper statistical significance to visual discoveries. Tools of mathematical statistics are employed to reduce composite null hypotheses to single reference distributions: conditioning on a minimal sufficient statistic, bootstrap plug-ins, and posterior predictive sampling. The protocols also have the potential to shift the perception of exploration-based findings in the scientific communities and dramatically increase the impact that these findings are allowed to have. The testing protocols will be made accessible with implementation in the open-source R language.
Data graphics are an essential part of communicating information. But how reliable is the information that we gather from them? The investigators will develop a rigorous framework for visual inference modeled after formal statistical testing. This framework allows the reader of a graphic to determine whether structure is real or spurious (is that a man in the moon, or just some rocks?). These protocols have the potential to shift the perception of exploration-based findings in the scientific community and dramatically increase the impact of exploratory work. Some aspects of the protocols are so intuitive that they can be used for general audiences and integrated in the teaching of introductory statistics at from grade school to college.
Which of these things is not like the other ones? -- People are shown a plot of the data amongst a field of "null plots" and asked to answer this main question, playing a game similar to the one made popular on Sesame Street. Using this approach, this research has established that the human visual system can work similarly to statistical hypothesis tests, which provides huge potential for improving understanding about variability in data amongst the general community, for tackling problems that cannot be solved by classical statistical tests, and providing ways to improve data visaulization techniques. The research evaluated the use of the lineup protocol as a visual test: a chart of the real data is inserted in a set of, usually 19, charts created from essentially random data. If a human observer can pick the data plot from this lineup, this is evidence towards concluding that the data has some non-random element(s) in it. The lineup protocol provides a framework to quantify the strength of a graphical findings in a probabilistic manner: if the data really is random, the chance that an observer accidentally picks the data plot is 1 in 20. Using Amazon's MTurk service, we conducted experiments to study different features and uses of the lineup protcol. Our first three experiments validated the lineup protocol in the setting of normal models: they showed that in highly controlled situations the power of conventional tests is superior to the lineup protocol, but the lineup protocol shows the same kind of power structure: power of lineups increases as the signal in the data increases. In situations, where the power of conventional tests is severely affected by noise, the results from lineup tests are not affected, because the human perceptual system filters noise from the displays to emphasize the main relationship. The lineup protocol produces results consistent with practical significance, where the effect size is effectively what people respond on. The lineup protocol also allows us to compare the effectiveness of competing data plot designs. Our studies have shown us that polar charts, such as pie-charts, are much less effective than rectangular charts, such as histograms or barcharts. Boxplots have also proven to be effective in a multitude of situations, and together with jittered dotplots for small sample sizes, boxplots were the most effective display for people to perceive location shifts between samples. Among the most common variations of boxplots, the traditional boxplot design turned out to be the most versatile tool in comparing distributions. The sine illusion is a well-known situation tricking the human perception into perceiving varying differences between curves that are, in fact, parallel. Using results from a lineup study we were able to quantify the extent of the illusion and measure the amount of correction necessary to counteract its effects. These effects are important in problems associated with reading time series plots, and particularly differences between series, such as stock prices for two companies, or exchange rates of two currencies. Many data problems today, particularly in bioinformatics brings data that have more variables than samples, for example, many thousands of gene expression measurements are recorded on small numbers of plant tissues samples. This produces sparse data where all sorts of patterns can occur by chance. Our research established that the lineup protocol can be used to gauge just how much separation between groups can occur by chance in gene expression data. The application that we examined was on differences between species of wasps, but it could be extended to problems like differences between malignant and benign tumors in human tissue. Many statistical models rely on data plots as one of the major methods to assess the model fit. The lineup protocol provides these diagnostics in a rigorous framework, and allows us to check assumptions and investigate output of complex statistical models, such as hierarchical linear models. We can use lineups for diagnostics of residual structures and assess residual distributions. The research has established that visual diagnostics using lineups of residual structures are as powerful as conventional tests, but have fewer constraints and are therefore applicable in a wider range of situations. Furthermore, leveraging the human perceptual system, they also prove to be more robust against data impurities such as outliers.
