1.0 Introduction

Part I. Foundational Philosophy and Concepts

1. The History Behind the Development of the VAS Software

In broad terms, the primary goal of analytic practice is “prediction” in the face of uncertainty that forms a rational basis for action. In 1931 Dr. Walter Shewhart described the scientific basis for prediction in his book Economic Control of Quality of Manufactured Product. Shewhart’s process behavior chart methodology focused on the analysis of the variation in the performance measurement of interest, and rational methods to separate out the “signals” in the observed data from the “noise” masking those signals. His primary focus was on applications in manufacturing.

In 1950 Dr. W. Edwards Deming, who was mentored by Dr. Shewhart, introduced the concept of “analytic studies” in Chapter 7 of his book Some Theory of Sampling. Throughout the remainder of his life he continued to expand Dr. Shewhart’s methodology to analytic studies related to the management and optimization of organizational performance in both business and industry.

In 2023 the book The Scientific Basis for Modern Analytic Practice (Bishop 2023)^[1] was published that describes the statistical concepts, theory and methods related to the rigorous formulation and the scientific basis for prediction in “analytic studies”. The methodology presented in that book is based on Dr. Shewhart’s and Dr. Deming’s pioneering work. In addition, the book provides a road map in the form of an “applied research protocol” which applied researchers can follow in the design and execution of analytic studies.

1.1 The Objectives Associated with the Development of the VAS Software¶

The Variation Analysis System (VAS) is an open-source, user-friendly software package found in the Python library under the name “Process Behavior”. It has been developed to give researchers in all fields of endeavor the ability to apply the methods of analysis presented in Dr. Bishop’s book. The applications supported by VAS include: applied research in the physical and social sciences; the improvement in organizational performance; the evaluation and improvement in employee performance; the design and implementation of statistical process control systems in manufacturing and administrative processes; and the analysis of data produced from experimental designs.

There are three primary objectives associated with the development of the VAS software system.

1. Expanding the opportunity to engage a much broader population of individuals in applied research and predictive analytics through the introduction of easily understood robust, inductive methods of statistical analysis.

2. Access to an open-source, user-friendly software system designed to provide the expanded population of researchers and analysts with the ability to apply the methods of analysis in practice without the need to become proficient in statistical theory or computer coding methodologies.

3. Improvement in the communication between researchers, data scientists, information technologists, subject matter experts and decision makers. This is achieved through the use of graphical methods for presenting the analytic results that are easily explained by the analyst and easily understood by the decision maker.

The remainder of this documentation manual presents the scientific philosophy, and the statistical concepts, theory, methods and algorithms forming the basis for VAS. If you are familiar with the works described in Section 1.1, you should read Chapter 4 and then skip to Part VII which begins the specific discussion of the heuristic, analytic methodology executed by the VAS software system.

1.2 The Fundamental Problem is Variation¶

In all the applications the empirical data forming the basis for prediction are produced by some type of physical or social “process”. Every process is associated with a set of “causes” that create variation in the data. The set of causes is referred to as a “cause system of variation”. The variation produced by the cause system is a fundamental problem for analysts and decision makers because it creates uncertainty in the predictions, and therefore risk in the actions taken by the decision makers. The goal in many applications is the discovery and control of the causes of variation affecting the data.

1.3 Recovering Signals from Noise¶

In non-technical terms the primary goal of analytic practice is often referred to as the discovery of “signals” contained in the data that are hidden by the “noise” in the data. The extraction of the signals from the noise depends, in part, upon our ability to identify and understand the “sources” of the causes of variation, and the nature of the variation the causes produce in the response variable of interest.^[2] Such knowledge allows us to determine if the causes produce “controlled variation” in which case prediction may be possible, or whether the causes produce “uncontrolled variation” in which case prediction is difficult at best, and at worst, is not even possible.^[3]

When the variation is “controlled” knowledge of the potential causes of variation allows the analyst to:

1. develop statistical models for the response variable of interest;

2. estimate the effects of practically important causes of variation on the response variable;

3. make predictions about the behavior of the response variable in time and space.

In some cases the knowledge created also allows the analyst to:

1. exploit important causes of deterministic and stochastic variation to the benefit of the decision maker;

2. remove unwanted causes of variation from the process;

3. monitor future data to detect changes in the process behavior due to:

   a. purposeful changes made to the design of the process, and

   b. the introduction of causes of variation producing unwanted, uncontrolled variation.

1.4 The Need for Employing a Rigorous Problem Formulation Methodology¶

Today’s analytic world is dominated by AI, and the development of sophisticated statistical and IT algorithms. Some claim that in this new world rigorous problem formulation methodologies are now of secondary importance. But the risks introduced by the uncertainty associated with variation, and the ease of executing analytic methods without a detailed understanding of the cause system producing the data, makes it even more important that analysts develop predictive models based on well-developed critical thinking skills, rigorous problem formulation and scientific principles.

1.5 The Structure of Analytic Studies¶

VAS is designed to analyze data produced within the context of the “analytic study”. The distinguishing feature of the analytic study is that it involves a temporal component related to making predictions about the future based on past experience. That is, inferences or predictions in the analytic study are “extrapolations” into the future based on the analysis of historical data. The theory and inductive methods of analysis also apply to the case where the extrapolations are in space.

In the analytic study data are obtained from samples taken from the physical or social process producing the data. The initial focus of the analysis is on understanding the sources and causes of variation affecting the available data. Based on this information it can be determined if the variation in the data is sufficiently “controlled” to make predictions, at least in the short term, about the properties of future data that will be produced by the process. Decisions and actions are based on these predictions. Almost all applications encountered in practice involve analytic studies.^[4]

1.6 The Heuristic Nature of the VAS Methodology¶

There is a significant body of statistical literature that presents many methods of data analysis that are based on deductively valid probability and statistical theory. It is often argued that these methods provide the analyst with the ability to measure and control risk with the use of probability theory. But in analytic studies the development of predictive models forming the basis for decision making is an inductive, experiential process. The predictions involve extrapolations from the current data into the future or space. The application of probability theory to measure risk is based on a number of assumptions, particularly the assumption that the cause system of variation producing the historical data will continue function in the same way in the future or in space. But the inductive process of predicting future events involves the collection and analysis of empirical data that very often do not conform to these assumptions.^[5]

In most analytic studies encountered in practice, the uncertainties and risks associated with data analysis cannot be directly measured or controlled using probability theory.^[6] It is often the case that the claimed accuracy and precision of specific parameter estimates, and the calculation of related measures of risk based on probability theory are a mathematical exercise not justified by fact. In addition, many of the standard statistical methods and modern computer algorithms are mathematically complicated which reduces the number of individuals who can understand the methods of analysis and become productive analysts. This complexity also creates significant communication problems between analysts, information technologists, subject matter experts and the decision makers.

A heuristic technique is any approach to problem solving or discovery that employs a practical method that is not guaranteed to be perfect, optimal or even rational, but is nevertheless a sufficient approximation to achieve the analytic goal. In the case of VAS, the analytic goal is the separation of the practically important signals from the noise in the data, the determination of whether the variation is controlled or uncontrolled, and the assessment of the contribution of specific causes of variation to the total variation observed in the data.

While the techniques forming the basis for the VAS methodology are grounded in deductively valid probability and statistical theory, the methods are heuristic in nature.^[7] The application of heuristic methods to analytic studies is justified in two important ways. First, it is justified based on the philosophy expounded by Dr. Walter Shewhart in 1931.

“The fact that the criterion which we happen to use has a
fine ancestry of highbrow statistical theorems does not justify its use.
Such justification must come from empirical evidence that it works.”

Second, as explained by Dr. W. Edwards Deming, there is often a large gap between deductively valid statistical theory and practice.

“Graduates in mathematical statistics, when taking up practice, discover yawning gaps
between theory and practice: the better their theoretical training, the wider the gap.”

So, in practice, there is no guarantee that any methodology, even if based on deductively valid theory, can produce risk-free results. There is no guarantee that the methodology executed by VAS produces optimal or theoretically correct results, only that it produces practically useful results in many applications.^[8]

And finally, we base the justification of the methods employed within VAS based on the philosophy of Dr. Albert Einstein.

“Everything should be made as simple as possible, but not simpler.”