2.0 The Scientific Basis for Prediction in Analytic Studies

Part II. The Scientific Philosophy Supporting the VAS Methodology

As noted above, the methods of analysis and algorithms executed by VAS are based on the philosophy of science, the scientific basis for prediction, and the inductive methods of data analysis presented by Dr. Walter Shewhart in his 1931 book Economic Control of Quality of Manufactured Product.^[9] The analytic methods used within VAS are linked to deductively valid statistical theory, including linear model theory, but only as “weak conditional” support for the heuristic, inductive analyses described in this manual.^[10]

Many classical methods of statistical analysis depend upon probability theory as the scientific basis for measuring and controlling the risks associated with inferences and predictions. Such theory is applicable for “enumerative” studies where the inference or prediction is an interpolation from the sample back into the frame from which the sample was selected.^[11] But inferences and predictions in the analytic study are extrapolations from historical sample(s) into the future or space. The use of probability theory as the scientific basis for measuring and controlling risk requires the assumption that “the future behaves like the past” to be true in fact. This assumption cannot be justified in most analytic studies. Probability theory is useful in creating guidelines for analysis in the analytic study. But the scientific basis for prediction is not probability theory, rather it is the execution of the scientific method augmented with the “weak conditional” use of probability theory, statistical methods and modern algorithmic techniques. And the basis for building credibility in predictions and inferences within the scientific method is not probability theory, but the nontrivial replication of results.^[12]

2.1 The Four Fundamental Principles for Prediction in Analytic Studies¶

In analytic studies historical data that are subject to a set of causes of variation are used to make predictions about the future. But as Dr. Shewhart noted in 1931 just because data are subject to variation does not necessarily mean that probability theory and statistical methods can be used for prediction.

“What can we say about the future behavior of a phenomenon acting

under the influence of unknown or chance causes?”

“I doubt that, in general, we can say anything.”

For prediction to be possible in an analytic study a reasonable degree of “stability” or “control” must exist over time or space with respect to the sources and causes of variation producing the empirical data forming the basis for the analysis and predictions. Stated another way, for prediction to be possible the physical or social process under study must produce both the historical and future data “under essentially the same conditions”. Dr. Shewhart went on to emphasize the following point of logic.

“Hence, if we are to be able to make predictions even within limits, we must find some criterion to apply to observed variability in the data to determine whether or not the cause system producing it is such that as to make future predictions possible.”

Dr. Shewhart’s counsel, and an understanding of the conditional role of probability theory in analytic studies, leads to four fundamental principles forming the scientific basis for making predictions in analytic studies.

1. The principle of process design.

2. The principle of controlled variation.

3. The principle of testing predictions.

4. The principle of the nontrivial replication of results.

2.1.1 The Principle of Process Design¶

The first fundamental principle relates to the fact that the sources and causes of variation, and the type of variation they produce in the response variable of interest, depend upon the design of the physical or social process producing the data. Therefore we state the first fundamental principle of analytic studies as follows.

“The sources and causes of variation, and the nature of the variation they produce in the empirical data forming the basis for prediction, are directly linked to the design of the process producing the data and the laws, or lack of laws, governing the system.”

Identifying important sources and causes of variation, and understanding the nature of the variation they produce in the empirical data forming the basis for prediction, almost always involves the direct analysis of the design of the process under study.^[13]

2.1.2 The Principle of Controlled Variation¶

The second principle relates to the requirement of control.

“The variation in the empirical data forming the basis for prediction must

be controlled in the sense that the nature of the causes of variation does not change in

practically significant ways over time or space.”

When the causes of variation are acting in a controlled manner over time the variation they produce is predictable at least within limits. Theoretically, controlled variation can be described by a statistical distribution that does not change over time. In this case mathematical, probability and statistical theory are applicable, and the foundation for prediction is based on the conditional application of deductively valid and sound methods of statistical analysis.^[14]

2.1.3 The Principle of Testing Predictions¶

One of the cornerstone principles of the scientific approach to analytic studies is that any prediction resulting from algorithms produced by mathematical or statistical models must be “testable”. As noted by Sir Karl Popper:

“Only a prediction that is testable is scientifically acceptable.” ^[15]

The required testing can take many forms depending upon the application, but the data used in the test must not come from the same set of data that produced the prediction. The data forming the test must be based on new data produced by the process in the future or space.

In the case of testing statistical models much has been written about the “validation” of statistical models. Model validation is often based on the analysis of “model residuals” and/or “holdout samples” selected from the data used to formulate the statistical model. However, the model residuals and holdout sample data come from the cause system in its current state. The analysis of model residuals or holdout samples are therefore a form of “interpolation” and do not provide the logically rigorous testing required by the scientific method. What is required is the comparison of predictions to actual outcomes produced by the cause system of variation in the future. ^[16]

2.1.4 The Principle of the Nontrivial Replication of Results¶

The fourth fundamental principle relates to the method used to build credibility in inferences and predictions derived from the underlying models or working theories created from the analysis of historical data. The development of credibility in analytic studies is based on the “nontrivial replication of results over time and space”. While in some applications it is somewhat difficult to operationally define the concept of the nontrivial replication of results, and in some cases it is a judgment call by the analyst, we usually “know it when we see it”. The fourth fundamental principle of analytic studies is stated in the following way.

“The nontrivial replication of results is the confirmatory tool used in
analytic studies to build credibility in inferences and predictions.”

In summary, the scientific philosophy and fundamental principles on which the VAS methodology rests leads to the following analytic precepts.

1. The majority of applied research deals with analytic studies.

2. The primary goal of an analytic study is prediction in time or space.

3. The fewest number of assumptions supported by empirical evidence should be posited that describe the structure of the data forming the basis for analysis and prediction.

4. Positing assumptions to make the mathematics easier without supporting empirical evidence is not scientifically sound.

5. The simplest analytic method required to analyze the data should be used, but not simpler.

6. Predictions produced by analytic models must be testable.

7. Credibility in the inferences and predictions forming the basis for decision making is not based on probability calculations, but on the nontrivial replication of results.^[17]

8. Decisions are not made on the basis of statistical analyses alone, but often involve important information “external” to the data being analyzed that can have a profound influence on the decisions made, and actions taken, by the decision maker.^[18]