Toward Norms for the Development of Models

Today, modeling and data issues pervade social science research. Likewise, the discipline of finance is now confronting issues of model risk. Assuming the nature of financial economics is similar to that of the social sciences, then perhaps the former might take a lesson from the latter. Prof W James Bradley and Prof Kurt C Schaefer offer the following “norms” in order that researchers might avoid the misue of models and data (1998, pp. 145-151):

1. For each situation under analysis, practitioners benefit from a detailed understanding of the situation, from different disciplinary perspectives when possible, before the problem is modeled.

2. Practitioners should learn what is important enough to measure before trying to measure it, then decide which of the five measurement scales [i.e., nominal, ordinal, interval, ratio, absolute] is appropriate. One must then live within the bounds imposed by the characteristics of that measurement scale.

3. Random error terms convey information about the abstractions, approximations, ignorance, and measurement problems that are involved in the model we have constructed. The drawing of inferences should therefore involve a careful inspection of the residual errors between our data and our model.

4. “Probabilities” in the social and human sciences are degrees of warranted belief, not relative frequencies. But this means that the classical ratio scale of probabilities is not appropriate in the situations we are discussing. Therefore, the level of confidence in a result cannot be stated as a single number; significance involves a judgment about the reasonableness of the entire model and its data.

5. Levels of statistical significance are always somewhat arbitrary, but we should be especially skeptical in cases when (a) the social processes under study are extremely complex, with many auxiliary hypotheses complicating the primary hypotheses; (b) the entity being measured is not clearly definable, or there is a poorly developed theory of the entity and its relationship to the measureable variables, or the measurement instrument is not precise and reliable; (c) inappropriate measurement scales are used; (d) the statistical methods (and, when present, functional forms) employed are not consistent with the measurement scale; (e) the specification of the model and its functional form are not clearly justified by reference to the actual situation being modeled; (f) the error residuals are not observed and analyzed (e.g., some ANOVA and correlation studies); and (g) the quality of the data and reliability of the source are questionable. We should be particularly skeptical when the analyst does not fully disclose the relevant information on these topics. In fact, it should be a professional norm that the statement of one’s results must, as a matter of habit, discuss these details.

Reference: Bradley, W J & Schaefer, K C (1998), The Uses and Misuses of Data and Models: The Mathematization of the Human Sciences, Thousand Oaks, CA: Sage.