The Power of Story - The Story Paradigm

by Tom Atlee © Co-Intelligence.org

In the field of co-intelligence, stories are more than dramas people tell or read. Story, as a pattern, is a powerful way of organizing and sharing individual experience and exploring and co-creating shared realties. It forms one of the underlying structures of reality, comprehensible and responsive to those who possess what we call narrative intelligence. Our psyches and cultures are filled with narrative fields of influence, or story fields, which shape the awareness and behavior of the individuals and collectives associated with them.

Story-reality is the reality that we see when we recognize that every person, every being, every thing has a story and contains stories -- and, in fact, is a story -- and that all of these stories interconnect, that we are, in fact, surrounded by stories, embedded in stories and made of stories. When poet Murial Rukeyser tells us "the universe is made of stories, not atoms," she's describing story-reality. Ultimately, story-reality includes any and all actual events and realities, but experienced as stories, not as the more usual patterns -- objects-and-actions; matter, energy, space, time; patterns of probability; etc. Story-reality is made up of lived stories.

Lived stories are those real-life, actual stories that are happening in the real world all around us all the time. The actual unfolding events relating to any one actual entity or subject comprise that entity's or subject's lived story. Everything that exists has, embodies and participates in many lived stories. The way to co-intelligently engage in story-reality is to become sensitive to lived stories... to learn about the lived stories of people, places, things... to share our own lived stories... to discover how all these stories intersect, who or what is in the foreground and background of each other's lived stories. Ultimately, this provides the guidance we need to find our own most meaningful place in the universal story.

While analysis is good for control and prediction, story-sensibility is good for understanding meaning and role. [italics added]

Narrative intelligence is the ability (or tendency) to perceive, know, think, feel, explain one's experience and influence reality through the use of stories and narrative forms.

It includes:

• the ability and tendency to organize experience and ideas using stories and narrative patterns (an excellent example of this is the use of myth, which defines and discusses concepts -- such as archetypes -- in narrative form)
• the tendency to understand things better when they are presented in the form of a story (and sometimes to have trouble understanding things when they aren't presented as stories)
• the capacity to sense the importance of context, character, history, etc., in any explanation -- and dissatisfaction when these are omitted
• dissatisfaction with isolated events and abstract ideas, out of context
• an ability to sense or imagine the stories of people, objects, places; the ability to accurately guess where something (or someone) comes from, what has happened to it, where it is going, what it means
• curiosity about the stories behind things, and an ability to investigate such stories
• a tendency to make up stories, plausible or fantastic, to illustrate a point
• the ability to maintain a repertoire of stories (real and imaginary) to convey meanings; the ability to access that repertoire
• the ability to sort out and describe what has happened to oneself or others, often with a richness of context and detail, and often with great relish
• the ability to place and remember events in sequence
• the ability to envision chains and webs of causation
• the tendency to build scenarios (stories of possibilities); an ability to plan and think strategically
• a love of stories
• the ability and tendency to see people, places and things in terms of their function in a story (very helpful for novelists picking up tidbits from the lives around them for use in their creative work)
• resonance with the stories of others; the ability to see another's viewpoint when presented with the stories which underlie or embody that viewpoint
• the ability to discover themes in the events of a life or story
• the ability to recognize (or select) certain elements as significant, as embodying certain meanings that "make sense of things"
• the ability to build a story out of randomly-selected items
• the ability to use stories as memory-enhancing devices (such as remembering a phone number by making the digits into characters and weaving them into a story).

Story fields are fields of influence or patterns of dynamic potential that permeate psycho-social space and influence the lives of those connected to them. They are made up of many mutually-reinforcing stories (myths, news, soap operas, lives, memories) and story-like phenomena (roles, metaphors, archetypes, images). A story field paints a particular picture of how life is or should be, and shapes the life within its range into its image.

The American Way of Life is a powerful story field, which includes everything from principles like freedom and the pursuit of happiness, to stories of cowboys and rags-to-riches heroes, to metaphors like the melting pot and the safety net, to images like the Statue of Liberty and the flag. It is communicated by movies, men in business suits, advertisements, college catalogues, and mall displays -- among many, many other things. It takes immense effort to resist or change it. Anyone or anything which doesn't live within this story-sea and move with its currents doesn't seem quite American.

Psychological, organizational or social transformation is usually preceded or accompanied by a change in the story field governing that system. It is therefore usually non-productive to try to change forms and habits without changing the story fields that hold them in place. Once the story field is changed, subsidiary patterns tend to realign rapidly. (This process is part of what has been called a paradigm shift.)

Co-intelligent cultural transformation necessarily includes the co-generation of co-intelligent story fields. This would include examples of co-intelligence in action, visions of how things could be more co-intelligent, biographies of co-intelligent people, fiction illustrating the dynamics of co-intelligence, co-intelligent myths and poems, academic reframing of numerous other subjects in terms of co-intelligence, people actually living co-intelligently, the clarification and use of special roles (like elder and partner) associated with co-intelligence, etc.

Reproduced with permission of Co-Intelligence.org

Evidence of Indeterminism

As a graduate student during the 1980's, I became fascinated by the colorful, but seemingly chaotic wave patterns exhibited by electricity in plasma bulbs. I soon concluded that the eratic wave patterns provided intuitive evidence of indeterminism in nature (though not proof). Thirty years later, I am still persuaded that indeterminism trumps determinism as a guiding philosophical proposition. This conclusion eventually shaped my views and approach to financial economics leading to an advocacy of stochastic modeling methodologies in research and practice.

I recorded the video that follows using a plasma lamp from my study and post it here for others to view and ponder. Notice how the waves respond as my finger touches the globe.



Follow the link below for primers on determinism and indeterminism, as well as metaphysics.

Dialogos of Eide

Visualize This

Visualize This by Nathan Yau (Wiley, 2011) is mandatory reading for business intelligence specialists -- more at:



According to the publisher:

Data doesn't decrease; it is ever-increasing and can be overwhelming to organize in a way that makes sense to its intended audience. Wouldn't it be wonderful if we could actually visualize data in such a way that we could maximize its potential and tell a story in a clear, concise manner? Thanks to the creative genius of Nathan Yau, we can. With this full-color book, data visualization guru and author Nathan Yau uses step-by-step tutorials to show you how to visualize and tell stories with data. He explains how to gather, parse, and format data and then design high quality graphics that help you explore and present patterns, outliers, and relationships.

• Presents a unique approach to visualizing and telling stories with data, from a data visualization expert and the creator of flowingdata.com, Nathan Yau
• Offers step-by-step tutorials and practical design tips for creating statistical graphics, geographical maps, and information design to find meaning in the numbers
• Details tools that can be used to visualize data-native graphics for the Web, such as ActionScript, Flash libraries, PHP, and JavaScript and tools to design graphics for print, such as R and Illustrator
• Contains numerous examples and descriptions of patterns and outliers and explains how to show them

Follow the links below to learn more.

Visualize This

Patterns of Beauty in Finance

This is an occasional paper I released arguing that there is beauty in the patterns discovered by financial economists that transcends symbolism and nominalism in what only the mind can understand within its conceptual space.

Download

101 Things Every Six Sigma Black Belt Should Know

by Thomas Pyzdek, Copyright © 2003

1. In general, a Six Sigma Black Belt should be quantitatively oriented.
2. With minimal guidance, the Six Sigma Black Belt should be able to use data to convert broad generalizations into actionable goals.
3. The Six Sigma Black Belt should be able to make the business case for attempting to accomplish these goals.
4. The Six Sigma Black Belt should be able to develop detailed plans for achieving these goals.
5. The Six Sigma Black Belt should be able to measure progress towards the goals in terms meaningful to customers and leaders.
6. The Six Sigma Black Belt should know how to establish control systems for maintaining the gains achieved through Six Sigma.
7. The Six Sigma Black Belt should understand and be able to communicate the rationale for continuous improvement, even after initial goals have been accomplished.
8. The Six Sigma Black Belt should be familiar with research that quantifies the benefits firms have obtained from Six Sigma.
9. The Six Sigma Black Belt should know or be able to find the PPM rates associated with different sigma levels (e.g., Six Sigma = 3.4 PPM).
10. The Six Sigma Black Belt should know the approximate relative cost of poor quality associated with various sigma levels (e.g., three sigma firms report 25% COPQ).
11. The Six Sigma Black Belt should understand the roles of the various people involved in change (senior leader, champion, mentor, change agent, technical leader, team leader, facilitator).
12. The Six Sigma Black Belt should be able to design, test, and analyze customer surveys.
13. The Six Sigma Black Belt should know how to quantitatively analyze data from employee and customer surveys. This includes evaluating survey reliability and validity as well as the differences between surveys.
14. Given two or more sets of survey data, the Six Sigma Black Belt should be able to determine if there are statistically significant differences between them.
15. The Six Sigma Black Belt should be able to quantify the value of customer retention.
16. Given a partly completed QFD matrix, the Six Sigma Black Belt should be able to complete it.
17. The Six Sigma Black Belt should be able to compute the value of money held or invested over time, including present value and future value of a fixed sum.
18. The Six Sigma Black Belt should be able to compute present value and future value for various compounding periods.
19. The Six Sigma Black Belt should be able to compute the breakeven point for a project.
20. The Six Sigma Black Belt should be able to compute the net present value of cash flow streams, and to use the results to choose among competing projects.
21. The Six Sigma Black Belt should be able to compute the internal rate of return for cash flow streams and to use the results to choose among competing projects.
22. The Six Sigma Black Belt should know the COPQ rationale for Six Sigma (i.e., the Six Sigma Black Belt should be able to explain what to do if COPQ analysis indicates that the optimum for a given process is less than Six Sigma).
23. The Six Sigma Black Belt should know the basic COPQ categories and be able to allocate a list of costs to the correct category.
24. Given a table of COPQ data over time, the Six Sigma Black Belt should be able to perform a statistical analysis of the trend.
25. Given a table of COPQ data over time, the Six Sigma Black Belt should be able to perform a statistical analysis of the distribution of costs among the various categories.
26. Given a list of tasks for a project, their times to complete, and their precedence relationships, the Six Sigma Black Belt should be able to compute the time to completion for the project, the earliest completion times, the latest completion times and the slack times. The Six Sigma Black Belt should also be able to identify which tasks are on the critical path.
27. Give cost and time data for project tasks, the Six Sigma Black Belt should be able to compute the cost of normal and crash schedules and the minimum total cost schedule.
28. The Six Sigma Black Belt should be familiar with the basic principles of benchmarking.
29. The Six Sigma Black Belt should be familiar with the limitations of benchmarking.
30. Given an organization chart and a listing of team members, process owners, and sponsors, the Six Sigma Black Belt should be able to identify projects with a low probability of success.
31. The Six Sigma Black Belt should be able to identify measurement scales of various metrics (nominal, ordinal, etc).
32. Given a metric on a particular scale, the Six Sigma Black Belt should be able to determine if a particular statistical method should be used for analysis.
33. Given a properly collected set of data, the Six Sigma Black Belt should be able to perform a complete measurement system analysis, including the calculation of bias, repeatability, reproducibility, stability, discrimination (resolution) and linearity.
34. Given the measurement system metrics, the Six Sigma Black Belt should know whether or not a given measurement system should be used on a given part or process.
35. The Six Sigma Black Belt should know the difference between computing sigma from a data set whose production sequence is known and from a data set whose production sequence is not known.
36. Given the results of an AIAG Gage R&R study, the Six Sigma Black Belt should be able to answer a variety of questions about the measurement system.
37. Given a narrative description of “as-is” and “should-be” processes, the Six Sigma Black Belt should be able to prepare process maps.
38. Given a table of raw data, the Six Sigma Black Belt should be able to prepare a frequency tally sheet of the data, and to use the tally sheet data to construct a histogram.
39. The Six Sigma Black Belt should be able to compute the mean and standard deviation from a grouped frequency distribution.
40. Given a list of problems, the Six Sigma Black Belt should be able to construct a Pareto Diagram of the problem frequencies.
41. Given a list which describes problems by department, the Six Sigma Black Belt should be able to construct a cross tabulation and use the information to perform a Chi-square analysis.
42. Given a table of x and y data pairs, the Six Sigma Black Belt should be able to determine if the relationship is linear or non-linear.
43. The Six Sigma Black Belt should know how to use non-linearity’s to make products or processes more robust.
44. The Six Sigma Black Belt should be able to construct and interpret a run chart when given a table of data in time-ordered sequence. This includes calculating run length, number of runs and quantitative trend evaluation.
45. When told the data are from an exponential or Erlang distribution the Six Sigma Black Belt should know that the run chart is preferred over the standard X control chart.
46. Given a set of raw data, the Six Sigma Black Belt should be able to identify and compute two statistical measures each for central tendency, dispersion, and shape.
47. Given a set of raw data, the Six Sigma Black Belt should be able to construct a histogram.
48. Given a stem & leaf plot, the Six Sigma Black Belt should be able to reproduce a sample of numbers to the accuracy allowed by the plot.
49. Given a box plot with numbers on the key box points, the Six Sigma Black Belt should be able to identify the 25th and 75th percentile and the median.
50. The Six Sigma Black Belt should know when to apply enumerative statistical methods, and when not to.
51. The Six Sigma Black Belt should know when to apply analytic statistical methods, and when not to.
52. The Six Sigma Black Belt should demonstrate a grasp of basic probability concepts, such as the probability of mutually exclusive events, of dependent and independent events, of events that can occur simultaneously, etc.
53. The Six Sigma Black Belt should know factorials, permutations and combinations, and how to use these in commonly used probability distributions.
54. The Six Sigma Black Belt should be able to compute expected values for continuous and discrete random variables.
55. The Six Sigma Black Belt should be able to compute univariate statistics for samples.
56. The Six Sigma Black Belt should be able to compute confidence intervals for various statistics.
57. The Six Sigma Black Belt should be able to read values from a cumulative frequency ogive.
58. The Six Sigma Black Belt should be familiar with the commonly used probability distributions, including: hypergeometric, binomial, Poisson, normal, exponential, Chi-square, Student’s t, and F.
59. Given a set of data the Six Sigma Black Belt should be able to correctly identify which distribution should be used to perform a given analysis, and to use the distribution to perform the analysis.
60. The Six Sigma Black Belt should know that different techniques are required for analysis depending on whether a given measure (e.g., the mean) is assumed known or estimated from a sample. The Six Sigma Black Belt should choose and properly use the correct technique when provided with data and sufficient information about the data.
61. Given a set of subgrouped data, the Six Sigma Black Belt should be able to select and prepare the correct control charts and to determine if a given process is in a state of statistical control.
62. The above should be demonstrated for data representing all of the most common control charts.
63. The Six Sigma Black Belt should understand the assumptions that underlie ANOVA, and be able to select and apply a transformation to the data.
64. The Six Sigma Black Belt should be able to identify which cause on a list of possible causes will most likely explain a non-random pattern in the regression residuals.
65. If shown control chart patterns, the Six Sigma Black Belt should be able to match the control chart with the correct situation (e.g., an outlier pattern vs. a gradual trend matched to a tool breaking vs. a machine gradually warming up).
66. The Six Sigma Black Belt should understand the mechanics of PRE-Control.
67. The Six Sigma Black Belt should be able to correctly apply EWMA charts to a process with serial correlation in the data.
68. Given a stable set of subgrouped data, the Six Sigma Black Belt should be able to perform a complete Process Capability Analysis. This includes computing and interpreting capability indices, estimating the % failures, control limit calculations, etc.
69. The Six Sigma Black Belt should demonstrate an awareness of the assumptions that underlie the use of capability indices.
70. Given the results of a replicated 22 full-factorial experiment, the Six Sigma Black Belt should be able to complete the entire ANOVA table.
71. The Six Sigma Black Belt should understand the basic principles of planning a statistically designed experiment. This can be demonstrated by critiquing various experimental plans with or without shortcomings.
72. Given a “clean” experimental plan, the Six Sigma Black Belt should be able to find the correct number of replicates to obtain a desired power.
73. The Six Sigma Black Belt should know the difference between the various types of experimental models (fixed-effects, random-effects, mixed).
74. The Six Sigma Black Belt should understand the concepts of randomization and blocking.
75. Given a set of data, the Six Sigma Black Belt should be able to perform a Latin Square analysis and interpret the results.
76. Ditto for one way ANOVA, two way ANOVA (with and without replicates), full and fractional factorials, and response surface designs.
77. Given an appropriate experimental result, the Six Sigma Black Belt should be able to compute the direction of steepest ascent.
78. Given a set of variables each at two levels, the Six Sigma Black Belt can determine the correct experimental layout for a screening experiment using a saturated design.
79. Given data for such an experiment, the Six Sigma Black Belt can identify which main effects are significant and state the effect of these factors.
80. Given two or more sets of responses to categorical items (e.g., customer survey responses categorized as poor, fair, good, excellent), the Six Sigma Black Belt will be able to perform a Chi-Square test to determine if the samples are significantly different.
81. The Six Sigma Black Belt will understand the idea of confounding and be able to identify which two factor interactions are confounded with the significant main effects.
82. The Six Sigma Black Belt will be able to state the direction of steepest ascent from experimental data.
83. The Six Sigma Black Belt will understand fold over designs and be able to identify the fold over design that will clear a given alias.
84. The Six Sigma Black Belt will know how to augment a factorial design to create a composite or central composite design.
85. The Six Sigma Black Belt will be able to evaluate the diagnostics for an experiment.
86. The Six Sigma Black Belt will be able to identify the need for a transformation in y and to apply the correct transformation.
87. Given a response surface equation in quadratic form, the Six Sigma Black Belt will be able to compute the stationary point.
88. Given data (not graphics), the Six Sigma Black Belt will be able to determine if the stationary point is a maximum, minimum or saddle point.
89. The Six Sigma Black Belt will be able to use a quadratic loss function to compute the cost of a given process.
90. The Six Sigma Black Belt will be able to conduct simple and multiple linear regression.
91. The Six Sigma Black Belt will be able to identify patterns in residuals from an improper regression model and to apply the correct remedy.
92. The Six Sigma Black Belt will understand the difference between regression and correlation studies.
93. The Six Sigma Black Belt will be able to perform Chi-square analysis of contingency tables.
94. The Six Sigma Black Belt will be able to compute basic reliability statistics (MTBF, availability, etc).
95. Given the failure rates for given subsystems, the Six Sigma Black Belt will be able to use reliability apportionment to set MTBF goals.
96. The Six Sigma Black Belt will be able to compute the reliability of series, parallel, and series-parallel system configurations.
97. The Six Sigma Black Belt will demonstrate the ability to create and read an FMEA analysis.
98. The Six Sigma Black Belt will demonstrate the ability to create and read a fault tree.
99. Given distributions of strength and stress, the Six Sigma Black Belt will be able to compute the probability of failure.
100. The Six Sigma Black Belt will be able to apply statistical tolerancing to set tolerances for simple assemblies. The Six Sigma Black Belt will know how to compare statistical tolerances to so-called “worst case” tolerancing.
101. The Six Sigma Black Belt will be aware of the limits of the Six Sigma approach.

Reproduced with kind permission of Six Sigma Training

Source: Pyzdek, T (2003), 101 Things Every Six Sigma Black Belt Should Know, Six Sigma Training.

Related Posts

Jason Silva: To Understand is to Perceive Patterns

Related Posts

Mathematics, Problem-Solving, and Critical Thinking

The following is extracted from a speech presented by Grace Fu to winners of the Singapore Mathematical Society's Annual Prize Presentation Ceremony in 2008:

Mathematics will imbue you with problem-solving skills such as the ability to understand a problem, identify the relevant information, look for relationships and patterns, make your own conjectures and apply mathematical knowledge and tools to solve or prove them. Mathematics, thus, provides good opportunities for the training of the mind to think critically and adapt to new situations, something that will be valuable to your future careers.

The words above are instructive for anyone seeking to join the ranks of analytics professionals worldwide. The ability to solve complex problems through critical thinking and reasoning is essential in today's analytics-based economy. Singapore is recognized around the world as a leader in the conceptual mathematics movement.

Grace Fu Hai Yien (Chinese: 傅海燕; pinyin: Fù Hǎiyiàn)

Source: Fu, G (2008), Speech at the Singapore Mathematical Society Annual Prize Presentation Ceremony.

Related Posts:

Pascal on Analysis

Conceptual Mathematics in Singapore

Spreadsheets Are Back

A recent poll of over one thousand LinkedIn members returned some interesting insights into spreadsheet usage patterns in companies. Respondents were posed with the following statement, and were then asked to provide a single response as follows:

I use spreadsheets _____ in my work.
· Never
· Rarely
· Monthly
· Weekly
· Daily

The results found that 80% of respondents use spreadsheets on a daily basis, while another 11% use spreadsheets weekly. In all, over 90% of respondents are apparently using spreadsheets at least weekly in their jobs.

The other interesting finding was that spreadsheet ubiquity was at its greatest in enterprise and large firms where a full 85% of resondents reported using spreadsheets on a daily basis, while another 10% reporting weekly usage.

One respondent left a comment claiming to have selected "daily" only because "constantly" and "hourly" were not offered as options. Still another respondent voiced surprise that "daily" users were less than 95%. One apparent critic of spreadsheets commented that the poll was "a waste of time."

Results were generally even across age groups. However, males reported somewhat higher daily spreadsheet usage than females. The survey was open to all LinkedIn users between April 24 and May 19, 2009. There were 1,094 voluntary participants in the survey.

More

What is Business Intelligence?

business intelligence

The use of data to discover what is happening with a company. An example of business intelligence is the use of a computer program to look at financial statements to detect irregularities or other patterns. A company may do this to improve its own efficiency, an outside observer may do it to make recommendations or to invest, and a competitor may do it to find potential advantages.

Source: The Free Dictionary

Related Posts

Why Policymakers Need to Take Note of High-Frequency Finance

by Richard Olsen © VoxEU.org

Why should high-frequency finance be of any interest to policymakers interested in long-term economic issues? This column argues that the discipline can revolutionise economics and finance by turning accepted assumptions on their head and offering novel solutions to today’s issues.

I believe high-frequency finance is turning aspects of economics and finance into a hard science. The discipline was officially inaugurated at a conference in Zurich in 1995 that was attended by over 200 of the world’s top researchers. Since then, there have been a large number of publications including a book with the title Introduction to High-frequency Finance. “High-frequency data” is a term used for tick-by-tick price information that is collected from financial markets. The tick data is valuable, because they represent transaction prices at which assets are bought and sold. The price changes are a footprint of the changing balance of buyers and sellers.

The term “high-frequency finance” has a deeper meaning and is a statement of intent indicating that research is data-driven and agnostic. There are no ex ante theories or hypotheses. We let the data speak for itself. In natural sciences this is how research is often conducted. The first step towards discovery is pure observation and coming up with a description of what has been observed – this may sound easy but is not at all the case. Only in a second step, when the facts are clearly established, do natural scientists start formulating hypotheses that are then verified with experiments.

In high-frequency finance:

• The first step involves the collecting and scrubbing of data.
• The second step is to analyse the data and identify its statistical properties.

Here one looks for stylised facts which are significant and not just spurious. Due to the masses of data points available for analysis (for many financial instruments one can collect more than 100,000 data points per day), identification of structures is straightforward, either there is a regularity or there is none.

• The third step is to formalise observations of specific patterns and seek tentative explanations, theories to explain them.

The abundance of data in high-frequency finance has profound implications for the statistical relevance of its results. Unlike in other fields of economics and finance, where there is not sufficient data to back up the inferences, this is not an issue in high-frequency finance. The results are unambiguous and turn economics and finance into a hard science, just as is the case for natural sciences. This is not a bad thing.

High-frequency data as an answer to singularity of macro events

Today we are all grappling with the global financial crisis and have to make hard decisions. In living memory, we have not seen a crisis of a similar scale, so policymakers are in a vacuum and do not have any comparable historical precedents to validate their policy decisions.

If the global economy had been in existence for 100,000 years, this would be a different matter. We would have had many crises of a similar scale, and we could use these previous events as a benchmark to evaluate the current crisis. The modern economy with financial markets linked together through high speed communication networks trading trillions of dollars on a daily basis is a new phenomenon that did not exist even 20 years ago. People refer to the events of 1929 and subsequent years, but while these events can be used as one possible point of reference, they are not meaningful in the statistical sense. On a macro level, we can make observations but no inferences because we do not have the historical data. There is a void that researchers and policymakers need to acknowledge.

Fractals: Understanding macro structure from micro data

High-frequency finance can fill the void with its huge amounts of data – if we embrace fractal theory that explains how phenomena are the same even if they occur at different scales. Fractal theory suggests that we can search for explanations of the big crisis by moving to another time scale, the short term.

At a second-by-second level, there are an abundance of crises and systemic shocks; just imagine the occurrence of the many price jumps due to unexpected news releases and political events or large market orders. Albeit on a short-term time scale, we study how regime shifts occur and how human beings react. The large number of occurrences allows for meaningful analysis. We study all facets of a crisis, how traders behave prior to the crisis, how they react to the first onslaught, how they panic, when the going gets hard and finally, how their frame of reference which previously was a kind of anchor and gave them a degree of security breaks down and how later, when the shock has passed, the excitement dies down, there is the aftershock depression and then eventually how gradual recovery to a new state of normality begins.

The everyday events sum up and shape the tomorrow

High-frequency finance has another big selling point, one which policymakers should take note of: the study of market events on a tick-by-tick basis brings to the surface the detailed flows of buying and selling that occur in the market. From this information, it is possible to build maps of how market participants build up positions and how asset bubbles develop over time. By tracking price action on a tick-by-tick basis, it is possible to infer the composition of those bubbles similar to the work of geologists studying rock formations. Researchers can identify, who has been buying and selling, on what time horizons they trade, how resilient they are to price shocks, what makes them turn their position and become net sellers as buyers. Based on this information we can make inferences of the likely collapse of those bubbles.

High-frequency finance opens the way to develop "economic weather maps". Just as in meteorology, where the large scale models rely on the most detailed information of precipitation, air pressure and wind, the same is true for the economic weather map. We have to start collecting data on a tick-by-tick level and then iteratively build large scale models. Today, the development of such a global economic weather map has barely started. The "scale of market quake" (a free Internet service) is a first instalment, but the start of an exciting development.

High-frequency finance holds out the hope of turning aspects economics and finance into a hard science by the sheer volume of data and its ability to set events into their appropriate context by mapping rare events into a short-term time scale with a near infinity of events, albeit at a shorter-term time scale. Second, the tracking of events on a tick-by-tick basis opens the door to identify underlying flows and develop economic weather maps. Surely that’s not a bad thing?

References

Bisig T, Dupuis, A, Impagliazzo, V, and Olsen, R (2009), “The scale of market quakes”, working paper, September.

Gençay, R, Dacorogna, M, Müller, U, Olsen, R, and Pictet, O (2001), An Introduction to High Frequency Finance, Academic Press.

Mandelbrot, B (1997), Fractals and Scaling in Finance, Springer.

Mandelbrot, B, Hudson, R (2004), The (Mis)behavior of Markets, Basic Books.

Republished with permission of VoxEU.org

The Beauty of Finance

Here's a quote by Prof Emanuel Derman (2004, p. 270) that every financial analyst can take to heart:

I like to think in Goethean terms of what we do in quantitative finance. We try to make as beautiful and truthful a description as we can of what we observe. We’re involved in intuiting [i.e., sensing], inventing, or concocting approximate laws and patterns. We combine both art and science in creating understanding. We use our intuition, our scientific knowledge and our pedagogical skills to paint a picture of how to think qualitatively, and then within limits, quantitatively, about the world of human affairs, and in so doing, we influence and are influenced by other people’s thoughts.

Reference: Derman, E (2004). My Life as a Quant: Reflections on Physics and Finance. Hoboken, NJ: John Wiley.

Future Skill Shortages in the US Economy? Sorting Out the Evidence

by David Neumark, Hans Johnson, and Marisol Cuellar Mejia © 2011 VoxEU.org

The impending retirement of the baby-boom cohort represents the first time in the history of the US that such a large and well-educated group of workers will exit the labour force. Despite the gloomy outlook of recent research, this column suggests there is little likelihood of large-scale skill shortages emerging by the end of this decade.

Ageing workforces pose challenges to governments around the world. While fiscal issues surrounding pension and social security have been very much in the news, a less well-known issue concerns skills.  The impending retirement of the baby-boom cohort brings with it the potential for skill shortages. The boomers are well-educated, having come into adulthood as the nation was rapidly expanding post-secondary educational opportunities. In earlier decades, younger workers replacing older workers were both much more educated and much more numerous. But the baby boomers are nearly as educated as current younger cohorts (Figure 1) and are large in number. Thus, their retirement will slow the growth of skill levels in the workforce, leading to shortages if skill demands continue to increase.

Figure 1. Number of adults with at least a bachelor’s degree by age group (25-44 and 45-64)


Source: Decennial Census (1990 and 2000); American Community Survey (2008)

Carnevale et al (2010) recently projected large shortages by the end of this decade: “By 2018, the postsecondary system will have produced 3 million fewer college graduates than demanded by the labour market” (p 16). But Harrington and Sum (2010) criticise these projections, instead seeing over-education or “mal-employment” – college workers in jobs that do not require college degrees – as “perhaps the most pressing problem facing college graduates in the nation today….”

Our projections of skill supplies and demands for the US economy stake out a middle ground. We foresee rising demand for highly-educated workers. But in the near term this rising demand will by and large be met by rising education levels among the US population, suggesting little risk of a substantial workforce skills gap. At the same time, there are greater risks of skill shortages in states with large and growing, and less-educated, immigrant populations. And over the longer-term, as more baby boomers retire, there is greater risk of substantial skill shortages nationwide.

Projections of skill supplies and demands through 2018

Our demand projections rest on US Bureau of Labour Statistics (BLS) projections of employment growth by occupation to 2018 (Lacey and Wright 2009). To project the education requirements of future jobs, we could also rely on the BLS, which classifies occupations by educational requirements. However, using data from the American Community Survey (ACS), we find substantial labour market returns, within occupations, to educational levels beyond those that the BLS deems “required” (see Neumark et al 2011). We therefore instead use empirical evidence on employment practices to estimate and project workforce skills needs, starting with the baseline education distribution of workers by occupation in 2008 and applying recent trend growth in education distributions within occupation (using ACS and Decennial Census data). Applying these estimates to the occupational projections, we obtain projected skill (education) demands.

To project supply, we construct new population forecasts that take account of nativity (unlike US Census Bureau population projections), and we project population by education, and labour force participation. Three important factors underlie our projections:

• First, that young adults will continue to experience improvements in educational attainment compared to the preceding cohorts;
• Second, that there will be continued upgrading of educational attainment levels of older workers; and
• Third, that labour force participation rates will continue to rise for more highly-educated older adults, and that past patterns in retirement will prevail for the baby boom as it reaches retirement ages.

In Table 1, we compare our preferred educational attainment projections (supply) with the employment projections (demand), in levels and shares. These projections do not point to significant impending shortages of skilled workers in the US through 2018, as the projected demand and supply shares by education are quite similar. We do see projected shortages for people with an Associate’s degree (356,000), and some excess supply of less-educated workers (those with some college or a high school degree or less). Our comparisons are based on projected total labour force supply of workers, and do not include forecasts of unemployment. If we adjust the 2018 supply projections for unemployment rates by education category as observed in 2008, then the projected shortage of workers with an Associate’s degree or higher expands to around 800,000, still far less than Carnevale et al predict.

Conflicting evidence

We have explored numerous explanations for why Carnevale et al (2010) project much more substantial skill shortages. The difference is primarily attributable to the data they use (the Current Population Survey, or CPS). In particular, the CPS data show a higher share with college degrees at the 2008 baseline, and faster growth of these shares over time, both of which lead to considerably higher projected demand for workers with college degrees in 2018. But the CPS data appear problematic. First, the CPS data appear to overstate the share with Associate’s degrees, because the CPS equates occupational or vocational programmes with college degrees, whereas the ACS data do not. The CPS data also show much faster growth rates in the share with Bachelor’s degrees or higher.

We verified that these data differences explain the differences between our demand projections and theirs. Moreover, in both data sets the shares in each education category in 2008 appear anomalous, whereas education trends through 2007 were much more similar. We therefore redid the demand-side forecasts using data from 2000-2007 (rather than 2000-2008) to estimate the within-occupation trends in education, in which case the entire difference between the projections was attributable to the different baseline educational distribution in the CPS. Finally, because Carnevale et al use a supply projection from a completely different source, it seemed likely that using CPS on both the demand and supply sides of the market should substantially reduce the sensitivity of the projections to the definition of education, which we verified. Using CPS data on both sides of the market leads to much milder projections of skill shortages than the dramatic shortages that Carnevale et al project.

The Harrington and Sum criticism of the projections in Carnevale et al (2010) could equally well be directed at our projections. They argue against using observed educational distributions within occupations to measure educational requirements, and instead believe that the BLS determinations of skill requirements are accurate. Although they do not develop projections, our full paper (Neumark et al, 2011) shows that if we project skill demands based on BLS skill requirements, we project massive oversupply of skilled workers. So Harrington and Sum are right that conclusions about skill shortages depend critically on how one measures skill requirements.

However, the argument that BLS skill requirements are accurate is belied by the evidence that there are substantial economic returns, within occupations, to education levels above those “required” according to the BLS. Although Harrington and Sum (forthcoming) present some evidence that appears to suggest the opposite, their evidence is based on earnings regressions that omit occupation controls, leading to spurious evidence of lower estimated returns to education for those in occupations that use less-educated workers. For example, Harrington and Sum (2010) tell the “story” of bartenders (college degree is not required) and compensation and benefits managers (college degree required). The question is not whether bartenders earn less than compensation and benefits managers, but whether the return to education within for bartenders is less than the return to education for compensation and benefits managers. We therefore conclude that the Harrington and Sum critique of using observed rather than “required” education to capture skill demands is unfounded.

Skill shortages in some states?

Although we do not project significant near-term skill shortages nationwide, the situation could differ in states with large and growing Hispanic immigrant shares in which older adults nearing retirement ages are notably better educated than young adults. States that fit this profile include California, Texas, Florida, Arizona, Colorado, New Mexico, and Nevada. The importance of these demographic changes is illustrated by a simple exercise where we project supply for the nation, but substituting California’s ethnic composition in 2018 for that of the entire US. In other words, we ask the question, would there be a national skill shortage if the country had California’s demographic mix? The answer is yes: we project a deficit of 3.1 million workers with an associate’s degree or higher. Of course, domestic migration could ameliorate some of these shortages.

Skill shortages in the longer term?

The longer-term perspective is also less sanguine. Our projections extend to 2018 because the BLS occupation projections end there. But the majority of boomers (two of every three) will be younger than age 65 in 2018, whereas by 2030 all of the boomers will have passed age 65. We expect that projections of the US economy to 2030 would show a continuation of greater rates of growth in industries and occupations that employ highly-educated workers, consistent with the long-standing trend in the US, and accentuated by the increased demand for healthcare as the baby boomers enter old age. Yet the size of the baby boom cohort coupled with its high education levels imply that the replacement of older with younger cohorts will not lead to rising education levels to anything like the extent to which it did in the past (Figure 1). It is plausible, then, that general skill shortages would be much more evident in projections extended out a couple more decades.

The research on the projections was supported by the Gates Foundation and the AARP Foundation. The views expressed are the authors’, and do not reflect the views of PPIC or the AARP or Gates Foundations.

References

Carnevale, Anthony P, Nicole Smith, and Jeff Strohl (2010), Help Wanted: Projections of Jobs and Education Requirements Through 2018, Centre on Education and the Workforce, Georgetown University.

Harrington, Paul E and Andrew M Sum (2010, November), “College Labour Shortages in 2018?”, New England Journal of Higher Education.

Harrington, Paul E and Andrew M Sum (2011), “Recent Projections of Labour Shortages Through 2018: From Great Recession to Labour Shortages? A Critical Look at the Evidence”, forthcoming in Monthly Labour Review.

Lacey, T Alan, and Benjamin Wright (2009), “Occupational Employment Projections to 2018”, Monthly Labour Review, 132(11), November, 82-123.

Neumark, David, Hans P Johnson, and Marisol Cuellar Mejia (2011), “Future Skill Shortages in the US Economy?”, NBER Working Paper No. 17213.

Republished with permission of VoxEU.org

Visualizing Human Migration Patterns

Even Westvang uses Deluge software to track the migrations of 300,000 Norwegians using individual tax filing data (which is public domain in Norway). The result is a visualization treat!



Imagine how this visualization technique might be applied to other data sets. Follow the links below to learn more about Even Westvang's work, Deluge, and how this visualization was produced.

Deluge by Even Westvang

Related Posts