Open Space Research?


Dennis Embry sent this question from PAXIS Institute in Tuscon, Arizona…

I am looking for peer-reviewed, published studies on Open Space procedures. I am looking for either randomized control group studies or high-quality interrupted time series studies showing actual impact on outcomes of organizations or communities. What I have found so far are discussions of processes and proximal outcomes. I see for example Open Space is often used for community problems, and I am curious if there is evidence that it actually leads to reductions of those problems or increase in related positive outcomes.

The only research I know of is posted in the ResearchActivities page of our ResourcesCollection… but maybe we can shake out some other responses for Dennis? Please comment here for Dennis and others interested in this sort of research. We’ll be glad to add any new materials to our Research page, as well.

 

Thanks for alerting me to the “interupted time series” methodology. New to me. It sounds like a usful approach to discernable behaviours or results of an “intervention”. Does it also measure the internal states of individuals or community culture as part of the process. I’m convinced measuring from all perspectives is useful.

One the wikki site Peg and I have posted two different approaches to gathering “perceptions” of OST events — I stopped putting in data after no one followed my exaples. I guess no one was interested it the questions related to what language people used to describe their experience. My recent publicaion in Alban and Bunker uses pre and post data of a Hay employee survey to suggest some impact of OST on an organization. Its availale thru my web site at http://www.spiritedorg.com. So, why haven’t you done an “interupted time series study” of OST.

It occurs to me that we might be asking Open Space to do more than it really can. Why would we expect a single meeting or series of a few meetings to be able to make a measurable dent in some of the large social issues mentioned above? My guess is that we do not ever hold other types of meetings, public hearings, and studies to the same standard. Has anybody ever researched the effectiveness of city council meetings or public comment hearings or even ballot questions?

When we do research on drugs, we don’t necessarily test whether a drug will cure a disease. We test whether it does what it is designed to do. I think we need to think about testing Open Space in the same way. Then decide if that effect might be part of the overall solution.

We know that Open Space allows…

-large numbers of people to gather
-identify all the issues that matter to them
-discuss them in ways that are satisfying and productive
-document them in ways that can be understood by others active in whatever field we’re in
-sort them into high and low priority levels, identify meaningful actions in the short and long-term
-identify resources available and needed
-build a roster of people who are ready willing and able to be actively supporting the resolution of whatever question is at hand — as responsible owners of the situation.

Now, all of these things can be tested very directly, and all of them happen very easily in Open Space. And few of them can be accomplished as well or as quickly by any of the formal processes usually used by governments, companies or community groups.

The question for the governors, or other policy people, is whether or not they want to do these things. Are these things, on any given issue or question, likely to be useful. There is no promise that these things will solve the question, BUT… does anybody think that the question can be solved WITHOUT doing these things. If so, then they should just solve it. Otherwise, Open Space is likely to be very useful in doing all of these things in short, simple order.

It would be very interesting to invite facilitators and sponsors to consider these very specific effects of Open Space and ask them how well these things happened in each OS they’d been part of, with room to comment on what specific thing(s) might have been done to make or sharpen or support each effect.

Mostly, though, I think it’s important to test and validate what Open Space actually promises to do. Invite, gather, identify, discuss, document, sort, connect, support, make plans. Not end addiction, cure cancer, raise profits, etc. These bigger things likely take more than a few meetings and depend on the people involved, not the meeting tech.

Dear Jeff,you are precisely right. Tony Biglan is a close colleague, and he and have have done the first intentional interrupted time series (multiple baseline) across states. We mobilized states on tobacco control issues using low-cost strategies in every county, and we were able to show state level effects, despite all the noise. That is the beauty of such designs: they filter out the noise and confuounds, in ways unlikte most randomized control group studies. When people say it is not possible to measure scientifically things that are preseumptively good, then I presume were are entering the arena of religious practice.

A community could be its own control across multiple outcomes. For example, open space could be used for three different “problems” of communities, which unique dependent measures. For example, one might take the problem of Meth addiction, obesity, and some form of conservation. All those problems have been previously studied in randomized control or interrupted time series studies. One could convene open space events sequentially on each those say, 12 months apart. One could measure community indicators of these problems as the dependent variables, and there cerrtainly ones that could be followed or exist already.

In the same vein, open space could be evaluated sequentially on different problems in organzations. If one had 10 organizatinos, the order of focus of the open space events could be randomized, adding to the quality of the experimental design.

For instance, a school could do such a study on increasing achievement, reducing bullying, increasing attendance, etc. Again, there are good quality studies on reducing these things that have used either randomized control group studies in the past or have used these type of interrupted time series.

I hope this discussion stimulates people to start thinking about serious science of culturatl change.

Dennis Embry

It seems more likely to me that the “interrupted time series” method could be used to assess OST interventions. Biglen, Ary, and Wagenaar introduce the method in their article “The Value of Interrupted Time-Series Experiments for Community Intervention Research” (Prevention Science journal) at http://www.springerlink.com/content/r615u05320743n01/ as follows:

“Time-series designs enable the development of knowledge about the effects of community interventions and policies in circumstances in which randomized controlled trials are too expensive, premature, or simply impractical. The multiple baseline time-series design typically involves two or more communities that are repeatedly assessed, with the intervention introduced into one community at a time. It is particularly well suited to initial evaluations of community interventions and the refinement of those interventions. This paper describes the main features of multiple baseline designs and related repeated-measures time-series experiments, discusses the threats to internal validity in multiple baseline designs, and outlines techniques for statistical analyses of time-series data. Examples are given of the use of multiple baseline designs in evaluating community interventions and policy changes.”

Whew. So we see the interest to isolate variables, for example to show that OST makes a difference and not just because the community that invites OST might be already more inclined toward, or on a trajectory toward, transformative change.

I’m glad that people are thinking about these quantitative/experimental research issues. As a qualitative researcher I’m quite happy with the hundreds of success stories. Haven’t any of these been translated into numbers using a Likert scale?

“randomized control group studies or high-quality interrupted time series studies showing actual impact on outcomes of organizations or communities.” — I think these research methodologies are not approriate for studying OST. The complexity of the context of any OST event means that finding a “control” group is almost impossible. I worked with one researcher to explore having parallel OST, Appreciaitve Inquiry and Future Search events in three different aboriginal commumnites to then compare the “outcomes”. No community was similar enough to actually perform a control function. The differences in community leadership, event planning approaches, event leadership, community participation culture, follow-up processes, etc create too many variables for a traditonal study or a multi-variate analysis.

I’m trying to imagine how such research would look. You’d have to have two similar communities, each trying to bring about a measurable community outcome, such as (to make up examples) increasing affordable housing or decreasing mercury emissions. Or maybe “cleaning up a city park.” Then you’d have to have a planning meeting in each community, one using OST and the other using another process. You’d have to hold other factors constant — numbers and qualities of participants, hours, type of setting, etc., plus amount of resources available to get the work done, external issues to be worked through (zoning, etc.) — and then measure, at some future point, which community had made the most progress toward the goal.

But there would be so many variables… could you possibly say that changed outcome measures were due to using OST?

To the best of my knowledge there is nothing that meets the research criteria proposed, certainly not prospectively, and probably not retrospectively. However, something that comes close to what Dennis may be looking for relates to a little gathering we did in Rome for Israelis and Palestinians. The first part is reported in a piece I did (http://openspaceworld.com/opening_space_for_peace.htm) and I understand that Tova Averbuch, Avner Haramati(Israelis), and Carol Daniel (Palestinian)have done. I don’t happen to have a copy of it, but maybe they will be forthcoming.