#13 on design-based/experiment research readings

Summary

Design-based research (Brown, 1992; Collins, 1992) has been emerging for the studying of learning in context through the systematic design and study of instructional strategies and tools. This research paradigm shifting brought in many vision, revision and arguments.
My current reading includes a series of monographs and papers surrounding around this theme- call it experiment design research, design-based research, or teaching experiment - in the CI8395 list. These eight articles provide an array of advocate and critique of its structures, functions, limitations as well as potential solutions.

Most of these articles either champion or espouse the new possibilities and contribution to improve and innovate the educational theories and practice. One of them, Shavelson et al’s “On the Science of Education Design Studies”, on the other hand, argues that design studies must comport with guiding scientific principles and provide adequate warrants for their knowledge claims. They acknowledge the nature of the messiness of the educational ecology study. For design studies are complex, multivariate, multilevel, and interventionist, making warrants particularly difficult to establish. They critique on the typically heavy usage of design studies on narrative accounts to communicate and justify the findings. They argue that narratives often purport to be true, but there is nothing in narrative form that guaranteed veracity. The solution they propose is to provide a framework that links design-study research questions as they evolve over time with corresponding research methods. In this way, integration can be seen of research methods focused on discovery with methods focused on validation of claims. That is what I agree upon- the tough task that a well designed design-based research needs to envision and to employ the fittest methods and documentations from an array of available research methods and methodologies into the ongoing research processes. It, by nature, also tests the heuristics and wisdom of researchers and teams to deploy appropriate multiple methods and triangulation to provide warrants and knowledge claims.

The above article is the major critique from the eight on design-based/experiment research. The rest of papers are mainly supporting or implementing (such as model building or framework constructing based on the tenet of design-based research) this methodology. In the article, “Design-based Research: An Emerging Paradigm for Educational Inquiry”, the Baumgartner et al remark that design-based research blends empirical educational research with the theory-driven design of learning environment, is an important methodology of understanding how , when, and why educational innovations work in practice. Design based researchers’ innovations embody specific theoretical claims about teaching and learning, and help us understand the relationships among educational theories, designed artifact, and practice. Design is central in efforts to foster learning, create usable knowledge, and advance theories of learning and teaching in complex settings. Design-based research also many contribute o the growth of human capacity for subsequent education reform.

I think I might have an opportunity to carry out a small scale of design research in my coming semester if I choose to teach the brand new LiveMeeting with Sociology. The following sections are my study notes to reinforce such a temptation from these articles:

Prospects for design-based research in education: the promise the design-based research can provide: a exploring possibilities for creating novel learning and teaching environment b. developing theories of learning and instruction that are contextually based c. advancing and consolidating design knowledge, and d. increasing or capacity for educational innovation.

Be careful! Challenges faced by design-based research methods: the issues of reliability and validity of data collection and interpretation are different from the controlled experiment. Design based research relies on techniques used in other paradigms which I am familiar with are case study, ethnography, hermeneutic phenomenology, historiography, ethnomethodology, which depend on thick description datasets, systematic analysis of date with carefully defined measures and consensus in building within the field around interpretation of the data. When trying to promote the objectivity, while attempting to facilitating the interpretation, design-based researchers regularly find themselves in t he dual roles of advocate and critic. It is possible to employ specific research methods to question the designer-researcher’s tacitly held assumptions. The methods of documenting process of enactment with triangulation of multiple sources to provide critical evidence to establish warrants for claimed outcomes.

In “Design Experiments in Educational Research” , Cobb et al Cobb et al (2003) propound that design-based research can help create and extend knowledge about developing, enacting, and sustaining innovative environments.
A good design-based research has the following characteristics, according to Cobb et al’s suggestions:
1.The central goals of designing learning environments and developing theories or “prototheories” of learning are intertwined.
2. Development and research take place through continuous cycles of design, enactment, analysis and redesign (Cobb, 2001; Collins 1992).
3. Research on designs must lead to sharable theories that help communicate relevant implications to practitioners and other educational designers (cf. Brophy, 2002).
4. Research must account for how design function in authentic settings. It must not just document success or failure but also focus on interactions that refine our understanding of the learning issues involved. The development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interests.
5. Design experiments are pragmatic as well as theoretical in orientation – both of the design and of the resulting ecology of learning- is at the heart of this methodology.

The range and settings vary in both research type and scope:
1. One on one (teacher-experimenter and student)design experiments in which a research team conducts a series of teaching sessions with a small number of students. The aim is to create a small scale version of a learning ecology so that it can be studied in depth and detail (Cobb & Steffe, 1983; Steffe & Thompson, 2000). I think I might try this one if things come together for me to do it!
2. Classroom experiments in which a research team collaborates with a teacher (who might be a research team in which a research team collaborates with a teacher (who might be a research team member) to assume responsibility for instruction (Cobb, 2000; Confrey & Lachance, 2000; Gravemeijer, 1994).
3. Preservice teacher development experiments in which a research team helps organize and study the education of prospective teachers (Simon, 2000).
In-service teacher development studies in which researchers collaborate with teachers to support the development of a professional community (Lehrer & Schaulble, 2000; Stein, Silver, & Smith, 1998).
4. School and school district restructuring experiments in which a research team collaborates with teachers, school administrators, and other stakeholders to support organizational change (Confrey, Bell, & Carrejo, 2001).

They also identified 5 crosscutting features of design study:
1. The purposes of design experimentation is to develop a class of theories about both the process of learning and the means that are designed to support that learning. It can be the learning of individual students, a classroom community, a professional teaching community, a school or school district as an organization. The means for supporting learning encompass the affordances and constraints of material artifacts, teaching and learning practices, and policy levers etc.

2. It is necessary to document learning ecologies at multiple levels (Kelly & Lesh, 2000). Example: A research team focuses on the norms and practices of a professional teaching community, the participating teachers’ pedagogical reasoning and instructional practices, and their students’ reasoning in a particular content domain.

3. The highly interventionist nature of the methodology- design studies are typically test-beds for innovation. The design developed while preparing for an experiment draws on prior research and attempts to cash in the empirical and theoretical results of that research. The process of engineering the forms of leaning being studies provided the research team with a measure of control when compared with purely naturalistic investigation. Design experiments have two faces: prospective and reflective. On the prospective side, designs are implemented with a hypothesized learning process and the means of supporting it in mind in order to expose the details of the process to scrutiny. On the reflective side, design experiments re conjecture-driven tests, often at several levels of analysis.

4. The above two aspects result in the iterative design.

5. The pragmatic nature-theories developed during t he process of experiment are humble in terms of domain specific learning processes, and also are accountable to the activity of design. The theories do real work. General philosophical orientations to educational matters- such as constructivism – are important to educational practice, but they tend to fail to provide detailed guidance in organizing instruction. The question is Does theory informs prospective design and in what way? Design experiment also tend to emphasize and intermediate theoretical scope (diSessa, 1991) that is located between a narrow account of a specific system.

Preparing for design experiment, according to Cobb et al’s perspective:
1. Clarifying the theoretical intent: What is the point of the study? Example: the relationship between classroom norms of a discipline and student learning, or diversity of students’ prior experiences can be capitalized upon as a resource to ensure that all student have access to significant disciplinary ideas.
2. Specify the significant disciplinary ideas and forms of reasoning that constitute the prospective goals or endpoints for student learning. This usually involves drawing on the synthesizing the prior research literature to identify central organizing ideas for s domain.
3. Specifies the assumptions about the intellectual and social starting points of the envisioned forms of learning. These works include current student capabilities, practices, their initial interpretation and understanding as part of the pilot work.
4. Formulate a design that embodies testable conjectures about both significant shifts in student reasoning and the specific means of supporting these shifts.

Conducting a Design experiment
A primary goal for the study is to improve the initial design by testing and revising conjectures as informed by ongoing analysis of both the students’ reasoning and the learning environment. There are 4 functions they require ongoing direct engagement in the research setting and the associated planning and interpretive activates:

1. A clear view of the anticipated learning pathways and the potential of support must be maintained and communicated within the research team,
2. Cultivation of ongoing relationships with practitioners
3. Seek to develop a deep understanding of the ecology of learning – a theoretical target for the research
4. Regular debriefing.

One of the characteristics of the design experiment methodology is that the research team deepens its understanding of the phenomenon under investigation while the experiment is in progress. It is standard procedure in most engineering disciplines to keep records to support the retrospective analysis of the experiment (Edelson, 2002). Accordingly, the research team may employ audio record f meeting and logs to document the evolving conjectures, together with the observations that re views as either supporting or questioning a conjecture.

This is an interesting article! For research across disciplines and across methods and methodologies, McCandliss et al’s article propose that design-experiments might be productively combined with methods of inquiry common in more traditional elaborative science and considers the potential benefits of such a dialectic. The authors hope to promote a constructive dialogue to help formulate an infrastructure for the science of education that synthesized theoretical insights supported by a wide array of investigational methodologies (Posner & McCandliss, 1993).

The article retrospected to the recent Congressional and U.S. Department of Education policy statements mark a radical shift in the shaping of future educational research methodology, calling for randomized controlled trials as the primary source of “scientific evidence” relevant to improve practice (Shavelson, Phillips, Towne, & Feuer, 2003). There is a basic tension between the types of methods and frameworks advanced in these recent calls for evidence-based practice and those that have proven to be useful in the leading models for design experiments. Ann Brown’s (1992) research provided vision to this tension.She envisioned dynamic relationships between classroom-based and elaborative-based research. Her work provided specific examples of observations, conjectures, and artifacts that might be transported across these two research contexts. She perceived such exchange as bi-directional supporting a mutually beneficial cross-fertilization of tow very different research contexts. Unfortunately many of the dominant design experiment approaches have provide little or no provision for intellectual exchange with laboratory science methods.

After perusing these articles, the potentiality, functionalities, limitations and challenges of the design-based/experiments research come alive in front of me.