In Response To: Alan Schoenfeld
King's College, London
Educational research and design: developing the debate
This article is a response to Schoenfeld, AH. (2009) Bridging the Cultures of Educational Research and Design. Educational Designer, 1(2). The view that I want to present is that Alan Schoenfeld’s paper opens up a very important debate in a challenging and incisive way, but that both the foundation and the range of the argument need further development.
Curriculum, pedagogy and assessment2
I start from Schoenfeld’s statement that, “the primary goal of design is to produce materials that make a difference” (p.4). Assuming that the difference desired is improved learning of students, one has to ask – what determines the quality of such learning? A simple model is that it is through the interaction of three main factors.
The first is the curriculum, seen as an inter-twining of learning quality and of content. The designer may take these for granted, so that the “difference” is to achieve the accepted aims more effectively, or may wish to push the boundaries by showing that outcomes, of greater value than those currently accepted, are both possible and desirable.
The second is pedagogy. Educational materials may be seen as serving the interaction between curriculum and pedagogy, helping the teacher to implement the aims within the real constraints of the educational system . Insofar as the materials are novel attempts to make a difference, their success will depend, at least in part, on the difference they make to the teacher’s classroom practice. A prudent innovation has to include changing the practice as far as possible - but no further.
A third determinant is assessment. A distinction has to be made here: for formative assessment, seen as the day to day interactive feedback between teachers and learners, the materials ought to be designed to promote activities which can provoke and enrich such interactions, thereby enhancing the role of these materials in strengthening the curriculum-pedagogy link. This aspect ought to be intrinsic to any good design. The situation may be different for summative assessment: in some systems there may be freedom to design the assessment instruments to reinforce the innovative aims, but in others the designer may have to match ambition to the constraint that teachers have to work to accountability pressures exerted through externally imposed tests. In this latter case, the “curriculum” changes, for good or ill, into the “curriculum as interpreted in the high-stakes tests”.
My argument here is that designers have to work with the interactions between these three components, and to pay attention to the constraints and affordances that they entail.
The innovative classroom3
In the above model, the teacher is a key player, often in determining the materials which are to be used, always in determining how they are to be used. I suggest that this raises two issues. One is to question further what would constitute making a difference. I suggest that the vision here must include changes in pedagogy, and not just the continued use of the pre-existing practices with no more than the replacement of the support materials.
The second issue is the need to explore ways in which consideration of the key role of the teacher might be included in any model of the designer-researcher relationship. One possible vision of successful materials is that they are accepted by teachers, and used in place of previous materials, so securing the desired improvements. In referring to Swan’s (2008) account of his development of materials, Schoenfeld acknowledges that more is involved, in that success depends on the ways in which teachers make use of them; but he does not subsequently expand on this point. A different vision is that the main aim is to transform the learning work in their classrooms by changing the ways in which teachers interact with learners, helping teachers to change by provision of materials that support the innovation. In my account of innovations in which I have been involved (Black, 2008a), I made the point that changes in the way teachers implemented an innovation were the central, and often the most difficult, part of the work. Indeed, for the innovation project in formative assessment, there were no “materials” for student use, and the only published outcomes were a book and booklets for teachers - yet it cannot be said that this was not a designed innovation .
There is extensive literature that discusses the problems of achieving change in ways that motivate and help sustained development in teachers’ practice. Insofar as it is descriptive, this literature shows how difficult it might be to produce sustainable change. The key title “Continuing Professional Development” (CPD) reflects this point. An introduction to a special issue of a science education journal devoted to the topic of evidence-based CPD (Harrison et al., 2008) states that:
The focus for this evidence-based CPD approach is the process of collection, analysis and reflection on evidence arising from classrooms and how this provided the impetus and motivation for teachers to transform their practice, (p.578).
This quotation implies involvement of the teacher in the research-design process, a point that I develop further in the next section.
The designer’s planning should include a strategy for “dissemination” or “taking to scale”. In this context, the key term is “sustainable”. The half-life of materials has not, to my knowledge, been documented, but I would guess that 10 years from the first date of ‘roll-out’ might be an average figure. Insofar as materials catalyse changes which have a lasting effect on teachers’ understanding of their role and their practice, one might hope for a more lasting effect. One study of this issue (Coburn, 2003) emphasises that scalability has four dimensions: the depth of the reforms, their sustainability, their spread, and the shift to ‘ownership’ by teachers. Studies of the combination of sustainability and of scalability have identified the need both for a detailed model of teacher learning communities which provide collegial support and stimulus at a school or district level (Wiliam, 2007/8), and for a model for implementing any new inputs – which should include consideration of how materials and the associated CPD might so effect changes in pedagogy that the desired changes in students’ learning are achieved (Thompson, L. & Wiliam, D., 2005).
Designers, researchers and teachers; models of interaction4
To attempt, in this discussion, to embrace the full range of styles, ideologies and methodologies of research would be impractically ambitious and of doubtful value. I choose here to describe two possible models of researcher-designer interaction. For the first, the designer may fashion the product, guided, both in initiating and in refining, by feedback from researchers. The process could include pilot and/or trial studies of classroom use in which the evaluative skills of researchers would be crucial to collecting and interpreting data that would serve to improve the product. This I understand to be the model that is presented in Schoenfeld’s discussion. Insofar as the teacher’s role is discussed, it is as a user to be consulted and assisted rather than as an active participant in generation. It would be necessary to add to any such model the design of dissemination/scalability, and here again research results may provide helpful ideas about the most effective strategies.
In the second model, the first design involves both researchers, the designer and experienced teachers. The innovation is then put out to trial. But now the role of the teachers taking part in the trial is made explicit – they are co-developers. Their own experiences, and their achievements in transforming the initial ideas so that they become more practical and workable, will be seen as establishing new knowledge about the ideas, which will serve their re-formulation. The role of the researcher as formative evaluator will be expanded to include both observing outcomes and documenting any transformations into practice that the teachers will accomplish, including their reasons and justifications. This is similar to, but more comprehensive than, an approach described in Black and Wiliam (2003) in work which was based on established research findings, but which aimed, on the basis of these, at the formulation of new knowledge about practical implementations needed to achieve effective changes in classroom practice. It was on the basis of these new findings that subsequent dissemination was founded. More generally, it is worth noting here that a significant part of the designer’s task in any innovation task is the development, for and with teachers, of formative and summative assessment skills to work with, or support, the innovation’s design. Whilst the summative aspect may be constrained by accountability pressures, these are relatively infrequent and teachers have opportunity and responsibility to implement their own summative assessments to guide pupils and others over the years between high-stakes test events (ARG, 2006; Gardner et al., 2008; Black et al., 2010).
There is room for many variations in, and combinations between, these two models. Yet they differ in two important ways. The first is in the role that they aim to give to teachers in the development of a design. The second relates to the designers foresight in the preparation for sustainability and scalability: the more active involvement of teachers may mean that the outcome is more robust, both in being closer to classroom and school system realities, and in being more attractive to a wider range of users because they can identify with the trial teachers involved who may well be the leading ambassadors for the innovation. What is implied here is that the outcome of design is not merely a set of books and other artefacts to be disseminated by publishers. A more ambitious outcome is envisaged, of a change in pedagogy disseminated through a programme of CPD led by the designers, researchers and the trial teachers, in which all participants are supported and inspired by educational materials which both represent and underpin the changes which the trio have collaborated to develop.
Whilst there is a danger of expanding consideration of the design of educational materials into a theory of everything educational, I argue that it is essential to locate consideration of design in a broad context of innovation in education. Schoenfeld’s paper is an important and necessary contribution here, for research is one aspect of this context of particular importance to the designer. The burden of the argument above is that teachers’ development is another important aspect that ought to be taken seriously – by planning in terms of a tri-partite partnership between designers, researchers and practising teachers.
ARG (2006) The role of teachers in the assessment of learning. Assessment Reform Group, London : Institute of Education.
Also available at: http://www.assessment-reform-group.org/
Black, P. (2008a). Strategic Decisions: Ambitions, Feasibility and Context. Educational Designer 1 (1)
Available at: http://www.educationaldesigner.org/ed/volume1/issue1/article1
Black, P. (2008b). Making teacher change happen. Chapter 12 pp.145-156 in Gewirtz, S., Mahony, P., Hextall, I. and Cribb, A. (eds.) Changing Teacher Professionalism: International trends, challenges and ways forward. London: Routledge.
Gardner, J., Harlen, W., Hayward, L. and Stobart, T. (2008). Changing assessment practice; process, principles and standards. Assessment Reform Group.
Available at: http://www.assessment-reform-group.org/
Schoenfeld, A. (2009) Bridging the cultures of educational research and design. Educational Designer. 1 (2)
Available at: http://www.educationaldesigner.org/ed/volume1/issue2/article5
Swan, M. (2008). A designer speaks: Malcolm Swan. Educational Designer. 1(1).
Available at: http://www.educationaldesigner.org/ed/voulme1/issue1/article3
 The terms ‘pedagogy’, ‘instruction’, ‘teaching, are used with a variety of meanings; some authors use them as interchangeable, others would regard any discussion of pedagogy as involving far more than classroom instruction - including the social, cultural and political contexts within which teachers carry out instruction. It is used here to denote the work of teachers in the classroom and their closely associated activities.
 I do not consider here the use of ICT or of other distance learning artefacts which are designed for individuals learning on their own. Even in systems which rely on such artefacts, interaction with teachers or tutors usually plays an important part - as in the programmes of the UK’s Open University.
 The contrast between this project and an earlier conventional curriculum innovation is discussed in detail in Black (2008b).
 I do not distinguish here between a first, small-scale or pilot stage in the initial development of an innovation, and the second larger scale trial of the acceptability of the developed product. In terms of this distinction, and where the two are conducted separately, my comments are more relevant to a pilot stage.
About the Author8
Paul Black was Professor of Science Education and Director of the Centre for Science and Mathematics Education, at Chelsea College in London and subsequently at the Department of Education and Professional Studies, King's College London. He is currently engaged in a research and development work to improve classroom practices in formative assessment. Paul has been a visiting Professor of Education at Stanford University, California.
For many years he was involved closely with curriculum development work with the Nuffield Foundation in science and in design and technology, at primary, secondary and tertiary levels. He was chair of the government’s Task Group on Assessment and Testing in 1987-88 and deputy chairman of the National Curriculum Council from 1989 to 1991. He has served on three committees on the USA National Research Council.
He took his first degree in physics, and subsequently obtained his Ph.D. in Crystallography at the Cavendish Laboratory in Cambridge in 1954. Between 1956 and 1976 he was a faculty member in the Department of Physics in the University of Birmingham but his interests gradually moved from research in physics to research and development in science education, culminating with his move to Chelsea.