DEVELOPING ALTERNATE CRITERIA FOR THE EVALUATION OF TEXTBOOKS

Abstract

Textbooks are one of the major teaching and learning resources used in schools. In Pakistan, the Curriculum Wing (CW), Ministry of Education is the apex national institution that frames and approves curricula for all subjects up to class XII. The CW is empowered to prescribe and withdraw textual materials for schools in Pakistan. Textbook evaluation and approval is one of the key responsibilities of CW. Textbook development and evaluation has always been a subject of debate in Pakistan owing to number of reasons. This study has been designed to explore and develop professional resources for facilitating objective evaluation of teaching and learning materials. The study intends to achieve the forth-coming objectives: 1) identify indicators and criteria of quality textbooks through literature and good practices employed internationally in the field; 2) critically appraise the existing indicators and criteria used in Pakistan for evaluating the textbooks by comparing them with the indicators; and 3) develop alternative criteria for review, evaluation and approval of quality textbooks in Pakistan. The key premise of the study is based on the understanding that it is difficult to assess quality in textbooks if there are no pre-set indicators for them. This study takes the approved textbook as a product and attempts to apply the general principles used for assessing the quality of any given product with product specific customization. The principal conceptual framework for developing indicators for quality textbooks used for this study is Garvin’s framework for quality products which has eight dimensions: a) vPerformance; b) Features; c) Reliability; d) Conformance; e) Durability; f) Serviceability; g) Aesthetic; and h) Perceived Value. The related literature was reviewed and a theoretical framework was developed on the basis of the findings. Criteria for textbook evaluation along with indicators of quality textbooks through literature and good practices employed internationally were identified. To critically appraise the existing criteria and indicators used in Pakistan for evaluating the textbooks by comparing them with these identified indicators, the study was delimited to grade three textbooks in the subjects of science and mathematics. Out of the sampled participants 230 (95.8%) teachers, 67 (55.8%) subject specialists /teacher educators and 51(85%) experts participated in the study. A sub group 23 (76%) of identified expert was requested to participate in building consensus on indicators and validate their compliance conditions for the alternate criteria for textbook evaluation. Both qualitative and qualitative data were collected for the study. For quantitative data, the research instruments were administered to: a) school teachers of three districts- two from Punjab and one Islamabad Capital Territory; b) subject specialists /teacher educators from Govt. Colleges for Elementary Teachers in the province of the Punjab; and c) experts from institutions associated with textbooks development and review in Pakistan. Textbook Evaluation Forms for Teacher/Teacher Educators and Textbook Experts were developed in the light of the identified indicators to seek their judgments on the approved and published books. The Cronbach Alpha values of these forms were 0.938 and 0.914 respectively. Interview protocols were developed to elicit the opinion of experts about various aspects of quality textbooks. Guidelines for content analysis of the approved textbooks were also developed. Data was mostly collected in workshop settings. Based on the results, alternate criteria and indicators for the objective evaluation of quality textbooks in Pakistan were developed. In the light of the data collected through the instruments, indicators for quality textbooks were agreed upon through the Delphi technique. For quantitative data; mean, percentages, ANOVA, Post Hoc, Kruskal-Wallies tests etc. were used for data analysis whereas for qualitative data; themes were identified for developing major categories of evidences. The study revealed inconsistencies in the curriculum reports (booklets) and vis-à- vis in approved textbooks. There was lack of homogeneity with regard to the level of understanding demanded by each of the approved textbooks. The criteria given to the viNational Textbook Review Committee to evaluate textbooks with respect to conformity of the curriculum objectives were not explicit for all members of the committee. Proper coverage of scope of the content in the approved textbooks was also an issue. In some books, it was not covered completely and in some of textbooks, other than the topics mentioned in the national curriculum were included. The approved textbooks both in the private and public sectors have given different amounts of space for the explanation of the same topics. Different approved textbooks adopted different approaches to layout, quality of paper, title, page layouts, pictures and line diagrams, presentation of culture, use of letter type and size of font, printing etc. According to the data collected through Textbook Evaluation Form (for experts) based on the identified indicators of quality textbooks, the approved textbooks in Pakistan are lacking in: a) content covers objectives given in the national curriculum; b) content is helpful in promoting ideology of Pakistan; c) content promotes conservation of environment; d) author(s) of the books are known for their mastery in the subject; e) binding and cover of the books are durable; and f) end of chapter assessments are according to student's mental development level. Keeping in view the identified indicators, the alternate criterion along with compliance indicators for the textbook evaluation was developed. In order to make the criterion more tangible, rubrics for each of the compliance indicators were developed in consultation with experts. These rubrics were designed to review each chapter or section of a textbook /supplementary teaching and learning material against the given indicators. The developed criterion was validated by the experts. In response to its three objectives, the researcher has a) identified indicators and criteria of quality textbooks, b) critically appraised the existing indicators and criteria used in Pakistan for evaluating the textbooks and c) developed alternate criteria for textbook evaluation.