Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.philol.msu.ru/~kedr/brno-kedr.pdf Дата изменения: Fri Apr 18 13:41:44 2003 Дата индексирования: Mon Oct 1 22:35:08 2012 Кодировка:

Distance Education and the problem of CD-ROMs and Web-based Language Courses' evaluation
Kedrova Galina, Ass. Prof., Director of the Center of New Technologies for Humanities, Kisterjova Maria, Graduate student Moscow State Lomonossov University, Philological Faculty 119992 Moscow, Vorobjovy Gory, MGU Russia Phone: 007 095 939-14-78; Fax: 007 095 939-55-96 Email: kedr@philol.msu.ru

masha_kis@mtu-net.ru
ABSTRACT This paper will present the main results of the original research work in progress targeted at elaboration of a universal system of parameters (criteria) for modern electronic educational resources evaluation. During the initial phase of the project our efforts were concentrated upon the problem of unification and standardisation of the data extracted from various lists of assessment criteria widely used for educational CD-ROMs and Web-based courses comparisons. The relevance of this information is determined by predominance of the Constructionist concept of all cognitive processes in modern Distance Education practice. This theoretical approach postulates electronic learning resources as the key component of general educational scheme: tutor -e-manuals - student, while the third one (student) becomes the main subject of the whole process. Both these reasons enhance the responsibility of DE providers and administrators in regard to the selection of relevant and efficient didactic material as well as information support of education in general. Another essential aspect of modern DE technologies was outlined by German educator O.Peters (Fern Universitaet in Hagen, Germany). His view of parallelism between the organisation and operational processes in Distance Education and basic principles of any industrial process we consider as very promising. Peters emphasises that in both cases work processes should be formalised and products should be standardised. So, the problem of evaluation and standardisation turns to be the crucial one for all aspects of contemporary both distance and traditional (conventional) education especially valuable for the domain of electronic learning resources production. In our work we've suggested that the great variety (thematic and qualitative) of educational CD-ROMs and Web-based courses should be examined, classified and evaluated. Such comparative analysis should be based upon an optimal set of objective criteria that could become in the future a productive source for creating an adequate system of universal educational facilities standards. Moving this way we've had to confront several serious problems. One of the main problems is the great diversity of theoretical assessment models. The empirical data we've obtained through thorough examination of various systems of evaluation criteria convincingly demonstrates that at present there are no globally accepted standard methods for the evaluation of educational CD-ROMs, therefore, neither acknowledged decision procedure allows a conclusion to be drawn about advantages of one e-course over another or about their adequate didactic application. There is no adopted common terminology within the subject either. Our purpose thus was to analyse and compare different lists of evaluation criteria of CD-ROM courses in use. The criteria under consideration were revised, updated and grouped into several basic structural and functional clusters. These groups of assessment criteria correspond to main coordinates in the process of teaching and learning, obvious and obligatory criteria being included into each section. Thus obtained the new optimised list was tested with several electronic courses of Russian, English, Portuguese and German languages both on CD and on the Web. The final enlarged and experimentally approved repertoire of evaluation criteria will be submitted for discussion.


Introduction
At the moment we are in a situation where e-publishers, software companies, colleges` and university teachers and professors are offering thousands of computer-based manuals, e-textbooks, electronic guides and grammars, electronic reference books both on CD and in the Web. These products are designed to sustain modern Distance Learning in every form: as more traditional (conventional) DE ,,by correspondence" or as a fully online courses; as a basic curriculum for self-study or as an additional support for traditional learning, thereby ostensibly altering centuries-old methods of teaching and learning in general. Few of these products, however, make significant improvements to either the cost or quality dimensions of student learning; instead, they frequently try to replicate face-to-face pedagogues and organizational frameworks - to our regret we should constate that in most cases with less success. Common explanation for the situation deals with low quality of a product or lack of professional skills in IT use as well as in Web-pedagogy in general among both teachers and students. However, we do not consider these the only reasons, nor the critical ones. Despite the widespread strong tendency for adoption of educational CD-ROMs and Web-based educational technologies by educational institutions, teachers and learners, one knows very little about their role in education in general and even less about general guidelines of technology-supported pedagogy. The relevance of this information is determined by the constructivist concept of modern Distance Education's fundamentals. This theoretical approach postulates the key position of the intermediate component of "tutor - educational e-books & manuals - student" triad with an especially active role of the third one. Both factors enhance the responsibility of distant courses administrators in regard to the selection of relevant information support. Thus CMS and LMS based on a concept of e-course become the core component of any DE program. Another essential aspect of modern DE technologies was outlined by German educationalist O.Peters (Fern Universitaet in Hagen, Germany). His view of parallelism between the organisation and operational processes in Distance Education and basic principles of any industrial process we consider as very promising. Peters emphasises that distance study has a ,,special relationship with the industrial production process in so far as the production of study materials in itself is an industrial process built into the whole teaching process as a constituent part, quite unlike the production of textbooks, for example". [1] Later he outlined main parallels between distance teaching and the production process in general. These coincidences were phrased in following statements: 1. According to the principle of rationalisation, individual work as was traditional in the craftsmen's trades changes at an early stage to a production based on the division of labour (e.g., in factories), and this later leads to the development of assembly lines and mass production. This tendency is now obvious for DE institutions and programs. 2. Work processes initially characterised by the use of tools are increasingly restructured by mechanisation and, later, automation. The same situation is observed either for e-courses authoring or in the market of distant programs in general. Peters also notes that it must necessarily follow that these fundamental changes lead to the following results: · The preparatory phase becomes increasingly important. · Success depends, among other things, on systematic planning and organisation. Hence, scientific measures of control are needed. · Work processes must be formalised and products standardised. · The production process is objectified. · Each developmental step towards increased mechanisation leads to changes in the function of those involved in the production process. · Small concerns are no longer able to raise the investment needed for developmental work and technical equipment. A strong tendency towards concentration and centralisation becomes noticeable. The same as for industrialisation in 19th and 20th centuries we consider thus the problem of standardisation to be the crucial one for all forms of DE today and as an especially important issue for instructional materials. Main parameters of those standards could be derived experimentally from observation of the current use of educational CD-ROMs and Web-based learning content. Therefore we state that the multiplicity and great diversity of educational CD-ROMs and Web-based courses should be


assessed, evaluated and classified. We also hypothesise that evaluative principles might correlate with the basic parameters of SCORM, new object-oriented ideology under development for ADL Environments.

Methodology
Our purpose thus was to analyse and compare different lists of evaluation criteria specially composed and commonly used for CD-ROM courses efficiency comparisons. Comparative analysis of their potential efficacy for education turns out to be very informative for elaborating an adequate system of universal global educational facilities standards. However the empirical data obtained for various systems of evaluation criteria demonstrate that at present there are no globally accepted standard methods for evaluating educational CD-ROMs, therefore neither admitted system allows a conclusion to be drawn about advantages of one e-course over another or about their adequate didactic application. There is no adopted common terminology within the subject either. Therefore, as one proceeds with evaluative efforts, evaluators are have to employ a wide range of evaluation models, methods, and strategies that might take into account the salient features of the assessment object, that is: particular teaching and learning material as well as particular teaching and learning models in use. Our experimental data (evaluative lists for CD-ROM courses of foreign languages) were collected from following sources: http://polyglot.lss.wisc.edu/lss/workshop/ http://calico.org/index.html http://nflrc.hawaii.edu/aboutus/ithompson/flmedia/prog_frame1.htm http://dbs.tay.ac.uk/instil/projects.htm http://www.internettime.com/itimegroup/forum/biblio http://www.tesol.edu/isaffil/intsec/f-call.htmlhttp://www.academic.com/ http://www.skillsoft.com/ http://www.firstclass.com/products/Education/ http://www.knowledgeability.biz/weblearning/ http://www.history-of-call.org/ http://iteslj.org/links/TESL/Internet/Web_Authoring/ http://home.att.net/~walt.crawford/cdrom.htm http://www.archimuse.com/papers/cidoc/cidoc.mmwg.eval.1.html http://clear.msu.edu/dennie/reference/ http://www.lrc.salemstate.edu/aske/lgsoftware.htm http://nflrc.hawaii.edu/aboutus/ithompson/flmedia/evaluation/general/gencriteria.htm http://www.lmp.ucla.edu/default.asp http://www.ed2go.com/ http://www.edhelper.com/ http://info.coursecompass.com/website/faq.html http://dbs.tay.ac.uk/instil/links2sites.html http://www.blackboard.com/ The next step was to analyse an experimental data set of evaluation criteria lists for CD-ROM courses with a view to optimise and unificate (to make congenerous) all the parameters. The resulted (optimised) list was applied to evaluate several CD-ROM courses of Russian, English, Portuguese and German languages. Thus we have tested an updated list and subsequently have classified the assortment of criteria by several thematic groups, obvious and obligatory criteria being included to every section. These sections determined the whole assessment's activity network.

Results
The following plan of a foreign language CD-ROM courses summary of these investigations, special attention has been (pedagogical models). Every section of the scheme comprises 1). Table 1. Updated (optimised and universalised) evaluation 1. Imprint information 2. Who is the audience: 3. Intended using: (mostly Elementary level) evaluation is a paid to the problem of teaching methods the most essential aspects only (see Table criteria list.

Who produced the package? (What (if any) is their academic standing?) Which Grade Level does the course address? For self-instruction; as a textbook supplement etc.


4. Scope: 5. Goals: 6. Currency:

7. General content:

7.1.Coverage: 7.2. Sources of the information: 8. Navigation:

8.1.Indexing:

Equivalence in terms of hours, semesters and years of instruction; Number of lessons or instructional units; Are the goals and objectives for the program as a whole, each unit and each activity clearly described? Currency and accuracy - is the info up to date and accurate? Is the course updated? How often? Is the file unique? Does the file overlap with other similar tools? Is the course biased culturally or nationally? What is its cultural and moral tone? Is the course topically aimed? Which subjects/topics does the course cover? How accurate is the information? Is there an equivalent to the file in printed formats? Is navigation easy? Is a natural language searching available? Can one search by abbreviations? Can "sounds like" searching be done (e. g., "telefon" retrieves "telephone")? Is there a contents menu? How comprehensive is the indexing of individual journals? Are abbreviations easily understood? Are full titles given? Is there a contents page with relevant links? Is the course targeted at the right level and is it appropriate for the way you teach or would it mean changing or modifying your teaching style? Is the course suitable for learners of various ages? Is a good speaking practice given? Does it aim at specific outcomes? What is the proportion of various kinds of excersises? Authenticity of tasks; Vocabulary and structures covered ­ what are they and are they appropriate? How long does it take to do the task(s)? Does the course allow a curriculum? Is the course consistent with graded programme of study? Does the course include any assessment? Is the course suitable for the National Curriculum subject area? Does the course give pre-tests and post-tests? Is the course easy to use for teachers and for learners? Does the file create real-world learning situations, applications, or simulations that parallel real-life?

9. Methodology:

9.1 Curriculum: 9.2 Assessment:

9.3 Learning:


Does the course encourage creative/critical thinking skills? Is there cooperative environment? Is it noticed how long does it take to study the course? Is on-screen help available at all times? Is general help available online? Are tutorials available both on screen and in print? Is there a user manual? Is there a quick reference card? Does the user receive feedback? 10.1 Feedback: Is there the availability of learner-controlled feedback? Can the software track learner interaction with the program? Is there full text available for display? 11. User interface: Do the hypertext links work well? Is an easy exit from the program available at all times? Are the commands throughout the course consistent? Is the course a guided tour or are there 11.1. Level and type of interactivity: opportunities for interaction (is it internetenabled)? Do graphics, video and audio help users to concentrate on content? Are there choices depending on user's level? 11.2 Customization: Can users customize the operation of the program? Do animation, video and sound serve a 12. Multimedia's characteristics: pedagogical purpose? Are images appropriate or do they detract from the meaning? Is the course a network product? Does the course include networkability? How extensively is speech recognition/processing 13. Speech recognition: utilized in each unit of the program? What is the function of speech recognition/processing in this program (voice navigation, pronunciation instruction, speaking practice)? How does the system react to nonnative speech (recognizes it despite of mistakes, asks for repetition, does not recognize it)? The further phase of our work was the new criteria's set validation. While testing our evaluation criteria set for Web-based educational material we've had to complete the list with new items. Taking into account the very specific nature of the Internet technologies we've considered as additional (though obligatory) the following new evaluation parameters: 14. Web characteristics: Are links to more information on the topic provided? How many dead links are on the page? Are the links current or updated regularly? Another specific feature of the Web-information deals with various conditions of access to information source. While any CD-ROM course is a commercial product ex definitione the best part of Web-based 10. Help:


resources is "access-free" (open source). Thus another new criteria for Web-based course evaluation should be added to the list. This criteria is formulated as follows: Is it free or is there a fee to obtain the information?

Discussion
The initial set of evaluative criteria under consideration were revised, updated and grouped into several basic structural and functional clusters. These groups of assessment criteria correspond to main coordinates in the process of teaching and learning, obvious and obligatory criteria being included into each section. Thus obtained the new optimised list was tested with several electronic courses of Russian, English and Portuguese languages both on CD and on the Web. The final enlarged and experimentally approved repertoire of evaluation criteria was submitted for discussion. This new set was analysed from structural and functional point of view. Our set of evaluation criteria thus consists of basic functional clusters, each corresponding to an essential constituent of the teaching and learning process. They are: 1) User's profile; 2) Organisational model of learning process; 3) Specific parameters of content; 4) Interfacial properties: 4.1. interactivity; 4.2. adaptivity; 4.3. design optimisation and informativity 5) Pedagogical models; 6) Custom values. The projection of this classes onto the initial list of evaluation criteria is presented in table 2. Table 2. Relation betw een functional clusters and evaluation criteria. Functional cluster Criteria (from the list) 1. User's profile 2-3. Which Grade Level does the course address? For self-instruction; as a textbook supplement etc. 2. Organisational model of learning process 4-5. Equivalence in terms of hours, semesters and years of instruction; Number of lessons or instructional units; Are the goals and objectives for the program as a whole, each unit and each activity clearly described? 3. Specific parameters of content 6-7, 7.1, 7.2. Currency and accuracy - is the info up to date and accurate? Is the course updated? How often? Is the file unique? Does the file overlap with other similar tools? Is the course biased culturally or nationally? What is its cultural and moral tone? Is the course topically aimed? Which subjects/topics does the course cover? How accurate is the information? Is there an equivalent to the file in printed formats? 4. Interfacial properties 8, 11.1. Is navigation easy? Is a natural language searching available? 4.1. interactivity Can one search by abbreviations? Can "sounds like" searching be done (e. g., "telefon" retrieves "telephone")? Is the course a guided tour or are there opportunities for interaction (is it internet-


4.2. adaptivity

4.3. design optimisation and informativity

5) Pedagogical models

enabled)? Do graphics, video and audio help users to concentrate on content? 8.1, 11, 12. Is there a contents menu? How comprehensive is the indexing of individual journals? Are abbreviations easily understood? Are full titles given? Is there a contents page with relevant links? Is there full text available for display? Do the hypertext links work well? Is an easy exit from the program available at all times? Are the commands throughout the course consistent? Do animation, video and sound serve a pedagogical purpose? Are images appropriate or do they detract from the meaning? Is the course a network product? Does the course include networkability? 10, 10.1, 11.2. Is on-screen help available at all times? Is general help available online? Are tutorials available both on screen and in print? Is there a user manual? Is there a quick reference card? Does the user receive feedback? Is there the availability of learner-controlled feedback? Can the software track learner interaction with the program? Are there choices depending on user's level? Can users customize the operation of the program? 9, 9.1, 9.2, 9.3. Is the course targeted at the right level and is it appropriate for the way one teaches or would it mean changing or modifying one's teaching style? Is the course suitable for learners of various ages? Is a good speaking practice given? Does it aim at specific outcomes? What is the proportion of various kinds of excersises? Authenticity of tasks; Vocabulary and structures covered ­ what are they and are they appropriate? How long does it take to do the task(s)? Does the course allow a curriculum? Is the course consistent with graded program of study? Does the course include any assessment? Is the course suitable for the National Curriculum subject area?


6) Custom values

Does the course give pre-tests and post-tests? Is the course easy to use for teachers and for learners? Does the file create real-world learning situations, applications, or simulations that parallel real-life? Does the course encourage creative/critical thinking skills? Is there cooperative environment? Is it noticed how long does it take to study the course? 1, 10, 13. Who produced the package? (What (if any) is their academic standing?) Is the course easy to use for teachers and for learners? Is on-screen help available at all times? Is general help available online? Are tutorials available both on screen and in print? Is there a user manual? Is there a quick reference card? How extensively is speech recognition/processing utilized in each unit of the program? What is the function of speech recognition/processing in this program (voice navigation, pronunciation instruction, speaking practice)? How does the system react to nonnative speech (recognizes it despite of mistakes, asks for repetition, does not recognize it)?

Conclusion
The optimised universal set of evaluation criteria that we've obtained as a result of our investigation is supposed to be a highly efficacious tool for assessment of any e-course, either on CD-ROM or on the Web. We suppose that our criteria should be taken into account at every stage of e-course construction and could be recommended as obligatory targets for authors, content-providers and programmers. Our basic functional clusters of evaluative parameters may form a solid base for elaboration of a global system of e-learning norms and standards.

Bibliography
[1]Keegan D. Foundations of distance education. - L.; N.Y., 1996. - XIII, 224 p. [2]Moore M. Theory of transactional distance // Theoretical principles of distance education / Ed. by Keegan D. - L.; N.Y., 1993. - p. 22-38. [3] . . . . .: , 2002. - 136 . [4]Forsyth, I. (1997), Teaching and learning materials and the Internet, Kogan Page. [5]Peters Otto. Distance education and industrial production: a comparative interpretation in outline. - in D. Sewart, D. Keegan and B. Holmberg (eds) (1983) Distance Education: International Perspectives, London and New York: Croom Helm Routledge, pp. 95-113. [6]Rosemberg, J.: e-Learning Strategies for Delivering Knowledge in the Digital Age. McGraw-Hill, New York,2001.

Biography


Kedrova Galina, Ass. Prof., Department of Russian Language, Philological Faculty of Moscow State Lomonossov University (Moscow, Russia), Director of the Centre of New Technologies for the Humanities, Ph.D. Area of scientific interest comprises: Theoretical and Applied Linguistics, Informatics, Computer-Assisted Language Learning (CALL), Web-pedagogy, General and Russian Phonetics, E-courses engineering. Kisterjova Maria, Graduate Student of the Department of Romance-German Philology (Portuguese Language), Philological Faculty of Moscow State Lomonossov University. Area of scientific interest comprises: Computer-Assisted Language Learning (CALL), E-courses evaluation, Methodical aspects of E-courses authoring (Portuguese Language).