You are at: Home
> E-Library > Paper
9710a > Paper9710a01
![](images/index_logo_lg.jpg)
![](images/index_trademark.gif)
LEARNWARE QUALITY
BACKGROUND PAPER
Prepared by Dr. Kathryn Barker
President, FuturEd
for:
Learnware Quality Working Group
Of the "Network of Networks" initiative of the
Education and Training Provider Network Project
October 27, 1997
TABLE OF CONTENTS
1.Purpose, Scope and Context
2.Criteria for Evaluating Learnware Quality
3.Who is Doing What…Resources for the Learnware Quality Initiative
4.Possible Next Steps
APPENDICES
Appendix 1: Working Definitions
Appendix 2: Related Topics that are Not Included in the Scope of the Study
Appendix 3: Context for the Study
Appendix 4: Print Evaluation Criteria
Appendix 5: Criteria for Evaluating Internet Information
Appendix 6: Library Selection Criteria for WWW Resources
Appendix 7: Internet Information Resources Evaluation Sites
1.Purpose, Scope and Context
This paper – a literature review of the current status of learnware quality assurance – has been prepared by Dr.
Kathryn Barker under contract to the Education and Training Provider Network. For purposes of this study,
learnware is computer software by which interactive education or training programs may be delivered to
individuals by electronic means, such as CD-ROM, Internet, or Intranet. Terminology is discussed more fully in
Appendix 1.0.
The purpose of the background paper was, as per the terms of reference of the Consultant’s Study, to identify
already-established initiatives which support quality assurance in electronically-provided education and training
courses. Accordingly, Dr. Barker of FuturEd, within the budget and timeline parameters, undertook to:
-
gather and sort information from
working group members and those sources they recommend, and:
-
on-line searches of sources relevant to,
( e.g., quality and quality standards)
-
types of learnware (e.g., various learning technologies)
-
elements of learnware (e.g., instructional design)
-
the use of learnware (e.g., technological efficacy, ease of use, distance learning).
-
prepare a comprehensive synthesis and background paper for the Working Group that includes:
-
existing written guidelines, standards, codes, or principles stipulating criteria for quality in
technologically-delivered learning materials and information about the organizations, in Canada and
elsewhere, which have developed them;
-
summary information on the nature of the guidelines, their primary constituency, methods of
implementation and assessment, content and accreditation issues and other categories as identified by
the Working Group and within the scope of learnware and quality assurance;
-
information on multi-person organizations which provide consulting, training, or information support
services to assist in the achievement of quality learnware; and
-
samples and appendices as required
Scope of the Study
This background paper was intended to provide an inventory of quality assurance initiatives specific to learnware.
The focus of the inquiry is the unique interface between format (technical standards), process (utility standards)
and content (information standards) for a particular type of software.
This concept is situated at the intersection of many related concepts, for example:
-
quality assurance in services, i.e., education and training (formal and non-formal learning), e.g.,
-
quality assurance in products, e.g.,
-
technology-based learning,
e.g.,
-
information dissemination, e.g.,
All of these concepts have some relevance; however, all can be examined separately and at great length.
Quality assurance in learnware touches on these, but it must be limited. Therefore, a lengthy discussion of
topics that are directly related but not included in the scope of this study is found in
Appendix 2.0.
Context for the Study
The environment in which this study takes place is one in which there is a growing concern for quality assurance
in education and training, in educational software, in the uses of technology in education, in distance education /
distance learning combined with the potential of the learnware market. A more complete discussion of this
context is found in Appendix 3.0.
2.
Criteria For Evaluating Learnware Quality
In a nutshell, there are not a lot of complete lists of criteria for evaluating the quality of learnware, particularly from
the consumer’s point of view. Here’s what was found in the time-limited on-line search that was conducted.
Toward the notion of quality standards for learnware
- In the literature bounded by the terms learnware, quality, software, education, and standards there are a wide variety of approaches to defining quality or determining evaluation criteria. The simplicity and inadequacy of what follows serves to emphasize the need for a more systems-based, holistic, consumer-oriented approach to quality assurance in learnware.
- Seven Steps to Responsible Software Selection
Prepared in 1995 by the ERIC Clearinghouse on Information and Technology, this list was created for educators to use. The not-so-useful steps are to:
- Step 1: Analyze needs
- Step 2: Specify requirements
- Step 3: Identify promising software
- Step 4: Read relevant reviews
- Step 5: Preview software
- Step 6: Make recommendations
- Step 7: Get post-use feedback
- The Multimedia Design Considerations supplied by J. Treuhaft of Algonquin College of Applied Arts and Technology
'Best Practice and Worst Practice Labels'
- In its successful application for SSHRC funding, EvNet (Network for the Evaluation of Education and
Training Technologies) listed the following best and worst practices in the application of technology in
education.
- Worst Practice Labels
- ‘high powered drill and kill’ programs
- ‘coldware’ user unfriendly
- skill destroying
- blocks access, or differentiates access by social characteristics (income, employment status, age, region, ethnicity, gender)
- loss of control
- random, aimless browsing of the Internet: time waster
- destruction of community and collaborative learning
- unidirectional (teacher to student sponge lecturing); no interactivity
- unidimensional (text or graphics or sound or animation or video)
- ‘old wine in new bottles’: repackaging old material with new graphics and sound bites on the world wide web
- distancing (technology creates barriers between people who prefer distant e-mail to personal contacts) speed for speed’s sake:pressure
to respond NOW to e-mail
- lack of personal support for finding solutions to computer glitches
- expensive and costly
- loss of privacy
- electronic gaming effects (violence; sexism)
- destruction of artistic talent; loss of art forms and culture
- Best Practice Labels:
- learner driver
- enhances control
- friendly and intimate; people centred (warmware)
- democratizes and deregulates the educational experience
- creates opportunities for meaningful interaction
- knowledge building, creative, artistic, constructive
- eliminates routine tasks, allowing more time for higher order thinking and learning
- inclusivity, equity of access (gender, race, income, region)
- multidimensional (text, sound, graphics, video and animation in a balanced symphony that is not overpowering)
- enabling (use of technology to overcome handicaps)
- extending the senses
- enhances user control over time; self-paced; not instructor paced or technology paced (any time, any place learning)
Presumably, although these criteria refer to educational technology that very broadly includes learnware, the negative labels are to be avoided and the positive labels are informal quality criteria.
Technology and the State Adoption Process
The goal of this recommended process, which is made available on the Internet by the Software Publishers Association, is "to provide recommendations to publishers, state textbook administrators, and state technology officers to serve as guidelines for lowering the barriers to the adoption of technology-based products." The recommended step-by-step process for adopting technology, including software, includes such pre-adoption issues as:
- Reviewer training for a fair review of key elements (unspecified) of specified products
- Product durability
- Hardware specifications
Recommendations from the US Secretary of Education to schools
- The 1995 US Education Secretary’s Conference on Educational Technology made the following recommendations for schools:
- Make sure software follows your curriculum objectives.
- Review software and allow end users to try it out before you buy.
- Obtain teacher and student feedback on software after they have used it for a while.
- Buy something on trial for one school, and if it works, buy it for others.
- Train teachers how to evaluate software.
Valuable Viable Software in Education
In order to provide examples of "successful software", a study was carried out by Educom’s Educational Uses of Information Technology Program (EUIT) in partnership with the Annenberg/CPB Project. Successful software is that which has value and is viable.
Value: Software is valuable if it has the capability of being used to help improve teaching and learning. Indicators of value can
include evaluation results, awards won, testimonials from users, and the like. A software’s value is independent of the extent to which it is used.
Viability: Software is viable if it is used by enough people for a long period of time that all its investors (original developers, funders, publishers, institutional support staff, faculty and students) can justly feel that they each have received an adequate return on their own investments in developing, acquiring, and/or learning to use the software.
The major findings of the study were that:
- If computer software is not adequately upgraded to cope with changes in operating systems, it probably will not last long enough to achieve viability. Most cases of valuable viable software have life cycles of as much as a decade, during which they have been upgraded several times.
- Most of the valuable viable software fall into the category of worldware: word processors, spreadsheets, statistical and other mathematical software, software for managing library services, telecommunications software, tools for design and composition in fields as diverse as architecture and music, tools for modeling and simulation, research databases on-line and on discs, programming languages, and software that increases access for disable people. The authors note that "The sheer size of the list hints at how pervasive, subtle, and profound the influence of software in education has become."
The key elements of the recommended National Strategy for Lifelong Learning on the Information Highway (from the Working Group on Learning and Training of the Information Highway Advisory Council, 1995) which are:
- attention to user needs
- affordability
- connection to learning professionals and organizations
- Canadian content
- accessibility
- management of negative impact
Existing quality standards for learnware
At this time, a small number of "sets of standards" have been developed, typically by an organization or agency with a particular perspective, e.g.,:
- company standards or company-specific standards for how courseware is produced, what learning models are
assumed, how information is formatted on the computer screen, how courseware producers work, and so on,
e.g., AMTECH – an Italian courseware producer
Oracle Corporation has produced the Oracle Learning Architecture with which it wants to establish the standard for education and training over the Internet. Oracle is drafting guidelines for learning objects that:
- play freely across multiple education server platforms,
- are presented via multiple instructional design environments, and
- have brokers to provide standard retrieval and routing services
Microsoft, Apple and many other companies are aware of the value of being the leader in educational technology standards. The standards leader can determine core technology upon which courseware is based. This will benefit society by stimulating the development of more and better courseware.
Standards development organizations which have standards architecture and ontologies such as
The Institute of Electrical and Electronic Engineers (IEEE) through its P1484 project with a mission to
develop standards, guidelines, and recommended practices for the area of computer-based learning, with
the goal of enabling tools, courseware, information, and services to be provided on a component basis.
IEEE P1484 has five task groups:
- Task Model Group which will characterize task descriptions, member prerequisites, learning objective, and so on.
- Reference Models Group which will establish guidelines for implementation of computer-based learning system components and define service systems that are available to all components
- Tools Protocol Group which will address educational systems consisting of one or more user tools and one or more instructional agents
- Session Management Group will specify a language for managing sessions in learning technology systems
- Authoring Tools Group which will produce guidelines for tools to author computer-based courses
The IEEE P1848 has developed a "computer-based learning system" model (attached). IEEE is attempting to be a forum for companies such as Oracle and Apple.
The American National Standards Institute (ANSI) which, though its ETG (Education Task Group) is:
- cataloguing existing standards that relate to both education and the Global Information Infrastucture, and developing relationships with standards-developing organizations
- identifying needs for new standards
- seeking standards developers to develop needed standards
- promoting understanding and use of standards with the education community
The Industry Research Educational Multimedia Task Force of the Directorate General of the European Union has as a goal to standardize educational technology. the purpose of the IMS Project is to develop standards for instructional use of the WWW to address the integration of information technology into teaching and learning; the goals of the IMS Project are to:
- increase the Web’s use as an instructional tool
- enable the widespread integration of information technology into teaching and
learning
- promote individualized learning environments
- reduce the learning curve for students and faculty
- increase the potential for collaboration and sharing among institutions
- stimulate the development of a robust commercial market for instructional software
- accelerate adoption and diffusion of technologies
Educational organizations
Educom – a nonprofit consortium of American higher education institutions that facilitates the introduction, use, and management of information resources in education
Educom’s National Learning Infrastructure Initiative which addresses the technological prerequisites needed to create an environment in which technology-mediated learning can flourish, such as open standards for networked learning applications
Here are the standards, then, that appear to exist to date.
The Industry Canada Quality Guidelines for Selecting Computer-based and Multimedia Programs for Training
A set of quality guidelines was prepared by Dr. Lynette Gillis in a 1996 project managed by the Knowledge Connection Corporation and supported by Industry Canada and several major corporate training organizations. In 1997 these guidelines were pilot tested and improved, in a project led by Dr. Gillis, with further support from Industry Canada, Knowledge Connection Corporation and major corporate trainers. Industry Canada plans to produce a digitized version for electronic distribution, and will present a progress report at the October 27th meeting of the E/TPN Working Group on Learnware Quality. Unlike other quality guidelines listed in this report, the Gillis/Industry Canada tool is not simply a "grocery list" of criteria, but rather an organized process for decisions concerning the selection of quality learnware. Criteria are presented in the context of a structured decision process.
Critical Success Factors
According to a publication of the Office of Learning Technology, the six critical success factors in the development of learnware, which are de facto criteria for quality, are:
- 1.partnerships in development
- 2.identification of learner needs
- 3.content with credibility
- 4.sufficient markets
- 5.appropriate delivery platform
- 6.links to accreditation
The four key characteristics of effective software:
- presentability – the software’s overall appeal to the user
- accountability – includes evaluation and objective accountability
- customizability – adaptability and flexibility
- extensibility – easy to augment, supportive of the creation of new things
The Code of Ethics for professionals who use educational technology, developed by the Association for Educational Communications and Technology (ACET), which include:
- commitment to the individual (9 standards)
- commitment to society (5 standards)
- commitment to the profession (9 standards)
Internet-related criteria lists
In the development of criteria for evaluating the quality of learnware, the evaluation criteria for websites and Internet information may make an important contribution. The criteria for evaluating Internet information ranges from the simplistic – e.g., Coolium -- to the highly complex, as per the University of Georgia.
To achieve "Coolium", the criteria for what makes the Cool Site for the Day, the webpage designer is directed to:
- create something useful
- be entertaining if you can’t be useful
- be aware of bandwidth
- think twice about your own picture
- surprise us
From the consumer’s point of view, there might be something to this!
At the other end of the spectrum, Wilkinson and others at the University of Georgia have developed a list including 11 criterion and 125 indicators. The criterion found in Evaluating the Quality of Internet Information Sources: Consolidated Listing of Evaluation Criteria and Quality Indicators could be the same as those used to evaluate learnware. They are:
- site access and usability (18 indicators)
- resource identification (13 indicators)
- author identification (9 indicators)
- authority of author (5 indicators)
- information structure and design (19 indicators)
- relevance and scope of content (6 indicators)
- validity of content (9 indicators)
- accuracy and balance of content (8 indicators)
- navigation within the document (12 indicators)
- quality of the links (13 indicators)
- aesthetic and affective aspects (13 indicators)
After conducting a study, the indicators of (1) information quality and (2) site quality were ranked in importance by experienced Internet users. This, too, could serve as the criteria for evaluating learnware quality.
In the middle, the Internet Public Library uses the following selection policy for quality information sources, i.e., products / services which:
- are high in useful content, preferably those which provide information in their own
- right rather than simply providing pathways to information
- are updated consistently
- are designed is such a way that any graphics are an attractive complement to the information rather than a flashy distraction from it
- provide text-only interfaces for non-graphical browsers
- show evidence of having been proofread carefully
- contain only "live" links, only to documents which are as relevant as the primary document
Resources that are selected / approved by the IPL receive the IPL Ready Reference Seal.
According to J. Jakevicius at the Idaho State University, the following is a list of recurring criteria when Internet resource evaluation is considered: content, authority, publisher-source, reference/awards, facts, documentation, bias, links and stability.
At the University of Washington, J. Alexander and M. Tate adapted five traditional print evaluation criteria to web resources. Located in Appendix 4.0, this process could be further adapted to learnware quality.
Possibly the most useful list of evaluation criteria, reproduced completely in
Appendix 5.0, was developed by A. Smith in New Zealand.
A different kind of Internet resource evaluation guide, reproduced in
Appendix 6.0, covers the same concepts.
An example of the various Internet rating styles can be found in the annotated list of Evaluation Sites reproduced in
Appendix 7.0.
Quality Indicators (of Internet Information Sources) as Ranked by Experienced Internet Users: both Information Quality and Site Quality
- According to the University of Wisconsin, the Ten C’s for Evaluating Internet Resources are:
- Content
- Credibility
- Critical thinking
- Copyright
- Citation
- Continuity
- Censorship
- Connectivity
- Comparability
- Context
Actual rating sheets for evaluating Internet sites have been produced by Teacher’s CyberGuide, and others.
3.
Who is doing what… Resources for the Learnware Quality Initiative
A directory of Canadian learnware developers, a service of the Training Technology Monitor
http://www.traintec.com/LW_directory.html
which includes company profiles: the people, tools and technologies, custom-developed products and off-the-shelf products
Recent studies of the Training Technology Monitor
http://traintec.com/studies.html including:
- Decision Tools: What to Consider When Partnering for Learnware
- Partnering for Learnware: Critical Success Factors in the Use of Technology-based Training by Human Resources Sector Councils and Industry Associations
- Partnering for the Use of Technology in Education: Who Does What
Learnware Evaluations, a service of the Knowledge Connection Corporation
http://www.kcc.ca/project/learn/home.html
that includes formative or alpha testing, impact studies of developed programs, and product evaluations against KCC criteria for learnware quality to ensure that programs are instructionally sound, easy to use and technically reliable, and a good fit with the learners, the task and instructional setting
An evaluation of the quality of information products and services on the internet by H. Tillman -- Evaluating Quality on the Net at
http://www.tiac.net/users/hope/findqual.html
– against the following generic criteria for evaluation:
- stated criteria for inclusion of information
- authority of author or creator
- comparability with related sources
- stability of information
- appropriateness of format
- software / hardware / multimedia requirements
Web evaluation forms for use by primary, intermediate and secondary grades (users) that include indicators for design, content, technical elements, and credibility
http://www.siec.k12.in.us/~west/edu/rubric1.html
Lucent Technologies’ Center for Excellence in Distance Learning (CEDL) which:
- creates distance learning solutions for customers
- collaborates with LT Business units, LT Bell Laboratories, and leading universities
- provides, at http://www.lucent.com/cedl/tips.html
10 Tips for Implementing a Distance Learning Program
- provides evaluation services for distance learning
The products and services of the National Council for Educational Technology (NCET) in the United Kingdom, specifically:
The study of quality in distance education in Australia, e.g.,
- a quality assurance framework, the elements of which are commitment (the intention of all staff to deliver efficient and effective service to students and themselves), systems (processes within the centre that help staff to offer clients the best possible service, and measurement (quantifiable, observable results highlighting opportunities for the centre to improve its products and services) at
http://ericir.syr.edu
Organizations concerned with learning technologies in Canada:
WICHE (Western Interstate Commission for Higher Education), at
http://www.wiche.edu/ has undertaken a number of projects, among them:
- A proposed catalogue of instructional materials which includes "criteria for submitting materials for inclusion in the catalogue" and "the catalogue from the learner’s perspective"
- Western Virtual University
An International Buyers Guide to Technology in Education is available from ISTE (International Society for Technology in Education); information is at
http://www.kenpubs.co.uk/iste
The Software Publishers Association http://www.spa.org/
provides, e.g.,
- The Education Market Section – a group of more than 750 member companies with an interest in education software
- A SPA Software Packaging Award
- The Education Market Report
- Education Technology Promotion Guide
Educom – a nonprofit consortium of higher education institutions that facilitates the introduction, use, and access to and management of information resources in teaching, learning, scholarship, and research -- has a large number of related projects, including, e.g.,
- The IMS Project (Instructional Management Systems) with information at
http://www.imsproject.org/ National Learning Infrastructure Initiative (NLII) intended to create new collegiate learning environments that harness the power of information technology to improve the quality of teaching and learning, contain or reduce rising costs, and provide greater access to American higher education.
- A related research project is currently under way at Douglas College in BC. Dr. Bruce Landon’s project is "A Comparative Analysis of Online Instructional Software" and he is limiting his study to particular software.
The TeleLearning Network, a national Center of Excellence project at
http://www.telelearn.ca/ is conducting a variety of research projects that are related to this study, e.g.,
- Project 2.3.2: Towards a Strategy for Supporting Commercialization of Education and Training Software
- Project 3.4: Design and Evaluation of Multimedia Tools
- Project 6.2.1: Course Design and Development Workbench
- Project 6.2.3: Instructional Design Methodology for Computer-Supported Collaborative Learning
NCRTEC (North Central Regional Technology in Education Consortium)
http://www.ncrtec.org/ has a downloadable computer program called "The Learning with Technology Profile Tool." It is intended to help educators think carefully about their practice in the areas of engaged learning and technology.
A list of Sources for Reviews of Educational CD-ROMs and Software, dated 1995, is located at
http://www.iat.unc.edu/guides/irg-31.html
A Course Developer’s Standards Guide (Draft July 1997), developed by the East-West Project, is available at
http://teleeducation.nb.ca/eastwest/standards/
From/at the American Association for Higher Learning, among other things:
- the AAHE Teaching, Learning and Technology (TLT) Roundtable Program seeks to improve teaching and learning through more effective use of information technologies while controlling costs. Rights and Responsibilities for Electronic Learners at
http://www.aahe.org/tltr-bor.html
- the Flashlight Project which is developing evaluation tools for studying uses of computers, video and
telecommunications
"The only" scholarly journal for the educational technology field focusing entirely on R&D: Educational Technology Research and Development
http://206.67.154.75/Pubs/etr_d.html
Grants to schools, via the NFIE (National Foundation for the Improvement of Education, at
http://www.hfie.org/ ), grants from the Microsoft Corporation (proceeds from Bill Gates’ book) to use technology to improve student learning
An Education Technology Promotion Guide, from the ISTE (International Society for Technology in Education) Private Sector Council and the Software Publishers Association, intended to help educators increase awareness, gain support and raise money for the integration of technology into their schools and districts (contact:
cust_svc@ccmail.uoregon.edu )
- ISTE, representing more than 40,000 educators, promotes appropriate uses of technology to support and
improve teaching and learning
- ISTE sells an Educational Software Preview Guide listing more than 700 titles of favorably reviewed software for
K-12 classroom use (see http://www.iste.org/publish/prevguid.html
)
THE Online (Technological Horizons in Education) publishes courseware assessments and evaluations, but not online (?)
http://www.thejournal.com/
In 1995, the Information Highway Advisory Council claimed to be "examining the pros and cons of establishing national standards – both pedagogical and technical – for the delivery of learning and training on the information highway" and asked "What are the main considerations to be taken into account with respect to this issue?" (from a Working Consultation Document entitled Lifelong Learning on the Information Highway)
A framework for thinking about standards specific to Intelligent Educational Systems is provided by Task
Ontology Design for Intelligent Educational/Training Systems, available as a position paper for the 1996 ITS
Workshop on Architectures and Methods for Designing Cost-Effective and Reusable ITSs, at
http://advlearn.lrdc.pitt.edu/its-arch/papers/mizuguchi.html
4.
Possible Next Steps
Having reviewed this document, Jim McPherson has concluded that:
The initiatives identified are, for the most part, regional or institutional initiatives. Increasingly learnware is crossing national and sectoral borders (between countries and from academia to industry). While there are lots of guidelines, principles and standards, there is no internationally-coordinated set of guidelines representing a consensus of professionals and consumers in many countries. If Canada wants to export learnware, it will need to do it within the context of Quality Guidelines that are developed and accepted internationally…not just as a Canadian initiative. Consumer input would facilitate ensuring general acceptance by all parties.
The recently-established Global Alliance for Transnational Education (GATE) could provide a logical framework or launching pad for an initiative which will need to be launched with stature and credibility. Canadians could negotiate a collaborative relationship with GATE to achieve this, and could have the work co-funded by GATE and Canadian sponsors.
To that end, possible next steps are to:
- Establish and mandate a Canadian Learnware Quality group to work with peers in other countries on the development of a clear and communicable set of guidelines for learnware quality for both the education and corporate training sectors. The international group is herein tentatively called "Learnware Quality International" (LQI).
- The LQI group would ensure that the learnware "consumer", or the ultimate learner constituency, is represented in this development process.
- Using the LQI guidelines development group as a nucleus, build an international network of qualified instructional design professionals who could help developers and educators to ensure that their learnware products comply with international guidelines. Compliance could be communicated to consumers through the use of an internationally-recognized seal of quality, indicating that guidelines have been followed and an evaluation report is available from a qualified instructional design professional.
Clearly, from the amount of information that exists, there is no need to reinvent the wheel.
Additional References
Collins, A. (September 1991). The Role of Computer Technology in Restructuring Schools. Phi Delta Kappan, p.
28-36.
Commonwealth of Learning. (1994). Quality Assurance in Higher Education. Vancouver: author.
Dunning, P. (1997). Education in Canada: An Overview. Toronto: Canadian Education Association.
Educom Review Staff. (1995). Roger Shank: End Run to the Goal Line, in Educom Review
http://www.educom.edu/web/pubs/review/reviewArticles/30114.html
Harasim, L. (undated). On-line Education: A New Domain.
http://www.icdl.open.ac.uk/mindweave/chap4.html
Jones, G. (1997). Cyberschools: An Education Renaissance. Englewood, CO: Jones Digital Century Inc.
Kennedy, M. and Kettle, B. (Summer 1995). Using a Transactionist Model in Evaluating Distance Education Programs.
Canadian Journal of Educational Communication, p. 159-170.
Office of Learning Technologies. (1997). Critical Success Factors.
http://olt-bta.hrdc-crhc.gc.ca/info/online/green/factors.html
Oppenheimer, T. (July 1997). The Computer Delusion. The Atlantic Monthly at
http://www.theatlantic.com/issues/97jul/computer.html
Ravitch, D. (1995). National Standards in American Education: A Citizen’s Guide. Washington: Brookings Institute.
Reibel, J. (1994). The Institute for Learning Technologies: Pedagogy for the 21st Century. Columbia University: Institute
for Learning Technologies. At http://www.ilt.columbia.edu/ilt/papers/ILTpedagogy.html
Smith, Alastair G. (1997). Testing the Surf: Criteria for Evaluating Internet Information Resources. The Public-Access
Computer Systems Review 8, no. 3. http://info.lib.uh.edu/pr/v8/n3/smit8n3.html
Supply and Services Canada. (1996) Highlights of Departmental S&T Action Plans in Response to Science and
Technology for the New Century. Ottawa: Government of Canada.
Twigg, C. (1996). Academic Productivity: The Case for Instructional Software. Report from the Broadmoor Roundtable
at http://www.educom.edu/program/nlii/keydocs/broadmoor.html
Wellburn, E. (1996). The Status of Technology in the Education System: A Literature Review. Victoria: BC Ministry of
Education, Skills, and Training; Community Learning Network http://www.etc.bc.ca/lists/nuggets/EdTech_report.html
Western Virtual University. (1996). Enhancing the Marketplace for Instructional Materials: Best Practices in
Implementation of Advanced Educational Technologies at http://www.wiche.edu/
Wilkinson, G. (1997). Evaluating the Quality of Internet Information Sources.
http://itech1.coe.uga.edu/faculty/gwilkinson/webeval.html
Appendix 1.0
- Working Definitions
Operationalizing "learnware"
- For purposes of this study, learnware is computer software by which interactive education or training programs may
be delivered to individuals by electronic means, such as CD-ROM, Internet, or Intranet.
Possible synonyms or related terms include:
- Instructional software – software that instructs students in a particular subject matter (e.g., PLATO)
- distributed learning resources (vis-à-vis distance learning)
- technology-based learning
- interactive educational technology
- educational / training courseware
- digital educational technology
- technology-mediated learning
- A study conducted by Educom and the Annenberg Project differentiated between:
- worldware – a term coined to denote software that has substantial market beyond instruction;
- curricular software – software that is developed and used exclusively for instruction, with the aim of wide distribution, either highly flexible or focused on very specific teaching or learning tasks; and
- home brew – software that is developed with the aim of use only by its developer(s).
- In a document from EvNet (Network for the Evaluation of Education and Training Technologies), a distinction is made
between:
- warmware – people-centred
- coldware – user unfriendly
- One view of the uses of learnware is that developed by Bruce and Levin from the University of Illinois, who combine
Dewey’s classifications of learning – inquiry, communication, construction, and expression – with the uses of
educational technology to create a useful taxonomy. It divides the uses of technology, e.g., learnware, into the following four categories.
- media for inquiry (e.g., visualization software, mathematical models, digital libraries, spreadsheets)
- theory building
- data access
- data collection
- data analysis
- media for communication (e.g., word processing, electronic mail, social spreadsheets)
- document preparation
- communication with others
- collaborative media
- teaching media
- media for construction (e.g., robotics, construction of graphs and charts)
- media for expression (e.g., drawing and painting programs, animation software)
This may be a useful classification for the types of learnware.
Related terms
Distance learning / distance education
- Distance learning is defined as a system and a process of connecting learners with distributed learning resources. Distance learning and distance education are terms that are often used synonymously, although the former focuses on the learner and the latter on the provider.
- Distance education is a teaching / learning environment characterized by:
- the separation of teacher and learner during at least the majority of the instructional process;
- the influence of an educational organization, including some form of student evaluation; and
- the use of educational media and technologies to unite teacher and learner, to carry course content and provide two-way interaction.
Instructional technology and educational technology
- According to the Association for Educational Communications and Technology (AECT), instructional technology is defined as the theory and practice of design, development, utilization, management, and evaluation of processes and resources for learning.
- Educational technology includes a wide range of systems for developing and delivering instruction and for administering the learning process. Educational technology encompasses:
- hardware (from the low- to high-tech spectrum, e.g., television sets, satellite dishes, computers, compact disk interactive equipment, audioconferencing terminals, video projectors); and
- design tools (such a computer programs, simulations, video programs, written materials and their instructional design); as well as
the setting in which it takes place (e.g., on-campus: a classroom, a laboratory or a learning centre; off-campus: a learning centre in a company or in a public facility; at the workstation, or at home; through distance education; by individual learning or learning in groups);
- courseware, which might cover the full content of a course, or support lectures with drill-and-practice exercises, or provide technical updates, or impart basic skills, or technical skills, or "soft" skills;
- admininstrative support systems, ranging from test items that are embedded in drill-and-practice exercises to student information systems that track learners’ overall progress; and
- information and communications networks that might link colleges among each other or that might link
different departments and functions within a college.
Multimedia
- is the capability to process various types of media – i.e., text, graphics, still images, animation, video, and special effects – on the same computer at the same time.
Multiple mode or multi-mode
- refers to the use of technologies such as print, audio cassettes, e-mail, audio, audiographic and/or videoconferencing and/or broadcast television in combination and in physically different formats.
Open learning is particularly characterized:
- by the removal of restrictions, exclusions and privileges;
- by the accreditation of students’ previous experience;
- by the flexibility of the management of the time variable; and
- by substantial changes in the traditional relationships between professors and students.
User tools
- are standard software applications that the student might use in the educational context, such as spreadsheets, text editors, or graphing tools.
Instructional agents
- are software modules able to provide guidance to students using user tools in pursuit of some educational goal. A communication protocol supports communication between instructional agents and user tools
Appendix 2.0
- RELATED TOPICS THAT ARE NOT INCLUDED IN THE SCOPE OF THE STUDY
Topics that are directly related, but NOT included in this study of quality assurance and learnware, are the following.
- The technical elements of software quality
for example, software metrics, software processes, and software standards that are typically engineering issues
For more information on software standards:
The Software Engineering Institute http://www.sei.cmu.edu/
provides a detailed list of SEI Products and Services that includes, e.g.,:
- State of software technology and practice
- Software Process Improvement
- Performance Improvement Techniques
- Human Resources Development, e.g.,
- Software Design for Academics
- Software Design for Practitioners
- Software Requirements Engineering for Academics
- Software Specification
Rutkowski, A. (1994) Today’s Cooperative Competitive Standards Environment for Open Information and Telecommunication Networks and the Internet Standards-Making Model at
http://www.isoc.org/papers/standards/amr-on-standards.html
Internet standards at the technical and developmental level
For more information on internet standards:
- The Internet Society – a non-governmental international organization for global cooperation and coordination for the internet and its internetworking technologies and applications -- at
http://www.isoc.org/
- The Internet Standards developed by the Internet Engineering Task Force and promulgated by the Internet Society as international standards at, e.g., MIT World Wide Web Consortium
- The list of user needs regarding internet standards development developed by NSSN (National Standards Systems Network), at
http://www.nssn.org/user.html
- Network wiring standards http://www.nctp.com/nws.html
- The American National Standards Institute (ANSI at http://www.ansi.org/) sponsored the Information Infrastructure Standards Panel within the national voluntary standards system to facilitate development of standards critical to the Global Information Infrastructure (GII). Standards to facilitate interconnection and interoperability are critical, according to ANSI. Terms of reference for IISP are found at
http://www.ansi.org/iisp/96-0182.html
- Crocker, D. (1993). Making Standards the IETF Way (Internet Engineering Task Force of the Internet Society at
http://www.isoc.org/papers/standards/crocker-on-standards.html
Internet standards and evaluation criteria for information sources
For more information on evaluation of information sources on the Internet:
Library selection criteria for WWW resources (both criteria: access, design, and content) and other
evaluation websites at http://www6.pilot.infi.net/~carolyn/criteria.html
Standards and quality assurance for information technology (IT)
For more information on IT policies and practices in education:
Specific applications of learnware, e.g., on-line education or computer conferencing
For more information on applications of technology in education :
Standards and quality assurance in distance education
/ distance learning other than that provided by learnware
For more information on standards, quality and evaluation of distance education and/or distance learning:
- The Commonwealth of Learning produces two sets of guidelines related to the remote delivery of distance
education courses and programs – one for students, and one for institutions. Available at
http://www.col.org/guideli.htm
- Principles of Good Practice for Electronically Offered Academic Degree and Certificate Programs are
available from the Western Cooperative for Educational Technology , WICHE at
http://www.wiche.edu/Telecom/projects/principles.htm
- Guiding Principles for Distance Learning in a Learning Society have been produced by the American
Council of Education, Centre for Adult Learning and Educational Credentials
- Quality and Standards in Distance Education: Report to the 1992 Distance Education Conference
(Australian: available through ERIC http://www.ericir.syr.edu/
- Student Support as a Factor Affecting the Quality of Australian Distance Education: The Findings of the
Project to Investigate Quality and Standards in Distance Education (available through ERIC)
- A Question of Quality (available through ERIC)
Effectiveness of Distance Learning Courses – Student’s Perceptions
http://www.ssu.misouri.edu/AgEd/NAERM/s-a-4.html
- from the Commonwealth of Learning, Guidelines for Remote Delivery of Courses: Guide for Students and
Guide to Institutional Responsibilities at http://www.col.org/guideli.html
- a list of 40 journals in distance education found at
www.icdl.open.ac.uk/info/journals.html
- a list of distance learning guides, organizations, and research with URLs at
http://www.mcrel.org/connect/disted.html
- Nortel’s (1977) Interactive Distance Learning Vendor Selection Checklist, available from KCC
Standards in education and training
For examples of standards in education / training:
- Model Indicators of Program Quality for Adult Education Programs (Office of Vocational and Adult
Education, US Department of Education, July 1992)
- GATE (Global Alliance for Transnational Education at standards (at
http://www.edugate.org/directory.html
),
together with the Centre for Quality Assurance in International Education (CQAIE at
http://www.cqaie.nche.edu/
) and the American Association of Collegiate Registrars and Admissions
Officers (AACRAO at http://www.aacrao.com/
), which will create a database that will compare, contrast
and evaluate global academic standards
- An annotated list of organizations concerned with developing education standards is located at
http://putwest.boces.org/Standards.html
- the Seven Principles for Good Practice in Undergraduate Education, promoted by the American
Association for Higher Education, at http://www.aahe.org/ehrmann.html
- specific content/skill areas, e.g.,
Quality assurance in education / learning technology in general
For more information:
Technical aspects of learnware
The task of understanding, defining and describing quality in learnware is linked to, e.g.,
- intelligent educational systems (IES), the components of which are:
- functions necessary to meet the educational goal,
- components necessary for functionality, and
- control of the components.
- knowledge engineering (the technology of building expert systems) and task ontology (see Misoguchi
paper)
- learning technology systems architecture (which is under development by Farance Edutool)
- advanced educational software systems, such as intelligent tutoring systems
Finally, this study has been delimited in the following ways, that is, on the following premises.
Learnware is considered, in this study, as a general or generic concept, although individual products may be:
- designed to target a specific age cohort (e.g., primary students, adult learners),
- designed for specific content areas or skill areas, and/or
- of particular interest, in both a positive and negative sense, to individuals with disabilities
The use of technology in education and training is assumed to have advantages and disadvantages. This
study is not an endorsement of either position.
The development and implementation of standards, i.e., who, how and why / why not to develop standards
is controversial, to say the least, and assumed to be outside the realm of this paper.
The issues of quality and quality assurance via, e.g., performance indicators and measures, in education /
training is assumed to be a desirable goal. It is, however, not completely defined or described in this paper.
Appendix 3.0
- CONTEXT FOR THIS STUDY
The environment in which this study takes place is one in which there is a growing concern for quality assurance in
education and training, in educational software, in the uses of technology in education, in distance education / distance
learning combined with the potential of the learnware market.
The interest in quality assurance in education
- Fundamental to an interest in the quality of learnware is an interest in the quality of education/training – an interest that
is widely held and increasingly strident.
- In the context of products and services such as education/training, quality may be defined as:
- doing the job effectively and appropriately
- having the characteristics of being well thought out, prepared with care, implemented with responsibility;
having a firm direction but is flexible enough to cope with contextual variation, and being positively
responsive to comment and criticism.
- An example of the definition of a quality educational experience, arrived at through stakeholder consensus, includes the
following elements: the quality of learning materials, the availability of materials, support for students through well
trained staff, a well managed system, monitoring and feedback mechanisms to improve the system.
- Stated more succinctly, quality education is education that produces an independent learner.
- The improvement of quality, a principle underpinning the vision of a transformed higher education system in South
Africa, is linked to the capacity and commitment of the teacher, the appropriateness of the curriculum, and the way
standards are set and assessed.
- The elements of quality assurance in education – e.g., standards for curriculum, evaluation and certification – have
direct applicability to quality learnware. This paper, however, is not about general standards in education and training. It
is, rather, mainly about initiatives dealing with quality of instructional design, technological efficiency, and overall
effectiveness by which subject matter has been "packaged" for the delivery of a successful learning experience.
- In short, there is a growing interest in the delivery of high quality education and training that meets, e.g.,
- the recommended national Training Standards which include all elements of the learning system: inputs and resources, processes and practices, and outputs and outcomes
- the seven principles for good practice in undergraduate education, i.e., good practice that:
1.encourages contacts between students and faculty
2.develops reciprocity and cooperation among students
3.uses active learning techniques
4.gives prompt feedback
5.emphasizes time on task
6.communicates high expectations
7.respects diverse talents and ways of learning
The interest in quality assurance in educational software
- Many organizations and individuals have called for standards in information and educational technology, of which
learnware is a component. For example:
- Participants of the 1995 Educational Technology Conference of the US Secretary of Education urged that the federal government publish a "consumer report" on software or develop standards for judging software quality.
- The Global Alliance for Transnational Education (GATE) has launched a "seal of approval" to ensure quality in
education that is delivered transnationally.
- Criteria to assess the quality of learnware are of interest to, e.g., librarians, in their traditional role of evaluating,
selecting and organizing information, who decide whether an information source (learnware) should be linked to a
resource guide or library website, and judge the quality or appropriateness of information for a particular query or user.
Librarians have identified that sample indicators of quality in learnware might be:
- the ease of identifying the scope and authority
- clarity of purpose
- currency (not out-of-copyright older editions)
- format
- compatibility with common multimedia formats
- ease of use, i.e. workability
- conviviality (ease with which user interacts with learnware)
- connectivity (whether or not one can connect to a site, i.e., not continually overloaded)
- cost
- Before quality can be assured, it must be defined; and that is the major thrust of this paper – various definitions and
criteria for quality in learnware.
The current situation is characterized by philosophical discussions of intelligent educational systems, on the one hand,
and ad hoc implementation on the other, i.e., a situation in which there is
- a lack of knowledge engineering of educational systems
- a lack of well-designed common vocabulary and frameworks for educational systems
- a need to formalize intelligent educational tasks at the right level of abstraction
- a need to explicitly define roles and tasks of learners and systems in intelligent educational processes
- a proposed solution in task ontology and ontology engineering principles
- a preliminary design of task ontology of IESs (*next steps)
In this context, there have been calls for standards, e.g.:
- standards to make advanced educational software systems viable, i.e., to be able to add component functionality incrementally, and to be able to interoperate with commercial software and Internet resources.
According to Rada and Schoening (undated), digital educational technology has been non-standard for most of its 50 year history; however, the WWW provides a standard platform for educational technology that encourages the decomposition of educational technology tools into exchangeable components: technical standards that are critical if computers are to contribute to student-centered learning – enabling students to learn at their own pace, in their own style, and according to their own interests and strategy; hence, the creation of the PLS Initiative: (Personal Learning Systems )
The value of standards has been acknowledged. The existence of standards in many other technology areas has
proven to accelerate the adoption and diffusion of technologies, e.g., in the microcomputer industry where competition
around a common set of standards has brought the research and development capacity of the entire industry to focus
on a single set of problems.
Quality assurance in the uses of educational technology
Learnware is but a subset of the larger field of educational technology. It is conceivable, therefore, that the criteria for evaluating the effectiveness or quality of educational / instructional technology could be applied to learnware. Quality in the use of educational technologies is viewed from many different perspectives, as per the
following:
- Quality assurance in what educational technology could achieve.
- According to a 1996 paper from the BC Ministry of Education, Skills and Training, entitled "The Status of Technology in the Education System: A Literature Review," the potential of technology is to assist with such educational goals as:
- Individualization
- Increasing proficiency at accessing, evaluating, and communicating information
- Increasing quantity and quality of students’ thinking and writing
- Improving students’ ability to solve complex problems
- Nurturing artistic expression
- Increasing global awareness
- Creating opportunities for students to do meaningful work
- Providing access to high-level and high-interest courses
- Making students feel comfortable with tools of the Information Age
- Increasing the productivity and efficiency of schools
If these are, in fact, the goals to be achieved, then quality learnware could be measured against them, in whole or in part.
Similarly, Frayer and West (1997) identify the following ways in which instructional technology can support learning:
- Enabling active engagement in construction of knowledge
- Making available real-world situations
- Providing representations in multiple modalities (e.g., 3-D, auditory, graphic, text)
- Drilling students on basic concepts to reach mastery
- Facilitating collaborative activity among students
- Seeing interconnections among concepts through hypertext
- Learning to use the tools of scholarship
- Simulating laboratory work
If there are benefits of learning technologies vis-à-vis student learning, then those benefits could serve as quality criteria. Clearly, there is a need here to elaborate on these criteria rather extensively.
NCREL (North Central Regional Educational Laboratory – funded by the US government) has developed a "technology effectiveness framework" which posits that the intersection of two continua – learning and technology performance – defines the effectiveness of a particular technology in student learning. The framework’s horizontal axis is learning, which progresses from passive at the low end to engaged and sustained at the high end. The vertical axis is technology performance, which progresses from low to high. This framework, found at the end of this appendix, could be used to evaluate the effectiveness of particular technologies, such as learnware.
Quality assurance in the appropriate uses of technology
Technology has the capacity to deliver better forms of student assessment, i.e., what the International Society for Technology in Education calls "authentic testing" which involves the following factors:
- faithful representation of the contexts encountered in a field of study or in the real-life tests faced by adults engaging and important problems and questions
- non-routine and multistage tasks and real problems
- self-assessment
- trained assessor judgement
- the assessment of habits of mind and patterns of performance
From a different perspective, this list is the criteria, developed by the Open University in the UK, intended to differentiate between different media, helps to understand the various uses and appropriate uses of technology.
- ease of use - ease of use and avoidance of technical hitches
- availability - availability of teaching when needed
- access - access to other resources
- questions - opportunity to hear other students’ questions
- contacts - contact with other teachers
- experts - opportunity to hear experts in the field
- Acc/Exp - opportunity to question experts
- integration - ease of integrating material with existing work
- status - improved status due to use of the medium
- synergy - synergy of medium with other projects
The categories for comparison used are:
- learners’ needs
- usage
- effectiveness
- perceived value
- comparative value
Quality assurance in distance education / distance learning
Learnware is often used in the context of distance education / distance learning, and the impact of technology on
learning effectiveness. Advocates for distance learning claim that it makes learning and training more accessible, more
convenient, more effective and more cost-efficient for the learners and for the education provider. Distance learning,
and learnware, can be used for formal education, continuing education, advanced professional education and
management/employee development.
The environment for distance learning is characterized as one in which remote students have special needs
including advising needs, access needs, communication needs, and administrative needs. In the traditional context – distance
education delivered by traditional learning organizations for course / program credit – these needs are met through
appropriate institutional support structures. However, in the distance delivery context – self-directed learners who may
or may not want credit from traditional learning institutions – the learnware must assist distance learners to:
- take greater responsibility for their own learning;
- become more active in asking questions and obtaining help;
- be prepared to deal with technical difficulties in the two-way flow of information.
To develop independent and self-reliant distance learners (possibly, learnware learners), research indicates that the
following three approaches are commonly advocated:
- the service model approach which focuses on the integration of quality, by providers, into distance delivery and
courseware (learnware) through, e.g.,
- quality assurance methods in courses and curricula
- high quality support services
- integration of the study of communication itself into the curriculum
- the TQM model of consumer-oriented quality in methods and materials
- a stakeholder analysis model which focuses on defining quality for distance education, i.e., involving more than the learning providers in the defining quality and setting benchmarks
- quality improvement model which involves ongoing evaluation, e.g.,
- qualitative assessment techniques to understand stakeholder values
- quantitative evaluation to provide indicators of quality and areas of concern
According to Lucent Technologies, a division of Bell Labs, the benefits of distance learning to organizations include:
- increased resource productivity through shared instructors, more student "face time," and access to special resources or experts;
- great cost effectiveness through increased numbers and frequency of training, reduced travel costs, and shared resources;
- greater impact through multiple sites, real time updates, and learning in the worksite;
- higher quality through more sources, remote experts, and increased student-student and student-teacher interaction.
According to Seligman (1992), the five elements of quality, specifically for the improvement of quality in distance are:
- materials must be learner friendly, academically respectable, able to be used by the average student, interesting in content and layout, and relevant;
- learning materials and any peripheral media or equipment must be available;
- tutors and students need to become familiar with distance learning methodology and practice;
- the whole system must be managed effectively; and
- monitoring, evaluation, and feedback must be viewed as important.
All of these statements of values can be construed as criteria for evaluation or quality assessment. Criteria for
evaluating distance learning, then, have some application in this study.
The nature of the learnware market
Educational Learnware
The market for learnware is studied both by the industry and by educational bureaucracies within government. In short,
both have concluded that the market for learnware is "a mess."
For example, a 1995 study of the educational software market, undertaken by the RAND Corporation, reached the
following conclusions concerning the K-12 area.
- First, the K-12 software market is shaped by several institutional realities. Key acquisition decisions are sometimes controlled by a small number of actors. Traditions and practice inherited from print-based educational materials are slowly changing, but continue to dominate. A significant proportion of computer hardware in schools is obsolete by today’s standards both restricting the appeal of the school market and shaping school demand in ways that are unattractive for private sector software developers.
- Second, at $15 or $16 (US) per student annually, school budgets for software are low. Vendor revenues are quickly eaten up by the costs of development, preparing different run-time versions of the software for different school computer platforms, maintaining inventory and delivering the product. Too little remains for product improvement and innovation.
- Third, the important ILS (integrated learning systems) segment of the market is stagnant and built on outmoded premises about learning. New concepts of instructional management to match emerging educational reform goals are needed.
- Fourth, the school market for occasional-use supplemental software is limited to the elementary grades. Participants in the RAND study were unanimous in agreeing that high-quality content software for the middle and secondary school grades was practically non-existent.
- Fifth, home market revenue for high-production-value "edutainment" software is positioned to overtake revenue for educationally correct, but low-production-value software in tow to three years. The visibly more sophisticated product may set a new standard of parental expectation for the quality of school software, creating a demand on the school budget for a higher-priced, educationally correct and visually sophisticated software product.
- Sixth, the trend towards image-enriched, visually more sophisticated educational software would be accelerated by the successful entry of "Hollywood-based" firms or subsidiaries like Lucas and Disney into the educational software market.
In summary, the structure of school budgets, which requires that educational
technology be acquired from the vanishingly small fraction of the budget that remains after all other requirements are met, a current emphasis in the schools on Internet service that tends to emphasize hardware acquisition, and an expanding home market for "edutainment" imagery in CD-ROM format combine to create an uncertain school market for an increasingly sophisticated educational software industry.
A reason that the market is considered to be "a mess" is because the quality of educational software is uneven at best.
For example, the California Software Clearinghouse evaluated 528 educational software packages between 1991 and
1994, and found that 376 were acceptable, 201 could be recommended for school use, only 63 were exemplary, and,
in the end, only 8 were adopted as classroom materials.
That having been said, there is an untapped market for quality learnware. The Software Publishers Association 1997
Education Market Report, including both Canada and the US, concludes, e.g., that:
- The 1996-97 total K-12 expenditures for educational technology are $261.7 billion, which is an average per pupil expenditure of $6,148.
- Estimates of the total to be spent on instructional software in the 1996-97 school year range from $494 million (Quality Education Data), to $565 million (Peter Li Education Group, Wujcik and Associates), to $728 million (SIMBA Information Inc.), which includes $42 million in school-to-home software sales, to $1.66 billion for all K-12 application software.
Trends from the report indicate that:
- More technology planning and funding will be needed to keep pace with increasing enrollment;
- More hardware, software, training and support will be needed for students and educators; and
- More educators and students will be on the Internet.
The full report is available from SPA via the SPA website.
Educom also assembled leading educators to study the learnware market, and to specifically consider the implications of learning technologies vis-à-vis educational productivity in post-secondary education. They concluded that:
- Instructional software is essential if they were to increase educational productivity because:
- Approximately 80% of the costs of colleges and universities are attributable to personnel costs.
- Colleges and universities need to find ways to substitute capital for labour in order to improve productivity.
- "Bolt-on" applications of IT cannot address the productivity problem.
- Instructional software is needed to create less labour intensive models of teaching and learning.
- Targeting areas of high student demand will have the greatest likelihood of success because:
- Large undergraduate enrollments are concentrated in relatively few academic areas.
- These high-demand subject areas are those in which instructional software may be more easily developed.
- The "One Percent Solution" offers a promising change strategy.
- A mass market is needed to justify initial risk and investment.
- Colleges and universities cannot create the requisite software on their own because:
- Traditional "soft money" development strategies have failed to produce a scalable body of instructional software.
- The business of colleges and universities is education students and conducting research; it is not the development, production,
distribution, and marketing of products.
- What is needed is "self-sustained commercialization."
- Market success depends on instructional software that is:
- disaggregated, therefore, enables mass customization
- disintermediated, i.e., usable without human mediation
- differentiated, i.e., combined and used in different ways, and
- diffuse, i.e., increasing in value the more it is used.
Learnware for Corporate Training
The corporate training market is judged by many to be the most lucrative, in terms of accessibility and size, to
pursue. A 1996 Market Assessment Study of New Media Learning Materials was produced by Industry Canada
with Support from Human Resources Development Canada. This report identified major training requirements in
key industrial sectors, recognized a significant training role for new media learning materials, and outlined
significant opportunities for Canada's fragile but growing learnware industry.
The NCREL Technology Effectiveness Framework
Reproduced from http://www.ncrel.org/sdrs/edtalk/tef.htm
Now that we have meaningful and appropriate indicators for engaged learning and for high technology performance, we
can use them to measure the extent to which individual technologies and technology-enhanced programs are effective
- that is, the extent to which they support engaged learning.
To this end, we (NCREL) have developed the technology effectiveness framework. This framework posits that the
intersection of two continua - learning and technology performance - defines the effectiveness of a particular
technology in student learning. The framework's horizontal axis is learning, which progresses from passive at the low
end of the continuum to engaged and sustained at the high end. The vertical axis is technology performance, which
progresses from low to high. (This is illustrated in Table 3 which was not electronically reproducible).
When we cross the two continua, four major learning and technology patterns emerge:
- Pattern A - Engaged learning and high technology performance
- Pattern B - Engaged learning and low technology performance
- Pattern C - Passive learning and high technology performance
- Pattern D - Passive learning and low technology performance
How to use the framework
The framework gives educators, researchers, and policymakers a way to evaluate technology and
technology-enhanced programs and curricula against the learning goals they have for their student. Before doing so,
however, these decision makers need to define their learning goals. That's where the trajectories for change come in.
Directions for Change
The framework encompasses four positive (desirable) directions for change:
- Type I trajectory: D - B
This is movement from passive learning and low technology performance to engaged learning
and low technology performance.
- Type II trajectory: B - A
This is movement from engaged learning and low technology performance to engaged learning
and high technology performance.
- Type III trajectory: C - A
This is movement from passive learning and high technology performance to engaged
learning and high technology performance.
- Type IV trajectory: D - A
This is movement from passive learning and low technology performance to engaged learning
and high technology performance.
It is obviously counterproductive to move from D (passive learning with the least functional technologies) to C (passive
learning with more functional, and more costly, technologies). If a school or group is not using technology to enhance
engaged learning, there is little reason to pay the higher cost for greater functionality.
Once the school or school district establishes its curricular goals, the trajectories can guide it in determining what
technologies can move learners toward these goals.
This framework provides a powerful matrix for analyzing particular technologies and programs in broad terms.
Decision makers can use it as they select and work toward specific curricular goals to promote engaged learning.
Researchers, curriculum developers, and staff developers can use the framework to design technologies and
technology-enhanced programs. And schools can use the framework to evaluate technology and its costs. In doing so,
the critical questions are:
- What are the learning goals (i.e., the vision of learning) to which technology is applied?
- How are these learning goals moving the school toward reform?
- How will a technology-enhanced curriculum support instruction that addresses
those learning goals?
- Does the technology-enhanced approach help restructure the school to meet its plan for educational reform?
- Do the students achieve the learning goals using the technology-enhanced curriculum?
- Can the school implement cost-efficient technologies given its goals and current realities?
- Can the school extend or adapt less functional technologies so that they are more functional in supporting a global community of learners in sustained learning that is challenging and authentic?
- Are there funding strategies or partnerships that can reduce the cost?
- How can a school continuously plan to use technology to reach for more powerful learning goals and reform?
Appendix 4.0
- PRINT EVALUATION CRITERIA
The material on this page was created by Jan Alexander and Marsha Tate, Reference Librarians at Wolfgram Memorial
Library, Widener University, Chester, PA. We would like to thank them for making their work available to us.
Review of the Five Traditional Print Evaluation Criteria
- Criterion #1: Accuracy
- How reliable and free from error is the information?
- Are there editors and fact checkers?
- Criterion #2: Authority
- What are the author’s qualifications for writing on this subject?
- How reputable is the publisher?
- Criterion #3: Objectivity
- Is the information presented with a minimum of bias?
- To what extent is the information trying to sway the opinion of the audience?
- Criterion #4: Currency
- Is the content of the work up-to-date?
- Is the publication date clearly labeled?
- Criterion #5: Coverage
- What topics are included in the work?
- Are the topics included explored in depth?
Adapting Five Traditional Print Evaluation Criteria to Web Resources
- Criterion #1: Accuracy of Web Resources
- Almost anyone can publish on the Web
- Many Web resources not verified by editors and/or fact checkers
- Web Standards to ensure accuracy yet to be fully developed
- Criterion #2: Authority of Web Resources
- Often difficult to determine authorship of Web Sources
- If author’s name is listed, his/her qualifications frequently absent
- Publisher responsibility often not indicated
- Criterion #3: Objectivity of Web Resources
- Goals/aims of persons or groups presenting material often not clearly stated
- Web often functions as a "virtual soapbox"
- Criterion #4: Currency of Web Resources
- Dates not always included on Web pages
- If included, a date may have various meanings:
- Date information first written
- Date information placed on Web
- Date information last revised
- Criterion #5: Coverage of Web Resources
- Web coverage may differ from print coverage
- Often hard to determine extent of Web coverage
Appendix 5.0
- Criteria for Evaluating Internet Information
Reproduced from http://info.lib.uh.edu/pr/v8/n3/smit8n3.html
The following set of criteria for evaluating internet information resources was developed by A. Smith (1997) in the
context of the librarians task of evaluating, selecting and recommending information resources. Smith’s "toolbox of
criteria," listed below, may be most relevant to the content side of learnware (as opposed to the process side).
According to Smith, not all criteria apply to all resources and librarians would choose criteria from the toolbox. Criteria
for evaluating Internet information resources are the following.
1. Scope
- What items are included in the resource? Is the scope only implied, or is it stated through metainformation such
as an introduction? Does the actual scope of the resource match expectations? Aspects of the scope include:
- Breadth: What aspects of the subject are covered? Is the resource focused on a narrow area or does it include
related topics?
- Depth: What is the level of detail provided about the subject? This is related to the level of audience for which the
resource has been designed, mentioned below.
- Time: Is the information in the resource limited to certain time periods?
- Format: A resource that provides links may restrict its scope to certain classes of resources. For example,
Telnet, Gopher, or FTP (File Transfer Protocol) resources may be excluded from an WWW-oriented site.
2. Content
- Is the information fact or opinion? Does the site contain original information or simply links? Sites can be useful
both as information resources in themselves and as links to other information. However, users can be frustrated
by lists of resources which look promising, but turn out to simply contain more links.
- Does the resource stand alone, or has it been abstracted from another source, perhaps losing meaning or links
in the process?
- Specific factors related to the content include the accuracy, authority, currency, and uniqueness of a resource.
- Accuracy - Is the information in the resource accurate? A resource may be checked against other resources or
against information that the evaluator has. Are there political, ideological, or other biases? The Internet has become a prime marketing and advertising tool, and it is advisable to ask what motivation the author has for placing this information on the Net. Frequently, the answer is that the information is placed to advertise, or to support a particular point of view.
- Authority - Does the resource have some reputable organization or expert behind it? Does the author have standing
in the field? Are sources of information stated? Is the information verifiable? Can the author be contacted
for clarification or to be informed of new information?
- Currency - Is the resource updated or static? If it is updated, how frequently does this occur? Are dates of update stated, and do these correspond to the information in the resource? Does the organization or person hosting the resource appear to have a commitment to its ongoing maintenance and stability? The date stamping of files, which can be determined by many browsers, indicates the date of change in the physical file; this may not reflect the currency of the information.
- Uniqueness - Is the content of the resource available in other forms (at other sites, on a Gopher, in print, on CD-ROM)? What advantages does this particular resource have? If the resource is derived from another format, does it have all the features of the original? Have extra features been added? Does it complement another resource, for example, by providing updates to a printed source?
- On the Internet, a resource may be available from a number of different sources. For example, the World
Fact Book is available widely at various locations, in different editions, and in FTP, Gopher, and HTML
formats. This kind of redundancy may be valuable--a particular site may not be available when required,
and an alternative or mirror site may have to be used. Some users may not be able to access certain
types of resources; for example Telnet or image-based Web sites, so the availability of alternative formats
may be useful.
- Links Made to Other Resources - If the value of the site lies in its links to other resources, are the links kept up to date, and made to appropriate resources? Are the links made in such a way that it is clear that an external site is being referred to? There are potential copyright issues with sites that, for instance, enclose an external link in frames so that the source of the information is unclear.
- Quality of Writing - Is the text well written? While hypertext linking and multimedia are important elements of the Web, the bulk of the information content on the Web still lies in text, and quality of writing is important for the content to be communicated clearly.
3. Graphic and Multimedia Design
- Is the resource interesting to look at? Do the visual effects enhance the resource, distract from the content, or
substitute for content? If audio, video, virtual reality modeling, or other effects are used, are they appropriate to
the purpose of the source? A related criterion is navigational design, mentioned below in the context of
browsability and organization.
4. Purpose and Audience
- What is the purpose of the resource? Is it clearly stated? Does the resource fulfill the stated purpose? The
purpose needs to fit the intended audience for the resource. Who are the intended users of this resource? At
what level is the resource pitched: toward a subject expert, a layperson, or a student? Will the resource satisfy
the needs of the intended users? Does the user group at which the site is aimed have the connectivity to access
the resource?
5. Reviews
- What do reviewing services say about the site? The use of reviewing journals has been a mainstay of the
development of printed collections; librarians in the Internet environment will need to become familiar with the
strengths and weaknesses of the tools reviewing Internet resources.
6. Workability
- Is the resource convenient, and can it be used effectively? This is the area where criteria for Internet resources
differ most from print sources. An issue in providing access to electronic documents is whether a library should
provide links to the originating site or "acquire" the publication for local access. Poor workability may indicate that
the library should store the data locally, if intellectual property considerations allow this. Aspects of workability fall
into a variety of areas.
7. User Friendliness
- Is the resource easy to use? Are any necessary special commands clear? Is help information available? Have user interface issues been addressed, such as menu design and readability of screens?
8. Required Computing Environment
- Can the resource be accessed with standard equipment and software, or are there special software, password, or network requirements? Has the resource been designed to work well with one software and user interface (for example, the latest Netscape release on a T1 connection)? Is it difficult to use with others (for example, Lynx at 2400 bits per second)? It is useful to test resources with a variety of browsers and connections. Telnet resources may pose problems to users who have not installed a Telnet client. Images and other multimedia may create problems if users have not installed the correct helper application.
- While the extent to which older browsers are currently used is a source of argument, there are still Lynx-only, frames-challenged, and visually impaired users, and sites should attempt to meet their needs. This criterion is less important where users are in a defined computing environment, such as that provided by workstations in a particular library.
9. Searching
- How effectively can information be retrieved from the resource? Is a useful search engine provided? What operators and ranking features are available? Is use of the search engine interface intuitive? Does the search engine index the whole resource?
10. Browsability and Organization
- Is the resource organized in a logical manner to facilitate the location of information? Is the organizational scheme appropriate (e.g., chronological for a historical source or geographical for a regional resource)?
11. Interactivity
- Where interactive features such as forms and CGI (Common Gateway Interface) scripts are provided, do they work? Do they add value to the site?
12. Connectivity
- Can the resource be accessed reliably, or is it frequently overloaded or offline? Is the connection one of limited
bandwidth, so that pages take a long time to load or keystrokes a long time to echo? Is a local mirror site
available, or do international traffic charges have to be incurred?
13. Cost
- Currently, Internet information resources are perceived as being free. However, costs do exist, and they are likely
to become more important. Costs can be divided into: (1) costs of connecting to the resource, and (2) costs
associated with the use of the intellectual property contained in the resource. Internet users paying traffic
charges already have to consider the costs of connection, and they may want include this in criteria for selection.
For example, they may favor text-based rather than image-intensive sites, if the information content is the same.
Increasingly, there will be sites where a charge is made for the intellectual content of the site. Libraries have
been dealing with pay-per-use online services such as Dialog for many years, but the Internet has created an
opportunity to make services available to end users for a fee. Libraries have a role in negotiating subscriptions
and site licenses for organizational access to services that charge.
Appendix 6.0
- Library Selection Criteria for WWW Resources
Reproduced from http://www6.pilot.infi.net/~carolyn/criteria.html
Carolyn Caywood c1995. (An early version appeared on page 169 of the May/June, 1996 issue of Public Libraries.)
Libraries are beginning to select Internet resources to be linked on their websites. Just as book selection has been, this
process needs to be guided by a policy with stated criteria. The following are my suggested criteria for assessing the
value of a Web site to a library's users. Few sites meet all criteria, so the benefits must be weighed against the lacks.
Given the speed of change on the Internet, resources need to be re-evaluated on a regular schedule to determine if
they still meet these criteria.
ACCESS
- Is the site still useable with an ASCII browser like Lynx?
- Is it written in standard html, or have proprietary extensions been used?
- Does it use standard multimedia formats?
- Do parts of it take too long to load?
- Is it usually possible to reach the site, or is it overloaded?
- Is it stable, or has the URL changed?
- Is it open to everyone on the Internet, or do parts require fees?
- If there is a charge, can the library pay it on a subscription basis for multiple access points?
- Are any rules for use stated up front?
DESIGN
- Are the individual Web pages concise, or do you have to scroll forever?
- Do essential instructions appear before links and interactive portions?
- Do all the parts work?
- Is using the site intuitive, or are parts likely to be misunderstood?
- Can you find your way around and easily locate a particular page from any other page?
- Is the structure stable, or do features disappear between visits?
- Does it look and feel friendly?
- Is it conceptually exciting? Does it do more than can be done with print?
- Can the user interact in satisfying ways?
- Are interactions secured if they involve private information?
- If Java or ActiveX, extensions like frames, or plug-ins are employed, do they actually make the site easier to use?
- How do they affect users with older browsers?
CONTENT
- Is the scope clearly stated? Are its limits stated? Is the title informative? Does the content fit the scope?
- Are headings clear and descriptive, or do they use jargon meaningful only to the creator?
- Is text well-written with acceptable grammar and spelling? What is the quality of multimedia files?
- Is the content organized by the needs of the user, or does it reflect an internal hierarchy?
- Is it a dead end, or are there links to take you further?
- Are the content and the links clearly described and suitable to the expected audience?
- Is the content up-to-date? Are out-of-date links promptly corrected?
- Is the content amplified over time, or is one visit all it's worth?
- Is the origin of the content documented? Is it authoritative? Is it verifiable? Is it accurate?
- Is it clear who is responsible for the site?
- Are biases and opinions clearly identified? Is there any conflict of interest?
- Is the content unique, or readily available elsewhere? Has copyright been respected?
- Is there community interest in the content or the site itself? Are there reviews of the site?
- Is the amount of bandwidth commensurate with the usefulness of the content, or are graphics or multimedia
included simply to show off?
Appendix 7.0
- Internet Information Resources Evaluation Sites
Reproduced from http://info.lib.uh.edu/pr/v8/n3/smit8n3.html
1.0 The Argus Clearinghouse
http://www.clearinghouse.net/
The Clearinghouse provides clearly laid out criteria that are used for evaluating the resource guides it includes. The
criteria are based on level of resource description, level of resource evaluation, design, organizational schemes, and
metainformation. These are useful criteria for evaluation of resources, although they are specifically intended for the
evaluation of resource guides. The criteria are listed at http://www.clearinghouse.net/ratings.html
2.0 Best of 1996 Social Sciences, Humanities & Asian-Pacific Studies WWW Resources
http://coombs.anu.edu.au/SpecialProj/QLTY/BEST/Method96.html
This is an example of a "best of" competition, but with an academic bent. Entries are to be rated under the criteria of
quality, structure, and presentation. Brief definitions of these criteria are listed under "Rating Procedure."
3.0 CyberHound
http://www.thomson.com/cyberhound/
Cyberhound is a service of Gale Research, well known for their print reference works. In addition to being a search and
directory service, Cyberhound offers reviews according to criteria under the headings content, design, technical merit,
and entertainment. The criteria are found at http://www.thomson.com/cyberhound/frames/content.html#rating
4.0 CyberStacks
http://www.public.iastate.edu/~CYBERSTACKS/
This experimental site arranges selected Internet information resources in science and technology by Library of
Congress classification. Its criteria (authority, accuracy, clarity, uniqueness, recency, reviews, and community needs)
are stated to be the same as those laid down for conventional resources in the American Library Association's
Reference Collection Development: a Manual, published in 1992. This does not address issues such as workability
which are more specific to Internet information resources. CyberStacks' criteria are at URL:
http://www.public.iastate.edu/~CYBERSTACKS/signif.htm
5.0 Infofilter
http://www.usc.edu/users/help/flick/Infofilter/
As of July 1997, this project has ceased operation. It lists criteria of authority, content, organization, currency, search
engine, graphic design, and innovative use of the medium. The criteria are listed at
http://www.usc.edu/users/help/flick/Infofilter/template.html
under "Review."
6.0 The Internet Public Library
http://www.ipl.org/
The IPL states that its collection policy for selecting ready reference resources is based on content, updating, the
graphics being complementary rather than distracting, availability of text interfaces, evidence of proofreading, and
whether the document is a primary one. The selection policy is at http://www.ipl.org/ref/RR/Rabt.html#policy
7.0 Magellan Internet Guide
http://www.mckinley.com/
For ratings in the Magellan directory, McKinley Group uses the criteria of depth, ease of exploration, and net appeal.
The latter is assessed by asking, "Is it innovative? Does it appeal to the eye or the ear? Is it funny? Is it hot, hip, or
cool? Is it thought-provoking?" The ratings explanation is reproduced at http://www.lib.ua.edu/maghelp.htm#howdoes
8.0 SiteGrade
http://www.sitegrade.com/
This site assigns "letter grades to websites in order to encourage responsible use of the World Wide Web medium so
that the widest possible audience can enjoy it." Fairly detailed criteria are stated at
http://www.sitegrade.com/criteria/
9.0 Stevie's Web Site Ratings
http://www.steview.com/cgi-bin/STEVIE/rat_home
This is quite a complex voting system. Users can rate a site on access speed, applicability to different age groups,
ease of navigation, educational/informative quality, entertainment quality, appearance, timeliness, and usefulness.
Ratings are continually averaged, and the latest figures and top ten sites are available from the Web site.
10.0 World Wide Web Virtual Library Maintainers:
(Criteria Used to Select Links for Resources' Catalogues)
In 1995 an email poll of WWWVL site maintainers was used to accumulate a range of criteria used by the maintainers
in selecting material for their sites. This provides a wide range of criteria, available at
http://www.ciolek.com/WWWVLPages/QltyPages/QltyLinks.html
The author maintains a page, which is part of the World Wide Web Virtual Library, with links to a number of resources relating to evaluation criteria for Internet
information resources. See: http://www.vuw.ac.nz/~agsmith/evaln/evaln.htm
![](images/index_logo_small.jpg)