This Quick Guide looks at the nature of assessment in a changed (and changing) Australian Vocational Education and Training (VET) context, and considerations and issues for online teaching.
Specifically this Guide looks at why the topic of online assessment is important, key research projects that are looking at online assessment in the VET context, current uses of online assessment, issues such as validity, reliability, fairness, and flexibility (including the hot topic of plagiarism), current barriers to the use of online mediums for assessment, and the impact on selection of assessment strategies and assessment management processes.
This Guide will be of relevance principally to teachers and designers of content and teaching processes, and in particular those operating within the Australian Quality Training Framework (AQTF) which governs teaching, learning and assessment standards within registered training organisations. It may also be of interest to managers and policy makers within VET.
Assessment plays an important part in the learning process - both summative and formative assessments inform progress and guide learning; it is essential to the accreditation process; and, results are used in all sorts of ways to measure outcomes and success of the student, teacher, course, or institute. Self assessment encourages student independence, and helps students develop the necessary skills for autonomous (and lifelong) learning. Assessment, especially when embedded within an 'authentic' learning tasks or exercises, can also be an essential part of the learning experience.
Principal reasons why online assessment is of fundamental importance:
Compared to the number of articles and research on online teaching and learning, there are few articles that specifically address online assessment (Anderson 2001) and many practitioners are seeking guidelines (McLoughlin & Luca 2001). A number of VET research projects are currently in progress aimed at addressing this void.
Note when searching the literature there is a wealth of experience to be mined in the distance education literature as well as the online literature. Distance education is built around autonomous and independent learning and has a long history in pedagogy, learning support, and systematising product development and learning management for off-campus teaching.
Assessment within VET is governed by the principles set out in the Australian Quality Training Framework Standards for Registered Training Organisations. These principles apply regardless of the mode of delivery. Online assessment like other forms of assessment needs to be valid, reliable, fair, and flexible.
Further, it is a requirement for training and assessment to be equitable for all persons, taking account of cultural and linguistic needs (AQTF, Section 8.1 viii). Assessment should be sufficiently flexible and diverse to account for differing needs and circumstance of students undertaking online study.
Flexibility can mean:
The extent to which these choices are available in online learning depends on many factors, including pragmatic issues of managing learning and learners, teaching strategies employed, requirements to start and finish on given dates and so forth.
Key issues for the online learning context are:
Assessment for online is underpinned by the same principles of validity, reliability, flexibility and fairness, and uses many of the same strategies used in traditional face-to-face teaching (Booth, Clayton, Hyde, Hartcher & Hungar 2002). What differs mostly is the context of the assessment, the interactions between assessor and those being assessed, and collection and administration processes.
Online can be used:
or
Online assessment can provide:
Some of the online options listed in the literature include:
|
|
Assessment can be by: teacher, self, peer, or by an external person or group.
Online assessment, like online teaching, is still very much in an embryonic state (Hyde, Booth & Wilson forthcoming). The most common online assessment strategies used within VET currently are: using computer communications as a transfer medium to submit and comment upon assigned work such as essays; compiling and submitting portfolios; and for computer testing, i.e. short answer, multiple choice type questions and quizzes. There is growing interest in communication tools for learning interactions (Booth, Hartcher & Hyde 2002) and evidence that this can work well, but the use of conferencing and online group work as assessable activities is not yet wide-spread. Whilst numerous papers report innovative examples of online assessment such as group discussion, simulation tools, various types of case studies and project work, and peer and self assessment, their use is not yet common place (McLoughlin & Luca 2001; Freeman & McKenzie 2000).
Online does not as yet lend itself easily to the testing of competencies that require the demonstration of a particular skill or work-based task (Booth, Hartcher & Hyde 2002).
Both summative and formative assessment have very important roles to play in flexible learning. Currently online is used more for 'summative' assessment than for formative assessment (Anderson 2001; Booth, Hartcher & Hyde 2002).
"Authentic" online assessment that is integrated with the learning experience, not separated from it, is considered to be one of the most effective ways of measuring student's learning. According to Anderson (2001) it promotes skills transfer and deeper level understandings - 'learners are assessed within a relevant context and are required to collaborate extensively to produce a product that demonstrates the knowledge, understanding and skills they have learned ...and the ability to manage ...activities'.
Challenges include:
Communication and interaction are essential elements within learning. There are views that 'required' participation and input into online discussion forums, chats, emails, face-to-face seminars provides tangible benefit to the learner and impacts upon learning outcomes; it helps keep learners motivated and involved (Toolbox Central Job Aid [3] ); enables students to support each other's learning; and provides opportunities for peer and teacher feedback to individuals and student groups (Salmon 2000). Some have found from experience that unless participation is assessable, students do not feel compelled to participate hence reducing the effectiveness for others (Hyde, Booth & Wilson forthcoming). There are counter views also about compulsory requirements for interaction and collaborative endeavour and its benefits for learners (see McLoughlin and Luca, page 419).
Maor (1998) claims that this presents three interesting dilemmas:
Maor discusses these dilemmas in an interpretive research study on student online participation conducted in a post-graduate Internet-based distance education unit teasing out the arguments and decisions faced by many online teachers.
Another difficulty for teachers is authentication - knowing that the person participating is in fact the person they profess to be.
Hyde, Booth and Wilson in their forthcoming publication provide some useful guidelines on assessment strategies that involve online communication and interaction.
An associated issue is that of measuring techniques and the ethical considerations. Online technologies can provide the means to track and record student work and conversations. Extreme care needs to be taken to ensure privacy and confidentiality requirements are observed and that the learning environment is established in a way that students feel comfortable and confident to work in the online setting.
Other areas of simultaneous opportunity and challenge afforded by online teaching is that of peer and student self-assessment. Two informative papers are:
Major issues identified were student readiness and ability to effectively participate in these forms of assessment, and design issues to ensure integrity and fairness.
As with group assignments in traditional face-to-face on-campus teaching, online assessment requires the selection of appropriate and defendable assessment criteria. Assessment might be of the product or output of the group work, on process only, or a mixture of both. With online assessments consideration also needs to be given to disadvantages that may occur because of student access to technology.
Authentication and plagiarism are topics of great concern to teachers and sceptics of online teaching. Concerns are fuelled by reports such as those reported in The Australian (Buckell 2002) wherein it was claimed that 8 per cent of university students were found to have pilfered large amounts of text from the web. Hyde, Booth & Wilson (forthcoming) suggest that the potential for more cheating and plagiarism to occur in online, as opposed to other modes, is probably overstated as cheating has always been with us and there are strategies that can be adopted to minimise its occurrence.
There are two types of plagiarism: intended copying of direct passages of other people's work and ideas without acknowledgement, and, lack of acknowledgement through poor or incorrect referencing techniques. The latter points to the need for inclusion of instruction and guides on referencing to include the need to acknowledge sources obtained from the web and the correct ways of acknowledging online material. Students need to know that information freely available on the web does not mean 'free to copy' without due acknowledgement or permission! Online courses should display clear guidelines.
With regard to combating plagiarism software programs are available to assist in the detection of plagiarised work. The software used by CAVAL on behalf of the six Universities involved in the study reported in The Australian was a web-based detection program called 'Turnitin.com', but a variety of other software programs are also available.
Detection is but one avenue. It is far better to prevent than detect. McLoughlin and Luca (2001, quoting Berge et al.) claim that "wisdom might be served by using alternative form of assessment of student understanding". For example using a series of sequential exercises, building upon one another so that issues of security become less of a problem. Other strategies include providing a mix of assessment strategies that allow students to demonstrate ability and construction of knowledge in a variety of ways that require them to give examples, or draw from class or student-teacher discussions over a period of time; encouraging the development of portfolios of work; integrating teaching strategies that focus on student enquiry and investigation; and adopting authenticated assessment approaches in which the assessment is situated in a real or simulated workplace (Hyde, Booth & Wilson forthcoming). Clearly there is also merit in getting to know one's students although this is less easy in high enrolment classes.
Hyde, Booth and Wilson (forthcoming) suggests "teachers need to adopt a risk management approach to the issue of authenticating assessment and make judgements concerning the identification of critical assessments that absolutely must be authenticated" and those which are less critical.
Issues of security in online assessment most often relate to:
Teachers and students need to be assured that adequate security is in place. The importance of management support cannot be overstressed.
Assessment strategies and tasks should be aligned to the learning objectives and learning process, and the connection should be made explicit to students so that they can see the relevance of the assessment to the learning task (Leask 2001; Hyde, Booth & Wilson forthcoming; Toolbox Central Job Aid [4] ). It should be planned from the inception of the teaching program or learningware design (Booth, Hartcher & Hyde, 2002, p. 3; Hyde, Booth & Wilson forthcoming).
McLoughlin and Luca (2001) demonstrate just one way student activity and assessment processes can be integrated in participatory forms of assessment:
|
Modes of student interaction |
Student contributions to the unit |
Interactive assessment activities |
|
Attending Practicing Discussing Articulating |
Design and develop team web site |
Intra-team peer review |
|
Post solutions to weekly problems |
Intra-team peer review |
|
|
Post weekly journals |
Inter-team peer review and commitment to tasks |
|
|
Develop electronic portfolio |
Critique and peer feedback on portfolios |
|
|
Develop a team contract outlining team responsibilities |
Negotiate roles and commitments for each team member |
This model shows how each component relates to the pedagogic principles of attending, practicing, discussing and articulation, identified in Laurillard (1996), that underpin effective learning.
Using a framework such as this may be a useful way of exploring the connections between activity and assessment in some teaching situations. Teachers working together may come up with other similar models.
When developing online assessment it is recommended that teachers and practitioners consult with others on the nature of the assessment task to ensure the assessment is valid, viable, and authentic (Toolbox Central Job Aid [5] ).
Online learning by its very nature requires a degree of independence in students and the ability to problem solve (that is, the ability to solve learning problems as well as technical problems). Working off-campus away from teachers and peers means that students have to take on greater responsibilities themselves if they are to succeed in online courses.
In addition, there is a view within VET and higher education that students be encouraged to take on further responsibilities for their own learning in keeping with expectations of employees, and the community at large. There has been a demonstrable shift to more student-centred approaches to teaching and learning that encourage students to do this. It follows that this same principle should be extended to assessment (Anderson 2001).
Self-assessment, peer assessment, discovery learning, reflection and articulation are just some of the ways students are being encouraged to manage their own learning. Other ways include options to negotiate assessment activities, mode of presentation, timing of assessment (though this is not always practical or desirable), and decisions about grading or mark allocations. The challenge remains as always though to find measurable evidence to show attainment of competencies and desired attributes.
Barriers to effective use of online assessment identified by Paulsen (2000) in an international study to inform decision makers included: public and institutional regulations, traditions for physical attendance, technical limitations, student identification, and detection of plagiarised digital material.
Bonk, reporting on his research of online learning experiences within educational institutions (2001) and training corporations (2002), claimed that non-use and feedback from research surveys indicate a need for more sophisticated and a greater range of online assessment options to increase both use and effectiveness of online assessment. And, that there were shortcomings in the current range of options.
Clearly technology lends itself better to some forms of assessment than to others, but mostly barriers often relate to teachers knowing sufficient about the available options and being able to confidently connect the appropriate assessment options to the different teaching and learning strategies they might employ. Teachers also need to feel confident they can overcome some of the issues associated with online assessment.
The pragmatics of time, available resources, cost, and lack of support are also often listed as barriers (Booth, Hartcher & Hyde 2002; Anderson 2001).
The short answer is - yes, it does. Or at very least electronic communications have impacted very significantly on teaching and learning options which in turn will challenge traditional assessment regimes. For example:
Such challenges of course need to be considered within the strategic and governing frameworks within which the provider operates.
As well, online assessment techniques will broaden and be refined as technical capabilities expand, access becomes less problematic, and the status of teaching, learning and assessment online matures.
Of prime concern to many is the potential conflict between flexibility and approaches that are valid, reliable and fair. It requires careful consideration, judgement, and a balanced approach to achieve all criteria.
Off-the-shelf and custom built systems to manage or monitor student learning and assessment are increasing in number and are used to provide efficiency, faster and improved turn-around of feedback to students, or as part of a total technical solution in support of online learning. Along with this array of options is a state of confusion over system choice. Teachers are looking for guidance. As well, time needs to be allocated to developing suitable assessment material.
The efficiency argument is a debatable point. Whilst there is potential to save on teacher time, the effort in establishing quality testing templates and databanks of questions and so forth is often under-estimated. There are some concerns also that some online assessment systems encourage surface recall rather than deeper level learning (Hyde, Booth & Wilson, forthcoming) making it all the more critical for assessment to be designed around the learning purpose not dictated by technologies available.
As online learning becomes more mainstream, providers recognise that online learning is an option in all teaching and not something to be accounted for separately, assessment systems will be integrated with other student management systems and business processing. Evaluators of the Australian Flexible Learning Framework found that institutions and providers are only just starting to grapple with the need for integrated management systems to improve quality, efficiency and measurement capabilities (KPMG 2002).
On a pragmatic level writing online assessment tasks to be read by students requires careful attention to detail. Unlike in face-to-teaching, it can be time-consuming for students to seek clarification or further explanation. Students should be told:
Providing examples wherever possible can avoid misunderstandings and can assist students interpret the requirements.
Students should also be given the opportunity to rehearse wherever this can appropriately be done. Practice tests, past exams, additional or extension activities are just some of the ways to do this. In on-campus teaching students can attend library or be given the opportunity to rehearse in class. In online courses the student may or may not have this option, hence opportunities should be built into the online learning provisions if they do not.
Assessing student readiness for online learning and online assessment is another important consideration. It should not be assumed that students are ready and able to cope with new forms of learning and assessment. Preparation may be required and support necessary (Hyde, Booth & Wilson forthcoming).
The term 'assessment' is often intertwined with 'evaluation' as is the case in Bonk's (2002) study of training provider experiences of online learning as assessment. Student and user feedback quite logically are a key process in the refinement of systems and assessment decisions.
In providing a range of tools and resources to draw from, Anderson (2001) warns of the dangers in educators adopting an 'atomistic approach' to assessing knowledge and skills rather than taking a bigger picture view and visualising and preparing holistic online assessment material.
Considering principles and quality measures for online assessment in the context of online learning has caused people to revisit the important elements of assessment and to reflect on the whys, and consequence of assessment, as well as the how to and options. It has raised consciousness on the topic (Anderson 2001). This consciousness now needs to be applied within mainstream frameworks such as the AQTF and a culture of critical reflection, sharing, and continuous refinement needs to be established.
Anderson, J 2001, Final Report: Flexible Learning Leaders.
Research investigation on Online Assessment as an Integral Part of Flexible
Online Delivery.
http://flexiblelearning.net.au/leaders/past_fellows/2001/janice_anderson.htm
[AQTF] Australian Quality Training Framework: Standards
for Registered Training Organisations, Australian National Training Authority.
http://www.anta.gov.au/publication.asp?qsID=86
Bonk, CJ 2001, Online Teaching in an Online World, CourseShare.com.
http://www.publicationshare.com/
Bonk, CJ 2002, Online Training in an Online World, CourseShare.com.
http://www.publicationshare.com/
Booth, R, Clayton, B, House, R, and Roy, S 2002, Maximising
Confidence in Assessment Decision-making: A Resource to Assist Assessors to
Maintain the Quality of their Assessment Systems. NCVER, Adelaide.
http://www.ncver.edu.au/research/proj/nr9029.pdf
Booth, R, Clayton, B, Hyde, P, Hartcher, R & Hungar, S unpublished, Creative, Quality Online Assessment. Progress report to FLAG, July 2002.
Booth, R, Clayton, B, Hyde, P, Hartcher, R & Hungar, S
forthcoming, The development of quality on-line assessment in VET. Project
description at
http://www.ncver.edu.au/cgi-bin/gda.pl?id=2074
Booth, R, Hartcher, R & Hyde, P 2002, 'Creating Quality
Online Assessment - The Experience to Date', Paper presented AVETRA 5th Annual
Conference, Caulfield, Victoria. Papers being prepared for publication.
To be available at:
http://www.avetra.org.au/
Buckell, J 2002, 'Plagiarism at 8 per cent', The Australian,11
September.
http://www.theaustralian.news.com.au/printpage/0,5942,5071391,00.html
Clayton, B & Booth, R 2000, 'How Flexible is Assessment
in Online Delivery?' Paper presented NET*Working 2000 Conference.
http://flexiblelearning.net.au/nw2000/talkback/p34.htm
Clayton, B & Booth, R forthcoming. How Flexible is Assessment
in the Flexible Delivery of VET. Project Description:
http://www.ncver.edu.au/cgi-bin/gda.pl?id=1712
Freeman, M & McKenzie, J 2000, 'Self and Peer Assessment of Student Teamwork: Designing, implementing and evaluating SPARK, a confidential, web based system', in Flexible Learning for a Flexible Society, Proceedings of ASET-HERDSA 2000 Conference, Toowoomba, Qld 2-5 July, ASET and HERDSA. CD-ROM of Conference proceedings can be ordered at http://www.herdsa.org.au/r&dpricelist.html
Hyde, P, Booth, R, & Wilson, P forthcoming, in Online Learning Research Readings, NCVER, Adelaide.
KPMG in conjunction with Lifelong Learning Associates 2002,
Evaluation of the Australian Flexible Learning Framework 2000-2001, Australian
National Training Authority.
http://flexiblelearning.net.au/evaluation/The_resources.htm
Laurillard, D 1996, quoted in McLoughlin & Luca (2001), How should UK higher education make best use of new technology? Paper presented at ALT-C. Glasgow, United Kingdom.
Leask B 2001, Issues in Online Delivery: Assessment - Workshop
notes, Flexible Learning Centre, University of South Australia
http://www.unisanet.unisa.edu.au/learningconnection/otl/
Maor, D 1998, 'How Does One Evaluate Students' Participation
and Interaction in an Internet-based Unit?' in Teaching and Learning in Changing
Times, Proceedings of the 7th Annual Teaching Learning Forum,4-5 February
1998, The University of Western Australia.
http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf98/maor.html
McLoughlin C and Luca J 2001, 'Quality in Online Delivery: What Does it Mean for Assessment in e-Learning Environments?' in Meeting at the Crossroads proceedings of Australasian Society for Computers in Learning in Tertiary Education (ASCILITE), 9-12 December, Melbourne. http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/mcloughlinc2.pdf
McMahon, M & Luca, J 2001, 'Assessing Students' Self-Regulatory
Skills', in Meeting at the Crossroads, Proceedings of Australasian Society
for Computers in Learning in Tertiary Education (ASCILITE), 9-12 December, Melbourne.
http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/mcmahonm.pdf
Paulsen, MF 2000, An International Analysis of Web-based
Education and Strategic Recommendations for Future Development of Online Education,
Powerpoint presentation, IVETTE Workshop, Barcelona.
http://xiram.doe.d5.ub.es/ivette-w/doc34.html
Radloff, A & de la Harpe, B 2001, 'Expanding What and How
We Assess: Going Beyond the Content', Teaching and Learning Forum 2001.
http://cea.curtin.edu.au/tlf/tlf2001/radloff.html
Resource Generator, Training Package Assessment Guides.
http://www.resourcegenerator.gov.au/loadpage.asp?TPAG.htm
Salmon, G 2000, E-moderating: the key to teaching and learning online, Kogan Page, London.
Slay, J 2001, Issues in Online Delivery: Assessment,
Flexible Learning Centre, University of South Australia.
http://www.unisanet.unisa.edu.au/learningconnection/otl/
TAFE Frontiers Project, Assessment for Online Learning,
Linda Wyse & Associates. Project in progress. Details at:
http://www.tafefrontiers.com.au/content/projects/assess.html
TAFE NSW 2001, Report of the Review of Product Research
and Development within TAFE NSW.
http://www.tafe.nsw.edu.au/publications/reviproc/welcome.htm
Toolbox Central - Job Aids and Guides relating to Assessment
and accompanying resources: Plan Online Assessment; Develop Learning and Assessment
Strategy; Identify Assessment Activities; Choosing Assessment Strategies; Assessment
Issues.
http://toolboxcentral.flexiblelearning.net.au/
West-One Services, Flexible Learning Innovation 2002 Projects
- Assessment Generator. Information about project at:
http://flexiblelearning.net.au/innovations/2002.htm
Williams, JB 2000, 'Flexible Assessment for Flexible Delivery:
On-Line Examinations that Beat the Cheats'. Paper presented at Moving Online
Conference, 2000. Gold Coast, Australia.
http://www.scu.edu.au/schools/sawd/moconf/mocpapers/moc33.pdf
[1] Clayton & Booth 2000, quoting Issues Paper, ANTA National Flexible Delivery Task Force,1996.
[2] Toolbox Central - Job Aid 'Plan Online Assessment'.
[3] Toolbox Central - Job Aid on 'Assessment Issues'.
[4] Toolbox Central - Job Aid on 'Develop Learning and Assessment Strategy'.
[5] Toolbox Central - Job Aid on 'Assessment Issues'.
[6] NCVER (2000) Student Outcomes Survey 2000, National Report, Adelaide, cited in TAFE NSW 2001.
Backroad Connections Pty Ltd 2002, Online Assessment
(Version 1.00), Australian Flexible Learning Framework Quick Guides series,
Australian National Training Authority
an initiative within
the Australian Flexible Learning Framework for the National Vocational Education
and Training System 2000-2004.
URL: http://flexiblelearning.net.au/guides/assessment.html
Last updated: 2 December 2002
Maintained by: Jack Gilding <jack@backroad.com.au>