Technology Enhanced Assessment for Learning in a Distance Education IT Degree Programme in Sri Lanka

This paper reports on a research study to investigate the impact of assessment practices on the learning process, student learning outcomes and student attitudes to learning in a large-scale distance education IT degree programme in Sri Lanka. The national context for higher education is described, especially the national challenges for widening access to, and participation in, higher education. The study is set within a developmental context which has involved a continual process of improvement through the use of technology enhanced learning. This process has addressed the need to improve retention and progression rates and the need to develop assessment practices in order to improve support for student learning. The paper includes a discussion of related work with regard to the development of online learning communities, promotion of collaborative learning and development of assessment for learning approaches. Findings from the research study are presented which arise from an analysis of students’ enrolment, retention and progression rates in relation to three developmental phases of the programme. The outcomes of a survey of students’ perspectives on their experience of assessment practices within this developmental context are also reported. The discussion identifies successes, areas for further improvement and directions for future research and development in relation to technology enhanced assessment for learning.


I. INTRODUCTION
A. The university system in Sri Lankaaddressing issues of access and participation ri Lanka has a state university system that is made up of fifteen universities, three campuses and seventeen other higher education institutes [25] with a total enrolment of 73,398 undergraduate students and employing 4,341 permanent and 643 temporary academic staff [26]. All higher education institutes are overseen by the University Grants Commission (UGC) created under the University Act of 1978. Another institution awards degrees in IT and Computer Science and, while it has no affiliation to the UGC, it is sanctioned by the government to award degrees. There are also many other private institutions which are not sanctioned by the government to award degrees. These institutions act as proxies of foreign universities that offer degrees in Sri Lanka. Admission to these universities and institutes under the UGC scheme is based on the results of the G.C.E. Advance Level [A/L] examination, a national examination conducted by the Ministry of Education. The subjects students are offered at the G.C.E. A/L examination dictate which study programmes students may take at university. In recent years the number of students who sit for the examination has been approximately 210,000 of whom over 130,000 qualify with the minimum marks required for university admission. However currently, universities in Sri Lanka can only accommodate about 21,000 new admissions per year, representing around 10% of all students sitting for the A/L examination and 16% of those who qualify for admission to universities. These students attend full-time courses on campus [internal] and, like primary and secondary education, tertiary education is free of charge in Sri Lanka. Further, students may be entitled to a monthly scholarship and hostel facilities during their stay at university.
In addition to these full-time internal degrees offered by universities and institutions, there are study programmes referred to as external degrees. On most occasions, the individuals participating in these are part-time students and take the courses off campus [external]. Admission to these degree programmes may be based on the G.C.E. A/L results and/or a qualifying examination.

B. The BIT degree programmewidening participation in higher education
The Bachelor of Information Technology (BIT) degree programme is offered by the University of Colombo School of Computing (UCSC) as an external degree that allows students with an interest in Information Technology [IT] to study for a degree over a period of three years [1]. It has an annual enrolment of over 1,500 students who work at a distance with variable levels of support from local study centres. Students have the option of continuing or leaving the programme with accreditation upon the completion of each year. If a student meets the minimum criteria in their examinations, he or she is awarded a Diploma in IT [DIT] at the end of the first year, a Higher Diploma in IT [HDIT] at the end of the second year and, at the end of the final year, the degree of Bachelor of IT [BIT]. Fig. 1 illustrates the progression path of the BIT degree programme.
Students attending the BIT study programme have diverse education levels as well as diverse objectives. This includes students who are aiming for a degree soon after their A/L examination; those pursuing parallel degrees or seeking a second degree in different subject areas or related subject of the all content in the syllabus was verified by a senior scrutinizer until it is improved to cceptable for all parties. All these precautions were taken since learn centric syllabus in the is the first document to the student and it showed a guide map for self-learning. At the end r, the feedback from industry experts, facilitators/instructors and students were gathered to takes and improve the standard. The course code was changed to reflect these changes in s.
curriculum development, the evaluation structure and guideline was prepared to safe guard ent of learning content and evaluation. Section 5 describes the evaluation in detail. A nsideration was taken when these syllabus were prepared to deliver the course online. y were independent from the face2face interaction.

IGN OF VIRTUAL CAMPUS
n and development of Virtual Campus is an incremental process and the introduction of anagement system to provide the interaction between teachers and students is the very first Virtual Campus system. Learning Management System (LMS) is a kind of information ent system which provides various types of facilities to develop interaction according to pes of learning activities and learning resources. Simply, LMS is similar to "water tank" used to manage and distribute water for registered customers in a particular context. water tank without water couldn't provide its service irrespective of facilities in the water resources and learning activities are similar to water in a water tank. The quality and f the learning resources and activities must be addressed irrespective of LMS. However, a always add value to provide a good learning service to subscribed users of the system. lecting a suitable learning management system (LMS) is a critical decision in designing a pus. A learning management system together with learning content/activities to provide the C58 Fig. 1: BIT curriculum structure and Learning path [10] The number of students in the BIT programme varies greatly from year to year, although student numbers progressing through the programme follow a similar pattern. In some years the withdrawal rate has been as high as 55% and the failure and withdrawal rates have been identified as major issues for the BIT study programme. The limited assistance for learning from the UCSC and issues associated with the assessment system have been highlighted as problem areas during the evaluation of the BIT degree programme. Many international students have dropped out because there is not much support for learning and, unlike in Sri Lanka, there are no supporting teaching institutions in their countries. Further, the price of textbooks outside South Asia has created problems leading to international students leaving the programme.

C. Learning and teaching in the BIT degree programme
Students in the BIT degree programme use various study methods including attendance at private institutions that conduct classes for the BIT study programme, self-study and group study. It is very common for students to adopt a combination of these methods but it is important to point out that some students only engage in self-study.
In most cases students have access to the student manuals in the form of detailed texts about the subject matter which are created for each course in addition to reference books, the BIT Virtual Learning Environment [VLE] with online learning content, assignments, practice tests, CDs with TV programmes, videos along with notes and handouts given by the institutions they attend.
The UCSC plays a very limited role in the actual learning process and acts more as an administrative body. The administration of students together with the preparation, conducting and marking of examinations and student accreditation are the UCSC's main activities. The learning process is aided by the UCSC through the designation of a detailed syllabus with the learning outcomes, topics to be covered and specific references, as well as the development of student manuals and creation of assignments and practice tests and online learning content. The BIT VLE was added as a result of the eBIT project funded by the Swedish International Development Cooperation Agency [SIDA] from 2005 to 2010 [24] with the aim of improving the support provided for students through technology enhanced learning. The VLE consists of online learning content, documents, slideshows, videos and discussion forums.
Students do not have any direct contact with the academic staff who lead the courses. The lecturer in charge of a course has a distinct set of activities he or she must perform. This includes preparing the course syllabus, developing the student manual, moderating the content developed for VLE, creating TV and video programmes, composing assignments and practice tests and creating and marking the final examination.
In its latest attempt to aid the students' learning process, the UCSC has allocated an e-Facilitator who is a dedicated to each course in order to address student questions. The student to e-Facilitator ratio is approximately 876. Even though this ratio seem high it is important to note that the academic staff that lead the course supports the e-Facilitators. Furthermore, many students attend private institutions that conduct classes for the BIT study programme and get assistance from the teachers at those institutions. E-Facilitators respond to students' queries both by email as well as on the online forum, hence they follow an asynchronous mode of assistance. Their assistance is available for students throughout the semester.

D. Traditional approaches to assessmentissues and challenges
At the end of each course, students sit an examination conducted by the UCSC at a designated examination centre. The final examination contributes 100% of the marks and determines a student's final grade for a particular course. The examination centres are located in a few major cities in Sri Lanka as well as some other countries that have a substantial number of students. The examination consists of 40-60 multiple choice questions [MCQs]. These questions tend to test factual knowledge rather than higher order skills and were also found to encourage students to engage in rote learning.
An associated issue to emerge related to guessing on the part of students which raised concerns about the validity of the MCQ assessment system used in the BIT study programme. At the outset of the programme, the BIT examinations discouraged guessing by penalising wrong answers and carrying forward its effects, though this approach was changed in the subsequent revised syllabus in order to increase students' confidence levels.
A further issue to arise relates to the students' problems with language as the BIT study programme is conducted in English, which is a second language for most students. This also leads to some students finding it difficult to understand the questions, thus raising issues of effectiveness in terms of the assessment of learning and the issue of equity in terms of ensuring equal opportunities for all students.
Against this background, and as a result of ongoing evaluation processes from the outset of the programme, there has been a continuous process of quality improvement in place. The major aims of this process have been to improve progression and retention rates, to respond to assessmentrelated issues and challenges, and to respond to associated language issues. These aims have been addressed through developments which have utilised the affordances of technology enhanced learning, including enhanced opportunities for online discussion, dialogue and communication.
The remaining sections of this paper firstly consider related work concerning the development on online learning communities, the promotion of collaborative learning and development of assessment for learning approaches. The third section outlines the methodology adopted which includes an overview of the developmental context, an outline of the approach taken to this research, including research questions and research methods. Findings from the research study are presented in the following section on results which arise from an analysis of students' enrolment, retention and progression rates in relation to three developmental phases of the programme. The outcomes of a survey of students' perspectives on their experience of assessment practices within this developmental context are also reported. The final discussion section identifies successes, areas for further improvement and directions for future research and development in relation to technology enhanced assessment for learning.

A. Learning communities and collaborative learning
An inherent issue in the BIT degree programme is the lack of a teacher. This creates an environment in which the student is isolated unless he or she attends an institute that supports the BIT syllabus. One response to this scenario has been to create the conditions for an online learning community in which students collaborate and take responsibility for their own learning and for supporting that of their peers. Such a learning community is defined by Mioduser, Nachmias et al. [17] and Mioduser and Nachmias [16] as a novel educational system based on a combination of three components: a virtual community [social dimension], hosted by an appropriate virtual environment [technological dimension], and embodying advanced pedagogical ideas [educational dimension]. Based on related work, Ludwig-Hardman and Dunlap [15] define a learning community as a group of people connected via technology-mediated communication who actively engage with one another in collaborative learnercentred activities to intentionally foster the creation of knowledge, while sharing a number of values and practices, including diversity, mutual appropriation and progressive discourse. What is clear from both definitions is that three elements are vital; a technological element, a social element and an educational element. If just one of these elements is missing, a learning community is unlikely to develop. As Lowell and Persichitte [14] state, "simply requiring learner interaction in asynchronous environments does not promote a sense of community". The three factors mentioned by Hiltz [11] can be clearly related to the above elements and can also be identified in Fig. 2 which depicts a Conditional Matrix proposed by Brown [4] in which the inner circles define greater levels of engagement in class and in dialogue, and feelings of belonging to a community.
Although the eBIT programme development was designed to support online learning communities and collaborative learning through the use of forums on the BIT VLE and the Elgg social learning environment, students were found to not be using them extensively. For such a learning community to be successful the challenge is seen to be the creation of a "critical mass" of participants in order to sustain and replenish the community [6] [18] [13].

B. Assessment for Learning
The problems encountered with the use of MCQs are reflected within the relevant literature through the work of other researchers in the field of teaching and learning in higher education such as Scouller and Prosser [20], Scouller [21], Gipps [9] and Paxton [18]. Whilst innovative MCQ development involving such techniques as "confidence measurement" [8] and "computer adaptive testing" [5] are regarded as having potential, such developments do not fully eliminate guessing and or fully address the argument that MCQs feed the answers rather than having the student construct them. A response to supporting the development of a learning community is seen to be by assigning a grade in the form of an extrinsic rewards scheme [23]. However, Brindley et al. [3] argue for a wider approach and in particular for the development of instructional strategies which accomplish the same goal more effectively and with added benefits for the learner. Such a strategy has been developed within the eBIT programme by adopting an approach based on the idea of assessment for learning [2] which means that the first priority of assessment is to serve the purpose of promoting students' learning. Using such a strategy has been seen as one way of combining the approaches proposed by Swan et al. [23] and Brindley et al. [3] to encourage participation. The eBIT phase has involved the development of support in terms of assessment for learning by providing practice tests and a number of optional assignments. However, the assignments are mandatory if the students want to obtain the DIT or the HDIT at the end of the first year and second year, respectively. As these tasks were initially designed to be undertaken as individual activities they did not support the building of a learning community. An approach to improve this situation has involved the use of a Forum on the BIT VLE and the Elgg social learning environment. Coupled with assessment through the award of a grade component, this has been intended to help motivate students to participate in the community and to help create the critical mass needed to sustain it. For those students who are more intrinsically motivated, the learning opportunities within the community are seen to encourage their continued participation.

C. The BIT e-learning environment
The free and open sources software platform Moodle is used as the e-learning environment of the BIT. In order to meet the requirements of the BIT degree programme the e-Learning Centre of the UCSC has developed a customised version of Moddle. Due to a majority of the students registering the BIT degree programme lacking the skills and profesiency in ICT, four courses related to ICT literacy and proficiency were developed for the 1 st semester of Year 1. A further four courses related to basic knowledge and skill required for learning subject matter in Year 2 and Year 3, were developed for the 2 nd semester of Year 1 [10]. Hence, the learning activities of Year 1 courses were created with the intent of engaging the learner with use of interactive multimedia objects. These courses were also supported with online forums for collaboration and interaction, formative assessment tasks through practice quizzes and assignments and the support of learning facilitators. Year 2 and 3 had no learning activities developed with interactive multimedia, but was supported through similar online forums, formative assessment tasks and learning facilitators.

A. The developmental contextimproving quality through technology enhanced learning
Since its outset in 2000 the BIT degree programme has undergone a number of major revisions, which can be classified in three major phases. The first phase (2000)(2001)(2002)(2003) can be described as the "pre-LMS phase" during which the learning support available to students was mainly based on a static website [www.bit.lk] to provide information, detailed syllabi with recommended text books, model papers and model answers, a number of PowerPoint slides, a few lessons through video, and private institutional support.
The second phase (2003)(2004)(2005)(2006) involving the introduction of a Learning Management System (LMS) can be described as the "LMS phase" during which time more support for learning was offered to students in the form of a more dynamic web environment, detailed syllabi with learning objectives, recommended text books and reference page numbers, practice quizzes and assignments and the introduction of a collaborative learning model. This phase had the drawback of the lack of a constructive alignment between syllabus, learning resources and assessment. Also during this phase, the initiative to promote collaborative learning was not successful in the way that had been envisaged. A consequence of this was the creation of a negative impression that the BIT lacked studentlearning support. Further, there was no real connection between processes of continuous assessment and the end-ofsemester examinations.
The third phase or "eBIT phase" (2007-11) reflects the impact of the SIDA funded eBIT project [24] which involved enhanced e-learning content, quizzes and activities for year 1 and activities and quizzes for year 2. A radical change occurred in this phase with the development of much more support for students' learning through rigorous adherence to a constructive alignment between the syllabus, learning content and assessment. The implementation of rich learning content and learning activities aimed at improving students' learning was a vast improvement over the previous phase. Another improvement was the introduction of formative assessment and the creation of a clear relationship between formative and summative assessment in the eBIT phase. This phase also involved the use of forum and chat facilities within the BIT virtual learning environment [VLE] to enhance student interaction and collaborative learning. It also saw the introduction of "UCSC TV" for streaming related video content [29].
BIT students previously had very few opportunities to have their issues and questions addressed and the introduction of an e-Facilitator for each course has helped address this issue in the eBIT phase. A further development during this phase was the introduction of the use of social media as a result of the impact arising from participation in a European Commission funded e-Learning project [7]. This involved the creation of a social network web environment "the eBIT Community" using the Elgg open source social networking engine as a further attempt to support students' collaborative learning and interaction.

B. Approach to the research
The research associated with this development has adopted a case study approach which, following Stake [22], is seen as the study of the particularity and complexity of a single case, coming to understand its activity within important circumstances. Further, it can be seen as a case of a design for technology enhanced learning that focuses on teachingstudying-learning processes and the design of teaching situations, pedagogical activities and learning environments with a particular emphasis on constructively aligned pedagogical activities that support assessment for learning [12]. It is also part of a wider study that has been reported in Usoof et al. [28] and Usoof [27].

C. Research questions
The two key research questions addressed by this study are: 1. How has the introduction of ICT to and revisions of the BIT degree programme affected the progression and accreditation achievements of the students? 2. What is the impact of current assessment practices on the learning process, student learning outcomes and students' attitudes?

D. Research methods
In responding to these questions, two research methods have been adopted. Firstly, an analysis has been conducted in relation to the three phases which have been compared in terms of students' enrolment, retention and progression rates. Secondly, a survey, and associated analysis, was conducted in order to illuminate students' perspectives on their experience of assessment practices within this developmental context.

A. Data analysis of enrolment, retention and progression rates
Enrolment refers to the number of registered students for a particular academic year of the BIT degree programme. This can be looked at in two different ways. First, the number of new students who joined the BIT degree programme and, second, the number of students remaining in the entire BIT degree programme.
The average first-year enrolment for the three phases is 3,630 for the Pre-LMS, 1,678 for the LMS and 1,772 for the eBIT. The drop in enrolment can mainly be attributed to the fact that, during the inception, most students thought they could "just do a degree" and that a conception arose that the BIT was difficult to follow and lacked student support.
The average enrolment levels for the three years  Year 1 to Year 2 The VLE of the BIT degree programme provides firstyear students with student manuals for each course, e-learning content, practice quizzes, assignments, learning activities and a CD with video programmes. The first-year students have a great deal of learning support and can also call upon the assistance of the e-Facilitator assigned to each course. In addition to the learning support, changes were also made to the evaluation criteria in the eBIT phase.
A fourfold improvement in the progression of first-year students to the second year can be seen when comparing the Pre-LMS phase to the eBIT phase and a twofold improvement when comparing the LMS phase to the eBIT phase of the BIT degree programme. The average first-time successes were 11% for the Pre-LMS phase, 21% for the LMS phase and 41% for the eBIT phase.

Year 2 to year 3
Unlike for the first-year students, the second-year students do not receive as much support from the VLE. They are provided with quizzes and learning activities. In comparing the second-year figures, it is important to note that repeat students benefited from the fact that the Math II and Computer Networks papers were made optional.
There is no considerable difference in the first-time success rates between the three phases of the BIT beyond 2001/2002. Even though there has been an increase in the number of students achieving the HDIT certification, it is not visible as a percentage since there has been a rise in the number of students following the second year. The average first-time successes were 51% for the Pre-LMS phase, 55% for the LMS phase and 50% for the eBIT phase.
It is clear from the above figures that there has been a great improvement in the first year of the BIT degree programme in student progression and retention, but there is no change in the figures for the second year beyond 2001/2002. The considerable support arising from the use of technology enhanced learning provided for students in the first year does appear to have had a positive impact. It seems clear that the extremely high withdrawal rate, which was a key issue of the BIT, has been addressed to a certain extent by the advent of technology enhanced learning in the first year. Further, the numbers of new students enrolling in the BIT degree programme has stabilised at around 1,500-2000 each year.

B. Students' perspectives on assessment practices
The survey designed to illuminate students' perspectives on their experience of assessment practices within this developmental context was conducted during 2008-2009 as an online questionnaire which was posted on the LMS of the BIT degree programme. The questionnaire targeted first-year students of the BIT degree taking the Computer Systems I course. The total number of respondents was 45 while 121 students viewed the questionnaire, thus making a 37% response ratio. 73% of the respondents were male, which closely reflects the overall gender balance of students in the BIT programme.
The majority of students who completed the questionnaire were taking the BIT degree programme in order to pursue a career in the IT industry or gain knowledge in ICT. Further, 76% were doing the study programme part-time and approximately 50% of the candidates were employed. While most of the students were taking the degree after their advance level examination, about 10% were pursuing a parallel degree or had completed a bachelor's degree. 64% of the students were either taking or hoping to take an additional study programme.

C. Survey findings
With respect to the type of examination and grading methods, the students most preferred computer-based continuous online assignments from home with a value of 1.9, followed by paper and pencil tests with a value of 2.1 on a 5point Likert scale, with 1 being the most preferred. The least preferred were printed reports/dissertations and digital media reports with values of 2.4 on a 5-point Likert scale, with 1 being the least preferred. 67% indicated end-of-course examinations coupled with continuous assignments as their most preferred grading method, while 22% preferred continuous assignments during the course to determine the final grade.
All of the respondents thought that audio and video in addition to text and graphics would help them express their knowledge better, while 82% preferred a change in the way they are assessed in BIT, 29% preferred a practical test the most, 24% preferred web portfolios/online examinations and 16% preferred open book/take home tests. As the preferred mode of expression, 64% indicated written and typed text, 62% audio-video and 67% graphics and images.
On the issue of the language of study, 86% had a "C" grade for O/L English at least and 67% had a "C" grade for A/L English at least. In addition, 51% were hoping to take an additional English course whilst 60% would rather use a different language to English as their preferred language of expression.
In relation to time pressure during examinations, 80% thought that this adversely affected the quality of their answers, while 38% felt nervous and anxious during examinations. Further, 82% believed they perform better in real life than in examinations. Yet, ironically, 53% thought that examinations were fair in assessing their actual ability and knowledge. Concerning preferences for assessment types, the most preferred were MCQs, open book/take home tests and practical tests. The least preferred were reports/dissertations and essay-type papers.

7
As for study methods of the BIT programme, 82% engaged in self-study, 13% in face-to-face group study, 20% in online group study and 58% attended group classes. In relation to technology use, 93% used computers often and 96% used the Internet often for their education.
With regard to studying, the main goal of 18% of the students was simply to pass the examinations and 13% believed that passing the examinations was the most important aspect. With regard to study methods, 58% of the students also stated that the one they use for multiple choice question examinations is different to when they study for non-MCQ examinations. Overall, 93% of the students had experienced MCQ examinations, but only 13% had experienced reports/dissertations, 29% viva/interviews and 47% web portfolios/online examinations.

D. Summary of the findings
In summary, the majority of the students prefer continuous assessment with an end-of-semester examination, they prefer the use of multimedia for examinations, they have an acceptable command of English but prefer their own mother tongue and they lack confidence about the assessment system and methods. In addition, standardised examinations affect students in a negative way. The majority of students engage in self-study and about 42% have very little interaction with other students or teachers. Almost all students were used to using computers and the Internet for their education. Finally, they have varying experiences with different types of assessment.

V. DISCUSSION
The analysis of the progression and retention rates for the BIT degree programme across the different phases has revealed a clear improvement in the student withdrawal rates and first-time passing of examinations. This pattern can only be attributed to the careful design considerations of the eBIT phase aimed at improving support for student learning. Despite the clear success in the design for learning, the attempts to create a learning community and a collaborative learning environment were not as successful. The main reason for this seems to be the lack of activity modelling, encouragement of community building and creation of collaborative learning.
The analysis of the survey finding show that students prefer not to deviate from assessment methods which are familiar, but however understand the importance of formative assessment. The survey also reveals that many students target their approaches to learning based on the method of assessment with a primary aim of passing examination than focusing on the learning aspects. In addition, the students have identified that the use of ICT and different media allows them to express their skills and abilities better. Furthermore, the survey reveals that students recognise the importance of collaboration for learning.
As a result of this study, a further project was established with a group of volunteer first-year students taking the Computer Systems I course. This has involved the use of community building and collaborative learning activities to encourage the creation of a learning community along with the use of peer assessment as the building block for learning within the community. The outcomes of this study are the subject of a further paper and the findings will be used to design future learning communities and assessment activities aimed at promoting learning and improving the BIT degree programme.