Re-evaluation of community of inquiry model with its metacognitive presence construct

Among the discussion-content analytical tools in the field of e-learning research, the community of inquiry (CoI) model is extensively applied and continuously improved by its users. This model investigates the types of elements that are manifested through inquiry-based learning processes in online discussions. They are social, cognitive, teaching and metacognitive presences. These elements are essential for meaningful student interactions to take place in online learning environments. In particular, the metacognitive presence construct of the CoI model discovers the students’ ability of self and co-regulation of learning in an online learning environment. However, the metacognitive presence construct of the CoI model has not been evaluated along with the other components of the model. Therefore, in this paper the CoI model was re-evaluated to determine its reliability in analysing discussions in online courses on information technology related subjects. The evaluation is conducted with four online courses designed and developed for a distance learning programme in Sri Lanka. The paper discusses the modifications that were needed to make the model more applicable for conducting discussion-content analysis in similar types of online learning environments, and reports on the results of the final evaluation. Furthermore, the findings of the study imply that the theoretical framework of the CoI model needs to be improved to properly enclose the metacognitive presence component. In spite of this, the study adds points to the CoI model supporting for its well applicability and reliability in analysing online discussion content in information technology related courses.


I. INTRODUCTION
t has become a common practice at higher educational departments conducting distance learning programmes to use e-learning to deliver instructional materials to students. Other than the materials, an e-learning environment can provide students with platforms to engage in discussions. Online discussions based on forums can be created to support both place and time-independent communications, providing more opportunities for students to converse with other students and teachers. If a forum is kept open for discussion throughout a course then it can become a record of how students and teachers interacted with each other. At the completion of the course, this forum can serve as a rich source of information for the course coordinators and designers to analyse and understand student interactions during discussions.
The content can be analysed using different types of analytical instruments, in order to study factors such as student participation and interaction, as well as cognitive, metacognitive and social cues (e.g. [1] and [2]), critical thinking (e.g. [3], [4], and [5]) and group development [6]. Analytical instruments are critically examined with respect to two parameters: validity and reliability. Reliability is a factor determining the quality of an analytical model [7]. Reliability of an analytical model is measured with statistical techniques such as percentage agreement, Cohen's Kappa or Pearson's correlation [8], [9]. A reliable analytical model strengthens the validity of the results in the content analysis. Therefore, results of a discussion content analysis should be preceded by an assessment of reliability of the analytical model. Since most of the issues associated with validity and reliability can be mitigated with sound analytical models having -discrete categories, and clear indicators‖ [10, p. 2], analysts should try to improve analytical models by appropriately modifying their lists and the definitions of categories and indicators.
Among the instruments used for online discussion content analysis, the model based on the Community of Inquiry (CoI) framework is extensively used (e.g. [11] and [12]) in the field of online learning, and it has been continually refined and adapted by researchers [13]. The CoI framework mainly focuses on the nature of educational transaction in an online learning environment, and it has emerged in the context of asynchronous text-based online discussions [14]. The framework reflects the critical thinking processes that could exist in an online discussion. The analytical tool based on this framework is named the Community of Inquiry (CoI) model [15], [16]. It can be used to identify social, cognitive, teaching and metacognitive presence in online discussions. While social, cognitive and teaching presence are essential to foster meaningful interactions, cognitive presence is considered to be the most important element to identify critical thinking activities in a CoI [4]. Metacognitive presence elements are closely linked to critical thinking and higher learning in a discussion [16].
Reviewing research on CoI framework and its analytical model, Garrison and Arbaugh [15] report that there is a need of evaluating the model with all its components and further studying the relationships between the components of the CoI theoretical framework. Moreover, they encourage future research to evaluate the CoI model and the framework in disciplines other than education. As an attempt to meet this requirement, Shea et al. [17] have examined the CoI modelexcluding metacognitive presence component, with minor modifications. However, their findings also highlight the necessity of improving the analytical model further, in particular, to enhance the reliability of the social presence construct. In spite of that, a study of Akyol and Garrison [16] has introduced a new component named ‗metacognitive presence', to the CoI model which has not been evaluated with the rest of the other components of the CoI model.
Metacognitive skill is considered essential for students learning in distance/ online learning environments [18]. Studying metacognitive presence in online discussions can reveal students' latent knowledge and regulatory skills further enabling researchers to discern student learning processes in online learning environments [16]. This understanding can help to design self-directed online learning activities for distance learning programmes. Therefore, we decided to re-evaluate the CoI model with its metacognitive presence construct. This paper reports on our research to re-evaluate and attempts to improve the CoI model with its four components: social, cognitive, teaching and metacognitive presence to make it more reliable for analysing forum discussions in online courses.
Re-evaluation of community of inquiry model with its metacognitive presence construct T. A. Weerasinghe, K. P. Hewagamage, and R. Ramberg The CoI model has become a useful tool for investigating students' interactions in inquiry based online discussions [19], [17]. However, it needs further improvements to make it a more reliable tool to investigate students' learning in online discussions [20]. The model and the framework have further not been examined with its new component, the metacognitive presence construct. The new construct has a potential for disclosing latent knowledge and regulatory skills which cannot be identified with the cognitive presence construct alone. Therefore, we wish to determine whether we can reliably use the metacognitive construct along with the rest of the components of the CoI model. For this purpose, we evaluate the CoI model by analysing a set of discussions selected from four online courses.

B. Theoretical Framework and Model
From its inception, the community of inquiry (CoI) framework has had three components enclosing three types of elements essential for meaningful interactions to take place in a community [4]. The three types of elements and the components are named social, cognitive and teaching presences. The teaching presence component represents the role of the teacher, which is often carried out with the collaboration of a number of individuals who are not teachers [21]. In [22], Shea and Bidjerano recommended that this framework should consider the online learner's self-regulatory learning behaviour and self-efficacy. In line with this reasoning, they suggested adding to the framework another element: learner presence. This stood independently from the rest of the components of the framework. Akyol and Garrison, however, did not agree with this modification reporting that -creation of a learning presence construct would implicitly assign teaching presence to only that of the teacher‖ and it -is incongruent with the premise of a community of inquiry‖ [16, p. 189]. Instead, they introduced the metacognitive presence component, which was placed at the intersection of the teaching and cognitive presence of the CoI framework (see Fig1). Fig1. Community of Inquiry (CoI) Framework (Adapted from [16]) In accordance with this framework, the CoI model has four coding schemes: social, cognitive, teaching and metacognitive presence.
Social presence is defined as -the ability of learners to project themselves socially and emotionally in a CoI‖ [4, p. 94]. The social presence coding scheme has three categories: affective, open communication and group cohesion. Garrison, Anderson and Archer define these categories respectively -in terms of the participants identifying with the community, communicating purposefully in a trusting environment and developing interpersonal relationships‖ [14, p. 7]. Cognitive presence is the main component of the CoI framework. It describes -the extent to which learners are able to construct and confirm meaning through sustained discourse in a critical community of inquiry‖ [23, p. 1]). The cognitive presence coding scheme has four categories that represent the phases of an inquiry process in a collaborative learning environment. The phases are triggering event, exploration, integration and resolution. Triggering event is the initiation phase of a critical inquiry where an issue, dilemma or problem is identified or recognised. The next phase is ‗exploration' where learners tend to grasp the nature of the problem and move to explore relevant information. In the integration phase learners construct meaning from the ideas generated in the exploratory phase. The last phase of the critical inquiry model is ‗resolution'. It indicates a resolution of dilemma or problem that caused the triggering event.
Teaching presence is defined as -the design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes‖ [21, p. 5]. Teaching presence represents the role of teaching, which is carried out by the collaborative involvement of participants in a community [21]. This component of the analytical tool has three categories: design and organisation, facilitating discourse and direct instruction. The design and organisation category considers the role of a teacher during the designing and planning process of online learning activities, while the other two categoriesfacilitating discourse and direct instructioninvestigate signs of teaching presence during students' engagement in learning activities.
Metacognitive presence is the new component that Akyol and Garrison [16] introduced to the CoI model. Referring to contemporary research into metacognition and learning (e.g. [24] and [25]), they defined metacognition in an online learning community as -the set of higher knowledge and skills to monitor and regulate manifest cognitive processes of self and others‖ [16, p. 184]. Moreover, motivational states for learning are considered in describing metacognitive presence in online discussions.
The component has three categories: knowledge of cognition (KC), monitoring of cognition (MC) and regulation of cognition (RC). According to [16], KC -refers to awareness of self as a learner in a broad sense‖, where -knowledge includes entering knowledge and motivation associated with the inquiry process, academic discipline, and expectancies‖ (p. 184). KC characterises pre-task metacognitive states; in other terms, more general aspects of metacognition observed at anytime. In comparison, MC and RC represent activity-based metacognitive states, which can be observed during the learning process.

II. CONTEXT
The examination of the CoI framework and model was carried out by analysing eight discussion threads in the online learning environment of the BIT (Bachelor of Information Technology) degree programme (www.bit.lk) at the University of Colombo School of Computing (UCSC), Sri Lanka. The students of the degree programme did not receive any lectures or feedback from the university teachers on site or in a physical lecture room. Instead, the university distributed course materials using a virtual learning environment. The online courses were designed and developed by the teams of content developers and instructional designers at the e-Learning Centre of the UCSC. There were discussion forums for each course, which provided an opportunity for the students to discuss their concerns. A facilitator was there to assist students in the forums to find answers to their problems.
The courses were offered in English and the recommended language to be used in the online discussions was English. Even though English language competency was considered as an entry requirement for the BIT programme, this was not the first language of the majority of the students. Therefore, their understandings of others' responses were different from native English speakers and the expressions were biased towards their mother tongue.

III. METHOD
The model was examined by evaluating it twice. The first evaluation was conducted using the CoI model improved by Shea et al. [17]. This model is composed of three separate coding schemes for investigating indications of social, cognitive and teaching presence. Along with these, we used a coding scheme for identifying signs of metacognitive presences. It was developed based on the classification of metacognitive presence elements in online discussions by Akyol and Garrison [16]. The first evaluation was carried out by analysing four discussion threads (Sample 1) randomly selected from four online courses in the BIT programme. The coding schemes were improved based on the comments of the coders and the issues that arose at the first evaluation. The improved coding schemes were re-evaluated by analysing eight online discussions that consisted of two online discussions from each of the four courses (see Table 1). These eight included the four discussion threads (Sample 1) that we previously analysed in the first evaluation and four other randomly selected discussion threads (Sample 2).

A. Discussion Threads
There were altogether 99 student messages (S-posts) and 17 facilitator messages (F-posts) in the selection of eight discussions. This represented more than 10% of the total number of messages in the discussion threads having at least five messages and dealing with student inquiries on the online courses. As reported in Table 1, the courses had different types of subject content.

B. Coders and Coding Process
We realised that our students had attempted to form their messages in English while thinking in Sinhala. As a result, they had used phrases that could not be understood by a coder who did not know Sinhala well. Therefore, we selected the coders from Sri Lanka. Both of them were university teachers of information technology in the country. One was in the research team and the other was an external researcher who participated only in the evaluation.
The coders were instructed to follow the same coding procedure that Shea et al. [17] applied in a study to re-examine the CoI model. Additionally, during the discussion for negotiation, when there were disagreements between the two coders, the due reasons for the disagreement were enquired and noted down. Also, the coders' comments and suggestions to improve the coding schemes were gathered.

C. Unit of Analysis
Garrison, Anderson and Archer [22] stated that messagelevel unit was the most appropriate unit of analysis for identifying cognitive presence in their discussions. As messages are demarcated clearly in a discussion, it is easy to consider messages as the unit of analysis if students have posted only one idea in one message. However, in our case, students formed messages including more than one idea in one message. Also, they had not structured the messages into paragraphs, having one idea in one paragraph. Therefore, we decided to use the chunk of a message as the unit of the analysis. A chunk could be a complete message or a meaningful segment of a message, with a cue of a presence that is described in the CoI model.

D. Inter-rater reliability measurements
The coding decisions of the two coders were evaluated for inter-rater reliability using Cohen's Kappa (K) and Holsti's coefficient of reliability (CR). The reason for applying two reliability measurements: K and CR, was to eliminate the weaknesses associated with individual reliability measurements and, thereby, to increase the validity of the results. Cohen kappa values were calculated using the equation on [26, p. 155]. Holsti's coefficient of reliability was calculated referring to [27, p. 140]. In reference to prevailing research, Rourke, Anderson, Garrison and Archer [28] report that inter-rater reliability should be more than 0.8000 for RC and more than 0.7500 for K in order to consider it as a very good agreement. The results of the present evaluation are discussed with respect to this satisfactory level of agreement.

IV. RESULTS
The evaluation of the existing CoI model with its three components: social, cognitive and teaching presence and the coding scheme prepared based on the analytical model of metacognitive presence did not result in initial IRR values at satisfactory levels. Especially the cognitive and metacognitive presence coding schemes were relatively less reliable than the two other coding schemes. The initial IRR values of the social presence coding scheme ranged from RC=0.5333-0.8354 and K=0.3182-0.6393 while that of the teaching presence coding scheme ranged from RC= 0.2857-0.8000 and K= -0.1667-0.5714. Also, a considerable number of negotiated IRR values were below RC=0.8000 and K= 0.7500. For instance, the negotiated IRR of the cognitive presence coding scheme ranged from RC= 0.6522-0.7957 and K= 0.5014-0.6513. This signified that there was a poor agreement (according to [29]) between the two coders. Furthermore, the comments of the coders revealed that they had encountered difficulties in following the coding schemes.

A. Difficulties and Suggested Modifications
In order to make the coding schemes valid and highly reliable the schemes should be well comprehensible and easy to use. This can be achieved by adding more meaning and clarity to the descriptions of categories and indicators [10]. Doing this should assure that components of the model (presences) are properly described by the categories and indicators are with sufficient details to -reflect the essence of the categories‖. [20, p. 68]. With this insight and considering the difficulties faced by the coders and their suggestions to improve the coding schemes, necessary modifications were introduced to the analytical model. The modifications made to the components of the model and their rationales are discussed below.

Social presence coding scheme
During the first evaluation, the coders experienced a difficulty in distinguishing conventional expressions from unconventional ones. Also, there were indications of students expressing emotions and tone of voice by using big letters, capital letters or colour text. Therefore, the two indicators, conventional and unconventional expressions were combined and the definition of the combined-indicator was modified accordingly, including other signs of affective expressions. The coders had long discussions that ended up with disagreements. For instance, at one occasion the coders did not come to an agreement due to ambiguity of the two indicators: ‗expressing emotions' in the affective category and ‗expressing appreciation' in the open communication category in the social presence coding scheme. The coders commented that it was not easy to determine whether an expression was not emotional when expressing an appreciation. For instance, they found chunks such as ‗Wow!' and ‗Best!', which could be interpreted both as appreciations and as emotions. Therefore, we decided to modify the indicator under open communication to ‗encouraging or complementing' The coders revealed that they were not sure of what ‗expressing values' exactly meant. Shea et al. [17] had also noted that the highly subjective nature of ‗expressing values' caused problems to the reliability of the social presence coding scheme. Therefore, we decided to clarify what ‗values' specifically means. Further, the comments of the coders suggested that ‗expressing values' could also be considered in the definition of the ‗self-disclosure' indicator. Therefore, we included ‗expressing values' in the ‗self-disclosure' indicator and modified the definition of ‗self-disclosure' to consider personal values such as beliefs, vision, attitudes and interests.
There were a considerable number of clues that urged us to create two other indicators for the open communication category: ‗accepting mistakes' for the recognition of each other's contribution and ‗expressing willingness to support'. Moreover, we modified the indicator ‗asking questions' to ‗requesting support', because we determined that it would be more suitable to identify students' requests for clarifications and information, which might not be in the form of a question.
The examples provided for each category indicator of the social presence coding scheme were not helpful to our team. Therefore, we created examples relevant to our context and added them to clarify each category indicator.

Cognitive presence coding scheme
During the evaluation, the coders found it very difficult to distinguish the chunks to be matched with the ‗exploration' and ‗integration' categories in the tool. They commented on the ambiguity of the information in the coding scheme and suggested that the socio-cognitive processes and the examples provided in the coding scheme should be improved. Also, the coders noted that the instructions given in the coding scheme were not clear; the instructions implied that some chunks could be matched both with the ‗integration' and ‗exploration' categories. These comments and issues that the coders encountered were considered in improving the coding scheme and the modifications were done by referring to the explanations available in [4], [30] and [14].
The coders encountered messages containing expressions of satisfaction after solving the problem which caused the ‗triggering event'. These messages could be interpreted as clues of resolutions. Therefore, the coders suggested us to add another indicator -‗judging or evaluating and expressing satisfaction' to the ‗resolution or application' category.
Additionally, the text in the examples column was replaced with a few sample codes that were more familiar to our context (see Appendix). Five questions or problems were added to exemplify plausible causes of triggering events. Probable replies to these questions were added as examples for exploration, integration and resolution. Our intention was to make the analytical tool more easily understood by the coders. Also, we removed the extra instructions provided in the descriptor column to let the coders look freely for relevant clues and segment messages and to make coding decisions.

Teaching presence coding scheme
The teaching presence component of the model was evaluated by analysing both facilitator's and students' messages in four discussion threads. This signified the importance in changing the category indicators to make the tool more applicable to identify signs of teaching presence that emerged with the students' teaching activities during the discussions.
The first category of the teaching presence construct is ‗designing and organisation' (DO). Shea et al. [17] had six indicators for this category. The first two indicators were ‗setting curriculum and communicating assessment methods to be used in the course' and ‗designing methods'. The last was ‗macro-level comments about the course', which was defined as ‗provides rationale for assignment/topic'. These indicators were found irrelevant to our context, because in our courses, the facilitator or the students were not supposed to set the course curriculum and assignments, design learning activities or provide macro-level instructions to do the course activities. These kinds of course planning and designing activities had been carried out by the instructional designers before starting the courses. However, the facilitator could use the discussion forums to post announcements relevant to the subject under discussion. Therefore, we added a new indicator, ‗informing notices', to the DO category.
Based on the comments of the coders, the two indicators in the ‗facilitating discourse' category were slightly modified to make them more suitable to the context. The modified indicators were ‗encouraging, acknowledging or reinforcing student contribution' and ‗drawing in participants and prompting discussions' (FD-d). The discussions in the environment did not have any time limitations or restrictions. Therefore, the second indicator, FD-d, could also be considered as encouraging student participation. For this reason, in order to make the two indicators clearly comprehensible and applicable to the environment, we slightly modified the indicators to ‗acknowledging or reinforcing student contribution' and ‗encouraging or motivating students'.
The list of indicators in the direct instruction (DI) category was modified by adding two new indicators: ‗providing specific instructions' and ‗encouraging doing activities'. These were determined during the first evaluation where the coders noticed that the students had provided task-specific instructions to peers and encouraged them to try out challenging activities. Moreover, based on the finding of the evaluation, two other indicators -‗offering useful illustrations' and ‗making explicit references'were modified by including ‗or examples' and ‗providing extra learning resources' respectively. The existing model had another indicator, ‗supplying clarifying information', which was quite similar to the two other indicators that were modified. However, the definition of the indicator described the teaching role of providing additional explanations. Therefore, in order to reduce the ambiguity of the indicator and make it more meaningful the existing indicator was replaced with ‗providing additional explanations'.
Moreover, the comments of the coders urge the importance of replacing the existing indicators of the assessment category with three new indicators. They are ‗judging, evaluating or checking the relevancy of message content', ‗assessing another student's knowledge' and ‗assessing an activity reported or presented in a message'.

Metacognitive presence coding scheme
The coders suggested that we should add examples to clarify the meaning of the indicators in the coding scheme. They reported that it was difficult to understand the meaning of the term ‗knowledge of cognition' and to differentiate the meaning of ‗monitoring cognition' and ‗regulation of cognition'. Therefore, in order to make the metacognitive presence coding scheme easily comprehended a set of examples were added to each category. Appropriate examples were selected from the sample coding in [16] and the rest were created. Furthermore, we removed some descriptive terms and added two indicators to the coding scheme.
According to [31], knowledge of cognition (KC) incorporates general learning strategies and tasks that describe when, why or how one can perform an activity. In line with this reasoning, Akyol and Garrison reported that -KC includes knowledge about cognition, cognitive strategies and tasks‖ [16, p. 184]. However, in order to describe category definitions of ‗monitoring cognition' and ‗regulation of cognition' (RC), they used the terms ‗task knowledge' and ‗strategies' respectively and that was incongruent with the description provided by Pintrich. Therefore, we removed the terms which created confusion and added one more indicator, ‗knowledge about general learning strategies and tasks', to the KC category. Also, we modified the list of indicators in the RC category by adding ‗suggesting taking an action' under the list of ‗applying strategies'. This addition was made after noticing that our students had posted messages including statements such as -Let's discuss about open office software‖ and -It's better to give up‖. These statements could be interpreted as suggestions to take actions during the application of a strategy and thus considered as clues of RC.

B. Results of the 2nd Evaluation
The coding schemes with modifications were re-evaluated for their reliability. This second evaluation (Eval 2) was conducted two months after the first evaluation (Eval 1). The cognitive presence component of the CoI framework was explained to the coders referring to [4]. The same set of discussion threads (Sample 1) was given to the coders and the same procedure was applied during the analysis. Finally, a new set of discussions (Sample 2) was analysed to determine whether we can use the adapted analytical model reliably to analyse online discussions in our courses.
The coders analysed 99 students' messages and 17 facilitator's messages. They identified different numbers of chunks and altogether there were 599 coding decisions that had to be checked or discussed for negotiation. There was a significant increase even in the initial IRR values of all the coding schemes (see Graph 1 and 2). The negotiated IRR values of the schemes ranged from 0.9000 to 1.0000. The reliability values resulted at the evaluation conducted with a new set of discussion threads (Sample 2) are shown in Table 2. Most of the IRR values of the evaluation conducted with Sample 2, reached levels of agreement which were more than RC=0.8000 = and K=0.5000. The negotiated IRR values ranged from RC=0.8600-1.0000 and K=0.8818-1.0000 (see Table 2). These can be interpreted as very good agreements between the two coders. Therefore, we can assume that the modifications we made could improve the schemes.
Moreover, all the coding schemes seemed well applicable in our context. Each coding scheme supported to identify considerable numbers of clues. However -though it was encountered at very few instances-there were coder disagreements regarding decisions related to the analysis using social, cognitive and metacognitive presence coding schemes.
At one instance, the coders had a disagreement regarding a decision pertaining to the ‗open communication' and the ‗affective' categories of the social presence coding scheme. The reason for this discrepancy was the ambiguity of the ‗expressing emotions' indicator in the affective category and the ‗expressing appreciation' indicator in the ‗open communication' category. The disagreement related to the cognitive presence coding scheme was due to un-clarity of a message posted by a student. The two coders interpreted it in two different ways that led one coder to match the whole message with the ‗exploration' category while the other coder matched it with the ‗integration' category.
Each category of the metacognitive presence coding scheme could capture more than 29% of clues -out of the total number of metacognitive presences that could be identified. However, there were 4% of disagreements. One coder explains that those chunks could be considered as signs of bringing previously acquired knowledge to the discussion and thus they can be matched with the ‗knowledge of cognition' category. However, the other coder disagreed with the decision saying that there was not enough information to consider those chunks having signs of previously acquired knowledge.

C. Relationship between metacognitive presence construct and other components
The metacognitive presence coding scheme captured a considerable number of the chunks that could also be captured either by the teaching presence component or by the cognitive presence component of the tool. Moreover, we found that some of the chunks that had been identified using the cognitive presence or the teaching presence coding schemes were segmented further in matching with the categories in the metacognitive presence coding scheme. Also, there were chunks that did not necessarily belong to the chunks with clues of cognitive presence or teaching presence, but had signs of metacognitive presence. For instance, in one discussion, the students considered what a closed system and an open system could be and tried to determine whether a prison could be a closed system or an open system. One student, who was found to have integrated his idea with others' opinions, added: The last part of this message: -I think this question...‖ could be treated separately from the rest of the message. During the analysis using the cognitive presence coding scheme, one coder suggested that it might be considered as a resolution. She reasoned that it might be interpreted as an attempt to seek a resolution or to end the discussion. The other coder did not agree and both decided to ignore that line in the message. However, when using the metacognitive presence coding scheme, both coders divided this line into two chunks and matched the first, -I think this...too much‖ with the ‗monitoring cognition' category and the second, -It's better to give up‖ with the ‗regulation cognition' category. In another instance, a student provided a detailed description of binary arithmetic and reported: -...If you feel anything not clear here please contact me. It is a pleasure to help you! I too had to struggle for days to have a good understanding. So, never give up! Wish you good luck!‖ The coders matched the chunk -I too had to struggle for days to have a good understanding!" with categories in the metacognitive and social presence coding schemes, but not with any of the categories in the teaching or cognitive presence schemes.

D. Typical issues and guidelines
Methodological issues in online discussion content analysis have frequently been discussed in the literature (e.g. [32], [10] and [14]). In such articles, suggestions and advice that analysts can adhere to in order to handle issues related to reliability have been proposed. For instance by incorporating multiple coders, and using Cohen's Kappa to compensate for chance agreement and triangulation methods to increase the validity of the results. However, there is still a need for easily applicable instructions that can support novice analysts to achieve valid results.
During our analysis, the main problem that we encountered was associated with the issue of culture. The students seemed to have formed messages in English while thinking in their mother tongue. Therefore, the coders had to pay extra attention to interpreting messages in the discussion threads. Further, the comments of the coders and our experience emphasised the importance of taking necessary steps to reduce the difficulty in analysing online discussion content in general. This resulted in the formulation of guidelines that could be followed in analysing online discussions. These guidelines are discussed below.
Reformulate messages where it is essential In some cases we had to reform the messages to make them more comprehensible. Therefore, we suggest that the analyst of discussion threads should read all the discussion threads and try to understand the discussion before starting the analysis. While doing this, the analyst can carefully improve the clarity of the messages where it is essential.
Study the context of the discussion In order to understand a discussion, the analyst may need to know the information related to the context of the discussion, which can be obtained from the online course environment where the discussion emerges. This contextual information will probably be essential to make decisions during the coding process. Therefore, if discussions are in printed form then the analyst should go back to the online learning environment and study the contexts of each discussion before starting the analysis.
Understand the inquiry process The reason for using the CoI model was to understand the student inquiry processes that emerged for the purpose of solving problems related to subjects covered in the online courses. This understanding was mainly connected to the analysis conducted using the cognitive presence coding scheme. Therefore, analysts who wish to use the analytical tool of the CoI framework should have a thorough understanding of the CoI framework and the inquiry process.
Comprehend the coding schemes Analysts should be able to comprehend not only the inquiry process and the CoI framework, but also the category definitions and indicators in each coding scheme of the CoI model. This will aid the analysts to investigating the chunks more precisely and, as a result, increase the reliability of the instrument. When there is more than one coder working with the analysis, they should grasp the instructions and information in the coding schemes together and build up a mutual understanding of the coding schemes.
Consider only one coding scheme at a time The CoI model has four components: social, cognitive, teaching and metacognitive presence. Each of these components has three or four categories and altogether there are fourteen categories. Hence, it is not easy for a coder to either remember all the categories and their definitions or to refer to the schemes back and forth during the analysis. Therefore, analysts who are interested in investigating all the elements covered in the CoI model should use only one coding scheme at a time until all the discussions are analysed. This process should be repeated with all the coding schemes.
Double check the work The coders who participated in our evaluation missed a considerable number of clues that could be identified by the model. This emphasises the importance of rechecking the analysis with a coding scheme before going to work with the next scheme.

V. DISCUSSION
The evaluation reported on in this paper aimed to determine whether the CoI model could be used to analyse online discussion content in a learning environment prepared for distance learners in an Asian country. The evaluation was conducted with a sample set of discussion threads in four online courses that covered subject content relating to information technology. Out of these courses, three (i.e. Course 2, 3 and 4) had practical as well as theoretical subject content, while the other course did not have any laboratorybased learning activities.

A. Reliability of the analytical tool
The model was adapted considering our experience and the suggestions brought out by cotemporary researchers for better reliability of the model. For instance, Rourke, Anderson, Garrison, & Archer [33], who developed the social presence analytical tool, examined it by analysing discussions in two graduate-level courses; one in workplace learning and the other in distance learning. They reported that there were issues in investigating clues of expressions of emotions and humour. Shea et al. [17] used an adapted version of the CoI model and re-examined the model by analysing discussion content from two courses in business management at a State college in the United States. They experienced problems with the indicators ‗expressions of values' and other indicators that Rourke et al. [33] also confounded in the social presence coding scheme. Furthermore, we solved another issue encountered by our coders due to the ambiguity of the two indicators -‗expressing emotions' in the affective category and ‗expressing appreciation' in the open communication category.
After doing necessary modification and re-evaluating each construct of the analytical model, we could ensure that the constructs were more reliable than before. The reliability values of the adapted model were at higher levels than the reliability values found by Shea et al. [17]. Therefore, we believe that the CoI model with our modifications can be used reliably to analyse discussions in the online courses. More specifically, the results of the current study imply that the modifications made to the coding schemes are appropriate and relevant for conducting discussion content analysis in our online courses. Further research is welcome to affirm that the adapted model is more appropriate and reliable in other disciplines as well.

B. Theoretical framework and metacognitive presence
The theoretical framework of the CoI analytical model is composed of three major componentssocial presence, cognitive presence and teaching presence. Akyol and Garrison [16] introduced a new component, ‗metacognitive presence' to the theoretical framework and they placed it at the intersection of the cognitive and teaching presence components of the framework. However, in our evaluation, we found cues of metacognitive presence which could be matched neither with cognitive presence nor with teaching presence. This implies metacognitive presence does not fall only on the intersection of teaching presence and cognitive presence. It goes beyond the boundaries of cognitive and teaching presence. Therefore, the findings of the present study imply that the CoI framework may need further improvements to properly enclose the metacognitive presence construct.

VI. CONCLUSION
The present study was conducted to evaluate the CoI analytical model and the findings affirm that the model with our modifications was reliable and more applicable to analyse online discussions in our context. Based on the experience of our coders, a set of guidelines was formulated to lessen the difficulties pertaining to online discussion content analysis in general. Consequently, the validity of the results that analysts can find with this improved model and by following the set of guides will get increased. Therefore, this study adds more value to the CoI model and suggests that researchers will be able to use this model as a highly useful and reliable analytical tool. Hopefully, the implications from findings of the future research using this model will bring up more practical and fruitful suggestions to enhance students' learning experience in online learning environments.
The coding process that we employed in the present study was very time consuming. This signifies the requirement of future research to develop application software to automate the analytical process and thereby evaluate students' learning in online discussions. For instance, the improved CoI model can be used to develop an application to automatically rate students' messages as soon as they are posted to online discussions. This will help teachers to more efficiently evaluate student activities in online discussions and consequently, students will get motivated to participate in online discussions and actively engage in the inquiry-based learning.
Moreover, the present study encourages future research work to investigate possible improvements that can be done to the theoretical framework of the CoI model and properly enclose metacognitive presence construct which is useful for declaring information related to distance online learning environments. "I just can't stand it when…!!!;" "ANYBODY OUT THERE!" "What does it mean!?!?"

APPENDIX
S-AE2 Using expressions of humour Using conventional strategies such as humorous banter, teasing and joking. "He heee..., no need to think, what the book says is correct and I"m wrong." S-AE3 Self-disclosure Disclosing details of personal life, expressing vulnerability or sharing personal beliefs, vision, attitudes and interests.
"Where I work this is what we do..." "I just don"t understand this question." "I believe that we have right to see the assignment marks." Open communication Encouraging or complementing Encouraging others or complementing others" or the content of their messages. "I really like your interpretation of the reading." "Don"t worry. You will find it easy." S-OC6 Expressing agreement or disagreement Agreeing or disagreeing with others or content of their messages. "I was thinking the same thing. You really hit the nail on the head." S-OC7 Accepting mistakes Acknowledging one"s own mistakes and expressing gratitude for being made aware of them (recognition of each other"s contribution).
"Sorry, it"s a typing mistake." "Oh! I have made a big mistake. Addressing or referring to the group using inclusive pronouns Addressing the group as we, us, our group or as friends. "Our textbook refers to ..." "I think we veered off track..."

S-GC3
Salutations and greetings Indicating purely social functions: greetings or closures. "Hi all" "Good luck!" S-GC4 Social sharing Sharing information unrelated to the course content but which helps maintain group cohesion.
"We are having the most beautiful weather here. I will take some photos and send you soon." "I have understood blended learning to be a …" "I like your eloquently worded definition…" "Good points." "…. well today I have learned something about a …" "I think I have been able to think of an example for almost each of the models presented in …" "You make an interesting point when you observe …" "I am not certain why this is true a priori." "Am I correct?" "You all have done very well…" "I am interested in reading from Tom's list." "I think I need to see more supporting research for the idea that …" "One of your solution is …Would it be feasible within …" "Also, I think I am going to need help in understanding" "I am just curious about the social processes … and how they might help learning" "Shall we give up this now and go to the next topic?" "Will we be able to finish this on time?" "Let"s use the new software and share our experience." "This has been discussed in the forum... Please post your question there." "Try to address issues that others have raised when you post."

Regulation of Cognition
T-OD4 Establishing netiquette Helps students understand and practice the kinds of behaviours that are acceptable in online learning, e.g., providing documentation on polite forms of online interaction.
"Keep your messages short." "Remember, all uppercase letters is the equivalent of "shouting." Facilitating discourse (T-FD)  T1. "In section 5, page 152 of the student manual says "solid states"... Could you please explain what it means?" T2. "I think the statement "the Internet uses TCP standards in data transmission" is correct. But in a Quiz, it is considered as incorrect. Can it be a mistake? Please explain." C-TE2 Sense of puzzlemen t Questions or messages that take the discussion in a new direction.
T3. "Sometimes ago, I studied what "bit" and "byte" are. But now, I can"t remember and I am confused. Can someone explain what they are?" T4. "I wanted to print cell borders of a Calc worksheet. But failed. Is there anybody who has done it before?" T5. "Are touch-screen laptops better than normal laptops?" Providing examples of how problems were solved or evidences of successful application.
T4  "How I printed a Calc worksheet with cell borders was..." "It did not work at first. But when I selected some lines of text and tried again then it worked..." C-RA2 Defending solutions Defending why a problem was solved in a specific manner.
T4  "Here is the modified list of steps to print a worksheet with cell borders. I did a small change to the second step of Mahela"s procedure. Because I could not open the Print dialog box by following it as it was." C-RA3 Judging or evaluating and expressing satisfactio n Judgement or evaluation followed by an expression of satisfaction after solving the problem or issue that caused the triggering event.