An Analysis of Student Self-Assessment of Online, Blended, and Face-to-Face Learning Environments: Implications for Sustainable Education Delivery

Abstract

Sidney R. Castle (Corresponding Author) Department of Educational Administration, National University 11395 North Torrey Pines Road, San Diego, California 92037, United States Tel: 1-858-642-8363 E-mail: [email protected]

Chad J. McGuire Department of Public Policy, University of Massachusetts, Dartmouth 285 Old Westport Road, Dartmouth, Massachusetts 02747, United States Tel: 1-508-999-8520 E-mail: [email protected]

Online delivery has the potential to offer significant benefits in achieving multiple goals related to sustainable education. For example, students from a variety of backgrounds can access educational opportunity, allowing for vast dissemination of education. In addition, the methods employed in online learning are generally much lower in carbon intensity, providing an added operational benefit to online education. Beyond these stated benefits, we must also identify what components of online education are deemed effective from the student’’s perspective. This article summarizes a recent study conducted by the authors on overall student self-assessment of learning at a major online university, and compares these results with general pedagogical assumptions regarding the perceived benefits of online learning. The goal is to highlight what factors students find important in ensuring quality learning outcomes in the online learning environment. The extension of this work is to link successful components of online delivery to the general achievement of sustainability in education delivery methods.

Keywords: Online, Education, Sustainability

1. Introduction

The use of online learning is increasing in higher education. There are a number of reasons why this is occurring, including its potential to provide flexible access to content and instruction at any time, from any place and cost-effectiveness for institutions of higher education. Online learning can increase the availability of learning experiences for learners who cannot, or who choose not to, attend traditional face-to-face (onsite) offerings, it offers an opportunity to disseminate course content more cost-efficiently, and also enables higher student to faculty ratios while maintaining a level of outcome quality equivalent to face-to-face instruction.

It has also been suggested that online learning can enhance the quality of learning experiences and outcomes by allowing for a complex and varied community of learners (Bransford, Brown, & Cocking 1999; Riel & Polin 2004; Schwen & Hara 2004; Vrasidas & Glass 2004). In addition, other research has indicated the merits of online learning by focusing on the benefits of asynchronous discourse, suggesting such discourse is inherently self-reflective and therefore more conducive to deep learning than the synchronous-type discourse one would expect in a fully face-to-face setting (Harlen & Doubler 2004; Hiltz & Goldman 2005; Jaffee, Moir, Swanson, & Wheeler 2006).

If the preceding premises hold true, then one would expect to find student self-assessment of learning to generally reflect the conjectures identified above. For example, students who have engaged in ““deep learning”” would more readily identify a greater satisfaction in self-learning. These results suggest, at a minimum, that online learning can be an effective tool in achieving sustainability outcomes in various measures. For instance, based on what has been presented, online education can provide a sustainable vehicle for the delivery of education, while also ensuring a high degree of student satisfaction, both in terms of actual and perceived learning outcomes.

To test these premises, the authors conducted a statistical analysis on a significant data set of student self-evaluation, attempting to discern the relationship between student self-assessment of learning and the teaching modality employed (onsite, online, hybrid, etc.). The results of this analysis suggest student

36 ISSN 1913-9020 E-ISSN 1913-9039www.ccsenet.org/ies International Education Studies Vol. 3, No. 3; August 2010

self-evaluation of learning was highest amongst onsite course offerings, followed by hybrid and then online offerings. While there may be many potential explanations for this trend, the trend itself suggests the conjectures identified above, even if true, do not necessarily correspond with student self-assessment of learning. This is an important trend to consider, especially where the future success of online courses is likely dependent and driven by student demand. While we offer potential reasons for the trends identified, further research is needed to more comprehensively identify the pedagogical connections between learning satisfaction and the different course delivery modalities current employed.

2. Methodology

The sample for the study consisted of 4,038 course assessment summaries over fiscal year 2009 –– 2010 year-to-date. Individual courses average 25-30 students per course. Single courses, such as independent studies, were excluded from the study.

The responses taken for statistical analysis were derived from a thirty-question student assessment employed by the university, which contains a five point Likert scale. The thirty questions were clustered into four categories: student self-assessment of learning; assessment of teaching; assessment of course content; and assessment of web-based technology. This study focused on the results for the category of student self-assessment of learning, which was derived from seven of the thirty-question student assessment. An example of this category is provided in Table 1. For the analysis, a focus was placed on Question 4 identified in Table 1: ““I gained significant knowledge about this subject.”” The purpose was to minimize the clustering effect that potentially weighted certain questions to favor onsite or hybrid learning, for example, Question 2 in Table 1: ““My oral communication skills have improved.”” Oral communication skills are admittedly diminished in the traditional online learning environment, especially where the majority of the course activities rely on asynchronous communications.

Course summary data of student self-assessment was sorted by academic level (undergraduate and graduate), and finally by course delivery modalities of Online, Hybrid, and Onsite. Each of the three major course delivery modalities were modified in some instances to include online supplements for onsite courses, online courses with synchronous audio/visual components and these represented the final fourth-level of sorting. Descriptive statistics were used to determine trends in both undergraduate and graduate courses across the three major course delivery modalities across the 2009-2010 fiscal year-to-date. Next, data was subjected to ANOVA statistical analysis and then, if statistically significant differences were found, to post-hoc Scheffe analysis to determine where those differences occurred. The data was then analyzed for trends between the different teaching modalities, comparing these modalities to student self-assessment of learning.

3. Results

The results for both undergraduate and graduate self-assessment of learning by teaching modality employed are shown in Figure 1.

Figure 1 represents the variety of teaching modalities employed by National University during the 2009-2010 academic year. The main categories of consideration are the onsite, online, and hybrid delivery methods. However, other teaching modalities, such as the onsite with online supplement (““Onsite/Online Sup””) are provided to help potentially explain how various combinations of teaching modalities may combine to aid in a more higher student self-assessment of learning.

The primary results indicate that both undergraduate and graduate students across various disciplines generally prefer onsite learning to either online or hybrid teaching modalities. However, the data shows that undergraduate students tend to prefer hybrid to online teaching, while graduate students generally prefer online to hybrid teaching. In addition, there is a general trend in the data results indicating both undergraduate and graduate students generally score onsite forms of education delivery the highest, but also score hybrid and online modalities high where they are part of specialized course instruction, for example where independent studies use online components but still meet face-to-face, or where online courses are supplemented with synchronous technologies, such as ClassLive Pro (CLP).

The differences in undergraduate student self-assessments of learning, as measured by question 4, between the seven different course delivery modalities is statistically significant at the .001 level. The two areas of statistically significant differences that are found are between ““traditional”” onsite v. ““traditional”” online modalities (.001 level) and between ““traditional”” onsite v. online courses with a synchronous communication component (.001 level). These online courses utilized a CLP component for supplementary instructor-student interactions. While not statistically significant, it is interesting to note that undergraduate students reported the

Published by Canadian Center of Science and Education 37

www.ccsenet.org/ies International Education Studies Vol. 3, No. 3; August 2010

highest self-assessment of learning (question 4) for onsite courses that incorporate an online resource component.

The differences in graduate student self-assessments of learning, as measured by question 4, between the ten different course delivery modalities is statistically significant at the .001 level. ““Traditional”” onsite courses are statistically significantly different from hybrid courses, ““traditional”” online courses, online courses with a synchronous communication component, and online practicum/project/thesis courses all at the .001 level. It is interesting to note that online courses taught exclusively with synchronous communication technology reported the highest level of student assessment of learning (question 4).

4. Discussion

The results obtained in the sampling data suggest student self-assessment of learning (a measure of overall satisfaction) is dependent on factors beyond the simple delivery of online course content. When measured against other modalities, online delivery of education ranks below more traditional methods of instruction, particularly onsite instruction. However, online courses that employ technologies that more closely mimic onsite, face-to-face interactions (for example, synchronous interactions via live video and audio feeds) tend to show higher levels of student satisfaction than entirely asynchronous online delivery. The implication for using online course delivery to maximize sustainable outcomes is clear; online courses should employ a mix of synchronous interaction opportunities to maximize student satisfaction opportunities. However, the authors recommend this use of increasing technology be approached with some caution.

One of the biggest assumptions commonly made in the development of e-learning programs is that the more visually appealing a program, the more learning that will occur; therefore, it is easy to assume that the way to create a premier e-learning course is to simply add more media (such as animation, video and illustration). This is not necessarily true since the purpose of media elements should be to deliver the content and instructional methods, not to make a program merely look appealing. Visual appeal therefore is simply a byproduct of good instructional design. A truly premier e-learning course is one that will look attractive, feel vibrant, encourage participation, and incorporate activities that support the learning objectives and various learning styles of its participants. In addition, it will combine elements of synchronous and asynchronous learning in a way that maximizes student engagement while maintaining the core course objectives and goals.

Online course delivery can be an effective way of obtaining multiple goals in sustainable education. It offers the benefits of educational access to a wide array of potential students, while also limiting the carbon intensity of course delivery. However, the benefits suggested in the literature, including the notion of ““deep learning”” through reflective asynchronous delivery methods, are not clearly reflected in this study of student self-assessment of learning. Rather, students seem to desire a mixed balance of synchronous and asynchronous delivery methods when engaging in the online environment. This is an important factor to consider when thinking about the overall benefits to online education in meeting sustainability goals.

One variable that was not controlled for in this study is the impact of the particular instructor on student self-assessment of learning. The instructor-student relationship is an important factor in student learning, and can likely impact student self-assessment of learning regardless of quality of the online course design. However, we can assume instruction will vary equally in both online and onsite course delivery modes such that this factor would weigh equally in both categories of assessment. Still, further research may be appropriate to better understand if there are statistically significant differences between the general quality between onsite and online instruction.

5. Conclusion

Student learning is the result of the interaction of three distinct components that differ in importance in different course delivery modalities. The most important component across all course delivery modalities is course content; outstanding advanced technology and instructor competence and the ability to ““connect”” with students have little impact if course content material does not facilitate and reinforce the learning experience. The positive impact of efforts to constantly upgrade course content material, especially with respect to online instruction, is clearly seen in student assessments. Once course content has been addressed, then instructor teaching skills and the ability to connect with students and motivate them is likely an important component. This connecting between student and instruction may be seen in the data where onsite courses, or courses that emulate the onsite experience, reported the highest levels of student self-assessment of learning.

The third component related to successfully student learning are the different course delivery modalities themselves. The data clearly show that given course content and instructor competence, modalities that afford

38 ISSN 1913-9020 E-ISSN 1913-9039

www.ccsenet.org/ies International Education Studies Vol. 3, No. 3; August 2010

the greatest degree of instructor-student interaction provide the highest levels of student self-assessment of learning. Advanced technology including components such synchronous audio/video communication may enhance the learning experience but are likely of lesser importance than course content and instructor competence.

Course assessment instruments need to be carefully reviewed and more consideration given to how component questions are clustered and reported. Further research needs to focus on how student and instructor interaction with course content elements impacts upon the student learning experience. The trend toward online v. onsite instruction is remarkable and has quickly impacted higher education. The online learning environment requires new teaching skills and an awareness of how to use emerging technology but, unfortunately, many current faculty members are ill prepared to successfully operate in the online environment. Schools of higher education must balance their efforts to develop premier e-learning courses with a corresponding reeducation program for current faculty to prepare them to operate in the new emerging educational paradigm. This is likely a necessary prerequisite to obtaining sustainable education delivery outcomes using the online environment.

References

Bransford, J. D., Brown, R. L. & Cocking, R.R. (1999). How people learn: Brain, mind, experience, and school. Washington, D.C.: National Academy Press.

Harlen, W. & Doubler, S. (2004). Can teachers learn through enquiry online? Studying professional development in science delivered online and on-campus. International Journal of Science Education 26 (10), 1247––67.

Hiltz, S. R. & Goldman R., eds. (2005). Learning together online: Research on asynchronous learning networks. Mahwah, N.J.: Lawrence Erlbaum.

Jaffe, R., Moir, E., Swanson, E. & Wheeler, G. (2006). EMentoring for Student Success: Online mentoring and professional development for new science teachers. In Online professional development for teachers: Emerging models and methods, ed. Dede, C. 89––116. Cambridge, Mass.: Harvard Education Press.

Riel, M. & Polin, L. (2004). Online communities: Common ground and critical differences in designing technical environments. In Designing for virtual communities in the service of learning, ed. Barab, S.A., Kling, R., & Gray, J.H. 16––50. Cambridge, Mass.: Cambridge University Press.

Schwen, T. M. & Hara, N. (2004). Community of practice: A metaphor for online design. In Designing for virtual communities in the service of learning, ed. Barab, S.A., Kling, R., & Gray, J.H. 154-78. Cambridge, Mass.: Cambridge University Press.

Vrasidas, C. & Glass, G.V. (2004). Teacher professional development: Issues and trends. In Online professional development for teachers, ed. Vrasidas C. and Glass, G.V. 1––12. Greenwich, Conn.: Information Age.

Table 1. Example of student self-assessment of learning cluster, combining seven different assessment questions together to achieve a mean score of student’’s self-assessment of learning.

Published by Canadian Center of Science and Education 39

www.ccsenet.org/ies International Education Studies Vol. 3, No. 3; August 2010

Figure 1. Trends of Self-Assessment of Learning by Teaching Modality

Undergraduate and Graduate trends in self-assessment of learning by teaching modality for courses taught between July 2009 and January 2010. It is clear onsite yields the highest average self-assessment of learning in both the undergraduate and graduate courses examined. However, undergraduate students seem to identify better learning outcomes with hybrid than fully online formats, while graduate students seem to identify higher learning outcomes with online than hybrid teaching formats.