Comparing Measures of Student Performance in Hybrid and MOOC Physics Courses
Abstract
In this paper we use seven quantitative measures of student performance to compare the performance of students
enrolled in three physics courses (two hybrid and one MOOC) that have some common features. We find that,
despite the fact that these courses have different audiences, aims, and methods, the measures presented here place the students from all three courses on the same scale and reveal performance similarities. All measures are compared pairwise and the sign of the correlation between each pair is consistent for all courses. The percentage-based measures all positively correlate with each other and with Item Response Theory measure, while the measures based on average number of submissions positively correlate together but anti-correlate with some percent-based and IRT measures. Our findings suggest that for all course types students who get a higher fraction of problems correct tend to use fewer submissions to do so and have a higher IRT skill, while students in a MOOC choose more frequently to not attempt a problem upon opening it than students enrolled in hybrid courses.
References
Bao, L., Cai, T., Koenig, K., Fang, K., Han, J., Wang, J., … Wu, N. (2009). Physics. Learning and scientific reasoning. Science (New York, N.Y.), 323(5914), 586–7.
http://doi.org/10.1126/science.1167740
Beichner, R., Saul, J., & Abbott, D. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. … -Based Reform of …. Retrieved from http://www.percentral.com/PER/per_reviews/media/volume1/SCALE-UP-2007.pdf
Bergner, Y., Droschler, S., Kortemeyer, G., Rayyan, S., Seaton, D., & Pritchard, D. E. (2012). Model-Based Collaborative Filtering Analysis of Student Response Data: Machine-Learning Item Response Theory. International Educational Data Mining Society, 95–102.
Beuckman, J., Rebello, N. S., & Zollman, D. (2007). Impact of a Classroom Interaction System on Student Learning. In AIP Conference Proceedings (Vol. 883, pp. 129–132). AIP. http://doi.org/10.1063/1.2508709
Bonham, S., Beichner, R., & Carolina, N. (2001). Online homework: Does it make a difference? The Physics Teacher, 39(5), 293. http://doi.org/10.1119/1.1375468
Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003). Comparison of student performance using web and paper-based homework in college-level physics. Journal of Research in Science Teaching, 40(10), 1050–1071. http://doi.org/10.1002/tea.10120
Bowley, A. L. (1928). The Standard Deviation of the Correlation Coefficient. Journal of the American Statistical Association, 23(161), 31–34.
Breslow, L., Pritchard, D. E., Jennifer, D., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Research & practice in assessment, 8(March 2012), 13–25.
Cheng, K. K., Thacker, B. A., Cardenas, R. L., & Crouch, C. (2004). Using an online homework system enhances students’ learning of physics concepts in an introductory physics course. American Journal of Physics, 72(11), 1447. http://doi.org/10.1119/1.1768555
Colvin, K., Champaign, J., & Liu, A. (2014). Comparing Learning in a MOOC and a Blended On-Campus Course. In Proceedings of the 7th International Conference on Educational Data Mining (pp. 343–344). Retrieved from http://educationaldatamining.org/EDM2014/uploads/procs2014/posters/19_EDM-2014- Poster.pdf
Colvin, K. F., Champaign, J., Liu, A., Zhou, Q., Fredericks, C., & Pritchard, D. E. (2014). Learning in an introductory physics MOOC: All cohorts learn equally, including an oncampus class. The International Review of Research in Open and Distributed Learning, 15(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1902/3038
Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science (New York, N.Y.), 332(6031), 862–4.
http://doi.org/10.1126/science.1201783
Docktor, J., Heller, K., Henderson, C., Sabella, M., & Hsu, L. (2008). Gender Differences in Both Force Concept Inventory and Introductory Physics Performance. In AIP Conference Proceedings (pp. 15–18). AIP. http://doi.org/10.1063/1.3021243
Docktor, J. L., & Mestre, J. P. (2014). Synthesis of discipline-based education research in physics. Physical Review Special Topics - Physics Education Research, 10(2), 020119. http://doi.org/10.1103/PhysRevSTPER.10.020119
Dubson, M., Johnsen, E., Lieberman, D., Olsen, J., & Finkelstein, N. D. (2014). Apples vs. Oranges: Comparison of Student Performance in a MOOC vs. a Brick-and-Mortar Course. Retrieved August 23, 2015, from http://www.compadre.org/per/items/detail.cfm?ID=13512 edX. (n.d.). Retrieved from https://www.edx.org/
Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1–26. Retrieved from http://projecteuclid.org/euclid.aos/1176344552
Ellison, S. L. R. (2006). In defence of the correlation coefficient. Accreditation and Quality Assurance, 11(3), 146–152. http://doi.org/10.1007/s00769-006-0087-y
Finkelstein, N. D., & Pollock, S. J. (2005). Replicating and understanding successful innovations: Implementing tutorials in introductory physics. Physical Review Special Topics - Physics Education Research, 1(1), 010101. http://doi.org/10.1103/PhysRevSTPER.1.010101
Fraenkel, J., Wallen, N., & Hyun, H. (2014). How to Design and Evaluate Research in Education (9th Edition). McGraw-Hill Education.
Fraser, J. M., Timan, A. L., Miller, K., Dowd, J. E., Tucker, L., & Mazur, E. (2014). Teaching and physics education research: bridging the gap. Reports on Progress in Physics. Physical Society (Great Britain), 77(3), 032401. http://doi.org/10.1088/0034-4885/77/3/032401
Ghadiri, K., Qayoumi, M., & Junn, E. (2013). The transformative potential of blended learning using MIT edX’s 6.002 x online MOOC content combined with student team-based learning in class. Environment. Retrieved from https://www.edx.org/sites/default/files/upload/edtech- paper.pdf
Harper, K. A., Etkina, E., & Lin, Y. (2003). Encouraging and analyzing student questions in a large physics course: Meaningful patterns for instructors. Journal of Research in Science Teaching, 40(8), 776–791. http://doi.org/10.1002/tea.10111
Henderson, C. (2002). Common Concerns About the Force Concept Inventory. The Physics Teacher, 40(9), 542. http://doi.org/10.1119/1.1534822
JO, I.-H., PARK, Y., KIM, J., & SONG, J. (2014). Analysis of Online Behavior and Prediction of Learning Performance in Blended Learning Environments. Educational Technology International, 15(2), 71–88. Retrieved from http://www.newnonmun.com/article=188862
Konstan, J. A., Walker, J. D., Brooks, D. C., Brown, K., & Ekstrand, M. D. (2015). Teaching Recommender Systems at Large Scale. ACM Transactions on Computer-Human Interaction, 22(2), 1–23. http://doi.org/10.1145/2728171
Kortemeyer, G. (2014a). An Empirical Study of the Effect of Granting Multiple Tries for Online Homework. Physics Education. Retrieved from http://arxiv.org/abs/1407.2276
Kortemeyer, G. (2014b). Extending item response theory to online homework. Physical Review Special Topics - Physics Education Research, 10(1), 010118.
http://doi.org/10.1103/PhysRevSTPER.10.010118
Kortemeyer, G., Albertelli, G., Bauer, W., Berryman, F., Bowers, J., Hall, M., … Speier, C. (2003). The LearningOnline Network With Computer Assisted Personalized Approach (LON-CAPA).
Kortemeyer, G., Kashy, E., Benenson, W., Bauer, W., Kashy, E., & Bauer, W. (2008). Experiences using the open-source learning content management and assessment system LON-CAPA in introductory physics courses. American Journal of Physics, 76(4), 438. http://doi.org/10.1119/1.2835046
Lieberman, D., Dubson, M., Johnsen, E., Olsen, J., & Finkelstein, N. (2014). Physics I MOOC - Educational Outcomes. In Physics Education Research Conference. LON-CAPA. (n.d.). Retrieved from http://www.lon-capa.org/ Mechanics Review. (n.d.). Retrieved December 31, 2014, from
https://www.edx.org/course/mechanics-review-mitx-8-mrevx
Mestre, J., Hart, D. M., Rath, K. A., & Dufresne, R. (2002). The Effect of Web-Based Homework on Test Performance in Large Enrollment Introductory Physics Courses. Journal of Computers in Mathematics and Science Teaching, 21(3), 229–251. Retrieved from /p/9259/
Park, EunsikLee, Y. (2001). Estimates of Standard Deviation of Spearman’s Rank Correlation Coefficients With Dependent Observations. Communications in Statistics - Simulation and Computation, 30(1).
Patterson, R. (2014). Can Behavioral Tools Improve Online Student Outcomes? Experimental Evidence from a Massive Open Online Course. Retrieved from
http://www.ilr.cornell.edu/sites/ilr.cornell.edu/files/cheri_wp165_0.pdf
Pawl, A., Barrantes, A., & Pritchard, D. E. (2009). Modeling applied to problem solving. In AIP Conf. Proc. (Vol. 1179, pp. 51–54). American Institute of Physics (AIP). Retrieved from http://dspace.mit.edu/handle/1721.1/76354
Perkins, K. K. (2005). Correlating Student Beliefs With Student Learning Using The Colorado Learning Attitudes about Science Survey. In AIP Conference Proceedings (Vol. 790, pp. 61–64). AIP. http://doi.org/10.1063/1.2084701
Pollock, S. J. (2009). Longitudinal study of student conceptual understanding in electricity and magnetism. Physical Review Special Topics - Physics Education Research, 5(2), 020110. http://doi.org/10.1103/PhysRevSTPER.5.020110
Pollock, S. J., & Finkelstein, N. D. (2008). Sustaining educational reforms in introductory physics. Physical Review Special Topics - Physics Education Research, 4(1), 010110. http://doi.org/10.1103/PhysRevSTPER.4.010110
Rayyan, S., Pawl, A., Barrantes, A., Teodorescu, R., & Pritchard, D. E. (2010). Improved Student Performance In Electricity And Magnetism Following Prior MAPS Instruction In Mechanics, 2–5. Retrieved from http://dspace.mit.edu/handle/1721.1/63094
Rayyan, S., Seaton, D. T., Belcher, J., Pritchard, D. E., & Chuang, I. (2013). Participation And performance In 8 . 02x Electricity And Magnetism
http://doi.org/10.1126/science.1167740
Beichner, R., Saul, J., & Abbott, D. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. … -Based Reform of …. Retrieved from http://www.percentral.com/PER/per_reviews/media/volume1/SCALE-UP-2007.pdf
Bergner, Y., Droschler, S., Kortemeyer, G., Rayyan, S., Seaton, D., & Pritchard, D. E. (2012). Model-Based Collaborative Filtering Analysis of Student Response Data: Machine-Learning Item Response Theory. International Educational Data Mining Society, 95–102.
Beuckman, J., Rebello, N. S., & Zollman, D. (2007). Impact of a Classroom Interaction System on Student Learning. In AIP Conference Proceedings (Vol. 883, pp. 129–132). AIP. http://doi.org/10.1063/1.2508709
Bonham, S., Beichner, R., & Carolina, N. (2001). Online homework: Does it make a difference? The Physics Teacher, 39(5), 293. http://doi.org/10.1119/1.1375468
Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003). Comparison of student performance using web and paper-based homework in college-level physics. Journal of Research in Science Teaching, 40(10), 1050–1071. http://doi.org/10.1002/tea.10120
Bowley, A. L. (1928). The Standard Deviation of the Correlation Coefficient. Journal of the American Statistical Association, 23(161), 31–34.
Breslow, L., Pritchard, D. E., Jennifer, D., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Research & practice in assessment, 8(March 2012), 13–25.
Cheng, K. K., Thacker, B. A., Cardenas, R. L., & Crouch, C. (2004). Using an online homework system enhances students’ learning of physics concepts in an introductory physics course. American Journal of Physics, 72(11), 1447. http://doi.org/10.1119/1.1768555
Colvin, K., Champaign, J., & Liu, A. (2014). Comparing Learning in a MOOC and a Blended On-Campus Course. In Proceedings of the 7th International Conference on Educational Data Mining (pp. 343–344). Retrieved from http://educationaldatamining.org/EDM2014/uploads/procs2014/posters/19_EDM-2014- Poster.pdf
Colvin, K. F., Champaign, J., Liu, A., Zhou, Q., Fredericks, C., & Pritchard, D. E. (2014). Learning in an introductory physics MOOC: All cohorts learn equally, including an oncampus class. The International Review of Research in Open and Distributed Learning, 15(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1902/3038
Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science (New York, N.Y.), 332(6031), 862–4.
http://doi.org/10.1126/science.1201783
Docktor, J., Heller, K., Henderson, C., Sabella, M., & Hsu, L. (2008). Gender Differences in Both Force Concept Inventory and Introductory Physics Performance. In AIP Conference Proceedings (pp. 15–18). AIP. http://doi.org/10.1063/1.3021243
Docktor, J. L., & Mestre, J. P. (2014). Synthesis of discipline-based education research in physics. Physical Review Special Topics - Physics Education Research, 10(2), 020119. http://doi.org/10.1103/PhysRevSTPER.10.020119
Dubson, M., Johnsen, E., Lieberman, D., Olsen, J., & Finkelstein, N. D. (2014). Apples vs. Oranges: Comparison of Student Performance in a MOOC vs. a Brick-and-Mortar Course. Retrieved August 23, 2015, from http://www.compadre.org/per/items/detail.cfm?ID=13512 edX. (n.d.). Retrieved from https://www.edx.org/
Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1–26. Retrieved from http://projecteuclid.org/euclid.aos/1176344552
Ellison, S. L. R. (2006). In defence of the correlation coefficient. Accreditation and Quality Assurance, 11(3), 146–152. http://doi.org/10.1007/s00769-006-0087-y
Finkelstein, N. D., & Pollock, S. J. (2005). Replicating and understanding successful innovations: Implementing tutorials in introductory physics. Physical Review Special Topics - Physics Education Research, 1(1), 010101. http://doi.org/10.1103/PhysRevSTPER.1.010101
Fraenkel, J., Wallen, N., & Hyun, H. (2014). How to Design and Evaluate Research in Education (9th Edition). McGraw-Hill Education.
Fraser, J. M., Timan, A. L., Miller, K., Dowd, J. E., Tucker, L., & Mazur, E. (2014). Teaching and physics education research: bridging the gap. Reports on Progress in Physics. Physical Society (Great Britain), 77(3), 032401. http://doi.org/10.1088/0034-4885/77/3/032401
Ghadiri, K., Qayoumi, M., & Junn, E. (2013). The transformative potential of blended learning using MIT edX’s 6.002 x online MOOC content combined with student team-based learning in class. Environment. Retrieved from https://www.edx.org/sites/default/files/upload/edtech- paper.pdf
Harper, K. A., Etkina, E., & Lin, Y. (2003). Encouraging and analyzing student questions in a large physics course: Meaningful patterns for instructors. Journal of Research in Science Teaching, 40(8), 776–791. http://doi.org/10.1002/tea.10111
Henderson, C. (2002). Common Concerns About the Force Concept Inventory. The Physics Teacher, 40(9), 542. http://doi.org/10.1119/1.1534822
JO, I.-H., PARK, Y., KIM, J., & SONG, J. (2014). Analysis of Online Behavior and Prediction of Learning Performance in Blended Learning Environments. Educational Technology International, 15(2), 71–88. Retrieved from http://www.newnonmun.com/article=188862
Konstan, J. A., Walker, J. D., Brooks, D. C., Brown, K., & Ekstrand, M. D. (2015). Teaching Recommender Systems at Large Scale. ACM Transactions on Computer-Human Interaction, 22(2), 1–23. http://doi.org/10.1145/2728171
Kortemeyer, G. (2014a). An Empirical Study of the Effect of Granting Multiple Tries for Online Homework. Physics Education. Retrieved from http://arxiv.org/abs/1407.2276
Kortemeyer, G. (2014b). Extending item response theory to online homework. Physical Review Special Topics - Physics Education Research, 10(1), 010118.
http://doi.org/10.1103/PhysRevSTPER.10.010118
Kortemeyer, G., Albertelli, G., Bauer, W., Berryman, F., Bowers, J., Hall, M., … Speier, C. (2003). The LearningOnline Network With Computer Assisted Personalized Approach (LON-CAPA).
Kortemeyer, G., Kashy, E., Benenson, W., Bauer, W., Kashy, E., & Bauer, W. (2008). Experiences using the open-source learning content management and assessment system LON-CAPA in introductory physics courses. American Journal of Physics, 76(4), 438. http://doi.org/10.1119/1.2835046
Lieberman, D., Dubson, M., Johnsen, E., Olsen, J., & Finkelstein, N. (2014). Physics I MOOC - Educational Outcomes. In Physics Education Research Conference. LON-CAPA. (n.d.). Retrieved from http://www.lon-capa.org/ Mechanics Review. (n.d.). Retrieved December 31, 2014, from
https://www.edx.org/course/mechanics-review-mitx-8-mrevx
Mestre, J., Hart, D. M., Rath, K. A., & Dufresne, R. (2002). The Effect of Web-Based Homework on Test Performance in Large Enrollment Introductory Physics Courses. Journal of Computers in Mathematics and Science Teaching, 21(3), 229–251. Retrieved from /p/9259/
Park, EunsikLee, Y. (2001). Estimates of Standard Deviation of Spearman’s Rank Correlation Coefficients With Dependent Observations. Communications in Statistics - Simulation and Computation, 30(1).
Patterson, R. (2014). Can Behavioral Tools Improve Online Student Outcomes? Experimental Evidence from a Massive Open Online Course. Retrieved from
http://www.ilr.cornell.edu/sites/ilr.cornell.edu/files/cheri_wp165_0.pdf
Pawl, A., Barrantes, A., & Pritchard, D. E. (2009). Modeling applied to problem solving. In AIP Conf. Proc. (Vol. 1179, pp. 51–54). American Institute of Physics (AIP). Retrieved from http://dspace.mit.edu/handle/1721.1/76354
Perkins, K. K. (2005). Correlating Student Beliefs With Student Learning Using The Colorado Learning Attitudes about Science Survey. In AIP Conference Proceedings (Vol. 790, pp. 61–64). AIP. http://doi.org/10.1063/1.2084701
Pollock, S. J. (2009). Longitudinal study of student conceptual understanding in electricity and magnetism. Physical Review Special Topics - Physics Education Research, 5(2), 020110. http://doi.org/10.1103/PhysRevSTPER.5.020110
Pollock, S. J., & Finkelstein, N. D. (2008). Sustaining educational reforms in introductory physics. Physical Review Special Topics - Physics Education Research, 4(1), 010110. http://doi.org/10.1103/PhysRevSTPER.4.010110
Rayyan, S., Pawl, A., Barrantes, A., Teodorescu, R., & Pritchard, D. E. (2010). Improved Student Performance In Electricity And Magnetism Following Prior MAPS Instruction In Mechanics, 2–5. Retrieved from http://dspace.mit.edu/handle/1721.1/63094
Rayyan, S., Seaton, D. T., Belcher, J., Pritchard, D. E., & Chuang, I. (2013). Participation And performance In 8 . 02x Electricity And Magnetism
Published
2017-02-27
How to Cite
BALINT, Trevor A et al.
Comparing Measures of Student Performance in Hybrid and MOOC Physics Courses.
European Journal of Physics Education, [S.l.], v. 6, n. 3, p. 32-45, feb. 2017.
ISSN 1309-7202.
Available at: <https://eu-journal.org/index.php/EJPE/article/view/47>. Date accessed: 24 apr. 2024.
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution 4.0 International License.
The copyright for all articles belongs to the authors. All other copyright is held by the journal.
