Comparing Measures of Student Performance in Hybrid and MOOC Physics Courses

  • Trevor A Balint George Washington University
  • Raluca Teodorescu Montgomery College
  • Kimberly Colvin University at Albany SUNY
  • Youn-Jeng Choi University of Alabama
  • David E. Pritchard Massachusetts Institute of Technology


In this paper we use seven quantitative measures of student performance to compare the performance of students
enrolled in three physics courses (two hybrid and one MOOC) that have some common features. We find that,
despite the fact that these courses have different audiences, aims, and methods, the measures presented here place the students from all three courses on the same scale and reveal performance similarities. All measures are compared pairwise and the sign of the correlation between each pair is consistent for all courses. The percentage-based measures all positively correlate with each other and with Item Response Theory measure, while the measures based on average number of submissions positively correlate together but anti-correlate with some percent-based and IRT measures. Our findings suggest that for all course types students who get a higher fraction of problems correct tend to use fewer submissions to do so and have a higher IRT skill, while students in a MOOC choose more frequently to not attempt a problem upon opening it than students enrolled in hybrid courses.


Bao, L., Cai, T., Koenig, K., Fang, K., Han, J., Wang, J., … Wu, N. (2009). Physics. Learning and scientific reasoning. Science (New York, N.Y.), 323(5914), 586–7.
Beichner, R., Saul, J., & Abbott, D. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. … -Based Reform of …. Retrieved from
Bergner, Y., Droschler, S., Kortemeyer, G., Rayyan, S., Seaton, D., & Pritchard, D. E. (2012). Model-Based Collaborative Filtering Analysis of Student Response Data: Machine-Learning Item Response Theory. International Educational Data Mining Society, 95–102.
Beuckman, J., Rebello, N. S., & Zollman, D. (2007). Impact of a Classroom Interaction System on Student Learning. In AIP Conference Proceedings (Vol. 883, pp. 129–132). AIP.
Bonham, S., Beichner, R., & Carolina, N. (2001). Online homework: Does it make a difference? The Physics Teacher, 39(5), 293.
Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003). Comparison of student performance using web and paper-based homework in college-level physics. Journal of Research in Science Teaching, 40(10), 1050–1071.
Bowley, A. L. (1928). The Standard Deviation of the Correlation Coefficient. Journal of the American Statistical Association, 23(161), 31–34.
Breslow, L., Pritchard, D. E., Jennifer, D., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Research & practice in assessment, 8(March 2012), 13–25.
Cheng, K. K., Thacker, B. A., Cardenas, R. L., & Crouch, C. (2004). Using an online homework system enhances students’ learning of physics concepts in an introductory physics course. American Journal of Physics, 72(11), 1447.
Colvin, K., Champaign, J., & Liu, A. (2014). Comparing Learning in a MOOC and a Blended On-Campus Course. In Proceedings of the 7th International Conference on Educational Data Mining (pp. 343–344). Retrieved from Poster.pdf
Colvin, K. F., Champaign, J., Liu, A., Zhou, Q., Fredericks, C., & Pritchard, D. E. (2014). Learning in an introductory physics MOOC: All cohorts learn equally, including an oncampus class. The International Review of Research in Open and Distributed Learning, 15(4). Retrieved from
Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science (New York, N.Y.), 332(6031), 862–4.
Docktor, J., Heller, K., Henderson, C., Sabella, M., & Hsu, L. (2008). Gender Differences in Both Force Concept Inventory and Introductory Physics Performance. In AIP Conference Proceedings (pp. 15–18). AIP.
Docktor, J. L., & Mestre, J. P. (2014). Synthesis of discipline-based education research in physics. Physical Review Special Topics - Physics Education Research, 10(2), 020119.
Dubson, M., Johnsen, E., Lieberman, D., Olsen, J., & Finkelstein, N. D. (2014). Apples vs. Oranges: Comparison of Student Performance in a MOOC vs. a Brick-and-Mortar Course. Retrieved August 23, 2015, from edX. (n.d.). Retrieved from
Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1–26. Retrieved from
Ellison, S. L. R. (2006). In defence of the correlation coefficient. Accreditation and Quality Assurance, 11(3), 146–152.
Finkelstein, N. D., & Pollock, S. J. (2005). Replicating and understanding successful innovations: Implementing tutorials in introductory physics. Physical Review Special Topics - Physics Education Research, 1(1), 010101.
Fraenkel, J., Wallen, N., & Hyun, H. (2014). How to Design and Evaluate Research in Education (9th Edition). McGraw-Hill Education.
Fraser, J. M., Timan, A. L., Miller, K., Dowd, J. E., Tucker, L., & Mazur, E. (2014). Teaching and physics education research: bridging the gap. Reports on Progress in Physics. Physical Society (Great Britain), 77(3), 032401.
Ghadiri, K., Qayoumi, M., & Junn, E. (2013). The transformative potential of blended learning using MIT edX’s 6.002 x online MOOC content combined with student team-based learning in class. Environment. Retrieved from paper.pdf
Harper, K. A., Etkina, E., & Lin, Y. (2003). Encouraging and analyzing student questions in a large physics course: Meaningful patterns for instructors. Journal of Research in Science Teaching, 40(8), 776–791.
Henderson, C. (2002). Common Concerns About the Force Concept Inventory. The Physics Teacher, 40(9), 542.
JO, I.-H., PARK, Y., KIM, J., & SONG, J. (2014). Analysis of Online Behavior and Prediction of Learning Performance in Blended Learning Environments. Educational Technology International, 15(2), 71–88. Retrieved from
Konstan, J. A., Walker, J. D., Brooks, D. C., Brown, K., & Ekstrand, M. D. (2015). Teaching Recommender Systems at Large Scale. ACM Transactions on Computer-Human Interaction, 22(2), 1–23.
Kortemeyer, G. (2014a). An Empirical Study of the Effect of Granting Multiple Tries for Online Homework. Physics Education. Retrieved from
Kortemeyer, G. (2014b). Extending item response theory to online homework. Physical Review Special Topics - Physics Education Research, 10(1), 010118.
Kortemeyer, G., Albertelli, G., Bauer, W., Berryman, F., Bowers, J., Hall, M., … Speier, C. (2003). The LearningOnline Network With Computer Assisted Personalized Approach (LON-CAPA).
Kortemeyer, G., Kashy, E., Benenson, W., Bauer, W., Kashy, E., & Bauer, W. (2008). Experiences using the open-source learning content management and assessment system LON-CAPA in introductory physics courses. American Journal of Physics, 76(4), 438.
Lieberman, D., Dubson, M., Johnsen, E., Olsen, J., & Finkelstein, N. (2014). Physics I MOOC - Educational Outcomes. In Physics Education Research Conference. LON-CAPA. (n.d.). Retrieved from Mechanics Review. (n.d.). Retrieved December 31, 2014, from
Mestre, J., Hart, D. M., Rath, K. A., & Dufresne, R. (2002). The Effect of Web-Based Homework on Test Performance in Large Enrollment Introductory Physics Courses. Journal of Computers in Mathematics and Science Teaching, 21(3), 229–251. Retrieved from /p/9259/
Park, EunsikLee, Y. (2001). Estimates of Standard Deviation of Spearman’s Rank Correlation Coefficients With Dependent Observations. Communications in Statistics - Simulation and Computation, 30(1).
Patterson, R. (2014). Can Behavioral Tools Improve Online Student Outcomes? Experimental Evidence from a Massive Open Online Course. Retrieved from
Pawl, A., Barrantes, A., & Pritchard, D. E. (2009). Modeling applied to problem solving. In AIP Conf. Proc. (Vol. 1179, pp. 51–54). American Institute of Physics (AIP). Retrieved from
Perkins, K. K. (2005). Correlating Student Beliefs With Student Learning Using The Colorado Learning Attitudes about Science Survey. In AIP Conference Proceedings (Vol. 790, pp. 61–64). AIP.
Pollock, S. J. (2009). Longitudinal study of student conceptual understanding in electricity and magnetism. Physical Review Special Topics - Physics Education Research, 5(2), 020110.
Pollock, S. J., & Finkelstein, N. D. (2008). Sustaining educational reforms in introductory physics. Physical Review Special Topics - Physics Education Research, 4(1), 010110.
Rayyan, S., Pawl, A., Barrantes, A., Teodorescu, R., & Pritchard, D. E. (2010). Improved Student Performance In Electricity And Magnetism Following Prior MAPS Instruction In Mechanics, 2–5. Retrieved from
Rayyan, S., Seaton, D. T., Belcher, J., Pritchard, D. E., & Chuang, I. (2013). Participation And performance In 8 . 02x Electricity And Magnetism
How to Cite
BALINT, Trevor A et al. Comparing Measures of Student Performance in Hybrid and MOOC Physics Courses. European Journal of Physics Education, [S.l.], v. 6, n. 3, p. 32-45, feb. 2017. ISSN 1309-7202. Available at: <>. Date accessed: 18 jan. 2021.