Designing a Multi-Faceted SOLO Taxonomy to Track Program Design Skills Through an Entire Course

Abstract

This paper explores how to assess students’ skills in program design and how those skills evolve across an entire CS1 course. We gathered various data from students, including programming samples and transcripts from interview and think-aloud sessions. As we coded the data, a progression resembling a SOLO taxonomy appeared to emerge bottom-up. As we refined this with top-down perspective from our curricular goals, we ended up with a novel multi-faceted SOLO taxonomy to track students’ progress. We also identified data that don’t fit a SOLO progression, yet reflect relevant traits and habits about design. In applying our framework, we learned several lessons about defining SOLO taxonomies and study protocols that leverage them. The major contributions of this work are (1) a taxonomy with separate but inter-related progressions for different design skills (that is applied to an entire course rather than a single assignment), along with (2) various methodological lessons about applying and designing assessments around SOLO taxonomies in this context.