**Authors**
Arushi Arora
Frederick Reiber
Jennifer Wong-Ma
Matthew Ferland
Michael Luu
Michael Shindler
Randy Huynh
Varun Nagaraj Rao

**License**
CC-BY-4.0

What is an Algorithms Course? Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. Michael Luu Matthew Ferland∗ Varun Nagaraj Rao∗ University of California, Irvine University of Southern California Princeton University Irvine, CA, USA Los Angeles, CA, USA Princeton, NJ, USA luum6@uci.edu mferland@usc.edu varunrao@princeton.edu Arushi Arora Frederick Reiber Jennifer Wong-Ma Randy Huynh Boston Universtiy Michael Shindler arushia2@uci.edu Boston, MA, USA jwongma@uci.edu randylh@uci.edu freddyr@bu.edu mikes@uci.edu University of California, Irvine University of California, Irvine Irvine, CA, USA Irvine, CA, USA ABSTRACT to their wide applicability, algorithmic skills are foundational to Algorithms courses are a core part of many CS programs, but have improving student programming ability [15]. received little focus in computing education, lacking statistical data Educators play an important role in establishing curriculum stan- about how they are generally taught. To remedy this, we present the dards and ensuring graduates are competent. The ACM Computer results of the first large-scale comprehensive survey of undergradu- Science Curricula 2013 [8] makes such an attempt for undergraduate ate introductory algorithms courses at four-year institutions in the CS education. It provides “realistic and adoptable recommendations” United States. Questions in the survey targeted instructor informa- for 18 knowledge areas, including “Algorithms and Complexity.” tion, course concepts, the ways students are evaluated, challenges This curricula was formed through a combination of survey data instructors encountered, and instructor envisioned improvements. and and several rounds of community feedback. However, this lacks We received 87 responses from 34 different states, across a wide detailed (public) data about the content and structure of courses. variety of 4-year institutions. The results indicate that algorithms It is important to understand the current content and structure courses vary dramatically in most surveyed areas. of courses for several reasons. First, it can help establish curricu- lum standards like those published by the ACM [1, 2, 8]. Second, CCS CONCEPTS the complexity of transferring course credits between institutions can be reduced when content is aligned and prerequisites are well- • Social and professional topics → Computer science educa- calibrated [3]. And third, it can help inform future curriculum revi- tion. sions and education research, such as the prioritization of topics for inclusion in a concept inventory (CI), like one created for data KEYWORDS structures [10]. For more about Computer Science CIs, see [14]. algorithms education, survey, curricula We set out to answer two research questions - What is the cur- ACM Reference Format: rent content and structure of introductory algorithms courses? and Michael Luu, Matthew Ferland, Varun Nagaraj Rao, Arushi Arora, Randy What associations and trends emerge from the analysis of the content Huynh, Frederick Reiber, Jennifer Wong-Ma, and Michael Shindler. 2023. and structure of algorithms courses? To answer these questions, we What is an Algorithms Course? Survey Results of Introductory Under- have conducted the first large-scale comprehensive survey of in- graduate Algorithms Courses in the U.S.. In Proceedings of the 54th ACM troductory algorithms courses at institutions in the U.S. that offer Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023), a four-year undergraduate degree in CS or a closely related field March 15–18, 2023, Toronto, ON, Canada. ACM, New York, NY, USA, 7 pages. (e.g. software design, information technology). We sent the survey https://doi.org/10.1145/3545945.3569820 to 411 faculty from across all 50 states who have recently taught such a course. We received 87 responses. The rest of the paper is 1 INTRODUCTION structured as follows - we discuss data collection and survey design The study of the design and analysis of algorithms is fundamental (Section 3), we present quantitative results of our survey (Section 4) to undergraduate computer science (CS) education [8]. In addition and we analyze our results to obtain additional data-driven insights ∗ Equal Contribution (Section 5). Overall we see major divergences in course structure and content This work is licensed under a Creative Commons Attribution International 4.0 License. 2 RELATED WORK SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Prior work on algorithms education has primarily focused on identi- © 2023 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9431-4/23/03. fying student misconceptions. Farghally et al. [4] study misconcep- https://doi.org/10.1145/3545945.3569820 tions across a list of algorithm analysis topics, including asymptotic SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Michael Luu et al. analysis and recursive analysis, among others. Shindler et al. [12] In the second part, based on the core topics from the ACM Com- perform a replication study of [19] that targeted misconceptions puter Science Curricula 2013 [8], we asked where the topic is first about dynamic programming. Özdener [9]’s study identifies stu- taught (i.e., prerequisite course, the algorithms course in question, dents’ misconceptions about time-efficiency of algorithms, while other elective or required courses, nowhere in the curriculum, un- Velázquez-Iturbide [16]’s work addresses misconceptions about sure, or other). Multiple core topics under Basic Analysis were com- optimization problems and their corresponding algorithms. bined into the topic “Asymptotic Analysis” to simplify and shorten Other research related to algorithms concerned effective teach- the survey length. Similarly, we included an abridged version of the ing and evaluation strategies [5, 11, 18] and means to incorporate Proof Techniques section from the Discrete Structures knowledge responsible-computing content into the course structure [6]. area to identify how proofs were integrated into the course. The More relevant to our work is the study conducted by Hertz [7]. “Basic Automata, Computability and Complexity” and “Advanced Their survey was focused on CS 1 and CS 2 course topics, whereas Data Structures, Algorithms, and Analysis” sections were preceded our survey concerned all algorithms course topics. Similar to our by a question on whether the instructor taught any topics that own work, they found significant divergence between courses. could fall under either section and if not, the section was skipped. Finally, we asked three open ended questions, on issues encoun- tered, desired course changes, and any other comments. 3 RESEARCH METHODS 3.1 Data Collection 3.3 Threats to Validity The first step of our data collection process entailed creating a (non- The most significant threat to the validity of our survey is in the comprehensive) list of academic institutions to include in the study. selection process of courses and institutions. In the first wave, we We only included universities in the United States that had a 4-year selected the initial universities ourselves. As such, the first institu- computer science (or related) program. We also ensured that each tions we reached out to were more often well-known universities. of the 50 states was represented, at least to some extent. Our second and third waves addressed this problem by attempting Once our list of institutions had been constructed, we began ex- to be almost fully comprehensive in previously uncovered states amining the catalog descriptions of undergraduate courses at each along with a few other states. However, it should be noted that this institution in order to identify a suitable course. If the course title in- means we do not have a perfectly random sample. Secondly, while dicated the class was introductory and solely an algorithms course “properly titled” introductory courses would always be surveyed, (e.g., “Design of Algorithms,” “Analysis of Algorithms,” “Fundamen- programs that didn’t have such a course were only surveyed if they tal Algorithms,” etc.), we included it. Otherwise, if there was no included a topic from “Algorithmic Strategies.” This criterion, how- such course, we would include another course with “algorithm(s)” ever, may cause some confirmation-bias, since this pre-supposes in the title so long as there was evidence that at least one topic that algorithms courses need to cover these strategies. from the ACM’s “Algorithmic Strategies” was included. This condi- Another threat comes from the six adjunct respondents. While tion was most often applied to “Data Structures and Algorithms” adjuncts likely know the course they are teaching very well, it is courses, which were often a terminal algorithmic course. possible that they are less familiar with the general curriculum of After creating lists of undergraduate algorithms courses, for each the institution as a whole. A survey by the TIAA Institute indicates course, we compiled a list of up to three instructors who had taught around 23% of adjuncts have jobs outside of academia, and 26% it within the past 2 years. To find the instructors for each course, we teach at multiple institutions [17]. As such, this could lead to certain applied a variety of methods. The most common way was through information being omitted or misrepresented. looking at publicly viewable course schedules which list instructors’ Finally, there are threats related to doing surveys. All of the contact information, though other sources were employed, such data is given voluntarily by those who responded, which can bias as course websites. After finalizing the list of emails, we sent an the sample. There can be errors when filling out surveys. For ours invitation to participate in the survey to all instructors in that email particularly, we allowed the final two sections to be skipped if the list. Each survey respondent was offered a $25 Amazon gift card instructor indicated they did not teach any topic in the section. for completing the survey. It could be that in these cases, the topics are taught in different locations at these universities. There is also the issue of sample size; 3.2 Survey Design we only received responses from 87 instructors. When designing the survey, our goal was to gain insight into both the course structure and the topics typically taught. The survey 4 RESULTS contained two main parts. The first, consisting of three sections, Our initial list had 615 higher-education institutions. Of these, 495 focused on the organization of the course. These sections ask about had a 4-year computer-science adjacent program, and 373 of those general course details, such as grading breakdowns, dedicated class had an applicable algorithms course. From this, we were able to section times, programming assignments, written assignments, and send out emails to 411 instructors from 302 institutions whose in-class assessments (exams and quizzes). contact information we were able to find. We were able to gather One of our motivations behind these questions was to determine responses from 87 instructors across 79 institutions in 34 states. emphasis on the theoretical components versus applied components 57 of these responses were from doctoral universities with 36 be- of the courses. As such, we asked questions about both the content ing from an R1 university, 12 from an R2 university, and 9 from a and number of assignments. doctoral/professional university as defined by the Basic Carnegie Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Classification [13]. 14 of these responses were from master’s degree- granting institutions, and 16 were from baccalaureate colleges. These rates mirror the ACM 2013 curricula survey [8]. 70 of these institutions used a semester system, seven used a quarter system, and two classes had a term length of 12-13 weeks. As answered by respondents, of the distinct courses, 70 were upper-division courses, 11 were lower-division, and 2 were indicated as both (by different respondents teaching the same course). 4.1 Instructor Background We found that over 90% of instructors were professors, comprising 31 assistant professors, 25 associate professors, and 24 full profes- sors. 6 respondents were adjunct faculty, and one was a graduate student teacher. We also asked about their involvement in teaching in comparison to research. We found that only 11% of instructors considered themselves more focused on research, with 54% being more teaching-oriented. Additionally, we categorized the respon- dents’ research areas into theoretical and non-theoretical1 . Of the 82 that did any amount of research, we classified 39 of them as doing theoretic research. Figure 1: Grading criteria for course categories 4.2 Course Structure 4.2.1 General Course Details. Participants were asked to describe 21% had assignments that involved plotting the algorithms with the grading criteria of their courses. We provided categories for data (eg: the time elapsed at various input sizes). participation, programming assignments, in-class assessments (ex- ams, quizzes, etc), and written homework assignments2 (Figure 1). 4.3 Course Topics Among these categories, participation and programming assign- ments typically accounted for the lowest portion of the total course In figure 2, we present graphs of all of the results from this part of grade, with nearly 29% of respondents weighting programming the survey. We use the written part of this section to note what we assignments at 0%, and over 55% of respondents weighting partic- believe are the highlights. ipation at 0%. In-class assessments, in contrast, were usually the 4.3.1 Algorithmic Strategies. This first part of the Course Topics highest portion of the course grade, with over 26% of instructors section deals with topics concerning algorithm design techniques. noting that they weighted it at over 60% of the total grade. Of the 79 responses3 , we observe that nearly all of the responses For the “other” category, responses varied. There were presenta- cover divide-and-conquer and greedy algorithms with one and four tions, student-made class notes, digital quizzes, reading assignments responses respectively stating the topic is covered in a different (with a flipped-classroom), take-home exams, and some logistical course. The same is true for dynamic programming, with only seven complications due to different grading systems. responses stating they do not cover it. The other topics were taught 4.2.2 Evaluation of Student Mastery. Among the 81 instructors in between 24% and 58% of classes. who gave non-programming homework assignments, the average 4.3.2 Fundamental Data Structures and Algorithms (DSA). Multiple number of assignments was 8.1, with many indicating that they topics related to data structures, like binary search, numerical algo- required proof writing, algorithm design, complexity analysis, and rithms, binary trees, hash tables, and sorting algorithms, are taught algorithm tracing. Similarly, for the 85 instructors who gave in-class in a prerequisite course for 45 to 60% of responses. Moreover, graph assessments, an average of around 4.3 were given, again with the topics, such as graph representations as well as breadth- and depth- same five areas as the homework. first search, are taught in over 50% and over 60% of the courses respectively. Notably, shortest-path and minimum-spanning tree 4.2.3 Programming Assignments. Of the 66 respondents (76%) with algorithms were taught by over 80% of the respondents. programming assignments, they assigned on average 5.2 through- out the academic term. There was significant variation among 4.3.3 Proof Techniques. We split the proof techniques section into which languages were accepted - Java (89%), Python (47%), C++ two main categories: basic logic (ie: notions of implication, equiva- (36%), and C (13%). 13% accepted any language. 91% of instructors lence, converse, inverse), and mathematical proof techniques. Proof indicated their assignments including having students implement techniques were taught in algorithms by only 11 instructors (13%), algorithms, 82% had assignments where they design their own, and and only 1 taught elementary logic. Both topics were covered in pre- vious courses by 47 respondents (54%), with 2 indicating only proofs 1 We considered theoretical to be any research involving algorithms, data structures, were taught in a perquisite, and 7 (8%) indicating that logic was theory of computation, complexity, or other heavily mathematically-oriented fields. 2 One respondent did not use a weighted grading system. They instead had specific 3 Due to an early error in the survey, we did not receive responses from 8 instructors requirements for each grade. They gave >60% to homework, quizzes, and participation for this section and the Fundamental Data Structures and Algorithms section. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Michael Luu et al. (a) Algorithmic Strategies (b) Fundamental Data Structures and Algorithms (c) Basic Analysis (d) Proof Techniques (e) Basic Automata Computability and Complexity (f) Advanced Data Structures Algorithms and Analysis Figure 2: Distribution of where Sub-Topics are taught as defined by ACM Computer Science Curricula 2013 [8] Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Theme Description Q1 1 Difficulties regarding students’ insufficient background, par- ticularly in these areas: A) Mathematical and proof-writing skills B) Programming and implementation skills 2 Difficulties in creating problems for in-class assessments and homework assignments, especially those with answers that cannot be easily found online 3 Difficulties with illustrating relevance and real-world applica- tions for algorithms 4 Difficulties with a particular algorithms topic that students struggled with Figure 3: Breakdown of themes of 80 responses for Q1 5 Difficulties in having students adequately apply algorithm (left)4 and 46 responses for Q2 (right) design techniques to new problems Q2 6 More programming assignments and focus on implementation previously taught but not proofs. Most other instructors indicated of algorithms that it was taught in a required (but non-prerequisite) course. 7 Cover less topics, move topics to other courses (existing or new), or replace topics with other topics 4.3.4 Basic Analysis. The basic analysis section of our survey con- 8 Ensure prerequisite courses cover necessary knowledge tained several categories, such as asymptotic analysis and recur- 9 More emphasis on proofs and mathematical aspects rence relations. The majority of respondents (over 50% or higher 10 Add lab/recitation section to the course for more chances to for each topic) indicated all eight topics as being covered in their practice problems own algorithms courses; around 10%-20% of respondents noted that Table 1: Themes in instructors’ difficulties with teaching these basic analysis topics were covered in a prerequisite course. algorithms (Q1) and in changes instructors’ would like to implement, assuming full control over course (Q2) 4.3.5 Basic Automata, Computability, and Complexity. Automata and complexity theory was skipped by 52% of instructors surveyed. However, the other 48% cover at least P vs NP, 45% cover NP- completeness, and 10% also cover the halting problem. Only 2% to 4% of instructors covered other complexity and theory of computation 5 ANALYSIS topics, but those topics appeared elsewhere in the curriculum in an elective, prerequisite, or required course. 5.1 Implications of Instructors’ Background As previously noted, 39 respondents reported performing research 4.3.6 Advanced Data Structures, Algorithms, and Analysis. Topics in theoretical computer science or mathematical fields while 47 in advanced DSA and analysis were skipped by 47% of instruc- were involved in other kinds of research or none at all. For the tors. Out of the remaining 53% of instructors, graph theoretical 37 of those 39 theory researchers who gave written homework algorithms such as topological sort and finding strongly connected assignments, 33 (89%) indicated that they require students to write components were common with 73% of instructors who answered proofs, all 37 required algorithm runtime analysis, 35 (95%) required this section covering it (38% overall). Network flow is also fairly designing algorithms, and 22 (59%) required tracing algorithms. For common, covered by 64% of this section’s respondents (34% overall). the 44 instructors involved in other fields of research who gave Balanced tree-related algorithms were usually covered in prereq- homework, 28 (64%) required proofs, 39 (89%) required algorithm uisite courses. Instructors who answered this section indicate that runtime analysis, 37 (84%) required algorithm design, and 35 (82%) the remaining topics from this section are either typically covered required tracing algorithms. by an elective course or not covered in the curriculum at all. Notably, those involved in theoretical research required proofs more often and tracing algorithms less often than those not involved 4.4 Free Response Questions in theoretical research, suggesting some difference in emphasis on In the final section of the survey, we asked respondents three op- the mathematics of algorithms versus the implementation. This is tional free-response questions about difficulties encountered in also reflected in the in-class assessments, sporting similar trends to teaching their course (Q1), changes they would like to implement the data regarding assignments. Although, it is significant to note assuming they had full control over their course (Q2), and any other that only 61% and 49% of theory and non-theory researching instruc- final thoughts they believed would be important for us to know. tors respectively expect students to write proofs in assessments. We found commonalities among the responses and categorized Furthermore, the grading criteria for programming assignments, them into common themes with Q1 and Q2 each having 5 different written assignments, and exams did not significantly differ between themes (with 2 sub-themes for Q1), listed in Table 1. Percentage instructors involved in theoretical and non-theoretical research. breakdowns of these themes are presented in Figure 3. 4 The stacked bar graph represents responses showing Theme 1 and subthemes 1A and section showcasing the percentage of responses that did not fall under 1A or 1B, often 1B (responses may have exhibited either or both as shown) with the remaining blue due to plainly stating that prerequisite knowledge was lacking. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Michael Luu et al. 5.2 Themes from Free Response Questions the topics). Given these observations, it is possible that the “typical” When teaching algorithms, by far the most common issue instruc- DSA topics concern tree traversal and graph algorithms. Further- tors reported facing was insufficient background on prerequisite more, 𝑂 (𝑛 log 𝑛) sorting algorithms are the only DSA topic that has concepts, with 54 of the 80 (68%) responses to Q1 exhibiting this any moderate associations with any Algorithmic Strategic topics, (Theme 1). 40 out of those 54 (74%) made references to students’ most notably having an Cramér’s V score of 0.34 with divide-and- weak foundation in mathematics and proof-writing. Only nine re- conquer, suggesting 𝑂 (𝑛 log 𝑛) sorting algorithms may be covered sponses (17%) expressed concern with a lack of programming skills to serve as examples for algorithmic paradigms. For the advanced to properly implement algorithms (the final 10 responses (19%) not algorithms topics, they have moderate intra-associations and weak elaborating on the prior knowledge students were lacking). inter-associations, possibly implying these topics are rarely covered Another concern raised was the assessment of students’ algo- together. Balanced trees, graph-theoretic algorithms, and network rithm design skills (Themes 4 and 7). Instructors frequently com- flow, which show up in a higher percent of algorithms/prerequisite mented on the difficulty in creating assessments and assignments courses, have less association with the other advanced topics. for students to apply their knowledge in designing algorithms. A We did not find any high associations between institution data key difficulty is to come up with novel problems, unique from those and instructor’s research area and our survey variables, suggesting discussed online. Another is that some students are unable to apply that algorithms course structure and topics are not particularly algorithm design beyond problems encountered in class. dependent institution type or professor research background. Respondents also noted the abstract nature inherent in learning these topics as a struggle. Themes 5 and 6 illustrate this as in- 6 FUTURE WORK structors note students’ struggles in recognizing the importance of A more focused survey could expand upon the themes we derived. algorithms and their relevance with real-world applications. Some Other research could look for and apply solutions to these issues. instructors listed particular algorithmic concepts that students Finally, conducting surveys in other areas can further increase our had trouble with as well, most commonly dynamic programming, understanding of the current state of CS education. greedy algorithms, and NP-completeness. Of the 46 responses we collected for Q2, we witness two cat- 7 CONCLUSION egories of themes: those expressing desires to add to the course In this study, we found that while instructors often use the term in some way, like adding more material, emphasizing particular “algorithms” to refer to a specific course, the actual course being topics, or providing more opportunities to apply what they learned referenced has a large variation between academic institutions: it (Themes 6, 9, 10) and those which seek to address the difficulty of can be oriented around mathematics, programming, or somewhere the course by reducing the amount of material covered or ensuring in between. Even within these divisions, instructors often select prerequisite courses cover the concepts needed (Themes 7 and 8). very different subsets of topics. In addition, instructors regularly feel students are unprepared for their courses and have a variety of 5.3 Associations Among Survey Variables and new directions they wish to take it - whether to focus on certain Institution Data course topics or to make the class more or less mathematical. We looked to see if there were any associations between the survey As educators and researchers, we need to take into account variables and additional data we collected on the institutions we the diversity in how algorithms courses are taught. Future studies surveyed5 . We also used our previous categorization of instructors’ should keep this in mind, and either make efforts to be applicable research in algorithms or non-algorithms adjacent research as a to a variety of courses or be tailored to a specific version of the variable. We obtained these associations by calculating the Cramér’s course. When interacting with new transfer students or graduate V measure for each pair of variables6 . students, we should be careful not to make assumptions about the We found that there are some moderate associations between topics covered in prior algorithms courses. Finally, as educators, variables from the same sections of the survey. We first see this with we should make use of this heterogeneity to pull elements from the variables from the “Evaluation of Student Mastery” sections other courses into our own to continue to improve and refine. where question types on the homework and exams were correlated with one another. Likewise, for the Fundamental DSA topic sec- ACKNOWLEDGEMENTS tion, there are moderate associations between quadratic sorting We are extremely grateful to the 87 respondents who filled out the algorithms, hash tables, binary trees, and heaps, having Cramér’s V survey. We are grateful to Michael Dillencourt for helpful com- values in the range 0.18 and 0.55. On the other hand, graph represen- ments with early drafts of the survey. We would also like to thank tations, depth- and breadth-first search, shortest-path algorithms, an anonymous reviewer from a different paper that one author pub- and minimum spanning tree algorithms, the most common topics lished at CCSC-Southwest 2022, which helped to highlight the need from the DSA section, have Cramér’s V measures less than 0.13 with for this work. Varun Nagaraj Rao was supported in part by NSF the other topics (and associations between 0.13 and 0.78 amongst awards #1916153, #1956435, and #1943584. Matthew Ferland was 5 This includes the institution’s Basic Carnegie Classification, its research activity, and supported in part by Simons Investigator Award for fundamental whether or not the institution was public/private and religious/nonsectarian. and curiosity-driven research and NSF grant CCF-1815254. We also 6 We note that these associations values do not indicate the kinds of relationships thank UCI’s Academic Senate Council on Research, Computing and the variables have and whether variables are positively or negatively correlated with one another in one manner. We provide interpretations of these associations with the Libraries (CORCL) for the research fund that enabled us to provide intention of yielding additional potential avenues of consideration and future study. gift cards as incentives for participants. Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada REFERENCES Computing Machinery, New York, NY, USA, 111–119. https://doi.org/10.1145/ [1] ACM. 2005. Computing Curricula 2005. https://www.acm.org/binaries/content/ 3291279.3339404 assets/education/curricula-recommendations/cc2005-march06final.pdf [11] Michael Shindler, Matt Ferland, Aaron Cote, and Olivera Grujic. 2020. Experience [2] ACM. 2020. Computing Curricula 2020. https://www.acm.org/binaries/content/ report: preemptive final exams for computer science theory classes. Journal of assets/education/curricula-recommendations/cc2020.pdf Computing Sciences in Colleges 35, 10 (2020), 9–14. [3] Philip E Agre. 1999. Information technology in higher education: The “global [12] Michael Shindler, Natalia Pinpin, Mia Markovic, Frederick Reiber, Jee Hoon academic village” and intellectual standardization. On the Horizon 7, 5 (1999), Kim, Giles Pierre Nunez Carlos, Mine Dogucu, Mark Hong, Michael Luu, Brian 8–11. Anderson, Aaron Cote, Matthew Ferland, Palak Jain, Tyler LaBonte, Leena Mathur, [4] Mohammed F Farghally, Kyu Han Koh, Jeremy V Ernst, and Clifford A Shaffer. Ryan Moreno, and Ryan Sakuma. 2022. Student misconceptions of dynamic 2017. Towards a concept inventory for algorithm analysis topics. In Proceedings programming: a replication study. Computer Science Education 32, 3 (2022), of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. 288–312. ACM, Seattle, WA, USA, 207–212. [13] Lee S Shulman. 2001. The Carnegie classification of institutions of higher educa- [5] Mohammed Fawzi Seddik Farghally. 2016. Visualizing algorithm analysis topics. tion. Ph. D. Dissertation. Virginia Tech. [14] Cynthia Taylor, Michael Clancy, Kevin C Webb, Daniel Zingaro, Cynthia Lee, [6] Kathi Fisler, Sorelle Friedler, Kevin Lin, and Suresh Venkatasubramanian. 2022. and Leo Porter. 2020. The practical details of building a CS concept inventory. In Approaches for Weaving Responsible Computing into Data Structures and Algo- Proceedings of the 51st ACM Technical Symposium on Computer Science Education. rithms Courses. In Proceedings of the 53rd ACM Technical Symposium on Computer ACM, Portland, OR, USA, 372–378. Science Education V. 2. ACM, Providence, RI, USA, 1049–1050. [15] Pinar Mihci Türker and Ferhat Kadir Pala. 2020. The effect of algorithm education [7] Matthew Hertz. 2010. What do "CS1" and "CS2" mean? Investigating differences on students’ computer programming self-efficacy perceptions and computational in the early courses. In Proceedings of the 41st ACM technical symposium on thinking skills. International Journal of Computer Science Education in Schools 3, Computer science education. ACM, Milwaukee, WI, USA, 199–203. 3 (2020), 19–32. [8] The Joint Task Force on Computing Curricula Association for Computing Ma- [16] J Ángel Velázquez-Iturbide. 2019. Students’ Misconceptions of Optimization Algo- chinery (ACM) IEEE Computer Society. 2013. Computer Science Curricula 2013. rithms. In Proceedings of the 2019 ACM Conference on Innovation and Technology https://dl.acm.org/doi/pdf/10.1145/2534860 in Computer Science Education. ACM, Aberdeen, UK, 464–470. [9] Nesrin Özdener. 2008. A comparison of the misconceptions about the time- [17] Paul Yakoboski. 2018. Adjunct faculty: Who they are and what is their experience. efficiency of algorithms by various profiles of computer-programming students. [18] Jeffrey Young and Eric Walkingshaw. 2018. A domain analysis of data structure Computers & Education 51, 3 (2008), 1094–1102. and algorithm explanations in the wild. In Proceedings of the 49th ACM Technical [10] Leo Porter, Daniel Zingaro, Soohyun Nam Liao, Cynthia Taylor, Kevin C. Webb, Symposium on Computer Science Education. ACM, Baltimore, MD, USA, 870–875. Cynthia Lee, and Michael Clancy. 2019. BDSI: A Validated Concept Inventory for [19] Shamama Zehra, Aishwarya Ramanathan, Larry Yueli Zhang, and Daniel Zingaro. Basic Data Structures. In Proceedings of the 2019 ACM Conference on International 2018. Student misconceptions of dynamic programming. In Proceedings of the Computing Education Research (Toronto ON, Canada) (ICER ’19). Association for 49th ACM technical symposium on Computer Science Education. ACM, Baltimore, MD, USA, 556–561.