Authors Arushi Arora Frederick Reiber Jennifer Wong-Ma Matthew Ferland Michael Luu Michael Shindler Randy Huynh Varun Nagaraj Rao
License CC-BY-4.0
What is an Algorithms Course? Survey Results of Introductory
Undergraduate Algorithms Courses in the U.S.
Michael Luu Matthew Ferland∗ Varun Nagaraj Rao∗
University of California, Irvine University of Southern California Princeton University
Irvine, CA, USA Los Angeles, CA, USA Princeton, NJ, USA
luum6@uci.edu mferland@usc.edu varunrao@princeton.edu
Arushi Arora Frederick Reiber Jennifer Wong-Ma
Randy Huynh Boston Universtiy Michael Shindler
arushia2@uci.edu Boston, MA, USA jwongma@uci.edu
randylh@uci.edu freddyr@bu.edu mikes@uci.edu
University of California, Irvine University of California, Irvine
Irvine, CA, USA Irvine, CA, USA
ABSTRACT to their wide applicability, algorithmic skills are foundational to
Algorithms courses are a core part of many CS programs, but have improving student programming ability [15].
received little focus in computing education, lacking statistical data Educators play an important role in establishing curriculum stan-
about how they are generally taught. To remedy this, we present the dards and ensuring graduates are competent. The ACM Computer
results of the first large-scale comprehensive survey of undergradu- Science Curricula 2013 [8] makes such an attempt for undergraduate
ate introductory algorithms courses at four-year institutions in the CS education. It provides “realistic and adoptable recommendations”
United States. Questions in the survey targeted instructor informa- for 18 knowledge areas, including “Algorithms and Complexity.”
tion, course concepts, the ways students are evaluated, challenges This curricula was formed through a combination of survey data
instructors encountered, and instructor envisioned improvements. and and several rounds of community feedback. However, this lacks
We received 87 responses from 34 different states, across a wide detailed (public) data about the content and structure of courses.
variety of 4-year institutions. The results indicate that algorithms It is important to understand the current content and structure
courses vary dramatically in most surveyed areas. of courses for several reasons. First, it can help establish curricu-
lum standards like those published by the ACM [1, 2, 8]. Second,
CCS CONCEPTS the complexity of transferring course credits between institutions
can be reduced when content is aligned and prerequisites are well-
• Social and professional topics → Computer science educa-
calibrated [3]. And third, it can help inform future curriculum revi-
tion.
sions and education research, such as the prioritization of topics
for inclusion in a concept inventory (CI), like one created for data
KEYWORDS
structures [10]. For more about Computer Science CIs, see [14].
algorithms education, survey, curricula We set out to answer two research questions - What is the cur-
ACM Reference Format: rent content and structure of introductory algorithms courses? and
Michael Luu, Matthew Ferland, Varun Nagaraj Rao, Arushi Arora, Randy What associations and trends emerge from the analysis of the content
Huynh, Frederick Reiber, Jennifer Wong-Ma, and Michael Shindler. 2023. and structure of algorithms courses? To answer these questions, we
What is an Algorithms Course? Survey Results of Introductory Under- have conducted the first large-scale comprehensive survey of in-
graduate Algorithms Courses in the U.S.. In Proceedings of the 54th ACM troductory algorithms courses at institutions in the U.S. that offer
Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023),
a four-year undergraduate degree in CS or a closely related field
March 15–18, 2023, Toronto, ON, Canada. ACM, New York, NY, USA, 7 pages.
(e.g. software design, information technology). We sent the survey
https://doi.org/10.1145/3545945.3569820
to 411 faculty from across all 50 states who have recently taught
such a course. We received 87 responses. The rest of the paper is
1 INTRODUCTION
structured as follows - we discuss data collection and survey design
The study of the design and analysis of algorithms is fundamental (Section 3), we present quantitative results of our survey (Section 4)
to undergraduate computer science (CS) education [8]. In addition and we analyze our results to obtain additional data-driven insights
∗ Equal Contribution
(Section 5). Overall we see major divergences in course structure
and content
This work is licensed under a Creative Commons Attribution
International 4.0 License.
2 RELATED WORK
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Prior work on algorithms education has primarily focused on identi-
© 2023 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9431-4/23/03. fying student misconceptions. Farghally et al. [4] study misconcep-
https://doi.org/10.1145/3545945.3569820 tions across a list of algorithm analysis topics, including asymptotic
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Michael Luu et al.
analysis and recursive analysis, among others. Shindler et al. [12] In the second part, based on the core topics from the ACM Com-
perform a replication study of [19] that targeted misconceptions puter Science Curricula 2013 [8], we asked where the topic is first
about dynamic programming. Özdener [9]’s study identifies stu- taught (i.e., prerequisite course, the algorithms course in question,
dents’ misconceptions about time-efficiency of algorithms, while other elective or required courses, nowhere in the curriculum, un-
Velázquez-Iturbide [16]’s work addresses misconceptions about sure, or other). Multiple core topics under Basic Analysis were com-
optimization problems and their corresponding algorithms. bined into the topic “Asymptotic Analysis” to simplify and shorten
Other research related to algorithms concerned effective teach- the survey length. Similarly, we included an abridged version of the
ing and evaluation strategies [5, 11, 18] and means to incorporate Proof Techniques section from the Discrete Structures knowledge
responsible-computing content into the course structure [6]. area to identify how proofs were integrated into the course. The
More relevant to our work is the study conducted by Hertz [7]. “Basic Automata, Computability and Complexity” and “Advanced
Their survey was focused on CS 1 and CS 2 course topics, whereas Data Structures, Algorithms, and Analysis” sections were preceded
our survey concerned all algorithms course topics. Similar to our by a question on whether the instructor taught any topics that
own work, they found significant divergence between courses. could fall under either section and if not, the section was skipped.
Finally, we asked three open ended questions, on issues encoun-
tered, desired course changes, and any other comments.
3 RESEARCH METHODS
3.1 Data Collection 3.3 Threats to Validity
The first step of our data collection process entailed creating a (non- The most significant threat to the validity of our survey is in the
comprehensive) list of academic institutions to include in the study. selection process of courses and institutions. In the first wave, we
We only included universities in the United States that had a 4-year selected the initial universities ourselves. As such, the first institu-
computer science (or related) program. We also ensured that each tions we reached out to were more often well-known universities.
of the 50 states was represented, at least to some extent. Our second and third waves addressed this problem by attempting
Once our list of institutions had been constructed, we began ex- to be almost fully comprehensive in previously uncovered states
amining the catalog descriptions of undergraduate courses at each along with a few other states. However, it should be noted that this
institution in order to identify a suitable course. If the course title in- means we do not have a perfectly random sample. Secondly, while
dicated the class was introductory and solely an algorithms course “properly titled” introductory courses would always be surveyed,
(e.g., “Design of Algorithms,” “Analysis of Algorithms,” “Fundamen- programs that didn’t have such a course were only surveyed if they
tal Algorithms,” etc.), we included it. Otherwise, if there was no included a topic from “Algorithmic Strategies.” This criterion, how-
such course, we would include another course with “algorithm(s)” ever, may cause some confirmation-bias, since this pre-supposes
in the title so long as there was evidence that at least one topic that algorithms courses need to cover these strategies.
from the ACM’s “Algorithmic Strategies” was included. This condi- Another threat comes from the six adjunct respondents. While
tion was most often applied to “Data Structures and Algorithms” adjuncts likely know the course they are teaching very well, it is
courses, which were often a terminal algorithmic course. possible that they are less familiar with the general curriculum of
After creating lists of undergraduate algorithms courses, for each the institution as a whole. A survey by the TIAA Institute indicates
course, we compiled a list of up to three instructors who had taught around 23% of adjuncts have jobs outside of academia, and 26%
it within the past 2 years. To find the instructors for each course, we teach at multiple institutions [17]. As such, this could lead to certain
applied a variety of methods. The most common way was through information being omitted or misrepresented.
looking at publicly viewable course schedules which list instructors’ Finally, there are threats related to doing surveys. All of the
contact information, though other sources were employed, such data is given voluntarily by those who responded, which can bias
as course websites. After finalizing the list of emails, we sent an the sample. There can be errors when filling out surveys. For ours
invitation to participate in the survey to all instructors in that email particularly, we allowed the final two sections to be skipped if the
list. Each survey respondent was offered a $25 Amazon gift card instructor indicated they did not teach any topic in the section.
for completing the survey. It could be that in these cases, the topics are taught in different
locations at these universities. There is also the issue of sample size;
3.2 Survey Design we only received responses from 87 instructors.
When designing the survey, our goal was to gain insight into both
the course structure and the topics typically taught. The survey 4 RESULTS
contained two main parts. The first, consisting of three sections, Our initial list had 615 higher-education institutions. Of these, 495
focused on the organization of the course. These sections ask about had a 4-year computer-science adjacent program, and 373 of those
general course details, such as grading breakdowns, dedicated class had an applicable algorithms course. From this, we were able to
section times, programming assignments, written assignments, and send out emails to 411 instructors from 302 institutions whose
in-class assessments (exams and quizzes). contact information we were able to find. We were able to gather
One of our motivations behind these questions was to determine responses from 87 instructors across 79 institutions in 34 states.
emphasis on the theoretical components versus applied components 57 of these responses were from doctoral universities with 36 be-
of the courses. As such, we asked questions about both the content ing from an R1 university, 12 from an R2 university, and 9 from a
and number of assignments. doctoral/professional university as defined by the Basic Carnegie
Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada
Classification [13]. 14 of these responses were from master’s degree-
granting institutions, and 16 were from baccalaureate colleges.
These rates mirror the ACM 2013 curricula survey [8]. 70 of these
institutions used a semester system, seven used a quarter system,
and two classes had a term length of 12-13 weeks. As answered by
respondents, of the distinct courses, 70 were upper-division courses,
11 were lower-division, and 2 were indicated as both (by different
respondents teaching the same course).
4.1 Instructor Background
We found that over 90% of instructors were professors, comprising
31 assistant professors, 25 associate professors, and 24 full profes-
sors. 6 respondents were adjunct faculty, and one was a graduate
student teacher. We also asked about their involvement in teaching
in comparison to research. We found that only 11% of instructors
considered themselves more focused on research, with 54% being
more teaching-oriented. Additionally, we categorized the respon-
dents’ research areas into theoretical and non-theoretical1 . Of the
82 that did any amount of research, we classified 39 of them as
doing theoretic research.
Figure 1: Grading criteria for course categories
4.2 Course Structure
4.2.1 General Course Details. Participants were asked to describe
21% had assignments that involved plotting the algorithms with
the grading criteria of their courses. We provided categories for
data (eg: the time elapsed at various input sizes).
participation, programming assignments, in-class assessments (ex-
ams, quizzes, etc), and written homework assignments2 (Figure 1).
4.3 Course Topics
Among these categories, participation and programming assign-
ments typically accounted for the lowest portion of the total course In figure 2, we present graphs of all of the results from this part of
grade, with nearly 29% of respondents weighting programming the survey. We use the written part of this section to note what we
assignments at 0%, and over 55% of respondents weighting partic- believe are the highlights.
ipation at 0%. In-class assessments, in contrast, were usually the 4.3.1 Algorithmic Strategies. This first part of the Course Topics
highest portion of the course grade, with over 26% of instructors section deals with topics concerning algorithm design techniques.
noting that they weighted it at over 60% of the total grade. Of the 79 responses3 , we observe that nearly all of the responses
For the “other” category, responses varied. There were presenta- cover divide-and-conquer and greedy algorithms with one and four
tions, student-made class notes, digital quizzes, reading assignments responses respectively stating the topic is covered in a different
(with a flipped-classroom), take-home exams, and some logistical course. The same is true for dynamic programming, with only seven
complications due to different grading systems. responses stating they do not cover it. The other topics were taught
4.2.2 Evaluation of Student Mastery. Among the 81 instructors in between 24% and 58% of classes.
who gave non-programming homework assignments, the average 4.3.2 Fundamental Data Structures and Algorithms (DSA). Multiple
number of assignments was 8.1, with many indicating that they topics related to data structures, like binary search, numerical algo-
required proof writing, algorithm design, complexity analysis, and rithms, binary trees, hash tables, and sorting algorithms, are taught
algorithm tracing. Similarly, for the 85 instructors who gave in-class in a prerequisite course for 45 to 60% of responses. Moreover, graph
assessments, an average of around 4.3 were given, again with the topics, such as graph representations as well as breadth- and depth-
same five areas as the homework. first search, are taught in over 50% and over 60% of the courses
respectively. Notably, shortest-path and minimum-spanning tree
4.2.3 Programming Assignments. Of the 66 respondents (76%) with
algorithms were taught by over 80% of the respondents.
programming assignments, they assigned on average 5.2 through-
out the academic term. There was significant variation among 4.3.3 Proof Techniques. We split the proof techniques section into
which languages were accepted - Java (89%), Python (47%), C++ two main categories: basic logic (ie: notions of implication, equiva-
(36%), and C (13%). 13% accepted any language. 91% of instructors lence, converse, inverse), and mathematical proof techniques. Proof
indicated their assignments including having students implement techniques were taught in algorithms by only 11 instructors (13%),
algorithms, 82% had assignments where they design their own, and and only 1 taught elementary logic. Both topics were covered in pre-
vious courses by 47 respondents (54%), with 2 indicating only proofs
1 We considered theoretical to be any research involving algorithms, data structures, were taught in a perquisite, and 7 (8%) indicating that logic was
theory of computation, complexity, or other heavily mathematically-oriented fields.
2 One respondent did not use a weighted grading system. They instead had specific 3 Due to an early error in the survey, we did not receive responses from 8 instructors
requirements for each grade. They gave >60% to homework, quizzes, and participation for this section and the Fundamental Data Structures and Algorithms section.
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Michael Luu et al.
(a) Algorithmic Strategies (b) Fundamental Data Structures and Algorithms
(c) Basic Analysis (d) Proof Techniques
(e) Basic Automata Computability and Complexity (f) Advanced Data Structures Algorithms and Analysis
Figure 2: Distribution of where Sub-Topics are taught as defined by ACM Computer Science Curricula 2013 [8]
Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada
Theme Description
Q1
1 Difficulties regarding students’ insufficient background, par-
ticularly in these areas:
A) Mathematical and proof-writing skills
B) Programming and implementation skills
2 Difficulties in creating problems for in-class assessments and
homework assignments, especially those with answers that
cannot be easily found online
3 Difficulties with illustrating relevance and real-world applica-
tions for algorithms
4 Difficulties with a particular algorithms topic that students
struggled with
Figure 3: Breakdown of themes of 80 responses for Q1 5 Difficulties in having students adequately apply algorithm
(left)4 and 46 responses for Q2 (right) design techniques to new problems
Q2
6 More programming assignments and focus on implementation
previously taught but not proofs. Most other instructors indicated of algorithms
that it was taught in a required (but non-prerequisite) course. 7 Cover less topics, move topics to other courses (existing or
new), or replace topics with other topics
4.3.4 Basic Analysis. The basic analysis section of our survey con- 8 Ensure prerequisite courses cover necessary knowledge
tained several categories, such as asymptotic analysis and recur- 9 More emphasis on proofs and mathematical aspects
rence relations. The majority of respondents (over 50% or higher 10 Add lab/recitation section to the course for more chances to
for each topic) indicated all eight topics as being covered in their practice problems
own algorithms courses; around 10%-20% of respondents noted that Table 1: Themes in instructors’ difficulties with teaching
these basic analysis topics were covered in a prerequisite course. algorithms (Q1) and in changes instructors’ would like to
implement, assuming full control over course (Q2)
4.3.5 Basic Automata, Computability, and Complexity. Automata
and complexity theory was skipped by 52% of instructors surveyed.
However, the other 48% cover at least P vs NP, 45% cover NP-
completeness, and 10% also cover the halting problem. Only 2% to 4%
of instructors covered other complexity and theory of computation 5 ANALYSIS
topics, but those topics appeared elsewhere in the curriculum in an
elective, prerequisite, or required course. 5.1 Implications of Instructors’ Background
As previously noted, 39 respondents reported performing research
4.3.6 Advanced Data Structures, Algorithms, and Analysis. Topics in theoretical computer science or mathematical fields while 47
in advanced DSA and analysis were skipped by 47% of instruc- were involved in other kinds of research or none at all. For the
tors. Out of the remaining 53% of instructors, graph theoretical 37 of those 39 theory researchers who gave written homework
algorithms such as topological sort and finding strongly connected assignments, 33 (89%) indicated that they require students to write
components were common with 73% of instructors who answered proofs, all 37 required algorithm runtime analysis, 35 (95%) required
this section covering it (38% overall). Network flow is also fairly designing algorithms, and 22 (59%) required tracing algorithms. For
common, covered by 64% of this section’s respondents (34% overall). the 44 instructors involved in other fields of research who gave
Balanced tree-related algorithms were usually covered in prereq- homework, 28 (64%) required proofs, 39 (89%) required algorithm
uisite courses. Instructors who answered this section indicate that runtime analysis, 37 (84%) required algorithm design, and 35 (82%)
the remaining topics from this section are either typically covered required tracing algorithms.
by an elective course or not covered in the curriculum at all. Notably, those involved in theoretical research required proofs
more often and tracing algorithms less often than those not involved
4.4 Free Response Questions in theoretical research, suggesting some difference in emphasis on
In the final section of the survey, we asked respondents three op- the mathematics of algorithms versus the implementation. This is
tional free-response questions about difficulties encountered in also reflected in the in-class assessments, sporting similar trends to
teaching their course (Q1), changes they would like to implement the data regarding assignments. Although, it is significant to note
assuming they had full control over their course (Q2), and any other that only 61% and 49% of theory and non-theory researching instruc-
final thoughts they believed would be important for us to know. tors respectively expect students to write proofs in assessments.
We found commonalities among the responses and categorized Furthermore, the grading criteria for programming assignments,
them into common themes with Q1 and Q2 each having 5 different written assignments, and exams did not significantly differ between
themes (with 2 sub-themes for Q1), listed in Table 1. Percentage instructors involved in theoretical and non-theoretical research.
breakdowns of these themes are presented in Figure 3.
4 The stacked bar graph represents responses showing Theme 1 and subthemes 1A and section showcasing the percentage of responses that did not fall under 1A or 1B, often
1B (responses may have exhibited either or both as shown) with the remaining blue due to plainly stating that prerequisite knowledge was lacking.
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada Michael Luu et al.
5.2 Themes from Free Response Questions the topics). Given these observations, it is possible that the “typical”
When teaching algorithms, by far the most common issue instruc- DSA topics concern tree traversal and graph algorithms. Further-
tors reported facing was insufficient background on prerequisite more, 𝑂 (𝑛 log 𝑛) sorting algorithms are the only DSA topic that has
concepts, with 54 of the 80 (68%) responses to Q1 exhibiting this any moderate associations with any Algorithmic Strategic topics,
(Theme 1). 40 out of those 54 (74%) made references to students’ most notably having an Cramér’s V score of 0.34 with divide-and-
weak foundation in mathematics and proof-writing. Only nine re- conquer, suggesting 𝑂 (𝑛 log 𝑛) sorting algorithms may be covered
sponses (17%) expressed concern with a lack of programming skills to serve as examples for algorithmic paradigms. For the advanced
to properly implement algorithms (the final 10 responses (19%) not algorithms topics, they have moderate intra-associations and weak
elaborating on the prior knowledge students were lacking). inter-associations, possibly implying these topics are rarely covered
Another concern raised was the assessment of students’ algo- together. Balanced trees, graph-theoretic algorithms, and network
rithm design skills (Themes 4 and 7). Instructors frequently com- flow, which show up in a higher percent of algorithms/prerequisite
mented on the difficulty in creating assessments and assignments courses, have less association with the other advanced topics.
for students to apply their knowledge in designing algorithms. A We did not find any high associations between institution data
key difficulty is to come up with novel problems, unique from those and instructor’s research area and our survey variables, suggesting
discussed online. Another is that some students are unable to apply that algorithms course structure and topics are not particularly
algorithm design beyond problems encountered in class. dependent institution type or professor research background.
Respondents also noted the abstract nature inherent in learning
these topics as a struggle. Themes 5 and 6 illustrate this as in- 6 FUTURE WORK
structors note students’ struggles in recognizing the importance of A more focused survey could expand upon the themes we derived.
algorithms and their relevance with real-world applications. Some Other research could look for and apply solutions to these issues.
instructors listed particular algorithmic concepts that students Finally, conducting surveys in other areas can further increase our
had trouble with as well, most commonly dynamic programming, understanding of the current state of CS education.
greedy algorithms, and NP-completeness.
Of the 46 responses we collected for Q2, we witness two cat- 7 CONCLUSION
egories of themes: those expressing desires to add to the course
In this study, we found that while instructors often use the term
in some way, like adding more material, emphasizing particular
“algorithms” to refer to a specific course, the actual course being
topics, or providing more opportunities to apply what they learned
referenced has a large variation between academic institutions: it
(Themes 6, 9, 10) and those which seek to address the difficulty of
can be oriented around mathematics, programming, or somewhere
the course by reducing the amount of material covered or ensuring
in between. Even within these divisions, instructors often select
prerequisite courses cover the concepts needed (Themes 7 and 8).
very different subsets of topics. In addition, instructors regularly
feel students are unprepared for their courses and have a variety of
5.3 Associations Among Survey Variables and new directions they wish to take it - whether to focus on certain
Institution Data course topics or to make the class more or less mathematical.
We looked to see if there were any associations between the survey As educators and researchers, we need to take into account
variables and additional data we collected on the institutions we the diversity in how algorithms courses are taught. Future studies
surveyed5 . We also used our previous categorization of instructors’ should keep this in mind, and either make efforts to be applicable
research in algorithms or non-algorithms adjacent research as a to a variety of courses or be tailored to a specific version of the
variable. We obtained these associations by calculating the Cramér’s course. When interacting with new transfer students or graduate
V measure for each pair of variables6 . students, we should be careful not to make assumptions about the
We found that there are some moderate associations between topics covered in prior algorithms courses. Finally, as educators,
variables from the same sections of the survey. We first see this with we should make use of this heterogeneity to pull elements from
the variables from the “Evaluation of Student Mastery” sections other courses into our own to continue to improve and refine.
where question types on the homework and exams were correlated
with one another. Likewise, for the Fundamental DSA topic sec- ACKNOWLEDGEMENTS
tion, there are moderate associations between quadratic sorting We are extremely grateful to the 87 respondents who filled out the
algorithms, hash tables, binary trees, and heaps, having Cramér’s V survey. We are grateful to Michael Dillencourt for helpful com-
values in the range 0.18 and 0.55. On the other hand, graph represen- ments with early drafts of the survey. We would also like to thank
tations, depth- and breadth-first search, shortest-path algorithms, an anonymous reviewer from a different paper that one author pub-
and minimum spanning tree algorithms, the most common topics lished at CCSC-Southwest 2022, which helped to highlight the need
from the DSA section, have Cramér’s V measures less than 0.13 with for this work. Varun Nagaraj Rao was supported in part by NSF
the other topics (and associations between 0.13 and 0.78 amongst awards #1916153, #1956435, and #1943584. Matthew Ferland was
5 This includes the institution’s Basic Carnegie Classification, its research activity, and supported in part by Simons Investigator Award for fundamental
whether or not the institution was public/private and religious/nonsectarian. and curiosity-driven research and NSF grant CCF-1815254. We also
6 We note that these associations values do not indicate the kinds of relationships
thank UCI’s Academic Senate Council on Research, Computing and
the variables have and whether variables are positively or negatively correlated with
one another in one manner. We provide interpretations of these associations with the Libraries (CORCL) for the research fund that enabled us to provide
intention of yielding additional potential avenues of consideration and future study. gift cards as incentives for participants.
Survey Results of Introductory Undergraduate Algorithms Courses in the U.S. SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada
REFERENCES Computing Machinery, New York, NY, USA, 111–119. https://doi.org/10.1145/
[1] ACM. 2005. Computing Curricula 2005. https://www.acm.org/binaries/content/ 3291279.3339404
assets/education/curricula-recommendations/cc2005-march06final.pdf [11] Michael Shindler, Matt Ferland, Aaron Cote, and Olivera Grujic. 2020. Experience
[2] ACM. 2020. Computing Curricula 2020. https://www.acm.org/binaries/content/ report: preemptive final exams for computer science theory classes. Journal of
assets/education/curricula-recommendations/cc2020.pdf Computing Sciences in Colleges 35, 10 (2020), 9–14.
[3] Philip E Agre. 1999. Information technology in higher education: The “global [12] Michael Shindler, Natalia Pinpin, Mia Markovic, Frederick Reiber, Jee Hoon
academic village” and intellectual standardization. On the Horizon 7, 5 (1999), Kim, Giles Pierre Nunez Carlos, Mine Dogucu, Mark Hong, Michael Luu, Brian
8–11. Anderson, Aaron Cote, Matthew Ferland, Palak Jain, Tyler LaBonte, Leena Mathur,
[4] Mohammed F Farghally, Kyu Han Koh, Jeremy V Ernst, and Clifford A Shaffer. Ryan Moreno, and Ryan Sakuma. 2022. Student misconceptions of dynamic
2017. Towards a concept inventory for algorithm analysis topics. In Proceedings programming: a replication study. Computer Science Education 32, 3 (2022),
of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. 288–312.
ACM, Seattle, WA, USA, 207–212. [13] Lee S Shulman. 2001. The Carnegie classification of institutions of higher educa-
[5] Mohammed Fawzi Seddik Farghally. 2016. Visualizing algorithm analysis topics. tion.
Ph. D. Dissertation. Virginia Tech. [14] Cynthia Taylor, Michael Clancy, Kevin C Webb, Daniel Zingaro, Cynthia Lee,
[6] Kathi Fisler, Sorelle Friedler, Kevin Lin, and Suresh Venkatasubramanian. 2022. and Leo Porter. 2020. The practical details of building a CS concept inventory. In
Approaches for Weaving Responsible Computing into Data Structures and Algo- Proceedings of the 51st ACM Technical Symposium on Computer Science Education.
rithms Courses. In Proceedings of the 53rd ACM Technical Symposium on Computer ACM, Portland, OR, USA, 372–378.
Science Education V. 2. ACM, Providence, RI, USA, 1049–1050. [15] Pinar Mihci Türker and Ferhat Kadir Pala. 2020. The effect of algorithm education
[7] Matthew Hertz. 2010. What do "CS1" and "CS2" mean? Investigating differences on students’ computer programming self-efficacy perceptions and computational
in the early courses. In Proceedings of the 41st ACM technical symposium on thinking skills. International Journal of Computer Science Education in Schools 3,
Computer science education. ACM, Milwaukee, WI, USA, 199–203. 3 (2020), 19–32.
[8] The Joint Task Force on Computing Curricula Association for Computing Ma- [16] J Ángel Velázquez-Iturbide. 2019. Students’ Misconceptions of Optimization Algo-
chinery (ACM) IEEE Computer Society. 2013. Computer Science Curricula 2013. rithms. In Proceedings of the 2019 ACM Conference on Innovation and Technology
https://dl.acm.org/doi/pdf/10.1145/2534860 in Computer Science Education. ACM, Aberdeen, UK, 464–470.
[9] Nesrin Özdener. 2008. A comparison of the misconceptions about the time- [17] Paul Yakoboski. 2018. Adjunct faculty: Who they are and what is their experience.
efficiency of algorithms by various profiles of computer-programming students. [18] Jeffrey Young and Eric Walkingshaw. 2018. A domain analysis of data structure
Computers & Education 51, 3 (2008), 1094–1102. and algorithm explanations in the wild. In Proceedings of the 49th ACM Technical
[10] Leo Porter, Daniel Zingaro, Soohyun Nam Liao, Cynthia Taylor, Kevin C. Webb, Symposium on Computer Science Education. ACM, Baltimore, MD, USA, 870–875.
Cynthia Lee, and Michael Clancy. 2019. BDSI: A Validated Concept Inventory for [19] Shamama Zehra, Aishwarya Ramanathan, Larry Yueli Zhang, and Daniel Zingaro.
Basic Data Structures. In Proceedings of the 2019 ACM Conference on International 2018. Student misconceptions of dynamic programming. In Proceedings of the
Computing Education Research (Toronto ON, Canada) (ICER ’19). Association for 49th ACM technical symposium on Computer Science Education. ACM, Baltimore,
MD, USA, 556–561.