DOKK Library

Looking at Algorithm Visualization through the Eyes of Pre-service ICT Teachers

Authors Fatih Saltan

License CC-BY-4.0

Universal Journal of Educational Research 4(2): 403-408, 2016                                   
DOI: 10.13189/ujer.2016.040213

  Looking at Algorithm Visualization through the Eyes of
                Pre-service ICT Teachers
                                                         Fatih Saltan

                     Department of Computer Education & Instructional Technology, Amasya University, Turkey

Copyright © 2016 by authors, all rights reserved. Authors agree that this article remains permanently open access under the
terms of the Creative Commons Attribution License 4.0 International License

Abstract         The study investigated pre-service ICT           algorithm visualization (AV). AV is defined as the graphical
teachers’ perceptions of algorithm visualization (AV) with        illustration of algorithms and data structure via software
regard to appropriateness of teaching levels and contribution     developed for this purpose [13, 35]. Almost all educational
to learning and motivation. In order to achieve this aim, a       communities have a positive perception of AV [28]. AV is
qualitative case study was carried out. The participants          preferred in programming education to facilitate student
consisted of 218 pre-service ICT teachers from four different     understanding of the way computer algorithms function [14].
universities. Data were obtained through an open-ended            New strategies are needed in this field, because learning
questionnaire (n=210) and interviews (n=8). The qualitative       computer programming is not easy, even for college students
data were analyzed using content analysis techniques. The         enrolled in computer-related disciplines. College students
results indicated that about half of the pre-service teachers     who have basic programming knowledge perceive
thought AV to be appropriate for use in elementary and            programming courses as difficult, because these courses
middle schools. A smaller number of participants thought          require higher-order thinking skills [37]. Researchers have
that using AV is appropriate in high schools and colleges.        investigated several problems that may cause failure of some
Almost all of the participants thought that AV effectively        novice programming students and dropout of the
contributes to learning and teaching computer programming.        programming course [8]. Some of the major problems
Participants explained this effect in terms of seven properties   identified for college students are abstract concepts, lack of
of algorithm visualization environments: Visualization,           understanding of the larger entities, lack of applications, and
Algorithmic Thinking, Enjoyable Learning, and Progressive         lack of practical and concrete learning situations [22]. A
Learning, Learning by Doing, Game-based Learning, and             recently conducted study categorized the problems a novice
Scaffolding. Moreover, results showed that most of the            learner faces as “programming knowledge (56%),
pre-service ICT teachers believed that AV contributes             programming skills (17%), understanding semantics (13%),
effectively to motivation. They explained this positive           and debugging (13%)” [29]. Regarding these problems,
contribution to motivation in terms of six properties of AV:      computer science educators and instructional technologists
Easy to use, Visual, Fun, Quick Produced, Active and              have been studying technological and pedagogical ways to
Game-based.                                                       make programming instruction easier and more effective.
                                                                  Surely, it is critical to employ instructional time in the most
Keywords       Algorithm Visualization, ICT Teachers,             appropriate ways possible. In this regard, much algorithm
Teaching/Learning Strategies, Computer Programming                visualization software has been developed to facilitate
                                                                  learning and teaching, and especially to support novice
                                                                  programmers. One of the first examples of algorithm
                                                                  visualization software is the Brown Algorithm Simulator and
                                                                  Animator (BALSA) that assists students in learning the
1. Introduction                                                   fundamentals of computer programming through
   Teachers have a dual role as educators and innovators in       system-generated visualizations [3]. While some AVs can be
information societies. Until recent decades, it was generally     accessed for free, only a few of them maintain an educational
accepted that qualified teachers possessed content                quality [35]. It is also important to highlight that after the
knowledge and pedagogical knowledge, but today, it is             concept was debuted in the mid-1970s, more than 500
expected that good teachers learn emerging technologies and       algorithm visualization soft wares have been developed, but
utilize them in the classroom, along with content and             a comprehensive repository of AV software has yet to be
pedagogical knowledge [20]. In programming education,             established.
teachers have utilized a technology supported strategy called        In the 1990s, TANGO, software which allowed learners to
404                       Looking at Algorithm Visualization through the Eyes of Pre-service ICT Teachers

create visual algorithm animations by utilizing C language,        [32], the long term impact of algorithm visualization, like
was developed [36]. Pierson and Rodger [31] also created a         other innovations in education, depends on the degree to
Java based system, named JAWAA, which was a “simple                which teachers utilize them in practice.
command language for creating animations of data structures           It is important to know what teachers think about AV
and displaying them with a Web browser” (p.267). In the            specifically ICT teachers. This study attempts to address this
present decade, more interactive algorithm visualization           important need in the literature by investigating the
environments have been introduced. For example, Carnegie           perception of pre-service ICT teachers about the use of AV
Mellon University developed Alice, which enables novice            for teaching computer programming.
programmers to develop attractive three-dimensional
graphic animations [6]. Another popular algorithm                  1.1. The purpose of the study
visualization software is Scratch that developed by
Massachusetts Institute of Technology, where allows                   The purpose of this study was to investigate pre-service
learners to develop “interactive, media-rich projects” (p.1)       ICT teachers’ perceptions of algorithm visualization with
[24, 39, 41]. Scratch allowed learners to begin programming        regard to appropriateness of teaching levels and contribution
with creating animations instead of writing code that make         to learning and motivation. Within the scope of this purpose,
learners more motivated [39]. This nature of the Scratch           the following specific research questions were addressed:
provides creative developments and discovery learning [40].        1. What do pre-service ICT teachers think about the
Recent studies indicated that Scratch take learners interest             teaching levels for which AV use is suitable?
for programming education [42]. It allowed learners utilize        2. What are the views of pre-service ICT teachers
computational constructs [43] and improve programming                    regarding the contribution of AV to teaching and
skills [44]. Surely, it was helpful to development of positive           learning computer programming?
perceptions toward programming [45]. In her study Ke also          3. What are the views of pre-service ICT teachers
reported that Scratch is useful to make reflections about daily          regarding the contribution of AV to motivation for
experiences [46]. Like the Scratch, Greenfoot is a recently              learning programming?
developed algorithm visualization program that aims to teach
computer programming in a visual way to those who have no
prior programming experience, and it has gained great              2. Method
popularity [21]. App Inventor (AI) is a new algorithm
visualization platform that allowed learners create mobile            In order to understand pre-service ICT teachers’
applications for Android-based smart phones without writing        perception of algorithm visualization, this study employed a
code [47]. Alpha version of AI was launched in 2009[48].           qualitative study. This approach was chosen in order to gain
Graphical interface of AI is like Scratch [50]. Morelli and        comprehensive and in-depth information from the selected
Colleagues asserted that AI is easy to learn, accessible and       subjects [1, 30, 38]. In this study, the researcher focused on
lead students to problem solving instead of coding [47]. It is     perceived appropriateness of AV to teaching levels and
also indicated that AI is a highly motivating and powerful         perceived contribution of AV to learning and motivation. In
instructional tool [49].                                           order to ensure the trustworthiness of the findings, the
   Instructors have been using AV software for different           credibility of the study was addressed [23]. Triangulation, an
educational aims in different contexts. Generally they use         important technique to establish credibility, was utilized in
AV to maintain individual learning, give homework,                 this study. Data were gathered using two common
facilitate lectures and support laboratory assignments [15,        techniques: interview and questionnaire. Furthermore, coder
27]. In K-12 levels, Information and Communication                 reliability was calculated for analysis.
Technology (ICT) teachers utilize AV software to make it
easier for students to start learning computer programming.        2.1. Samples
Starting programming education at an early age has been
shown to foster mathematical and problem solving skills [5].          The participants in this study consisted of 214 pre-service
Another study, which investigated the contribution of              ICT teachers who were in their last year of study in the
specific AV software to students’ achievement, showed that         computer education and instructional technology
AV contributes to the development of mathematical and              departments at four different universities. Most of the
analytical thinking, problem solving and development of            participants (71%) graduated from a vocational high school.
logic [4]. Generally, studies on algorithm visualization have      The participants included 104 females and 110 males. Both
focused on the contributions of AV to learning, and students’      female (M=3.93) and male (M=4.03) pre-service teachers
motivation and perceptions. Few studies have investigated          had about four years of previous programming experience.
the perceptions of teachers about the usage of AV. Teachers        The pre-service teachers had all experienced using algorithm
are decision maker and key actors in instructional                 visualization to some degree. At a minimum, they had used
environments; their beliefs shape all instructional activities     one AV program like Scratch, Alice or Greenfoot.
[7, 11]. According to Robertson, Macvean, and Howland              Demographics of the participants are summarized in Table 1.
                                     Universal Journal of Educational Research 4(2): 403-408, 2016                                      405

       Table 1. Demographics of the Pre-service ICT teachers         In this regard, one pre-service ICT teacher said, “AV is
                 Vocational        Non-Vocational
                                                                     simple and suitable for young ages. It can improve learning
                 High School        High School                      and memory at those ages.”
    Female            72                  32               104          Likewise, pre-service ICT teachers who highlighted that
     Male             81                  29               110       AV is not useful for students in upper grades believe that AV
     Total           153                  61               214       is too simplistic and does not cover the important topics in
                                                                     computer programming. For example, one pre-service ICT
                                                                     teacher said,
2.2. Data Collection
                                                                        "It is unnecessary for us to use AV in college and high
   Data was gathered through a questionnaire and interviews.         school. The logic behind algorithms is an issue that can be
The questionnaire consisted of 4 demographic questions and           understood in one lesson. It will be better for us to learn more
5 open-ended questions. All pre-service teachers answered            helpful computer programs. At least, this software should not
the questionnaire voluntarily. Interviews consisting of              be use any more."
semi-structured questions were conducted with eight
                                                                              Table 2. The appropriate teaching grades for using AV
pre-service teachers, and the interviews were recorded via a
recording device.                                                              Themes/Categories                  Frequency (n=196)
                                                                          Elementary and middle school                 51(26%)
2.3. Data Analysis                                                             Only middle school                      32(16%)
                                                                             Only elementary school                    25(13%)
   The data were analyzed using content analysis techniques
                                                                                Only high school                       25(13%)
[30, 26]. In this study, the analysis was organized according
to the four stages defined by Yildirim and Simsek [38]. In the                  K12+ University                        25(13%)
first two stages, data were coded and themes were developed                  Middle and high school                    20(10%)
from these codes. In the last two stages, themes were                                 K12                              18(9%)
organized and described.
   First, the interviews were transcribed. The answers to the
                                                                     3.2. What Are the Views of Pre-service ICT Teachers
open-ended questions were coded using Nvivo8, a
                                                                          Regarding the Contribution of AV to Teaching and
qualitative data analysis software. Then, themes were
                                                                          Learning Computer Programming?
developed from the codes. In this process, the researcher
made sure that themes were internally consistent and distinct           Analysis of the interviews and the responses of the
from each other [25]. The themes were verified and                   pre-service ICT teachers to open-ended questions showed
confirmed by two independent researchers. Inter-coder                that all but fourteen participants (7%) thought that AV
reliability was calculated as 78 percent.                            contributes effectively to learning and teaching computer
                                                                     programming. For example one participant stated,
                                                                        “AV software is an easy and enjoyable program that every
3. Results                                                           teacher should learn and should teach to students. It uses a
                                                                     drag and drop approach, like Legos, and does not require
3.1. What Do Pre-service ICT Teachers Think about the                writing code. It is very good because it ensures that students
     Teaching Levels for Which Use of AV Is Suitable?                learn computer programming while using their imaginations
                                                                     to design a game-like application.”
   Pre-service ICT teachers were asked for which teaching
                                                                       Table 3. Reasons why pre-service ICT teachers perceive AV as useful
levels AV use is suitable and why. The data analysis
indicated that about half of the participants (55%) thought                  Themes/Categories                   Frequency(n=128)
that AV should be used in elementary or middle school.                          Visualization                        45 (35%)
While some of the pre-service ICT teachers (13%) thought
                                                                            Algorithmic Thinking                     34 (27%)
that AV was appropriate only for high school, some
                                                                             Enjoyable Learning                      20 (16%)
participants (10%) regarded AV as suitable for both middle
school and high school. Moreover, some of the participants                   Progressive Learning                     10(8%)
(9%) indicated that AV can be used in all grades, K-12. Also,                 Learning by Doing                        9(7%)
some participants (13%) regarded the use of AV to be                        Game-based Learning                        6(5%)
suitable in grades K-12 and college. The participants’                        Clues as Scaffold                        4(3%)
opinions are presented under the seven categories in Table 2.
When it was investigated why participants chose lower                  Results showed that pre-service ICT teachers regard AV
grades instead of high grades, content analysis showed that          as a useful instructional strategy for several reasons.
participants mainly prefer using AV in lower grades because          Resulting categories are listed in Table 3. On the other hand,
they believe that it is simple, visual and helpful for children.     participants who did not find AV useful for teaching and
406                       Looking at Algorithm Visualization through the Eyes of Pre-service ICT Teachers

learning highlight that AV is too simple for learning coding.                Table 4. Reasons why using AV motivates learners
One of them said,                                                          Themes/Categories                 Frequency(n=135)
   “It (AV software) does not make it easier to learn                        Simple & Easy                       41 (30%)
computer programming. On the contrary, it takes users to a
                                                                                 Visual                          29 (21%)
utopian world. AV software is not appropriate for the
seriousness of learning computer programming.”                                   Funny                           26 (19%)
   Many pre-service ICT teachers stated that AV enhances                   Quick Development                     21 (16%)
learning and teaching because it provides a visual learning           Interaction & Active Learning               12(9%)
environment. They stated that AV makes programming                               Game                              6(4%)
codes and structures tangible. Therefore, they perceived AV
to be most suitable for when students first start learning            About one in three participants noted that the reason using
programming. They also indicated that AV programs support          AV motives learners is because it is simple and easy to use
learning by providing clear and apprehensible interfaces. In       (30%). Pre-service teachers also indicated that using AV
this regard, one of them stated, “Yeah, easy. Programming          makes learning visual (21%) and fun (19%), so it generates
code is abstract, but students can see the structures and          necessary motivation for learners. For example, an interview
outputs in AV software. It is easier to learn.”                    participant said,
   Another theme that explains why pre-service ICT teachers           “It is being visual and working without writing code
regard AV as effective is algorithmic thinking (27%). Many         makes students more motivated. Therefore, AV encourages
pre-service ICT teachers stated that AV facilitates                learners to work on programming.” Another important
algorithmic thinking by letting users understand the logic         category that emerged as a part of the motivation cultivated
behind computer programs. Therefore, it contributes to             by AV was quick development (16%). Pre-service ICT
teaching and learning computer programming.                        teachers thought that learners feel satisfied when they
   It was also found that pre-service ICT teachers believe that    immediately see the results of what they do in AV programs.
AV contributes to learning programming because AV                  Thus, using AV drives them to continue learning. In this
ensures the learning process is fun. It was highlighted that       regard, one participant said,
using AV programs were enjoyable on the country of the                “I think AV is motivating. It is because we can
traditional instructional approaches. A participant indicated      immediately see results of code block we created using AV.
   “AV facilitates learning computer programming. It is            The resulting products (programs) let individuals to be more
because compare to other programming language, AV is               motivated and willing to progress create more complex
easier to understand and more enjoyable. AV software is            programs”.
more suitable and beneficial because it contains visual               Moreover, some of the ICT teachers emphasized
expressions and visual content which the students like.”           interaction with AV programs (9%) as a source of motivation
   Three other important themes emerged regarding the              for learners. The analysis showed that these teachers
instructional methods that AV employed. Some pre-service           believed that active involvement in AV environments
ICT teachers indicated that AV programs utilize progressive        successfully keeps learners’ attention. Lastly, a few
learning (8%). Learners can progress in stages with this           participants noted that the game-like nature (4%) of AV
approach. Participants also asserted that AV employs the           programs improves learners’ motivation. They highlighted
learning-by-doing method (7%). Learners are actively               that students, especially young students, want to play
involved when using AV.                                            computer games, so AV appeals to them.
   The third instructional method highlighted by pre-service
ICT teachers was game-based learning (5%). They indicated
that, like the other two methods, usage of game-method AV          4. Discussion
facilitates learning and teaching.
   Lastly, a few participants noted that AV programs provide          The goal of this study was to investigate pre-service ICT
clues as scaffolding (3%), which contributes to instruction.       teachers’ perceptions of algorithm visualization with regard
                                                                   to appropriateness of teaching levels and contribution to
3.3. What Are the Views of Pre-service ICT Teachers                learning and motivation. Data analysis revealed that about
     Regarding the Contribution of AV to Motivation for            half of the pre-service teachers thought that AV is
     Learning Programming?                                         appropriate for use in elementary and middle school. A
                                                                   smaller number of participants thought that using AV is
   Results showed that while most of the pre-service ICT           appropriate in high school and college. To date, there has not
teachers (87%) thought that AV contributes effectively to          been an extensive study showing in which teaching levels
motivation, some of them (13%) did not. Further analysis           AV is most preferred and in which teaching levels using AV
indicated that participants who believe using AV making a          is most effective. However, the findings of this study
positive contribution to motivation gave reasons that fell         correlate with the emphasis of “novice programmer” in
under six main categories. The resulting categories are listed     previous studies [e.g. 33]. In the literature, there is a
in Table 4.                                                        significant emphasis on learners who have just begun
                                   Universal Journal of Educational Research 4(2): 403-408, 2016                                     407

learning computer programming. It has been supposed that            [2]   Brennan, K., & Resnick, M. New frameworks for studying
novice programmers are the potential audience of AV-based                 and assessing the development of computational thinking,
                                                                          2012 annual meeting of the American Educational Research
instruction [24]. Although the main audience of the most of               Association, 1-25, 2012.
the popular AV software developers is K-12 students, some
developers also address college students and life-long              [3]   Brown, M. H., & Sedgewick, R. A system for algorithm
                                                                          animation, ACM SIGGRAPH Computer Graphics, Vol.18,
learners.                                                                 Issue.3, 177-186, 1984.
   The study also showed that almost all of the pre-service
ICT teachers thought that AV contributes effectively to             [4]   Calder, N. Using Scratch: An Integrated Problem-Solving
learning and teaching computer programming. Participants                  Approach to Mathematical Thinking, Australian Primary
                                                                          Mathematics Classroom, Vol.15, Issue.4, 9-14, 2010.
explained this effect with seven properties of algorithm
visualization environments: Visualization, Algorithmic              [5]   Çetin, E. The Effect of Computer Programming Education on
Thinking, Enjoyable Learning, and Progressive Learning,                   Children’s Problem-Solving Skills, (Unpublished master’s
                                                                          thesis), Gazi University, Ankara, 2012.
Learning by Doing, Game-based Learning, and Scaffolding.
   Similarly, a recent study showed that AV allows users,           [6]   Cooper, S., Dann, W., Pausch, R. Alice: a 3-D Tool for
especially novice programmers, to focus mainly on design                  Introductory Programming Concepts, Journal of Computing
and development rather than programming syntax [9]. In this               Sciences in Colleges, Vol.15, Issue.5, 107-116, 2000.
respect, another study found that using AV decreases the            [7]   Ertmer P. A. Teacher pedagogical beliefs: the final frontier in
cognitive load through “avoidance of the handling of the                  our quest for technology integration?, Educational
syntax errors” [17]. Moreover, Brennan and Resnick [2]                    Technology Research and Development, Vol.53, Issue.4,
indicate that designing interactive media in visual algorithm             25-39, 2005.
environments help learners develop computational thinking           [8]   Federici, S. A minimal, extensible, drag-and-drop
skills. On the other hand, several studies found little or no             implementation of the C programming language, 2011
evidence that AV is effective or efficient [e.g. 10,16]. In their         conference on Information technology education, 191-196,
comprehensive meta-analysis study, Hundhausen, Douglas
and Stasko [12], indicate that educators do not want to use         [9]   Grover, S., & Pea, R. Computational Thinking in K–12 A
AV, mainly because they do not believe using AV is as                     Review of the State of the Field, Educational Researcher,
effective and efficient as traditional approaches. However,               Vol.42, No.1, 38-43, 2013.
the present study showed that teachers have good reasons to         [10] Gurka, J., Citrin, W. Testing effectiveness of algorithm
utilize AV and most believe in its effectiveness.                        animation, IEEE Symposium on Visual Languages (VL’96).
   In addition, most of the pre-service ICT teachers thought             182–189, 1996.
that AV contributes effectively to motivation. They                 [11] Hammond L. D. Constructing 21st-century teacher education,
explained this positive contribution to motivation with six              Journal of Teacher Education, Vol.57, No.3, 300-314, 2006.
properties of AV which are Ease of Use, Visual, Fun, Quick          [12] Hundhausen, C. D. Toward effective algorithm visualization
Production, Active and Game-based Learning. In previous                  artifacts: designing for participation and communication in an
studies, the lively and popular aspect of AV software has                undergraduate algorithms course, (Unpublished Ph.D.
been clearly highlighted [12, 17, 36]. In the present study,             dissertation), Department of Computer and Information
several ICT teachers indicated that AV cannot contribute to              Science, University of Oregon, 1999.
learners’ motivation because it is too simple. However, most        [13] Hundhausen, C. D. Integrating algorithm visualization
ICT teachers saw its simplicity as a motivating factor. Most             technology into an undergraduate algorithms course:
novice learners want to see an immediate outcome. The                    ethnographic studies of a social constructivist approach,
simple development environment of AV provides this                       Computers & Education, Vol.39, Issue.3, 237-260, 2002.
satisfaction and builds confidence. Both satisfaction and           [14] Hundhausen, C. D., & Brown, J. L. Designing, visualizing,
confidence are major aspects of the Kellers’ ARCS                        and discussing algorithms within a CS 1 studio experience:
motivation theory [17]. Moreover, the attention of the                   An empirical study, Computers & Education, Vol.50, Issue.1,
learners, another aspect of the ARCS theory, is gained with              301-326, 2008.
the visualization and interaction in AV programs.                   [15] Hundhausen, C. D., Douglas, S. A., & Stasko, J. T. A
   Overall, this study showed that pre-service ICT teachers              meta-study of algorithm visualization effectiveness, Journal
thoroughly evaluate the usage of AV in teaching computer                 of Visual Languages and Computing, Vol.13, Issue.3,
                                                                         259-290, 2002.
programming. They indicated the contributions and
appropriateness to teaching level of AV.                            [16] Jarc, D. J., Feldman, M. B., and Heller, R. S. Assessing the
                                                                         benefits of interactive prediction using Web-based algorithm
                                                                         animation courseware, 31st SIGCSE Technical Symposium
                                                                         on Computer Science Education (SIGCSE’00), 377–381,

REFERENCES                                                          [17] Kelleher, C., & Pausch, R. Lowering the barriers to
                                                                         programming: A taxonomy of programming environments
[1]   Bogdan, R. & Biklen, S. K. Qualitative Research for                and languages for novice programmers, ACM Computing
      Education, Allyn & Bacon, Boston, 1992.                            Surveys (CSUR), Vol.37, Issue.2, 83-137, 2005.
408                        Looking at Algorithm Visualization through the Eyes of Pre-service ICT Teachers

[18] Kelleher, C., Pausch, R., & Kiesler, S. Storytelling Alice            structure visualization system for computer science education,
     motivates middle school girls to learn computer programming,          ACM SIGCSE Bulletin, Vol. 28, No. 1, 140-144, 1996.
     SIGCHI conference on Human factors in computing systems,
     1455-1464, 2007.                                                 [36] Stasko, J. T. Using student-built animations as learning aids,
                                                                           ACM Technical Symposium on Computer Science
[19] Keller, J. M. Development and use of the ARCS model of                Education, 25-29, 1997.
     motivational design, Journal of Instructional Development,
     Vol. 10, Issue. 3, 2-10, 1987.                                   [37] Tan, P. H., Ting, C. Y., & Ling, S. W. Learning difficulties in
                                                                           programming courses: undergraduates' perspective and
[20] Koehler, M., & Mishra, P. What is technological pedagogical           perception, International Conference on Computer
     content knowledge (TPACK)?, Contemporary issues in                    Technology and Development 2009 (ICCTD'09), 42-46,
     technology and teacher education, Vol.9, No.1, 60-70, 2009.           2009.
[21] Kölling, M. The greenfoot programming environment, ACM           [38] Yıldırım, A., & Şimşek, H. Sosyal Bilimlerde Nitel Araştırma
     Transactions on Computing Education (TOCE), Vol.10,                   Yöntemleri, Seçkin Yayıncılık, Ankara, 2006.
     Issue.4, 14, 2010.
                                                                      [39] Ouahbi, I., Kaddari, F., Darhmaoui, H., Elachqar, A., &
[22] Lahtinen, E., Ala-Mutka, K., & Järvinen, H. M. A study of the         Lahmine, S. Learning Basic Programming Concepts by
     difficulties of novice programmers, ACM SIGCSE Bulletin,              Creating Games with Scratch Programming Environment.
     Vol. 37, Issue. 3, 14-18, 2005.                                       Procedia-Social and Behavioral Sciences, 191, 1479-1482,
[23] Lincoln, Y. S., & Guba, E. G. Naturalistic inquiry, CA: Sage,
     Beverly Hills, 1985.                                             [40] Kobsiripat, W. Effects of the Media to Promote the Scratch
[24] Maloney, J., Resnick, M., Rusk, N., Silverman, B., &                  Programming Capabilities Creativity of Elementary School
     Eastmond, E. The scratch programming language and                     Students. Procedia-Social and Behavioral Sciences, 174,
     environment, ACM Transactions on Computing Education                  227-232, 2015.
     (TOCE), Vol. 10, Issue. 4, 16, 2010.                             [41] Brennan, K. Best of Both Worlds: Issues of Structure and
[25] Marshall, C., & Rossman, G. B. Designing Qualitative                  Agency in Computational Creation, In and Out of School.
     Research (4th ed.), Sage, Thousand Oaks, 2006.                        Ph.D. Dissertation. MIT Media Lab, 2012.

[26] Miles, M. B., & Huberman, A. M. Qualitative Data Analysis:       [42] Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N.,
     An Expanded Sourcebook (2nd ed.), Sage, California, 1994.             Eastmond, E., Brennan, K., Millner, A., Rosenbaum, J., S.,
                                                                           Silverman, B., & Kafai, Y. Scratch: programming for all.
[27] Naps, T. L. Algorithm visualization in computer science               Communications of the ACM, 52(11), 60-67, 2009.
     laboratories, ACM SIGCSE Bulletin, Vol.22, Issue.1,
     105-110, 1990.                                                   [43] Meerbaum-Salant, O., Armoni, M., & Ben-Ari, M. Learning
                                                                           computer science concepts with scratch. Computer Science
[28] Naps, T., Rößling, G., Almstrum, V., Dann, W., Fleischer, R.,         Education, 23(3), 239-264, 2013.
     Hundhausen, C., Korhonen, A., Malmi, L., Mcnally, M.,
     Rodger, S., and Ángel Velázquez-Iturbide J. Exploring the        [44] Begosso, L. C., & da Silva, P. R. Teaching computer
     role of visualization and engagement in computer science              programming: A practical review. In 2013 IEEE Frontiers in
     education, Working Group Reports from ITiCSE on                       Education Conference (FIE) (pp. 508-510). IEEE, 2013.
     Innovation and Technology in Computer Science Education
     (ITiCSE-WGR’02), 131-152, 2002.                                  [45] Genç, Z., & Karakuş, S. Tasarımla Öğrenme: Eğitsel
                                                                           Bilgisayar Oyunları Tasarımında Scratch Kullanımı. 5th
[29] Özmen, B., Altun, A. Undergraduate Students' Experiences in           International Computer & Instructional Technologies
     Programming: Difficulties and Obstacles. Turkish Online               Symposium, 2012.
     Journal of Qualitative Inquiry, Vol. 5, Issue. 3, 9-27, 2014.
                                                                      [46] Ke, F. An implementation of design-based learning through
[30] Patton, M. Q. Qualitative Research and Evaluation Methods             creating educational computer games: A case study on
     (3th ed), Thousand Oaks, CA: Sage, 2002.                              mathematics learning during design and computing.
[31] Pierson, W. C. & Rodger, S. H. Web-based animation of data            Computers & Education, 73, 26-39, 2014.
     structures using JAWAA, twenty-ninth SIGCSE technical            [47] Morelli, R., de Lanerolle, T., Lake, P., Limardo, N., Tamotsu,
     symposiums on Computer science education, 267–271, 1998.              E., & Uche, C. Can android app inventor bring computational
[32] Robertson, J., Macvean, A., & Howland, K. Robust                      thinking to k-12. In Proc. 42nd ACM technical symposium on
     evaluation for a maturing field: the train the teacher method,        Computer science education (SIGCSE'11), 2011.
     International Journal of Child-Computer Interaction, Vol. 1,     [48] H. Abelson. App Inventor for Android. Google Research
     Issue. 2, 50-60, 2013.                                                Blog, 
[33] Robins, A., Rountree, J., & Rountree, N. Learning and                 app-inventor-for-android.html, July 31, 2009.
     teaching programming: A review and discussion, Computer
                                                                      [49] Wagner, A., Gray, J., Corley, J., and Wolber, D., Using App
     Science Education, Vol. 13, Issue. 2, 137-172, 2003.
                                                                           Inventor in a K - 12 Summer Camp, SIGCSE '13 , 621 – 626,
[34] Shaffer, C. A., Cooper, M. L., Alon, A. J. D., Akbar, M.,             2013.
     Stewart, M., Ponce, S., & Edwards, S. H. Algorithm
     visualization: The state of the field, ACM Transactions on       [50] Resnick, M., Maloney, J., Monroy - Hernandez, A., Rusk, N.,
     Computing Education (TOCE), Vol. 10, Issue. 3, 9, 2010.               Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E.,
                                                                           Silver, J., Silverman, B., and Kafai, Y., S cratch:
[35] Shaffer, C. A., Heath, L. S., & Yang, J. Using the Swan data          Programming for All, CACM , 52, (11), 60 - 67, 2009.