FUB evaluation of DynaLearn final software

Transcrição

1 Deliverable number: Deliverable title: FUB evaluation of final software Delivery date: Submission date: Leading beneficiary: Status: Dissemination level: Authors: 2011/10/31 Extension deadline 2012/01/ /06/03 University of Brasilia (FUB) Version 07 (final) PU (public) Paulo Salles, Adriano Souza, Gisele Morison Feltrini, Isabella Gontijo de Sá, Carlos Eduardo Marinelli, Pedro Assumpção Costa e Silva, Emília Braga, Vanessa Tunholi, Camila Pianca, Pedro de Podestà Uchoa. Project number: Project acronym: Project title: Starting date: Duration: Call identifier: Funding scheme: Engaging and informed tools for learning conceptual system knowledge February 1st, Months FP7-ICT Collaborative project (STREP)

2 Abstract This Deliverable describes 14 activities aiming at the evaluation of software. These activities involved 108 persons, including secondary school students, secondary school deaf students, hearing school students in acceleration class and experts in biodiversity conservation. Conceptual understanding, the use of Virtual Characters and Semantic Technology were addressed. Among the evaluators, two groups deserved special attention given their specific situations: the LTS (Long Term Studies) students, who had a broad view of, and gave their opinion in each activity, and the deaf students, because of their special needs. The results showed that was positively evaluated, being thus accepted. The results of conceptual modelling activities with LTS students, Hearing students and Deaf students demonstrate the capabilities to improve the students conceptual understanding. The students also felt very motivated to use the Hamster in their modelling activities. Grounding feature also proved to be a very good tool for understanding concepts. The results of Long Term Studies showed that the students improved their understanding of along the year. Their evaluation of the modelling elements showed that they believe that the qualitative models are a good way to represent concepts, to understand different phenomena because the language used is easy and can be learned without major problems, thus motivating them to use. The Deaf Students found it helpful to use the models to develop their understanding of the potential behaviour of systems. Learning Space 2 and 4 were the ones that contributed most for their understanding of concepts. Regarding identification of relevant information from a given text the students were not unanimous, but still the majority found it easy or very easy. Finally, 95,6% of all the evaluators considered that is a good or very good tool for educational purposes, and 82,9% agreed or totally agreed that modelling in opened up new ways for thinking about systems. These results confirm the potential of for making a difference in science education. Acknowledgements We would like to thank the participants in our evaluation activities, for dedicating their time and energy to our project. Thanks to the Secretary of Education of the Federal District Brasília, the Directors and the staff where our evaluation activities were carried out. Thanks also to the project colleagues, particularly those involved in WP7, for their support and valuable contributions during the evaluation design and results analyses. Internal reviewer Bert Bredeweg (UvA) Page 2 / 160

3 Document History Version Modification(s) Date Author(s) 01 Report Structure Adriano Souza, Paulo Salles 02 First draft Adriano Souza, Paulo Salles 03 Second draft Adriano Souza, Paulo Salles, Carlos Eduardo Marinelli, Isabella Gontijo. 04 Third draft Paulo Salles, Adriano Souza 05 Fourth draft Paulo Salles, Adriano Souza 06 Version sent to internal review Paulo Salles, Adriano Souza 07 Final version Paulo Salles, Adriano Souza Page 3 / 160

4 Contents Abstract 2 Acknowledgements 2 Internal reviewer 2 Document History 3 Contents 4 1. Introduction General framework for the evaluation activities Previous evaluation activities 8 2. Overview of all activities List of participants Distribution of participants in evaluation activities Overview on all the evaluation activities Environmental science topics explored in evaluation activities features, evaluation instruments and statistical analyses Methodology Special evaluation studies Evaluation based on Long Term Studies Evaluation activities with deaf students Evaluation activity with conservation biology experts Evaluation activity with pre-secondary school students (CED03) Evaluation activity with secondary school students (CEM01) Evaluation activities in Conceptual modelling Exploring Causality Activity Activity Activity Systems thinking and qualitative systems dynamics 23 Page 4 / 160

5 Activity Activity Modelling progression towards LS Activity Activity Activity Conservation biology experts Activity Evaluation activities with Teachable Agents Activity Activity Activity Activity Evaluation activity with Grounding Activity Results Results obtained on Conceptual modelling Causality influences and proportionalities Activity Activity Activity Systems thinking and qualitative systems dynamics Activity Activity Modelling progression in LS4, LS5 and LS Activity Activity Activity Conservation biology experts 39 Page 5 / 160

6 Activity Results obtained with the Teachable Agent Activity Activity Activity Activity Results obtained with Grounding Activity Results of special evaluation studies Results of evaluation based on long term studies Questionnaires obtained in the first and second semester of activities Results of evaluation with deaf students Questionnaires about modelling elements Discussion and concluding remarks 51 References 54 Appendix A: Activity 1 55 Appendix B: Activity 2 66 Appendix C: Activity 5 72 Appendix D: Activity 6 74 Appendix E: Activity 7 81 Appendix F: Activity Appendix G: Activity Appendix H: Activity Appendix I: Activity Appendix J: Activity Appendix K: Activity Appendix L: Activity Appendix M: Activity Page 6 / 160

7 Page 7 / 160

8 1. Introduction The work described in this Deliverable follows the approach established in Task 7.3 regarding the evaluation of ( final software: Based on the finished Curriculum and content models (WP6), each case study beneficiary will develop lessons and an evaluation plan focussing on several different topics in the curriculum (including at least some of the advanced models they have developed themselves). Each case study will focus on different aspects of the curriculum and features of the final software. Using the prepared lessons and evaluation plan, each beneficiary will evaluate the prototype and the curriculum in real educational settings. Accordingly, FUB has carried out 14 evaluation activities, focussing on conceptual modelling, teachable agents and grounding as well as using, in most of them, advanced models produced in Task 6.4, particularly models described in D6.4.1 (Salles et al. 2011). These activities involved 108 evaluators, including pre-college students, deaf students and experts in conservation biology General framework for the evaluation activities According to D7.1 (Mioduser et al. 2010), the basic elements for the evaluation activities should be organized in Lesson Plans. Consequently, Lesson Plans for all the activities described in this Deliverable were prepared and can be found in the appendices Previous evaluation activities An overview of the whole evaluation activities developed in Task 7.2 is presented in D In the first round of evaluation activities, described in D7.2.1 (Salles et al. 2010), FUB has produced 10 activities, exploring conceptual modelling features as well as topics about conservation biology, population biology and metapopulation. These studies captured evidence that qualitative reasoning models significantly improves the students learning of biological concepts and their capacity of making inferences involving biological concepts. Also it proved to be a very good motivating approach for learners. For the future, the orientation was to keep the evaluation activities and to focus on Conceptual Modelling activities specifically in activities approaching causality, systems thinking and modelling progression, Virtual Characters interactions activities and Semantic Technology features. This Deliverable presents the evaluation activities carried out by FUB as follows. Section 2 presents information about the participants of the evaluation activities, the type of activities, duration, number of Page 8 / 160

9 participants and the environmental science topics addressed in these evaluation activities. Section 3 presents the methodological aspects of each activity in a summarized way, with tables gathering the most relevant information. The results of each activity are discussed in Section 4. The discussion of the results and the concluding remarks are provided in the Section 5. Page 9 / 160

10 2. Overview of all activities An overview of the evaluation activities carried on by FUB during the third year of the project, based on the work developed in Task 7.2 and described in D7 2.1 (Salles et al., 2011) is presented in this chapter List of participants The following institutions participated in the evaluation activities organized by FUB: Centro de Ensino Médio 01 of Sobradinho (CEM 01): a public secondary school located at Sobradinho, a city nearby Brasília. Three groups of students participated actively in this evaluation effort: (a) 3 rd year students that participated in what was called Long Term Studies (LTS), a series of meetings along 2011 during which a number of activities were carried out, exploring different functionalities in ; (b) a group of 1 st year students (CEM01-J); and (c) a group of 1 st year students (CEM01-K). Centro Educacional 06 of Taguatinga (CED 06-T), a public secondary school located at Taguatinga, a city nearby Brasília, that receives a number of deaf students. Their Biology teacher has been a partner in research about using Qualitative Reasoning in science education of the deaf. Among these students, a group has participated in evaluation activities. Centro Educacional 03 of Sobradinho (CED 03-S), a public fundamental and secondary school also located in Sobradinho. Students in a special class the Acceleration class participated in FUB s evaluation activities. These students should have finished their fundamental studies, but given that they did not succeed in getting the required scores in some disciplines during the 9 th series, they receive support to catch up and enter the 1 st year of secondary school. Experts is a small but important group of researchers on Conservation Biology that are interested in using for promoting science education among managers and staff that work in legally protected areas. CEM 01 Sobradinho: (CEM01 LTS): 5 students, two of them being female and 3 being male, between years old; (CEM01 J): 27 students, 14 of them being females and 13 being males, between years old, as shown in Figure 1: Page 10 / 160

11 Figure 1. Age frequency of CEM01-J students (CEM01 K): 27 students, 14 of them being females and 13 being males, between years old, as shown in Figure 2: CED 03 Sobradinho: Figure 2. Age frequency of CEM01-K students. (CED03): 16 students, 8 of them being female and 8 being male; 15 of these students are 15 years old and one is 17. CED 06 Taguatinga: Experts: (CED06 Deaf) The group of deaf students consists of 29 students (17 females and 12 males) between 15 and 22 years old. This population is distributed in the 1 st year (15 students, 7 of them being females, and 8 being males), 2 nd year (11 students, 5 of them being females and 4 being males) and 3 rd year (5 students, all females). With respect to the level of deafness, they are classified as mild to moderate (45%) and severe to profound (55%). Although 52% are lip readers, only 38% assume they understand Portuguese. A large number of these students (79%) understand and communicate well in the Brazilian Sign Language (Libras). (Experts) The group includes a researcher from the Brazilian Agricultural Research Organization (Embrapa), two researchers and lecturers at University of Brasilia, and a researcher and PhD student at University of Brasilia. All of them are experienced researchers and have scientific interest in conservation biology. Page 11 / 160

12 The Table 1 presents more details about the expert evaluators. Table 1. Identification and work experience of expert evaluators. Expert # E-1 E-2 E-3 E-4 Identification Researcher and Lecturer, 36 years old Researcher, 29 years old Researcher and Lecturer, 37 years old Researcher, 35 years old Experience Sixteen years acting in the execution of studies and planning strategies for biodiversity conservation and environmental development of protected areas Participation in projects of biological inventory to define priority areas for conservation Researcher and teacher in the area of biodiversity Masters and doctorate in forest restoration. Working for five years as a researcher in restoration and conservation of genetic resource in situ Distribution of participants in evaluation activities Altogether, 14 activities were developed and involved 108 persons, summing up 19 sessions, with 58 hours duration. The table below (Table 2) presents the details: the number of activities, sessions and hours per type of participant: Table 2. Number of activities, sessions and hours according to groups of students. Students Number of activities Number of sessions Hours Long term students Hearing students Deaf students Experts Total Three of the key features were evaluated: Conceptual modelling, Virtual Characters, and Semantic Technology. Table 3 presents the groups of evaluators involved in these activitites. Table 3. Number of activities and participants by groups of evaluators in each feature evaluated. Students Conceptual Modelling (CM) Virtual Characters (TA) Semantic technology (ST) Activities Participants Activities Participants Activities Participants Long term students Hearing students Deaf students Experts Total Note: if a student participates in more than one activity (s)he is counted as 1 person in each activity. Page 12 / 160

13 2.3. Overview on all the evaluation activities The following table presents the full list of evaluation activities run by FUB, implementing Task 7.3: four features were evaluated, namely conceptual modelling (CM), teachable agent (TA), grounding and model debugging. These activities explored all Learning Spaces (LS) but not # 3. Table 4. Topics approached and learning spaces evaluated. feature CM TA Activity # Topic LS # Participants (#) Duration (h) 1 Causal aspects of 1,2,4 LTS (4) 6 Deforestation, habitation, energy 6 Causal aspects of 4 LTS (3) 3 Osmosis and diffusion 13 Systems Thinking on Water cycle 4 Deaf (27) 6 14 Systems Thinking on Water cycle 4 CEM01-J (27) 3 10 Water erosion 4 LTS (3) 3 11 Water erosion 5 LTS (3) 3 12 Algal bloom 6 LTS (3) 9 16 Environmentalists X Lumber dealers 2 CEM01-J (19) 3 18 Conservation Biology all experts (4) 4 2 Photosynthesis - Respiration 2 LTS (4) 3 7 Food chain 2 CED03 (16) 3 15 Environmentalists X Lumber dealers 2 Deaf (7) 6 17 Environmentalists X Lumber dealers 2 CEM01-K (27) 3 Grounding 5 Agriculture in the Cerrado 4 LTS (2) 3 TOTALS Note: if a student participates in more than one activity (s)he is counted as 1 person in each activity Environmental science topics explored in evaluation activities During these events of evaluation, the following topics (D6.1, Salles et al. 2009) and models on environmental science produced in Tasks 6.2 and 6.4 were used: Page 13 / 160

14 Table 5. Topics by activity, feature and learning space Topic Activity # DL Feature Learning Space Deforestation, Habitation and Energy 1 CM LS1 and LS4 Osmosis and diffusion 6 CM LS4 Hydrological erosion 10, 11 CM LS4 and LS5 Algal bloom 12 CM LS6 Water cycle 13, 14 CM LS4 Environmentalists vs lumber dealers 16 CM LS2 Environmentalists vs lumber dealers 15, 17 TA LS2 Photosynthesis and respiration 2 TA LS2 Food chain 7 TA LS2 Cerrado agriculture 5 Grounding LS features, evaluation instruments and statistical analyses Following D7.1 (Mioduser et al. 2009), evaluation activities were organized in lesson plans, after the definition of hypotheses, research questions and appropriated procedures and evaluation instruments. Lesson plans for all the evaluation activities are included in this document as Appendices. The following table summarizes the instruments used to collect information and opinions about learning by modelling with. Table 6. Evaluation instruments used in each evaluation activity developed by FUB in functionalities Activity # Evaluation instruments Conceptual Modelling (CM) 1,6, 10, 11, 12, 13, 14, 16, 18 Pre and Pos tests; Models; Concept maps; Exercises; Interviews; Specific questionnaires Teachable Agent (TA) 2, 7, 15, 17 Pre- and Pos-tests; Models; Interviews; TA Specific questionnaire Grounding 5 Pre- and Pos-tests; Reports; Interviews Pre and Post-tests have taken different formats: written questions, true-false statements, multiple choice questions, diagrams to be completed, causal models to be filled in with direct influences and proportionalities and/or with positive and negative signs, and combinations. As required in this type of Page 14 / 160

15 evaluation approach, these instruments were carefully designed and balanced. Appropriate statistical analyses were carried out, and are referred in every activity presented below. Models produced by the students were assessed by their structure (correct use of entities, configurations, dependencies, quantity and quantity spaces etc.), their expressiveness and correct results of the simulations (see Annexes). Two members of FUB group marked independently the scores to each model. In the case of divergences these were further discussed. Statistical analyses were carried out, and are referred to below, when needed. Interviews were also used, particularly with participants of the Long Term Studies group. These were semistructured conversations, in which some questions were pre-defined by the researchers, and the interviewed had freedom to address other issues of interest. Concept maps offer the learners the possibility of creating well-organized maps of their knowledge. LS1 provides a tool for building concept maps, and as such, to capture initial impressions of the system at study. Comparing concept maps produced by learners with standards prepared by the teacher may serve as evaluation tools, and scores can be used to measure progress towards models of better quality. Independent analysis of concept maps (see Appendix A) by two members of FUB s team was used in most of the evaluation activities. Exercises aiming at identifying entities, processes, quantities and further implementing models or exploring modelling techniques were used, and some of these exercises were used as evaluation tools. Reports were also taken for evaluation when modelling activities involved the development of specific modelling skills. Surveys are among the most important instruments to collect the opinion of evaluators. Two questionnaires were used: the Motivation- Attitude Questionnaire, completed by almost all the participants of evaluation activities, and variations of the Questionnaire about, slightly modified versions to be completed by deaf students, for example, or exploring details about the modelling language completed by the Long Term Studies group of students. Participants of activities with Teachable Agent were asked to complete a TA-specific questionnaire, in which learners were presented with opinions about the role of virtual characters and the way they are presented, their features and the question-answering interaction. All these types of questionnaires used Likert-scale questions, combined with open questions (see Appendices D, E, H and I). Statistical analyses In many evaluation activities a bootstrapping method was used for statistical analyses. This approach is very useful when the sample size is insufficient for straightforward statistical inference. Bootstrapping is a method of randomization with replacement that empowers the analyses. If the underlying distribution is well-known, bootstrapping provides a way to account for the distortions caused by the specific sample that may not be fully representative of the population. Otherwise bootstrapping is useful when powerful calculations have to be performed and a small pilot sample is available. It is known that most power and sample size calculations are heavily dependent on the standard deviation of the statistic of interest. An alternative method to get an impression of the variation of the statistic is to use a small pilot sample, perform bootstrapping on it and then calculate the variance (Adèr et al., 2008). In the evaluation activities presented here, the following procedure was adopted. First, collected data were tested for the normality by means of the Shapiro-Wilk normality test. If the data fits to a normal distribution, the paired t-test was used; if not, nonparametric bootstrapping methods running 1000 randomizations were used. This procedure was also used in analyses of concept maps. To approach models and diagrams, analyses Page 15 / 160

16 of variance were performed. Tests were run in R (R Development Core Team, 2010), at the significance level of 5%. Page 16 / 160

17 3. Methodology 3.1. Special evaluation studies This session will describe main considerations about special studies, those which deserve particular considerations such as a long term evaluation effort; special needs (deafness); low or high scholar performance; and domain expertise (conservation biology) Evaluation based on Long Term Studies This evaluation was performed with a small group (4 students in total) of secondary school students from CEM01 Sobradinho, Federal District, Brasilia. They had previous contact with software in evaluation activities developed at CEM01 School in The evaluation started in May 2011 and finished in December 2011, lasting 8 months. Because of the long duration of the evaluation activities with these students, compared with the time spent in evaluation activities with other groups, this group was named Long Term Studies group. Also the investigators created a close relationship to provide a better assistance and care during the activities and to have a broader view of the students feelings, difficulties and advantages of using Evaluation activities with deaf students The work with the deafs went beyond the regular evaluation activities. The teacher has a MSc degree and her dissertation includes the development of vocabulary to present qualitative reasoning models and simulations, and a tutorial about how to build models and related instructional materials. The material is organized on a DVD, which was used as a supporting material to introduce to teachers (see D7.2.1, Salles et al. 2010). In 2011, the biology teacher has prepared a series of activities to actually introduce into the deafs classroom. The students had classes from August to October 2011 twice a week, each session last 50 minutes in a total of 12 sessions at CED06 School and other 12h of activities (4 sessions of 3h each) in the Informatics Lab at University of Brasilia (UnB). The number of students participating in those activities was 31, from which 15 were from the first year of the secondary school, 11 were from the second year, and 5 were from the third year. The lessons were taught using Brazilian Sign Language, also known as "Libras" (from "Língua Brasileira de Sinais"). The advantage is that the biology teacher is also fluent in Libras and interpreter. In class the students built along with the teacher a model about algae bloom using LS2, after that other subject explored was Diffusion, they had to draw diagrams identifying areas with high and low substance concentration. Also the teacher made Diffusion models in in Learning Spaces 2, 3 and 4 using Page 17 / 160

18 Datashow and then they built their own model using the teacher model as reference. After those activities they answered a detailed questionnaire about attitude and motivation Evaluation activity with conservation biology experts A group of four experts in conservation biology was involved in the evaluation activity 18 and only in it. They are knowledgeable people, and their role was to evaluate aspects of the conceptual modelling workbench in order to assess the potential of this tool to be used by persons involved in the management of legally protected areas. The activity was held at University of Brasilia. A power point presentation gave them an overview of general aspects of the project and of the software. The experts had the opportunity to explore in LS1 and LS2, and build models in these LS. Given the objective of this activity was to collect their opinion about, no lesson plan was prepared and no hypothesis was tested. At the end of the activity they answered a specific questionnaire prepared for them Evaluation activity with pre-secondary school students (CED03) This group consists of students from CED03 School at Sobradinho, Brasilia, Federal District. They are enrolled in an acceleration class, that is, they had low performance and failed the final school year. They had then to attend the same class again the following year. However being enrolled in this special class they can catch up the class they are supposed to be according to their age. The activity with these students was performed in the Informatics Lab at CED03 School in a 3 hours session. They worked in pairs and the main subject explored was food chain using Teachable Agent. The students have never experienced the software, and it was a completely new approach for them Evaluation activity with secondary school students (CEM01) Other regular students from CEM01 School at Sobradinho also participated in three evaluation activities, one using Teachable Agent use mode in and the other 2 activities was used only the conceptual modelling features of. The first group of students (CEM01-J) was supervised by a biology teacher and they attended in two activities both conceptual modelling only. She selected the students with better performance and higher scores in the regular tests and exercises applied in the classroom to attend in the evaluation activities of. The second group (CEM01-K) was also supervised by a biology teacher. This group is characterized by students with average or low performance on the tests in the classroom. Students of this group attended in evaluation activity using Teachable Agent use mode. Page 18 / 160

19 Students of both groups are enrolled in the first year of the secondary school and they already had lessons addressing ecological topics such as water cycle. One of the topics approached using was the debate between environmentalists and lumber dealers regarding conservation versus economic issues Evaluation activities in Conceptual modelling Nine activities are related to the use of Conceptual modelling workbench to develop conceptual understanding about themes on environmental science, reasoning and modelling skills. The development of reasoning skills was very important for the educational activities reported in this Deliverable. Evaluation activities 1, 6, 10, 11, 12, 13, 14 and 16 are directly related to students on the following skills: S1- Understanding natural phenomena; S2- Mastering natural and modelling languages, and mastering the capability to translate among languages; S3- Identifying central and peripheral information, presented in different contexts (texts, literature, models). Comparing possible solutions for a problem; S4- Applying adequate methods for problem analysis, formulation of suitable solutions, selecting and implementing an optimal solution; S5- Formulating and articulating adequate and consistent argumentation. The following general hypotheses provide the basis for these evaluation activities: Modelling activities enhance conceptual understanding. Modelling activities develop in students the following abilities (reasoning skills): o o o Understand central relationships and interrelations; The ability to identify components of a system and processes within a system The ability to organize the system's components within a framework of relationships These general hypotheses resulted in two research questions, summarized as follows.. Does students involvement in modelling activities contribute to: A better understanding of the concepts explored in classroom and textbooks? Develop specific reasoning skills? Improve the students understanding of how to implement causal relations, particularly direct influences (Is) and qualitative proportionalities (Ps)? The specific hypotheses and research questions are presented in each activity. Page 19 / 160

20 Exploring Causality Three activities involving regular students (Activity 16) and long term study (LTS) students (Activities 1 and 6) are described in this section. The goal was to have a better view and to investigate how these students deal with causality while exploring modelling language and functionalities Activity 1 Activity Learning Content Domain Target Sample Evaluation Spaces Title Spaces Population Size Activity 1: Expressing causal LS01-LS02 and LS04 Deforestation; Habitation; and Secondary school students 04 1 CM 2 VC 3 ST relations; Working with Is Energy (CEM01-LTS) a 1, 2 and Ps CU b 3,4 SR/ M c M/A d SDL Evaluation Questions 1. Are the learners able to identify the key elements of the system (entities, quantities and processes)? 2. Are the learners able to identity processes, rates and to understand how concurrent processes shape system behaviour? 3. Are the learners able to differentiate processes (direct influences) and propagation the processes effects (proportionalities)? 4. What are the criteria for the learners to select the elements of the system to be included in the model (contour conditions)? Evaluation Design & Instruments O 1 X 1 X 2 O 3 X 3 O 4 X 4 O 5 O 6 Introduction lecture Pre-test: building of conceptual map about phosphorus cycle (Appendix A) Evolving Deforestation model Post Model 01 Complete Is and Ps Evolving Habitation model Post Model 02 Complete Is and Ps Evolving Energy model Post Model 03 Complete Is and Ps Post-test: building conceptual map about sulphur cycle (Appendix A) The lesson plan is available in the appendix A and provides an overview of the whole activity. Page 20 / 160