openmodeller A framework for species modeling Fapesp process: 04/ Partial Report #3 (April 2007 March 2008)

Transcrição

1 openmodeller A framework for species modeling Fapesp process: 04/ Partial Report #3 (April 2007 March 2008)

2

3 Index INTRODUCTION...1 OBJECTIVES...1 SUMMARY OF MAIN ACTIVITIES AND RESULTS...2 Locality data component...2 Locality data server...2 Clients (drivers) that read locality data...3 Data cleaning...3 Environmental data component...4 Pre-analysis component...5 Modeling component...5 Post-analysis component...6 Desktop interface...7 Web interface...7 Integration with TerraLib...8 Evaluation of parallelization techniques and analysis of parallel algorithms for biodiversity data analysis...9 Initial implementation using services architecture...10 Other developments...10 Model repository...10 Study Cases...10 Seminars...12 Training...12 Publications...14 Journal papers...14 Conference papers...15 Other papers...16 In preparation...16 GENERAL COMMENTS...17 ANNEX 1. CHANGELOG...19 ANNEX 2. ACTIVITIES RELATED TO THE SPECIESLINK NETWORK...21 Centralized data repository...21 Centralized query...24 Sensitive data...27 Future developments...28 ANNEX 3. REPORT ON A TOOL TO ANALYZE DATA ON SPECIES DISTRIBUTION USING PARALLEL PROCESSING...29

4 ANNEX 4. PARALLEL VERSIONS OF THE PROJECTION FOR COMPUTER CLUSTERS...45 ANNEX 5. MANUAL FOR THE INSTALLATION OF A SERVICES PLATFORM...47

5 1 Introduction The goal of this project is to develop a framework to facilitate the work of scientists in predictive modeling of species distribution. The four year project funded by Fapesp involves three institutions: CRIA (Centro de Referência em Informação Ambiental), Poli (Escola Politécnica da USP), and INPE (Instituto Nacional de Pesquisas Espaciais). This report summarizes the activities carried out during the project s third year, from April 2007 to March Objectives This project s original aim is to develop a flexible and robust modeling framework to predict species distribution. Main objectives are: Develop a component-based modeling framework with reusable modules compliant with web services technology Enable use and comparison of different modeling algorithms through the same framework Enable modeling pre-analysis and pos-analysis using specific components Develop multiple interfaces (web, desktop, command line, web service) Facilitate access to distributed biological and environmental data networks Allow usage of high performance computing with species distribution modeling. Carry out use cases to test and validate the framework. Figure 1 illustrates the original proposal. The project addressed the following components: locality data, environmental data, pre-analysis, modeling, post analysis, and interfaces. Study cases testing the framework and training are other important activities of this project. Original architecture USER Interface (desktop, web, console, soap server, swig wrapper, others) data cleaning georeferencing Modeling Pre-analysis Post-analysis Locality component Algorithms Environmental component Specimen data (occurrence points) Environmental data (layers) GBIF specieslink GEOSS Figure 1. Proposed modeling framework architecture

6 Summary of main activities and results During the third year of the project there were 8 doctoral students, 3 master students, 6 undergraduate students and 6 fellowships for technical training directly involved with the project, besides the direct involvement of staff members from each institution (CRIA, Poli, and INPE). There were five new releases of the openmodeller framework since March 2007 (the complete changelog can be found in annex 1). Each release included bug fixes and new features developed during the project. New features included new algorithms (Support Vector Machines and Envelope Score), a new infrastructure for unit testing, ROC curve to measure algorithm performance, jackknifing to help choosing environment layers in modeling experiments, and a new unified compiling strategy for all platforms. Now both the library and the desktop interface have packages for all three major platforms: GNU/Linux, Mac OSX and Windows. The latest version of openmodeller Desktop (1.0.6) is getting an average of 600 downloads per month, which confirms the expectation in the beginning of the project that it would become the most popular interface of the library. The computer cluster was purchased, installed and is available to researchers. Different user interfaces are being developed to allow access in a transparent and friendly way. Research on parallelizing algorithms such as GARP has been conducted and a first version of this algorithm is available for testing. Soon it will be available to end users. The number of publications since the last report includes 7 journal papers, 13 conference papers among others that are in preparation. The specieslink network 1, which constitutes the main source of species occurrence data in the Brazilian territory and whose project originated openmodeller, almost doubled the number of records freely and openly available on-line, reaching approximately 2.3 million records of voucher specimens in February There were also two new releases of TerraLib and TerraView that are already compatible with the latest version of the openmodeller framework. The publicity gained from our regular software releases and interactions with other individuals and institutions contributed to the use of openmodeller by other interfaces and tools, such as the GBIF portal. The number of international publications that mention openmodeller is continuously increasing 2. The number of potential areas for future collaboration with the wider scientific community now includes institutions such as NASA. 2 Locality data component Locality data server The fact that CRIA is also responsible for the development of the specieslink network 3, which today represents the main source of on-line species occurrence data in Brazil, is an opportunity to develop a locality data service for openmodeller. The development of the network is also crucial for study cases

7 For this purpose, this last year specieslink changed its architecture from using an online dynamic distributed search system to a centralized search approach (see annex 2). A system that periodically searches the network and harvests all new and modified records and stores them in a central database was designed and developed. This has improved performance and makes it easier to serve all specieslink records for other applications as there is now a single endpoint. The newest DarwinCore 4 version was adopted to map the specieslink central database fields. The next step will be to install a TAPIR 5 compatible data provider software to serve this data to openmodeller. This development was co-funded by the JRS Biodiversity Foundation. Clients (drivers) that read locality data The openmodeller locality component involves a set of generic clients (drivers) that can read locality data from different sources using different formats. Currently there are drivers for TAB-delimited text files and Terralib databases. To access the specieslink data service, a new driver is being developed to provide direct access to any TAPIR/DarwinCore provider. Data cleaning Ecological niche modeling requires high quality data, so within the openmodeller framework, the idea is to develop an application where users may test data quality. During the first two years of the project, work was concentrated in developing a more universal application (data tester). The idea originated in a project that was carried out in collaboration with GBIF to design and develop an extensible Java framework for performing tests against XML data sets and reporting on data quality. At the same time, the development of data-cleaning applications is a continuous study for the specieslink network. CRIA s staff has been developing tools to find data inconsistencies and typing errors and producing on-line reports for each data provider (biological collections) for the last 5 years. This field of research has proved to be very dynamic, and through the constant interaction with collections (the specieslink network today has more then 130 collection) new features are frequently added. What the team was basically doing was developing applications for the specieslink network, that is the main source of species occurrence data on the Internet for Brazil, and then rewriting the tests to be used by data tester. This has not proved to be efficient so we have changed our strategy. Last year, work was concentrated in making the specieslink network more efficient through a centralized data repository and in adding new features to the data cleaning applications. Plans are to develop a Web interface for users to be able to submit data sets and retrieve reports with data quality evaluations. The following data fields are currently being checked: Names Dates Locality Consistency between Collector, collector number and name between herbaria

8 4 Brazil does not have a validated list of species names. The data cleaning tools check names with the Catalogue of Life 6 but in a recent study we estimate that less then 5% of the approximately 204 thousand unique names that are part of the specieslink network are included in the Catalogue of Life. For this reason a phonetic database of names of the network was created to help find typing errors. When a name is phonetically equal but written differently it is presented as a suspect record. The data cleaning report returns the following information: Name (family, genus, species, subspecies), Number of records written this way in the collection, Number of records written this way in the network, Status of this name in the Catalogue of Life, and Catalogue number. Geographic error detection includes checking whether given coordinates are consistent with the administrative regions provided by the original records, such as whether the values registered for latitude/longitude fall within the municipality registered. The system also checks whether data on country state municipality are coherent. Information as to records that have values of latitude or longitude equal to zero and values that are points in the sea are also shown. If the collection is of terrestrial species, then all coordinates that occur in the sea are errors. Another tool to detect possible errors is checking for outliers on latitude and longitude using a statistical method based on reverse-jackknifing procedure. Additionally, the system calculates latitude/longitude values to nongeoreferenced records based on the official coordinates of municipalities in Brazil. The system also checks the date of the record. Each collection informs the date of their oldest record, so the system compares this date with the date informed for each record and also compares the date of the record with the date it was last updated. Dates prior to the oldest date informed by the collection, and record dates that are more recent then the date the record was updated are presented as suspect records. The last feature is of particular interest to herbaria that commonly exchange duplicate exsiccates. Common fields for duplicates are the collector s name and number. The system assumes that if different records from different herbaria have the same collector s name and number they are duplicates and therefore should have the same species name. If not, they are presented as suspect records. The results are presented as a table where duplicate records are shown with the different name and the name of the specialist responsible for its identification and the identification date. This way, herbaria can benefit from specialists visits to different herbaria. Environmental data component In the future, we envision a system that will serve environmental layers as a service, such as GBIF and the specieslink network are serving species occurrence points. Such a system is under discussion 7 and a new protocol called Web Coverage Server (WCS) is already available for this purpose. New versions of GDAL can optionally access remote rasters through WCS. The corresponding driver was compiled, tested and seemed to work properly. Since GDAL can be used by openmodeller, this increases the number of environmental data sources that can be accessed in a distributed data scenario. Initial tests were limited by the small number of WCS see GEO and GEOSS at

9 servers available. To overcome this limitation and gain more knowledge and experience with WCS, a new WCS server is being developed on top of TerraLib. CRIA is also investigating ways to manage the large number of environmental layers accumulated over the years. Each layer represents a different variable and is stored as an individual file (or directory) in its own raster format. The first attempt was to create a metadata file containing keywords for each layer. A data harvester program was then developed to read all metadata files and store this information in a relational database so that queries could be performed. A prototype web interface was also developed to display all keywords as tag clouds with incremental filters being applied after each keyword selection. This approach is still being evaluated and may be eventually used in the openmodeller Web interface. 5 Pre-analysis component A jackknife-based procedure is the first technique implemented in openmodeller to help in identifying the most relevant environment layers that can be used in each modeling experiment. Jackknife is currently implemented as a separate class and it expects an algorithm as a parameter. The specified algorithm will generate a model for different subsets of layers (each time excluding one of the layers from the complete set) and the results are tested with an independent set of points. Jackknife calculates bias and variance estimates of statistical parameters (we are currently using model accuracy). In addition, it measures the contribution of each variable (environment layer) for the chosen parameter estimate (model accuracy). This method is still being tested in order to find the best openmodeller algorithm(s) that can be used for this purpose. A second technique implemented in openmodeller is the Chi Square method described at Li, L., et al. (2006) "An Integrated Bayesian Modelling Approach for Predicting Mosquito Larval Habitats". This method also expects a set of layers as input. For each pair of layers a contingence matrix is built. This matrix records the number of occurrences in both layers, considering them categorized in classes. The Chi-square test is applied with the significance level of The algorithm returns, for each variable, the number of layers with a correlation at the significance level of A generic API to interact in the same way with different pre-analysis techniques involving input layers is still being studied and discussed. The implementation is planned for the final year of the project. Modeling component openmodeller changed its compilation system to use CMake 8 so that the same build system could be used across all platforms. This change allows openmodeller to be more easily deployed in the three major platforms: GNU/Linux, Mac OS X and Windows. The adoption of a CMake build system for all platforms provides a significant reduction in maintenance, complexity of the code base, and duplication of effort. Additionally, for GNU/Linux two packages are now being generated in each release using the two main formats: Red Hat Package Manager (rpm) and Debian (deb). Considering the frequency in which changes are made to the library by different developers, one of the efforts during the last year was to develop a new infrastructure 8 Cross Platform Make:

10 for unit testing. Unit tests provide a way to check that specific parts of the source code are working as expected. Several C++ frameworks for unit tests were investigated before deciding to use CxxTest 9 for the openmodeller library. This process was documented in the openmodeller Wiki 10. A set of unit tests were developed with the chosen framework, although they still don t cover the entire library. Now that we have documentation about how to write, compile and run unit tests, the plan is to gradually increase the number of tests and to follow a test-driven development strategy for future work on the library. Two new algorithms are available in openmodeller: Support Vector Machines (SVM) and Envelope Score. SVM is a machine learning technique based on concepts from the statistical learning theory. SVMs have been used in many different applications that involve pattern recognition. Recently, SVM has also been applied to the problem of creating species distribution models, but there is still considerable scope for further research. SVMs are known for their good generalization ability and robustness to high dimensional datasets. The openmodeller implementation of SVM was carried out in a partnership with the Instituto de Ciências Matemáticas e de Computação 11 of the University of São Paulo. Envelope Score is a lax bioclimatic envelope algorithm where the probability of presence in a specific point is proportional to the number of environment layers whose envelope (min-max range calculated in model creation) contains the corresponding environment value for the point. The primary motivation for implementing Envelope Score was to generate more meaningful models in paleoclimate scenarios, where the traditional Bioclim algorithm produces overly constrained models. This work was done in collaboration with Chris Yesson from the University of Reading. The Aquamaps algorithm already mentioned in the last report (implemented in collaboration with the Incofish 12 project) is currently being updated to incorporate recent changes made by the authors. After validation, this new version should be officially released as part of openmodeller to become an alternative for modeling marine organisms. Another change in existing algorithms was the inclusion of the Chebyshev metric in Environmental Distance. Another two algorithms are planned for the final year of this project. The first will be based on Maximum Entropy (MaxEnt) techniques. Maximum entropy has been successfully applied in many different areas including species distribution models. The second new algorithm will be based on Neural Networks using Multilayer Perceptron with Backpropagation. When available, both algorithms will be compared with the existing ones. 6 Post-analysis component The Receiver Operating Characteristic (ROC) curve and its associated measure (AUC - area under the curve) were implemented as part of the openmodeller library. ROC curve data and AUC are now automatically calculated after model creation using command-line tools, Web Service calls, and the Desktop interface (which includes a graphical display of the curve). When no absence points are used, openmodeller generates background points to calculate the curve. Besides automatic calculation for training data, the same class can be used for any extrinsic tests for more about Incofish see

11 7 A new method for model validation should be incorporated into the Web Services interface during the last year of the project. This method will allow any number of external model validations using any of the available methods (AUC, accuracy or others) to be performed remotely. Desktop interface Two additional releases (1.0.5 & 1.0.6) were made and a third is planned. The current generation has numerous improvements, including: The running of experiments is now threaded so the application remains responsive and interactive during the running of an experiment. This includes the ability to browse individual model outputs while the experiment run is still underway. For model preparation, the data fetcher tools have been updated - fetching data from the new GBIF Rest interface is now possible. In addition, users can specify minimum size search results. Lastly the data fetcher tools have been re-implemented as a more user friendly wizard interface. Version now displays an AUC graph and ROC score for each model. When preparing a model the use of absence data is now supported. In addition, the visualization of model outputs in the mapping tab shows absence and presence data with separate symbology (by default red for absence and green for presence). The environmental values at each presence or absence point are now presented in a new table tab which has been optimized for large datasets. The model visualization now supports the overlay of a user defined 'context layer' (vector geographic boundaries) to aid users in placing the model in an appropriate geographic context. Once an experiment is completed, users can visualize the models in a new thumbnail view with options to sort by algorithm or by taxon. This, for example, facilitates easy visual comparison of different algorithm outputs for the same taxon. The addition of a new post processing tool allows users to create 'hotspot' maps and 'consensus' maps. Hotspot maps aggregate the outputs from two or more models using a specified probability threshold. The result is a raster showing the number of taxa predicted present per raster cell. The consensus map tool allows the outputs of multiple algorithms for the same taxon to be combined to create a single unified raster where the value in each cell represents the number of algorithms that predict presence for that cell. Regular updates were made to ensure that the openmodeller Desktop application remains current and compatible with the most recent version of the openmodeller library. The creation of a detailed unit testing framework using QTest and CTest with similar goals in mind to the test framework being developed for the openmodeller library. Web interface Three new prototype Web interfaces involving openmodeller were developed during the last year. Two of them (one in PHP and the other in Flex) were outcomes of the

12 Biogeo Interoperability Workshop 13. This workshop was organized by a specific task group from the TDWG Geospatial Interest Group 14 and hosted by CRIA. The aim of this workshop was actually to test biodiversity and geospatial protocols and data standards and to investigate how they could interoperate. This was done by developing two demo applications which basically consisted of Web interfaces making use of specific Web services. One of the services to be tested was based on the openmodeller Web Services interface (OMWS) developed by the openmodeller team to facilitate remote modeling. The final report of the workshop 15 includes comments about OMWS. Both applications developed during the workshop are Web interfaces that can generate niche models. The other prototype was produced in a partnership between CRIA, University of Colorado and GBIF. The result was also a Web interface making use of specific services to produce niche models 16. However, one of the objectives was to develop a Java library to access the modeling service so that GBIF could use it. GBIF is incorporating this functionality into their data portal with a scheduled release in April Being integrated into the GBIF portal will provide access to openmodeller to a very wide audience. These prototypes, as well as the first Web interface developed as part of this project, offered a good opportunity to learn about this kind of interface and to test ideas. However, we feel that such interface should also be able to interact with openmodeller as a local library, which would require C++ or Python as the programming language. So a final and full-featured Web interface will be developed as part of this project during its final year. 8 Integration with TerraLib Two new versions of TerraLib and TerraView were released during the last year: and The openmodeller driver to access TerraLib databases and the TerraView plugin are currently being updated for compatibility with version of TerraLib and TerraView. In this version TerraLib is now built as a dynamic library, which will help solve previous integration issues with the openmodeller library. TerraView and TerraLib version allow the integration of new data sources that are external to the TerraLib database: shapefiles, data collected from WMS servers and third party TerraLib databases. This will expand the capabilities of using distributed occurrence and environmental data for TerraView/openModeller users. TerraView now includes sample point generation functionality. It allows users to generate points over the layer of occurrence and/or environmental data in a random or stratified way. These points can be used to get values from modeling results and also for post-analysis operations. The integration between the R and TerraLib is also being updated to the latest version of TerraLib

13 Evaluation of parallelization techniques and analysis of parallel algorithms for biodiversity data analysis The following tasks were carried out to improve the performance of openmodeller by applying parallelization techniques: Software installation and configuration in the new cluster environment. Development of a portal for job submission on the cluster. Performance analysis of openmodeller. Parallelization of one of the existing algorithms (P-GARP). Parallelization of the model projection step. The following tasks are currently in progress: Adapt the existing openmodeller Web Services scheduler script to submit jobs to the cluster. Merge P-GARP and the parallelized projection into the openmodeller code repository. Deploy P-GARP and the parallelized projection in the cluster. Develop new strategies to improve the parallel versions. Four different versions of the original openmodeller projection code were produced and tested. The first version used the OpenMP library 17 to parallelize model projection in multiprocessor computers. Tests were performed in one node of the cluster (each node has dual processor cores). The initial results were not good (see annex 3 only in Portuguese) due to the intensive hard disk access rate of the application. The other two versions used Message Passing Interface (MPI) 18 on the cluster. In version 2, each node of the cluster generates a file in the TIFF format for each part of the distribution map, and in the end one of the nodes merges all files. In version 3, each node generates an intermediate custom file for each part of the map. This file contains the associated probability for each pixel. In the end, one of the nodes receives these intermediate files from the other nodes to generate the final map. Version 4 was based on version 3, but without generating intermediate files. The associated probability for each pixel is stored in a buffer which is transferred to node 0. More details can be found in annex 3 and 4 (only in Portuguese). There was a 5.7 fold reduction in execution time using version 4 with ten nodes compared to the sequential execution time (sequential version of openmodeller running in one node) when the map extent is limited to Brazil using the Bioclim algorithm. A 3.1 fold improvement was obtained when the map extent is South America. These versions are still being studied and improved. P-GARP is a parallel version of the GARP algorithm designed as part of this project for efficient use on clusters. The current implementation was tested for correctness and performance. P-GARP models were successfully compared with GARP models using the same input. P-GARP is currently available for testing but only for cluster identified users. The ideas that were used to develop P-GARP can be applied in the parallelization of other algorithms to improve performance. P-BestSubsets and HighP-BestSubsets are parallel versions of the openmodeller GARP Best Subsets algorithm, using GARP and P-GARP, respectively. The design of both new algorithms has been completed. P-BestSubsets is partially implemented, but is still not available. HighP- BestSubsets implementation has not been initiated

14 A Website for job submission to the cluster was built. It provides an interface for end users to request either a sequential openmodeller execution running on one node or a parallel execution running on several nodes. 10 Initial implementation using services architecture To help identify openmodeller components that can be processed in alternative parallel and distributed ways using a high-performance infrastructure, a detailed performance analysis was carried out during the second year of the project. In this study the openmodeller framework was split into several components and submitted to a typical workload defined with the help of end users. Results can be found in the performance analysis online report 19 (only available in Portuguese). A Web system was also developed to collect performance metrics of the openmodeller components. Building on the above mentioned work, studies for establishing a new Service Oriented Architecture model (or SOA-based) for openmodeller were initiated. These studies were also based on previous studies about architecture and reference models for openmodeller, the SOA reference model, and a comparative study between precision agriculture and biodiversity modeling information systems (all published as part of this project). First results include the definition of a reference architecture for ecological niche modeling and the application of SOA for service identification in ecological niche modeling and GIS systems. A new version of openmodeller using the new SOA-based architecture is being developed. Phase 1 consists of providing a service-oriented platform to deliver and integrate services attending to established SOA requirements. A preliminary version is available using Jboss, Apache Tomcat, Apache Axis and Apache Ant (all open source tools) but not yet integrated with openmodeller (see annex 5, only available in Portuguese). The next step consists of breaking openmodeller into several separate services. These will reflect the pre-analysis, modeling and post-analysis stages of the modeling process. Other developments Model repository Karla Donato Fook's doctoral thesis proposes a Web Services architecture that will support collaboration in a species distribution modeling network. By using this new architecture, users will be able to share model instances (definition of parameters, data used, chosen algorithms) and potential distribution maps. This will allow them to access other models and compare results. The proposed architecture was partially implemented. In 2007 two prototypes were released and presented to CRIA and INPE users for feedback. A first final release is expected to be delivered in April, Study Cases The following studies were carried out in collaboration with CRIA during the period between March 2007 and February 2008 with the objective of testing the framework, training and involving users from other institutions, and producing papers to disseminate openmodeller and its applications: 19

15 11 Comparison of two different algorithms (GARP and Maxent) in modeling the potential habitat of manned wolf (Chrysocyon brachyurus). The main objective was to know the consequences of actual habitat fragmentation for this species. Renata Kawashima & Eduardo Mantovani (Instituto Nacional de Pesquisas Espaciais INPE) and Marinez F. Siqueira (CRIA). Status: published. Performance test of two different algorithms (GARP and SVM Support Vector Machine) in modeling Cerrado tree species (Stryphnodendron obovatum). The main objective was to compare the accuracy of these algorithms and test the effect of using a high number of environmental layers in the process. Ana Carolina Lorena (Universidade Federal do ABC), Renato De Giovanni (CRIA), André C.P.L.F.Carvalho & Ricardo C. Prati (Universidade de São Paulo - USP, Campus de São Carlos). Status: accepted. Application of potential species modeling to known geographic distribution of Hennecartia omphalandra (Monimiaceae). Marcus Gonzales & Ariane L. Peixoto, (Escola Nacional de Botânica Tropical Jardim Botânico do Rio de Janeiro), and Marinez F. Siqueira (CRIA). Status: in preparation. Potential distribution modeling of species threatened of extinction from State of Minas Gerais, Brazil. Luciana H. Yoshino Kamino (Universidade Federal de Minas Gerais UFMG) and Marinez F. Siqueira (CRIA). Status: in preparation. Potential distribution modeling and model validation using openmodeller. Francisco C. Barreto (Universidade Federal de Viçosa UFV) and Marinez F. Siqueira (CRIA). Status: in preparation. Ecological niche modeling in the Brazilian Atlantic Forest: a comparative evaluation of presence-only methods for modeling the geographic distribution of anurans. João Gabriel Giovanelli & João Alexandrino (Universidade do Estado de São Paulo - UNESP, Campus de Rio Claro) and Marinez F. Siqueira (CRIA).. Status: in preparation. The following study cases were carried out at INPE: Fabio Iwashita s master thesis assessed the sensibility of species distribution models according to the precision of locality data. Two algorithms implemented in openmodeller (GARP and BIOCLIM) and Maxent were evaluated. The paper from this thesis will be submitted in March, 2008 (Iwashita, in preparation). For the state of São Paulo, species of genus Croton (Euphorbiaceae) and species from the Tribo Cynodonteae (Poaceae Chloridoideae) where studied in collaboration with the Instituto Botânico de São Paulo (IBt). The first analysis of the spatial distribution of the Tribo Cynodonteae indicated that the sampling effort has to be intensified to enable a better understanding of the biogeography and conservation status of the group (Santos et al., 2007). For the Croton genus, the importance of the soil variable in species distribution modeling for this group was analyzed (Caruzo et al. 2007). For both taxa, experiments considering species distribution models with different algorithms and variables can be further tested to better discuss the biological processes related to the species distribution A database for studying Arecaceae (Palmae) distribution in Brazil is under development. Occurrence data from the most important Brazilian and international herbaria were incorporated into the database. At the moment, a data cleaning process to correct the geographical coordinates and taxonomic nomenclature is being performed. The database with Arecaceae occurrence data will enable a study about the spatial distribution of Brazilian palms and also help testing openmodeller algorithm implementations.

16 12 Different environmental data and species distribution algorithms were tested to achieve better results and optimize modeling procedures. In particular, Normalized Difference Vegetation Index (NDVI) was used to model the genus Coccocypselum (Rubiacea) using GARP-OM Best Subsets algorithm and Maxent (Costa et al., in preparation). A study case is under preparation to analyze the impact of climate change over the life forms in the boundaries between savanna and tropical rain forest in the Brazilian Amazon. A database containing occurrence data from savanna and forest vegetation is under construction and will be used to model the current and the predicted distribution of dominant life forms (Box model algorithm) as well as for a number of selected species using algorithms implemented in openmodeller. Seminars In order to integrate the teams and try to make all have a clearer concept of the project as a whole, a number of seminars and meetings are held. During the last year of the project three seminars and two general meetings were held. Date: 14th June, 2007 Location: Poli Presentations: Using the new interface to submit Condor jobs in the cluster (Nilton Cézar de Paula, Poli), Plans for Unit Tests in openmodeller (Albert Massayuki Kuniyoshi, Poli), News about openmodeller Desktop (Tim Sutton, CRIA), Model Repository (Karla Fook, INPE). Date: 16th August, 2007 Location: INPE Presentations: Support Vector Machines (Ana Carolina Lorena, UFABC), Status of the cluster environment (César Bravo, Poli), Discussions about the project. Date: 18th October, 2007 Location: Poli Presentations: Adaptive Systems (Prof. João José Neto, Poli), AdaptGarp (César Bravo, Poli), Using openmodeller Desktop with the cluster (Tim Sutton, CRIA). The study group about Biodiversity Modeling that was created at INPE, called Referata Biodiversa 20, proceeded with the monthly meetings. Over the last year, the group discussed issues such as climate changes and potential vegetation models, spatial dependence in predictive vegetation modeling, status and problems related to digital database of biological collections, statistical methods to modeling evaluation, among others. Besides contributing towards the integration of INPE s team, it also offers the opportunity of interaction with other groups such as the Instituto de Biociências (USP) and the Instituto de Botânica de São Paulo (IBt-SP). Training Dr. Marinez F. Siqueira from CRIA offered training courses, lectures, and acted as advisor to graduate students working with openmodeller. Last year activities include: 20

17 13 Instructor: III course Potential distribution of species. Instituto de Pesquisas Ecológicas IPE. Nazaré Paulista, SP, Brazil. Apr/2008. Instructor: Advanced course Potential distribution of species. CENARGEM/EMBRAPA. Feb/2008. Co-adviser of Luciana H. Yoshino Kamino. PhD degree: Modelagem de espécies de plantas ameaçadas de extinção de Minas Gerais. Pós-graduação em Biologia Vegetal. Laboratório de Sistemática Vegetal. Depto de Botânica /ICB /UFMG. Jan/2008. Co-adviser of Francisco Candido Cardoso Barreto. PhD Degree. Potential distribution modeling and models validation in openmodeller. Programa de Pós-Graduação em Entomologia. Universidade Federal de Viçosa - UFV, Brazil. Feb/2008. Instructor: II course Potential distribution of species. Instituto de Pesquisas Ecológicas IPE. Nazaré Paulista, SP, Brazil. Sep/2007. Instructor: I course Potential distribution of species. Instituto de Pesquisas Ecológicas IPE. Nazaré Paulista, SP, Brazil. Mar/2007. Lecture: Modelagem de distribuição geográfica de espécies. In: XVIII Semana de Estudos da Ecologia. Instituto de Biociências, UNESP, Campus Rio Claro September, Lecture: Modelagem de distribuição potencial de espécies. In: Faculdades Integradas Metropolitanas de Campinas METROCAMP. 6th October, Lecture: Acesso a dados de coleções biológicas. In Faculdades Integradas Metropolitanas de Campinas METROCAMP. October 6, Lecture: Mudanças ambientais: possíveis impactos na biodiversidade. In: Programa de Extensão da Escola Nacional de Botânica Tropical. Seminários em Ciência e Tecnologia. Jardim Botânico do Rio de Janeiro. 20 th April, Lecture: Environmental satellite data: applications in studies of biodiversity. Strategies for Open and Permanent Access to Scientific Information in Latin America: Focus on Health and Environmental Information for Sustainable Development. Atibaia, SP May Dra. Silvana Amaral presented the openmodeller project as part of INPE s activity in the following events: Visit of the Ministry of Forestry of Indonesia at INPE, June/2007, presentation entitled Species Distribution Modeling in the Amazônia ; Lecture in the Post-Graduation in Remote Sensing at INPE (24/10/2007), in the course Tópicos Especiais em Floresta (SER 455-3), presentation entitled Modelos de Distribuição de Espécies ; Rede GEOMA Symposium, Petrópolis-RJ (29-31/10/2007), presenting the paper Estudos de Modelagem de Distribuição de Espécies no Componente Biodiversidade na Rede GEOMA. During this year the following people were involved with openmodeller through scholarships and training: Doctoral students: Cristina Giannini, Instituto de Biociências da Universidade de São Paulo (IB/USP), since 07/2007. Elisângela Silva da Cunha Rodrigues, EPUSP (CAPES scholarship), since 07/2007; Fabiana Soares Santana, EPUSP, since 02/2007;

18 14 Fabrício Rodrigues, EPUSP (CAPES scholarship), since 07/2007; Francisco Candido Cardoso Barreto, UFV; Karla Donato Fook, INPE, since 03/2004; Luciana H, Yoshino Kamino, UFMG; Nilton Cézar de Paula, EPUSP, since 06/2006. Master students: Fabio Iwashita, INPE Remote Sensing Program, finished in 03/2007 under the supervision of Dr. Silvana Amaral; João Gabriel R. Giovanelli, UNESP, Rio Claro; Marcos Gonzales, ENBT/JBRJ. Undergraduate students during the period of this report: Albert Massayuki Kuniyoshi EPUSP; Alex Oshika, student of Computer Engineering EPUSP; Danilo de Jesus da Silva Bellini, student of Electric Engineering EPUSP (CNPq scholarship); Luciano Bergantini Lippi, student of Computer Engineering EPUSP; Marcos Cabral Santos, student of Computer Engineering EPUSP (Fapesp scholarship); Mariana Ramos Franco, student of Computer Engineering EPUSP (Fapesp scholarship). FAPESP Technical Training scholarships: Alexandre Copertino Jardim, scholarship type TT4, since 10/2007; Dr. César Alberto Bravo Pariente, EPUSP, scholarship type TT5, 12/ /2007; Luciana Satiko Arasato, scholarship type TT3, since 10/2007; Missae Yamamoto, scholarship type TT5, since 10/2007; Renata Luiza Stange, EPUSP, scholarship type TT4a, since 01/2008; Tim Sutton, CRIA, scholarship type TT5, since 05/2006. Publications Journal papers Canhos, V.P., Siqueira, M.F.; Marino, A.; Canhos, D.A.L. Análise da vulnerabilidade da biodiversidade brasileira frente às mudanças climáticas globais. Parcerias Estratégicas. Centro de Gestão e Estudos Estratégicos. Accepted in March, De Marco Jr, P. & Siqueira, M.F. Como determinar a distribuição potencial de espécies sob uma abordagem conservacionista? Megadiversidade, Belo Horizonte. (Submitted in December 2007). Muñoz, M.E.S., Giovanni, R., Siqueira, M.F., Sutton, T., Brewer, P., Scachetti- Pereira, R., Canhos, V.P. & Canhos, D.A.L. openmodeller: A Generic Approach to Potential Distribution Modelling of Species. Geoinformatica. (Submitted in December 2007). Pereira, R. S. & Siqueira, M. F. Algoritmo Genético para Produção de Conjunto de Regras (GARP). Megadiversidade, Belo Horizonte. (Article in press)

19 Santana, F.S., Bravo, C., Saraiva, A.M. & Correa, P.L.P. Parallel Genetic Algorithm for Rule-set Production. Environmental Modelling and Software. (Submitted in March 2008) Santana, F. S., Siqueira, M. F., Saraiva, A. M. & Correa, P. L. P A reference business process for ecological niche modelling. Ecological Informatics Journal, v. 3 p Siqueira, M.F. & Durigan, G Modelagem da distribuição geográfica de espécies lenhosas de cerrado no Estado de São Paulo. Revista Brasileira de Botânica. v.30. p Siqueira, M.F., Durigan, G., De Marco Jr., P. & Peterson, A.T. Something from Nothing: Using Landscape Similarity and Ecological Niche Modeling to Find Rare Plant Species. Journal for Nature Conservancy. (Submitted in December 2007) Conference papers Amaral, S., Costa, C.B., Iwashita, F., Ximenes, A. & Valeriano, D.M. (2007). Estudos de Modelagem de Distribuição de Espécies no Componente Biodiversidade na Rede GEOMA. I Simpósio da Rede Geoma, Petrópolis, RJ. Amaral, S., Costa, C.B. & Rennó, C.D. (2007). Normalized Difference Vegetation Index (NDVI) improving species distribution models: an example with the neotropical genus Coccocypselum (Rubiaceae). Anais do XIII Simpósio Brasileiro de Sensoriamento Remoto, Florianópolis, Brasil, INPE, p (marte.dpi.inpe.br/col/dpi.inpe.br/sbsr@80/2006/ /doc/ pdf). Araujo, J. M., Correa, P.L.P. & Saraiva, A. M. A Framework for Species Distribution Modeling: a performance evaluation approach, I2TS'2007 Proceedings of the 6th International Information and Telecommunication Technologies Symposium, Brasília: IEEE R9, Editors: Fundação Bardall de Educação e Cultura; Boukerche, A, Loureiro, A.A.F., Melo, A.C.M.A. and Gondim, P.R.L. p Oral presentation. Bravo, C., Neto, J.J, & Santana, F.S. Unifyinig Genetic Representation and Operators in an Adaptive Framework. Analysis of Genetic Representations and Operators, AGRO Bravo, C., Neto, J.J, Santana, F.S. & Saraiva, A.M. Towards an adaptive implementation of genetic algorithms. Anais da XXXIII Conferência Latinoamericana de Informática CLEI 2007; Taller Latinoamericano de Informática para la Biodiversidad INBI 2007, San José, Costa Rica. Proceedings of the CLEI Centro Latinoamericano para Estudios en Informatica, v.1 p Caruzo, M.B., Costa, C. B., Amaral, S. & Cordeiro, I. (2007). Aplicação de classes de solo em modelos de distribuição de espécies: um exemplo com Croton L. (Euphorbiaceae). Paper presented at Congresso Nacional de Botânica, São Paulo. Kawashita, R.S., Siqueira, M.F. & Mantovani, E. (2007). Dados do monitoramento da cobertura vegetal por NDVI na modelagem da distribuição geográfica potencial do lobo-guará (Chrysocyon bracyurus). XIII Simpósio Brasileiro de Sensoriamento Remoto. Florianópolis, SC. v.13. p Kuniyoshi, M. A. & Correa, P. L. P. Aplicação de Testes Unitários no openmodeller, Anais do 15º Simpósio Internacional de Iniciação Científica da USP, São Carlos, Abstract. Poster presentation. Lorena, A. C., Siqueira, M. F., Giovanni, R., Carvalho, A. C. P. L. F. & Prati, R. C. Potential Distribution Modelling Using Machine Learning. In: The Twenty First International Conference on Industrial, Engineering & Other Applications of Applied 15

20 Intelligent Systems (IEA/AIE), Wroclaw, Poland. Lecture Notes in Artificial Intelligence, v. 5027, Springer-Verlag, (Accepted) Santana, F. S., Murakami, E., Saraiva, A. M. & Correa, P. L. P. A comparative study between precision agriculture and biodiversity modelling information systems. 6th Biennal Conference of the European Federation of IT in Agriculture, Glasgow: C.Parker, S.Skerratt, C.Park, J.Shields, v. 1. p Oral presentation. Santana, F. S., Murakami, E., Saraiva, A. M., Bravo, C. & Correa, P. L. P. Uma arquitetura de referência para sistemas de informação para modelagem de nicho ecológico, Anais do 6º Congresso Brasileiro de Agroinformática SBIAgro 2007, Campinas: Embrapa Informática Agropecuária, Editors: S.Tiernes, L.H.A. Rodrigues. p Oral presentation. Santana, F. S., Pinaya, J.L.D., Saraiva, A. M., Correa, P. L. P., Becerra, J.L.R. & Bravo, C. Aplicação de SOA para identificação de serviços em sistemas de modelagem de nicho ecológico e GIS, I2TS'2007 Proceedings of the 6th International Information and Telecommunication Technologies Symposium, Brasília: IEEE R9, Editors: Fundação Bardall de Educação e Cultura; Boukerche, A, Loureiro, A.A.F., Melo, A.C.M.A. and Gondim, P.R.L. Santos, A.L. dos, Wanderley, M.G.L., Bestetti, C.B. & Amaral, S. (2007). Diversidade da tribo Cynodonteae (Poaceae: Chloridoideae) no Estado de São Paulo. Paper presented at Congresso Nacional de Botânica, São Paulo, SP. Other papers Sutton, T., Giovanni, R. & Siqueira, M.F. Introducing openmodeller - A fundamental niche modelling framework. OSGeo Journal Volume 1. ISSN Available at Franco, M.F. Arcabouço para distribuição e modelagem de espécies uma análise de desempenho. Scientific Report FAPESP. FAPESP Process: 2006/ July pp. Stange, R.L. Manual de Instalação da Plataforma de Serviços. Internal Technical Report, February, pp. In preparation Costa, C.B. & Amaral, S., Presence-only modeling method for predicting species distribution: an example with the neotropical Rubiaceae genus Coccocypselum P. Br. Biota Neotropica. Giovanelli, J.G.R., Siqueira, M.F., Haddad, C.F.B. & Alexandrino, J. Ecological niche modeling in the Brazilian Atlantic Forest: a comparative evaluation of presence-only methods for modelling the geographic distribution of anurans. Gonzalez, M., Peixoto, A.L. & Siqueira, M.F. Chorology of Hennecartia omphalandra Poisson (Monimiaceae), a Miocene species from the South American Atlantic Forest. Iwashita, F., Amaral, S., Monteiro, A.M.V. Species distribution models sensibility to geographical positioning data. Journal of Biogeography. Santana, F.S., Sato, L., Bravo, C. & Saraiva, A.M. Performance improvement strategies for ecological niche modelling. 16

21 General Comments During the project we found that certain aspects of the architecture would be better if approached in a different way. Although a fully componentized architecture is still being researched, we are concentrated on providing a cohesive and simplified Web Service modeling infrastructure. Instead of creating individual Web Services for each openmodeller component as originally proposed, we are aiming at consolidating the main functional areas into a single modeling Web Service. We realized that some of the components, such as the locality and environmental components would be better implemented with a focus on supplying data to the modeling environment rather than to end-users. For example, TAPIR, WFS and DiGIR services all provide a robust and well-established protocol for obtaining occurrence data, so implementing another service with a similar goal would not make much sense. Similar logic applies to the environmental component, where protocols such as WCS are already wellestablished. The idea is to be able to retrieve data from these kinds of services. An additional motivation is to keep the public Web Services API simple in order to ease integration with third parties and facilitate maintenance. For these reasons, the core pre and post analysis functionality are being incorporated into the main modeling Web Service. 17 New architecture USER Interface (desktop, web, console, soap server, swig wrapper, others) modelling, pre-analysis,post-analysis Algorithm 1 data cleaning georeferencing openmodeller Algorithm 2 locality component environmental component Algorithm n Specimen data (occurrence points) Environmental data (layers) GBIF specieslink GEOSS Figure 2. New architecture for openmodeller Another part of the architecture that we are likely to change relates to the data cleaning component. Originally we conceived a Web Service for data cleaning but we feel that it is better to work on the existing data cleaning infrastructure of the specieslink network. These gains will benefit end users who will retrieve data from specieslink using the new openmodeller TAPIR driver. The publicity gained from our regular software releases and interactions with other individuals and institutions has resulted in a number of potential areas for future collaboration with the wider scientific community. These include:

22 18 An informal offer from Dr. Neil Caithness at the University of Oxford (UK) to host openmodeller services at the OxGrid Campus Grid Computing Centre and the National Grid Service for the UK. Informal discussions with various people from the American Museum of Natural History and NatureServe on how we can help them to integrate openmodeller into their current niche modeling processes. Informal discussions with Brian Hamlin (UC Berkeley, USA) towards including openmodeller in future large scale modeling experiments they are planning. openmodeller has also been selected for the second generation of the LifeMapper 21 project being developed by the University of Kansas. Another initiative using openmodeller is a GEOSS 22 demonstration project being developed by GBIF and the Italian National Research Council. The result of this demonstration project should be used by another project called Ecological Model Web 23 being developed by the Ecological Forecasting Program at NASA. In addition we have been able to engage with users of our software from various countries around the world through our users' mailing list and IRC presence. This has enabled the introduction of a Taiwanese translation of openmodeller Desktop which was contributed by one of our users. A major concern refers to the continuity of these developments. We consider openmodeller too important an initiative to depend on a project based grant or to be left solely as an open source initiative without substantial funding. Therefore, this last year will also be decisive as to planning its continuity and sustainability

23 19 Annex 1. Changelog Release ( ) SVN revision: 4209 Fixed bug when GDAL failed to read a raster row (in which case the row used to have zero values). Now the row is filled with nodata values. XML request for model creation now supports additional options to filter occurrences (using spatiallyunique or environmentallyunique sampler functions). Changed nodata value of the default raster type (ByteHFA) to 101. Masks must now use nodata to indicate masked areas (zeros will not work anymore). om_testmodel now generates pseudo-absence points when there are no absences to be tested. Display confusion matrix cell values in om_console and om_testmodel. Renamed getlayerfilename to getlayerpath and getmaskfilename to getmaskpath in EnvironmentPtr class. Refactored om_pseudo, om_create and om_project to use the getopts command line library. Created man pages for om_pseudo, om_create and om_project. New parameter in om_pseudo to speficy the proportion of absence points to be generated. New algorithm AquaMaps (for marine organisms). Removed algorithms minimum distance and distance to average since EnvironmentalDistance provides the same functionality. "type" property of AlgParamMetadata changed from char * to a new enumeration called AlgParamDatatype. Values can be Integer, Real and String. Fixed bug in the environmental distance algorithm: probabilities were negative for points whose distance to the nearest point was beyond the maximum allowed distance. New classes to perform jackknife and chi-square in the environmental layers. Updated TerraLib drivers for compatibility with TerraLib version Release ( ) SVN revision: 3806 Fixed compilation issues under Windows. Included new command line program to generate pseudo occurrences. Minor improvements in console tools (absences are now displayed in om_viewer and om_niche). Code clean up. Release ( ) SVN revision: 3661

24 20 Fixed MSVC compilation problems. Fixed bug in deserialization of the new GARP algorithm (OM GARP). Fixed crash in one-class SVM when input points contained absences. Fixed bug in deserialization of environmental distance algorithm with Mahalanobis distance. Implemented serialization/deserialization for the new GARP Best Subsets (OM GARP). New algorithm "Envelope Score". Fixed bug in the pseudo-absence generation of the SVM algorithm when no absences were passed as a parameter. Release 0.5 ( ) SVN revision: 3527 New algorithm "Support Vector Machines" (C-SVC, nu-svc and one-class SVM). Added support for multiple normalization techniques (two implementations are available: ScaleNormalizer and MeanVarianceNormalizer). New method to cancel jobs (model creation or model projection). Sample serialization is now based on the original (unnormalized) environment values. New infrastructure for unit tests using cxxtest. Release ( ) Included ROC curve as part of model statistics. Added metric Chebyshev in the Environmental Distance algorithm. Log object is now a singleton. Minor bugfixes.

25 21 Annex 2. Activities related to the specieslink network Centralized data repository During this year, a new architecture was designed to include a centralized data repository. Figure 1 shows the diagram prior to this activity. All cache nodes include a DiGIR provider that is also present in collections that are serving data directly to the DiGIR portal. All queries were distributed and the network was beginning to have problems with performance. This development was co-financed through a project with the JRS Biodiversity Foundation. The former architecture already had a database with a subset of DarwinCore fields In order to analyze the data and apply data cleaning tools. A webservice application called mapcria has also been developed to be used by different applications to dynamically produce maps through a web interface. Network Network Manager Manager Reports Web Site mapcria mapcria webservice webservice Data cleaning Data cleaning Data analysis Database subset Distributed query DiGIR DiGIR Portal Portal Geographic data Maps Maps PostGIS PostGIS WMS WMS DiGIR DiGIR Cache Cache node node Cache node Cache node Collections with a DiGIR provider SOAP Collections with splinker Figure 3. Diagram of network architecture With funding from JRS Biodiversity Foundation and Fapesp the architecture was altered and now includes a central repository of all data served by the collections. The repository was installed in a Dell PowerEdge 1900 server, running Linux Fedora Core 6 operating system. The server has two powerful Intel Xeon 3GHz CPUs with 8GB of RAM memory. The whole system is being developed using open-source software: the web server is Apache, the database management system is PostgreSQL and the programming language is Perl.

26 22 Network Network Manager Manager Reports Web Site mapcria mapcria webservice webservice Indicators Indicators Data cleaning Data cleaning Data analysis Centralized Query interface Query interface query Central Repository Central Repository Geographic data Maps Maps PostGIS PostGIS WMS WMS Data Harvester Data Harvester Distributed query DiGIR Portal DiGIR Portal DiGIR DiGIR Collections with a DiGIR provider Cache node Cache node Cache Cache node node SOAP Collections with splinker Figure 4. specieslink s new architecture Introducing a Central Repository to the architecture also meant developing a data harvester and a new query interface. The first idea was to store a subset of the DarwinCore fields selecting only those of interest for data cleaning and ecological niche modeling applications. As the analysis of fields for different purposes evolves, the team decided to store all data that was being provided by the collections. This way the system is already designed to use and provide all fields. The Central Repository uses PostgreSQL 24 on Linux, an open source relational database system that has more than 15 years of active development and a proven architecture. CRIA s team has about 10 years experience with the software that has proven to be robust with a very good performance, and has a number of resources available such as transaction control, maintenance of the referential integrity and automatic triggers. It also has native programming interfaces for C/C++, Java,.Net, Perl, Python, Ruby, Tcl, ODBC, among others, and good documentation. Its SQL implementation strongly conforms to the ANSI-SQL 92/99 standards and supports compound functional indexes which can use any of its B-tree, R-tree, hash, or GiST storage methods. GiST (Generalized Search Tree) serves as a foundation for many public projects that use PostgreSQL such as PostGIS. PostGIS is a project which adds support for geographic objects in PostgreSQL, allowing it to be used as a spatial database for geographic information systems (GIS). Other advanced features include table inheritance, a rules systems, and database events. Table inheritance puts an object oriented slant on table creation, allowing database designers to derive new tables from other tables, treating them as base classes. 24

27 All these implementations were used in the development of the centralized database. A general table (splink) was created to store textual data following the DarwinCore data model and using inheritance mechanisms to create tables for each collection of the network (figure 3). 23 Figure 5. Diagram of the main table (splink) with secondary tables for each collection PostGIS was used to facilitate the creation of maps, making geographic queries more efficient. A table was created for this specific purpose, using the geographic object point to store all georeferenced data of the network. The data harvester was developed using Perl and recognizes any changes in the network through the DiGIR portal that accesses all DiGIR providers. Once a day, the

28 system checks to see if the database has been updated. If there is any change, data analysis processes (data cleaning, network manager, and indicators) are triggered. 24 Centralized query 25 A classification system was added to the metadata to enable users to select the type of collection they wish to search. XML files were created for each collection and these are used by different applications such as the network manager, indicators, and the centralized query. Data outputs of the system include selecting what fields should be presented (small subset, locality, all), in what format (XML, HTML, Excel), and whether the user wishes to plot the georeferenced data on a map. Below is a print screen of a query where the collections selected were those that are classified as plants, herbarium, and voucher. Possible options for different outputs are also shown. This interface was written in Perl and is available in English and Portuguese languages. 25 available at

29 The classification system is also used by both management and monitoring system and indicators. The next figure shows a print screen of the map that is produced. Points are plotted with different colors for each collection and layers such as roadways, rivers, protected areas, among many others can be added. 25

30 26 Through the map interface it is now possible to retrieve information about the layers that were activated. Figure 6. Occurrence points of a species plotted on a map