AppliFish2 Synchronization

synchronization has 2 parts; the VREVirtual Research Environment. holding or serving the content, and the Mobile Apps providing or needing (sub)sets of the VREVirtual Research Environment. content.

VREVirtual Research Environment.

Push Features

The requirements for pushing from iMarine into Apps are to update the SQLite db containing the Species Fact Sheets;

• Entire new fact-sheets;
• Fact-sheets updates; new statistics, links, local names, maps, etc.

Pull Features - Generate New content - App side

There are many field data collection apps for environmental and biological data.

• A good example is the very widely used app for ornithologists: [1]
• Another 'random' app is [2]

Applifish couyld be equipped with similar features.

Pull feature - Generate new content - backend

• New species fact sheets may be published by FAO. These will be discoverable through a generic URL structure http://figisapps.fao.org/figis/geoserver/factsheets/species_e.html?species=XXX-Y (embed mode) See i.e. http://www.fao.org/figis/geoserver/factsheets/species_e.html?species=BET-m
• Another possibility is that species fact-sheets information is extracted from the fishFinderVRE
• For selected species, WoRMS data may be the base of Species Fact Sheets. This has to be discussed with Ward.

• For AppliFish data were extracted from the FAO FishFinder Aquatic Species fact sheets. The webservice for extracting selected pieces of information through specific XPaths download the results in CSV or JSON format (F.Fiorellato).

An initial api is available here: http://figisapps.fao.org/vrmf/samples/species/FS/

The list of endpoints (URLs) for the available services are:

Full data extraction

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/all.<format>
       Example (extract ALL data in CSV format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/all.csv
       Example (extract ALL data in JSON format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/all.json 

Subset (by 3-alpha-code) data extraction

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/3a/<comma separated list of 3-alpha codes>.<format>
       Example (extract data for Bluefin Tuna, Swordfish and Albacore in CSV format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/3a/BFT,SWO,ALB.csv
       Example (extract all data for Bluefin Tuna, Swordfish and Albacore in JSON format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/3a/BFT,SWO,ALB.json 

List all associations between 3-alpha-code and factsheet IDs

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/FS/list.<format>
       Example (extract all associations in CSV format): http://figisapps.fao.org/vrmf/samples/services/species/FS/list.csv
       Example (extract all associations in JSON format): http://figisapps.fao.org/vrmf/samples/services/species/FS/list.json 

Rescan the species factsheets (to be invoked when species factsheets are added, deleted or updated in order to reflect updates in the dataset)

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/reinitialize 

The actual XPaths are:

   FAO3AlphaCode: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAO3AlphaCode
   FAOName.en: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAOName/fi:En
   FAOName.fr: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAOName/fi:Fr
   FAOName.sp: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAOName/fi:Sp
   Image (width 300)[!] fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:Image[1] Please note that in several cases there are multiple images and drawings
   ScientificName: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fint:ScientificName
   Family fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName/fi:Family
   PersonalAuthor: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName/ags:PersonalAuthor Please note that Author and year need to be concatenated <author, year> and putin brackets if the attribute “ChangedGenus” (fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName) is = Y
   Year: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName/fi:Year
   DiagnosticFeat: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesProfile/fi:DiagnosticFeat (all sub elements to be concatenated)
   AreaText: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:GeoDist fi:AqSpeciesText+ fi:AreaText
   HabitatBio: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio all text for its sub-elements but skipping the content of the following sub-elements:
       Bathymetry node fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio/fi:DepthBehav/fi:Bathymetry
       Reproduction: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio/fi:Reproduction
       Feeding: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio/fi:Feeding 
   InterestFisheries only FisheriesText: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:InterestFisheries/fi:FisheriesText (all sub elements to be concatenated)
   Local names: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:LocalName then each name (and its country) is by entry fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:LocalName/fi:LocalNameEntry[XXXX] we suggest a format as <localName> (Country)

App SQLite data and structure

The results are copied into a simple SQLite database, depending on the quality of the source data into a temporary table, e.g. if alfa3 was not provided, if texts were too long, if names between data providers did not match(some 25% of names were not identical between different sources), or if the structure of the result did not allow for a direct copy.

some examples of transformation required before copying into the results table are:

• For the statistics; following the unit (tonne / numbers), merge all statistics into 1 column;

"t | 2000, 654 | 2001, 616 | 2002, 427 | 2003, 468 | 2004, 321 | 2005, 422 | 2006, 415 | 2007, 465 | 2008, 328 | 2009, 311 | 2010, 207"

• for the local name, merge information from FAO, FB/SLB, WoRMS into 1 column. Since the result from WoRMS was normalized, a de-nomalization / group_concat was needed to achieve a result similar to:

"Thresher, English | Renard, French | Zorro, Spanish  | Common thresher, England | Whiptail shark, South africa | Lluynog mor, Wales | Renard, France | Rabosa, Spain | Arequim, Portugal | Peixe rato, Madeira | Tubarâo raposo, Azores | Pesce volpe, Italy | Pas sabljas, Adriatic | Raefhajen, Sweden | Onagazame, Japan | Zorro, Cuba | Lisitska morskayia, Russia | Zorro cauda longa, Mozambique"  

• For name mismatching, a manual effort was done, as there is (was) no service available to radily match all.

All results are copied into the main App table:

CREATE TABLE "fishmapp" ("id" integer PRIMARY KEY NOT NULL ,"alfa3_code" text,"Scientific_name" text,"English_name" text,"French_name" text,"Spanish_name" text,"Author" text,"Family" text,"Ordo" text,"Local_names" text,"AquaMapsName" text,"Stats" text,"fact_distribution" text,"fact_production" text,"fact_link" text,"species_image" text, "iucn_status" TEXT, "species_group" TEXT, "sizes" TEXT, "dt_upd" DATETIME)