Difference between revisions of "AppliFish2 Synchronization"

Revision as of 15:54, 24 April 2013

synchronization has 2 parts; the VREVirtual Research Environment. holding or serving the content, and the Mobile Apps providing or needing (sub)sets of the VREVirtual Research Environment. content.

VREVirtual Research Environment.

Push Features

No requirements yet described for pushing from iMarine into Apps

Pull Features

• New species fact sheets may be published by FAO. These will be discoverable through a generic URL structure http://figisapps.fao.org/figis/geoserver/factsheets/species_e.html?species=XXX-Y (embed mode) See i.e. http://www.fao.org/figis/geoserver/factsheets/species_e.html?species=BET-m
• Another possibility is that species fact-sheets information is extracted from the fishFinderVRE
• For selected species, WoRMS data may be the base of Species Fact Sheets. This has to be discussed with Ward.

• For AppliFish data were extracted from the FAO FishFinder Aquatic Species fact sheets. The webservice for extracting selected pieces of information through specific XPaths download the results in CSV or JSON format (F.Fiorellato).

An initial api is available here: http://figisapps.fao.org/vrmf/samples/species/FS/

The list of endpoints (URLs) for the available services are:

Full data extraction

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/all.<format>
       Example (extract ALL data in CSV format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/all.csv
       Example (extract ALL data in JSON format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/all.json 

Subset (by 3-alpha-code) data extraction

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/3a/<comma separated list of 3-alpha codes>.<format>
       Example (extract data for Bluefin Tuna, Swordfish and Albacore in CSV format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/3a/BFT,SWO,ALB.csv
       Example (extract all data for Bluefin Tuna, Swordfish and Albacore in JSON format): http://figisapps.fao.org/vrmf/samples/services/species/FS/extract/3a/BFT,SWO,ALB.json 

List all associations between 3-alpha-code and factsheet IDs

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/FS/list.<format>
       Example (extract all associations in CSV format): http://figisapps.fao.org/vrmf/samples/services/species/FS/list.csv
       Example (extract all associations in JSON format): http://figisapps.fao.org/vrmf/samples/services/species/FS/list.json 

Rescan the species factsheets (to be invoked when species factsheets are added, deleted or updated in order to reflect updates in the dataset)

   Syntax: http://figisapps.fao.org/vrmf/samples/services/species/reinitialize 

The actual XPaths are:

   FAO3AlphaCode: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAO3AlphaCode
   FAOName.en: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAOName/fi:En
   FAOName.fr: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAOName/fi:Fr
   FAOName.sp: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:FAOName/fi:Sp
   Image (width 300)[!] fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:Image[1] Please note that in several cases there are multiple images and drawings
   ScientificName: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fint:ScientificName
   Family fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName/fi:Family
   PersonalAuthor: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName/ags:PersonalAuthor Please note that Author and year need to be concatenated <author, year> and putin brackets if the attribute “ChangedGenus” (fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName) is = Y
   Year: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesIdent/fi:SciName/fi:Year
   DiagnosticFeat: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesProfile/fi:DiagnosticFeat (all sub elements to be concatenated)
   AreaText: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:GeoDist fi:AqSpeciesText+ fi:AreaText
   HabitatBio: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio all text for its sub-elements but skipping the content of the following sub-elements:
       Bathymetry node fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio/fi:DepthBehav/fi:Bathymetry
       Reproduction: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio/fi:Reproduction
       Feeding: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:HabitatBio/fi:Feeding 
   InterestFisheries only FisheriesText: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:InterestFisheries/fi:FisheriesText (all sub elements to be concatenated)
   Local names: fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:LocalName then each name (and its country) is by entry fi:FIGISDoc/fi:AqSpecies/fi:AqSpeciesFeature/fi:LocalName/fi:LocalNameEntry[XXXX] we suggest a format as <localName> (Country)

App SQLite structure

The results are copied into a simple SQLite database, depending on the qulaity of the source data into a temporary table, e.g. if alfa3 was not provided, if texts were too long, to match names between data providers (some 25% of names were not identical between differnt sources)

All results are copied into the main App table:

CREATE TABLE "fishmapp" ("id" integer PRIMARY KEY NOT NULL ,"alfa3_code" text,"Scientific_name" text,"English_name" text,"French_name" text,"Spanish_name" text,"Author" text,"Family" text,"Ordo" text,"Local_names" text,"AquaMapsName" text,"Stats" text,"fact_distribution" text,"fact_production" text,"fact_link" text,"species_image" text, "iucn_status" TEXT, "species_group" TEXT, "sizes" TEXT, "dt_upd" DATETIME)