Difference between revisions of "Rule frame"

From D4Science Wiki
Jump to: navigation, search
(→‎VALID project Action rules)
(→‎VALID project Action rules)
Line 97: Line 97:
 
| align="center" | inter-entity
 
| align="center" | inter-entity
 
| align="center" | Daily Check (N+1?)
 
| align="center" | Daily Check (N+1?)
−
| align="center" | timestamp, Port information
+
| align="center" | timestamp, Port information,VMS, e-LogBook
 
| align="center" |  
 
| align="center" |  
 
| align="center" | New
 
| align="center" | New
 
| align="center" |  
 
| align="center" |  
 
| align="center" | A logbook entry (Fishing Activity Report -FAR-  declaration) must be available for every calendar day with VMS positions at sea
 
| align="center" | A logbook entry (Fishing Activity Report -FAR-  declaration) must be available for every calendar day with VMS positions at sea
−
| align="center" | (If return = "FALSE") Raise Alert ("No VMS reporting for the last 12 hours contact port authorities for thorough check on vessel (calls managePort(PortName))
+
| align="center" | (If return = "FALSE") Raise Alert ("No FAR reporting for the last day contact port authorities for thorough check on vessel (calls managePort(PortName))
−
|}
+
|-
 +
| align="center" | inter-entity
 +
| align="center" | When RTP message is received
 +
| align="center" | Species in FAR(Fishing Activity Report = species catches) declarations, DIS (Discarded Species) declarations, TRA (Transhipments) declarations, DEP (Departure from port) declaration = number of species on board when vessel leaves port, RTP (Return To Port = Landing declaration)
 +
| align="center" |
 +
| align="center" | New
 +
| align="center" |
 +
| align="center" | Comparison of logbook entries regarding species (catches, quantities on board, transhipments, discards) with species stated in the landing declaration
 +
| align="center" | (If return = "FALSE") Raise Alert ("")
 +
|-
 +
| align="center" | tuple
 +
| align="center" | When RTP message is received
 +
| align="center" | PNO (Prior Notification) message, RTP (Return To Port) message , for given fisheries (Herring, Mackerel, horse mackerel) to prepare landing control
 +
| align="center" |
 +
| align="center" | New
 +
| align="center" |
 +
| align="center" | A PNO is sent at least 4h prior to real return to port (RTP)
 +
| align="center" | (If return = "FALSE") Raise Alert ("")
 +
|-
 +
| align="center" | inter-entity
 +
| align="center" | When FAR message is received
 +
| align="center" | FAR (Fishing Activity Report) declarations, Fishing authorisations
 +
| align="center" |
 +
| align="center" | New
 +
| align="center" |
 +
| align="center" | Verify whether the combinations of fishing area, gear and species entered  in a declaration of catches, are consistent with valid combinations according to a valid fishing authorization. (excluding catches taken for the purpose of scientific research)
 +
| align="center" | (If return = "FALSE") Raise Alert ("")
 +
|-
 +
| align="center" | inter-entity
 +
| align="center" | When all declarations of commercialization are received
 +
| align="center" | Landing declaration, Declarations of commercialization
 +
| align="center" |
 +
| align="center" | New
 +
| align="center" |
 +
| align="center" | Check whether landed species match species stated in the declarations of commercialisation (usage, transport declarations, take-over declarations).
 +
| align="center" | (If return = "FALSE") Raise Alert ("")
 +
|-
 +
| align="center" | inter-entity
 +
| align="center" | When all sales note are received
 +
| align="center" | Sale notes, Fleet register
 +
| align="center" |
 +
| align="center" | New
 +
| align="center" |
 +
| align="center" | Check that vessel selling species is registered with a valid licence
 +
| align="center" | (If return = "FALSE") Raise Alert ("")|}
  
 
=== Taxon name reconciliation rules ===
 
=== Taxon name reconciliation rules ===

Revision as of 08:08, 30 July 2013

Rule Frame

One approach to capturing requirements and potentially define on the implementation facilities, is to use a rule frame. This classifies and describes the expected types of rules to apply to data once an action has been triggered.

The rule frame described in this example related to the validation of a imaginary dataset that is collected from a Regional Fisheries Organization, and has been loaded as a CSV in ICIS. It now needs to be reconciled with previous datasets from the same reporting stream, and be published in various locations.

There are 2 basic types of rules:

  • Contraints check if a condition is violated (return a boolean), and this triggers some action (from an alert, to stopping the transaction) that is not related to a modification of data.
  • Action rules trigger an event that can be a transaction, a form navigation, or something similar.

There are also several types of rules:

  • Attribute rules operate on one input value, and usually check only it's validity according to a predefined range.
  • Tuple rules check if two values in a row that have a defined relationship do not violate the relationship.
  • Entity rules check if the values in a tuple 'fit' within the data-table (or structured dataset) they belong to. Exaples are time-series of VMS positions where two consecutive records may have temporal and spatial distances that should be observed.
  • Inter-entity rules check if the values in one table-row are consistent with the values in another (set of) tables. Examples are if the given lat-long are within an EEZ, if the given genus / species name is valid, if the portname is open for vassels of the reporting category, and much, much more.
  • OTA are Other Triggered Actions, and basically cover all actions that are not modeled with the data. These can e.g. be import and export actions, logging, alerting, backing-up, or sharing.

Rules have to fired by an event, that can be user-triggered (mouse-events, click events), data manipulation (CRUD), or by a clock. In order to perform the operation correctly, they must fire at the right time, and in the correct sequence. The timing is usually related to maintaining data integrity, thus before the manipulation is persisted. The sequence is a topic for another page. This page is to collect user level requirements only, more developer oriented pages will have to describe the implementation and fire-sequence.

Check rules

Type Trigger When Status Implemented by Comment Action
Attr 0 <= temp <= 30 AI / AU New Check that Seawater is warmer then 0, and cooler than 30 Raise Alert ("Your water temp is out of range")
Tuple dtStart <= dtEnd BI / BU New A check to ensure start dates are before end-dates Raise Alert ("A start date must be before the end-date")
Entity Unique key on genus, species, subspecies BI / BU New Check that species are unique Raise Error ("This species already exists")
Inter Entity Valid Lat/Long App logic New Find if Lat/Long fits within BBox of distributions Raise Alert ("This position seems unvalid")

Action rules

Type Trigger Params Returns Status Implemented by Comment Action
tuple getGeoName Lat / Long geoName New Retrieve the nearest geoName, of any kind (If return = "") Raise Alert ("No name found")
tuple getGeoNameCityOrPort Lat / Long geoName New Retrieve the nearest geoName, of type "City" / "Port" (If return = "") Raise Alert ("No name found")

VALID project Action rules

Please note that no action has been defined for the VALID rules. The below proposed actions are tentative. The table below presents the 7 core VALID rules.

Type Trigger Params Returns Status Implemented by Comment Action
tuple Daily Check (N+1?) timestamp, PortName New Check that delay between two VMS reporting <= 12h (If return = "FALSE") Raise Alert ("No VMS reporting for the last 12 hours contact port authorities for thorough check on vessel (calls managePort(PortName))
inter-entity Daily Check (N+1?) timestamp, Port information,VMS, e-LogBook New A logbook entry (Fishing Activity Report -FAR- declaration) must be available for every calendar day with VMS positions at sea (If return = "FALSE") Raise Alert ("No FAR reporting for the last day contact port authorities for thorough check on vessel (calls managePort(PortName))
inter-entity When RTP message is received Species in FAR(Fishing Activity Report = species catches) declarations, DIS (Discarded Species) declarations, TRA (Transhipments) declarations, DEP (Departure from port) declaration = number of species on board when vessel leaves port, RTP (Return To Port = Landing declaration) New Comparison of logbook entries regarding species (catches, quantities on board, transhipments, discards) with species stated in the landing declaration (If return = "FALSE") Raise Alert ("")
tuple When RTP message is received PNO (Prior Notification) message, RTP (Return To Port) message , for given fisheries (Herring, Mackerel, horse mackerel) to prepare landing control New A PNO is sent at least 4h prior to real return to port (RTP) (If return = "FALSE") Raise Alert ("")
inter-entity When FAR message is received FAR (Fishing Activity Report) declarations, Fishing authorisations New Verify whether the combinations of fishing area, gear and species entered in a declaration of catches, are consistent with valid combinations according to a valid fishing authorization. (excluding catches taken for the purpose of scientific research) (If return = "FALSE") Raise Alert ("")
inter-entity When all declarations of commercialization are received Landing declaration, Declarations of commercialization New Check whether landed species match species stated in the declarations of commercialisation (usage, transport declarations, take-over declarations). (If return = "FALSE") Raise Alert ("")
inter-entity When all sales note are received Sale notes, Fleet register New Check that vessel selling species is registered with a valid licence (If return = "FALSE") Raise Alert ("")|}

Taxon name reconciliation rules

Type Trigger When Status Implemented by Comment Action
Attr Taxonomic name is null Raise error
Attr Taxonomic name written in all-caps "Convert to proper caps: change everything to lower, except first character of he name, and the first character following a '('" Convert
Attr Taxonomic name contains invalid characters and is listed as valid name~'^[A-Za-z -()]+$' "taxonomic names can only have valid latin characters (no diacritical marks, no punctuation like '.' or ':', no ligatures…). Brackets are allowed around a subgenus name. Hyphens are allowed in botany, not in zoology (see below). Inmany older sources, we do find diacritical marks and especially ligatures, so we have to be able to accommodate them in the database; but they should not be listed as valid" Raise alert
Attr Taxonomic name contains more than 1 pair of brackets [)]{2,}'" Raise alert
Attr Taxonomic name has inbalanced brackets name !~ '^[A-Z][a-z-]+( [(][A-Z][a-z-]+[)])?' Raise alert
Attr year of description <1735 or >present Botanical names start from 1735 with the firs edition of the 'Systema Naturae' of Linnaeus Raise alert
Attr no parent taxon "parent_id is null, or taxon record with id=parent_id does not exist" We don't know where to place the name in the taxonomic classification Raise error
Attr no 'valid' taxon "valid_id is null, or taxon record with id=valid_id does not exist" We don't know whether the name is the name of a valid taxon or not Raise error
Attr no authority Often authority is not listed Raise warning?
Attr "no rank, or rank not in rank table" "Raise allert, raise error"
Tuple "kingdom='Animalia', year of description < 1758" "Zoological names start from 1758, with the 10th edition of the 'Systema Naturae' of Linnaeus" Raise alert
Tuple [need to deal with Bacteria here: need to be published in the bacterial names database before they can be valid; start of bacterial nomenclature: 1973 or something; all earlier names can not be valid]
Entity combination of taxonomic name and author is not unique "There are very few exceptions to this rule; not sure it is, at this point, worth complicating our systems to accommodate them" Raise error
Inter-entity rank of parent is lower or equal to rank of child "For example, species declared parent of genus" Raise error
Inter-entity parent-child relationship jumps mandatory ranks needs the 'ranks' tables with mandatory and optional parents as in WoRMS; needs list of exceptions "For example, a family is declared as the parent of a species; genus is a mandatory rank. Exception for some high-level taxa with very little descedants, such as phylum Phoronida (these are formally listed and recognised with the Code for Zoological Nomenclature)" Check for exception
Inter-entity "parent-child relationship jumps mandatory ranks, and parent is not listed as one of the exception cases" Raise alert
Inter-entity siblings have different rank All children of the same parent have to have the same rank; exception when parent name is '<name> incertae sedis' Raise alert
Inter-entity invalid parent for a valid taxon A valid taxon can not be a child of an invalid parent; all descendants of an invalid taxon should also be invalid Raise alert
Inter-entity taxon points to invalid taxon as its valid taxon "Needs to result in error, not only alert - danger of 'circular synonyms' (record A points to B points to C points to A), resulting in infinite loops when looking for a valid name for a taxon" "Only valid taxa can be declared 'valid' taxon for another (invalid) taxon. Alternative, for a more complex system, is to keep the complete 'audit trail' of changes of synonymy. So, if taxon A was first (in history) declared a synonym of taxon B, then taxon B declared a synonym of taxon C: record A points to record B as valid taxon, not to record C; this ay we can follow the history of a name (which often makes it easier to understand why a particular change has taken place). In OBIS now: record A would point to record C. In WoRMS: not consistent (at least not last time I looked); the original intention was, following our example again, to store C as valid taxon for A, not B." Raise error
Inter-entity "A species is listed under different subgenera, and/or both with and without subgenus, and there is more than one of these combinations that is listed as valid" "A specific epitheton has to be unique within a genus; if it occurs in combination with different subgenera, or once with, once without subgenus, only one of those combinations can be valid" Raise alert
Inter-entity "A name is listed both as a genus and a subgenus, and the parent of the subgenus is not the genus, and both are listed as valid" "E.g. subgenus 'Oneus (Twous)' and genus 'Twous' are both listed as valid. WoRMS is further normalised than OBIS, and stores the name parts (genus name, specific and infra-specific epithetons) in atomised fields. The complete name is calculated from these name parts. In OBIS, only the full name is stored, raising the possibility of creating descendant names that conflict with the ancestors." Raise error
Inter-entity species or infra-specific name conflicts with name of parent E.g. listed parent name is 'Oneus' but species name is 'Twous three' Raise error
Tuple rank>genus and name!~'^[A-Z][a-z]+( incertae sedis)?$' "A supra-generic name is a single word ('uninomen'), starting with a capital, followed by an optional phrase 'incertae sedis'"
Inter-entity name~'^[A-Z][a-z]+( incertae sedis)?$' and no other taxa points at this taxon as their parent "A 'Taxon incertae sedis' is not a real taxon, but a container for all names within 'Taxon' that have no clear position in the classification. " flag for deletion
Tuple rank=genus and kingdon='Animalia' and name!~'^[A-Z][a-z]+$' "A genus name is a single word, starting with a capital; there are very few names that have other than a-z. Have to look up the exceptions"
Tuple rank=genus and kingdon!='Animalia' and name!~'^[A-Z][a-z-]+$' "A genus name is a single word, starting with a capital; there are very few names that have other than a-z. In other than Animalia, also a hyphen is valid character"
Tuple rank=species and kingdom='Animalia' and name!~'^[A-Z][a-z]+( [(][A-Z][a-z]*[)])? [a-z]+$' "A species name consists of two words: a genus name followed by a specific epitheton; an epitheton starts with a lower case letter. Between the genus name and the specific epitheton is an optional subgenus name, enclosed in '(', ')'; a subgenus name starts with an upper case letter"
Tuple rank=species and kingdom!='Animalia' and name!~'^[A-Z][a-z-]+( [(][A-Z][a-z-]*[)])? [a-z-]+$' Allow hyphens for taxa hat aren't animals
Tuple rank=family and kingdom='Animalia' and name!~'idea$' Same for other standard endings of animal taxon names
Tuple rank=family and kingdom='Plantae' and name!~'aceae$' "A few well-known exceptions, such as 'Compositae'. These exceptions are deprecated, and properly-formed alternatives exist for all of them, but last time I checked their use was still alowed, and they definitely have to be included in the database (but pointing at the properly-formed alternative as the valid name)." Check for exception
Tuple rank<subspecies and kingdom='Animalia' and taxon listed as valid Zoological nomenclature no longer recognises ranks below subspecies
Tuple rank=subspecies and kingdom='Animalia' and name!~'^species name [a-z]$' "An infraspecific epitheton starts with a lower case; in zoological names, the infraspecific indicator is not written, as this can only be a subspecies"
Tuple rank=subspecies and kingdom!='Animalia' and name!~'^species name (subsp[.]|ssp) [a-z]$' "An infraspecific epitheton starts with a lower case; for other than animal names, the rank is indicated by writing either 'subsp.' or 'ssp'. Similar rules for variety, subvariety, forma, subforma"
Attr "specific epitheton is adjective, and gender doesn't correspond with gender of genus" "Needs taxonomic dictionary to find out whether epitheton is adjective or noun, and gender of genus; and if epitheton is adjective, stem + male/female/neutral forms"
Attr "specific epitheton is noun in apposition, but the ending of the epitheton has been mangled to correspond with the gender of the genus" Raise alert
Inter-entity "two or more names have identical epithetons (with possible exception of gender-dependent ending) and identical non-null authority (with the possible exception of a pair of '(', ')' surrounding the authority, or brackets followed by an extra author string), and more than one of these names are listed as valid " "Caused by new combinations between species and genus, that were not caught in the database" Increase probability that these are undetected synonyms
Inter-entity "two or more names have identical epithetons (with possible exception of gender-dependent ending) and similar non-null authority (with the possible exception of a pair of '(', ')' surrounding the authority, or brackets followed by an extra author string), or null authority, within the same say phylum, and more than one of these names are listed as valid " Need to use higher taxon (phylum in the exaple) as a parameter in a comparison tool "Same but with uncertainty on the author; the closer the the two names are in he classification, the more likely that the two names are synonyms" Increase probability that these are undetected synonyms
Inter-entity "two or more names of rank species or below differ only in the gender-dependent ending of the epitheton, and do not have the same valid name" Increase probability that these are undetected spelling variations
Inter-entity "two or more names of rank species or below differ only in the way a genitive is formed from the non-latin stem, and do not have the same valid name" "male genitives 'i' or 'ii', or female 'ae' or 'iae' are likely to be the same; also quite often the gender itself is wrong so that all 4 can be treated as likely spelling variations. In case of native Latin names, there is usually no doubt whether the ending should have the extra i: Macrus -> Marci; Antonius -> Antonii; Maria -> Mariae; Margareta -> Margaretae. When non-latin names are latinised, this can be done by appending either -us or -ius, leading to either -i or -ii in the genitive. " Increase probability that these are undetected spelling variations
Inter-entity "two names are listed as synonyms, but the rank of the lower ancestor is too high" "Can happen because of silly mistakes, database errors caused by homonymous genus names… Exceptions caused by synonymous high taxa (especially for protozoans)" Check for exception
Inter-entity "two or more names are identical if all 'y's are replaced with 'i's, and do not have the same valid name" "Same for other substitution rules to implement fuzzy matching a la Rees: {sck}, {h}…" "Catch a number of very common sources of spelling mistakes, specific for taxonomyc names" Increase probability that these are undetected spelling variations
Inter-entity "the Levenshtein distance between two names is lower than a treshold, and names do not have the same valid name" treshold a function of the length of the names Catch general misspellings Increase probability that these are undetected spelling variations; increase inversely propostional with the distance
Attr name ~* ' [(]lpil[)]$' OBIS-specific (and duplicated in GBIF) "lpil' stands for 'Lowest Possible Identification Level', is not part of the name, and does not convey any information that is not contained in the name itself; should be stripped from the name" strip ' (lpil)' from name
Attr [Aa]ff[.]?) ' OBIS-specific (and duplicated in GBIF) """'Cfr' and 'Aff' are not parts of the name proper, but a judgement on the reliability or the certainty of the identification; in OBI, such names are stripped from the qualifier and reduced in precision to the first reliable taxonomic name. E.g. species 'Oneus cfr twous' would be reduced to genus 'Oneus'" aff[.]?) ' from name; reduce to higher taxonomic level if needed (sacrificing precision for accuracy)
Attr "species' name has two or more epithetons, often separated by '-' or '/'" OBIS-specific (and duplicated in GBIF) Person identifying the specimen wasn't able to make his mind up between several different possible identifications strip extra information from name; reduce to higher taxonomic level if needed (sacrificing precision for accuracy)
Attr taxonomic name has extra copy of authority OBIS-specific (and duplicated in GBIF) name~*authority'$' "The OBIS 'scientific name' field, according to its definition, des not include the authority; several data providers have been reporting scientific name with authority included, and these cases where not always caught before the names went itno the taxonomic tables" strip extra information from name
Attr taxonomic name has information on gender or life stage of the specimens OBIS-specific (and duplicated in GBIF) needs vocabulary of 'gender' and 'lifestage' to recognise the terminology in the candidate names strip extra information and store in separate fields
Retrieved from "https://wiki.d4science.org/index.php?title=Rule_frame&oldid=14377"