Difference between revisions of "Rule frame"
From D4Science Wiki
(→Taxon name reconciliation rules) |
(→Taxon name reconciliation rules) |
||
| Line 142: | Line 142: | ||
{| border="1" cellpadding="4" cellspacing="0" valign="top" | {| border="1" cellpadding="4" cellspacing="0" valign="top" | ||
| − | ! width="50"|'''Type''' !! width=" | + | ! width="50"|'''Type''' !! width="140"|'''Trigger''' !! width="120"|'''When''' !! width="120"|'''Status''' !!width="120"|'''Implemented by''' !! width="330"|'''Comment''' !! width="120"|'''Action''' |
|- valign="top" | |- valign="top" | ||
| Attr || Taxonomic name is null || || || || || Raise error | | Attr || Taxonomic name is null || || || || || Raise error | ||
Revision as of 10:39, 24 July 2013
Rule Frame
One approach to capturing requirements and potentially define on the implementation facilities, is to use a rule frame. This classifies and describes the expected types of rules to apply to data once an action has been triggered.
The rule frame described in this example related to the validation of a imaginary dataset that is collected from a Regional Fisheries Organization, and has been loaded as a CSV in ICIS. It now needs to be reconciled with previous datasets from the same reporting stream, and be published in various locations.
There are 2 basic types of rules:
- Contraints check if a condition is violated (return a boolean), and this triggers some action (from an alert, to stopping the transaction) that is not related to a modification of data.
- Action rules trigger an event that can be a transaction, a form navigation, or something similar.
There are also several types of rules:
- Attribute rules operate on one input value, and usually check only it's validity according to a predefined range.
- Tuple rules check if two values in a row that have a defined relationship do not violate the relationship.
- Entity rules check if the values in a tuple 'fit' within the data-table (or structured dataset) they belong to. Exaples are time-series of VMS positions where two consecutive records may have temporal and spatial distances that should be observed.
- Inter-entity rules check if the values in one table-row are consistent with the values in another (set of) tables. Examples are if the given lat-long are within an EEZ, if the given genus / species name is valid, if the portname is open for vassels of the reporting category, and much, much more.
- OTA are Other Triggered Actions, and basically cover all actions that are not modeled with the data. These can e.g. be import and export actions, logging, alerting, backing-up, or sharing.
Rules have to fired by an event, that can be user-triggered (mouse-events, click events), data manipulation (CRUD), or by a clock. In order to perform the operation correctly, they must fire at the right time, and in the correct sequence. The timing is usually related to maintaining data integrity, thus before the manipulation is persisted. The sequence is a topic for another page. This page is to collect user level requirements only, more developer oriented pages will have to describe the implementation and fire-sequence.
Check rules
| Type | Trigger | When | Status | Implemented by | Comment | Action |
|---|---|---|---|---|---|---|
| Attr | 0 <= temp <= 30 | AI / AU | New | Check that Seawater is warmer then 0, and cooler than 30 | Raise Alert ("Your water temp is out of range") | |
| Tuple | dtStart <= dtEnd | BI / BU | New | A check to ensure start dates are before end-dates | Raise Alert ("A start date must be before the end-date") | |
| Entity | Unique key on genus, species, subspecies | BI / BU | New | Check that species are unique | Raise Error ("This species already exists") | |
| Inter Entity | Valid Lat/Long | App logic | New | Find if Lat/Long fits within BBox of distributions | Raise Alert ("This position seems unvalid") |
Action rules
| Type | Trigger | Params | Returns | Status | Implemented by | Comment | Action |
|---|---|---|---|---|---|---|---|
| tuple | getGeoName | Lat / Long | geoName | New | Retrieve the nearest geoName, of any kind | (If return = "") Raise Alert ("No name found") | |
| tuple | getGeoNameCityOrPort | Lat / Long | geoName | New | Retrieve the nearest geoName, of type "City" / "Port" | (If return = "") Raise Alert ("No name found") |
VALID project Action rules
Please note that no action has been defined for the VALID rules. The below proposed actions are tentative.
| Type | Trigger | Params | Returns | Status | Implemented by | Comment | Action |
|---|---|---|---|---|---|---|---|
| tuple | chkPort | PortName, timestamp | New | Check if the reported port is valid | (If return = "") Raise Alert ("No port found, want to check the name Y/N (calls managePort(PortName)") | ||
| entity | chkPortMooring | [[PortName], [timestamp]] | New | Check if the reported stay in a port is valid; no other location can be reported in between reported location | (If return = "FALSE") Raise Alert ("No valid port mooring reported") | ||
| inter-entity | chkSteaming | [[Lat], [long], [timestamp]] | New | Check if the reported positions are valid steaming according to chkSea, chkSpeed, chkLicense, chkFishArea | (If return = "FALSE") Raise Alert ("No valid port mooring reported") | ||
| OTA | managePort | [PortName, timestamp opt] | New | Check if the PortName does exist given the timestamp, and gives Create Update options | (If return = "TRUE") Raise Alert ("Your changes have been sent for review") | ||
| tuple | chkVMSInterval | timestamp, PortName | New | Check that delay between two VMS reporting <= 2h | (If return = "FALSE") Raise Alert ("No VMS reporting for the last 2 hours contact (calls getVesselsDetails("RadioCode",VMSVesselRadioCode)) | ||
| tuple | chkVMSInterval | timestamp, PortName | New | Check that delay between two VMS reporting <= 12h | (If return = "FALSE") Raise Alert ("No VMS reporting for the last 12 hours contact port authorities for thorough check on vessel (calls managePort(PortName)) |
Taxon name reconciliation rules
| Type | Trigger | When | Status | Implemented by | Comment | Action |
|---|---|---|---|---|---|---|
| Attr | Taxonomic name is null | Raise error | ||||
| Attr | Taxonomic name written in all-caps | "Convert to proper caps: change everything to lower, except first character of he name, and the first character following a '('" | Convert | |||
| Attr | Taxonomic name contains invalid characters and is listed as valid | name~'^[A-Za-z -()]+$' | "taxonomic names can only have valid latin characters (no diacritical marks, no punctuation like '.' or ':', no ligatures…). Brackets are allowed around a subgenus name. Hyphens are allowed in botany, not in zoology (see below). Inmany older sources, we do find diacritical marks and especially ligatures, so we have to be able to accommodate them in the database; but they should not be listed as valid" | Raise alert | ||
| Attr | Taxonomic name contains more than 1 pair of brackets | [)]{2,}'" | Raise alert | |||
| Attr | Taxonomic name has inbalanced brackets | name !~ '^[A-Z][a-z-]+( [(][A-Z][a-z-]+[)])?' | Raise alert | |||
| Attr | year of description <1735 or >present | Botanical names start from 1735 with the firs edition of the 'Systema Naturae' of Linnaeus | Raise alert | |||
| Attr | no parent taxon | "parent_id is null, or taxon record with id=parent_id does not exist" | We don't know where to place the name in the taxonomic classification | Raise error | ||
| Attr | no 'valid' taxon | "valid_id is null, or taxon record with id=valid_id does not exist" | We don't know whether the name is the name of a valid taxon or not | Raise error | ||
| Attr | no authority | Often authority is not listed | Raise warning? | |||
| Attr | "no rank, or rank not in rank table" | "Raise allert, raise error" | ||||
| Tuple | "kingdom='Animalia', year of description < 1758" | "Zoological names start from 1758, with the 10th edition of the 'Systema Naturae' of Linnaeus" | Raise alert | |||
| Tuple | [need to deal with Bacteria here: need to be published in the bacterial names database before they can be valid; start of bacterial nomenclature: 1973 or something; all earlier names can not be valid] | |||||
| Entity | combination of taxonomic name and author is not unique | "There are very few exceptions to this rule; not sure it is, at this point, worth complicating our systems to accommodate them" | Raise error | |||
| Inter-entity | rank of parent is lower or equal to rank of child | "For example, species declared parent of genus" | Raise error | |||
| Inter-entity | parent-child relationship jumps mandatory ranks | needs the 'ranks' tables with mandatory and optional parents as in WoRMS; needs list of exceptions | "For example, a family is declared as the parent of a species; genus is a mandatory rank. Exception for some high-level taxa with very little descedants, such as phylum Phoronida (these are formally listed and recognised with the Code for Zoological Nomenclature)" | Check for exception | ||
| Inter-entity | "parent-child relationship jumps mandatory ranks, and parent is not listed as one of the exception cases" | Raise alert | ||||
| Inter-entity | siblings have different rank | All children of the same parent have to have the same rank; exception when parent name is '<name> incertae sedis' | Raise alert | |||
| Inter-entity | invalid parent for a valid taxon | A valid taxon can not be a child of an invalid parent; all descendants of an invalid taxon should also be invalid | Raise alert | |||
| Inter-entity | taxon points to invalid taxon as its valid taxon | "Needs to result in error, not only alert - danger of 'circular synonyms' (record A points to B points to C points to A), resulting in infinite loops when looking for a valid name for a taxon" | "Only valid taxa can be declared 'valid' taxon for another (invalid) taxon. Alternative, for a more complex system, is to keep the complete 'audit trail' of changes of synonymy. So, if taxon A was first (in history) declared a synonym of taxon B, then taxon B declared a synonym of taxon C: record A points to record B as valid taxon, not to record C; this ay we can follow the history of a name (which often makes it easier to understand why a particular change has taken place). In OBIS now: record A would point to record C. In WoRMS: not consistent (at least not last time I looked); the original intention was, following our example again, to store C as valid taxon for A, not B." | Raise error | ||
| Inter-entity | "A species is listed under different subgenera, and/or both with and without subgenus, and there is more than one of these combinations that is listed as valid" | "A specific epitheton has to be unique within a genus; if it occurs in combination with different subgenera, or once with, once without subgenus, only one of those combinations can be valid" | Raise alert | |||
| Inter-entity | "A name is listed both as a genus and a subgenus, and the parent of the subgenus is not the genus, and both are listed as valid" | "E.g. subgenus 'Oneus (Twous)' and genus 'Twous' are both listed as valid. WoRMS is further normalised than OBIS, and stores the name parts (genus name, specific and infra-specific epithetons) in atomised fields. The complete name is calculated from these name parts. In OBIS, only the full name is stored, raising the possibility of creating descendant names that conflict with the ancestors." | Raise error | |||
| Inter-entity | species or infra-specific name conflicts with name of parent | E.g. listed parent name is 'Oneus' but species name is 'Twous three' | Raise error | |||
| Tuple | rank>genus and name!~'^[A-Z][a-z]+( incertae sedis)?$' | "A supra-generic name is a single word ('uninomen'), starting with a capital, followed by an optional phrase 'incertae sedis'" | ||||
| Inter-entity | name~'^[A-Z][a-z]+( incertae sedis)?$' and no other taxa points at this taxon as their parent | "A 'Taxon incertae sedis' is not a real taxon, but a container for all names within 'Taxon' that have no clear position in the classification. " | flag for deletion | |||
| Tuple | rank=genus and kingdon='Animalia' and name!~'^[A-Z][a-z]+$' | "A genus name is a single word, starting with a capital; there are very few names that have other than a-z. Have to look up the exceptions" | ||||
| Tuple | rank=genus and kingdon!='Animalia' and name!~'^[A-Z][a-z-]+$' | "A genus name is a single word, starting with a capital; there are very few names that have other than a-z. In other than Animalia, also a hyphen is valid character" | ||||
| Tuple | rank=species and kingdom='Animalia' and name!~'^[A-Z][a-z]+( [(][A-Z][a-z]*[)])? [a-z]+$' | "A species name consists of two words: a genus name followed by a specific epitheton; an epitheton starts with a lower case letter. Between the genus name and the specific epitheton is an optional subgenus name, enclosed in '(', ')'; a subgenus name starts with an upper case letter" | ||||
| Tuple | rank=species and kingdom!='Animalia' and name!~'^[A-Z][a-z-]+( [(][A-Z][a-z-]*[)])? [a-z-]+$' | Allow hyphens for taxa hat aren't animals | ||||
| Tuple | rank=family and kingdom='Animalia' and name!~'idea$' | Same for other standard endings of animal taxon names | ||||
| Tuple | rank=family and kingdom='Plantae' and name!~'aceae$' | "A few well-known exceptions, such as 'Compositae'. These exceptions are deprecated, and properly-formed alternatives exist for all of them, but last time I checked their use was still alowed, and they definitely have to be included in the database (but pointing at the properly-formed alternative as the valid name)." | Check for exception | |||
| Tuple | rank<subspecies and kingdom='Animalia' and taxon listed as valid | Zoological nomenclature no longer recognises ranks below subspecies | ||||
| Tuple | rank=subspecies and kingdom='Animalia' and name!~'^species name [a-z]$' | "An infraspecific epitheton starts with a lower case; in zoological names, the infraspecific indicator is not written, as this can only be a subspecies" | ||||
| Tuple | rank=subspecies and kingdom!='Animalia' and name!~'^species name (subsp[.]|ssp) [a-z]$' | "An infraspecific epitheton starts with a lower case; for other than animal names, the rank is indicated by writing either 'subsp.' or 'ssp'. Similar rules for variety, subvariety, forma, subforma" | ||||
| Attr | "specific epitheton is adjective, and gender doesn't correspond with gender of genus" | "Needs taxonomic dictionary to find out whether epitheton is adjective or noun, and gender of genus; and if epitheton is adjective, stem + male/female/neutral forms" | ||||
| Attr | "specific epitheton is noun in apposition, but the ending of the epitheton has been mangled to correspond with the gender of the genus" | Raise alert | ||||
| Inter-entity | "two or more names have identical epithetons (with possible exception of gender-dependent ending) and identical non-null authority (with the possible exception of a pair of '(', ')' surrounding the authority, or brackets followed by an extra author string), and more than one of these names are listed as valid " | "Caused by new combinations between species and genus, that were not caught in the database" | Increase probability that these are undetected synonyms | |||
| Inter-entity | "two or more names have identical epithetons (with possible exception of gender-dependent ending) and similar non-null authority (with the possible exception of a pair of '(', ')' surrounding the authority, or brackets followed by an extra author string), or null authority, within the same say phylum, and more than one of these names are listed as valid " | Need to use higher taxon (phylum in the exaple) as a parameter in a comparison tool | "Same but with uncertainty on the author; the closer the the two names are in he classification, the more likely that the two names are synonyms" | Increase probability that these are undetected synonyms | ||
| Inter-entity | "two or more names of rank species or below differ only in the gender-dependent ending of the epitheton, and do not have the same valid name" | Increase probability that these are undetected spelling variations | ||||
| Inter-entity | "two or more names of rank species or below differ only in the way a genitive is formed from the non-latin stem, and do not have the same valid name" | "male genitives 'i' or 'ii', or female 'ae' or 'iae' are likely to be the same; also quite often the gender itself is wrong so that all 4 can be treated as likely spelling variations. In case of native Latin names, there is usually no doubt whether the ending should have the extra i: Macrus -> Marci; Antonius -> Antonii; Maria -> Mariae; Margareta -> Margaretae. When non-latin names are latinised, this can be done by appending either -us or -ius, leading to either -i or -ii in the genitive. " | Increase probability that these are undetected spelling variations | |||
| Inter-entity | "two names are listed as synonyms, but the rank of the lower ancestor is too high" | "Can happen because of silly mistakes, database errors caused by homonymous genus names… Exceptions caused by synonymous high taxa (especially for protozoans)" | Check for exception | |||
| Inter-entity | "two or more names are identical if all 'y's are replaced with 'i's, and do not have the same valid name" | "Same for other substitution rules to implement fuzzy matching a la Rees: {sck}, {h}…" | "Catch a number of very common sources of spelling mistakes, specific for taxonomyc names" | Increase probability that these are undetected spelling variations | ||
| Inter-entity | "the Levenshtein distance between two names is lower than a treshold, and names do not have the same valid name" | treshold a function of the length of the names | Catch general misspellings | Increase probability that these are undetected spelling variations; increase inversely propostional with the distance | ||
| Attr | name ~* ' [(]lpil[)]$' | OBIS-specific (and duplicated in GBIF) | "lpil' stands for 'Lowest Possible Identification Level', is not part of the name, and does not convey any information that is not contained in the name itself; should be stripped from the name" | strip ' (lpil)' from name | ||
| Attr | [Aa]ff[.]?) ' | OBIS-specific (and duplicated in GBIF) | """'Cfr' and 'Aff' are not parts of the name proper, but a judgement on the reliability or the certainty of the identification; in OBI, such names are stripped from the qualifier and reduced in precision to the first reliable taxonomic name. E.g. species 'Oneus cfr twous' would be reduced to genus 'Oneus'" | aff[.]?) ' from name; reduce to higher taxonomic level if needed (sacrificing precision for accuracy) | ||
| Attr | "species' name has two or more epithetons, often separated by '-' or '/'" | OBIS-specific (and duplicated in GBIF) | Person identifying the specimen wasn't able to make his mind up between several different possible identifications | strip extra information from name; reduce to higher taxonomic level if needed (sacrificing precision for accuracy) | ||
| Attr | taxonomic name has extra copy of authority | OBIS-specific (and duplicated in GBIF) | name~*authority'$' | "The OBIS 'scientific name' field, according to its definition, des not include the authority; several data providers have been reporting scientific name with authority included, and these cases where not always caught before the names went itno the taxonomic tables" | strip extra information from name | |
| Attr | taxonomic name has information on gender or life stage of the specimens | OBIS-specific (and duplicated in GBIF) | needs vocabulary of 'gender' and 'lifestage' to recognise the terminology in the candidate names | strip extra information and store in separate fields |
