Difference between revisions of "R algorithm integration with Statistical Manager"
From D4Science Wiki
(→Hypothesis and Thesis) |
|||
| Line 1: | Line 1: | ||
== Hypothesis and Thesis == | == Hypothesis and Thesis == | ||
| − | This | + | This experiment is performed by FAO in order to test and assess how data managers / developers can plug easily algorithms (especially R algorithms) in the infrastructure, through the Statistical Manager tool, and respond quickly to data analysis needs while benefiting of iMarine computing resources. |
| − | The product of this experiment is a basic service that allows to convert a SDMX dataset, provided through a SDMX service URL, to the CSV format. | + | The product of this experiment is a basic service that allows to convert a SDMX dataset, provided through a SDMX service URL, to the CSV format. Other similar experiments will handle more complex algorithms, and will complete the outcomes of this first basic experiment |
| − | The | + | The scope of these algorithm integration experiments is: |
| − | * to assess how a data manager / developer can plug an algorithm by their own, | + | * developer/algorithm integrator oriented |
| − | * to identify potential improvements | + | ** to assess how a data manager / developer can plug an algorithm by their own, |
| + | ** to identify potential improvements for the ease, speed and sustainability of the R algorithm integration procedure | ||
| + | * end-user oriented | ||
| + | ** to assess user friendliness of the Statistical Manager data analysis tool | ||
== Outcome == | == Outcome == | ||
Revision as of 17:15, 19 June 2014
Hypothesis and Thesis
This experiment is performed by FAO in order to test and assess how data managers / developers can plug easily algorithms (especially R algorithms) in the infrastructure, through the Statistical Manager tool, and respond quickly to data analysis needs while benefiting of iMarine computing resources.
The product of this experiment is a basic service that allows to convert a SDMX dataset, provided through a SDMX service URL, to the CSV format. Other similar experiments will handle more complex algorithms, and will complete the outcomes of this first basic experiment
The scope of these algorithm integration experiments is:
- developer/algorithm integrator oriented
- to assess how a data manager / developer can plug an algorithm by their own,
- to identify potential improvements for the ease, speed and sustainability of the R algorithm integration procedure
- end-user oriented
- to assess user friendliness of the Statistical Manager data analysis tool
Outcome
The results of this experiment that the procedure of integrating R scripts as algorithms is a quick, straightforward and sustainable to be considered by institutions that wish to plug data analysis algorithms. The benefits are the following:
- The e-infrastructure, by means of the Statistical Manager, provides a fast, straightforward, well-documented and sustainable procedure of algorithm integration, highlightly recommended for institutions
- In term of software tools & programming language, some basic knowledge is required:
- basic knowledge in Java programming is required.
- knowledge of an IDE (e.g. Eclipse) and SVN is recommended
- Additional knowledge of Maven is optional, but required only if data managers intend to build a separate Java project to deliver the algorithms (as done in this exercice).
- Through this procedure, the e-infrastructure offers a powerful tool to institutions, especially research institutions, to expose scripts (often scattered among offices) to be exposed as web-services, and make benefits of the e-infrastructure computing resources
Activity Workflow
- The activity was done by familiarizing with the Statistical Manager, relying both on the documentation and a tutorial video made available to facilitate the integration of algorithms.
- A basic R script was created to test the Statistical Manager. This script allows to convert a SDMX-ML dataset to CSV.
- In order to integrate the R script, a separate Java Maven project was created (with the aim to add further algorithm later).
- Few exchange with the Statistical Managers developers was required for the project settings, an highlighted some few scatter in the documentation
- The R script was integrated in the project, tested and sent to Statistical Manager team for its deployment
- Additional exchange with the team took place, to have some clarifications on:
- algorithms inputs (difference between a File input and remote resource - URL - input)
- the need for data managers to indicate the eventual R package dependencies to install prior to the algorithm deployment
- how to add the algorithm within a given category of algorithms (for display purpose in the Statistical Manager user interface)
- The algorithm was successfully deployed and is currently operational in the development portal, and usable in the rich user interface of the Statistical Manager.
Conclusion
TBD
Recommendations & future developments
TBD
Experimentation
TBD
