Difference between revisions of "Procedure Infrastructure upgrade"

Latest revision as of 16:42, 2 September 2015

Preparation of the upgrade plan

This page Upgrade page contains the list of resources upgrades performed so far on the D4ScienceAn e-Infrastructure operated by the D4Science.org initiative. infrastructure. The different categories of resources are listed for the D4science VOs + a dedicated section for the other interventions is present ( which included VOs dismissal and upgrades on the EUBrazilOpenBio VOVirtual Organization;). The upgrades described in this section of the wiki are the ones related to gCube software: the resources for UMD/gLite middleware are under decommission, while the Third Party services upgrades are discussed case by case by the Service Activity Team.

When dealing with whatever type of upgrade on the infrastructure we should make sure that a dedicated wiki page is created listing the different actions to be performed on each resource involved. In case the upgrade then lead to disruption of the infrastructure services, the Users should be notified with Downtime notification using the [usermanagement portlet] for downtime of the whole infra, or the specific portlet at level of VOVirtual Organization; or VREVirtual Research Environment. in case the downtime affects only a subset of the infrastructure services; the related Downtime then should be tracked on the Downtimes wiki page. More info for related to this procedure are available at the dedicate wiki page. Usually we tend to be less intrusive as possible when doing an upgrade, but most of the time ( expecially in the case of the portlal upgrades) a period of downtime is always needed.

In order to keep trace of the different gCube resources deployed on the production infrastructure we use the wiki . The gCube Resources page infact lists the different nodes and container installed, with the VOVirtual Organization;/VREs associated and the services deployed.

gCube upgrades

As said we will focus on the upgrades involving gCube software. Usually we tend to split the upgrades by scopes: given that the infrastructure is made of a scope hierarchy and resources are associated to different scopes. Therefore on the upgrade page related to gCube (http://wiki.d4science.org/index.php/Resources_Upgrade#gCube_Upgrade), the different scopes are listed in separate wiki pages. In addition a separate page is dedicated to the infrastructure portals.

For each scope the different activities to be applied are listed; the main categories of activities, that are then explained in details in the next section, are:

• Containers
• gCube Services
• Portal components

Upgrade Activities

When dealing with an upgrade in gCube we have different categories of components to analyze:

• Containers
  • gHN
  • Smartgears
• gCube Services
  • gCF based
  • Smargears based
• Portal and Portlets

Containers upgrade

The infrastructure is equipped with 2 types of containers :

• gHN : customisation of a Globus WS core 4.0.4 container
• SmartGears : a set of extension to any Servlet based container to enable registration and activation on the infra of webapps ( at the moment only tomcat is installed)

The main documentation regarding the containers is the gCube Admin guide:

• gHN : https://gcore.wiki.gcube-system.org/gCube/index.php/Administrator_Guide
• Smartgears: https://gcube.wiki.gcube-system.org/gcube/index.php/SmartGears_gHN_Installation

gHN

The gHN unfortunately implements an old flat classloader schema, which means that common and application libs are all shared and stored in the same folder. When a new version of the gHN is available we should understand which changes have been applied. The typical changes are:

• a new version of the libraries part of the gHN
• a new version of the web services part of the gHN
• a change on the configuration of the GHNgCube Hosting Node.

At this level of maturity the gHN upgrades difficultly introduce not backward compatibilities, which should make life easier..

In order to upgrade an existing installation we have been always using 2 type of scripts:

• upgrade script: The script downloads the new version of the gHN from Maven, removes the old libraries and copies the new ones by keeping the existing gCube Services installed. He modifies also the existing configuration if needed. In case one of the existing base gCube services ( Deployer, GHNgCube Hosting Node. Manager, ResultSet) needs to be updated it takes care also of undeploy the old version and deploy the new one
• clean script: The script runs a reinstallation of the gHN by keeping only the gCube service state. it can be needed in case of changes of java version ( e.g. move from java 6 built artifacts to java 7) as well

The latest versions of the scripts are available on svn at:

https://svn.research-infrastructures.eu/d4science/gcube/trunk/ghn-distribution/upgrade-scripts/

The rules behind the execution of the upgrade or the clean script in a running installation are :

• if the gHN is empty ( the gHN is not running any other web service other than the basic gCube ones) it's always better to run a clean script
• if most of the libraries part of the gHN, the gCube services installed and their deps have changes better to run a clean script
• if only part of the the libraries or the conf of the gHN has changed, better to always run an upgrade script.

Of course the main difference between the 2 scripts is the result. The upgrade script does not require any other action from the site manager, while the clean script requires that the gCube services previously installed on the node have to be reinstalled. ( see section on service upgrade)

SmartGears

In the case of SmartGears containers we are in front of the latest generation servlet containers, which implement separate classloaders for common and each applications and in most of the cases the upgrade is a really simple procedure.

The upgrade is a combination of 2 scripts:

• Download scripts : https://gcube.wiki.gcube-system.org/gcube/index.php/SmartGears_gHN_Installation#Upgrade, which downloads the version to install from Nexus
• Install script : https://gcube.wiki.gcube-system.org/gcube/index.php/SmartGears_gHN_Installation#Installation, which first removes the previous installed components if present.

gCube service upgrade

The upgrade of gCube Services depends on the nature of the services and the technology used for their development

gCF based

gCube Services gCF based are the oldest categories of Web ServiceSelf-contained, self-describing, modular application that can be published, located, and invoked across the Web. Web services perform functions that can be anything from simple requests to complicated business processes. Once a Web service is deployed, other applications (and other Web services) can discover and invoke the deployed service. developed in gCube: they are developed using the gCF ( gCore Framework) , they run in a gHN and they are packaged in a bundle called gar archive (i.e. deployer-service.gar). Also in this case we have 2 possible gar archives:

• Normal gar archive : which contains the service implementation, the config and the wsdl
• Full gar archive: which in addition contains all the service runtime dependencies.

The different service packaging is required because it reflects the different nature of the gCube services and their deployment model

Core services

The Core Service also named Enabling layer services https://gcube.wiki.gcube-system.org/gcube/index.php/Core_Services_Installation) are the minimal set of gCube Services needed to setup and manage a gCube infrastructure. Due to their nature and deployment scenario they should be installed /upgraded manually. Given that a manual installation is requested the Full gar archive is preferable cause it install together with a service the needed dependencies. Please note that not all the Core Services have a Full Gar cause this has been introduced when moving to Maven, and not all of them have been ported yet. In case the where the Full Gar is not available the service dependencies are listed on the Admin guide. In the case of manual upgrade the following actions should be performed:

Container stop:

$GLOBUS_LOCATION/bin/gcore-stop-container

Old service undeployment :

$GLOBUS_LOCATION/bin/gcore-undeploy-service ServiceName

New Service deployment:

$GLOBUS_LOCATION/bin/gcore-deploy-service ServiceName-full.gar

Please note that this operation removes together with the service implementation and dependencies also the configuration files ( among them the important jndi-config.xml). So in case of upgrades they should be recollected from the backup ( a GHNgCube Hosting Node. backup is always performed by the upgrade script)

Application services

The services that implement the Application logic and they are not fundamental in a gCube infrastructure deployment. They are installed using the Dynamic Deployment facilities implement by gCube amd part of the Resource and VREVirtual Research Environment. Management area (https://gcube.wiki.gcube-system.org/gcube/index.php/VRE_Management)

In details each gHN is equipped with a gCube service ( Deployer) which is responsible for the installation of the packages on the node. The Deployer is invoked by another gCube service ( the Resource Manager ) to deploy the packages taken from Maven. In the process another gCube Service is involved ( the Software Gateway ) which is responsible for the dependencies resolution. In order to upgrade a service using the Dynamic deployment, we should make sure that the previous version of the service has been previously removed. This process therefore is always used in junction with an upgrade of the gHN of type clean, that removes the previously installed service and its dependencies.

The upgrade using the Dynamic Deployment facilities can be performed directly from the Portal trough graphical interface.( LINK TO INCLUDE)

SmartGears based

As said SmartGears based services are JAVA JAXWS or JAXRS service deployed on a servlet based container ( Tomcat at the moment). They are packaged as a web application archive ( war) therefore each archive includes by default the service implementation and its dependencies. At the moment no gCube Core service has been deployed using this technology therefore only Application services are present in this category.

There are two ways of upgrading an existing gCube Service Smartgears based;

• Using the Tomcat Manager application ( where is enabled )
• Manually removing the previous version of the service and placing the new war under the $CATALINA_HOME/webapps folder.

The upgrade of a gCube Service SmartGears based can be performed exploiting the underlying container facilities of hot deployment, but this feature is not suggested in a production environment due to Memory Leaks. Therefore a container restart is always recommended.

Portal upgrade

The Infrastructure portals are the gateways to the infrastructure for most of the users therefore their upgrade is quite a crucial activity cause any issue is immediately visible to the users. When planning an upgrade of the portals there are 3 groups of components to take into account:

• Portal Bundle ( containing the base software and configuration)
• Common libraries
• Portlets and servlets

more details on the portal are available at: https://gcube.wiki.gcube-system.org/gcube/index.php/GCube_Portal_Installation

Portal Bundle

The portal bundle contains the base software and configuration to run a gCube portal. It's composed by a Liferay Portal ( based on tomcat), a gHN and some other base gCube components. The current dependencies towards the gHN components are planned to be dropped in future gCube releases, but at the moment they are still present. The base layout of the portal bundle is:

• PortalBundle
  • tomcat
    • conf
    • webapps
      • ROOT
  • gCore
    • lib
    • config

The configuration is splitted among different places , gCore/config, tomcat/webapps/ROOT, tomcat/conf, so in case of configuration changes during an upgrade it's important to include in the dedicated wiki all the files that have to be modified.

For the upgrade and configuration of the gHN libraries the same script described in the previous section is used. While individual changes on the base webapps are installed manually.

Common Libraries

The common libraries are gCube or third party libraries that are not part of the gHN libraries but are dependencies of portal components and deployed on the common classpath of the portal. They are included inside the gCore/lib folder with the following directory layout

• gCore/lib : contains the gcore libraries as described in the previous section
  • _application-support-layer : contains the ASL libs and their dependencies
  • _home_-ibrary : contains the Home library libs and their dependencies
  • _social-library : contains the social Library libs and their deps
  • _ext : contains any other gCube and third party dependencies not included in the Portlet war.

In the case of an upgrade the replacement of the common libraries should be listed in the wiki page, by grouping them using the above layout.

Portlets and servlets

The Portlets and servlets components are deployed under the PortalBundle/tomcat/webapps folder. Apart from the technology these 2 groups of components differ for the deployment point of view as well:

• portlets: they are deployed exploiting the facilities offered by Liferay, the war should be dropped within the PortalBundle/deploy folder.
• servlets: they are threated as any other webapp deployed into tomcat so the war should be installed under the PortalBundle/tomcat/webapps folder.

The upgrade of a portlet/servlet simply includes the steps to remove the previous deployed webapp from the PortalBundle/tomcat/webapps folder ( by removing the portlet/servlet folder ) and then deploy the new versions as described above.

In case of a portlet we should also take into account the upgrade of the portal layouts( for the VOVirtual Organization; and VREs) . The layouts of the different VOVirtual Organization; and VREs infact are stored in the portal DB and when the new deployed portlet changes its identifier we should make sure to upgrade this relation. This can be done:

• manually upgrading the layout by re-adding the new version of the portlet to the layout
• modify the DB to update the reference to the portlet identifier on the layout.