Setting up an NG2 on Ubuntu
From BeSTGRID
This guide is tightly meshed with the Setting up an NG2 guide. It tries to avoid redundancies and refers back frequently to its parent. Please refer to the parent and the LRM-specific pages additionally to this guide.
At time of writing the descriptions are based on the (64 bit) server release of Ubuntu 10.04 LTS (Long Term Support, code name "Lucid Lynx"). It is likely to work equally well with slightly older or newer releases, 32 bit releases and with (some minor) modifications with current Debian releases as well.
Note: Some of these notes may not be 100% in proper chronological order. The order has been retained from the original NG2 install notes to keep them in sync. But it should be quite obvious that certain configuration can only be integrated into VDT once VDT has been installed. Please, keep this in mind when using this guide.
Contents |
[edit] Preliminaries
See also Setting up an NG2#Preliminaries.
[edit] OS requirements
- Scripting shell:
- Many of the VDT supplied scripts are not safe for a POSIX compliant shell, but require the presence of a bash. To circumvent this problem altogether, reconfigure /bin/sh to use bash (rather than the faster dash used by Ubuntu's default).
- See also here: https://wiki.ubuntu.com/DashAsBinSh
- Reconfigure the dash to disable it (say "no" when prompted for the install of dash for /bin/sh)
(Note: The VDT team has been informed, and they will submit an upstream patch to the pacman maintainers to resolve this issue for future releases.)
$ sudo dpkg-reconfigure dash
- SMTP mail relay:
- Configure the host as an SMTP relay,
- Install the package postfix, and configure the host as a "satellite system" using your institution's SMTP server's FQDN as a relay host
- Packages
- Install/configure the NTP daemon (package ntp)
- Install the mailutils package
- Install the xinetd daemon package
- Install the chkconfig package (system tool to enable or disable system services)
- Network configuration requirements
- Use the FQDN (ng2.your.site.domain) in /etc/hostname
- The line in /etc/hosts containing an additional localhost IP (127.0.1.1) must be commented, as it "hurts" GridFTP transfers. Destinations may be indicated to remote hosts as being local.
#127.0.1.1 ng2.your.site.domain ng2
[edit] Preparing the installation
[edit] Adding machine users
Add the machine users for the host, they do not need a login shell, so we'll give them /bin/false:
$ sudo useradd grid-bestgrid -s /bin/false $ sudo useradd grid-admin -s /bin/false
Set up the users' home directory by configuring the host for NFS mounts (and potentially the auto mounter), if these users' home directories are "living" on the NFS.
[edit] Configuring local scheduler access
See also Setting up an NG2#Configuring local scheduler access.
This is dependent on the LRM used. The description given here is for Torque/PBS.
This host (NG2) is to submit computation jobs to the Torque/PBS cluster head node. For this, the Torque/PBS client tools have to be installed, which are then used by the Grid submission tools from VDT.
- Install the torque-client package.
- Edit/enter the PBS head node name in /var/lib/torque/server_name
- Set up ports for PBS in /etc/services (in "Local Services"):
pbs_server 15000/tcp # added for Grid services pbs_dis 15001/tcp # added for Grid services pbs_dis 15001/udp # added for Grid services pbs_mom 15002/tcp # added for Grid services pbs_mom 15003/tcp # added for Grid services pbs_mom 15003/udp # added for Grid services pbs_sched 15004/tcp # added for Grid services
[edit] GridPulse
Get and install the APAC GridPulse system monitor:
- Get the RPM from the ARCS RPM repository, in this case here: http://projects.arcs.org.au/dist/production/5/i386/noarch/APAC-gateway-gridpulse-0.3-4.noarch.rpm
- Convert it with the alien tool to a Debian package (do not use the --scripts option), copy the Debian package to the host and install it.
- Hack the script /usr/local/bin/gridpulse to fit Ubuntu!
This one should work for a start: gridpulse (Note: renamed for upload on the wiki.) - Create a file /usr/local/lib/gridpulse/system_packages.pulse and add the following line to it:
apac-gateway-gridpulse
- Add further installed RPM packages during the course of this guide to it. For this full guide the list will look like this:
apac-gateway-gridpulse ggateway pbs-telltail apac-mip apac-mip-module-py apac-mip-globus apac-mip-module-py-config apac-glue-schema
- Add a crontab entry for executing the script every 20 minutes:
3,23,43 * * * * /usr/local/bin/gridpulse grid_pulse@lists.arcs.org.au >/dev/null 2>&1
[edit] Ggateway
Get and install the Ggateway package:
- Get the RPM from here: http://projects.arcs.org.au/dist/production/5/i386/noarch/Ggateway-1.0.2-2.noarch.rpm
- Convert it with the alien tool to a Debian package (do not use the --scripts option), copy the Debian package to the host and install it.
- Copy the APAC job manager gateway script into the VDT setup
$ sudo cp /usr/local/src/pbs.pm.APAC /opt/vdt/globus/lib/perl/Globus/GRAM/JobManager/pbs.pm
- Edit the script /opt/vdt/globus/lib/perl/Globus/GRAM/JobManager/pbs.pm to suit the local setup (particularly the mpiexec invocation and the paths to the PBS "q*" tools, which are under /usr/bin for the Ubuntu packages):
BEGIN
{
$mpiexec = 'mpiexec';
$qsub = '/usr/bin/qsub';
$qstat = '/usr/bin/qstat';
$qdel = '/usr/bin/qdel';
$cluster = 1;
$cpu_per_node = 1;
$remote_shell = '/usr/bin/ssh';
}
- Some further configuration for PBS may be necessary here. See: Setting up an NG2/PBS specific parts
- Add the installed ggateway packages to a line in the gridpulse checker configuration (/usr/local/lib/gridpulse/system_packages.pulse)
[edit] Configure LRM log replication from LRM server to the NG2
See also Setting up an NG2#Configure LRM log replication from LRM server to the NG2.
[edit] Configuring local scheduler access
See also Setting up an NG2#Configuring local scheduler access.
To link up the Torque/PBS log output to the local NG2, a "log sender" (pbs-telltail) on the Torque/PBS head node is required to pick up the logs and "send" them, and on the NG2 a local "log maker" (pbs-logmaker) is required.
- Get the RPM from the ARCS RPM repository: http://projects.arcs.org.au/dist/production/5/i386/noarch/pbs-telltail-1.0.2-8.noarch.rpm
- Convert it with the alien tool to a Debian package (do not use the --scripts option), copy the Debian package to the NG2.
- On the PBS head node:
- Install the pbs-telltail package.
- Configure/enable the pbs-telltail daemon (edit REMOTES in /etc/init.d/pbs-telltail to enable the (additional) NG2).
- On the NG2 gateway:
- Install the pbs-telltail package.
- Copy this init script to /etc/init.d/: pbslogmaker (Note: renamed to Pbs-logmaker.sh for upload to this wiki, rename to pbs-logmaker.)
- Enable it for automatic start/shutdown with the system, and start it directly:
$ sudo update-rc.d pbs-logmaker defaults $ sudo service pbs-logmaker start
- Remove the package provided init script /etc/rc.d/init.d/pbs-logmaker on Debian/Ubuntu systems.
Note: Using the Ubuntu Torque/PBS install, the PBS log is located under /usr/spool/pbs/server_logs.
(The CentOS based BeSTGRID documentation uses /usr/spool/PBS/server_logs.)
[edit] Installing VDT
See also Setting up an NG2#Installing VDT.
Most of these steps are much more easily performed with a root shell. To obtain one use the following:
$ sudo su -
Following steps using a root shell use a preceding shell prompt "#".
- If your institution requires a proxy, configure the environment (change host and port accordingly).
- It might also make sense to add this to a (newly created) script under /etc/profile.d/proxy.sh to manifest it for all future login shells.
# export http_proxy=http://institute.proxy.server.ac.nz:8080
- Create a new base directory for VDT:
# mkdir -p /opt/vdt
- Download and setup pacman (the packager used)
# cd /opt/vdt # wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz # tar xfz pacman-latest.tar.gz # cd pacman-3.29 && source setup.sh && cd ..
- Install Grid tools from VDT
(Note: It is very important to use the -pretend-platform switch at the first usage of pacman. For Ubuntu Karmic (9.10) and Lucid (10.04) "Debian-5" worked well.)
# cd /opt/vdt # export VDTMIRROR=http://vdt.cs.wisc.edu/vdt_200_cache # pacman -pretend-platform Debian-5 -get $VDTMIRROR:Globus-WS $VDTMIRROR:PRIMA-GT4 $VDTMIRROR:Globus-WS-PBS-Setup # # To also get the recommended packages for the install: # pacman -get $VDTMIRROR:VOMS-Client $VDTMIRROR:MyProxy-Client $VDTMIRROR:UberFTP $VDTMIRROR:Globus-Base-SDK $VDTMIRROR:GSIOpenSSH
- Make the environment variable setup script created by VDT load in the default profile
# ln -s /opt/vdt/setup.sh /etc/profile.d/vdt.sh # . /etc/profile
- Fix a problem with the MySQL/Globus-WS setup:
- You will have to enable the VDT services for the init.d scripts to show up as per [Setting_up_an_NG2#Start_VDT_services], then go back and do the steps missed?
- Move the init scripts mysql5 and globus-ws from /etc/rc.d/init.d/ to the "proper" location in /etc/init.d/
- Enable the linking of the script into the init process using update-rc.d (the fields "20" and "25" are important for starting order of the services), then start them
# update-rc.d mysql5 defaults 20 # update-rc.d globus-ws defaults 25 # service mysql5 start # service globus-ws start
- Create an xinetd startup file for gsiftp /etc/xinetd.d/gsiftp:
service gsiftp
{
socket_type = stream
protocol = tcp
wait = no
user = root
instances = UNLIMITED
cps = 400 10
server = /opt/vdt/vdt/services/vdt-run-gsiftp.sh
disable = no
}
[edit] Post-install configuration
Follow all steps according to Setting up an NG2#Post-install configuration.
[edit] Setup job reporting
Follow all steps according to Setting up an NG2#Setup job reporting.
[edit] Setup MDS/MIP
See also Setting up an NG2#Setup MDS/MIP.
The steps described here approach the configuration of the MIP script in a slightly different manner than the NG2 install guide. Here, we are following the steps given in the ARCS documentation, by installing the APAC-mip-module-py-config package, and then configuring a "master configuration Perl script" (/usr/local/mip/config/apac_py/mip-config.pl). The complete MIP configuration then should be generated through this script without manual touchups necessary. It has worked, but the approach seems a bit "filthy" to me. But hey, you make a choice and try it out. Here is the link to the ARCS documentation in question: https://projects.arcs.org.au/trac/systems/wiki/InfoSystems/InstallConfigSteps
[edit] Install/Configure
- Install the python-lxml Ubuntu package.
- Get the RPM packages APAC-mip, APAC-mip-module-py, APAC-mip-globus, APAC-mip-module-py-config and APAC-glue-schema in their latest version from:
http://projects.arcs.org.au/dist/production/5/i386/noarch/ - Convert them with the alien tool to a Debian package (do not use the --scripts option), copy the Debian package to the host and install them.
Packages: apac-mip, apac-mip-module-py, apac-mip-globus and apac-glue-schema - Add the installed packages to a line in the gridpulse checker configuration (one per line, use the Debian package name): /usr/local/lib/gridpulse/system_packages.pulse
- Correct the script /usr/local/mip/mip according to the following diff:
--- mip~ 2008-01-24 18:50:12.000000000 +1300
+++ mip 2010-05-25 16:00:58.789952820 +1200
@@ -1,12 +1,12 @@
#!/bin/bash
LANG=C
-. /usr/local/osg/setup.sh
-cd /home/eshook/Projects/MIP/mip
+export PYTHONPATH="/usr/local/mip/modules/apac_py:$PYTHONPATH"
+cd /usr/local/mip
if [ ! -z "$1" ]; then
if [ "$1" == "-remote" ]; then
- ./mip-remote.pl /home/eshook/Projects/MIP/mip/config
+ ./mip-remote.pl /usr/local/mip/config
elif [ "$1" == "-int" -o "$1" == "-integrator" ]; then
- ./integrator.pl /home/eshook/Projects/MIP/mip/config
+ ./integrator.pl /usr/local/mip/config
else
./mip.pl $1
fi
- Generate and edit a MIP configuration file (see Setting up an NG2#Configuring MIP for further information on this):
- Check the MIP generated configuration file for the requirements /usr/local/mip/config/apac_config.py, and make sure the listed element names in /usr/local/mip/config/default.pl are matching!
- Continue configuring as described.
- Before testing, define the "default" location (with a sym link) and install MIP using the provided install script:
$ cd /usr/local/mip/modules $ sudo ln -s apac_py default $ cd .. $ sudo ./install_mip
- Test the install, see Setting up an NG2#Testing and finishing up
- Also validate the output (this requires the libxml2-utils package for XML validation on the system, it may need to be installed)
$ sudo /usr/local/mip/config/globus/mip-exec.sh -validate
- If the Python warning (/usr/local/mip/modules/default/ComputingElement/computingelement.py:284: DeprecationWarning: the sets module is deprecated) is disturbing you, modify the file /usr/local/mip/modules/default/ComputingElement/computingelement.py according to this diff (do not forget to delete any editor's backup files, e. g. computingelement.py~):
--- computingelement.py~ 2008-07-31 13:12:57.000000000 +1200
+++ computingelement.py 2010-06-28 13:51:55.752474535 +1200
@@ -281,10 +281,8 @@
print "</VOView>"
- import sets
-
print "<ACL>"
- for rule in sets.Set(ce.ACL):
+ for rule in set(ce.ACL):
print "\t<Rule>%s</Rule>" % rule
print "</ACL>"
- For the step Setting up an NG2#Activating MDS in Globus, the package APAC-mip-globus has been installed already (see above). So skip the "yum install ..." step, and proceed straight to continue with the steps starting with "Secure your MDS".
[edit] Integrating MIP for Globus
See also these two resources:
The "boiled down" instructions:
- In this case of the stream of these instructions $GLOBUS_LOCATION is /opt/vdt/globus
- Activate the Globus integration of MIP
$ sudo /usr/local/mip/config/globus/mip-globus-config -l /opt/vdt/globus install
- Restart the Globus service
[edit] Testing MDS
- Initialise a Grid proxy certificate and request the resource's service entries
$ grid-proxy-init $ wsrf-query -s https://ng2.your.site.domain:8443/wsrf/services/DefaultIndexService "//*[local-name()='Site']"
[edit] Polishing Globus
Do not proceed with the instructions of this step (see Setting up an NG2#Polishing Globus), as this issue has been taken care of already by choosing different starting indices (mysql5 at "20" and globus-ws at "25") when installing VDT above.
[edit] Making local cluster accounts ready for the grid
Proceed as stated in Setting up an NG2#Making local cluster accounts ready for the grid.
[edit] Testing
Proceed as stated in Setting up an NG2#Testing.
