Setup NG2 at University of Canterbury
From BeSTGRID
NG2 is a job submission gateway based on Globus Tookit 4.x. At the University of Canterbury, NG2 submits jobs only to the BeSTGRID prototype cluster, while Vladimir:Setup NG2HPC and NG2SGE allow to submit jobs to more powerful resources. The NG2 was setup as a gateway prototype, and also works as the MIP integrator for the other gateways.
|
NOTE: This page is a historic relict and is not up to date. If you are looking for instructions on how to setup a job submission gateway for BeSTGRID, please see the Setting up an NG2 page instead. |
Contents |
[edit] Preliminary setup
[edit] Install LAM/MPI
- Installed to allow code to compile (... if compile job submitted to Fork port)
- Will be replaced by MPICH2 (already installed on ngcompute)
http_proxy=http://gridws1:3128 yum install lam
[edit] Install PBS client
- Installed in the same way as described in grid
- Should have instead installed torque-client from APAC-Repo.
[edit] Setup client NFS
- Shared home directories mounted from grid.
- as described in ngcompute)
[edit] APACGrid NG2 Setup
- Follow http://www.vpac.org/twiki/bin/view/APACgrid/VmdetailsNg2
- Install basic APACGrid packages:
yum install Gbuild Gpulse
- Install PBS client
- APACGrid recommends yum install Gtorque-client (would use /usr/spool/PBS)
- I installed torque from source code with /var/spool/torque. This causes some difficulties - pbs-telltail and pbs-logmaker startup scripts have to be adjusted accordingly. Apart from that, everything runs fine - and globus reads the server logs from the directory identified by $PBS_HOME.
[edit] Main NG2 Setup
- The main part of the install starts with
/usr/local/sbin/BuildNg2Vdt161.sh
- Precisely, I did not run the script but instead executed it step-by-step on the command-line.
- The only real difference is that I do not have a GUMS server yet, and I would have to comment the PRIMA initialization out of the script.
- In the end, there were several steps I had to do manually to start the container:
- Install certificate into /etc/grid-security/host{cert,key}.pem owned by root
- Copy the certificate also into /etc/grid-security/container{cert,key}.pem owned by daemon:daemon
- pacman -pretend-platform linux-rhel-4 -get http://vdt.cs.wisc.edu/vdt_161_cache:EDG-Make-Gridmap
- create empty /opt/vdt/edg/etc/grid-mapfile-local (or from template /opt/vdt/edg/etc/grid-mapfile-local.template - actually empty :-) )
- create /opt/vdt/edg/etc/edg-mkgridmap.conf
- group vomss://vomrs.apac.edu.au:8443/voms/APACGrid?/APACGrid/BeSTGRID grid-bestgrid
- group vomss://vomrs.apac.edu.au:8443/voms/APACGrid?/APACGrid/NGAdmin grid-admin
- /opt/vdt/edg/sbin/edg-mkgridmap
- vdt-control --on
- visudo - add the following to the end:
Runas_Alias GLOBUSUSERS = ALL, !root
daemon ALL=(GLOBUSUSERS) \
NOPASSWD: /opt/vdt/globus/libexec/globus-job-manager-script.pl *
daemon ALL=(GLOBUSUSERS) \
NOPASSWD: /opt/vdt/globus/libexec/globus-gram-local-proxy-tool *
- The extra /etc/sudoers lines are necessary. The build script installs a slightly different configuration (for use with PRIMA), and the when Gridmap is used (my setting until I install a GUMS server), the command given to sudo is slightly different and must be explicitly permitted.
- I have also created four user accounts for VOMS mapping. My current system configuration has NFS shared homes among grid (server), ngcompute, and ng2. Password files are however separate - so a user must be created on all these three machines.
# on grid, ngcompute, and ng2: adduser -u 95005 grid-admin adduser -u 95006 grid-user adduser -u 95007 grid-bio adduser -u 95008 grid-bestgrid adduser -u 95039 grid-startup adduser -u 95040 grid-cloud # on just one (create .globus directory and scratch directory within that) for I in grid-admin grid-user grid-bio grid-bestgrid grid-startup grid-cloud ; do mkdir -p /home/$I/.globus/scratch chown -R $I.$I /home/$I/.globus done
[edit] PBS logs
- yum install pbs-telltail
- may report error:
Starting pbs-logmaker Can't access directory: /usr/spool/PBS/server_logs [FAILED]
- The error may be ignored - it is necessary to edit /etc/rc.d/init.d/pbs-logmaker and change the pbs-logmaker argument to the actual PBS_HOME directory.
- may report error:
- Continue with
chkconfig pbs-logmaker on service pbs-logmaker start
On your cluster headnode (ngcompute here), also install pbs-telltail, and copy /usr/local/pbs-telltail/pbs-telltail.RH to /etc/rc.d/init.d/pbs-telltail (edit the PBS_HOME directory in the file as needed). Then:
# on ngcompute chkconfig pbs-logmaker off service pbs-logmaker stop chkconfig pbs-telltail on service pbs-telltail start
[edit] PBS.pm
- yum install Ggateway
- Copy /usr/local/src/pbs.pm.APAC to /opt/vdt/globus/lib/perl/Globus/GRAM/JobManager/pbs.pm
- I did the following modifications:
--- pbs.pm.APAC 2007-04-21 18:15:05.000000000 +1200
+++ pbs.pm.ngcompute 2007-05-10 16:18:56.000000000 +1200
@@ -338,7 +338,12 @@
my $cmd_script ;
my $stdin = $description->stdin();
my $mpiexec_params = ''; # APAC-Specific 'multiple' simplification
- if ($description->jobtype() eq 'multiple') { $mpiexec_params='-comm none ' }
+ ### if ($description->jobtype() eq 'multiple') { $mpiexec_params='-comm none ' }
+ ### disabled by Vladmir: does not work with mpich2 mpiexec
+
+ ### extra by Vladimir: make mpiexec use the right count (mpich2 won't get it from the PBS environment)
+ if ($description->count()>0) { $mpiexec_params .= " -np " . $description->count() . " "; };
+ print JOB "#DEBUG: count=" . $description->count() . " or " . $description->count . "\n";
$cmd_script_name = $self->job_dir() . '/scheduler_pbs_cmd_script';
- pbs.pm.APAC was passing -comm none to mpiexec; this was not understood by MPICH2 mpiexec
- MPICH2 mpiexec does not get the number of nodes from PBS environment; I added explicit -np # parameter.
- Also, APAC pbs.pm has weird behavior when an MPI job has <count> specified but no <hostCount> is given. As my test setup has only 1 node with 4 CPU, I modified pbs.pm to assume hostCount=1 when not specified.
@@ -214,15 +215,21 @@
#print JOB '#PBS -o ', $description->stdout(), "\n";
#print JOB '#PBS -e ', $description->stderr(), "\n";
- if ($description->host_count()==0 || $cluster==0)
+ #### HACK FIXME TODO Vladimir 2007-06-05
+ ## We don't want the APACGrid code below to set nodes=$count
+ ## we have only _one_ node, with 4 CPUs
+
+ $host_count = ( $description->host_count()==0 ) ? 1 : $description->host_count();
+
+ if ($host_count==0 || $cluster==0) ### this will never be called
{ # APAC-specific .. hostCount treatment matches Globus Primer description
($nodes) = sort{$b<=>$a} ( $description->count(), 1 );
print JOB '#PBS -l nodes=', $nodes, "\n";
}
else
{
- print JOB '#PBS -l nodes=', $description->host_count(), ':ppn=',
- myceil($description->count() / $description->host_count()), "\n";
+ print JOB '#PBS -l nodes=', $host_count, ':ppn=',
+ myceil($description->count() / $host_count), "\n";
}
$library_vars{LD_LIBRARY_PATH} = 0;
[edit] Audit capability
- Just run /usr/local/sbin/AddAuditNg2Vdt161.sh
- modifies /opt/vdt/globus/etc/gram-service/jndi-config.xml with connection parameters to audit database
- adds Audit logger configuration to /opt/vdt/globus/container-log4j.properties
- /etc/cron.hourly/auditquery (installed with Ggateway) emails Job-DN mappings to GOC
- Sample email:
From: root <root@ng2.canterbury.ac.nz> Subject: ng2.canterbury.ac.nz JobID 20070515 To: grid_pulse@vpac.org Job-DN: 276.ngcompute.canterbury.ac.nz /C=AU/O=APACGrid/O=BeSTGRID/OU=University of Canterbury/CN=Vladimir Mencl Job-DN: 1b77e5d6-0299-11dc-a003-00163e84b502:8513 /C=AU/O=APACGrid/O=BeSTGRID/OU=University of Canterbury/CN=Vladimir Mencl
Important: In addition, you have to make sure your cluster is reporting the job statistics to GOC.
[edit] Configuring PRIMA
Note: in March 2008, I have installed a GUMS server and switched the gateways to use PRIMA. Due to the whitespace issue, I had to install an updated PRIMA library properly encoding and decoding the whitespace. I have used the one I compiled when helping to solve the problem, and which was installed on the VDT161 instance on Ng2SGE.
After doing that, it was sufficient to turn PRIMA on with:
/opt/vdt/vdt/setup/configure_prima_gt4 --enable --gums-server nggums.canterbury.ac.nz
[edit] Future recommendations
- To configure PRIMA to use a GUMS server (once I have it), the command will be
/opt/vdt/vdt/setup/configure_prima_gt4 --enable --gums-server $Gums_Server
- On any future installs, double check that GLOBUS_TCP_PORT_RANGE is set to 40000-41000 (in /opt/vdt/post-setup/APAC01.sh)
- Globus otherwise uses arbitrary listener ports for credential delegation service instances. This may conflict with a firewall either at your site, or with a client connection firewall at other sites.
- All new users must have their ~/.globus/scratch directory created. Globus automatically creates ~/.globus, but not the scratch directory (${GLOBUS_SCRATCH_DIR})
[edit] Comments
- The original pbs.pm file did not work for me. Pbs.pm created a job script file that:
- treated the job as multijob (not job type specified)
- tried to ssh the command to the nodes listed in $PBS_NODEFILE
- the ssh command sequence had an additional command to store the exit code in a file
- the script tried to read the value from that file
- ssh command failed (Host key verification failed.).
- ng2 (short hostname) is not in /etc/ssh/ssh_known_hosts on ng2
- Anyway, ngcompute does not have host authentication set up to ssh onto itself.
- In the end, the script failed: it read an empty value from the exit-code file, and caused a bash syntax error:
/var/spool/torque/mom_priv/jobs/204.ngcompu.SC: line 53: [: too many arguments
- This issue could be (probably) solved by allowing ssh host authentication from ng2 to ng2
- Not relevant anymore: using APAC pbs.pm
- An observation about PBS:
- qstat reports all jobs only locally
- remotely, qstat reports only jobs owned by the current user
- Note: Ggateway also installs
/etc/cron.hourly/auditquery /etc/logrotate.d/grid
[edit] MDS / MIP setup
The MDS setup needs some preparatory reading. Good starting pages are http://www.vpac.org/twiki/bin/view/APACgrid/UsingMDSFour and http://www.vpac.org/twiki/bin/view/APACgrid/PlanResource. The installation procedure is described at http://www.vpac.org/twiki/bin/view/APACgrid/MdsVMDeployment. It worked for me to just follow the procedure, but you really should now what you are doing. Read first.
[edit] Secure your MDS
Depending on when you installed your gateway, you may have to run an extra script from the Gbuild package to secure your MDS. On ng2, run
yum update Gbuild /usr/local/sbin/SecureMdsVdt161.sh
The script:
- creates $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/client-security-config.xml (use containercert)
- modifies $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/upstream.xml (use above client-security-config)
- creates empty /etc/grid-security/mds-grid-mapfile
- replaces $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/index-security-config.xml (require mds-grid-mapfile authorization for non-read-only methods)
- modifies $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/server-config.wsdd (add reference to index-security-config)
This script ties your MDS down so that only authorized hosts (those that have a mapping in /etc/grid-security/mds-grid-mapfile) can publish to the local MDS. This is necessary, because your compromised MDS would also compromise the upstream central server.
However, this script itself also breaks internal registration of the RFT service and the GRAM jobmanager. The viewpoint of APACGrid is that this is not planned but welcome casualty. It is not needed for current APACGrid operations. For new services deployed, the GLUE schema data format is preferred. Gerson however approved to publish the information into the repository now.
This is how I re-enabled the service registration on my machine. In /opt/vdt/globus/etc/
- copy globus_wsrf_mds_index/client-security-config.xml to gram-service/client-security-config.xml and to globus_wsrf_rft/client-security-config.xml
- edit globus_wsrf_rft/registration.xml and add
<SecurityDescriptorFile>etc/globus_wsrf_rft/client-security-config.xml</SecurityDescriptorFile>
past RefreshIntervalSecs - similarly, edit gram-service/registration.xml and add
<SecurityDescriptorFile>etc/gram-service/client-security-config.xml</SecurityDescriptorFile>
past RefreshIntervalSecs - add the mapping for local host into /etc/grid-security/mds-grid-mapfile
"/C=AU/O=APACGrid/O=BeSTGRID/OU=University of Canterbury/CN=host/ng2.canterbury.ac.nz" grid-mds
in my case - service globus-ws stop; service globus-ws start
[edit] Installing MIP
- Setup JCU yum repo
wget -P /etc/yum.repos.d/ http://ng0.hpc.jcu.edu.au/apac/gateway/rpms/jcu-apac.repo
- Install MIP
yum install APAC-mip-module-py APAC-mip-globus
[edit] Configure local MIP data
The configuration is described at http://www.vpac.org/twiki/bin/view/APACgrid/ConfigureAPACInfoServiceProvider (and the instructions below are borrowed from there :-) ).
- Run /usr/local/mip/mip and you should get about 10 lines of statically configured information.
- Edit /usr/local/mip/config/apac_config.py
- use your queue name in computeElement.Name = 'queue_name'
- check the paths for
- computeElement.qstat = '/usr/bin/qstat'
- computeElement.pbsnodes = '/usr/bin/pbsnodes'
- Turn on the reporting for the cluster and computing elements
- edit /usr/local/mip/config/default.pl, and uncomment the lines
- Cluster => [ "cluster1", ], and
- ComputingElement => [ "compute1", ]
- edit /usr/local/mip/config/default.pl, and uncomment the lines
[edit] Enable the JobManager field
The JobManager field will be particularly important on the LoadLeveler-integrated NG2HPC.
To publish the JobManager field in MDS, it's not enough to set the field in /usr/local/mip/config/apac_config.py:computeElement.JobManager = 'PBS'
It is also necessary to change /usr/local/mip/modules/apac_py/ComputingElement/computingelement.py to output this field (patch below).
TODO: check what other fields are not published by APAC MIP
--- computingelement.py.ORIG 2007-04-23 17:29:39.000000000 +1200
+++ computingelement.py 2007-07-26 14:17:03.000000000 +1200
@@ -229,7 +229,7 @@
# overridable values
- for key in ['ApplicationDir', 'DataDir', 'DefaultSE', 'ContactString', 'Status', 'HostName', 'GateKeeperPort', 'Name', 'LRMSType', 'LRMSVersion', 'TotalCPUs', 'FreeCPUs', 'MaxWallClockTime', 'MaxCPUTime', 'RunningJobs', 'FreeJobSlots', 'MaxRunningJobs', 'MaxTotalJobs', 'TotalJobs', 'Priority', 'WaitingJobs']:
+ for key in ['ApplicationDir', 'DataDir', 'DefaultSE', 'ContactString', 'JobManager', 'Status', 'HostName', 'GateKeeperPort', 'Name', 'LRMSType', 'LRMSVersion', 'TotalCPUs', 'FreeCPUs', 'MaxWallClockTime', 'MaxCPUTime', 'RunningJobs', 'FreeJobSlots', 'MaxRunningJobs', 'MaxTotalJobs', 'TotalJobs', 'Priority', 'WaitingJobs']:
if config.__dict__[key] is not None:
ce.__dict__[key] = config.__dict__[key]
@@ -241,7 +241,7 @@
ce.FreeJobSlots = ce.FreeCPUs
# print
- for key in ['ApplicationDir', 'DataDir', 'DefaultSE', 'ContactString', 'Status', 'HostName', 'GateKeeperPort', 'Name', 'LRMSType', 'LRMSVersion', 'TotalCPUs', 'FreeCPUs', 'MaxWallClockTime', 'MaxCPUTime', 'RunningJobs', 'FreeJobSlots', 'MaxRunningJobs', 'MaxTotalJobs', 'TotalJobs', 'Priority', 'WaitingJobs']:
+ for key in ['ApplicationDir', 'DataDir', 'DefaultSE', 'ContactString', 'JobManager', 'Status', 'HostName', 'GateKeeperPort', 'Name', 'LRMSType', 'LRMSVersion', 'TotalCPUs', 'FreeCPUs', 'MaxWallClockTime', 'MaxCPUTime', 'RunningJobs', 'FreeJobSlots', 'MaxRunningJobs', 'MaxTotalJobs', 'TotalJobs', 'Priority', 'WaitingJobs']:
if ce.__dict__[key] is not None:
print "<%s>%s</%s>" % (str(key), str(ce.__dict__[key]), str(key))
[edit] Configure provided Software information
You also need to configure information about software installed. This is in detail described at http://www.vpac.org/twiki/bin/view/APACgrid/ConfigureAPACSoftwareInfoProvider. One option mentioned there is feeding the information from the [APAC Software Map]. However, our sites are of course not registered there, so this option is not readily available. The contact person for the software map is [Ben Evans], however, I have not yet succeeded to establish contact with him. Temporarily, a usable solution is to configure the software list locally in an XML file.
The Softare Information Provider (SIP) configuration file in $MIP/config, the of the file name is <cluster>_<subcluster>_SIP.ini. For me, the name of the file should be ngcompute_sub1ngcompute_SIP.ini... but actually, MIP requires it to be called default_sub1ngcompute_SIP.ini. Reason uknown, resolved with a symlink :-(
This configuration file specifies the URI for the software information source. For a file-based source, I'm using
uri: file:softwareInfoData/ngcompute-localSoftware.xml
To create the file, copy /usr/local/mip/modules/apac_py/SubCluster/softwareInfoData/localSoftware.xml to ngcompute-localSoftware.xml in the same directory. The important part of my contents is:
<SoftwarePackages ...>
<SoftwarePackage LocalID="MrBayes/3.1.2" xmlns="http://grid.apac.edu.au/glueschema/Spec/V12/R1">
<Name>MrBayes</Name>
<Version>3.1.2</Version>
<Module>mrbayes/3.1.2</Module>
<ACL>
<glue:Rule>/APACGrid/*</glue:Rule>
</ACL>
<SoftwareExecutable LocalID="mrbayes-sequential">
<Name>mb</Name>
<Path>/opt/shared/bin/</Path>
<SerialAvail>true</SerialAvail>
<ParallelAvail>false</ParallelAvail>
</SoftwareExecutable>
<SoftwareExecutable LocalID="mrbayes-parallel">
<Name>mb-mpich2</Name>
<Path>/opt/shared/bin</Path>
<SerialAvail>false</SerialAvail>
<ParallelAvail>true</ParallelAvail>
</SoftwareExecutable>
</SoftwarePackage>
<SoftwarePackage ...>
....
</SoftwarePackages>
Now validate your data with
/usr/local/mip/config/globus/mip-exec.sh -validate
Note that I have not installed neither the package modules, neither its successor SoftEnv, and thus the element <Module>mrbayes/3.1.2-mpi</Module> does not work - pbs.pm translates it to module load mrbayes/3.1.2-mpi, which only generates the message "modules: command not found". I have temporarily hacked pbs.pm to add the directory /opt/shared/bin (where executables for installed packages reside) to PATH:
@@ -284,6 +291,8 @@
"export NODE_SCRATCH\n";
print JOB "$rsh_env";
+ print JOB "PATH=\$PATH:/opt/shared/bin"; ### Canterbury: hack before modules are installed
+
print JOB "\n#Change to directory requested by user\n";
print JOB 'cd ' . $description->directory() . "\n";
print JOB "$modulestring";
[edit] Turn MIP on
If MIP is configured correctly on your system and mip-exec.sh produces valid information, it is the time to integrate MIP into globus. A rough outline of this procedure is at http://wiki.arcs.org.au/bin/view/Main/IntegrateGridAusInfoServiceProvider. A detailed description of how to do the integration manually is at http://wiki.arcs.org.au/bin/view/Main/ManualMIPGlobusIntegration. Luckily, the last mentioned page is not needed anymore, and the integration is done simply with the comamnd
/usr/local/mip/config/globus/install -l /opt/vdt/globus
I recommend studying this command first to see what is changed - it patches the files downstream.xml, upstream.xml, server-config.wsdd, and hierarchy.xml in $GLOBUS_LOCATION/etc/globus_wsrf_mds_index, and installs mip-exec as a information provider into globus.
After running the install command, edit $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/hierarchy.xml and make sure you specify the correct primary MDS server (the install script incorrectly specifies ng2.sapac.edu.au instead of mds.sapac.edu.au) and preferably also the backup MDS server (ngmds.hpcu.uq.edu.au). The upstream entries in the config element should be
<upstream>https://mds.sapac.edu.au:8443/wsrf/services/DefaultIndexService</upstream> <upstream>https://ngmds.hpcu.uq.edu.au:8443/wsrf/services/DefaultIndexService</upstream>
And restart the globus WS container in order for all the changes to take effect
service globus-ws stop; service globus-ws start
[edit] Checking MDS contents
If everything works fine, you should see your site now listed at http://www.sapac.edu.au/webmds/webmds?info=indexinfo&xsl=apacgluexsl
You can also query your own MDS service
wsrf-query -s https://ng2:8443/wsrf/services/DefaultIndexService
query the service at Canterbury
wsrf-query -s https://ng2.canterbury.ac.nz:8443/wsrf/services/DefaultIndexService
or see the dump of all sites in the central APAC MDS server
wsrf-query -s https://mds.sapac.edu.au:8443/wsrf/services/DefaultIndexService "//*[local-name()='Site']"
or see just a single site as its contained in APAC MDS server
wsrf-query -s https://mds.sapac.edu.au:8443/wsrf/services/DefaultIndexService "//*[local-name()='Site' and @UniqueID='CANTERBURY']"
Note that in order for the wsrf-query command to work, certain jars with Java XML binding for the GLUE types must be installed - these come with the MDS server. Thus, this command should work on your ng2 machine (but may not work on the client grid machine).
[edit] Additional Modifications
- PBS job tagging to make distinguished name and email address of the user submitting the job available to PBS.
- Fixing startup order to avoid problems when Globus RFT starts before MySQL.
