Usenet 1994 October/usenetsourcesnewsgroupsinfomagicoctober1994disk2.iso/misc/volume25/QBATCH/part03/DESCRIPTION

Text File | 1991-11-04 | 6KB | 118 lines

     The QBATCH system and its related programs were
     written by Alan D. Saunders and are
     Copyright (c) Vita Services Ltd. 1990 and
     Copyright (c) Vita Fibres Ltd. 1991

     PLEASE read the NOTICE file for details.

QBATCH is a versatile queued batch processing system for *NIX.

Each queue consists of a file containing information about the queue
itself, and about all jobs currently present in the queue.  When the
program 'qp' is run for a given queue, it will fork a child process for
each job in the queue in turn, and wait for it to complete.  If there
are no jobs present in the queue, qp will wait for a signal from one of
the support programs, which will 'tell' it that another job has joined
the queue, or that it should terminate.

Features:

There can be as many queues as the system can support. Queues are
named, and run asynchronously.  The processing of jobs running in one
queue are totally independent of those in any other (subject of course
to the independence of potentially shared resources such as data files
and devices).

Queues can be started and stopped independently.  If a job is running
in a queue which receives a stop signal from the 'qs' program (NOT
SIGSTOP) qp will allow the job to run to completion before terminating
unless the kill option is specified.

The priority (nice value) of all jobs running in a queue is defined
when the queue is created, so prioritised queues can be configured.

Jobs running in a queue consist of shell scripts, created (or copied)
by the js program.  The default shell is the bourne shell, but this can
be overridden at submit time with the -s option to js.   The jobs run
with the uid and gid of the submittor, and run in the environment
ruling when the job was submitted.  All environment variables except
those excluded deliberately are passed to the job, and the job will
start in the submittors current working directory.  Fixed context
queues may be configured which overrule the above, they may be
configured to run under a specified default shell, with a specified
uid and gid (NOT root), and with a pre-defined environment and working
directory.  Access to fixed context queues may be restricted by user-
name and group.

Stdout and stderr of running jobs is redirected to a 'monitor' This
will normally be the queue monitor (specified or defaulted when the
queue was created), but may be specified for individual jobs.  The
pathspec of both the job file, and the monitor, may be accessed with
the 'qf' program viz:  "tail -f `qf -m work`"  where 'work' is the name
of a queue.

The status and contents of queues can be listed at any time using the
'ql' program.  Information can include the uid and gid under which jobs
will run (are running), user name and tty of submittor,  date and time
of submission, date and time current job started, and a meaningful job
name or description.

If rc.QBATCH is configured in the system init script, all queues which
were running when the system went down will be restarted. Any jobs
present in the queues will also be restarted.

Facilities:

inform user by mail, or tty message when job is complete.
cancel jobs in queues, or kill running jobs.
change the running order of jobs in queues.
suspend and resume processing of jobs.
start and stop the queue process engine at any time (even under cron or from
      another queue)
disable submissions to a queue, and re-enable them.
stop a queue, and wait until the current job has finished (useful for ensuring
      that the system is idle before a backup)
list the status and contents of a queue.
repeat (or kill and repeat) the currently running job in a queue.
examine or edit the jcl of jobs in a queue, and examine the monitors.
monitor (and perhaps account) the times taken by jobs in the queue.

Prevent deadlocks between batch programs accessing the same devices or data
files by ensuring they are submitted to the same queue, then they can't run in
parallel.

Prevent cpu hogging jobs from swamping the system by submitting them to
low-priority queues.

Have a separate queue for humungous jobs that is started and stopped by cron
out of working hours.

Ensure all overnight work is running in queues, then a backup job can stop the
queues, wait for current jobs to finish, and back up before restarting the
queues.

Protect sensitive applications and data with file and directory permissions,
then permit named users or groups to submit jobs to a fixed context queue.

Queues can only be created by root, so the system administrator can tune the
configuration to provide optimum throughput.  It is frequently the case,
particularly if a system is swapping heavily, or is short of swap space, that
two jobs running one after the other in a queue, will process faster overall
than if they were run concurrently.

Of course, porting and installing QBATCH isn't the end. Queue discipline has to
be imposed in order for it to be of any benefit.  Scripts and commands
triggering batch jobs will have to be changed to use QBATCH, and jobs normally
run interactively which impose heavy system loads, will have to be
reconfigured. (consider for example renaming 'make' as 'MAKE' and writing a
'make' script which submits 'MAKE' to a queue).  Of course there will always be
the awkward user who decides to submit a large job into a high priority queue.
Analysing the queue monitors should show this up, they contain a certain amount
of profile timing for jobs run in the queue, and the uid and gid of the submittor.

This should give you a taste of what QBATCH has to offer.

Regards .. Alan