Usenet 1994 October/usenetsourcesnewsgroupsinfomagicoctober1994disk2.iso/unix/volume19/cnews2/part01

Text File | 1989-06-29 | 46KB | 1,194 lines

Subject:  v19i078:  Cnews production release, Part01/19
Newsgroups: comp.sources.unix
Sender: sources
Approved: [e-mail]

Submitted-by: utzoo!henry
Posting-number: Volume 19, Issue 78
Archive-name: cnews2/part01

[  This is a rewrite of the transport and expire portions of the Usenet
   software.  It was designed to be fast and flexible.  --r$  ]

: ---CUT HERE---
mkdir doc
mkdir conf
mkdir notebook
mkdir batch
mkdir contrib
mkdir contrib/nntpmail
mkdir contrib/nntpmail/post_via_mail
mkdir contrib/nntpmail/mailing_lists
mkdir contrib/nntpmail/mailing_lists/appendfile
mkdir contrib/nntpmail/nntp_support
mkdir contrib/nntpmail/mail_to_group
mkdir contrib/rn.mod
mkdir expire
mkdir h
mkdir hfake
mkdir input
mkdir libbig
mkdir libbsd42
mkdir libc
mkdir libcnews
mkdir libfake
mkdir libsmall
mkdir libstdio
mkdir libusg
mkdir libv7
mkdir libv8
mkdir man
mkdir misc
mkdir nntpdiffs
mkdir nntpdiffs/diff
mkdir nntpdiffs/src
mkdir nntpdiffs/src.allnew
mkdir relay
mkdir relay/regress
mkdir relay/regress/out
mkdir relay/regress/out/test
mkdir relay/regress/out/test/a
mkdir relay/regress/out/test/b
mkdir relay/regress/out/test/c
mkdir relay/regress/out/control
mkdir relay/regress/master
mkdir relay/ads
mkdir relay/anews
mkdir relay/ctl
mkdir relay/aux
mkdir relay/altctl
mkdir relay/sh
mkdir rna
mkdir rna/lib
echo 'COPYRIGHT':
sed 's/^X//' >'COPYRIGHT' <<'!'
X/*
X * Copyright (c) University of Toronto 1985, 1986, 1987, 1988, 1989.
X * All rights reserved.
X * Written mostly by Geoffrey Collyer and Henry Spencer.
X * This software is not subject to any license of the American Telephone
X * and Telegraph Company or of the Regents of the University of California.
X *
X * Permission is granted to anyone to use this software for any purpose on
X * any computer system, and to alter it and redistribute it freely, subject
X * to the following restrictions:
X *
X * 1. The authors are not responsible for the consequences of use of this
X *    software, no matter how awful, even if they arise from flaws in it.
X *
X * 2. The origin of this software must not be misrepresented, either by
X *    explicit claim or by omission.  Since few users ever read sources,
X *    credits must appear in the documentation.
X *
X * 3. Altered versions must be plainly marked as such, and must not be
X *    misrepresented as being the original software.  Since few users
X *    ever read sources, credits must appear in the documentation.
X *
X * 4. This notice may not be removed or altered.
X */
!
echo 'PATCHDATES':
sed 's/^X//' >'PATCHDATES' <<'!'
X23-Jun-1989
!
echo 'README':
sed 's/^X//' >'README' <<'!'
XThis is C News, superseding assorted preliminary releases.  9 June 1989
X
XC News is a reimplementation of the transport and storage subsystems of the
Xnews software -- basically, everything except news readers.  We supply a
Xsimple news reader (written by Michael Rourke, included by permission,
Xslightly modified [so bugs are probably our fault]) as a replacement for
XB-News readnews suited to use by occasional users.  For regular news users,
Xthere are several more sophisticated readers widely available, and all should
Xwork with C News.  We use Larry Wall's "rn" ourselves; we have not included
Xit because this distribution is already rather big.
X
XC News's major advantage over B News is that it is much faster.  Timings
Xquite a while ago gave C News a speed advantage of roughly a factor of 25
Xin processing incoming batches.  This has probably improved a bit since.
XC News is now, on good machines with good C libraries, mostly system-call
Xbound.  Use of system calls has been optimized with some care, so it's
Xunlikely that further big speed improvements can be made at user level
Xwithout sacrificing backward compatibility.  See our paper in the Winter '87
XUsenix proceedings for some discussion of how the speed was achieved.
X
XC News also wins over B News on simplicity and robustness.  We provide
X(in our opinion) everything that's necessary, and avoid the frills that
Xrun up the complexity and decrease reliability.  We have not attempted to
Xprovide every feature anyone can think of, and have no plans to do so
Xin future either.  (This is one reason why we've stayed out of the news-
Xreader business, which generally has a bad case of feature-of-the-week
Xsyndrome.)
X
XC News's files are fully compatible with those of B News for any program
Xthat does not read log files and does not inspect the middle field of the
Xhistory file closely.  (The one major program that does is "nntp"; we include
Xdiffs for NNTP 1.5 which handle our middle-field format, interface it to
Xour input subsystem, and generally make it quite a bit faster than the
Xoriginal.)  C News complies fully with RFC 1036 (nee RFC 850), the official
Xdefinition of news interchange format.
X
XC News is, by intent and *we think* in practice, compatible with B News
Xat the level of most interfaces to the normal user, which basically means
Xthe semantics and options of "inews".  It is *not* compatible on the
Xsystem-administration level, although we think most of the changes are
Ximprovements or worthwhile simplifications.  The "postnews" that we supply
Xis not compatible with that of B News; it is purely interactive, as news
Xthat is already formatted can simply be fed to "inews -h".
X
XFor those who now run one of our ancient pre-alpha versions, many things
Xhave changed, and in particular the four-field history file format is gone.
XC News has also changed quite a bit since the alpha release that went out
Xon Usenet some time ago.
X
XFor our beta testers, build and the Makefiles have changed quite a bit but
Xthe software itself needed only fairly trivial fixes.
X
XWe know of three things that could still use work in this release:
X
X    1. The documentation could use work, especially for naive customers.
X    As it stands, it pretty much assumes a general knowledge of news
X    software.
X
X    2. There are a great number of small improvements that could
X    be made to the installation process, especially to permit still more
X    customization via the "build" program.
X
X    3. The fgetmfs function (in the libc directory) assumes that fgets
X    does not alter the buffer beyond the end of the string.  We are not
X    sure how portable this is, although it works on all our beta-test
X    systems, and may revise fgetmfs someday.
X
XThe active file format is the 4-field one that B news introduced midway
Xthrough 2.10, with minor additions: an `x' in the 4th field means
X``discard articles in this newsgroup'', and `=group' in the 4th field
Xmeans ``file articles for this newsgroup under `group' instead''.
X
XThe history file format is like B with one exception:  the second field,
Xwhich few programs ever look at, now consists of two subfields separated
Xby a tilde (~).  The first is the arrival date as a decimal number, the
Xsecond is the expiry date (if any) as a human-readable date (as emitted by
Xrnews) or a decimal number (after expire has gotten its hands on it once).
XExpire is tolerant of human-readable dates in both those places, but other
Xthings may not be.  The best way to get the history file into the new
Xformat is to rebuild it completely (this is RELATIVELY quick).
X
XThe sys file format is like a late-model B news with two extensions.
XFirst, the second field (groups and distributions) may optionally be
Xsplit into two subfields (newsgroups and distributions, respectively)
Xwith a slash. This permits solutions to various tricky problems that can
Xarise in odd situations if it is impossible to tell what's a newsgroup
Xname and what's a distribution.  Second, there is a new flag in the
Xthird field:  f is like F except that its output has the size
Xinformation that the C batcher wants for accurate limiting of batch
Xsize.
X
XThe way the news articles themselves are stored is totally unchanged; we
Xhave been unable to think of any changes that are worth the trouble.
X
XThere are some new control files in /usr/lib/news, to control the smarter
Xexpire and batcher.  (Old C News sites, note some format changes.)
X
XFile organization:  the one change is that programs are now kept mostly
Xin /usr/lib/newsbin, with /usr/lib/news reserved for control files etc.
X
XB News sites note:  /bin/rnews is now a front end for the input spooler.
XThe real news-filing program is called relaynews and is not in /bin.
X
XC News is meant to run adequately on a 16-bit machine, although this has
Xnot been tested thoroughly since utzoo became a Sun.
X
XMost (by intent all) of the programs understand seven key environment
Xvariables: NEWSARTS specifies location of articles (default
X/usr/spool/news), NEWSCTL specifies location of control files (default
X/usr/lib/news), NEWSBIN gives location of programs (default
X/usr/lib/newsbin), NEWSUMASK gives the umask to be used in creating
Xfiles (default 002), NEWSPATH gives the path used to find "normal" programs
X(default /bin:/usr/bin), NEWSMASTER is the address to which problem
Xreports should be mailed (default usenet), and NEWSCONFIG is the full
Xpathname of a little file that shell programs can use to pick up local
Xdefault settings for all these things (the equivalent for the C programs
Xis in the C News library) (default /usr/lib/news/bin/config).  The
Xenvironment variables override the defaults for testing and for operation
Xin funny situations.  Note that one or two things (e.g. relaynews), as
Xdistributed, will insist on renouncing setuid privileges if invoked with
Xthese overrides.
X
XBe warned that the nntp stuff in nntpdiffs, and the simple news reader in
Xrna, have not been gone over very well to make sure that they use the
Xstandard configuration mechanisms.  Hardwired pathnames may be present there,
Xand in general the stuff there is not well fitted into our automatic-install
Xmachinery.
X
XSee ROADMAP for a run-down on what's where in the distribution.  See
Xdoc/install for how to install it.  Conf/build is the interactive setup
Xprogram that does most of the work, or rather, sets up shell files which,
Xwhen run, do most of the work.  Even if your system is odd enough that
Xyou don't want to run the shell files conf/build generates -- doit.bin and
Xfriends -- as is, you are most strongly advised to run conf/build and use
Xthe resulting shell files as guides, rather than trying to "wing it"
Xyourself.  Getting the library built correctly is *NOT* trivial, because
Xwe try to cater to all 57000 different variants of Unix and there are a lot
Xof decisions that have to be made.  This is why we supply a 31KB shell
Xprogram to generate the configure+compile+install instructions, rather than
Xjust sending a complete pre-cooked recipe embodied in a Makefile:  there are
Xtoo many variables affecting how it should be done.
X
XC News has been tested pretty thoroughly.  We're also thoroughly sick of
Xit and make no promises that there will ever be another release.  We may,
Xrepeat *may*, provide updates via some appropriate newsgroup (currently
Xthe best choice is "news.software.b", although there is some sentiment for
Xfolding all the subgroups there into just "news.software"; we oppose
Xcreation of "news.software.c" because we don't think there will be enough
Xtraffic to justify a whole newsgroup).
X
XIf you've found a problem, we definitely do want to hear about it.  But,
Xwe *do not* want to see 2000 lines of diff listing!  What we want to see
Xis a concise human-readable description of what the problem is and how,
Xif at all, you solved it.  If we want the diff listing, we will ask.
XSimilarly, we are interested in hearing about changes and improvements,
Xbut want to see terse descriptions first.
X
XIf you want us to consider changes/fixes/etc, send them to us, don't just
Xpost them to the net.  We don't necessarily read all possibly-relevant
Xgroups.  Only postings from us are officially part of C News.
X
XTo send comments, complaints, problem reports, etc., do *not* mail to
XGeoff or Henry personally, but to:
X
X    [e-mail]
Xaka    [e-mail]
Xaka    utzoo!utstat!c-news
X
XUtstat.toronto.edu is directly on the Internet but does not have a lot of
XUUCP connections.  However, it does talk to utzoo.  Utzoo connects to half
Xthe known universe (well, not quite, but try via allegra, att, attcan,
Xattunix, decvax, dptcdc, floyd, hoptoad, kitty, linus, mnetor, pyramid,
Xsuncan, utai, utgpu, watmath, or yunexus).
X
X                    Geoff Collyer
X                    Henry Spencer
!
echo 'ROADMAP':
sed 's/^X//' >'ROADMAP' <<'!'
Xbatch    Output batcher.  It should work with B News, but not as well --
X    it really wants to be told sizes of articles as well as names.
Xconf    Configuration stuff, including the "build" shell program that
X    does most of the work of installing C News.
Xcontrib    Software from other contributors, possibly useful but not really
X    an official part of C News.
Xdoc    User documentation, including "install" which discusses how to
X    install C News.
Xexpire    Expire and friends, including history rebuilding and active-file
X    updating (neither of which is done by expire in C News).  Ought
X    to work with B News.  Fast, and permits control of expiry time
X    and such on a per-group basis.
Xh    Header files for include (see below).
Xhfake    Header files that your system ought to have but possibly doesn't,
X    for include (see below).
Xinclude    Actually not in distribution -- "build" constructs this from h and
X    hfake as required.
Xinput    Input spooler and processing.  Includes the "rnews" that goes in
X    /bin or wherever.
Xlibsmall
Xlibbig    Libraries for a couple of key functions that care whether they
X    have a large address space to work with.
Xlibbsd42
Xlibusg
Xlibv7
Xlibv8    Libraries for details that are Unix-variety-specific.
Xlibc    Stuff that ought to be in your C library but probably isn't.
Xlibcnews    General C News library functions.
Xlibfake    Stuff that might be in your C library, but might not be.
Xlibstdio    Fast re-implementations of some crucial stdio functions.
X    These are compatible with most Bell-derived stdios, and quite a
X    bit faster than most of the pre-System-V ones (which includes
X    those of many 4BSD variants).
Xman    Manual pages.
Xmisc    Miscellaneous internal utilities and maintenance programs.
Xnntpdiffs    Amendments to NNTP version 1.5, for Internet users, to work
X    with C News history-file format and to be a good deal faster.
Xnotebook    The C News Implementor's Notebook -- pieces of documentation
X    aimed more at gurus than novices.
Xrelay    Relaynews and friends.  The heart of C News -- actual reception
X    and filing of articles.
Xrna    A simple version of readnews (by Michael Rourke at UNSW) for
X    naive news users.
!
echo 'doc/install':
sed 's/^X//' >'doc/install' <<'!'
X.de Fn
X\\&\\$1\\fB\\$2\\fP\\$3
X..
X.TL
XInstalling ``C News'' Network News Software
X.AU
XGeoff Collyer
X.AI
XDepartment of Statistics
XUniversity of Toronto
X.AB
XThis document describes the flow
Xof network news within and between machines,
Xwhat each component of C news does,
Xand
Xhow to install C news.
X.AE
X.SH
XIntroduction
X.PP
XNetwork news
X(or
X.I netnews
Xfor short)
Xconsists of a collection of messages formatted similarly to
XARPAnet mail
X(see ARPA Internet RFC 1036 for details),
Xwidely spread.
XThe logical network,
Ximposed on top of various real networks,
Xformed by the set of all interconnected sites
Xexchanging network news is called ``Usenet''
Xand was formed in 1979,
Xradiating out from Duke University.
XNetnews is propagated
Xbetween cooperating machines
Xby a flooding algorithm,
Xwith some loop prevention heuristics:
Xeach machine sends its neighbours news that the
Xneighbours have (probably) not yet seen.
X.PP
XFlow of netnews between machines
Xmay be achieved by use of any network
Xwhich can transmit an arbitrary stream of
X(at least 7-bit)
XASCII code, unmodified.
XIf a network cannot transmit ASCII intact
X(e.g. BITNET),
Xit is possible to encode netnews
Xbefore transmission across the network
Xand
Xdecode it upon reception from the network.
XSince one cannot be certain that
Xone's network neighbours will be up and
Xreachable at all times,
Xoutgoing netnews must be queued,
Xat least in the case of network trouble.
XTo date,
Xat least
Xthese networks,
Xprotocols and media
Xhave been used to transmit netnews correctly:
XUUCP,
XRS232,
XNNTP,
XEthernet\(rg,
Xthe ARPA Internet,
XDatakit\(rg,
XACSnet,
Xmagnetic tape,
XSMTP,
Xand
XBITNET,
Xthough at least the last two require some form of encapsulation
Xto avoid corruption of articles;
XSMTP because some common implementations get the newline-CRLF
Xmappings wrong,
Xthus throwing off byte counts in batches,
Xand
XBITNET because of its Procrustean chopping,
Xexpanding,
Xmapping,
Xbashing
Xand
Xsmashing
Xof all data sent through it
X(sending lines of 80 or fewer characters of
Xletters of either case and digits and plus and minus
Xappears to be safe).
X.PP
XNetnews arrives via some network,
Xwhich causes a command to be executed upon arrival
X(e.g.
X.I rnews ).
X.I rnews
Xtypically spools the inbound netnews for later processing.
XEventually
X(often within the hour
Xor at the end of the business day),
Xthe input queue is run
Xand the netnews is stored locally,
Xtypically under
X.I /usr/spool/news ,
Xand queued for transmission to netnews neighbours.
XOnce stored on disk,
Xnetnews may be read by any of a collection of unprivileged news readers,
Xincluding
X.I cat (1).
X.I Expire
Xis run typically each night
Xto remove old netnews from disk.
X.PP
XC News was originally written to provide a
Xmuch faster and smaller,
Xmore robust,
Xreliable
Xand
Xcorrect
Ximplementation of netnews software than B 2.11 news.
XWe believe that C News is also
Xfaster,
Xsmaller
Xand
Xmore robust
Xthan B 3.0 news.
XB 3.0 news has many more features;
Xtake that as you will.
X.SH
XC News Components.
X.PP
X.I Rnews
Xinvokes
Xthe input subsystem,
Xwhich
Xspools incoming netnews in its original form,
Xas received,
Xtypically in compressed batches.
XPeriodically,
Xthe input queue should be run
Xby invoking
X.I newsrun ,
Xthus uncompressing any compressed input
Xand feeding it to
X.I relaynews .
X.PP
X.I Relaynews
Xfiles incoming netnews as articles on disk
Xand
Xinitiates spooled transmission to other machines,
Xoften by simply writing the names of the disk files
Xcontaining the articles on the ends of `batch' files of file names.
XQuite a bit of policy from RFC 1036 is embedded in
X.I relaynews .
X.I inews
Xis a complex front-end for
X.I relaynews
Xwhich implements much of the per-site policy on news posting.
X.I inews
Xis a fairly-slow shell script which is adequately fast for most sites,
Xbut it will need to be replaced by something more streamlined
Xfor sites which gateway mailing lists into netnews,
Xfor example.
X.PP
XThe output
X.I batcher
Xreads lists of file names
Xand
Xgenerates news batches
X(see RFC 1036 or
X.I news (5)
Xfor the format)
Xof prescribed sizes
Xand queues the batches
Xfor transmission to other sites.
XThe batcher is run asynchronously with
X.I relaynews ,
Xtypically once an hour
Xoutside of business hours.
X.PP
X.I Expire
Xis generally run once per night
Xto remove from disk news articles older than a few days.
X.I Expire
Xcan use different expiry criterion for different newsgroups
Xand can archive articles instead of removing them.
X.I Expire
Xalso runs asynchronously
Xwith
X.I relaynews .
X.PP
XThere are many news readers.
XC News
Xcomes with a limited news reader
X(\c
X.I readnews
Xby Michael Rourke)
Xsufficient to replace
Xlong-winded
X.I /etc/motd s
Xbut you will want a heavy-duty news reader
Xif you plan to read more than local news groups.
XWe recommend
X.I rn
Xby Larry Wall,
Xavailable from
Xyour netnews neighbours
Xor
Xyour nearby
X.B comp.sources.unix
Xarchive site.
XThere are others:
X.I vn
Xand
X.I vnews
Xare two.
X.SH
XPreparation for Installation
X.PP
XNetnews consumes a lot of disk space
Xand
Xoften a lot of transmission time.
XHere are some relevant measurements regarding a full netnews feed
Xas of the time of writing
X(January 1989),
Xtaken from
X.I news.lists :
Xa day's netnews is about 3Mb and 1,400 articles
Xin 450 newsgroups.
XYears ago,
Xsites often kept 14 days of netnews on disk,
Xbut now many sites keep news for 3 to 5 days,
Xthus allowing for the occasional long weekend.
XThus a full news feed expired after 4 days will consume
Xabout 12Mb.
XSome people feel that news volume is doubling roughly every 16 months.
XIf this is true,
Xwe can expect volume to increase by a factor of 10
Xin about 4 years
Xto 30Mb per day or 115Mb for 4 days.
XIt is thus wise planning to set aside a lot of disk space for netnews.
XThere are two ways to cope with ever-increasing volumes of netnews:
Xrefuse to accept more newsgroups,
Xor
Xexpire news after shorter intervals on disk.
XA current full feed takes just over 7 hours to transmit at 1200 baud,
Xso for dial-up connections
Xone clearly wants the fastest modems one can afford.
X.PP
XClearly,
Xtransmitting a full news feed is a non-trivial commitment of resources,
Xso you may have some difficulty in finding a site willing to supply one.
XSuch a site may in turn expect you to feed yet other sites.
XYou will need to agree with your prospective netnews neighbour(s)
Xupon transfer media,
Xprotocols
Xand
Xnetworks.
X.PP
XBefore proceeding to install C News, you should read this document through
Xto the end, probably read the companion document
X\fIThe Interface Between C News And The Outside World\fR,
Xand possibly read selected items in the
X\fIC News Implementor's Notebook\fR and the manual pages.
X.PP
XYou will need to
Xassign a user id and group id to netnews
X(often new ids called ``news'');
Xinitialise these directories:
XNEWSCTL
X(typically
X.B /usr/lib/news ),
XNEWSBIN
X(\c
X.B /usr/lib/newsbin ),
Xand
XNEWSARTS
X(\c
X.B /usr/spool/news );
Xand
Xinstall each subsystem.
XNEWSCTL and NEWSARTS
Xare logically one subtree,
Xdefining a news data base,
Xbut are split for backward compatibility with
Xolder news software,
Xparticularly old news readers.
XNEWSBIN
Xcontains programs and shell scripts
Xwhich might be common amongst machines sharing
Xa common architecture
X(e.g. Sun 3's);
X.Fn NEWSCTL /bin
Xmay override these.
XThe goal is to install the subsystems,
Xintegrate them into a working news system,
Xand
Xconfigure the news system to communicate with other news systems.
X.PP
XThere are a few key files that must exist before any serious
Xattempt may be made to operate the news software.
X.Fn NEWSCTL /active
Xis the list of newsgroups that this site knows
X(is willing to accept or individually reject),
Xand must be owned by the
XNEWS
Xuserid
X(the userid that owns
X.Fn NEWSBIN /relay/relaynews ,
Xtypically
X.I news ).
XYou will probably want to get your initial
X.Fn "" active
Xfile from your upstream feed
Xand edit it to suit the set of groups you wish to receive.
XBe sure to make the second field more than five digits wide,
Xby adding leading zeroes
X(ten digits is a conservative width).
X.Fn NEWSCTL /sys
Xdefines the newsgroups that this site is willing to accept
Xand describes how they are to be forwarded to its neighbours.
X.Fn NEWSCTL /server
Xcontains the hostname of your file server,
Xif you have multiple machines sharing news
Xvia a network file system.
X.Fn NEWSCTL /whoami
Xsimilarly contains the name by which a cluster of machines
Xsharing news is to be known for purposes of news.
X.Fn NEWSCTL /mailname
Xis optional and contains the full
X(possibly dotted)
Xname by which your cluster is known for purposes
Xof mail.
X.Fn NEWSCTL /organisation
X(or
X.Fn NEWSCTL /organization
Xif you insist)
Xcontains the name of your organisation,
Xwhich will be copied into the
X.B Organization:
Xheader of articles posted locally,
Xby default.
X.Fn NEWSCTL /mailpaths
Xdefines mail routes to machines which contain aliases
Xfor postings to moderated newsgroups.
X.Fn NEWSCTL /log ,
X.Fn NEWSCTL /errlog ,
X.Fn NEWSCTL /history ,
X.Fn NEWSCTL /history.dir ,
Xand
X.Fn NEWSCTL /history.pag
Xmust exist and be owned by the
XNEWS
Xuserid.
XTentative versions of all these files are built by the installation
Xprocedures, but it is quite likely that you'll have to edit some of them.
X.SH
XC News Installation
X.PP
XProceed to the
X.Fn \& conf
Xdirectory of the distribution.
XThere is a major shell file here, named
X.Fn \& build ,
Xthat will interrogate
Xyou at length and construct shell files to do the real work.
XYou may need to do
X``chmod\ +x\ build'' before running it, to make it executable.
X.PP
XYou will probably need your system's manuals on hand to answer
X.Fn \& build 's
Xquestions.
XAnother terminal (or another window, on a bitmap display) would also be
Xuseful.
XYou'd best be prepared to take notes, also, as
X.Fn \& build
Xwill occasionally suggest that something be checked when you're done.
X.PP
X.Fn \& Build
Xitself does not alter any files or perform any installation
Xchores:  all it does is create shell files in the
X.Fn \& conf
Xdirectory.
XIf you already know something about news software, or are merely
Xsuspicious that
X.Fn \& build
Xhasn't done everything right, you should
Xprobably read the shell files before running them.
XThere are four of them:
X.Fn \& doit.root ,
X.Fn \& doit.bin ,
X.Fn \& doit.news ,
Xand
X.Fn \& again.root .
X.PP
X.Fn \& Doit.root
Xsets up the major directories for news, and sets their
Xownerships correctly.
XIt typically will have to be run as \fIroot\fR to have adequate permissions
Xfor all of this.
XIt is brief.
X.PP
X.Fn \& Doit.bin
Xdoes most of the work of building and installing the programs.
XIt should be run as the user who owns the distribution directories and will
Xown the executable programs under NEWSBIN.
X.PP
X.Fn \& Doit.news
Xdoes some other small chores and installs control files.
XIf any of the control files already exist, it will complain and refuse
Xto overwrite them, as a safety precaution.
XIt should be run as the owner of the news files.
XSince many of the files it is installing are built by
X.Fn \& doit.bin ,
Xthat should be run first.
X.PP
XFinally,
X.Fn \& again.root
Xtends to ownership and permission changes on
Xa few programs that need to run set-userid.
XIt requires the ability to change ownerships and to set permissions on
Xthe files afterwards, which usually means running it as \fIroot\fR.
XIt too is brief.
X.PP
XThere are undoubtedly strange systems out there that
X.Fn \& build
Xand friends are not smart enough to cope with.
XIn such cases it will be necessary to edit the shell files before running
Xthem, or to use them as guides and do the work by hand.
XIn particular, systems that require strange options in
X.Fn \& Makefile s
Xwill need to have those inserted by hand.
X.PP
XIf you want to test pieces of C News without installing them, some
X(not all) of the subsystems have a ``make\ r'' feature that runs a
Xregression test.
XNote:  almost all of these require that
X.Fn NEWSCTL /bin/config ,
Xor its local equivalent, be in place already so that shell files
Xcan find out what PATH (etc.) they should be using.
X.PP
XNote that it is easy to build a test news system which is completely
Xindependent of other existing news systems on the same machine,
Xor
Xto build one which shares its
X.Fn NEWSBIN ""
Xwith another C news system,
Xor shares input of articles
X(e.g. running an old B news system
Xand a new C news system off the same stream of input
Xuntil you are confident that your new C news system is operating
Xto your satisfaction).
XSee
X.I subst (1)
Xfor the mechanism which permits quickly changing
Xall the places that know the location of
X.Fn NEWSCTL /bin/config.
XAfter that,
Xedit this news system's
X.Fn NEWSCTL /bin/config
Xand things should be set up for separate existence.
X.SH
XFirst News
X.PP
XWhen you arrange to get a news feed from your neighbor, you should also
Xask him to send you the current set of articles in the newsgroup
X\fBnews.announce.newusers\fR.
XSeveral of these are very important reading for people who are new to the net.
X.SH
XUnusual Systems
X.PP
XWe believe that C News runs fine on 16-bit machines, but it hasn't been
Xtested very thoroughly on them lately.
XIt will not perform quite as well with the more limited space.
X.PP
XMachines with very old compilers can be a headache.
XThere are some hooks in
X.Fn \& h/news.h
Xfor doing without ``void'' and ``unsigned long'',
Xtwo particular problem areas, but they have to be arranged manually;
X.Fn \& build
Xdoes not know about them.
X.PP
XSome very old systems cannot do \fIsetuid(geteuid())\fR, which makes it
Ximpossible for a daemon to make directories and get the ownership right.
XWe provide a small program,
X.Fn \& setnewsids ,
Xto run setuid-root.
X.Fn \& Relaynews
Xknows about it and invokes it if \fIsetuid(geteuid())\fR fails;
Xit then sets permissions correctly and re-invokes
X.Fn \& relaynews .
XThe code is short enough to be read and understood in full, so that the
Xsuspicious system administrator can be sure that this setuid-root program
Xis not going to do something awful.
!
echo 'doc/README':
sed 's/^X//' >'doc/README' <<'!'
XThis is user documentation.  See ../notebook for implementor's notes.
X
XEverything here is set up to use the -ms macros.  No complex formatting
Xis attempted, and only the three standard fonts (R, I, B) are used.
X
Xinstall        somewhat sketchy how-to-install document
Xinstall.out    nroffed version of install
Xinterface    technical summary of how C News works and its interface to
X        the rest of the system
!
echo 'doc/interface':
sed 's/^X//' >'doc/interface' <<'!'
X.DA "9 June 1989"
X.TL
XThe Interface Between C News And The Outside World
X.AU
XHenry Spencer
X.AI
XDept. of Zoology
XUniversity of Toronto
X.SH
XIntro and Generalities
X.PP
XC News relates to the ``outside world'',
Xthe system it is installed on, in a number of ways.
XThis document attempts to enumerate them and explain what goes on.
X.PP
XIn general, C News attempts to rely only on ``common basic Unix'',
Xand in particular it is not particularly BSD-specific or System-V-specific.
XSpecifically,
Xit makes no use of ornate locking mechanisms, silly interprocess-communication
Xschemes, peculiar networking primitives,
Xextensions to C, or other annoyances.
X.SH
XDirectories And Userids
X.PP
XMost of the components of C News live in \fI/usr/lib/newsbin\fR,
Xwith control files in \fI/usr/lib/news\fR and spooling areas
X(for current news, inbound traffic, and outbound traffic) in
X\fI/usr/spool/news\fR.
XSee the document
X\fIDirectory Layout and PATH in C News\fR
Xfor elaboration on the structure,
X\fIFiles in /usr/lib/news (aka NEWSCTL)\fR for a summary of control files,
Xand
X\fIConfiguration Mechanisms in C News\fR
Xfor how to change the locations of the directories.
X.PP
XThere is an extensive subdirectory structure under \fI/usr/lib/newsbin\fR,
Xwith only a few heavily-used utilities in the top-level directory.
XIn the following,
Xprograms not explicitly described as going elsewhere are all under there
Xsomewhere.
X.PP
XC News's spool areas and major control files need to be owned by a
Xspecific userid, normally \fInews\fR.
X(We believe that nothing except some of the Makefiles knows this name.)
XWe suggest that this userid should not own anything else on the system.
XThe programs in \fI/usr/lib/newsbin\fR can be
Xowned by \fIbin\fR (or anyone else) except for those that need to be setuid.
X(On our systems the non-setuid ones are owned by \fIbin\fR.)
X.SH
XUnix Utilities
X.PP
XC News requires a fairly complete Unix or equivalent.
X(We take no position on whether 4BSD, or System V, is Unix or not;
Xour private opinion is that neither truly deserves the name any more,
Xalthough we occasionally change our minds about which is less deserving.)
X(Note also that ``Unix'' is used here as an abbreviation for
X\fBThe UNIX Operating System\fR\(rg,
Xa registered trademark of AT&T;
Xwe acquired our bad habits and incorrect capitalization from
XUnix [sic] documentation supplied by the Bell System in the mid-1970s.)
X.PP
XIn particular, C News relies heavily on shell files.
X``Shell'' here means, of course, the standard shell, written by
XSteve Bourne.
XIf your \fI/bin/sh\fR is not a Bourne shell or \fIvery\fR good imitation,
Xyou're in trouble.
XNote, in particular, that some old versions of the Korn shell differ from
Xthe Bourne shell in some important details of quoting behavior, and
Xwe \fIknow\fR this causes problems with C News.
XYou need a standard shell.
X.PP
XTo the best of our ability, all our shell files begin with ``#!\ /bin/sh''.
XIf your shell misinterprets this as a request to run the C shell, you're
Xin trouble.
XIf your shell doesn't know about ``#'' comments at all, you're in trouble.
XWe hope that no modern shell makes either of these mistakes.
X.PP
XWe believe that none of our stuff relies on relatively modern shell features
Xlike shell functions or \fIgetopts\fR (as distinct from \fIgetopt\fR).
XIf \fItest\fR and \fIecho\fR are not built-in commands in your shell,
Xefficiency will suffer
Xbut everything should still run.
XIt's possible that a few obscure things will break if your shell does not
Xsupport ``['' as a synonym for \fItest\fR, although we try to avoid this usage.
X.PP
XWithin the shell files, C News makes heavy use of a wide variety of Unix
Xutilities, notably \fIsed\fR and \fIawk\fR.
XAny \fIawk\fR should do; in particular, nothing needs the ``new \fIawk\fR''
X(we don't run it yet).
XAlthough we have tried to avoid it, it is possible that some things depend
Xon \fIawk\fR recognizing ``\et'' inside strings, which very old \fIawk\fRs
Xdidn't.
X.PP
XIf your Unix does not put all standard Unix programs in
Xthe standard directories\(em\fI/bin\fR and
X\fI/usr/bin\fR\(emyou will need to modify C News's default PATH to include
Xwhatever bizarre directories are involved.
XSee the \fIConfiguration Mechanisms\fR document for details.
XIn particular, for some insane reason, 4BSD relocated a few standard
Xutilities like \fIwc\fR to \fI/usr/ucb\fR, which causes trouble
X(and not just to C News!).
XWe simply put some links in \fI/usr/bin\fR to fix this on our systems,
Xand this is what we recommend you do, but if you can't, you'll have to
Xchange the PATH.
X.PP
XSeveral parts of C News rely on being able to send mail to an administrative
Xaccount (the name of which can be chosen; see \fIConfiguration Mechanisms\fR).
XThe assumption is that a message, without any RFC822 headers, can be piped
Xinto \fI/bin/mail\fR (or whatever \fImail\fR is first in the PATH)
Xinvoked with a single argument which is the addressee,
Xand be delivered to the addressee's mailbox intact, possibly embellished
Xwith headers.
XThat is, with news's standard PATH, if
X.DS
Xecho hi there Joe | mail joe
X.DE
Xdoes not result in the message ``hi there Joe'' arriving in the mailbox
X``joe'', you're going to have to fake it.
XSee \fIDirectory Layout\fR for insight on where to put a fake \fImail\fR
Xso that C News components will use it rather than \fI/bin/mail\fR.
X.PP
XSimilarly, several parts of C News rely on being able to invoke \fIhostname\fR
Xwithout arguments, and have it return a single word which is the name of the
XCPU the code is running on.
XAgain, you'll have to fake it if you don't have \fIhostname\fR
Xor if it outputs something strange.
X.PP
XInput reception and output batching both want to use the \fIcompress\fR
Xdata-compression program.
X\fICompress\fR has been published on Usenet and is shipped with many Unix
Xsystems these days, but if you don't have it, we recommend that you get
Xit from your neighbors or the \fIcomp.sources.unix\fR archives.
XIt is possible to configure C News so that it doesn't use \fIcompress\fR
Xto compress news being transmitted, but you don't want to.
X\fICompress\fR is good and it is fast.
X.SH
XLibraries
X.PP
XC News is mostly self-contained as far as libraries go.
XIt does need some things that are not yet universal, like a full set
Xof string functions (e.g. \fIstrtok\fR);
Xwe provide reasonably portable versions of these for places that lack them.
X.PP
XOne thing that C News needs, because it is both useful
Xand a user-visible part of B News,
Xis the \fIdbm\fR library.
XAT&T has stupidly omitted it from System V.
XWe include an emulation or two that are claimed to work reasonably well.
XIf your system doesn't have a real \fIdbm\fR, however, our recommendation
Xis that you harass your supplier about it.
X.SH
XNetworking
X.PP
X(We're talking here about networking in the sense of NFS and all those fun
Xthings, not \fIuucp\fR or TCP/IP.)
XC News is designed to run on systems which consist of a conglomeration of
Xmachines on a local network, with only one of them acting as news server.
XBe warned, though, that if you're doing this, you need to have \fIrsh\fR
Xor some reasonably equivalent way of executing a program on another CPU.
XIf you've only got one machine,
Xjust make sure you \fIdon't\fR have a \fI/usr/lib/news/server\fR file and
Xforget the whole issue.
X.SH
XHighly System-Specific Things
X.PP
XThere are two small utilities within C News that are inevitably highly
Xsystem-specific and have a high probability of needing changes to match
Xyour system.
XBoth normally live in \fI/usr/lib/newsbin\fR proper.
X.PP
X\fISpacefor\fR is invoked by various components of C News to find out
Xhow much disk space is available.
XThe space margins in it are ones we find reasonable, but you may need
Xto adjust them.
XOn an old System V system in particular (one where \fIdf\fR can't be applied
Xto any directory name, but needs to be given a name in \fI/dev\fR), you
Xwill also probably need to modify the locations to be \fIdf\fRed.
XYou will also need to fix \fIspacefor\fR if your system's \fIdf\fR yields
Xresults in some strange format or in some strange units.
XWe believe that we get it right for stock 4BSD,
Xmany (but probably not all) System Vs, modern Irix, and real Unix (Version 7).
XBeware introducing major inefficiencies:
X\fIspacefor\fR is called a lot.
X(Our current one could stand to be faster, in fact.)
XBeware, also, that \fIspacefor\fR is not intended to permit routine operation
Xusing filesystems that are on the brink of being full;
Xthe limits it imposes are only approximate, and no attempt has been made
Xto tune its clients to exploit every last available block.
X.PP
X\fIQueuelen\fR is invoked by the batcher to determine how long the current
X\fIuucp\fR queues are, so it can judge whether it should suspend batching
Xof output to a given site.
XThere is too much diversity in \fIuucp\fRs for us to try to do this for all
Xpossible versions.
XWe think we get it right for the two most common flavors
X(HDB, aka BNU, and old subdirectory versions).
XOur versions measure queue length in batches, not bytes,
Xbut it would not be hard to change this:
X\fIqueuelen\fR, the \fIbatchparms\fR control file, and the \fIsendbatches\fR
Xhow-many-batches-should-I-try-to-add logic need to agree on the units of
Xmeasurement, but (we think) nothing else cares.
X.PP
XIt is just possible that you might have to modify \fInewshostname\fR,
Xwhich also lives in \fI/usr/lib/newsbin\fR itself.
X\fINewshostname\fR delivers the name which should be applied to the
Xwhole system (not just the particular CPU) for news purposes.
XIt tries to be fairly clever about different ways of getting the name,
Xand in particular will take it out of \fI/usr/lib/news/whoami\fR if
Xthat file exists, but if you're doing something odd on a strange system,
Xchanges may be needed.
X.SH
XInput
X.PP
XInput from the net via \fIuucp\fR (or equivalent) shows up as batches of
Xnews to be fed to \fIrnews\fR or its obsolete synonym \fIcunbatch\fR.
XThese two programs normally live in \fI/bin\fR, although nothing in the
Xcode knows about this and they can go elsewhere (one of our systems does this).
XBoth are simple front ends that invoke \fIinput/newsspool\fR to store the
Xbatch, while taking precautions to report trouble and not to overflow disks.
XNeither \fIrnews\fR nor \fIcunbatch\fR needs to be setuid.
X.PP
XInput via NNTP over the Internet (or equivalent) uses rather different
Xmachinery but ends up creating a saved batch in much the same way as
X\fIinput/newsspool\fR does.
X.PP
X\fIInput/newsspool\fR is a small C program that saves a batch,
Xwriting into a file in \fI/usr/spool/news/in.coming\fR.
XIt must be able to create files there, and \fIinput/newsrun\fR (see below)
Xneeds to be able to read them and delete them.
XThis gets a little tricky because \fInewsspool\fR will usually be
Xrun by \fIuuxqt\fR as userid \fIuucp\fR (or something like that),
Xnot as \fInews\fR, which \fInewsrun\fR needs to run as.
XThe recommended solution is to have
X\fInewsspool\fR owned by \fInews\fR and setuid.
XAn alternative is to give the \fIin.coming\fR directory
Xthe userid of \fInews\fR and the groupid of \fIuucp\fR, or vice versa,
Xand set permissions so that either can access it.
XOne of our systems ran that way for a while.
X.PP
XTo actually process incoming news, \fIinput/newsrun\fR gets invoked
Xto decompress the spooled batches and
Xfeed them to \fIrelay/relaynews\fR (see below).
XThere is an option for \fInewsspool\fR to invoke
X\fInewsrun\fR when a batch is spooled,
Xbut a (usually)
Xpreferable method is to have \fIcron\fR invoke \fInewsrun\fR once an hour.
X\fINewsrun\fR does its own locking to prevent multiple occurrences running
Xsimultaneously.
XThere is a related program, \fIinput/newsrunning\fR, that can be used
Xto set or clear a flag that stops \fInewsrun\fR;
Xthis may be a useful tactic if \fInewsrun\fR should not run at
Xcertain times.
XBoth \fInewsrun\fR and \fInewsrunning\fR must be run as \fInews\fR.
X.PP
XWhen a user posts news, he (or his news reader) does it by feeding the
Xarticle to \fI/bin/inews\fR.
XIn C News, \fIinews\fR is a complex shell file that attends to preliminaries
Xand then invokes \fIrelay/relaynews\fR.
X\fIInews\fR does not need to be setuid (indeed, we make no use of setuid
Xshell files at all, since they are grossly insecure).
X\fIRelaynews\fR is the heart of C News, the program that actually pulls
Xbatches apart and places articles into the database.
XIf users are to be able to post news, \fIrelaynews\fR should be owned
Xby \fInews\fR and setuid.
X.SH
XNews Readers
X.PP
XC News is fully compatible with B News to any news-reader program that
Xdoes not inspect the middle field of \fI/usr/lib/news/history\fR too closely.
XStandard B News news readers work fine.
XWe supply a simple news reader
X(written by, and included with permission of, Michael Rourke)
Xas a naive-user replacement for the B News
X\fIreadnews\fR.
XMore complex programs are preferable for serious news enthusiasts.
XWe recommend Larry Wall's \fIrn\fR
X(which we use, unmodified),
Xbut there are others.
X.SH
XOutput
X.PP
X\fIRelay/relaynews\fR normally queues up news for transmission to other
Xsystems by appending article names and sizes to batch files
Xin subdirectories under \fI/usr/spool/news/out.going\fR.
XThese are then processed by \fIbatch/sendbatches\fR, which should be run
Xregularly, as \fInews\fR, by \fIcron\fR.
X\fISendbatches\fR can be configured to use a variety of transmission
Xmechanisms, the usual being \fIuux\fR.
X.SH
XExpiry And Related
X.PP
XNews articles are removed, possibly with archiving to an archive area,
Xby the expiry subsystem.
X\fIExpire/doexpire\fR should be invoked now and then,
Xas \fInews\fR, by \fIcron\fR.
XWe suggest nightly.
X\fIDoexpire\fR actually invokes \fIexpire/expire\fR to do the dirty work.
X.PP
XC News \fIexpire\fR does not have an option to rebuild the
X\fI/usr/lib/news/history\fR file from scratch,
Xsince that has nothing to do with expiry.
XTo rebuild \fIhistory\fR,
Xe.g. if it has been destroyed,
Xuse \fIexpire/mkhistory\fR.
X.PP
XSome news readers need to have the third field of \fI/usr/lib/news/active\fR
Xupdated occasionally to show the lowest article number still present in each
Xnewsgroup.
XFrankly, we think such news readers simply need to be fixed.
XC News \fIexpire\fR does not do this updating.
XFor those who use such news readers, however, \fIexpire/upact\fR will do
Xsuch an update.
XIt should be run as \fInews\fR.
X.PP
X\fIRelay/relaynews\fR
Xdoes not implement the ``Supersedes:'' header, which we loathe
Xas a crude solution to the wrong problem.
XAlas, the network-map distribution in
X\fIcomp.mail.maps\fR relies heavily on it.
XOur preferred approach is to process map articles as they arrive and
Xthen expire them normally (using \fIexpire\fR's expiry-date-override
Xfeatures to insist that they expire promptly).
XHowever, for those who don't do this, and don't want to have megabytes
Xof obsolete map data piling up,
X\fIexpire/superkludge\fR will remove superseded files in specified
Xnewsgroups.
XIt should be run as \fInews\fR by \fIcron\fR, perhaps nightly.
X.SH
XReboots and Administration
X.PP
XIf the system crashes, things like locks must be cleaned out if C News
Xis to function properly after reboot.
X\fI/etc/rc\fR, or equivalent, should run \fImaint/newsboot\fR during reboot,
Xas \fInews\fR.
X.PP
XCertain log files can grow without bounds if not renamed/removed now and
Xthen.
XWe recommend running \fImaint/newsdaily\fR once a day.
XIt tends to logs, keeping a generation or two for use in trouble tracking,
Xand also sends mail to the news administrator in the event that something
Xfunny has happened.
X.PP
XIn general, C News does not attempt to break locks,
Xon the philosophy that a stale lock may mean something is badly wrong.
X(See
X\fILocking in C News\fR for details on locking methods.)
XThe various programs will either give up, to return later, or wait
Xpatiently for the lock to go away.
XIf one doesn't keep an eye on things, a problem of some kind can hang up
Xthe news system for quite a while.
XRunning \fImaint/newswatch\fR once in a while\(emwe recommend a few times
Xa day\(emwill alert administrators to signs of trouble.
XExcept on grossly slow systems, C News locks should never hang around for
Xany great length of time.
!
echo 'doc/Makefile':
sed 's/^X//' >'doc/Makefile' <<'!'
Xinstall.out:    install
X    nroff -Tdumb -ms install >$@
!
echo done