home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
DP Tool Club 12
/
CD_ASCQ_12_0294.iso
/
news
/
2381
/
fcc110
/
fcc.doc
< prev
next >
Wrap
Text File
|
1993-10-12
|
29KB
|
743 lines
FCC Version 1.10
10/5/93
What is FCC
-------------------------------------------------------------------------
FCC is a file access library which may be used in place of 'standard'
file access functions. FCC performs the functions of a disk cache and
disk doubler. FCC enhances standard file access functions in two ways:
- FCC uses an efficient LRU write back cache. FCC supports cache
block sizes as small as 128 bytes for increased performance
when randomly accessing small blocks.
- FCC can optionaly provide sequential and random read/write access
to a compressed file. Like a disk doubler, FCC performs these
functions transparently. All you our your application is aware
of is that the physical size of the FCC file is smaller.
Uses for FCC
-------------------------------------------------------------------------
- To speedup file access. FCC is much faster than a disk cache, and
the cache parameters are under program control. Programs which
randomly access small blocks of information will benefit the most.
- To reduce the disk requirements of applications. Both for
distribution, and when installed. FCC may be used in place of
the Windows LZ functions.
- To hide sensitive data. Data compression ensures that proprietary
data can not be easily decoded.
- To update existing products. Many database type programs waste
a lot of disk space. FCC can remove this waste with file
compression, requiring only minor changes to the application.
- To efficiently store flat file databases. With FCC, you do not
need to be concerned about balancing field lengths with disk
usage.
Why use FCC
-------------------------------------------------------------------------
FCC does not have any runtime royalties, and does not require a separate
compression program (like COMPRESS.EXE). In addition, FCC provides full
write access to compressed files.
Programming Interface
-------------------------------------------------------------------------
The following programming interface is not language specific. FCC
may be used by almost all Windows languages and tools. For language
specific interface information see the files:
FCC.H
FCC.BAS
The FCC functions closely resemble the standard C file access functions
(open, close, lseek, read, write). This is also the model used by
many operating systems. An internal file pointer is kept for each opened
file. It is used to direct the file positioning used in read and write
operations.
In the standard model there are 7 functions:
open - open or create a file for reading and/or writing
close - close an opened file
seek - set the internal file pointer to a specific file location
read - read from a file
write - write to a file
tell - retrieve the position of the internal file pointer
length - retrieve the length of a file
The open function returns a handle to the opened file, which is used
to identify the file to the other functions.
FCC also has 5 additional functions:
version - determine the version of FCC being used
exists - determine whether a file exists and/or is compressed
plength - retrieve physical length of a compressed file
flush - write dirty cache blocks to disk
copy - copy one file to another
The main difference between the FCC model and the standard model is in
file opening. The additions to the standard open function are:
- A way to indicate the cache size.
- A way to indicate the cache block size.
- A way to indicate whether or not a created file should be a
compressed file.
The FCC functions are presented below:
FCC_open
----------------------------------------------------------------------
PURPOSE:
Open or create (and open) a file.
PARAMETERS:
handle - Filled in with a FCC file handle (if no error).
name - Name of file to open or create.
flags - Open flags. May be one of:
FCC_READ - Open for reading.
FCC_WRITE - Open for reading and writing.
FCC_CREATE - Create and open for reading and writing.
compress - Compression method (only used with FCC_CREATE).
May be one of:
FCC_CDEFAULT - Default compression.
FCC_CNONE - No compression.
kbytes - The size of the file cache in kilobytes (1 - 2048).
blcksize - The size of cache blocks in bytes (128 - 16384).
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
The blksize used when creating a compressed file will be used
in all subsequent access to the file. To change the blksize,
copy the file with a different block size.
When a file is opened, the internal file pointer is set to 0.
FCC_close
----------------------------------------------------------------------
PURPOSE:
Close an opened FCC file.
PARAMETERS:
handle - FCC file handle.
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
All dirty cache buffers will be written to disk before the file
is closed. This may take some time if the cache used is very
large.
FCC_seek
----------------------------------------------------------------------
PURPOSE:
Set the internal file pointer to a specific location.
PARAMETERS:
handle - FCC file handle.
pos - File position offset.
where - May be one of the following:
FCC_SEEKSET - Seek to position.
FCC_SEEKCUR - Seek from current position.
FCC_SEEKEND - Seek from end of file.
newpos - Filled in with the new internal file pointer position.
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
The internal file pointer is 0 based (i.e. the first byte in a
file is located at position 0.)
To position the file pointer for appending, use
FCC_seek(handle, 0, FCC_SEEKEND, NULL).
If newpos is set to NULL, it will not be filled in.
FCC_read
----------------------------------------------------------------------
PURPOSE:
Read from a file.
PARAMETERS:
handle - FCC file handle.
buff - Memory buffer where data should be placed.
bytes - Bytes to read (0-FFFE).
rbytes - Filled in with number of bytes read (if no error).
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
rbytes may be filled in with a value which is less than the
actual number of bytes requested.
FCC_write
----------------------------------------------------------------------
PURPOSE:
Write to a file.
PARAMETERS:
handle - FCC file handle.
buff - Memory buffer containing data to write.
bytes - Bytes to write (0-FFFE).
wbytes - Filled in with number of bytes written (if no error).
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
wbytes may be filled in with a value which is less than the
actual number of bytes requested.
FCC_tell
----------------------------------------------------------------------
PURPOSE:
Retrieve the current internal file pointer position.
PARAMETERS:
handle - FCC file handle.
pos - Filled in with the current file pointer position
(if no error).
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
The value placed in pos is 0-based relative to the beginning of
the file. If the file pointer is positioned at the first byte
of the file, pos contains 0.
FCC_tell functions identically to:
FCC_seek(handle, 0, SEEK_CUR, &pos);
FCC_length
FCC_plength
----------------------------------------------------------------------
PURPOSE:
Retrieve the logical (FCC_length) or physical (FCC_plength) length
of a file.
PARAMETERS:
handle - FCC file handle.
length - Filled in with the file length (if no error).
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
If the file is not compressed, FCC_plength functions identically
to FCC_length. If the file is compressed, FCC_plength will
typically return a value smaller than FCC_length.
FCC_exists
----------------------------------------------------------------------
PURPOSE:
Determine if a file exists and/or is compressed. In addition,
FCC_exists may be used to determine a files length.
PARAMETERS:
name - Name of file.
version - Filled in with:
-1 - The file does not exits.
0 - The file exists but is not FCC compressed.
else - The FCC version used to compress (if no error).
length - If not NULL, filled in with the logical file length.
plength - If not NULL, filled in with the physical file length.
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
The value placed in version will be the integer equivalent of the
FCC version used to compress. Version 1.00 will be 100, version
10.15 will be 1015. All versions numbers will be between 100 and
9999.
FCC_version
----------------------------------------------------------------------
PURPOSE:
Determine the FCC_version
PARAMETERS: none
RETURN:
The version of the FCC library. The version is the same as
described in FCC_exists.
FCC_copy
----------------------------------------------------------------------
PURPOSE:
Copy one file to another. The source file may be any file,
including FCC compressed files. The destination file may
optionaly be compressed.
PARAMETERS:
src - Source file.
dst - Destination file (or NULL, or "").
blksize - Block size parameter to both FCC_open calls. Choose
the largest block size for maximum compression, and
the fastest copy.
compress - Indicates whether the destination file should be
compressed. May be one of:
FCC_CDEFAULT - Compress destination file.
FCC_CNONE - Do not compress destination file.
progress - See section Copy Progress below.
RETURN:
0 - No error.
else - An FCC error code.
NOTES:
The destination file will have the same date and time of the
source file (if no error).
If dst is NULL or "", the file will be copied in place. In
other words, src will be copied to itself.
- To decompress a file:
FCC_copy(filename, "", blksize, FC_CNONE, NULL)
- To compress or de-fragment a file:
FCC_copy(filename, "", blksize, FCC_CDEFAULT, NULL)
Copy Progress
-------------------------------------------------------------------------
This section is only relevant to the function FCC_copy.
When copying a file using FCC_copy, you may want your application to
be able to abort the copy and/or display progress information to the
operator. The progress parameter to FCC_copy is used for this purpose.
If you do not want any progress information, you may pass a NULL
parameter for progress. In MS Visual Basic this is equivalent to
passing ByVal 0&.
The structure FCC_COPYT is used for progress/abort purposes. The fields
in FCC_COPYT are:
wHnd - window handle
message - message to be sent to wHnd
wparam - first message parameter
lparam - second message parameter
cancel - if not zero, copy will be aborted
length - logical length of input file
plength - physical length of input file
curpos - bytes copied so far
Periodically, FCC_copy will send the following Windows message:
SendMessage(wHnd, message, wparam, lparam)
If, after sending this message, the cancel field of progress has been
set to a value other than 0 by the application, FCC_copy will return
with the 'error' FCC_ERR_CANCELED. In addition, the application may
use the values in length, plength and curpos to display progress
information.
The MS Visual Basic example demonstrates how the progress parameter
to FCC_copy is used.
There are several important notes about the VB example:
- The message sent in this example is WM_KEYUP with wparam set to 256.
In other words, a false key up message is sent to the main VB
Window. The false key is 256, which will not conflict with any
Windows keystrokes.
- DoEvents is called to process any user input. If DoEvents is not
called, Windows will not be able to process any user input or
display progress information.
- When the false WM_KEYUP message is sent, progress information is not
automatically displayed. The display of progress information is
done by a timer control. This is done to reduce CPU overhead
and for an 'aestheticly pleasing' once/per half second display.
FCC Error Codes
-------------------------------------------------------------------------
FCC_ERR_OPEN
Could not open a file. This error will typically be encountered
when trying to open a file which does not exist.
FCC_ERR_WRITE
Could not write to a file. This error will typically be encountered
if a disk drive fails or becomes full.
FCC_ERR_READ
Could not read from a file. This error will typically be encountered
if a disk drive fails.
FCC_ERR_CLOSE
A file could not be closed. This error will typically be encountered
if a disk drive fails.
FCC_ERR_SEEK
A seek could not be performed. Because seek operations should not
generate an error, this error is very unlikely.
FCC_ERR_CACCESS
Data in a compressed file could not be retrieved. This error
will typically be encountered if a FCC compressed file has become
corrupt or has been altered without FCC.
FCC_ERR_VERSION
A file could not be opened because it is recognized as a
compressed FCC file, but the version is not supported.
FCC_ERR_NOMEM
Insufficient memory.
FCC_ERR_PARAM
An invalid parameter was passed to a FCC function. This error
will typically only be encountered when a program is being written.
Make sure each of the passed parameters is valid.
FCC_ERR_CANCELED
The cancel field of progress in FCC_copy was set to a value other
than 0.
FCC_ERR_UNKNOWN
An unknown error occurred.
Disabling the Write Back Cache
-------------------------------------------------------------------------
There is no way to explicitly disable the write back cache. You may,
however, write a function which does so. In C:
int FCC_writeflush(int handle, void *buff, long bytes, long *wbytes)
{
int err;
err = FCC_write(handle, buff, bytes, wbytes);
if (err == 0)
err = FCC_flush(handle);
else
FCC_flush(handle);
return (err);
}
This function writes and flushes in one operation.
Choosing Cache and Block Sizes
-------------------------------------------------------------------------
When opening a file with FCC, you have to tell FCC how much memory to
use for the cache, and what cache block size to use. The cache will
have (kbytes * 1024) / blksize blocks. In other words, if you specify
a 1MB (1024K) cache and a 256 byte block size, there will be 4096
cache blocks. Each cache block can contain blksize bytes of file data.
A least recently used (LRU) block allocation mechanism is used.
In general, you should make the cache size as big as possible (up to
the size of the file of course). Choosing the block size is more
difficult. The following rules will usually work in most cases:
- When accessing a file sequentially, choose a block size as
big as possible (up to the size of the cache size).
- When accessing a file randomly, choose a block size between 1/2
and 4 times the size of the typical access size (the bytes
parameter to FCC_read and FCC_write).
The other consideration when choosing a block size is compression. The
larger the block size, the smaller the file will be. The penalty, however,
is slower random access. For example, suppose you randomly access a
compressed file, and the typical access size is 128 bytes. File access
using a block size of 32K will be much slower than with a block size of
2K. A block size of less than 1K for compressed files is not
recommended.
It is best to experiment with different combinations until an adequate
size vs. speed tradeoff is reached.
Zeroing Unused Data
-------------------------------------------------------------------------
When writing to a compressed file, all unused space should be set to one
value (zero is suggested). For example: Suppose you are writing an
address book database. For each record you are reserving 30 bytes for
the last name. Since most last names have less than 30 letters, a lot of
space will be unused in each record. By setting this unused space to
one value, it may be compressed by FCC. The last name 'Jones' should
be stored as 'Jones' followed by 25 0's.
Fragmentation
-------------------------------------------------------------------------
When randomly accessing a compressed file, it may become fragmented.
This causes the physical file size to become larger than it needs to
be. By simply copying the FCC file, the physical size will be become
as small as possible. For example: Suppose you have been randomly
accessing the file MY.DAT, and now you want to make it as small as
possible. FCC_copy may be used for this purpose:
FCC_copy("MY.DAT", "", blksize, FCC_CDEFAULT, NULL)
Other Versions / Other Products
-------------------------------------------------------------------------
There are other versions and products which do not have a
standard pricing plan:
- Other versions of FCC with faster and/or better data compression.
- Buffer based data compression algorithms.
- Stream based data compression algorithms.
- MS-DOS and OS/2 libraries.
Ordering Information
-------------------------------------------------------------------------
To purchase FCC 1.10, print (or copy) this ordering
form. You will receive the commercial version of FCC 1.10, and
the right to distribute FCCWIN.DLL for use with your applications.
The price shown is an introductory price, valid through 12/31/93.
After this date, call for the current pricing.
Product ID: FCC110
Price: $95 US (non-US shipping add $5 US)
Make Payment To: Four Lakes Computing
Payment Methods: US Check, US Money Order, US Dollars,
American Express International Money Order.
Send Payment To: Four Lakes Computing
1135 Williamson #4
Madison, WI 53703 USA
Address: __________________________________________________
__________________________________________________
__________________________________________________
__________________________________________________
Phone: _____________________ Disk: ___ 5 1/4 ___ 3 1/2
If you have any questions, contact:
US Phone: 608-256-3382
Compuserve ID: 70662,2501
The Test Program FCCTST.EXE
-------------------------------------------------------------------------
FCCTST.EXE is a Windows program which may be used to test FCC. It
is written in Microsoft Visual BASIC. The source code for FCCTST.EXE
is contained in the files:
FCC.BAS - FCC BASIC Interface
FCCERR.BAS - Converts FCC error codes to English messages.
FCCTST.FRM - Visual BASIC form.
FCCTST.MAK - Visual BASIC project file.
FCCTST.EXE allows you to copy files. The source file may be any file,
including FCC compressed files. You may choose for the destination
file to be compressed.
FCCTST.EXE also allows you to test random read and write access to both
compressed and uncompressed files.
The controls in FCCTST.EXE are explained below:
Source File:
----------------------------------------------------------------------
This is the source file for a copy operation. The file you specify
will NOT be modified by FCCTST.EXE. The choice of Source File is not
really important if compression is not being used (any file of the
desired size will work). If compression is being used, Source File
should contain the type of data you are interested in compressing.
Destination File:
----------------------------------------------------------------------
This is the destination file for the copy operation, and the file that
will be modified during random access. THIS FILE MAY BE CREATED OR
MODIFIED BY FCCTST.EXE. The default destination file is c:\fcc.tmp.
Copy
----------------------------------------------------------------------
Copies the source file to the destination file.
Random
----------------------------------------------------------------------
Randomly accesses the destination file until cancel is pressed.
Cancel
----------------------------------------------------------------------
Cancels a file copy or random access.
Cache Size (K):
----------------------------------------------------------------------
Used to determine the cache size for the destination file.
Block Size
----------------------------------------------------------------------
Used to determine the block size for FCC_copy and FCC_open.
When accessing a compressed file, it is suggested that block
size be no less than 1024. For maximum compression, choose the
maximum Block Size for the destination file (16384).
Compress
----------------------------------------------------------------------
Determines whether the destination file in a copy operation should
be compressed.
Random Size
----------------------------------------------------------------------
Indicates the size of reads (and possibly writes) used in a random
access operation. A random number between 1 and Random Size will
be used for each access. Data will be read from the destination file,
and if Random Writes is enabled, will be written to another location
in the destination file.
This parameter is important for testing cache efficiency during
random access. If the average size of access you want to test is
1024 bytes, choose a Random Size of 2048.
Random Writes
----------------------------------------------------------------------
Indicates that random writes (as well as reads) should be performed
in a random access operation. It is interesting to note that the
physical size of a compressed file will increase for awhile when
accessed randomly with Random Writes enabled. This is because of
file fragmentation.
Random %
----------------------------------------------------------------------
Indicates the location of random access:
10 - 90% of the random access requests will be made to the first
10% of the file. The other 10% will come from the entire file.
30 - 70% of the random access requests will be made to the first
30% of the file. The other 30% will come from the entire file.
100 - The file is accessed randomly.
This is an important parameter for testing the effectiveness of the
cache. As most files are not accessed truly randomly, this parameter
allows you to test typical file access, and adjust the cache parameters
for the best performance.
Progress Information
----------------------------------------------------------------------
Source File Size
Indicates how big the input file is. The logical size will be in
parentheses if it differs from the physical size.
Destination File Size
Indicates how big the destination file is. Note that this may
grow during randow access (with writes enabled) due to fragmentation.
Bytes Processed
Indicates how many bytes have been copied, or how many bytes have
been processed in a random access operation.
K/Sec
Indicates how may kilobytes have been processed per second. This
is the main indicator to you of cache/compression efficiency. It
will typically range between 5 and 2000. This is the gauge you
use to select optimal FCC parameters.