Writing Adaptors for Conductor (Last updated: $Date: 2001/10/17 16:46:03 $)

1. Introduction

Adaptors operate on one direction of data flow. Each adaptors should be a subclass of the Adaptor class and should implement handleFlow(). Each adaptor has its own thread. The handleFlow() method is called at the start of the life of the thread. When handleFlow() returns, the adaptor will be removed from the data stream. If the stream has not closed when handleFlow() returns, data will continue to flow unadapted.

2. Instantiation

Each adaptor should include a constructor that takes two arguments. The first argument is a window object (see below). The second argument is a String containing parameters provided by the planner. If no parameters are specified, the string may be null. The constructor should call the super-class constructor.

3. The Adaptor Window

The part of the data stream that an adaptor has access to is called a window. An instance variable window is inherited by all adaptors from the Adaptor base class, providing all window operations. A window has two pointers into the data stream, a head and a tail. Adaptors adjust their view on the data stream by moving their head and tail along the stream using the window.expand() and window.contract() operations. The expand() operation adds more bytes to the window by moving the tail. The contract() operation shrinks the window by moving the head (passing those bytes to the next window). You can think of this as the adaptor window moving along the data stream (or the data stream moving past the window ... it's all relative).
                 window      window
                  tail        head
                    |           |
                    v           v
====================+===========+========================>  data stream
                   /             \
                  |     window    |
                  +---------------+
                 <--             <--
               expand()        contract()
Several flavors of expand() are provided: expand(), expand(int), expand(char), expand(byte). With no arguments, expand will block until at least one byte is available and then expand as far as possible. If an integer option is specified, the window is expanded by at least that number of bytes (perhap more). If a byte argument is provided, the window will expand until the specified byte is added to the window. Passing a char argument is equivalent to passing a byte argument. The char form is provided only to prevent such calls (particularly calls like expand('a') from calling the int form). Each of these forms returns the number of bytes added to the window.

Several flavors of contract() are provided: contract(), contract(Pointer), contract(int). With no arguments, all data in the window is pushed out of the window. When an integer argument is provided, that number of bytes are pushed off the head of the window. If a Pointer is given as an argument, the window is contracted up to that pointer (see Section 4).

4. Events

Some window operations can throw AsyncEventException. When this occurs, one of the following happened: the downstream endpoint closed, the upstream endpoint closed, or SwitchPlan occurred. Subclasses of AsyncEventExcption, such as ClosureException, InputClosedException, OutputClosedException, and PlanChangedException are actually thrown to allow these events to be differentiated. The methods inputClosed(), outputClosed(), and switchPlanInWindow(), can also be used to determine the window state. When the input closes, no more data will be available for expand() operations. Final data manipulation should be done, the window should be contracted, and the output should be closed using closeOutput(). If the output closes, no more data can be output using contract(). Normally, the adaptor should exit at this point.

The interface for handling replanning and failures is not yet complete. You may wish to assume that a SwitchPlan event will not occur.

The null adaptor (FMG.ConductorExt.adaptors.NullAdaptor) provides a nice framework for handling exceptions.

5. Data Access

Access to the window is possible using one or more DataAccessPointer objects. A DataAccessPointer can be moved within the window to allow an adaptor to read and modify data relative to the position of the pointer.

Several window operations are provided to operate on pointers.

insertPointer(Pointer)
Insert a pointer at the head of the window.
contract(Pointer)
Contract the window up to the position of the pointer.
getPosition(Pointer)
Count the number of bytes from the head of the window to the pointer.
countRemaining(Pointer)
Count the number of bytes from the pointer to the tail of the window.
A pointer can also be inserted relative to an existing pointer by calling insertPointer() on the existing pointer:
insertPointer(Pointer)
Insert the given pointer upstream of this pointer.
A pointer can be moved within the window using the following pointer operators:
moveToByte(byte)
Advance the pointer to immediately before the next instance of the given byte.
movePastByte(byte)
Advance the pointer to immediately after the next instance of the given byte.
advance(int)
Advance the pointer past int bytes.
advance()
Advance the pointer to the end of the window.
Care should be taken to ensure that no attempt is made to advance beyond the tail of the window.

Access to the data stream is possible using the following pointer operators:

distanceTo(byte)
Determine the distance to the next instance of the given byte.
getByte(int)
Get the byte at a position relative to the pointer.
getBytes(byte[], int, int, int)
Copy a chunk of bytes from the stream into the given byte array.
Modifications to the data stream are possible using the following operators:
replaceByte(int, byte)
Replace a byte at the given position with the given new value.
replaceBytes(byte[], int, int, int, int)
Replace a group of bytes at the given position with another group of bytes from the given array.
replaceBytes(byte[], int, int, DataAccessPointer)
Replace a group of bytes deliniated by two DataAccessPointers with another group of bytes from the given array.
Methods that modify the data stream automatically maintain proper segmentation. If you replace a chunk of bytes, segments are combined as needed to put those bytes in the same segment. It is up to the adaptor to operate on the data in semantically meaningful ways. For instance, a stream cipher should replace each byte individually, leaving one-byte segments. An adaptor that adds lowsrc attributes to img tags in HTML should relace everything between "<" and ">" with one (or more) replace operation(s).

For operations that require multiple operations which form a single semantic operation or for operations that extend beyond the edge of the window, use the BlockModPointer. The BlockModPointer adds the methods blockmodStart() and blockModEnd(). When in block modification mode, the pointer causes all segments operated on to become part of a single segment. Also, all data passed as the BlockModPointer advances while in block modification mode will become part of that segment.

When a pointer is no longer needed, it can be removed from the window using removeSelf().

6. Inter-adaptor Communication

Adaptors can communicate with one another using the inter-adaptor communication facility. Inter-adaptor communiation is localized to each node. No inter-node communication facility is currently present, besides the data stream itself. The IAC object is used to provide two caches to each stream. One cache is for stream-specific data, the other is for node-global data. The Adaptor methods getStreamIAC() and getNodeIAC() are provided to obtain access to IAC objects.

Each IAC object provides the following operations:

lookup(String)
Get an object stored under a particular key. If it's not present, return null.
waitFor(String)
Get an object stored under a particular key. If it's not present, wait for it to be stored.
store(String, Object)
Store an object under a particular key. If another object exists with that key, overwrite it.
store(String, Object, boolean)
Store an object under a particular key, with optional overwrite.
remove(String)
Remove an object stored under a particular key.

7. Support for Standard Interfaces

Converting tools and existing code to operate on an adaptor window can be tricky. If the code in question uses the InputStream. and OutputStream interfaces, than you may wish to use the WindowInputStream and WindowOutputStream classes (in the FMG.ConductorExt.adaptors.tools package), which map the java I/O interface into window operations. Using these classes, a piece of adapting code that expects these interfaces can transparently read data from the window, arbitrarily process that data and write the results back to the window. However, since Conductor cannot determine what modifications (if any) were performed on the data, all data processed in this way becomes a single semantic segment.

8. Filesystem Access

All adaptor filesystem accesses should be made through the StorageAccess object. (Currently nothing prevents an adaptor from simply using the java.io classes, but someday this should be fixed.)

The StorageAccess object provides both authenticated and unauthenticated access to the file system. Unauthenticated access is allowed only within a Conductor "working directory," which is, by default, located in /tmp/conductor/anonymous. Authenitcate access to a user's personal working directory is also allowed. A user's personal working directory is typically a sub-directory called "conductor" in their home directory. If the user's home directory is not available, a sub-directory is created for the user in the working area, typically /tmp/conductor/username. In addition, authenticated users may access any arbitrary files in the filesystem that are accessible by the authenticated user.

The working directory can (optionally) be divided according to Conductor module. In particular, each adaptor may wish to use a unique module name to prevent storage conflicts. Four constructors are available for the StorageAccess object to create unauthenticated or authenticated objects with or without a specific module name.

Once created, a StorageAccess object allows files to be open for reading and writing. An authenticated user can use writeFile(String) and readFile(String) to gain access to arbitrary files in the file system (subject to the rights of that user). All users can use writeWorkingFile(String), writeWorkingFile(String, boolean), and readWorkingFile(String) to obtain access to files in their working area. These methods return objects of type StorageAccess.ReadAccess or StorageAccess.WriteAccess, which simply provide access to an input or output stream and the absolute name of the file.

9. Controlling Downstream Buffering

Data written by adaptors is buffered downstream before it is transmitted over the physical chanel. Conductor provides buffering between adaptors and between each adaptor and the module that writes messages to a socket. Sockets themselves also have internal buffers for data written to the socket but not yet physically sent.

Some adaptors may which to control the amount of buffering allowed downstream. For instance, an adaptor that wishes to stop the flow of data (perhaps in defference to a higher proiority stream) can only do so properly if downstream buffering is limited.

The AdaptorWindow method requestDownstreamBuffering(int) allows an adaptor to specify a change in the buffering downstream from that adaptor. The change is effective for all data contracted out of the window following the request. The argument specifies the desired level of buffering. Acceptable values are MAX_BUFFERING, MIN_BUFFERING, and DEFAULT_BUFFERING.

Note: This method has only been tested for use in the last adaptor on a particular node. Use with care.

10. Security Adaptors

Conductor can provide a secure planning mechanism. When secure planning is used, Conductor can also provide key distribution for encryption and decryption adaptors.

To allow key distribution, an adaptor must be prepared to generate a key for distribution by Conductor. Security adaptors should implement two extra methods: getKeyLabel(String) and generateKey(String). When a pair of adaptors (say encryption and decryption) are used, only both should implement these methods.

getKeyLabel(String) returns a String which provides a key label. After selecting a set of adaptors for a stream, Conductor will query each adaptor for a key label, passing the parameters that will be passed to the adaptor at instantiation as a string argument. Conductor will request generation of and distribute one key for each unique label. So, a different key label should be used for each non-interopeable key. For instance, if two adaptors encrypt using DES, they can have the same key label. However, an adaptor that encrypts using a differen algorithm should have a different label, since it would need a differen key.

Key generation is accomplished through a call to generateKey(String), which returns a byte array. The argument provided is the argument that will be provided to the adpator at instantiation. Each generated key is security delivered to each node that requires a key with that key label. The keys are delivered to adaptors using the IAC mechanism. Thus, each adaptor can obtain the key by calling

	(byte[])getStreamIAC().waitFor(getKeyLabel(parameter))
More information about these methods may be found in the class documentation for the SecurityBox class.

More information on secure planning is also available here.

11. Paired Adaptors

Many adaptations require two adaptors to work as a pair (e.g., encryption and decryption, compression and decompression, etc). Conductor's failure recovery capability will ensure that if one of a pair of adaptors fails, they will both be correctly removed from stream processing (and perhaps replaced). In order for this to work, however, an adaptor must indicate to Conductor that it is paired.

Paired adaptors should implement the following static method:

   public static CompositionType compositionType(String parameters);
Unpaired adaptors can either omit implementation of this method or return CompositionType.NONE. The upstream adaptor of a pair should return CompositionType.PUSH, while the downstreawm adaptor should return CompositionType.POP. Note that the adaptor parameter (as specified in the plan) is passed to this method, allowing an adaptor to use its parameters to determine which role to take.
Conductor is a product of Mark Yarvis (yarvis@fmg.cs.ucla.edu) and the FMG Research Group at UCLA's Department of Computer Science.
Copyright © 2001 The Regents of the University of California. All Rights Reserved.