Storage Handler Commands

The configuration for the storage handler is a of storage operations which are performed sequentially.

<component name="StorageHandlerOpen" subType="default" factoryName="aspire-storage-handler">
  <commands>
    <mkdir path="data/vectors" />
      
    <delete path="data/vectors/{XML:APPLICANT_ID}.bak" />
    
    <rename fromPath="data/vectors/{XML:APPLICANT_ID}.vec" 
            toPath=="data/vectors/{XML:APPLICANT_ID}.bak" "/>
      
    <open componentRef="/system/vectorPipeline/storeVector"
              path="data/vectors/{XML:APPLICANT_ID}.vec" />
      
    <open variable="statsFileOut" path="data/vectors/stats.txt" objectType="PrintWriter" />
  </commands>
</component>

Command: <open> for custom storage for components

Use the following configuration for opening storage objects required by some other component in the system. This is especially useful for components which do things like write data for sub-jobs.

For example, this command was first implemented to create sequence files of Mahout vectors. A storage handler on the parent pipeline first opens the sequence file by sending an <open> command to the "StoreVector" component, which creates the storage object and stores it into the AspireObject as a variable.

Then for each job processed by the "StoreVector" component, StoreVector will fetch the object from the AspireObject variable (traversing up the sub-job/parent-job hierarchy if necessary) and will then store the vector using the previously opened storage object.

Finally, when the parent job is complete, a second storage handler will be called to close the storage object (or the pipeline manager will do it if no one else does).

Example

 <open componentRef="../sub-doc-pipeline/storageVector"
       path="data/vectors/{XML:APPLICANT_ID}.vec />

Command: <open> for file I/O

The Storage Handler can also be used to open file system files for I/O. The resulting file objects are stored as variables in the AspireObject which can then be used by Groovy scripts or other components for reading and writing files.

For example

     <open variable="statsFileOut" path="data/vectors/stats.txt" 
           objectType="PrintWriter" append="true" encoding="UTF-8" />

Command: <close>

Closes open storage. This operation will get the named variable from the AspireObject, and then will close the object (if it implements Closeable). Note that any variable on the document can be closed - it does not have to have been previously opened by a prior storage handler (although it typically is).

Example

 <close variable="statsFileOut"  />

Command: <closeAll>

Closes all variables opened previously by an earlier storage handler. Does not close all variables in the AspireObject, just the ones opened by any previous storage handler.

Contains no attributes.

Example:

 <closeAll/>

Command: <mkdir> for files and directories

Creates the specified directory. If the directory already exists, does nothing and does not report an error. Throws an exception if unable to create the directory. Automatically creates all parent directories if any of them do not already exist.

Example

 <mkDir path="data/vectors" />

Command: <delete> for files and directories

Deletes the specified directory or file. If the directory or file does not already exist, does nothing and does not report an error.

When deleting a directory, automatically deletes all of the files contained within the directory.

Returns an exception error if the deletion was unsuccessful (for example, a locked file which could not be deleted).

Example

 <delete path="data/vectors/{XML:APPLICANT_ID}.bak" />

Command: <rename> for files and directories

Renames a file or directory. If the "from" path does not exist, does nothing and does not report an error. Returns an exception error if the rename could not proceed (for example, due to a locked file or if the destination exists).

Example

 <rename fromPath="data/vectors/{XML:APPLICANT_ID}.vec" 
           toPath="data/vectors/{XML:APPLICANT_ID}.bak" />

Command: <write> for files and variables

Copy the content of specified variable to a given file location. If the content of the variable is instance of InputStream, Serializable or AspireObject write that content to the file otherwise write whole AspireObject to the file.

Implementing A Custom Component Source Object

To implement a component which requires custom storage management, do the following.

Step 1:  Add the "com.searchtechnologies:aspire-storage-handler" as a dependency to your project in your "pom.xml" file.

Step 2:  Have your pipeline stage implement AspireStorageInterface.

For example

 public class MyPipelineStage extends ComponentImpl implements AspireStorageInterface {
   .
   .
   .
 }

Step 3:  Create the code for the open() method required by AspireStorageInterface.

For example

 @Override
 public Closeable open(Job j, Element config) throws AspireException {
   String pathToOpen = StandardPathBuilder.build(this, config, j);
   
   // Create, initialize, and return the appropriate object for your custom storage
   //
   // Note that the object must implement "Closeable".
   //
   // If you use the "StandardPathBuilder", then the <open> command specified in the
   // storage handler must have an @path attribute specifying the location which contains
   // the storage to open.
   //
   MyNewStorageObject storageObject = new MyNewStorageObject(... pathToOpen ... );
   
   // Now, be sure to save the object in a variable so you can access it later.
   AspireObject doc = (AspireObject)job.get();
   
   doc.putVariable("MyComponentStorageVar", storageObject);
   
   return storageObject;
 }

Step 4:  Inside your pipeline stage, access the variable which contains the storage object as follows:

 @Override
 public void process(Job j) throws AspireException {
   MyNewStorageObject myStorage = 
     (MyNewStorageObject) JobDocumentHierarchyMap.get(j, "MyComponentStorageVar");
   
   .
   .
   .
 }

Note:  The use of JobDocumentHierarchyMap.get() which traverses the job hierarchy to fetch the specified variable from the appropriate AspireObject. This allows the storage handler to open the storage on a parent job, but then for the storage to be actually used in a sub-job pipeline.