Extracting custom mapped metadata for use in Amazon CloudSearch (Aspire 2)

The following is a quick tutorial on how to map custom metadata fields, extract them using the Aspire S3 connector, and send the data to Amazon CloudSearch.

Step 1: Create custom metadata fields in an S3 bucket

Create two custom metadata fields, “x-amz-meta-field1” and “x-amz-meta-field2” in a sample S3 bucket.

No mapping or clean-up is necessary to remove the “x-amz-meta-” prefix from custom metadata fields as the Aspire S3 connector removes that by default.

After crawling with the S3 connector, it will look like this in the AspireObject:

 <doc>
    <fetchUrl>Test/ReadMe.txt</fetchUrl>
    <docType>FILE</docType>
    <lastModified>2012-11-15T17:27:58Z</lastModified>
    <dataSize>316</dataSize>
    <contentType source="ExtractTextStage/Content-Type">text/plain</contentType>
    <crawlId>10</crawlId>
    <content source="ExtractTextStage">Read me file content</content>
    <connectorSpecific type="s3">
      <field name="ETag">cf31821f5aa14facf1bcd714eeb31f6a</field>
      <field name="Content-Length">316</field>
      <field name="Last-Modified">Thu Nov 15 11:27:58 CST 2012</field>
      <field name="Accept-Ranges">bytes</field>
      <field name="Content-Type">text/rtf</field>
      <field name="field1">test1</field>
      <field name="field2">test2</field>
    </connectorSpecific>
    <snapshotUrl>003 my-first-s3-bucket-1-0000000001/Test/ReadMe.txt</snapshotUrl>
    <action>add</action>
    ...
    ...
  </doc>

See S3 Application Bundle connector documentation for more details.

Step 2: Define custom fields in Amazon CloudSearch

The fields in Amazon CloudSearch are defined as follows:

Both fields are type “text” and have the “result” facet column checked so that those can be seen in the search results.

Step 6: Update the XSL used by the Publish to CloudSearch application

To update the XSL used by your Publish to CloudSearch application, do the following:

1. Copy the modified XSL to ${aspire.home}/config/xsl/ folder in your Aspire distribution (where you are running Aspire from). Download File:CustomMetadata-AspireToCloudSearch.xsl to see the final version of the XSL.
2. Go to the Aspire Home and update your installed Publish to CloudSearch application.
   1. Click on the Content Source.
   2. Click on the Workflow tab and drag and drop or double-click on the existing Publish to CloudSearch application.
   3. Enter the path to the updated XSL. This path is relative to ${aspire.home}, so there is no leading “/” character. You can use the same “config/xsl/aspireToCloudSearch.xsl”.
   4. Click on the “Add” button.

After that, run your S3 connector source and you should see the custom fields on CloudSearch.

Handling deletes

To handle deletes, install a scripting application to encode the document URL to MD5. The AspireObject for deletes is slightly different and does not contain the MD5 by default. Example:

  <doc>
     <folder>Test/1995/ed465240.pdf</folder>
     <fetchUrl>my-s3-bucket-1-0000000001/Test/1995/ed465240.pdf</fetchUrl>
     <docType>FILE</docType>
     <snapshotUrl>004 my--s3-bucket-1-0000000001/Test/1995/ed465240.pdf</snapshotUrl>
     <bucket>my-s3-bucket-1-0000000001</bucket>
     <action>delete</action>
     <connectorSource type="s3">
        <parentUrl/>
        <displayName>TestS3</displayName>
        <indexFolders>true</indexFolders>
        <scanSubFolders>true</scanSubFolders>
      </connectorSource>
     <crawlId>21</crawlId>
  </doc>

Adding the scripting application

1. Go to the Content Source configuration and click on the workflow tab.
2. Click on the Base Functions drag and drop the Custom (Custom Groovy Script) to the OnPublish event.
3. Edit description of the groovy script.
4. In the Script text field, enter the Groovy script below, then click on the “Add” button:

import com.searchtechnologies.aspire.framework.utilities.StringUtilities;
   if(doc.getText("action") == "delete") {
   doc.add("md5", StringUtilities.stringToMD5(doc.getText("fetchUrl")));
   }

• The import is required to be able to use the class StringUtilities.
• The method doc.getText will read a direct child of the AspireObject.
• <doc><action/></doc> will always be “delete” in when a delete was detected in S3. We only want to do something special in those situations. 
• The method doc.add will add a new child to the AspireObject. In this case, named “md5”.
• The method StringUtilities.stringToMD5 converts a string to its MD5 value.

Make sure you add the new scripting into the OnPublish event. In this example, the scripting application is named “HandleDeletes”. Place it before your PublishToCloudSearch application.

After this, you will start seeing deletions in CloudSearch. More documentation on Amazon CloudSearch document deletion here. Finally, all this customization is required because Amazon CloudSearch fields are not predefined. This means, that this the XSL customization has to be done for each set of CloudSearch index fields you have.