1. Launch Aspire (if it's not already running).
2. Go to Launch Control.
3. Browse to: http://localhost:50505

For details on using the Aspire Content Source Management page, see Admin UI.

1. From Content Source, click Add Source to add a new content source or select and existing one.
2. Click File System Connector.

In the General tab in the Content Source Configuration window, specify basic information for the content source:

1. Enter a content source name in the Name field. 
   1. This is any useful name which you decide is a good name for the source. It will be displayed in the content source page, in error messages, etc.
2. Click on the Scheduled pulldown list and select one of the following: Manually, Periodically, Daily, Weekly or Advanced.
   1. Aspire can automatically schedule content sources to be crawled on a set schedule, such as once a day, several times a week, or periodically (every N minutes or hours).For the purposes of this tutorial, you may want to select Manually and then set up a regular crawling schedule later.
3. Click on the Action pull-down list to select one of the following: Start, Stop, Pause, or Resume.
   1. This is the action that will be performed for that specific schedule.
4. Click on the Crawl pull-down list and select an option such as: Incremental, Full, Real Time, or Cache Groups.
   1. This will be the type of crawl to execute for that specific schedule.
      

After selecting a Scheduled, specify the details, if applicable:

• Manually: No additional options.
• Periodically: Specify the "Run every:" options by entering the number of "hours" and "minutes."
• Daily: Specify the "Start time:" by clicking on the hours and minutes drop-down lists and selecting options.
• Weekly: Specify the "Start time:" by clicking on the hours and minutes drop-down lists and selecting options, then clicking on the day checkboxes to specify days of the week to run the crawl.
• Advanced: Enter a custom CRON Expression (e.g. 0 0 0 ? * *)

In the Connector tab, specify the connection information to crawl the File System.

1. URL: Enter the folder path you want to crawl.

   • For Windows: D:\folder\folder1\

   • For Linux: /home/user/folder/folder1/
2. Partial Scan: Check if you want to run a partial scan (i.e. only scan a portion of a large directory)
   
   • Sub Directory to Scan: Enter a relative path (relative to the URL path)
     
3. Index Folders?: Index subfolders as items. If the check box is cleared, only files will be indexed.
4. Scan Recursively?: Scan through subfolders' child nodes.
5. Scan Excluded Items: If selected, sub items will be scanned from excluded containers (excluded by the Include/Exclude patterns).
6. Use Fixed ACLs: Select to use a fixed acl that will be attached to all of the fetched files.
   • Users: Fixed user acls
     • Domain: User's domain
     • Name: Username
     • Type: Select Allow or Deny
   • Groups: Fixed group acls
     • Name: Group's name
     • Type: Select Allow or Deny
7. Include/Exclude patterns: Enter regex patterns to include or exclude files/folders based on URL matches.

"Connector" tab:

• Locate the "Extract Text Properties" section
  • Check "Extract Text Configuration"
    • To show the configuration parameters
  • Check "Disable text extraction"
    • To disable the text extraction

If you need text extraction, you will need to add a text extraction stage in to the work flow later

In the Workflow tab:

install the binary writer either by dragging it from the Applications section of the workflow configuration or by adding a custom application with the group id “com.searchtechnologies.aspire” and the artifact id “app-hdfs-binary writer”

Configure the parameters

• HDFS Base
  • The base directory on the hdfs file system under which all content will be written
  • Form
    • hdfs://<server>:<port>/<path>
  • Example
    • hdfs://localhost:9000/myBase
• Get content source from document
• When checked, the content source name with be automatically extracted from the Aspire document. The field holding the value can be configured below
• Content source field
  • The field in the Aspire document in which the content source name is held, used when getting the content source from the document
  • Form
    • /path/to/field
  • Example
    • /doc/sourceId
• Content source
  • The static value of content source name to use when not getting the content source from the document
  • Form
    • name
  • Example
    • myContentSource
• Use default ID field
  • By default, the id field is taken from the fetchUrl field. If you want to use a different field, uncheck this box.
• Id field
• When not using the default field for the document id, enter the field that contains the document id here.
• Form
  • /path/to/field
• Example
  • /doc/fetchUrl
• Hash bits
  • The number of bits of the hash to use for the HDFS directory name. The filename is constructed using the MD5 hash of the document ID and the original filename. The file is then stored in a directory whose name is the least significant n bits of the hash, where n is the number of bits given here.
  • Form
    • <number>
  • Example
    • 16
• Directory bits
  • The number of bits of the hash to use for the grouping the directories.
  • Form
    • <number>
  • Example
    • 8
  • Suppress deletes
    • Checking this box will prevent binary files being deleted when a delete action is encountered. Instead a “marker file” will be left to indicate the binary was deleted
• HDFS Options
  • Security
    • Choose the type of security to use to access the HDFS file system- Kerberos or none
  • User principle
    • The principal user for Kerberos
  • Key tab file
    • The user's key tab file
    • Form
      • File path
    • Example
      • config/myUser.keytab
  • Add resources
    • Check this box if you need to add Hadoop resources to the configuration (such as site-core.xml)
    • Resource file
      • The path to a resource file to add to the configuration
      • Form
        • /path/to/file
      • Example
        • config/core-site.xml
  • Block size
    • The size of block to be used when accessing the HDFS file system
    • Form
      • <number> <unit>
    • Example
      • 32mb
  • Buffer size
    • The size of buffer to be used when accessing the HDFS file system
    • Form
      • <number> <unit>
    • Example
      • 32kb
  • Replication
    • The HDFS replication factor
    • Form
      • <number>
    • Example
      • 3
• Debug
  • Check this box to enable debug messages