Step 1. Launch Aspire and Open the Content Source Management Page
Step 2. Select or Add a
new File System Content Content Source
To specify exactly which shared folder to crawl, create a new Select a "Content Source" to work with.
Specify Basic Information
In the Workflow tab:
- Specify the workflow steps for the jobs that come out of the crawl.
- Drag and drop rules to determine which steps an item should follow after being crawled.
These rules could be where to publish the document or transformations needed on the data before sending it to a search engine. See Workflow for more information.
For this tutorial, drag and drop the Publish To File rule found under the Publishers tab to the onPublish Workflow tree.
- Specify a Name and Description for the Publisher.
- Click Add.
- Click Save and Done and you'll be sent back to the Home page.
Image Removed
Now that the HDFS writer is set up, a crawl can be initiated.
- When crawling the content will be written to HDFS
Now that the content source is set up, the crawl can be initiated.
- Click on the crawl type option to set it as Full. ("Incremental" is the default so the first time it'll work like a full crawl).
- After the first crawl, set it to Incremental to crawl for any changes done in the repository).
- ClickStart.
During the crawl, you can do the following:
- Click Refresh on the Content Sources page to view the latest status of the crawl.
- The status will show RUNNING while the crawl is going, and CRAWLED when it is finished.
- Click Complete to view the number of documents crawled so far, the number of documents submitted, and the number of documents with errors.
If there are errors, you will get a clickable "Error" flag that will take you to a detailed error message page.
If you only want to process content updates from the File System (documents that are added, modified, or removed), then
- Click Incremental instead of Full. The File System connector will automatically identify only changes which have occurred since the last crawl.
- If this is the first time that the connector has crawled, the action of the "Incremental" button depends on the exact method of change discovery.
- It may perform the same action as a "Full" crawl (crawling everything), or it may not crawl anything. Thereafter, the Incremental button will only crawl updates.
Info |
Statistics are reset for every crawl.