Step 3: Initiate a Full Crawl
Now that the content source is set up, the crawl can be initiated.
- Click on the crawl type option to set it as "Full" (is set as "Incremental" by default and the first time it'll work like a full crawl.
- After the first crawl, set it to "Incremental" to crawl for any changes done in the repository).
- Click Start.
During the Crawl
- Click on the "Refresh" button on the Content Sources page to view the latest status of the crawl.
- The status will show RUNNING while the crawl is going, and CRAWLED when it is finished.
- Click on "Complete" to view the number of documents crawled so far, the number of documents submitted, and the number of documents with errors.
If there are errors, you will get a clickable "Error" flag that will take you to a detailed error message page.
Step 4: Initiate an Incremental Crawl
If you only want to process content updates from the Aspider Web Crawler (documents which are added, modified, or removed), then click on the "Incremental" button instead of the "Full" button.
- The Aspider Web Crawler connector will automatically identify only changes which have occurred since the last crawl.
- If this is the first time that the connector has crawled, the action of the "Incremental" button depends on the exact method of change discovery.
- It may perform the same action as a "Full" crawl crawling everything, or it may not crawl anything. Thereafter, the Incremental button will only crawl updates.