The Hadoop Distributed File system (HDFS) connector will crawl content from any given HDFS Cluster using the WebHDFS http interface.

Features

...

Some of the features of the HDFS connector include:

• Performs incremental crawling (so that only new/updated documents are indexed)
• Metadata extraction
• Is search engine independent
• Runs from any machine with HTTP access to the given HDFS Namenode
• Filter the crawled documents by paths (including file names) using regex patterns
• Supports Archive file processing, for more info visit Archive files processing