...

• Performs incremental crawling (so that only new/updated documents are indexed)
• Metadata extraction
• Is search engine independent
• Runs from any machine with HTTP access to the given HDFS Namenode
• Filters the crawled documents by paths (including file names) using regex patterns
• Supports Kerberized Clusters by using a delegation token.
• Supports Archive file processing; for more information, visit Archive files processing

WebHDFS Operations

...

Only two operations are used by this connector:

• http://<host>:<port>/webhdfs/v1/<path>?op=OPEN
  • Used to fetch the files data to be used to extract its content.
    
    
• http://<host>:<port>/webhdfs/v1/<path>?op=LISTSTATUS
  • Used to scan a directory and get relevant file information like:
    • Last-Modified dates for incremental crawls.
    • Group and Owner