This section describes in details each of the components involved in the Connector Framework.
Component's Bundle
The main component's bundle jar component the framework uses is aspire-connector-framework, this bundle contains all the Stages, components and also provides interfaces for the specific connector implementations.
Connector AppBundle
In Aspire every component needs to be referenced from an AppBundle or application.xml file, which describes the job execution flow. For the Connector Framework we have one common AppBundle called app-rap-connector.
This AppBundle is automatically loaded when Aspire detects it needs to load a connector. It contains all the PipelineManagers, Pipelines and references to the Connector Framework Components and Stages from the aspire-connector-framework bundle.
PipelineManagers and Pipelines
Main (PipelineManager)
- controlPipeline
QueuePipelineManager
- jobStartEndPipeline
- crawlEndPipeline
ScanPipelineManager
- scanControlPipeline1
- scanControlPipeline2
- scanErrorPipeline
ScanChildrenPipelineManager
- scannedItemsPipeline
ProcessPipelineManager
- crawlStartEndPipeline
- crawlStartEndErrorPipeline
- processControlPipeline1
- processControlPipeline2
- addUpdatePipeline
- fetchAndExtractPipeline
- addUpdateWorkflowPipeline
- publishWorkflowPipeline
- deletePipeline
- deleteWorkflowPipeline
- errorPipeline
Components
CrawlController
ScanQueueLoader
ProcessQueueLoader
ProcessDeletes
CrawlEnd
ScanReleaseController
Scan
FlagContainer
IncludeExclude
CheckSnapshot
GenerateHierarchy
EnqueueScan
AddUpdateSnapshot
EnqueueProcess
ProcessReleaseController
ProcessCrawlRoot
PopulateOrDelete
FetchUrl
EnqueueScan
AddUpdateSnapshot
MarkProcessComplete
Overview
Content Tools