The Aspider Web Crawler Apache Kafka Connector can be configured using the Aspire Admin UI. It requires the following entities to be created:

• Connection
• Connector
• Seed

Create Connection 

1. On the Aspire Admin UI, go to the connections page
2. All existing connections will be listed. Click on the new button
3. Enter the new connection description. 
4. Select Apache Kafka from the Type list.
5. General:
   1. Servers: The Kafka server(s) hostnames fololowed followed by the port. Ex. localhost:9092
   2. Stop on scan error: Check if you want the connector to stop crawing crawling when an error is encountered.
   3. Enable Stop Consumer: If checked, the crawl will stop at a certain timeout.
   4. Schema Registry Configuration: If checked, the schema will be retrieved from Schema Registry. Schema Registry credentials are required.
      1. Schema Registry URL: the schema registry URL to connect to.
      2. Username: Schema Registry User Name.
      3. Password: Schema Registry Password.
6. Starting offset for ful full crawl: 
   1. Extract text: the auto offset reset configuration. Starting offsets: Starting offset for a full crawl.
   2. Debug: Debug messages on/off.

Create Connector Instance

For the creation of the Connector object using the Admin UI, check this page.

Create Seed 

1. On the Aspire Admin UI, go to the seeds page
2. All existing seed will be listed. Click on the new button
3. Enter the new seed description.
4. Select Apache Select Apache Kafka from the Type list.
5. Seed:
   1. Topic: Select the topic you want to crawl.
6. Connector:
   1. Connector: Select a Kafka connector.
7. Connection:
   1. Connection: Select a Kafka connection.
8. Workflow:
   1. Workflows: Select the workflow(s) for the seed.