Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The Aspider Web Crawler Apache Kafka Connector can be configured using the Aspire Admin UI. It requires the following entities to be created:

  • Connection
  • Connector
  • Seed

Easy Heading Free
navigationTitleOn this Page
wrapNavigationTexttrue
navigationExpandOptionexpand-all-by-default

Create Connection 


  1. On the Aspire Admin UI, go to the connections page page Image Added
  2. All existing connections will be listed. Click on the new button buttonImage Added
  3. Enter the new connection description. 
  4. Select Apache Kafka from the Type list.
  5. General:
    1. Servers: The Kafka server(s) hostnames followed by the port. Ex. localhost:9092
    2. Stop on scan error: Check if you want the connector to stop crawling when an error is encountered.
      1. Schema Registry URL: the schema registry URL to connect to.
      2. Username: Schema Registry User Name.
      3. Password: Schema Registry Password.
  6. Starting offset for full crawl:
    1. Starting offsets: Starting offset for a full crawl.
    2. Debug: Debug messages on/off.


Create Connector Instance


For the creation of the Connector object using the Admin UI, check this page.


Create Seed 


  1. On the Aspire Admin UI, go to the seeds page pageImage Added
  2. All existing seed will be listed. Click on the new button buttonImage Added
  3. Enter the new seed description.
  4. Select Apache Select Apache Kafka from the Type list.
  5. Seed:
    1. Topic: Select the topic you want to crawl.
  6. Connector:
    1. Connector: Select a Kafka connector.
  7. Connection:
    1. Connection: Select a Kafka connection.
  8. Workflow:
    1. Workflows: Select the workflow(s) for the seed.