The Kafka connector fetches messages from a Kafka message stream.
Features
1) Performs incremental and full crawls.
- An incremental crawl obtains all messages that have been added after the last full crawl, and then keeps running until it is paused or stopped.
- A full crawl obtains all available messages from the topic stream at the time of crawl, and then stops.
2) Contains configurable message offsets (for full crawls).
Content Retrieved
The Kafka connector retrieves messages for the subscribed topic. The content body is the message value, which is retrieved by the text extraction component.
If you need the content element populated, then text extraction must be enabled.
Include
The following metadata are included:
- Message key
- Message value (same as content if text extraction is enabled)
- Topic name
- Partition number
- Message offset
Limitations
Cannot track expired messages; it cannot send updates or deletes to the publisher.