The Apache Kafka Connector can be configured using the Rest API. It requires the following entities to be created:

Credential
Connection
Connector
Seed

Below are the examples of how to create the Connection and the Seed. For the Connector, please check this page.

Easy Heading Free

navigationTitle	On this Page
wrapNavigationText	true
navigationExpandOption	expand-all-by-default

Create Connection

Field	Required	Default	Multiple	Notes	Example
type	Yes	-	No	The value must be kafkabe Kafka.	kafka
description	Yes	-	No	Name of the connection object.	"MyApache KafkaConnection"
throttlePolicy	No	-	No	Id ID of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids IDs of the routing policies that this connection will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
properties	Yes	-	No	Configuration object
server	yes	-	Yes	The Kafka server(s) hostnames fololowed followed by the port.	"localhost:9092"
stopOnScanError	yes	true	no	Check if you want the connector to stop crawing crawling when an error is encountered.	"true"
offsets	yes	earliest	no	Starting offset for ful a full crawl.	"offsets": { "altPull": "Alpha", "debug": "Test Text Outer field set" }

Example

Code Block

theme	RDark
title	POST aspire/_api/connections

{
  "type": "kafka",
  "description": "kafkaConnection_1",
  "properties": {
    "server": "localhost:9093",
    "stopOnScanError": true,
    "offsets": {
      "altPull": "Alpha",
      "debug": "Test Text Outer field set"
    }
  }
}

Update Connection

Field	Required	Default	Multiple	Notes	Example
id	Yes	-	No	Id ID of the connection to update	"89d6632a-a296-426c-adb0-d442adcab4b0",
description	No	-	No	Name of the connection object.	"MyImage RemovedMyApacheKafka Connection"
throttlePolicy	No	-	No	Id ID of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids IDs of the routing policies that this connection will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
properties	Yes	-	No	Configuration object

Example


server	yes	-	Yes	The Kafka server(s) hostnames followed by the port.	"localhost:9090"
stopOnScanError	yes	true	no	Check if you want the connector to stop crawling when an error is encountered.	"false"
offsets	yes	earliest	no	Starting offset for a full crawl.	"offsets": { "altPull": "Alpha", "debug": "Test Text Outer field set" }

Example

Code Block

theme	RDark

{
  "id": "890083fd-39a2-47e7-a213-0051f4715cda",
  "type": "kafka",
  "description": "kafkaConnection description",
  "credential

Code Block

theme	RDark

{
  "type": "kafka",
  "description": "Kafka Demo",
  "properties": {
      "multipleStartPoints": "false",
      "url": "//opt//aspire",
      "partialScan": "true",
      "subDirUrl": null,
      "indexContainersthrottlePolicy": "false"null,
      "scanRecursivelyproperties": "true",
  {
    "scanExcludedItemsserver": "falselocalhost:9090",
      "ignoreScanErrorsstopOnScanError": "false",
      "ignoreSymLinksoffsets": "false",{
      "useACLsaltPull": "falseAlpha",
      "aclsdebug": null,
"Test Text Outer field   "includes": null,set"
      "excludes": null}
    }
}

Update Connection

Field

Required

Default

Multiple

Notes

Example

idYes-NoId of the connection to update"89d6632a-a296-426c-adb0-d442adcab4b0",descriptionNo-NoName of the connection object.

"MyImage RemovedConnection"

throttlePolicyNo-NoId of the throttle policy that applies to this connection object."f5587cee-9116-4011-b3a9-6b235b333a1b"routingPoliciesNo[ ]YesThe ids of the routing policies that this connection will use.

["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]

propertiesYes-NoConfiguration objectstopOnScanErrorNotrueNoIf enabled, the crawl will stop if there is an error on the scan phase.true / falseurlYes-NoPath of the base directory to crawl. All the seeds will be prefixed with this value to form the full path. All seeds will be prefixed with this value to form the full path"C:\\Directory"

Example

Create Connector Instance

For the creation of the Connector object using the Rest API check this page

Update Connector Instance

For the update of the Connector object using the Rest API check this page

Create Seed

Field	Required	Default	Multiple	Notes	Example
seed	Yes	-	No	<seed description>	"topic"
type	Yes	-	No	The value must be kafkabe dbconnector.	"kafka"
description	Yes	-	No	Name of the seed object.	"My Apache Kafka My ApacheKafka Connector Seed"
connector	Yes	-	No	The id ID of the connector to be used with this seed. The connector type must match the seed type.	"82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31"
connection	Yes	-	No	The id ID of the connection to be used with this seed. The connection type must match the seed type.	"602d3700-28dd-4a6a-8b51-e4a663fe9ee6"
workflows	No	[ ]	Yes	The ids IDs of the workflows that will be executed for the documents crawled.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
throttlePolicy	No	-	No	Id ID of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids IDs of the routing policies that this seed will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
tags	No	[ ]	Yes	The tags of the seed. These can be used to filter the seed.	["tag1", "tag2"]
properties	Yes	-	No	Configuration object
seed	Yes	-	No	Seed to crawl.	"topic"

Example

Code Block

theme	RDark
title	POST aspire/_api/seeds

{
  "seed": "topic",
  "type": "kafka",
    "seeddescription": "directorykafkaSeed_1",
    "connector": "82f7f0a49c59ad21-8d284321-47ce41c9-8c9daaad-e3ca414b0d312a5e30145b83",
    "descriptionconnection": "kafka_Test_Seed890083fd-39a2-47e7-a213-0051f4715cda",
    "connectionworkflows": [
    "602d3700f51f5162-28ddfaaf-4a6a43dc-8b51-e4a663fe9ee6"aa89-3b872160f504"
  ],
  "tags": [],
  "properties": {
    "workflowsseed": ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]"topic"
  }
}

Update Seed

Field

Required

Default

Multiple

Notes

Example

id


seed	Yes	-	No

Id of the seed to update."2f287669-d163-4e35-ad17-6bbfe9df3778"

<seed description>	"topic2"
type	Yes

seedNo

-

No

The value must be dbconnector.	"kafka"
description

No

Yes

-

No

Name of the seed object.

"

MyApache KafkaSeed

My Kafka connector Seed"

connector

No

Yes

-

No

The

id

ID of the connector to be used with this seed. The connector type must match the seed type.	"82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31"
connection

No

Yes

-

No

The

id

ID of the connection to be used with this seed. The connection type must match the seed type.	"602d3700-28dd-4a6a-8b51-e4a663fe9ee6"
workflows	No	[ ]	Yes	The

ids

IDs of the workflows that will be executed for the documents crawled

.["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]workflows.addNo[ ]YesThe ids of the workflows to add.["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]workflows.removeNo[ ]YesThe ids of the workflows to remove

.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
throttlePolicy	No	-	No

Id

ID of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The

ids

IDs of the routing policies that this seed will use.

["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7",

"b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]routingPolicies.addNo[ ]YesThe ids of the routingPolicies to add.[

"b4d2579f-1a0a-4a8b-9fd4-d42780003b36

"]routingPolicies.removeNo[ ]YesThe ids of the routingPolicies to remove.["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7

"]
tags	No	[ ]	Yes	The tags of the seed. These can be used to filter the seed.	["tag1", "

tag3

tag2"]

tags.add

properties	Yes	-	No

[ ]

Configuration object
seed	Yes

The tags to add["tag4"]

-

No

Seed to crawl.

"topic2"

tags.removeNo[ ]YesThe tags to remove["tag2"]

Example

Code Block

theme	RDark
title	PUT aspire/_api/seeds/2f287669-d163-4e35-ad17-6bbfe9df3778

{
    "id": "2f2876695f7b2877-d163ccc1-4e3541f7-ad17a182-6bbfe9df37780733712bd7a8",
  "type": "kafka",
  "seed": "C:\\test\\topic_copy",
    "connector": "82f7f0a49c59ad21-8d284321-47ce41c9-8c9daaad-e3ca414b0d312a5e30145b83",
    "description": "kafkakafkaSeed_Test_Seed1",
  "throttlePolicy": null,
  "throttlePolicyconnection": "6b8b5f23890083fd-fc7739a2-47a147e7-9b58a213-106577162e7b0051f4715cda",
    "routingPoliciesworkflows": [
    "313de87cf51f5162-3cb9faaf-4fe043dc-a2cbaa89-17f75ce7d0c73b872160f504"
  ],
  "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"tags": [],
    "connectionproperties": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",{
    "workflowsseed": ["b255e950-1dac-46dc-8f86-1238b2fbdf27"],"topic_copy"
  }
}

Page tree

Versions Compared

Old Version 2

New Version Current

Key

Create Connection

Example

Update Connection

Example

Example

Update Connection

Example

Create Connector Instance

Update Connector Instance

Create Seed

Example

Update Seed

Example

Page tree

Page History

Versions Compared

Old Version 2

New Version Current

Key

Create Connection

Example

Update Connection

Example

Example

Update Connection

Example

Create Connector Instance

Update Connector Instance

Create Seed

Example

Update Seed

Example