Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
type | No | - | No | The value must be "rdb-tables". | "rdb-tables" |
description | No | - | No | Name of the credential object. | "My RDB Credential" |
properties | No | - | No | Configuration object | |
username | No | - | No | User name. | "admin" |
password | No | - | No | Password. | "adminPassword" |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "type": "rdb-tables", "description": "My RDB Credential", "properties": { "username": "admin", "password": "adminPassword" } } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
type | No | - | No | The value must be "rdb-tables". | "rdb-tables" |
description | No | - | No | Name of the connection object. | "My RDB Connection" |
throttlePolicy | Yes | - | No | Id of the throttle policy that applies to this connection object. | "6b235b333a1b" |
routingPolicies | Yes | [ ] | Yes | The ids of the routing policies that this connection will use. | ["17f75ce7d0c7", "d42780003b36"] |
credential | No | - | No | Id of the credential | "6b235b333a1b" |
properties | No | - | No | Configuration object | |
jdbcUrl | No | - | No | The JDBC url for your RDBMS server and database. Use database marker {DATABASE} to denote the database | "jdbc:mysql://db:3306/{DATABASE}" |
jdbcDriverJar | No | - | No | Path to the JDBC driver jar file for your RDBMS. | "/lib/myjdbcdriver.jar" |
jdbcDriverClass | Yes | see notes | No | The name of the default JDBC driver class (if the class name from the META-INF/services/java.sql.Driver file in the driver Jar file should not be used), or if that file does not exist in the driver jar file (Oracle) | "java.sql.Driver" |
jdbcDriverClasspath | Yes | the driver jar file | No | The class path for external jars required by the jdbc driver | |
stopOnError | Yes | true | No | When selected, the scan stops if the JDBC driver throws an error while getting a row, and the crawl halts. When unselected, the connector attempts to get subsequent rows | false |
useSlices | Yes | false | No | Select this option to divide the full SQL into multiple slices. For example,if you have a 10 Million row table to scan, executing the 10 M query will take a while. After it completes, the connector starts sending items. By activating 10 slices, the scan is split into 10 1 Million scans, which takes less time and you can view results faster. This only works when the idColumn contains an integer. | true |
numSlices | Yes | - | No | The number of SQL slices to split fullSQL. Slicing the full SQL should improve the performance significantly if a big database is to be crawled. Only works when the id column is an integer | 10 |
percentAsMod | Yes | false | No | Use % Operator for Modulo. Check this option if you want to specify the MODULO operator to use for a particular Database system that doesn't recognize the MOD() function. "MOD()" is available for MySQL, PostgreSQL and Oracle. But systems like Microsoft SQL Server you must check this option | true |
customFetchSize | Yes | false | No | Check this box if you need to specify a fetch size to the JDBC driver to use when getting results | true |
fetchSize | Yes | 50 | No | This option indicates to the JDBC driver how it should do paging when retrieving results | 100 |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "type": rdb-tables, "description": "RDB_TEST", "properties": { "jdbcUrl":"jdbc:mysql://%slocalhost:3307/{DATABASE}", "jdbcDriverJar" : "/lib/myjdbcdriver.jar", "jdbcDriverClass": null, "jdbcDriverClasspath": null, "stopOnError": true, "useSlices": false, "numSlices": 2, "percentAsMod": false, "customFetchSize": true, "fetchSize": 10 } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
id | No | - | No | Id of the connection to update | "d442adcab4b0", |
description | Yes | - | No | Name of the connection object. | "My RDB Connection" |
throttlePolicy | Yes | - | No | Id of the throttle policy that applies to this connection object. | "b3a9-6b235b333a1b" |
routingPolicies | Yes | [ ] | Yes | The ids of the routing policies that this connection will use. | ["17f75ce7d0c7", "d42780003b36"] |
credential | Yes | - | No | Id of the credential | "6b235b333a1b" |
properties | Yes | - | No | Configuration object | |
(see create connection) |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "id": "89d6632a-a296-426c-adb0-d442adcab4b0", "description": "RDB_TEST", "properties": { "jdbcUrl":"jdbc:mysql://%slocalhost:3307/{DATABASE}", "jdbcDriverJar" : "/lib/myjdbcdriver.jar", "jdbcDriverClass": null, "jdbcDriverClasspath": null, "stopOnError": true, "useSlices": false, "numSlices": 2, "percentAsMod": false, "customFetchSize": true, "fetchSize": 10 } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
seed | No | - | No | The name of the database. It will replace the marker {DATABASE} used in the field jdbcUrl of connection object | "test_db" |
type | No | - | No | The value must be "rdb-tables". | "rdb-tables" |
description | No | - | No | Name of the seed object. | "My RDB Seed" |
connector | No | - | No | The id of the connector to be used with this seed. The connector type must match the seed type. | "e3ca414b0d31" |
connection | No | - | No | The id of the connection to be used with this seed. The connection type must match the seed type. | "e4a663fe9ee6" |
workflows | Yes | [ ] | Yes | The ids of the workflows that will be executed for the documents crawled. | ["5696c3f0bda4"] |
throttlePolicy | Yes | - | No | Id of the throttle policy that applies to this seed object. | "6b235b333a1b" |
routingPolicies | Yes | [ ] | Yes | The ids of the routing policies that this seed will use. | ["17f75ce7d0c7", "d42780003b36"] |
tags | Yes | [ ] | Yes | The tags of the seed. These can be used to filter the seed | ["tag1", "tag2"] |
properties | No | - | No | Configuration object | |
fullSQL | |||||
idColumn | |||||
stringIdColumn | |||||
postCrawlSQL | |||||
Incremental Crawl | |||||
preUpdateSQL | |||||
updateSQL | |||||
postUpdateSQL | |||||
postUpdateFailedSQL | |||||
seqColumn | |||||
actionColumn | |||||
useBounding | |||||
boundingSQL | |||||
ACL | |||||
aclColumn | |||||
aclSQL |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "type": "group-expansion", "seed": "seed_placeholder", "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31", "description": "GroupExpansion_Test_Seed", "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b", "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"], "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6", "workflows": ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"], "tags": ["tag1", "tag2"], "properties": { "seedsIncluded": [ "5b9f4074-01a3-4d86-8642-32a6f27179d4" ], "seedsExcluded": [], "seedsIdentities": [ { "id": "5b9f4074-01a3-4d86-8642-32a6f27179d4", "identityKeyType": "local", "identityKeyAttr": "someAttrName" } ] } } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
id | No | - | No | Id of the seed to update | "2f287669-d163-4e35-ad17-6bbfe9df3778" |
(see the "Create seed" for other fields) |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "id": "2f287669-d163-4e35-ad17-6bbfe9df3778", "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31", "description": "FileSystem_Test_Seed", "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b", "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"], "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6", "workflows": ["b255e950-1dac-46dc-8f86-1238b2fbdf27", "f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"], "tags": ["tag", "tag2"], "properties": { "seedsIncluded": [ "5b9f4074-01a3-4d86-8642-32a6f27179d4" ], "seedsExcluded": [], "seedsIdentities": [ { "id": "5b9f4074-01a3-4d86-8642-32a6f27179d4", "identityKeyType": "local", "identityKeyAttr": "someAttrName" } ] } } |