Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
type | No | - | No | The value must be "rdb-tables". | "rdb-tables" |
description | No | - | No | Name of the credential object. | "My RDB Credential" |
properties | No | - | No | Configuration object | |
username | No | - | No | User name. | "admin" |
password | No | - | No | Password. | "adminPassword" |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "type": "rdb-tables", "description": "My RDB Credential", "properties": { "username": "admin", "password": "adminPassword" } } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
type | No | - | No | The value must be "rdb-tables". | "rdb-tables" |
description | No | - | No | Name of the connection object. | "My RDB Connection" |
throttlePolicy | Yes | - | No | Id of the throttle policy that applies to this connection object. | "6b235b333a1b" |
routingPolicies | Yes | [ ] | Yes | The ids of the routing policies that this connection will use. | ["17f75ce7d0c7", "d42780003b36"] |
credential | No | - | No | Id of the credential | "6b235b333a1b" |
properties | No | - | No | Configuration object | |
jdbcUrl | No | - | No | The JDBC url for your RDBMS server and database. Use database marker {DATABASE} to denote the database | "jdbc:mysql://db:3306/{DATABASE}" |
jdbcDriverJar | No | - | No | Path to the JDBC driver jar file for your RDBMS. | "/lib/myjdbcdriver.jar" |
jdbcDriverClass | Yes | see notes | No | The name of the default JDBC driver class (if the class name from the META-INF/services/java.sql.Driver file in the driver Jar file should not be used), or if that file does not exist in the driver jar file (Oracle) | "java.sql.Driver" |
jdbcDriverClasspath | Yes | the driver jar file | No | The class path for external jars required by the jdbc driver | |
stopOnError | Yes | true | No | When selected, the scan stops if the JDBC driver throws an error while getting a row, and the crawl halts. When unselected, the connector attempts to get subsequent rows | false |
useSlices | Yes | false | No | Select this option to divide the full SQL into multiple slices. For example,if you have a 10 Million row table to scan, executing the 10 M query will take a while. After it completes, the connector starts sending items. By activating 10 slices, the scan is split into 10 1 Million scans, which takes less time and you can view results faster. This only works when the idColumn contains an integer. | true |
numSlices | Yes | - | No | The number of SQL slices to split fullSQL. Slicing the full SQL should improve the performance significantly if a big database is to be crawled. Only works when the id column is an integer | 10 |
percentAsMod | Yes | false | No | Use % Operator for Modulo. Check this option if you want to specify the MODULO operator to use for a particular Database system that doesn't recognize the MOD() function. "MOD()" is available for MySQL, PostgreSQL and Oracle. But systems like Microsoft SQL Server you must check this option | true |
customFetchSize | Yes | false | No | Check this box if you need to specify a fetch size to the JDBC driver to use when getting results | true |
fetchSize | Yes | 50 | No | This option indicates to the JDBC driver how it should do paging when retrieving results | 100 |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "type": rdb-tables, "description": "RDB_TEST", "properties": { "jdbcUrl":"jdbc:mysql://localhost:3307/{DATABASE}", "jdbcDriverJar" : "/lib/myjdbcdriver.jar", "jdbcDriverClass": null, "jdbcDriverClasspath": null, "stopOnError": true, "useSlices": false, "numSlices": 2, "percentAsMod": false, "customFetchSize": true, "fetchSize": 10 } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
id | No | - | No | Id of the connection to update | "d442adcab4b0", |
description | Yes | - | No | Name of the connection object. | "My RDB Connection" |
throttlePolicy | Yes | - | No | Id of the throttle policy that applies to this connection object. | "b3a9-6b235b333a1b" |
routingPolicies | Yes | [ ] | Yes | The ids of the routing policies that this connection will use. | ["17f75ce7d0c7", "d42780003b36"] |
credential | Yes | - | No | Id of the credential | "6b235b333a1b" |
properties | Yes | - | No | Configuration object | |
(see create connection) |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "id": "89d6632a-a296-426c-adb0-d442adcab4b0", "description": "RDB_TEST", "properties": { "jdbcUrl":"jdbc:mysql://localhost:3307/{DATABASE}", "jdbcDriverJar" : "/lib/myjdbcdriver.jar", "jdbcDriverClass": null, "jdbcDriverClasspath": null, "stopOnError": true, "useSlices": false, "numSlices": 2, "percentAsMod": false, "customFetchSize": true, "fetchSize": 10 } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
seed | No | - | No | The name of the database. It will replace the marker {DATABASE} used in the field jdbcUrl of connection object | "test_db" |
type | No | - | No | The value must be "rdb-tables". | "rdb-tables" |
description | No | - | No | Name of the seed object. | "My RDB Seed" |
connector | No | - | No | The id of the connector to be used with this seed. The connector type must match the seed type. | "e3ca414b0d31" |
connection | No | - | No | The id of the connection to be used with this seed. The connection type must match the seed type. | "e4a663fe9ee6" |
workflows | Yes | [ ] | Yes | The ids of the workflows that will be executed for the documents crawled. | ["5696c3f0bda4"] |
throttlePolicy | Yes | - | No | Id of the throttle policy that applies to this seed object. | "6b235b333a1b" |
routingPolicies | Yes | [ ] | Yes | The ids of the routing policies that this seed will use. | ["17f75ce7d0c7", "d42780003b36"] |
tags | Yes | [ ] | Yes | The tags of the seed. These can be used to filter the seed | ["tag1", "tag2"] |
properties | No | - | No | Configuration object | |
fullSQL | No | - | No | Full. SQL. Run a "SELECT" query to retrieve all documents. This query is used only for full scans. Use the "WHERE" clause to specify any required condition for crawling just the desired documents. If slicing is enabled, add a "WHERE" clause: "SELECT idCol, col1, col2, col3 FROM data_table WHERE {SLICES} | "SELECT * FROM table" |
idColumn | |||||
stringIdColumn | |||||
postCrawlSQL | |||||
Incremental Crawl | |||||
preUpdateSQL | |||||
updateSQL | |||||
postUpdateSQL | |||||
postUpdateFailedSQL | |||||
seqColumn | |||||
actionColumn | |||||
useBounding | |||||
boundingSQL | |||||
ACL | |||||
aclColumn | |||||
aclSQL |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "type": "group-expansion", "seed": "seed_placeholder", "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31", "description": "GroupExpansion_Test_Seed", "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b", "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"], "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6", "workflows": ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"], "tags": ["tag1", "tag2"], "properties": { "seedsIncluded": [ "5b9f4074-01a3-4d86-8642-32a6f27179d4" ], "seedsExcluded": [], "seedsIdentities": [ { "id": "5b9f4074-01a3-4d86-8642-32a6f27179d4", "identityKeyType": "local", "identityKeyAttr": "someAttrName" } ] } } |
Field | Optional | Default | Multiple | Notes | Example |
---|---|---|---|---|---|
id | No | - | No | Id of the seed to update | "2f287669-d163-4e35-ad17-6bbfe9df3778" |
(see the "Create seed" for other fields) |
Code Block | ||||
---|---|---|---|---|
| ||||
{ "id": "2f287669-d163-4e35-ad17-6bbfe9df3778", "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31", "description": "FileSystem_Test_Seed", "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b", "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"], "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6", "workflows": ["b255e950-1dac-46dc-8f86-1238b2fbdf27", "f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"], "tags": ["tag", "tag2"], "properties": { "seedsIncluded": [ "5b9f4074-01a3-4d86-8642-32a6f27179d4" ], "seedsExcluded": [], "seedsIdentities": [ { "id": "5b9f4074-01a3-4d86-8642-32a6f27179d4", "identityKeyType": "local", "identityKeyAttr": "someAttrName" } ] } } |