The File System Connector can be configured using the Rest API. It requires the following entities to be created:
- Connection
- Connector
- Seed
Bellow are the examples of how to create the Connection and the Seed. For the Connector please check this page.
Create Connection
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| type | Yes | - | No | The value must be "filesystem". | "filesystem" |
| description | Yes | - | No | Name of the connection object. | "MyFileSystemConnection" |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this connection object. | "f5587cee-9116-4011-b3a9-6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this connection will use. | ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"] |
| properties | Yes | - | No | Configuration object | |
| url | Yes | - | No | Path of the base directory to crawl. All the seeds will be prefixed with this value to form the full path. All the seeds will be prefixed with this value to form the full path | "C:\Directory" |
| ignoreSymLinks | No | false | No | If enabled symbolic links will not be processed and links in the root items will cause an error. | true / false |
| stopOnScanError | No | true | No | If enabled, the crawl will stop if there is an error on the scan phase. | true / false |
| indexContainers | No | false | No | Enable to index the directories. | true / false |
| scanRecursively | No | true | No | Enable to scan discovered directories recursively. | true / false |
| include | No | [ ] | Yes | Patterns to match against document URL, if any of them match, the document will be included in the crawl. | [ ".*pdf$", ".*docx$" ] |
| exclude | No | [ ] | Yes | Patterns to match against document URL, if any of them match, the document will be excluded from the crawl. | [ ".*png$", ".*jpeg$" ] |
| scanExcludedItems | No | false | No | Enable to force the scan of excluded directories, so child items within the scope can be found. | true / false |
| staticAcl | No | [ ] | Yes | Static ACL configuration object | |
| name | Yes | - | No | Name of the static ACL. | "group1" |
| domain | No | "" | No | Domain of the static ACL. | "testDomain" |
| entity | No | "user" | No | Entity (user / group) represented by the static ACL. | "user" / "group" |
| access | No | "allow" | No | Access (allow / deny) granted by the ACL. | "allow" / "deny" |
Example
POST aspire/_api/connections
{
"type": "filesystem",
"description": "FileSystem Test Connector",
"properties": {
"url": "C:\\Directory",
"ignoreSymLinks": true,
"stopOnScanError": true,
"indexContainers": true,
"scanExcludedItems": true,
"ignoreSymLinks": true,
"includes": ".*\\.txt",
"excludes": ".*\\.png",
"staticAcl": [{
"name": "test-user",
"domain": "test-domain",
"entity": "user",
"access": "allow"
}, {
"name": "test-group",
"domain": "",
"entity": "group",
"access": "deny"
}
]
}
}
Update Connection
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| id | Yes | - | No | Id of the connection to update | "89d6632a-a296-426c-adb0-d442adcab4b0", |
| type | Yes | - | No | The value must be "filesystem". | "filesystem" |
| description | No | - | No | Name of the connection object. | "MyFileSystemConnection" |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this connection object. | "f5587cee-9116-4011-b3a9-6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this connection will use. | ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"] |
| properties | Yes | - | No | Configuration object | |
| url | Yes | - | No | Path of the base directory to crawl. All the seeds will be prefixed with this value to form the full path. All seeds will be prefixed with this value to form the full path | "C:\\Directory" |
| ignoreSymLinks | No | false | No | If enabled symbolic links will not be processed and links in the root items will cause an error. | true / false |
| stopOnScanError | No | true | No | If enabled, the crawl will stop if there is an error on the scan phase. | true / false |
| indexContainers | No | false | No | Enable to index the directories. | true / false |
| scanRecursively | No | true | No | Enable to scan discovered items recursively. | true / false |
| include | No | [ ] | Yes | Patterns to match against document URL, if any of them match, the document will be included in the crawl. | [ ".*pdf$", ".*docx$" ] |
| exclude | No | [ ] | Yes | Patterns to match against document URL, if any of them match, the document will be excluded from the crawl. | [ ".*png$", ".*jpeg$" ] |
| scanExcludedItems | No | false | No | Enable to force the scan of excluded directories, so child items within the scope can be found. | true / false |
| staticAcl | No | [ ] | Yes | Static ACL configuration object | |
| name | Yes | - | No | Name of the static ACL. | "group1" |
| domain | No | "" | No | Domain of the static ACL. | "testDomain" |
| entity | No | "user" | No | Entity (user / group) represented by the static ACL. | "user" / "group" |
| access | No | "allow" | No | Access (allow / deny) granted by the ACL. | "allow" / "deny" |
Example
PUT aspire/_api/connections/89d6632a-a296-426c-adb0-d442adcab4b0
{
"id": "89d6632a-a296-426c-adb0-d442adcab4b0",
"type": "filesystem",
"description": "FileSystem Test Connector",
"properties": {
"url": "C:\\Directory",
"ignoreSymLinks": true,
"stopOnScanError": true,
"indexContainers": true,
"scanRecursively": true,
"scanExcludedItems": true,
"includes": ".*\\.txt",
"excludes": ".*\\.png",
"staticAcl": [{
"name": "test-user",
"domain": "test-domain",
"entity": "user",
"access": "allow"
}, {
"name": "test-group",
"domain": "",
"entity": "group",
"access": "deny"
}
]
}
}
Create Seed
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| seed | Yes | - | No | The subdirectory to crawl. This value will be appended to the url of the connection. | "directory" |
| type | Yes | - | No | The value must be "filesystem". | "filesystem" |
| description | Yes | - | No | Name of the seed object. | "MyFileSystemConnection" |
| connector | Yes | - | No | The id of the connector to be used with this seed. The connector type must match the seed type. | "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31" |
| connection | Yes | - | No | The id of the connection to be used with this seed. The connection type must match the seed type. | "602d3700-28dd-4a6a-8b51-e4a663fe9ee6" |
| workflows | No | [ ] | Yes | The ids of the workflows that will be executed for the documents crawled. | ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"] |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this connection object. | "f5587cee-9116-4011-b3a9-6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this seed will use. | ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"] |
| tags | No | [ ] | Yes | The tags of the seed. These can be used to filter the seed | ["tag1", "tag2"] |
Example
POST aspire/_api/seeds
{
"type": "filesystem",
"seed": "directory",
"connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31",
"description": "FileSystem_Test_Seed",
"throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b",
"routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"],
"connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",
"workflows": ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"],
"tags": ["tag1", "tag2"],
"properties": {
"seed": "directory"
}
}
Update Seed
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| id | Yes | - | No | Id of the seed to update. | "2f287669-d163-4e35-ad17-6bbfe9df3778" |
| seed | No | - | No | The subdirectory to crawl. This value will be appended to the url of the connection. | "directory" |
| description | No | - | No | Name of the seed object. | "MyFileSystemConnection" |
| connector | No | - | No | The id of the connector to be used with this seed. The connector type must match the seed type. | "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31" |
| connection | No | - | No | The id of the connection to be used with this seed. The connection type must match the seed type. | "602d3700-28dd-4a6a-8b51-e4a663fe9ee6" |
| workflows | No | [ ] | Yes | The ids of the workflows that will be executed for the documents crawled. | ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"] |
| workflows.add | No | [ ] | Yes | The ids of the workflows to add. | ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"] |
| workflows.remove | No | [ ] | Yes | The ids of the workflows to remove. | ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"] |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this connection object. | "f5587cee-9116-4011-b3a9-6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this seed will use. | ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"] |
| routingPolicies.add | No | [ ] | Yes | The ids of the routingPolicies to add. | ["b4d2579f-1a0a-4a8b-9fd4-d42780003b36"] |
| routingPolicies.remove | No | [ ] | Yes | The ids of the routingPolicies to remove. | ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7"] |
| tags | No | [ ] | Yes | The tags of the seed. These can be used to filter the seed | ["tag1", "tag3"] |
| tags.add | No | [ ] | Yes | The tags to add | ["tag4"] |
| tags.remove | No | [ ] | Yes | The tags to remove | ["tag2"] |
Example
PUT aspire/_api/seeds/2f287669-d163-4e35-ad17-6bbfe9df3778
{
"id": "2f287669-d163-4e35-ad17-6bbfe9df3778",
"type": "filesystem",
"seed": "directory",
"connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31",
"description": "FileSystem_Test_Seed",
"throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b",
"routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"],
"connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",
"workflows": ["b255e950-1dac-46dc-8f86-1238b2fbdf27", "f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"],
"tags": ["tag", "tag2"],
"properties": {
"seed": "directory"
}
}
Overview
Content Tools
Copyright © 2024 Accenture
