Versions Compared

Old Version 1

changes.mady.by.user Ricardo Fonseca

Saved on

compared with

New Version Current

changes.mady.by.user Pablo Bonilla

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Panel

Easy Heading Free
navigationTitleOn this Page
wrapNavigationTexttrue
navigationExpandOptionexpand-all-by-default

Introduction

The SharePoint Online connector will crawl content from any SharePoint Online site collection URL. The connector will retrieve Sites, Lists, Folders, List Items and Attachments, as well as other pages (in .aspx format). This connector supports SharePoint running in the Microsoft 365 offering.

This is not a Microsoft 365 connector, the individual repository offerings within Microsoft 365, such as OneDrive, Calendar, Tasks, Yammer will have their own connectors.

Environment and Access Requirements

Repository Support

The File System supports crawling the following the repositories

RepositoryVersionConnector VersionSharePointMicrosoft 3655.0

Account Privileges

The connector offers two authentication options to access the SharePoint REST API: user account or Azure AD application.

User Account

To configure a user crawl account use the following see ServiceNow - Crawl Account Access.

Info

To use a user crawl account on multiple site collections, you'll have to follow the steps on each site collection the access is needed.

Azure AD Application

To configure an Azure AD application for crawling, see ServiceNow - Azure AD Access.

Info

Using an Azure AD Application will grant access to all site collections under the tenant.

Environment Requirements

The connector uses SharePoint's REST API, so the Aspire Worker nodes must have internet access to connect to the Microsoft 365 environment. Optionally, you can configure a proxy on the connector to enable internet access.

Framework and Connector Features

Framework Features

NameSupportedContent CrawlingyesIdentity CrawlingyesSnapshot-based IncrementalsyesNon-snapshot-based IncrementalsyesDocument Hierarchyyes

Connector Features

The SharePoint Online connector has the following features:

  • Item filtering using include and exclude regex patterns.  This is based on the item's display url.
  • Access Control Lists (acls) fetching, for document level security.
  • Support for BCS external lists.
  • Performs non-snapshot-based incremental crawling (so that only new/updated documents are indexed) using SharePoint's change log tokens.

Content Crawled

The SharePoint Online connector is able to crawl the following objects:

NameType Relevant MetadataContent Fetch & ExtractionDescriptionSitescontainer
  • Last Modified Date
N/AAny site or subsite underneath a seed. Not the same as the .aspx page for a SharePoint SiteListscontainer
  • Last Modified Date
  • Data size
N/AAny type of SharePoint list including (but not limited to): Document Libraries, External Lists, Calendars, Task Lists, etc.FolderscontainerN/AList Item Folders found on lists like Document Libraries or Link Lists.

ListItems

documentYesListItems can take a number of different formats. For example, documents (pdf, doc, ppt, etc), calendar events or announcements. For more info on how ListItems content types work go to the MSDN article.AttachmentsdocumentYesA document attached to a SharePoint List Item.

Limitations

Due to API limitations, SharePoint Online connector has the following limitations:

  • The connector uses the REST API to access SharePoint database(s) directly; it doesn't use web crawling
    • Crawling is only supported using a Site or a List as a root url.

    Features

    ServiceNow connector features:

    • Retrieval of supported elements using the RESTful API.
    • Basic authentication.

    Retrieved Content

    Currently, the ServiceNow connector retrieves the following types of documents:

    • Knowledge Articles and Attachments
    • Catalog Items and Attachments


    Table Usage

    Knowledge Content

    To retrieve Knowledge content, the connector accesses the tables:

    • kb_knowledge: Table with the Knowledge Articles.
    • kb_category: Table with the details of the article’s category.
    • kb_knowledge_base: Table for the Knowledge Base information.
    • sys_attachment: Table to extract attachments.
    • sys_attachment_doc: Table to extract attachments.
    • kb_uc_cannot_read_mtom: Table to relate Cannot Read user criteria and Knowledge Bases.
    • kb_uc_can_read_mtom: Table to relate Can Read user criteria and Knowledge Bases.
    • kb_uc_can_contribute_mtom: Table to relate Can Contribute user criteria and Knowledge Bases.
    • sys_user_role: Table to extract the Public role sys id.
    • user_criteria: Table of user criteria.
    • sys_user: Table with the detail of users (Used for author details).

    The roles needed to access these tables are: Knowledge, Knowledge_Admin and User_Admin or Itil.

    Catalog Content

    To retrieve Catalog Items’ content, the connector accesses the tables:

    • sc_cat_item: Table with the Catalog Items.
    • sc_category: Table with the details of the Catalog item’s category.
    • sc_catalog: Table with the Catalog of the Catalog Items.
    • sc_cat_item_user_criteria_mtom: Table to relate Available For user criteria and Catalog Items
    • sc_cat_item_user_criteria_no_mtom: Table to relate No Available For user criteria and Catalog Items
    • sys_attachment: Table to extract attachments.
    • sys_attachment_doc: Table to extract attachments.
    • user_criteria: Table of user criteria.

    The roles needed to access these tables are: Catalog and User_Admin or Itil.

    Identity Extraction

    These are the tables required to extract the list of users that can access Knowledge Bases.

    • kb_knowledge_base
    • u_acl_table_users (custom table and depends on a custom web service)

    The roles needed to access these tables are: Knowledge_Admin.



    Limitations

    Due to API limitations, the ServiceNow connector has the following limitations:


    For Knowledge Articles 

    The connector will crawl active and published articles. 

    Depending on the chosen security, there could be additional filtering of articles:

    • Public articles: from the base of active and published articles, the connector allows only public items, which means articles with Public Knowledge Bases. Furthermore, the connector verifies if Can Read or Can Contribute fields are set, but only public user criteria are allowed. Finally, if Cannot Read is set, the item will be ignored.
    • User Criteria: This option does not filter. This type of security will add the Can Read, Cannot Read and Can Contribute user criteria values as ACLs. For these ACLs, a web service is required to extract the list of allowed users to read each user criteria.
    • Knowledge Base: This option does not filter. This type of security will add the sys_id of the Knowledge Base's article value as an ACL. For these ACLs, a web service is required to extract the list of allowed users to read each Knowledge Base.


    For Catalog items:

    The connector will crawl by default active items with role field empty (there is an option to remove this filter).

    Depending on the chosen security, there could be additional filtering of items:

    • Public Catalog items: using the base of active items with empty roles (if checked), the connector will check if Available For field is set. The value should be the public User Criteria. In addition, if No Available For fields is set, the item will be ignored from the crawl.
    • User Criteria: This option does not filter. This type of security will add the Available For or No Available For user criteria values as ACLs. In case there is no UC, the ACL will be Public. For these ACLs, a web service is required to extract the list of allowed users to read each user criteria.


    Due to technical constraints on the security model of ServiceNow, the security support for other content should be reviewed on a case by case basis.

    Future Development Plans

    The following features are not currently implemented, but are on the development plan:

    • Public content from other endpoints using a custom query.

    Anything we should add? Please let us know.


    Disable checkingPremium suggestionsDisable checkingPremium suggestionsDisable checkingPremium suggestionsDisable checkingPremium suggestionsDisable checkingPremium suggestionsDisable checkingPremium suggestions