You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Current »

FAQs

General 

Unable to render {include} The included page could not be found.

Authentication Issues

(Cookie Based) I have added the "username" and "password" fields but I can't still authenticate.

Sometimes just the username and password fields are not enough to authenticate to a site. Some sites requires some custom fields or even the "submit" button in order to successfully authenticate you. So you may have to add them as custom fields in the Aspider Configuration.

You can also open the browser inspect mode in order to break down the authentication request and make sure you are not missing any field.

(Cookie Based) How to include those dynamic "hidden" elements into the Aspider authentication request?

Don't worry about those hidden fields inside the form, Aspider will automatically include them in the request, you don't have to do anything.

(Cookie Based) My initial login is successful but shortly after, Aspider can't connect to already discovered URLs.

Watch out for "logout" pages which usually send requests to the browsers to clear their cookies. If Aspider is requested to clear its cookies for logging out, it will do that and will not try to re-login.

Suggestion: Add an exclusion pattern for the "log out" pages.

  • If "Scan Excluded Items" is selected, make sure the "Do not follow patterns" also contains a pattern that matches the "log out" page.

(Cookie Based) My site requires two different HTTP requests in order to authenticate, can Aspider handle that?

Unfortunately not at the moment, Aspider Cookie Based Authentication is built to send only one request, but we are already considering improvements for it.

(Cookie Based) Can Aspider handle javascript based authentications?

If your login form relies on javascript for sending the Authentication request your form element probably won't have an "action" attribute, which Aspider use for sending the POST request to. So you wouldn't be able to authenticate.

(Cookie Based) Which versions of SSL are supported by the Aspider web crawler?

As of Java 8, Aspider supports and has been tested to work on the following protocols:

  • TLSv1
  • TLSv1.1
  • TLSv1.2


SSLv2 and SSLv3 are not supported by Aspider.


  • No labels