General
Why is does an incremental crawl last as long as a full crawl?
The RDB via Snapshots Connector performs incremental crawls based on snapshot files, which are meant to match the exact documents that have been indexed by the connector to the search engine. On an incremental crawl the connector fully crawls the RDBMS content the same way as a full crawl, but it only indexes the modified, new or deleted documents during that crawl.
Common Errors
Connection errors
If you can't get a connection to the database, you'll see an exception similar to this:
Couldn't get an RDB connection
com.searchtechnologies.aspire.services.AspireException: Unable to open connection to jdbc:postgresql://localhost/RDBMS_Test_DB (aspire_user) (component='/My_RDB_Source/RDBMSRDB', componentFactory='aspire-multiple-rdb') at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.createConnection(MultiRDBConnectionPool.java:535) at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.getConnection(MultiRDBConnectionPool.java:477) at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.access$000(MultiRDBConnectionPool.java:387) at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool.getConnection(MultiRDBConnectionPool.java:131) at com.searchtechnologies.aspire.components.rdbconnector.RDBScanner.doCrawl(RDBScanner.java:420) at com.searchtechnologies.aspire.scanner.AbstractPushScanner.performScan(AbstractPushScanner.java:30) at com.searchtechnologies.aspire.scanner.AbstractScanner.scanProcess(AbstractScanner.java:701) at com.searchtechnologies.aspire.scanner.AbstractScanner.process(AbstractScanner.java:323) at com.searchtechnologies.aspire.application.JobHandlerImpl.runNested(JobHandlerImpl.java:158) at com.searchtechnologies.aspire.application.JobHandlerImpl.run(JobHandlerImpl.java:80) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:410) at org.postgresql.Driver.connect(Driver.java:280) at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.createConnection(MultiRDBConnectionPool.java:525) ... 12 more Caused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at org.postgresql.core.PGStream.<init>(PGStream.java:60) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101) ... 21 more
SQL Errors
Errors in the SQL configured for the job will produce exceptions such as:
2014-02-10T19:54:07Z ERROR: Error executing select sql for full crawl (SELECT "id", "name", "company", "email", birthday FROM Test_table WHERE "id" <= 1000 AND MOD(id,10) = 9 ) org.postgresql.util.PSQLException: ERROR: relation "test_table" does not exist Position: 57 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302) at com.searchtechnologies.aspire.components.rdbconnector.RDBScanner.executeSelectSQL(RDBScanner.java:57) at com.searchtechnologies.aspire.components.rdbconnector.RDBScanner.doCrawl(RDBScanner.java:466) at com.searchtechnologies.aspire.scanner.AbstractPushScanner.performScan(AbstractPushScanner.java:30) at com.searchtechnologies.aspire.scanner.AbstractScanner.scanProcess(AbstractScanner.java:701) at com.searchtechnologies.aspire.scanner.AbstractScanner.process(AbstractScanner.java:323) at com.searchtechnologies.aspire.application.JobHandlerImpl.runNested(JobHandlerImpl.java:158) at com.searchtechnologies.aspire.application.JobHandlerImpl.run(JobHandlerImpl.java:80) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
Overview
Content Tools