Wednesday, May 13, 2009

Have a site that recently changed IP, but the crawler is still using an old IP

When the IP is refreshed depends on the Time To Live received from the DNS server.However in crawler versions 6.4.16 and below this behavior was not correct and the dns cache was not updated. The issue is fixed in 6.4.17+, however you can refresh the dns cache the following way:

Stop the crawler and on the ubermaster and the master nodes,

Remove (consider taking a backup) the file

$FASTSEARCH/data/crawler/config/dnscache.hashdb.

Now start the crawler.

The DNS cache should now be reset.