Resolving Sitefinity 4 Search Not Including Content Pages

I recently encountered an issue where, after creating a Sitefinity SearchIndex for NewsItems, Static HTML and all Content, only the NewsItems were being returned - no Content Pages or Static HTML pages appeared in the results. The response I got was to upgrade to Sitefinity 4.1, SP 1, which wasn't the complete answer, but at least the newer admin UI helped to troubleshoot the problem:

(SearchIndex creation screen)


After going through the upgrade process, helpful error messages began to appear when I re-indexed my SearchIndex, indicating errors were being thrown when certain pages were being crawled.

So the problem really was that one particular call, in this case:
SiteMapBase.GetCurrentNode();
was throwing a null reference exception, only during the Sitefinity crawling process, and quietly killing the addition of the Content Pages to the SearchIndex. After resolving the issue, the full site indexed correctly, and searching returned results as expected. There's no feedback when the SearchIndex is successfully reindexed, however, but you'll see the page subtly refresh. (Maybe a good feature for 4.2)

(SearchIndex admin screen)


After going through this process, I found that you could also troubleshoot indexing issues by launching your app in debug mode, going to the admin tool and re-indexing your problem SearchIndex... The Visual Studio debugger will then point out any problem code.