Is the Domino server setting "Make this site accessible to web search site crawlers" still relevant?

Hello,

A quick admin entry for a change. 

Imagine you are running IIS with the WAS Plug-in and serving content in Domino databases.  You have some links that contain a query string e.g. "?openview&category=monkeylegs".  You want web crawlers to follow these links so you enable the setting "Make this site accessible to web search site crawlers".  So far so good.

That setting applies to the Domino Web Engine and substitutes a url's usual query string delimiter "?" with an exclamation point (!) in urls that it generates.  So "http://blah.talk.com/topics.nsf?category=politics"  becomes "http://blah.talk.com/topics.nsf!category=politics".

IIS restricts the number of characters in a part of a path ("/.../" or "/.....?")  to something like 260 -270 characters.  When that limit is exceeded IIS returns a 400 invalid url error.

IIS knows that the exclamation point denotes the start of the query string.  Apparently IIS doesn't know that "!" could be being used instead of "?" to denote the query string.  And what IIS appears to do is treat the "!" as any other character in a path. 

So IIS + Domino + "!" + looong query string = Error 400 Invalid Url

Obviously you have choices to avoid this problem:
  • Use Apache instead of IIS (though I haven't tested this behaviour in Apache)
  • Shorten the query string
  • Put your parameters in the form and use POST instead of GET
  • Avoid using "!" and use "?" instead
The sticking point is can Google and other crawlers follow "?open..." links nowadays?  I think Google can which would mean that the "!" setting is largely redundant and if other crawlers can it's completely redundant. 

Fortunately Domino allows "?" and "!" to be used interchangeably regardless of the server setting (it only seems to affect urls that are generated by Domino) so bookmarked urls will work either way.

I hope you've been able to follow my logic, do you have an opinion/insight?

BTW I've got a terrible headache right now which is not directly linked to this post but might affect it's clarity

TTFN


    

 
Trackbacks
  • Trackbacks are closed for this entry.
Comments
  • No comments exist for this entry.
Leave a comment

Comments are closed.