SingleUserBlog

Cleaning Up ASP.NET Sessions in Google

Brendan Kowitz — Wed, 06 Dec 2006 13:49:27 GMT

ASP.NET and Dirty Urls

There are two things that have been bothering me about pages that are getting indexed in Google from an ASP.NET application. The first is somehow there are ASP.NET Session Urls ending up in the Google index. This is bad because searchers that actually do click these links are likely to get a 500 error (internal server error) because they will be trying to access a page of an expired session.

How is Google finding all these 'bad' urls?

Well apparently there is no browser definition in ASP.NET 2.0 for the Googlebot's useragent string, so when the spider hits your ASP.NET page it's browser capabilities are not defined.

Edit: The default browser capabilities are defined to use cookies, the issue occurs because the base Mozilla definition is defined to NOT use cookies. If the browser is not able to accept cookeis .NET gets around this by inserting the session information into the Url and issues a 302 (content temporarily moved) in the response header.

This default behaviour is a good and a bad thing. It's good in the fact that if I'm browsing an asp.net site on a pda that doesn't support cookies I still can. However just about every search engine spider ever created has it's own UserAgent string making it a tough task to issue the standard non-crufted url. One solution to fixing the session urls being indexed in Google is to tell your asp.net application that Googlebot supports cookies and the problem is solved.

To read more about the solution please see my next post.

Dynamic Captcha build-up

Another dynamic aspect that is used on this site are Captcha images, and yes Google's image spider finds those too. Upon trying a Google image search on my domain, it's littered with Captcha images! I've also added an exclude to the robots.txt file for this.

Solution for Captcha images build-up and stop-gap solution for Session Urls

Here is my "robots.txt" file so far for my SingleUserBlog install. *Note the last two lines, "Disallow: /(A(*" should exclude any ASP.NET session urls, (this is not recommended unless you have fixed the Mozilla detection hole). The last line should exclude any captcha images from being indexed.

User-agent: *
Disallow: /LoginPage.aspx
Disallow: /Administration/
Disallow: /(A(*
Disallow: /Captcha.ashx*$

ASP.NET Mobile Content

Thu, 26 Oct 2006 20:31:57 GMT

A few weeks ago before everything started getting really busy both personally and work wise, I still somehow found the time to put together a mobile version of my blog content.

Device Detection
In the beginning I was just using the ASP.NET property Request.Browser.IsMobileDevice for device detection, but for some reason WindowsCE is not considered a mobile device by this property, so I included an additional 'if Request.Browser.Platform.Equals("WinCE")'. The mobile devices request will be redirected to www.kowitz.net/mobile/ which displays only one post on the screen at a time using 100% mobile controls and pages.

While I was creating my Mobile Page, I found in the docs that in ASP.NET 1.1 the adaptive rendering for mobile devices was only done in the ASP.NET mobile controls. Apparently this has all been combined into the regular controls in .NET 2.0. My first thought was 'wow, thats great, my pages should be compatible with mobile devices without having to do a thing'. However after viewing my normal domain with an O2 Neo WindowsCE device there's just waaay to much content for the small screen.

This leads me to three possible solutions to dealing with mobile devices reading my blog.

The first is simply; get an rss reader for your mobile device, you get none of the pretty style sheet nonsense since it probably won't display correctly anyway.
Second is redirect the device to a different url which is specific to mobile screens.
The third option is the one I'd like to explore a little more which is; assuming that the regular controls will use their super adaptive rendering powers on mobile screens, then simply change the masterpage to a 'mobile specific' version.

Issues:
I would like to test the normal controls using a WAP only phone / browser just to make sure everything works the way it should before I fully move away from my 'Mobile Only' page for a masterpage.

Observations:
Because my posts contain HTML formatting, this formatting may or may not be compatible on the target mobile device. The MobileLiteral control takes care of this by filtering out incompatible HTML tags, now to be specific, its easier to list the tags that it doesn't filter, even <hr> is gone. This is not so much as a problem as the <img> tag being filtered out of the posts. My original idea for this problem was to simply regex the post to extract the image links, then redirect all my images through my own 'ImageResizingHandler' which would then shrink the image on-the-fly to fit the width of the mobile screen if it was too big...but...because these tags a wiped out, this won't work...yet.

Future:
Well, my next steps are to test the rendering under a more primitive device then my O2 Neo. Somehow improve the mobile devices detection (even Opera Mobile slips through). Then assuming all that works I've only got the images problem to figure out.

Url Rewriting using IHttpModule vs ISAPI Rewrite

Thu, 14 Sep 2006 21:25:01 GMT

Why not use IHttpHandler for rewriting:

You can find the previous post I've done on this subject here, it has a c# source code example and will also explain some of the downfalls of using the IhttpHandler for rewriting. In summary my basic feeling on this is IhttpHandler is great if you know the Url that is going to be requested eg. /Rss.ashx every time an rss reader requests this Url, the reader will never know that Rss.ashx doesn't necessarily have to exist as the response is picked up and written back by the handler.

How is IHttpModule more useful url rewriting?

When it comes down to rewriting urls where the requested Url is not known or is dynamically generated IHttpModule makes it easier to manipulate the request as it passes down the asp.net stack. So creating an ASP.NET url rewriter it's just a simple matter of attaching a httpmodule that can identify the dynamic url and rewrite it before it requests a page that doesn't exist (eg a 404).

The way I chose to do this in pure asp.net was to use regex matching. Every request that comes in is tried against a regex pattern, if the url makes a match it is rewritten to the handling page (or possibly a IHttpHandler) to return the response.

But nothing beats ISAPI rewrite, right?

I'd say yes and no to this, ISAPI rewrite uses 'almost' the same principle as my module, except it rewrites the Url as it enters IIS and before the request even gets to the ASP.NET request handler. This means the client will think they are at a particular url eg /rss/ but the request could be transformed by ISAPI rewrite to /feeds/rss.ashx. One thing ISAPI rewrite can do that the .NET module can not is it is able to hide the page's language extension (eg. .aspx), but doing this is not a huge benefit unless you want to change your site's programming language which would most likely break all your referring links if the extension is present.

The ASP.NET way:

So now for the things that the ASP.NET way does better. Because the rewriting is done at the ASP.NET level, .NET knows about it, this means ASP.NET can auto-magically resolve all those virtual ~/paths correctly. There are also some other SEO advantages to doing this such as utilizing the keywords in your url as opposed to having a meaningless url full of query parameters which will get you nothing, well it might get some SEO points, but not as much as you could.

The ideal solution for ASP.NET:

I think the ideal way to do rewriting for ASP.NET is use both ISAPI in combination with a .NET module for the reasons listed above. ISAPI rewrite gives that edge of power over IIS that a class inside ASP.NET can't, so urls can look like directories ('/') rather then .aspx pages, plus the ISAPI rewrite configuration file is very convenient and configurable. The .NET module comes in handy to rewrite the url from a dynamic location to the real location of the page and is able to resolve all the ~/ virtual directories in the process.

An example of this that I've done using a combination on another site is as follows:

Request comes to: http://www.site.com/second-level/content-title/
ISAPI rewrites to: http://www.site.com/second-level/content-title/ResolvePage.aspx
My .NET module watches for requests to ResolvePage.aspx and does an ASP.NET Context.RewritePath() to the real content page location of http://www.site.com/Content.aspx?path=content-title
This then causes Content.aspx to render 'as if' it was at the requested location eg.
http://www.site.com/second-level/content-title/ResolvePage.aspx so all the virtual paths are correct in the requesting browser.

And there it is a friendly, programming language free, keyword rich url living in harmony with ISAPI and ASP.NET. Search engines are happy, users have nice clean hackable urls to look at and remember and are happy. Everybody is just happy.

Edited using Windows Live Writer, thanks Matt!

Powered By SUB

Wed, 06 Sep 2006 02:55:29 GMT

I have been aching to try out Live Writer to do a post...but no success yet :(

Many people have probably seen the "Powered By ASP.NET 2.0" button.

So I thought it was only fair to convert it into a Powered by SUB button.

Or here's another more subtle one.

Online Presence Webpart

Fri, 26 May 2006 21:30:10 GMT

You may have noticed that over the past few weeks I have been trying to build more interactive elements into this blog, some of these include.

Most viewed posts webpart.
Rate a post.
Highest Rated posts webpart.
Concurrent user indicator.
MSN client, online / offline status.
Sessions per hour graph.

My intention was that this blog would start to expose more of the interesting content and create a slight online presence awareness by subtly reacting to how it is interacted with.

Most of the features I just listed are part of a Sqlite stats database that I’ve been logging certain sessions/hits/ratings to. I hope to make this available in the near future with a slightly improved version of the Sqlite datalayer.

But for now here is how to install the code I am using to control the MSN client and a preview of the Concurrent user indicator.

How to install this webpart.

Step 1: Create yourself a new ‘bot’ msn account that your website will use, than add yourself to the bot’s contact list.

Step 2: Add these 2 events to your Global.asax file

protected void Session_OnStart()
{
kowitz.SingleUserBlog.Services.OnlineUsers RecordUser(Session.SessionID);
kowitz.WebsiteMSN.MSNInstance.SignOn();
}

protectedvoid Session_OnEnd()
{
kowitz.SingleUserBlog.Services. OnlineUsers.DeleteUser(Session.SessionID);
if (kowitz.SingleUserBlog.Services.SqliteBlogStatRecorder.ConcurrentUsers() <= 0)
{
kowitz.WebsiteMSN.MSNInstance SignOff();
}
}

Step 3: Copy the webpart control into the Webparts folder. Copy the images into your images directory and copy the two dlls (kowitz.WebsiteMSN.dll and XihSolutions.DotMSN.dll) into your Bin directory.

Step 4: Add the webpart to your Masterpage’s catalogue zone.

<uc1:MessengerOnlinePresenceID="MessengerOnlinePresence1"Title="Messenger Online Presence"runat="server"/>

Step 5: Add this to your BlogConfiguration.xml file:

<MyMSNAccount> MyMSNAccount@hotmail.com </MyMSNAccount>
<BotMSNAccount> msnbot@mywebsite.net</BotMSNAccount>
<BotMSNPassword>mybotspassword</BotMSNPassword>

Step 6, Allowing your bot to message you when someone leaves feedback: Add this line to the void CommentForm1_CommentAdded() function (just after the call to "Notifications.NotifyAdministrator(subject, body);") in your Posts.aspx page.
kowitz.WebsiteMSN.MSNInstance.SendMeAMessage(body);

Finally

Your blog should now be logging onto MSN when there is at least 1 active session on your site.

As I mentioned before, a lot more of the things I’ve added are part of the Sqlite stats db / blog datalayer I’ve been making and hopefully I’ll also be releasing this soon.

Download

MessengerOnlinePresence webpart

Your Blog has signed-in

Sat, 20 May 2006 23:20:48 GMT

Today I was browsing the website ProgrammableWeb, which lists the Web APIs by a lot of well known sites. One of the things that caught my eye was MSN, and then I noticed they had a listing for Messenger. After clicking details I soon found it was only a description of how Javascript can invoke a chat window from the installed messenger client.

I wanted to take this idea further and maybe integrate something Messengery into SUB. After a single search I come across DotMSN, a C# library for Messenger. So& as an experiment I've integrated the library into SUB. When someone begins to browse my site (like you are now), the first person there invokes the blog to sign-in. After the blog has signed in, it then checks the accounts in its contact list to check if I'm online, so without doing anything, you know if I'm on the net, and I know somebody is on my blog.

So how can we communicate? Well for now I've added some functionality into the comments event, when you leave a comment, SUB will send it via Messenger to me as well as to my set up email account as normal.

I guess there are limitless things you could do with a blog that signs into msn but I think I'll leave it at that for now and see how it goes. What do you think?

New webparts, SUB Monthly Archive List

Mon, 08 May 2006 11:41:05 GMT

I've had a couple of experimental things going on here in this blog. Firstly is the Archive page, it now renders the archive list in a similar fashion to a plug-in Al has created for wordpress. As for the whole query problem with the wordpress version, the SUB version of this does exactly 1 query for each month listed or 0 queries if that month has already been cached.

The other experimental webpart I've been running is the "Popular Posts" component (displayed on the left). Everytime a post is hit it increments the daily count, then it selects the top x number of posts over the last 21 days. This is also similar to a wordpress plug-in Al has running on his site. This webpart I'll probably be making more modifications to until I'm happy with the way it's behaving, so if your interested in it, stay tuned.

Files
Archive webpart: DetailedMonthlyPostListing.zip

New SUB Theme

Thu, 04 May 2006 11:08:29 GMT

Just for something random I decided to re-arrange the objects in the banner and adjust the link font sizes.

Webparts without SqlExpress or SqlServer

Mon, 01 May 2006 11:34:13 GMT

One of the things that has been annoying me for some time about Webparts is that silly dependency on SqlExpress/SqlServer. My webhosting at Jumba comes with MySQL or something and not SqlServer. So up until this point to get around the dependency on SqlServer I simply disabled all the calls from SqlBlogPersonalizationProvider and that did the trick, but I had to manually do all the layout stuff and couldn’t use the webparts properly. A while ago when Microsoft released the source code for the old AccessPersonalizationProvider from Beta 1 I thought I’d come back and address this issue.

Here’s what happened.
I plugged the Access providers into SUB, it all compiled started to run, then Bang! It all broke. Apparently I need an existing Access database with the correct schema and tables. I’m sure this wouldn’t have been very difficult to put together but I don’t have that much time anymore.

Fortunately I found this blog entry where Dion provides details on how to store the serialized byte array that the provider outputs as a Base64 string in the aspx page. I didn’t exactly want the page modifying itself so I simply modified it to store all the goodies in an Xml document in the App_Data folder.

Yey, so now I can use the Shared-Scope webpart features without SqlServer. Another advantage of this is my Xml document is around 5kb where I know the starting size for that SqlExpress db is 10mb. In future I would like to get a portable database solution working such as the Access one or even a Sqlite version.

View XmlFilePersonalizationProvider Source

Update: Sqlite Dataprovider for SUBv2

Fri, 10 Mar 2006 01:45:01 GMT

Here is the new Sqlite Dataprovider for SingleUserBlog v2.0.1. It is now supports string IDs so any existing SingleUserBlog can migrate easily.

Find out more information at its new home here.

Please feel free to leave any comments, feedback or problems on this post.