Tuesday, March 25, 2008

Automating .NET web application builds with CruiseControl.NET

I've been trying to get around a completely automated build process for sometime now. Finally, I got to work out through the details for a recent web application project in .NET. Now that I have most of the things out of the way I suppose its fair to say that its fairly straight forward but an occasional head cracking must happen. I hope that this post might help someone else starting out with CruiseControl.NET.

To understand what's happening I suppose its necessary to explain the environment in which this takes place:

  • ASP.NET application, developed with Visual Studio 2005, using VB.NET
  • NUnit for unit testing
  • NAnt for automated build (version 0.86-beta-1)
  • Selenium for automated web tests (version 0.8.2)
  • CruiseControl.NET for continuous integration (version 1.3.0.2918)
  • Subversion as a source code repository
Hm. Not too bad, I suppose. If I can figure out how to throw in an issue-tracking system into the pot and start distributing installer packages with automated release notes that would be just about as far as you can push that thing. But, lets see on the fundamentals first.

I have 3 projects in the Visual Studio solution: web app, class library (with the core set of classes) and an NUnit test project. Class library uses Microsoft Enterprise Library 3.1 (May 2007) - specifically its Data and Logging services.

Web application references the core class library (just like the NUnit project does).

You can see the top-level folder structure for the project. It was inspired by Jean Paul S. Boodhoo's series of articles on automating a build with NAnt.

Web application resides in src/app/WebApp folder. Core class library is in src/app/Core and NUnit is in src/test/NUnit.

tools/ folder contains NAnt and NUnit dlls. lib/ contains MS Enterprise Library DLLs referenced by the project.

build.bat batch file basically calls NAnt passing in the default.build as the build file where all (or most of) the juice happens.

NAnt

This is really the central piece of the puzzle that took the most time, so lets try and explain that bit.

The technique of using template files to generate actual files needed for certain things during the build process is borrowed from automating your build with NAnt (part 6) by Jean Paul S. Boodhoo. That is used to create the script file to generate a database as well as populate it with default seed data. In addition, app.config file is generated in the same way before deployment. Example target that does the merging:
<target name="convert.template.unicode">
<copy file="${target}.template" tofile="${target}" overwrite="true" inputencoding="Unicode">
<filterchain>
<replacetokens>
<token key="INITIAL_CATALOG" value="${initial.catalog}" />
<token key="ASPNETACCOUNT" value="${aspnet.account}" />
<token key="OSQL_CONNECTION_STRING" value="${osql.ConnectionString}" />
<token key="CONFIG_CONNECTION_STRING" value="${config.ConnectionString}" />
<token key="DBUSER" value="${db.user}" />
<token key="DBPASSWORD" value="${db.password}" />
<token key="DBPATH" value="${database.path}"/>
</replacetokens>
</filterchain>
</copy>
</target>
Example above is used to replace various tokens with the values required in the environment where build is taking place. The only difference from the original appearance in Boodhoo's article is in the use of inputencoding="Unicode" in the copy task which ensures that a file that is encoded in unicode gets copied into a file with the same encoding (by default OEM code page is used which means your unicode files become perhap latin-1252 encoded and makes the loose special characters).

Local.properties.xml is also used to merge user-specific properties into the build file - things like passwords, paths where SQL Server data/log files are located, etc.

I have .NET 1.1 and 2.0 both installed on my development machines. IIS runs with 1.1 being the default ASP.NET version on the default website. This means that when you deploy a new virtual directory on the site it picks up 1.1 as ASP.NET version. In this case, that would be a problem since the virtual directory must be 2.0. Luckily that is easily changable with the use of aspnet_regiis.exe. You will notice that the deploy target in the build file essentially executes it to get:
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -s W3SVC/1/ROOT/WebApp
The above will essentially change the the ASP.NET version for the virtual folder that was created during deployment through the build script. In my case I have IIS 5.0 with a single instance and my web application's virtual folder name is WebApp.

I've found it useful to suppress warning messages from the compiler. You can do that within the vbc NAnt task with the following:

<vbc target="library" rootnamespace="NantCCProject.HR" output="build\NantCCProject.HR.dll" debug="${debug}" verbose="true">
<nowarn>
<!-- ignore all code paths don't return a value warning. -->
<warning number="42104"/>
<warning number="42105"/>
<warning number="40000"/>
</nowarn>
...
</vbc>


Selenium

To facilitate selenium testing, I've placed the core selenium files inside the web application as well as a folder with all the tests. selenium is in src/app/WebApp/resources/selenium/core and its tests are at src/app/WebApp/resources/selenium/tests folder.

Integrating Selenium into CruiseControl.NET means that the tests should get executed through NAnt. There is a good starting point on how to perform the integration in the OpenQA wiki. Essentially, we'll need a custom NAnt task that executes the selenium tests.

CruiseControl.NET configuration

Following is the CruiseControl.NET server configuration file for the project:

<cruisecontrol>
<!--
cruise control config file -->
<!--
<project name="
NantCCProject" />
-->
<project name="
NantCCProject">
<workingDirectory>E:\programming\DOT.NET\cruise\checkouts\
NantCCProject</workingDirectory>
<artifactDirectory>E:\programming\DOT.NET\cruise\builds\
NantCCProject\artifacts</artifactDirectory>
<webURL>http://localhost:8283/ccnet</webURL>
<modificationDelaySeconds>10</modificationDelaySeconds>
<labeller type="defaultlabeller">
<prefix>1.0.</prefix>
<incrementOnFailure>true</incrementOnFailure>
</labeller>
<triggers>
<intervalTrigger seconds="60" />
</triggers>
<state type="state" directory="E:\programming\DOT.NET\cruise\builds\
NantCCProject\state" />
<sourcecontrol type="svn">
<tagOnSuccess>true</tagOnSuccess>
<trunkUrl>http://localhost/svn-
NantCCProject/NantCCProject/trunk</trunkUrl>
<tagBaseUrl>http://localhost/svn-
NantCCProject/NantCCProject/releases</tagBaseUrl>
<workingDirectory>E:\programming\DOT.NET\cruise\checkouts\
NantCCProject</workingDirectory>
</sourcecontrol>
<tasks>
<nant>
<executable>nant.exe</executable>
<baseDirectory>E:\programming\DOT.NET\cruise\checkouts\
NantCCProject</baseDirectory>
<buildFile>default.build</buildFile>
<targetList>
<target>db</target>
<target>seed</target>
<target>compile</target>
<target>test</target>
<target>asp.compile</target>
<target>deploy</target>
<target>selenium</target>
</targetList>
<buildTimeoutSeconds>300</buildTimeoutSeconds>
</nant>
</tasks>
<publishers>
<buildpublisher />
<merge>
<files>
<file>E:\programming\DOT.NET\cruise\checkouts\NantCCProject\build\*Test-Result.xml</file>
</files>
</merge>
<xmllogger />
</publishers>
</project>
</cruisecontrol>


A few things to explain regarding cruise control config file:
  • I'm using SVN setup on localhost. This is served through an Apache 2.0 (config for that shortly), hence SVN access is through http://localhost/svn-... URL.
  • There are a few NAnt tasks that get executed - in the order specified: db (create database), seed (pump default data), compile, test, ASP.NET compile, deploy and finally selenium.
  • Every build is labeled as [prefix].[build] where prefix is "1.0." (I did not go into one of the more elaborate labeling schemes, but CC is quite happy to entertain more complicated requests on this)
  • This is using a simple publisher which essentially makes a folder upon every successful build named [prefix].[build] and copies into it whatever was checked-out from SVN
  • Upon successful builds, source code is tagged by way of actually making a copy of the snapshot of files inside the SVN's [ROOT]/releases/[prefix].[build] folder. SVN core files are obtained from the SVN's [ROOT]/trunk folder.
Misc notes
  • Make sure you use rootnamespace attribute in your vbc NAnt task. Set it as per what Visual Studio sets it, otherwise you'll get compilation problems.
  • Your NAnt installation may have a default .NET framework set to something other then the one you require. That can be controlled with the:
<property name="nant.settings.currentframework" value="net-2.0" />
  • hmmm. perhaps enough for now.
Resources

Monday, March 24, 2008

Windows explorer chrashing after changing regional setting

Recently I've changed my regional setting from English/United States (default) to Bosnian. On startup of the machine it is not possible to start windows explorer using the windows key + E shortcut - it shows the "Windows explorer has encountered a ..." dialog box. Interestingly, double clicking My Computer shortcut does work and shows the explorer window. However, attempting to do a right-click on anything inside the explorer crashes the thing again. I'm contemplating a switch back to English/United States locale...

Friday, March 21, 2008

MS SQL Server and codepage

I have a simple problem (I think): my office development machine uses Windows' regional setting with codepage 1250. My laptop is using default windows United States/English code page (latin, 1252 codepage).

SQL Server 2000 on the office machine was installed when that machine was also with the US/English regional setting. Only recently have I changed that.

With my recent project I'm getting some frustration with inserting and viewing data through enterprise manager. I've managed to get my office machine to properly insert data through osql (command line invocation from NAnt). In addition, enterprise manager properly shows all the special diacritic characters. The database is set as having collation of Croatian_CI_AS (and all the varchar columns are as well).

Now I am not 100% certain here but I believe that it should be possible to have my laptop on the US/English regional setting and whatever the codepage it uses properly insert and view the inserted data in my laptop SQL Server instance. So far, no luck on that front.

I can successfully insert the data through a script (file is encoded as UCS-2 Little Endian with help from Notepad++). However, looking at the data I can't see the special characters - they are all converted to their closest equivalent (Č becomes C). While researching a bit, I came across this really old post on MS SQL 6.0 and 4.x - SQL Server Code Pages and AutoAnsiToOEM behavior (SQL Server 6.x and earlier was, I learnt, non-unicode complaint at a time so there were troubles there apparently). A more interesting read was another article on translation of characters between code pages of server and client. The second one got me really worried since it essentially says that you're not supposed to store data from one codepage on the client in a server with a different codepage.

My trouble is that the client happily uses 1250 codepage throughout the organization and I can't use NVARCHAR to represent text but must use VARCHAR (there are existing tables to work with). I hate the idea of having to change my regional setting of the laptop just because of this one project - it appears that there ought to be a better way to do this.

Well. Right now, I'm stuck a bit. Hopefully will dig out more.

Update 23 Mar 2008:
to my great surprise I just realized that Enterprise Manager does not show diacritics in my varchar columns on the machine whose codepage is United States/English BUT when I display the data on my web page I get the proper characters shown! That was a real surprise. I cannot enter diacritics via Enterprise Manager either - they are turned into their non-diacritic siblings.

In addition (more surprises for me), SQL Query Analyzer shows the diacritics properly! This sort of suggests that I just might be able to use those differing codepages on my development machines just fine, provided that I don't try and enter data directly in Enterprise Manager. I do have to try to insert some characters via Query Analyzer to see what happens with that.

Thursday, March 20, 2008

osql and input file encodings

I have been trying to run MS SQL Server's osql utility (SQL Server 2000) to insert some default data. My SQL Server's target database collation is Croatian_CI_AS. After some time I've realized a few things about the ways in which you can save the input file.

Using Notepad++ as my editor I thought that formatting a file as UTF-8 would cause everything to be wonderful and magically work. Not so. In fact, osql will not even read the UTF-8 encoded file (this was a real surprise)! That was a weird thing but I kept on getting this error message about first line and some weird character displayed there. Turns out that osql accepts unicode or ANSI encoded files (and there's no way to control the input file encoding from the command line so we're stuck with that ... which is not that bad after all).

So, no.1: don't format your input file for osql as UTF-8. That's a different animal from a unicode encoded file.

Notepad++ offers several formatting options. Turns out that UCS-2 Little Endian is the wizard. For those interested, read on UCS-2 on wikipedia. I just realized it was actually a 16-bit Unicode encoding. Just looking at it in the list of encodings in Notepad++ didn't ring bells.

I'm not sure why UCS-2 Big Endian encoding does not play nice (I get gibberish for most of my special characters).

Formatting a file as ANSI in Notepad++ and then running through osql was a bit of a surprise that it did not work (special characters like diacritics became gibberish, although osql did execute everything). I even changed my Windows code page to Croatian in hope that it will all somehow sort itself out but it didn't. From what I can tell, osql will read the ANSI encoded file as probably English Latin encoding and loose the special characters in the process.

So, bottom line, UCS-2 Little Endian is our friend. Use that for encoding files that are to be executed with osql. Oh, and probably get a good editor like Notepad++.

Tuesday, March 18, 2008

CSS selectors precedence (and ways to waste time)

If you ever wondered how CSS selectors work and how to figure out precedence of CSS rules, read the CSS specificity spec and CSS selectors spec.

The gist of if? Think of it as four values: a,b,c,d.
  • a= count 1 if the declaration is from is a 'style' attribute rather than a rule with a selector, 0 otherwise (= a) (In HTML, values of an element's "style" attribute are style sheet rules. These rules have no selectors, so a=1, b=0, c=0, and d=0.)
  • b= count the number of ID attributes in the selector (= b)
  • c= count the number of other attributes and pseudo-classes in the selector (= c)
  • d= count the number of element names and pseudo-elements in the selector (= d)
Oh, and I've learnt this the hard way: don't start your IDs or CSS class names with a number. FF/IE simply ignore them. I suppose there is a spec somewhere for this as well ... sigh...

Monday, March 17, 2008

MySQL copying data folder

I was under the impression that you can copy data/ folder from one MySQL server to the other and you'd get the all the of the databases properly transferred over. Not so. It sort of works - you will get all your databases transferred, but chances are that your users will soon enough start noticing how certain things no longer work in applications.

My only advice is that you need to do a "proper" transfer of databases. Either mysqldump each one or get a tool that can do this quicker (Navicat has a wonderful mechanism of transferring data across servers).

Bottom line: forget copying data subfolders for MySQL server migrations. Do it slowly and properly instead.

Sunday, March 16, 2008

Extending HTML container element's height around inner floats

A common problem with using floats in a site is that their container does not expand around them. Imagine that #left-side has float:left and #right-side has float:right, both having width: 49% CSS properties. Example:
<div id="wrapper"> some title of header
<div id="left-side">some content</div>
<div id="right-side">some content</div>
</div>
Usually this markup would render the wrapper as high as the wrapper's text height (one row only). What you'd expect is that it goes around all of the inside floats. It does not unless you give the container the following CSS property:
overflow:auto;
However, it does not stop there. As expected, this would do the trick well in Firefox (2.x), but IE (as expected,again) does not entertain the whole idea unless a really weird CSS hack is applied (to the container element):
_height:1%;
Seems to produce desired results in IE as well.

Friday, March 14, 2008

Generating SQL INSERT statements for existing data in SQL Server

I've been probably faced with the same problem of seed data during development and I've never managed to get around having a simple procedure to insert default data in some quick and automated way. Well, finally I've bumped into this absolutely amazing stored procedure that does exactly that. Get the procedure from here, courtesy of Narayana Vyas Kondreddi's code library. Works like a charm!

To create the INSERT statements for a selected table use:
EXEC sp_generate_inserts 'yourtablename'
To get data for all your tables you need to create one stored procedure call (as per above) for each table. Following SQL script will do just that:
SELECT 'EXEC sp_generate_inserts ' +
'[' + name + ']' +
',@owner = ' +
'[' + RTRIM(USER_NAME(uid)) + '],' +
'@ommit_images = 1, @disable_constraints = 1'
FROM sysobjects
WHERE type = 'U' AND
OBJECTPROPERTY(id,'ismsshipped') = 0
That's it. Above code will create one stored procedure call to create insert statements per table which you then need to execute separately to get the actual inserts. Really great script.

(updated 23 Mar 2008)
Another useful script for generating stored procedures for CRUD (found it here). It is a script file that you must change each time you run against a different database so I suppose it would make sense to put all that into a stored procedure.

Thursday, March 13, 2008

"MySQL server has gone away" query error message with Drupal

For no obvious reason my Drupal administration site started showing this huge list of SQL queries with an error message "MySQL server has gone away query" followed by some arbitrary sql. After reading this I tried increasing the max. packet size to 24MB (sure sound huge!) and luckily all came back to life again.

Not sure what to do when you dont have access to this parameter though. It looks like drupal was trying to store really big chunks of data in a single query (I'm guessing cache-related). Perhaps there's a way to disable this which would in turn make the error go away?

VB6 date conversions

Came across a problem with converting strings into dates in a VB6 app. Basically our date strings were in the format of "MM/dd/yyyy" but after using CDate(...) to convert the string in that format into a date it would have different results depending on the string itself. For example, for a "01/13/2008" CDate(...) would return the correct date of 13th Jan 2008. However, given a "01/12/2008" it would produce 1st Dec 2008.

I am guessing this behavior probably has something to do with my regional settings as well but I gave up trying to figure that part and just started using VB6 DateSerial(year, month, day) function to create my dates. Works wonderfully. Wish I had seen it before.