July 6, 2009 by Arvind Shyamsundar

t-SQL Anti-Pattern: Index Key Order and Query Expression Order

This is really not a T-SQL anti-pattern as much as it is a database design issue, but we see it so often that it’s worthwhile bringing it up and clarifying things.

For illustrating the scenario, let’s examine the table Person.Contact in the AdventureWorks database. It has 2 columns called FirstName and LastName. Let’s say an application frequently queries this table with these columns in the WHERE clause. The query looks like this:

SELECT ContactID from Person.Contact

WHERE FirstName = ‘Carla’ and LastName = ‘Adams’

In order to support the query for seeking to the data, we create this index:

create nonclustered index idx_contact_names on Person.Contact(FirstName, LastName)

Now, let’s say there’s another application which fires another query on this table, and that query looks like this:

SELECT ContactID from Person.Contact

WHERE LastName = ‘Alberts’ and FirstName = ‘Amy’

Notice the difference between the 2 queries: the predicate ordering in the expression is different. Now, for the problem: some developers will now create another index, with the column order as (LastName, FirstName). That is not required. If you view the execution plan for both the queries, you will notice that the index is being used!

If you end up creating 2 indexes for the above scenario, SQL Server will effectively use only one of them for queries such as the above. The other index will only add to the overhead of index maintenance required during DML operations. So it is a redundant index and as such should be dropped.

Conclusion

The predicate order is independent of the choice of index to be created / the choice of index being used. SQL Server will use the index which yields a good plan.

June 21, 2009 by Arvind Shyamsundar

Tip of the day: “An attempt was made to load a program with an incorrect format” .NET P/INVOKE issue

The other day I was using a 3rd party utility which was built on the .NET platform. My primary work computer happens to be a x64 installation. So on this computer when I fired the utility up, and tried to perform some tasks it would error with a .NET Exception which basically had the following characteristics:

– Message: “An attempt was made to load a program with an incorrect format”

– Exception: System.BadImageFormatException

After some troubleshooting it turned out that this utility was trying to load a plain-old DLL (which exported some functions) presumably using P/Invoke. The DLL was built for 32-bit platforms. Now it turns out that by design a 64-bit process (the 3rd party utility would run as a 64-bit process owing to the 64-bit .NET runtime) would be prevented from loading a non-COM 32-bit DLL (32-bit COM DLLs are loaded in a DLLHOST.EXE surrogate when invoked from a 64-bit client process, BTW…) with the above exception.

To configure the utility to run as a 32-bit .NET process, it turns out you can use the CORFLAGS utility. You run it as follows to switch the 32-bit execution mode ON:

corflags utility.exe /32Bit+

To turn it off, just use /32Bit- in the above command line.

May 23, 2009 by Arvind Shyamsundar

VISUAL STUDIO REMOTE DEBUGGER HISTORY

Let’s say you use the Visual Studio Remote Debugger extensively, and with a wide variety of remote targets. Very quickly the list of qualifiers (see the image below to understand what I refer to) can grow quickly with ‘noise’ items.

If you were curious about where these are stored, they happen to be under one of the sub-keys of the following registry key:

HKCUSoftwareMicrosoftVisualStudio9.0DebuggerPort MRU

Before you get any further ideas, I must include the statutory warning:

WARNING: Using Registry Editor incorrectly can cause serious problems that may require you to reinstall Windows. Microsoft cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk. If you edit the registry, click the following article number to view the article in the Microsoft Knowledge Base:

Description of the Microsoft Windows Registry

April 11, 2009 by Arvind Shyamsundar

Rolling upgrade of 1 instance from 2-node, 2-instance sql failover cluster to slipstreamed sql 2008 SP1

Now that SQL Server 2008 SP1 is released (download link here), it is time to test a very interesting capability: Service Pack slipstreaming. That means that you can save some time by doing an ‘integrated’ installation of SQL 2008 which includes the SP1 binaries and avoid having to apply SP1 later on. Windows had this capability for some time but only now is it officially supported for SQL Server 2008.

Scenario

This walk-through was conducted on a 2-node, 2-SQL instance Windows 2003 x86 cluster. The 2 instances were:

VIRSQL2K – default instance of SQL Server 2000 SP4 (2039 build)
VIRSQL2K5SQL2K5 – named instance of SQL Server 2005 SP2 (3042 build)

Objective and Tools

We want to upgrade the SQL 2000 instance to SQL 2008, in-place. Currently we will not be touching the SQL 2005 instance (maybe I will do another blog post for that if there are any specific observations.) The objective was to minimize downtime of either instance while performing an in-place upgrade to SQL Server 2008 SP1. To achieve the objective, we use 2 new features which are available in SQL Server 2008:

Rolling upgrades – by using this approach to upgrade, we minimize the time that the instance which is being upgraded is not available to end users. Typically in SQL 2005 or 2000, both version and service pack / hotfix upgrades required total downtime wherein the instance will not be available. Comparatively with rolling upgrade to SQL 2008 we can have minimum downtime, which is restricted to the amount of time it takes to upgrade system databases and the user databases to the SQL 2008 format. In my testing this minimum downtime was measured at around a minute.
- http://support.microsoft.com/kb/958734 talks about rolling updates / upgrades
Slipstreamed setup – this is a new feature introduced in SQL Server 2008 SP1. If you were not aware about this capability, I would request you to study the details from the following resources:
- http://blogs.msdn.com/petersad/archive/2009/02/25/sql-server-2008-creating-a-merged-slisptream-drop.aspx
- http://blogs.msdn.com/psssql/archive/2009/03/17/how-to-fix-your-sql-server-2008-setup-before-you-run-setup-part-ii.aspx

Step-by-step

1. Pre-requisites

Install the Windows Installer 4.5 (found on SQL 2008 DVD) prior to your actual downtime. Do not reboot at this stage.
Install the hotfix for KB article 937444 (from http://support.microsoft.com/kb/937444). The SQL issue pertaining to this OS hotfix is documented in http://support.microsoft.com/kb/955828
Review all the cluster specific best practices documented in my previous post.
Install .NET Framework 3.5 SP1 (found on SQL 2008 DVD) prior to your actual downtime.
At a suitable window (you may couple this with your actual upgrade downtime) you need to reboot to complete the process.
Do these steps on both nodes. Take care to stagger the reboots mentioned above to avoid total non-availability of your SQL instances.

1. Prepare the merged (slipstreamed) media

In my case I was just dealing with x86 instances, so I optimized some steps from the blogs above

From SQL 2008 DVD, copy the x86 subfolder and the files from the root (setup.exe etc.), I did not copy the IA64 and x64 folders. I copied these to c:sql2008 on my local nodes
Extract the SQL 2008 SP1 contents to c:sql2008pcu on each of the nodes. To do this I used
- SQLServer2008SP1-KB968369-x86-ENU.exe /x:c:sql2008pcu /q
I then copied setup.exe and setup.rll from the c:sql2008pcu folder to c:sql2008, overwriting the older ones
I then copied the FILEs (NOT the sub-folders, and also ignore the Microsoft.SQL.Chainer.PackageData.dll from c:sql2008pcux86) of the c:sql2008pcux86 folder, into the c:sql2008x86 folder.
- If you accidentally copied Microsoft.SQL.Chainer.PackageData.dll (like I did once) you will receive an error message when you launch setup: ‘specified action LandingPage is not supported for the sql server patching operation’. (In fact the blog post from Peter also talks about this error.)
Peter’s original blog post and also the current one from Bob Ward mention copying the sqlsupport.msi files over. I did do that, though Peter mentioned it is no longer required. Copy the sqlsupport.msi from c:sql2008pcux86setup1033 to c:sql2008x86setup (note that there is no 1033 in the destination, that is not a typo.) You can overwrite the older version and you may note that the older version was actually larger than the new one!
At this time you are ready to roll.

2. Run setup on passive node

I ran setup from the command prompt, specifying the PCUSource by hand:

cd c:sql2008
setup.exe /Action=UPGRADE /PCUSource=c:sql2008pcu

Initially I select the SQL 2000 instance, which is active on the other node:

In the cluster security policy screen, you need to enter the service account domain group names. Now something related to this step caused a problem for later on, I will explain it later:

At the Upgrade Rules screen in setup, you can verify that we are slipstreaming:

Also in Upgrade Rules you might be warned that any other SQL instances active on this node will be restarted due to cluster resource DLL update. This is very important if you have not planned on those other instances being restarted. So you should note it and factor it into your upgrade plans:

Later the cluster upgrade report clearly tells us it is going to upgrade just this (passive) node:

Once again we check we are slipstreaming:

The rest of setup was fairly uneventful and at the end I checked the sqlservr.exe version on this (passive) node:

If you check the old SQL 2000 installation folder, you will note that the binaries and other folders have been cleaned up.

3. Change group for failed Full-Text Search resource

At this stage you may notice that the full-text search service cluster resource is either stopped or plain does not even show up. In my case I received the following message in cluster admin. I just moved out the fulltext resource into another cluster group for the moment.

4. Failover and watch the upgrade of system databases happen

At this stage, you are ready to failover the SQL 2000 instance from (let’s say) NodeA (which is still running SQL 2000 and has not been touched by upgrade process) to the NodeB (on which the SQL 2000 service has been upgraded, binaries-only, to SQL 2008). What is supposed to happen after the move group, is that the SQL 2000 instance is temporarily unavailable, SQL 2008 service starts up on NodeB, it then upgrades the system databases to SQL 2008 ‘format’ and also upgrades the user databases to SQL 2008 ‘format’. This is technically a point of ‘no-return’ from which roll back means reinstalling SQL 2000 and restoring from backups.

In my case, things did not go well initially, and on failover from NodeA to NodeB, SQL 2008 instance failed to come online on NodeB, restarted and couple of times and then flipped back to NodeA. Checking the event log showed this message:

initerrlog: Could not open error log file ”. Operating system error = 3(The system cannot find the path specified.).

I verified the startup parameters using Configuration Manager on NodeB and also double-checked using Enterprise Manager on NodeA, and they were correct. The only other possibility was a permissions issue, and I used Process Monitor from SysInternals to track down the issue. It turned out that my SQL 2008 service account (which was the same as what the 2000 version used) did not have access to a registry key. Then it dawned on me that the domain group membership I referred to previously, was not granted. To fix this was easy, used AD Users and Computers and added my service account into this group:

After this was fixed, I was able to move group again to NodeB at which stage the database upgrades happened and the instance was online. Here are some random snippets from the SQL errorlog at the time of upgrade, just to show you what happened under the hood:

2009-04-11 04:08:06.43 Server Microsoft SQL Server 2008 (SP1) – 10.0.2531.0 (Intel X86)
2009-04-11 04:08:17.14 spid7s Database ‘master’ running the upgrade step from version 654 to version 655.
2009-04-11 04:08:18.84 spid8s Converting database ‘model’ from version 539 to the current version 655.
2009-04-11 04:08:33.61 spid7s Database ‘master’ is upgrading script ‘sqlagent90_msdb_upgrade.sql’ from level 0 to level 2.
2009-04-11 04:09:26.75 spid7s Recovery is complete. This is an informational message only. No user action is required.

Total time from start to finish: 1:01 minutes. That means that my instance was potentially unavailable just for this long. That is pretty impressive!

5. Run setup on previously active node

The process on NodeA was pretty much the same as described above. All of the steps described above are relevant, except for #4 above, because the databases are already upgraded once. The only thing you may come across if you have not handled the full-text search resource issue would be an error in setup: ‘The device is not ready’:

I am not 100% sure, but it seems related to the fact that full text was not moved out to another group. Moving it out fixed the above setup error on NodeA.

6. Verify setup

Just a casual check here using Management Studio (yes, the management tools get upgraded from Query Analyzer / Enterprise Manager to SSMS) and verify that we are indeed on SQL 2008 SP1:

Key Learnings

Read the blog posts mentioned above very carefully while preparing the merged drop
Watch out for the exact DLL to be excluded (Microsoft.SQL.Chainer.PackageData.dll).
Pre-create the domain group for SQL Server 2008 service account in the AD (only required if you are on Windows 2003)
Add the SQL 2000 service account to the SQL 2008 domain group before you run the setup program
Change group for the failed Full-Text Search resource from the SQL group to any other group
Due to resource DLL update, be prepared for a restart of any other SQL instance on the node you are upgrading (such as in a N-instance cluster)

I hope you enjoyed this blog post and if you do like it, please do take a second to rate this post and also leave a few comments if you can. See you later!

January 30, 2009 by Arvind Shyamsundar

Debugging Toolbox

This one is a quickie for an easy reference to most commonly used debugging tools and links. I hope you find it useful, and kindly indicate your feedback on this page by using the comments section or by rating the post!

Debugging Toolbox

Tool	Key Usage Scenarios	Download location
WinDbg	Interactive production debugging Dump analysis Local kernel debugging	http://www.microsoft.com/whdc/devtools/debugging/default.mspx
CDB	Scriptable, low overhead capture of dumps Low overhead interactive debugging
ADPlus	Scripted automation for capturing dumps
DebugDiag	Can setup rules for capturing dumps in an unattended, logged-off scenario Automated analysis of basic crash, hang and leak scenarios	http://www.iis.net
SysInternals Process Monitor	Interactive process and thread level details Handle usage	http://www.sysinternals.net
TLIST	Listing services Listing process tree Listing which processes have loaded a module	http://www.microsoft.com/whdc/devtools/debugging/default.mspx
Performance Monitor	Interactive display of performance data on the system (System Monitor) Logged-off unattended capturing of performance logs	Shipped with the OS
Application Verifier	Enable various checks to trap deep rooted issues earlier such as Orphaned critical sections Heap corruption Unsafe API usage Simulate low memory conditions	http://msdn.microsoft.com/en-us/library/aa480483.aspx http://www.microsoft.com/downloads/details.aspx?FamilyID=C4A25AB9-649D-4A1B-B4A7-C9D8B095DF18&displaylang=en
PREFast	Static source code analysis to detect potential buffer / stack overruns etc.	http://www.microsoft.com/whdc/devtools/tools/prefast.mspx
XPerf / XPerfInfo	Profiler / tracing of user mode applications	http://msdn.microsoft.com/en-us/library/cc305187.aspx http://msdn.microsoft.com/en-us/performance/cc825801.aspx http://blogs.msdn.com/seema/archive/2008/10/08/xperf-a-cpu-sampler-for-silverlight.aspx
UserDump	Automated capture of dumps	http://www.microsoft.com/downloads/details.aspx?FamilyID=e089ca41-6a87-40c8-bf69-28ac08570b7e&DisplayLang=en
LeakDiag	Leak diagnostics	ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/LeakDiag

Native code debugging sites and blogs

http://www.advancedwindowsdebugging.com

http://blogs.msdn.com/ntdebugging

http://www.osronline.com

http://www.alex-ionscu.com

http://www.dumpanalysis.org

http://www.nynaeve.net/

http://blogs.thinktecture.com/ingo/archive/2006/08/05/414674.aspx

.NET articles:

http://msdn2.microsoft.com/en-us/library/ms954594.aspx

http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer

http://msdn.microsoft.com/msdnmag/issues/05/03/Bugslayer

http://support.microsoft.com/kb/892277

http://msdn.microsoft.com/msdnmag/issues/05/07/Debugging/

http://msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut/default.aspx

.NET debugging blogs:

http://blogs.msdn.com/yunjin/

http://blogs.msdn.com/tess/

http://blogs.msdn.com/jmstall/

http://blogs.msdn.com/mvstanton/

http://blogs.msdn.com/cbrumme/

http://blogs.msdn.com/maoni/

http://blogs.msdn.com/toddca/

http://blogs.msdn.com/suzcook/

http://www.debugtricks.com

January 3, 2009 by Arvind Shyamsundar

SQL Server 2000 / 2005 Failover Cluster Pre-installation Checklist

Here’s a quick checklist before you install an instance of SQL Server on a failover cluster; or any service pack or hotfix for an instance. Following this list should eliminate 90% of the most common causes of setup failure.

Please note that this checklist does not cover all the ‘normal’ requirements for installation of a SQL Server failover cluster instance. For that you may refer to this excellent white paper on the subject. I also recommend that you view my colleague Uttam Parui’s webcast as well prior to attempting any installations on Windows Server 2008. He covers many issues specific to Windows 2008 as well.

Common points for all SQL 200x versions

Checklist item	Description
1. Name resolution	Verify network name resolution of physical nodes using machine name and FQDN (fully qualified DNS name) using PING
	Verify network name resolution of SQL Server virtual network name using PING of network name and FQDN
2. Verify admin shares	From each node, try to access the C$ and ADMIN$ shares of each of the other nodes.
3. Verify network binding order	Click Start, point to Settings, click Control Panel, and then double-click Network and Dial-up Connections. On the Advanced menu, click Advanced Settings. In the Connections box, make sure that your bindings are in the following order, and then click OK: External public network Internal private network (Heartbeat) [Remote Access Connections] The idea is to have the public network as the highest in the binding order.
4. Verify the cluster network priority and ensure that private network is for internal communications only.	Start Cluster Administrator. Click the cluster name at the root of Administrator. On the File menu, click Properties. On the Network Priority tab, verify that the private network is listed at the top. If it is not, use the Move Upbutton to increase its priority. Click the private network, and then click Properties. Click to select the Enable this network for cluster usecheck box. Click Internal cluster communications only (private Network). The idea here is to make sure that the private network only carries internal cluster traffic and that it is listed as the #1 preferred network to carry that traffic. Refer to http://support.microsoft.com/kb/258750 for this and the previous point.
5. Ensure that all the nodes are in perfect time synchronization with your DC	Using NET TIME command you can check (and optionally set) the local time on each node. Failure to synchronize the time on the nodes will cause Kerberos authentication failures which in some cases can cause setup to fail.
6. Verify that there are no ‘special’ characters in resource names	Avoid the following special characters in the resource names: > < “ ‘ & (, ) + Characters with a dieresis, a tilde, or a space character. Reference: http://support.microsoft.com/kb/302491
7. Ensure all the physical disk resources in the cluster are online	Though this issue primarily has been observed for SQL Server 2005, it may be a good idea to check it for all versions. Reference: http://blogs.msdn.com/sqlserverfaq/archive/2008/12/17/sql-server-2005-setup-failing-to-install-on-cluster-with-unexpected-failure-while-performing-the-scc-checks.aspx
8. Remote Registry	Ensure that the remote registry service is started
9. Service Account User Rights Assignment	Ensure that your SQL Server service account has the following privileges: Act as part of the operating system (Microsoft Windows 2000 only Adjust memory quotas for a process Bypass traverse checking Log on as a batch job Log on as a service Replace a process-level token Lock Pages in Memory (Only for SQL Server Service account to use AWE on 32-bit) Reference: http://support.microsoft.com/kb/283811, http://support.microsoft.com/kb/840219 and http://support.microsoft.com/kb/254321
10. Service Account should be member of local administrators	Unfortunately the SQL Server 2000 program mandates this pre-requisite. But if you carefully follow the steps to assign the user rights, NTFS permissions and registry permissions as per the above mentioned KB articles, then you can change the account later on post setup.

SQL Server 2005 specific points

Checklist item	Description
1. Verify required services are started	The following services must be started on both nodes: Task Scheduler Remote Registry COM+ System Application Server Cryptographic services
2. MSDTC cluster resource requirement	If you are installing anything other than the database engine component on Windows Server 2003, then you must create the MSDTC resource in the cluster prior to running setup. Reference: http://support.microsoft.com/kb/301600
3. Folder structure for installation files	If you are installing from a network dump, make sure that the Servers and Tools sub-folders are preserved correctly. For example in the D:SQLServer2005 folder, create the following two subfolders: Servers Tools
4. Remote desktop connections on passive nodes.	Except for the node from where the installation is being run, please log off from any Remote Desktop or Terminal Services sessions on all the other nodes. It may be a good idea to log off from the console session as well.
5. Non-administrator Service Account	The SQL Server 2005 service account need not be a member of local administrators. As long as it has the user rights (from point #9 in the previous table) assigned to it, it will work without any local administrative privileges. The SQL 2005 setup program automatically grants the necessary permissions and user rights to the service accounts chosen. Reference: http://support.microsoft.com/kb/283811

Please let me know if you liked this article and if it was useful for your purposes, by rating the article and optionally providing any comments. If you have any questions as well please leave them as comments; I will respond to them!

December 31, 2008 by Arvind Shyamsundar

Some notes on database snapshots

Overview

I recently conducted some tests to double check the exact behavior of database snapshots when:

Snapshots are created on volumes which are much smaller than the actual database size
Metadata-only operations such as DROP TABLE are executed
DDL operations execute on the main database and the snapshot runs out of disk space

Here is my testing methodology and the customary summary of observations.

Scenario

Drive C: contains the original data file for database TestSS. Data file size = 1.95GB
TestSS.dbo.T1 is a table which contains 1GB of data
TestSS_1 is a snapshot which was created on TestSS after the 1GB data insert was completed.

Test Case A: Can we create a snapshot on volume smaller than the database size?

The snapshot TestSS_1 was created on volume T: which is 10MB in size.
Size on disk for snapshot at creation = 192KB
The snapshot creation succeeded without any errors.

If you are curious as to what I mean by ‘size on disk’, please refer to this snapshot:

It is a property of the NTFS sparse file that the actual space occupied on disk is much lesser initially than the ‘Size’ which is displayed for that file in Explorer.

Test Case B: Metadata-only operations and snapshot files

Next, I executed a DROP TABLE T1 on TestSS, which returned immediately without any errors. At this time the snapshot did not have a large ‘on disk size’ – just 1.48MB
At this time the data is accessible through the snapshot just fine. If you do a select count(1) from testSS..T1 it fails (as expected). So the main operation (DROP TABLE) worked just fine.

How is this possible? It turns out that certain operations such as TRUNCATE TABLE, DROP TABLE etc. directly operate on the IAM chain and possibly other internal allocation pages to efficiently ‘clean up’ data and metadata. In such cases the copy-on-write to the snapshot is deferred till such time that the actual data page and extent are touched.

Test Case 3: What effect does failure to write to the snapshot have on the main database?

Next, I reverted the database TestSS to the snapshot TestSS_1 and then dropped the TestSS_1 snapshot. How do you revert a database to a snapshot? You use the RESTORE DATABASE … command like this: RESTORE DATABASE TestSS FROM DATABASE_SNAPSHOT = TestSS_1
I recreated TestSS_1 immediately using the CREATE DATABASE … AS SNAPSHOT …
Next, I executed DELETE T1 to force a logged data operation and it executed for some time before finally returning these errors:

Msg 5128, Level 17, State 2, Line 1

Write to sparse file ‘t:Datatestss_1.ss’ failed due to lack of disk space.

Msg 5128, Level 17, State 2, Line 1

Write to sparse file ‘t:Datatestss_1.ss’ failed due to lack of disk space.

Msg 5128, Level 17, State 2, Line 1

Write to sparse file ‘t:Datatestss_1.ss’ failed due to lack of disk space.

Though, interestingly the main DELETE did not fail; all rows from the table T1 have been deleted. It is by-design that any failures to write to the snapshot are not going to affect the primary operation on the database.

Summary

So in conclusion this is what we can learn from the above. Most of it is logical when you think about it, but it is better to be sure than to guess!

Database snapshots can most definitely be created on volumes which are much smaller than the original database size
Metadata only operations initially do not affect the snapshot. Only when the underlying pages are really ‘cleaned up’ by subsequent system processes, will the copy of the affected pages be moved into the database snapshot’s sparse file
Failures to write to the snapshot will never affect the original database on which the DDL is being executed

Additional Reading

Please also refer to the excellent white paper on Database Snapshots Performance Considerations Under IO Intensive Workloads for some more information on snapshots and their effect on performance.
Paul Randal’s blogs about snapshots.
Good information on snapshot internals is at the SQL CAT Team website.

December 30, 2008 by Arvind Shyamsundar

Howto: Have a Clustered instance of SQL Server listen on multiple virtual IP addresses

Recently a colleague asked me if there were any changes specific to Windows Server 2008 which allow SQL Server (clustered instance) to listen on multiple subnets. There was only one change in Windows 2008 to my knowledge (which I have detailed in this post later on) but that had nothing specific to do with SQL listening on different subnets. I did some testing and have posted the results below.

Network configuration

Assuming the following network configuration on the local nodes, there are 3 networks on this cluster:

Cluster_Public: subnet is 255.255.255.0 (used only for mixed communications in the cluster)
Second_Public: subnet is 255.255.0.0 (used only for public communications in the cluster)
Cluster_Private: subnet is 255.0.0.0 (used only for private communications in the cluster)

SQL Server 2000

This version had the capability to bind to multiple virtual IP addresses. When setting up SQL Server, on the following page, you can specify both the IP addresses. The same page can also be subsequently accessed by re-running the setup program and selecting the Advanced option to Maintain Failover Clustering.

Assuming that setup succeeds, you can check the SQL Server error log (shown below) and verify that the SQL instance is successfully listening on both virtual IPs:

The key to this is actually that the SQL Server resource in the cluster is now dependent on the 2 VIPs:

However, due to a limitation in SQL 2000, we cannot directly do this in the cluster administrator, you always have to use the setup program to do the same. As we will see later on the limitation no longer exists in SQL 2005 or 2008.

SQL Server 2005 / 2008

Things are different and easier in these versions. One option is at setup time, if you want to configure SQL Server to listen on multiple virtual IPs, you need to specify them in the setup program:

However, later on if you want to maintain the cluster and ADD a new virtual IP, all you need to do to have SQL listen on multiple Virtual IP addresses is to follow the steps below:

Take the SQL Server network name offline
Add an additional IP resource with the new (additional) VIP for SQL to listen on
Add the new IP resource as an additional dependency for the sql network name resource
Bring the SQL Server resource online, the errorlog should show sql listening on the 2 IPs:

Windows 2008 cluster, multiple subnets, ‘OR’ dependencies and SQL Server

So, with all this background, what has changed in Windows 2008? Well, it is a question of dependencies and how they affect the dependent resource (in this case SQL Server.) By default, in Windows 2000 and 2003, the failure of any one of multiple virtual IP address resource dependencies for SQL Server, will cause the SQL Server resource to restart. This is because the dependencies by default work in a ‘AND’ mode.

In Windows 2008, there is now support for each node to be on a different subnet (see this MVP article for a nice explanation) due to which support for an OR dependency has been introduced. However, for SQL Server to co-exist in this environment, both the subnets must be up for SQL Server to function correctly. I will cover this topic (SQL 2005 / 2008 on Windows Server 2008 with nodes in different subnets) in a later post.

December 30, 2008December 4, 2020 by Arvind Shyamsundar

Concurrency of Full, Differential and Log backups on the same database

NOTE: As of Dec 2020, the information in this blog post should be considered out-of-date, and provided for historical reference only.

Overview

Many a time, we are asked by customers about the effect of overlapping backup schedules: for example, whether they can backup the transaction log at the same time that a long-running complete backup of the database is taking place. Books Online does talk about the concurrency, but does not cover all versions or combinations.

Here is a short test case to illustrate the concepts involved. At the end of the post, I have provided a summary of what works in what version.

Test Case

I ran the 3 types of backups on a SQL 2005 / 2008 database at the same time:

COMPLETE a.k.a. FULL (backup database mydb to disk = ‘c:tempmydb.bak’ with init)
DIFFERENTIAL (backup database mydb to disk = ‘c:tempmydb_diff.bak’ with differential, init)
LOG (backup log mydb to disk = ‘c:tempmydb.trn’ with init)

Here is the output of each command for verbose reference. Also included immediately below is a snapshot from sysprocesses showing the blocking of the DIFFERENTIAL backup and the concurrent execution of the LOG backup.

spid	blocked	waittype	lastwaittype	waitresource	cmd
61	0	0x0062	ASYNC_IO_COMPLETION		BACKUP DATABASE
61	0	0x0086	BACKUPBUFFER		BACKUP DATABASE
61	0	0x0087	BACKUPIO		BACKUP DATABASE
62	61	0x0004	LCK_M_U	DB: 16 [BULKOP_BACKUP_DB]	BACKUP DATABASE (this is the differential backup)
68	0	0x0087	BACKUPIO		BACKUP LOG
68	0	0x0087	BACKUPIO		BACKUP LOG

Output from FULL backup

SQL Server parse and compile time:

CPU time = 0 ms, elapsed time = 0 ms.

Processed 17512 pages for database ‘mydb’, file ‘mydbData’ on file 1.

Processed 1 pages for database ‘mydb’, file ‘mydbLog’ on file 1.

BACKUP DATABASE successfully processed 17513 pages in 18.986 seconds (7.206 MB/sec).

SQL Server Execution Times:

CPU time = 70 ms, elapsed time = 20256 ms.

SQL Server parse and compile time:

CPU time = 0 ms, elapsed time = 0 ms.

Output from DIFFERENTIAL backup

SQL Server parse and compile time:

CPU time = 0 ms, elapsed time = 0 ms.

Processed 32 pages for database ‘mydb’, file ‘mydbData’ on file 1.

Processed 1 pages for database ‘mydb’, file ‘mydbLog’ on file 1.

BACKUP DATABASE WITH DIFFERENTIAL successfully processed 33 pages in 0.321 seconds (0.801 MB/sec). <– actual time for the differential backup to complete

SQL Server Execution Times:

CPU time = 51 ms, elapsed time = 19777 ms. <– total time to execute the command, including wait and runnable time

SQL Server parse and compile time:

CPU time = 0 ms, elapsed time = 0 ms.

Note that the time taken by the command is just 0.321 seconds but the elapsed time is 19.777 seconds, which clearly demonstrates the blocking.

Output from LOG backup

SQL Server parse and compile time:

CPU time = 0 ms, elapsed time = 0 ms.

Processed 3 pages for database ‘mydb’, file ‘mydbLog’ on file 1.

BACKUP LOG successfully processed 3 pages in 7.958 seconds (0.002 MB/sec).

SQL Server Execution Times:

CPU time = 60 ms, elapsed time = 15106 ms.

SQL Server parse and compile time:

CPU time = 0 ms, elapsed time = 0 ms.

Time taken by command is around 52% of time elapsed, which is probably due to the fact that we are on a single-CPU system and the task had to wait for runnable time on the CPU.

Summary

Here is the summary of the testing I conducted. I hope the summary is useful to all of you.

	What is requested?
What is already running? (rows below)	Complete	Differential	Log
Complete backup is running	Blocks in all versions *	Blocked in all versions *	Concurrently allowed in SQL 2005 / 2008. Not allowed in 2000.
Differential backup is running	Blocks in all versions *	Blocked in all versions *	Concurrently allowed in SQL 2005 / 2008. Not allowed in 2000.
Log backup already is running	Concurrently allowed in SQL 2005 / 2008.	Concurrently allowed in SQL 2005 / 2008.	Blocked in all versions *

* Note: wait type of the blocked process will be LCK_M_U with a waittype of BULKOP_BACKUP_DB

December 22, 2008 by Arvind Shyamsundar

SQL Server setup log locations and more

In order to efficiently troubleshoot problems involving failed setup, we need to use the setup logs. Once you know where the log files are located, you can start troubleshooting. What to look for in the log files is a separate topic in itself, but at least if you know where the log files are, then you can attempt to look for logical clues in the files. So, where are these log files located? Here’s a summary, listed by product and version. I took some efforts to compile the list of available rollup hotfixes and their relevant KB articles, hopefully that in itself is a very useful thing.

So, I hope the list and the notes below are very useful for you; please feel free to rate the article and provide comments as well!

Product Major Version ¹	Product Level ²	Version post setup ³	Bulletins / KB Articles	Location of setup logs ⁴
SQL Server 2000 (8.0)	RTM	8.00.194	N/A	%WINDIR%sqlstp.log
	SP4	8.00.2039	888799	%WINDIR%sqlsp.log
	Post-SP4 GDR security rollup ⁷	8.00.2050	MS08-040; 948110	%WINDIR%SQL2000-KB948110-v8.00.2050-x86-ENU.log; SQLServerClient_Hotfix_KB948110_*.log
	[Recommended] Post-SP4 QFE security rollup ^8.a	8.00.2273	MS08-040; 948111	%WINDIR%SQL2000-KB948111-v8.00.2273-x86-ENU.log; SQLServerClient_Hotfix_KB948111_*.log ¹¹
	IA64 versions of the security rollups ¹³	Per respective packages mentioned above		%WINDIR%HotfixHotfix.log %WINDIR%HotfixSQL64_Hotfix_KBnumber_.log
	Clustered instance ¹⁴	N/A		RTM: %WINDIR%Sqlclstr.log ⁵; %WINDIR%sqlstpN.log from both nodes Service Pack only: %WINDIR%sqlspN.log from both nodes; %WINDIR%remsetup.log from active node Hotfix only: Relevant hotfix files as described above from both nodes
SQL Server 2005 (9.0)	RTM	9.00.1399	N/A	%ProgramFiles%Microsoft SQL Server90SetupBootstrapLOGFiles
	SP1	9.00.2047	913090	%WINDIR%Hotfix ¹⁰
	SP2	9.00.3042	921896	%ProgramFiles%Microsoft SQL Server90Setup BootstrapLOGHotfix; files will have KB921896 as part of their name.
	Post-SP2 GDR security rollup (engine only)	9.00.3068	MS08-040; 948109	Same as above; files will have KB948109 as part of their name.
	Post-SP2 GDR security rollup (engine + Reporting Services)	9.00.3073	MS08-052; 954606	Same as above; files will have KB954606 as part of their name.
	Post-SP2 QFE security rollup (engine only)	9.00.3233	MS08-040; 948108	Same as above; files will have KB948108 as part of their name.
	Post-SP2 QFE security rollup (engine + Reporting Services)	9.00.3282	MS08-052; 954607	Same as above; files will have KB954607 as part of their name.
	CU #11 for SP2 ⁸	9.00.3301	958735	Same as above; files will have KB958735 as part of their name.
	[Recommended] SP3	9.00.4035	955706	Same as above; files will have KB955706 as part of their name.
	CU #1 for SP3 ⁸	9.00.4207	959195	Same as above; files will have KB959195 as part of their name.
	Clustered instance	N/A		%WINDIR%tasksSchedLgU.Txt ⁶; Setup bootstrap will copy all the files from remote node to active node.
SQL Server 2008	RTM	10.0.1600.22	N/A	%ProgramFiles%Microsoft SQL Server100Setup BootstrapLog ⁹
SQL Server 2008	CU #2	10.00.1779	958186	%ProgramFiles%Microsoft SQL Server100Setup BootstrapLog ⁹
	Clustered instance	N/A		At the time of writing no special considerations are known.

Notes:

This is just the common name which we all tend to use, the value in brackets is the major version number used by the product itself.
To determine the product level use SELECT SERVERPROPERTY(‘ProductLevel’). However this will not reflect the GDR / QFE hotfix status. It just shows the latest service pack.
To determine the exact SQL Server build number / version, use SELECT SERVERPROPERTY(‘ProductVersion’). (More information about other product levels and version numbers is summarized in this KB article.)
Here %WINDIR% refers to the location where Windows is installed (typically C:WINDOWS) and %PROGRAMFILES% denotes location of the Program Files folder (typically C:PROGRAM FILES).
This file is only generated for SQL 2000 clustered instance installation. Service Pack installation will update this file, but subsequent hotfix installers will not update this file.
The SQL Server 2005 RTM and service pack installers will launch the remote node setup program using Task Scheduler service. This log file is used to check if the remote task was actually started or not. Check this file (in addition to the regular setup files) on each node of your SQL failover cluster.
GDR vs. QFE releases can be a confusing topic. Luckily they are explained very well in the following blog links:
- http://blogs.msdn.com/psssql/archive/2007/04/06/post-sql-server-2005-service-pack-2-sp2-fixes-explained.aspx
- http://blogs.msdn.com/sqlreleaseservices/archive/2008/04/27/a-changed-approach-to-service-packs.aspx
In general, note that applying a Cumulative Update should be done only if you are facing specific issues fixed by that update; doing so will take you to the QFE tree and you should apply the appropriate QFE rollup (if and when available.) This note applies to all rollups and Cumulative Updates.
- 8.a However for SQL 2000 SP4, we might recommend being on the QFE tree due to the fact that there will be no further service packs for that product.
There are no separate folders for RTM and hotfix / Service Pack installers. Instead, look at the relevant sub-folder of this folder with the latest time stamp to get to the latest installation files. Historical logs will be maintained in the subfolder with the respective timestamps.
SQL Server 2005 SP1 is unique in the SQL 2005 chain because it stores its logs under the WindowsHotfix folder. Under that there are subfolders for each component, such as SQL9, SQLTools9 etc.
The exact filename is dependent on the name of the self-extracting package which was executed. The SQL 2000 QFE rollup generates many more log files than the GDR installer. This is expected because the GDR security rollup only contains security fixes. The QFE security rollup contains bug fixes and the requisite security fixes. See this KB article for details.
For what to look for in the log files, consider this topic from SQL Books Online. An additional reference for troubleshooting Windows Installer logs is found here.
The hotfix installer for SQL 2000 IA64 is described in this KB article.
Cluster delta specific instructions only; regular log files still apply.

Arvind Shyamsundar's technical blog

Arvind Shyamsundar is a Principal PM @ MSFT Azure Data, working on Azure SQL. Data geek. Apache Accumulo and Fluo PMC. SQL MCM, ex-Principal PFE (MSFT Services). These are my own opinions and not those of Microsoft.

Category / Uncategorized