Archive for the ‘SQL’ Category

Query MS CRM XML Entity with XQuery and LINQ

In our company we have been working on a large MSCRM 2011 project. Last week, I was asked to create a report listing all the visible fields for each entity in the solution. During my search, I found a few different ways to do it, but the most fun way was querying the XML.
The XML is created when the solution is exported. It’s normally several levels deep and even when the structure would depend on the solution the XML listed below is an example of how it would looks like.

<Entities>
    <Entity>
      <Name LocalizedName="Account" OriginalName="Account">Account</Name>
      <ObjectTypeCode>1</ObjectTypeCode>
      <EntityInfo>
        <entity Name="Account">
          <LocalizedNames>
            <LocalizedName description="Account" languagecode="1033" />
          </LocalizedNames>
          <LocalizedCollectionNames>
            <LocalizedCollectionName description="Accounts" languagecode="1033" />
          </LocalizedCollectionNames>
          <Descriptions>
            <Description description="Business." languagecode="1033" />
          </Descriptions>
   <attributes>
		<attribute PhysicalName="Category">
			…
	<attribute>
   </attributes>
</entity>
      </EntityInfo>
   <FormXml>
        <forms type="main">
          <systemform>
            <formid>{b053a39a}</formid>
            <form>
              <hiddencontrols>
                <data id="addressid" datafieldname="addressid" classid="{}" />
                <data id="address2” datafieldname="address2" classid="{}" />
              </hiddencontrols>
		  <tabs>
                <tab name="general" verticallayout="true" id=”" IsUserDefined="0">
                  <labels>
                    <label description="General" languagecode="1033" />
                  </labels>
                  <columns>
                    <column width="100%">
<sections>
  <section name="account information" showlabel="false" showbar="false" id="{}" IsUserDefined="0">
                          <labels>
                            <label description="Information" languagecode="1033" />
                          </labels>
                          <rows>
                            <row>
                              <cell id="{}">
                                <labels>
                                  <label description="Name" languagecode="1033" />
                                </labels>
                                <control id="name" classid="{}" datafieldname="name" />
                              </cell>
                              <cell id="{}">
                                <labels>
                                  <label description="Phone" languagecode="1033" />
                                </labels>
                                <control id="telephone1" classid="{}" datafieldname="telephone1" />
                              </cell>
                            </row>
                            <row>
                              <cell id="{}" showlabel="false" >
                                <labels>
                                  <label description="Contact" languagecode="1033" />
                                </labels>
                                <control id="primarycontactid" classid="{}" datafieldname="primarycontactid">
                                  <parameters>
                                    <FilterRelationshipName>customer_accounts</FilterRelationshipName>
                                    <DependentAttributeName>parentcustomerid</DependentAttributeName>
                                    <DependentAttributeType>account</DependentAttributeType>
                                    <AllowFilterOff>true</AllowFilterOff>
                                  </parameters>
                                </control>
                              </cell>
                              <cell id="{}" showlabel="true" >
                                <labels>
                                  <label description="Other Phone" languagecode="1033" />
                                </labels>
                                <control id="telepho" classid="{}" datafieldname="telep" />
                              </cell>
                            </row>
                            <row>
                              <cell id="{}">
                                <labels>
                                  <label description="Parent" languagecode="1033" />
                                </labels>
                                <events>
                                  <event name="setparams" application="true" active="true">
                                    <InternalHandlers>
                                      <Handler handlerUniqueId=" " enabled="true" />
                                    </InternalHandlers>
                                  </event>
                                </events>
                                <control id="parentaccountid" classid="{}" datafieldname="parentaccountid" />
                              </cell>
                              <cell id="{ }">
                                <labels>
                                  <label description="Fax" languagecode="1033" />
                                </labels>
                                <control id="fax" classid="{}" datafieldname="fax" />
                              </cell>
                            </row>
                            <row>
                              <cell id="{ }">
                                <labels>
                                  <label description="E-mail" languagecode="1033" />
                                </labels>
                                <control id="emailadd" classid="{}" datafieldname="emailadd" />
                              </cell>
                              <cell id="{ }">
                                <labels>
                                  <label description="Web Site" languagecode="1033" />
                                </labels>
                                <control id="websiteurl" classid="{}" datafieldname="websiteurl" />
                              </cell>
                            </row>
                          </rows>
                        </section>
             </sections>
                    </column>
                  </columns>
                </tab>
</tabs>
</form>
     </systemform>
        </forms>
      </FormXml>
    </Entity>
</Entities>

'

Take into consideration that we have several entities and those entities have multiple tabs, sections, controls and labels. The listing above is just an example of what MS CRM exports in XML format. It shouldn’t be consider as a real entity. Having said that, lets continue with the real fun.

My task was to find all the visible and enabled fields in the solution. I also needed to know, if the fields belong to the Account entity or any other entity and I just needed to retrieve the fields from the Main form not the Mobile form.

Before we go further, is important to say that I had to retrieve the Label description, but I had to check the cell element, “Showlabel” attribute which controls the visibility as you can see on the next example.

<cell id="{}" showlabel="true" >
  <labels>
    <label description="Other Phone" languagecode="1033" />
  </labels>
  <control id="telepho" classid="{}" datafieldname="telep" />
 </cell>

'

Like I said before, there are different ways to do this. Here are a couple examples.
The first I used was XQuery and FLWOR expressions.

Basically it searches through the XML verifies if the “showlabel” attribute is either not there or set to true and retrieves the names.


SELECT ent.vals.value('@OriginalName', 'varchar(max)') AS Entity, ent.vals.value('@datafieldname', 'varchar(max)') FieldName
FROM @xml.nodes('for $en in //Entity
return ($en/Name,(for $q in ($en/FormXml/forms/systemform/form/tabs/tab/columns/column/sections/section/rows/row/cell)
where (empty($q/@showlabel) or ($q/@showlabel=&amp;quot;true&amp;quot;))
return $q/control)) ') ent(vals)

'

It returns the Entity name in one column follow by the Fields. It looks like this:

Another way to query is using Linq to XML. The statement looks like this:


var xrmEntities =   from e in doc.Elements(&amp;quot;Entity&amp;quot;)
                        from c in e.Elements(&amp;quot;FormXml&amp;quot;).Elements(&amp;quot;forms&amp;quot;).Elements(&amp;quot;systemform&amp;quot;).Elements(&amp;quot;form&amp;quot;)
                                    .Elements(&amp;quot;tabs&amp;quot;).Elements(&amp;quot;tab&amp;quot;)
                                    .Elements(&amp;quot;columns&amp;quot;).Elements(&amp;quot;column&amp;quot;)
                                    .Elements(&amp;quot;sections&amp;quot;).Elements(&amp;quot;section&amp;quot;)
                                   .Elements(&amp;quot;rows&amp;quot;).Elements(&amp;quot;row&amp;quot;)
                                    .Elements(&amp;quot;cell&amp;quot;)
                        where (string)c.Attribute(&amp;quot;showlabel&amp;quot;) == &amp;quot;true&amp;quot; || c.Attribute(&amp;quot;showlabel&amp;quot;) == null
                        from l in c.Elements(&amp;quot;labels&amp;quot;)
                        let formName = e.Element(&amp;quot;Name&amp;quot;).Attribute(&amp;quot;LocalizedName&amp;quot;).Value
                        let fieldName = l.Element(&amp;quot;label&amp;quot;).Attribute(&amp;quot;description&amp;quot;).Value
                        orderby formName, fieldName
                        select new {Entity = formName, Field = fieldName };
    xrmEntities.Count().Dump();
    xrmEntities.Dump();

//”doc”  variable loads the XML file.

The result returned by Linq is probably easier to understand. It looks like this.

Either way returns the Fields Names so, It’s up to you to decide what to use Linq or XQuery. Both of them are good things to learn as a developer.

Tags: , , , , , , , ,


Denali New T-SQL features.

SQL Server Denali comes with a number of new features that will help to make our life easier or more complicated depending on your point of view. However, I think it will make our life a lot easier.

A few of the new features that I will talk about today are the Try/Catch block, execute a store procedure With Result Sets and the IIF function.

TRY/CATCH

It was originally introduced in SQL 2005. However it lacked flexibility and we were not able to re-throw the error like we do it in C# or VB.net. However, now we can using “THROW”. Here is an example of how it works:

DECLARE @REG VARCHAR(50);
DECLARE @MORE INT;

	BEGIN TRY
		SELECT @REG= RegionDescription FROM DBO.Region R
		INNER JOIN Territories T ON T.RegionID = R.RegionID	 

		SELECT @MORE= @REG

	END TRY

	BEGIN CATCH
		PRINT 'You need to cast the varchar before assign it to a int';
		THROW;

	END CATCH

This is just a simple example of how it works, however there are many different ways to format the message so you can display or return the right message for your code or UI. You can find additional information at Microsoft

WITH RESULT SETS

Another new feature that was called to my attention is the execution of stored procedures “WITH RESULT SETS”. With this new feature you won’t need the high cost efficient functions. Something else that is nice with this feature is that you can rename the columns returned by the stored procedure. Also if you were using a temp table in order to see the data you will not need it any more with the new “WITH RESULT SETS.” Here is an example:

-- Create the store procedure in the Northwind database
CREATE PROCEDURE [dbo].[spTerritories] (@Region int)
as
BEGIN
	SELECT TerritoryID, TerritoryDescription, RegionID FROM Territories
	WHERE RegionID = @Region
END
GO

EXEC spTerritories @Region = 1
WITH RESULT SETS
(
	(
		PlaceID varchar(100),
		PlaceDescription text,
		Region float
	)
)

Now, regarding cost, the new approach is a lot more efficient than the previous versions. In fact if you run something like this:

DECLARE @TestTable TABLE (TerritoryID varchar(MAX), TerritoryDescription text,
    RegionID float )

INSERT INTO @TestTable
EXEC spTerritories @Region = 1

SELECT * FROM @TestTable

It will cost you around 5 times more than just executing the store procedure With Result Sets. Take a look at the image below.

The blue square is the execution of the store procedure with the result sets that was 14% of the batch. The red squares make a total of 14 + 57 + 14 = 85% of the whole batch, so there is no doubt that using WITH RESULT SETS is more efficient that the old way.

IIF Function

Another new feature in SQL Denali are the built in functions. I will not write about all of them but you can find great examples of them at Pinal Dave’s blog. You can find a reference to his blog at the end of this post.

The one that caught my attention was the new IIF function. It basically replaces the Case statement for simple scenarios. Obviously if you have a long nested IIF function the CASE may be your best option. However the new IIF improves the code a lot.

Here are a couple of examples using tables from the Northwind database.

SELECT IIF(RegionID >= 2, 'Good Region', 'The best') as RegionScore,
		TerritoryID, TerritoryDescription, RegionID
FROM  dbo.Territories

Another example would be something like this. In this case I’m nesting the IIF function.

SELECT IIF(RegionID >=2,
		IIF(
			(SELECT e.FirstName
			FROM  Employees  e
			LEFT JOIN EmployeeTerritories et
                            on e.EmployeeID = et.EmployeeID
			WHERE et.TerritoryID =ttr.TerritoryID) is null, 'No Data',
			(
			        SELECT  e.FirstName
				FROM Employees  e
				LEFT JOIN EmployeeTerritories et
                                    on e.EmployeeID = et.EmployeeID
				WHERE et.TerritoryID =ttr.TerritoryID
		       )
			),'No Employee Assigned') as RegionScore,
		TerritoryID, TerritoryDescription, RegionID
FROM  dbo.Territories  ttr

Some of the other built-in functions are specified on Pinal’s website. You can take a look at them Clicking Here. Microsoft is another reference for any information I wrote on this post.

Tags: , , , , , , , ,


New SQL “Denali” is here

Technology users are growing around the world, as well as data and device dependency , whether it be pc, laptop , tablet or smart phone. Even when the economy doesn’t look very promising, generally speaking there are some other sectors that are not being hit as hard as others, technology is one of them. In fact many websites such as CNMBC.com, USNEWS.com and others are ranking technology carriers as the most growing carriers in the market right now. Demand on this career is growing exponentially as well as tools to supply such demand.

One of the new tools just releases for testing is “Denali”. This is new version of Microsoft SQL Server. This new version is cloud-ready information platform and looks pretty processing. It has the ability to interact with OData and sync information between multiple platforms.

Microsoft is also promising that the OS patching and downtime can be reduced significantly by 50-60% less by running SQL server on Windows Server Core.

Regarding the performance, Microsoft is introducing In-Memory Column Store or better known as “Apollo” on this version. Basically Apollo brings together the Vertipaq technology that was developed in Analysis Services and a new quory excecution paradigm called batch processing to provide speed up for common data warehouse queries.

Some other features that new Denali brings up are:

  • “Crescent”, is a reporting services project. According to Microsoft, it is easy to use and provides with highly interactive web-based data exploration, visualization, and presentation experience to users of all levels.
  • BI Semantic Model which is a single model for user with multiple ways of building business intelligence solutions.
  • “Juneau”, is the project for SQL Server Developer Tools. Actually “Juneau” unifies the development for database development across Database, BI and web and supports SQL Server and SQL Azure.

In order to install Denali there are some requirements you have to take in consideration:

  • First Denali does not install or enable Windows PowerShell; however Windows PowerShell 2.0 is an installation prerequisite.
  • .Net 3.5 PS1 is a requirement and you have to enable .NET framework 3.5 SP1 before you install Denali if you are installing it on a Windows Server 2008 R2 .Net 4.0 is a requirement too.

Regarding hardware and OS Denali can be installed on a Windows Server 2008 and 2008 R2 SP1, Windows 7 SP1 and Vista SP2. It requires a minimum of 1 GB on memory and a 64 version requires a 1.4Ghz process or a 1.0 GHz process if you install the 32-bit version.

Installing Denali was not a big problem. However it took more than 30 minutes installing.

When I start using it I didn’t noticed a big difference between this version and 2008 R2 Version. Here is an email of the new Management Studio UI.

The new UI is powered by Visual Studio so you will fell that you are working on it instead of regular SQL Management Studio.
In future blogs I will discuss about new TSQL features and Reporting tools as well as BI tools that Denali provides.

If you are interested on download and Install Denali you can download it from: Microsoft
Also if you want to learn more about it this video may help you. It talks about new Denali features. The speaker is Dandy Weyn and I can tell that he does a great job presenting the product.
Take a look at it. Microsoft Technet Videos

Tags: , , , ,


Table-Valued Parameters

Being able to pass a table as a parameter to a stored procedure is a welcome addition to SQL Server 2008. In previous versions you had to use an XML document or a delimited string, both of which required additional processing. Below is a simple example that populates the Shippers table in the Northwind database. See table-valued parameters in the MSDN documentation for more information.

-- Create a new user-defined table type that matches
-- the Shippers table in the Northwind database
-- (excluding the PK/Identity field).
CREATE TYPE dbo.ShipperTable AS TABLE (
    CompanyName     NVARCHAR(40) NOT NULL
    ,Phone          NVARCHAR(24) NULL
)
GO

-- Create a new stored procedure with a table valued parameter
-- to insert the data (you must specify the READONLY clause).
CREATE PROCEDURE dbo.AddShipper (
    @Shipper    ShipperTable READONLY
)
AS
    INSERT INTO dbo.Shippers
        SELECT  CompanyName, Phone FROM @Shipper
GO

-- Create a variable based on the new table type
DECLARE @LocalShipper ShipperTable

-- Populate the table variable using the table value constructor
-- syntax. In the value expression each list must be enclosed in
-- parentheses and separated by a comma.
INSERT INTO @LocalShipper VALUES
 ('DHL', '(480) 555-1234')
,('FEX', '(480) 555-5678')
,('UPS', '(480) 555-9012')

-- Insert the new shippers using the table variable as a parameter
EXEC AddShipper @Shipper = @LocalShipper

SELECT * FROM dbo.Shippers

-- Clean up
DELETE FROM dbo.Shippers WHERE CHARINDEX(CompanyName, 'DHL,FEX,UPS') > 0
DROP PROCEDURE AddShipper
DROP TYPE ShipperTable

Tags: , , , ,


MS CRM SQL Server Traces – CRM Brings the Noise – and I Filter it!

Every once in a while when doing customization or development for MS CRM I need to run a SQL trace. Anyone who has ever watched a trace of SQL Server for MS CRM (or any enterprise level application, for that matter) knows that a lot of SQL gets generated.

In my case, I usually don’t want to see all the “background noise” SQL that the app generates for it’s internal processing – I’m more interested in what SQL is running to select, insert or update records. SQL Server has the option to add filters to filter out items you don’t want to see in a SQL trace.

Assuming you already know how to create a filter in SQL Server, you can add each of the following statements as a filter to the column “TextData” and filter out a lot of things that you would prefer not to see in the results. I hope it helps, of course, but I warn you that if I am filtering out something that you actually do want to see in the results, don’t blame me:)

NOTE: This is for CRM 4.0 and has not been testing in MSCRM 2011. I would say that it probably will not work very well.

%exec sp_executesql N'With TopSuspendedAsyncOperations(AsyncOperationId)%
%if exists (select * from WorkflowWaitSubscriptionBase (nolock) where IsModified = 1)%
%exec GetDBVersion @DBVersion=@p1 output%
%exec sp_reset_connection%
%exec sp_executesql N'update AsyncOperationBase%
%-- network protocol: LPC%
%exec sp_executesql N'SELECT OrganizationId, EventId, EventData, CreatedOn FROM Notification%
%exec sp_executesql N'select subscription.WorkflowWaitSubscriptionId, subscription.AsyncOperationId, subscription.EntityName,%
%if exists (select * from WorkflowWaitSubscriptionBase (nolock) where IsModified = 1)%
%and DeletionStateCode = 0',N'@lockedState int,@hostId nvarchar(62),@pausingStatus int,@cancelingStatus%
%exec sp_executesql N'SELECT Id FROM DataEncryptionKey  WHERE (((IsConfigurationRow = @IsConfigurationRow0))%
%exec sp_executesql N'SELECT Id, ColumnName, BigIntColumn, IntColumn, SmallIntColumn, TinyIntColumn,%
%exec sp_executesql N'SELECT Id, CreatedOn, Enabled, KeyType, ScaleGroupId FROM CrmKey  WHERE (((Id = @Id0)) ) AND (IsDeleted =
%exec sp_executesql N'SELECT Id, CreatedOn, Enabled, KeyType, ScaleGroupId FROM CrmKey%
%exec sp_executesql N'select sdkmessagefilter0.PrimaryObjectTypeCode as ''primaryobjecttypecode'',%
%exec sp_executesql N'SELECT LocalizedLabelId AS localizedlabelid,%
%exec sp_executesql N'select sdkmessage0.SdkMessageId as ''sdkmessageid'' from SdkMessage%
%exec sp_executesql N'update WorkflowLogBase set <a href="mailto:ModifiedBy=@ModifiedBy0">ModifiedBy=@ModifiedBy0</a>, <a href="mailto:ModifiedOn=@ModifiedOn0">ModifiedOn=@ModifiedOn0</a>, Message=NULL,%
%exec p_GetFullName @organizationid=%
%exec sp_executesql N'select workflow0.UIData as ''uidata'', workflow0.IsCrmUIWorkflow as ''iscrmuiworkflow'',%
%exec sp_executesql N'select systemuser0.AccessMode as ''accessmode'', systemuser0.SystemUserId%
%exec sp_executesql N'select workflowlog0.AsyncOperationId as ''asyncoperationid'',%
%exec sp_executesql N'select workflow0.WorkflowId as ''workflowid'', workflow0.PrimaryEntity as ''primaryentity'',%
%exec sp_executesql N'delete from WorkflowCompletedScopeBase%
%exec sp_executesql N'SELECT Id, KeyType, ActiveKeyId, Enabled%
%exec sp_executesql N'SELECT Id, ScaleGroupId, KeyType,%
%select sdkmessageprocessingstepimage0.CreatedOn as ''createdon''%
%exec sp_executesql N'SELECT Label AS label,%
%SELECT EntityRelationshipId AS entityrelationshipid,%
%SELECT CascadeAssign AS cascadeassign,%
%exec sp_executesql N'SELECT EntityId AS entityid,%
%exec sp_executesql N'SELECT LogicalName AS logicalname,%
%exec sp_executesql N'SELECT T1.LocalizedLabelId AS localizedlabelid,%
%exec sp_executesql N'SELECT RelationshipId AS relationshipid,%
%exec sp_executesql N'IF EXISTS (SELECT 1 FROM dbo.MatchCode%
%SELECT MatchCode FROM dbo.MatchCode%
%exec sp_executesql N'update WorkflowWaitSubscriptionBase%
%exec sp_executesql N'select distinct AsyncOperationId from WorkflowWaitSubscriptionBase%
%exec sp_executesql N'SELECT Id, HelpServerUrl, InstallOn, IsRegistered,%
%exec sp_executesql N'insert into AsyncOperationBase%
%exec sp_executesql N'SELECT T1.AttributeId AS attributeid,%
%exec sp_executesql N'select sdkmessagerequestinput%
%exec sp_executesql N'SELECT Id, KeyType, ScaleGroupId, ActiveKeyId%
%exec sp_executesql N'select sdkmessageprocessingstep0%
%AttributeId AS attributeid FROM AttributeLogicalView%
%FROM AttributePicklistValueLogicalView WHERE AttributeId%
%exec sp_executesql N'SELECT T1.Label AS label,%
%select VersionNumber from EntityView%
%select VersionNumber from AttributePicklistValueView%
%select VersionNumber from EntityRelationshipView%
%select VersionNumber from RelationshipView%
%select VersionNumber from LocalizedLabelView%
%select VersionNumber from AttributeView%
%select VersionNumber from AttributeLookupValueView%
%select VersionNumber from EntityRelationshipRoleView%
%select VersionNumber from EntityRelationshipRelationshipsView%
%select VersionNumber from ViewAttributeView%
%exec sp_executesql N'select transactioncurrency0.ImportSequenceNumber%
%exec sp_executesql N'SELECT EntityId AS entityid FROM EntityLogicalView%
%exec sp_executesql N'SELECT AttributeTypeId AS attributetypeid,%
%exec sp_executesql N'SELECT Id FROM Organization  WHERE (((UniqueName%
%FROM AttributePicklistValueLogicalView WHERE AttributeId%
%exec sp_executesql N'SELECT T1.Label AS label,%
%exec sp_executesql N'exec p_GrantInheritedAccess @referencingId, @referencingOTC,%
%exec sp_executesql N'select contact0.OwningBusinessUnit as ''owningbusinessunit''%
%case when WorkflowState is null then 0 else 1 end%
%exec sp_executesql N'select asyncoperation0.AsyncOperationId as ''asyncoperationid''%
%exec sp_executesql N'SELECT Id, LOWER(SqlServerName) FROM Organization%
%isvalidforadvancedfind FROM EntityLogicalView%
%exec sp_executesql N'SELECT T1.AttributeTypeId AS attributetypeid,%
%exec sp_executesql N'SELECT ReferencedEntityId AS referencedentityid,%
%exec sp_executesql N'select workflowdependency0.CreatedBy as ''createdby'',%
%exec sp_executesql N'select workflow0.WorkflowId as%
%exec sp_executesql N'select asyncoperation0.OwningUser%
%exec sp_executesql N'select workflow0.WorkflowId as%
%create table #CascadeCollect(id INT IDENTITY(1,1)%
%exec sp_executesql N'if (NOT EXISTS(select 0 from PrincipalObjectAccess%
%exec sp_executesql N'select systemuser0.OrganizationId as%
%exec sp_executesql N'select privilege0.PrivilegeId as%
%exec sp_executesql N'SELECT Id, LicenseKey, ProductId, InstallOn FROM ConfigSettings%
%exec sp_executesql N'select asyncoperation0.OwningUser as %
%#CascadeCollect%
%select organization0.SystemUserId as%
%insert into WorkflowLogBase%
%select sdkmessageprocessingstepimage0.SdkMessageProcessingStepImageId%
%exec sp_executesql N'exec p_GetCrmUserId @OrganizationId, @AuthInfo'%
%SELECT Id, DefaultOrganizationId FROM Deployment%
%exec sp_executesql N'SELECT Id, UserId FROM SystemUserOrganizations%
%exec sp_executesql N'SELECT Id, AuthInfo FROM SystemUserAuthentication%
%SELECT M.ObjectId FROM dbo.MatchCode%
%exec sp_executesql N'select subscription.WorkflowWaitSubscriptionId%
%and StatusCode in (@pausingStatus, @cancelingStatus)%
%exec sp_executesql N'select timezonedefinition0.TimeZoneCode%
%exec GetMyRunningJobs%
%exec sp_executesql N'select sdkmessagefilter0%
%exec sp_executesql N'select sdkmessagerequestfield0.Name%
%KeyType, ScaleGroupId FROM CrmKeySetting%
%select report0.Name as ''name'', report0.ReportTypeCode as%
%from TimeZoneRule as timezonerule0%
%exec sp_executesql N'select queue0.QueueId%
%truncate table #SyncEntry%
%create table #SyncEntry%
%exec sp_executesql N'update Subscription set LastSyncStartedOn%
%exec sp_executesql N'select subscriptionclients%
%exec sp_executesql N'select displaystring0.DisplayStringId%
%drop table #SyncEntryIds;%
%select convert(bigint, min_active_rowversion()-1)%
%select ObjectTypeCode, SyncState,%
%insert into SubscriptionSyncInfo%
Enjoy

Tags: , , ,


Retrieve the Primary Key column for a SQL Server table

During a recent project I was dynamically scripting data from related database tables and needed to know the primary key column from a parent table to use with the child tables. I normally use the Information_Schema Views (ISV) for most metadata related queries, but there are always several ways to achieve the same results and I wanted to see if another approach might be easier. I tried the following options and ended up using the ISV solution, it was the simplest and the often the best standard approach.

  1. SystemTables
  2. Stored Procedure
  3. Information_Schema Views

Using the Northwind database the following queries will return the primary key column for the Customers table. The system table approach is shown in two queries with the second one using a CTE instead of a sub-select.

-----------------------------------------------------------
-- Using the system tables
-----------------------------------------------------------
SELECT  sc.name AS PrimaryKey
FROM    syscolumns sc
            JOIN sysobjects so ON so.id = sc.id
WHERE   so.name = 'Customers'
AND     sc.colid IN (   SELECT  sik.colid
                        FROM    sysindexkeys sik
                                    JOIN sysobjects so ON sik.[id] = so.[id]
                        WHERE   sik.indid = 1
                        AND     so.name = 'Customers')
-- CTE
-- Get the id of the primary key column
;WITH PKColumnIdCTE AS
(
    SELECT  sik.colid
    FROM    sysindexkeys sik
                JOIN sysobjects so ON sik.id = so.id
    WHERE   sik.indid = 1
    AND     so.name = 'Customers'
)
-- Get the name of the primary key column
SELECT  sc.name AS PrimaryKey
FROM    syscolumns sc
            JOIN sysobjects     so ON so.id = sc.id
            JOIN PKColumnIdCTE  pk ON pk.colid = sc.colid
WHERE   so.name = 'Customers'


-----------------------------------------------------------
-- Using the system stored procedure
-----------------------------------------------------------
-- This uses the system tables internally and returns the
-- primary key information for the table as a row.
EXEC sp_Pkeys 'Customers'

-- You can extract the key by capturing the result in a temp
-- table and querying to select the column.
DECLARE @_PKInfo TABLE
(
     DbName     VARCHAR(128)
    ,OwnerName  VARCHAR(128)
    ,TableName  VARCHAR(128)
    ,ColumnName VARCHAR(128)
    ,KeySeq     INT
    ,PkName     VARCHAR(128)
)
INSERT INTO @_PKInfo EXEC sp_Pkeys 'Customers'
SELECT ColumnName AS PrimaryKey FROM @_PKInfo


-----------------------------------------------------------
-- Using the information_schema views
-----------------------------------------------------------
SELECT  ccu.column_name AS PrimaryKey
FROM    information_schema.table_constraints tc
            JOIN information_schema.constraint_column_usage ccu
                ON tc.constraint_name = ccu.Constraint_name
WHERE   tc.constraint_type = 'Primary Key'
AND     tc.table_name = 'Customers'

Tags: , , ,


Comparing Schema Information

During a recent purge/archive project I needed to compare two tables in different SQL Server databases to see if the schemas were the same. Basically, I was moving data from a production database to an archive database and if new fields were added or changed to any of the production tables, I needed to know and sync those changes to the related archive tables. Below are two stored procedures that I created initially in the master database to simplify the process and to allow the stored procedures to be executed from any database.

The first stored procedure (sp_GetSchema) simply retrieves the column names and other related data for the specified database table using the INFORMATION_SCHEMA.COLUMNS view (Phil Haack has a great article about Using INFORMATION_SCHEMA Views to bulletproof Sql Change Scripts). It uses dynamic SQL to construct the query because the database name needs to be included. The schema information from the production and archive tables are then compared using the second stored procedure (sp_CompareSchema).

CREATE PROCEDURE sp_GetSchema
(
    @database	NVARCHAR(128)
    ,@table		NVARCHAR(128)
)
AS
SET NOCOUNT ON

DECLARE @sql    NVARCHAR(MAX)

SET @sql = '
    SELECT  column_name
            ,CASE   WHEN is_nullable = ''YES''
                    THEN ''NULL''
                    ELSE ''NOT NULL''
                    END AS is_nullable
            ,CASE   WHEN CHARINDEX(data_type, ''decimal, numeric'') > 0
                        THEN UPPER(data_type)
                            + ''(''
                            + CONVERT(NVARCHAR, numeric_precision)
                            + '', ''
                            + CONVERT(NVARCHAR, numeric_scale)
                            + '')''
                    WHEN CHARINDEX(data_type, ''varchar, nvarchar,
                            varbinary'') > 0
                            AND character_maximum_length = -1
                        THEN UPPER(data_type) + ''(MAX)''
                    WHEN CHARINDEX(data_type, ''char, nchar, varchar,
                            nvarchar, varbinary'') > 0
                        THEN UPPER(data_type)
                            + ''(''
                            + CONVERT(NVARCHAR, character_maximum_length)
                            + '')''
                        ELSE UPPER(data_type)
                    END AS data_type
            ,ordinal_position
            ,table_name
            ,''' + @database + ''' as dbname
    FROM    ' + @database + '.INFORMATION_SCHEMA.COLUMNS
    WHERE  table_name = ''' + @table + ''' '

EXEC( @sql )

SET NOCOUNT OFF

Here’s an example of executing the sp_GetSchema stored procedure and the output generated.

EXEC sp_GetSchema @database='Action', @table='Order'
column_name  is_nullable data_type ordinal_position table_name dbname
------------ ----------- --------- ---------------- ---------- ------
OrderId      NOT NULL    INT       1                Order      Action
CustomerId   NOT NULL    INT       2                Order      Action
OrderDate    NOT NULL    DATETIME  3                Order      Action
RequiredDate NULL        DATETIME  4                Order      Action
ShippedDate  NULL        DATETIME  5                Order      Action
Freight      NULL        MONEY     6                Order      Action
Version      NOT NULL    TIMESTAMP 7                Order      Action

A few things to note about the sp_CompareSchema stored procedure. It uses temporary tables to house the schema information (retrieved via sp_GetSchema) for the tables to compare because you cannot insert the result from a stored procedure into a table variable and it compares the column names, data type and nullability. I am using the characters “T”, “A” and “D” to indicate a different in type, a column to be added and a column to be deleted respectively in the schema diff table. Technically, you could move the sp_GetSchema logic into the sp_CompareSchema stored procedure, but I am using it for other purposes so I kept it separate.

CREATE PROCEDURE sp_CompareSchema
(
    @database1  NVARCHAR(128)
    ,@database2 NVARCHAR(128)
    ,@table1    NVARCHAR(128)
    ,@table2    NVARCHAR(128)
)
AS

    SET NOCOUNT ON

    DECLARE @ErrorCode      INT
    DECLARE @ErrorMsg       NVARCHAR(500)
    DECLARE @alterScript    NVARCHAR(MAX)

    SET @alterScript = ''

    -----------------------------
    -- CREATE TEMP TABLES
    -----------------------------
    IF( OBJECT_ID('#SCHEMA_T1') IS NOT NULL ) DROP TABLE #SCHEMA_T1
    IF( OBJECT_ID('#SCHEMA_T2') IS NOT NULL ) DROP TABLE #SCHEMA_T2

    -- Schema for table1
    CREATE TABLE #SCHEMA_T1
    (
        column_name     NVARCHAR(128)
        ,is_nullable    NVARCHAR(64)
        ,data_type      NVARCHAR(32)
        ,position       INT
        ,table_name     NVARCHAR(128)
        ,database_name  NVARCHAR(128)
    )
    -- Schema for table2
    CREATE TABLE #SCHEMA_T2
    (
        column_name     NVARCHAR(128)
        ,is_nullable    NVARCHAR(64)
        ,data_type      NVARCHAR(32)
        ,position       INT
        ,table_name     NVARCHAR(128)
        ,database_name  NVARCHAR(128)
    )
    -- Schema difference table
    DECLARE @SCHEMA_DIFF TABLE
    (
        id              INT IDENTITY(1, 1)
        ,database_name  NVARCHAR(128)
        ,column_name    NVARCHAR(128)
        ,is_nullable    NVARCHAR(64)
        ,data_type      NVARCHAR(32)
        ,diff_type      NVARCHAR(5)
    )

    -----------------------------
    -- POPULATE TEMP TABLES
    -----------------------------
    -- Get the fields for database1.table1 (cannot insert the result from a sproc into a table variable)
    INSERT INTO #SCHEMA_T1 (column_name, is_nullable, data_type, position, table_name, database_name)
        EXEC sp_GetSchema @database=@database1, @table=@table1

    -- Get the fields for database2.table2
    INSERT INTO #SCHEMA_T2 (column_name, is_nullable, data_type, position, table_name, database_name)
        EXEC sp_GetSchema @database=@database2, @table=@table2

    -----------------------------
    -- COMPARE THE FIELDS
    -----------------------------
    -- T - indicates a difference in Data Types...ALTER column in Target
    -- A - indicates a missing Column in the Target...ADD column to Target
    -- D - indicates a missing Column in the Source...DROP column from Target
    INSERT INTO @SCHEMA_DIFF
        SELECT  s.database_name, s.column_name, s.is_nullable, s.data_type,
                CASE WHEN (s.data_type != t.data_type) THEN 'T'
                     WHEN (s.column_name IS NOT NULL AND t.column_name IS NULL) THEN 'A' ELSE '' END AS diff_type
        FROM    #SCHEMA_T1 s
                FULL OUTER JOIN #SCHEMA_T2 t on s.column_name = t.column_name
        WHERE   s.data_type != t.data_type
        OR      (s.column_name IS NOT NULL AND t.column_name IS NULL)
        UNION ALL
        SELECT  t.database_name, t.column_name, t.is_nullable, t.data_type,
                CASE WHEN (s.column_name IS NULL AND t.column_name IS NOT NULL) THEN 'D' ELSE '' END AS diff_type
        FROM    #SCHEMA_T1 s
                FULL OUTER JOIN #SCHEMA_T2 t on s.column_name = t.column_name
        WHERE   s.column_name IS NULL AND t.column_name IS NOT NULL

    SELECT * FROM @SCHEMA_DIFF

    DROP TABLE #SCHEMA_T1
    DROP TABLE #SCHEMA_T2

    SET NOCOUNT OFF

Here’s an example of executing the sp_CompareSchema stored procedure and the output generated.

Exec sp_CompareSchema
    @database1  = 'AdventureWorks'
    ,@database2 = 'AdventureWorksLT'
    ,@table1    = 'Product'
    ,@table2    = 'Product'
id  database_name     column_name             is_nullable  data_type       diff_type
------------------------------------------------------------------------------------
1   AdventureWorks    MakeFlag                NOT NULL     BIT             A
2   AdventureWorks    FinishedGoodsFlag       NOT NULL     BIT             A
3   AdventureWorks    SafetyStockLevel        NOT NULL     SMALLINT        A
4   AdventureWorks    ReorderPoint            NOT NULL     SMALLINT        A
5   AdventureWorks    SizeUnitMeasureCode     NULL         NCHAR(3)        A
6   AdventureWorks    WeightUnitMeasureCode   NULL         NCHAR(3)        A
7   AdventureWorks    DaysToManufacture       NOT NULL     INT             A
8   AdventureWorks    ProductLine             NULL         NCHAR(2)        A
9   AdventureWorks    Class                   NULL         NCHAR(2)        A
10  AdventureWorks    Style                   NULL         NCHAR(2)        A
11  AdventureWorks    ProductSubcategoryID    NULL         INT             A
12  AdventureWorksLT  ProductCategoryID       NULL         INT             D
13  AdventureWorksLT  ThumbNailPhoto          NULL         VARBINARY(MAX)  D
14  AdventureWorksLT  ThumbnailPhotoFileName  NULL         NVARCHAR(50)    D

The data from the @SCHEMA_DIFF table could be used to build an alter script that would be applied against the archive table to sync the schema changes.

Tags: , , ,


Common Table Expression T-SQL

Common table expressions, a feature introduced by Microsoft SQL Server 2005, are a very powerful feature that may be used in lieu of temporary tables. They may be thought of as a temporary result set that is defined within the execution scope of a query. Using a common table expression you are able to create a reference to itself, which exposes a recursive ability that is very helpful for identifying a hierarchy amongst table elements. 

For our example we will use NorthWind’s employee table. This table is designed in such a way that the ReportsTo column is a foreign key field which refers to the primary key field EmployeeID. 

 

 

 WITH Managers AS
(
--Anchor Member
SELECT EmployeeID, FirstName, LastName, ReportsTo 
FROM Employees
WHERE ReportsTo is null
UNION ALL
--Recursive Member
SELECT e.employeeID,e.FirstName, e.LastName, e.ReportsTo
FROM Employees e
INNER JOIN Managers m 
ON e.ReportsTo = m.employeeID
)
SELECT * FROM Managers 

Semantics 

1. Split the Common Table Expression into anchor and recursive members. 

                1(a). The anchor is the root level node of the result set. In our example Andrew Fuller has no report to property. Thus, we declare our anchor value with ReportsTo = NULL 


SELECT EmployeeID, FirstName, LastName, ReportsTo 
FROM Employees
WHERE ReportsTo is null

 

                 1(b). The recursive member must be joined to the anchor using UNION ALL 


UNION ALL
--Recursive Member
SELECT e.employeeID,e.FirstName, e.LastName, e.ReportsTo
FROM Employees e
INNER JOIN Managers m 
ON e.ReportsTo = m.employeeID

Any rows returned are inserted into the CTE, which are also then available (seen) on the next recursion. Recursion occurs because of the query referencing the CTE itself based on the Employee in the Managers CTE as input. The recursive query is repeated until it returns an empty result set. 

2. The final step is to query the results of the Common Table Expression.


SELECT * FROM Managers

 

A drawback of this feature is that a Common Table Expression must be called immediately after definition. Any subsequent query referencing the Common Table Expression will fail.

Tags: , , , , ,


ACID Rules

A few days ago, I had to look for some information regarding the ACID rules and I didn’t really find anything relevant that helped me understand the topic very well. I even checked many SQL Books because I thought I would be able to find good information in them, but even though they were supposed to be “good books” , I still found nothing good . So I decided to put together all the information I found and a little bit about the topic.

First of all we need to understand what the letters A.C.I.D. stand for and what the idea behind it is. The A.C.I.D. letters stand for atomicity, consistency, isolation and durability. The meaning of those words will be discussed with more detail in order to have a better understanding and give us a better idea of the topic as a whole.

Atomicity

When we deal with data, we want to be sure that if one thing is happening, another thing will happen as well. In other words atomicity means that something has to happen or nothing will happen at all.

To illustrate the principle of Atomicity, imagine writing an Insert statement. If you have not written any yet, you can do it by typing something like this:

 INSERT INTO table_name
 VALUES (value1, value2, value3,...)

The statement would insert the values in the database. But let’s also imagine that the Server has a problem and at that precise moment something unexpected happened? Would we lose the information? Would the system crash cause the data to be damaged or lost? In order to follow the principle of atomicity, the database engine would never insert that information into the table. Therefore the information that the table had, before you ran the insert, would remain intact and none of the changes would have taken place. That serves the Atomicity principle.

Another option to prevent those problems and to follow the Atomicity principle correctly, the database engines have some features such as Transaction Management, Locks and Concurrency.

Consistency

This may be the most important principle of the ACID model. As we know, the data has to be consistent. Every time we run a transaction on the data base, the data may change. If that happens, the data has to transform from one consistent state of data into another consistent state of data. If something happens that breaks down the principle, the transaction has to be rolled back. How Data Base engines roll back transactions is another topic that we may explore some other day however I think it’s fair enough to mention that we can also control how the Data base server rolls back the transactions.

Let’s remember that Rollback basically goes back to the last committed transaction known by the data base. It restores that information on the rows affected by the update that failed. This behavior provides consistency and atomicity to the data.

Isolated

To better explain this term, let us imagine that we have a busy data base server receiving multiple transactions on the same table. Those transactions could be affecting one or multiple records. The ability to keep those transactions separated one from another and the data consistent is what we call Isolated. In other words, this term requires that multiple transactions occur at the same time and not impact each other’s execution. This is something that we have to take into consideration especially from the Data Base Administration point of view.

To avoid that from happening, SQL server uses a locking mechanism. This mechanism arbitrates when a process is allowed to modify data as well as insuring the read are consistent. Locks occur at three different levels and there are three different types. They can be applied in a row, page or table level.

Durability

This is the last topic of the ACID model but not less important than the others. When we talk about durability we are not necessarily saying that the information will last forever but the concept means that we have to ensure that the data will remain safe and it won’t be lost even if the server crashes. In order to prevent loss of data, we have to keep backups and transaction logs. They will facilitate the data restoration.

There is no doubt that all the other terms ensure the durability of the data, however we haven’t thought about external problems and situations that the Data base can’t control, such as server crashes or unexpected shut downs. In order to prevent those problems, the data bases provide many different tools to provide durability. The most common way to do this is by doing backups.

There are many different solutions we can currently find. It really doesn’t matter what you implement, the important thing is that you implement some kind of backup.

Tags: , , , , , , , , , ,


Disabling SQL Server Constraints

Adding and removing constraints from an existing SQL Server database table is performed using the ALTER TABLE statement. The article Using Microsoft SQL Server Constraints provides more information on constraints and some good examples and Books Online has specific details, if needed.

The following statement is an example of adding a primary key constraint to the Account table:

ALTER TABLE [Account] ADD CONSTRAINT [PK_Account] PRIMARY KEY( AccountId )

A common issue that I’ve run into is not adding/removing constraints, but rather needing to disable constraints when deleting data or importing data from related tables, including the primary keys. Disabling constraints is a two-step process because the constraints must also be re-enabled. For example, the following statement disables the ‘FK_Account’ Foreign Key constraint in the Account table.  The keyword “NOCHECK” disables the constraint and “CHECK” enables it.

ALTER TABLE [Account] NOCHECK CONSTRAINT [FK_Account]

Issuing a statement or two isn’t bad, but does require that one knows the name of the constraint to disable. What if you had to deal with five, ten, fifteen or more tables? I needed an automated way of identifying the constraints and creating the ALTER statements based on the constraint type.

Using the INFORMATION_SCHEMA.TABLE_CONSTRAINTS view I was able to automate the process of creating and executing the statements using a variation of the script below. This script will list ALL the Foreign Key constraints and the corresponding table for all tables in the current database. The script can be filtered as needed, but for my purposes, all tables were needed.

SELECT  constraint_name, table_name
FROM    INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE   constraint_type = 'FOREIGN KEY'
ORDER   BY table_name

The above script was converted to the stored procedure below to disable/enable all the constraints of a specific type for all tables.

CREATE PROCEDURE [dbo].[ToggleConstraints]
(
    @type    VARCHAR(128) = 'FOREIGN KEY'
    ,@enable BIT = 1
)
AS
SET NOCOUNT ON

DECLARE @__CONSTRAINTS TABLE(Id INT IDENTITY(1,1), ConstraintName VARCHAR(128), TableName VARCHAR(128))
DECLARE @rows            INT
        ,@index          INT
        ,@sql            NVARCHAR(2000)
        ,@tableName      VARCHAR(128)
        ,@constraintName VARCHAR(128)

-- Retrieve the constraints using the INFORMATION SCHEMA View and place into
-- a table variable based on the specified type. Primary Key and Unique Key
-- constraints are excluded by default to prevent duplicate data from being
-- inserted, but can also be included by changing the constraint_type.

INSERT INTO @__CONSTRAINTS(ConstraintName, TableName)
SELECT constraint_name, table_name
FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS
WHERE constraint_type = @type
ORDER BY table_name

-- Set defaults
SELECT @rows = @@ROWCOUNT, @index = 1

-- Generate and execute the sql script
WHILE( @index <= @rows )
    BEGIN
        SELECT  @tableName = TableName, @constraintName = ConstraintName
        FROM    @__CONSTRAINTS
        WHERE   Id = @index

        -- Toggle the constraint based on the @enable parameter
        SET @sql = 'ALTER TABLE [' + @tableName + '] ' + CASE WHEN @enable = 0 THEN 'NOCHECK' ELSE 'CHECK' END + ' CONSTRAINT [' + @constraintName + ']'

        PRINT @sql

        -- EXEC sp_executesql @sql

        SET @index = @index + 1
    END

SET NOCOUNT OFF

Simply uncomment the EXEC line if you want to actually run the generated statements.