Hamilton SharePoint User Group

Thursday, Feb 16 2012

SQL Server Reporting Services with SharePoint SQL Server Reporting Services provides a rich reporting environment, and integration with SharePoint makes it seem seamless to end users. How does Reporting Services impact the SharePoint environment? This presentation will walk through the basic features of Reporting Services, and architectural considerations when installing in a SharePoint farm. In addition, some of the differences included in the upcoming SQL Server 2012 version of Reporting Services will be discussed.

Toronto SharePoint User Group

Wednesday, April 18, 2012

What’s new in SQL Server 2012 for SharePoint SQL Server 2012 brings a wealth of new features to the core database used by SharePoint. However, it also brings a number of exciting new Business Intelligence features right to your SharePoint users. This session will walk through a number of the new features that have a direct impact on SharePoint administrators, designers and end users, with in depth demonstrations of how to configure and use them. These features include significant architectural changes to Reporting services, the new BISM or tabular engine for Analysis Services and PowerPivot, and the new end user focused reporting tool, PowerView.

SharePoint Summit

May 15, 2012

Reporting for Duty – Best Practices for Reporting Services with SharePoint   Reporting Services and SharePoint have been working together since SharePoint 2003. SQL Server 2005 SP1 brought the ability to use Reporting Services in Integrated Mode through a SharePoint Add in, and with SQL Server 2012, Reporting Services is a fully fledged SharePoint Service Application, and some features, the new PowerView in particular, are only available through SharePoint Integrated Mode. Reporting Services bring a wealth of benefits to your SharePoint farm, but can also have a significant impact on it. This session will discuss the do’s and don’ts for a successful Reporting Services implementation. It will cover architectural considerations through to Report design, for both the Reporting Services Add In (SQL Server 2005-2008) and the new Reporting Services Service Application (SQL Server 2012).

My only complaint about the resort was that it was a bit far from Playa Del Carmen, and when in the area, I always dive with the Abyss Dive Center – I originally certified there, and have done all of my certifications there since. Also diving with me this time was my son John, and our good friends Allyson and Rich.

Around December-March every year, bull sharks can be found right off the beach in Playa Del Carmen, and last year, I had hoped to see a few of them. Alas, at the time, they had left the area for some inexplicable reason, only to return after I had come home. This year it was different. This time I got to dive with them twice, and each time there were between 10-20 checking us out. One morning we saw 5 at one of the regular dive sites. All of my diving pictures for the trip can be found here but just to highlight a few…

To celebrate my reconfirmation as MVP on Jan 1, I had my son take the shot of me below

A few other shots of the attractions in question:

In addition to the photos, I wanted to try out the video capabilities of my Canon T2i. From the footage, I was able to put this video together:

In addition to the sharks, on our last diving day, we headed inland to the cenotes. For those of you that don’t know, cenotes are underground rivers that only exist in the Yucatan peninsula, and you can dive in them. Since there are frequent openings into air, it’s not quite cave diving, but it’s close, and on a nice sunny day, like the one we had, the light show can be spectacular.

That pretty much wraps up the fun for this year – now it’s back to work. I have a mountain in front of me and a lot of writing to do to!

Happy New Year

The most common scenario that will be encountered, given the vintages of the products will be an RS upgrade from SQL Server 2005 to 2008 R2. In addition, the in place upgrade is relatively painless (in the short term….) so I’ll be walking through a DB attach upgrade, which is just as applicable to RS as it is to SharePoint. Finally as I’ve written about previously, in a small farm, it’s likely a better idea to add the Reporting Services bits to a SharePoint front end server, than to add the SQL server to the SharePoint farm, and that will also be a part of our scenario.

The first question to answer is “why bother”? One of the advantages to using RS in SharePoint Integrated mode is that unlike Native mode, the reports, data connections and models are stored directly in SharePoint. It is therefore possible to just create a new RS database, and move forward. However, since subscriptions, schedules, and cache profiles are still in the database, it’s likely worth it to do the upgrade.

Step 1 – Back Up The Asymmetric Key

Reporting Services itself uses 2 SQL databases. One of the databases is for temporary operations, but the other database stores a number of important, and sensitive items for this reason, all sensitive items in the database are encrypted with a key. If we want to get access to these items, we need the key. To do so, we need to back it up from the source server before we move ahead.

Run the Reporting Services Configuration Manager on the source RS server, and select “Encryption Keys”. Click the Backup button, select (and remember) a password, and then save the key to the file system. 

Once saved, copy the key file to the destination RS server (likely your SharePoint 2010 front end server).

Step 2 – Back Up the Reporting Services Databases

Run SQL Server Management Studio on the Server where the Reporting Services databases are located. Run full backups of the two RS databases. When complete, copy the backups to the destination SQL server (likely the server that will host the SharePoint 2010 databases).

Step 3 – Restore the Reporting Services Databases

Using SSMS, restore the two databases to the host SQL server. Once restored, it’s likely a good idea to set the recovery model to Simple, and the Compatibility level to SQL Server 2008. These steps aren’t required, but are recommended, unless you have a reason for not doing so.

Step 4 – Run the RS Configuration

If Reporting Services hasn’t yet been installed on the SharePoint server, do so, otherwise, proceed to configuration by running the Reporting Services Configuration Wizard on the destination RS Server. Configure the basic steps, and then when it comes to Database configuration, select the option to choose an existing database.

Select the server where you restored the files in step 3, and select the primary RS database (the one without the word “Temp” in it”).

Complete the configuration wizard.

Step 5 – Connect SharePoint to Reporting Services

From SharePoint Central Administration, select General Application Settings, and then Reporting Services Integration.

Complete  the integration configuration, and then select OK

So far so good – now we’re ready for some gotchas. If you now click on the “Set Server Defaults” link in the Reporting Services section, you likely get a rather nasty looking error. You’ll also experience this error if you access ewither of the two RS URLs defined in the RS configuration wizard. The error is:

The report server installation is not initialized. (rsReportServerNotActivated)

This error happens when the server can’t access configuration information, and the most common cause of that is that it can’t decrypt the content. In our case, it can’t because we haven’t yet restored our key.

Step 6 – Restore the Asymmetric Key

On our new RS server, we need to run the RS Configuration Manager, Select Encryption Keys, and then click the Restore button. You will be prompted for the file that you created and copied in Step 1, and this is where remembering the password comes in very handy.

Once this is done, we can close the configuration manager and return to Central Administration. However, now when we try to access access any aspect of RS we get a new error:

The feature: “Scale-out deployment” is not supported in this edition of Reporting Services. (rsOperationNotSupported)

The reason for this error is that when we restored the key, it added an entry in the Keys table in the Reporting Services database, causing RS to think that we’re using multiple Reporting Services servers. This is what’s known as a scale-out deployment, and is only supported in the Enterprise version of Reporting Services. Obviously this isn’t a problem for anyone running Enterprise, but if not, it’s a showstopper.

The way to fix this is to remove the old server entry in the Keys table. Using SQL Server Management Studio, connect to the Reporting Services database, and open the dbo.Keys table. The old entry should be easy to spot as it will have the old server name. Simple delete the row.

Once the offending entry is deleted, RS should be good to go.

7. Fix up the content type names

I have posted about this already, but often, an upgrade will break the Content Type names for the Reporting Services content types. Just follow the steps in this post to clean them up.

8. Reconnect Reports with Data Sources And/Or Republish

In addition, moving connection files and reports around in SharePoint can cause them to be disconnected from each other, or for the connection files to be disabled. It’s a good idea to navigate to all of your reports to make sure that they are connected, or better yet, to republish from the source if you had previously used BIDS to publish reports.

Given that we are using Office 365, I checked in the community hub and found this post that indicated that there were were other users experiencing the same problem, but in their case, it was related to the fact that they were using the default “onmicrosoft.com” domain. However, there were some DNS instructions in the post that gave me a clue. Unfortunately, once I added the indicated DNS entry, I still had the same problem.

I decided to check the DNS settings for my domain in the Administration panel. and as it turns out, the DNS instructions have been updated since I had originally set up our Office 365. There are two CNAME entries that are required for the WP7 Lync application to work with Office 365:

Obviously, yourname.com should be replaced with your actual domain name.

Once I set these up, I was off to the races with my mobile Lync client.

The names displayed were the internal names that are used to support multilingual installations. What I believe caused this problem was the the content database had been upgraded to 2010 prior to the add in being installed on the farm, so the English names could not be resolved.

All existing items using the types continued to work, and reports could still be deployed using Business Intelligence Development Studio (BIDS). However some UI dependent functions, like Report Builder could not be used without the proper names. Luckily, the fix is relatively straightforward. All that we need to do is to set the correct names manually. To get the correct farm, all that is necessary is to inspect an unaffected farm to see the section below.

In order to fix this, you need to first navigate to the root of your site collection, and then select Site Actions – Site Settings, and then click on Site content types in the Galleries section.

At this point, you should see the affected section, it will be right at the top. We need to fix the content types one at a time, so click on one of the affected content types. The Content type editor window will appear.  From that Window, simply select Name, description, and group.

You will then be able to edit the values for Name, Description, and Group.

For the first one that you edit, you’ll need to select the New Group option, and enter the correct name of the group, as indicated above. For the other two content types, you’ll be able to select the correct group from the existing group list.

Once you’ve edited all three content types, that’s it! You’re done.

Well, sort of…

Unfortunately the name change will only affect new libraries , or newly configured libraries. For libraries that already have these content types attached to them, you’ll need to repeat the process for each library that has been affected. I imagine that it’s quite possible to write a PowerShell script that will do this for all libraries, but so far, the remediation effort hasn’t warranted it.

A sparkline is essentially a mini-graph – a visual representation of a single row of data. In my case, it is the results, by party (as indicated by colour) for a given poll. It looks like this:

Once I got this report looking the way that I wanted it to, I deployed it to Azure Reporting Services. However, when I ran the report, it took an incredibly long time to load (3 minutes and 30 seconds). I immediately blamed Azure RS for this, as it’s still a preview edition. However, further testing revealed the same behaviour on an on-premise deployment. The puzzling thing was that it rendered very quickly using the preview in BIDS or in Report Builder.

Adding to the mystery is the fact that the report renders fairly quickly when called from the Reporting Service web service to create a PDF file, or even an HTML file. I managed to discover this when I decided to pre-create a number of these reports in PDF format to reducre the load time for users (more on how I did this in an upcoming post).

The oddest part is precisely where the performance problem shows up. When the report is run using a browser, the browser thread’s CPU utilization goes up to 100% of available resources (a 2 core machine CPU runs up to 50%). This is happening on the client side.

On a hunch, I tried removing the sparklines. Presto, the load time dropped to 20 seconds. So I don’t get to use my sparklines in the live report which is unfortunate, but at least I found the culprit. What I wish I knew was why this was happening, or if there’s something I’m simply doing wrong.

If anyone has any ideas, I’m all ears.

The Render method of the ReportExecution2005 web service would appear something like below:

report = rs.Render("EXCEL", deviceinfo, _  extension, mimeType, encoding, warnings, streamIDs)

However, if you are using the Reporting Services web service, nothing changes. The Render method of the ReportExecution2005 web service would appear something like below:

The first argument is the file format to render into. With SSRS, the above code will render into the 2003 file format (.xls). The web service defaults to the old format.

I was unable to find any documentation at all on the correct argument to get the new file formats, so I poked around a little with my debugger and the ListRenderingExtensions method. As a result, I thought I’d share what I think is the comprehensive list of rendering formats for Reporting Services in SQL Server 2012. These are taken straight from the code.

Therefore, if I want to render to the description “Excel” the argument needs to be “EXCELOPENXML”, and to render to the description “Excel 2003″ the argument is “EXCEL”. That’s not at all confusing.

In any event, hopefully this helps someone (likely me in the future..)

Yesterday, as part of the ongoing Panellist Spotlight series for SharePoint Shoptalk, I presented a session on how to move your data from SharePoint, and in to a data warehouse, and then consume the warehoused data directly within SharePoint. These presentations are recorded, and posted online, and if you’re interested, you can view the entire presentation below:

In it, I demonstrate how SQL Server Integration Services (SSIS) can be used to extract the data from SharePoint, and to store it in a SQL Server data warehouse. Then I walk through the creation of an external content type, and an external list using SharePoint Designer and SharePoint Business Connectivity Services (BCS). Finally, I create a report using SQL Server Reporting Services (SSRS) in SharePoint Integrated mode.

Thanks to Jamees Wright for organizing the session, and to my fellow panellist, Laura Rogers.

My company is small (currently 6 people) and widely distributed. Cloud solutions make perfect sense to us,and we have been using Exchange Online for over 2 years now. Our requirements for SharePoint went beyond what was possible in BPOS’ offering, but since migrating to Office 365 6 months ago, the  new SharePoint online fits the bill, and more and more of our corporate assets live there now.

UnlimitedViz is currently primarily a SharePoint services company focused on Business Intelligence, and a significant portion of those services involve architecting SharePoint environments at a lower level, which involves sizing servers, making resource decisions, etc. I personally love designing solutions and watching them come to life. We are certainly more than capable to maintain our own SharePoint infrastructure, so why would we want to use an admittedly more limited version of the product that is maintained by someone else?

Pretty much because it’s maintained by someone else.

As mentioned above we’re small, and we need to be focused on what we do best, which is providing services to our customers, and building product. Maintaining internal systems, no matter how good we are at it, is a distraction, and a significant cost, both capital and operational. The per user cost of Office 365 is pretty simple to justify from just a cost standpoint, but there are many more benefits that are brought to the table.

No matter what the location of a team member, they can easily access what they need to. Lync brings that down to the voice and IM communication level. No need to mess around with access methods, VPNs, Firewalls, Reverse Proxy servers and the like. We can get to our content easily on site, at home via whatever device we happen to need. Granted, I could set that stuff up on-premise, but now I don’t have to! I also know that my data is safe, and the performance is going to be good. Two months ago, Exchange online suffered an outage for about two hours (the only hiccup I’ve experienced so far). My initial reaction was “what can I do to fix this”, but that was quickly superseded by  “It’s not my problem to fix”, so I just sat back and got other work done.

As we bring more customers onto Office 365, supporting them just gets simpler. A simple client request can be acted upon immediately by launching a browser window, and connecting to their site, seamlessly. With most onsite installations, I need to start a virtual machine, connect through a VPN client, and then hope that the correct tools are installed on the VM, or the client site, depending on the access mechanism. I try to keep a VM image available for every type of VPN client used, which is a hopeless and necessary task due to the incompatibilities between clients. In my opinion, the world will be a better place when VPN clients are eliminated (or at least consolidated”).

Customers using Office 365 don’t need VPN clients, and it makes it that much easier (and cheaper for them) for us to support them.

There a a whole bunch of great features about Office 365 (Shared OneNote files accessed via Windows Phone, browser and client is a good one, not to mention Lync), but the reason that I really like it is that it’s solid, it works, and it lets my business focus on using its tools, not maintaining them.

Since originally posting this, the content provider has added the cookie requirement for download of the files. This is in line with Microsoft’s wishes that the videos be downloaded by conference attendees only. In order to batch download the videos, you will need to follow the procedure outlined for the PPTX files below. The addresses of the video files are still the same – only the requirement for an authentication cookie has changed. Thanks to George Winters for pointing this out.

UPDATE 11/10/2011 – The Files Have Moved.

I have updated the post below accordingly. Also note the file names are also different. The new source is significantly faster – thank you MSFT!

I’ve seen a number of questions about this, and I’m in the process of collecting all of the presentations and slide decks from the conference, so I thought that I’d share my process. It’s relatively Straightforward to batch these up.

The videos are all in one location, and the presentations are in another. The base folder for the videos is (now) http://cdn.tri-digital.com/spc/videos/ and the naming format for the files is SPCsessionnumber.wmv.zip, so for session 301, the filename is SPC301.zip. You can simply punch the full URL into a browser and the files will start downloading. In order to batch download a bunch, I am using the open source WGET utility, recommended to me by Ziad Wakim.  I simply created a batch file that calls WGET for all of the presentations that I wanted. Each line in the batch file looks like:

"C:\Program Files (x86)\GnuWin32\bin\wget.exe" http://cdn.tri-digital.com/spc/videos/SPC261.wmv.zip

The process is the same for the slide decks, but the site base is  http://www.mssharepointconference.com/SessionPresentations and the file name format is identical to the videos with the .zip replaced with .pptx. However, the conference site expects a cookie, so you’ll first need to log into the myspc site. Once done, export the cookie for the site to a text file (In IE use File – Import and Export – Export to a file) that’s available to the WGET command. Once it’s a simple command switch on the end of the WGET command to load in the cookie for each request. If the cookie is exported to a file names cookies.txt, then the command will look something like:

wget http://www.mssharepointconference.com/SessionPresentations/SPC413.pptx  –load-cookies cookies.txt

Another batch file with a line for each slide deck you want, and you can run it and download everything at once. One note – you’ll need to be patient. Given the speeds, I don’t think that I’m the only one who has thought of this….

Hope it helps.

In all of the sessions mentioned above, reference was made to add ins, extensions, or other tools that make working with SharePoint and Azure a great deal easier. I took note of most of them, and in the process of summarizing them, thought that I should amalgamate them with my own current list of dev tools, and post it here. Extensions can and should be installed via the extensions manager in Visual Studio, and I’ll note them below.

*Available through the Visual Studio Extension Manager

There conference was sold out, and while I never got an official number of attendees, the numbers that I heard ranged from 7,000 to 8,000. This is the attendance for a conference for a single product, which is really quite an achievement. In fact, at the keynote, Jared Spataro announced that SharePoint had sold 125 million licences to 65,000 customers making it the fastest growing product in Microsoft’s history. If SharePoint were a separate company, it would be in the top 50 software companies worldwide.

The keynote contained a single new announcement – that Business Connectivity Services will be available in Office365 before the end of the year. This is great news for my organization given that we specialize in both Office 365 and in Business Intelligence, but then, I may have been aware of this previously.

The demo highlight of the keynote was when Richard Riley demonstrated a 10 TB SharePoint farm with a single content database experience a SQL Server failure and fail over to a hot standby. This was a very impressive demonstration, but it was really just a demo of the new “Always On” features in the upcoming SQL Server “Denali”. I also feel the need to point out that just because you can have a 10 GB content database, it doesn’t mean that you should. As my friend Ziad Wakim whispered to me during the demo “try backing THAT up!”.

The sessions were the real meat of the show. In fact, I only skipped one time slot. In choosing my sessions, I was looking for 3 things – to fill in any gaps, or to confirm my field practices or to pick up demonstration tips, to see how Denali was being positioned within SharePoint, and to see any new development patterns and practices, particularly around Office365 and Azure. I didn’t get to everything that I  wanted to, but I wanted to acknowledge the ones that I did.

On the development front, Andrew Connell laid out what I think will be the definitive design pattern for working with external data in Office365 and Azure. Fellow Critical Path Training founder Ted Pattison drove home the UI patterns in his talk on using jQuery and HTML5 with SharePoint. Between them, they mentioned a number of useful tools and add ins which I’ll document shortly in an upcoming post. As always, both presenters were highly entertaining.

On the BI front, I attended sessions by Carolyn Chau (Reporting Services), Kevin Donovan (Analysis Services), Peter Myers (PowerPivot), and John Hancock (PerformancePoint and PowerPivot). They outlined the BI features of SQL Server Denali that specifically pertain to SharePoint, in particular project Crescent, and the new version of PowerPivot. PowerPivot is now more than just an analysis tool, it’s also a designer for Analysis Services. Project Crescent is a power user tool meant to support ad hoc reporting. In fact, in my opinion, Crescent has the same relationship to Reporting Services that the current version of PowerPivot has to Analysis Services. In Carolyn’s last session she also noted that Denali would be released in Q1 2012 – we now have a “date”!

In the “best practices” category (I hate that term….) there were a number of standouts. Darvish  Shadravan and Asif Rehmani gave some great insights into Infopath. What many don’t understand is how important Infopath is to SharePoint, particularly to Office365 where coding is significantly more problematic. Darvish was also monitoring a Twitter feed during the session to hand out swag, which I thought was a great idea. I’ll be using something similar for my next presentation for feedback at least. Asif was clear and concise as usual, and both sessions were highly entertaining.

Eric Shupps led an early morning session on performance tuning, an increasingly important topic for SharePoint farms. I took quite a few notes. I attended the session to confirm my deployment approaches, which I did, but also came away with a laundry list of items to add. Entertaining, and a very good use of my time.

The king of the User Profile service, Spencer Harbar delivered a clear and concise talk on the User Profile service. I follow Spencer’s blog quite closely, so many of the points that he made I was already aware of. However, I did pick up one technical tidbit that I was previously unaware of. The service requires that the synchronization account be granted the “Replicating Directory Changes” permission in Active Directory. This at times terrifies AD administrators, and it turns out that the name is bad. All that the permission does is to allow the account to read a change log. Spencer also made an excellent point in that the deployment of a user directory is 90% political and only 10% technical.

Finally, Christina Wheeler wrapped up the session with a spirited, if challenged talk on packaging Branding elements with Visual Studio. I have done extensive work with both branding and with solution packaging, but oddly, have never put the two together. I certainly will in the future. Christina was badly ill with bronchitis, and had to battle a possessed projection system, but did so without ever losing her sense of humour. I didn’t know Christina before attending, but will certainly seek her out in the future.

The best part of SPC is of course the networking events, of which there were several every night. Even though the big Disney night was a letdown in my opinion, the people more than made up for it. I left the conference knowing much more than I did going in, rekindled some (very!) old friendships and started a few new ones. Overall, a great conference.

Now the countdown begins to SPC 2012!!!!

Back to Vegas (where it should be!) and only a year from now. Does that mean that 2012 will bring big things to the world of SharePoint? I think that you can bet on it.

Given that the last two conferences have used the same url, I would recommend keeping an eye on http://www.mssharepointconference.com for any updates.

What may be less commonly known is how this works. This history list is really just a view of items that are contained in a hidden list on the site, and whenever an item is logged to the list, it gets created in the history list. For regular SharePoint workflows, this list is located at http://yoursiteurl/Lists/Workflow History, and for For Nintex workflows, you can find it at http://yoursiteurl/Lists/NintexWorkflowHistory. So, why does this matter? Well, if you need to audit what is has happened with your workflows, this is where the information is contained.

There is, however a catch. By default, SharePoint will run a Timer job named “Workflow Auto Cleanup” daily that will remove all of the the tasks associated with a workflow, and all of the history links for workflows that are over 60 days old. This is done for performance reasons. Well, unless your audit requirements only go back two months, this isn’t going to work for you.

Try doing a search for “Workflow History” and you’ll see that this has caused a number of issues (especially for those that have found it out after the fact. The good news for those people is that the workflow history list isn’t actually purged (which is also bad news, as we’ll see shortly), and those links can be recreated through reporting. However, the most common guidance found on this topic is to simply disable the automatic cleanup job, as outlined in this very poorly named Technet article.

The problem with disabling the job is that performance will suffer, potentially badly. Assume that we have an approval workflow that runs on a list that will receive 2500 approvals annually. This is a reasonably  sized list (for SharePoint 2010). Now lets also assume that during the life of the workflow, 10 items get logged to the history list. This means that in a given year, 25,000 items are being logged to the history list, which is beyond the default list view threshold of administrators, and would be considered a very large list.

What is needed is a way to balance the auditing requirements with the list size constraints of SharePoint. 25,000 items may be a large SharePoint list, but it’s trivial to a relational database like SQL. What the remainder of this article will do is to discuss how to use Microsoft’s Business Intelligence tools to extract workflow history data into a data warehouse, and then safely purge it from the workflow history list. This will be a lengthy one.

Step 1 – Extract And Load

In my opinion, one of the most underutilized tools in Microsoft’s arsenal is SQL Server Integration Services (SSIS). Almost every SharePoint site has it, and very few know about it. It is Microsoft’s ETL (Extract, Transform, and Load) tool, and it is used for taking data from source systems, performing operations on it, and loading it into a destination system, which is typically a data warehouse in the form of one or more SQL databases. This is precisely what we need to do with our Workflow History data. You can read more about SSIS here.

The problem however is that SSIS does not support SharePoint list data as a data source. Yes, ultimately all SharePoint data is stored in SQL content databases, but we all know that we’re supposed to stay out of there. SharePoint data should only be accessed via UI constructs, the SharePoint APIs, or the SharePoint web services.

Happily, a Codeplex project was created several years ago that adds both source and destination SSIS adapters for SharePoint list data, and yes, it works well with SharePoint 2007 data. What this project does is to encapsulate calls to the SharePoint web services into SSIS data adapters. Because it uses the SharePoint web services (not the API), there is no requirement for the SharePoint bits to be installed where it is being run.

You can find an excellent tutorial on how to set this up here, so I’m not going to go into any details on that. I will however cover the basic steps below. First, however I want to outline the logic involved.

What we want to be able to do is to maintain a complete log of workflow history. We also want to be able to keep the history in SharePoint for a period of time (60 days by default), and then be able to purge it, knowing that it’s secure in the data warehouse. Therefore, we need to take an initial dump of the data, and then be able to add only new items to it. The design of the data warehouse will also support multiple site history lists.

The solution will consist of 2 tables in a SQL data warehouse (a staging table and the actual archive table). The SSIS package will perform the following steps:

1. Empty the staging table
2. Extract the entire Workflow History List (SP) into the staging table (SQL)
3. Query the archive table for the most recent entry
4. Extract all items from the staging table more recent than the entry in step 3 into the archive table, and add in a site identifier.

First, open up Business Intelligence Development Studio (BIDS). BIDS is really just Visual Studio with all of the SQL BI project types added, and is normally installed when SQL is installed. If not, you can install it from the SQL media. You do not need SQL server installed to use it, but it does have some advantages.

From the Business Intelligence Projects section, select “Integration Services Project”, and give it a solution and project name. You’ll then be presented with the SSIS design canvas. The first thing that you’ll want to do is to create two connection managers – one for SharePoint, and one for SQL. In the Connection Managers pane right click anywhere in the window and select “New Connection”

Scroll down, and select SPCRED – Connection manager for SharePoint connections, give it a name, and select the credentials. If you use the credentials of the executing process, it will use your credentials when you test it, but the credentials of the SQL Server Agent process if you schedule it to run automatically. Alternatively, you can enter the credentials of a proxy account, which is what I typically do. Repeat this process, only this time select OLEDB and configure the connection to your SQL Data Warehouse database (if you haven’t already done so, you’ll need to create a SQL database to house the archive).

Next, from the Toolbox, drag a Data Flow Task onto the Design surface. Your surface should look something like below:

Double click on the Data Flow task, and the Data Flow window will open (you can also click on the Data Flow Tab). Here, from the toolbox, drag a SharePoint List Source, and an OLE DB Destination task onto the surface. Double click on the SharePoint List source, then click in the area to the right of the SharePoint Credential Connection, and set the Connection manager to the manager that you created above.

Next, click on the Component properties tab, and enter valid values for the SharePoint source site URL, and the list name. The List name will either be WorkflowHistory for standard SharePoint workflows, or NintexWorkflowHistory, for Nintex workflows.

Click OK. Next, grab the green arrow at the bottom of the SharePoint List source, and connect it to the OLE DB Destination. Double click on the OLE DB Destination, and select the New button beside the Name of the table field. What this allows us to do is to create our Temp Table in the Data Warehouse with the appropriate schema for our Workflow History List. Once the create table widow is open, simple change the name of the table to what you want (in this case wfhStaging).

As soon as you click OK, the table is created in SQL. Next, click the Mappings tab on the left, and confirm that all of the fields are mapped correctly from the SharePoint list, to the SQL table. No changes should be required. When complete, click OK, and the data flow is ready for testing. From the BIDS debug menu, select Start debugging. After a pause, the process will run, and the boxes will turn  yellow and green as the process executes. If all works properly, you will see something like the screen below:

Both boxes green indicate that the process completed successfully, and there will be an indicator showing the number of rows that were transferred. You can confirm this by opening up SQL Server Management Studio, selecting your Data Warehouse database and running the following query:

SELECT Count(id) From wfhStaging

At this point, we need to stop our debugging process and switch back to the Control Flow tab. Given that we want to repopulate the staging table whenever we run this package, we need to first clear it at the beginning of the run. Drag an “Execute SQL Task” from the toolbox onto the design surface above or to the left of the data flow task. Us its arrow to connect it to the Data Flow task, and then double click on it. Select your OLE DB connection as its connection property and enter the following SQL (substituting your table name) as its SQL Statement:

TRUNCATE TABLE wfhStaging

Next, we will need to create and populate our actual archive table. To do this, drag another Data Flow task onto the design surface. Connect the output from the first data flow task to it and then double click on it. Drag an OLEDB Source, a derived column, and an OLEDB destination onto the design surface.

We want to be able to store the workflow history for multiple sites in the same data warehouse table. To do this, we need to add another identifier column to the schema of the workflow history list that will uniquely identify the source site. In our case we will use the relative site URL. The derived column action will add this column to each row as it is processed.

Configure the OLEDB Source to read from the Staging table. Then, connect the OLEDB Source to the Derived Column action with the green arrow. Double click on the derived column action. Under the Derived Column Name, enter the name of the new column. Leave the Derived Column action as “add as new column”, and for the expression we will simply use a literal string for the site relative site URL. When complete, the action should appear as below.

Click OK to close the dialog, and then connect the derived column action to the OLEDB Destination action with the green arrow. Double click the OLEDB Destination action and repeat the steps taken above to create the staging table, only this time, you’ll create the actual archive table. This time, when you click on the Mappings tab, note that the SiteURL column has been added at the bottom. Don’t run debug at this point, as it will run the entire package. Click back on the Control Flow tab,  right click on the new Data Flow action, and select Execute Task. Just that task will run, and if you move back to the Data Flow tab, you should see that the same number of rows have been added to the archive table.

Now we need to ensure that only the new columns from the staging table are moved into the archive table. To do this, we will change the select statement in the OLEDB source of the second data flow task. Firstly, we’ll need to know what the date/time latest record in the archive table for this site. The SQL statement for this looks like

SELECT MAX(Modified) From wfhArchive
Where SiteURL='/SalespersonChangeRequest'

So therefore, we can embed that statement into the select statement for our staging table. However, we still need to accommodate the case where there are no records in the archive table, where the above statement returns a NULL value. To deal with this, we can use the ISNULL TSQL function, and our complete staging table select statement will be

SELECT * FROM wfhStaging
Where Modified >
  ISNULL(
      (SELECT MAX(Modified) From wfhArchive
       Where SiteURL='/SalespersonChangeRequest'
       ),
       '1900-01-01')

Translated into English, this basically says “Find the value for modified of the most recent record of any items with SiteURL set to /SalesPersonChangeRequest. If you don’t find any, set it to 1900-01-01. Then, get me everything from the staging table with a modified date more recent.”

Now that we have our SQL, we need to modify our OLEDB Source action. Double click on it, and then change the Data access mode from “Table or view” to “SQL command”. Then, add the select statement to the SQL command text window. At completion, the window should appear as follows:

Once done click back to the Control Flow tab, and then start Debugging (you can also just press the F5 key to start debug). The first Data Flow task should write all of the source records to the table, and the second should write none (assuming nothing has happened to the source since you did the initial extract. You can try deleting some of the records from the archive table, and rerunning the package – they should get replaced. That was step 1.

Step 2 – Schedule the package

Now that we have our package, we want it to run periodically (usually nightly). We do this by deploying the package to the server, and then scheduling to run with the SQL job agent.

To deploy the package, we need to first create a deployment utility for it. To do this, we must first select the Project in the Solution Explorer pane, and then select Project-Properties from the menu. The Configuration Properties window is then opened. In the left pane, select the Deployment Utility tab, and ensure that the CreateDeploymentUtility is set to True.

Also – take note of the DeploymentOutputPath value.

run the deployment utility for it. The deployment utility is  stored in a subfolder of the package project. You can find the folder of the project by selecting the project in the Solution Explorer pane, and then examining its FullPath  property in the Properties pane. Open the project path in Windows Explorer, and then navigate to the DeploymentOutputPath as noted above. In that folder, you’ll find a file named yourprojectname.SSISDeploymentManifest. When ready to deploy, double click on it, and the Deployment wizard will start.

The deployment wizard is straightforward and self explanatory. You’ll want to select “SQL Server deployment” on the first screen, then the SQL server that you wish to deploy to (usually (local) ), and select a location for the Package Path (the root is likely fine). Once the wizard is complete, you are ready to schedule the package.

Open Up SQL Server Management Studio, and connect the destination server. If the SQL Server Agent service is not running, start it (you’ll want to make sure that it is started automatically). Expand the Agent node, and then expand the Jobs node. Right click on Jobs, and select New Job.

Give the Job a name, then click on the Steps tab. Click the New button to create a new step, and give it a name. In the Type dropdown, select “SQL Server Integration Services Package”. In the General section, select the SQL server that holds the package, and then use the ellipsis in the Package field to select the package you deployed above.

Next, click the Schedules tab, click the new button, give the schedule a name, and select when you want the job to run. Save the schedule, and then save the job (click OK). Your job should now appear in the Jobs folder. To test it, right click on it and select “Start Job at step”. The job will run and you will see its progress in a dialog.

There are many options for scheduling SSIS jobs, and for error handling, and I would strongly recommend investigating them.

Step 3 – Purge the Workflow History Data

As mentioned above, the workflow cleanup job removes workflow history associations, but does not actually delete the items from the list, allowing that list to grow large. If you use Nintex, there’s a Nintex command that will take care of this for you:

NWAdmin.exe –o PurgeHistoryListData  -siteUrl urlToSite
 [-workflowName workflowName] [-days daysSinceLastActivity]
 [-lastActivityBefore datetime DateFormat)] [-state All|Running|Completed|Cancelled|Error]
 [-deletedLists] [-clearAll [-workflowItemId id -workflowListName "list name"]] [-verbose]
 [-reportOnly] [-batchSize numberDefaultIs500]
 [-pauseAfterBatch] [-maxItemsToDelete number]

This command is run on a front end server. To keep things up to date it would need to be scheduled. However, if you’re using out of the box workflows, there is no equivalent command. You could just access the history list and remove old data, but since SharePoint has built in tools for this, I recommend using them. These features are contained in the Information management policy settings of any list.

Open the Workflow history list (your site url + Lists/WorkflowHistory or NintexWorkflowHistory). Open the List settings page, and select “Information management policy settings” in the Permissions and Management section. If you don’t see this option, you may need to enable the relevant features.In the Content type policies, select the Workflow History content type, and then select “Enable Retention”. Once enabled, you will be able to select “Add a retention stage”.

The retention stage is what we will use to delete the workflow history items (which, given the name, is somewhat counter-intuitive, don’t you think?). Date occurred is when the event occurred, so it is likely our best time indicator, and I would suggest a period of time at least double to the automatic cleanup task, which is 60 days. Finally, we want the item to be deleted at this point, so we select “Permanently Delete” from the Action dropdown. When complete, the stage will appear as follows:

Once we save our policy, the expired items will be deleted the next time the timer jobs run.

And that’s all there is to it!

Now that we’ve taken the data out of SharePoint, it’s no longer obviously available to end users. If this is important, we will need to build some Reporting Services reports, and integrate them back into the appropriate locations in SharePoint. This will (hopefully) be the subject of an upcoming post.

I recently had the opportunity to perform an in place upgrade of a WSS 3.0 site that we had built a few years ago to Foundation, and I of course decided to use SSE instead. As it turns out, the upgrade wasn’t quite as straightforward as I had hoped.

Normally, when you perform an in place upgrade from WSS to Foundation, you first install the bits, and then run the Products and Technologies Configuration Wizard, which in turn detects the pre-existing WSS installation and offers to upgrade it. Unfortunately, this doesn’t happen with SSE. The Wizard only prompted for a new or existing Farm. 

The next step was to uninstall SSE, and to Install Foundation, once this was done, the Wizard did detect the existing installation, and properly upgraded the entire farm. Once this was done, I thought “why wait?” and I went ahead and laid down the bits for Search Server Express and then for Office Web Applications.

Everything seemed alright, but when I tried to start the services on the server, they simply weren’t there. It also wasn’t possible to create the corresponding Service Applications for either SSE or for the Office Web Applications. After much head pounding, I decided to uninstall everything, OWA, SSE, and Foundation (CAREFULLY as outlined here..), and then Install SSE alone, joining it to the Pre-existing farm.

Once that was done, everything showed up properly, and I was able to properly start the appropriate search services, and OWA services, and to create the appropriate service applications.

So as it turns out, order of operations is pretty important in this scenario. If you want to upgrade from WSS 3.0 to Search Server Express 2010 (using the in place upgrade approach), you’ll want to follow these steps:

1. Install SharePoint Foundation 2010 on your server
2. Run the Products Configuration Wizard, and perform the upgrade
3. Uninstall SharePoint Foundation from the server, removing it from the farm
4. Install Search Server Express 2010 on the Server
5. Run the Products Configuration Wizard, and re-join the existing farm
6. Test the site to ensure that it’s functional
7. (optional) Install any appropriate Service Packs and/or hot fixes for SSE
8. (optional) Run the Products Configuration Wizard to update the databases (if step 7 was performed)
9. (optional) If desired, Install Office Web Applications, and any appropriate Service Packs for OWA
10. Run the Products Configuration Wizard to complete the OWA installation
11. Start all necessary services, create the necessary service applications (search is a big one….)
12. Create a basic Search Center and configure your site collection to use it.

Hopefully this helps any other folks in the same situation.

For quite some time now, Microsoft has been able to read and write data from SharePoint lists as if they were active Access tables. This is distinct from Access Services, which ships with the Enterprise version of SharePoint Server. Access Services lets you “convert” your entire Access application to a SharePoint site at which point the Access client is no longer required (for a user).

Our situation was much simpler. We were dealing with a Power user that was good with Access, and had leveraged the list read/write capability quite heavily with 2007. However, after the upgrade, Access 2007 couldn’t open some of the lists that it could previously. Compounding this problem was that Access 2010 didn’t have this problem on the lists in question. The browser could also open these lists just fine.

The answer to this one came from what appeared to be a different problem. Some of the other lists in the site couldn’t be opened by the browser. Instead, the user received the message “The query cannot be completed because the number of lookup columns it contains exceeds the lookup column threshold enforced by the administrator”:

SharePoint has a bad reputation for “Unknown Error” messages, but this one is really quite good. This one pointed squarely at the list throttling features available in SharePoint 2010 that of course weren’t there in 2007. Basically, 2010 allows an administrator to throttle, or prevent poorly performing functions from slowing down the system for everyone. One such expensive operation is performing lookups, and the default limit is set to 8.

Dina Ayoub has a good post here on the throttling features if you would like to learn more, but the important thing to note here is that this setting affects not just lookup fields, but Person/Group and Workflow Status fields as well, so if you have 8 or more of them, the list will simply stop working.

This setting is scoped to the application level, so if it is changed, you affect all site collections in that application. (It also means that you can’t change it at all in Office 365.) You set it through the Resource Throttling settings in Central Administration. Once in CA, click on Application Management, highlight the application to be changed, and in the General Settings dropdown, pick Resource Throttling.

Scroll down to the section titled “List View Lookup Threshold”:

Here, you can simply increase its value to where you need it.

Changing the values should be done with considerable care. These throttling features were implemented for very good reasons, and changing them risks overloading your SQL server. A much better approach would be to go back and rethink the design of your list, if that’s an option. If it isn’t then this is a decent plan B. You can always buy more hardware…….

So this fixed our post upgrade list issue in the browser, how does this relate to our Access problem? Well, it turns out that they were one and the same, just manifesting differently. It seems that Access does something when it opens a list that adds a few more lookup type items to the Query, or at least it behaves that way. It also appears that Access 2010 and Access 2007 behave differently in this regard. In the end, increasing this value sufficiently solved the Access problems.

I haven’t found anything definitive out there, but anecdotally at least, you should be aware that when you use Access to open up a SharePoint list, you pay a “”List View Lookup Threshold” penalty.

Customization is not without its drawbacks. Just because you can build on SharePoint, it doesn’t mean that you should. With custom code comes all of the costs of ownership of running custom code, and the upgrade difficulties that come with it. If you can possibly use a no-code solution, you probably should. A recent customer challenge that we had is a good demonstration of that, and provides a good example of a highly customized solution being replaced with a no code solution.

Back in the days of SharePoint 2007 (!) we ran across a number of requirements for a cascaded combo box control. When filling out a form, we wanted the options available to one combo box to be driven by the selection in another. Unfortunately, there was no real way to do this with SharePoint forms. It can be done with InfoPath through filtering, but that was only available through InfoPath form libraries. We needed to provide this capability with standard lists.  Our solution was to implement a custom column that provided this capability.

This worked perfectly well for a few years, but when it came time to upgrade to SharePoint 2010, we found that the custom control didn’t quite work. One option would of course be to fire up Visual Studio and make the necessary modifications. However, since SharePoint 2010 allows you to use InfoPath to edit standard SharePoint lists now, this was determined to be the better choice, and we could ditch the custom controls. Another drawback of the custom columns is that InfoPath will simply not work with them, making solid form design difficult.

However, given that this was an upgrade, we wanted to make sure that all of the existing data was retained, so we couldn’t just trash the existing data, and start new. We needed to somehow convert the old column data to standard SharePoint data. We were able to do this through the PowerShell Import/Export function, with a few tweaks.

Step 1 – Upgrade the Content Database

Before you can use this method, the content databases will need to be in SP2010 format. If you’re doing an in place upgrade, this will happen automatically, but if you’re using the DB attach method, you’ll need to first install the custom solutions on the new farm, or else the database mount procedure could fail.

Step 2 – Export the List Data

Most SharePoint administrators are familiar with the STSADM import and export commands that allow you to export and import either single sites or entire site collections. This capability still exists in SharePoint 2010 with both STSADM and with PowerShell, but the PowerShell commands also allow you to export individual lists.

When ready, export your list with the following PowerShell command:

Export-SPWeb -Identity siteurl -Path outputpath -ItemUrl listurl –IncludeUserSecurity      -IncludeVersions All -NoFileCompression –Verbose

The NoFileCompression is important because we need to edit one of the output files after the export is complete.

Step 3 – Modify The List

Now that the data has been saved outside of the list, you can go ahead and remove the custom columns from the list. Once you’ve done so, be sure to replace them with identically named columns. If possible, you should also use the SharePoint column type that the custom columns were originally derived from. In my example below, I replace a Cascaded Lookup (custom) column with a Text column.

Once this is done, you’ll also want to delete all of the current items in the list. You could simply delete the list itself, but if you have any workflows, or custom forms, they’ll get deleted too. Also, a new list will have a new internal GUID, which may not work for you.

Another option is to use the Data Sheet view to copy the old field values to Excel, so that you can copy them back after the fields are changed. This approach will update the last edited date and the last edited by, so that may not be acceptable. If it is, you can omit steps 2,4 and 5.

Step 4 – Edit the Manifest File

In the path that was used in outputpath in step 2, there will be a number of files. Find any named manifest.xml or manifestx.xml (where x is a number) and edit them. These files contain the metadata for your list. Simply search for the name of your custom field type (in my case, CascadingDropDown) and replace it with the standard type name (like Text). Once all occurrences have been modified, save the file(s).

Step 5 – Import the List

Once the manifest has been modified to match the new list schema, you can bring in the exported list, with essentially the reverse of the PowerShell used in Step 2:

Import-SPWeb -Identity siteurl -Path inputpath -IncludeUserSecurity -UpdateVersions Overwrite -NoFileCompression -Verbose

Once the PowerShell completes, you should have all of your data in a list with no custom columns. From there, you can use InfoPath to modify your forms, or anything else that is possible with the standard column types.

One caveat though. We had customized the standard New/Edit/View aspx forms with SharePoint designer. Before we could move toward our InfoPath goal, we needed to recreate these forms as standard forms.

I wrote recently about overcoming an issue with orphaned documents when doing a database attach upgrade from SharePoint 2007 to SharePoint 2010. During that same database upgrade, I managed to run into a second nasty error. After running the PowerShell mount-spContentDatabase command, the progress would go to approximately 15% complete, and then the process would fail. Investigation of the upgrade log revealed the following errors:

[powershell] [SPContentDatabaseSequence] [ERROR] [8/26/2011 8:56:01 AM]: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence failed.
[powershell] [SPContentDatabaseSequence] [ERROR] [8/26/2011 8:56:01 AM]: Exception: The statistics '_dta_stat_1365579903_2_3_4_188_190_27_28_187_192' is dependent on column 'tp_ContentType'.
The statistics '_dta_stat_1365579903_28_192_190_4_27_187_188_3_26_2' is dependent on column 'tp_ContentType'.
The statistics '_dta_stat_1365579903_3_4_188_190_27_28_187' is dependent on column 'tp_ContentType'.
ALTER TABLE DROP COLUMN tp_ContentType failed because one or more objects access this column.

This was happening  when the upgrade process was altering the Database schema, and removing the tp_ContentType column from the table. I know SQL quite well from a BI perspective, and well enough from a SharePoint perspective, but I had never before bumped into a problem with SQL statistics.

Statistics are created in conjunction with indices to help optimize performance. Kim Tripp talks about them in a good article here, and Idera has a nice in-depth discussion on them here. However nice they may be, they were preventing my upgrade, so how important were they?

Well, as it turns out, a well meaning SQL DBA had decided that it would be a good idea to optimize the SharePoint databases, and had run a process that created a number of these statistics. Unfortunately, these ones implemented referential integrity constraints that were preventing the schema to be changed, and causing the upgrade to fail.

Luckily, these statistics followed a consistent naming convention. I was fairly easily able to go through each table in the content database and remove the offending statistics (in my case, all of the ones that began with “_dta”. There were other statistics in there, and they were left alone.

Once the statistics were removed, the upgrade proceeded smoothly. Once again, I was forced to manipulate the content database directly to solve a problem, which is not at all a good idea generally. However, in this case, it was done to undo some other changes previously implemented.

Hopefully this can save someone else some grief in the future.

Recently, I was at a client site doing a database attach upgrade, and we ran into a warning about orphaned objects. Orphaned objects will cause the upgrade to fail, so it needs to be remedied prior to the upgrade.

Orphaned objects are items that exist in the database, but aren’t properly connected to anything in the site collection. Orphaned objects are typically sites or lists. Joel Oleson has a good article on deleting orphaned sites, Orphaned lists are trickier with much of the guidance pointing to removing and then re-adding the content database. Also there is an stsadm command that should repair orphaned objects. The syntax is:

stsadm –o databaserepair –url yoururl –databasename contentDB –deletecorruption

Using the –deletecorruption directive goes ahead and fixes the problem, without it, it just tells you where the problem lies.

Unfortunately, nothing worked in my case. Running databaserepair simply resulted in a return that looked like the following:

<OrphanedObjects Count="1">
  <Orphan Type="SPList" Id="{787A6375-ABA3-4475-AE64-230853EB4448}" SiteId="{13AB0F9E-386B-4128-916C-E70BFC6A45F3}" />
</OrphanedObjects>

This discussion (in which the response marked as an answer isn’t the answer) got me pointed in the right direction. I am loathe to do anything directly in the content database, but if I had an orphan, I could at least find it there. I therefore opened up the AllLists table in the content database and did a SQL search for my list, like:

select * from AllLists Where tp_ID=’787A6375-ABA3-4475-AE64-230853EB4448’

Unfortunately, I got no results. This was baffling.

I had the GUID of the offending list, but it didn’t exist in the list table. I started rooting around elsewhere. It wasn’t too long before I found a reference to it in the AllDocs table, in the ListId column. To me, that meant that I didn’t have an orphaned list, but an orphaned document that was referring to a non-existent list..

After determining that the document was in fact disposable, I deleted its record with some simple SQL

DELETE FROM AllDocs Where ListID = ‘787A6375-ABA3-4475-AE64-230853EB4448’

Once that was done, I ran the preupgrade check, and all was clear – no more orphaned objects. We could proceed to the next roadblock… (more on that later).

This solution worked in my case, but as it involves monkeying with the content database, use at your own risk, and whatever you do – have a backup of the database available!