Reference tables don't belong in the database!

Now I should admit that over half of the tables in my database violate that statement, so I guess I have a lot of work ahead of me if I want to comply with my new resolution – even to justify it;  but I do believe there is a better approach – if only achievable in future development efforts.

To clarify, the content of these exiled tables rarely, if ever, change by any other means than direct statements by a member of IT; I don't mean transactional tables that are updated by procedural system logic.  Germane to my point, they essentially enumerate references to objects that do not reside in the database - often not even within the organization’s domain – like countries,  titles (Mr., Mrs, Miss), software configurations, status codes, etc. 

For example, a title (Mr., Mrs.) belongs to the global domain ontology.  It is not data, rather a concept to which we refer when capturing data about customers, users, vendors, etc.  Yet we store our own versions so we can all play games when our systems need to integrate – I’m sorry, I don’t have an ‘Officer’, go fish.  “Monsieur?”.. Je regrette, je ne parle pas francais.   I digress. 

Likewise, a status code seems fairly innocuous, until your trading partner receives it on an excel report or via an integration and then has to create their own local mapping  that will gradually erode as each system evolves, adding – quite irresponsibly, I might add - to each partner’s technical debt.  Even within our own organizations, we shouldn’t assume that our databases are accessible to all systems; there are indeed cases similar to this among integrated proprietary systems.

Since these references are pointing to objects outside the database realm anyway, I’m simply suggesting that we skip the middle man and point directly to the object with a URI (uniform resource identifier).  It’s not a revolutionary concept, the ever-burgeoning semantic web movement has been pushing this agenda for years. Its just an adaptation that seems to make sense.   That foreign key column in your ‘Customer’ table that references your ‘Title’ table becomes a simple varchar that contains a URI that can be scaled universally so that all consumers reference the same concept as opposed to an address within an isolated database. 

The URI does not necessarily have to be dereferencable; if it were to follow follow convention, it could be human readable.  Rather than an arbitrary status code, I might return “http://company.org/applicationx/status/pending.”  It’s a bit verbose, but at least my sales rep would know what it means…. and developers could avoid the join when looking at data in the database.  I could also format the URI as a RESTful URL where integrated applications could get a localized representation directly from the source, minimizing synchronization concerns.  Most important, the practice will encourage consumers of data to anticipate change that was inevitable to begin with, rather than to assume that their will be some formal change process that never succeeds at anything but inhibiting change.

Imagine a trading partner who, or even an internal system that gives you direct access to all of its system codes and reference lists via RESTful url, advising that they are subject to some nature of change without warning.  No formal change process that won’t work anyway, and as long as you account for the possibility of change, the integration maintains lubricated.  And dealing with the change is not that complicated.  You anticipate connection problems like a DNS – cache the known and slap on a TTL (time to live).  Build generic processes to handle additions for those elements that drive process.  And by all means, store the URI as is so you can handle cases when the partner no longer offers values that you have already consumed. 

My initiative had less momentum a couple of years ago, back when virtualization was a Citrix desktop and “the cloud” was nothing more than a Visio stencil used to represent stuff beyond our comprehension. Oh, wait! It’s still used for that. But we’re far from that comfort, well into the days when the surface area of our systems extend far beyond the warmth and safety of our own domain, and the location of our data and executables is abstracted from us by virtual servers in private clouds. Identifying our resources in a uniform way that transcends localized addressing schemes seems a logical cooperation.

I’m just marking these words in Microsoft’s new blog “Building Windows 8”, or “B8” as they call it.  I wonder if that’s pronounced “Bait” or “B – 8”.   We’ll see how that all works out.  No pessimism, or optimism…. just remarkable words.

“Windows 8 reimagines Windows for a new generation of computing devices, and will be the very best operating system for hundreds of millions of PCs”

Here’s good summary of powerful NoSQL alternatives; none of them are new, but it reminded me of that one hour round table at the IANS conference in D.C. last year during which the moderator spoke non-stop for approximately one hour about the marvels of  “eventual consistency” and distributed file systems like Cassandra (a la Facebook) and Hadoop, among others.  I don’t know what that had to do with security, but at the time I did not really comprehend the strength of that movement, so it was certainly worth the hour. 

I’m not saying that relational databases don’t have their place, I’m only recognizing the value of databases that can potentially challenge the CAP theorem, even if by pulling a sort of kobayashi maru by side-stepping the ‘Consistency’ requirement.  And I’m interested to see if Neo4J will scale to handle distributed graphs.

If your application pool in IIS 7 is set to run under the ‘ApplicationPoolIdentity’ account, you have likely encountered the need to change the privileges (usually increase) for that account, and may have discovered that the actual account is a little difficult to identify.    While the nuance is well documented, I find myself re-researching the solution every time I run into it, so I figured I’ll document it for myself; at least that way I won’t have to filter through Google results.

Assuming you already know how to set file permissions using the ‘Select Users, Computers, Service Accounts or Groups’ dialog….

• Make sure ‘Built-in security principals’ is selected in ‘Object Types’.
• Make sure your machine names is selected in Locations.
• Search for ‘IIS AppPool\DefaultAppPool’
• Set its permissions appropriately.

per Wikipedia….

In many disciplines a greenfield is a project that lacks any constraints imposed by prior work. The analogy is to that of construction on greenfield land where there is no need to remodel or demolish an existing structure. Such projects are often coveted by engineers.

I love industry terms, especially for those concepts that you otherwise wouldn’t have known there was even a term.

A long-time client recently remarked that my solution delivered "Standardized Customization", confirming how pivotal those design decisions were ten years prior, when I initially conceived much of the architecture that still prevails today in that software.  Core to that success, I believe, was a fundamental understanding that one can not anticipate the changes that will inevitably occur, leading me to focus on flexibility rather than simply a final product.  This, in my opinion, is the key ingredient in any successful implementation, from the smallest features to mission critical solutions - there is no final product. 

The world evolves, thus does business, thus must software.

Silly me!  I thought I’d get through Sharepoint 2010 - ‘hello world’ without any bumps.  But, of course, when I went to add my newly created webpart to my newly created page, I was treated to this little reminder of what I should well expect after fourteen years of Microsoft programming.

After four hours of searching, several ineffective antidotes, and a lot of valuable education, my aggravation was finally relieved by an amalgam of all aforementioned prescriptions.  I did exercise scientific method, so I can offer this with some confidence.  

Mine is a Windows 7 “development” install, which was done well after VS2010 + all applicable windows updates.

Symptoms

• Event Log Entries
• Event ID: 7034
• “The Sharepoint 2010 User Code Host service terminated unexpectedly.  This has happened n times.
• ULS Log Entries – log path should be “C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS”
• Performance Counter OS (pdh) call failed with error code PDH_INVALID_HANDLE.
• PDH failure on counter \MachineName\ASP.NET\\Requests Current with error Unknown error (0xc0000bbc)
• SharePoint 2010 User Code Host Service will not start (the event log isn’t lying).

Resolution

The issue seems to be because SharePoint is unable to create the performance counters for its process.  While the authors of the various blogs that I encountered seemed to have rather isolated symptoms, I was lucky to have a bit of several…

• Make Sure the account under which the Sharepoint 2010 User Code Host Service is running is in the following groups.  After doing so, stop and start the SharePoint 2010 User Code Host Service.
• Performance Monitor Users
• WSS_WPG
• WSS_ADMIN_WPG
• The service account should have write access to the logs directory
• “C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS”
• The service account should be that which was configured during install.  I understand that SharePoint also makes other assumptions based on that account, so that if you were to change it – as some suggest – to an account with greater privilege, you will likely encounter other issues.
• Check that your performance counters are set-up properly
• The registry key HKLM\Software\Microsoft\Windows NT\CurrentVersion\Perflib\009 should contain three entries (Default), Counter and Help
• You should have a file named perfc009.dat in c:\windows\system32.
• If these conditions aren’t satisified, run the following command from you VS2010 command prompt.  lodctr /r
• The value for “Disable Performance Counters” under registry key HKLM\System\CurrentControlset\services\PerfProc\Performance should be 0.

Yes, my configuration violated all of these; I don’t know what I did so terribly wrong during my install, other than assuming that Microsoft’s installer would do its job.  Bitter, I know, yet I’ll keep on sharpening my axe, as it were, with Microsoft’s toolset.

Installing Sharepoint Server 2010 on a Windows 7 Client for development purposes, you get the following error during configuration.

Could not load file or assembly 'System.Web.DataVisualization, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies.

To fix… make sure you have .NET Framework 3.5 installed.

 http://www.microsoft.com/downloads/details.aspx?FamilyID=
AB99342F-5D1A-413D-8319-81DA479AB0D7&displaylang=en

And the MSChart Component

http://www.microsoft.com/downloads/details.aspx?familyid=
130F7986-BF49-4FE5-9CA8-910AE6EA442C&displaylang=en

There’s a white board next to my kids’ computer on which there is nothing but scribbled credentials for all of the sites that require a unique login and password; most of them comprised of some random forgettable number because all of the memorable names, even the silliest, were already taken by some other kid who I curse now for making me endure the ‘enter and confirm new password’ dialog one more freaking time!  But I digress.

But all of the sites my kids visit, and most of the sites I visit aren’t storing sensitive data.  O.K, maybe they ask for names and birth dates, I suppose to ensure that the user is of reasonable age for the site’s content – as though “are you under or over this age?” were insufficient – but let’s be honest, If you’re giving them you’re real name and age, they are probably not your greatest security risk.  I don’t even give them a valid email address!  I use something like mailinator, or a dummy email account that I never check.  It’s psychegophantom@hotmail.com; go ahead, email it all you want.  My kids have so many fake names, they’re no longer sure what their real names are.  All I’m saying is…  if a site is not storing (or I am not offering) sensitive data, it shouldn’t require another proprietary set of credentials, right?  Right!  If they’ve got my bank account or SS#, though, I’m willing to give a blood sample if it were possible.

So as I entertain projects that might persist a user’s non-sensitive data, I think it was my responsibility to engage a solution like OpenID.  And, as a software developer, I can say that it was certainly a much easier route than developing or implementing another proprietary credential management tool.  Save the few misunderstandings that could have been avoided had I just RTFM, this solution would have taken no more than 15 minutes to implement. 

For an overview of OpenID, visit http://openid.net/get-an-openid/what-is-openid/

A Basic OpenID implementation in C# – Using DotNetOpenAuth

Download the .Net library at http://www.dotnetopenauth.net.  You can get the source there, too, if you want an in depth understanding of the underlying mechanics.

Create a login page to guide the user through the process.  An example.. http://www.jamespritz.com/login/login.aspx

I chose the OpenIdLogin control simply so I could provide common providers while still allowing the user to enter the discovery url of their preference.  It also allows the page to function without javascript.

<%@ Register Assembly="DotNetOpenAuth" Namespace="DotNetOpenAuth.OpenId.RelyingParty" TagPrefix="rp" %>

<form id="login" runat="server">

<rp:OpenIdLogin ID="OpenIdLogin1" runat="server" CssClass="openid_login" RequestCountry="Request" RequestEmail="Require" RequestGender="Request" RequestNickname="Request" RequestTimeZone="Require" RememberMeVisible="False" PolicyUrl="~/PrivacyPolicy.aspx" TabIndex="1" OnLoggedIn="OpenIdLogin1_LoggedIn" OnLoggingIn="OpenIdLogin1_LoggingIn"/>

</form>

The OpenId_LogginIn event occurs before attempting to communicate with the identity provider, so you can gather information about the request before it is made. 

The OpenId_LoggedIn event occurs in response to the identity provider’s action; this is where you inspect the results and determine if the provider was able to authenticate the user.  You can also inspect the ClaimsResponse on form load, but this seems to flush the response cache, which will cause an error if you trap the loggedin event as well.

        protected void OpenIdLogin1_LoggedIn(object sender, OpenIdEventArgs e)
        {
            ClaimsResponse r = e.Response.GetExtension<ClaimsResponse>();
            switch (e.Response.Status)
            {
                case AuthenticationStatus.Authenticated:
                    {
                        //gather the user info from r  
                        break;
                    }
                default:
                    {
                        //handle the other cases
                        break;
                    }
            }         
        }

That’s it!  You can simply persist the user’s info in a session variable for the duration of their session.  Or you can build a more sophisticated solution that persists user information and allows you to log them in automatically.

First, take a look at this code.  Take it in… feel it… be it.

   1:      public class Identity<T>

   2:      {

   3:          public T Value { get; private set; }

   4:          public Identity(T value) { this.Value = value; }

   5:      }

   6:   

   7:      public static class IdentityExtensions

   8:      {

   9:          public static Identity<T> ToIdentity<T>(this T value) 

  10:          { 

  11:              return new Identity<T>(value); 

  12:          }

  13:   

  14:          public static Identity<V> SelectMany<T, U, V>

  15:              (this Identity<T> id, Func<T, Identity<U>> k, Func<T, U, V> s)

  16:          {

  17:              return s(id.Value, k(id.Value).Value).ToIdentity();

  18:          }

  19:          public static Identity<U> SelectMany<T, U>

  20:              (this Identity<T> id, Func<T, Identity<U>> k) 

  21:          { 

  22:              return k(id.Value); 

  23:          }

  24:      }

The code below exercises the classes above, so from it you should get a general idea of what’s going on, if you hadn’t already a clue.

   1:      class Program

   2:      {

   3:   

   4:          static void Main(string[] args)

   5:          {

   6:   

   7:              var r = 5.ToIdentity()

   8:                  .SelectMany(x => 6.ToIdentity(), (x, y) => x + y);

   9:   

  10:              var b = from x in 5.ToIdentity()

  11:                      from y in 6.ToIdentity()

  12:                      select x + y;

  13:   

  14:              Console.WriteLine(r.Value);

  15:              Console.WriteLine(b.Value);

  16:   

  17:              Console.ReadKey();

  18:   

  19:          }

  20:      }

This code frightens me; not terrifically, but enough to keep me honest and humble.  As it were, I see only code; I don’t see the blonde, brunette or the redhead.  I can discern the LINQ, extension methods, lambda expressions and generics, but there is obviously something bigger happening…. magic, if you will.  Now I don’t believe in magic, but I am certainly willing to call it that if it saves me from the spiraling rabbit hole that an educational journey often becomes.  And I’ll admit, LINQ has always been magic.

Now you might be asking yourself what all this has to do with Reactive Extensions (Rx).  If so, then great!  This isn’t pointless.  If you are unfamiliar with Rx, follow the link… I don’t regurgitate.  Better yet, go to  Erik Meijer’s intro, which should pique your interest.  If you are familiar with Rx and have explored the  Hands on Lab, then you clearly didn’t follow the rabbit on page 8.

The code represents monads, which are apparently the theory behind LINQ, which is fundamental to Rx.  In fact, some suggest that LINQ should have been called Language INtegrated Monads.  I wish it were.  My mind always wants to draw a direct correlation to query in the SQL sense, but Query in the LINQ sense has more to do with functional programming.  I extracted this code from Wes Dyer’s blog entry on  The Marvels of Monads  If monads are new to you, or you don’t get the fundamentals of LINQ, I highly recommend following through the steps in the article; don’t just cut and paste my reduction.  if you’re hoping for an explanation of Monads, I don’t think it can be done, as Wikipediahas so generously proven; bring some Excedrin along for the journey.  I think it is just something you have to ‘get’, and I think Wes’ article presents it in such a manner.

The journey from Rx to monads included many more resources; I’ve reference only the ones that I believe helped me gain a fundamental understanding of LINQ.   The others were just static.