This is has been an awful week; things have just not gone as planned from the get-go. I’m gonna run through several issues in this post, and perhaps someone will find some value in them; lessons learned may help others avoid the pain of my ways.
VSTS: DB Pro
This week, I’ve spent way too much time just trying to figure out how to use Visual Studio; I’ve mentioned in previous posts that I’m a database developer, but that I’ve spent most of my career working with the standard SQL Server DBA tools: Query Analyzer & Management Studio. When we migrated to TFS, my manager encouraged me to adopt VSTS:DB Pro as a method of developing, and I’ve put some real effort into learning how the system works. I’m slowly becoming accustomed to it, but there’s still some very quirky things that I’ve yet to figure out.
Issue 1: Logins. Whenever I import a database into a database project, I get a long list of errors because we have Windows accounts associated with users in the database; since those users don’t exist on my laptop, the project throws all kinds of ugly errors. Googling the error reveals very little information; the only method I’ve found to eliminate the errors is to drop the associated users and schemas, which of course, can lead to VERY BAD THINGS in deployment scripts. I’m sure there is a method to resolve it; I just haven’t found it yet.
Issue 2: The Silent Source Control Failure. Most of my projects involve references to vendor databases; we have a policy of not modifying vendor databases, so when I need to write a stored procedure that accesses data from a third-party source, I create a linked database that sits on the server beside the vendor database, and include my proc in there. I don’t touch their code, but am able to customize my queries to pull data. When setting up projects in VSTS:DB Pro, I usually make my custom database the primary project, and include a database reference to a project based on the vendor db. This usually works out OK, but I ran into an error yesterday where the project wizard would complete, but the project was unable to load. Nothing I did would fix it; I finally removed the vendor db from the solution, and was able to load my project (albeit with tons of errors because of the lack of references).
The problem? One of the stored procs in the vendor db has an extremely long name (nearly 100 characters); combined with the path to the workspace, the name of the file exceeds the 255 character limit for Windows file management. TFS apparently doesn’t know how to tell me that the file name is too long, so it just refuses to open. Try wasting 2 hours of precious development time tracking that down, and you can imagine how frustrating that is.
Haste makes waste.
A few months ago, we made a decision to adopt a version of Scrum to standardize our development processes; we’ve made up some of our rules along the way, and for the most part, we’ve done OK with it. However, the pressure to show progress and meet the needs of the organization means that sometimes I’ve made the wrong decision in my desire to please our Product Users. One case was today; one of our rules is that we a) strive to only deploy once a month (at the end of a sprint), and b) if we need to deploy an emergency patch, we only deploy on one day of the week (Wednesdays).
One of our users found an issue, and a production DBA researched it and made a recommendation for a relatively simple code change; I was out of the office Monday, so I only saw the recommendation on Tuesday. The issue was important, and the code looked good, so I thought I could get away with rolling it out today; unfortunately, I didn’t do due diligence on the code, and neglected to check for dependencies on that particular proc. We deployed my change today at 8:30 AM; by 3:30 PM, I was rolling it back. Luckily, the only side effect was that we generated a few extra tickets for our clients, but still, it was a painful experience.
The one thing I want to carry away from this experience is that as our system becomes increasingly complex, the less likely it is that a simple change can be made in less than a day. The code may be simple, but the context of the code in terms of surrounding processes SHOULD warrant time for thorough review. I need to quit reacting to the deemed urgency of the issue, and think about long-term stability instead.
Even the best deployments can have issues.
Overall, I think our deployment process works well, but even a good process can have points of failure. Even though we have several sets of eyes examining all of our code before it goes into production, things can still slip by. For example, at 3:45 this afternoon, I was informed that a stored procedure in production was not returning all of the data; the procedure pulls data from a third-party database, and makes certain assumptions about the nature of that data. In this case, we have a standard naming syntax for all of the devices we manage; ClientID, followed by a space, then device name (e.g., 999 Joe’s Server). The stored proc had a where clause that ended with LIKE ‘[0-9][0-9][0-9] %’; the missing row of data had a device name with the space omitted.
Now, it wasn’t entirely my fault because I had written the code to spec; however, I should have anticipated that the naming convention could be slightly off (and still be parseable; if they had a 2-digit client id, it would be a different story); in the end, our company looked bad to a client because of typo, and that’s not good. Easy to fix the code, but hard to regain the trust of the client.
Tomorrow WILL be a better day.
I’m not much for mumbo-jumbo, but I do believe that life is what you make of it. If I go back to work tomorrow and assume that it’s going to be a bad day, then I’ll probably be proven right. I’m changing my attitude as of this moment, and I choose to believe that the priority issue that I address tomorrow (while it may not be the issue I want to work on) will be the issue I was meant to work on.