SQLServerPedia Syndication

#DevOps – Starting the path

Last week, I had the pleasure of attending my first DevOps conference (DevOpsDays Nashville), and the first conference that had absolutely nothing to do with SQL Server since graduate school. The conference was great; I met some new folks, and had some great moments of insight early on. The DevOps community is a very welcoming community, but as a relatively new explorer, I quickly got lost. The sessions ranged from culture to technology, and somewhere in between; the speakers were very casual, and to be honest, a little unstructured. It was very different than my previous experiences at tech conferences, where even professional development talks were goal-focused.

One the last day of the event, during one of the keynotes, I encountered the following tweet:

Paul Reed’s piece is interesting, because it helped me begin to form a schema around all that is DevOps; he defines the following frameworks underneath the umbrella of DevOps:

Automation and Application Lifecycle Management – basically, the tools needed to insure micro-deployments and increased time-to-market. The focus here is on Continuous IntegrationDeploymentDelivery, and Configuration Management.

Cultural Issues – Reed breaks these out into a relatively fine grain in a manner that he didn’t do with technology, but he defines three subsections of culture: Organizational Culture, Workflow, and Diversity. Each of these areas has a slightly different area of focus, but they all deal with the “soft, fuzzy, human interactions that lie just beyond the technology.” Organizational Culture can refer to breaking down silos, restructuring, and/or management & leadership. Workflow refers to the application lifecycle methodology, such as the conjunction of Agile and Lean methods.  Diversity is a focus on bringing different voices into technology, including feminist, LGBTQ, and minority perspectives.

The Indescribable – finally, Reed finishes with basically an “everything else” definition. Some organizations which have pioneered DevOps practices don’t really call them that; they’ve just organically grown their own processes to focus on efficiency, including minimizing silos and rapid delivery.

I have to admit, it helped me to reframe my perspective on various lectures using this structure; that may not be true to the spirit of DevOps itself (which de-emphasizes artificial constraints), but it’s given me some room for thought in terms of the areas I want to study and learn. The technology is interesting, but I don’t see myself working in those areas anytime soon; my real focus will be on organizational culture and workflow. As a manager, I want to increase my efficiency by building a team, and I think the path I’m on is to figure out how to push my team of system engineers and database administrators to prepare for continuous integration.

More to come.

Getting my #DevOps learn on…

On Thursday (Nov 10), I’m going to be going to DevOpsDays Nashville; I’m excited.  This will be the first conference in several years that I’ve attended where I wasn’t either an organizer or a speaker.  Hoping to meet some new folks and learn a lot; I’m really pushing myself to get out my comfort zone (databases), and soak up some new knowledge regarding IT culture, automation, and agile operations.

If you’re around and want to say hi, follow me on Twitter: @codegumbo

 

Oh, The Places You’ll Go! – #SQLSeuss #SQLPASS

Last week, I had the privilege to speak at the annual PASS Summit; I got to present two different sessions, but the one I’m the most proud of was my Lightning Talk: Oh, the Places You’ll Go! A Seussian Guide to the Data Platform. I bungled the presentation a bit (sorry for those of you who want to listen to it), but I feel pretty good about the content. I’ve presented it below, with the slides that I used for the talk.

The goal of this presentation was to explore the Microsoft Data Platform from the perspective of a SQL Server professional; I found this great conceptual diagram of the platform from this website a while back, and wanted to use it as a framework. I figured the best way to teach a subject was the same way I teach my 3-year-old: a little bit of whimsy.

Enjoy.

You have brains in your head

And SQL Skills to boot

You’ll soar to great heights

On the Data Platform too

You’re on your own, and you know what you know,

And YOU are the one who’ll decide where to go.

You’ve mastered tables, columns and rows, OHHHHH MYYYY

You may even have dabbled in a little B.I.

You’re a data professional, full of zest,

But now you’re wondering “What comes next?”

Data! It’s more than just SQL,

And there’s a slew of it coming, measured without equal.

Zettabytes, YotaBytes, XenoBytes and more

All coming our way, faster than ever before.

So what should we do? How should we act?

Should we rest on our laurels? Should we lie on our backs?

Do we sit idly by, while the going gets tough?

No… no, we step up our game and start learning new stuff!

 

Oh, the places you’ll go!

ARCHITECTURE

Let’s start with the Theories,

The things you should know

Designing systems as services,

Are the route you might go.

Distributed, scalable

Compute on Demand

The Internet of Things

And all that it commands.

Infrastructure is base,

Platform is in line

Software and data

Rest on top of design

Once you’ve grasped this

Once you’ve settled in

You’ve embraced cloud thinking

Even while staying on-prem.

But beyond the cloud, there’s data itself.

Structured, polyschematic, binary, and log

Centralized or on the edge,

Some might say “in the fog”

Big Data, Fast Data, Dark, New and Lost

All of it needs management, all at some cost

There’s opportunity there to discover something new

But it will take somebody, somebody with skills like you.

Beyond relational, moving deep into insight

We must embrace new directions, and bring data to life

And there’s so many directions to go!

ADMINISTRATORS

For those of you who prefer administration

System engineering and server calibration

You need to acknowledge, and you probably do

You’ll manage more systems, with resources few.

Automation and scripting are the tools of the trade

Learn powershell to step up your game.

Take what you know about managing SQL

And apply it to more tech; you’ll be without equal

Besides the familiar, disk memory CPU

There’s virtualization and networking too

In the future you might even manage a zoo,

Clustering elephants, and a penguin or two.

 

But it all hinges on answering things

Making servers reliable and performance tuning,

Monitoring, maintenance, backup strategies

All of these things you do with some ease.

And it doesn’t matter if the data is relational

Your strategies and skills will make you sensational

All it takes is some get up, and little bit of go

And you’re on your way, ready to know.

So start building a server, and try something new

SQL Server is free, Hadoop is too.

Tinker and learn in your spare time

Let your passions drive you and you’ll be just fine

DEVELOPERS

But maybe you’re a T-SQL kind of geek,

And it’s the languages of data that you want to speak

There’s lots of different directions for you

Too many to cover, but I’ll try a few

You could talk like a pirate

And learn to speak R

Statistics, and Science!

I’m sure you’ll go far

Additional queries for XML and JSON

Built in SQL Server, the latest edition.

You can learn HiveQL, if Big Data’s your thing

And interface with Tez, Spark, or just MapReducing

U_SQL is the language of the Azure Data Lake

A full-functioned dialect; what progress you could make!

There’s LINQ and C-Sharp, and so many more

Ways to write your code against the datastores

You could write streaming queries against Streaminsight

And answer questions against data in flight.

And lest I overlook, or lest I forget,

There’s products and processes still to mention yet.

SSIS, SSAS, In-memory design

SSRS, DataZen, and Power BI

All of these things, all of these tools

Are waiting to be used, are waiting for you.

You just start down the path, a direction you know

And soon you’ll be learning, your brain all aglow

And, oh, the places you’ll go.

And once you get there, wherever you go.

Don’t forget to write, and let somebody know.

Blog, tweet, present what you’ve mastered

And help someone else get there a little faster.

Feel free to leave a comment if you like, or follow me on Twitter: @codegumbo

One (Last) Trip to the Emerald City for #SQLPASS

On Monday, I’m flying out to the Emerald City (Seattle, WA) for the annual gathering of Microsoft database geeks known as the PASS Summit; as always, I’m excited to see friends and learn new stuff. However, this will probably be my last Summit. Over the last few years, my career trajectory has taken me away from database development and administration, and it’s time that I start investing in the things that now interest me (technology management, and operational culture). My goals for the next year are to attend conferences like the DevOps Enterprise Summit and the SRECon; I want and need to learn more about making IT efficient, and managing large-scale applications.

I’m not entirely disconnecting from the SQL community; I still plan to stay active and involved in our local chapter (AtlantaMDF), and part of the organizing committee for our SQL Saturdays. I still want to be a data-driven professional; I’m just not a data professional. That’s a subtle distinction, but it’s important to me. I’ll still sling code part-time and for hobbies, but I’m really trying to hone in on what I enjoy these days, and it’s process, procedures, management, and cultural change in IT (all IT, not just SQL Server).

So, this year will be different for me; instead of trying to network and schmooze and elevate my own SQL skillset, I’m going to hang out in sessions like “Overcoming a Culture of FearOps by Adopting DevOps” ,Agile Development Fundamentals: Continuous Integration with SSDT“, and
Fundamentals of Tech Team Leadership“. I may visit some courses on Cloud development and Analytics, but mostly, I want to enjoy spending some time with folks that I may not see again for a while.

I truly hope to see you there; I owe a lot to all of you, so I’m probably going to have a huge bar tab after buying rounds. Should be an exciting week.

#DevOps “We are all developers”

https://youtu.be/RYMH3qrHFEM

While thinking about the Implicit Optimism of DevOps, I started running through some of the cultural axioms of DevOps; I’m not sure if anyone has put together a comprehensive list, but I have a few items that I think are important. Be good at getting better is my new mantra, and now, I’m fond of saying “We are all developers”. I remember eating lunch at SQL Saturday Atlanta 2016 listening to a database developer describing this perspective to a DBA, and hearing how strongly the DBA objected to that label. I tentatively agreed with the developer, but recently, I’ve gotten more enamored with that statement.

Having worked as both a developer and an administrator, I get it; there’s an in-group mentality. The two sides of the operational silo are often working toward very different goals; developers are tasked with promoting change (new features, service packs, etc). DBA’s are tasked with maintaining the stability of the system; change is the opposite of stability. Most technical people I know are very proud of their work, which means that there’s often a desire for accuracy in the work we do. If a DBA is trying to make a system stable, and you call them a developer (think: change instigator), then it could be perceived as insulting.

It’s not meant to be.

Efficient development (to me) revolves around the three basic principles of:

  1. Reduce – changes should be highly targeted, small in scope, and touch only what’s necessary.
  2. Reuse – any process that is repeated should be repeated consistently; and,
  3. Recycle – code should be shared with stakeholders, so that inspiration can be shared.

From that perspective, there’s lots of opportunities to apply development principles to operational problems. For my DBA readers (all three of you), think about all the jobs you’ve written to automate maintenance. Think about the index changes you’ve suggested and/or implemented. Think about the reports you’ve written to monitor the performance of your systems. Any time you’ve created something to help you perform your job more efficiently, that’s development.

DevOps is built on the principle of infrastructure as code, with an emphasis on giving developers the ability to build the stack as needed. Google calls its implementation of DevOps principles Site Reliability Engineering, and characterizes it as “what you get when you treat operations as if it’s a software problem”. Microsoft is committed to DevOps as part of its application lifecycle management (although it’s notably cloud-focused). When dealing with large-scale implementations, operations can benefit from the application of the principles of efficient development.

We are all developers; most of us have always been developers. We just called it something else.

The Implicit Optimism of #DevOps

One of my favorite podcasts lately is DevOps Café; John Willis and Damon Edwards do a great job of talking about the various trends in IT management, and have really opened my eyes to a lot of different ways of thinking about problems in enterprise systems administration. On a recent podcast, John interviewed Damon about his #DOES15 presentation, “DevOps Kaizen: Practical Steps to Start & Sustain a Transformation“. During that conversation, Damon mentioned a phrase that really resonated with me: Be Good at Getting Better.

At the heart of the DevOps philosophy is the desire to improve delivery of services through removal of cultural blockages. Success isn’t measured by the amount of code pushed out the door or the number of releases; it’s the ability to continuously improve over time. Companies that experiment (even with ideas that don’t work) learn a different way to approach any problem that they face. The freedom to experiment means that failure is not an outcome; it’s a method of improvement.

The optimism of that appeals to me; I think if you’re focusing on continuous improvement, then you’ve implicitly accepted two fundamental principles of optimism:

  1. Change is necessary for growth, and
  2. Things CAN improve (you just need to figure out how).

There’s some beauty in that; if you’re an organization facing overwhelming technical debt, it’s not uncommon to sink into a spiral of despair, where changes are infrequent for fear of breaking something. Mistrust breeds, as organizations point fingers at other teams for “failing to deliver”. You quit working toward solutions, and instead focus on fighting fires and maintaining some sort of desperate last stand.

You’re better than that.

DevOps is a cultural change; it’s an optimistic philosophy focused on changing IT culture while being open to different strategies for doing so. If you can commit to Be Good at Getting Better, you can change. It may be slow, it may be frustrating, but every day is an opportunity to incrementally move the ball forward in delivering quality business services. The trick is not to focus on where to begin, but simply to begin.


#BigData is coming; what should SQL Server people do about it?

I’ve been presenting a lot on Big Data (specifically Hadoop) from the perspective of a SQL Server DBA, and I’ve made a couple of recent observations.  I think most people are aware of the fact that data generation is growing at a staggering rate, with some estimates as high as 44 zettabytes by the year 2020; what I think is lacking in the SQL Server community is a rapid movement among database professionals to expand their skills to highly scalable Big Data platforms (like Hadoop) or streaming technologies.  Don’t get me wrong; I think there’s people out there who have made the transition (like Michelle Ufford; SQLFool, now Hadoopsie), and are willing to share their knowledge, but by and large, I think most SQL Server professionals are accustomed to working with our precious relational system.

Why is that?  I think it boils down to three reasons:

  1.  The SQL Server platform is a complex product, with ever increasing opportunities to learn something new.  SQL 2016 is about to drop, and it’s a BIG release; I expect most SQL Server people to wrap themselves up in new features and learn something new soon.  There’s always going to be a need for deep expertise, and as the product continues to mature and grow, it requires deeper knowledge.
  2. Big Data tools are vast, untamed, and very organic.  Those of us accustomed to the Microsoft development cycle are used to having a single official product drop every couple of years; Big Data tools (like Hadoop) are open-source, prone to various forks, and very rapidly developed.  It’s like drinking from a firehose.
  3. It’s not quite clear how it all fits together.  We know that Microsoft has presented some interesting data technologies as of late, but it’s not quite clear how the pieces all work together; should SQL Server pros learn Azure, HDInsight, Hadoop?  What’s this about U-SQL?  StreamInsight, Spark, Cortana Analytics?

The first two reasons aren’t easily solved; they require a willingness to learn and a commitment to study (both of which are difficult resources to commit).  The third issue, however, can be easily addressed by the following graphic.

This is Microsoft’s generic vision of a complete end-to-end analytics platform; for the data professional, it’s a roadmap of skills to learn.  Note that relational engines (and their BI cousins) remain a part of the vision, but they’re only small pieces in an ever-increasing ecosystem of database tools.

So here’s the question for you; what should SQL Server people do about it?  Do we continue to focus on a very specific tool set, or do we push ourselves (and each other) to learn more about the broader opportunities?  Either choice is equally valid, but even if you choose to become an expert on a single platform in lieu of transitioning to something new, you should understand how other tools interact with the relational system.

What are you going to learn today?

Reason 1234 for attending #SQLPass: The stretch

I’ve done a lousy job of encapsulating my thoughts about the Professional Association for SQL Server’s Summits for the last few years, but here’s a quick thought about one of the reasons I love this conference: the stretch.

What do I mean by stretch?  It’s the exposure to new ideas, new concepts, things I may never use.  It’s easy for technical people to get obsessed about our struggles of the day; we invest a lot of mental energy into figuring out problems that require immediate solutions, so we often pick educational sessions that fit into our current solution requirements (i.e. I’m having issues with ETL, so let me go pick up a few sessions on BIML).  That’s satisfying, but it’s not a stretch.

The stretch is thinking about issues and problems which I don’t see ahead.  It’s the creative detours, the opportunities to explore ideas that are just, well, fun to think about.  There’s two benefits to this; first, by allowing the mind to wander, it actually gives me a chance to have an inspirational moment about my current issues (the “aha” moment).  Second, by learning a little bit about something foreign to me, it gives me an advantage if the situation ever arises that I may need to know something about that topic (i.e. we don’t use Azure now, but we might in the future).

What’s your stretch at Summit?  I’m going to squeeze in a class on R, and maybe some USQL.

Coding naked.

Me, circa 1992. This is not the most embarrassing picture of that time period.

Me, circa 1992. This is not the most embarrassing picture of me from that time period.

True story: when I was in college, I was the lead singer (if you can call it that) for a goth rock Christian rave band called Sandi’s Nightmare.  I had zero musical talent, and relied solely on the strengths of my band-mate (Brad; you still rock); however, while my singing ability was dubious at best, I was pretty theatrical.  We had a total of 3 awesome shows before we moved on to other things, but it was a lot of fun while it lasted.  Our biggest hit was a cover of Lust Control’s Dancing Naked, a fantastic little number about David’s celebration of the ark returning; his passion and excitement overwhelmed his embarrassment.

I promise, I’m not making this stuff up; it was pretty edgy for the Bible belt of  Northeast Louisiana in the early 90’s.  I mean, I had an earring and everything.   The point of this post (besides reveling in my musical abilities and tastes of 25 years ago) is that sometimes we just need to be naked.

Let me explain; I made choices that in hindsight are difficult to defend (like my haircut). However, there was a reason for the choice at the time, and even though it may not have been the greatest idea, I’ve learned something from every experience I had.  While there are things that I would do differently if given the choice and knowledge I have today, I don’t regret many things.  That’s what the metaphor of nakedness is all about; being open and honest about not only your successes, but your choices along the way.  You can’t change the past, but you can learn from it.

Becoming a Naked Developer

I’ve spent most of my professional career as a SQL developer working for one company (13 years in November);  I was the first developer hired by that company, and my job was to build an integration platform transforming syslog data into reports, all on a shoestring budget.  I strung something together in a couple of weeks, and in the first month, I took a process that normally took two weeks to do by hand down to a day; my boss threatened to kiss me (I was flattered, but declined).

Was it pretty (the code, not the boss)?  No.  Was it successful?  Yes.

Fast forward a couple of years to 2004: I’m a member of small team building a solution to analyze more data than I’ve ever seen in my life (terabytes of data from firewalls and Windows machines); we had to import and semantically normalize data in near-real-time.  We built a solution using DTS, and then moved it to .NET and MSMQ.

Was it pretty? No.  Was it successful?  Yes.  Part of that code is still being used today, and my company has thrived off of it.  We were acquired by a larger company 8 years ago, and we’ve kept moving forward.

Every few years, we’d talk about doing a rewrite, and we’d bring in some outside expertise to evaluate what we’ve done and help us come up with a migration plan.  Some consultant would come in, look at what we did, and shake their heads at us.  Each time, we (the remaining original developers) would attend meeting after meeting as our code was dissected and judged; we sat naked (metaphorically) in front of a team of experts who told us everything that was wrong with the choices that we made.  At the end of each visit, we’d regroup and complain about the agony of judgement. In all honesty, we would have done things differently, given our current experience (and modern technology).  However, the simple truth is: we won the day.  We continue to win the day.

During some of these initial code reviews, the consultants were jerks.  They’d spend a lot of time bemoaning how horrid everything was; I’d be embarrassed and try to explain what I would have done differently if only I had the time.  The consultants would eventually propose some complex solution that would cost hundreds of thousands of dollars to implement.  Our company would consider the offer, stick it in a file somewhere, and we’d keep on keeping on.   Clunky code that works is better than expensive code with the same outcome.

Over time, I learned to live with the embarrassment (as you can tell from this post); I started listening to what the consultants would say and could quickly figure out which ones had good advice, and which ones were there to just be seagulls.  People that focused on solving the immediate problems were much easier to talk to than the people who wanted to rip out everything and start over.  Don’t get me wrong; sometimes you need to start over, but there is  a cost associated with that decision.  It’s usually much easier to start where you are, rather than trying to reinvent the wheel.

What’s my point?

First and foremost, don’t dwell on your past mistakes.  If you can honestly say that you built something that worked within the constraints of the time period, no matter how clunky or silly it seems now, then it was the right decision.  Working software works.  Laugh at it, learn from it, and let it go; repair what you can, when you can, but keep moving forward.

Second, learn to code naked.  Whatever it is that you’re trying to do at the moment, share it.  Open yourself up to code review, to criticism; exposure brings experience.  Granted, you’ll meet some seagulls along the way, but eventually, you’ll figure out who’s there to help you grow as a developer.  Foster those relationships.

Third, encourage people to grow; recognize that the same grace you desire for your coding mistakes is applicable to others as well.  Be kind, be gentle, but be honest; seek to understand why they made the design choices they did, and be sensitive to the fact that if it works, it works.  Sometimes we get caught up on trying to figure out the right way to do something, and ultimately end up doing nothing because it’s too difficult to implement it according to the textbook.

Life is too short to not enjoy what you do; try to share that joy without embarrassment.

THIS is the most embarrassing picture of me from that time period.  I thought being an actor was the best way to meet girls… Yes, those are tights I am wearing…

 

 

 

Agile Perspectives: Fixed or Variable Length Iterations

Got into a deep discussion with a colleague tonight over different approaches to Agile development (yes, I’m a geek; why do you ask?).  I’m a Fixed Length Iteration kind of guy, particularly since I’ve spent most of time on the Operational side of the house recently.  He’s a Variable Length Iteration fan, and I thought the arguments on both sides of the fence were compelling enough that I wanted to blog about them.

Really, I’m doing this for my adoring fans.

Actually, I’m doing this because I feel guilty about not blogging, and thought this was at least SOMETHING to warrant paying for a domain year after year.

Anyway, here’s the breakdown of the two arguments; let me start by spelling out some common assumptions.  I’m assuming that you’re reading this because you have some interest in development, and more specifically, some interest in agile development.  If you have those interests, then I’m also assuming that you know what an iteration is.  If not, the hyperlinks in the previous statements should help educate you and steer you clear away from this blog to more fascinating subjects.

The Argument for a Fixed Length Iteration

There’s a couple different versions of the Fixed Length Iteration, based on either a calendar (January, February, etc.) or a cycle (every four weeks); the goal is to commit to a ship date, and stick to it time after time.  I’m more of a fan of a calendar-based Fixed Length Iteration; the actual length of the iteration varies (February is short), but development is consistently wrapping up every month, and every month code is being shipped out the door.  My reasons for supporting this model are as follows:

  1. People are cyclical creatures. We live from paycheck to paycheck; we hate Mondays and work for the weekend.  Having a fixed length cycle (with some minor variation if you based that cycle on a calendar) helps in estimating and planning what gets done and when.
  2. A fixed cycle forces developers to think in terms of components. If you’re expected to ship working code at the end of each month, you start writing code that is bite-size, so that you can leave out pieces that aren’t finished (rather than waiting until the whole thing is done).  Bite-sized code is easier to test and deploy with reduced risk.
  3. A fixed cycle means that it’s easy to change directions without stopping development efforts. Sometimes business priorities change; a fixed length cycle allows developers to continue moving forward until the end of the iteration and change gears there.  The start of a new iteration is never far away, so most businesses can wait until the next month rather than waiting till an unknown end of a sprint.

The Argument for a Variable Length Iteration

I’m trying to give this perspective a fair shake, even though I don’t subscribe to it; in a variable length iteration model allows business and development to scope out work at the beginning of an iteration and estimate a ship date regardless of the cycle or the calendar; the goal is to allow code to mature and be more stable.  My friend subscribes to this model because:

  1. Variable length iterations resemble traditional projects. There’s a scope estimate, requirements gathering, and work estimates (and traditional slippage).  Most agile purists immediately scream “WATERFALL”, and there’s some truth to that, but a variable length iteration is comfortable to business.  It’s like Mr. Roger’s house shoes.
  2. They lend themselves to continual shipment of code. If the iterations are variable, developers can focus on one task from start to end, and begin to operate on separate iterative cycles; if you have a development staff of 5, you could theoretically have up to 5 separate iterations going on at the same time; when a developer finishes, they ship their contribution without waiting on the sprint to end.
  3. This fragmentation of the iteration allows for sudden, non-disruptive change. If there are multiple iterations occurring for each independent block of code, and business needs change for one line, then only that line has to shift gears.  There’s no impetus to wait; you stop where you are on the one piece, and move on to the next piece.

Your Choice?

I’d love feedback on this; as I’ve stated from the outset, I’m a fixed length iteration guy.  What did I miss?  Are there other benefits to the Variable Length model that I’m missing?