Professional Development

#DASH2023: Three Things I Learned

Recently attended the 2023 DataDog DASH conference, and it was a lot of fun. This was the first in-person multi-day conference for me in a while (I did crash the PowerBI SQLSaturday Atlanta conference back in February, but attended no sessions and mostly just went to see friends). I had a blast; the conference space was amazing, and the content was thought-provoking. Here’s my key takeaways.:

Take your team to conferences.

I’ve been a manager for over 10 years now, and I’ve struggled to convince upper management to send multiple key individuals to conferences. Thankfully, my director at Grainger is a big believer in education, and not only did he encourage me to send a team, but he also wanted to come as well. It was awesome to have feedback on topics both upstream and downstream. I had team members with a variety of experiences (junior, mid, and senior), and they raised some insightful questions. We split up often, so I still managed to make some new networking contacts but it was good to come back together and discuss new ideas.

Your fires are no worse than anybody else’s fires.

Sometimes being on the production side of the development pipeline you get the feeling that the world is burning and there’s absolutely nothing you can do to save it. While we were at the conference, my slack channels were screaming about several ongoing issues that my team was having to deal with. At times, it’s overwhelming.

But talking to folks at the conference from companies in all types of verticals from health care to automotive to financial to manufacturing, we’re all dealing with the same issues. Changing enterprise systems is hard, and modern development methods often accelerate faster than production systems can respond. Additionally, systems that have been in place for years have often grown connections to other systems in unexpected ways; things break, and they break fast. Observability systems offer hope, but we shouldn’t feel like we’re the only ones struggling with implementing that vision.

AI, AI, AI,AI, AI, AI

Artificial Intelligence and Large Language Models are here. DataDog offers some compelling use cases to accelerate MTTx (Mean Time to Detect, Acknowledge, Respond, Repair, Resolve), but it will take some time to get the plumbing set up for it to provide value. Additionally, users have to be trained on how to do their jobs with a co-pilot. They have to trust the system, and know when to dive deeper than the initial responses. They have to know how to phrase questions in such a way to help the assistant understand them, and they have to understand what the assistant is suggesting. That’s going to take time.

Practical #kanban – map your flow to your work

When I started my new gig a little over a month ago, one of the first problems I wanted to tackle was the flow of work. My team does a lot, and some of the work is very structured and repetitive, whereas other work is much more fluid. It’s really two teams with different foci; they run different sync meetings, and had different work intake processes but they were sharing a board.

The board was a typical software-development style kanban board; board columns included things like:

  • Ready – User stories pulled out of backlog based on business priorities
  • In Progress – work that was ongoing
  • On Hold – work that was waiting for other team input
  • Validate – Work that needed sign-off from someone
  • Closed – recently “done” items

For a fluid SRE team where the project work can be ambiguous, a flow like this is OK (with some caveats that I’ll explain below); In Progress is a “squishy” state for work; you don’t know what the steps are, or how long it’s going to take at a glance. However, SRE work can be “squishy”; you don’t know if you’re going to be writing a script to automate a technical process or helping coordinate a large scale project to implement SLO’s for a new service. Having a card sitting In Progress is just a visual reminder of a task list.

For structured, repeatable work, though, this style of board has limitations; if the steps of being In Progress are known, then you can highlight obstacles in your process by teasing them out. The regulated work team doesn’t know how often projects come in, but when they do, they followed a series of steps for every project (no matter how much data was involved. There was also a lot of regular back-and-forth between this team and client teams early in the process; using the old board, cards were constantly moving back and forth between In Progress and On Hold. It made WIP limits on the board very difficult to track.

We spun up a new board to better represent the flow, focusing on replacing the concept of In Progress with more specific states tailored to the flow. We also identified the primary constraint(s) for each state, be it Client, Team Member, or Systems:

  • Ready: Projects (not single stories) pulled from the backlog in terms of priority (Client)
  • Requirements: A discussion phase with clients to finalize the expectations (Client\Team Member)
  • Prep: Project scripts are developed; data is gathered. (Team Member)
  • Produce: Actual work is in-flight (Systems)
  • Analysis: Write-up from project (Team Member)
  • Validate – Work that needed sign-off from someone (Client)
  • Closed – recently “done” items

Note that WIP can now be defined for multiple states (Prep, Produce, and Analysis); each of these states are governed by a limited number of resources; team members can only have 1 project in the Analysis state at a time, for example. Note also that the cards represent the entire project, not just a task inside a project. It’s an assembly line mentality, and allows us to quickly see where constraints are acting as a bottleneck.

With the regulated workflow removed from the original board, it let us focus on some rules for the fluid SRE work (the caveats referenced above). One of the basic decisions needed for a kanban flow is what does a card represent? For assembly lines, a unit of work could be the whole project, but for fluid software development efforts, it gets really soft.

We decided to go with a timebox method; a unit of work should take no more than a day of focused work. A card could be really small or reasonably large, but if you think a project will take more than a day, then split the work out. This has psychological advantages; it’s far better to see things getting marked as “done” than to just watch a card sit in the In Progress column for weeks.

Another rule we implemented was transferring ownership of the card in the Validate phases back to the original requester. We then started a clock; if a card sat in the Validate phase for more than 7 days, we flipped the ownership back to the team member, and closed the card.

So far, the team seems to be working well with the two new improvements; after 30 days, we should have enough data to see where we can continue to improve the flow.

Trello Power-Ups and Automation

As I posted last time, I’ve been using Trello as a Kanban board to help with my job search. The default functionality for Trello is effective, but there are some free components you can add to minimize the time managing the workflow (so you can focus on the work itself). In Trello, these are broken up into two broad classes:

  • Power-Ups: add-ins that have been developed either by Atlassian or contributors, and
  • Automation: native functionality to Trello that are triggered by actions or dates.

Power-Ups

Adding a Power-Up to an existing board is easy; just push the menu button near the top right of the board and click on the add power-ups button. As you can see in the screenshot, I have two power-ups that I use in my job search board.

Card Age Badge – this puts a colored icon on each card that shows how old that card is. This allows me to track individual job submissions or leads and follow-up on older cards. At this point, I’m mostly using it to indicate when a job has “ghosted” me, and I no longer consider them active applications.

List Limits – an important component of kanban is WIP, and Trello does not capture that by default. This Power-Up lets you set a limit per column (and tracks the number of work items in each column). I use it as Reverse WIP measurement; I want to keep at least 15 applications in progress. List Limits will color the background of a column when you exceed the limit, so for my Applied column I set a limit of 15, and as long as the background is colored, I can focus on other things.

Automation

To access automation, click the Automation menu item next to the Power-Ups button. Because my needs are simple, I only use the Rules feature for my setup. Trello’s implementation of rules is very nice in that it creates natural language statements to show what is to be done and when. I have the following rules:

when a comment is posted to a card by me, move the card to the top of the list

when the red "Rejected" label is added to a card by me, move the card to the top of list "Dead Lead"

when the dark black "Ghost" label is added to a card by me, move the card to the top of list "Dead Lead"
The interface is very easy to set up; each rule is simply a trigger (something that happens) and an action (something that gets done). The three rules above do two things; they allow me to move a card to the top of the stack by simply adding a comment to the card (this helps me to remember to follow up on it), and to move a card to the Dead Lead list when I either get rejected or I think the posting has ghosted me (no response after 14 days).

Board Final* View

Here’s what my board looks like today; as always, there’s room for improvement, but this allows me to focus on the job of finding a job while minimizing the overhead of managing the job search.

Good luck out there! And as always, if you want to see what kinds of roles I’m interested in, check out my LinkedIn profile! Stuart Ainsworth | LinkedIn

Roll With The Punches – #opentowork #sqlfamily #datafam

For a lot of Salesforce employees, January 4, 2023 was not a good day. I was standing in my kitchen, getting my kid ready for school when I got the email at 6:16 AM EST.

Hi Stuart,
As we announced earlier today, we’re reducing our workforce by about 10 percent, mostly over the coming weeks. Unfortunately, as part of this reduction your role is being eliminated.

Shock. Confusion. Acceptance. Thankfully, the severance package is good, and I have time to find the next opportunity.

I had only recently joined Salesforce (on February 7, 2022) after a lengthy career at Jack Henry. I had big plans. I was going places. I liked my job with Tableau on the CI Infrastructure team, and had been exposed to some great ideas, and had big plans to tackle some interesting challenges this year. In fact, I was already planning on writing a version of this blog post in preparation for the upcoming V2MOM process. One of my values can be summed up with the quote from Mike Tyson:

Plans change. They sometimes shatter like glass. But how do you recover when you’re reeling in pain from something? You go back to the fundamentals.

The fundamentals are those skills you’ve built over the years through practice. They should be automatic in times of crisis. Boxers constantly work on the fundamentals of their craft well before a big fight. They can walk into a ring with big plans, but when the blows start landing and the plans fall apart, what saves them is always going to be how well they can marshal the fundamentals.

As a manager in the technology space looking for my next opportunity, I’m going to sharpen the three following fundamentals:

First, my network of peers is large, and filled with good people. I was frankly overwhelmed by the number of people that reached out to ask how they could help and offered up leads. I am grateful, and it just makes me want to continue to invest more in building relationships.

Second, my technical expertise is T-shaped; I’ve got some in-depth knowledge of database architecture and performance tuning, and I’ve been exposed to a lot of ideas for software deployment across the lifecycle. I’m going to continue to work on some weaker areas (Git, Linux, Powershell) while I search for the right opportunity.

Third, I’m a Work From Home master. I’ve been working from home for the last 17 years of my career. You want a manager who understands how to motivate a remote team and build a remote culture? Boom, I can do it. You need somebody who can set deadlines and expectations around work\life balance? That’s me.

I got this.

If you wanna see what my career looks like, add me on LinkedIn: Stuart Ainsworth | LinkedIn

Code Complete

19 years, 3 months.

My last day at Jack Henry and Associates will be February 4, 2022. I am extremely grateful for the opportunities I’ve had. Jack Henry has been very supportive of working from home for the last 14 years, and they’ve fostered me as a learning-center manager. The work-life balance and benefits were extremely tough to beat over the last 19 years. I’ve built and supported some really cool things in the managed security services space. I’ve grown up here.

Here’s a selfie (now) next to a photo of me from 2002 (when I joined JHA).

I’m a dinosaur by IT career standards; honestly, I didn’t think I’d write this blog post anytime soon. However, COVID-19 has changed the way we work, and more companies are creating remote leadership positions.

On February 7th, 2022, I’ll start a new position at Salesforce, working as a Senior Manager of Systems Engineering, supporting the CI/CD infrastructure for Tableau. I’m ecstatic about the opportunity, but I will greatly miss all of my colleagues from JHA (Gladiator), particularly my direct reports.

I still plan on blogging intermittently, and I’ll be active and involved in the community (in fact Salesforce actively encourages volunteerism), so I’m not going anywhere. But yet, I’m moving up, and am looking forward to figuring out the next few challenges in my career.

Let’s build something awesome.

Fix More, Whine Less

I get paralyzed by the thought of NOT doing something correct, so I end up doing nothing. I’ve got to work on that. A three sentence blog post is better than no blog post. Fixing one tiny recurring error is better than just watching the logs fill up. It may be a lost cause, or it may be a moment of hope; whatever.

Just fix something. Something small. A lot of small somethings can make a big something.

To be a #DevOps Leader…

…you have to do less work, and do more to improve work.

I know, it’s been said 1000 different ways already, but it sunk in this week as I struggled to put out the bazillionth little forest fire that had crept up after our last deployment.  It’s so easy to get back in the rhythm of doing whatever it takes to keep the system stable, even if it means sacrificing time with family, projects you want to make headway, general health and well-being… IT work is built on the backs of heroes.

But it isn’t sustainable.  I look back over the last month and wonder what the hell it is I accomplished.  My team is worn out; I’m worn out.  We managed to squeak in a major project, but we had to drop the ball on 100 other little things to get it done.  And, to rub salt in the wound, the new project added new work to the backlog, which just means that we may have won the battle, but it’s an expensive win.

My job as a leader is to make sure people know how much wins like that cost; not to discourage them from trying to make a win, but to make better strategic decisions so that we don’t sacrifice something important in order to get some other important thing done.

Monday’s coming, and I’m going to start over. Again.  And i”m going to keep starting over until things get better.

#DOES18 – Wednesday Takeaways

Last day of the DevOps Enterprise Summit 2018, and I had a few aha! moments that really clicked for me.

  1. Jon Hall’s presentation on swarming was a great practical explanation on how to manage swarms.  My team needs to be more available to Tier 1 and Tier 2 to swarm on issues and resolve them.  It’s not quite the same model as the model Jon presented, but it’s a step.
  2. As an organization, we don’t need a plan so much as we need a set of common principles: commitment to communicate (make work visible to teams, management, customers… etc.), and commitment to best practices in development (SOA, Agile, product not project, etc.).  That’s the first step; once we’ve identified those principles, then there needs to be an establishment of processes that reflect those principles.
  3. Uncertainty is good.  We don’t need to stay in a particular mode in so much as we need to find the problem and solve quickly.

Keynotes are up, if you want to check them out.  Great stuff in there.

#DOES18 – Tuesday Takeaways

Thoughts from yesterday’s DevOps Enterprise Summit in Las Vegas

  1. Managing workflow is different than managing configuration changes or code.  It’s related, but you can have separate systems for individual team’s processes and procedures (i.e., Azure DevOps for developers, ServiceNow for ops), but you want to make work visible to all affected teams.
  2. Making work visible was a theme I heard over and over again; so much of the “throw it over the wall” mentality stems from the fact that ops does magic. 
  3. Processes are necessary, but they must be as light as possible or they won’t meet the need (they get in the way of rapid response).  People often adapt by ditching the process anyway.