Show your Debt

I had an interesting twitter debate a couple weeks ago with a bunch of people I really respect and admire. A good number of them totally disagreed with what I was saying though, so in order to maintain my large ego, I’m going to chalk it up to 140 character limitations and try to expand on my thoughts here.

The crux of the discussion is this. Our team had recently imposed a limit on the number of refactoring or technical debt cards in process at any given time in our system. The reason for this was that we had a couple of iterations where we accomplished some significant changes to the code base which will result in longer term savings and easier maintenance, but the product owner and sales organizations that we work with didn’t really get to see much visible progress on the software for a couple of weeks. We felt that we should spread out the tech debt cards so that the product owner and stakeholders could still get to see progress every week, yet still empower our team to play the tech debt cards as they see fit. So my open question on twitter was:

The consistent push back against my comment was some form of “Why do you have refactoring items on your board?”, as my colleagues mention here:

A lot of people felt that we should just have completed the refactoring as we finished the card. I am not saying that we are not doing refactoring as we are delivering stories and I’m not saying we have an extensive list of refactoring items. We have a solid team with many members who teach and coach others on TDD and the red, green, refactor cycle.  The refactoring items or technical debt that we’re tracking are typically orthogonal to a user story we’re currently working on. Consider a fictional example to illustrate the point.

Say you are building a website that sells index cards. The website may have some domain objects like a user, product, address, etc… You have decided to use validation framework X to validate the data that users enter for these objects.  This works well until you run into the account domain object. Here there is much more extensive acceptance criteria around the validation of credit card information which isn’t easily accomplished using validation framework X. The team completes a spike and determines that validation framework Y is the right approach so the account creation user story is completed with a new validation framework Y. The team collectively decides that having two validation frameworks is confusing and since Y is the superset of X, that they should standardize on using Y throughout the system. What do you do?

I think many of my peers seem to be suggesting that we should just complete the validation framework swap while working on the account creation user story. I don’t like this for a number of reasons. First, the completed account creation user story is delayed from moving through the rest of the system. I’d rather my product owner or exploratory tester start hammering on the new story (and validation framework) before I start expanding this to the rest of the system. We would be able to see any deficiencies in the new framework prior to swapping everything over and avoid a potentially painful conversion back. If the validation framework swap was a day or more worth of work, then we’d also run the risk of not demoing a completed user story to the rest of the team when really that work is done.

I also think the team would be better served to prioritize this work when they feel it is most important. Collectively. Completing large refactoring or technical work in under the guise of a user story is fairly disingenuous to the product owner. Maybe we’re just lucky because we have a product owner who really sees the potential maintenance costs of not doing these refactorings, but we’ve never felt like we needed to be anything but open and honest about the work that lays ahead. Added bonus here is that if a tech debt card, like swapping out your entire validation framework needs to be played, it can be planned in a way that may not cause significant merge pain for the rest of the team.

I think the discussion on twitter may have been largely a semantic argument about what is technical debt versus refactoring, but I want to make sure my points here are clear. Whether calling technical cards a user story, refactoring card, or tech debt card, make sure that they are planned, prioritized, and completed in the same visible manner as other cards in the system. The board should be a mirror of the work, the work shouldn’t have to be forced to fit into the board.

Why we went Kanban

I don’t think any single process is the best fit for all teams. Teams and projects using Scrum, Kanban, Waterfall, or any other custom process you can think of have been successful in the past. In my opinion, the most important thing to keep in mind is that the process should be constantly evolving, similar to the software that you are building.

We recently started a new project off with 1 week iterations as they seemed to be a good balance between a client who was coming from a waterfall background and our team, who was experienced in agile development eager to get show off completed functionality as soon as possible. A few retrospectives ago, we found a number of us were openly questioning why we weren’t using Kanban. So as any adaptive, continually improving team would do, we switched. There are a number of reasons for moving to a pull based model, but here were the top ones for our team.

Estimates were WASTE

We performed our demo, retrospective, and planning on Friday of each week. This was usually accomplished in 4 hours or less but by far, the most significant portion of that time was spent discussing and estimating the stories for the next week. We continually struggled to balance how much discussion was needed to properly understand the solution without totally solving the problem within the discussion. With a team of 4 or more developers, the time spent determining whether or not a card was a 3 pointer or a 5 pointer was not trivial. It was however WASTE, and it leads me to my next point.

Track output, not arbitrary numbers

We had been tracking overall burn down based on the number of high level stories identified anyway so the specific number of points that we accomplished in any given week were just a number. Across projects that I’ve been a part of recently, the points and further, velocity as a weekly total of those points, generally winds up becoming at best an arbitrary number and at worst a point of contention or undo pressure. Clients wind up believing that the team is gaming the number of points and the team has to listen to the endless sighs and groans when a card is estimated at 5 points. Screw the points. Get rid of them and instead track either how many cards are being accomplished over time, or better yet the cycle time of cards.

Planning sessions are no substitute for conversations

Another bad habit that we had gotten into was trying to provide cursory information on a card or in TFS work items, discussing that during the planning session, and then just kicking the feature over to the developers to work on during the week. Cards need to be a promise for a future conversation. Without some level of collaboration between product owner and developer, you really don’t get all of the detail flushed out. Further, doing that conversation in a long meeting a few days prior to starting the work inevitably results in details being lost or having to be rehashed in the form of defects or new cards. Since moving to Kanban, we began to execute conversations about the cards in real time much more often. This has inevitably led to less defects or missing requirements coming back into the system.

What’s going to work?

Teamwork! Our project had a couple iterations where our product owner and QA were frustrated by the lack of time they had to test the application before demoing to end users. Ultimately the team is to blame and should’ve realized that having 8 features partially tested and completed is less valuable than having 4 features DONE. When developers are prohibited from pulling another feature forward until stalled cards in QA are moved to done, they immediately begin helping in areas they might otherwise thumb their nose at. It also facilitates in focusing product owners and BA types on keeping a nice stack of prioritized stories ready for development. 

Better Stories

Kanban doesn’t write your user stories or even provide better visibility into your user stories. It does however, eliminate some boundaries that are counter productive. When a team feels that a user story must be decomposed into chunks that fit nicely into iterations they may begin to create cards that have many interdependencies. If a project has a bunch of interdependencies within user stories, it will also have significant issues with stories not being testable until other stories are done. Is it worse to have a story that spans an arbitrary segment of time or to have 4 developers all stepping on each other trying to complete small portions of a single large story? In my opinion the former is much better and Kanban doesn’t prohibit it.

I don’t believe everyone should change their process to Kanban. I do believe everyone should evolve their process, though. Kanban definitely seems to be the better fit for our group, but I’m confident it will continue to change before our project is done. In the last couple of weeks using it compared to the prior months of doing weekly iterations, we are communicating more, resolving defects in real time, and collaborating with our product owner much more tightly. This works for us, but as always, use your noodle to determine what works for you. Most importantly, don’t be afraid to change it up!

Numbers photo courtesy of c a r a m e l on flickr. User Stories photo courtesy of jakuza on flickr.

Thank You Coimbatore

It’s difficult to put into words what a eye opening experience it’s been coaching our development team in Coimbatore, India over the last three weeks. As with anyone, my view of people and the world is largely limited by lack of experience. An interaction here or there with an India based development team has really been all that I have had to base my experiences on and with this limited experience I’ve come to realize I’ve developed some unrealistic beliefs about IT in India. I’m extremely thankful for the opportunity to dispel many of the myths that I’ve had about this gigantic and growing population of developers in India. Here are the top ones:

Myth #1: Indian Developers are  Subpar

I had assumed that the caliber of development with the team over here would be pretty substandard. I’ve worked with a couple of different offshoring teams and interacted with a number of people who have done the same. The common lessons learned across those interactions have been, don’t give the offshore team any challenging work and relentlessly review the code when it comes back in. Many people I know attribute this to a lack of technical knowledge with the offshore developers. People believe they are only a year or so out of school and that they don’t  have anywhere close to the amount of programming time that their counterparts in the US have.

Maybe this holds true for bigger segments of developers, but we were extremely fortunate to have six really talented developers on our team. Additionally, within the sixty developer company where they work they are more the norm than the exception. Each team member was extremely comfortable in C#, .NET, SQL, and a number of other technical skills. When technologies were new to the team, they were able to quickly and effectively pick up working knowledge of the tools with a little help from us. We quickly changed our plan from trying to teach technical skill to teaching craft and improving process within our first couple days here. In my opinion the difference between the developers I work with back home and the ones over here is mainly just experience with test driving development.

Myth #2: Communication Will be Easy

Everyone speaks English right? Well at least a country that was colonized by England speaks English, right? Uh, no. The difference between the Indians who have frequently interacted with Americans and those who haven’t was astounding. Communicating with our taxi driver was largely primitive grunting and pointing. There were two members of our group were fluent in English and had little trouble understanding us when we slowed our pace of speech down to a reasonable level. The other five members of the team had difficulty understanding us unless we were gesturing, speaking very slowly and enunciating very well. Neither Tim nor I were able to master speaking to this group effectively.

We did find however that everyone on our team was able to read English and all of their code is written in English so we were able to use that as a common denominator. With a little bit of translation help then we were off and running. We’d favor walking through code on a projector as a team and showing by example the practices that we were trying to teach. Ideas and concepts that we felt were extremely important were captured as reference in our wiki on GitHub. It’s a difficult problem to solve when so much of our success relies on frequent, open and honest communication. As we move forward with 8000 miles between us, we’ll leverage IM, shared living documents, and timely feedback with daily standups, weekly retrospectives and demos.

Myth #3: Cultural Differences are not that Wide

I knew that there would be cultural differences but I really had no idea how different the people are here from back home. I had brushed up on the differences with a great book, Speaking of India by Craig Storti, but to understand the impact they have they really must be experienced. The most challenging one to overcome was the rigid hierarchy in the workplace. When the owner of the company was in the room, no team member spoke except for the team lead and he would only reply with a “Yes” or “Yes, sir”. When the team was asked for feedback they would defer to the team lead every time. If he was out of the room, they would defer to the “secondary team lead”.

This rigid hierarchy was stifling communication which as I mentioned before must be open, direct and honest. We didn’t really try to overcome cultural standards that have been ingrained in this group since the beginning of their lives. Instead we picked our opportunities to have direct conversation with individuals and to solicit feedback from the group as much as possible when they were their most honest (bosses out of the room).

Myth #4: Little Effect can be had in Three Weeks

I’ve worked with a number of really good trainers, coaches, and teachers of Agile techniques and everyone routinely agrees that to get somewhat proficient at Test Driving, Refactoring, and Emergent Design takes months.  I have no intention of being in India, away from my family for months, so the question of how much change we could enact in three weeks was really concerning. I’m not foolish or egotistical enough to think that the developers we’ve worked with are now proficient at TDD, Refactoring, or anything else that we’ve tried to teach them. Further, I know that they will start to regress once we’ve returned to the states. I do believe, however, that these guys already had an inherent thirst for knowledge. We were able to show them the benefits of every practice we talked about and you could see in their eyes and hear in their conversations that they were excited and eager to continue the voyage. Will they become proficient in the same time as if we were with them? Probably not. Will they continue down the path? Definitely. These guys already had the spark of interest going, the most important thing that we were able to accomplish over here was to just fan the flame.

In all, this trip has taught me to question all of my preconceived notions about people. I am hopefully returning from India, humbled, grateful, and much more respectful of the great country and it’s people. Despite conditions that many Americans would find below standard, the people of India are as welcoming and hospitable as any I have ever met. I am thankful for the opportunity to learn more about their culture and I look forward to continuing to build some great software with this team.

Senthil, Antony, Chenthil, Perumal, Sathish, Govind, and Priya, you were all extremely gracious and patient with the boisterous Americans. You made us feel very much at home when we were in need and we will forever be thankful!

தங்க யு கோயம்புத்தூர்

Hire Craft over Technology

Consider this blog post an open plea with hiring managers and recruiters. The vast majority of you focus on the wrong thing when trying to hire developers. Really good developers are exponentially more productive than mediocre ones, but you can’t tell the difference because you are looking at the wrong thing. If you disagree, please ask yourself: “What is the first thing you assess in a candidate when considering them for a project?” I’m sure a few will answer differently but by and large most will say that they are trying to make sure that someone has adequate experience with the technologies in use on the project. I’ve seen hundreds of good, bad and mediocre developers at work and I’m here to tell you that the focus on technical skill should be secondary. Focus instead on hiring developers who embrace quality, take pride in their work, and consistently practice their skill. These developers are craftsmen, not programmers.

Taken at a glance, it doesn’t seem to be a bad thing to focus on technology. Technical skill is mandatory for most development positions. Most hiring managers and recruiters couldn’t fathom bringing someone onto a project written in C# if they professed a novice or rudimentary knowledge of C#. The problem with this mentality however is that there are a number of C# developers who are proficient with the language, yet build very costly systems due to the quality of their code. A developer may know Java inside and out, but it doesn’t mean they know how to build software.

Put yourself in a different mindset, that of a person hiring potential carpenters to construct your new home. If you were interviewing the carpenters would you quiz them continually on their knowledge of a power nailer? No, you’d ask them more appropriately about the differences in constructing load bearing walls from non-load bearing. How to frame a doorway. What procedures they use for ensuring a square corner and flat walls. These questions are all striving to assess what level of quality the carpenter builds into their final product. Why do we not assess the same thing with developers?

I believe it namely boils down to either misperception or laziness.

Misperception

The main misperception in hiring managers is that technical skill equals speed. It may in fact equate to short term speed, but without an additional focus on quality the legacy codebase quickly becomes a giant time sink that is difficult to change. Studies have shown that the vast percentage of our project costs are in the maintenance phase, not the construction phase, yet how many dollars have been lost by companies emphasize saving time on the 20% at the expense of the 80%? Unless you are constructing throwaway code or something that will be extremely short lived, do not focus on the speed to build. Focus instead on the quality and your overall speed will increase. Focus on your speed and your overall quality will decrease.

Laziness

I actually think the misperception referenced above is being debunked more and more by hiring managers every day. Sadly though, many recruiters and hiring managers I know are in a rut where they still screen development candidates exactly as they did many years ago. They build simple quizzes to screen for technical knowledge, followed by phone interviews that screen for technical skill and the good ones even follow this up with a programming interview that again determines technical knowledge. Whether it’s ignorance, apathy, or laziness I’m not sure, but if these groups want to see an increase in overall quality (and therefore speed), they need to start asking the difficult questions focused on craft, not technology. I still do a decent number of pairing interviews and I never dictate that someone use a specific language in the interview process. I think my most memorable candidate actually used a language and editor that he wasn’t that familiar with and still you could tell that he had intimate knowledge of TDD and emergent design. I immediately wanted to hire him for any project that we had available and I knew he would be productive and successful regardless of the environment. Can you say the same for the C# developer who has 15 years of experience which are really 1 year projects repeated 15 times?

Craft is not something that is easily learned. K. Anders Ericsson noted that it takes 10,000 hours of targeted practice to reach mastery. If you happen to get one of those passionate developers who have 10,000 hours under their belt, then consider yourself lucky. If you can’t find those people then do the next best thing and hire someone who truly gives a shit and is practicing their craft. You’ll know you’ve found them when they talk much more about quality than technology.

Hammer photo courtesy of thefixer on flickr

Things I Have Learned About Driving in India

 

In our first few days over here in India, we’ve been privy to some of the most exhilarating car rides I’ve ever been a part of. By exhilarating, I mean that we have actually almost had 3 head on collisions and we’ve stopped in front of oncoming traffic to take pictures while we were passing an ox drawn cart. We really believe that our driver could have some success back in the states, so we’re actively recruiting him for NASCAR. Here are a few of the tidbits we’ve learned from this expert:

1. Lanes are merely a suggestion and should rarely be followed. Instead, find the best opening in the road to position you for massive future  acceleration.
2. If you are behind a bus with 230 people in it, pass carefully. They always have the right away.
3. If you are behind an elephant in the road, pass carefully. They always have the right away. Also, do not tailgate.
4. Horns are by far the best means to communicate that you are on someone’s left, on someone’s right, are about to hit someone, are going to pass someone, are going to ram someone if they don’t let you pass, or that you are sorry for accidentally running into someone.
5. If horns are not conveying your intent properly, resort to flipping your headlights at them.
6. Motorcycles are by far the easiest way to get around. And they can safely transport a family of four. And they can transport 18 month old children.
7. If you are not simultaneously pressing the gas and the break at some point during your drive, then you are doing it wrong.
8. When encountering a traffic light which counts down until you can go, do not be a loser and wait for 0. Go at 5 at a minimum, 10 if you are feeling lucky.

Wish us luck for the next few weeks. I’ll have the unbridled joy of this little guy for every ride…

Trials and Tribulations of getting Jenkins on Windows pulling from GitHub

I spent the better part of the last two days getting a Jenkins build which runs on a Windows 2008 server to pull from GitHub. You might think, “Wow, Todd really must be an (idiot|rube|dolt|moron|manager) to need 2 days to setup a CI server”. Well, you might be right, but even a simpleton like myself can google the shit out of a problem and come up with some type of frankensteined solution in a day or less. Apparently though, I am one of 5 people on this planet to actually use the cluster of Jenkins/Windows/Github. In case there are more of you out there, I thought I would try to spell out exactly what I did so that you don’t have to suffer (and so that when I forget how I got this thing to work and the server needs to be rebuilt in a year I can do it).

Step 0: I’m assuming you already have a Windows server with Jenkins up and running on it. I’m also going to assume that you installed it as a service. if you aren’t there yet, follow the instructions here: https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+as+a+Windows+service

Step 1: Install yourself some msysgit from here: http://code.google.com/p/msysgit/

Step 2:  Head to Manage Plugins from the main Jenkins menu, select the Git plugin from the list of available plugins, and install that puppy. You’ll need to restart the service for this to take effect.

Step 3: I had to explicitly configure Jenkins to use the git.cmd instead of git.exe. I believe this properly sets the HOME variable so that your ssh and netrc files can be located when Jenkins fires off a build. From the Manage Jenkins link at the top level, you should have the option of adding a git installation. Provide the full path to the <GIT_HOME>/cmd/git.cmd file in this line as shown here:

Step 4: Generate and upload an ssh key for your github account. Make sure you are logged in as the same user that the Jenkins service runs under. Good instructions here: http://help.github.com/win-set-up-git/

Step 5: Configure a _netrc file and store it in the home directory of the user Jenkins runs under. My _netrc file looks like this:

machine github.com
login pillarci
password notmypassword

Step 6: Set an environment variable for HTTPS_PROXY if you have a firewall.

Step 7: Test that you can pull down from the command line. Given your repository name, you should be able to go to the command line and run something like

git clone https://github.com/SomeOrg/SomeRepo.git

Step 8: Configure your build on Jenkins. Please note that the repository name should not have the username in it and it should be using the https url, not the git url. Something like this:

Step 9: Kick that build off and pray. At this step I had a number of obtuse error messages and hung builds. If either of those instances occur with you, proceed to steps 10-47.

Possible Step 10 – 47: Check for a .git folder in your workspace without a sibling directory containing your repo. Nuke it if so and retry the build.

That should be all you need to get up and running. Again, I had some issues, many of which were the fault of the user, many due to the lack of info out on the internets. Luckily I work with a really smart guy named Justin Searls and he had already laid a bunch of this pavement for me. Happy GitHubbing!

Step 4.5 – Spring DM Extender Logging

I’m going to sneak in a bonus blog post in the Enterprise OSGi series as I think some people may have had issues in getting the last step to work and will be relatively stuck without a little help in debugging what the heck is going on with Spring DM Extender. Enabling logging on a bundle that you did not write may seem tricky. Some may even attempt to download the source, tweak it and re-bundle the extender. A cool feature of OSGi is the ability to extend bundles that you’ve not written by developing a Fragment. Fragments allow bundles to be closed for modification, but open for extension. They are typically used to customize web bundles with separate UI skins (as we’ll see later), internationalization, separate OS installations, and a few other niche cases. In our case, we’ll leverage a fragment to customize the log level of the dm extender so that we can figure out what is going on when say, our bundle doesn’t start and there are absolutely zero messages explaining why. Let’s get started by creating another bundle

I passed another couple of flags to maven on this command to ensure that it doesn’t create any internal or interface classes that we’ll have to delete. In our simple case of enabling some logging, we really only need to add a single log4j.properties file to get the magic to happen. I placed it in a src/main/resources/ to keep in line with a typical maven project and added some simple verbose logging configuration.

log4j.rootLogger=info, A

log4j.appender.A=org.apache.log4j.ConsoleAppender

log4j.appender.A.layout=org.apache.log4j.PatternLayout

log4j.appender.A.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

The last order of business is to adjust the manifest to attach this fragment to a host bundle. As usual, we accomplish this task by adding the following line to the BND file

Fragment-Host: com.springsource.org.apache.log4j

Compiling the project and running pax-provision should provide a slew of information regarding the process spring uses to look for bundles and resolve dependencies

If you’ve had issues, this will hopefully provide some clues as to why the dm extender was not able to find your dependencies. My most frequent mistake is to misspell the directory containing the context files. In the next post, we’ll resume our regularly scheduled programming by providing a database backed persistence layer in our application.

Examples, as always available on github.