MIT Social Media

My lack of blog posts here is justified, I assure you.

Over the last six months I have been working with my esteemed colleagues Rafael Maranon, Azamat Abdymomunov, and Charles Atencio, to get the MIT Social Media Club up and Running.. Great things will come in 2011. For now, you can read some of my musings on Social Media here.

The Hidden Cost of Agile Software Development

Perhaps my favorite class so far at MIT is the “Business of Software” MIT Sloan class taught by Professor Michael Cusumano. This is a seminar-style course aimed at anyone interested in the business of software (products and services) and software-based digital platforms. Anyone interested in working for, founding, or analyzing and consulting for companies relying on software technology or digital platforms such as mobile phones should find this course of interest. Many of the issues we discussed were also highly relevant for companies whose businesses are heavily dependent on software, such as in e-business, digital content and publishing, financial services, or embedded software for industrial applications. The class material was augmented by visits from various CEOs and software professionals, who joined our classes each week to provide a first person view of any given software sector. The list of visitors was long and included: Brian Halligan (CEO of Hubspot), Ted Schadler (Forrester Research), Harvard Business School Prof. Tom Eisenmann (platforms strategy concepts), Jeremy Allaire (CEO of Brightcove and the innovator behind macromedia Flash), Zia Yusuf (Stanford D-School, serial entrepreneur, and former SAP executive VP for platform ecosystem development) and so on..

One topic near and dear to me that we examined was that of large-scale software development approaches. Specifically, we compared the Waterfall and the Iterative software development approaches. We discussed the pros and cons of each, and developed an understanding of the internal (firm) and external (market) conditions that may call for one versus the other. I will not go into that discussion at this time; these two development styles have been compared and pitted against each other for decades now...

However, it seemed to me that the overall impression that people have is that an Iterative/Agile model of software development was the superior approach in today’s market, as more and more software development organizations are adopting or transitioning to the Agile/iterative approach, especially the software behemoth du jour: Google. This may be true, but Agile software development comes with it’s own unique issues and complexities that require special attention to Software Configuration Management (SCM), and therefore added costs. Note that Configuration Management is also one of the Key Process Areas (KPAs) in the Capability Maturity Model (CMM) for a level 2 organization. Another way to put this is: Agile may allow you to develop a software product faster, but not necessarily cheaper. I argue that in fact Agile requires more stringent SCM and planning which incurs more management effort and cost. Let’s take a closer look at some of the SCM issues that a manager might face when adopting an Agile development model. My point of view is based on my personal experience with large distributed software system development, having spent the last 12 years in software development for aerospace and defense systems at Raytheon Network Centric Systems and Raytheon Integrated Defense System.

Version Control:
I had never heard why Windows Longhorn was a failure until I heard about the way in which the Windows group had operated: nightly builds were constantly breaking as various component teams would “check-in” their changes to the common code base . “Making even small changes in one part of the product led to unpredictable and destabilizing consequences in other parts since most of the components were tied together in complex and unpredictable ways” (M. Cusumano, 2006, What Lies ahead for Microsoft and Windows.)
This reminded me of one of the major challenges I face in my professional life when managing multiple teams of developers working simultaneously on several “threads” in an iterative manner (we call features or capabilities threads – multiple threads of software development are woven into a particular release.) Many a developer has known the shame of walking in to the office early in the morning and having others give him/her the “you broke the nightly build” look.
The answer to this problem is Version Control: you allow parallel development activities to occur on “branches” and “sub-branches” that fork off from an exiting baseline. Each team has its own “view”. This allows each team to develop their piece in a stable environment. Later, all of the threads are “merged” together into a common baseline. In the example in figure 1 below, the same file “prog.c” is modified by two threads of development “db_optimize” and “pat_usability”; db_optimize is merged back to the main baseline and is part of release 1.4.

Figure 1 – Example Version Tree (source: IBM Software Information Center)

Version Control is just one of the tools for managing the configuration of one of the essential work products of a software development organization: source code files. Many products exist in the industry to allow for the implementation of such version control strategies. Personally I am quite experienced with the Rational ClearCase suite of tools which provide us, among other things, with a version tree browser, a “diff tool” which highlights the differences between any two given versions of a file, and a merge tool which facilitates the reconciliation of conflicts between two versions of a given file. Another popular tool is CVS (Concurrent Versions System) which is Open Source and supports many environments. I must also pay homage here to the grandfather of them all, SCCS (Source Code Control System,) which I used in the old UNIX command-line days. While it is considered obsolete today, I still use it to control versions of my intranet webpage.
To be clear, these Version Control concepts are not unique to Agile development. As I hinted with my UNIX command-line reference, Version Control concepts predate Agile, and are used also in Waterfall models.

My point however is that Agile calls for even more stringent SCM, and that the complexities of your version tree and therefore of your merge efforts are going to increase dramatically as you create more branches and sub-branches to cater to more parallel and iterative cycles. In such an environment, you have to plan and coordinate the merging of the sub-threads and perform some higher level testing of your functionality.

A common pitfall for a software product development manager is to think that deveklopment work is complete when all of the threads have each individually completed their iterations. This is not the case! One has to plan for problems and account for some refactoring or rework that may arise when merge conflicts are irreconcilable.

Having managed a large team working in this fashion for the lasts few years, here are some rules of thumb that I now live by:
1- Merge early and merge often: The last thing you want to do is merge everything near the end of your schedule, right before a big test or sell-off, only to encounter a host of conflicts that break your schedule and possibly your reputation. It is much better to identify conflicts and address them early.
2- Make sure your team understands the branching strategy and are all on the same page. Before the start of every release’s development cycle, I hold a “SCM Strategy” meeting with all of the development leads, as well as the integration team, to go over a branching plan and agree to get team buy-in.
3- Have a CM-meister. In our case, it was very hard convincing senior management that we needed an individual working full time to manage our branches, views, etc. and coordinate merges. This person needed to be an expert ClearCase user. We never got such a person, so it became task by committee.
4- Always plan for SCM activities in your project schedule. In the past, our Integrated Master Schedule only included tasks for requirements analysis, design, code, etc up to sell-off but had nothing for merges, syncs, etc. This resulted in the “development is complete, but why can’t we start tests yet?” question as we scrambled to reconcile merge conflicts and get the software to run. I have now learned to plan and leave room for SCM – the size of this effort is calculated as a function of the number of threads in the release and the number of other active releases (bug fixes are ported across releases, further complicating things.)

-->Proper version control helps reign in some of the chaos that may result from Agile development, however its complexities translate directly into added costs for the program as a whole.

Interface Control:
Take the example of an interface changing on one branch, while procedure calls to the old interface were added in another branch, resulting in a code merge conflict that has to be remedied when the two threads come together. Such situations can be avoided by using a strict change control mechanism. In our case, interfaces between components, as well as common/shared data structures, are treated as a contract between software subsystems. Any change of contract has to go through a formal review and authorization process. In order to change an interface, one must submit an Interface Change Request (ICR.) All of the stakeholders are then tasked to review and approve the change, and the ICR is reworked until it is approved – then the implementation of the change is coordinated across the threads of development. This process is driven/managed by the Software Change Control Board (SCCB.) The workload of the SCCB also increases dramatically when multiple concurrent threads of development are occurring, and what once used to be a part-time role for a few senior software architects becomes a full-time responsibility.

--> Slow turn-around by the SCCB can a bottleneck to development and eat heavily into your schedule as teams of development wait for concurrence for an interface change.

Change Management/Request tools:
Now that you have many concurrent threads of development and interface changes to manage, how are you ever going to document all of this activity, compile objective evidence for CMMI auditors, or even know what is going on in any given release? Clearly you need a database to house information related to changes being made to the software, documentation, and other work products. You also need a place to document code changes, and manage the authorization and review process.

This is where change management tools come in. My company has its own proprietary tool, but there are many other tools on the market such as Borland’s StartTeam, or even 37Signals’ Basecamp.
I highly recommend using such tools to track all changes to your software. This becomes more important as the size of the project increases. In fact this is an imperative for any self-respecting software organization. These tools are also very valuable as defect metrics collection mechanisms and can provide you with a wealth of information. They allow you to collect important statistics and for example identify problem areas that need special attention. These type of statistics recently led me to a finding that our process for performing SWIT was inadequate as a large and disproportionate number of defects that should have been caught in that phase were escaping it and caught in subsequent phases of the program (and we all know the sooner you find it the cheaper it is to fix.)
Change tracking also adds another dimension of documentation: beyond well-commented code, the iTracker system allows me to further investigate why certain changes to the code might have been made. File versions on the version tree are labeled with corresponding CR numbers, and now you have a full story of the evolution of any given file.

--> Again, this is not a new concept that comes with Agile development. However, as before, the activity level increases and the number of concurrent development teams explodes, the burden placed on the people who have to manage Change Request system increases exponentially.

Conclusion
One may assume that Agile means “less overhead” however I have learned that when you take down some of the barriers and allow the engineering/programming part of software development to happen in a more creative, agile and iterative manner, there is a higher cost associated on the management side of this style of development. The process requires control at a higher level to reign in the inevitable chaos and avoid the spaghetti-code phenomenon. While an Agile approach may result in quicker beta releases and more innovative software features, if not managed properly it can turn into a nightmare of broken nightly builds and never-ending refactoring (Longhorn, anyone?) With proper SCM tools and techniques, much of the risk can be mitigated, albeit at a higher cost.

Think Global. Act Local - after the scavenger hunt, of course.

Here we are in mid-April and I still haven't blogged about one of my favorite events that happened back in January... Just goes to show you how little time I have to reminisce when busy with a full System Design Management course load and while maintaining part time employment as a senior engineer at Raytheon.

It's only been three months since IAP (Independent Activities Period ... MIT lingo) during which the cohort of 2010 was put through "SDM bootcamp." Essentially, the month of January was dominated by cohort-building activities which consisted of early morning till late night (much to the chagrin of my wife and 18 month old son.) These sessions included Lectures, Workshops, and several team-based Design Challenges.

The program was generous enough to provide catered breakfast, lunch, and dinner during the whole month (In retrospect this was probably because they knew that we wouldn't even have time to run up the street for a sandwich) and I even gained a few pounds, prompting me to organize the SDM "Biggest Loser" weight loss contest with some of my fellow "gravitationally challenged" classmates... more on that later.

The Design Challenges were particularly interesting .. Here's how they went: The cohort was broken up into teams and assigned challengeas which were to be completed in a week or two. Teams were intentionally diverse and optimized for maximum conflict - I am kidding, but interesting team dynamics ensued -- a paranoid person might have thought that we were part of some social experiment and that there was a mad scientist observing us through a two-way mirror in a Zimbardo-like experiment.

We had three Challenges , each interesting enough to deserve its own blog post - Like a reverse polish notation calculator, I will pop my memory stack and present these backwards, starting with the one that's most recent and fresh in my mind : DC3.

For starters, DC3 turned out to be two challenges in the guise of one:

First, there was a scavenger hunt! This allowed, nay forced, us to discover the MIT campus and surroundings by looking for a mix of arcane and historic MIT-related landmarks and curiosities. The list of these is way too long to publish here. There were 50 items. In some cases you had to get a picture of the whole team in front of these places/items. I admit that, after having had knee surgery in December, the thought of running up and down the Charles river looking for "Nerd Crossing" didn't quite excite me, but I actually went through with "most" of it : we thought that we could meet the letter of the challenge requirements by photoshopping (hmm spell check didn't complain about "photoshopping," is it an official word now?) our team into the picture.

A few pictures from that hunt are below... we had to provide a charcoal rubbing of the plaque used by MIT students for good luck... Turns out that Rubbing the nose of the George Eastman plaque in Building 6 is supposed to bring good luck.

Another required item: picture of the team in front of the bar at the Muddy Charles Pub

There were several bar and pub-related scavenger hunt items, to our delight:)

Below, we found two Nerd Crossings:

After the fun and games it was time to get down to business. The real challenge!

The name of it was " Think Global. Act Local."

Each team gets a large systems issue to research. They must make a connection to a local organization. (Local organizations must be non-profit.) Each team will make a final presentation - a “sales pitch” for why SDM should support the organization they advocate.

Prizes for will be given for the top presentation in each category plus an audience choice award. Each prize is a $500 donation to the non-profit organization presented by the team.

Topics are intentionally broad. Each team will have to determine how they will decompose and or define the problem. E.g. For large scale disaster recovery, the team can identify major challenges, and choose to focus on one such as how to meet the need for potable water after a disaster. Topics :

· Improving Large Scale Disaster Recovery (e.g. earthquake, tsunami, hurricane)

· Enabling economic development with minimal carbon impact

· Promoting STEM subjects and increase participation at higher education levels

My team, team number 2 (or "team deuce" ) was assigned "Enabling economic development with minimal carbon impact."

It was amazing to see all of the team presentations and the ideas that were generated in just one week!

In my next blog post I will tell you all about my own DC3 experience in finding, connecting with, and pitching a non-profit that is all about helping build a sustainable future - How about that for a cliff hanger?

Toolkits, Platforms, Product Lines, and System Design & Management

Today I meant to write about Platforms, Product Lines, Product Line Architectures, and so on... I was especially fueled to do so after a particularly fruitful day back at the office, where I and a few other engineers at Raytheon NCS attended a rather interesting presentation describing a joint effort between Raytheon (my employer) and Microsoft termed "ASPEN" (Advanced Software Productivity Environments, I believe.) I hate to be a tease - but since it is almost all company proprietary, all I can tell you is that it had me thinking all day about toolkits, platforms and product lines... As if I needed that, after a week during which: a) "Mr Platforms" himself, Marc H Meyer (author of "The Power of Product Platforms" and "Fast Path to Corporate Growth") was a guest lecturer in one class. b) I started reading professor Cusumano's book, "Platform Leadership" c) Also started writing a paper related to software toolkits for Von Hippel's class "User Centered Innovation in the Internet Age."

But since I can't share my Raytheon-related thoughts with you, you're just going to have to trust me that it's some really cool stuff. Instead let me turn my ramblings towards the System Design & Management program (SDM) and try to start answering the question, nay cliff-hanger, I closed my first blog post with... It had to do with why I picked SDM -- Let me start with three simple facts:

1) System Design & Management is co-sponsored by the MIT Sloan School of Management and the School of Engineering (SoE), and resides within the Engineering Systems Division.


2) SDM is not an MBA program, but a program that offers a master's in engineering *and* management.

3) SDM Fellows can take advantage of the best of all worlds via a rich curriculum in engineering and management provided by Sloan and the SoE and even the Harvard Business School.

Innovation and Disruptive Technologies

Today, in my Technology Strategy class, I was delighted by our guest speaker for the day: Irving-Wladawsky-Berger. He was here to tell us about IBM's survival story, which I will not attempt to convey here as he does a much better job of it in his own blog. In the late 80s and 90s, he led the effort to formulate IBM’s Internet strategy and to develop and bring to market leading-edge Internet technologies that could be integrated into IBM’s mainstream business. He subsequently led a number of companywide initiatives like Linux, Grid Computing and the On Demand Business initiative. Under the leadership the Lou Gerstener, these strategies pulled IBM back from the brink of extinction. Gerstner is credited with saving IBM from going out of business in the early 1990s. In his memoir, Who Says Elephants Can't Dance?, he describes his arrival at the company in April 1993, when an active plan was in place to disaggregate the company. The prevailing wisdom of the time held that IBM's core mainframe business was headed for obsolescence.

This was a great lecture and I wish we had hd even more time to pick Irving's brain. I came away from this class with the following points in mind:
- Innovation is not only about adapting to changes in the marketplace/environment, but it is also about positioning yourself such that those changes work in your favor.
- Today's high level of attention to Innovation is attributable to the fact that the world is changing at an ever-increasing rate.
- One should hold a Darwinian view of competition. Most companies do not buck the trend.

- Most companies die because they either failed to Innovate or chose to do so too late in the game.

- Sometimes managers, no matter ow highly paid, are in over their heads.

Some sobering statistics:
- Of the 172 companies that have appeared on the Fortune 500 list between 1955 and 1995, only 5 percent grew their revenue above the overall inflation rate.
- Only 16 percent of 1,008 companies tracked from 1962 to 1998 survived.
- The average lifetime of a firm is now one-third of what it was in the 1930s; on average, large companies in North America and Europe now fail within just 20 years .

What must a business do in order to not just survive but thrive when facing a disruptive innovation?
. . . embrace the disruptive innovative . . .
. . . become the disruptors rather than the disruptees . . .
. . . and ultimately win over the established competitors . . .

Much more on Innovation and Technology strategy later. Have I mentioned that my professor for this class is none other than the James Utterback, author of "Mastering the Dynamics of Innovation," and of "Design-Inspired Innovation" ??? Jim inspired a whole generation of Systems and Innovation thinkers, including Clayton Christensen (a former student of Jim's, and author of "the Innovator's Dilemma") who will also be making a guest appearance in our class. Every single book or article I have come accross which talks about Innovation, Corporate Strategy, or any sub-topic therein (e.g. Dynamic Capabilities) cite or reference Jim Utterback's work. Personally, I put him right up there with Joseph Schumpeter.

We all more-or-less had this idea of "staying relevant" and are pretty aware that companies must innovate to stay alive. I think that these days, most self-respecting large corporations have some sort of Innovation strategy (mine does.) But coming from a pure engineering "design it, buil it, test it, and manage the process" background, this class has broadened my horizons and opened my eyes to a whole new world of Innovation Theory, Disruptive Technologies, Creative Destruction, and how enterprises must take these into account. I am sure that I will be blogging much, much more about Innovation before my time is up.

BlogDive

So, here I am: Recently volunteered-for/appointed-to the position of Communications Lead on the MIT System Design and Management Industrial Relations Committee (IRC.) My 'big idea' for spreading the word about the System Design and Management program was "publish, publish, publish..." Encouraging my colleagues in the 2010 cohort to blog and share their ideas, especially with respect to their experiences at MIT. I must note that this notion of "Inbound Marketing" was implanted in my brain by Brian Halligan, HubSpot CEO and founder, whom I met and briefly talked to after a guest appearance in Professor Cusumano's class, The Business of Software (get the book, it's good).

Not wanting to fall into the "do as I say, not as I do" category, I have decided to put my money where my mouth is and start my own blog. The first hurdle encountered when deciding to do so is "where am I going to find the time?" Well, let's just take the dive and see how long we can hold our breath... I think this s going to be a useful exercise - I needed somewhere to brain dump all the crazy ideas and novel notions floating around my head. Or maybe I will publish some of the papers I end up writing for some of my classes. One thing is for sure: I will use this to blog about all the interesting people I get to meet and hear in class, and all of the cool concepts and themes that I encounter along the way..

This year I started the System Design and Management program, a graduate program at MIT's Engineering Systems Division and the Sloan School of Management. There are around 50 others in my cohort, and from day one I was amazed (as I am to this day) with the depth and breadth of experience, culture, and background of this cohort. You name it, and someone in the System Design and Management program knows it. We've got representation from various engineering disciplines. We've got fellows from numerous industries. We're from all over the world. We have on average around 10 years of professional experience... Needless to say there is never a dull moment here.

So why MIT? That's easy and self-explanatory. But why did I choose the System Design and Management program? How does it differ from a traditional MBA? Those are questions I cannot answer in one post. But over the course of time it will become painfully obvious, as over the next two year I use this platform to share my experiences. But for now, I will borrow my favorite line from a presentation I attended last year: "System Design and Management is for those who want to lead engineering, not leave it."