One thing that kind of bothered me in my recent CSM training course was a consultant's comment that he didn't feel any need to try to "sell" Scrum to resistant management, implying that, if they didn't buy into it, it was their loss -- but winning over management is almost certainly a big part of what developers in that organization are expecting a consultant to HELP THEM WITH!

Just as management needs some level of developer "buy-in" for a project to have any chance of success, anyone hoping to transition a traditional organization to Agile needs to be able to SELL the concept, in terms a traditional business manager can understand. "Just trust us" is the arrogant Agile counterpart to management's "Just do it" -- and about as effective.

Agile is a TOUGH SELL! You're asking what may be a whole STACK of non-programming middle managers to abandon established practices, to override PMP-influenced resistance (which may finally be diminishing, now that the Project Management Institute has begun to take an interest in Agile), to buy into subversive concepts like "self-organizing teams", based on rationale they typically don't have the first-hand development experience to understand... knowing that if it all goes wrong, THEY'RE the ones who'll be held accountable.

Emphasizing Agile's ability to deliver better quality software, much faster, using fewer resources (better, faster, cheaper) is critical to convincing management to at least give Agile a TRY, but avoiding an outcome that permanently DISCREDITS Agile requires knowing where you can compromise the "purity" of your model, while holding the line on items necessary for Agile to actually produce benefits. Setting up a system of code review and mentoring, to ensure code is kept clean, maintainable, and extensible, is more important than insisting a large team of developers with poor dev. skills write sloppy unit tests to support their convoluted code. Generating some amount of unecessary documentation is preferrable to having your product fail a customer audit, or to have your project remain unfunded by corporate managers who don't buy into your "trust us" argument.

To offer reassurance and manage expectations, you'll need to provide justification for the changes that will need to take place, be able to offer credible alternatives to traditional methods of maintaining accountability, and openly acknowledge potential areas of conflict (for example, Agile organizations typically require a smaller number of more highly-skilled developers, which may eventually lead to issues of what to do with long-time employees who are under-performing in a department which is over-staffed, under the new standard).

By choosing your battles, and focusing on a few key issues like obsessive attention to code quality, replacing Big Design Up-Front with a commitment to iterative "refinement" of an initial outline spec., sqeezing in at least SOME amount of regular customer proxy feeback (usually from a busy marketing rep. who doesn't understand why you can't just go off and build the entire app. as specified), limiting the number of items under development to force developers out of their silos and encourage them to collaborate, holding regular informal reviews to try to improve the dev. process based on observed results... you are surreptitiously "infecting" the organization with traces of Agile, so you can break down resistance and eventually take over the host.

Don't make the mistake of the last organization I was in, where we managed to get buy-in on adopting specific Agile PRACTICES (continuous integration, unit testing...), but failed utterly to get commitment to the more important Agile PRINCIPLES. Management still insisted that we begin a three year project by dividing all work into "tasks", which were distributed to 15+ developers in two globally-distributed teams to "estimate", despite the fact that only three of us had significant first-hand experience with the technologies we would be using, and many developers suffered from questionable coding skills... with disasterous results.

Kanban is interesting here again, because it seems better able to overcome initial management and developer resistence, by requiring only very minor changes to ANY existing development process (including waterfall), with teams reportedly adopting more and more agile techniques naturally, over the course of the project, as productivity improvements become obvious to everyone involved.

"The (Pirate's Code) is more what you'd call 'guidelines' than actual rules."

- Captain Barbossa, Agile pragmatist, Pirates of the Carribean, 2003

One of the more confusing aspects of Scrum for me has been the fact that "adapting" Scrum to fit your team's specific circumstances is always encouraged... but doing something like applying the "Kanban" technique to Scrum upsets many traditionalists.

"The surprising thing for me is that many smart Agile people - people I know to be intelligent insightful people - seem bugged by Kanban... I'm seeing Agile people behave as strangely about Kanban as traditional process folks behaved about Agile. They seem threatened." -- Jeff Patton

I've had a hard time getting some Scrum fundamentalists to even DISCUSS why they feel certain parts of Scrum should be "off limits" to change -- a strange idea, considering that Scrum, as it is typically applied today, is already very different from the way it was originally described. This seems inconsistent with Agile principles that encourage change based on continued analysis and experimentation, especially since the majority of people applying Kanban seem to be experienced Scrum practitioners, who report great success with this Scrum + Kanban approach in solving real-world issues that their teams struggled with under "straight" Scrum.

There is a subculture of agilists who seem to promote an "all or nothing", "enlightened vs. heathen" attitude, reflected in statements along the lines of "if you're not doing X, Y, and Z (or if you try to add Q), you're doing it wrong and you're not Agile". Many senior Agile dev. community members have explicitly spoken out against this, as counter-productive and just plain wrong. Kent Beck included an apology in the preface to the second edition of his book, Extreme Programming Explained, specifically to try to address this for the XP community, and published works by other industry thought leaders including Mary Poppendieck, Alan Shalloway etc. have made similar points. IBM has pushed for ALL its developers to adopt SOME form of agile, while explicitly REJECTING the idea that you need to follow any specific set of practices or methodology.

The Dancing Agile Elephant: IBM Software Group's Transition to Lean and Agile Development

Having worked both in highly-productive start up environments, and some amazingly dysfunctional corporate beauracracies, I'm an absolute supporter of developing software in a way that's consistent with Agile PRINCIPLES, and it is frustrating to think, after finally convincing your corporate management to take a serious look at what Agile has to offer, they'll encounter a discussion space filled with a mixture of hippie-dippy idealism and bickering that's likely just to make them roll their eyes and dismiss the topic out-of-hand.

"Badges? We don't need no stinkin' badges!" - Blazing Saddles, 1974

I became a "Microsoft Certified Professional" (MCP) for .Net back in '03, got my MCP for WPF last summer, and just picked up a Certified ScrumMaster (CSM) badge, all of which means... SQUAT!

Any experienced developer can tell you an MCP certificate (or a CS degree) says absolutely NOTHING about whether someone can actually write production quality code or help a company ship products, and there's been some criticism over the way the term "Certified ScrumMaster" is sometimes promoted and abused. For the record, a Microsoft certification essentially proves you can memorize API trivia out of a study guide, and a CSM certification just says you've had two days of classroom exposure to the basic concepts of Scrum.

Of course, there's a LITTLE more to it than that. A Microsoft certification guarantees baseline familiarity with the recommended way to handle common Windows dev. tasks, and a CSM badge guarantees exposure to fundamental Agile concepts, along with a general idea of how to apply them, and obviously indicates a commitment to helping to refine a company's development process. When interviewing new hire candidates at my last three companies, I always asked about certification as evidence of some level of "devotion to craft", beyond simply working on "assigned features", but I was always more impressed if the candidate was familiar with notable developer community authors, or attended local dev. events.

From the job candidate perspective, certification is most useful in helping get past HR and other non-programming middle manager filters. From management's perspective, the risk in being overly impressed by a Microsoft certification is alot less than the risk of turning an inexperienced CSM loose on your enterprise organization's development process for your mission critical software. CSM training and resources often have a kind of naive idealism to them ("Let's play a game to illustrate collaboration!"), and I think a CSM who hasn't personally experienced harsh corporate realities is in danger of having management and technical staff quickly flip the "Bozo Bit" on them. For example, there's a HUGE difference in promoting "self-organizing teams" in an environment of top-notch, experienced, highly-motivated developers, versus in a bloated, complacent, corporate development environment.

I've been keeping up with Agile practices and approaches since the late '90's, and I'm currently most impressed by approaches that combine Scrum and Kanban, as described in Henrik Kniberg's newly-published "Kanban and Scrum, Making The Most of Both". The book is succinct and the clearest explanation I've come accross, of what all this "Kanban" stuff is that everyone keeps talking about, and how it relates to Scrum

Update (2/13): Scrum co-creator Ken Schwaber recently parted ways with the Scrum Alliance, and started Scrum.org to offer more rigorous developer training.

It's been a very interesting couple of weeks...

I attended the local .Net User's Group "Education Day" event covering the Model-View-ViewModel architecture pattern, presented by Microsoft's WPF Program Manager, Karl Shifflet, and Jamie Rodriguez .  Karl missed our breakfast meeting, but we arranged to talk on the phone, and he apparently thought enough of my e-mail summarizing some of the issues to forward it to HIS boss.  Microsoft dev. evangelist Bruno Terkaly has also forwarded some of my feedback on various topics, and we joked that I'd end up on some kind of internal list of "trouble makers" at Microsoft (I listen for the black helicoptors at night...).

My e-mail included the following: "Looking at forums and sites like StackOverflow.com and listening to questions at the end of presentations, you'll keep hearing confused developers asking "Do I need to learn Blend?"  "For data access, are we supposed to use ADO.Net, or DAAB, or Linq to SQL (and is that now deprecated), or Entity Framework (and what about the posted 'vote of no-confidence')...?"  "Are we supposed to REJECT the WPF architecture encouraged by VStudio in order to insert some MVPoo mini-framework?"  "Should we use MVPoo version A, B, C, or go with Prism?" etc. etc. etc.   Microsoft reps seem to treat these as trivial concerns because they are each focused on their own specific area of technology, not considering that a dev. team on an actual project is in the position of having to essentially investigate ALL the available options in order to pick the best approach for EACH area of technology (because their project involves data access, AND remote comms, AND a UI...), and they have to do this in the limited time available before a project is under active development, and they have to train all the members of the dev. team (who are generally NOT MVP caliber developers)."

The gist of our conversation was that Microsoft is aware of many of these issues, and has been discussing them internally.  Karl was also surprised to hear that Blend is NOT included in the professional-level MSDN subscription, and said he'd recommend changing that, which would be great.

I attended the first meeting of a local SilverLight User's group, hosted in Microsoft's San Francisco offices.  This was the first event I've ever been to, where the number of designers and design-related discussion had any kind of parity with development concerns.  Presenters included Scott Stanfield and other representatives from Vertigo, as well as design/devs from the San Franciso offices of the advertising firm, McCann Worldgroup.

I also attended the day-long MSDN Developer's Conference, billed as a "mini-PDC", covering updates to topics and technologies presented last September, and was honored/priviledged to be invited to attend not one, but two social gatherings, where I got to hobnob with Microsoft and local dev. community leaders, bending their ears on dev.-related issues -- especially after the first couple of free pints...  I reluctantly passed up a free pass to MIX, since I just can't justify the hotel and other costs at this time (and videos for the sessions will be available online anyway).

For the second time in as many months, I've gotten e-mail from someone who noticed my CodeProject posts, asking if I'd be interested in doing some WPF consulting.  This time the request came directly from the CEO of an east coast medical equipment manufacturer, and I've forwarded some WPF resource links that should help their devs get up to speed, and offered to put together a simple demo to illustrate some architecture and design options just for fun -- since many of the requirements don't seem all that different from another CodeProject demo I've been working on anyway.

My social calendar hasn't been this busy since college!  Even the ILLUSION of being included in serious dev. discussions has me hooked, as I discovered from monitoring, the "WPF Disciples" discussion thread.  You have to filter out alot of empty "banter", but I swear I learn something completely unexpected from that site at least once a week (variations on approaches to M-V-VM, markup extensions as a localization technique etc.), and it really makes you feel like you're sitting directly together with the people who will have the most influence in guiding the direction of the WPF platform.

I've been meeting regularly with Bruno Terkaly, Microsoft Dev. Evangelist for Northern California, who has updated his WPF presentations to include discussion of issues I've suggested are important to developers moving to WPF -- but which always seem to be left out or misrepresented in Microsoft presentations and published resources.  Bruno has reported getting some "thank-you"s from audience members specifically related to the new material, just as I got "thank-you"s from developers in reaction to my original blog post on the subject.

WPF is a GREAT technology, but developers need open, honest discussion of the pain points they can expect to run into as they make the transition.  Examples of less-than-helpful guidance includes the often-repeated advice to simply "hand the project to the designers", while all the developers in the audience are wondering "WHAT designers?  We can't even get approval to hire TESTERS!"  Xaml is frequently portrayed as low-level tooling support, which can safely be ignored, while failing to mention that Visual Studio lacks design-time support for basic WPF features (DataTemplates, for example), and Expression Blend, which DOES offer support, is promoted as a tool for designers.  Any suggestion that developers avoid working directly with Xaml is especially unfortunate, since it takes just a few weeks to become completely comfortable editing Xaml directly, and this approach is MUCH more productive, and leads naturally to better-refactored, more maintainable markup.

After describing similar issues involving other Microsoft technologies, including difficulty in trying to make sense of Microsoft's recently-announced "Azure" cloud computing platform (Bruno says I can take credit for the fact that essential white papers are now grouped together and accessible when you click on the link labelled... er... "White Papers"), I was very kindly invited to attend an informal upcoming get-together of local Microsoft dev. community figures, where I'm hoping to compare notes.  I've also managed to finagle a breakfast meeting this weekend with Microsoft's Karl Shifflet, where I'll have the opportunity to discuss these issues directly with the WPF Program Manager.  Last night, I attended an excellent presentation on Azure, by David Chappelle, where I mentioned some of these issues, and was surprised to hear him express similar sentiments (confusion over the dual storage systems in Azure, the lame attempt to paint the lack of schema support as a feature instead of the honest -- and more palatable -- reality that ALL cloud vendors are adopting a very similar approach...).  He also seemed to agree with my impression that these issues are occuring more frequently, with evidence of development by independent teams (who don't talk to each other)... showing through at the seams.

I've suggested Microsoft could benefit from having internal developer ADVOCATES, focused on identifying and fixing these kinds of issues, rather than just developer "evangelists", who operate more as an extension of Microsoft marketing. Responsibilities for such a team might include...

• Working to identify and resolve sources of frustration and confusion in all public-facing content.
• Acting as a dev. community resource, soliciting reports of new issues, and maintaining a public (?) list of issues and their resolution status.
• Ensuring that cases where perceived issues are actually justified (or temporary), are clearly explained, as close to the start of the release cycle as possible.
• Using the history of reported issues to improve internal guidance for online documentation and SDK content and install experiences (including the possibility of relying more on pre-configured vhd images).
• Working to see that all mainstream documentation, sample code, tool-generated code, and public presentations, reflect industry "best practices", or that such information is at least mentioned alongside the usual focus on point-and-click solutions.

The idea is a little naiive, of course, and developers already have access to a wide variety of ways in which to report issues, but the current “scattershot” approach, leaving it up to each individual team to report and respond to issues in its own way, may be part of the problem.  A small team that served as the developer community voice INSIDE Microsoft, that was explicitly tasked with helping to identify and resolve these issues, regardless of which dev. or documentation group might actually be technically responsible for that particular area… could reduce frustration among Microsoft's target customer base, and probably also reduce the time dev. and marketing team members spend, having to repeat answers to the same questions, from a variety of sources.

I'm really enjoying the opportunity to discuss these issues with knowledgable people in the Windows dev. community, and I'd love to think that something in one of these discussions could trigger real change.

Our most important client wanted to draw attention to a major new feature in the next version of their successful commercial software product with a "flashing red button".  We tried to explain that this would be irritating, and our task should be to enable the user to quickly LOCATE and understand HOW TO USE features, not to impose our own idea of what their individual workflow should be.  I asked the client's marketing reps to imagine a word processor featuring a blinking "SPELLCHECK... SPELLCHECK... SPELLCHECK..." button.  They insisted...  We resisted...  Several weeks later, one of the marketing reps called to say that they now completely agreed that a flashing red button had been terrible idea... but could we add a "cycling button", which would endlessly transition through a set of colorful images?  We eventually delivered the product, free of any "throbbing" UI elements, to rave reviews -- and a Software Publishers Association "Codie" award.

Effective UI development requires attention to many different areas...

Buy-in:  Attention to UI is still an afterthought at many organizations ("our customers pay for FEATURES!"), and it sometimes takes a competing (possibly technically inferior) product, that wins over reviewers with its intuitive design and polished graphics, to change this.  Until you can overcome the attitude that developers can just use MS Paint to generate artwork for the shipping product, you'll be very limited in what you can accomplish in terms of design.

Interactive Design:  Guidance on principles and practices considered to contribute to "good UI" has long been available in published references from a number of notable industry authors (see the list below).  In addition to making the effort to actually STUDY the subject, it helps to be able to mentally "block out" your deep understanding of the underlying technical issues, and see interaction requirements from a user's non-technical perspective.  Common UI sins include exposing far too many options in the top-level UI, failing to group associated features or define interactions for similar tasks in a consistent way, failing to create a visual hierarchy to promote feature "discovery", creating deeply nesting submenus under submenus or dialog buttons that pop up more dialogs, and many others.

Aesthetics:  This is what most managers and marketing people tend to think of when they hear the term "UI design", with managers justifiably skeptical that the company should invest time and money just to "make things pretty", while marketing department representatives are envisioning how the application will look, projected on the JumboTron, at the next trade show.  A general consensus on the definition of "good enough" is important when considering changes that are purely aesthetic -- but much less so when it comes to issues of usability.  Aesthetics are almost completely subjective: "It's too round, the borders need to be thicker/darker, it 'feels' too crowded, the animation isn't smooth enough, it's too blue...".  A subtopic which is less subjective involves "color theory", and developers can dramatically improve their designs by relying on resources like Adobe's "Kuler" to help them select a limited set of colors that are guaranteed to look good together.

Engineering Effort:  Developers who focus exclusively on server-side/infrastructure/business logic issues, HUGELY underestimate the engineering effort required to produce an intuitive and compelling UI.  Speakers at dev. training events will often wave their hands in front of a thinly populated data grid near the end of a presentation on the latest data access/remote communications mini-framework, and make some offhand remark to the effect that "someone" can hang a pretty UI on all of this later...  In reality, I don't think I've ever seen a project where the time and effort required to implement a non-trivial user interface didn't turn out to be many times greater than what was required to pound out the underlying business logic and support layers, where implementation of the UI didn't require the attention of the teams's best developers, or where there weren't significant benefits to refactoring lower layers to present information in a form more convenient to the UI.

Design Extremes:  More than one member of the Windows developer community has warned about WPF's potential to inflict interfaces consisting of dancing 3D teapots pouring colored streams of text onto virtual curling pages...  At the other extreme, some developers will continue to simply glom together 3rd party controls, and rob their projects of the benefits of the DESIGN capabilities built into the framework.

Designer-Developer Interop:  Some people have been mislead to believe that designers can essentially make indiscriminate changes in Blend, while developers make indiscriminate changes to the shared project source in Visual Studio, and that as long as everything works in one environment, it will work in the other... and that designers and developers can work in harmony while remaining largely ignorant of each other's area of concern. Many teams, including some inside Microsoft, are finding that this is simply not true.

UI Testing:  The process of writing unit tests for business logic generally results in better code, converting methods that were formerly "private" into testable "public" methods in new classes with more narrowly-defined responsibilities, for example.  The structure of "good" UI code, on the other hand, is dictated by the UI development framework being used (Windows Forms, WPF etc.), and trying to unit test the presentation layer often requires conforming to a dramatically different coding style, which is not inherently better, but required by the test framework (implementing all control handlers as empty stubs that call into new "presenter" classes, for example -- although there are arguably other benefits to this particular technique that extend beyond testing).  Unit tests are also completely unable to test effects that are purely visual, which make up a HUGE proportion of UI bugs (scrolling is not smooth, controls don't line up, layout doesn't conform to UI guidelines, X is misspelled, quickly dragging the window causes the video image to dissappear while audio continues to play...), while much of what CAN be detected would rarely have escaped notice, even without unit tests (verify that clicking buttonX launches featureX...).  The cost-benefit trade off for trying to unit test the presentation layer becomes even more questionable with WPF,  since the majority of UI behavior in a well-designed WPF application will be defined in Xaml and, even if you could test the markup, you would essentially be testing Microsoft's Xaml parser and data binding implementations.

A less common type of UI testing consists of formal "usability" testing.  On the project mentioned earlier, our development team got to spend a week in an actual usability lab and (once we got over the fun of zooming the hidden cameras in and out on the test subject's bald spot), we found it to be a very humbling experience to watch user after user struggle with features we "knew" were completely intuitive, while we jumped around behind the one-way mirror begging the user to "Just click the Toolbar button!"  Formal usability testing is very valuable if you can justify the expense, but interpreting test results can be tricky -- thinking that maybe the user would have noticed that Toolbar option IF ONLY WE'D USED A FLASHING RED BUTTON!

 

Recommended UI Design Resources:

• "User Interface Design For Programmers", Joel Spolsky:  Concise & authoritative.  Best place to start!
• "Don't Make Me Think", Steve Krug:  Web-focused, but very relevant and a very quick read
• "Designing Interfaces", Jenifer Tidwell
• Jensen Harris blog (MS Office Design Mgr.):  Fascinating details of MS design efforts on Office 2007
• "About Face", Alan Cooper:  Sometimes controversial MS Windows UI luminary
• "Designing Web Usability", Jacob Nielsen: Leading authority on UI issues
• "The Design of Everyday Things", Donald Norman:  A recognized classic
• "Software For Use", Larry Constantine:  In-depth, textbook treatment of the subject 
• Edward R. Tufte (series of 5 books on design) :  The books themselves are works of art