One thing that kind of bothered me in my recent CSM training course was a consultant's comment that he didn't feel any need to try to "sell" Scrum to resistant management, implying that, if they didn't buy into it, it was their loss -- but winning over management is almost certainly a big part of what developers in that organization are expecting a consultant to HELP THEM WITH!
Just as management needs some level of developer "buy-in" for a project to have any chance of success, anyone hoping to transition a traditional organization to Agile needs to be able to SELL the concept, in terms a traditional business manager can understand. "Just trust us" is the arrogant Agile counterpart to management's "Just do it" -- and about as effective.
Agile is a TOUGH SELL! You're asking what may be a whole STACK of non-programming middle managers to abandon established practices, to override PMP-influenced resistance (which may finally be diminishing, now that the Project Management Institute has begun to take an interest in Agile), to buy into subversive concepts like "self-organizing teams", based on rationale they typically don't have the first-hand development experience to understand... knowing that if it all goes wrong, THEY'RE the ones who'll be held accountable.
Emphasizing Agile's ability to deliver better quality software, much faster, using fewer resources (better, faster, cheaper) is critical to convincing management to at least give Agile a TRY, but avoiding an outcome that permanently DISCREDITS Agile requires knowing where you can compromise the "purity" of your model, while holding the line on items necessary for Agile to actually produce benefits. Setting up a system of code review and mentoring, to ensure code is kept clean, maintainable, and extensible, is more important than insisting a large team of developers with poor dev. skills write sloppy unit tests to support their convoluted code. Generating some amount of unecessary documentation is preferrable to having your product fail a customer audit, or to have your project remain unfunded by corporate managers who don't buy into your "trust us" argument.
To offer reassurance and manage expectations, you'll need to provide justification for the changes that will need to take place, be able to offer credible alternatives to traditional methods of maintaining accountability, and openly acknowledge potential areas of conflict (for example, Agile organizations typically require a smaller number of more highly-skilled developers, which may eventually lead to issues of what to do with long-time employees who are under-performing in a department which is over-staffed, under the new standard).
By choosing your battles, and focusing on a few key issues like obsessive attention to code quality, replacing Big Design Up-Front with a commitment to iterative "refinement" of an initial outline spec., sqeezing in at least SOME amount of regular customer proxy feeback (usually from a busy marketing rep. who doesn't understand why you can't just go off and build the entire app. as specified), limiting the number of items under development to force developers out of their silos and encourage them to collaborate, holding regular informal reviews to try to improve the dev. process based on observed results... you are surreptitiously "infecting" the organization with traces of Agile, so you can break down resistance and eventually take over the host.
Don't make the mistake of the last organization I was in, where we managed to get buy-in on adopting specific Agile PRACTICES (continuous integration, unit testing...), but failed utterly to get commitment to the more important Agile PRINCIPLES. Management still insisted that we begin a three year project by dividing all work into "tasks", which were distributed to 15+ developers in two globally-distributed teams to "estimate", despite the fact that only three of us had significant first-hand experience with the technologies we would be using, and many developers suffered from questionable coding skills... with disasterous results.
Kanban is interesting here again, because it seems better able to overcome initial management and developer resistence, by requiring only very minor changes to ANY existing development process (including waterfall), with teams reportedly adopting more and more agile techniques naturally, over the course of the project, as productivity improvements become obvious to everyone involved.
"The (Pirate's Code) is more what you'd call 'guidelines' than actual rules."
- Captain Barbossa, Agile pragmatist, Pirates of the Carribean, 2003
One of the more confusing aspects of Scrum for me has been the fact that "adapting" Scrum to fit your team's specific circumstances is always encouraged... but doing something like applying the "Kanban" technique to Scrum upsets many traditionalists.
"The surprising thing for me is that many smart Agile people - people I know to be intelligent insightful people - seem bugged by Kanban... I'm seeing Agile people behave as strangely about Kanban as traditional process folks behaved about Agile. They seem threatened."
-- Jeff Patton
I've had a hard time getting some Scrum fundamentalists to even DISCUSS why they feel certain parts of Scrum should be "off limits" to change -- a strange idea, considering that Scrum, as it is typically applied today, is already very different from the way it was originally described. This seems inconsistent with Agile principles that encourage change based on continued analysis and experimentation, especially since the majority of people applying Kanban seem to be experienced Scrum practitioners, who report great success with this Scrum + Kanban approach in solving real-world issues that their teams struggled with under "straight" Scrum.
There is a subculture of agilists who seem to promote an "all or nothing", "enlightened vs. heathen" attitude, reflected in statements along the lines of "if you're not doing X, Y, and Z (or if you try to add Q), you're doing it wrong and you're not Agile". Many senior Agile dev. community members have explicitly spoken out against this, as counter-productive and just plain wrong. Kent Beck included an apology in the preface to the second edition of his book, Extreme Programming Explained, specifically to try to address this for the XP community, and published works by other industry thought leaders including Mary Poppendieck, Alan Shalloway etc. have made similar points. IBM has pushed for ALL its developers to adopt SOME form of agile, while explicitly REJECTING the idea that you need to follow any specific set of practices or methodology.
The Dancing Agile Elephant: IBM Software Group's Transition to Lean and Agile Development
Having worked both in highly-productive start up environments, and some amazingly dysfunctional corporate beauracracies, I'm an absolute supporter of developing software in a way that's consistent with Agile PRINCIPLES, and it is frustrating to think, after finally convincing your corporate management to take a serious look at what Agile has to offer, they'll encounter a discussion space filled with a mixture of hippie-dippy idealism and bickering that's likely just to make them roll their eyes and dismiss the topic out-of-hand.
"Badges? We don't need no stinkin' badges!" - Blazing Saddles, 1974
I became a "Microsoft Certified Professional" (MCP) for .Net back in '03, got my MCP for WPF last summer, and just picked up a Certified ScrumMaster (CSM) badge, all of which means... SQUAT!
Any experienced developer can tell you an MCP certificate (or a CS degree) says absolutely NOTHING about whether someone can actually write production quality code or help a company ship products, and there's been some criticism over the way the term "Certified ScrumMaster" is sometimes promoted and abused. For the record, a Microsoft certification essentially proves you can memorize API trivia out of a study guide, and a CSM certification just says you've had two days of classroom exposure to the basic concepts of Scrum.
Of course, there's a LITTLE more to it than that. A Microsoft certification guarantees baseline familiarity with the recommended way to handle common Windows dev. tasks, and a CSM badge guarantees exposure to fundamental Agile concepts, along with a general idea of how to apply them, and obviously indicates a commitment to helping to refine a company's development process. When interviewing new hire candidates at my last three companies, I always asked about certification as evidence of some level of "devotion to craft", beyond simply working on "assigned features", but I was always more impressed if the candidate was familiar with notable developer community authors, or attended local dev. events.
From the job candidate perspective, certification is most useful in helping get past HR and other non-programming middle manager filters. From management's perspective, the risk in being overly impressed by a Microsoft certification is alot less than the risk of turning an inexperienced CSM loose on your enterprise organization's development process for your mission critical software. CSM training and resources often have a kind of naive idealism to them ("Let's play a game to illustrate collaboration!"), and I think a CSM who hasn't personally experienced harsh corporate realities is in danger of having management and technical staff quickly flip the "Bozo Bit" on them. For example, there's a HUGE difference in promoting "self-organizing teams" in an environment of top-notch, experienced, highly-motivated developers, versus in a bloated, complacent, corporate development environment.
I've been keeping up with Agile practices and approaches since the late '90's, and I'm currently most impressed by approaches that combine Scrum and Kanban, as described in Henrik Kniberg's newly-published "Kanban and Scrum, Making The Most of Both". The book is succinct and the clearest explanation I've come accross, of what all this "Kanban" stuff is that everyone keeps talking about, and how it relates to Scrum
Update (2/13): Scrum co-creator Ken Schwaber recently parted ways with the Scrum Alliance, and started Scrum.org to offer more rigorous developer training.
It's been a very interesting couple of weeks...
I attended the local .Net User's Group "Education Day" event covering the Model-View-ViewModel architecture pattern, presented by Microsoft's WPF Program Manager, Karl Shifflet, and Jamie Rodriguez . Karl missed our breakfast meeting, but we arranged to talk on the phone, and he apparently thought enough of my e-mail summarizing some of the issues to forward it to HIS boss. Microsoft dev. evangelist Bruno Terkaly has also forwarded some of my feedback on various topics, and we joked that I'd end up on some kind of internal list of "trouble makers" at Microsoft (I listen for the black helicoptors at night...).
My e-mail included the following: "Looking at forums and sites like StackOverflow.com and listening to questions at the end of presentations, you'll keep hearing confused developers asking "Do I need to learn Blend?" "For data access, are we supposed to use ADO.Net, or DAAB, or Linq to SQL (and is that now deprecated), or Entity Framework (and what about the posted 'vote of no-confidence')...?" "Are we supposed to REJECT the WPF architecture encouraged by VStudio in order to insert some MVPoo mini-framework?" "Should we use MVPoo version A, B, C, or go with Prism?" etc. etc. etc. Microsoft reps seem to treat these as trivial concerns because they are each focused on their own specific area of technology, not considering that a dev. team on an actual project is in the position of having to essentially investigate ALL the available options in order to pick the best approach for EACH area of technology (because their project involves data access, AND remote comms, AND a UI...), and they have to do this in the limited time available before a project is under active development, and they have to train all the members of the dev. team (who are generally NOT MVP caliber developers)."
The gist of our conversation was that Microsoft is aware of many of these issues, and has been discussing them internally. Karl was also surprised to hear that Blend is NOT included in the professional-level MSDN subscription, and said he'd recommend changing that, which would be great.
I attended the first meeting of a local SilverLight User's group, hosted in Microsoft's San Francisco offices. This was the first event I've ever been to, where the number of designers and design-related discussion had any kind of parity with development concerns. Presenters included Scott Stanfield and other representatives from Vertigo, as well as design/devs from the San Franciso offices of the advertising firm, McCann Worldgroup.
I also attended the day-long MSDN Developer's Conference, billed as a "mini-PDC", covering updates to topics and technologies presented last September, and was honored/priviledged to be invited to attend not one, but two social gatherings, where I got to hobnob with Microsoft and local dev. community leaders, bending their ears on dev.-related issues -- especially after the first couple of free pints... I reluctantly passed up a free pass to MIX, since I just can't justify the hotel and other costs at this time (and videos for the sessions will be available online anyway).
For the second time in as many months, I've gotten e-mail from someone who noticed my CodeProject posts, asking if I'd be interested in doing some WPF consulting. This time the request came directly from the CEO of an east coast medical equipment manufacturer, and I've forwarded some WPF resource links that should help their devs get up to speed, and offered to put together a simple demo to illustrate some architecture and design options just for fun -- since many of the requirements don't seem all that different from another CodeProject demo I've been working on anyway.
My social calendar hasn't been this busy since college! Even the ILLUSION of being included in serious dev. discussions has me hooked, as I discovered from monitoring, the "WPF Disciples" discussion thread. You have to filter out alot of empty "banter", but I swear I learn something completely unexpected from that site at least once a week (variations on approaches to M-V-VM, markup extensions as a localization technique etc.), and it really makes you feel like you're sitting directly together with the people who will have the most influence in guiding the direction of the WPF platform.
"Princeton could USE a man like Joel."
- Hope for all us unconventional interview candidates, from "Risky Business" 1983
I've been living off my incredible shrinking stock portfolio, hanging out in local coffee shops and restaurants with my laptop, working with WPF and other development technologies... for too long now, and I'm itching to APPLY these new technologies and practices, to something challenging and interesting, working with people who are "smart, and get things done".
In addition to pointing out that I'll be starting my job search in the depths of a nation-wide employment freeze, friends have predicted that hiring managers will reject me because of my recent "sabbatical", but I think any company that would consider intense work with next generation development technologies on my own dime to be a BAD thing is probably one I wouldn't want to work at anyway 
I'm obviously delusional enough to think I can still afford to be picky, and there are some situations I'm really hoping to AVOID this time around...
- The New Number 6: "This 'best practices' stuff all sounds very interesting, but right now what we really need is for someone to take over John's code. We're down to five developers now that John got fed up and quit, no one undertands his stuff, and marketing has commited us to delivering a big stack of new features in eight months."
- How Hard Could It Be?: "We can't spare any of our hardware engineers to spend a weekend knocking out this 'simple' software, so we want to hire you. We've already done the hard part of defining the database schema, and we just need you to add a few things and get together with marketing to slap on some kind of pretty UI."
- Brain Silos: "Joe is designing data access, Tim is handling remote comms, Fred is responsible for security... you don't need to concern yourself with those issues, and they don't need to be a part of discussions involving your area."
- Plug-and-Play Developers: "We're going to need to pull you and Joe off the project for eight weeks to jump on a new marketing request. Remedial Roger's not doing anything right now, so he can fill in until you get back, and then maybe we'll have you split your time 60/40 between projects X and Y..."
- Sacred Tablets: "I'm not sure what you mean by 'Agile', but we follow the process described in this 5 inch thick binder, written ten years ago"
- Invertebrates: "Yes, we've all been saying for years how we should be doing X, Y, and Z, but what can we do?"
I have this fantasy where I work in an environment where we avoid doing things we already know are dumb before we even start, and where we actually FIX things we discover we got wrong along the way. I'm not holding out hope for an environment as enlightened as Fog Creek, but a company should fall within some REASONABLE RANGE on these issues.
Some organizations believe they "can't afford" to spend time refactoring their development process, especially in the current economic environment, when its probably the adoption of a more streamlined iterative process, featuring cross-functional teams, relying on a smaller number of more capable individuals, that is likely to have the most dramatic long-term positive impact on a company's software development costs, schedules, and quality.
I've been meeting regularly with Bruno Terkaly, Microsoft Dev. Evangelist for Northern California, who has updated his WPF presentations to include discussion of issues I've suggested are important to developers moving to WPF -- but which always seem to be left out or misrepresented in Microsoft presentations and published resources. Bruno has reported getting some "thank-you"s from audience members specifically related to the new material, just as I got "thank-you"s from developers in reaction to my original blog post on the subject.
WPF is a GREAT technology, but developers need open, honest discussion of the pain points they can expect to run into as they make the transition. Examples of less-than-helpful guidance includes the often-repeated advice to simply "hand the project to the designers", while all the developers in the audience are wondering "WHAT designers? We can't even get approval to hire TESTERS!" Xaml is frequently portrayed as low-level tooling support, which can safely be ignored, while failing to mention that Visual Studio lacks design-time support for basic WPF features (DataTemplates, for example), and Expression Blend, which DOES offer support, is promoted as a tool for designers. Any suggestion that developers avoid working directly with Xaml is especially unfortunate, since it takes just a few weeks to become completely comfortable editing Xaml directly, and this approach is MUCH more productive, and leads naturally to better-refactored, more maintainable markup.
After describing similar issues involving other Microsoft technologies, including difficulty in trying to make sense of Microsoft's recently-announced "Azure" cloud computing platform (Bruno says I can take credit for the fact that essential white papers are now grouped together and accessible when you click on the link labelled... er... "White Papers"), I was very kindly invited to attend an informal upcoming get-together of local Microsoft dev. community figures, where I'm hoping to compare notes. I've also managed to finagle a breakfast meeting this weekend with Microsoft's Karl Shifflet, where I'll have the opportunity to discuss these issues directly with the WPF Program Manager. Last night, I attended an excellent presentation on Azure, by David Chappelle, where I mentioned some of these issues, and was surprised to hear him express similar sentiments (confusion over the dual storage systems in Azure, the lame attempt to paint the lack of schema support as a feature instead of the honest -- and more palatable -- reality that ALL cloud vendors are adopting a very similar approach...). He also seemed to agree with my impression that these issues are occuring more frequently, with evidence of development by independent teams (who don't talk to each other)... showing through at the seams.
I've suggested Microsoft could benefit from having internal developer ADVOCATES, focused on identifying and fixing these kinds of issues, rather than just developer "evangelists", who operate more as an extension of Microsoft marketing. Responsibilities for such a team might include...
- Working to identify and resolve sources of frustration and confusion in all public-facing content.
- Acting as a dev. community resource, soliciting reports of new issues, and maintaining a public (?) list of issues and their resolution status.
- Ensuring that cases where perceived issues are actually justified (or temporary), are clearly explained, as close to the start of the release cycle as possible.
- Using the history of reported issues to improve internal guidance for online documentation and SDK content and install experiences (including the possibility of relying more on pre-configured vhd images).
- Working to see that all mainstream documentation, sample code, tool-generated code, and public presentations, reflect industry "best practices", or that such information is at least mentioned alongside the usual focus on point-and-click solutions.
The idea is a little naiive, of course, and developers already have access to a wide variety of ways in which to report issues, but the current “scattershot” approach, leaving it up to each individual team to report and respond to issues in its own way, may be part of the problem. A small team that served as the developer community voice INSIDE Microsoft, that was explicitly tasked with helping to identify and resolve these issues, regardless of which dev. or documentation group might actually be technically responsible for that particular area… could reduce frustration among Microsoft's target customer base, and probably also reduce the time dev. and marketing team members spend, having to repeat answers to the same questions, from a variety of sources.
I'm really enjoying the opportunity to discuss these issues with knowledgable people in the Windows dev. community, and I'd love to think that something in one of these discussions could trigger real change.
Our most important client wanted to draw attention to a major new feature in the next version of their successful commercial software product with a "flashing red button". We tried to explain that this would be irritating, and our task should be to enable the user to quickly LOCATE and understand HOW TO USE features, not to impose our own idea of what their individual workflow should be. I asked the client's marketing reps to imagine a word processor featuring a blinking "SPELLCHECK... SPELLCHECK... SPELLCHECK..." button. They insisted... We resisted... Several weeks later, one of the marketing reps called to say that they now completely agreed that a flashing red button had been terrible idea... but could we add a "cycling button", which would endlessly transition through a set of colorful images? We eventually delivered the product, free of any "throbbing" UI elements, to rave reviews -- and a Software Publishers Association "Codie" award.
Effective UI development requires attention to many different areas...
Buy-in: Attention to UI is still an afterthought at many organizations ("our customers pay for FEATURES!"), and it sometimes takes a competing (possibly technically inferior) product, that wins over reviewers with its intuitive design and polished graphics, to change this. Until you can overcome the attitude that developers can just use MS Paint to generate artwork for the shipping product, you'll be very limited in what you can accomplish in terms of design.
Interactive Design: Guidance on principles and practices considered to contribute to "good UI" has long been available in published references from a number of notable industry authors (see the list below). In addition to making the effort to actually STUDY the subject, it helps to be able to mentally "block out" your deep understanding of the underlying technical issues, and see interaction requirements from a user's non-technical perspective. Common UI sins include exposing far too many options in the top-level UI, failing to group associated features or define interactions for similar tasks in a consistent way, failing to create a visual hierarchy to promote feature "discovery", creating deeply nesting submenus under submenus or dialog buttons that pop up more dialogs, and many others.
Aesthetics: This is what most managers and marketing people tend to think of when they hear the term "UI design", with managers justifiably skeptical that the company should invest time and money just to "make things pretty", while marketing department representatives are envisioning how the application will look, projected on the JumboTron, at the next trade show. A general consensus on the definition of "good enough" is important when considering changes that are purely aesthetic -- but much less so when it comes to issues of usability. Aesthetics are almost completely subjective: "It's too round, the borders need to be thicker/darker, it 'feels' too crowded, the animation isn't smooth enough, it's too blue...". A subtopic which is less subjective involves "color theory", and developers can dramatically improve their designs by relying on resources like Adobe's "Kuler" to help them select a limited set of colors that are guaranteed to look good together.
Engineering Effort: Developers who focus exclusively on server-side/infrastructure/business logic issues, HUGELY underestimate the engineering effort required to produce an intuitive and compelling UI. Speakers at dev. training events will often wave their hands in front of a thinly populated data grid near the end of a presentation on the latest data access/remote communications mini-framework, and make some offhand remark to the effect that "someone" can hang a pretty UI on all of this later... In reality, I don't think I've ever seen a project where the time and effort required to implement a non-trivial user interface didn't turn out to be many times greater than what was required to pound out the underlying business logic and support layers, where implementation of the UI didn't require the attention of the teams's best developers, or where there weren't significant benefits to refactoring lower layers to present information in a form more convenient to the UI.
Design Extremes: More than one member of the Windows developer community has warned about WPF's potential to inflict interfaces consisting of dancing 3D teapots pouring colored streams of text onto virtual curling pages... At the other extreme, some developers will continue to simply glom together 3rd party controls, and rob their projects of the benefits of the DESIGN capabilities built into the framework.
Designer-Developer Interop: Some people have been mislead to believe that designers can essentially make indiscriminate changes in Blend, while developers make indiscriminate changes to the shared project source in Visual Studio, and that as long as everything works in one environment, it will work in the other... and that designers and developers can work in harmony while remaining largely ignorant of each other's area of concern. Many teams, including some inside Microsoft, are finding that this is simply not true.
UI Testing: The process of writing unit tests for business logic generally results in better code, converting methods that were formerly "private" into testable "public" methods in new classes with more narrowly-defined responsibilities, for example. The structure of "good" UI code, on the other hand, is dictated by the UI development framework being used (Windows Forms, WPF etc.), and trying to unit test the presentation layer often requires conforming to a dramatically different coding style, which is not inherently better, but required by the test framework (implementing all control handlers as empty stubs that call into new "presenter" classes, for example -- although there are arguably other benefits to this particular technique that extend beyond testing). Unit tests are also completely unable to test effects that are purely visual, which make up a HUGE proportion of UI bugs (scrolling is not smooth, controls don't line up, layout doesn't conform to UI guidelines, X is misspelled, quickly dragging the window causes the video image to dissappear while audio continues to play...), while much of what CAN be detected would rarely have escaped notice, even without unit tests (verify that clicking buttonX launches featureX...). The cost-benefit trade off for trying to unit test the presentation layer becomes even more questionable with WPF, since the majority of UI behavior in a well-designed WPF application will be defined in Xaml and, even if you could test the markup, you would essentially be testing Microsoft's Xaml parser and data binding implementations.
A less common type of UI testing consists of formal "usability" testing. On the project mentioned earlier, our development team got to spend a week in an actual usability lab and (once we got over the fun of zooming the hidden cameras in and out on the test subject's bald spot), we found it to be a very humbling experience to watch user after user struggle with features we "knew" were completely intuitive, while we jumped around behind the one-way mirror begging the user to "Just click the Toolbar button!" Formal usability testing is very valuable if you can justify the expense, but interpreting test results can be tricky -- thinking that maybe the user would have noticed that Toolbar option IF ONLY WE'D USED A FLASHING RED BUTTON!
Recommended UI Design Resources:
I've come accross several good posts recently, reminding developers not to get sucked into a "Red Queen's Race", trying to keep up with every new development tool and technology, and every popular development author's latest book.
On the other hand, your job as a developer includes the responsibility to promote tools and practices that enable your company to create more reliable, more aesthetically pleasing versions of its software, delivered much more quickly -- and quite possibly requiring fewer (but better trained) developers. Somehow, you have to sort through all the hype, to identify the subset of tools and practices that are actually destined to become industry standards, and you need to understand these technologies well enough (and early enough) to design your CURRENT code and architecture in a way that will make eventual migration to the new standard as painless as possible.
You'd think, with several months free of any of those pesky JOB responsibilities, that I'd have no problem catching up on my backlog of unexplored technologies and unread books, even with the distraction of daytime TV shows like "MythBusters" and "Dirty Jobs", and regular trips to the beach, but even after dropping some of the more "speculative" technologies, like F#, Acropolis, and the ADO.Net Entity Framework, from my study list, I still feel like I'd need to take the rest of 2008 off, just to cover the essentials... and maybe 2009 too!
One of the first things I wanted to do this summer, was to set up a realistic simulation of an acceptable enterprise development environment, using virtual machines, in order to practice with, and test, Agile tools and processes. I finally settled on using VirtualPC 2007 to create VS2005 and VS2008 Pro Beta2 workstations, with associated continuous integration VPC servers running a whole slew of open source analysis tools. I can check for issues developing as a Least Priviledged User on my Vista laptop host system, compared with developing with full admin. priviledges on Windows XP running inside a virtual machine. To create a fresh work or test environment, I can simply copy a "base" VPC image, without having to reinstall the OS. It may even be possible for the server to automatically publish the compiled output to a fresh VPC image for testing. If you want to set up something similar, you'll save yourself A LOT of time and misery if you refer to the links below. One hint: Use the NAT networking option to share the host system's internet access and virus scan software, for tasks like downloading OS updates, and switch to a Loopback Adapter when you want to communicate between VPC images, or between your host and a VPC image. You'll also want alot of disk space (each VPC dev. workstation ends up around 9GB.), and alot of RAM (My system has 2GB).
VPC Development and Debugging White Paper
Creating Smaller Virtual Machines
For more information and guidance on CI server-based development, I recommend the recently-published Continuous Integration", from the Martin Fowler series. Setting up your server to automatically regenerate the database, complete with a clean set of demo data, in order to validate schema changes, is just one example of possibilities I'd never considered.
I found a couple of new books, covering enterprise Agile and Agile project management, to be enjoyable, quick reads, suitable for managers who may just be starting to seriously consider Agile practices, but the books didn't contain anything that would be new to developers who have been keeping up with Agile community discussions over the years.
"Scaling Software Agility", from the Agile Software Development series
"Manage It!", from the Pragmatic Programmers series
I've been working through the LINQ examples in the recently published "Introducing Microsoft LINQ", and I definitely recommend that book as a quick way to get up to speed on the three flavors of LINQ (to Objects, to XML, and to Databases), and the new C# 3.0 language features LINQ is built on.
Its a strange kind of vacation...
Halfway through a four year stint in what was then a "peacetime" army (how quaint does THAT concept seem these days?), I was attached to a Special Forces team for the duration of a three month Arabic language course at the JFK Special Warfare Training Center, at Fort Bragg, North Carolina. I noticed my new classmates didn't seem to share my unit's obsession with spit shined boots and starched uniforms and buzz cuts, and I was stunned on our first day, when a low-ranking team member cut off an attempt, by an overzelous regular army officer classmate, to assign us some unrelated busy work, saying "No, lieutenant. That's not why we're here" -- leaving the young officer red faced and stuttering.
Over the next few weeks, in answer to my many questions, members of the team described HALO freefalls, sniper, SERE (survival, escape, resistance, and evasion), and Ranger schools, the stress of trying to keep a goat alive through a series of burns and bullet wounds inflicted as a part of medic training, techniques for constructing demolitions using common household materials, the experience of being "drowned" in a swimming pool by military scuba instructors, and much more. One afternoon, the team's medic suggested I join them at the local hospital after class, where they would practice giving each other injections. I expected to start off with some carefully-supervised sessions using oranges or something, but they just handed out vials and syringes, and the team's weapons man guided me through the steps, on his own arm. When I had trouble piercing the center of the vial cap because my hand was shaking, he just arched an eyebrow. When I failed to jab the needle deeply enough into his shoulder, and had to gradually PUUUUUUSH it in the rest of the way before depressing the plunger, he simply drew a deep breath slowly in through his teeth. Shrugging off my embarrassed apologies, he gave an evil grin and said, "My turn." Of course, I didn't feel a thing, and my own attempt on a second victim went much more smoothly, but I still turned down an invitation to come back the next week when they were scheduled to practice inserting IVs. The no-nonsense attitude and competence displayed by every member of the team was such a contrast to my experience in the regular army, that I spoke to a recruiter about transferring -- until I found out I'd have to extend my enlistment an additional three years.
The make up of a special forces team reflects many of the same values promoted by Agile methodologies, consisting of a small cross-functional team of expert individuals, who share a commitment to ruthless pragmatism. An S.F. team consists of a team leader, with one expert and an assistant to cover each of four specialized areas: weapons, communications, demolitions, and battlefield medicine. Team members continuously cross-train one another in their specialties, constantly attend courses to acquire new skills, and receive language, culture, and jungle, mountain, or dessert training appropriate to whatever region is their group's focus. They can't afford to suffer fools, or foolish processes, lightly.
In contrast to the pragmatic "adapt whatever proves useful" philosophy embraced by most Agile advocates today, the first generation of Agile proponents tried to impose their own brand of process fundamentalism, and only gradually mellowed, partly through the influence of thoughtfull critiques, such as in the book, "Extreme Programming Refactored: The Case Against XP". Most published Agile references now emphasize that Agile practices are not a "one size fits all" proposition, and it's the shared underlying Agile PRINCIPLES, rather than any specific methodology, which are important for organizations to cultivate.
"Critics of the first edition have complained that it tries to force them to program in a certain way... I'm embarrassed to say that was my intention... in this edition, I have tried to rephrase my message in a positive, inclusive way" -- Kent Beck, in the preface to his second edition of "Extreme Programming Explained".
Unfortunately, some development groups remain unfamiliar with even the most the basic concepts of Agile development, with management still determined to try to nail down a "complete" design and "accurate" schedule up front, dividing responsibility for work into separate silos, preferring innocuous mission statements and shiny quality commitment plaques to the potential confrontation inherent in rigorous code reviews and post-mortems... After ten years of intense discussion in the software developer community, with support for Agile practices increasingly baked directly into the developer tool set, that's hard to excuse.
As irrational as some management decisions affecting your development group may sometimes appear to be, at least they aren't likely to get you killed. When my Arabic language course ended, I returned to my own unit, just in time for our commander to order us to get our newly-redesigned camoflauge uniforms starched and pressed. The uniforms came packaged with warnings from the manufacturer, clearly stating that starching and pressing would destroy infrared dispersing properties embedded into the fabric, making them a bright target for anyone with a night vision scope. I suspect that more than a few individuals in a Special Forces unit would have stood up to remind the commander of why they were there...
Two months into my first job out of college, in the Tokyo offices of a large U.S. investment banking firm, I received an e-mail asking me to verify figures to be presented to the Board of Directors in New York, describing a 65% reduction in trades processing costs, and predicting a similar improvement the following year. I nervously asked the head of Tokyo operations if he could show me how he had arrived at his figures, and he gave an exasperated sigh, and demonstrated how he'd taken the (unusually low) daily trade volumes from January, plotted a straight line through the current month of August (when we were experiencing the highest trade volumes in the firm's eight year history in Japan), and then simply EXTENDED the line for another twelve months. I stood there with my mouth open... and then hesitantly pointed out that projecting the growth of an embryo would give you a one year old child who was a hundred feet tall, and that his figures assumed fixed costs, while I'd already heard we were running unsustainable levels of overtime, and adding staff might require leasing more office space in what was, at the time, the most expensive real estate market in the world... He said that, since these numbers were my responsibility, he certainly wouldn't present them to the board if I wasn't comfortable with them...
Flash forward a few years, to a massive effort to completely revamp a company's flagship client-server software product, requiring coordination between engineering teams in europe and the U.S., involving new technologies like WCF, Windows Forms, and ADO.Net, applying new development procedures like unit testing, FxCop oversight, and comprehensive code review for the very first time, and involving more than a few developers whose skills were... less than optimal. I'd been pushing to have the entire local team work closely with the two of us who had significant .Net experience, to try to generate, as quickly as possible, an installable, functional UI shell, over a minimal (but architectuarally accurate) remote comms/data access framework. This would let us iron out basic code quality, architecture, and dev. process issues AS A GROUP, and give us a more accurate basis for monitoring progress, as we gradually added features to a production quality base. The project manager vetoed the plan, saying we couldn't have everyone working together on the same tasks, and that we had to come up with a detailed schedule. Three years of tasks were therefore broken out, assigned to individuals to estimate, interdependencies carefully charted, and tasks then reassigned to DIFFERENT individuals for implementation. Several months later, we had a collection of untested, independently-designed feature FRAGMENTS, which had yet to undergo a single code review, with no idea of when we might ever see a running executable representing the actual integrated product. After the third panicked meeting in a single week, on how to address feature schedules already double their original estimates, with the discussion focused mainly on the possibly of cutting features (with a ship date still TWO AND A HALF YEARS AWAY), and the need to revise estimates and reshuffle interdependencies in MS Project, not to mention the PM's comment that she hoped she wouldn't have to cancel everyone's vacations for the duration of the project... I announced my resignation.
Reality doesn't conform to carefully "calculated" projections. As a software project manager, your contribution is not to tweak Gantt chart or MS Project scenarios, but to help identify, and RUTHLESSLY ADDRESS, issues interfering with the optimal performance of your team, adjusting any initial estimates or predictions based on observed (hopefully, ever-improving) results. No amount of estimation is going to help in the situation described above, without first establishing some actual BASIS for making estimates. Developers still need to identify tasks, and provide estimates in no larger than half-day increments, but focusing mainly on the subset of features they are ACTUALLY PREPARING TO IMPLEMENT. Unless you are building a virtual copy of a previously implemented product, involving the same group of individuals, working with identical technologies under the same process... you just have to accept the fact that any detailed time estimates farther than 3 months out are very likely to be VERY wrong, and adjusting those estimates shouldn't be treated as anything other than a natural, expected part of the development cycle. Obviously, detailed estimates are intended to avoid a situation where you approach the "end" of a three year project, with no clear idea of when, if ever, the product will actually be ready to ship, and modern "agile" methodologies try to address this by advocating the creation of a production quality framework as quickly as possible, by prioritizing features to ensure the most important items are addressed first, by requiring all code to be fully tested and of shipping quality at the time it is written, by not saddling the project with subpar performers or senseless beauracracy, and by defining short intervals at which to re-evaluate the schedule and feature set, based on open, honest, evaluation of OBSERVED results.
Several things in a developer's e-mail just didn't sound right (they reported problems explicitly checking mouse click coordinates to display a custom control context menu). Reviewing the code with the developer in their cube, I pointed out Windows Forms features that made much of their code unecessary, suggested a couple strategies to try to address some legitimate issues associated with their feature, and bookmarked relevant sections in the developer's copy of Chris Sell's "Windows Forms 2.0 Programming", that I had insisted we buy (along with Löwy's "Programming .Net Components, 2nd ed.") for everyone in the department. The developer's response: "Do I really have to read all this? Its almost working, and we've got a schedule to keep."
My job has included interviewing potential new hires at my last three companies, and the one quality I'm always desperate to find, is evidence of sincere PASSION for software engineering. Staffing with passionate developers eliminates situations like the one above, while dealing with "corporate" developers just makes you want to poke your eyes out with a sharp stick on a daily basis -- or makes you want to poke THEIR eyes out, which apparently violates some sort of HR policy.
The "corporate" developer has little interest in software engineering, beyond what's required to check their currently-assigned feature off of their "To Do" list, and they're not about to spend time attending local .Net user group meetings, or catching up on the latest development books, or monitoring dev. community RSS feeds. Their code usually shows little evidence of any attempt at refactoring, reflects ignorance of features explicitly designed into the dev. platform to address common usage scenarios, ignores obvious opportunities to employ common design or architecture patterns, and sometimes seems specifically designed to make it as difficult as possible for anyone else ever to understand, maintain, debug, or extend, their particular "contribution" to the code base. Obviously, you can't hire someone purely on the basis of passion, but passionate developers tend to quickly pick up whatever they need to know, and understand it at a much less superficial level. Passionate developers also care deeply about the quality of the code and architecture that they are inflicting on their co-workers. Some of them may secretly even... blog.
Beware, however, of the passionate highly-skilled developer who seems obsessed with finding fault with Microsoft's recommended approach to almost anything. These are the developers whose code will have no relationship to anything found in any standard Windows development reference, ensuring confusion for every experienced Windows developer who joins the project in each new dev. cycle, and eliminating any chance of a clean migration path as Windows dev. technologies continue to evolve. These developers are always able to provide low-level technical justification for their choices that is hard to argue with, but they never seem to be able to demonstrate any real problems with the simpler (usually higher-level), expected approach, that a target end user would actually care about -- or even be able to detect.
Developer passion and complacency are equally contagious, so be sure that your hiring, and FIRING, decisions leave no doubt as to the kind of engineering department you're trying to build.
The fact that the Blend tool was used to design the non-trivial UI of Blend itself, is pretty convincing evidence that the first release version is probably already suitable for use on many real-world projects, but there are a few practical concerns that always seem to be glossed over in Blend-related discussions.
How exactly is this new designer-developer coordination supposed to work? An interesting Channel9 interview with members of Microsoft's User Experience Team in the UK details some valuable lessons learned through trying to apply this developer-designer collaboration on actual projects (jump to minute 16:00 to skip intro material). The team discovered that designers needed some level of understanding of Windows development and WPF, in order to avoid problems such as designers creating image elements where they should have restyled control templates, or designers breaking links to underlying business logic by "replacing" controls with restyled versions. The developers also found the auto-generated Xaml provided by the designers to be difficult to read and edit in those inevitable situations where it became necessary or beneficial to do so. The UK team finally settled on the very interesting approach of having the designers generate a "wire frame" DRAWING of the overall UI, which the developers use as a guide in beginning the actual project, while the designers go on to create separate resource library projects, featuring the production quality UI elements that the developers plug into their own carefully-crafted production code. Other development teams have reported success using a single set of shared project files, but its helpful to be aware of a full range of real-world proven alternatives before rushing to apply Blend on your company's next mission critical project.
Realistically, what are the chances that your company will hire professional desigers, or approve funds to hire one of the few consulting firms with experience in this area, like Vertigo? I recently worked at a chemical equipment manufacturer, which contracted with a professional industrial design firm for their complete line of hardware enclosures, while relying on the artistic talents of internal developers using MS Paint for their accompanying software, and a marketing manager at a different company once asked why I couldn't just use clip art throughout the UI for the "commercial quality" digital video capture and editing application they'd hired me to write (using DirectShow, to accompany one of the first generation 1394 Host Bus Adapters). For many organizations, this attitude is likely to continue -- at least until that first trade show where a competitor demonstrates a professionally-designed product that takes full advantage of new WPF graphics capabilities. Even if a company is willing to allocate the funds, there is still the issue of trying to FIND designers who have the combination of skills described in the UK group video. This is not a situation where you can simply retask your company's web developer or marcom person, which still leaves it up to the DEVELOPERS to acquire expertise with Blend, at least in the near term.
On the architecture front, Blend promises to enable designers to "reskin" an application at will, but this separation of UI from business logic occurs at a much higher level than what is currently supported through code-level frameworks such as CAB and SCSF. Whereas these code-level frameworks often attempt to create extra abstraction layers (presenters, controllers, etc.) to decouple UI "views" from the underlying business logic, they still assume that developers will be required to "code" the UI. Microsoft has already announced that CAB/SCSF will be replaced by Acropolis, which will offer some degree of support for Blend (currently, limited to the ability to open "PartViews"), but is not expected to ship "for quite some time".
Finally, Blend is specific to building WPF applications, and WPF is a brand new technology, with a steep learning curve, which will not even enjoy a significant degree of Visual Studio designer support until the release of Visual Studio 2008 at the end of this year. Some corporate managers may feel it is safer to hold off on any commitment to Blend, or designers, or possibly even WPF, until things sort themselves out. I disagree, in cases where a change in technology clearly represents a major advance that will need to be adopted eventually anyway, and I'll have more about THAT attitude in a future post. A beta of Blend 2.0 is already available and, as the tool continues to mature, and the practical issues I've mentioned are resolved, we can expect Blend-based development to become the standard for a significant percentage of Windows dev. projects in the not-too-distant future. I recently decided to check out an online course in Blend at the local junior college, and the text we are using, "Microsoft Expression Blend Bible", seems to be a pretty good introduction to its features
A few hours of playing around with one of the earliest CTP builds of "Sparkle" (now, "Blend") got me very excited at its potential to address some familiar UI design PROCESS issues...
1) Our client's team of marketing product managers had flown in from corporate headquarters for the annual project kickoff meeting. After reminding us that we had to deliver in ten months in order to meet Christmas store deadlines, they proudly announced that it had been decided that EVERYTHING in this year's version of the product would be designed around the concept of a "concierge". If the user wanted some information, they would consult the "concierge". We said, "Fine," would this take the form of some kind of animated assistant, or were they envisioning a special screen that the user would navigate to in the application? Would the feature involve recorded or synthesized speech? Would it have an online component?" We were told that focus groups were already hard at work, considering such questions as "If you were to invite the concierge to dinner, what would he look like?". Four months later, a group of us flew East to participate in formal usability lab studies on the product shell, and I took aside one of the PMs, to try to pin down this still-undefined concept of "concierge". He told me the design consultants were still busy, finalizing the product color scheme, and he was shocked when I told him they could safely delay a decision on the final RGB color values almost to the end of the project (this was MFC, so I simply #defined place-holder color codes in a header file), whereas implementing the "concierge" might require significant effort. Six months later, we shipped a product (to good reviews) whose only reference to "Concierge" consisted of a single modal dialog.
2) In preparation for a major revamp of the company's flagship product UI, myself and several other developers in US and German engineering departments spent several months separately implementing a number of interactive mockups using Windows Forms, while I used PhotoShop to generate over 200 additional static "screen shots" of UI alternatives that would have been too time consuming to code...
3) I've almost always had a major role in defining the UI of the products I've worked on, writing many custom UI elements (in Windows Forms and, previously, with MFC), and working with end users or user proxies (ie: Marketing) to design the overall UI experience. This is one of my favorite parts of the development process, and I've read most of the popular design references (Cooper, Krug, Nielsen, Spolsky, Tufte...), so I've always been surprised at the number of developers with good technical skills who seem to have no affinity for, or interest in, user interface design. In some cases, the developers simply seem unable to envision how complex and unintuitive their proposed application workflow will appear to the target (non-technical) end user. In other cases, it's pretty clear that they consider any amount of attention to aesthetics or design to be completely irrelevant to the task of engineering an application, even to the extent of failing to size and line up controls properly, or to follow basic Windows UI design guidelines.
For those who have been busy with existing projects, and may still be unaware, Microsoft's Expression Blend is a UI layout tool (and much more) for WPF applications, that enables designers to generate the actual production UI for an application themselves, rather than having to settle for the developers' distortion of some carefully-rendered initial design. Because they share the same project file format, developers can load a Blend project in Visual Studio to add business logic to the designer's work, and designers can load a Visual Studio project in Blend to professionally "style" basic control layouts stubbed out by the developers. Blend's capabilities extend far beyond mere layout however. Without leaving the design environment, designers can bind controls to XML data, or to class instances defined in code-behind files, or bind property values between controls. Designers can hook up events, create and trigger animations (defined in WPF as any change in the UI over time, including the appearance and location of controls -- as opposed to the more traditional definition, which was specific to image animation), completely restyle the appearance of system controls, or implement entirely new controls.
Sound a little too good to be true? A little skeptical that your same company management which ships products featuring artwork created by developers using MS Paint will suddenly approve funds to hire "interaction design professionals" -- and just where do you find these strange, half-programmer, half-artist, creatures anyway? More about all that in my next post.
|